You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by John Jackson <jo...@gmail.com> on 2023/08/08 05:47:05 UTC
Apache SOLR from Version 8.6.0 - Joining between Multiple Collections and Multiple Shards in each collection
Please consider SOLR version greater than 8.6.0 for this query. There are
many questions regarding this issue but all are before version 8.6.0 and at
that time SOLR was not supporting Join between multiple shards and multiple
collection.
<https://stackoverflow.com/posts/76578595/timeline>
Now refer below links, it mention that SOLR supports Join between multiple
collection and multiple shards. Refer Solr 8.6.0 Release Highlights: in
this news. https://solr.apache.org/news.html
Also, SOLR provides guidance how to configure and achieve the same. Refer
Join Query Parser Definition in solrconfig.xml in below link
https://solr.apache.org/guide/8_6/other-parsers.html
I have also gone through below link for configuration
https://kmwllc.com/index.php/2020/07/15/the-cross-collection-join-query/
Please review below configuration for local and other collection, schema
structure and join query
*Local Collection Name - loc_1 collection*
*Other Collection Name - oth_1 collection*
*loc_id (loc_1 collection) is a join key with oth_id (other collection)*
1. loc_1 Collection (local collection) -
schema.xml
<field name="loc_id" type="int" indexed="true" stored="true"/>
<field name="loc_auth" type="int" indexed="true" stored="true"/>
solrconfig.xml file
<queryParser name="join" class="org.apache.solr.search.JoinQParserPlugin">
<str name="routerField">loc_id</str>
</queryParser>
1. oth_1 Collection (remote collection) -
schema.xml
<field name="oth_id" type="int" indexed="true" stored="true" />
<field name="oth_auth" type="int" indexed="true" stored="true"
multiValued="true" />
solrconfig.xml file
<cache name="hash_oth_id"
class="solr.CaffeineCache"
size="128"
initialSize="0"
regenerator="solr.NoOpRegenerator"/>
I can successfully execute below query in Local environment with single
Shard only and retrieve data from both collection. I am passing in local
collection with fq.
loc_auth:123 OR {!join from=oth_id to=loc_id fromIndex=oth_1}oth_auth:123
But, above query fails with Multiple Shard on QA environment so, with help
of above links, I have modify solrconfig.xml file as per above. Then
execute the same query by just adding method=crossCollection. Please refer
below join query for cross collection which I am passing in fq.
loc_auth:123 OR {!join method=crossCollection from=oth_id to=loc_id
fromIndex=oth_1}oth_auth:123
It throws error as given here. Please any one can help if I am missing some
configuration or need any changes in Shared configuration or need to
re-create the shared structure.
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader">
<bool name="zkConnected">true</bool>
<int name="status">500</int>
<int name="QTime">8</int>
</lst>
<lst name="error">
<lst name="metadata">
<str name="error-class">org.apache.solr.common.SolrException</str>
<str name="root-error-class">java.lang.RuntimeException</str>
</lst>
<str name="msg">java.io.IOException:
java.util.concurrent.ExecutionException: java.io.IOException: params
method=crossCollection&q=oth_auth:123&fl=oth_id&sort=oth_id+asc&qt=/export&wt=javabin&distrib=false</str>
<str name="trace">org.apache.solr.common.SolrException:
java.io.IOException: java.util.concurrent.ExecutionException:
java.io.IOException: params
method=crossCollection&q=oth_auth:123&fl=oth_id&sort=oth_id+asc&qt=/export&wt=javabin&distrib=false
at org.apache.solr.search.join.CrossCollectionJoinQuery$CrossCollectionJoinQueryWeight.getDocSet(CrossCollectionJoinQuery.java:300)
at org.apache.solr.search.join.CrossCollectionJoinQuery$CrossCollectionJoinQueryWeight.scorer(CrossCollectionJoinQuery.java:311)
at org.apache.lucene.search.Weight.scorerSupplier(Weight.java:148)
at org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:379)
at org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:344)
at org.apache.solr.search.SolrIndexSearcher$FilterImpl$FilterSet.iterator(SolrIndexSearcher.java:2366)
at org.apache.solr.search.Filter$1.scorer(Filter.java:133)
at org.apache.lucene.search.Weight.bulkScorer(Weight.java:182)
at org.apache.lucene.search.ConstantScoreQuery$1.bulkScorer(ConstantScoreQuery.java:121)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:656)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:443)
at org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:211)
at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1595)
at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1412)
at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:596)
at org.apache.solr.handler.component.QueryComponent.doProcessUngroupedSearch(QueryComponent.java:1500)
at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:390)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:369)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.ja
Re: Apache SOLR from Version 8.6.0 - Joining between Multiple Collections and Multiple Shards in each collection
Posted by Mikhail Khludnev <mk...@apache.org>.
Hello, John.
This exception occurs in the caller side: a node executing join requests
streaming expor from another node.
The root cause of the problem might be found in the callee node logs.
On Tue, Aug 8, 2023 at 8:48 AM John Jackson <jo...@gmail.com> wrote:
> Please consider SOLR version greater than 8.6.0 for this query. There are
> many questions regarding this issue but all are before version 8.6.0 and at
> that time SOLR was not supporting Join between multiple shards and multiple
> collection.
>
> <https://stackoverflow.com/posts/76578595/timeline>
>
> Now refer below links, it mention that SOLR supports Join between multiple
> collection and multiple shards. Refer Solr 8.6.0 Release Highlights: in
> this news. https://solr.apache.org/news.html
>
> Also, SOLR provides guidance how to configure and achieve the same. Refer
> Join Query Parser Definition in solrconfig.xml in below link
> https://solr.apache.org/guide/8_6/other-parsers.html
>
> I have also gone through below link for configuration
> https://kmwllc.com/index.php/2020/07/15/the-cross-collection-join-query/
>
> Please review below configuration for local and other collection, schema
> structure and join query
>
> *Local Collection Name - loc_1 collection*
>
> *Other Collection Name - oth_1 collection*
>
> *loc_id (loc_1 collection) is a join key with oth_id (other collection)*
>
> 1. loc_1 Collection (local collection) -
>
> schema.xml
>
> <field name="loc_id" type="int" indexed="true" stored="true"/>
> <field name="loc_auth" type="int" indexed="true" stored="true"/>
>
> solrconfig.xml file
>
> <queryParser name="join" class="org.apache.solr.search.JoinQParserPlugin">
> <str name="routerField">loc_id</str>
> </queryParser>
>
>
> 1. oth_1 Collection (remote collection) -
>
> schema.xml
>
> <field name="oth_id" type="int" indexed="true" stored="true" />
> <field name="oth_auth" type="int" indexed="true" stored="true"
> multiValued="true" />
>
> solrconfig.xml file
>
> <cache name="hash_oth_id"
> class="solr.CaffeineCache"
> size="128"
> initialSize="0"
> regenerator="solr.NoOpRegenerator"/>
>
> I can successfully execute below query in Local environment with single
> Shard only and retrieve data from both collection. I am passing in local
> collection with fq.
>
> loc_auth:123 OR {!join from=oth_id to=loc_id fromIndex=oth_1}oth_auth:123
>
> But, above query fails with Multiple Shard on QA environment so, with help
> of above links, I have modify solrconfig.xml file as per above. Then
> execute the same query by just adding method=crossCollection. Please refer
> below join query for cross collection which I am passing in fq.
>
> loc_auth:123 OR {!join method=crossCollection from=oth_id to=loc_id
> fromIndex=oth_1}oth_auth:123
>
> It throws error as given here. Please any one can help if I am missing some
> configuration or need any changes in Shared configuration or need to
> re-create the shared structure.
>
> <?xml version="1.0" encoding="UTF-8"?>
> <response>
>
> <lst name="responseHeader">
> <bool name="zkConnected">true</bool>
> <int name="status">500</int>
> <int name="QTime">8</int>
> </lst>
> <lst name="error">
> <lst name="metadata">
> <str name="error-class">org.apache.solr.common.SolrException</str>
> <str name="root-error-class">java.lang.RuntimeException</str>
> </lst>
> <str name="msg">java.io.IOException:
> java.util.concurrent.ExecutionException: java.io.IOException: params
>
> method=crossCollection&q=oth_auth:123&fl=oth_id&sort=oth_id+asc&qt=/export&wt=javabin&distrib=false</str>
> <str name="trace">org.apache.solr.common.SolrException:
> java.io.IOException: java.util.concurrent.ExecutionException:
> java.io.IOException: params
>
> method=crossCollection&q=oth_auth:123&fl=oth_id&sort=oth_id+asc&qt=/export&wt=javabin&distrib=false
> at
> org.apache.solr.search.join.CrossCollectionJoinQuery$CrossCollectionJoinQueryWeight.getDocSet(CrossCollectionJoinQuery.java:300)
> at
> org.apache.solr.search.join.CrossCollectionJoinQuery$CrossCollectionJoinQueryWeight.scorer(CrossCollectionJoinQuery.java:311)
> at org.apache.lucene.search.Weight.scorerSupplier(Weight.java:148)
> at
> org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:379)
> at
> org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:344)
> at
> org.apache.solr.search.SolrIndexSearcher$FilterImpl$FilterSet.iterator(SolrIndexSearcher.java:2366)
> at org.apache.solr.search.Filter$1.scorer(Filter.java:133)
> at org.apache.lucene.search.Weight.bulkScorer(Weight.java:182)
> at
> org.apache.lucene.search.ConstantScoreQuery$1.bulkScorer(ConstantScoreQuery.java:121)
> at
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:656)
> at
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:443)
> at
> org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:211)
> at
> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1595)
> at
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1412)
> at
> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:596)
> at
> org.apache.solr.handler.component.QueryComponent.doProcessUngroupedSearch(QueryComponent.java:1500)
> at
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:390)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:369)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.ja
>
--
Sincerely yours
Mikhail Khludnev