You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Nicolas Franck <Ni...@UGent.be> on 2018/07/26 12:04:28 UTC
query other solr collection from within a solr plugin
I'm writing a solr plugin in java that has to query another solr collection to gather
information. What is the best way to do this?
For now I'm just using a SolrClient ( CloudSolrClient ), but has several disadvantages:
* you have to extract from core metadata where your server resides, and setup your SolrClient accordingly.
* you are just knocking at the same door
* search has to go over http for the same core.
Is there a better way? Are there any examples?
Thanks in advance
Nicolas Franck
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: query other solr collection from within a solr plugin
Posted by Nicolas Franck <Ni...@UGent.be>.
thanks, I'll have a look!
On 26 Jul 2018, at 14:28, Mikhail Khludnev <mk...@apache.org>> wrote:
[subquery] calls remote cloud collections if collection parameter (which is somewhat not well known, documented) is supplied
https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/response/transform/SubQueryAugmenterFactory.java#L334
On Thu, Jul 26, 2018 at 3:05 PM Nicolas Franck <Ni...@ugent.be>> wrote:
I'm writing a solr plugin in java that has to query another solr collection to gather
information. What is the best way to do this?
For now I'm just using a SolrClient ( CloudSolrClient ), but has several disadvantages:
* you have to extract from core metadata where your server resides, and setup your SolrClient accordingly.
* you are just knocking at the same door
* search has to go over http for the same core.
Is there a better way? Are there any examples?
Thanks in advance
Nicolas Franck
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org<ma...@lucene.apache.org>
For additional commands, e-mail: dev-help@lucene.apache.org<ma...@lucene.apache.org>
--
Sincerely yours
Mikhail Khludnev
Re: query other solr collection from within a solr plugin
Posted by Nicolas Franck <Ni...@UGent.be>.
Alright, as I expected.
Thanks for all your help!
> On 1 Aug 2018, at 16:22, Shawn Heisey <ap...@elyograg.org> wrote:
>
> On 8/1/2018 12:59 AM, Nicolas Franck wrote:
>> @Mikhail Khludnev: thanks for your response
>>
>> You mean something like this (source collection is "collection1", and I want to query "collection2"):
>>
>> SolrClient solrClient = new EmbeddedSolrServer(req.getCore());
>> ModifiableSolrParams newParams = new ModifiableSolrParams();
>> newParams.add("collection","collection2");
>> SolrDocumentList docs = solrClient.getById(ids, newParams);
>>
>> which is basically the same as:
>>
>> http://localhost:8983/collection1_shard1_replica_n1/select?collection=collection2&&qt=/get&ids=myid
>>
>
> An instance of EmbeddedSolrServer has no http access. Also, it can't do SolrCloud -- because SolrCloud requires http. EmbeddedSolrServer is a complete Solr server running in standalone (not cloud) mode, without http access.
>
> If you're doing this code within a Solr plugin, then you can't start an EmbeddedSolrServer on one of the cores from the Solr install. Any cores you try to use will already be open, so the embedded server will not be able to open them. Since you're already running inside a Solr server, there's no reason to start *another* Solr server.
>
> I don't have any other ideas for you. I've written a couple of update processors for Solr, but nothing that does queries.
>
>> but in both cases, I get this error:
>>
>> org.apache.solr.common.SolrException: Can't find shard 'collection1_shard1' at org.apache.solr.handler.component.RealTimeGetComponent.sliceToShards(RealTimeGetComponent.java:897) at
>>
>>
>> apparently it forgets its own cloud information? What am I missing here?
>
> As already mentioned, EmbeddedSolrServer can't do SolrCloud.
>
> Thanks,
> Shawn
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: query other solr collection from within a solr plugin
Posted by Shawn Heisey <ap...@elyograg.org>.
On 8/1/2018 12:59 AM, Nicolas Franck wrote:
> @Mikhail Khludnev: thanks for your response
>
> You mean something like this (source collection is "collection1", and
> I want to query "collection2"):
>
> SolrClient solrClient = new EmbeddedSolrServer(req.getCore());
> ModifiableSolrParams newParams = new ModifiableSolrParams();
> newParams.add("collection","collection2");
> SolrDocumentList docs = solrClient.getById(ids, newParams);
>
> which is basically the same as:
>
> http://localhost:8983/collection1_shard1_replica_n1/select?collection=collection2&&qt=/get&ids=myid
>
An instance of EmbeddedSolrServer has no http access. Also, it can't do
SolrCloud -- because SolrCloud requires http. EmbeddedSolrServer is a
complete Solr server running in standalone (not cloud) mode, without
http access.
If you're doing this code within a Solr plugin, then you can't start an
EmbeddedSolrServer on one of the cores from the Solr install. Any cores
you try to use will already be open, so the embedded server will not be
able to open them. Since you're already running inside a Solr server,
there's no reason to start *another* Solr server.
I don't have any other ideas for you. I've written a couple of update
processors for Solr, but nothing that does queries.
> but in both cases, I get this error:
>
> org.apache.solr.common.SolrException: Can't find shard
> 'collection1_shard1' at
> org.apache.solr.handler.component.RealTimeGetComponent.sliceToShards(RealTimeGetComponent.java:897)
> at
>
>
> apparently it forgets its own cloud information? What am I missing here?
As already mentioned, EmbeddedSolrServer can't do SolrCloud.
Thanks,
Shawn
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: query other solr collection from within a solr plugin
Posted by Mikhail Khludnev <mk...@apache.org>.
Well. I certainly haven't invoked RTG by this way. You might need to debug
and understand what's going on. I briefly looked through code, maybe
supplying shards param is an option (see
RealTimeGetComponent.createSubRequests(ResponseBuilder)), but it implies
"gathering cloud info".
On Wed, Aug 1, 2018 at 9:59 AM Nicolas Franck <Ni...@ugent.be>
wrote:
> @Mikhail Khludnev: thanks for your response
>
> You mean something like this (source collection is "collection1", and I
> want to query "collection2"):
>
> SolrClient solrClient = new EmbeddedSolrServer(req.getCore());
> ModifiableSolrParams newParams = new ModifiableSolrParams();
> newParams.add("collection","collection2");
> SolrDocumentList docs = solrClient.getById(ids, newParams);
>
> which is basically the same as:
>
>
> http://localhost:8983/collection1_shard1_replica_n1/select?collection=collection2&&qt=/get&ids=myid
>
> but in both cases, I get this error:
>
> org.apache.solr.common.SolrException: Can't find shard
> 'collection1_shard1' at
> org.apache.solr.handler.component.RealTimeGetComponent.sliceToShards(RealTimeGetComponent.java:897)
> at
>
>
> apparently it forgets its own cloud information? What am I missing here?
>
> BTW: is it necessary to recreate a client like this on every request,
> giving the nature of a solrcloud where the shards can change?
>
> On 27 Jul 2018, at 16:04, Mikhail Khludnev <mk...@apache.org> wrote:
>
> Sure, it's up to you. But if for matter of fact, it EmbeddedSolrServer
> request has "collection" param, it pulls shards from zk and executed
> distributed request.
> see
> http://people.apache.org/~mkhl/searchable-solr-guide-7-3/transforming-result-documents.html#cores-and-collections-in-solrcloud
>
> https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java#L323
>
> On Fri, Jul 27, 2018 at 11:48 AM Nicolas Franck <Ni...@ugent.be>
> wrote:
>
>> From what I've seen now, it seems that you can only directly connect to a
>> specific core on your own node,
>> right? Should have expected that: it is local ;-)
>>
>> Then I'll stick to the old solution that worked after all.
>>
>> Thanks for all the advice
>>
>> On 26 Jul 2018, at 14:28, Mikhail Khludnev <mk...@apache.org> wrote:
>>
>> [subquery] calls remote cloud collections if collection parameter (which
>> is somewhat not well known, documented) is supplied
>>
>> https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/response/transform/SubQueryAugmenterFactory.java#L334
>>
>>
>> On Thu, Jul 26, 2018 at 3:05 PM Nicolas Franck <Ni...@ugent.be>
>> wrote:
>>
>>> I'm writing a solr plugin in java that has to query another solr
>>> collection to gather
>>> information. What is the best way to do this?
>>>
>>> For now I'm just using a SolrClient ( CloudSolrClient ), but has several
>>> disadvantages:
>>>
>>> * you have to extract from core metadata where your server resides, and
>>> setup your SolrClient accordingly.
>>> * you are just knocking at the same door
>>> * search has to go over http for the same core.
>>>
>>> Is there a better way? Are there any examples?
>>>
>>> Thanks in advance
>>>
>>> Nicolas Franck
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>>
>>
>> --
>> Sincerely yours
>> Mikhail Khludnev
>>
>>
>>
>
> --
> Sincerely yours
> Mikhail Khludnev
>
>
>
--
Sincerely yours
Mikhail Khludnev
Re: query other solr collection from within a solr plugin
Posted by Nicolas Franck <Ni...@UGent.be>.
@Mikhail Khludnev: thanks for your response
You mean something like this (source collection is "collection1", and I want to query "collection2"):
SolrClient solrClient = new EmbeddedSolrServer(req.getCore());
ModifiableSolrParams newParams = new ModifiableSolrParams();
newParams.add("collection","collection2");
SolrDocumentList docs = solrClient.getById(ids, newParams);
which is basically the same as:
http://localhost:8983/collection1_shard1_replica_n1/select?collection=collection2&&qt=/get&ids=myid
but in both cases, I get this error:
org.apache.solr.common.SolrException: Can't find shard 'collection1_shard1' at org.apache.solr.handler.component.RealTimeGetComponent.sliceToShards(RealTimeGetComponent.java:897) at
apparently it forgets its own cloud information? What am I missing here?
BTW: is it necessary to recreate a client like this on every request, giving the nature of a solrcloud where the shards can change?
On 27 Jul 2018, at 16:04, Mikhail Khludnev <mk...@apache.org>> wrote:
Sure, it's up to you. But if for matter of fact, it EmbeddedSolrServer request has "collection" param, it pulls shards from zk and executed distributed request.
see http://people.apache.org/~mkhl/searchable-solr-guide-7-3/transforming-result-documents.html#cores-and-collections-in-solrcloud
https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java#L323
On Fri, Jul 27, 2018 at 11:48 AM Nicolas Franck <Ni...@ugent.be>> wrote:
From what I've seen now, it seems that you can only directly connect to a specific core on your own node,
right? Should have expected that: it is local ;-)
Then I'll stick to the old solution that worked after all.
Thanks for all the advice
On 26 Jul 2018, at 14:28, Mikhail Khludnev <mk...@apache.org>> wrote:
[subquery] calls remote cloud collections if collection parameter (which is somewhat not well known, documented) is supplied
https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/response/transform/SubQueryAugmenterFactory.java#L334
On Thu, Jul 26, 2018 at 3:05 PM Nicolas Franck <Ni...@ugent.be>> wrote:
I'm writing a solr plugin in java that has to query another solr collection to gather
information. What is the best way to do this?
For now I'm just using a SolrClient ( CloudSolrClient ), but has several disadvantages:
* you have to extract from core metadata where your server resides, and setup your SolrClient accordingly.
* you are just knocking at the same door
* search has to go over http for the same core.
Is there a better way? Are there any examples?
Thanks in advance
Nicolas Franck
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org<ma...@lucene.apache.org>
For additional commands, e-mail: dev-help@lucene.apache.org<ma...@lucene.apache.org>
--
Sincerely yours
Mikhail Khludnev
--
Sincerely yours
Mikhail Khludnev
Re: query other solr collection from within a solr plugin
Posted by Mikhail Khludnev <mk...@apache.org>.
Sure, it's up to you. But if for matter of fact, it EmbeddedSolrServer
request has "collection" param, it pulls shards from zk and executed
distributed request.
see
http://people.apache.org/~mkhl/searchable-solr-guide-7-3/transforming-result-documents.html#cores-and-collections-in-solrcloud
https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java#L323
On Fri, Jul 27, 2018 at 11:48 AM Nicolas Franck <Ni...@ugent.be>
wrote:
> From what I've seen now, it seems that you can only directly connect to a
> specific core on your own node,
> right? Should have expected that: it is local ;-)
>
> Then I'll stick to the old solution that worked after all.
>
> Thanks for all the advice
>
> On 26 Jul 2018, at 14:28, Mikhail Khludnev <mk...@apache.org> wrote:
>
> [subquery] calls remote cloud collections if collection parameter (which
> is somewhat not well known, documented) is supplied
>
> https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/response/transform/SubQueryAugmenterFactory.java#L334
>
>
> On Thu, Jul 26, 2018 at 3:05 PM Nicolas Franck <Ni...@ugent.be>
> wrote:
>
>> I'm writing a solr plugin in java that has to query another solr
>> collection to gather
>> information. What is the best way to do this?
>>
>> For now I'm just using a SolrClient ( CloudSolrClient ), but has several
>> disadvantages:
>>
>> * you have to extract from core metadata where your server resides, and
>> setup your SolrClient accordingly.
>> * you are just knocking at the same door
>> * search has to go over http for the same core.
>>
>> Is there a better way? Are there any examples?
>>
>> Thanks in advance
>>
>> Nicolas Franck
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
>
> --
> Sincerely yours
> Mikhail Khludnev
>
>
>
--
Sincerely yours
Mikhail Khludnev
Re: query other solr collection from within a solr plugin
Posted by Nicolas Franck <Ni...@UGent.be>.
From what I've seen now, it seems that you can only directly connect to a specific core on your own node,
right? Should have expected that: it is local ;-)
Then I'll stick to the old solution that worked after all.
Thanks for all the advice
On 26 Jul 2018, at 14:28, Mikhail Khludnev <mk...@apache.org>> wrote:
[subquery] calls remote cloud collections if collection parameter (which is somewhat not well known, documented) is supplied
https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/response/transform/SubQueryAugmenterFactory.java#L334
On Thu, Jul 26, 2018 at 3:05 PM Nicolas Franck <Ni...@ugent.be>> wrote:
I'm writing a solr plugin in java that has to query another solr collection to gather
information. What is the best way to do this?
For now I'm just using a SolrClient ( CloudSolrClient ), but has several disadvantages:
* you have to extract from core metadata where your server resides, and setup your SolrClient accordingly.
* you are just knocking at the same door
* search has to go over http for the same core.
Is there a better way? Are there any examples?
Thanks in advance
Nicolas Franck
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org<ma...@lucene.apache.org>
For additional commands, e-mail: dev-help@lucene.apache.org<ma...@lucene.apache.org>
--
Sincerely yours
Mikhail Khludnev
Re: query other solr collection from within a solr plugin
Posted by Mikhail Khludnev <mk...@apache.org>.
[subquery] calls remote cloud collections if collection parameter (which is
somewhat not well known, documented) is supplied
https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/response/transform/SubQueryAugmenterFactory.java#L334
On Thu, Jul 26, 2018 at 3:05 PM Nicolas Franck <Ni...@ugent.be>
wrote:
> I'm writing a solr plugin in java that has to query another solr
> collection to gather
> information. What is the best way to do this?
>
> For now I'm just using a SolrClient ( CloudSolrClient ), but has several
> disadvantages:
>
> * you have to extract from core metadata where your server resides, and
> setup your SolrClient accordingly.
> * you are just knocking at the same door
> * search has to go over http for the same core.
>
> Is there a better way? Are there any examples?
>
> Thanks in advance
>
> Nicolas Franck
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
--
Sincerely yours
Mikhail Khludnev
Re: query other solr collection from within a solr plugin
Posted by Nicolas Franck <Ni...@UGent.be>.
thanks, I'll have a look!
> On 26 Jul 2018, at 14:15, Upayavira <uv...@odoko.co.uk> wrote:
>
> Go look in the source for the Join query parser. It does this.
>
> Upayavira
>
> On Thu, 26 Jul 2018, at 1:04 PM, Nicolas Franck wrote:
>> I'm writing a solr plugin in java that has to query another solr
>> collection to gather
>> information. What is the best way to do this?
>>
>> For now I'm just using a SolrClient ( CloudSolrClient ), but has several
>> disadvantages:
>>
>> * you have to extract from core metadata where your server resides, and
>> setup your SolrClient accordingly.
>> * you are just knocking at the same door
>> * search has to go over http for the same core.
>>
>> Is there a better way? Are there any examples?
>>
>> Thanks in advance
>>
>> Nicolas Franck
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: query other solr collection from within a solr plugin
Posted by Upayavira <uv...@odoko.co.uk>.
Go look in the source for the Join query parser. It does this.
Upayavira
On Thu, 26 Jul 2018, at 1:04 PM, Nicolas Franck wrote:
> I'm writing a solr plugin in java that has to query another solr
> collection to gather
> information. What is the best way to do this?
>
> For now I'm just using a SolrClient ( CloudSolrClient ), but has several
> disadvantages:
>
> * you have to extract from core metadata where your server resides, and
> setup your SolrClient accordingly.
> * you are just knocking at the same door
> * search has to go over http for the same core.
>
> Is there a better way? Are there any examples?
>
> Thanks in advance
>
> Nicolas Franck
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org