You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by tuxedomoon <da...@yahoo.com> on 2015/03/11 19:23:33 UTC
Can a single SolrServer instance update multiple collections?
I have a SolrJ application that reads from a Redis queue and updates
different collections based on the message content. New collections are
added without my knowledge, so I am creating SolrServer objects on the fly
as follows:
def solrHost = "http://myhost/solr/" (defined at startup)
def solrTarget = solrHost + collectionName
SolrServer solrServer = new CommonsHttpSolrServer(solrTarget)
updateResponse = solrServer.add(solrdoc)
This does work but obviously creates a new CommonsHttpSolrServer instance
for each message. I assume GC will eliminate these but is there a way to do
this with a single SolrServer object?
The SOLR host is version 3.5 and I am using the 3.5 jars for my application
(not sure if that is necessary).
--
View this message in context: http://lucene.472066.n3.nabble.com/Can-a-single-SolrServer-instance-update-multiple-collections-tp4192480.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Can a single SolrServer instance update multiple collections?
Posted by Shawn Heisey <ap...@elyograg.org>.
On 3/11/2015 4:28 PM, Shawn Heisey wrote:
> When I have some time to actually work on the code, I'm going to write
> it using 4.x classes because that's what I have immediate access to,
> but if you do 5.x, SolrServer becomes SolrClient, and HttpSolrServer
> becomes HttpSolrClient.
At the URL below is the code I came up with. It shows how to do an add,
a commit, and a query where the Solr core (collection) is specified as
part of the request, rather than the server connection:
http://apaste.info/lRi
I did test this code successfully, although there was one difference in
that code (/update instead of /update/javabin) because my dev Solr
server is running 4.9.1, not 3.5. The code I've shared uses SolrJ 4.x,
but is tailored to a server running 3.x with a typical 3.x config. I
hope this code will work as-is ... and if it doesn't, that it will be
easy for you to figure out what I did wrong.
If you want to figure out how to use SolrRequest to implement a query
with a specific handler path, you could probably implement all of this
in SolrJ 3.5, where SolrQuery#setRequestHandler does not exist. I'm
sure that if you look at the SolrQuery class and the
CommonsHttpSolrServer#query method from the 3.5 source code, you could
piece together how to do this.
It might be a good idea to abstract these procedures for add, commit,
and query into your own local methods that include the collection
parameter. If you need it, you can also implement
UpdateRequest.ACTION.OPTIMIZE in a similar manner to the way that I used
UpdateRequest.ACTION.COMMIT.
See the following issue for the recent work that will go into a new 5.x
version (probably 5.1), which adds the capability you are seeking
directly to HttpSolrClient, implementing abstract methods from SolrClient:
https://issues.apache.org/jira/browse/SOLR-7201
Thanks,
Shawn
Re: Can a single SolrServer instance update multiple collections?
Posted by tuxedomoon <da...@yahoo.com>.
@Shawn
I'm getting the Bad Request again, with the original code snippet I posted,
it appears to be an 'illegal' string field.
SOLR log
-----------------------------------------------------------------------------
INFO:
{add=[mgid:arc:content:jokers.com:694d5bf8-ecfd-11e0-aca6-0026b9414f30]} 0 7
Mar 12, 2015 12:15:09 AM org.apache.solr.common.SolrException log
SEVERE: org.apache.solr.common.SolrException: ERROR:
[doc=mgid:arc:content:jokers.com:694d5bf8-ecfd-11e0-aca6-0026b9414f30]
multiple values encountered for non multiValued field image_url_s:
[mgid:file:gsp:movie-assets:/movie-assets/cc/images/shows/miami-beach/episode-thumbnails/specials/iamstupid-the-movie_4x3.jpg,
mgid:file:gsp:movie-assets:/movie-assets/cc/images/shows/miami-beach/episode-thumbnails/specials/iamstupid-the-movie_4x3.jpg]
at
org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:246)
at
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:60)
at
org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:115)
at
org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:158)
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:79)
SolrJ Log shows the doc being sent (this is the offending field only)
----------------------------------------------------------------------------------------
<field name="image_url_s"></field>
I will investigate on the feeds side, the existing SolrJ code is not the
culprit. But I'd still like a more elegant solution. If a SolrJ 5 client
can talk to a 3.5 host I'm willing to go there. I know I'm not the only one
who would like to address collections on the fly.
thx
Dan
--
View this message in context: http://lucene.472066.n3.nabble.com/Can-a-single-SolrServer-instance-update-multiple-collections-tp4192480p4192545.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Can a single SolrServer instance update multiple collections?
Posted by Shawn Heisey <el...@elyograg.org>.
On 3/11/2015 3:35 PM, tuxedomoon wrote:
> I can definitely upgrade to SolrJ 4.x and would prefer that so as to target
> 4.x cores as well. I'm already on Java 7.
>
> One attempt I made was this
>
> UpdateRequest updateRequest = new UpdateRequest();
> updateRequest.setParam("collection", collectionName);
> updateRequest.setMethod(SolrRequest.METHOD.POST);
> updateRequest.add(solrdoc);
> UpdateResponse updateResponse = updateRequest.process(solrServer);
>
> but I kept getting Bad Request which I suspect was a SOLR/SolrJ version
> conflict.
>
> I'm all ears!
Can you share the full stacktrace? If you can't see it on the client,
grab it from the server log.
The "collection" request parameter is only useful if you're running
SolrCloud. The 3.x version and 4.x/5.x in non-cloud mode should ignore it.
UpdateRequest objects are created by default with a POST method, you
don't need to include that.
When I have some time to actually work on the code, I'm going to write
it using 4.x classes because that's what I have immediate access to, but
if you do 5.x, SolrServer becomes SolrClient, and HttpSolrServer becomes
HttpSolrClient. I think everything else will be the same. If I'm wrong
about that, it very likely will not be very hard to fix.
Thanks,
Shawn
Re: Can a single SolrServer instance update multiple collections?
Posted by tuxedomoon <da...@yahoo.com>.
@Shawn,
I can definitely upgrade to SolrJ 4.x and would prefer that so as to target
4.x cores as well. I'm already on Java 7.
One attempt I made was this
UpdateRequest updateRequest = new UpdateRequest();
updateRequest.setParam("collection", collectionName);
updateRequest.setMethod(SolrRequest.METHOD.POST);
updateRequest.add(solrdoc);
UpdateResponse updateResponse = updateRequest.process(solrServer);
but I kept getting Bad Request which I suspect was a SOLR/SolrJ version
conflict.
I'm all ears!
Dan
--
View this message in context: http://lucene.472066.n3.nabble.com/Can-a-single-SolrServer-instance-update-multiple-collections-tp4192480p4192520.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Can a single SolrServer instance update multiple collections?
Posted by Shawn Heisey <ap...@elyograg.org>.
On 3/11/2015 12:23 PM, tuxedomoon wrote:
> I have a SolrJ application that reads from a Redis queue and updates
> different collections based on the message content. New collections are
> added without my knowledge, so I am creating SolrServer objects on the fly
> as follows:
>
> def solrHost = "http://myhost/solr/" (defined at startup)
>
> def solrTarget = solrHost + collectionName
> SolrServer solrServer = new CommonsHttpSolrServer(solrTarget)
> updateResponse = solrServer.add(solrdoc)
>
>
> This does work but obviously creates a new CommonsHttpSolrServer instance
> for each message. I assume GC will eliminate these but is there a way to do
> this with a single SolrServer object?
>
> The SOLR host is version 3.5 and I am using the 3.5 jars for my application
> (not sure if that is necessary).
What you want to accomplish should be possible, with some attention to
how SolrJ code is used.
We won't talk about SolrCloud, since you're not running Solr 4.x or
5.0. Upgrading the server side is generally more involved than
upgrading the client side, and switching to SolrCloud can be a fairly
major conceptual leap.
To do what I'm thinking about, you will need to ugprade SolrJ. When
SolrCloud is not involved, cross-version compatibility between Solr and
SolrJ is pretty good, although there can be some hiccups when crossing
the 3.x/4.x barrier relating to the update handlers. Those hiccups are
normally easy to fix, but they are something you need to be aware of.
Once you've decided on whether you're upgrading Solr and which version
of SolrJ you will upgrade to, we can get down to the actual Java code
you'll need. Note that recent 4.x and 5.0 versions require Java 7, so
if you're still on Java 6, you'll be limited to version 4.7.2.
It might even be possible to do this with SolrJ 3.5, but I am already
pretty familiar with how you can do it using new features in 4.x, and
since you're going to need to change the source code anyway, you might
as well take advantage of more modern client functionality that will
make the code easier to understand.
Just FYI, there are changes coming (currently planned for SolrJ 5.1)
that will make this VERY easy.
Thanks,
Shawn