You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Billy Newman <ne...@gmail.com> on 2012/12/13 17:46:32 UTC

Solrj connect to already running solr server

I have deployed the solr.war to my application server.  On deploy I
can see the solr server and my core "general" start up.

I have a timer that fires every so ofter to go out and 'crawl' some
services and index into Solr.  I am using Solrj in my application and
I am having trouble understanding the best way to get data in my
already running Solr Server.

I am trying to use the EmbeddedSolrServer to connect to the "general"
core that is already running:

String solrHomeProperty = System.getProperty("solr.solr.home");
File solrHome = new File(solrHomeProperty);
CoreContainer coreContainer = new CoreContainer(solrHomeProperty);
coreContainer.load(solrHomeProperty, new File(solrHome, "solr.xml"));
SolrServer solrServer = new EmbeddedSolrServer(coreContainer, "general");

The problem here is that I think this is trying to start a new Solr
server, and it collides with the Solr server that is already running,
as I get the following exception:

SEVERE [org.apache.solr.core.CoreContainer] Unable to create core: general
java.nio.channles.OverlappingFileLockException
...

I know that I can also use HttpSolrServer, but I don't really want to
connect Http when I am already in  the application server.

What is the suggested way to connect to an already running solr server
using Solrj.  Am I using EmbeddedSolrServer wrong, or is it expected
behavior for it to try and start a new server?

Thanks in advance!!!

Re: Solrj connect to already running solr server

Posted by Chris Hostetter <ho...@fucit.org>.
: I am trying to use the EmbeddedSolrServer to connect to the "general"
: core that is already running:

that is not going to work.  EmbeddedSolrServer is solely for running solr 
entirely contained inside your application, w/o using a servlet container 
-- it doesn't support any of the HTTP based features of Solr (ie: 
replication, solr cloud, distributed search, admin ui, etc...)

: The problem here is that I think this is trying to start a new Solr

Exactly.

: I know that I can also use HttpSolrServer, but I don't really want to
: connect Http when I am already in  the application server.

An HTTP connection to localhost, using things like keep-alive, should have 
almost negligable overhead compared to using EmbeddedSolrServer -- and 
leaves you open for all of the scalability advantages of distributing solr 
to multiple machines.


-Hoss

Re: How heavy is HttpSolrServer

Posted by Shawn Heisey <so...@elyograg.org>.
On 1/3/2013 3:02 PM, Benjamin, Roy wrote:
> I currently create a new instance of HttpSolrServer for each update. It's convenient when sharding
> over a hundred shards in a heavily threaded updating client.  How heavy is this class?  Would
> it really be worth using a pool (map of pools really) to hold on to previously created instances?

Each Solr core should have *one* HttpSolrServer instance for your entire 
application, and it can be static if necessary.  It is reusable and 
entirely thread-safe.  I don't know how heavy it is, but it does build 
its own HttpClient, and HttpClient in turn creates who knows how many 
additional objects.

It *might* be possible to have one server instance for each host/port 
combination with multiple cores and use query.setRequestHandler to pick 
which core you want, but I have not tested this theory.  I've added it 
to my never-ending TODO list.

A note about using this mailing list: Please don't reply to an existing 
message when you want to create a new thread.  You must create a brand 
new message and address it to solr-user@lucene.apache.org.  When you 
reply to an existing message, message threading headers cause your 
message to appear as part of the original thread, and knowledgeable 
people who are uninterested in the original thread might never see your 
message.  This is even more likely to happen to people who follow this 
mailing list on web-based forums that link to the mailing list.

Thanks,
Shawn


How heavy is HttpSolrServer

Posted by "Benjamin, Roy" <rb...@ebay.com>.
I currently create a new instance of HttpSolrServer for each update. It's convenient when sharding
over a hundred shards in a heavily threaded updating client.  How heavy is this class?  Would 
it really be worth using a pool (map of pools really) to hold on to previously created instances?

Thanks
Roy

Re: Solrj connect to already running solr server

Posted by Per Steffensen <st...@designware.dk>.
Per Steffensen skrev:
> Billy Newman skrev:
>> I have deployed the solr.war to my application server.  On deploy I
>> can see the solr server and my core "general" start up.
>>
>> I have a timer that fires every so ofter to go out and 'crawl' some
>> services and index into Solr.  I am using Solrj in my application and
>> I am having trouble understanding the best way to get data in my
>> already running Solr Server.
>>
>> I am trying to use the EmbeddedSolrServer to connect to the "general"
>> core that is already running:
>>
>> String solrHomeProperty = System.getProperty("solr.solr.home");
>> File solrHome = new File(solrHomeProperty);
>> CoreContainer coreContainer = new CoreContainer(solrHomeProperty);
>> coreContainer.load(solrHomeProperty, new File(solrHome, "solr.xml"));
>> SolrServer solrServer = new EmbeddedSolrServer(coreContainer, 
>> "general");
>>
>> The problem here is that I think this is trying to start a new Solr
>> server, and it collides with the Solr server that is already running,
>> as I get the following exception:
>>
>> SEVERE [org.apache.solr.core.CoreContainer] Unable to create core: 
>> general
>> java.nio.channles.OverlappingFileLockException
>> ...
>>   
> You create a new CoreContainer and it will create and start the 
> SolrCores described in your solr.xml. But they are already running. 
> You dont want to create a new CoreContainer but look up the existing 
> one instead, and use that for your EmbeddedSolrServer. The 
> CoreContainer is available through SolrDispatchFilter.getCores, so 
> basically you need to get hold of the SolrDispatchFilter instance.
>
> You do not say much about where/how your "other code" (the code with 
> you timer job) runs. Is it in the same webapp as Solr (you might have 
> hacked the web.xml of Solr), is it another webapp running side by side 
> with the solr-webapp in the same webcontainer, is it a ejb-app or what 
> is it. Depending on where/how your "other code" runs there are 
> different ways to get hold of the SolrDispatchFilter. You can probably 
> get it through JNDI but in some containers this means that you need to 
> do some JNDI-name-wireing between your apps. There are other "easier" 
> ways depending on where/how your "other code" runs - I believe you 
> want to google for things like ServletContext, getRequestDispatcher, 
> getNamedDispatcher etc.
> You might also have to consider the actual container you are using 
> (for Solr its Jetty out-of-the-box, but you might run on tomcat or 
> something else), even though I believe the specs allow you to do what 
> you want - and if the spec dictates a way, all webcontainers (that 
> want to be certified) have to support it.
>
> Hope it helps you. Else I might get time to help you in a little more 
> concrete way. Have been teaching webapp and ejb-app stuff many years 
> ago, so I might be able to dust off my knowledge about it in order to 
> help you.
Chris mentions that EmbeddedSolrServer might not work even though you 
are able to get hold of the existing CoreContainer. I do not know much 
about EmbeddedSolrServer so I cant argue about that. Dont know the exact 
design gold for EmbeddedSolrServer, but you should be able to get hold 
of the CoreContainer.
>
>> I know that I can also use HttpSolrServer, but I don't really want to
>> connect Http when I am already in  the application server.
>>
>> What is the suggested way to connect to an already running solr server
>> using Solrj.  Am I using EmbeddedSolrServer wrong, or is it expected
>> behavior for it to try and start a new server?
>>
>> Thanks in advance!!!
>>
>>   
>
>


Re: Solrj connect to already running solr server

Posted by Per Steffensen <st...@designware.dk>.
Billy Newman skrev:
> I have deployed the solr.war to my application server.  On deploy I
> can see the solr server and my core "general" start up.
>
> I have a timer that fires every so ofter to go out and 'crawl' some
> services and index into Solr.  I am using Solrj in my application and
> I am having trouble understanding the best way to get data in my
> already running Solr Server.
>
> I am trying to use the EmbeddedSolrServer to connect to the "general"
> core that is already running:
>
> String solrHomeProperty = System.getProperty("solr.solr.home");
> File solrHome = new File(solrHomeProperty);
> CoreContainer coreContainer = new CoreContainer(solrHomeProperty);
> coreContainer.load(solrHomeProperty, new File(solrHome, "solr.xml"));
> SolrServer solrServer = new EmbeddedSolrServer(coreContainer, "general");
>
> The problem here is that I think this is trying to start a new Solr
> server, and it collides with the Solr server that is already running,
> as I get the following exception:
>
> SEVERE [org.apache.solr.core.CoreContainer] Unable to create core: general
> java.nio.channles.OverlappingFileLockException
> ...
>   
You create a new CoreContainer and it will create and start the 
SolrCores described in your solr.xml. But they are already running. You 
dont want to create a new CoreContainer but look up the existing one 
instead, and use that for your EmbeddedSolrServer. The CoreContainer is 
available through SolrDispatchFilter.getCores, so basically you need to 
get hold of the SolrDispatchFilter instance.

You do not say much about where/how your "other code" (the code with you 
timer job) runs. Is it in the same webapp as Solr (you might have hacked 
the web.xml of Solr), is it another webapp running side by side with the 
solr-webapp in the same webcontainer, is it a ejb-app or what is it. 
Depending on where/how your "other code" runs there are different ways 
to get hold of the SolrDispatchFilter. You can probably get it through 
JNDI but in some containers this means that you need to do some 
JNDI-name-wireing between your apps. There are other "easier" ways 
depending on where/how your "other code" runs - I believe you want to 
google for things like ServletContext, getRequestDispatcher, 
getNamedDispatcher etc.
You might also have to consider the actual container you are using (for 
Solr its Jetty out-of-the-box, but you might run on tomcat or something 
else), even though I believe the specs allow you to do what you want - 
and if the spec dictates a way, all webcontainers (that want to be 
certified) have to support it.

Hope it helps you. Else I might get time to help you in a little more 
concrete way. Have been teaching webapp and ejb-app stuff many years 
ago, so I might be able to dust off my knowledge about it in order to 
help you.

> I know that I can also use HttpSolrServer, but I don't really want to
> connect Http when I am already in  the application server.
>
> What is the suggested way to connect to an already running solr server
> using Solrj.  Am I using EmbeddedSolrServer wrong, or is it expected
> behavior for it to try and start a new server?
>
> Thanks in advance!!!
>
>