You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Shawn Heisey <so...@elyograg.org> on 2013/02/04 22:25:24 UTC
Need to create SolrServer objects without checking server availability
I have a SolrJ app that talks to four Solr servers. For each server, it
creates 14 HttpSolrServer objects, each for a different core.
Once it's running, the app knows how to deal with errors, especially
those caused when the server goes down.
The problem is at program startup -- when 'new HttpSolrServer(url)' is
called, it goes and makes sure that the server is up and responsive. If
any of those 56 object creation calls fails, then my app won't even start.
Due to the design of my app, a server being down at startup is not
really a problem. When the program tries to update an index that's
down, that fact will be noticed. If the situation continues for long
enough, an alarm email is generated, but the program will continue to
operate correctly on the servers that are still up.
As far as I can tell, there is no built-in way to bypass the health
check at object creation. Is there an alternate way of initializing
these objects that would bypass this?
Someone on the IRC channel brought up the possibility of initializing
the HttpClient myself instead of letting the Solr object do it. If the
health check is actually in HttpClient, this might work, if there's a
way to initialize the HttpClient without a health check. I've actually
been wondering if it makes any sense to re-use a single HttpClient
object across all 56 server objects.
Should I file a Jira for an alternate SolrServer constructor that
doesn't perform a healthcheck?
Thanks,
Shawn
Re: Need to create SolrServer objects without checking server availability
Posted by Shawn Heisey <so...@elyograg.org>.
On 2/4/2013 3:33 PM, Michael Della Bitta wrote:
> Ah, OK, sorry to be terse!
>
> 1. Create a class that implements SolrServer from the SolrJ project:
> http://lucene.apache.org/solr/4_1_0/solr-solrj/org/apache/solr/client/solrj/SolrServer.html
> 2. Make the constructor of that class take as arguments the config you
> need to make an HttpSolrServer object and store them in member fields.
> 3. Make a private field for an HttpSolrServer instance but leave it null.
> 4. Create a private method called getSolrServer() that creates an
> HttpSolrServer and assigns it to the private field and returns it if
> successful, otherwise it catches and logs the exception that is
> thrown.
> 5. For each of the methods in SolrServer that you need to override,
> have it call getSolrServer() and call the same method on the returned
> instance if it's not null, e.g.,
>
> public UpdateResponse add(SolrInputDocument doc)
> throws SolrServerException,
> IOException {
>
> SolrServer solrServer = getSolrServer();
> if (solrServer != null) {
> return solrServer.add(doc);
> }
>
> return new UpdateResponse();
> }
>
> You may want to return something other than an empty UpdateResponse
> when things fail. That's up to you.
>
> 6. Refactor your project to use your SolrServer implementation.
Generally speaking, is it preferred to use a wrapper approach like
you've described or add a method to SolrServer and its implementations?
Whatever I come up with, I'd like to contribute to the project, and if
I'm doing that, I would think it's better to improve what's already
there than add a class.
Thanks,
Shawn
Re: Need to create SolrServer objects without checking server availability
Posted by Michael Della Bitta <mi...@appinions.com>.
Ah, OK, sorry to be terse!
1. Create a class that implements SolrServer from the SolrJ project:
http://lucene.apache.org/solr/4_1_0/solr-solrj/org/apache/solr/client/solrj/SolrServer.html
2. Make the constructor of that class take as arguments the config you
need to make an HttpSolrServer object and store them in member fields.
3. Make a private field for an HttpSolrServer instance but leave it null.
4. Create a private method called getSolrServer() that creates an
HttpSolrServer and assigns it to the private field and returns it if
successful, otherwise it catches and logs the exception that is
thrown.
5. For each of the methods in SolrServer that you need to override,
have it call getSolrServer() and call the same method on the returned
instance if it's not null, e.g.,
public UpdateResponse add(SolrInputDocument doc)
throws SolrServerException,
IOException {
SolrServer solrServer = getSolrServer();
if (solrServer != null) {
return solrServer.add(doc);
}
return new UpdateResponse();
}
You may want to return something other than an empty UpdateResponse
when things fail. That's up to you.
6. Refactor your project to use your SolrServer implementation.
That should be it!
Michael Della Bitta
------------------------------------------------
Appinions
18 East 41st Street, 2nd Floor
New York, NY 10017-6271
www.appinions.com
Where Influence Isn’t a Game
On Mon, Feb 4, 2013 at 5:05 PM, Shawn Heisey <so...@elyograg.org> wrote:
> On 2/4/2013 2:38 PM, Michael Della Bitta wrote:
>>
>> Hi Shawn,
>>
>> Why don't you write a delegating SolrServer class that lazily
>> instantiates an HttpSolrServer and catches and logs exceptions when
>> something's down?
>
>
> I only about half understood that, and I'm not sure how to do it. I'm
> willing to learn, though.
>
> Thanks,
> Shawn
>
Re: Need to create SolrServer objects without checking server availability
Posted by Shawn Heisey <so...@elyograg.org>.
On 2/4/2013 2:38 PM, Michael Della Bitta wrote:
> Hi Shawn,
>
> Why don't you write a delegating SolrServer class that lazily
> instantiates an HttpSolrServer and catches and logs exceptions when
> something's down?
I only about half understood that, and I'm not sure how to do it. I'm
willing to learn, though.
Thanks,
Shawn
Re: Need to create SolrServer objects without checking server availability
Posted by Michael Della Bitta <mi...@appinions.com>.
Hi Shawn,
Why don't you write a delegating SolrServer class that lazily
instantiates an HttpSolrServer and catches and logs exceptions when
something's down?
Michael Della Bitta
------------------------------------------------
Appinions
18 East 41st Street, 2nd Floor
New York, NY 10017-6271
www.appinions.com
Where Influence Isn’t a Game
On Mon, Feb 4, 2013 at 4:25 PM, Shawn Heisey <so...@elyograg.org> wrote:
> I have a SolrJ app that talks to four Solr servers. For each server, it
> creates 14 HttpSolrServer objects, each for a different core.
>
> Once it's running, the app knows how to deal with errors, especially those
> caused when the server goes down.
>
> The problem is at program startup -- when 'new HttpSolrServer(url)' is
> called, it goes and makes sure that the server is up and responsive. If any
> of those 56 object creation calls fails, then my app won't even start.
>
> Due to the design of my app, a server being down at startup is not really a
> problem. When the program tries to update an index that's down, that fact
> will be noticed. If the situation continues for long enough, an alarm email
> is generated, but the program will continue to operate correctly on the
> servers that are still up.
>
> As far as I can tell, there is no built-in way to bypass the health check at
> object creation. Is there an alternate way of initializing these objects
> that would bypass this?
>
> Someone on the IRC channel brought up the possibility of initializing the
> HttpClient myself instead of letting the Solr object do it. If the health
> check is actually in HttpClient, this might work, if there's a way to
> initialize the HttpClient without a health check. I've actually been
> wondering if it makes any sense to re-use a single HttpClient object across
> all 56 server objects.
>
> Should I file a Jira for an alternate SolrServer constructor that doesn't
> perform a healthcheck?
>
> Thanks,
> Shawn
Re: Need to create SolrServer objects without checking server availability
Posted by Shawn Heisey <so...@elyograg.org>.
On 2/12/2013 12:27 PM, Chris Hostetter wrote:
>
> : The problem is at program startup -- when 'new HttpSolrServer(url)' is called,
> : it goes and makes sure that the server is up and responsive. If any of those
> : 56 object creation calls fails, then my app won't even start.
>
> What exactly is the exception are you getting? i don't think anything in
> HttpSolrServer explicitly tries to test the server on creation.
>
> : Someone on the IRC channel brought up the possibility of initializing the
> : HttpClient myself instead of letting the Solr object do it. If the health
> : check is actually in HttpClient, this might work, if there's a way to
> : initialize the HttpClient without a health check. I've actually been
> : wondering if it makes any sense to re-use a single HttpClient object across
> : all 56 server objects.
>
> It probably would make sense for you to create a single HttpClient object
> that you re-use in all of your HttpSolrServer instances -- but i'm not
> sure how that would help your problem, since the HttpClient constructed
> implicitly by HttpSolrServer(String) doesn't know anything about the
> baseUrl.
>
> HttpClient literally can not test the health of the URL when it is
> created, because it doesn't know anything about any URLs until a request
> is executed.
I gathered up the exception, looked it over very closely ... and it's
all my fault! User error all the way on this one.
In my code, as soon as I create the object, I make a call that gets the
dataDir and instanceDir. I initially wrote this code so long ago that I
had forgotten this fact.
I have now changed it so this query is only made when the information is
requested, not when the object initializes. Aside: it turns out that
nothing in my code actually USES those getter methods! Now my program
will start up even if Solr is down. I'll look into re-using an http
client on all objects.
Thanks for the prodding, Hoss!
Shawn
Re: Need to create SolrServer objects without checking server
availability
Posted by Chris Hostetter <ho...@fucit.org>.
: The problem is at program startup -- when 'new HttpSolrServer(url)' is called,
: it goes and makes sure that the server is up and responsive. If any of those
: 56 object creation calls fails, then my app won't even start.
What exactly is the exception are you getting? i don't think antying in
HttpSolrServer explicitly tries to test the server on creation.
: Someone on the IRC channel brought up the possibility of initializing the
: HttpClient myself instead of letting the Solr object do it. If the health
: check is actually in HttpClient, this might work, if there's a way to
: initialize the HttpClient without a health check. I've actually been
: wondering if it makes any sense to re-use a single HttpClient object across
: all 56 server objects.
It probably would make sense for you to create a single HttpClient object
that you re-use in all of your HttpSolrServer instances -- but i'm not
sure how that would help your problem, since the HttpClient constructed
implicitly by HttpSolrServer(String) doesn't know anything about the
baseUrl.
HttpClient literally can not test the health of the URL when it is
created, because it doesn't know anything about any URLs until a request
is executed.
-Hoss