You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Ritesh Kumar <ri...@hotwaxsystems.com> on 2018/07/02 13:35:07 UTC

Creating single CloudSolrClient object which can be used throughout the application

Hello Team,

I have got a static method which returns CloudSolrClient object if Solr is
running in Cloud mode and HttpSolrClient object otherwise.

When running bulk indexing service, this method is called from within the
indexing service to get the appropriate client object. Each time, this
method creates a new client object. The problem is, when the bulk indexing
service is run, after a while, connection error occurs (could not connect
to zookeeper running at 0.0.0.0:2181. It seems the Zookeeper runs out of
connections.

Configuration:
One Zookeeper - maxClientCnxns=60
Two Solr nodes, running in the Cloud mode.

After looking out for the solution, I could find that CloudSolrClient is
thread safe provided it collection remains the same.

How can I create an object of CloudSolrClient such that it is used
throughout the application without creating a new object each time the data
is indexed.

Best,
Ritesh Kumar

Re: Creating single CloudSolrClient object which can be used throughout the application

Posted by Erick Erickson <er...@gmail.com>.
Just a try block or a try-with-resources block? I.e.

try (CloudSolrClient csc = new ClousSolrClient.....) {
}

try-with-resources is _designed_ to call close when the
block is exited, there's no mystery involved at all.

Best,
Erick

On Wed, Jul 4, 2018 at 10:13 PM, Ritesh Kumar
<ri...@hotwaxsystems.com> wrote:
> Hello Shawn,
>
> I wasn't explicitly closing the client object but I fetched the client
> object inside the try block and this seems to automatically destroy the
> client object.
>
> Taking it out of the try block worked like magic.
>
> Problem solved!
>
> Best
>
> On Wed, Jul 4, 2018 at 10:40 PM Shawn Heisey <ap...@elyograg.org> wrote:
>
>> On 7/4/2018 2:41 AM, Ritesh Kumar wrote:
>> > I did exactly as you told, created a public static synchronized method.
>> The
>> > problem still exists.
>>
>> I wasn't addressing the connection problem.  I was addressing the
>> question in the subject -- one client object that you can use
>> everywhere.  But I think the problem you're having now is different than
>> the initial problem, based on the error message:
>>
>> > Maybe returning the client object if it is not null is causing
>> > " java.lang.IllegalStateException: Connection pool shut down" error. It
>> > does run fine for just one time.
>>
>> This means that you are still calling client.close() after you use the
>> client.  That must be removed.  The entire point of the close() call is
>> to release the internal objects to garbage and make the client unusable.
>>
>> Thanks,
>> Shawn
>>
>>

Re: Creating single CloudSolrClient object which can be used throughout the application

Posted by Ritesh Kumar <ri...@hotwaxsystems.com>.
Hello Shawn,

I wasn't explicitly closing the client object but I fetched the client
object inside the try block and this seems to automatically destroy the
client object.

Taking it out of the try block worked like magic.

Problem solved!

Best

On Wed, Jul 4, 2018 at 10:40 PM Shawn Heisey <ap...@elyograg.org> wrote:

> On 7/4/2018 2:41 AM, Ritesh Kumar wrote:
> > I did exactly as you told, created a public static synchronized method.
> The
> > problem still exists.
>
> I wasn't addressing the connection problem.  I was addressing the
> question in the subject -- one client object that you can use
> everywhere.  But I think the problem you're having now is different than
> the initial problem, based on the error message:
>
> > Maybe returning the client object if it is not null is causing
> > " java.lang.IllegalStateException: Connection pool shut down" error. It
> > does run fine for just one time.
>
> This means that you are still calling client.close() after you use the
> client.  That must be removed.  The entire point of the close() call is
> to release the internal objects to garbage and make the client unusable.
>
> Thanks,
> Shawn
>
>

Re: Creating single CloudSolrClient object which can be used throughout the application

Posted by Shawn Heisey <ap...@elyograg.org>.
On 7/4/2018 2:41 AM, Ritesh Kumar wrote:
> I did exactly as you told, created a public static synchronized method. The
> problem still exists.

I wasn't addressing the connection problem.  I was addressing the 
question in the subject -- one client object that you can use 
everywhere.  But I think the problem you're having now is different than 
the initial problem, based on the error message:

> Maybe returning the client object if it is not null is causing
> " java.lang.IllegalStateException: Connection pool shut down" error. It
> does run fine for just one time.

This means that you are still calling client.close() after you use the 
client.  That must be removed.  The entire point of the close() call is 
to release the internal objects to garbage and make the client unusable.

Thanks,
Shawn

Re: Creating single CloudSolrClient object which can be used throughout the application

Posted by Ritesh Kumar <ri...@hotwaxsystems.com>.
Hello Shawn,

I did exactly as you told, created a public static synchronized method. The
problem still exists.

Maybe returning the client object if it is not null is causing
" java.lang.IllegalStateException: Connection pool shut down" error. It
does run fine for just one time.

pseudo code:
private static SolrClient client = null;
public static synchronized SolrClient getSolrClient(collectionName) {
        if (client != null) {
            return client;
        }
        if (Boolean.parseBoolean(isSolrCloudEnabled)) {
            client = new
CloudSolrClient.Builder().withZkHost(ZKHOSTS).build();
            ((CloudSolrClient)client).setDefaultCollection(collectionName);
        } else {
            client = new HttpSolrClient.Builder(getSolrHost(delegator,
collectionName)).build();
        }
        return client;
    }

If I remove this code and simply create a new client object still keeping
the method synchronized, everything seems to be running fine.

Am I missing something?

On Tue, Jul 3, 2018 at 6:04 AM Shawn Heisey <ap...@elyograg.org> wrote:

> On 7/2/2018 7:35 AM, Ritesh Kumar wrote:
> > I have got a static method which returns CloudSolrClient object if Solr
> is
> > running in Cloud mode and HttpSolrClient object otherwise.
>
> Declare that method as synchronized, so that multiple usages do not step
> on each other's toes.  This will also eliminate object visibility issues
> in multi-threaded code.  The modifiers for the method will probably end
> up being "public static synchronized".
>
> In the class where that method lives, create a "private static
> SolrClient" field and set it to null.  In the method, if the class-level
> field is not null, return it.  If it is null, create the HttpSolrClient
> or CloudSolrClient object just as you do now, set the default collection
> if that's required, then assign that client object to the class-level
> field and return it.
>
> Remove any client.close() calls that you have currently.  You can close
> the client at application shutdown, but this is not actually necessary
> if application shutdown also halts the JVM.
>
> You could also use the singleton paradigm that Erick mentioned, but
> since you already have code to obtain a client object, it's probably
> more straightforward to just modify that code as I have described, and
> don't close the client after you use it.
>
> Thanks,
> Shawn
>
>

Re: Creating single CloudSolrClient object which can be used throughout the application

Posted by Shawn Heisey <ap...@elyograg.org>.
On 7/2/2018 7:35 AM, Ritesh Kumar wrote:
> I have got a static method which returns CloudSolrClient object if Solr is
> running in Cloud mode and HttpSolrClient object otherwise.

Declare that method as synchronized, so that multiple usages do not step
on each other's toes.  This will also eliminate object visibility issues
in multi-threaded code.  The modifiers for the method will probably end
up being "public static synchronized".

In the class where that method lives, create a "private static
SolrClient" field and set it to null.  In the method, if the class-level
field is not null, return it.  If it is null, create the HttpSolrClient
or CloudSolrClient object just as you do now, set the default collection
if that's required, then assign that client object to the class-level
field and return it.

Remove any client.close() calls that you have currently.  You can close
the client at application shutdown, but this is not actually necessary
if application shutdown also halts the JVM.

You could also use the singleton paradigm that Erick mentioned, but
since you already have code to obtain a client object, it's probably
more straightforward to just modify that code as I have described, and
don't close the client after you use it.

Thanks,
Shawn

Re: Creating single CloudSolrClient object which can be used throughout the application

Posted by Ritesh Kumar <ri...@hotwaxsystems.com>.
Yes, the client object is closed each time.

The bulk indexing service calls the service which, let's say, indexes all
the orders from the database. So, a service is called *asynchronously* from
within the bulk service which indexes order related data individually for
each order. There may be more than one such bulk services.

Pseudo code:
createOrdersIndex () {
    allThe Orders.iterate {
        createOrderIndex(orderId)
   }
}
createOrderIndex() {
    get the client object
    client.add(SolrInputDocument)
    client.close()
}

The individual service indexing each order is scheduled, each one having a
client object. So there is a possibility that the services have not yet
finished execution, meaning the client objects are not closed yet,
resulting in connection timeout.

This is how I have prepared the CloudSolrClient object
client = new CloudSolrClient.Builder().withZkHost("zkHosts").build();
client.setDefaultCollection(collectionName)

Hope, this gives a clear picture of the problem I am facing.



On Mon, Jul 2, 2018 at 7:41 PM Erick Erickson <er...@gmail.com>
wrote:

> It's recommended to use one object of course. That said, you should
> not be having a connection problem just because you create new ones
> all the time. Are you closing it after you're done with it each time?
>
> As to your question about how to reuse the same one, the "singleton
> pattern" is one solution.
>
> Best,
> Erick
>
> On Mon, Jul 2, 2018 at 6:35 AM, Ritesh Kumar
> <ri...@hotwaxsystems.com> wrote:
> > Hello Team,
> >
> > I have got a static method which returns CloudSolrClient object if Solr
> is
> > running in Cloud mode and HttpSolrClient object otherwise.
> >
> > When running bulk indexing service, this method is called from within the
> > indexing service to get the appropriate client object. Each time, this
> > method creates a new client object. The problem is, when the bulk
> indexing
> > service is run, after a while, connection error occurs (could not connect
> > to zookeeper running at 0.0.0.0:2181. It seems the Zookeeper runs out of
> > connections.
> >
> > Configuration:
> > One Zookeeper - maxClientCnxns=60
> > Two Solr nodes, running in the Cloud mode.
> >
> > After looking out for the solution, I could find that CloudSolrClient is
> > thread safe provided it collection remains the same.
> >
> > How can I create an object of CloudSolrClient such that it is used
> > throughout the application without creating a new object each time the
> data
> > is indexed.
> >
> > Best,
> > Ritesh Kumar
>

Re: Creating single CloudSolrClient object which can be used throughout the application

Posted by Erick Erickson <er...@gmail.com>.
It's recommended to use one object of course. That said, you should
not be having a connection problem just because you create new ones
all the time. Are you closing it after you're done with it each time?

As to your question about how to reuse the same one, the "singleton
pattern" is one solution.

Best,
Erick

On Mon, Jul 2, 2018 at 6:35 AM, Ritesh Kumar
<ri...@hotwaxsystems.com> wrote:
> Hello Team,
>
> I have got a static method which returns CloudSolrClient object if Solr is
> running in Cloud mode and HttpSolrClient object otherwise.
>
> When running bulk indexing service, this method is called from within the
> indexing service to get the appropriate client object. Each time, this
> method creates a new client object. The problem is, when the bulk indexing
> service is run, after a while, connection error occurs (could not connect
> to zookeeper running at 0.0.0.0:2181. It seems the Zookeeper runs out of
> connections.
>
> Configuration:
> One Zookeeper - maxClientCnxns=60
> Two Solr nodes, running in the Cloud mode.
>
> After looking out for the solution, I could find that CloudSolrClient is
> thread safe provided it collection remains the same.
>
> How can I create an object of CloudSolrClient such that it is used
> throughout the application without creating a new object each time the data
> is indexed.
>
> Best,
> Ritesh Kumar