You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by RAUNAK AGRAWAL <ag...@gmail.com> on 2018/01/02 15:31:28 UTC
SolrJ with Async Http Client
Hi Guys,
I am trying to write fully async service where solr calls are also async.
Just wondering did anyone tried calling solr in non-blocking mode or is
there is a way to do it? I have come across one such project
<https://github.com/inoio/solrs> but wondering is there anything provided
by solrj?
Thanks
Re: SolrJ with Async Http Client
Posted by Erick Erickson <er...@gmail.com>.
re: upgrade Jetty:
See SOLR-11810
On Fri, Jan 5, 2018 at 2:08 PM, Shawn Heisey <ap...@elyograg.org> wrote:
> On 1/5/2018 11:35 AM, Gus Heck wrote:
> > IIRC http2 allows for multiple (non blocking) requests over a single
> > connection, so this Jira might be relevant:
> > https://issues.apache.org/jira/browse/SOLR-7442
>
> HTTPClient will have http/2 support in version 5.0 -- but only in the
> async client, not the client that SolrJ currently uses.
>
> Parts of the 5.0 version are in alpha at the moment, one part is in
> beta. I do not have any information about when it will be fully
> released as a stable version. We can't fully implement a new SolrJ that
> uses the new HC version until they officially release it.
>
> I have been looking into upgrading to HC 5.0 so that Solr will be
> prepared once that version gets released, but the job is turning out a
> lot more difficult than I had imagined it to be, so it's not really
> going anywhere because I don't have very much time to devote to the
> work. And I haven't done anything yet for switching to the async
> client. Issue here:
>
> https://issues.apache.org/jira/browse/SOLR-11780
>
> To fully support http/2, there will also need to be some work in Jetty,
> and probably an upgrade to the latest release of Jetty so we can be sure
> we have all their bugfixes.
>
> My efforts will not bring async operation to SolrJ, even if I manage to
> figure out how to use the async HttpClient. I have no idea how to
> implement that. There is a very old issue in Jira for it.
>
> https://issues.apache.org/jira/browse/SOLR-3383
>
> Thanks,
> Shawn
>
>
Re: SolrJ with Async Http Client
Posted by Shawn Heisey <ap...@elyograg.org>.
On 1/5/2018 11:35 AM, Gus Heck wrote:
> IIRC http2 allows for multiple (non blocking) requests over a single
> connection, so this Jira might be relevant:
> https://issues.apache.org/jira/browse/SOLR-7442
HTTPClient will have http/2 support in version 5.0 -- but only in the
async client, not the client that SolrJ currently uses.
Parts of the 5.0 version are in alpha at the moment, one part is in
beta. I do not have any information about when it will be fully
released as a stable version. We can't fully implement a new SolrJ that
uses the new HC version until they officially release it.
I have been looking into upgrading to HC 5.0 so that Solr will be
prepared once that version gets released, but the job is turning out a
lot more difficult than I had imagined it to be, so it's not really
going anywhere because I don't have very much time to devote to the
work. And I haven't done anything yet for switching to the async
client. Issue here:
https://issues.apache.org/jira/browse/SOLR-11780
To fully support http/2, there will also need to be some work in Jetty,
and probably an upgrade to the latest release of Jetty so we can be sure
we have all their bugfixes.
My efforts will not bring async operation to SolrJ, even if I manage to
figure out how to use the async HttpClient. I have no idea how to
implement that. There is a very old issue in Jira for it.
https://issues.apache.org/jira/browse/SOLR-3383
Thanks,
Shawn
Re: SolrJ with Async Http Client
Posted by Gus Heck <gu...@gmail.com>.
IIRC http2 allows for multiple (non blocking) requests over a single
connection, so this Jira might be relevant:
https://issues.apache.org/jira/browse/SOLR-7442
-Gus
On Wed, Jan 3, 2018 at 10:22 AM, Walter Underwood <wu...@wunderwood.org>
wrote:
> HTTPClient is non-blocking. Send the request, then the client gets control
> back. It only blocks when you do the read. So one thread can send multiple
> requests then check for each response.
>
> wunder
> Walter Underwood
> wunder@wunderwood.org
> http://observer.wunderwood.org/ (my blog)
>
>
> > On Jan 3, 2018, at 5:11 AM, RAUNAK AGRAWAL <ag...@gmail.com>
> wrote:
> >
> > Yes, I am talking about event driven way of calling solr, so that I can
> > write pure async web service. Does SolrJ provides support for
> non-blocking
> > calls?
> >
> > On Wed, Jan 3, 2018 at 6:22 PM, Hendrik Haddorp <hendrik.haddorp@gmx.net
> >
> > wrote:
> >
> >> There is asynchronous and non-blocking. If I use 100 threads to perform
> >> calls to Solr using the standard Java HTTP client or SolrJ I block 100
> >> threads even if I don't block my program logic threads by using async
> >> calls. However if I perform those HTTP calls using a non-blocking HTTP
> >> client, like netty, I basically only need a single eventing thread in
> >> addition to my normal threads. The advantage is less memory usage and an
> >> often better scaling. I would however expect that the main advantage
> would
> >> be on the server side.
> >>
> >>
> >> On 02.01.2018 22:02, Gus Heck wrote:
> >>
> >>> It's not very clear (to me) what your use case is, but generally
> speaking,
> >>> asynchronous requests can be achieved by using
> threads/executors/futures
> >>> (java) or ajax (javascript). The link seems to be a scala project, I'm
> >>> sure
> >>> scala has analogous facilities.
> >>>
> >>> On Tue, Jan 2, 2018 at 10:31 AM, RAUNAK AGRAWAL <
> agrawal.raunak@gmail.com
> >>>>
> >>> wrote:
> >>>
> >>> Hi Guys,
> >>>>
> >>>> I am trying to write fully async service where solr calls are also
> async.
> >>>> Just wondering did anyone tried calling solr in non-blocking mode or
> is
> >>>> there is a way to do it? I have come across one such project
> >>>> <https://github.com/inoio/solrs> but wondering is there anything
> >>>> provided
> >>>> by solrj?
> >>>>
> >>>> Thanks
> >>>>
> >>>>
> >>>
> >>>
> >>
>
>
--
http://www.the111shift.com
Re: SolrJ with Async Http Client
Posted by Walter Underwood <wu...@wunderwood.org>.
HTTPClient is non-blocking. Send the request, then the client gets control back. It only blocks when you do the read. So one thread can send multiple requests then check for each response.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Jan 3, 2018, at 5:11 AM, RAUNAK AGRAWAL <ag...@gmail.com> wrote:
>
> Yes, I am talking about event driven way of calling solr, so that I can
> write pure async web service. Does SolrJ provides support for non-blocking
> calls?
>
> On Wed, Jan 3, 2018 at 6:22 PM, Hendrik Haddorp <he...@gmx.net>
> wrote:
>
>> There is asynchronous and non-blocking. If I use 100 threads to perform
>> calls to Solr using the standard Java HTTP client or SolrJ I block 100
>> threads even if I don't block my program logic threads by using async
>> calls. However if I perform those HTTP calls using a non-blocking HTTP
>> client, like netty, I basically only need a single eventing thread in
>> addition to my normal threads. The advantage is less memory usage and an
>> often better scaling. I would however expect that the main advantage would
>> be on the server side.
>>
>>
>> On 02.01.2018 22:02, Gus Heck wrote:
>>
>>> It's not very clear (to me) what your use case is, but generally speaking,
>>> asynchronous requests can be achieved by using threads/executors/futures
>>> (java) or ajax (javascript). The link seems to be a scala project, I'm
>>> sure
>>> scala has analogous facilities.
>>>
>>> On Tue, Jan 2, 2018 at 10:31 AM, RAUNAK AGRAWAL <agrawal.raunak@gmail.com
>>>>
>>> wrote:
>>>
>>> Hi Guys,
>>>>
>>>> I am trying to write fully async service where solr calls are also async.
>>>> Just wondering did anyone tried calling solr in non-blocking mode or is
>>>> there is a way to do it? I have come across one such project
>>>> <https://github.com/inoio/solrs> but wondering is there anything
>>>> provided
>>>> by solrj?
>>>>
>>>> Thanks
>>>>
>>>>
>>>
>>>
>>
Re: SolrJ with Async Http Client
Posted by Joel Bernstein <jo...@gmail.com>.
Streaming expressions has an event driven architecture built in. There are
two blogs that describe how it works.
This describes the message queues:
http://joelsolr.blogspot.com/2016/10/solr-63-batch-jobs-parallel-etl-and.html
This describes an async model of execution:
http://joelsolr.blogspot.com/2017/01/deploying-solrs-new-parallel-executor.html
After you've read through the two blogs let me know if you have questions
about how to apply this to your use case.
Joel Bernstein
http://joelsolr.blogspot.com/
On Wed, Jan 3, 2018 at 8:11 AM, RAUNAK AGRAWAL <ag...@gmail.com>
wrote:
> Yes, I am talking about event driven way of calling solr, so that I can
> write pure async web service. Does SolrJ provides support for non-blocking
> calls?
>
> On Wed, Jan 3, 2018 at 6:22 PM, Hendrik Haddorp <he...@gmx.net>
> wrote:
>
> > There is asynchronous and non-blocking. If I use 100 threads to perform
> > calls to Solr using the standard Java HTTP client or SolrJ I block 100
> > threads even if I don't block my program logic threads by using async
> > calls. However if I perform those HTTP calls using a non-blocking HTTP
> > client, like netty, I basically only need a single eventing thread in
> > addition to my normal threads. The advantage is less memory usage and an
> > often better scaling. I would however expect that the main advantage
> would
> > be on the server side.
> >
> >
> > On 02.01.2018 22:02, Gus Heck wrote:
> >
> >> It's not very clear (to me) what your use case is, but generally
> speaking,
> >> asynchronous requests can be achieved by using threads/executors/futures
> >> (java) or ajax (javascript). The link seems to be a scala project, I'm
> >> sure
> >> scala has analogous facilities.
> >>
> >> On Tue, Jan 2, 2018 at 10:31 AM, RAUNAK AGRAWAL <
> agrawal.raunak@gmail.com
> >> >
> >> wrote:
> >>
> >> Hi Guys,
> >>>
> >>> I am trying to write fully async service where solr calls are also
> async.
> >>> Just wondering did anyone tried calling solr in non-blocking mode or is
> >>> there is a way to do it? I have come across one such project
> >>> <https://github.com/inoio/solrs> but wondering is there anything
> >>> provided
> >>> by solrj?
> >>>
> >>> Thanks
> >>>
> >>>
> >>
> >>
> >
>
Re: SolrJ with Async Http Client
Posted by RAUNAK AGRAWAL <ag...@gmail.com>.
Yes, I am talking about event driven way of calling solr, so that I can
write pure async web service. Does SolrJ provides support for non-blocking
calls?
On Wed, Jan 3, 2018 at 6:22 PM, Hendrik Haddorp <he...@gmx.net>
wrote:
> There is asynchronous and non-blocking. If I use 100 threads to perform
> calls to Solr using the standard Java HTTP client or SolrJ I block 100
> threads even if I don't block my program logic threads by using async
> calls. However if I perform those HTTP calls using a non-blocking HTTP
> client, like netty, I basically only need a single eventing thread in
> addition to my normal threads. The advantage is less memory usage and an
> often better scaling. I would however expect that the main advantage would
> be on the server side.
>
>
> On 02.01.2018 22:02, Gus Heck wrote:
>
>> It's not very clear (to me) what your use case is, but generally speaking,
>> asynchronous requests can be achieved by using threads/executors/futures
>> (java) or ajax (javascript). The link seems to be a scala project, I'm
>> sure
>> scala has analogous facilities.
>>
>> On Tue, Jan 2, 2018 at 10:31 AM, RAUNAK AGRAWAL <agrawal.raunak@gmail.com
>> >
>> wrote:
>>
>> Hi Guys,
>>>
>>> I am trying to write fully async service where solr calls are also async.
>>> Just wondering did anyone tried calling solr in non-blocking mode or is
>>> there is a way to do it? I have come across one such project
>>> <https://github.com/inoio/solrs> but wondering is there anything
>>> provided
>>> by solrj?
>>>
>>> Thanks
>>>
>>>
>>
>>
>
Re: SolrJ with Async Http Client
Posted by Hendrik Haddorp <he...@gmx.net>.
There is asynchronous and non-blocking. If I use 100 threads to perform
calls to Solr using the standard Java HTTP client or SolrJ I block 100
threads even if I don't block my program logic threads by using async
calls. However if I perform those HTTP calls using a non-blocking HTTP
client, like netty, I basically only need a single eventing thread in
addition to my normal threads. The advantage is less memory usage and an
often better scaling. I would however expect that the main advantage
would be on the server side.
On 02.01.2018 22:02, Gus Heck wrote:
> It's not very clear (to me) what your use case is, but generally speaking,
> asynchronous requests can be achieved by using threads/executors/futures
> (java) or ajax (javascript). The link seems to be a scala project, I'm sure
> scala has analogous facilities.
>
> On Tue, Jan 2, 2018 at 10:31 AM, RAUNAK AGRAWAL <ag...@gmail.com>
> wrote:
>
>> Hi Guys,
>>
>> I am trying to write fully async service where solr calls are also async.
>> Just wondering did anyone tried calling solr in non-blocking mode or is
>> there is a way to do it? I have come across one such project
>> <https://github.com/inoio/solrs> but wondering is there anything provided
>> by solrj?
>>
>> Thanks
>>
>
>
Re: SolrJ with Async Http Client
Posted by Gus Heck <gu...@gmail.com>.
It's not very clear (to me) what your use case is, but generally speaking,
asynchronous requests can be achieved by using threads/executors/futures
(java) or ajax (javascript). The link seems to be a scala project, I'm sure
scala has analogous facilities.
On Tue, Jan 2, 2018 at 10:31 AM, RAUNAK AGRAWAL <ag...@gmail.com>
wrote:
> Hi Guys,
>
> I am trying to write fully async service where solr calls are also async.
> Just wondering did anyone tried calling solr in non-blocking mode or is
> there is a way to do it? I have come across one such project
> <https://github.com/inoio/solrs> but wondering is there anything provided
> by solrj?
>
> Thanks
>
--
http://www.the111shift.com
Re: SolrJ with Async Http Client
Posted by Emir Arnautović <em...@sematext.com>.
Not sure if alilgns with your expectations, but here is something that is declared as “async solr client”: https://github.com/inoio/solrs <https://github.com/inoio/solrs>
HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> On 2 Jan 2018, at 16:31, RAUNAK AGRAWAL <ag...@gmail.com> wrote:
>
> Hi Guys,
>
> I am trying to write fully async service where solr calls are also async.
> Just wondering did anyone tried calling solr in non-blocking mode or is
> there is a way to do it? I have come across one such project
> <https://github.com/inoio/solrs> but wondering is there anything provided
> by solrj?
>
> Thanks
Re: SolrJ with Async Http Client
Posted by Rick Leir <rl...@leirtech.com>.
Agrawal
There is good reading on the topic at
https://wiki.apache.org/solr/IntegratingSolr
Cheers -- Rick
On January 2, 2018 10:31:28 AM EST, RAUNAK AGRAWAL <ag...@gmail.com> wrote:
>Hi Guys,
>
>I am trying to write fully async service where solr calls are also
>async.
>Just wondering did anyone tried calling solr in non-blocking mode or is
>there is a way to do it? I have come across one such project
><https://github.com/inoio/solrs> but wondering is there anything
>provided
>by solrj?
>
>Thanks
--
Sorry for being brief. Alternate email is rickleir at yahoo dot com