You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stanbol.apache.org by Dileepa Jayakody <di...@gmail.com> on 2013/11/15 12:04:32 UTC
Can I increase the concurrent request number for enhancer?
Hi All,
I'm doing a project using Stanbol to enhance content of an external Solr
index.
What I basically do is I call Stanbol enhancer endpoint when indexing
documents in Solr to enhance particular field values by extracting Person,
Organization entities.
In Solr dataimport handler we can import large number of documents into
Solr index, and for each document indexed I'm calling Stanbol enhancer to
perform enhancements on the content.
I tested my setup manually indexing 1 document, and it works fine.
However when I trigger a dataimport in Solr with a big number of documents
being indexed, the Stanbol server throws Connection Refused erros as below
[1];
I assume this is because huge number of concurrent connections to Stanbol.
Is there a concurrent request threshold for Stanbol?
How can I increase it?
Thanks,
Dileepa
[1]
com.sun.jersey.api.client.ClientHandlerException:
java.net.ConnectException: Connection refused
at
com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:151)
at com.sun.jersey.api.client.Client.handle(Client.java:648)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:680)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:558)
at
com.solr.stanbol.processor.StanbolContentProcessorFactory.stanbolPost(StanbolContentProcessorFactory.java:115)
at
com.solr.stanbol.processor.StanbolContentProcessorFactory.access$1(StanbolContentProcessorFactory.java:109)
at
com.solr.stanbol.processor.StanbolContentProcessorFactory$StanbolContentProcessor.processAdd(StanbolContentProcessorFactory.java:70)
Re: Can I increase the concurrent request number for enhancer?
Posted by Dileepa Jayakody <di...@gmail.com>.
Thanks a lot Rupert.
On Fri, Nov 15, 2013 at 8:03 PM, Rupert Westenthaler <
rupert.westenthaler@gmail.com> wrote:
> Hi Dileepa,
>
> Stanbol can not throw a "Connection refused". This is thrown by the
> client - in your example the Solr ImportHandler - when it con not
> connect to the configured Stanbol Server. If you see this exception it
> means that the client is not able to establish an TCP connection with
> the receiving host (e.g. because there is no service listening to the
> specified Port).
>
> If you use the standalone server an embedded Jetty is used to provide
> the HTTP service. AFAIK Jetty does not refuse any connection, but uses
> an unlimited queue by default. However I found [2] that connections
> can be rejected on the OS level resulting in Connection Refused
> messages. If you use the WAR launcher you will need to look at the
> configuration of the used Web Container such as Apache Tomcat.
>
> I would recommend you to use the Stanbol Stress test tool for
> performance testing your Stanbol configuration. By that you can easily
> test what happens if you increase the number of concurrent threads.
>
> hope this helps
> Rupert
>
>
> [1] http://stanbol.apache.org/docs/trunk/utils/enhancerstresstest
> [2] http://wiki.eclipse.org/Jetty/Howto/High_Load#Queue_Sizes
>
> On Fri, Nov 15, 2013 at 12:04 PM, Dileepa Jayakody
> <di...@gmail.com> wrote:
> > Hi All,
> >
> > I'm doing a project using Stanbol to enhance content of an external Solr
> > index.
> > What I basically do is I call Stanbol enhancer endpoint when indexing
> > documents in Solr to enhance particular field values by extracting
> Person,
> > Organization entities.
> > In Solr dataimport handler we can import large number of documents into
> > Solr index, and for each document indexed I'm calling Stanbol enhancer to
> > perform enhancements on the content.
> >
> > I tested my setup manually indexing 1 document, and it works fine.
> > However when I trigger a dataimport in Solr with a big number of
> documents
> > being indexed, the Stanbol server throws Connection Refused erros as
> below
> > [1];
> > I assume this is because huge number of concurrent connections to
> Stanbol.
> >
> > Is there a concurrent request threshold for Stanbol?
> > How can I increase it?
> >
> > Thanks,
> > Dileepa
> > [1]
> > com.sun.jersey.api.client.ClientHandlerException:
> > java.net.ConnectException: Connection refused
> > at
> >
> com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:151)
> > at com.sun.jersey.api.client.Client.handle(Client.java:648)
> > at com.sun.jersey.api.client.WebResource.handle(WebResource.java:680)
> > at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
> > at
> com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:558)
> > at
> >
> com.solr.stanbol.processor.StanbolContentProcessorFactory.stanbolPost(StanbolContentProcessorFactory.java:115)
> > at
> >
> com.solr.stanbol.processor.StanbolContentProcessorFactory.access$1(StanbolContentProcessorFactory.java:109)
> > at
> >
> com.solr.stanbol.processor.StanbolContentProcessorFactory$StanbolContentProcessor.processAdd(StanbolContentProcessorFactory.java:70)
>
>
>
> --
> | Rupert Westenthaler rupert.westenthaler@gmail.com
> | Bodenlehenstraße 11 ++43-699-11108907
> | A-5500 Bischofshofen
>
Re: Can I increase the concurrent request number for enhancer?
Posted by Rupert Westenthaler <ru...@gmail.com>.
Hi Dileepa,
Stanbol can not throw a "Connection refused". This is thrown by the
client - in your example the Solr ImportHandler - when it con not
connect to the configured Stanbol Server. If you see this exception it
means that the client is not able to establish an TCP connection with
the receiving host (e.g. because there is no service listening to the
specified Port).
If you use the standalone server an embedded Jetty is used to provide
the HTTP service. AFAIK Jetty does not refuse any connection, but uses
an unlimited queue by default. However I found [2] that connections
can be rejected on the OS level resulting in Connection Refused
messages. If you use the WAR launcher you will need to look at the
configuration of the used Web Container such as Apache Tomcat.
I would recommend you to use the Stanbol Stress test tool for
performance testing your Stanbol configuration. By that you can easily
test what happens if you increase the number of concurrent threads.
hope this helps
Rupert
[1] http://stanbol.apache.org/docs/trunk/utils/enhancerstresstest
[2] http://wiki.eclipse.org/Jetty/Howto/High_Load#Queue_Sizes
On Fri, Nov 15, 2013 at 12:04 PM, Dileepa Jayakody
<di...@gmail.com> wrote:
> Hi All,
>
> I'm doing a project using Stanbol to enhance content of an external Solr
> index.
> What I basically do is I call Stanbol enhancer endpoint when indexing
> documents in Solr to enhance particular field values by extracting Person,
> Organization entities.
> In Solr dataimport handler we can import large number of documents into
> Solr index, and for each document indexed I'm calling Stanbol enhancer to
> perform enhancements on the content.
>
> I tested my setup manually indexing 1 document, and it works fine.
> However when I trigger a dataimport in Solr with a big number of documents
> being indexed, the Stanbol server throws Connection Refused erros as below
> [1];
> I assume this is because huge number of concurrent connections to Stanbol.
>
> Is there a concurrent request threshold for Stanbol?
> How can I increase it?
>
> Thanks,
> Dileepa
> [1]
> com.sun.jersey.api.client.ClientHandlerException:
> java.net.ConnectException: Connection refused
> at
> com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:151)
> at com.sun.jersey.api.client.Client.handle(Client.java:648)
> at com.sun.jersey.api.client.WebResource.handle(WebResource.java:680)
> at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
> at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:558)
> at
> com.solr.stanbol.processor.StanbolContentProcessorFactory.stanbolPost(StanbolContentProcessorFactory.java:115)
> at
> com.solr.stanbol.processor.StanbolContentProcessorFactory.access$1(StanbolContentProcessorFactory.java:109)
> at
> com.solr.stanbol.processor.StanbolContentProcessorFactory$StanbolContentProcessor.processAdd(StanbolContentProcessorFactory.java:70)
--
| Rupert Westenthaler rupert.westenthaler@gmail.com
| Bodenlehenstraße 11 ++43-699-11108907
| A-5500 Bischofshofen