You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by "Billig, Andreas" <an...@fokus.fraunhofer.de> on 2015/10/11 11:40:08 UTC
java.net.BindException during query evaluation
Hello team,
we are developing a constraints checker (schema-check under Closed World Assumption (OWL, RDFS, etc.)) for a quad store and implement it via SPARQL/HTTP.
We ran into java.net.BindException during query evaluation.
I have isolated the code and made an abstraction. So now with the following code
-----
for (int i = 0; i < 20000; i++) { // TODO (i:~16300)
QueryExecution qe =
QueryExecutionFactory.sparqlService("http://localhost:3030/Yyy/query", "select * {?x ?y ?z}"); // only two solutions to keep the result set small
System.out.println(i);
ResultSet rs = qe.execSelect();
while (rs.hasNext()) {
System.out.println(rs.next());
}
qe.close();
}
-----
I got an exception around i > 16000 (so may be some queuing is involved) with trace
------
HttpException: java.net.BindException: Address already in use: connect: Unexpected error making the query: java.net.BindException: Address already in use: connect
at com.hp.hpl.jena.sparql.engine.http.HttpQuery.rewrap(HttpQuery.java:417)
at com.hp.hpl.jena.sparql.engine.http.HttpQuery.execGet(HttpQuery.java:358)
at com.hp.hpl.jena.sparql.engine.http.HttpQuery.exec(HttpQuery.java:295)
at com.hp.hpl.jena.sparql.engine.http.QueryEngineHTTP.execResultSetInner(QueryEngineHTTP.java:346)
at com.hp.hpl.jena.sparql.engine.http.QueryEngineHTTP.execSelect(QueryEngineHTTP.java:338)
<own trace items>
...
Caused by: java.net.BindException: Address already in use: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
...
------
Is this a known problem? It would be nice to have some solution because I prefer not to go back to embedded TDB, where this error probably does not occur (?).
thanks in advance, Andreas
Re: java.net.BindException during query evaluation
Posted by Andy Seaborne <an...@apache.org>.
On 13/10/15 10:41, Billig, Andreas wrote:
> Dear Andy, great, thanks a lot.
>
> - HttpOp.setDefaultHttpClient(HttpClient);
> works perfectly (therefore I used it directly without taking care of my environment; If you still want my environment info, let me know). Again, very good framework with anticipatory API, very good support.
> I've also discussed the problem with colleagues and they said it's an mature http/1 issue (on every machine (windows or linux)) that will be obsolete with http/2 (->SPDY).
The default behaviour of the Jena code is to open a new connection each
time. It does not, by default, use cached connections; that leads to a
different problem where the cached connection is kept open in the client
and leads to a different cause of exhaustion of resources. And, as you
point out, concurrency is the issue with cached connections. The Jena
code is robust in the common case - hitting the server hard repeatedly
from a single thread is not the common case :-).
Yes, http/2 would multiplex the connection and make it much easier to
get request connection reuse and threading all working.
>
> What do you think: Will it be possible to initially set a http client "globally" separately for each thread (via ThreadLocal) ?
I don't know any reason why that could not be added if someone sent a
pull request.
It will have to deal with the issue about thread locals that you have to
explicitly delete them or else you get a build up of state.
Personally, rather than ThreadLocals in HttpOp, I'd look at allowing the
HttpClient for the execution to be provided externally HttpQuery not
just use the HttpOp.getDefaultHttpClient() all the time. Maybe add
constructors in QueryEngineHTTP, or setter style, to take a HttpClient
and pass to a new HttpQuery(string, HttpClient).
Andy
>
> Ciao, Andreas
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: Andy Seaborne [mailto:andy@apache.org]
> Gesendet: Sonntag, 11. Oktober 2015 12:32
> An: users@jena.apache.org
> Betreff: Re: java.net.BindException during query evaluation
>
> Hi,
>
> It's not clear what's happening - could you say some more about your environment?
>
> Is it always at the same point?
> Which OS? (Windows?)
> Which version of Jena?
> Which version of Java?
> What's the hardware? How many cores?
>
>
> Lots on stackoverflow but this looks close:
>
> http://stackoverflow.com/questions/4708649/java-net-bindexception-address-already-in-use-when-trying-to-do-rapid-socket
>
>
> On Linux:
>
> The Jena tests provoked a similar situation where we saw TCP sockets not getting recycled fast enough when the CPUs are all busy.
>
> Programs run into problems when opening/closing sockets at high frequency for a sustained period of time when the server is on the same system as the client, as it is here.
>
> If so, put a short wait in your loop.
>
> Run "netstat -t" ; if you see a lot of connections and they go away after a couple of minutes, then its zombie connections.
>
> It may help to reconfigure the low level HTTP setup with HttpOp.setDefaultHttpClient(HttpClient); Remote SPARQL uses the Jena default but that may not fix the issue.
>
> TDB in the same JVM does not use network connections.
>
> Andy
>
>
>
> On 11/10/15 10:40, Billig, Andreas wrote:
>> Hello team,
>>
>> we are developing a constraints checker (schema-check under Closed World Assumption (OWL, RDFS, etc.)) for a quad store and implement it via SPARQL/HTTP.
>> We ran into java.net.BindException during query evaluation.
>>
>> I have isolated the code and made an abstraction. So now with the
>> following code
>>
>> -----
>> for (int i = 0; i < 20000; i++) { // TODO (i:~16300)
>> QueryExecution qe =
>>
>> QueryExecutionFactory.sparqlService("http://localhost:3030/Yyy/query",
>> "select * {?x ?y ?z}"); // only two solutions to keep the result set
>> small
>>
>> System.out.println(i);
>>
>> ResultSet rs = qe.execSelect();
>>
>> while (rs.hasNext()) {
>> System.out.println(rs.next());
>> }
>>
>> qe.close();
>> }
>> -----
>>
>> I got an exception around i > 16000 (so may be some queuing is
>> involved) with trace
>>
>> ------
>> HttpException: java.net.BindException: Address already in use: connect: Unexpected error making the query: java.net.BindException: Address already in use: connect
>> at com.hp.hpl.jena.sparql.engine.http.HttpQuery.rewrap(HttpQuery.java:417)
>> at com.hp.hpl.jena.sparql.engine.http.HttpQuery.execGet(HttpQuery.java:358)
>> at com.hp.hpl.jena.sparql.engine.http.HttpQuery.exec(HttpQuery.java:295)
>> at com.hp.hpl.jena.sparql.engine.http.QueryEngineHTTP.execResultSetInner(QueryEngineHTTP.java:346)
>> at com.hp.hpl.jena.sparql.engine.http.QueryEngineHTTP.execSelect(QueryEngineHTTP.java:338)
>> <own trace items>
>> ...
>> Caused by: java.net.BindException: Address already in use: connect
>> at java.net.DualStackPlainSocketImpl.connect0(Native Method)
>> at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)
>> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
>> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
>> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
>> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>> at java.net.Socket.connect(Socket.java:579)
>> ...
>> ------
>>
>> Is this a known problem? It would be nice to have some solution because I prefer not to go back to embedded TDB, where this error probably does not occur (?).
>>
>> thanks in advance, Andreas
>>
>>
>>
>
java.net.BindException during query evaluation
Posted by "Billig, Andreas" <an...@fokus.fraunhofer.de>.
Dear Andy, great, thanks a lot.
- HttpOp.setDefaultHttpClient(HttpClient);
works perfectly (therefore I used it directly without taking care of my environment; If you still want my environment info, let me know). Again, very good framework with anticipatory API, very good support.
I've also discussed the problem with colleagues and they said it's an mature http/1 issue (on every machine (windows or linux)) that will be obsolete with http/2 (->SPDY).
What do you think: Will it be possible to initially set a http client "globally" separately for each thread (via ThreadLocal) ?
Ciao, Andreas
-----Ursprüngliche Nachricht-----
Von: Andy Seaborne [mailto:andy@apache.org]
Gesendet: Sonntag, 11. Oktober 2015 12:32
An: users@jena.apache.org
Betreff: Re: java.net.BindException during query evaluation
Hi,
It's not clear what's happening - could you say some more about your environment?
Is it always at the same point?
Which OS? (Windows?)
Which version of Jena?
Which version of Java?
What's the hardware? How many cores?
Lots on stackoverflow but this looks close:
http://stackoverflow.com/questions/4708649/java-net-bindexception-address-already-in-use-when-trying-to-do-rapid-socket
On Linux:
The Jena tests provoked a similar situation where we saw TCP sockets not getting recycled fast enough when the CPUs are all busy.
Programs run into problems when opening/closing sockets at high frequency for a sustained period of time when the server is on the same system as the client, as it is here.
If so, put a short wait in your loop.
Run "netstat -t" ; if you see a lot of connections and they go away after a couple of minutes, then its zombie connections.
It may help to reconfigure the low level HTTP setup with HttpOp.setDefaultHttpClient(HttpClient); Remote SPARQL uses the Jena default but that may not fix the issue.
TDB in the same JVM does not use network connections.
Andy
On 11/10/15 10:40, Billig, Andreas wrote:
> Hello team,
>
> we are developing a constraints checker (schema-check under Closed World Assumption (OWL, RDFS, etc.)) for a quad store and implement it via SPARQL/HTTP.
> We ran into java.net.BindException during query evaluation.
>
> I have isolated the code and made an abstraction. So now with the
> following code
>
> -----
> for (int i = 0; i < 20000; i++) { // TODO (i:~16300)
> QueryExecution qe =
>
> QueryExecutionFactory.sparqlService("http://localhost:3030/Yyy/query",
> "select * {?x ?y ?z}"); // only two solutions to keep the result set
> small
>
> System.out.println(i);
>
> ResultSet rs = qe.execSelect();
>
> while (rs.hasNext()) {
> System.out.println(rs.next());
> }
>
> qe.close();
> }
> -----
>
> I got an exception around i > 16000 (so may be some queuing is
> involved) with trace
>
> ------
> HttpException: java.net.BindException: Address already in use: connect: Unexpected error making the query: java.net.BindException: Address already in use: connect
> at com.hp.hpl.jena.sparql.engine.http.HttpQuery.rewrap(HttpQuery.java:417)
> at com.hp.hpl.jena.sparql.engine.http.HttpQuery.execGet(HttpQuery.java:358)
> at com.hp.hpl.jena.sparql.engine.http.HttpQuery.exec(HttpQuery.java:295)
> at com.hp.hpl.jena.sparql.engine.http.QueryEngineHTTP.execResultSetInner(QueryEngineHTTP.java:346)
> at com.hp.hpl.jena.sparql.engine.http.QueryEngineHTTP.execSelect(QueryEngineHTTP.java:338)
> <own trace items>
> ...
> Caused by: java.net.BindException: Address already in use: connect
> at java.net.DualStackPlainSocketImpl.connect0(Native Method)
> at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)
> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:579)
> ...
> ------
>
> Is this a known problem? It would be nice to have some solution because I prefer not to go back to embedded TDB, where this error probably does not occur (?).
>
> thanks in advance, Andreas
>
>
>
Re: java.net.BindException during query evaluation
Posted by Andy Seaborne <an...@apache.org>.
Hi,
It's not clear what's happening - could you say some more about your
environment?
Is it always at the same point?
Which OS? (Windows?)
Which version of Jena?
Which version of Java?
What's the hardware? How many cores?
Lots on stackoverflow but this looks close:
http://stackoverflow.com/questions/4708649/java-net-bindexception-address-already-in-use-when-trying-to-do-rapid-socket
On Linux:
The Jena tests provoked a similar situation where we saw TCP sockets not
getting recycled fast enough when the CPUs are all busy.
Programs run into problems when opening/closing sockets at high
frequency for a sustained period of time when the server is on the same
system as the client, as it is here.
If so, put a short wait in your loop.
Run "netstat -t" ; if you see a lot of connections and they go away
after a couple of minutes, then its zombie connections.
It may help to reconfigure the low level HTTP setup with
HttpOp.setDefaultHttpClient(HttpClient); Remote SPARQL uses the Jena
default but that may not fix the issue.
TDB in the same JVM does not use network connections.
Andy
On 11/10/15 10:40, Billig, Andreas wrote:
> Hello team,
>
> we are developing a constraints checker (schema-check under Closed World Assumption (OWL, RDFS, etc.)) for a quad store and implement it via SPARQL/HTTP.
> We ran into java.net.BindException during query evaluation.
>
> I have isolated the code and made an abstraction. So now with the following code
>
> -----
> for (int i = 0; i < 20000; i++) { // TODO (i:~16300)
> QueryExecution qe =
> QueryExecutionFactory.sparqlService("http://localhost:3030/Yyy/query", "select * {?x ?y ?z}"); // only two solutions to keep the result set small
>
> System.out.println(i);
>
> ResultSet rs = qe.execSelect();
>
> while (rs.hasNext()) {
> System.out.println(rs.next());
> }
>
> qe.close();
> }
> -----
>
> I got an exception around i > 16000 (so may be some queuing is involved) with trace
>
> ------
> HttpException: java.net.BindException: Address already in use: connect: Unexpected error making the query: java.net.BindException: Address already in use: connect
> at com.hp.hpl.jena.sparql.engine.http.HttpQuery.rewrap(HttpQuery.java:417)
> at com.hp.hpl.jena.sparql.engine.http.HttpQuery.execGet(HttpQuery.java:358)
> at com.hp.hpl.jena.sparql.engine.http.HttpQuery.exec(HttpQuery.java:295)
> at com.hp.hpl.jena.sparql.engine.http.QueryEngineHTTP.execResultSetInner(QueryEngineHTTP.java:346)
> at com.hp.hpl.jena.sparql.engine.http.QueryEngineHTTP.execSelect(QueryEngineHTTP.java:338)
> <own trace items>
> ...
> Caused by: java.net.BindException: Address already in use: connect
> at java.net.DualStackPlainSocketImpl.connect0(Native Method)
> at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)
> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:579)
> ...
> ------
>
> Is this a known problem? It would be nice to have some solution because I prefer not to go back to embedded TDB, where this error probably does not occur (?).
>
> thanks in advance, Andreas
>
>
>