You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Sebastian Trueg <tr...@truegeex.de> on 2019/10/31 10:30:24 UTC
SocketException with RDFConnection and models > 100 triples
Hi everyone,
trying to use RDFConnection with Jena 3.13.1 to put a Model into a
remote Fuseki instance I encountered very strange behavior. First off,
let me show my very simple code:
try(RDFConnection conn
= RDFConnectionFactory.connectPW(datasetUrl, "admin", "admin")) {
conn.put(graphUri, model);
}
This works fine on its own and for very small models in general. But as
soon as I repeat the exact same snippet of code, ie. run the same try
block twice I get a SocketException (Broken pipe) on the first call to
RDFConnection::put.
So, to sum up:
- Single put works fine.
- A subsequent call to put will result in the first one already throwing
an exception!
- Using a model with less than 100 triples results in both put
operations to succeed.
- In all this the Fuseki instance keeps on working.
Any ideas?
Regards,
Sebastian
Re: SocketException with RDFConnection and models > 100 triples
Posted by Sebastian Trueg <tr...@truegeex.de>.
One more interesting thing:
Using RDFConnection the Fuseki logs:
Body: Content-Length=-1, Content-Type=application/n-triples,
Charset=null => N-Triples : Count=2500 Triples=2500 Quads=0
While with my manual OkHttp implementation it logs:
Body: Content-Length=83755, Content-Type=text/turtle, Charset=utf-8 =>
Turtle : Count=657 Triples=657 Quads=0
I find it interesting that the Content-Length is set to -1 via
RDFConnection.
Regards,
Sebastian
On 31.10.19 14:14, Andy Seaborne wrote:
> Thanks.
>
> Presumably it works if you create the RDFConnection once and reuse the
> java object?
>
> Andy
>
> On 31/10/2019 12:10, Sebastian Trueg wrote:
>> Hi Andy,
>>
>> - Fuseki started via "fuseki-server".
>> - shiro config attached
>> - "pdm-data-model" dataset created via attached config
>> - Simple test app attached which takes the fuseki dataset url as
>> parameter.
>>
>> Hope this helps.
>>
>> Regards,
>> Sebastian
>>
>> On 31.10.19 11:55, Andy Seaborne wrote:
>>> Sebastian,
>>>
>>> Do you have a complete, minimal example, including the Fuseki setup for
>>> the user/password. Which Fuseki variant? war? full-jar? main?
>>>
>>> Andy
>>>
>>> PS Don't forget teh javadoc on connectPW : it's "basic auth"
>>>
>>> On 31/10/2019 10:30, Sebastian Trueg wrote:
>>>> Hi everyone,
>>>>
>>>> trying to use RDFConnection with Jena 3.13.1 to put a Model into a
>>>> remote Fuseki instance I encountered very strange behavior. First off,
>>>> let me show my very simple code:
>>>>
>>>> try(RDFConnection conn
>>>> = RDFConnectionFactory.connectPW(datasetUrl, "admin", "admin")) {
>>>> conn.put(graphUri, model);
>>>> }
>>>>
>>>> This works fine on its own and for very small models in general. But as
>>>> soon as I repeat the exact same snippet of code, ie. run the same try
>>>> block twice I get a SocketException (Broken pipe) on the first call to
>>>> RDFConnection::put.
>>>>
>>>> So, to sum up:
>>>> - Single put works fine.
>>>> - A subsequent call to put will result in the first one already
>>>> throwing
>>>> an exception!
>>>> - Using a model with less than 100 triples results in both put
>>>> operations to succeed.
>>>> - In all this the Fuseki instance keeps on working.
>>>>
>>>> Any ideas?
>>>>
>>>> Regards,
>>>> Sebastian
>>>>
>>
--
Sebastian Trueg
Managing Director
TrueGeeX UG (haftungsbeschränkt)
trueg@truegeex.de
http://www.linkedin.com/in/trueg
Mobile: 0049 1762 3244 664
Re: SocketException with RDFConnection and models > 100 triples
Posted by Sebastian Trueg <tr...@truegeex.de>.
You have nothing to be sorry for. I am actually very happy about your
great feedback and help.
I am not working today since it is a holiday in Germany, but I will see
if I can find a few minutes to test it anyway.
Regards,
Sebastian
On 01.11.19 10:57, Andy Seaborne wrote:
> Switching to putting in the length shoudl work. As this is for models
> (not sending large files), buffering might be the better choice after all.
>
> The below worked in limited testing.
>
> Sorry to rush this ...
>
> Andy
>
> RDFConnection:
>
> /** Create an HttpEntity for the graph */
> protected HttpEntity graphToHttpEntity(Graph graph, RDFFormat syntax) {
> if ( true )
> return graphToHttpEntityWithLength(graph, syntax);
>
> EntityTemplate entity = new
> EntityTemplate((out)->RDFDataMgr.write(out, graph, syntax));
> String ct = syntax.getLang().getContentType().getContentType();
> entity.setContentType(ct);
> return entity;
> }
>
> /** Create an HttpEntity for the graph */
> protected HttpEntity graphToHttpEntityWithLength(Graph graph, RDFFormat
> syntax) {
> String ct = syntax.getLang().getContentType().getContentType();
> ByteArrayOutputStream out = new ByteArrayOutputStream(128*1024);
> RDFDataMgr.write(out, graph, syntax);
> ByteArrayEntity entity = new ByteArrayEntity(out.toByteArray());
> entity.setContentType(ct);
> return entity;
> }
>
>
>
>
> On 01/11/2019 07:49, Sebastian Trueg wrote:
>> ah, interesting. I will dig a bit deeper then, maybe I will be able to
>> help. Thank you.
>>
>> On 31.10.19 23:26, Andy Seaborne wrote:
>>> Yes, Jena bug, possibly related to the -1 length.
>>>
>>> Recorded as:
>>> https://issues.apache.org/jira/browse/JENA-1776
>>>
>>> -1 is because Jena streams the content for a GSP PUT operation. It does
>>> not know the length at the start of request. To get the length you have
>>> to produce the content then send the request - which is not streaming. A
>>> "feature" of HTTP.
>>>
>>> So I'm guessing that the connection is not handled like the default (no
>>> auth) version in some way but I don't know why.
>>>
>>> RDFConnectionRemote has a builder and connectPW is just this:
>>>
>>>
>>> public static RDFConnection connectPW(String URL, String user, String
>>> password) {
>>> BasicCredentialsProvider credsProvider =
>>> new BasicCredentialsProvider();
>>> Credentials credentials =
>>> new UsernamePasswordCredentials(user, password);
>>> credsProvider.setCredentials(AuthScope.ANY, credentials);
>>> HttpClient client = HttpClients.custom()
>>> .setDefaultCredentialsProvider(credsProvider)
>>> .build();
>>>
>>> return RDFConnectionRemote.create()
>>> .destination(URL)
>>> .httpClient(client)
>>> .build();
>>> }
>>>
>>> so it is a matter for setting up the HttpClient correctly (setting it up
>>> so it is closed after each use? This is the compromise HTTP requires -
>>> stream once, or buffer and send length with connection reuse).
>>>
>>> The normal, no auth default is
>>> HttpOp.createPoolingHttpClientBuilder
>>>
>>> Andy
>>>
>>> FYI: I'm away, no development machine (if I even have internet access,
>>> and that's not certain), for a week or so.
>>>
>>>
>>> On 31/10/2019 14:45, Sebastian Trueg wrote:
>>>> So... a Jena bug then? Do you have an idea if there is something I can
>>>> configure in the HttpClient as a workaround?
>>>>
>>>>
>>>> On 31.10.19 15:18, Andy Seaborne wrote:
>>>>> I can make it happen sporadically using Fuseki main (so no shiro,
>>>>> not a
>>>>> webapp environment). It happens maybe one in four test runs.
>>>>>
>>>>> It looks like a client-side problem - it creates the HTTP connection
>>>>> each time and maybe something is cached in HttpClient. A hash-map-ism
>>>>> would explain the "sporadically". If so, the # triples effect maybe
>>>>> causing the timing to get changed a little.
>>>>>
>>>>> Andy
>>>>>
>>>>> On 31/10/2019 13:27, Sebastian Trueg wrote:
>>>>>> Interestingly it does not. Only going down to a really small
>>>>>> number of
>>>>>> triples does work.
>>>>>>
>>>>>> And to make sure I did implement manual HTTP DELETE+PUT via OkHttp
>>>>>> which
>>>>>> works without problems on the same Fuseki instance.
>>>>>>
>>>>>> Regards,
>>>>>> Sebastian
>>>>>>
>>>>>> On 31.10.19 14:14, Andy Seaborne wrote:
>>>>>>> Thanks.
>>>>>>>
>>>>>>> Presumably it works if you create the RDFConnection once and
>>>>>>> reuse the
>>>>>>> java object?
>>>>>>>
>>>>>>> Andy
>>>>>>>
>>>>>>> On 31/10/2019 12:10, Sebastian Trueg wrote:
>>>>>>>> Hi Andy,
>>>>>>>>
>>>>>>>> - Fuseki started via "fuseki-server".
>>>>>>>> - shiro config attached
>>>>>>>> - "pdm-data-model" dataset created via attached config
>>>>>>>> - Simple test app attached which takes the fuseki dataset url as
>>>>>>>> parameter.
>>>>>>>>
>>>>>>>> Hope this helps.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Sebastian
>>>>>>>>
>>>>>>>> On 31.10.19 11:55, Andy Seaborne wrote:
>>>>>>>>> Sebastian,
>>>>>>>>>
>>>>>>>>> Do you have a complete, minimal example, including the Fuseki
>>>>>>>>> setup
>>>>>>>>> for
>>>>>>>>> the user/password. Which Fuseki variant? war? full-jar? main?
>>>>>>>>>
>>>>>>>>> Andy
>>>>>>>>>
>>>>>>>>> PS Don't forget teh javadoc on connectPW : it's "basic auth"
>>>>>>>>>
>>>>>>>>> On 31/10/2019 10:30, Sebastian Trueg wrote:
>>>>>>>>>> Hi everyone,
>>>>>>>>>>
>>>>>>>>>> trying to use RDFConnection with Jena 3.13.1 to put a Model
>>>>>>>>>> into a
>>>>>>>>>> remote Fuseki instance I encountered very strange behavior. First
>>>>>>>>>> off,
>>>>>>>>>> let me show my very simple code:
>>>>>>>>>>
>>>>>>>>>> try(RDFConnection conn
>>>>>>>>>> = RDFConnectionFactory.connectPW(datasetUrl, "admin",
>>>>>>>>>> "admin")) {
>>>>>>>>>> conn.put(graphUri, model);
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> This works fine on its own and for very small models in general.
>>>>>>>>>> But as
>>>>>>>>>> soon as I repeat the exact same snippet of code, ie. run the
>>>>>>>>>> same try
>>>>>>>>>> block twice I get a SocketException (Broken pipe) on the first
>>>>>>>>>> call to
>>>>>>>>>> RDFConnection::put.
>>>>>>>>>>
>>>>>>>>>> So, to sum up:
>>>>>>>>>> - Single put works fine.
>>>>>>>>>> - A subsequent call to put will result in the first one already
>>>>>>>>>> throwing
>>>>>>>>>> an exception!
>>>>>>>>>> - Using a model with less than 100 triples results in both put
>>>>>>>>>> operations to succeed.
>>>>>>>>>> - In all this the Fuseki instance keeps on working.
>>>>>>>>>>
>>>>>>>>>> Any ideas?
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Sebastian
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>
--
Sebastian Trueg
Managing Director
TrueGeeX UG (haftungsbeschränkt)
trueg@truegeex.de
http://www.linkedin.com/in/trueg
Mobile: 0049 1762 3244 664
Re: SocketException with RDFConnection and models > 100 triples
Posted by Sebastian Trueg <tr...@truegeex.de>.
Hi Andy,
a follow-up to the problem: In order to successfully PUT bigger graphs
(roughly around 2k triples) I needed to enable expect-continue on the
request:
HttpOp.setRequestTransformer(req -> {
if (req instanceof HttpPut) {
((HttpPut) req).setConfig(RequestConfig.custom()
.setExpectContinueEnabled(true)
.build());
}
return req;
}
);
Otherwise I would constantly get a "broken pipe".
Hope this helps.
Regards,
Sebastian
On 01.11.19 10:57, Andy Seaborne wrote:
> Switching to putting in the length shoudl work. As this is for models
> (not sending large files), buffering might be the better choice after all.
>
> The below worked in limited testing.
>
> Sorry to rush this ...
>
> Andy
>
> RDFConnection:
>
> /** Create an HttpEntity for the graph */
> protected HttpEntity graphToHttpEntity(Graph graph, RDFFormat syntax) {
> if ( true )
> return graphToHttpEntityWithLength(graph, syntax);
>
> EntityTemplate entity = new
> EntityTemplate((out)->RDFDataMgr.write(out, graph, syntax));
> String ct = syntax.getLang().getContentType().getContentType();
> entity.setContentType(ct);
> return entity;
> }
>
> /** Create an HttpEntity for the graph */
> protected HttpEntity graphToHttpEntityWithLength(Graph graph, RDFFormat
> syntax) {
> String ct = syntax.getLang().getContentType().getContentType();
> ByteArrayOutputStream out = new ByteArrayOutputStream(128*1024);
> RDFDataMgr.write(out, graph, syntax);
> ByteArrayEntity entity = new ByteArrayEntity(out.toByteArray());
> entity.setContentType(ct);
> return entity;
> }
>
>
>
>
> On 01/11/2019 07:49, Sebastian Trueg wrote:
>> ah, interesting. I will dig a bit deeper then, maybe I will be able to
>> help. Thank you.
>>
>> On 31.10.19 23:26, Andy Seaborne wrote:
>>> Yes, Jena bug, possibly related to the -1 length.
>>>
>>> Recorded as:
>>> https://issues.apache.org/jira/browse/JENA-1776
>>>
>>> -1 is because Jena streams the content for a GSP PUT operation. It does
>>> not know the length at the start of request. To get the length you have
>>> to produce the content then send the request - which is not streaming. A
>>> "feature" of HTTP.
>>>
>>> So I'm guessing that the connection is not handled like the default (no
>>> auth) version in some way but I don't know why.
>>>
>>> RDFConnectionRemote has a builder and connectPW is just this:
>>>
>>>
>>> public static RDFConnection connectPW(String URL, String user, String
>>> password) {
>>> BasicCredentialsProvider credsProvider =
>>> new BasicCredentialsProvider();
>>> Credentials credentials =
>>> new UsernamePasswordCredentials(user, password);
>>> credsProvider.setCredentials(AuthScope.ANY, credentials);
>>> HttpClient client = HttpClients.custom()
>>> .setDefaultCredentialsProvider(credsProvider)
>>> .build();
>>>
>>> return RDFConnectionRemote.create()
>>> .destination(URL)
>>> .httpClient(client)
>>> .build();
>>> }
>>>
>>> so it is a matter for setting up the HttpClient correctly (setting it up
>>> so it is closed after each use? This is the compromise HTTP requires -
>>> stream once, or buffer and send length with connection reuse).
>>>
>>> The normal, no auth default is
>>> HttpOp.createPoolingHttpClientBuilder
>>>
>>> Andy
>>>
>>> FYI: I'm away, no development machine (if I even have internet access,
>>> and that's not certain), for a week or so.
>>>
>>>
>>> On 31/10/2019 14:45, Sebastian Trueg wrote:
>>>> So... a Jena bug then? Do you have an idea if there is something I can
>>>> configure in the HttpClient as a workaround?
>>>>
>>>>
>>>> On 31.10.19 15:18, Andy Seaborne wrote:
>>>>> I can make it happen sporadically using Fuseki main (so no shiro,
>>>>> not a
>>>>> webapp environment). It happens maybe one in four test runs.
>>>>>
>>>>> It looks like a client-side problem - it creates the HTTP connection
>>>>> each time and maybe something is cached in HttpClient. A hash-map-ism
>>>>> would explain the "sporadically". If so, the # triples effect maybe
>>>>> causing the timing to get changed a little.
>>>>>
>>>>> Andy
>>>>>
>>>>> On 31/10/2019 13:27, Sebastian Trueg wrote:
>>>>>> Interestingly it does not. Only going down to a really small
>>>>>> number of
>>>>>> triples does work.
>>>>>>
>>>>>> And to make sure I did implement manual HTTP DELETE+PUT via OkHttp
>>>>>> which
>>>>>> works without problems on the same Fuseki instance.
>>>>>>
>>>>>> Regards,
>>>>>> Sebastian
>>>>>>
>>>>>> On 31.10.19 14:14, Andy Seaborne wrote:
>>>>>>> Thanks.
>>>>>>>
>>>>>>> Presumably it works if you create the RDFConnection once and
>>>>>>> reuse the
>>>>>>> java object?
>>>>>>>
>>>>>>> Andy
>>>>>>>
>>>>>>> On 31/10/2019 12:10, Sebastian Trueg wrote:
>>>>>>>> Hi Andy,
>>>>>>>>
>>>>>>>> - Fuseki started via "fuseki-server".
>>>>>>>> - shiro config attached
>>>>>>>> - "pdm-data-model" dataset created via attached config
>>>>>>>> - Simple test app attached which takes the fuseki dataset url as
>>>>>>>> parameter.
>>>>>>>>
>>>>>>>> Hope this helps.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Sebastian
>>>>>>>>
>>>>>>>> On 31.10.19 11:55, Andy Seaborne wrote:
>>>>>>>>> Sebastian,
>>>>>>>>>
>>>>>>>>> Do you have a complete, minimal example, including the Fuseki
>>>>>>>>> setup
>>>>>>>>> for
>>>>>>>>> the user/password. Which Fuseki variant? war? full-jar? main?
>>>>>>>>>
>>>>>>>>> Andy
>>>>>>>>>
>>>>>>>>> PS Don't forget teh javadoc on connectPW : it's "basic auth"
>>>>>>>>>
>>>>>>>>> On 31/10/2019 10:30, Sebastian Trueg wrote:
>>>>>>>>>> Hi everyone,
>>>>>>>>>>
>>>>>>>>>> trying to use RDFConnection with Jena 3.13.1 to put a Model
>>>>>>>>>> into a
>>>>>>>>>> remote Fuseki instance I encountered very strange behavior. First
>>>>>>>>>> off,
>>>>>>>>>> let me show my very simple code:
>>>>>>>>>>
>>>>>>>>>> try(RDFConnection conn
>>>>>>>>>> = RDFConnectionFactory.connectPW(datasetUrl, "admin",
>>>>>>>>>> "admin")) {
>>>>>>>>>> conn.put(graphUri, model);
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> This works fine on its own and for very small models in general.
>>>>>>>>>> But as
>>>>>>>>>> soon as I repeat the exact same snippet of code, ie. run the
>>>>>>>>>> same try
>>>>>>>>>> block twice I get a SocketException (Broken pipe) on the first
>>>>>>>>>> call to
>>>>>>>>>> RDFConnection::put.
>>>>>>>>>>
>>>>>>>>>> So, to sum up:
>>>>>>>>>> - Single put works fine.
>>>>>>>>>> - A subsequent call to put will result in the first one already
>>>>>>>>>> throwing
>>>>>>>>>> an exception!
>>>>>>>>>> - Using a model with less than 100 triples results in both put
>>>>>>>>>> operations to succeed.
>>>>>>>>>> - In all this the Fuseki instance keeps on working.
>>>>>>>>>>
>>>>>>>>>> Any ideas?
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Sebastian
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>
--
Sebastian Trueg
Managing Director
TrueGeeX UG (haftungsbeschränkt)
trueg@truegeex.de
http://www.linkedin.com/in/trueg
Mobile: 0049 1762 3244 664
Re: SocketException with RDFConnection and models > 100 triples
Posted by Andy Seaborne <an...@apache.org>.
On 05/11/2019 14:43, Sebastian Trueg wrote:
> ok, it took me a while but now I tested it. With your proposed changes
> the graphs are properly written.
Thnaks -
There is a PR #628 and a JIRA JENA-1776 now.
Andy
> I simply wrote a wrapper around
> RDFConnectionRemove and RDFConnectionRemoteBuilder:
>
> public class RDFConnectionRemoteWithLength extends RDFConnectionRemote {
> protected RDFConnectionRemoteWithLength(Transactional txnLifecycle,
> HttpClient httpClient, HttpContext httpContext, String destination,
> String queryURL, String updateURL, String gspURL, RDFFormat outputQuads,
> RDFFormat outputTriples, String acceptDataset, String acceptGraph,
> String acceptSparqlResults, String acceptSelectResult, String
> acceptAskResult, boolean parseCheckQueries, boolean parseCheckUpdates) {
> super(txnLifecycle, httpClient, httpContext, destination,
> queryURL, updateURL, gspURL, outputQuads, outputTriples, acceptDataset,
> acceptGraph, acceptSparqlResults, acceptSelectResult, acceptAskResult,
> parseCheckQueries, parseCheckUpdates);
> }
>
> public static RDFConnectionRemoteBuilder create() {
> return new RDFConnectioRemoteWithLengthBuilder();
> }
>
> @Override
> protected HttpEntity graphToHttpEntity(Graph graph, RDFFormat syntax) {
> return graphToHttpEntityWithLength(graph, syntax);
> }
>
> protected HttpEntity graphToHttpEntityWithLength(Graph graph,
> RDFFormat syntax) {
> String ct = syntax.getLang().getContentType().getContentType();
> ByteArrayOutputStream out = new ByteArrayOutputStream(128*1024);
> RDFDataMgr.write(out, graph, syntax);
> ByteArrayEntity entity = new ByteArrayEntity(out.toByteArray());
> entity.setContentType(ct);
> return entity;
> }
> }
>
> public class RDFConnectioRemoteWithLengthBuilder extends
> RDFConnectionRemoteBuilder {
> @Override
> protected RDFConnectionRemote buildConnection() {
> return new RDFConnectionRemoteWithLength(txnLifecycle,
> httpClient, httpContext,
> destination, queryURL, updateURL, gspURL,
> outputQuads, outputTriples,
> acceptDataset, acceptGraph,
> acceptSparqlResults, acceptSelectResult, acceptAskResult,
> parseCheckQueries, parseCheckUpdates);
> }
> }
>
> Thanks a lot.
>
> Regards,
> Sebastian
>
> On 01.11.19 10:57, Andy Seaborne wrote:
>> Switching to putting in the length shoudl work. As this is for models
>> (not sending large files), buffering might be the better choice after all.
>>
>> The below worked in limited testing.
>>
>> Sorry to rush this ...
>>
>> Andy
>>
>> RDFConnection:
>>
>> /** Create an HttpEntity for the graph */
>> protected HttpEntity graphToHttpEntity(Graph graph, RDFFormat syntax) {
>> if ( true )
>> return graphToHttpEntityWithLength(graph, syntax);
>>
>> EntityTemplate entity = new
>> EntityTemplate((out)->RDFDataMgr.write(out, graph, syntax));
>> String ct = syntax.getLang().getContentType().getContentType();
>> entity.setContentType(ct);
>> return entity;
>> }
>>
>> /** Create an HttpEntity for the graph */
>> protected HttpEntity graphToHttpEntityWithLength(Graph graph, RDFFormat
>> syntax) {
>> String ct = syntax.getLang().getContentType().getContentType();
>> ByteArrayOutputStream out = new ByteArrayOutputStream(128*1024);
>> RDFDataMgr.write(out, graph, syntax);
>> ByteArrayEntity entity = new ByteArrayEntity(out.toByteArray());
>> entity.setContentType(ct);
>> return entity;
>> }
>>
>>
>>
>>
>> On 01/11/2019 07:49, Sebastian Trueg wrote:
>>> ah, interesting. I will dig a bit deeper then, maybe I will be able to
>>> help. Thank you.
>>>
>>> On 31.10.19 23:26, Andy Seaborne wrote:
>>>> Yes, Jena bug, possibly related to the -1 length.
>>>>
>>>> Recorded as:
>>>> https://issues.apache.org/jira/browse/JENA-1776
>>>>
>>>> -1 is because Jena streams the content for a GSP PUT operation. It does
>>>> not know the length at the start of request. To get the length you have
>>>> to produce the content then send the request - which is not streaming. A
>>>> "feature" of HTTP.
>>>>
>>>> So I'm guessing that the connection is not handled like the default (no
>>>> auth) version in some way but I don't know why.
>>>>
>>>> RDFConnectionRemote has a builder and connectPW is just this:
>>>>
>>>>
>>>> public static RDFConnection connectPW(String URL, String user, String
>>>> password) {
>>>> BasicCredentialsProvider credsProvider =
>>>> new BasicCredentialsProvider();
>>>> Credentials credentials =
>>>> new UsernamePasswordCredentials(user, password);
>>>> credsProvider.setCredentials(AuthScope.ANY, credentials);
>>>> HttpClient client = HttpClients.custom()
>>>> .setDefaultCredentialsProvider(credsProvider)
>>>> .build();
>>>>
>>>> return RDFConnectionRemote.create()
>>>> .destination(URL)
>>>> .httpClient(client)
>>>> .build();
>>>> }
>>>>
>>>> so it is a matter for setting up the HttpClient correctly (setting it up
>>>> so it is closed after each use? This is the compromise HTTP requires -
>>>> stream once, or buffer and send length with connection reuse).
>>>>
>>>> The normal, no auth default is
>>>> HttpOp.createPoolingHttpClientBuilder
>>>>
>>>> Andy
>>>>
>>>> FYI: I'm away, no development machine (if I even have internet access,
>>>> and that's not certain), for a week or so.
>>>>
>>>>
>>>> On 31/10/2019 14:45, Sebastian Trueg wrote:
>>>>> So... a Jena bug then? Do you have an idea if there is something I can
>>>>> configure in the HttpClient as a workaround?
>>>>>
>>>>>
>>>>> On 31.10.19 15:18, Andy Seaborne wrote:
>>>>>> I can make it happen sporadically using Fuseki main (so no shiro,
>>>>>> not a
>>>>>> webapp environment). It happens maybe one in four test runs.
>>>>>>
>>>>>> It looks like a client-side problem - it creates the HTTP connection
>>>>>> each time and maybe something is cached in HttpClient. A hash-map-ism
>>>>>> would explain the "sporadically". If so, the # triples effect maybe
>>>>>> causing the timing to get changed a little.
>>>>>>
>>>>>> Andy
>>>>>>
>>>>>> On 31/10/2019 13:27, Sebastian Trueg wrote:
>>>>>>> Interestingly it does not. Only going down to a really small
>>>>>>> number of
>>>>>>> triples does work.
>>>>>>>
>>>>>>> And to make sure I did implement manual HTTP DELETE+PUT via OkHttp
>>>>>>> which
>>>>>>> works without problems on the same Fuseki instance.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Sebastian
>>>>>>>
>>>>>>> On 31.10.19 14:14, Andy Seaborne wrote:
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> Presumably it works if you create the RDFConnection once and
>>>>>>>> reuse the
>>>>>>>> java object?
>>>>>>>>
>>>>>>>> Andy
>>>>>>>>
>>>>>>>> On 31/10/2019 12:10, Sebastian Trueg wrote:
>>>>>>>>> Hi Andy,
>>>>>>>>>
>>>>>>>>> - Fuseki started via "fuseki-server".
>>>>>>>>> - shiro config attached
>>>>>>>>> - "pdm-data-model" dataset created via attached config
>>>>>>>>> - Simple test app attached which takes the fuseki dataset url as
>>>>>>>>> parameter.
>>>>>>>>>
>>>>>>>>> Hope this helps.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Sebastian
>>>>>>>>>
>>>>>>>>> On 31.10.19 11:55, Andy Seaborne wrote:
>>>>>>>>>> Sebastian,
>>>>>>>>>>
>>>>>>>>>> Do you have a complete, minimal example, including the Fuseki
>>>>>>>>>> setup
>>>>>>>>>> for
>>>>>>>>>> the user/password. Which Fuseki variant? war? full-jar? main?
>>>>>>>>>>
>>>>>>>>>> Andy
>>>>>>>>>>
>>>>>>>>>> PS Don't forget teh javadoc on connectPW : it's "basic auth"
>>>>>>>>>>
>>>>>>>>>> On 31/10/2019 10:30, Sebastian Trueg wrote:
>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>
>>>>>>>>>>> trying to use RDFConnection with Jena 3.13.1 to put a Model
>>>>>>>>>>> into a
>>>>>>>>>>> remote Fuseki instance I encountered very strange behavior. First
>>>>>>>>>>> off,
>>>>>>>>>>> let me show my very simple code:
>>>>>>>>>>>
>>>>>>>>>>> try(RDFConnection conn
>>>>>>>>>>> = RDFConnectionFactory.connectPW(datasetUrl, "admin",
>>>>>>>>>>> "admin")) {
>>>>>>>>>>> conn.put(graphUri, model);
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> This works fine on its own and for very small models in general.
>>>>>>>>>>> But as
>>>>>>>>>>> soon as I repeat the exact same snippet of code, ie. run the
>>>>>>>>>>> same try
>>>>>>>>>>> block twice I get a SocketException (Broken pipe) on the first
>>>>>>>>>>> call to
>>>>>>>>>>> RDFConnection::put.
>>>>>>>>>>>
>>>>>>>>>>> So, to sum up:
>>>>>>>>>>> - Single put works fine.
>>>>>>>>>>> - A subsequent call to put will result in the first one already
>>>>>>>>>>> throwing
>>>>>>>>>>> an exception!
>>>>>>>>>>> - Using a model with less than 100 triples results in both put
>>>>>>>>>>> operations to succeed.
>>>>>>>>>>> - In all this the Fuseki instance keeps on working.
>>>>>>>>>>>
>>>>>>>>>>> Any ideas?
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Sebastian
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>>>
>
Re: SocketException with RDFConnection and models > 100 triples
Posted by Sebastian Trueg <tr...@truegeex.de>.
ok, it took me a while but now I tested it. With your proposed changes
the graphs are properly written. I simply wrote a wrapper around
RDFConnectionRemove and RDFConnectionRemoteBuilder:
public class RDFConnectionRemoteWithLength extends RDFConnectionRemote {
protected RDFConnectionRemoteWithLength(Transactional txnLifecycle,
HttpClient httpClient, HttpContext httpContext, String destination,
String queryURL, String updateURL, String gspURL, RDFFormat outputQuads,
RDFFormat outputTriples, String acceptDataset, String acceptGraph,
String acceptSparqlResults, String acceptSelectResult, String
acceptAskResult, boolean parseCheckQueries, boolean parseCheckUpdates) {
super(txnLifecycle, httpClient, httpContext, destination,
queryURL, updateURL, gspURL, outputQuads, outputTriples, acceptDataset,
acceptGraph, acceptSparqlResults, acceptSelectResult, acceptAskResult,
parseCheckQueries, parseCheckUpdates);
}
public static RDFConnectionRemoteBuilder create() {
return new RDFConnectioRemoteWithLengthBuilder();
}
@Override
protected HttpEntity graphToHttpEntity(Graph graph, RDFFormat syntax) {
return graphToHttpEntityWithLength(graph, syntax);
}
protected HttpEntity graphToHttpEntityWithLength(Graph graph,
RDFFormat syntax) {
String ct = syntax.getLang().getContentType().getContentType();
ByteArrayOutputStream out = new ByteArrayOutputStream(128*1024);
RDFDataMgr.write(out, graph, syntax);
ByteArrayEntity entity = new ByteArrayEntity(out.toByteArray());
entity.setContentType(ct);
return entity;
}
}
public class RDFConnectioRemoteWithLengthBuilder extends
RDFConnectionRemoteBuilder {
@Override
protected RDFConnectionRemote buildConnection() {
return new RDFConnectionRemoteWithLength(txnLifecycle,
httpClient, httpContext,
destination, queryURL, updateURL, gspURL,
outputQuads, outputTriples,
acceptDataset, acceptGraph,
acceptSparqlResults, acceptSelectResult, acceptAskResult,
parseCheckQueries, parseCheckUpdates);
}
}
Thanks a lot.
Regards,
Sebastian
On 01.11.19 10:57, Andy Seaborne wrote:
> Switching to putting in the length shoudl work. As this is for models
> (not sending large files), buffering might be the better choice after all.
>
> The below worked in limited testing.
>
> Sorry to rush this ...
>
> Andy
>
> RDFConnection:
>
> /** Create an HttpEntity for the graph */
> protected HttpEntity graphToHttpEntity(Graph graph, RDFFormat syntax) {
> if ( true )
> return graphToHttpEntityWithLength(graph, syntax);
>
> EntityTemplate entity = new
> EntityTemplate((out)->RDFDataMgr.write(out, graph, syntax));
> String ct = syntax.getLang().getContentType().getContentType();
> entity.setContentType(ct);
> return entity;
> }
>
> /** Create an HttpEntity for the graph */
> protected HttpEntity graphToHttpEntityWithLength(Graph graph, RDFFormat
> syntax) {
> String ct = syntax.getLang().getContentType().getContentType();
> ByteArrayOutputStream out = new ByteArrayOutputStream(128*1024);
> RDFDataMgr.write(out, graph, syntax);
> ByteArrayEntity entity = new ByteArrayEntity(out.toByteArray());
> entity.setContentType(ct);
> return entity;
> }
>
>
>
>
> On 01/11/2019 07:49, Sebastian Trueg wrote:
>> ah, interesting. I will dig a bit deeper then, maybe I will be able to
>> help. Thank you.
>>
>> On 31.10.19 23:26, Andy Seaborne wrote:
>>> Yes, Jena bug, possibly related to the -1 length.
>>>
>>> Recorded as:
>>> https://issues.apache.org/jira/browse/JENA-1776
>>>
>>> -1 is because Jena streams the content for a GSP PUT operation. It does
>>> not know the length at the start of request. To get the length you have
>>> to produce the content then send the request - which is not streaming. A
>>> "feature" of HTTP.
>>>
>>> So I'm guessing that the connection is not handled like the default (no
>>> auth) version in some way but I don't know why.
>>>
>>> RDFConnectionRemote has a builder and connectPW is just this:
>>>
>>>
>>> public static RDFConnection connectPW(String URL, String user, String
>>> password) {
>>> BasicCredentialsProvider credsProvider =
>>> new BasicCredentialsProvider();
>>> Credentials credentials =
>>> new UsernamePasswordCredentials(user, password);
>>> credsProvider.setCredentials(AuthScope.ANY, credentials);
>>> HttpClient client = HttpClients.custom()
>>> .setDefaultCredentialsProvider(credsProvider)
>>> .build();
>>>
>>> return RDFConnectionRemote.create()
>>> .destination(URL)
>>> .httpClient(client)
>>> .build();
>>> }
>>>
>>> so it is a matter for setting up the HttpClient correctly (setting it up
>>> so it is closed after each use? This is the compromise HTTP requires -
>>> stream once, or buffer and send length with connection reuse).
>>>
>>> The normal, no auth default is
>>> HttpOp.createPoolingHttpClientBuilder
>>>
>>> Andy
>>>
>>> FYI: I'm away, no development machine (if I even have internet access,
>>> and that's not certain), for a week or so.
>>>
>>>
>>> On 31/10/2019 14:45, Sebastian Trueg wrote:
>>>> So... a Jena bug then? Do you have an idea if there is something I can
>>>> configure in the HttpClient as a workaround?
>>>>
>>>>
>>>> On 31.10.19 15:18, Andy Seaborne wrote:
>>>>> I can make it happen sporadically using Fuseki main (so no shiro,
>>>>> not a
>>>>> webapp environment). It happens maybe one in four test runs.
>>>>>
>>>>> It looks like a client-side problem - it creates the HTTP connection
>>>>> each time and maybe something is cached in HttpClient. A hash-map-ism
>>>>> would explain the "sporadically". If so, the # triples effect maybe
>>>>> causing the timing to get changed a little.
>>>>>
>>>>> Andy
>>>>>
>>>>> On 31/10/2019 13:27, Sebastian Trueg wrote:
>>>>>> Interestingly it does not. Only going down to a really small
>>>>>> number of
>>>>>> triples does work.
>>>>>>
>>>>>> And to make sure I did implement manual HTTP DELETE+PUT via OkHttp
>>>>>> which
>>>>>> works without problems on the same Fuseki instance.
>>>>>>
>>>>>> Regards,
>>>>>> Sebastian
>>>>>>
>>>>>> On 31.10.19 14:14, Andy Seaborne wrote:
>>>>>>> Thanks.
>>>>>>>
>>>>>>> Presumably it works if you create the RDFConnection once and
>>>>>>> reuse the
>>>>>>> java object?
>>>>>>>
>>>>>>> Andy
>>>>>>>
>>>>>>> On 31/10/2019 12:10, Sebastian Trueg wrote:
>>>>>>>> Hi Andy,
>>>>>>>>
>>>>>>>> - Fuseki started via "fuseki-server".
>>>>>>>> - shiro config attached
>>>>>>>> - "pdm-data-model" dataset created via attached config
>>>>>>>> - Simple test app attached which takes the fuseki dataset url as
>>>>>>>> parameter.
>>>>>>>>
>>>>>>>> Hope this helps.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Sebastian
>>>>>>>>
>>>>>>>> On 31.10.19 11:55, Andy Seaborne wrote:
>>>>>>>>> Sebastian,
>>>>>>>>>
>>>>>>>>> Do you have a complete, minimal example, including the Fuseki
>>>>>>>>> setup
>>>>>>>>> for
>>>>>>>>> the user/password. Which Fuseki variant? war? full-jar? main?
>>>>>>>>>
>>>>>>>>> Andy
>>>>>>>>>
>>>>>>>>> PS Don't forget teh javadoc on connectPW : it's "basic auth"
>>>>>>>>>
>>>>>>>>> On 31/10/2019 10:30, Sebastian Trueg wrote:
>>>>>>>>>> Hi everyone,
>>>>>>>>>>
>>>>>>>>>> trying to use RDFConnection with Jena 3.13.1 to put a Model
>>>>>>>>>> into a
>>>>>>>>>> remote Fuseki instance I encountered very strange behavior. First
>>>>>>>>>> off,
>>>>>>>>>> let me show my very simple code:
>>>>>>>>>>
>>>>>>>>>> try(RDFConnection conn
>>>>>>>>>> = RDFConnectionFactory.connectPW(datasetUrl, "admin",
>>>>>>>>>> "admin")) {
>>>>>>>>>> conn.put(graphUri, model);
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> This works fine on its own and for very small models in general.
>>>>>>>>>> But as
>>>>>>>>>> soon as I repeat the exact same snippet of code, ie. run the
>>>>>>>>>> same try
>>>>>>>>>> block twice I get a SocketException (Broken pipe) on the first
>>>>>>>>>> call to
>>>>>>>>>> RDFConnection::put.
>>>>>>>>>>
>>>>>>>>>> So, to sum up:
>>>>>>>>>> - Single put works fine.
>>>>>>>>>> - A subsequent call to put will result in the first one already
>>>>>>>>>> throwing
>>>>>>>>>> an exception!
>>>>>>>>>> - Using a model with less than 100 triples results in both put
>>>>>>>>>> operations to succeed.
>>>>>>>>>> - In all this the Fuseki instance keeps on working.
>>>>>>>>>>
>>>>>>>>>> Any ideas?
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Sebastian
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>
--
Sebastian Trueg
Managing Director
TrueGeeX UG (haftungsbeschränkt)
trueg@truegeex.de
http://www.linkedin.com/in/trueg
Mobile: 0049 1762 3244 664
Re: SocketException with RDFConnection and models > 100 triples
Posted by Andy Seaborne <an...@apache.org>.
Switching to putting in the length shoudl work. As this is for models
(not sending large files), buffering might be the better choice after all.
The below worked in limited testing.
Sorry to rush this ...
Andy
RDFConnection:
/** Create an HttpEntity for the graph */
protected HttpEntity graphToHttpEntity(Graph graph, RDFFormat syntax) {
if ( true )
return graphToHttpEntityWithLength(graph, syntax);
EntityTemplate entity = new
EntityTemplate((out)->RDFDataMgr.write(out, graph, syntax));
String ct = syntax.getLang().getContentType().getContentType();
entity.setContentType(ct);
return entity;
}
/** Create an HttpEntity for the graph */
protected HttpEntity graphToHttpEntityWithLength(Graph graph, RDFFormat
syntax) {
String ct = syntax.getLang().getContentType().getContentType();
ByteArrayOutputStream out = new ByteArrayOutputStream(128*1024);
RDFDataMgr.write(out, graph, syntax);
ByteArrayEntity entity = new ByteArrayEntity(out.toByteArray());
entity.setContentType(ct);
return entity;
}
On 01/11/2019 07:49, Sebastian Trueg wrote:
> ah, interesting. I will dig a bit deeper then, maybe I will be able to
> help. Thank you.
>
> On 31.10.19 23:26, Andy Seaborne wrote:
>> Yes, Jena bug, possibly related to the -1 length.
>>
>> Recorded as:
>> https://issues.apache.org/jira/browse/JENA-1776
>>
>> -1 is because Jena streams the content for a GSP PUT operation. It does
>> not know the length at the start of request. To get the length you have
>> to produce the content then send the request - which is not streaming. A
>> "feature" of HTTP.
>>
>> So I'm guessing that the connection is not handled like the default (no
>> auth) version in some way but I don't know why.
>>
>> RDFConnectionRemote has a builder and connectPW is just this:
>>
>>
>> public static RDFConnection connectPW(String URL, String user, String
>> password) {
>> BasicCredentialsProvider credsProvider =
>> new BasicCredentialsProvider();
>> Credentials credentials =
>> new UsernamePasswordCredentials(user, password);
>> credsProvider.setCredentials(AuthScope.ANY, credentials);
>> HttpClient client = HttpClients.custom()
>> .setDefaultCredentialsProvider(credsProvider)
>> .build();
>>
>> return RDFConnectionRemote.create()
>> .destination(URL)
>> .httpClient(client)
>> .build();
>> }
>>
>> so it is a matter for setting up the HttpClient correctly (setting it up
>> so it is closed after each use? This is the compromise HTTP requires -
>> stream once, or buffer and send length with connection reuse).
>>
>> The normal, no auth default is
>> HttpOp.createPoolingHttpClientBuilder
>>
>> Andy
>>
>> FYI: I'm away, no development machine (if I even have internet access,
>> and that's not certain), for a week or so.
>>
>>
>> On 31/10/2019 14:45, Sebastian Trueg wrote:
>>> So... a Jena bug then? Do you have an idea if there is something I can
>>> configure in the HttpClient as a workaround?
>>>
>>>
>>> On 31.10.19 15:18, Andy Seaborne wrote:
>>>> I can make it happen sporadically using Fuseki main (so no shiro, not a
>>>> webapp environment). It happens maybe one in four test runs.
>>>>
>>>> It looks like a client-side problem - it creates the HTTP connection
>>>> each time and maybe something is cached in HttpClient. A hash-map-ism
>>>> would explain the "sporadically". If so, the # triples effect maybe
>>>> causing the timing to get changed a little.
>>>>
>>>> Andy
>>>>
>>>> On 31/10/2019 13:27, Sebastian Trueg wrote:
>>>>> Interestingly it does not. Only going down to a really small number of
>>>>> triples does work.
>>>>>
>>>>> And to make sure I did implement manual HTTP DELETE+PUT via OkHttp
>>>>> which
>>>>> works without problems on the same Fuseki instance.
>>>>>
>>>>> Regards,
>>>>> Sebastian
>>>>>
>>>>> On 31.10.19 14:14, Andy Seaborne wrote:
>>>>>> Thanks.
>>>>>>
>>>>>> Presumably it works if you create the RDFConnection once and reuse the
>>>>>> java object?
>>>>>>
>>>>>> Andy
>>>>>>
>>>>>> On 31/10/2019 12:10, Sebastian Trueg wrote:
>>>>>>> Hi Andy,
>>>>>>>
>>>>>>> - Fuseki started via "fuseki-server".
>>>>>>> - shiro config attached
>>>>>>> - "pdm-data-model" dataset created via attached config
>>>>>>> - Simple test app attached which takes the fuseki dataset url as
>>>>>>> parameter.
>>>>>>>
>>>>>>> Hope this helps.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Sebastian
>>>>>>>
>>>>>>> On 31.10.19 11:55, Andy Seaborne wrote:
>>>>>>>> Sebastian,
>>>>>>>>
>>>>>>>> Do you have a complete, minimal example, including the Fuseki setup
>>>>>>>> for
>>>>>>>> the user/password. Which Fuseki variant? war? full-jar? main?
>>>>>>>>
>>>>>>>> Andy
>>>>>>>>
>>>>>>>> PS Don't forget teh javadoc on connectPW : it's "basic auth"
>>>>>>>>
>>>>>>>> On 31/10/2019 10:30, Sebastian Trueg wrote:
>>>>>>>>> Hi everyone,
>>>>>>>>>
>>>>>>>>> trying to use RDFConnection with Jena 3.13.1 to put a Model into a
>>>>>>>>> remote Fuseki instance I encountered very strange behavior. First
>>>>>>>>> off,
>>>>>>>>> let me show my very simple code:
>>>>>>>>>
>>>>>>>>> try(RDFConnection conn
>>>>>>>>> = RDFConnectionFactory.connectPW(datasetUrl, "admin",
>>>>>>>>> "admin")) {
>>>>>>>>> conn.put(graphUri, model);
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> This works fine on its own and for very small models in general.
>>>>>>>>> But as
>>>>>>>>> soon as I repeat the exact same snippet of code, ie. run the
>>>>>>>>> same try
>>>>>>>>> block twice I get a SocketException (Broken pipe) on the first
>>>>>>>>> call to
>>>>>>>>> RDFConnection::put.
>>>>>>>>>
>>>>>>>>> So, to sum up:
>>>>>>>>> - Single put works fine.
>>>>>>>>> - A subsequent call to put will result in the first one already
>>>>>>>>> throwing
>>>>>>>>> an exception!
>>>>>>>>> - Using a model with less than 100 triples results in both put
>>>>>>>>> operations to succeed.
>>>>>>>>> - In all this the Fuseki instance keeps on working.
>>>>>>>>>
>>>>>>>>> Any ideas?
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Sebastian
>>>>>>>>>
>>>>>>>
>>>>>
>>>
>
Re: SocketException with RDFConnection and models > 100 triples
Posted by Sebastian Trueg <tr...@truegeex.de>.
ah, interesting. I will dig a bit deeper then, maybe I will be able to
help. Thank you.
On 31.10.19 23:26, Andy Seaborne wrote:
> Yes, Jena bug, possibly related to the -1 length.
>
> Recorded as:
> https://issues.apache.org/jira/browse/JENA-1776
>
> -1 is because Jena streams the content for a GSP PUT operation. It does
> not know the length at the start of request. To get the length you have
> to produce the content then send the request - which is not streaming. A
> "feature" of HTTP.
>
> So I'm guessing that the connection is not handled like the default (no
> auth) version in some way but I don't know why.
>
> RDFConnectionRemote has a builder and connectPW is just this:
>
>
> public static RDFConnection connectPW(String URL, String user, String
> password) {
> BasicCredentialsProvider credsProvider =
> new BasicCredentialsProvider();
> Credentials credentials =
> new UsernamePasswordCredentials(user, password);
> credsProvider.setCredentials(AuthScope.ANY, credentials);
> HttpClient client = HttpClients.custom()
> .setDefaultCredentialsProvider(credsProvider)
> .build();
>
> return RDFConnectionRemote.create()
> .destination(URL)
> .httpClient(client)
> .build();
> }
>
> so it is a matter for setting up the HttpClient correctly (setting it up
> so it is closed after each use? This is the compromise HTTP requires -
> stream once, or buffer and send length with connection reuse).
>
> The normal, no auth default is
> HttpOp.createPoolingHttpClientBuilder
>
> Andy
>
> FYI: I'm away, no development machine (if I even have internet access,
> and that's not certain), for a week or so.
>
>
> On 31/10/2019 14:45, Sebastian Trueg wrote:
>> So... a Jena bug then? Do you have an idea if there is something I can
>> configure in the HttpClient as a workaround?
>>
>>
>> On 31.10.19 15:18, Andy Seaborne wrote:
>>> I can make it happen sporadically using Fuseki main (so no shiro, not a
>>> webapp environment). It happens maybe one in four test runs.
>>>
>>> It looks like a client-side problem - it creates the HTTP connection
>>> each time and maybe something is cached in HttpClient. A hash-map-ism
>>> would explain the "sporadically". If so, the # triples effect maybe
>>> causing the timing to get changed a little.
>>>
>>> Andy
>>>
>>> On 31/10/2019 13:27, Sebastian Trueg wrote:
>>>> Interestingly it does not. Only going down to a really small number of
>>>> triples does work.
>>>>
>>>> And to make sure I did implement manual HTTP DELETE+PUT via OkHttp
>>>> which
>>>> works without problems on the same Fuseki instance.
>>>>
>>>> Regards,
>>>> Sebastian
>>>>
>>>> On 31.10.19 14:14, Andy Seaborne wrote:
>>>>> Thanks.
>>>>>
>>>>> Presumably it works if you create the RDFConnection once and reuse the
>>>>> java object?
>>>>>
>>>>> Andy
>>>>>
>>>>> On 31/10/2019 12:10, Sebastian Trueg wrote:
>>>>>> Hi Andy,
>>>>>>
>>>>>> - Fuseki started via "fuseki-server".
>>>>>> - shiro config attached
>>>>>> - "pdm-data-model" dataset created via attached config
>>>>>> - Simple test app attached which takes the fuseki dataset url as
>>>>>> parameter.
>>>>>>
>>>>>> Hope this helps.
>>>>>>
>>>>>> Regards,
>>>>>> Sebastian
>>>>>>
>>>>>> On 31.10.19 11:55, Andy Seaborne wrote:
>>>>>>> Sebastian,
>>>>>>>
>>>>>>> Do you have a complete, minimal example, including the Fuseki setup
>>>>>>> for
>>>>>>> the user/password. Which Fuseki variant? war? full-jar? main?
>>>>>>>
>>>>>>> Andy
>>>>>>>
>>>>>>> PS Don't forget teh javadoc on connectPW : it's "basic auth"
>>>>>>>
>>>>>>> On 31/10/2019 10:30, Sebastian Trueg wrote:
>>>>>>>> Hi everyone,
>>>>>>>>
>>>>>>>> trying to use RDFConnection with Jena 3.13.1 to put a Model into a
>>>>>>>> remote Fuseki instance I encountered very strange behavior. First
>>>>>>>> off,
>>>>>>>> let me show my very simple code:
>>>>>>>>
>>>>>>>> try(RDFConnection conn
>>>>>>>> = RDFConnectionFactory.connectPW(datasetUrl, "admin",
>>>>>>>> "admin")) {
>>>>>>>> conn.put(graphUri, model);
>>>>>>>> }
>>>>>>>>
>>>>>>>> This works fine on its own and for very small models in general.
>>>>>>>> But as
>>>>>>>> soon as I repeat the exact same snippet of code, ie. run the
>>>>>>>> same try
>>>>>>>> block twice I get a SocketException (Broken pipe) on the first
>>>>>>>> call to
>>>>>>>> RDFConnection::put.
>>>>>>>>
>>>>>>>> So, to sum up:
>>>>>>>> - Single put works fine.
>>>>>>>> - A subsequent call to put will result in the first one already
>>>>>>>> throwing
>>>>>>>> an exception!
>>>>>>>> - Using a model with less than 100 triples results in both put
>>>>>>>> operations to succeed.
>>>>>>>> - In all this the Fuseki instance keeps on working.
>>>>>>>>
>>>>>>>> Any ideas?
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Sebastian
>>>>>>>>
>>>>>>
>>>>
>>
--
Sebastian Trueg
Managing Director
TrueGeeX UG (haftungsbeschränkt)
trueg@truegeex.de
http://www.linkedin.com/in/trueg
Mobile: 0049 1762 3244 664
Re: SocketException with RDFConnection and models > 100 triples
Posted by Andy Seaborne <an...@apache.org>.
Yes, Jena bug, possibly related to the -1 length.
Recorded as:
https://issues.apache.org/jira/browse/JENA-1776
-1 is because Jena streams the content for a GSP PUT operation. It does
not know the length at the start of request. To get the length you have
to produce the content then send the request - which is not streaming.
A "feature" of HTTP.
So I'm guessing that the connection is not handled like the default (no
auth) version in some way but I don't know why.
RDFConnectionRemote has a builder and connectPW is just this:
public static RDFConnection connectPW(String URL, String user, String
password) {
BasicCredentialsProvider credsProvider =
new BasicCredentialsProvider();
Credentials credentials =
new UsernamePasswordCredentials(user, password);
credsProvider.setCredentials(AuthScope.ANY, credentials);
HttpClient client = HttpClients.custom()
.setDefaultCredentialsProvider(credsProvider)
.build();
return RDFConnectionRemote.create()
.destination(URL)
.httpClient(client)
.build();
}
so it is a matter for setting up the HttpClient correctly (setting it up
so it is closed after each use? This is the compromise HTTP requires -
stream once, or buffer and send length with connection reuse).
The normal, no auth default is
HttpOp.createPoolingHttpClientBuilder
Andy
FYI: I'm away, no development machine (if I even have internet access,
and that's not certain), for a week or so.
On 31/10/2019 14:45, Sebastian Trueg wrote:
> So... a Jena bug then? Do you have an idea if there is something I can
> configure in the HttpClient as a workaround?
>
>
> On 31.10.19 15:18, Andy Seaborne wrote:
>> I can make it happen sporadically using Fuseki main (so no shiro, not a
>> webapp environment). It happens maybe one in four test runs.
>>
>> It looks like a client-side problem - it creates the HTTP connection
>> each time and maybe something is cached in HttpClient. A hash-map-ism
>> would explain the "sporadically". If so, the # triples effect maybe
>> causing the timing to get changed a little.
>>
>> Andy
>>
>> On 31/10/2019 13:27, Sebastian Trueg wrote:
>>> Interestingly it does not. Only going down to a really small number of
>>> triples does work.
>>>
>>> And to make sure I did implement manual HTTP DELETE+PUT via OkHttp which
>>> works without problems on the same Fuseki instance.
>>>
>>> Regards,
>>> Sebastian
>>>
>>> On 31.10.19 14:14, Andy Seaborne wrote:
>>>> Thanks.
>>>>
>>>> Presumably it works if you create the RDFConnection once and reuse the
>>>> java object?
>>>>
>>>> Andy
>>>>
>>>> On 31/10/2019 12:10, Sebastian Trueg wrote:
>>>>> Hi Andy,
>>>>>
>>>>> - Fuseki started via "fuseki-server".
>>>>> - shiro config attached
>>>>> - "pdm-data-model" dataset created via attached config
>>>>> - Simple test app attached which takes the fuseki dataset url as
>>>>> parameter.
>>>>>
>>>>> Hope this helps.
>>>>>
>>>>> Regards,
>>>>> Sebastian
>>>>>
>>>>> On 31.10.19 11:55, Andy Seaborne wrote:
>>>>>> Sebastian,
>>>>>>
>>>>>> Do you have a complete, minimal example, including the Fuseki setup
>>>>>> for
>>>>>> the user/password. Which Fuseki variant? war? full-jar? main?
>>>>>>
>>>>>> Andy
>>>>>>
>>>>>> PS Don't forget teh javadoc on connectPW : it's "basic auth"
>>>>>>
>>>>>> On 31/10/2019 10:30, Sebastian Trueg wrote:
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> trying to use RDFConnection with Jena 3.13.1 to put a Model into a
>>>>>>> remote Fuseki instance I encountered very strange behavior. First
>>>>>>> off,
>>>>>>> let me show my very simple code:
>>>>>>>
>>>>>>> try(RDFConnection conn
>>>>>>> = RDFConnectionFactory.connectPW(datasetUrl, "admin",
>>>>>>> "admin")) {
>>>>>>> conn.put(graphUri, model);
>>>>>>> }
>>>>>>>
>>>>>>> This works fine on its own and for very small models in general.
>>>>>>> But as
>>>>>>> soon as I repeat the exact same snippet of code, ie. run the same try
>>>>>>> block twice I get a SocketException (Broken pipe) on the first
>>>>>>> call to
>>>>>>> RDFConnection::put.
>>>>>>>
>>>>>>> So, to sum up:
>>>>>>> - Single put works fine.
>>>>>>> - A subsequent call to put will result in the first one already
>>>>>>> throwing
>>>>>>> an exception!
>>>>>>> - Using a model with less than 100 triples results in both put
>>>>>>> operations to succeed.
>>>>>>> - In all this the Fuseki instance keeps on working.
>>>>>>>
>>>>>>> Any ideas?
>>>>>>>
>>>>>>> Regards,
>>>>>>> Sebastian
>>>>>>>
>>>>>
>>>
>
Re: SocketException with RDFConnection and models > 100 triples
Posted by Sebastian Trueg <tr...@truegeex.de>.
So... a Jena bug then? Do you have an idea if there is something I can
configure in the HttpClient as a workaround?
On 31.10.19 15:18, Andy Seaborne wrote:
> I can make it happen sporadically using Fuseki main (so no shiro, not a
> webapp environment). It happens maybe one in four test runs.
>
> It looks like a client-side problem - it creates the HTTP connection
> each time and maybe something is cached in HttpClient. A hash-map-ism
> would explain the "sporadically". If so, the # triples effect maybe
> causing the timing to get changed a little.
>
> Andy
>
> On 31/10/2019 13:27, Sebastian Trueg wrote:
>> Interestingly it does not. Only going down to a really small number of
>> triples does work.
>>
>> And to make sure I did implement manual HTTP DELETE+PUT via OkHttp which
>> works without problems on the same Fuseki instance.
>>
>> Regards,
>> Sebastian
>>
>> On 31.10.19 14:14, Andy Seaborne wrote:
>>> Thanks.
>>>
>>> Presumably it works if you create the RDFConnection once and reuse the
>>> java object?
>>>
>>> Andy
>>>
>>> On 31/10/2019 12:10, Sebastian Trueg wrote:
>>>> Hi Andy,
>>>>
>>>> - Fuseki started via "fuseki-server".
>>>> - shiro config attached
>>>> - "pdm-data-model" dataset created via attached config
>>>> - Simple test app attached which takes the fuseki dataset url as
>>>> parameter.
>>>>
>>>> Hope this helps.
>>>>
>>>> Regards,
>>>> Sebastian
>>>>
>>>> On 31.10.19 11:55, Andy Seaborne wrote:
>>>>> Sebastian,
>>>>>
>>>>> Do you have a complete, minimal example, including the Fuseki setup
>>>>> for
>>>>> the user/password. Which Fuseki variant? war? full-jar? main?
>>>>>
>>>>> Andy
>>>>>
>>>>> PS Don't forget teh javadoc on connectPW : it's "basic auth"
>>>>>
>>>>> On 31/10/2019 10:30, Sebastian Trueg wrote:
>>>>>> Hi everyone,
>>>>>>
>>>>>> trying to use RDFConnection with Jena 3.13.1 to put a Model into a
>>>>>> remote Fuseki instance I encountered very strange behavior. First
>>>>>> off,
>>>>>> let me show my very simple code:
>>>>>>
>>>>>> try(RDFConnection conn
>>>>>> = RDFConnectionFactory.connectPW(datasetUrl, "admin",
>>>>>> "admin")) {
>>>>>> conn.put(graphUri, model);
>>>>>> }
>>>>>>
>>>>>> This works fine on its own and for very small models in general.
>>>>>> But as
>>>>>> soon as I repeat the exact same snippet of code, ie. run the same try
>>>>>> block twice I get a SocketException (Broken pipe) on the first
>>>>>> call to
>>>>>> RDFConnection::put.
>>>>>>
>>>>>> So, to sum up:
>>>>>> - Single put works fine.
>>>>>> - A subsequent call to put will result in the first one already
>>>>>> throwing
>>>>>> an exception!
>>>>>> - Using a model with less than 100 triples results in both put
>>>>>> operations to succeed.
>>>>>> - In all this the Fuseki instance keeps on working.
>>>>>>
>>>>>> Any ideas?
>>>>>>
>>>>>> Regards,
>>>>>> Sebastian
>>>>>>
>>>>
>>
--
Sebastian Trueg
Managing Director
TrueGeeX UG (haftungsbeschränkt)
trueg@truegeex.de
http://www.linkedin.com/in/trueg
Mobile: 0049 1762 3244 664
Re: SocketException with RDFConnection and models > 100 triples
Posted by Andy Seaborne <an...@apache.org>.
I can make it happen sporadically using Fuseki main (so no shiro, not a
webapp environment). It happens maybe one in four test runs.
It looks like a client-side problem - it creates the HTTP connection
each time and maybe something is cached in HttpClient. A hash-map-ism
would explain the "sporadically". If so, the # triples effect maybe
causing the timing to get changed a little.
Andy
On 31/10/2019 13:27, Sebastian Trueg wrote:
> Interestingly it does not. Only going down to a really small number of
> triples does work.
>
> And to make sure I did implement manual HTTP DELETE+PUT via OkHttp which
> works without problems on the same Fuseki instance.
>
> Regards,
> Sebastian
>
> On 31.10.19 14:14, Andy Seaborne wrote:
>> Thanks.
>>
>> Presumably it works if you create the RDFConnection once and reuse the
>> java object?
>>
>> Andy
>>
>> On 31/10/2019 12:10, Sebastian Trueg wrote:
>>> Hi Andy,
>>>
>>> - Fuseki started via "fuseki-server".
>>> - shiro config attached
>>> - "pdm-data-model" dataset created via attached config
>>> - Simple test app attached which takes the fuseki dataset url as
>>> parameter.
>>>
>>> Hope this helps.
>>>
>>> Regards,
>>> Sebastian
>>>
>>> On 31.10.19 11:55, Andy Seaborne wrote:
>>>> Sebastian,
>>>>
>>>> Do you have a complete, minimal example, including the Fuseki setup for
>>>> the user/password. Which Fuseki variant? war? full-jar? main?
>>>>
>>>> Andy
>>>>
>>>> PS Don't forget teh javadoc on connectPW : it's "basic auth"
>>>>
>>>> On 31/10/2019 10:30, Sebastian Trueg wrote:
>>>>> Hi everyone,
>>>>>
>>>>> trying to use RDFConnection with Jena 3.13.1 to put a Model into a
>>>>> remote Fuseki instance I encountered very strange behavior. First off,
>>>>> let me show my very simple code:
>>>>>
>>>>> try(RDFConnection conn
>>>>> = RDFConnectionFactory.connectPW(datasetUrl, "admin", "admin")) {
>>>>> conn.put(graphUri, model);
>>>>> }
>>>>>
>>>>> This works fine on its own and for very small models in general. But as
>>>>> soon as I repeat the exact same snippet of code, ie. run the same try
>>>>> block twice I get a SocketException (Broken pipe) on the first call to
>>>>> RDFConnection::put.
>>>>>
>>>>> So, to sum up:
>>>>> - Single put works fine.
>>>>> - A subsequent call to put will result in the first one already
>>>>> throwing
>>>>> an exception!
>>>>> - Using a model with less than 100 triples results in both put
>>>>> operations to succeed.
>>>>> - In all this the Fuseki instance keeps on working.
>>>>>
>>>>> Any ideas?
>>>>>
>>>>> Regards,
>>>>> Sebastian
>>>>>
>>>
>
Re: SocketException with RDFConnection and models > 100 triples
Posted by Sebastian Trueg <tr...@truegeex.de>.
Interestingly it does not. Only going down to a really small number of
triples does work.
And to make sure I did implement manual HTTP DELETE+PUT via OkHttp which
works without problems on the same Fuseki instance.
Regards,
Sebastian
On 31.10.19 14:14, Andy Seaborne wrote:
> Thanks.
>
> Presumably it works if you create the RDFConnection once and reuse the
> java object?
>
> Andy
>
> On 31/10/2019 12:10, Sebastian Trueg wrote:
>> Hi Andy,
>>
>> - Fuseki started via "fuseki-server".
>> - shiro config attached
>> - "pdm-data-model" dataset created via attached config
>> - Simple test app attached which takes the fuseki dataset url as
>> parameter.
>>
>> Hope this helps.
>>
>> Regards,
>> Sebastian
>>
>> On 31.10.19 11:55, Andy Seaborne wrote:
>>> Sebastian,
>>>
>>> Do you have a complete, minimal example, including the Fuseki setup for
>>> the user/password. Which Fuseki variant? war? full-jar? main?
>>>
>>> Andy
>>>
>>> PS Don't forget teh javadoc on connectPW : it's "basic auth"
>>>
>>> On 31/10/2019 10:30, Sebastian Trueg wrote:
>>>> Hi everyone,
>>>>
>>>> trying to use RDFConnection with Jena 3.13.1 to put a Model into a
>>>> remote Fuseki instance I encountered very strange behavior. First off,
>>>> let me show my very simple code:
>>>>
>>>> try(RDFConnection conn
>>>> = RDFConnectionFactory.connectPW(datasetUrl, "admin", "admin")) {
>>>> conn.put(graphUri, model);
>>>> }
>>>>
>>>> This works fine on its own and for very small models in general. But as
>>>> soon as I repeat the exact same snippet of code, ie. run the same try
>>>> block twice I get a SocketException (Broken pipe) on the first call to
>>>> RDFConnection::put.
>>>>
>>>> So, to sum up:
>>>> - Single put works fine.
>>>> - A subsequent call to put will result in the first one already
>>>> throwing
>>>> an exception!
>>>> - Using a model with less than 100 triples results in both put
>>>> operations to succeed.
>>>> - In all this the Fuseki instance keeps on working.
>>>>
>>>> Any ideas?
>>>>
>>>> Regards,
>>>> Sebastian
>>>>
>>
--
Sebastian Trueg
Managing Director
TrueGeeX UG (haftungsbeschränkt)
trueg@truegeex.de
http://www.linkedin.com/in/trueg
Mobile: 0049 1762 3244 664
Re: SocketException with RDFConnection and models > 100 triples
Posted by Andy Seaborne <an...@apache.org>.
Thanks.
Presumably it works if you create the RDFConnection once and reuse the
java object?
Andy
On 31/10/2019 12:10, Sebastian Trueg wrote:
> Hi Andy,
>
> - Fuseki started via "fuseki-server".
> - shiro config attached
> - "pdm-data-model" dataset created via attached config
> - Simple test app attached which takes the fuseki dataset url as
> parameter.
>
> Hope this helps.
>
> Regards,
> Sebastian
>
> On 31.10.19 11:55, Andy Seaborne wrote:
>> Sebastian,
>>
>> Do you have a complete, minimal example, including the Fuseki setup for
>> the user/password. Which Fuseki variant? war? full-jar? main?
>>
>> Andy
>>
>> PS Don't forget teh javadoc on connectPW : it's "basic auth"
>>
>> On 31/10/2019 10:30, Sebastian Trueg wrote:
>>> Hi everyone,
>>>
>>> trying to use RDFConnection with Jena 3.13.1 to put a Model into a
>>> remote Fuseki instance I encountered very strange behavior. First off,
>>> let me show my very simple code:
>>>
>>> try(RDFConnection conn
>>> = RDFConnectionFactory.connectPW(datasetUrl, "admin", "admin")) {
>>> conn.put(graphUri, model);
>>> }
>>>
>>> This works fine on its own and for very small models in general. But as
>>> soon as I repeat the exact same snippet of code, ie. run the same try
>>> block twice I get a SocketException (Broken pipe) on the first call to
>>> RDFConnection::put.
>>>
>>> So, to sum up:
>>> - Single put works fine.
>>> - A subsequent call to put will result in the first one already throwing
>>> an exception!
>>> - Using a model with less than 100 triples results in both put
>>> operations to succeed.
>>> - In all this the Fuseki instance keeps on working.
>>>
>>> Any ideas?
>>>
>>> Regards,
>>> Sebastian
>>>
>
Re: SocketException with RDFConnection and models > 100 triples
Posted by Sebastian Trueg <tr...@truegeex.de>.
Hi Andy,
- Fuseki started via "fuseki-server".
- shiro config attached
- "pdm-data-model" dataset created via attached config
- Simple test app attached which takes the fuseki dataset url as
parameter.
Hope this helps.
Regards,
Sebastian
On 31.10.19 11:55, Andy Seaborne wrote:
> Sebastian,
>
> Do you have a complete, minimal example, including the Fuseki setup for
> the user/password. Which Fuseki variant? war? full-jar? main?
>
> Andy
>
> PS Don't forget teh javadoc on connectPW : it's "basic auth"
>
> On 31/10/2019 10:30, Sebastian Trueg wrote:
>> Hi everyone,
>>
>> trying to use RDFConnection with Jena 3.13.1 to put a Model into a
>> remote Fuseki instance I encountered very strange behavior. First off,
>> let me show my very simple code:
>>
>> try(RDFConnection conn
>> = RDFConnectionFactory.connectPW(datasetUrl, "admin", "admin")) {
>> conn.put(graphUri, model);
>> }
>>
>> This works fine on its own and for very small models in general. But as
>> soon as I repeat the exact same snippet of code, ie. run the same try
>> block twice I get a SocketException (Broken pipe) on the first call to
>> RDFConnection::put.
>>
>> So, to sum up:
>> - Single put works fine.
>> - A subsequent call to put will result in the first one already throwing
>> an exception!
>> - Using a model with less than 100 triples results in both put
>> operations to succeed.
>> - In all this the Fuseki instance keeps on working.
>>
>> Any ideas?
>>
>> Regards,
>> Sebastian
>>
--
Sebastian Trueg
Managing Director
TrueGeeX UG (haftungsbeschränkt)
trueg@truegeex.de
http://www.linkedin.com/in/trueg
Mobile: 0049 1762 3244 664
Re: SocketException with RDFConnection and models > 100 triples
Posted by Andy Seaborne <an...@apache.org>.
Sebastian,
Do you have a complete, minimal example, including the Fuseki setup for
the user/password. Which Fuseki variant? war? full-jar? main?
Andy
PS Don't forget teh javadoc on connectPW : it's "basic auth"
On 31/10/2019 10:30, Sebastian Trueg wrote:
> Hi everyone,
>
> trying to use RDFConnection with Jena 3.13.1 to put a Model into a
> remote Fuseki instance I encountered very strange behavior. First off,
> let me show my very simple code:
>
> try(RDFConnection conn
> = RDFConnectionFactory.connectPW(datasetUrl, "admin", "admin")) {
> conn.put(graphUri, model);
> }
>
> This works fine on its own and for very small models in general. But as
> soon as I repeat the exact same snippet of code, ie. run the same try
> block twice I get a SocketException (Broken pipe) on the first call to
> RDFConnection::put.
>
> So, to sum up:
> - Single put works fine.
> - A subsequent call to put will result in the first one already throwing
> an exception!
> - Using a model with less than 100 triples results in both put
> operations to succeed.
> - In all this the Fuseki instance keeps on working.
>
> Any ideas?
>
> Regards,
> Sebastian
>