You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by John Fereira <ja...@cornell.edu> on 2012/01/24 16:16:10 UTC
Fuseki (broken pipe)
After getting fuseki with SDB working yesterday I check it this morning and got a 500 Server Error, with a SQLException in executing SQL statement message.
This appears to be the same problem I was trying to fix by migrating from Joseki to Fuseki. After a bit of searching I came across this thread (http://tech.groups.yahoo.com/group/jena-dev/message/35766) which describes the same problem when using Joseki. The link I've included shows a class for doing database connection pooling but it appears to be a standalone class and there isn't any information about where it might be hooked into SDB. I've been using database connection pooling for a long time and have had better success with the C3p0 db connection pooling library and have it wired into a Tomcat servlet container but it seems to me a ConnectionPooling class could be wired into SDB, but it would be nice if it pulled in the database connection values from the config file instead of hardcoding them in the java class as the URL shown above does.
I've seen numerous references to connection management and connection pooling with SDB but haven't seen any sort of recipe for how to put it all together.
Re: Fuseki (broken pipe)
Posted by Damian Steer <d....@bristol.ac.uk>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 01/02/12 12:10, John Fereira wrote:
> It's now about 24 hours since I implemented Damians ReconnectSDB
> fix and my fuseki endpoint is still returning results without any
> errors. Thank you much, Damian.
No problem, glad it's working. I'll have a think about working this
into SDB proper.
What we really need is a pooling version, but so far that eludes me.
> Just for kicks I tried using the ReconnectSDB class with a joseki
> implementation but just couldn't seem to get the right combination
> of jena, arq, sdb, tdb versions without some sort of unresolved
> dependency.
Yes, variations in ARQ typically bite me at some point.
Damian
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk8pO7cACgkQAyLCB+mTtymWSwCg9jh1Z9nC8lORvTS9S3XDuuEu
r4QAoOVoM1M5XhCvBxD+7Lfc3q78sVfb
=NzMg
-----END PGP SIGNATURE-----
RE: Fuseki (broken pipe)
Posted by John Fereira <ja...@cornell.edu>.
It's now about 24 hours since I implemented Damians ReconnectSDB fix and my fuseki endpoint is still returning results without any errors. Thank you much, Damian.
Just for kicks I tried using the ReconnectSDB class with a joseki implementation but just couldn't seem to get the right combination of jena, arq, sdb, tdb versions without some sort of unresolved dependency.
RE: Fuseki (broken pipe)
Posted by John Fereira <ja...@cornell.edu>.
Okay, that works. I've got it running against a MySQL backed SDB without all the extra sdb-part and sdb-one-graph specs. I'll let you know how it's working after it's been running for 8 hours to see if it reconnects...thanks a lot for the fix.
> -----Original Message-----
> From: Damian Steer [mailto:d.steer@bristol.ac.uk]
> Sent: Monday, January 30, 2012 7:06 PM
> To: jena-users@incubator.apache.org
> Subject: Re: Fuseki (broken pipe)
>
>
> On 30 Jan 2012, at 20:10, John Fereira wrote:
>
> > Actually, I *was* using SDB-1.3.4 *not* SDB01.3.3
>
> Mystery solved.
>
> > I was able to get your reconnector built and launch Fuseki with a
> configuration file similar to yours (but using MySQL instead of H2) but
> got empty results when issue a query. I had run into the same issue
> before and had to run a configuration that specified a UNION graph.
> Here's what my current configuration files looks like. With it, fuseki
> works for 8 hours (when the mysql wait_timeout is reached). I guess
> what I need is to figure out how to merge your fusekisdbreconnect class
> with a store that uses a UNION graph.
>
>
> > <#sdb-part> rdf:type ja:RDFDataset ;
> > # If ja:namedGraph is used here, there is no correspondence
> > # with the name in the SDB store.
> > ja:defaultGraph <#sdb-one-graph> ;
> > .
> >
> > <#sdb-one-graph> a sdb:Model ;
> > sdb:dataset <#dataset> ;
> > sdb:graphName <urn:x-arq:UnionGraph> ; .
>
> This was pretty long, but if I read correctly you're just trying to get
> a dataset using the default union graph? That ought to be possible with
> SDB directly but clearly isn't (no equivalent of the TDB assembler
> option).That might cause performance issues for complex queries.
>
> I've added this feature to my little hack:
>
> <#dataset> rdf:type jumble:SDBConnect ;
> sdb:layout "layout2/index" ;
> jumble:defaultUnionGraph "true" ; # will switch option on globally,
> in fact
> sdb:connection [
> rdf:type sdb:SDBConnection ;
> sdb:sdbType "h2:file" ;
> sdb:sdbName "target/test-DB/H2" ;
> ]
> .
>
> Damian
Re: Fuseki (broken pipe)
Posted by Andy Seaborne <an...@apache.org>.
On 31/01/12 00:05, Damian Steer wrote:
>
> On 30 Jan 2012, at 20:10, John Fereira wrote:
>
>> Actually, I *was* using SDB-1.3.4 *not* SDB01.3.3
>
> Mystery solved.
>
>> I was able to get your reconnector built and launch Fuseki with a
>> configuration file similar to yours (but using MySQL instead of H2)
>> but got empty results when issue a query. I had run into the same
>> issue before and had to run a configuration that specified a UNION
>> graph. Here's what my current configuration files looks like.
>> With it, fuseki works for 8 hours (when the mysql wait_timeout is
>> reached). I guess what I need is to figure out how to merge your
>> fusekisdbreconnect class with a store that uses a UNION graph.
>
>
>> <#sdb-part> rdf:type ja:RDFDataset ; # If ja:namedGraph is used
>> here, there is no correspondence # with the name in the SDB store.
>> ja:defaultGraph<#sdb-one-graph> ; .
>>
>> <#sdb-one-graph> a sdb:Model ; sdb:dataset<#dataset> ;
>> sdb:graphName<urn:x-arq:UnionGraph> ; .
>
> This was pretty long, but if I read correctly you're just trying to
> get a dataset using the default union graph? That ought to be
> possible with SDB directly but clearly isn't (no equivalent of the
> TDB assembler option).That might cause performance issues for complex
> queries.
No reason not to add that to the assembler (caveat - it's a long time
since I looked at the code). It should be just a matter of adding it to
the dataset context.
It might be better to couple with handling this more uniformly across
stores (i.e. one URI) but a quick addition now to the SDB assembler
isn't going to invalidate existing behaviour.
Andy
> Damian
Re: Fuseki (broken pipe)
Posted by Damian Steer <d....@bristol.ac.uk>.
On 30 Jan 2012, at 20:10, John Fereira wrote:
> Actually, I *was* using SDB-1.3.4 *not* SDB01.3.3
Mystery solved.
> I was able to get your reconnector built and launch Fuseki with a configuration file similar to yours (but using MySQL instead of H2) but got empty results when issue a query. I had run into the same issue before and had to run a configuration that specified a UNION graph. Here's what my current configuration files looks like. With it, fuseki works for 8 hours (when the mysql wait_timeout is reached). I guess what I need is to figure out how to merge your fusekisdbreconnect class with a store that uses a UNION graph.
> <#sdb-part> rdf:type ja:RDFDataset ;
> # If ja:namedGraph is used here, there is no correspondence
> # with the name in the SDB store.
> ja:defaultGraph <#sdb-one-graph> ;
> .
>
> <#sdb-one-graph> a sdb:Model ;
> sdb:dataset <#dataset> ;
> sdb:graphName <urn:x-arq:UnionGraph> ;
> .
This was pretty long, but if I read correctly you're just trying to get a dataset using the default union graph? That ought to be possible with SDB directly but clearly isn't (no equivalent of the TDB assembler option).That might cause performance issues for complex queries.
I've added this feature to my little hack:
<#dataset> rdf:type jumble:SDBConnect ;
sdb:layout "layout2/index" ;
jumble:defaultUnionGraph "true" ; # will switch option on globally, in fact
sdb:connection [
rdf:type sdb:SDBConnection ;
sdb:sdbType "h2:file" ;
sdb:sdbName "target/test-DB/H2" ;
]
.
Damian
RE: Fuseki (broken pipe)
Posted by John Fereira <ja...@cornell.edu>.
Actually, I *was* using SDB-1.3.4 *not* SDB01.3.3
I was able to get your reconnector built and launch Fuseki with a configuration file similar to yours (but using MySQL instead of H2) but got empty results when issue a query. I had run into the same issue before and had to run a configuration that specified a UNION graph. Here's what my current configuration files looks like. With it, fuseki works for 8 hours (when the mysql wait_timeout is reached). I guess what I need is to figure out how to merge your fusekisdbreconnect class with a store that uses a UNION graph.
==================================================================================================
@prefix fuseki: <http://jena.apache.org/fuseki#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix tdb: <http://jena.hpl.hp.com/2008/tdb#> .
@prefix sdb: <http://jena.hpl.hp.com/2007/sdb#> .
@prefix ja: <http://jena.hpl.hp.com/2005/11/Assembler#> .
@prefix : <#> .
[] rdf:type fuseki:Server ;
# Services available. Only explicitly listed services are configured.
# If there is a service description not linked from this list, it is ignored.
fuseki:services (
<#service1>
)
.
# Declaration additional assembler items.
[] ja:loadClass "com.hp.hpl.jena.sdb.SDB" .
# SDB
# sdb:DatasetStore rdfs:subClassOf ja:RDFDataset .
sdb:Model rdfs:subClassOf ja:Model .
<#sdb-part> rdf:type ja:RDFDataset ;
# If ja:namedGraph is used here, there is no correspondence
# with the name in the SDB store.
ja:defaultGraph <#sdb-one-graph> ;
.
<#sdb-one-graph> a sdb:Model ;
sdb:dataset <#dataset> ;
sdb:graphName <urn:x-arq:UnionGraph> ;
.
<#dataset> rdf:type sdb:DatasetStore ;
sdb:store <#store> .
<#store> rdf:type sdb:Store ;
rdfs:label "SDB" ;
sdb:layout "layout2" ;
sdb:connection
[ rdf:type sdb:SDBConnection ;
# Using MySQL
sdb:sdbHost "localhost" ;
sdb:sdbType "MySQL" ;
sdb:sdbName "vivodb" ;
sdb:sdbUser "******" ;
sdb:sdbPassword "******" ;
sdb:driver "com.mysql.jdbc.Driver" ;
] ;
.
# Vivo dataset in sdb
<#service1> rdf:type fuseki:Service ;
fuseki:name "ds" ; # http://host:port/ds
fuseki:serviceQuery "query" ; # SPARQL query service
fuseki:serviceQuery "sparql" ; # SPARQL query service
# A separate ead-only graph store endpoint:
fuseki:serviceReadGraphStore "get" ; # SPARQL Graph store protocol (read only)
fuseki:dataset <#sdb-part> ;
.
==================================================================================================
-----Original Message-----
From: Damian Steer [mailto:d.steer@bristol.ac.uk]
Sent: Monday, January 30, 2012 12:19 PM
To: jena-users@incubator.apache.org
Subject: Re: Fuseki (broken pipe)
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 30/01/12 12:56, John Fereira wrote:
>
>
>> -----Original Message----- From: Damian Steer
>> [mailto:d.steer@bristol.ac.uk]
>> <https://github.com/shellac/ReconnectingSDB>
>>
>> (instructions to follow)
Instructions added. Let me know how it goes.
>> What versions of fuseki and SDB are you using?
>
> I'm currently using fusecki 0.2.1 and SDB 1.3.3. I think I tried SDB
> 1.3.4 but had some unresolved dependencies.
Hmm, I couldn't get that combination working.
Damian
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk8m0PQACgkQAyLCB+mTtymWBQCgqa5gSKS8WhkHL+5yNkGR7FpP
gYoAoP3317+uburcGn7kWuqQANoUX9t0
=aJ0G
-----END PGP SIGNATURE-----
Re: Fuseki (broken pipe)
Posted by Damian Steer <d....@bristol.ac.uk>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 30/01/12 12:56, John Fereira wrote:
>
>
>> -----Original Message----- From: Damian Steer
>> [mailto:d.steer@bristol.ac.uk]
>> <https://github.com/shellac/ReconnectingSDB>
>>
>> (instructions to follow)
Instructions added. Let me know how it goes.
>> What versions of fuseki and SDB are you using?
>
> I'm currently using fusecki 0.2.1 and SDB 1.3.3. I think I tried
> SDB 1.3.4 but had some unresolved dependencies.
Hmm, I couldn't get that combination working.
Damian
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk8m0PQACgkQAyLCB+mTtymWBQCgqa5gSKS8WhkHL+5yNkGR7FpP
gYoAoP3317+uburcGn7kWuqQANoUX9t0
=aJ0G
-----END PGP SIGNATURE-----
RE: Fuseki (broken pipe)
Posted by John Fereira <ja...@cornell.edu>.
> -----Original Message-----
> From: Damian Steer [mailto:d.steer@bristol.ac.uk]
> Sent: Monday, January 30, 2012 7:28 AM
> To: jena-users@incubator.apache.org
> Subject: Re: Fuseki (broken pipe)
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 30/01/12 11:59, John Fereira wrote:
>
> Hi John,
>
> >
> >> -----Original Message----- From: Damian Steer
> >> [mailto:d.steer@bristol.ac.uk] Sent: Saturday, January 28, 2012
> >> 8:44 AM To: jena-users@incubator.apache.org Subject: Re: Fuseki
> >> (broken pipe)
>
> >> 2) Before every SDB operation check the status of the connection
> >> (e.g. 'SELECT 1') and, if necessary, reconnect.
> >
> > As of java 6 there is a java.sql.Connection.isValid(int timeout)
> > method that could be used. What are the minimum java requirements
> > for SDB? If it's Java 6, using the isValid() method might be the
> > easiest way to fix this.
>
> Oh, that would be useful. Wonder how many drivers implement it?
Since it's in the java.sql.Connection class I suspect that the implementation is done there (possibly doing a SELECT 1 query) rather than expecting that the jdbc driver is going to implement a isValid() method.
>
> > On Friday I updated the mysql Connector/J jar file to the latest
> > version and added autoReconnect=true to the jdbcUrl. I don't see
> > a Stacktrace anymore.
>
> autoReconnect is a bad idea anyway, since it loses things like
> temporary tables.
>
> I had a crack at reopening SDB connections here:
>
> <https://github.com/shellac/ReconnectingSDB>
>
> (instructions to follow)
>
> What versions of fuseki and SDB are you using?
I'm currently using fusecki 0.2.1 and SDB 1.3.3. I think I tried SDB 1.3.4 but had some unresolved dependencies.
Re: Fuseki (broken pipe)
Posted by Damian Steer <d....@bristol.ac.uk>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 30/01/12 11:59, John Fereira wrote:
Hi John,
>
>> -----Original Message----- From: Damian Steer
>> [mailto:d.steer@bristol.ac.uk] Sent: Saturday, January 28, 2012
>> 8:44 AM To: jena-users@incubator.apache.org Subject: Re: Fuseki
>> (broken pipe)
>> 2) Before every SDB operation check the status of the connection
>> (e.g. 'SELECT 1') and, if necessary, reconnect.
>
> As of java 6 there is a java.sql.Connection.isValid(int timeout)
> method that could be used. What are the minimum java requirements
> for SDB? If it's Java 6, using the isValid() method might be the
> easiest way to fix this.
Oh, that would be useful. Wonder how many drivers implement it?
> On Friday I updated the mysql Connector/J jar file to the latest
> version and added autoReconnect=true to the jdbcUrl. I don't see
> a Stacktrace anymore.
autoReconnect is a bad idea anyway, since it loses things like
temporary tables.
I had a crack at reopening SDB connections here:
<https://github.com/shellac/ReconnectingSDB>
(instructions to follow)
What versions of fuseki and SDB are you using?
Damian
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk8mjMIACgkQAyLCB+mTtyk2rQCgjoem7xV6Q92tDBXEHtPJMVth
jlsAoIsZ4fBGHXmPD4pmGT0p5Nh7fVlj
=te5B
-----END PGP SIGNATURE-----
RE: Fuseki (broken pipe)
Posted by John Fereira <ja...@cornell.edu>.
> -----Original Message-----
> From: Damian Steer [mailto:d.steer@bristol.ac.uk]
> Sent: Saturday, January 28, 2012 8:44 AM
> To: jena-users@incubator.apache.org
> Subject: Re: Fuseki (broken pipe)
>
>
> On 27 Jan 2012, at 20:27, John Fereira wrote:
>
> > Sure, I could dump the RDF, but the site is going to be harvesting
> data from numerous data sources, probably on a daily basis and
> considering it's going to be hosted at a well known U.N. organization
> it's likely going to get really large. If I was just going to use RDF
> I could that directly to Elda and forget about using joseki/fuseki
> entirely (which I might do for a subset of the data anyway).
> >
> > The semantic web app isn't so much tightly coupled with SDB as much
> as SDB is sort of hardwired into it. I suppose replacing it with TDB
> would be possible but that would make it quite difficult to migrate to
> future release of the app.
>
> I've had a quick look at getting connection pooling working with fuseki
> and it looks tricky. The problem is that there's no obvious point to
> free the connection.
>
> Two approaches:
>
> 1) Wrap every DatasetGraph operation in a getConnection /
> freeConnection and use a pool. That might be ok for fuseki. Generally
> it won't work because you can get a graph and perform arbitrary
> operations on it.
>
> 2) Before every SDB operation check the status of the connection (e.g.
> 'SELECT 1') and, if necessary, reconnect.
As of java 6 there is a java.sql.Connection.isValid(int timeout) method that could be used. What are the minimum java requirements for SDB? If it's Java 6, using the isValid() method might be the easiest way to fix this.
On Friday I updated the mysql Connector/J jar file to the latest version and added autoReconnect=true to the jdbcUrl. I don't see a Stacktrace anymore. Instead, I'm seeing the following message:
The last packet successfully received from the server was 231,244,460 milliseconds ago. The last packet sent successfully to the server was 231,244,461 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
Re: Fuseki (broken pipe)
Posted by Damian Steer <d....@bristol.ac.uk>.
On 27 Jan 2012, at 20:27, John Fereira wrote:
> Sure, I could dump the RDF, but the site is going to be harvesting data from numerous data sources, probably on a daily basis and considering it's going to be hosted at a well known U.N. organization it's likely going to get really large. If I was just going to use RDF I could that directly to Elda and forget about using joseki/fuseki entirely (which I might do for a subset of the data anyway).
>
> The semantic web app isn't so much tightly coupled with SDB as much as SDB is sort of hardwired into it. I suppose replacing it with TDB would be possible but that would make it quite difficult to migrate to future release of the app.
I've had a quick look at getting connection pooling working with fuseki and it looks tricky. The problem is that there's no obvious point to free the connection.
Two approaches:
1) Wrap every DatasetGraph operation in a getConnection / freeConnection and use a pool. That might be ok for fuseki. Generally it won't work because you can get a graph and perform arbitrary operations on it.
2) Before every SDB operation check the status of the connection (e.g. 'SELECT 1') and, if necessary, reconnect.
Will investigate later.
Damian
RE: Fuseki (broken pipe)
Posted by John Fereira <ja...@cornell.edu>.
Sure, I could dump the RDF, but the site is going to be harvesting data from numerous data sources, probably on a daily basis and considering it's going to be hosted at a well known U.N. organization it's likely going to get really large. If I was just going to use RDF I could that directly to Elda and forget about using joseki/fuseki entirely (which I might do for a subset of the data anyway).
The semantic web app isn't so much tightly coupled with SDB as much as SDB is sort of hardwired into it. I suppose replacing it with TDB would be possible but that would make it quite difficult to migrate to future release of the app.
-----Original Message-----
From: Martynas Jusevicius [mailto:martynas@graphity.org]
Sent: Friday, January 27, 2012 3:16 PM
To: jena-users@incubator.apache.org
Subject: Re: Fuseki (broken pipe)
I wonder how it can be tightly coupled with SDB if SDB uses its internal table format which makes the database pretty much useless for relational DB tools?
If you can get the RDF dump, you can switch to any other triple store.
Even if the original data is in a relational DB, you can use a tool like D2RQ http://www4.wiwiss.fu-berlin.de/bizer/d2rq/ to extract RDF.
I'm just saying :)
Martynas
On Fri, Jan 27, 2012 at 9:01 PM, John Fereira <ja...@cornell.edu> wrote:
> Unfortunately, the semantic web application (VIVO) I'm working with is tightly coupled with SDB so switching to TDB is really an option (though I did just have a discussion with one of the developers to see if it could be done). Among other things, VIVO manages it's own SDB connections and *does* do connection pooling.
>
> -----Original Message-----
> From: Martynas Jusevicius [mailto:martynas@graphity.org]
> Sent: Friday, January 27, 2012 1:34 PM
> To: jena-users@incubator.apache.org
> Subject: Re: Fuseki (broken pipe)
>
> John,
>
> I was fighting with the same problem some time ago:
> http://tech.groups.yahoo.com/group/jena-dev/message/46541
> and I managed to solve it, as far as I remember.
>
> But my advice would be -- save yourself the trouble and start using TDB. TDB is actively developed, and performance on SDB using more complex queries quickly becomes unacceptable.
>
> Martynas
> http://graphity.org
>
> On Fri, Jan 27, 2012 at 7:25 PM, John Fereira <ja...@cornell.edu> wrote:
>>>On 24/01/12 15:16, John Fereira wrote:
>>>> After getting fuseki with SDB working yesterday I check it this
>>>> morning and got a 500 Server Error, with a SQLException in
>>>> executing SQL statement message.
>>>>
>>>> This appears to be the same problem I was trying to fix by
>>>> migrating from Joseki to Fuseki. After a bit of searching I came
>>>> across this thread
>>>> (http://tech.groups.yahoo.com/group/jena-dev/message/35766)
>>>> which describes the same problem when using Joseki. The link I've
>>>> included shows a class for doing database connection pooling but it
>>>> appears to be a standalone class and there isn't any information
>>>> about where it might be hooked into SDB. I've been using database
>>>> connection pooling for a long time and have had better success with
>>>> the C3p0 db connection pooling library and have it wired into a
>>>> Tomcat servlet container but it seems to me a ConnectionPooling
>>>> class could be wired into SDB, but it would be nice if it pulled in
>>>> the database connection values from the config file instead of
>>>> hardcoding them in the java class as the URL shown above does.
>>>>
>>>> I've seen numerous references to connection management and
>>>> connection pooling with SDB but haven't seen any sort of recipe for
>>>> how to put it all together.
>>>>
>>>
>>>If it's overnight, is this the MySQL timeout idle connections issue?
>>>MySQL, by default, kills connections after (IIRC) 8 hours of idleness.
>>
>> Yes, the connection closed overnight (and yes the idle value is 8 hours, based on the wait_timeout or interactive_timeout mysql variables). I guess my question is "why is a connection being held open for 8 hours?".
>>
>>>
>>>Connection pooling is a good solution to this, the crude way is ?autoReconnect=true or adjust the server settings to some very, very long period.
>>>
>>
>> I thought I'd read something that setting ?autoReconnect=true didn't seem to fix the problem. It seems that adjusting the idle timeout to a very long period would just result in running out of database connections eventually. Connection pooling *would* be a good solution as long as the connection is returned to the pool once the transaction is complete.
>>
>>>Adding connection pooling support to SDB would be good. There's
>>>nothing there except an assembler setting to set the number and a
>>>comment in SDBConnection about where it might hook in ... but that's
>>>as far as it goes
>>
>> There is a "pool" comment in SDBConnection but it looks like the place to add pooling would be in the SDBConnectionFactory (which also has a pooling comment). It currently obtains a java.sql.Connection using JDBC.createConnection. I think I can figure out where to inject connection pooling but I'm not quite sure what its' doing when sdb:jdbcUrl is not specified in the desc file.
>>
>>>
>>>Not sure of the status of c3po ... one preview release in 2010-05 but before that 2007-05. Maybe Apache Commons DBCP is a better bet.
>>
>> I'm certainly not married to c3p0...it's just what I found to be a bit more reliable a few years ago when I was implementing connection pooling in one of my web applications.
>>
Re: Fuseki (broken pipe)
Posted by Martynas Jusevicius <ma...@graphity.org>.
I wonder how it can be tightly coupled with SDB if SDB uses its
internal table format which makes the database pretty much useless for
relational DB tools?
If you can get the RDF dump, you can switch to any other triple store.
Even if the original data is in a relational DB, you can use a tool
like D2RQ http://www4.wiwiss.fu-berlin.de/bizer/d2rq/ to extract RDF.
I'm just saying :)
Martynas
On Fri, Jan 27, 2012 at 9:01 PM, John Fereira <ja...@cornell.edu> wrote:
> Unfortunately, the semantic web application (VIVO) I'm working with is tightly coupled with SDB so switching to TDB is really an option (though I did just have a discussion with one of the developers to see if it could be done). Among other things, VIVO manages it's own SDB connections and *does* do connection pooling.
>
> -----Original Message-----
> From: Martynas Jusevicius [mailto:martynas@graphity.org]
> Sent: Friday, January 27, 2012 1:34 PM
> To: jena-users@incubator.apache.org
> Subject: Re: Fuseki (broken pipe)
>
> John,
>
> I was fighting with the same problem some time ago:
> http://tech.groups.yahoo.com/group/jena-dev/message/46541
> and I managed to solve it, as far as I remember.
>
> But my advice would be -- save yourself the trouble and start using TDB. TDB is actively developed, and performance on SDB using more complex queries quickly becomes unacceptable.
>
> Martynas
> http://graphity.org
>
> On Fri, Jan 27, 2012 at 7:25 PM, John Fereira <ja...@cornell.edu> wrote:
>>>On 24/01/12 15:16, John Fereira wrote:
>>>> After getting fuseki with SDB working yesterday I check it this
>>>> morning and got a 500 Server Error, with a SQLException in
>>>> executing SQL statement message.
>>>>
>>>> This appears to be the same problem I was trying to fix by migrating
>>>> from Joseki to Fuseki. After a bit of searching I came across this
>>>> thread (http://tech.groups.yahoo.com/group/jena-dev/message/35766)
>>>> which describes the same problem when using Joseki. The link I've
>>>> included shows a class for doing database connection pooling but it
>>>> appears to be a standalone class and there isn't any information
>>>> about where it might be hooked into SDB. I've been using database
>>>> connection pooling for a long time and have had better success with
>>>> the C3p0 db connection pooling library and have it wired into a
>>>> Tomcat servlet container but it seems to me a ConnectionPooling
>>>> class could be wired into SDB, but it would be nice if it pulled in
>>>> the database connection values from the config file instead of
>>>> hardcoding them in the java class as the URL shown above does.
>>>>
>>>> I've seen numerous references to connection management and
>>>> connection pooling with SDB but haven't seen any sort of recipe for
>>>> how to put it all together.
>>>>
>>>
>>>If it's overnight, is this the MySQL timeout idle connections issue?
>>>MySQL, by default, kills connections after (IIRC) 8 hours of idleness.
>>
>> Yes, the connection closed overnight (and yes the idle value is 8 hours, based on the wait_timeout or interactive_timeout mysql variables). I guess my question is "why is a connection being held open for 8 hours?".
>>
>>>
>>>Connection pooling is a good solution to this, the crude way is ?autoReconnect=true or adjust the server settings to some very, very long period.
>>>
>>
>> I thought I'd read something that setting ?autoReconnect=true didn't seem to fix the problem. It seems that adjusting the idle timeout to a very long period would just result in running out of database connections eventually. Connection pooling *would* be a good solution as long as the connection is returned to the pool once the transaction is complete.
>>
>>>Adding connection pooling support to SDB would be good. There's
>>>nothing there except an assembler setting to set the number and a
>>>comment in SDBConnection about where it might hook in ... but that's
>>>as far as it goes
>>
>> There is a "pool" comment in SDBConnection but it looks like the place to add pooling would be in the SDBConnectionFactory (which also has a pooling comment). It currently obtains a java.sql.Connection using JDBC.createConnection. I think I can figure out where to inject connection pooling but I'm not quite sure what its' doing when sdb:jdbcUrl is not specified in the desc file.
>>
>>>
>>>Not sure of the status of c3po ... one preview release in 2010-05 but before that 2007-05. Maybe Apache Commons DBCP is a better bet.
>>
>> I'm certainly not married to c3p0...it's just what I found to be a bit more reliable a few years ago when I was implementing connection pooling in one of my web applications.
>>
RE: Fuseki (broken pipe)
Posted by John Fereira <ja...@cornell.edu>.
Unfortunately, the semantic web application (VIVO) I'm working with is tightly coupled with SDB so switching to TDB is really an option (though I did just have a discussion with one of the developers to see if it could be done). Among other things, VIVO manages it's own SDB connections and *does* do connection pooling.
-----Original Message-----
From: Martynas Jusevicius [mailto:martynas@graphity.org]
Sent: Friday, January 27, 2012 1:34 PM
To: jena-users@incubator.apache.org
Subject: Re: Fuseki (broken pipe)
John,
I was fighting with the same problem some time ago:
http://tech.groups.yahoo.com/group/jena-dev/message/46541
and I managed to solve it, as far as I remember.
But my advice would be -- save yourself the trouble and start using TDB. TDB is actively developed, and performance on SDB using more complex queries quickly becomes unacceptable.
Martynas
http://graphity.org
On Fri, Jan 27, 2012 at 7:25 PM, John Fereira <ja...@cornell.edu> wrote:
>>On 24/01/12 15:16, John Fereira wrote:
>>> After getting fuseki with SDB working yesterday I check it this
>>> morning and got a 500 Server Error, with a SQLException in
>>> executing SQL statement message.
>>>
>>> This appears to be the same problem I was trying to fix by migrating
>>> from Joseki to Fuseki. After a bit of searching I came across this
>>> thread (http://tech.groups.yahoo.com/group/jena-dev/message/35766)
>>> which describes the same problem when using Joseki. The link I've
>>> included shows a class for doing database connection pooling but it
>>> appears to be a standalone class and there isn't any information
>>> about where it might be hooked into SDB. I've been using database
>>> connection pooling for a long time and have had better success with
>>> the C3p0 db connection pooling library and have it wired into a
>>> Tomcat servlet container but it seems to me a ConnectionPooling
>>> class could be wired into SDB, but it would be nice if it pulled in
>>> the database connection values from the config file instead of
>>> hardcoding them in the java class as the URL shown above does.
>>>
>>> I've seen numerous references to connection management and
>>> connection pooling with SDB but haven't seen any sort of recipe for
>>> how to put it all together.
>>>
>>
>>If it's overnight, is this the MySQL timeout idle connections issue?
>>MySQL, by default, kills connections after (IIRC) 8 hours of idleness.
>
> Yes, the connection closed overnight (and yes the idle value is 8 hours, based on the wait_timeout or interactive_timeout mysql variables). I guess my question is "why is a connection being held open for 8 hours?".
>
>>
>>Connection pooling is a good solution to this, the crude way is ?autoReconnect=true or adjust the server settings to some very, very long period.
>>
>
> I thought I'd read something that setting ?autoReconnect=true didn't seem to fix the problem. It seems that adjusting the idle timeout to a very long period would just result in running out of database connections eventually. Connection pooling *would* be a good solution as long as the connection is returned to the pool once the transaction is complete.
>
>>Adding connection pooling support to SDB would be good. There's
>>nothing there except an assembler setting to set the number and a
>>comment in SDBConnection about where it might hook in ... but that's
>>as far as it goes
>
> There is a "pool" comment in SDBConnection but it looks like the place to add pooling would be in the SDBConnectionFactory (which also has a pooling comment). It currently obtains a java.sql.Connection using JDBC.createConnection. I think I can figure out where to inject connection pooling but I'm not quite sure what its' doing when sdb:jdbcUrl is not specified in the desc file.
>
>>
>>Not sure of the status of c3po ... one preview release in 2010-05 but before that 2007-05. Maybe Apache Commons DBCP is a better bet.
>
> I'm certainly not married to c3p0...it's just what I found to be a bit more reliable a few years ago when I was implementing connection pooling in one of my web applications.
>
Re: Fuseki (broken pipe)
Posted by Martynas Jusevicius <ma...@graphity.org>.
John,
I was fighting with the same problem some time ago:
http://tech.groups.yahoo.com/group/jena-dev/message/46541
and I managed to solve it, as far as I remember.
But my advice would be -- save yourself the trouble and start using
TDB. TDB is actively developed, and performance on SDB using more
complex queries quickly becomes unacceptable.
Martynas
http://graphity.org
On Fri, Jan 27, 2012 at 7:25 PM, John Fereira <ja...@cornell.edu> wrote:
>>On 24/01/12 15:16, John Fereira wrote:
>>> After getting fuseki with SDB working yesterday I check it this
>>> morning and got a 500 Server Error, with a SQLException in executing
>>> SQL statement message.
>>>
>>> This appears to be the same problem I was trying to fix by migrating
>>> from Joseki to Fuseki. After a bit of searching I came across this
>>> thread (http://tech.groups.yahoo.com/group/jena-dev/message/35766)
>>> which describes the same problem when using Joseki. The link I've
>>> included shows a class for doing database connection pooling but it
>>> appears to be a standalone class and there isn't any information about
>>> where it might be hooked into SDB. I've been using database
>>> connection pooling for a long time and have had better success with
>>> the C3p0 db connection pooling library and have it wired into a Tomcat
>>> servlet container but it seems to me a ConnectionPooling class could
>>> be wired into SDB, but it would be nice if it pulled in the database
>>> connection values from the config file instead of hardcoding them in
>>> the java class as the URL shown above does.
>>>
>>> I've seen numerous references to connection management and connection
>>> pooling with SDB but haven't seen any sort of recipe for how to put it
>>> all together.
>>>
>>
>>If it's overnight, is this the MySQL timeout idle connections issue?
>>MySQL, by default, kills connections after (IIRC) 8 hours of idleness.
>
> Yes, the connection closed overnight (and yes the idle value is 8 hours, based on the wait_timeout or interactive_timeout mysql variables). I guess my question is "why is a connection being held open for 8 hours?".
>
>>
>>Connection pooling is a good solution to this, the crude way is ?autoReconnect=true or adjust the server settings to some very, very long period.
>>
>
> I thought I'd read something that setting ?autoReconnect=true didn't seem to fix the problem. It seems that adjusting the idle timeout to a very long period would just result in running out of database connections eventually. Connection pooling *would* be a good solution as long as the connection is returned to the pool once the transaction is complete.
>
>>Adding connection pooling support to SDB would be good. There's nothing there except an assembler setting to set the number and a comment in SDBConnection about where it might hook in ... but that's as far as it goes
>
> There is a "pool" comment in SDBConnection but it looks like the place to add pooling would be in the SDBConnectionFactory (which also has a pooling comment). It currently obtains a java.sql.Connection using JDBC.createConnection. I think I can figure out where to inject connection pooling but I'm not quite sure what its' doing when sdb:jdbcUrl is not specified in the desc file.
>
>>
>>Not sure of the status of c3po ... one preview release in 2010-05 but before that 2007-05. Maybe Apache Commons DBCP is a better bet.
>
> I'm certainly not married to c3p0...it's just what I found to be a bit more reliable a few years ago when I was implementing connection pooling in one of my web applications.
>
Re: Fuseki (broken pipe)
Posted by Andy Seaborne <an...@apache.org>.
On 27/01/12 18:25, John Fereira wrote:
>> On 24/01/12 15:16, John Fereira wrote:
...
>> If it's overnight, is this the MySQL timeout idle connections
>> issue? MySQL, by default, kills connections after (IIRC) 8 hours of
>> idleness.
>
> Yes, the connection closed overnight (and yes the idle value is 8
> hours, based on the wait_timeout or interactive_timeout mysql
> variables). I guess my question is "why is a connection being held
> open for 8 hours?".
SDB doesn't itself do connection pooling. The idea is that it's done
outside by the app, on the basis that the app knows best.
http://incubator.apache.org/jena/documentation/sdb/javaapi.html#connection_management
But in Joseki,Fuseki they just want a handle to a thing to query or update.
There's a conceptual gap in the responsibilities. Pooled connections
fell down that gap. It ought to be an option for SDB.
No other system I know of times out connections and complains if they
are used after forced system close.
>
>>
>> Connection pooling is a good solution to this, the crude way is
>> ?autoReconnect=true or adjust the server settings to some very,
>> very long period.
>>
>
> I thought I'd read something that setting ?autoReconnect=true didn't
> seem to fix the problem.
The main argument against ?autoReconnect is that MySQL recommend against
it and they keep talking about removing it.
Andy
> It seems that adjusting the idle timeout to
> a very long period would just result in running out of database
> connections eventually. Connection pooling *would* be a good
> solution as long as the connection is returned to the pool once the
> transaction is complete.
>
>> Adding connection pooling support to SDB would be good. There's
>> nothing there except an assembler setting to set the number and a
>> comment in SDBConnection about where it might hook in ... but
>> that's as far as it goes
>
> There is a "pool" comment in SDBConnection but it looks like the
> place to add pooling would be in the SDBConnectionFactory (which also
> has a pooling comment). It currently obtains a java.sql.Connection
> using JDBC.createConnection. I think I can figure out where to
> inject connection pooling but I'm not quite sure what its' doing when
> sdb:jdbcUrl is not specified in the desc file.
>
>>
>> Not sure of the status of c3po ... one preview release in 2010-05
>> but before that 2007-05. Maybe Apache Commons DBCP is a better
>> bet.
>
> I'm certainly not married to c3p0...it's just what I found to be a
> bit more reliable a few years ago when I was implementing connection
> pooling in one of my web applications.
>
RE: Fuseki (broken pipe)
Posted by John Fereira <ja...@cornell.edu>.
>On 24/01/12 15:16, John Fereira wrote:
>> After getting fuseki with SDB working yesterday I check it this
>> morning and got a 500 Server Error, with a SQLException in executing
>> SQL statement message.
>>
>> This appears to be the same problem I was trying to fix by migrating
>> from Joseki to Fuseki. After a bit of searching I came across this
>> thread (http://tech.groups.yahoo.com/group/jena-dev/message/35766)
>> which describes the same problem when using Joseki. The link I've
>> included shows a class for doing database connection pooling but it
>> appears to be a standalone class and there isn't any information about
>> where it might be hooked into SDB. I've been using database
>> connection pooling for a long time and have had better success with
>> the C3p0 db connection pooling library and have it wired into a Tomcat
>> servlet container but it seems to me a ConnectionPooling class could
>> be wired into SDB, but it would be nice if it pulled in the database
>> connection values from the config file instead of hardcoding them in
>> the java class as the URL shown above does.
>>
>> I've seen numerous references to connection management and connection
>> pooling with SDB but haven't seen any sort of recipe for how to put it
>> all together.
>>
>
>If it's overnight, is this the MySQL timeout idle connections issue?
>MySQL, by default, kills connections after (IIRC) 8 hours of idleness.
Yes, the connection closed overnight (and yes the idle value is 8 hours, based on the wait_timeout or interactive_timeout mysql variables). I guess my question is "why is a connection being held open for 8 hours?".
>
>Connection pooling is a good solution to this, the crude way is ?autoReconnect=true or adjust the server settings to some very, very long period.
>
I thought I'd read something that setting ?autoReconnect=true didn't seem to fix the problem. It seems that adjusting the idle timeout to a very long period would just result in running out of database connections eventually. Connection pooling *would* be a good solution as long as the connection is returned to the pool once the transaction is complete.
>Adding connection pooling support to SDB would be good. There's nothing there except an assembler setting to set the number and a comment in SDBConnection about where it might hook in ... but that's as far as it goes
There is a "pool" comment in SDBConnection but it looks like the place to add pooling would be in the SDBConnectionFactory (which also has a pooling comment). It currently obtains a java.sql.Connection using JDBC.createConnection. I think I can figure out where to inject connection pooling but I'm not quite sure what its' doing when sdb:jdbcUrl is not specified in the desc file.
>
>Not sure of the status of c3po ... one preview release in 2010-05 but before that 2007-05. Maybe Apache Commons DBCP is a better bet.
I'm certainly not married to c3p0...it's just what I found to be a bit more reliable a few years ago when I was implementing connection pooling in one of my web applications.
Re: Fuseki (broken pipe)
Posted by Andy Seaborne <an...@apache.org>.
On 24/01/12 15:16, John Fereira wrote:
> After getting fuseki with SDB working yesterday I check it this
> morning and got a 500 Server Error, with a SQLException in executing
> SQL statement message.
>
> This appears to be the same problem I was trying to fix by migrating
> from Joseki to Fuseki. After a bit of searching I came across this
> thread (http://tech.groups.yahoo.com/group/jena-dev/message/35766)
> which describes the same problem when using Joseki. The link I've
> included shows a class for doing database connection pooling but it
> appears to be a standalone class and there isn't any information
> about where it might be hooked into SDB. I've been using database
> connection pooling for a long time and have had better success with
> the C3p0 db connection pooling library and have it wired into a
> Tomcat servlet container but it seems to me a ConnectionPooling class
> could be wired into SDB, but it would be nice if it pulled in the
> database connection values from the config file instead of hardcoding
> them in the java class as the URL shown above does.
>
> I've seen numerous references to connection management and connection
> pooling with SDB but haven't seen any sort of recipe for how to put
> it all together.
>
If it's overnight, is this the MySQL timeout idle connections issue?
MySQL, by default, kills connections after (IIRC) 8 hours of idleness.
Connection pooling is a good solution to this, the crude way is
?autoReconnect=true or adjust the server settings to some very, very
long period.
Adding connection pooling support to SDB would be good. There's nothing
there except an assembler setting to set the number and a comment in
SDBConnection about where it might hook in ... but that's as far as it goes
Not sure of the status of c3po ... one preview release in 2010-05 but
before that 2007-05. Maybe Apache Commons DBCP is a better bet.
At the moment, my work is on TDB and I'm unlikely to have time to look
at SDB in any reasonable timeframe.
Why not raise a JIRA for this? And a patch?
(We can't directly copy from that yahoo! message but I'm sure Bastian
would not mind submitting it to Apache if asked.)
Andy