You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@clerezza.apache.org by Bertrand Delacretaz <bd...@apache.org> on 2010/06/15 10:31:08 UTC
SPARQL query performance/scalability
Hi,
We are using some Clerezza modules to implement SPARQL queries in IKS FISE.
I'm wondering about the performance/scalability of SPARQL queries as
done in our ClerezzaSparqlQueryEngine [1], which is based on an
example that Manuel created at the first FISE hackathon.
That code collects all available triples from the triple store in an
ArrayList, creates an UnionGraph by converting that collection to an
array, and passes that graph to TcManager.executeSparqlQuery(...)
Unless I'm missing something, it looks like that would use tons of
memory if the TcManager stores lots of triples, is there a way to make
this more scalable?
Also, is TcManager.executeSparqlQuery(...) efficient and scalable, or
are there usage limitations that we should take into account?
Thanks for any help, and RTFM links are welcome of course.
-Bertrand
[1] http://iks-project.googlecode.com/svn/sandbox/fise/trunk/clerezza/clerezza-sparql/src/main/java/eu/iksproject/fise/clerezza/sparql/ClerezzaSparqlQueryEngine.java
Re: SPARQL query performance/scalability
Posted by Reto Bachmann-Gmuer <re...@trialox.org>.
On Tue, Jun 15, 2010 at 1:32 PM, Bertrand Delacretaz <bdelacretaz@apache.org
> wrote:
> How should I apply a sparql query to the complete set of triples of a
> given TcManager efficiently?
>
> IIUC it's something like
>
> MGraph defaultGraph = new UnionMGraph(...)
> ResultSet rs = tcManager.executeSparqlQuery(query, defaultGraph);
>
> but I'm not sure what the defaultGraph exactly means - could I use
> null if the sparql query contains a FROM clause, and would that be
> better?
>
> The default graph in section 8.2.1 of the sparql spec, it can be defined
with a FROM clause, if there is no such clause in the query the one passed
to the method is used. So yes it should be possible to set that argument to
null. A query may also provide multiple FROM clauses yielding to the same
result as with the union graph.
Performance-wise it should make no difference, but with the implementation
of the fastlane the approach with the from-clause(s) is potentially more
performant.
> >
> > The clerezza sparql endpoint doesn't need to have the triples in memory.
> You
> > can have a UnionGraph of multiple graphs even from different backends
> > without any data actually being copied around...
>
> Ok, so I guess what I'm missing is how to construct that UnionGraph
> efficiently, while allowing all triples to be queried.
>
> new UnionMGraph(tcm.getGraph(x), tcm.getGraph(y),....)
reto
Re: SPARQL query performance/scalability
Posted by Bertrand Delacretaz <bd...@apache.org>.
Hi Reto,
Thanks very much for you reply.
On Tue, Jun 15, 2010 at 11:07 AM, Reto Bachmann-Gmuer
<re...@trialox.org> wrote:
> ...The approach of putting triples in an ArrayList seems the limiting factor
> and I don't understand why you do this. TcManager is as scalable as the
> underlying storage backend (i.e. tdb or sesame)....
Ok - as I said I just copied that code from an example.
How should I apply a sparql query to the complete set of triples of a
given TcManager efficiently?
IIUC it's something like
MGraph defaultGraph = new UnionMGraph(...)
ResultSet rs = tcManager.executeSparqlQuery(query, defaultGraph);
but I'm not sure what the defaultGraph exactly means - could I use
null if the sparql query contains a FROM clause, and would that be
better?
>
> The clerezza sparql endpoint doesn't need to have the triples in memory. You
> can have a UnionGraph of multiple graphs even from different backends
> without any data actually being copied around...
Ok, so I guess what I'm missing is how to construct that UnionGraph
efficiently, while allowing all triples to be queried.
>
> The performance of a sparql query may depend on the order of the where
> clauses. For queries against a set of graphs from the same storage endpoint
> we will soon implement a transparent fastlane to the sparql-endpoint of the
> storage provider (if the storage provider supports sparql), but in your case
> as you are querying against a UnionGraph (and not against named actually
> stored graphs) this will not make any difference.
Ok, cool, thanks for the info!
-Bertrand
Re: SPARQL query performance/scalability
Posted by Reto Bachmann-Gmuer <re...@trialox.org>.
Hi Bertrand,
The approach of putting triples in an ArrayList seems the limiting factor
and I don't understand why you do this. TcManager is as scalable as the
underlying storage backend (i.e. tdb or sesame).
The clerezza sparql endpoint doesn't need to have the triples in memory. You
can have a UnionGraph of multiple graphs even from different backends
without any data actually being copied around.
The performance of a sparql query may depend on the order of the where
clauses. For queries against a set of graphs from the same storage endpoint
we will soon implement a transparent fastlane to the sparql-endpoint of the
storage provider (if the storage provider supports sparql), but in your case
as you are querying against a UnionGraph (and not against named actually
stored graphs) this will not make any difference.
Cheers,
reto
On Tue, Jun 15, 2010 at 10:31 AM, Bertrand Delacretaz <
bdelacretaz@apache.org> wrote:
> Hi,
>
> We are using some Clerezza modules to implement SPARQL queries in IKS FISE.
>
> I'm wondering about the performance/scalability of SPARQL queries as
> done in our ClerezzaSparqlQueryEngine [1], which is based on an
> example that Manuel created at the first FISE hackathon.
>
> That code collects all available triples from the triple store in an
> ArrayList, creates an UnionGraph by converting that collection to an
> array, and passes that graph to TcManager.executeSparqlQuery(...)
>
> Unless I'm missing something, it looks like that would use tons of
> memory if the TcManager stores lots of triples, is there a way to make
> this more scalable?
>
> Also, is TcManager.executeSparqlQuery(...) efficient and scalable, or
> are there usage limitations that we should take into account?
>
> Thanks for any help, and RTFM links are welcome of course.
>
> -Bertrand
>
> [1]
> http://iks-project.googlecode.com/svn/sandbox/fise/trunk/clerezza/clerezza-sparql/src/main/java/eu/iksproject/fise/clerezza/sparql/ClerezzaSparqlQueryEngine.java
>