You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Andy Seaborne <an...@apache.org> on 2021/04/05 13:34:30 UTC
Re: Suggestions for learning more about SPARQL query performance?
Hi Steve,
> QueryExecutionFactory lets me create a query on either a Model or a
> Dataset.What is the difference?Which one does ARQ actually operate
> on?Will ARQ create a Dataset and DataGraph if given (say) an in-memory
> Model to be queried?
All SPARQL execution is on a dataset (actually a DatasetGraph, lower
level than Dataset which maps to Models, Resources from the building
blocks of Graphs and DatasetGraph).
Model is a presentation API - it has no state and the RDF is stored in
Graphs or DatasetGraphs.
Execution is of the algebra using class OpExecutor.
The algebra is extended to have quads and blocks of quads. OpExecutor
has both triple/basic graph pattern and quad/quadpattern steps. TDB
rewrite the algebra to quad form then executes that.
So actually touching the data comes down to "execute OpQuadPattern" or
"execute OpBGP" in OpExecutor. TDB executes QuadPattern natively; it has
its own subclass of OpExecutor. (TDB also has an adapter to execute over
a single graph as well.)
If executing in triple form for an in-memory, general purpose dataset, a
collection of graphs. The "execute OpBGP" step calls a rgistered
StageGenerator, which is an interface and can be set per-dataset.
StageGenerator receives a multi-triple BGP and returns the solutions -
get to see the whole BGP.
(Currently so does TIM, the native in-memory DatasetGraph even though it
is a quad store).
To extend for a custom graph implementation:
* do nothing - the default StageGenerator will turn the BGP execution
into graph.find calls.
if the graph can do better for a BGP:
* Provide your own StageGenerator.
and if a dataset concerned with quads or you want to see more of the
execution beyond BGPs:
* Provide your own OpExecutor subclass.
Everything maps back and forth between triples and quads so full SPARQL
functionality is available uniformly, but maybe not as efficiently as
possible.
> When/where are indexes created and used?
Indexes are a feature of the storage, not the general execution strategy.
> optimizing the SPARQL algebra
Optimziation has two parts: rewriting the algebra into "better" algebra.
This may include moving filters about.
Reordering BPs/QuadPatterns is done at the Storage or the default
StageGenerator.
Andy
On 05/04/2021 16:29, Steve Vestal wrote:
>
> Thanks for the pointers to the excellent tutorial materials on how
> SPARQL queries are processed and how various things affect
> performance.After a little bit of digging and experimenting with ARQ,
> I’m not clear on how things go from an optimized SPARQL algebra
> expression to evaluation of leaf BGPs and when/where indexes are
> created and used.
>
> I assumed ARQ uses the Graph interface to access an underlying Model,
> which can be backed by any of an in-memory model, a TDB, my own class
> that implements the Graph interface, or an inference or union model
> backed by any of these.The Graph interface does not have a “find”
> method that accepts a multi-triple BGP as an input (as one of the
> tutorials described), just finds for single-triple patterns.
>
> QueryExecutionFactory lets me create a query on either a Model or a
> Dataset.What is the difference?Which one does ARQ actually operate
> on?Will ARQ create a Dataset and DataGraph if given (say) an in-memory
> Model to be queried?Graph, Dataset, and DatasetGraph all support only
> single-triple query patterns.The TDB documentation talks about
> optimizing the SPARQL algebra, but it is the ARQ API that has
> optimization configuration options.Some initial experiments with a
> couple of ARQ Context settings resulted in little impact on a test
> query issued to a series of increasingly larger in-memory
> models.When/where are indexes created and used?
>
> Thanks for any insight.
>
> On 3/18/2021 9:45 AM, Steve Vestal wrote:
>> Thanks. I'm looking to get smarter in general about formulating
>> queries, particularly those with non-trivial graph structure, e.g.,
>> more than just a shallow tree of properties rooted in one resource,
>> maybe dags, maybe with cycles. I am open to post-processing query
>> results. (I do that already, generating and post-processing queries
>> are steps in the overall algorithm.)
>>
>> On 3/18/2021 9:19 AM, Andy Seaborne wrote:
>>>
>>>
>>> On 17/03/2021 22:45, Steve Vestal wrote:
>>>> I'd like to dig a bit deeper into SPARQL query performance, better
>>>> understand how different query formulations affect that, how ARQ
>>>> configuration parameters might be used to tune that. Can anyone
>>>> recommend a place to start reading beyond the SPARQL book and
>>>> language definition?
>>>
>>> Hi Steve,
>>>
>>> It's a bit "it depends on the query.
>>>
>>> There was a presentation recently and while its not about ARQ, the
>>> fundamental point that getting the basic graph pattern matching
>>> working efficiently applies.
>>>
>>> http://www.lotico.com/index.php/SPARQL_Query_Optimization_with_Pavel_Klinov
>>>
>>>
>>> Do you have specific queries in mind or is this a general enquiry?
>>>
>>> Andy
>>
Re: Suggestions for learning more about SPARQL query performance?
Posted by Andy Seaborne <an...@apache.org>.
On 06/04/2021 12:23, Steve Vestal wrote:
> One more question. What is the default interface used to query an
> InfModel?
Graph.find
> Is a tailored StageGenerator used
No (but could be - it just isn't ATM)
> or a straight-up Graph or
> InfGraph interface?
Yes.
Andy
>
> On 4/5/2021 11:39 AM, Steve Vestal wrote:
>> Thanks. Are the following impressions sort of correct?
>>
>> If I have a moderately sizeable RDF model (say over 1M triples), I am
>> likely to get faster queries by loading it into TDB rather than read
>> it into an in-memory model.
>>
>> If I interface to an external database using the Graph interface (I
>> assume this is much easier than writing my own StageGenerator or
>> OpExecutor), then I should pay particular attention to how triples are
>> ordered in a query or in BGPs in the resulting algebra.
>>
>> Does the default StageGenerator execute finds in the order they appear
>> in a BGP in the algebra?
>>
>> Is there any reason to pick TDB over TDB2?
>>
>> On 4/5/2021 8:34 AM, Andy Seaborne wrote:
>>> Hi Steve,
>>>
>>>> QueryExecutionFactory lets me create a query on either a Model or a
>>>> Dataset.What is the difference?Which one does ARQ actually operate
>>>> on?Will ARQ create a Dataset and DataGraph if given (say) an
>>>> in-memory Model to be queried?
>>> All SPARQL execution is on a dataset (actually a DatasetGraph, lower
>>> level than Dataset which maps to Models, Resources from the building
>>> blocks of Graphs and DatasetGraph).
>>>
>>> Model is a presentation API - it has no state and the RDF is stored
>>> in Graphs or DatasetGraphs.
>>>
>>> Execution is of the algebra using class OpExecutor.
>>>
>>> The algebra is extended to have quads and blocks of quads. OpExecutor
>>> has both triple/basic graph pattern and quad/quadpattern steps. TDB
>>> rewrite the algebra to quad form then executes that.
>>>
>>> So actually touching the data comes down to "execute OpQuadPattern"
>>> or "execute OpBGP" in OpExecutor. TDB executes QuadPattern natively;
>>> it has its own subclass of OpExecutor. (TDB also has an adapter to
>>> execute over a single graph as well.)
>>>
>>> If executing in triple form for an in-memory, general purpose
>>> dataset, a collection of graphs. The "execute OpBGP" step calls a
>>> rgistered StageGenerator, which is an interface and can be set
>>> per-dataset.
>>>
>>> StageGenerator receives a multi-triple BGP and returns the solutions
>>> - get to see the whole BGP.
>>>
>>> (Currently so does TIM, the native in-memory DatasetGraph even though
>>> it is a quad store).
>>>
>>> To extend for a custom graph implementation:
>>>
>>> * do nothing - the default StageGenerator will turn the BGP execution
>>> into graph.find calls.
>>>
>>> if the graph can do better for a BGP:
>>>
>>> * Provide your own StageGenerator.
>>>
>>> and if a dataset concerned with quads or you want to see more of the
>>> execution beyond BGPs:
>>>
>>> * Provide your own OpExecutor subclass.
>>>
>>> Everything maps back and forth between triples and quads so full
>>> SPARQL functionality is available uniformly, but maybe not as
>>> efficiently as possible.
>>>
>>>> When/where are indexes created and used?
>>> Indexes are a feature of the storage, not the general execution
>>> strategy.
>>>
>>>> optimizing the SPARQL algebra
>>> Optimziation has two parts: rewriting the algebra into "better"
>>> algebra. This may include moving filters about.
>>>
>>> Reordering BPs/QuadPatterns is done at the Storage or the default
>>> StageGenerator.
>>>
>>> Andy
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 05/04/2021 16:29, Steve Vestal wrote:
>>>>
>>>> Thanks for the pointers to the excellent tutorial materials on how
>>>> SPARQL queries are processed and how various things affect
>>>> performance.After a little bit of digging and experimenting with
>>>> ARQ, I’m not clear on how things go from an optimized SPARQL algebra
>>>> expression to evaluation of leaf BGPs and when/where indexes are
>>>> created and used.
>>>>
>>>> I assumed ARQ uses the Graph interface to access an underlying
>>>> Model, which can be backed by any of an in-memory model, a TDB, my
>>>> own class that implements the Graph interface, or an inference or
>>>> union model backed by any of these.The Graph interface does not have
>>>> a “find” method that accepts a multi-triple BGP as an
> input (as one
>>>> of the tutorials described), just finds for single-triple patterns.
>>>>
>>>> QueryExecutionFactory lets me create a query on either a Model or a
>>
>>>> Dataset.What is the difference?Which one does ARQ actually operate
>>>> on?Will ARQ create a Dataset and DataGraph if given (say) an
>>>> in-memory Model to be queried?Graph, Dataset, and DatasetGraph all
>>>> support only single-triple query patterns.The TDB documentation
>>>> talks about optimizing the SPARQL algebra, but it is the ARQ API
>>>> that has optimization configuration options.Some initial experiments
>>>> with a couple of ARQ Context settings resulted in little impact on a
>>>> test query issued to a series of increasingly larger in-memory
>>>> models.When/where are indexes created and used?
>>>>
>>>> Thanks for any insight.
>>>>
>>>> On 3/18/2021 9:45 AM, Steve Vestal wrote:
>>>>> Thanks. I'm looking to get smarter in general about formulating
>>>>> queries, particularly those with non-trivial graph structure, e.g.,
>>>>> more than just a shallow tree of properties rooted in one resource,
>>>>> maybe dags, maybe with cycles. I am open to post-processing query
>>>>> results. (I do that already, generating and post-processing queries
>>>>> are steps in the overall algorithm.)
>>>>>
>>>>> On 3/18/2021 9:19 AM, Andy Seaborne wrote:
>>>>>>
>>>>>>
>>>>>> On 17/03/2021 22:45, Steve Vestal wrote:
>>>>>>> I'd like to dig a bit deeper into SPARQL query performance,
>>>>>>> better understand how different query formulations affect that,
>>>>>>> how ARQ configuration parameters might be used to tune that. Can
>>>>>>> anyone recommend a place to start reading beyond the SPARQL book
>>>>>>> and language definition?
>>>>>>
>>>>>> Hi Steve,
>>>>>>
>>>>>> It's a bit "it depends on the query.
>>>>>>
>>>>>> There was a presentation recently and while its not about ARQ,
>> the
>>>>>> fundamental point that getting the basic graph pattern matching
>>>>>> working efficiently applies.
>>>>>>
>>>>>> http://www.lotico.com/index.php/SPARQL_Query_Optimization_with_Pavel_Klinov
>>>>>>
>>>>>>
>>>>>> Do you have specific queries in mind or is this a general enquiry?
>>>>>>
>>>>>> Andy
>>>>>
>>>
>>
>
Re: Suggestions for learning more about SPARQL query performance?
Posted by Steve Vestal <st...@adventiumlabs.com>.
One more question. What is the default interface used to query an
InfModel? Is a tailored StageGenerator used or a straight-up Graph or
InfGraph interface?
On 4/5/2021 11:39 AM, Steve Vestal wrote:
> Thanks. Are the following impressions sort of correct?
>
> If I have a moderately sizeable RDF model (say over 1M triples), I am
> likely to get faster queries by loading it into TDB rather than read
> it into an in-memory model.
>
> If I interface to an external database using the Graph interface (I
> assume this is much easier than writing my own StageGenerator or
> OpExecutor), then I should pay particular attention to how triples are
> ordered in a query or in BGPs in the resulting algebra.
>
> Does the default StageGenerator execute finds in the order they appear
> in a BGP in the algebra?
>
> Is there any reason to pick TDB over TDB2?
>
> On 4/5/2021 8:34 AM, Andy Seaborne wrote:
>> Hi Steve,
>>
>>> QueryExecutionFactory lets me create a query on either a Model or a
>>> Dataset.What is the difference?Which one does ARQ actually operate
>>> on?Will ARQ create a Dataset and DataGraph if given (say) an
>>> in-memory Model to be queried?
>> All SPARQL execution is on a dataset (actually a DatasetGraph, lower
>> level than Dataset which maps to Models, Resources from the building
>> blocks of Graphs and DatasetGraph).
>>
>> Model is a presentation API - it has no state and the RDF is stored
>> in Graphs or DatasetGraphs.
>>
>> Execution is of the algebra using class OpExecutor.
>>
>> The algebra is extended to have quads and blocks of quads. OpExecutor
>> has both triple/basic graph pattern and quad/quadpattern steps. TDB
>> rewrite the algebra to quad form then executes that.
>>
>> So actually touching the data comes down to "execute OpQuadPattern"
>> or "execute OpBGP" in OpExecutor. TDB executes QuadPattern natively;
>> it has its own subclass of OpExecutor. (TDB also has an adapter to
>> execute over a single graph as well.)
>>
>> If executing in triple form for an in-memory, general purpose
>> dataset, a collection of graphs. The "execute OpBGP" step calls a
>> rgistered StageGenerator, which is an interface and can be set
>> per-dataset.
>>
>> StageGenerator receives a multi-triple BGP and returns the solutions
>> - get to see the whole BGP.
>>
>> (Currently so does TIM, the native in-memory DatasetGraph even though
>> it is a quad store).
>>
>> To extend for a custom graph implementation:
>>
>> * do nothing - the default StageGenerator will turn the BGP execution
>> into graph.find calls.
>>
>> if the graph can do better for a BGP:
>>
>> * Provide your own StageGenerator.
>>
>> and if a dataset concerned with quads or you want to see more of the
>> execution beyond BGPs:
>>
>> * Provide your own OpExecutor subclass.
>>
>> Everything maps back and forth between triples and quads so full
>> SPARQL functionality is available uniformly, but maybe not as
>> efficiently as possible.
>>
>>> When/where are indexes created and used?
>> Indexes are a feature of the storage, not the general execution
>> strategy.
>>
>>> optimizing the SPARQL algebra
>> Optimziation has two parts: rewriting the algebra into "better"
>> algebra. This may include moving filters about.
>>
>> Reordering BPs/QuadPatterns is done at the Storage or the default
>> StageGenerator.
>>
>> Andy
>>
>>
>>
>>
>>
>>
>>
>> On 05/04/2021 16:29, Steve Vestal wrote:
>>>
>>> Thanks for the pointers to the excellent tutorial materials on how
>>> SPARQL queries are processed and how various things affect
>>> performance.After a little bit of digging and experimenting with
>>> ARQ, I’m not clear on how things go from an optimized SPARQL algebra
>>> expression to evaluation of leaf BGPs and when/where indexes are
>>> created and used.
>>>
>>> I assumed ARQ uses the Graph interface to access an underlying
>>> Model, which can be backed by any of an in-memory model, a TDB, my
>>> own class that implements the Graph interface, or an inference or
>>> union model backed by any of these.The Graph interface does not have
>>> a “find” method that accepts a multi-triple BGP as an
input (as one
>>> of the tutorials described), just finds for single-triple patterns.
>>>
>>> QueryExecutionFactory lets me create a query on either a Model or a
>
>>> Dataset.What is the difference?Which one does ARQ actually operate
>>> on?Will ARQ create a Dataset and DataGraph if given (say) an
>>> in-memory Model to be queried?Graph, Dataset, and DatasetGraph all
>>> support only single-triple query patterns.The TDB documentation
>>> talks about optimizing the SPARQL algebra, but it is the ARQ API
>>> that has optimization configuration options.Some initial experiments
>>> with a couple of ARQ Context settings resulted in little impact on a
>>> test query issued to a series of increasingly larger in-memory
>>> models.When/where are indexes created and used?
>>>
>>> Thanks for any insight.
>>>
>>> On 3/18/2021 9:45 AM, Steve Vestal wrote:
>>>> Thanks. I'm looking to get smarter in general about formulating
>>>> queries, particularly those with non-trivial graph structure, e.g.,
>>>> more than just a shallow tree of properties rooted in one resource,
>>>> maybe dags, maybe with cycles. I am open to post-processing query
>>>> results. (I do that already, generating and post-processing queries
>>>> are steps in the overall algorithm.)
>>>>
>>>> On 3/18/2021 9:19 AM, Andy Seaborne wrote:
>>>>>
>>>>>
>>>>> On 17/03/2021 22:45, Steve Vestal wrote:
>>>>>> I'd like to dig a bit deeper into SPARQL query performance,
>>>>>> better understand how different query formulations affect that,
>>>>>> how ARQ configuration parameters might be used to tune that. Can
>>>>>> anyone recommend a place to start reading beyond the SPARQL book
>>>>>> and language definition?
>>>>>
>>>>> Hi Steve,
>>>>>
>>>>> It's a bit "it depends on the query.
>>>>>
>>>>> There was a presentation recently and while its not about ARQ,
> the
>>>>> fundamental point that getting the basic graph pattern matching
>>>>> working efficiently applies.
>>>>>
>>>>> http://www.lotico.com/index.php/SPARQL_Query_Optimization_with_Pavel_Klinov
>>>>>
>>>>>
>>>>> Do you have specific queries in mind or is this a general enquiry?
>>>>>
>>>>> Andy
>>>>
>>
>
Re: Suggestions for learning more about SPARQL query performance?
Posted by Andy Seaborne <an...@apache.org>.
On 05/04/2021 17:39, Steve Vestal wrote:
> Thanks. Are the following impressions sort of correct?
>
> If I have a moderately sizeable RDF model (say over 1M triples), I am
> likely to get faster queries by loading it into TDB rather than read it
> into an in-memory model.
Hard to tell which si faster.
At 1M triples, TDB will very likely be running with everything cached so
it's "in-memory" for query.
Speed difference will come down to the fact it is different code in
places. Algorithmically, they are much the same (comparing TIM, the
transactional in-memory dataset vs TDB).
If you try it out, do report your findings.
> If I interface to an external database using the Graph interface (I
> assume this is much easier than writing my own StageGenerator or
> OpExecutor), then I should pay particular attention to how triples are
> ordered in a query or in BGPs in the resulting algebra.
Yes.
But the default StageGenerator is some light reordering then calling
Graph.find. See the code
For RDF-star:
StageGeneratorGenericStar
QueryIterBlockTriplesStar
but if no embedded triple terms these are calling the same as:
StageGeneratorGeneric
QueryIterBlockTriples
QueryIterTriplePattern
> Does the default StageGenerator execute finds in the order they appear
> in a BGP in the algebra?
StageGeneratorGeneric - no.
There is a simple, fixed reordering.
Order by most grounded terms (being less keen on rdf:type).
> Is there any reason to pick TDB over TDB2?
Not really especially for query use cases.
Andy
>
> On 4/5/2021 8:34 AM, Andy Seaborne wrote:
>> Hi Steve,
>>
>>> QueryExecutionFactory lets me create a query on either a Model or a
>>> Dataset.What is the difference?Which one does ARQ actually operate
>>> on?Will ARQ create a Dataset and DataGraph if given (say) an
>>> in-memory Model to be queried?
>> All SPARQL execution is on a dataset (actually a DatasetGraph, lower
>> level than Dataset which maps to Models, Resources from the building
>> blocks of Graphs and DatasetGraph).
>>
>> Model is a presentation API - it has no state and the RDF is stored in
>> Graphs or DatasetGraphs.
>>
>> Execution is of the algebra using class OpExecutor.
>>
>> The algebra is extended to have quads and blocks of quads. OpExecutor
>> has both triple/basic graph pattern and quad/quadpattern steps. TDB
>> rewrite the algebra to quad form then executes that.
>>
>> So actually touching the data comes down to "execute OpQuadPattern" or
>> "execute OpBGP" in OpExecutor. TDB executes QuadPattern natively; it
>> has its own subclass of OpExecutor. (TDB also has an adapter to
>> execute over a single graph as well.)
>>
>> If executing in triple form for an in-memory, general purpose dataset,
>> a collection of graphs. The "execute OpBGP" step calls a rgistered
>> StageGenerator, which is an interface and can be set per-dataset.
>>
>> StageGenerator receives a multi-triple BGP and returns the solutions -
>> get to see the whole BGP.
>>
>> (Currently so does TIM, the native in-memory DatasetGraph even though
>> it is a quad store).
>>
>> To extend for a custom graph implementation:
>>
>> * do nothing - the default StageGenerator will turn the BGP execution
>> into graph.find calls.
>>
>> if the graph can do better for a BGP:
>>
>> * Provide your own StageGenerator.
>>
>> and if a dataset concerned with quads or you want to see more of the
>> execution beyond BGPs:
>>
>> * Provide your own OpExecutor subclass.
>>
>> Everything maps back and forth between triples and quads so full
>> SPARQL functionality is available uniformly, but maybe not as
>> efficiently as possible.
>>
>>> When/where are indexes created and used?
>> Indexes are a feature of the storage, not the general execution strategy.
>>
>>> optimizing the SPARQL algebra
>> Optimziation has two parts: rewriting the algebra into "better"
>> algebra. This may include moving filters about.
>>
>> Reordering BPs/QuadPatterns is done at the Storage or the default
>> StageGenerator.
>>
>> Andy
>>
>>
>>
>>
>>
>>
>>
>> On 05/04/2021 16:29, Steve Vestal wrote:
>>>
>>> Thanks for the pointers to the excellent tutorial materials on how
>>> SPARQL queries are processed and how various things affect
>>> performance.After a little bit of digging and experimenting with ARQ,
>>> I’m not clear on how things go from an optimized SPARQL algebra
>>> expression to evaluation of leaf BGPs and when/where indexes are
>>> created and used.
>>>
>>> I assumed ARQ uses the Graph interface to access an underlying Model,
>>> which can be backed by any of an in-memory model, a TDB, my own class
>>> that implements the Graph interface, or an inference or union model
>>> backed by any of these.The Graph interface does not have a “find”
>>> method that accepts a multi-triple BGP as an input (as one of the
>>> tutorials described), just finds for single-triple patterns.
>>>
>>> QueryExecutionFactory lets me create a query on either a Model or a
>
>>> Dataset.What is the difference?Which one does ARQ actually operate
>>> on?Will ARQ create a Dataset and DataGraph if given (say) an
>>> in-memory Model to be queried?Graph, Dataset, and DatasetGraph all
>>> support only single-triple query patterns.The TDB documentation talks
>>> about optimizing the SPARQL algebra, but it is the ARQ API that has
>>> optimization configuration options.Some initial experiments with a
>>> couple of ARQ Context settings resulted in little impact on a test
>>> query issued to a series of increasingly larger in-memory
>>> models.When/where are indexes created and used?
>>>
>>> Thanks for any insight.
>>>
>>> On 3/18/2021 9:45 AM, Steve Vestal wrote:
>>>> Thanks. I'm looking to get smarter in general about formulating
>>>> queries, particularly those with non-trivial graph structure, e.g.,
>>>> more than just a shallow tree of properties rooted in one resource,
>>>> maybe dags, maybe with cycles. I am open to post-processing query
>>>> results. (I do that already, generating and post-processing queries
>>>> are steps in the overall algorithm.)
>>>>
>>>> On 3/18/2021 9:19 AM, Andy Seaborne wrote:
>>>>>
>>>>>
>>>>> On 17/03/2021 22:45, Steve Vestal wrote:
>>>>>> I'd like to dig a bit deeper into SPARQL query performance, better
>>>>>> understand how different query formulations affect that, how ARQ
>>>>>> configuration parameters might be used to tune that. Can anyone
>>>>>> recommend a place to start reading beyond the SPARQL book and
>>>>>> language definition?
>>>>>
>>>>> Hi Steve,
>>>>>
>>>>> It's a bit "it depends on the query.
>>>>>
>>>>> There was a presentation recently and while its not about ARQ,
> the
>>>>> fundamental point that getting the basic graph pattern matching
>>>>> working efficiently applies.
>>>>>
>>>>> http://www.lotico.com/index.php/SPARQL_Query_Optimization_with_Pavel_Klinov
>>>>>
>>>>>
>>>>> Do you have specific queries in mind or is this a general enquiry?
>>>>>
>>>>> Andy
>>>>
>>
>
Re: Suggestions for learning more about SPARQL query performance?
Posted by Steve Vestal <st...@adventiumlabs.com>.
Thanks. Are the following impressions sort of correct?
If I have a moderately sizeable RDF model (say over 1M triples), I am
likely to get faster queries by loading it into TDB rather than read it
into an in-memory model.
If I interface to an external database using the Graph interface (I
assume this is much easier than writing my own StageGenerator or
OpExecutor), then I should pay particular attention to how triples are
ordered in a query or in BGPs in the resulting algebra.
Does the default StageGenerator execute finds in the order they appear
in a BGP in the algebra?
Is there any reason to pick TDB over TDB2?
On 4/5/2021 8:34 AM, Andy Seaborne wrote:
> Hi Steve,
>
>> QueryExecutionFactory lets me create a query on either a Model or a
>> Dataset.What is the difference?Which one does ARQ actually operate
>> on?Will ARQ create a Dataset and DataGraph if given (say) an
>> in-memory Model to be queried?
> All SPARQL execution is on a dataset (actually a DatasetGraph, lower
> level than Dataset which maps to Models, Resources from the building
> blocks of Graphs and DatasetGraph).
>
> Model is a presentation API - it has no state and the RDF is stored in
> Graphs or DatasetGraphs.
>
> Execution is of the algebra using class OpExecutor.
>
> The algebra is extended to have quads and blocks of quads. OpExecutor
> has both triple/basic graph pattern and quad/quadpattern steps. TDB
> rewrite the algebra to quad form then executes that.
>
> So actually touching the data comes down to "execute OpQuadPattern" or
> "execute OpBGP" in OpExecutor. TDB executes QuadPattern natively; it
> has its own subclass of OpExecutor. (TDB also has an adapter to
> execute over a single graph as well.)
>
> If executing in triple form for an in-memory, general purpose dataset,
> a collection of graphs. The "execute OpBGP" step calls a rgistered
> StageGenerator, which is an interface and can be set per-dataset.
>
> StageGenerator receives a multi-triple BGP and returns the solutions -
> get to see the whole BGP.
>
> (Currently so does TIM, the native in-memory DatasetGraph even though
> it is a quad store).
>
> To extend for a custom graph implementation:
>
> * do nothing - the default StageGenerator will turn the BGP execution
> into graph.find calls.
>
> if the graph can do better for a BGP:
>
> * Provide your own StageGenerator.
>
> and if a dataset concerned with quads or you want to see more of the
> execution beyond BGPs:
>
> * Provide your own OpExecutor subclass.
>
> Everything maps back and forth between triples and quads so full
> SPARQL functionality is available uniformly, but maybe not as
> efficiently as possible.
>
>> When/where are indexes created and used?
> Indexes are a feature of the storage, not the general execution strategy.
>
>> optimizing the SPARQL algebra
> Optimziation has two parts: rewriting the algebra into "better"
> algebra. This may include moving filters about.
>
> Reordering BPs/QuadPatterns is done at the Storage or the default
> StageGenerator.
>
> Andy
>
>
>
>
>
>
>
> On 05/04/2021 16:29, Steve Vestal wrote:
>>
>> Thanks for the pointers to the excellent tutorial materials on how
>> SPARQL queries are processed and how various things affect
>> performance.After a little bit of digging and experimenting with ARQ,
>> I’m not clear on how things go from an optimized SPARQL algebra
>> expression to evaluation of leaf BGPs and when/where indexes are
>> created and used.
>>
>> I assumed ARQ uses the Graph interface to access an underlying Model,
>> which can be backed by any of an in-memory model, a TDB, my own class
>> that implements the Graph interface, or an inference or union model
>> backed by any of these.The Graph interface does not have a “find”
>> method that accepts a multi-triple BGP as an input (as one of the
>> tutorials described), just finds for single-triple patterns.
>>
>> QueryExecutionFactory lets me create a query on either a Model or a
>> Dataset.What is the difference?Which one does ARQ actually operate
>> on?Will ARQ create a Dataset and DataGraph if given (say) an
>> in-memory Model to be queried?Graph, Dataset, and DatasetGraph all
>> support only single-triple query patterns.The TDB documentation talks
>> about optimizing the SPARQL algebra, but it is the ARQ API that has
>> optimization configuration options.Some initial experiments with a
>> couple of ARQ Context settings resulted in little impact on a test
>> query issued to a series of increasingly larger in-memory
>> models.When/where are indexes created and used?
>>
>> Thanks for any insight.
>>
>> On 3/18/2021 9:45 AM, Steve Vestal wrote:
>>> Thanks. I'm looking to get smarter in general about formulating
>>> queries, particularly those with non-trivial graph structure, e.g.,
>>> more than just a shallow tree of properties rooted in one resource,
>>> maybe dags, maybe with cycles. I am open to post-processing query
>>> results. (I do that already, generating and post-processing queries
>>> are steps in the overall algorithm.)
>>>
>>> On 3/18/2021 9:19 AM, Andy Seaborne wrote:
>>>>
>>>>
>>>> On 17/03/2021 22:45, Steve Vestal wrote:
>>>>> I'd like to dig a bit deeper into SPARQL query performance, better
>>>>> understand how different query formulations affect that, how ARQ
>>>>> configuration parameters might be used to tune that. Can anyone
>>>>> recommend a place to start reading beyond the SPARQL book and
>>>>> language definition?
>>>>
>>>> Hi Steve,
>>>>
>>>> It's a bit "it depends on the query.
>>>>
>>>> There was a presentation recently and while its not about ARQ,
the
>>>> fundamental point that getting the basic graph pattern matching
>>>> working efficiently applies.
>>>>
>>>> http://www.lotico.com/index.php/SPARQL_Query_Optimization_with_Pavel_Klinov
>>>>
>>>>
>>>> Do you have specific queries in mind or is this a general enquiry?
>>>>
>>>> Andy
>>>
>