You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Thorsten Möller <th...@unibas.ch> on 2011/11/21 10:54:20 UTC
TDB benchmark performance
Hi
I guess some of you have noticed the recent paper on comparing performance of prominent triple stores [1]. Do you consider it a fair benchmark (set up, data set, queries), considering the fact that TDB's performance isn't that good according to the results. Any objections, comments on the paper?
Thorsten
[1] http://iswc2011.semanticweb.org/fileadmin/iswc/Papers/Research_Paper/03/70310448.pdf
Re: TDB benchmark performance
Posted by Andy Seaborne <an...@apache.org>.
There's always things that can be done to improve performance. Finding
hot spots in the code would be a good start because they can infer with
everything else.
Reproducibility is important but I believe that it is also a 2shiny
thing" trap. What matters is "fit for purpose" in the environment it
will be used - and that is going to be a different setup.
That's why I'm trying to (find time to) write JenaPerf - it's a canned
collection of benchmarks anyone can run in their own environment.
And to measure "throughput per $".
Andy
On 22/11/11 10:14, Paolo Castagna wrote:
> Hi Thorsten, hi Bill,
> if the word science in "computer science" has any meaning, repeating the experiment would be interesting.
>
> In this case, fortunately, we are in the condition that repeating the experiment is possible (this is not always the case!). Even though we might change some of the settings for the Java heap and try
> stats.opt vs. fixed.opt to see if those have a significant impact on the results.
>
> The reproducibility of an experiment is one of the principles of the scientific method. However, benchmark developers do not always make that easy. :-)
>
> Paolo
>
> Bill Roberts wrote:
>> The report mentions that it uses stats-based optimisation for Jena/TDB. My experience has been that naive use of stats.opt in TDB can sometimes make it signficantly slower. I don't know the details of how the benchmark tests were done with regard to the stats.opt file, but impact of different optimisation approaches would be interesting to know.
>>
>>
>> On 22 Nov 2011, at 10:47, Thorsten Möller wrote:
>>
>>> Thanks. I was asking mostly to better understand *why* differences were that large. Reading statements such as "Jena-TDB can in general not deliver competitive performance with being by a factor 30-50 slower than the fastest system." [Section 7 of the paper] sets off alarm bells, meaning that I'm wondering whether the set up, data set, queries were fair. Assuming that query processing and the data backend are not naive in all the systems, I can imagine that differences are to some extent a matter of caching strategies. But then the query mixes become important since they might result in higher cache hits for one system than another.
>>>
>>> And I agree with Paolo that benchmarks should distinguish between interactive and analytic queries, which is nicely reflected by the different TPC suites (e.g. TPC-E versus TPC-H).
>>>
>>> Thorsten
>>>
>>>
>>>
>>>
>>>
>>> Am 21.11.2011 um 14:25 schrieb Andy Seaborne:
>>>
>>>> Some things could be clearer (and I was in the audience for the paper)
>>>>
>>>> 1/ "Java heap size set 8GB" - TDB does not use the heap for file caching.
>>>>
>>>> (this also messes up Sesame for another reason - a 32 Gb machine and they only used 8G for Sesame)
>>>>
>>>> 2/ Not clear if the machine was configured to use all memory for memory mapped files. It has been reported that some kernel configs are set to limit mmap usage.
>>>>
>>>> 3/ TDB stats and DBpedia data is "interesting". If they just used the default settings, I'd guess the stats (predicate frequencies) are likely to be wild.
>>>>
>>>> My personally feeling is that 1+2 is having an effect. It would interesting to run in direct mode with the in-JVM caches turned up to use 30G or so.
>>>>
>>>> But the real unclarity is the query set - they were asked about this at ISWC and did not give a clear an answer. It is gathered from DBpedia.org - but that has a timeout on queries that people often run into. This (a) skews the queries people ask anyway (b) may skew the queries collected as they did not say whether they collected timed out queries or successful queries.
>>>>
>>>> Andy
>>>>
>>>>
>>>>
>>>> On 21/11/11 10:47, Paolo Castagna wrote:
>>>>> Thorsten Möller wrote:
>>>>>> Hi
>>>>>>
>>>>>> I guess some of you have noticed the recent paper on comparing performance of prominent triple stores [1].
>>>>> Yep [1].
>>>>>
>>>>>> Do you consider it a fair benchmark (set up, data set, queries), considering the fact that TDB's performance isn't that good according to the results. Any objections, comments on the paper?
>>>>> Adding DBPSB to the list of JenaPerf benchmarks would probably be an useful contribution:
>>>>> https://svn.apache.org/repos/asf/incubator/jena/Experimental/JenaPerf/trunk/Benchmarks/
>>>>> I've had no time to run DBPSB myself or look in details each of the DBPSB queries, unfortunately. But, this is IMHO something useful to do.
>>>>>
>>>>> Benchmarks are useful to compare different systems.
>>>>> However, the best benchmark is the one you build yourself, using your data and the queries you need or your users are more likely to write.
>>>>> The aim of JenaPerf, as I see it, it to make as easy as possible to add new datasets and queries and run your benchmark.
>>>>>
>>>>> Personally, I tend to classify SPARQL queries in two broad categories: interactive and analytic.
>>>>> For interactive queries, I'd like to have sub-second response times. If that is not always possible, I desperately want a cache layer in front.
>>>>> For analytic queries, it does not really matter a few seconds or minutes (or hours!) difference. I just want the answer (in a reasonable amount of time).
>>>>>
>>>>> IMHO SPARQL (as SQL) is an unconstrained language in term of complexity of queries people can run... this makes a lot of things more "interesting" and challenging. (Compare with heavily constrained
>>>>> query languages from other NoSQL systems: MQL [2], CQL [3], ...)
>>>>>
>>>>> Last but not least, IMHO benchmarks are very useful tools in showing people and developers that there is room for improvement.
>>>>> Performances and scalability are very important and I welcome any progress TDB and ARQ makes on these aspects.
>>>>> But, I also value: being an open source project with an active community (and hopefully growing) of users and contributors around (including you! :-)), ease of installation, extensibility, compliance
>>>>> with W3C recommendations, ease of integration with other (Java) projects, quality and speed of support (via mailing list or commercial alternatives), etc.
>>>>>
>>>>> Do you want to help adding DBPSB to JenaPerf? :-)
>>>>>
>>>>> Paolo
>>>>>
>>>>> [1] http://markmail.org/message/n32zifhgl52kilkz
>>>>> [2] http://wiki.freebase.com/wiki/MQL
>>>>> [3] https://www.google.com/#q="Cassandra+Query+Language"
>>>>>
>>>>>
>>>>>> Thorsten
>>>>>>
>>>>>> [1] http://iswc2011.semanticweb.org/fileadmin/iswc/Papers/Research_Paper/03/70310448.pdf
>>>>>>
>>>>>>
>>
RE: TDB benchmark performance
Posted by "Dennis E. Hamilton" <or...@apache.org>.
OT: Made me smile, Paolo.
It is rare to recognize that, despite its theoretical foundations, computing is an empirical science; experimental methodology applies. And of course: Theoretically, there is no difference between theory and practice. In practice there is.
- Dennis
-----Original Message-----
From: Paolo Castagna [mailto:castagna.lists@googlemail.com]
Sent: Tuesday, November 22, 2011 02:15
To: jena-users@incubator.apache.org
Subject: Re: TDB benchmark performance
Hi Thorsten, hi Bill,
if the word science in "computer science" has any meaning, repeating the experiment would be interesting.
In this case, fortunately, we are in the condition that repeating the experiment is possible (this is not always the case!). Even though we might change some of the settings for the Java heap and try
stats.opt vs. fixed.opt to see if those have a significant impact on the results.
The reproducibility of an experiment is one of the principles of the scientific method. However, benchmark developers do not always make that easy. :-)
Paolo
[ ... ]
Re: TDB benchmark performance
Posted by Paolo Castagna <ca...@googlemail.com>.
Hi Thorsten, hi Bill,
if the word science in "computer science" has any meaning, repeating the experiment would be interesting.
In this case, fortunately, we are in the condition that repeating the experiment is possible (this is not always the case!). Even though we might change some of the settings for the Java heap and try
stats.opt vs. fixed.opt to see if those have a significant impact on the results.
The reproducibility of an experiment is one of the principles of the scientific method. However, benchmark developers do not always make that easy. :-)
Paolo
Bill Roberts wrote:
> The report mentions that it uses stats-based optimisation for Jena/TDB. My experience has been that naive use of stats.opt in TDB can sometimes make it signficantly slower. I don't know the details of how the benchmark tests were done with regard to the stats.opt file, but impact of different optimisation approaches would be interesting to know.
>
>
> On 22 Nov 2011, at 10:47, Thorsten Möller wrote:
>
>> Thanks. I was asking mostly to better understand *why* differences were that large. Reading statements such as "Jena-TDB can in general not deliver competitive performance with being by a factor 30-50 slower than the fastest system." [Section 7 of the paper] sets off alarm bells, meaning that I'm wondering whether the set up, data set, queries were fair. Assuming that query processing and the data backend are not naive in all the systems, I can imagine that differences are to some extent a matter of caching strategies. But then the query mixes become important since they might result in higher cache hits for one system than another.
>>
>> And I agree with Paolo that benchmarks should distinguish between interactive and analytic queries, which is nicely reflected by the different TPC suites (e.g. TPC-E versus TPC-H).
>>
>> Thorsten
>>
>>
>>
>>
>>
>> Am 21.11.2011 um 14:25 schrieb Andy Seaborne:
>>
>>> Some things could be clearer (and I was in the audience for the paper)
>>>
>>> 1/ "Java heap size set 8GB" - TDB does not use the heap for file caching.
>>>
>>> (this also messes up Sesame for another reason - a 32 Gb machine and they only used 8G for Sesame)
>>>
>>> 2/ Not clear if the machine was configured to use all memory for memory mapped files. It has been reported that some kernel configs are set to limit mmap usage.
>>>
>>> 3/ TDB stats and DBpedia data is "interesting". If they just used the default settings, I'd guess the stats (predicate frequencies) are likely to be wild.
>>>
>>> My personally feeling is that 1+2 is having an effect. It would interesting to run in direct mode with the in-JVM caches turned up to use 30G or so.
>>>
>>> But the real unclarity is the query set - they were asked about this at ISWC and did not give a clear an answer. It is gathered from DBpedia.org - but that has a timeout on queries that people often run into. This (a) skews the queries people ask anyway (b) may skew the queries collected as they did not say whether they collected timed out queries or successful queries.
>>>
>>> Andy
>>>
>>>
>>>
>>> On 21/11/11 10:47, Paolo Castagna wrote:
>>>> Thorsten Möller wrote:
>>>>> Hi
>>>>>
>>>>> I guess some of you have noticed the recent paper on comparing performance of prominent triple stores [1].
>>>> Yep [1].
>>>>
>>>>> Do you consider it a fair benchmark (set up, data set, queries), considering the fact that TDB's performance isn't that good according to the results. Any objections, comments on the paper?
>>>> Adding DBPSB to the list of JenaPerf benchmarks would probably be an useful contribution:
>>>> https://svn.apache.org/repos/asf/incubator/jena/Experimental/JenaPerf/trunk/Benchmarks/
>>>> I've had no time to run DBPSB myself or look in details each of the DBPSB queries, unfortunately. But, this is IMHO something useful to do.
>>>>
>>>> Benchmarks are useful to compare different systems.
>>>> However, the best benchmark is the one you build yourself, using your data and the queries you need or your users are more likely to write.
>>>> The aim of JenaPerf, as I see it, it to make as easy as possible to add new datasets and queries and run your benchmark.
>>>>
>>>> Personally, I tend to classify SPARQL queries in two broad categories: interactive and analytic.
>>>> For interactive queries, I'd like to have sub-second response times. If that is not always possible, I desperately want a cache layer in front.
>>>> For analytic queries, it does not really matter a few seconds or minutes (or hours!) difference. I just want the answer (in a reasonable amount of time).
>>>>
>>>> IMHO SPARQL (as SQL) is an unconstrained language in term of complexity of queries people can run... this makes a lot of things more "interesting" and challenging. (Compare with heavily constrained
>>>> query languages from other NoSQL systems: MQL [2], CQL [3], ...)
>>>>
>>>> Last but not least, IMHO benchmarks are very useful tools in showing people and developers that there is room for improvement.
>>>> Performances and scalability are very important and I welcome any progress TDB and ARQ makes on these aspects.
>>>> But, I also value: being an open source project with an active community (and hopefully growing) of users and contributors around (including you! :-)), ease of installation, extensibility, compliance
>>>> with W3C recommendations, ease of integration with other (Java) projects, quality and speed of support (via mailing list or commercial alternatives), etc.
>>>>
>>>> Do you want to help adding DBPSB to JenaPerf? :-)
>>>>
>>>> Paolo
>>>>
>>>> [1] http://markmail.org/message/n32zifhgl52kilkz
>>>> [2] http://wiki.freebase.com/wiki/MQL
>>>> [3] https://www.google.com/#q="Cassandra+Query+Language"
>>>>
>>>>
>>>>> Thorsten
>>>>>
>>>>> [1] http://iswc2011.semanticweb.org/fileadmin/iswc/Papers/Research_Paper/03/70310448.pdf
>>>>>
>>>>>
>
Re: TDB benchmark performance
Posted by Bill Roberts <bi...@swirrl.com>.
The report mentions that it uses stats-based optimisation for Jena/TDB. My experience has been that naive use of stats.opt in TDB can sometimes make it signficantly slower. I don't know the details of how the benchmark tests were done with regard to the stats.opt file, but impact of different optimisation approaches would be interesting to know.
On 22 Nov 2011, at 10:47, Thorsten Möller wrote:
> Thanks. I was asking mostly to better understand *why* differences were that large. Reading statements such as "Jena-TDB can in general not deliver competitive performance with being by a factor 30-50 slower than the fastest system." [Section 7 of the paper] sets off alarm bells, meaning that I'm wondering whether the set up, data set, queries were fair. Assuming that query processing and the data backend are not naive in all the systems, I can imagine that differences are to some extent a matter of caching strategies. But then the query mixes become important since they might result in higher cache hits for one system than another.
>
> And I agree with Paolo that benchmarks should distinguish between interactive and analytic queries, which is nicely reflected by the different TPC suites (e.g. TPC-E versus TPC-H).
>
> Thorsten
>
>
>
>
>
> Am 21.11.2011 um 14:25 schrieb Andy Seaborne:
>
>> Some things could be clearer (and I was in the audience for the paper)
>>
>> 1/ "Java heap size set 8GB" - TDB does not use the heap for file caching.
>>
>> (this also messes up Sesame for another reason - a 32 Gb machine and they only used 8G for Sesame)
>>
>> 2/ Not clear if the machine was configured to use all memory for memory mapped files. It has been reported that some kernel configs are set to limit mmap usage.
>>
>> 3/ TDB stats and DBpedia data is "interesting". If they just used the default settings, I'd guess the stats (predicate frequencies) are likely to be wild.
>>
>> My personally feeling is that 1+2 is having an effect. It would interesting to run in direct mode with the in-JVM caches turned up to use 30G or so.
>>
>> But the real unclarity is the query set - they were asked about this at ISWC and did not give a clear an answer. It is gathered from DBpedia.org - but that has a timeout on queries that people often run into. This (a) skews the queries people ask anyway (b) may skew the queries collected as they did not say whether they collected timed out queries or successful queries.
>>
>> Andy
>>
>>
>>
>> On 21/11/11 10:47, Paolo Castagna wrote:
>>> Thorsten Möller wrote:
>>>> Hi
>>>>
>>>> I guess some of you have noticed the recent paper on comparing performance of prominent triple stores [1].
>>>
>>> Yep [1].
>>>
>>>> Do you consider it a fair benchmark (set up, data set, queries), considering the fact that TDB's performance isn't that good according to the results. Any objections, comments on the paper?
>>>
>>> Adding DBPSB to the list of JenaPerf benchmarks would probably be an useful contribution:
>>> https://svn.apache.org/repos/asf/incubator/jena/Experimental/JenaPerf/trunk/Benchmarks/
>>> I've had no time to run DBPSB myself or look in details each of the DBPSB queries, unfortunately. But, this is IMHO something useful to do.
>>>
>>> Benchmarks are useful to compare different systems.
>>> However, the best benchmark is the one you build yourself, using your data and the queries you need or your users are more likely to write.
>>> The aim of JenaPerf, as I see it, it to make as easy as possible to add new datasets and queries and run your benchmark.
>>>
>>> Personally, I tend to classify SPARQL queries in two broad categories: interactive and analytic.
>>> For interactive queries, I'd like to have sub-second response times. If that is not always possible, I desperately want a cache layer in front.
>>> For analytic queries, it does not really matter a few seconds or minutes (or hours!) difference. I just want the answer (in a reasonable amount of time).
>>>
>>> IMHO SPARQL (as SQL) is an unconstrained language in term of complexity of queries people can run... this makes a lot of things more "interesting" and challenging. (Compare with heavily constrained
>>> query languages from other NoSQL systems: MQL [2], CQL [3], ...)
>>>
>>> Last but not least, IMHO benchmarks are very useful tools in showing people and developers that there is room for improvement.
>>> Performances and scalability are very important and I welcome any progress TDB and ARQ makes on these aspects.
>>> But, I also value: being an open source project with an active community (and hopefully growing) of users and contributors around (including you! :-)), ease of installation, extensibility, compliance
>>> with W3C recommendations, ease of integration with other (Java) projects, quality and speed of support (via mailing list or commercial alternatives), etc.
>>>
>>> Do you want to help adding DBPSB to JenaPerf? :-)
>>>
>>> Paolo
>>>
>>> [1] http://markmail.org/message/n32zifhgl52kilkz
>>> [2] http://wiki.freebase.com/wiki/MQL
>>> [3] https://www.google.com/#q="Cassandra+Query+Language"
>>>
>>>
>>>>
>>>> Thorsten
>>>>
>>>> [1] http://iswc2011.semanticweb.org/fileadmin/iswc/Papers/Research_Paper/03/70310448.pdf
>>>>
>>>>
>>
>
Re: TDB benchmark performance
Posted by Thorsten Möller <th...@unibas.ch>.
Thanks. I was asking mostly to better understand *why* differences were that large. Reading statements such as "Jena-TDB can in general not deliver competitive performance with being by a factor 30-50 slower than the fastest system." [Section 7 of the paper] sets off alarm bells, meaning that I'm wondering whether the set up, data set, queries were fair. Assuming that query processing and the data backend are not naive in all the systems, I can imagine that differences are to some extent a matter of caching strategies. But then the query mixes become important since they might result in higher cache hits for one system than another.
And I agree with Paolo that benchmarks should distinguish between interactive and analytic queries, which is nicely reflected by the different TPC suites (e.g. TPC-E versus TPC-H).
Thorsten
Am 21.11.2011 um 14:25 schrieb Andy Seaborne:
> Some things could be clearer (and I was in the audience for the paper)
>
> 1/ "Java heap size set 8GB" - TDB does not use the heap for file caching.
>
> (this also messes up Sesame for another reason - a 32 Gb machine and they only used 8G for Sesame)
>
> 2/ Not clear if the machine was configured to use all memory for memory mapped files. It has been reported that some kernel configs are set to limit mmap usage.
>
> 3/ TDB stats and DBpedia data is "interesting". If they just used the default settings, I'd guess the stats (predicate frequencies) are likely to be wild.
>
> My personally feeling is that 1+2 is having an effect. It would interesting to run in direct mode with the in-JVM caches turned up to use 30G or so.
>
> But the real unclarity is the query set - they were asked about this at ISWC and did not give a clear an answer. It is gathered from DBpedia.org - but that has a timeout on queries that people often run into. This (a) skews the queries people ask anyway (b) may skew the queries collected as they did not say whether they collected timed out queries or successful queries.
>
> Andy
>
>
>
> On 21/11/11 10:47, Paolo Castagna wrote:
>> Thorsten Möller wrote:
>>> Hi
>>>
>>> I guess some of you have noticed the recent paper on comparing performance of prominent triple stores [1].
>>
>> Yep [1].
>>
>>> Do you consider it a fair benchmark (set up, data set, queries), considering the fact that TDB's performance isn't that good according to the results. Any objections, comments on the paper?
>>
>> Adding DBPSB to the list of JenaPerf benchmarks would probably be an useful contribution:
>> https://svn.apache.org/repos/asf/incubator/jena/Experimental/JenaPerf/trunk/Benchmarks/
>> I've had no time to run DBPSB myself or look in details each of the DBPSB queries, unfortunately. But, this is IMHO something useful to do.
>>
>> Benchmarks are useful to compare different systems.
>> However, the best benchmark is the one you build yourself, using your data and the queries you need or your users are more likely to write.
>> The aim of JenaPerf, as I see it, it to make as easy as possible to add new datasets and queries and run your benchmark.
>>
>> Personally, I tend to classify SPARQL queries in two broad categories: interactive and analytic.
>> For interactive queries, I'd like to have sub-second response times. If that is not always possible, I desperately want a cache layer in front.
>> For analytic queries, it does not really matter a few seconds or minutes (or hours!) difference. I just want the answer (in a reasonable amount of time).
>>
>> IMHO SPARQL (as SQL) is an unconstrained language in term of complexity of queries people can run... this makes a lot of things more "interesting" and challenging. (Compare with heavily constrained
>> query languages from other NoSQL systems: MQL [2], CQL [3], ...)
>>
>> Last but not least, IMHO benchmarks are very useful tools in showing people and developers that there is room for improvement.
>> Performances and scalability are very important and I welcome any progress TDB and ARQ makes on these aspects.
>> But, I also value: being an open source project with an active community (and hopefully growing) of users and contributors around (including you! :-)), ease of installation, extensibility, compliance
>> with W3C recommendations, ease of integration with other (Java) projects, quality and speed of support (via mailing list or commercial alternatives), etc.
>>
>> Do you want to help adding DBPSB to JenaPerf? :-)
>>
>> Paolo
>>
>> [1] http://markmail.org/message/n32zifhgl52kilkz
>> [2] http://wiki.freebase.com/wiki/MQL
>> [3] https://www.google.com/#q="Cassandra+Query+Language"
>>
>>
>>>
>>> Thorsten
>>>
>>> [1] http://iswc2011.semanticweb.org/fileadmin/iswc/Papers/Research_Paper/03/70310448.pdf
>>>
>>>
>
Re: TDB benchmark performance
Posted by Andy Seaborne <an...@apache.org>.
Some things could be clearer (and I was in the audience for the paper)
1/ "Java heap size set 8GB" - TDB does not use the heap for file caching.
(this also messes up Sesame for another reason - a 32 Gb machine and
they only used 8G for Sesame)
2/ Not clear if the machine was configured to use all memory for memory
mapped files. It has been reported that some kernel configs are set to
limit mmap usage.
3/ TDB stats and DBpedia data is "interesting". If they just used the
default settings, I'd guess the stats (predicate frequencies) are likely
to be wild.
My personally feeling is that 1+2 is having an effect. It would
interesting to run in direct mode with the in-JVM caches turned up to
use 30G or so.
But the real unclarity is the query set - they were asked about this at
ISWC and did not give a clear an answer. It is gathered from
DBpedia.org - but that has a timeout on queries that people often run
into. This (a) skews the queries people ask anyway (b) may skew the
queries collected as they did not say whether they collected timed out
queries or successful queries.
Andy
On 21/11/11 10:47, Paolo Castagna wrote:
> Thorsten Möller wrote:
>> Hi
>>
>> I guess some of you have noticed the recent paper on comparing performance of prominent triple stores [1].
>
> Yep [1].
>
>> Do you consider it a fair benchmark (set up, data set, queries), considering the fact that TDB's performance isn't that good according to the results. Any objections, comments on the paper?
>
> Adding DBPSB to the list of JenaPerf benchmarks would probably be an useful contribution:
> https://svn.apache.org/repos/asf/incubator/jena/Experimental/JenaPerf/trunk/Benchmarks/
> I've had no time to run DBPSB myself or look in details each of the DBPSB queries, unfortunately. But, this is IMHO something useful to do.
>
> Benchmarks are useful to compare different systems.
> However, the best benchmark is the one you build yourself, using your data and the queries you need or your users are more likely to write.
> The aim of JenaPerf, as I see it, it to make as easy as possible to add new datasets and queries and run your benchmark.
>
> Personally, I tend to classify SPARQL queries in two broad categories: interactive and analytic.
> For interactive queries, I'd like to have sub-second response times. If that is not always possible, I desperately want a cache layer in front.
> For analytic queries, it does not really matter a few seconds or minutes (or hours!) difference. I just want the answer (in a reasonable amount of time).
>
> IMHO SPARQL (as SQL) is an unconstrained language in term of complexity of queries people can run... this makes a lot of things more "interesting" and challenging. (Compare with heavily constrained
> query languages from other NoSQL systems: MQL [2], CQL [3], ...)
>
> Last but not least, IMHO benchmarks are very useful tools in showing people and developers that there is room for improvement.
> Performances and scalability are very important and I welcome any progress TDB and ARQ makes on these aspects.
> But, I also value: being an open source project with an active community (and hopefully growing) of users and contributors around (including you! :-)), ease of installation, extensibility, compliance
> with W3C recommendations, ease of integration with other (Java) projects, quality and speed of support (via mailing list or commercial alternatives), etc.
>
> Do you want to help adding DBPSB to JenaPerf? :-)
>
> Paolo
>
> [1] http://markmail.org/message/n32zifhgl52kilkz
> [2] http://wiki.freebase.com/wiki/MQL
> [3] https://www.google.com/#q="Cassandra+Query+Language"
>
>
>>
>> Thorsten
>>
>> [1] http://iswc2011.semanticweb.org/fileadmin/iswc/Papers/Research_Paper/03/70310448.pdf
>>
>>
Re: TDB benchmark performance
Posted by Paolo Castagna <ca...@googlemail.com>.
Thorsten Möller wrote:
> Hi
>
> I guess some of you have noticed the recent paper on comparing performance of prominent triple stores [1].
Yep [1].
> Do you consider it a fair benchmark (set up, data set, queries), considering the fact that TDB's performance isn't that good according to the results. Any objections, comments on the paper?
Adding DBPSB to the list of JenaPerf benchmarks would probably be an useful contribution:
https://svn.apache.org/repos/asf/incubator/jena/Experimental/JenaPerf/trunk/Benchmarks/
I've had no time to run DBPSB myself or look in details each of the DBPSB queries, unfortunately. But, this is IMHO something useful to do.
Benchmarks are useful to compare different systems.
However, the best benchmark is the one you build yourself, using your data and the queries you need or your users are more likely to write.
The aim of JenaPerf, as I see it, it to make as easy as possible to add new datasets and queries and run your benchmark.
Personally, I tend to classify SPARQL queries in two broad categories: interactive and analytic.
For interactive queries, I'd like to have sub-second response times. If that is not always possible, I desperately want a cache layer in front.
For analytic queries, it does not really matter a few seconds or minutes (or hours!) difference. I just want the answer (in a reasonable amount of time).
IMHO SPARQL (as SQL) is an unconstrained language in term of complexity of queries people can run... this makes a lot of things more "interesting" and challenging. (Compare with heavily constrained
query languages from other NoSQL systems: MQL [2], CQL [3], ...)
Last but not least, IMHO benchmarks are very useful tools in showing people and developers that there is room for improvement.
Performances and scalability are very important and I welcome any progress TDB and ARQ makes on these aspects.
But, I also value: being an open source project with an active community (and hopefully growing) of users and contributors around (including you! :-)), ease of installation, extensibility, compliance
with W3C recommendations, ease of integration with other (Java) projects, quality and speed of support (via mailing list or commercial alternatives), etc.
Do you want to help adding DBPSB to JenaPerf? :-)
Paolo
[1] http://markmail.org/message/n32zifhgl52kilkz
[2] http://wiki.freebase.com/wiki/MQL
[3] https://www.google.com/#q="Cassandra+Query+Language"
>
> Thorsten
>
> [1] http://iswc2011.semanticweb.org/fileadmin/iswc/Papers/Research_Paper/03/70310448.pdf
>
>