You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Michael Brunnbauer <br...@netestate.de> on 2012/07/11 17:05:22 UTC
Fuseki 0.2.3 performance / StatsMatcher warnings
hi all
I have just replaced Joseki 3.4.3 with Fuseki 0.2.3 for our projects
www.foaf-search.net and www.semantic-search.info. I have used config-tdb.ttl
as starting point for the configuration. My first impression is that
it is *much* slower. Is this normal ? More configuration details on demand.
I also get these warnings now because of URIs containing spaces in the
stats.opt files:
17:00:10 WARN StatsMatcher :: Not understood: <http://www.w3.org/2003/12/exif/ns/Digital
17:00:10 WARN StatsMatcher :: Not understood: <http://www.w3.org/2003/12/exif/ns/Scene
Could this be the reason for the slow performance ? Can I fix the stats.opt
file by removing lines with bad URIs ?
Regards,
Michael Brunnbauer
--
++ Michael Brunnbauer
++ netEstate GmbH
++ Geisenhausener Straße 11a
++ 81379 München
++ Tel +49 89 32 19 77 80
++ Fax +49 89 32 19 77 89
++ E-Mail brunni@netestate.de
++ http://www.netestate.de/
++
++ Sitz: München, HRB Nr.142452 (Handelsregister B München)
++ USt-IdNr. DE221033342
++ Geschäftsführer: Michael Brunnbauer, Franz Brunnbauer
++ Prokurist: Dipl. Kfm. (Univ.) Markus Hendel
Re: Fuseki 0.2.3 performance / StatsMatcher warnings
Posted by Andy Seaborne <an...@apache.org>.
> For 10K transactions, 3.2s; for 10K transactions and simple query 5.7s.
Now 1.1s and 3.457s for 10k actions of which the JIT and misc setup is
significant because running x10
N = 100k => Time = 4.242 for what was the 1.1s case.
So that's faster (removing a bunch of unnecessary OS system calls).
Unless your machine has some disk-related costs, I can't make these
numbers explain what you see.
Andy
On 12/07/12 14:46, Andy Seaborne wrote:
> On 12/07/12 10:14, Andy Seaborne wrote:
>> On 11/07/12 20:57, Michael Brunnbauer wrote:
>>>> but it could be the transaction overhead
>>> +1
>>>
>>
>> Not that simple ... another prime candidate is not managing connections
>> in the same way e.g. connection caching as is managing the dataset with
>> open/close. These are Fuseki things.
>>
>> The TDB transaction overhead per se is a few wrapper objects (to enforce
>> read-only) and in TDB-only tests, has not shown up as significant in a
>> setup like your as I understand it.
>>
>> Just to check details: what's the config file for Fuseki that you are
>> using or these tests?
>>
>> Andy
>
>
> Michael sent me the config file - the only potentially interesting point
> is that the server has 4 large datasets all attached to the one
> instance. This might cause the OS to get unhappy - but maybe not.
>
> A simple test of transaction overhead put the cost of a read transaction
> under 1ms (raw TDB) so that does not in itself account for
>
> That puts it, at worse, comparable to the cost of a simple query.
>
> For 10K transactions, 3.2s; for 10K transactions and simple query 5.7s.
>
> (and there is maybe some speed-ups as I see a surprising cost showing up
> in YourKit but even that is 20% of the total time of my transaction
> churning so this is not explaining what Michael is seeing)
>
> I can't reproduce the situation: I do really need a similar setup
> because what's going on is not jumping out at me. The performance tests
> I've done do not show the effect you have.
>
> The things to try that occur to be are to simplify the situation:
>
> 1/ Run with only one dataset on the server
>
> 2/ Change the workload: do multiple calls of each query , so do
> q1,q1,q1,q1,q2,q2,q2,q2, etc
>
> Andy
>
>
>
>
Re: Fuseki 0.2.3 performance / StatsMatcher warnings
Posted by Andy Seaborne <an...@apache.org>.
On 05/08/12 12:03, Michael Brunnbauer wrote:
>
> Hello Andy,
>
> much better now. Thank you very much :-)
Good!
>
> I guess it's OK to rename stats.opt to stats.opt.old instead of moving it
> out of the directory ?
Yes - it's the specific name "stats.opt" that the code looks for.
Andy
>
> Regards,
>
> Michael Brunnbauer
>
> On Sat, Aug 04, 2012 at 05:49:41PM +0100, Andy Seaborne wrote:
>> Hi Michael.
>>
>> (Michael has sent me a copy of this database he's using - a bit big too
>> email even though it's only 157,809,969 triples and 44Gbytes).
>>
>> I can recreate this - I'm getting a total of 5006ms for the query set of
>> 17 queries you're using.
>>
>> With a fix, it's 150ms made up of approx 50ms of server-side execution,
>> 100ms of HTTP networking and results transmission for all 17 queries.
>>
>> With the workaround, it's about 170ms.
>>
>> Sorry - the fix will not make the next release, which is already built.
>> Also, my current fix does the right thing for your case but I want
>> make sure it's not got any concurrency problems.
>>
>>
>> Workaround:
>>
>> Remove the "stats.opt" from the database directory and create a file
>> "fixed.opt" in that directory. An empty "fixed.opt" is fine - it's not
>> actually read; it's the presence that matters. Caution - you need to
>> get rid of the stats.opt file as it's used in preference.
>>
>> You'll need to see if the change affects other, more complex queries.
>> dbpedia as a very unusual dataset at the best of times (42K different
>> unique properties). Depressingly, fixed.opt does a reasonably job of
>> optimizing. It simply looks for more tightly constrained triple
>> patterns and mildly avoid rdf:type.
>>
>> (The other optimizer option is "none.opt" when BGPs in queries are
>> executed in the order written. Good for control and experimentation.)
>>
>>
>> Explanation:
>>
>> I said:
>> [[
>> But the performance is not to do with stats - this is all single quad
>> lookup.
>> ]]
>>
>> Not quite true :-) While the statistics themselves don't matter, the
>> system is reading stats.opt too often. Normally, this isn't too
>> important because the file is small, heavily cached and fast to parse
>> (it still shouldn't do it). But the dbpedia stats.opt is big at 2.1
>> Mbytes and 42K entries.
>>
>> Andy
>
Re: Fuseki 0.2.3 performance / StatsMatcher warnings
Posted by Michael Brunnbauer <br...@netestate.de>.
Hello Andy,
much better now. Thank you very much :-)
I guess it's OK to rename stats.opt to stats.opt.old instead of moving it
out of the directory ?
Regards,
Michael Brunnbauer
On Sat, Aug 04, 2012 at 05:49:41PM +0100, Andy Seaborne wrote:
> Hi Michael.
>
> (Michael has sent me a copy of this database he's using - a bit big too
> email even though it's only 157,809,969 triples and 44Gbytes).
>
> I can recreate this - I'm getting a total of 5006ms for the query set of
> 17 queries you're using.
>
> With a fix, it's 150ms made up of approx 50ms of server-side execution,
> 100ms of HTTP networking and results transmission for all 17 queries.
>
> With the workaround, it's about 170ms.
>
> Sorry - the fix will not make the next release, which is already built.
> Also, my current fix does the right thing for your case but I want
> make sure it's not got any concurrency problems.
>
>
> Workaround:
>
> Remove the "stats.opt" from the database directory and create a file
> "fixed.opt" in that directory. An empty "fixed.opt" is fine - it's not
> actually read; it's the presence that matters. Caution - you need to
> get rid of the stats.opt file as it's used in preference.
>
> You'll need to see if the change affects other, more complex queries.
> dbpedia as a very unusual dataset at the best of times (42K different
> unique properties). Depressingly, fixed.opt does a reasonably job of
> optimizing. It simply looks for more tightly constrained triple
> patterns and mildly avoid rdf:type.
>
> (The other optimizer option is "none.opt" when BGPs in queries are
> executed in the order written. Good for control and experimentation.)
>
>
> Explanation:
>
> I said:
> [[
> But the performance is not to do with stats - this is all single quad
> lookup.
> ]]
>
> Not quite true :-) While the statistics themselves don't matter, the
> system is reading stats.opt too often. Normally, this isn't too
> important because the file is small, heavily cached and fast to parse
> (it still shouldn't do it). But the dbpedia stats.opt is big at 2.1
> Mbytes and 42K entries.
>
> Andy
--
++ Michael Brunnbauer
++ netEstate GmbH
++ Geisenhausener Straße 11a
++ 81379 München
++ Tel +49 89 32 19 77 80
++ Fax +49 89 32 19 77 89
++ E-Mail brunni@netestate.de
++ http://www.netestate.de/
++
++ Sitz: München, HRB Nr.142452 (Handelsregister B München)
++ USt-IdNr. DE221033342
++ Geschäftsführer: Michael Brunnbauer, Franz Brunnbauer
++ Prokurist: Dipl. Kfm. (Univ.) Markus Hendel
Re: Fuseki 0.2.3 performance / StatsMatcher warnings
Posted by Andy Seaborne <an...@apache.org>.
Hi Michael.
(Michael has sent me a copy of this database he's using - a bit big too
email even though it's only 157,809,969 triples and 44Gbytes).
I can recreate this - I'm getting a total of 5006ms for the query set of
17 queries you're using.
With a fix, it's 150ms made up of approx 50ms of server-side execution,
100ms of HTTP networking and results transmission for all 17 queries.
With the workaround, it's about 170ms.
Sorry - the fix will not make the next release, which is already built.
Also, my current fix does the right thing for your case but I want
make sure it's not got any concurrency problems.
Workaround:
Remove the "stats.opt" from the database directory and create a file
"fixed.opt" in that directory. An empty "fixed.opt" is fine - it's not
actually read; it's the presence that matters. Caution - you need to
get rid of the stats.opt file as it's used in preference.
You'll need to see if the change affects other, more complex queries.
dbpedia as a very unusual dataset at the best of times (42K different
unique properties). Depressingly, fixed.opt does a reasonably job of
optimizing. It simply looks for more tightly constrained triple
patterns and mildly avoid rdf:type.
(The other optimizer option is "none.opt" when BGPs in queries are
executed in the order written. Good for control and experimentation.)
Explanation:
I said:
[[
But the performance is not to do with stats - this is all single quad
lookup.
]]
Not quite true :-) While the statistics themselves don't matter, the
system is reading stats.opt too often. Normally, this isn't too
important because the file is small, heavily cached and fast to parse
(it still shouldn't do it). But the dbpedia stats.opt is big at 2.1
Mbytes and 42K entries.
Andy
Re: Fuseki 0.2.3 performance / StatsMatcher warnings
Posted by Michael Brunnbauer <br...@netestate.de>.
Hello Andy,
On Sun, Jul 15, 2012 at 08:21:50PM +0100, Andy Seaborne wrote:
>sparql-wrapper can be eliminated by running
>wget -O /dev/null 'http://ts.foaf-search.net:3030/foaf/query?query=....'
This does not make a difference.
> >Running in direct mode (-Dtdb:fileMode=direct and an info levelk logging
> >event happens) would see if it's some weird mmap issue to do with the
> >different Jetty's.
"-Dtdb:fileMode=direct" is supposed to be in JVM_ARGS ? Does not make a
difference (6,6s instead of 6,2s).
> also, running against the dataset not in Fuseki would eliminate Fuseki-isms:
> tdbquery --loc=... '...query...'
> tdbquery --set tdb:fileMode=direct --loc=... '...query...'
This actually takes longer because I found no way to do several queries at
once so the TDB is opened each time.
I can give you a test account or make the 45GB DBpedia TDB available for
download if you want.
Regards,
Michael Brunnbauer
--
++ Michael Brunnbauer
++ netEstate GmbH
++ Geisenhausener Straße 11a
++ 81379 München
++ Tel +49 89 32 19 77 80
++ Fax +49 89 32 19 77 89
++ E-Mail brunni@netestate.de
++ http://www.netestate.de/
++
++ Sitz: München, HRB Nr.142452 (Handelsregister B München)
++ USt-IdNr. DE221033342
++ Geschäftsführer: Michael Brunnbauer, Franz Brunnbauer
++ Prokurist: Dipl. Kfm. (Univ.) Markus Hendel
Re: Fuseki 0.2.3 performance / StatsMatcher warnings
Posted by Andy Seaborne <an...@apache.org>.
> Running in direct mode (-Dtdb:fileMode=direct and an info levelk logging
> event happens) would see if it's some weird mmap issue to do with the
> different Jetty's.
also, running against the dataset not in Fuseki would eliminate Fuseki-isms:
tdbquery --loc=... '...query...'
tdbquery --set tdb:fileMode=direct --loc=... '...query...'
Andy
>
> I have checked that the DB isn't being reopened each time - it's open
> once on start up and not within a run.
>
> Andy
>
>>
>> Regards,
>>
>> Michael Brunnbauer
>>
>
>
Re: Fuseki 0.2.3 performance / StatsMatcher warnings
Posted by Andy Seaborne <an...@apache.org>.
On 13/07/12 11:33, Michael Brunnbauer wrote:
>
> Hello Andy,
>
> On Thu, Jul 12, 2012 at 02:46:49PM +0100, Andy Seaborne wrote:
>> The things to try that occur to be are to simplify the situation:
>>
>> 1/ Run with only one dataset on the server
>
> Does not help.
>
>> 2/ Change the workload: do multiple calls of each query , so do
>> q1,q1,q1,q1,q2,q2,q2,q2, etc
>
> Immediately repeating a query does not make it fast.
>
> I now have a set of queries which is slow against the dbpedia dataset
> (156966301 quads) and very fast against the ontology dataset (12177 quads).
>
> There are results for some of the queries in both datasets. Queries that
> return no results do not seem to be faster.
>
> There is no significant disk activity while I run the queries (the machine has
> 48GB RAM and everything relevant is in memory after the first time I run the
> queries). I can see that Fuseki is under high cpu load (99%) while I run the
> queries. Fuseki has a virtual size high above 4GB - so it seems to use memory
> mapped IO for the TDB files.
Yes - files are memory mapped.
I'm sure you'll understand that it's hard to debug this without access
to a replicated setup. I can try to set up something if you make the
data available but I don't have access to a 48G machine. It is possible
that the large memory is an issue - I have read elsewhere of strange
things happening with mmap I/O. I don't see how the change in TDB
versions is going to make a difference because IIRC nothing has changed
except the read transaction wrapper which I've tested.
sparql-wrapper can be eliminated by running
wget -O /dev/null 'http://ts.foaf-search.net:3030/foaf/query?query=....'
for each query. That does not maintain the connection; but I don't
think sparql-wrappers use of Python http code does either.
The snapshot build of Fuseki 0.2.4-SNAPSHOT has the improvements I
discovered in passing.
The queries will all use the POSG index which may not have paged in but
repeated use should have eliminated that possibility.
Running in direct mode (-Dtdb:fileMode=direct and an info levelk logging
event happens) would see if it's some weird mmap issue to do with the
different Jetty's.
I have checked that the DB isn't being reopened each time - it's open
once on start up and not within a run.
Andy
>
> Regards,
>
> Michael Brunnbauer
>
Re: Fuseki 0.2.3 performance / StatsMatcher warnings
Posted by Michael Brunnbauer <br...@netestate.de>.
Hello Andy,
On Thu, Jul 12, 2012 at 02:46:49PM +0100, Andy Seaborne wrote:
> The things to try that occur to be are to simplify the situation:
>
> 1/ Run with only one dataset on the server
Does not help.
> 2/ Change the workload: do multiple calls of each query , so do
> q1,q1,q1,q1,q2,q2,q2,q2, etc
Immediately repeating a query does not make it fast.
I now have a set of queries which is slow against the dbpedia dataset
(156966301 quads) and very fast against the ontology dataset (12177 quads).
There are results for some of the queries in both datasets. Queries that
return no results do not seem to be faster.
There is no significant disk activity while I run the queries (the machine has
48GB RAM and everything relevant is in memory after the first time I run the
queries). I can see that Fuseki is under high cpu load (99%) while I run the
queries. Fuseki has a virtual size high above 4GB - so it seems to use memory
mapped IO for the TDB files.
Regards,
Michael Brunnbauer
--
++ Michael Brunnbauer
++ netEstate GmbH
++ Geisenhausener Straße 11a
++ 81379 München
++ Tel +49 89 32 19 77 80
++ Fax +49 89 32 19 77 89
++ E-Mail brunni@netestate.de
++ http://www.netestate.de/
++
++ Sitz: München, HRB Nr.142452 (Handelsregister B München)
++ USt-IdNr. DE221033342
++ Geschäftsführer: Michael Brunnbauer, Franz Brunnbauer
++ Prokurist: Dipl. Kfm. (Univ.) Markus Hendel
Re: Fuseki 0.2.3 performance / StatsMatcher warnings
Posted by Andy Seaborne <an...@apache.org>.
On 12/07/12 10:14, Andy Seaborne wrote:
> On 11/07/12 20:57, Michael Brunnbauer wrote:
>>> but it could be the transaction overhead
>> +1
>>
>
> Not that simple ... another prime candidate is not managing connections
> in the same way e.g. connection caching as is managing the dataset with
> open/close. These are Fuseki things.
>
> The TDB transaction overhead per se is a few wrapper objects (to enforce
> read-only) and in TDB-only tests, has not shown up as significant in a
> setup like your as I understand it.
>
> Just to check details: what's the config file for Fuseki that you are
> using or these tests?
>
> Andy
Michael sent me the config file - the only potentially interesting point
is that the server has 4 large datasets all attached to the one
instance. This might cause the OS to get unhappy - but maybe not.
A simple test of transaction overhead put the cost of a read transaction
under 1ms (raw TDB) so that does not in itself account for
That puts it, at worse, comparable to the cost of a simple query.
For 10K transactions, 3.2s; for 10K transactions and simple query 5.7s.
(and there is maybe some speed-ups as I see a surprising cost showing up
in YourKit but even that is 20% of the total time of my transaction
churning so this is not explaining what Michael is seeing)
I can't reproduce the situation: I do really need a similar setup
because what's going on is not jumping out at me. The performance tests
I've done do not show the effect you have.
The things to try that occur to be are to simplify the situation:
1/ Run with only one dataset on the server
2/ Change the workload: do multiple calls of each query , so do
q1,q1,q1,q1,q2,q2,q2,q2, etc
Andy
Re: Fuseki 0.2.3 performance / StatsMatcher warnings
Posted by Andy Seaborne <an...@apache.org>.
On 11/07/12 20:57, Michael Brunnbauer wrote:
>> but it could be the transaction overhead
> +1
>
Not that simple ... another prime candidate is not managing connections
in the same way e.g. connection caching as is managing the dataset with
open/close. These are Fuseki things.
The TDB transaction overhead per se is a few wrapper objects (to enforce
read-only) and in TDB-only tests, has not shown up as significant in a
setup like your as I understand it.
Just to check details: what's the config file for Fuseki that you are
using or these tests?
Andy
Re: Fuseki 0.2.3 performance / StatsMatcher warnings
Posted by Michael Brunnbauer <br...@netestate.de>.
Hello Andy,
On Wed, Jul 11, 2012 at 08:18:53PM +0100, Andy Seaborne wrote:
> Could you try:
>
> 1/ SELECT (count(*) AS ?count) WHERE ...
> and let me know the counts.
6
2
0
0
0
...
> 2/ SELECT (count(DISTINCT *) AS ?count) WHERE ...
> and let me know the counts.
6
2
0
0
0
...
> 3/ Change SELECT DISTINCT ?o to SELECT ?o
Not faster.
> Lots of POST, not GET
> What is sending the queries?
> (in case the network costs are the issue)
http://sparql-wrapper.sourceforge.net/ Version 1.4.1 from localhost
I did not change anything except replace Joseki with Fuseki.
> but it could be the transaction overhead
+1
> >> On how much data?
> >
> >The foaf TDB has 357GB. But I also have performance problems with the
> >dbpedia
> >TDB (45GB). Both use quads.
>
> And how many quads? I don't speak bytes!
The dbpedia TDB (which also makes problems) has 156966301 quads (I just
counted them) and the foaf TDB should have more than 600 mio.
> >>>Can I fix the stats.opt
> >>>file by removing lines with bad URIs ?
> >Done. Does not help (I have restarted fuseki).
> The warnings should go away.
Yes - they are gone.
> >>You should fix the URIs as well.
> >>(They aren't in the vocabulary either)
> There is no such <http://www.w3.org/2003/12/exif/ns/Digital Zoom Ratio>
> in the W3C vocabulary that I can find.
As usual, the bad stuff was crawled.
Regards,
Michael Brunnbauer
--
++ Michael Brunnbauer
++ netEstate GmbH
++ Geisenhausener Straße 11a
++ 81379 München
++ Tel +49 89 32 19 77 80
++ Fax +49 89 32 19 77 89
++ E-Mail brunni@netestate.de
++ http://www.netestate.de/
++
++ Sitz: München, HRB Nr.142452 (Handelsregister B München)
++ USt-IdNr. DE221033342
++ Geschäftsführer: Michael Brunnbauer, Franz Brunnbauer
++ Prokurist: Dipl. Kfm. (Univ.) Markus Hendel
Re: Fuseki 0.2.3 performance / StatsMatcher warnings
Posted by Andy Seaborne <an...@apache.org>.
On 11/07/12 19:59, Michael Brunnbauer wrote:
>
> Hello Andy,
>
> On Wed, Jul 11, 2012 at 07:04:21PM +0100, Andy Seaborne wrote:
>>> I have just replaced Joseki 3.4.3 with Fuseki 0.2.3 for our projects
>>> www.foaf-search.net and www.semantic-search.info.
>> Which version of TDB are you using with Joseki?
>
> I was using the Joseki-3.4.3 distribution which contains tdb-0.8.8.jar.
That's a big jump so there are various things to check out.
Could you try:
1/ SELECT (count(*) AS ?count) WHERE ...
and let me know the counts.
2/ SELECT (count(DISTINCT *) AS ?count) WHERE ...
and let me know the counts.
3/ Change SELECT DISTINCT ?o to SELECT ?o
(more inline)
>
>> It should make little or no difference (for the same version of TDB, and
>> little across TDB's for query).
>> Query? Update?
>
> Query
>
>> What operations are you performing?
>
> All of this should be done in 1-2 seconds instead of 8:
>
..
Lots of POST, not GET
What is sending the queries?
(in case the network costs are the issue)
but it could be the transaction overhead (although that would make it
rather high - this are read-only operations).
>> On how much data?
>
> The foaf TDB has 357GB. But I also have performance problems with the dbpedia
> TDB (45GB). Both use quads.
And how many quads? I don't speak bytes!
>
>> On what OS?
>
> Linux 3.2.21 with JDK 6u33-x64.
>
>> Spaces break the syntax, which is why you have "<...." and no ">"
>> What's the stats.opt got in it?
>
> 45991 lines. Here are the broken ones:
>
> (<http://www.w3.org/2003/12/exif/ns/Digital Zoom Ratio> 1)
> (<http://www.w3.org/2003/12/exif/ns/Scene Capture Type> 1)
>
>>> Can I fix the stats.opt
>>> file by removing lines with bad URIs ?
>>
>> Yes -
>
> Done. Does not help (I have restarted fuseki).
The warnings should go away.
>
>> also try "fixed.opt"
>
> huh ?
instead of a (broken) stats.opt , put in a file called "fixed.opt".
But the performance is not to do with stats - this is all single quad
lookup.
>
>> You should fix the URIs as well.
>> (They aren't in the vocabulary either)
>
> I don't understand.
There is no such <http://www.w3.org/2003/12/exif/ns/Digital Zoom Ratio>
in the W3C vocabulary that I can find.
Andy
>
> Regards,
>
> Michael Brunnbauer
>
Re: Fuseki 0.2.3 performance / StatsMatcher warnings
Posted by Michael Brunnbauer <br...@netestate.de>.
Hello Andy,
On Wed, Jul 11, 2012 at 07:04:21PM +0100, Andy Seaborne wrote:
> >I have just replaced Joseki 3.4.3 with Fuseki 0.2.3 for our projects
> >www.foaf-search.net and www.semantic-search.info.
> Which version of TDB are you using with Joseki?
I was using the Joseki-3.4.3 distribution which contains tdb-0.8.8.jar.
> It should make little or no difference (for the same version of TDB, and
> little across TDB's for query).
> Query? Update?
Query
> What operations are you performing?
All of this should be done in 1-2 seconds instead of 8:
20:51:42 INFO Fuseki :: [649] POST http://ts.foaf-search.net:3030/foaf/query
20:51:42 INFO Fuseki :: [649] Query = SELECT DISTINCT ?o WHERE { GRAPH ?g { <http://dbpedia.org/class/yago/Person100007846> <http://www.w3.org/2000/01/rdf-schema#label> ?o }}
20:51:42 INFO Fuseki :: [649] OK/select
20:51:42 INFO Fuseki :: [649] 200 OK
20:51:42 INFO Fuseki :: [650] POST http://ts.foaf-search.net:3030/foaf/query
20:51:42 INFO Fuseki :: [650] Query = SELECT DISTINCT ?o WHERE { GRAPH ?g { <http://dbpedia.org/class/yago/LivingPeople> <http://www.w3.org/2000/01/rdf-schema#label> ?o }}
20:51:43 INFO Fuseki :: [650] OK/select
20:51:43 INFO Fuseki :: [650] 200 OK
20:51:43 INFO Fuseki :: [651] POST http://ts.foaf-search.net:3030/foaf/query
20:51:43 INFO Fuseki :: [651] Query = SELECT DISTINCT ?o WHERE { GRAPH ?g { <http://www.opengis.net/gml/_Feature> <http://www.w3.org/2000/01/rdf-schema#label> ?o }}
20:51:43 INFO Fuseki :: [651] OK/select
20:51:43 INFO Fuseki :: [651] 200 OK
20:51:43 INFO Fuseki :: [652] POST http://ts.foaf-search.net:3030/foaf/query
20:51:43 INFO Fuseki :: [652] Query = SELECT DISTINCT ?o WHERE { GRAPH ?g { <http://dbpedia.org/class/yago/YagoGeoEntity> <http://www.w3.org/2000/01/rdf-schema#label> ?o }}
20:51:44 INFO Fuseki :: [652] OK/select
20:51:44 INFO Fuseki :: [652] 200 OK
20:51:44 INFO Fuseki :: [653] POST http://ts.foaf-search.net:3030/foaf/query
20:51:44 INFO Fuseki :: [653] Query = SELECT DISTINCT ?o WHERE { GRAPH ?g { <http://dbpedia.org/class/yago/GeoclassPopulatedPlace> <http://www.w3.org/2000/01/rdf-schema#label> ?o }}
20:51:44 INFO Fuseki :: [653] OK/select
20:51:44 INFO Fuseki :: [653] 200 OK
20:51:44 INFO Fuseki :: [654] POST http://ts.foaf-search.net:3030/foaf/query
20:51:44 INFO Fuseki :: [654] Query = SELECT DISTINCT ?o WHERE { GRAPH ?g { <http://purl.org/rss/1.0/channel> <http://www.w3.org/2000/01/rdf-schema#label> ?o }}
20:51:45 INFO Fuseki :: [654] OK/select
20:51:45 INFO Fuseki :: [654] 200 OK
20:51:45 INFO Fuseki :: [655] POST http://ts.foaf-search.net:3030/foaf/query
20:51:45 INFO Fuseki :: [655] Query = SELECT DISTINCT ?o WHERE { GRAPH ?g { <http://dbpedia.org/class/yago/Actor109765278> <http://www.w3.org/2000/01/rdf-schema#label> ?o }}
20:51:45 INFO Fuseki :: [655] OK/select
20:51:45 INFO Fuseki :: [655] 200 OK
20:51:45 INFO Fuseki :: [656] POST http://ts.foaf-search.net:3030/foaf/query
20:51:45 INFO Fuseki :: [656] Query = SELECT DISTINCT ?o WHERE { GRAPH ?g { <http://purl.org/ontology/mo/MusicArtist> <http://www.w3.org/2000/01/rdf-schema#label> ?o }}
20:51:46 INFO Fuseki :: [656] OK/select
20:51:46 INFO Fuseki :: [656] 200 OK
20:51:46 INFO Fuseki :: [657] POST http://ts.foaf-search.net:3030/foaf/query
20:51:46 INFO Fuseki :: [657] Query = SELECT DISTINCT ?o WHERE { GRAPH ?g { <http://purl.org/ontology/bibo/Collection> <http://www.w3.org/2000/01/rdf-schema#label> ?o }}
20:51:47 INFO Fuseki :: [657] OK/select
20:51:47 INFO Fuseki :: [657] 200 OK
20:51:47 INFO Fuseki :: [658] POST http://ts.foaf-search.net:3030/foaf/query
20:51:47 INFO Fuseki :: [658] Query = SELECT DISTINCT ?o WHERE { GRAPH ?g { <http://data.ordnancesurvey.co.uk/ontology/admingeo/CivilParish> <http://www.w3.org/2000/01/rdf-schema#label> ?o }}
20:51:47 INFO Fuseki :: [658] OK/select
20:51:47 INFO Fuseki :: [658] 200 OK
20:51:47 INFO Fuseki :: [659] POST http://ts.foaf-search.net:3030/foaf/query
20:51:47 INFO Fuseki :: [659] Query = SELECT DISTINCT ?o WHERE { GRAPH ?g { <http://dbpedia.org/class/yago/AmericanFilmActors> <http://www.w3.org/2000/01/rdf-schema#label> ?o }}
20:51:48 INFO Fuseki :: [659] OK/select
20:51:48 INFO Fuseki :: [659] 200 OK
20:51:48 INFO Fuseki :: [660] POST http://ts.foaf-search.net:3030/foaf/query
20:51:48 INFO Fuseki :: [660] Query = SELECT DISTINCT ?o WHERE { GRAPH ?g { <http://dbpedia.org/class/yago/TheFootballLeaguePlayers> <http://www.w3.org/2000/01/rdf-schema#label> ?o }}
20:51:48 INFO Fuseki :: [660] OK/select
20:51:48 INFO Fuseki :: [660] 200 OK
20:51:48 INFO Fuseki :: [661] POST http://ts.foaf-search.net:3030/foaf/query
20:51:48 INFO Fuseki :: [661] Query = SELECT DISTINCT ?o WHERE { GRAPH ?g { <http://dbpedia.org/class/yago/EnglishFootballers> <http://www.w3.org/2000/01/rdf-schema#label> ?o }}
20:51:48 INFO Fuseki :: [661] OK/select
20:51:48 INFO Fuseki :: [661] 200 OK
20:51:48 INFO Fuseki :: [662] POST http://ts.foaf-search.net:3030/foaf/query
20:51:48 INFO Fuseki :: [662] Query = SELECT DISTINCT ?o WHERE { GRAPH ?g { <http://dbpedia.org/class/yago/AmericanTelevisionActors> <http://www.w3.org/2000/01/rdf-schema#label> ?o }}
20:51:49 INFO Fuseki :: [662] OK/select
20:51:49 INFO Fuseki :: [662] 200 OK
20:51:49 INFO Fuseki :: [663] POST http://ts.foaf-search.net:3030/foaf/query
20:51:49 INFO Fuseki :: [663] Query = SELECT DISTINCT ?o WHERE { GRAPH ?g { <http://dbpedia.org/class/yago/AssociationFootballForwards> <http://www.w3.org/2000/01/rdf-schema#label> ?o }}
20:51:49 INFO Fuseki :: [663] OK/select
20:51:49 INFO Fuseki :: [663] 200 OK
20:51:49 INFO Fuseki :: [664] POST http://ts.foaf-search.net:3030/foaf/query
20:51:49 INFO Fuseki :: [664] Query = SELECT DISTINCT ?o WHERE { GRAPH ?g { <http://dbpedia.org/class/yago/AssociationFootballDefenders> <http://www.w3.org/2000/01/rdf-schema#label> ?o }}
20:51:50 INFO Fuseki :: [664] OK/select
20:51:50 INFO Fuseki :: [664] 200 OK
20:51:50 INFO Fuseki :: [665] POST http://ts.foaf-search.net:3030/foaf/query
20:51:50 INFO Fuseki :: [665] Query = SELECT DISTINCT ?o WHERE { GRAPH ?g { <http://purl.org/dc/terms/DCMIType/Text> <http://www.w3.org/2000/01/rdf-schema#label> ?o }}
20:51:50 INFO Fuseki :: [665] OK/select
20:51:50 INFO Fuseki :: [665] 200 OK
> On how much data?
The foaf TDB has 357GB. But I also have performance problems with the dbpedia
TDB (45GB). Both use quads.
> On what OS?
Linux 3.2.21 with JDK 6u33-x64.
> Spaces break the syntax, which is why you have "<...." and no ">"
> What's the stats.opt got in it?
45991 lines. Here are the broken ones:
(<http://www.w3.org/2003/12/exif/ns/Digital Zoom Ratio> 1)
(<http://www.w3.org/2003/12/exif/ns/Scene Capture Type> 1)
> >Can I fix the stats.opt
> >file by removing lines with bad URIs ?
>
> Yes -
Done. Does not help (I have restarted fuseki).
> also try "fixed.opt"
huh ?
> You should fix the URIs as well.
> (They aren't in the vocabulary either)
I don't understand.
Regards,
Michael Brunnbauer
--
++ Michael Brunnbauer
++ netEstate GmbH
++ Geisenhausener Straße 11a
++ 81379 München
++ Tel +49 89 32 19 77 80
++ Fax +49 89 32 19 77 89
++ E-Mail brunni@netestate.de
++ http://www.netestate.de/
++
++ Sitz: München, HRB Nr.142452 (Handelsregister B München)
++ USt-IdNr. DE221033342
++ Geschäftsführer: Michael Brunnbauer, Franz Brunnbauer
++ Prokurist: Dipl. Kfm. (Univ.) Markus Hendel
Re: Fuseki 0.2.3 performance / StatsMatcher warnings
Posted by Andy Seaborne <an...@apache.org>.
On 11/07/12 16:05, Michael Brunnbauer wrote:
>
> hi all
>
> I have just replaced Joseki 3.4.3 with Fuseki 0.2.3 for our projects
> www.foaf-search.net and www.semantic-search.info.
Which version of TDB are you using with Joseki?
> I have used config-tdb.ttl
> as starting point for the configuration. My first impression is that
> it is *much* slower. Is this normal ? More configuration details on demand.
It should make little or no difference (for the same version of TDB, and
little across TDB's for query).
Query? Update?
What operations are you performing? On how much data? On what OS?
> I also get these warnings now because of URIs containing spaces in the
> stats.opt files:
> 17:00:10 WARN StatsMatcher :: Not understood: <http://www.w3.org/2003/12/exif/ns/Digital
> 17:00:10 WARN StatsMatcher :: Not understood: <http://www.w3.org/2003/12/exif/ns/Scene
>
> Could this be the reason for the slow performance ?
It certainly will not help.
These are not a good sign.
Spaces break the syntax, which is why you have "<...." and no ">"
What's the stats.opt got in it?
> Can I fix the stats.opt
> file by removing lines with bad URIs ?
Yes - also try "fixed.opt"
You should fix the URIs as well.
(They aren't in the vocabulary either)
>
> Regards,
>
> Michael Brunnbauer
>
Andy