You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jena.apache.org by "Rob Vesse (Created) (JIRA)" <ji...@apache.org> on 2011/12/14 00:13:30 UTC
[jira] [Created] (JENA-178) SPARQL Results serialization and
parsing is slow with large result sets
SPARQL Results serialization and parsing is slow with large result sets
-----------------------------------------------------------------------
Key: JENA-178
URL: https://issues.apache.org/jira/browse/JENA-178
Project: Jena
Issue Type: Bug
Components: ARQ
Affects Versions: ARQ 2.8.9
Environment: Windows 7 Enterprise 64 bit
Reporter: Rob Vesse
The SPARQL XML and JSON Result formats are very slow when the result set is large. This is surprising to me since both formats are relatively simple and should lend themselves to fairly fast streaming serialization and parsing.
The following are observed performance figures comparing SPARQL XML, SPARQL JSON and SPARQL TSV results format. This is the averaged time over 5 runs to retrieve the first 50,000 triples from the dataset with a simple SELECT * WHERE { ?s ?p ?o } LIMIT 50000 via a HTTP request to Fuseki and iterate over the results on the client.
SPARQL XML = 15.25 seconds
SPARQL JSON = 10.9 seconds
SPARQL TSV = 0.54 seconds
Now obviously TSV is way simpler to serialize and parse than XML/JSON but these serializers and parsers should not be 20-30 times slower IMO
Also for comparison note that doing an equivalent CONSTRUCT { ?s ?p ?p } WHERE { ?s ?p ?o } LIMIT 50000 takes only about 2s and that is using RDF/XML serialization which I would have expected to be slower because RDF/XML is more complex to generate than either SPARQL XML/JSON results. I haven't dived into the code in detail to investigate why this is slow yet but do the Jena team have any thoughts on this?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-178) SPARQL Results serialization is slow
for some formats with large result sets
Posted by "Andy Seaborne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176151#comment-13176151 ]
Andy Seaborne commented on JENA-178:
------------------------------------
I changed all Stacks<> in ARQ to Deque<>. This was a Java6 change due that I'd not got round to doing.
This changed XML from about 7000 to 5000 (not completely sound test though - the machine was long running TDB tests at the time so the absolute number isn't meaningful).
The test program Jena178 does not allow for JITting. Attached is Jena178_ResultSetSpeed. It makes more difference to the first-run tests presumably because low-level code is being worked out and it benefits later code.
The JSON looks slow and IndentedWriter is a common component. It does do per-character poking. Making it final did not help. Checking for newlines in printed strings makes some difference.
BufferingWriter looks suspicious even after the encoder improvements after looking with visualvm. It should buffer in string-space, not in byte-space.
We could swap to StAX. Any reason why not?
Current figures:
https://svn.apache.org/repos/asf/incubator/jena/Scratch/AFS/Jena-Dev/trunk/src/dev/Jena178_ResultSetSpeed.java
-- checking for newlines inside strings
XMLSAX out took: 273
XMLStax out took: 272
XML out took: 1940
XML in took: 397
TSV out took: 128
TSV in took: 330
JSON out took: 1782
JSON in took: 746
-- not checking for newlines inside strings.
XMLSAX out took: 326
XMLStax out took: 274
XML out took: 840
XML in took: 452
TSV out took: 130
TSV in took: 306
JSON out took: 908
JSON in took: 761
JVM:
OpenJDK Runtime Environment (IcedTea6 1.11pre) (6b23~pre11-0ubuntu1.11.10)
OpenJDK 64-Bit Server VM (build 20.0-b11, mixed mode)
> SPARQL Results serialization is slow for some formats with large result sets
> ----------------------------------------------------------------------------
>
> Key: JENA-178
> URL: https://issues.apache.org/jira/browse/JENA-178
> Project: Jena
> Issue Type: Bug
> Components: ARQ
> Affects Versions: ARQ 2.9.0
> Environment: Windows 7 Enterprise 64 bit
> Reporter: Rob Vesse
> Assignee: Damian Steer
> Fix For: ARQ 2.9.1
>
> Attachments: Jena178.java, Jena178.patch, TestArqSerializerPerformance.java, XMLOutputSAX.java, XMLOutputStAX.java
>
>
> The SPARQL XML and JSON Result formats are very slow when the result set is large. This is surprising to me since both formats are relatively simple and should lend themselves to fairly fast streaming serialization and parsing.
> The following are observed performance figures comparing SPARQL XML, SPARQL JSON and SPARQL TSV results format. This is the averaged time over 5 runs to retrieve the first 50,000 triples from the dataset with a simple SELECT * WHERE { ?s ?p ?o } LIMIT 50000 via a HTTP request to Fuseki and iterate over the results on the client.
> SPARQL XML = 15.25 seconds
> SPARQL JSON = 10.9 seconds
> SPARQL TSV = 0.54 seconds
> Now obviously TSV is way simpler to serialize and parse than XML/JSON but these serializers and parsers should not be 20-30 times slower IMO
> Also for comparison note that doing an equivalent CONSTRUCT { ?s ?p ?p } WHERE { ?s ?p ?o } LIMIT 50000 takes only about 2s and that is using RDF/XML serialization which I would have expected to be slower because RDF/XML is more complex to generate than either SPARQL XML/JSON results. I haven't dived into the code in detail to investigate why this is slow yet but do the Jena team have any thoughts on this?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-178) SPARQL Results serialization is slow
for some formats with large result sets
Posted by "Damian Steer (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171560#comment-13171560 ]
Damian Steer commented on JENA-178:
-----------------------------------
I've also found that the BufferingWriter is using Bytes.toByteBuffer() on each write. This allocates then deallocates a char encoder from a pool, which appears to be relatively expensive. Perhaps BufferingWriter should buffer characters, then convert just before writing?
> SPARQL Results serialization is slow for some formats with large result sets
> ----------------------------------------------------------------------------
>
> Key: JENA-178
> URL: https://issues.apache.org/jira/browse/JENA-178
> Project: Jena
> Issue Type: Bug
> Components: ARQ
> Affects Versions: ARQ 2.8.9
> Environment: Windows 7 Enterprise 64 bit
> Reporter: Rob Vesse
> Attachments: Jena178.java, TestArqSerializerPerformance.java, XMLOutputSAX.java, XMLOutputStAX.java
>
>
> The SPARQL XML and JSON Result formats are very slow when the result set is large. This is surprising to me since both formats are relatively simple and should lend themselves to fairly fast streaming serialization and parsing.
> The following are observed performance figures comparing SPARQL XML, SPARQL JSON and SPARQL TSV results format. This is the averaged time over 5 runs to retrieve the first 50,000 triples from the dataset with a simple SELECT * WHERE { ?s ?p ?o } LIMIT 50000 via a HTTP request to Fuseki and iterate over the results on the client.
> SPARQL XML = 15.25 seconds
> SPARQL JSON = 10.9 seconds
> SPARQL TSV = 0.54 seconds
> Now obviously TSV is way simpler to serialize and parse than XML/JSON but these serializers and parsers should not be 20-30 times slower IMO
> Also for comparison note that doing an equivalent CONSTRUCT { ?s ?p ?p } WHERE { ?s ?p ?o } LIMIT 50000 takes only about 2s and that is using RDF/XML serialization which I would have expected to be slower because RDF/XML is more complex to generate than either SPARQL XML/JSON results. I haven't dived into the code in detail to investigate why this is slow yet but do the Jena team have any thoughts on this?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (JENA-178) SPARQL Results serialization is slow for
some formats with large result sets
Posted by "Andy Seaborne (Closed) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andy Seaborne closed JENA-178.
------------------------------
> SPARQL Results serialization is slow for some formats with large result sets
> ----------------------------------------------------------------------------
>
> Key: JENA-178
> URL: https://issues.apache.org/jira/browse/JENA-178
> Project: Jena
> Issue Type: Bug
> Components: ARQ
> Affects Versions: ARQ 2.8.9
> Environment: Windows 7 Enterprise 64 bit
> Reporter: Rob Vesse
> Assignee: Damian Steer
> Fix For: ARQ 2.9.1
>
> Attachments: Jena178.java, Jena178.patch, TestArqSerializerPerformance.java, XMLOutputSAX.java, XMLOutputStAX.java
>
>
> The SPARQL XML and JSON Result formats are very slow when the result set is large. This is surprising to me since both formats are relatively simple and should lend themselves to fairly fast streaming serialization and parsing.
> The following are observed performance figures comparing SPARQL XML, SPARQL JSON and SPARQL TSV results format. This is the averaged time over 5 runs to retrieve the first 50,000 triples from the dataset with a simple SELECT * WHERE { ?s ?p ?o } LIMIT 50000 via a HTTP request to Fuseki and iterate over the results on the client.
> SPARQL XML = 15.25 seconds
> SPARQL JSON = 10.9 seconds
> SPARQL TSV = 0.54 seconds
> Now obviously TSV is way simpler to serialize and parse than XML/JSON but these serializers and parsers should not be 20-30 times slower IMO
> Also for comparison note that doing an equivalent CONSTRUCT { ?s ?p ?p } WHERE { ?s ?p ?o } LIMIT 50000 takes only about 2s and that is using RDF/XML serialization which I would have expected to be slower because RDF/XML is more complex to generate than either SPARQL XML/JSON results. I haven't dived into the code in detail to investigate why this is slow yet but do the Jena team have any thoughts on this?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-178) SPARQL Results serialization and
parsing is slow with large result sets
Posted by "Rob Vesse (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168873#comment-13168873 ]
Rob Vesse commented on JENA-178:
--------------------------------
So a couple more notes - for these figures the dataset was LUBM 0 with materialized forward chained inferences giving approx 138,000 triples loaded into a TDB database. The performance problem I've seen is not dataset specific, I've seen similarily bad performance on a variety of datasets when queries yield large results. The Fuseki instance is running on the local machine so the HTTP overhead should be minimal.
I looked into the code and I can't seen anything untoward for SPARQL XML at least, it looks like results are serialized and parsed in a fully streaming fashion. Certainly serialization all appears to be done with string manipulation which should in theory at least be fast. I will try and create a minimal test case that just exercises the parser and the serializer on local in-memory data and would allow easier profiling to see what may be going wrong.
> SPARQL Results serialization and parsing is slow with large result sets
> -----------------------------------------------------------------------
>
> Key: JENA-178
> URL: https://issues.apache.org/jira/browse/JENA-178
> Project: Jena
> Issue Type: Bug
> Components: ARQ
> Affects Versions: ARQ 2.8.9
> Environment: Windows 7 Enterprise 64 bit
> Reporter: Rob Vesse
>
> The SPARQL XML and JSON Result formats are very slow when the result set is large. This is surprising to me since both formats are relatively simple and should lend themselves to fairly fast streaming serialization and parsing.
> The following are observed performance figures comparing SPARQL XML, SPARQL JSON and SPARQL TSV results format. This is the averaged time over 5 runs to retrieve the first 50,000 triples from the dataset with a simple SELECT * WHERE { ?s ?p ?o } LIMIT 50000 via a HTTP request to Fuseki and iterate over the results on the client.
> SPARQL XML = 15.25 seconds
> SPARQL JSON = 10.9 seconds
> SPARQL TSV = 0.54 seconds
> Now obviously TSV is way simpler to serialize and parse than XML/JSON but these serializers and parsers should not be 20-30 times slower IMO
> Also for comparison note that doing an equivalent CONSTRUCT { ?s ?p ?p } WHERE { ?s ?p ?o } LIMIT 50000 takes only about 2s and that is using RDF/XML serialization which I would have expected to be slower because RDF/XML is more complex to generate than either SPARQL XML/JSON results. I haven't dived into the code in detail to investigate why this is slow yet but do the Jena team have any thoughts on this?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (JENA-178) SPARQL Results serialization and
parsing is slow with large result sets
Posted by "Rob Vesse (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rob Vesse updated JENA-178:
---------------------------
Attachment: TestArqSerializerPerformance.java
Simple test case which generates a 50,000 triple file, loads this as an in-memory dataset and does a round-trip serialization test on SPARQL XML, JSON and TSV formats
> SPARQL Results serialization and parsing is slow with large result sets
> -----------------------------------------------------------------------
>
> Key: JENA-178
> URL: https://issues.apache.org/jira/browse/JENA-178
> Project: Jena
> Issue Type: Bug
> Components: ARQ
> Affects Versions: ARQ 2.8.9
> Environment: Windows 7 Enterprise 64 bit
> Reporter: Rob Vesse
> Attachments: TestArqSerializerPerformance.java
>
>
> The SPARQL XML and JSON Result formats are very slow when the result set is large. This is surprising to me since both formats are relatively simple and should lend themselves to fairly fast streaming serialization and parsing.
> The following are observed performance figures comparing SPARQL XML, SPARQL JSON and SPARQL TSV results format. This is the averaged time over 5 runs to retrieve the first 50,000 triples from the dataset with a simple SELECT * WHERE { ?s ?p ?o } LIMIT 50000 via a HTTP request to Fuseki and iterate over the results on the client.
> SPARQL XML = 15.25 seconds
> SPARQL JSON = 10.9 seconds
> SPARQL TSV = 0.54 seconds
> Now obviously TSV is way simpler to serialize and parse than XML/JSON but these serializers and parsers should not be 20-30 times slower IMO
> Also for comparison note that doing an equivalent CONSTRUCT { ?s ?p ?p } WHERE { ?s ?p ?o } LIMIT 50000 takes only about 2s and that is using RDF/XML serialization which I would have expected to be slower because RDF/XML is more complex to generate than either SPARQL XML/JSON results. I haven't dived into the code in detail to investigate why this is slow yet but do the Jena team have any thoughts on this?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (JENA-178) SPARQL Results serialization is slow
for some formats with large result sets
Posted by "Damian Steer (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Damian Steer updated JENA-178:
------------------------------
Attachment: Jena178.java
Here's a more direct demonstration (no query involved here). My results (subject to annoying IO issues on my desktop) are:
XML out took: 12053
XML in took: 1272
TSV out took: 749
TSV in took: 659
JSON out took: 9205
JSON in took: 1825
Interesting tidbit: TSV uses bindings not query solutions, unlike the other two.
> SPARQL Results serialization is slow for some formats with large result sets
> ----------------------------------------------------------------------------
>
> Key: JENA-178
> URL: https://issues.apache.org/jira/browse/JENA-178
> Project: Jena
> Issue Type: Bug
> Components: ARQ
> Affects Versions: ARQ 2.8.9
> Environment: Windows 7 Enterprise 64 bit
> Reporter: Rob Vesse
> Attachments: Jena178.java, TestArqSerializerPerformance.java
>
>
> The SPARQL XML and JSON Result formats are very slow when the result set is large. This is surprising to me since both formats are relatively simple and should lend themselves to fairly fast streaming serialization and parsing.
> The following are observed performance figures comparing SPARQL XML, SPARQL JSON and SPARQL TSV results format. This is the averaged time over 5 runs to retrieve the first 50,000 triples from the dataset with a simple SELECT * WHERE { ?s ?p ?o } LIMIT 50000 via a HTTP request to Fuseki and iterate over the results on the client.
> SPARQL XML = 15.25 seconds
> SPARQL JSON = 10.9 seconds
> SPARQL TSV = 0.54 seconds
> Now obviously TSV is way simpler to serialize and parse than XML/JSON but these serializers and parsers should not be 20-30 times slower IMO
> Also for comparison note that doing an equivalent CONSTRUCT { ?s ?p ?p } WHERE { ?s ?p ?o } LIMIT 50000 takes only about 2s and that is using RDF/XML serialization which I would have expected to be slower because RDF/XML is more complex to generate than either SPARQL XML/JSON results. I haven't dived into the code in detail to investigate why this is slow yet but do the Jena team have any thoughts on this?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (JENA-178) SPARQL Results serialization is slow
for some formats with large result sets
Posted by "Rob Vesse (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rob Vesse updated JENA-178:
---------------------------
Summary: SPARQL Results serialization is slow for some formats with large result sets (was: SPARQL Results serialization and parsing is slow with large result sets)
Changed title in light of my investigations which appear to show that it is serialization not parsing that is the issue
> SPARQL Results serialization is slow for some formats with large result sets
> ----------------------------------------------------------------------------
>
> Key: JENA-178
> URL: https://issues.apache.org/jira/browse/JENA-178
> Project: Jena
> Issue Type: Bug
> Components: ARQ
> Affects Versions: ARQ 2.8.9
> Environment: Windows 7 Enterprise 64 bit
> Reporter: Rob Vesse
> Attachments: TestArqSerializerPerformance.java
>
>
> The SPARQL XML and JSON Result formats are very slow when the result set is large. This is surprising to me since both formats are relatively simple and should lend themselves to fairly fast streaming serialization and parsing.
> The following are observed performance figures comparing SPARQL XML, SPARQL JSON and SPARQL TSV results format. This is the averaged time over 5 runs to retrieve the first 50,000 triples from the dataset with a simple SELECT * WHERE { ?s ?p ?o } LIMIT 50000 via a HTTP request to Fuseki and iterate over the results on the client.
> SPARQL XML = 15.25 seconds
> SPARQL JSON = 10.9 seconds
> SPARQL TSV = 0.54 seconds
> Now obviously TSV is way simpler to serialize and parse than XML/JSON but these serializers and parsers should not be 20-30 times slower IMO
> Also for comparison note that doing an equivalent CONSTRUCT { ?s ?p ?p } WHERE { ?s ?p ?o } LIMIT 50000 takes only about 2s and that is using RDF/XML serialization which I would have expected to be slower because RDF/XML is more complex to generate than either SPARQL XML/JSON results. I haven't dived into the code in detail to investigate why this is slow yet but do the Jena team have any thoughts on this?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-178) SPARQL Results serialization and
parsing is slow with large result sets
Posted by "Rob Vesse (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168904#comment-13168904 ]
Rob Vesse commented on JENA-178:
--------------------------------
>From my attached test case I get the following results:
Took 0.068863659s to do execSelect()
Took 9.720043614s to do outputAsXML()
Took 0.77990671s to do fromXML() and iterate over result set
Took 3.27026E-4s to do execSelect()
Took 6.075415556s to do outputAsJSON()
Took 1.291390952s to do fromJSON() and iterate over result set
Took 3.0173E-4s to do execSelect()
Took 0.382692228s to do outputAsTSV()
Took 0.821681056s to do fromTSV() and iterate over result set
So there is some HTTP overhead involved in my original figures but essentially it it still the case that XML serialization is 20x slower than TSV and JSON serialization is 15x slower than TSV
Parsing all these formats is pretty fast so looks to be the serializers that are at fault
> SPARQL Results serialization and parsing is slow with large result sets
> -----------------------------------------------------------------------
>
> Key: JENA-178
> URL: https://issues.apache.org/jira/browse/JENA-178
> Project: Jena
> Issue Type: Bug
> Components: ARQ
> Affects Versions: ARQ 2.8.9
> Environment: Windows 7 Enterprise 64 bit
> Reporter: Rob Vesse
> Attachments: TestArqSerializerPerformance.java
>
>
> The SPARQL XML and JSON Result formats are very slow when the result set is large. This is surprising to me since both formats are relatively simple and should lend themselves to fairly fast streaming serialization and parsing.
> The following are observed performance figures comparing SPARQL XML, SPARQL JSON and SPARQL TSV results format. This is the averaged time over 5 runs to retrieve the first 50,000 triples from the dataset with a simple SELECT * WHERE { ?s ?p ?o } LIMIT 50000 via a HTTP request to Fuseki and iterate over the results on the client.
> SPARQL XML = 15.25 seconds
> SPARQL JSON = 10.9 seconds
> SPARQL TSV = 0.54 seconds
> Now obviously TSV is way simpler to serialize and parse than XML/JSON but these serializers and parsers should not be 20-30 times slower IMO
> Also for comparison note that doing an equivalent CONSTRUCT { ?s ?p ?p } WHERE { ?s ?p ?o } LIMIT 50000 takes only about 2s and that is using RDF/XML serialization which I would have expected to be slower because RDF/XML is more complex to generate than either SPARQL XML/JSON results. I haven't dived into the code in detail to investigate why this is slow yet but do the Jena team have any thoughts on this?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-178) SPARQL Results serialization is slow
for some formats with large result sets
Posted by "Andy Seaborne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176157#comment-13176157 ]
Andy Seaborne commented on JENA-178:
------------------------------------
Using java.io.OutputStreamWriter and a java.io.BufferedWriter (8K buffer):
== On the 10th iteration (not checking for newlines)
XMLSAX out took: 275
XMLStax out took: 273
XML out took: 264
XML in took: 490
TSV out took: 124
TSV in took: 315
JSON out took: 580
JSON in took: 732
== On the 10th iteration (checking for newlines)
XMLSAX out took: 338
XMLStax out took: 275
XML out took: 439
XML in took: 462
TSV out took: 187
TSV in took: 324
JSON out took: 760
JSON in took: 773
Checked in (IndentedWriter left checking for in-string newlines for now).
revision 1224902
> SPARQL Results serialization is slow for some formats with large result sets
> ----------------------------------------------------------------------------
>
> Key: JENA-178
> URL: https://issues.apache.org/jira/browse/JENA-178
> Project: Jena
> Issue Type: Bug
> Components: ARQ
> Affects Versions: ARQ 2.9.0
> Environment: Windows 7 Enterprise 64 bit
> Reporter: Rob Vesse
> Assignee: Damian Steer
> Fix For: ARQ 2.9.1
>
> Attachments: Jena178.java, Jena178.patch, TestArqSerializerPerformance.java, XMLOutputSAX.java, XMLOutputStAX.java
>
>
> The SPARQL XML and JSON Result formats are very slow when the result set is large. This is surprising to me since both formats are relatively simple and should lend themselves to fairly fast streaming serialization and parsing.
> The following are observed performance figures comparing SPARQL XML, SPARQL JSON and SPARQL TSV results format. This is the averaged time over 5 runs to retrieve the first 50,000 triples from the dataset with a simple SELECT * WHERE { ?s ?p ?o } LIMIT 50000 via a HTTP request to Fuseki and iterate over the results on the client.
> SPARQL XML = 15.25 seconds
> SPARQL JSON = 10.9 seconds
> SPARQL TSV = 0.54 seconds
> Now obviously TSV is way simpler to serialize and parse than XML/JSON but these serializers and parsers should not be 20-30 times slower IMO
> Also for comparison note that doing an equivalent CONSTRUCT { ?s ?p ?p } WHERE { ?s ?p ?o } LIMIT 50000 takes only about 2s and that is using RDF/XML serialization which I would have expected to be slower because RDF/XML is more complex to generate than either SPARQL XML/JSON results. I haven't dived into the code in detail to investigate why this is slow yet but do the Jena team have any thoughts on this?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-178) SPARQL Results serialization is slow
for some formats with large result sets
Posted by "Damian Steer (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175584#comment-13175584 ]
Damian Steer commented on JENA-178:
-----------------------------------
Committed.
I've added a get/setFlushOnNewline option on the IndentedWriter.
As for the pool, I was surprised when it showed up in profiling. A good part of this is due to the BufferingWriter retrieving an encoder on every write, so it really exercises the pool.
I'll mark this as resolved although there's still scope for improvement.
> SPARQL Results serialization is slow for some formats with large result sets
> ----------------------------------------------------------------------------
>
> Key: JENA-178
> URL: https://issues.apache.org/jira/browse/JENA-178
> Project: Jena
> Issue Type: Bug
> Components: ARQ
> Affects Versions: ARQ 2.8.9
> Environment: Windows 7 Enterprise 64 bit
> Reporter: Rob Vesse
> Attachments: Jena178.java, Jena178.patch, TestArqSerializerPerformance.java, XMLOutputSAX.java, XMLOutputStAX.java
>
>
> The SPARQL XML and JSON Result formats are very slow when the result set is large. This is surprising to me since both formats are relatively simple and should lend themselves to fairly fast streaming serialization and parsing.
> The following are observed performance figures comparing SPARQL XML, SPARQL JSON and SPARQL TSV results format. This is the averaged time over 5 runs to retrieve the first 50,000 triples from the dataset with a simple SELECT * WHERE { ?s ?p ?o } LIMIT 50000 via a HTTP request to Fuseki and iterate over the results on the client.
> SPARQL XML = 15.25 seconds
> SPARQL JSON = 10.9 seconds
> SPARQL TSV = 0.54 seconds
> Now obviously TSV is way simpler to serialize and parse than XML/JSON but these serializers and parsers should not be 20-30 times slower IMO
> Also for comparison note that doing an equivalent CONSTRUCT { ?s ?p ?p } WHERE { ?s ?p ?o } LIMIT 50000 takes only about 2s and that is using RDF/XML serialization which I would have expected to be slower because RDF/XML is more complex to generate than either SPARQL XML/JSON results. I haven't dived into the code in detail to investigate why this is slow yet but do the Jena team have any thoughts on this?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (JENA-178) SPARQL Results serialization is slow
for some formats with large result sets
Posted by "Stephen Allen (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stephen Allen updated JENA-178:
-------------------------------
Attachment: XMLOutputStAX.java
XMLOutputSAX.java
The current XMLOutput manually generates the XML output.
I've created versions that use SAX and StAX, and they are both much faster (comparable to the TSV output). [1]
XMLOutput took: 33534
XMLOutputSAX took: 2141
XMLOutputSAX (no indenting) took: 1569
XMLOutputStAX (no indenting) took: 2780
TSV took: 716
JSON took: 15656
We may want to do something similar with JSON and use a library.
[1] They should both be streaming, but I'm less confident about what's going on in the SAX version. Also, I'm not quite sure how to do indenting in the StAX version, maybe I have to handle it manually?
> SPARQL Results serialization is slow for some formats with large result sets
> ----------------------------------------------------------------------------
>
> Key: JENA-178
> URL: https://issues.apache.org/jira/browse/JENA-178
> Project: Jena
> Issue Type: Bug
> Components: ARQ
> Affects Versions: ARQ 2.8.9
> Environment: Windows 7 Enterprise 64 bit
> Reporter: Rob Vesse
> Attachments: Jena178.java, TestArqSerializerPerformance.java, XMLOutputSAX.java, XMLOutputStAX.java
>
>
> The SPARQL XML and JSON Result formats are very slow when the result set is large. This is surprising to me since both formats are relatively simple and should lend themselves to fairly fast streaming serialization and parsing.
> The following are observed performance figures comparing SPARQL XML, SPARQL JSON and SPARQL TSV results format. This is the averaged time over 5 runs to retrieve the first 50,000 triples from the dataset with a simple SELECT * WHERE { ?s ?p ?o } LIMIT 50000 via a HTTP request to Fuseki and iterate over the results on the client.
> SPARQL XML = 15.25 seconds
> SPARQL JSON = 10.9 seconds
> SPARQL TSV = 0.54 seconds
> Now obviously TSV is way simpler to serialize and parse than XML/JSON but these serializers and parsers should not be 20-30 times slower IMO
> Also for comparison note that doing an equivalent CONSTRUCT { ?s ?p ?p } WHERE { ?s ?p ?o } LIMIT 50000 takes only about 2s and that is using RDF/XML serialization which I would have expected to be slower because RDF/XML is more complex to generate than either SPARQL XML/JSON results. I haven't dived into the code in detail to investigate why this is slow yet but do the Jena team have any thoughts on this?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-178) SPARQL Results serialization is slow
for some formats with large result sets
Posted by "Damian Steer (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171519#comment-13171519 ]
Damian Steer commented on JENA-178:
-----------------------------------
Exploring a little more I noticed that there were an awful lot of flushes going on. org.openjena.atlas.io.IndentedWriter was responsible:
public void newline()
{
....
// Note that PrintWriters do not autoflush by default
// so if layered over a PrintWirter, need to flush that as well.
flush() ;
}
This in practice negates the BufferingWriter buffering, and removing it halves the XML output time.
Next we have the RIOT BufferingWriter. Comparing:
writer = BufferingWriter.create(out); // RIOT
and
writer = new BufferedWriter(new OutputStreamWriter(out, "UTF-8")); // JDK
I found the former took 3 times longer to write random strings to a file.
> SPARQL Results serialization is slow for some formats with large result sets
> ----------------------------------------------------------------------------
>
> Key: JENA-178
> URL: https://issues.apache.org/jira/browse/JENA-178
> Project: Jena
> Issue Type: Bug
> Components: ARQ
> Affects Versions: ARQ 2.8.9
> Environment: Windows 7 Enterprise 64 bit
> Reporter: Rob Vesse
> Attachments: Jena178.java, TestArqSerializerPerformance.java, XMLOutputSAX.java, XMLOutputStAX.java
>
>
> The SPARQL XML and JSON Result formats are very slow when the result set is large. This is surprising to me since both formats are relatively simple and should lend themselves to fairly fast streaming serialization and parsing.
> The following are observed performance figures comparing SPARQL XML, SPARQL JSON and SPARQL TSV results format. This is the averaged time over 5 runs to retrieve the first 50,000 triples from the dataset with a simple SELECT * WHERE { ?s ?p ?o } LIMIT 50000 via a HTTP request to Fuseki and iterate over the results on the client.
> SPARQL XML = 15.25 seconds
> SPARQL JSON = 10.9 seconds
> SPARQL TSV = 0.54 seconds
> Now obviously TSV is way simpler to serialize and parse than XML/JSON but these serializers and parsers should not be 20-30 times slower IMO
> Also for comparison note that doing an equivalent CONSTRUCT { ?s ?p ?p } WHERE { ?s ?p ?o } LIMIT 50000 takes only about 2s and that is using RDF/XML serialization which I would have expected to be slower because RDF/XML is more complex to generate than either SPARQL XML/JSON results. I haven't dived into the code in detail to investigate why this is slow yet but do the Jena team have any thoughts on this?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-178) SPARQL Results serialization is slow
for some formats with large result sets
Posted by "Andy Seaborne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175473#comment-13175473 ]
Andy Seaborne commented on JENA-178:
------------------------------------
The patch looks good. I vote it gets applied. Neither of my points below affect that.
IndentedWriter ought to default to no line-based flushing.
I wonder if it ought to have a "setAutoFlush(boolean)" with the default being no flush on end of line.
I'm unpleasantly surprised that CharsetEncoder pool didn't help. I wonder if there is a bug and the pool is, in fact, empty so that a new encoder is created too often.
> SPARQL Results serialization is slow for some formats with large result sets
> ----------------------------------------------------------------------------
>
> Key: JENA-178
> URL: https://issues.apache.org/jira/browse/JENA-178
> Project: Jena
> Issue Type: Bug
> Components: ARQ
> Affects Versions: ARQ 2.8.9
> Environment: Windows 7 Enterprise 64 bit
> Reporter: Rob Vesse
> Attachments: Jena178.java, Jena178.patch, TestArqSerializerPerformance.java, XMLOutputSAX.java, XMLOutputStAX.java
>
>
> The SPARQL XML and JSON Result formats are very slow when the result set is large. This is surprising to me since both formats are relatively simple and should lend themselves to fairly fast streaming serialization and parsing.
> The following are observed performance figures comparing SPARQL XML, SPARQL JSON and SPARQL TSV results format. This is the averaged time over 5 runs to retrieve the first 50,000 triples from the dataset with a simple SELECT * WHERE { ?s ?p ?o } LIMIT 50000 via a HTTP request to Fuseki and iterate over the results on the client.
> SPARQL XML = 15.25 seconds
> SPARQL JSON = 10.9 seconds
> SPARQL TSV = 0.54 seconds
> Now obviously TSV is way simpler to serialize and parse than XML/JSON but these serializers and parsers should not be 20-30 times slower IMO
> Also for comparison note that doing an equivalent CONSTRUCT { ?s ?p ?p } WHERE { ?s ?p ?o } LIMIT 50000 takes only about 2s and that is using RDF/XML serialization which I would have expected to be slower because RDF/XML is more complex to generate than either SPARQL XML/JSON results. I haven't dived into the code in detail to investigate why this is slow yet but do the Jena team have any thoughts on this?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-178) SPARQL Results serialization is slow
for some formats with large result sets
Posted by "Stephen Allen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176034#comment-13176034 ]
Stephen Allen commented on JENA-178:
------------------------------------
I shaved ~100ms off of the output by eliminating string concatenation in XMLOutputResultSet. Still lagging behind SAX and StAX though. Committed in revision 1224827.
XML Before: 6829
XML After: 6733
TSV: 801
JSON: 6581
XML SAX: 2057
XML StAX: 1371
> SPARQL Results serialization is slow for some formats with large result sets
> ----------------------------------------------------------------------------
>
> Key: JENA-178
> URL: https://issues.apache.org/jira/browse/JENA-178
> Project: Jena
> Issue Type: Bug
> Components: ARQ
> Affects Versions: ARQ 2.9.0
> Environment: Windows 7 Enterprise 64 bit
> Reporter: Rob Vesse
> Assignee: Damian Steer
> Fix For: ARQ 2.9.1
>
> Attachments: Jena178.java, Jena178.patch, TestArqSerializerPerformance.java, XMLOutputSAX.java, XMLOutputStAX.java
>
>
> The SPARQL XML and JSON Result formats are very slow when the result set is large. This is surprising to me since both formats are relatively simple and should lend themselves to fairly fast streaming serialization and parsing.
> The following are observed performance figures comparing SPARQL XML, SPARQL JSON and SPARQL TSV results format. This is the averaged time over 5 runs to retrieve the first 50,000 triples from the dataset with a simple SELECT * WHERE { ?s ?p ?o } LIMIT 50000 via a HTTP request to Fuseki and iterate over the results on the client.
> SPARQL XML = 15.25 seconds
> SPARQL JSON = 10.9 seconds
> SPARQL TSV = 0.54 seconds
> Now obviously TSV is way simpler to serialize and parse than XML/JSON but these serializers and parsers should not be 20-30 times slower IMO
> Also for comparison note that doing an equivalent CONSTRUCT { ?s ?p ?p } WHERE { ?s ?p ?o } LIMIT 50000 takes only about 2s and that is using RDF/XML serialization which I would have expected to be slower because RDF/XML is more complex to generate than either SPARQL XML/JSON results. I haven't dived into the code in detail to investigate why this is slow yet but do the Jena team have any thoughts on this?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (JENA-178) SPARQL Results serialization is slow
for some formats with large result sets
Posted by "Damian Steer (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Damian Steer updated JENA-178:
------------------------------
Attachment: Jena178.patch
This patch:
* Stops the indenter flushing on every new line.
* Replaces the XML output char escape function.
* Uses the same CharsetEncoder for all BufferingWriter conversions.
Before:
XML: 2651.63ms +- 77.02ms
TSV: 52.13ms +- 1.90ms
JSON: 2346.25ms +- 306.44ms
After:
XML: 541.63ms +- 11.52ms
TSV: 51.75ms +- 1.92ms
JSON: 600.13ms +- 6.62ms
So much better, but still needs some work. I should look at the SAX and StAX performance.
> SPARQL Results serialization is slow for some formats with large result sets
> ----------------------------------------------------------------------------
>
> Key: JENA-178
> URL: https://issues.apache.org/jira/browse/JENA-178
> Project: Jena
> Issue Type: Bug
> Components: ARQ
> Affects Versions: ARQ 2.8.9
> Environment: Windows 7 Enterprise 64 bit
> Reporter: Rob Vesse
> Attachments: Jena178.java, Jena178.patch, TestArqSerializerPerformance.java, XMLOutputSAX.java, XMLOutputStAX.java
>
>
> The SPARQL XML and JSON Result formats are very slow when the result set is large. This is surprising to me since both formats are relatively simple and should lend themselves to fairly fast streaming serialization and parsing.
> The following are observed performance figures comparing SPARQL XML, SPARQL JSON and SPARQL TSV results format. This is the averaged time over 5 runs to retrieve the first 50,000 triples from the dataset with a simple SELECT * WHERE { ?s ?p ?o } LIMIT 50000 via a HTTP request to Fuseki and iterate over the results on the client.
> SPARQL XML = 15.25 seconds
> SPARQL JSON = 10.9 seconds
> SPARQL TSV = 0.54 seconds
> Now obviously TSV is way simpler to serialize and parse than XML/JSON but these serializers and parsers should not be 20-30 times slower IMO
> Also for comparison note that doing an equivalent CONSTRUCT { ?s ?p ?p } WHERE { ?s ?p ?o } LIMIT 50000 takes only about 2s and that is using RDF/XML serialization which I would have expected to be slower because RDF/XML is more complex to generate than either SPARQL XML/JSON results. I haven't dived into the code in detail to investigate why this is slow yet but do the Jena team have any thoughts on this?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-178) SPARQL Results serialization is slow
for some formats with large result sets
Posted by "Damian Steer (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169416#comment-13169416 ]
Damian Steer commented on JENA-178:
-----------------------------------
Little profiling suggests org.openjena.atlas.io.BufferingWriter.SinkOutputStream#send is eating time.
> SPARQL Results serialization is slow for some formats with large result sets
> ----------------------------------------------------------------------------
>
> Key: JENA-178
> URL: https://issues.apache.org/jira/browse/JENA-178
> Project: Jena
> Issue Type: Bug
> Components: ARQ
> Affects Versions: ARQ 2.8.9
> Environment: Windows 7 Enterprise 64 bit
> Reporter: Rob Vesse
> Attachments: Jena178.java, TestArqSerializerPerformance.java
>
>
> The SPARQL XML and JSON Result formats are very slow when the result set is large. This is surprising to me since both formats are relatively simple and should lend themselves to fairly fast streaming serialization and parsing.
> The following are observed performance figures comparing SPARQL XML, SPARQL JSON and SPARQL TSV results format. This is the averaged time over 5 runs to retrieve the first 50,000 triples from the dataset with a simple SELECT * WHERE { ?s ?p ?o } LIMIT 50000 via a HTTP request to Fuseki and iterate over the results on the client.
> SPARQL XML = 15.25 seconds
> SPARQL JSON = 10.9 seconds
> SPARQL TSV = 0.54 seconds
> Now obviously TSV is way simpler to serialize and parse than XML/JSON but these serializers and parsers should not be 20-30 times slower IMO
> Also for comparison note that doing an equivalent CONSTRUCT { ?s ?p ?p } WHERE { ?s ?p ?o } LIMIT 50000 takes only about 2s and that is using RDF/XML serialization which I would have expected to be slower because RDF/XML is more complex to generate than either SPARQL XML/JSON results. I haven't dived into the code in detail to investigate why this is slow yet but do the Jena team have any thoughts on this?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (JENA-178) SPARQL Results serialization is slow
for some formats with large result sets
Posted by "Damian Steer (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Damian Steer resolved JENA-178.
-------------------------------
Resolution: Fixed
Assignee: Damian Steer
> SPARQL Results serialization is slow for some formats with large result sets
> ----------------------------------------------------------------------------
>
> Key: JENA-178
> URL: https://issues.apache.org/jira/browse/JENA-178
> Project: Jena
> Issue Type: Bug
> Components: ARQ
> Affects Versions: ARQ 2.8.9
> Environment: Windows 7 Enterprise 64 bit
> Reporter: Rob Vesse
> Assignee: Damian Steer
> Attachments: Jena178.java, Jena178.patch, TestArqSerializerPerformance.java, XMLOutputSAX.java, XMLOutputStAX.java
>
>
> The SPARQL XML and JSON Result formats are very slow when the result set is large. This is surprising to me since both formats are relatively simple and should lend themselves to fairly fast streaming serialization and parsing.
> The following are observed performance figures comparing SPARQL XML, SPARQL JSON and SPARQL TSV results format. This is the averaged time over 5 runs to retrieve the first 50,000 triples from the dataset with a simple SELECT * WHERE { ?s ?p ?o } LIMIT 50000 via a HTTP request to Fuseki and iterate over the results on the client.
> SPARQL XML = 15.25 seconds
> SPARQL JSON = 10.9 seconds
> SPARQL TSV = 0.54 seconds
> Now obviously TSV is way simpler to serialize and parse than XML/JSON but these serializers and parsers should not be 20-30 times slower IMO
> Also for comparison note that doing an equivalent CONSTRUCT { ?s ?p ?p } WHERE { ?s ?p ?o } LIMIT 50000 takes only about 2s and that is using RDF/XML serialization which I would have expected to be slower because RDF/XML is more complex to generate than either SPARQL XML/JSON results. I haven't dived into the code in detail to investigate why this is slow yet but do the Jena team have any thoughts on this?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-178) SPARQL Results serialization is slow
for some formats with large result sets
Posted by "Stephen Allen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170646#comment-13170646 ]
Stephen Allen commented on JENA-178:
------------------------------------
Additional note, make sure to modify the test class to use a BufferedOutputStream, or performance will be terrible if you write to disk, especially on the StAX version.
> SPARQL Results serialization is slow for some formats with large result sets
> ----------------------------------------------------------------------------
>
> Key: JENA-178
> URL: https://issues.apache.org/jira/browse/JENA-178
> Project: Jena
> Issue Type: Bug
> Components: ARQ
> Affects Versions: ARQ 2.8.9
> Environment: Windows 7 Enterprise 64 bit
> Reporter: Rob Vesse
> Attachments: Jena178.java, TestArqSerializerPerformance.java, XMLOutputSAX.java, XMLOutputStAX.java
>
>
> The SPARQL XML and JSON Result formats are very slow when the result set is large. This is surprising to me since both formats are relatively simple and should lend themselves to fairly fast streaming serialization and parsing.
> The following are observed performance figures comparing SPARQL XML, SPARQL JSON and SPARQL TSV results format. This is the averaged time over 5 runs to retrieve the first 50,000 triples from the dataset with a simple SELECT * WHERE { ?s ?p ?o } LIMIT 50000 via a HTTP request to Fuseki and iterate over the results on the client.
> SPARQL XML = 15.25 seconds
> SPARQL JSON = 10.9 seconds
> SPARQL TSV = 0.54 seconds
> Now obviously TSV is way simpler to serialize and parse than XML/JSON but these serializers and parsers should not be 20-30 times slower IMO
> Also for comparison note that doing an equivalent CONSTRUCT { ?s ?p ?p } WHERE { ?s ?p ?o } LIMIT 50000 takes only about 2s and that is using RDF/XML serialization which I would have expected to be slower because RDF/XML is more complex to generate than either SPARQL XML/JSON results. I haven't dived into the code in detail to investigate why this is slow yet but do the Jena team have any thoughts on this?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (JENA-178) SPARQL Results serialization is slow
for some formats with large result sets
Posted by "Andy Seaborne (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andy Seaborne updated JENA-178:
-------------------------------
Fix Version/s: ARQ 2.9.1
> SPARQL Results serialization is slow for some formats with large result sets
> ----------------------------------------------------------------------------
>
> Key: JENA-178
> URL: https://issues.apache.org/jira/browse/JENA-178
> Project: Jena
> Issue Type: Bug
> Components: ARQ
> Affects Versions: ARQ 2.8.9
> Environment: Windows 7 Enterprise 64 bit
> Reporter: Rob Vesse
> Assignee: Damian Steer
> Fix For: ARQ 2.9.1
>
> Attachments: Jena178.java, Jena178.patch, TestArqSerializerPerformance.java, XMLOutputSAX.java, XMLOutputStAX.java
>
>
> The SPARQL XML and JSON Result formats are very slow when the result set is large. This is surprising to me since both formats are relatively simple and should lend themselves to fairly fast streaming serialization and parsing.
> The following are observed performance figures comparing SPARQL XML, SPARQL JSON and SPARQL TSV results format. This is the averaged time over 5 runs to retrieve the first 50,000 triples from the dataset with a simple SELECT * WHERE { ?s ?p ?o } LIMIT 50000 via a HTTP request to Fuseki and iterate over the results on the client.
> SPARQL XML = 15.25 seconds
> SPARQL JSON = 10.9 seconds
> SPARQL TSV = 0.54 seconds
> Now obviously TSV is way simpler to serialize and parse than XML/JSON but these serializers and parsers should not be 20-30 times slower IMO
> Also for comparison note that doing an equivalent CONSTRUCT { ?s ?p ?p } WHERE { ?s ?p ?o } LIMIT 50000 takes only about 2s and that is using RDF/XML serialization which I would have expected to be slower because RDF/XML is more complex to generate than either SPARQL XML/JSON results. I haven't dived into the code in detail to investigate why this is slow yet but do the Jena team have any thoughts on this?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira