You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jena.apache.org by "Rob Vesse (JIRA)" <ji...@apache.org> on 2012/06/21 20:08:43 UTC
[jira] [Created] (JENA-267) SPARQL JSON Parser is not streaming
Rob Vesse created JENA-267:
------------------------------
Summary: SPARQL JSON Parser is not streaming
Key: JENA-267
URL: https://issues.apache.org/jira/browse/JENA-267
Project: Apache Jena
Issue Type: Improvement
Components: ARQ
Affects Versions: ARQ 2.9.1
Reporter: Rob Vesse
Assignee: Rob Vesse
The current SPARQL JSON parser in ARQ parses the JSON by parsing the entire response into a in-memory data structure and then iterating over that. So for large results it is trivial to hit an OutOfMemory exception.
This task is to reimplement the SPARQL JSON parser to be fully streaming in its implementation in line with all the other results parsers
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-267) SPARQL JSON Parser is not streaming
Posted by "Andy Seaborne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401402#comment-13401402 ]
Andy Seaborne commented on JENA-267:
------------------------------------
+1 to formally defining some kind of DataBag with a contract of write out, write back in same order.
SequnceDataBag which simply inherits from DefaultDataBag at the moment.
Then BagFactory.newSequenceDataBag?
This leaves the default choice uncontracted and open for change later.
> SPARQL JSON Parser is not streaming
> -----------------------------------
>
> Key: JENA-267
> URL: https://issues.apache.org/jira/browse/JENA-267
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: ARQ 2.9.1
> Reporter: Rob Vesse
> Assignee: Rob Vesse
> Labels: json, sparql
>
> The current SPARQL JSON parser in ARQ parses the JSON by parsing the entire response into a in-memory data structure and then iterating over that. So for large results it is trivial to hit an OutOfMemory exception.
> This task is to reimplement the SPARQL JSON parser to be fully streaming in its implementation in line with all the other results parsers
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-267) SPARQL JSON Parser is not streaming
Posted by "Alex Hall (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398771#comment-13398771 ]
Alex Hall commented on JENA-267:
--------------------------------
Is there any requirement that the head come before the bindings in the SPARQL results JSON object? As far as I can tell, JSON objects are unordered so it's possible that a valid result object have the bindings before the head, which might make life difficult for a streaming parser.
> SPARQL JSON Parser is not streaming
> -----------------------------------
>
> Key: JENA-267
> URL: https://issues.apache.org/jira/browse/JENA-267
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: ARQ 2.9.1
> Reporter: Rob Vesse
> Assignee: Rob Vesse
> Labels: json, sparql
>
> The current SPARQL JSON parser in ARQ parses the JSON by parsing the entire response into a in-memory data structure and then iterating over that. So for large results it is trivial to hit an OutOfMemory exception.
> This task is to reimplement the SPARQL JSON parser to be fully streaming in its implementation in line with all the other results parsers
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-267) SPARQL JSON Parser is not streaming
Posted by "Stephen Allen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398998#comment-13398998 ]
Stephen Allen commented on JENA-267:
------------------------------------
The implementation of DefaultDataBag will actually keep things in the same order in which they were added (even though the documentation on DataBag says it doesn't guarantee it). We could formalize that particular behavior if it helps.
> SPARQL JSON Parser is not streaming
> -----------------------------------
>
> Key: JENA-267
> URL: https://issues.apache.org/jira/browse/JENA-267
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: ARQ 2.9.1
> Reporter: Rob Vesse
> Assignee: Rob Vesse
> Labels: json, sparql
>
> The current SPARQL JSON parser in ARQ parses the JSON by parsing the entire response into a in-memory data structure and then iterating over that. So for large results it is trivial to hit an OutOfMemory exception.
> This task is to reimplement the SPARQL JSON parser to be fully streaming in its implementation in line with all the other results parsers
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-267) SPARQL JSON Parser is not streaming
Posted by "Andy Seaborne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398891#comment-13398891 ]
Andy Seaborne commented on JENA-267:
------------------------------------
Tricky point.
If the objective of streaming is efficency/low-latency, not handling very large results sets (a significant proportion of heap), then the stream-if-you, buffer-if-you-can't can is a good solution.
For very large results, to keep a perfect contract, if the header is not available first, stream for a bit to see if you see the end, but if it gets too big, flush to disk (org.openjena.atlas.data.DataBag) then stream from disk after the headers arrive.
To weaken the contract, ignore the lack of header and stream parse (and lie for ResultSet.getResultVars). Even if we had a clean slate, the utility of getResultVars for formatting results is quite high so the best weakening of the contract I can think of is to add something like "it's the result vars if available". But at such scale, switching to TSV is probably acceptable.
It does suggest JSON results are best for small and mid-range result sets.
> SPARQL JSON Parser is not streaming
> -----------------------------------
>
> Key: JENA-267
> URL: https://issues.apache.org/jira/browse/JENA-267
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: ARQ 2.9.1
> Reporter: Rob Vesse
> Assignee: Rob Vesse
> Labels: json, sparql
>
> The current SPARQL JSON parser in ARQ parses the JSON by parsing the entire response into a in-memory data structure and then iterating over that. So for large results it is trivial to hit an OutOfMemory exception.
> This task is to reimplement the SPARQL JSON parser to be fully streaming in its implementation in line with all the other results parsers
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-267) SPARQL JSON Parser is not streaming
Posted by "Rob Vesse (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398780#comment-13398780 ]
Rob Vesse commented on JENA-267:
--------------------------------
Technically speaking no - the spec [1] certainly doesn't require it. In reality I would expect that any sensible producer of the format is always going to put the head element first.
One thing we can do is read and see if a head key is present as the first key, if not then we revert to the read everything into memory behavior.
[1]: http://www.w3.org/TR/sparql11-results-json/
> SPARQL JSON Parser is not streaming
> -----------------------------------
>
> Key: JENA-267
> URL: https://issues.apache.org/jira/browse/JENA-267
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: ARQ 2.9.1
> Reporter: Rob Vesse
> Assignee: Rob Vesse
> Labels: json, sparql
>
> The current SPARQL JSON parser in ARQ parses the JSON by parsing the entire response into a in-memory data structure and then iterating over that. So for large results it is trivial to hit an OutOfMemory exception.
> This task is to reimplement the SPARQL JSON parser to be fully streaming in its implementation in line with all the other results parsers
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-267) SPARQL JSON Parser is not streaming
Posted by "Rob Vesse (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398987#comment-13398987 ]
Rob Vesse commented on JENA-267:
--------------------------------
Using a DataBag is definitely a possibility, having got as far as looking at that part of things the main issue I see is that we may need to keep the results that we read sorted (because they may be the results of an ordered query)
I guess in theory even if you haven't see the header provided you'd seen a few results you should know what the variables used are (of have a decent guess at it) even without seeing the header so you could just buffer the first 1000 or so results if you don't see the header first and extrapolate the variables from there. Given this a DataBag may actually be overkill
I'll carry on experimenting with this
> SPARQL JSON Parser is not streaming
> -----------------------------------
>
> Key: JENA-267
> URL: https://issues.apache.org/jira/browse/JENA-267
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: ARQ 2.9.1
> Reporter: Rob Vesse
> Assignee: Rob Vesse
> Labels: json, sparql
>
> The current SPARQL JSON parser in ARQ parses the JSON by parsing the entire response into a in-memory data structure and then iterating over that. So for large results it is trivial to hit an OutOfMemory exception.
> This task is to reimplement the SPARQL JSON parser to be fully streaming in its implementation in line with all the other results parsers
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-267) SPARQL JSON Parser is not streaming
Posted by "Alex Hall (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398787#comment-13398787 ]
Alex Hall commented on JENA-267:
--------------------------------
Agreed, any sensible producer will put the head first. The only concern I would have is somebody using an off-the-shelf JSON serializer with no control over the order an object's keys are written. I think your proposed workaround to revert to in-memory behavior if the head isn't first is probably the best approach.
> SPARQL JSON Parser is not streaming
> -----------------------------------
>
> Key: JENA-267
> URL: https://issues.apache.org/jira/browse/JENA-267
> Project: Apache Jena
> Issue Type: Improvement
> Components: ARQ
> Affects Versions: ARQ 2.9.1
> Reporter: Rob Vesse
> Assignee: Rob Vesse
> Labels: json, sparql
>
> The current SPARQL JSON parser in ARQ parses the JSON by parsing the entire response into a in-memory data structure and then iterating over that. So for large results it is trivial to hit an OutOfMemory exception.
> This task is to reimplement the SPARQL JSON parser to be fully streaming in its implementation in line with all the other results parsers
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira