Support of streams in Fuseki

["Dibowski Henrik (CR/ADT3)" <He...@de.bosch.com.INVALID>]

Dear Jena team,

in our current project we are exploring a way how to extract triples from a Jena Fuseki dataset via SPARQL Construct queries. We want to write (i.e. stream) the returned statements directly on-the-fly into a file on our server, without the need of loading all the statements into main memory first, as this is the bottleneck especially for large amounts of data. Our TypeScript backend application uses Node.js and will connect directly via http (REST), using the SPARQL protocol API, with Fuseki. Our question: Does Fuseki support streams, i.e. the sending of the result in chunks? We are considering to use Node Fetch for enabling & using this on our application side. See also https://www.npmjs.com/package/node-fetch#streams Is that supported by Fuseki? Or is there some other way how to "stream" large results into files?

Best regards

Henrik

Re: Support of streams in Fuseki

[Andy Seaborne <an...@apache.org>]

Hi Henrik,

Requests to Fuseki are HTTP requests.

You could simply copy the HTTP response body stream into a file if you 
can get the body as a stream of bytes from the local HTTP client library.

Whether it streams or whether the HTTP client library in your 
application collects the whole response then passes it to the 
application is purely a client-side issue - Fuseki (or any other 
webserver) has no control over that.

By chunks do you mean HTTP chunk-encoding?

Fuseki does provide chunk-encoding on responses for CONSTRUCT
(Jetty does it because there isn't a Content-Length). Try it with "wget 
-S" or "curl -v" --> Transfer-Encoding: chunked)

Being a Transfer-Encoding intermediates servers may change this.

Or the response comes back in (sync|async) callbacks (c.f. HTTP/2)? That 
is a matter for client-side HTTP code.

I can't tell from the doc you point to whether it is referring to HTTP 
chunk encoding (which requires the sever to cooperator and the HTTP 
header used aren't shown) or locally collecting bytes together to return 
in blocks.

     Andy

On 24/09/2021 09:12, Dibowski Henrik (CR/ADT3) wrote:
> Dear Jena team,
> 
> in our current project we are exploring a way how to extract triples from a Jena Fuseki dataset via SPARQL Construct queries. We want to write (i.e. stream) the returned statements directly on-the-fly into a file on our server, without the need of loading all the statements into main memory first, as this is the bottleneck especially for large amounts of data. Our TypeScript backend application uses Node.js and will connect directly via http (REST), using the SPARQL protocol API, with Fuseki. Our question: Does Fuseki support streams, i.e. the sending of the result in chunks? We are considering to use Node Fetch for enabling & using this on our application side. See also https://www.npmjs.com/package/node-fetch#streams Is that supported by Fuseki? Or is there some other way how to "stream" large results into files?
> 
> Best regards
> 
> Henrik
>