Druid Scan with result streaming

[madhavi g <ma...@gmail.com>]

Hi,

We are using axios to make http call to druid via node js application.
Since the data might be huge we are seeing memory issues at the client side
while loading data to a variable. Instead, is there a way to receive data
as stream and process it.
Please share details and example if possible.

Thanks,
Madhavi.

Re: Druid Scan with result streaming

[Gian Merlino <gi...@apache.org>]

Actually I'd suggest the opposite approach: use Scan if you want a stream.
It can handle almost any number of results (you might want to set
druid.broker.http.maxQueuedBytes to 10000000 or so if you are doing large
resultsets; this will likely be the default in the future).

With Scan queries, Druid will stream the results back to you. All you have
to do is read the JSON array row by row instead of all at once, using a
streaming JSON parser. Sorry, I'm not sure what the best way is to do this
in node.js, but maybe someone else can chime in on that. By the way, you
could also use Druid SQL (https://druid.apache.org/docs/latest/querying/sql)
and its resultFormat parameter to get results as newline-delimited CSV or
JSON objects. This might be easier to parse in a streaming way, depending
on what tools you have available.


On Tue, Dec 10, 2019 at 9:13 PM Gaurav Bhatnagar <ga...@gmail.com> wrote:

> You can try using Select query instead of scan. Select provides pagination
> and takes more resources compared to scan queries. Scan queries provide
> result in stream mode. Here is link for further reading
> https://druid.apache.org/docs/latest/querying/scan-query.html
>
> On Wed, Dec 11, 2019 at 2:06 AM madhavi g <ma...@gmail.com> wrote:
>
> > Hi,
> >
> > We are using axios to make http call to druid via node js application.
> > Since the data might be huge we are seeing memory issues at the client
> side
> > while loading data to a variable. Instead, is there a way to receive data
> > as stream and process it.
> > Please share details and example if possible.
> >
> > Thanks,
> > Madhavi.
> >
>

Re: Druid Scan with result streaming

[Gaurav Bhatnagar <ga...@gmail.com>]

You can try using Select query instead of scan. Select provides pagination
and takes more resources compared to scan queries. Scan queries provide
result in stream mode. Here is link for further reading
https://druid.apache.org/docs/latest/querying/scan-query.html

On Wed, Dec 11, 2019 at 2:06 AM madhavi g <ma...@gmail.com> wrote:

> Hi,
>
> We are using axios to make http call to druid via node js application.
> Since the data might be huge we are seeing memory issues at the client side
> while loading data to a variable. Instead, is there a way to receive data
> as stream and process it.
> Please share details and example if possible.
>
> Thanks,
> Madhavi.
>