Making the result generation in Asterix DB eager

[Pratyoy Das <pr...@uci.edu>]

Hi all,
I am trying to get Asterix DB to return eager results instead of returning
all the results at the end when the job is completed. Currently the logic
of result generation waits for the job to complete, but I feel like there
should be a way to make the process eager, if we can figure out where the
results are being stored and keep polling the location. Any pointers that
can help me identify the correct code points to look at and potentially
modify would be appreciated.
Thanks!

-- 
Best Regards,
Pratyoy

Re: Making the result generation in Asterix DB eager

[Mike Carey <dt...@gmail.com>]

Pratyoy,

Attached is a description of the system's result management framework in 
its initial days - I don't think there have been huge changes since, 
surprisingly, but there have been some (e.g., to support management of 
results that are range-partitioned results of things like parallel 
sorts).  Shiva Jahangiri may have some expertise related to this due to 
the work that she has done on staged execution of queries - not sure...

Historically the reason we don't have a pull-based API - instead, having 
our push-based API - is because one of our design criteria was that an 
application shouldn't be able to impede the progress of a query if its 
consumption rate was lower than the query's production rate, 
particularly in the post client-server era where the consumer is on the 
wrong side of an internet connection.  (Our goal was for the system to 
be able to get a query done ASAP to free up the resources it uses for 
other queries.)

Cheers,

Mike

On 1/28/25 11:33 AM, Pratyoy Das wrote:
> Hi all,
> I am trying to get Asterix DB to return eager results instead of returning
> all the results at the end when the job is completed. Currently the logic
> of result generation waits for the job to complete, but I feel like there
> should be a way to make the process eager, if we can figure out where the
> results are being stored and keep polling the location. Any pointers that
> can help me identify the correct code points to look at and potentially
> modify would be appreciated.
> Thanks!
>