Livy interactive session

[Gos Os <go...@gmail.com>]

Hello folks,

I am new to Apache Livy and currently trying to understand how feasible
will Livy be for interactive query for a 300 user app.

Latency to first results is critical for customer experience.

The biggest concern I have is the 1000 limit associated with the
take/collect. Most of the ad-how queries will return more than 10k rows
easily.

In my view there are two options:

1- Livy batch submission with S3 as the destination. Then read the results
from the app from S3. This will not be the best experience as customer
can’t see results right away.

2- interactive query submission via Livy. Then add a mechanism to perform
pagination or write results to S3 if more than 1000 rows returned. The app
would know this query has more than 1000 rows and automatically start
paginating from S3 after 1000.

My question: how have other Livy users with a requirement of low latency to
first result solve this?

Thank you,
Gos.

Re: Livy interactive session

[Gos Os <go...@gmail.com>]

Thank you Sanchay and thank you for blog.

No dia terça-feira, 19 de julho de 2022, sanchay javeria <
sanchay.javeria@gmail.com> escreveu:

> I ran into this issue and solved it roughly the way you described your
> second approach. You can modify the SQLIntepreter
> <https://github.com/apache/incubator-livy/blob/master/repl/src/main/scala/org/apache/livy/repl/SQLInterpreter.scala#L97> to
> write the output dataframe to S3 instead, and on your client you can
> retrieve the results in a paginated manner from S3. I wrote about this
> problem in a blog post
> <https://medium.com/pinterest-engineering/interactive-querying-with-apache-spark-sql-at-pinterest-2a3eaf60ac1b>
> last year (see "Large Result Handling and Status Tracking") if you're
> interested.
>
> Best
>
> On Mon, 18 Jul 2022 at 11:38, Gos Os <go...@gmail.com> wrote:
>
>> Hello folks,
>>
>> I am new to Apache Livy and currently trying to understand how feasible
>> will Livy be for interactive query for a 300 user app.
>>
>> Latency to first results is critical for customer experience.
>>
>> The biggest concern I have is the 1000 limit associated with the
>> take/collect. Most of the ad-how queries will return more than 10k rows
>> easily.
>>
>> In my view there are two options:
>>
>> 1- Livy batch submission with S3 as the destination. Then read the
>> results from the app from S3. This will not be the best experience as
>> customer can’t see results right away.
>>
>> 2- interactive query submission via Livy. Then add a mechanism to perform
>> pagination or write results to S3 if more than 1000 rows returned. The app
>> would know this query has more than 1000 rows and automatically start
>> paginating from S3 after 1000.
>>
>> My question: how have other Livy users with a requirement of low latency
>> to first result solve this?
>>
>> Thank you,
>> Gos.
>>
>

Re: Livy interactive session

[sanchay javeria <sa...@gmail.com>]

I ran into this issue and solved it roughly the way you described your
second approach. You can modify the SQLIntepreter
<https://github.com/apache/incubator-livy/blob/master/repl/src/main/scala/org/apache/livy/repl/SQLInterpreter.scala#L97>
to
write the output dataframe to S3 instead, and on your client you can
retrieve the results in a paginated manner from S3. I wrote about this
problem in a blog post
<https://medium.com/pinterest-engineering/interactive-querying-with-apache-spark-sql-at-pinterest-2a3eaf60ac1b>
last year (see "Large Result Handling and Status Tracking") if you're
interested.

Best

On Mon, 18 Jul 2022 at 11:38, Gos Os <go...@gmail.com> wrote:

> Hello folks,
>
> I am new to Apache Livy and currently trying to understand how feasible
> will Livy be for interactive query for a 300 user app.
>
> Latency to first results is critical for customer experience.
>
> The biggest concern I have is the 1000 limit associated with the
> take/collect. Most of the ad-how queries will return more than 10k rows
> easily.
>
> In my view there are two options:
>
> 1- Livy batch submission with S3 as the destination. Then read the results
> from the app from S3. This will not be the best experience as customer
> can’t see results right away.
>
> 2- interactive query submission via Livy. Then add a mechanism to perform
> pagination or write results to S3 if more than 1000 rows returned. The app
> would know this query has more than 1000 rows and automatically start
> paginating from S3 after 1000.
>
> My question: how have other Livy users with a requirement of low latency
> to first result solve this?
>
> Thank you,
> Gos.
>