Flink SQL on Docker

[Tristan Culp via user <us...@flink.apache.org>]

Hello all,

I've been working with the Flink SQL Client lately, and have been trying to
use it on a Docker image. I've noticed that when it runs on an image,
however, it performs much slower and after 1 or 2 sql queries, stops
working altogether, leading me to have to restart the SQL client. I've
noticed through the Docker Desktop that the container running it has very
high CPU usage when running the client, and uses over 1 GB of memory usage
just after running start-cluster.sh. When running sql queries, the memory
usage steadily increases until it stops some point after 3 GB, and stops
working.

Does anyone have any recommendations to make the SQL client more reliable
in a Docker image, and to make queries return faster? It is relatively
seamless when doing the same things locally, and messing with the Docker
Desktop resources has caused issues.

Thanks for your help!

Re: Flink SQL on Docker

[Tristan Culp via user <us...@flink.apache.org>]

Hi, sorry about the late response.

Currently the sql client queries connect to a Kafka topic to read events,
so I don't think it'll be reproducible. I'm using Flink 1.15, and running
it in a Docker image on Mac OS with M1 chip.

I also noticed that multiple queries in a sql client session continue to
build up memory usage in Docker, until the client is exited on which all of
the memory is cleared. Is this because all of the incoming events are saved
in the heap? It seems that I can only get previous sql commands and not
results, but are those results actually being written somewhere and saved?

Thanks!

Best,
Tristan


On Mon, Oct 24, 2022 at 9:35 PM Shengkai Fang <fs...@gmail.com> wrote:

> Hi. Could you share the query you use in the tests and let us reproduce
> this problem offline? It's better you can provide us with more infos:
>
> - the Flink version you use
> - the logs in the sql-client and jm
> - It's better you can dump the memory to detect which uses the memory
>
> Best,
> Shengkai
>
> Tristan Culp via user <us...@flink.apache.org> 于2022年10月25日周二 05:30写道：
>
>> Hello all,
>>
>> I've been working with the Flink SQL Client lately, and have been trying
>> to use it on a Docker image. I've noticed that when it runs on an image,
>> however, it performs much slower and after 1 or 2 sql queries, stops
>> working altogether, leading me to have to restart the SQL client. I've
>> noticed through the Docker Desktop that the container running it has very
>> high CPU usage when running the client, and uses over 1 GB of memory usage
>> just after running start-cluster.sh. When running sql queries, the memory
>> usage steadily increases until it stops some point after 3 GB, and stops
>> working.
>>
>> Does anyone have any recommendations to make the SQL client more reliable
>> in a Docker image, and to make queries return faster? It is relatively
>> seamless when doing the same things locally, and messing with the Docker
>> Desktop resources has caused issues.
>>
>> Thanks for your help!
>>
>

Re: Flink SQL on Docker

[Shengkai Fang <fs...@gmail.com>]

Hi. Could you share the query you use in the tests and let us reproduce
this problem offline? It's better you can provide us with more infos:

- the Flink version you use
- the logs in the sql-client and jm
- It's better you can dump the memory to detect which uses the memory

Best,
Shengkai

Tristan Culp via user <us...@flink.apache.org> 于2022年10月25日周二 05:30写道：

> Hello all,
>
> I've been working with the Flink SQL Client lately, and have been trying
> to use it on a Docker image. I've noticed that when it runs on an image,
> however, it performs much slower and after 1 or 2 sql queries, stops
> working altogether, leading me to have to restart the SQL client. I've
> noticed through the Docker Desktop that the container running it has very
> high CPU usage when running the client, and uses over 1 GB of memory usage
> just after running start-cluster.sh. When running sql queries, the memory
> usage steadily increases until it stops some point after 3 GB, and stops
> working.
>
> Does anyone have any recommendations to make the SQL client more reliable
> in a Docker image, and to make queries return faster? It is relatively
> seamless when doing the same things locally, and messing with the Docker
> Desktop resources has caused issues.
>
> Thanks for your help!
>