[question][python][flink][kafka] Configuring java worker

[marc hurabielle <ma...@gmail.com>]

Hello beam community,

I have a question arround beam python, and kafka and flink runner.
As the kafka source need java, right now, the flink runner will try to
start a docker container.
I have trouble finding documentation arround this behavior. As we run flink
in kubernetes or in docker environment, it will be pretty difficult to do
docker in docker.

1_ Is it possible to run this part in a process if the dependency are
installed on the flink runner?

2_ Is it possible to configure to run a worker pool like for python? (when
using help there is no worker pool like in python)

3_ Is it possible to configure the host of the provision endpoints (as
localhost is hard to configure in docker inside docker, or any options to
configure the docker commands)?

Thank you a lot for your help.
Best regards

Re: [question][python][flink][kafka] Configuring java worker

[Ahmet Altay via user <us...@beam.apache.org>]

Adding relevant folks who might be able to help: @Chamikara Jayalath
<ch...@google.com> @John Casey <jo...@google.com>

On Wed, Aug 2, 2023 at 4:57 PM marc hurabielle <ma...@gmail.com>
wrote:

> Hello beam community,
>
> I have a question arround beam python, and kafka and flink runner.
> As the kafka source need java, right now, the flink runner will try to
> start a docker container.
> I have trouble finding documentation arround this behavior. As we run
> flink in kubernetes or in docker environment, it will be pretty difficult
> to do docker in docker.
>
> 1_ Is it possible to run this part in a process if the dependency are
> installed on the flink runner?
>
> 2_ Is it possible to configure to run a worker pool like for python? (when
> using help there is no worker pool like in python)
>
> 3_ Is it possible to configure the host of the provision endpoints (as
> localhost is hard to configure in docker inside docker, or any options to
> configure the docker commands)?
>
> Thank you a lot for your help.
> Best regards
>