Application mode - Custom Flink docker image with Python user code

[Sumeet Malhotra <su...@gmail.com>]

Hi,

I'm currently submitting my Python user code from my local machine to a
Flink cluster running in Session mode on Kubernetes. For this, I have a
custom Flink image with Python as per this reference [1].

Now, I'd like to move to using the Application mode with Native Kubernetes,
where the user code is embedded within the container image. For Java, the
process is described here [2]. This basically requires the user application
jar to be copied over to $FLINK_HOME/usrlib directory. I couldn't find any
documentation on where should Python user code live? Can it be packaged at
any location and invoked?

Thanks,
Sumeet


[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/standalone/docker/#enabling-python
[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#application-mode

Re: Application mode - Custom Flink docker image with Python user code

[Dian Fu <di...@gmail.com>]

Hi Sumeet,

It still has not provided special support to handle the dependencies for
the Application mode in PyFlink. This means that the dependencies could be
handled the same as the other deployment modes. However, it is indeed
correct that the dependencies could be handled differently in application
mode. For Python user code, I guess you could add them to the environment
variable PYTHONPATH? Then it's not necessary to specify them again in the
job.

Regards,
Dian

On Tue, Oct 26, 2021 at 3:23 PM Sumeet Malhotra <su...@gmail.com>
wrote:

> Hi,
>
> I'm currently submitting my Python user code from my local machine to a
> Flink cluster running in Session mode on Kubernetes. For this, I have a
> custom Flink image with Python as per this reference [1].
>
> Now, I'd like to move to using the Application mode with Native
> Kubernetes, where the user code is embedded within the container image. For
> Java, the process is described here [2]. This basically requires the user
> application jar to be copied over to $FLINK_HOME/usrlib directory. I
> couldn't find any documentation on where should Python user code live? Can
> it be packaged at any location and invoked?
>
> Thanks,
> Sumeet
>
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/standalone/docker/#enabling-python
> [2]
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#application-mode
>

Re: Application mode - Custom Flink docker image with Python user code

[Shuiqiang Chen <ac...@gmail.com>]

Hi Sumeet，

Actually, running pyflink jobs in application mode on kubernetes has been
supported since release 1.13.

To build a docker image with PyFlink installed, please refer to Enabling
Python[1]. In order to run the python code in application mode, you also
need to COPY the code files into the container image when building it. Then
you can submit the job by executing the following command:

$ ./bin/flink run-application \
    --target kubernetes-application
    -Dkubernetes.cluster-id={your-custom-cluster-id} \
    -Dkubernetes.container.image = {your-custom-image-name} \
    -pyfs {the python file path with the container image} \
    -pym {the-entry-module-name} \
    [the-entry-main-args](optional)

Hope this helps!

Best,
Shuiqiang



[1]
https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/deployment/resource-providers/standalone/docker/#enabling-python

Sumeet Malhotra <su...@gmail.com> 于2021年10月26日周二 下午3:23写道：

> Hi,
>
> I'm currently submitting my Python user code from my local machine to a
> Flink cluster running in Session mode on Kubernetes. For this, I have a
> custom Flink image with Python as per this reference [1].
>
> Now, I'd like to move to using the Application mode with Native
> Kubernetes, where the user code is embedded within the container image. For
> Java, the process is described here [2]. This basically requires the user
> application jar to be copied over to $FLINK_HOME/usrlib directory. I
> couldn't find any documentation on where should Python user code live? Can
> it be packaged at any location and invoked?
>
> Thanks,
> Sumeet
>
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/standalone/docker/#enabling-python
> [2]
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#application-mode
>