[jira] [Commented] (FLINK-17480) Support running PyFlink on Kubernetes

["Shuiqiang Chen (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-17480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187596#comment-17187596 ] 

Shuiqiang Chen commented on FLINK-17480:
----------------------------------------

Hi [~felixzheng], I have managed to run Python Flink jobs on Kubernetes in native session/application mode with the master branch of Flink these days.  The following shows the approaches:

1.  Native session mode

 An image with Flink, python and PyFlink environment installed is required to start a Python Flink session cluster. It could be the extension of the Flink official image as below:

 
{code:java}
FROM flink

# install miniconda to prepare a python environment
ENV LANG=C.UTF-8 LC_ALL=C.UTF-8
ENV PATH /opt/conda/bin:$PATH

RUN apt-get update --fix-missing && \
    apt-get install -y wget bzip2 ca-certificates curl git && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-4.5.11-Linux-x86_64.sh -O ~/miniconda.sh && \
    /bin/bash ~/miniconda.sh -b -p /opt/conda && \
    rm ~/miniconda.sh && \
    /opt/conda/bin/conda clean -tipsy && \
    ln -s /opt/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh && \
    echo ". /opt/conda/etc/profile.d/conda.sh" >> ~/.bashrc && \
    echo "conda activate base" >> ~/.bashrc

# install python flink
RUN pip install apache-flink{code}
 

Assuming that the built image is tagged with name and version `pyflink:latest`, users can start a PyFlink session cluster by executing the following command:

 
{code:java}
./bin/kubernetes-session.sh \
 -Dkubernetes.container.image=pyflink:latest \
 -Dkubernetes.cluster-id=my-flink-session \
 -Dtaskmanager.memory.process.size=4096m \
 -Dkubernetes.taskmanager.cpu=2 \
 -Dtaskmanager.numberOfTaskSlots=4 \
 -Dresourcemanager.taskmanager-timeout=3600000 \{code}
 

As the session cluster starts up, users are able to submit a Python Flink Job:

 
{code:java}
./bin/flink run -m jm-host:port -py <PATH-TO-YOUR-PYTHON-CODES-DIRECTORY>/my_python_flink_job.py{code}
2. Native application mode

Application mode requires users to create a single image containing their Job and the Flink runtime. Base on the built image with python and Python Flink environment provided in session mode section, users can build an Job image like:

 
{code:java}
FROM pyflink:latest
COPY /path/of/python/codes /opt/my_python_codes

# if there a third party python dependencies, users can install them when building the image
COPY /path/to/requirements.txt /opt/requirements.txt
RUN pip install -r requirements.txt 

# if the job requires external java dependencies, they should be built into the image as well
RUN mkdir -p $FLINK_HOME/usrlib
COPY /path/of/external/jar/dependencies $FLINK_HOME/usrlib/{code}
With the application image prepared and tagged as `pyflink:app`, users can submit a Python Flink application as below:

 

 
{code:java}
./bin/flink run-application -p 8 -t kubernetes-application \
  -Dkubernetes.cluster-id=my-flink-app \
  -Dtaskmanager.memory.process.size=4096m \
  -Dkubernetes.taskmanager.cpu=2 \
  -Dtaskmanager.numberOfTaskSlots=4 \
  -Dkubernetes.container.image=pyflink:app \
  -Dpython.files=/opt/my_python_codes \
  -c org.apache.flink.client.python.PythonDriver \
  local:///opt/flink/usrlib/flink-python_2.11-1.12-SNAPSHOT.jar \
  -pym <ENTRY_MODULE_NAME> (or -py /opt/my_python_codes/xxx.py)
{code}
Note that users must specify the following arguments:

 
 * `-Dpython.files=/opt/my_python_codes `: the path of the python codes in the image
 * `-c org.apache.flink.client.python.PythonDriver`:  the java entry main class to execute the python codes, it is a constant but must be specified.
 * `local:///opt/flink/usrlib/flink-python_2.11-1.12-SNAPSHOT.jar`:  the jar of python flink module, it's provided by Flink internal but must be specified. 

 

Above is  the overall illustration of how to run PyFlink jobs on kubernetes in native mode. And it seems there are some chances to improve the ease of use, maybe we can do that in the future.

Best,

Shuiqiang

 

> Support running PyFlink on Kubernetes
> -------------------------------------
>
>                 Key: FLINK-17480
>                 URL: https://issues.apache.org/jira/browse/FLINK-17480
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Deployment / Kubernetes
>            Reporter: Canbin Zheng
>            Priority: Major
>
> This is the umbrella issue for running PyFlink on Kubernetes in native mode.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)