Beam on Flink version compatibility

[Nir Gazit <ni...@gmail.com>]

Hey,
I'm trying to use Beam on Flink with Python, and for some reason it insists
on using Flink 1.10 even though Beam's website states
<https://beam.apache.org/documentation/runners/flink/#beam-capability> that
v1.12 is already supported. Digging through the code, it seems there's
a hardcoded
limitation
<https://github.com/apache/beam/blob/0350f2efa3cd4cf994a102625e479fa06dc5a643/sdks/python/apache_beam/options/pipeline_options.py#L1216>
for
that in the Python SDK.

Is there a reason for that?

Thanks!
Nir

Re: Beam on Flink version compatibility

[Kyle Weaver <kc...@google.com>]

The Java portion of the Flink runner was upgraded to 1.12, but Python and
job server containers were not [1]. This will be fixed in the next Beam
release (2.28.0).

If you want to use Flink 1.12 with Python now, you can follow the
instructions in the "Portable (Java/Python/Go)" tab [2]. But since official
Docker containers were not released, you will have to build them yourself
from source: ./gradlew :runners:flink:1.12:job-server-container:docker

[1] https://issues.apache.org/jira/browse/BEAM-11475
[2] https://beam.apache.org/documentation/runners/flink/

On Sun, Jan 10, 2021 at 9:38 AM Nir Gazit <ni...@gmail.com> wrote:

> Hey,
> I'm trying to use Beam on Flink with Python, and for some reason it
> insists on using Flink 1.10 even though Beam's website states
> <https://beam.apache.org/documentation/runners/flink/#beam-capability> that
> v1.12 is already supported. Digging through the code, it seems there's a hardcoded
> limitation
> <https://github.com/apache/beam/blob/0350f2efa3cd4cf994a102625e479fa06dc5a643/sdks/python/apache_beam/options/pipeline_options.py#L1216> for
> that in the Python SDK.
>
> Is there a reason for that?
>
> Thanks!
> Nir
>