You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Laura Corssac <la...@gmail.com> on 2022/12/13 16:59:03 UTC

Flink Kafka integration in Python

Dear flink community,

I hope you are well!

I have tried to integrate Kafka and Flink and was not successful. Could you
help me?

I am getting this error:

[image: Captura de Tela 2022-12-13 às 5.53.00 PM.png]

The code of the examples of Flink in Python. I just tried to add different
JARS, but was not successful.

[image: Captura de Tela 2022-12-13 às 5.55.44 PM.png]

The version of Flink I am using is 1.16 and I have started Kafka and
Zookeeper. Also, I am running the python code in a Miniconda environment.

I hope you can help me! And I would love to join the Flink community in Slack.

Thank you a lot for your attention.

Best regards,

Laura

Re: Flink Kafka integration in Python

Posted by Dian Fu <di...@gmail.com>.
Hi Laura,

The root cause could be seen from the following log:
```
Caused by: java.lang.UnsupportedOperationException: unsupported Java
version: 14
at
org.apache.beam.runners.core.construction.Environments$JavaVersion.forSpecification(Environments.java:121)
at
org.apache.beam.runners.core.construction.Environments.getJavaVersion(Environments.java:398)
at
org.apache.beam.runners.core.construction.Environments.getDefaultJavaSdkHarnessContainerUrl(Environments.java:455)
at
org.apache.beam.runners.core.construction.Environments.<clinit>(Environments.java:134)
... 21 more
```

It seems that you are using JDK 14, please use JDK 8 or JDK 11 instead.

Regards,
Dian

On Wed, Dec 14, 2022 at 4:10 PM Laura Corssac <la...@gmail.com>
wrote:

> Hi Dian!
>
> Thank you for your response.
>
> I found the .jar under the folder you mentioned. So I replaced the
> explicit jar addings by the following:
>
>
> env.add_jars("file:///Users/lauracorssac/miniconda3/envs/pyflink_38/lib/python3.8/site-packages/pyflink/lib/flink-sql-connector-kafka_2.11-1.13.0.jar")
>
> However, the problem persists.. 😕
>
> I am sending the log!
>
> Thank you,
>
> Laura
>
>
> On Wed, Dec 14, 2022 at 3:32 AM Dian Fu <di...@gmail.com> wrote:
>
>> Hi Laura,
>>
>> `org.apache.beam.runners.core.construction.Environments` is bundled in
>> the flink-python jar package which should be available in the PyFlink
>> distribution. Could you check the jars available in your environment, e.g.
>> the jars available in site-packages/pyflink/lib and
>> site-packages/pyflink/opt? Besides, it would also be helpful to upload the
>> full log which should be located under site-packages/pyflink/log.
>>
>> PS: you don't need to add the following jars:
>> flink-connector-kafka-1.16.0.jar, flink-core-1.16.0.jar and
>> flink-connector-base-1.16.0.jar explicitly.
>>
>> Regards,
>> Dian
>>
>> On Wed, Dec 14, 2022 at 1:01 AM Laura Corssac <la...@gmail.com>
>> wrote:
>>
>>> Dear flink community,
>>>
>>> I hope you are well!
>>>
>>> I have tried to integrate Kafka and Flink and was not successful. Could
>>> you help me?
>>>
>>> I am getting this error:
>>>
>>> [image: Captura de Tela 2022-12-13 às 5.53.00 PM.png]
>>>
>>> The code of the examples of Flink in Python. I just tried to add
>>> different JARS, but was not successful.
>>>
>>> [image: Captura de Tela 2022-12-13 às 5.55.44 PM.png]
>>>
>>> The version of Flink I am using is 1.16 and I have started Kafka and
>>> Zookeeper. Also, I am running the python code in a Miniconda environment.
>>>
>>> I hope you can help me! And I would love to join the Flink community in Slack.
>>>
>>> Thank you a lot for your attention.
>>>
>>> Best regards,
>>>
>>> Laura
>>>
>>>
>>>
>>>
>>>

Re: Flink Kafka integration in Python

Posted by Dian Fu <di...@gmail.com>.
Hi Laura,

`org.apache.beam.runners.core.construction.Environments` is bundled in the
flink-python jar package which should be available in the PyFlink
distribution. Could you check the jars available in your environment, e.g.
the jars available in site-packages/pyflink/lib and
site-packages/pyflink/opt? Besides, it would also be helpful to upload the
full log which should be located under site-packages/pyflink/log.

PS: you don't need to add the following jars:
flink-connector-kafka-1.16.0.jar, flink-core-1.16.0.jar and
flink-connector-base-1.16.0.jar explicitly.

Regards,
Dian

On Wed, Dec 14, 2022 at 1:01 AM Laura Corssac <la...@gmail.com>
wrote:

> Dear flink community,
>
> I hope you are well!
>
> I have tried to integrate Kafka and Flink and was not successful. Could
> you help me?
>
> I am getting this error:
>
> [image: Captura de Tela 2022-12-13 às 5.53.00 PM.png]
>
> The code of the examples of Flink in Python. I just tried to add different
> JARS, but was not successful.
>
> [image: Captura de Tela 2022-12-13 às 5.55.44 PM.png]
>
> The version of Flink I am using is 1.16 and I have started Kafka and
> Zookeeper. Also, I am running the python code in a Miniconda environment.
>
> I hope you can help me! And I would love to join the Flink community in Slack.
>
> Thank you a lot for your attention.
>
> Best regards,
>
> Laura
>
>
>
>
>