Datastream api implementation of a third party pyflink connector

["Wang, Zhongle" <Zh...@dell.com>]

Hi,

I'm working on a pyflink datastream connector for Pravega and wish to use a datasource other than Kafka.

Currently the Kafka connector for the python datastream api is implemented using a ` get_gateway` function which creates a binding to java in ` FlinkKafkaConsumer`.

So if I want to create a `FlinkPrevegaReader` that consumes other datasource like the Pravega, is it recommended to do in the same way? (we have a java reader/consumer implementation)
Or the gateway thing might be changed in the future?

PS: In this post(https://stackoverflow.com/questions/65009292/is-there-a-kinesis-connector-for-pyflink) Xingbo suggested that the Kinesis and other connectors will be added soon, but I'm not sure whether it uses the same technique mentioned above.

Thanks,
Zhongle Wang

RE: Datastream api implementation of a third party pyflink connector

["Wang, Zhongle" <Zh...@dell.com>]

Hi Xingbo,

Thanks for the reassurance.

PS: The java implementation of pravega connector is at https://github.com/pravega/flink-connectors/packages/

Best,
Zhongle Wang

From: Xingbo Huang <hx...@gmail.com>
Sent: Tuesday, July 20, 2021 9:58 AM
To: Wang, Zhongle
Cc: user@flink.apache.org
Subject: Re: Datastream api implementation of a third party pyflink connector

Hi Zhongle Wang,

Your understanding is correct. Firstly, you need to provide an implementation of a java connector, then add this jar to the dependency[1], and finally add a python connector wrapper.

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/python/faq/#adding-jar-files [ci.apache.org]<https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/python/faq/*adding-jar-files__;Iw!!LpKI!26AH0LoYsKDqTNM44mZDO2niLxfL9Nkc_P_uHd1FAlvLY8BwLogL1oLKOOn6CI7m7A$>

Best,
Xingbo

Wang, Zhongle <Zh...@dell.com>> 于2021年7月19日周一 下午5:43写道：
Hi,

I’m working on a pyflink datastream connector for Pravega and wish to use a datasource other than Kafka.

Currently the Kafka connector for the python datastream api is implemented using a ` get_gateway` function which creates a binding to java in ` FlinkKafkaConsumer`.

So if I want to create a `FlinkPrevegaReader` that consumes other datasource like the Pravega, is it recommended to do in the same way? (we have a java reader/consumer implementation)
Or the gateway thing might be changed in the future?

PS: In this post(https://stackoverflow.com/questions/65009292/is-there-a-kinesis-connector-for-pyflink [stackoverflow.com]<https://urldefense.com/v3/__https:/stackoverflow.com/questions/65009292/is-there-a-kinesis-connector-for-pyflink__;!!LpKI!26AH0LoYsKDqTNM44mZDO2niLxfL9Nkc_P_uHd1FAlvLY8BwLogL1oLKOOl6GKF5EQ$>) Xingbo suggested that the Kinesis and other connectors will be added soon, but I’m not sure whether it uses the same technique mentioned above.

Thanks,
Zhongle Wang



Internal Use - Confidential

Re: Datastream api implementation of a third party pyflink connector

[Xingbo Huang <hx...@gmail.com>]

Hi Zhongle Wang,

Your understanding is correct. Firstly, you need to provide an
implementation of a java connector, then add this jar to the dependency[1],
and finally add a python connector wrapper.

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/python/faq/#adding-jar-files

Best,
Xingbo

Wang, Zhongle <Zh...@dell.com> 于2021年7月19日周一 下午5:43写道：

> Hi,
>
>
>
> I’m working on a pyflink datastream connector for Pravega and wish to use
> a datasource other than Kafka.
>
>
>
> Currently the Kafka connector for the python datastream api is implemented
> using a ` get_gateway` function which creates a binding to java in `
> FlinkKafkaConsumer`.
>
>
>
> So if I want to create a `FlinkPrevegaReader` that consumes other
> datasource like the Pravega, is it recommended to do in the same way? (we
> have a java reader/consumer implementation)
>
> Or the gateway thing might be changed in the future?
>
>
>
> PS: In this post(
> https://stackoverflow.com/questions/65009292/is-there-a-kinesis-connector-for-pyflink)
> Xingbo suggested that the Kinesis and other connectors will be added soon,
> but I’m not sure whether it uses the same technique mentioned above.
>
>
>
> Thanks,
>
> Zhongle Wang
>
>
>