Socket stream source in Python?

[Philippe Rigaux <ph...@lecnam.net>]

Hi there

I would like to use a socket stream as input for my Flink workflow in Python. This works in scala with the socketTextStream() method, for instance

val stream = senv.socketTextStream("localhost", 9000, '\n')

I cannot find an equivalent in PyFlink, although it is briefly mentioned in the documentation. 

Any help is much appreciated.

Philippe

Re: Socket stream source in Python?

[Philippe Rigaux <ph...@lecnam.net>]

Thank you so much. 

> Le 31 janv. 2022 à 01:11, Francis Conroy <fr...@switchdin.com> a écrit :
> 
> Hi Philippe,
> after checking the source Flink master I think you're right, there is currently no binding from python to Flink socketTextStream (via py4j) in pyFlink. The py4j interface isn't too complicated to modify for some tasks and I suspect that it should be fairly trivial to extend pyflink to support this. I imagine that you could take   
> read_text_file
> in 'stream_execution_environment.py' as a starting point.
> Happy to provide some more information on this if you'd like.
> 
> Kind regards,
> Francis
> 
> On Sat, 29 Jan 2022 at 01:20, Philippe Rigaux <philippe.rigaux@lecnam.net <ma...@lecnam.net>> wrote:
> Hi there
> 
> I would like to use a socket stream as input for my Flink workflow in Python. This works in scala with the socketTextStream() method, for instance
> 
> val stream = senv.socketTextStream("localhost", 9000, '\n')
> 
> I cannot find an equivalent in PyFlink, although it is briefly mentioned in the documentation. 
> 
> Any help is much appreciated.
> 
> Philippe
> 
> 
> This email and any attachments are proprietary and confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily reflect or represent those of SwitchDin Pty Ltd. If you have received this email in error, please let us know immediately by reply email and delete it from your system. You may not use, disseminate, distribute or copy this message nor disclose its contents to anyone. 
> SwitchDin Pty Ltd (ABN 29 154893857) PO Box 1165, Newcastle NSW 2300 Australia

Re: Socket stream source in Python?

[Francis Conroy <fr...@switchdin.com>]

 Hi Philippe,
after checking the source Flink master I think you're right, there is
currently no binding from python to Flink socketTextStream (via py4j) in
pyFlink. The py4j interface isn't too complicated to modify for some tasks
and I suspect that it should be fairly trivial to extend pyflink to support
this. I imagine that you could take
read_text_file
in 'stream_execution_environment.py' as a starting point.
Happy to provide some more information on this if you'd like.

Kind regards,
Francis

On Sat, 29 Jan 2022 at 01:20, Philippe Rigaux <ph...@lecnam.net>
wrote:

> Hi there
>
> I would like to use a socket stream as input for my Flink workflow in
> Python. This works in scala with the socketTextStream() method, for instance
>
> val stream = senv.socketTextStream("localhost", 9000, '\n')
>
> I cannot find an equivalent in PyFlink, although it is briefly mentioned
> in the documentation.
>
> Any help is much appreciated.
>
> Philippe
>
>

-- 
This email and any attachments are proprietary and confidential and are 
intended solely for the use of the individual to whom it is addressed. Any 
views or opinions expressed are solely those of the author and do not 
necessarily reflect or represent those of SwitchDin Pty Ltd. If you have 
received this email in error, please let us know immediately by reply email 
and delete it from your system. You may not use, disseminate, distribute or 
copy this message nor disclose its contents to anyone. 
SwitchDin Pty Ltd 
(ABN 29 154893857) PO Box 1165, Newcastle NSW 2300 Australia