WebSocket/Https connector for Apache Beam (Java)?

[I-Feng Lin <i....@squaremetrics.com>]

Hello all,

I have an Apache Beam pipeline in Java where I would like to read data that
comes from a WebSocket and write data to the server through Https.

I have been looking for some connectors but so far my search was
unsuccessful. I know that it is possible to create custom connectors but I
want to check if there is anything already exists.

Thanks in advance,

Ifeng

Re: WebSocket/Https connector for Apache Beam (Java)?

[Ismaël Mejía <ie...@gmail.com>]

I was not aware of that websocket work thanks for sharing Alexey.
I-Feng a Websocket based IO would be a really nice contribution so
worth to bring it to the project if it is in your plans. Maybe worth
to sync with JB on that.

If by any chance you or any other person is interested in adding an IO
for raw sockets there used to be an UnboundedSocketSource as part of
the Beam Flink runner that was removed recently.
https://github.com/apache/beam/pull/8574

I think this could be easily re-wrapped and pushed into its own
module, and I think it would be useful not only for pipelines but also
for demo purposes or interactive stuff like the SQL CLI / IPython
notebooks.

On Wed, Jul 3, 2019 at 10:07 AM Alexey Romanenko
<ar...@gmail.com> wrote:
>
> Probably, this blog, which Romain Manni-Bucau wrote a while ago, could be helpful:
> https://rmannibucau.metawerx.net/post/apache-beam-websocket-output
>
> It’s not in Beam upstream but you can use it as your own transform.
>
> On 2 Jul 2019, at 13:34, I-Feng Lin <i....@squaremetrics.com> wrote:
>
> Hello all,
>
> I have an Apache Beam pipeline in Java where I would like to read data that comes from a WebSocket and write data to the server through Https.
>
> I have been looking for some connectors but so far my search was unsuccessful. I know that it is possible to create custom connectors but I want to check if there is anything already exists.
>
> Thanks in advance,
>
> Ifeng
>
>

Re: WebSocket/Https connector for Apache Beam (Java)?

[Alexey Romanenko <ar...@gmail.com>]

Probably, this blog, which Romain Manni-Bucau wrote a while ago, could be helpful:
https://rmannibucau.metawerx.net/post/apache-beam-websocket-output <https://rmannibucau.metawerx.net/post/apache-beam-websocket-output>

It’s not in Beam upstream but you can use it as your own transform.

> On 2 Jul 2019, at 13:34, I-Feng Lin <i....@squaremetrics.com> wrote:
> 
> Hello all,
> 
> I have an Apache Beam pipeline in Java where I would like to read data that comes from a WebSocket and write data to the server through Https.
> 
> I have been looking for some connectors but so far my search was unsuccessful. I know that it is possible to create custom connectors but I want to check if there is anything already exists.
> 
> Thanks in advance,
> 
> Ifeng
> 

Re: WebSocket/Https connector for Apache Beam (Java)?

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi,

I have a websocket and REST IO about that (I proposed the IO while ago).

As you are not the first one to ask for such IO, I will revive the IO.

Regards
JB

On 02/07/2019 13:34, I-Feng Lin wrote:
> Hello all,
> 
> I have an Apache Beam pipeline in Java where I would like to read data
> that comes from a WebSocket and write data to the server through Https.
> 
> I have been looking for some connectors but so far my search was
> unsuccessful. I know that it is possible to create custom connectors but
> I want to check if there is anything already exists.
> 
> Thanks in advance,
> 
> Ifeng
> 

-- 
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com