You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by "Tang Jijun (上海_技术部_数据平台_唐觊隽)" <ta...@yhd.com> on 2017/04/12 06:16:00 UTC
答复: Public in-progress I/O Transform list
Hi Stephen,
We need restIO module. I am looking forward to your pr about it. For restIO ,I create a jira https://issues.apache.org/jira/browse/BEAM-1946.
-----邮件原件-----
发件人: Jean-Baptiste Onofré [mailto:jb@nanthrax.net]
发送时间: 2017年4月12日 14:10
收件人: user@beam.apache.org
主题: Re: Public in-progress I/O Transform list
Hi Stephen,
Thanks !
I would add Socket IO and REST IO that I have on local branches (I'm waiting to close some of my PRs before opening new ones). I also have (not fully ready for
PR) CamelIO, IronMQIO, and others (I will update the website on the fly ;)).
MQTT IO is missing in the list. Gonna provide a website PR about that.
I also created ExecIO while ago but we didn't get a consensus about what we should do with it.
Regards
JB
On 04/12/2017 02:01 AM, Stephen Sisk wrote:
> Hi!
>
> We occasionally get questions about whether or not an I/O is planned
> to be added to Beam. I've added a list of known in-progress I/O
> Transforms to the "Built-in Transforms" page (where in-progress is
> defined as "has a JIRA issue"). The Built-In Transforms page is the
> publicly visible list of I/O transforms, so it's a natural place to let users know what's also in-progress.
>
> You can find the current list here:
> https://beam.apache.org/documentation/io/built-in/
>
> I may have missed one or two I/Os (there's no good way to find them in
> JIRA as far as I can tell), so if you're working on an I/O Transform
> that you intend to contribute to beam that's not listed, feel free to send a PR adding it to the list.
>
> Thanks,
> Stephen
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: 答复: 答复: Public in-progress I/O Transform list
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Sure, I will push on my github. I will let you know.
Regards
JB
On 04/12/2017 08:30 AM, Tang Jijun(\u4e0a\u6d77_\u6280\u672f\u90e8_\u6570\u636e\u5e73\u53f0_\u5510\u89ca\u96bd) wrote:
> Great work.Could you show me your code? Thanks.
>
> -----\u90ae\u4ef6\u539f\u4ef6-----
> \u53d1\u4ef6\u4eba: Jean-Baptiste Onofr� [mailto:jb@nanthrax.net]
> \u53d1\u9001\u65f6\u95f4: 2017\u5e744\u670812\u65e5 14:22
> \u6536\u4ef6\u4eba: user@beam.apache.org
> \u4e3b\u9898: Re: \u7b54\u590d: Public in-progress I/O Transform list
>
> Hi,
>
> Yes, I saw that and already provided some info.
>
> The RestIO I have works as follow:
>
> - Producer Read (working)
>
> pipeline.apply(RestIO.read().from("http://localhost:8080/rest"))
>
> It provides a PCollection<String> where String is JSON or XML. It's also possible to provide a data binding with withDataBinding() method.
>
> It's basically a DoFn as it's not possible to do any splitting.
>
> - Producer Write (working)
>
> pipeline.apply()....apply(RestIO.write().to("http://localhost:8080/rest"))
>
> Here, it's basically the same DoFn as producer using input PCollection as REST request.
>
> - Consumer Read (work in progress)
>
> pipeline.apply(RestIO.read().bind("http://localhost:9090/foo"))
>
> Here it's an unbounded source that bind a CXF connector (the initialSplit() method defines where the connector will be bind).
>
> I'm experiment this as it's more "ambitious" than the producer mode.
>
> Regards
> JB
>
> On 04/12/2017 08:16 AM, Tang Jijun(\u4e0a\u6d77_\u6280\u672f\u90e8_\u6570\u636e\u5e73\u53f0_\u5510\u89ca\u96bd) wrote:
>> Hi Stephen,
>> We need restIO module. I am looking forward to your pr about it. For restIO ,I create a jira https://issues.apache.org/jira/browse/BEAM-1946.
>>
>> -----\u90ae\u4ef6\u539f\u4ef6-----
>> \u53d1\u4ef6\u4eba: Jean-Baptiste Onofr� [mailto:jb@nanthrax.net]
>> \u53d1\u9001\u65f6\u95f4: 2017\u5e744\u670812\u65e5 14:10
>> \u6536\u4ef6\u4eba: user@beam.apache.org
>> \u4e3b\u9898: Re: Public in-progress I/O Transform list
>>
>> Hi Stephen,
>>
>> Thanks !
>>
>> I would add Socket IO and REST IO that I have on local branches (I'm
>> waiting to close some of my PRs before opening new ones). I also have
>> (not fully ready for
>> PR) CamelIO, IronMQIO, and others (I will update the website on the fly ;)).
>>
>> MQTT IO is missing in the list. Gonna provide a website PR about that.
>>
>> I also created ExecIO while ago but we didn't get a consensus about what we should do with it.
>>
>> Regards
>> JB
>>
>> On 04/12/2017 02:01 AM, Stephen Sisk wrote:
>>> Hi!
>>>
>>> We occasionally get questions about whether or not an I/O is planned
>>> to be added to Beam. I've added a list of known in-progress I/O
>>> Transforms to the "Built-in Transforms" page (where in-progress is
>>> defined as "has a JIRA issue"). The Built-In Transforms page is the
>>> publicly visible list of I/O transforms, so it's a natural place to let users know what's also in-progress.
>>>
>>> You can find the current list here:
>>> https://beam.apache.org/documentation/io/built-in/
>>>
>>> I may have missed one or two I/Os (there's no good way to find them
>>> in JIRA as far as I can tell), so if you're working on an I/O
>>> Transform that you intend to contribute to beam that's not listed, feel free to send a PR adding it to the list.
>>>
>>> Thanks,
>>> Stephen
>>
>> --
>> Jean-Baptiste Onofr�
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
> --
> Jean-Baptiste Onofr�
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
--
Jean-Baptiste Onofr�
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
答复: 答复: Public in-progress I/O Transform list
Posted by "Tang Jijun (上海_技术部_数据平台_唐觊隽)" <ta...@yhd.com>.
Great work.Could you show me your code? Thanks.
-----邮件原件-----
发件人: Jean-Baptiste Onofré [mailto:jb@nanthrax.net]
发送时间: 2017年4月12日 14:22
收件人: user@beam.apache.org
主题: Re: 答复: Public in-progress I/O Transform list
Hi,
Yes, I saw that and already provided some info.
The RestIO I have works as follow:
- Producer Read (working)
pipeline.apply(RestIO.read().from("http://localhost:8080/rest"))
It provides a PCollection<String> where String is JSON or XML. It's also possible to provide a data binding with withDataBinding() method.
It's basically a DoFn as it's not possible to do any splitting.
- Producer Write (working)
pipeline.apply()....apply(RestIO.write().to("http://localhost:8080/rest"))
Here, it's basically the same DoFn as producer using input PCollection as REST request.
- Consumer Read (work in progress)
pipeline.apply(RestIO.read().bind("http://localhost:9090/foo"))
Here it's an unbounded source that bind a CXF connector (the initialSplit() method defines where the connector will be bind).
I'm experiment this as it's more "ambitious" than the producer mode.
Regards
JB
On 04/12/2017 08:16 AM, Tang Jijun(上海_技术部_数据平台_唐觊隽) wrote:
> Hi Stephen,
> We need restIO module. I am looking forward to your pr about it. For restIO ,I create a jira https://issues.apache.org/jira/browse/BEAM-1946.
>
> -----邮件原件-----
> 发件人: Jean-Baptiste Onofré [mailto:jb@nanthrax.net]
> 发送时间: 2017年4月12日 14:10
> 收件人: user@beam.apache.org
> 主题: Re: Public in-progress I/O Transform list
>
> Hi Stephen,
>
> Thanks !
>
> I would add Socket IO and REST IO that I have on local branches (I'm
> waiting to close some of my PRs before opening new ones). I also have
> (not fully ready for
> PR) CamelIO, IronMQIO, and others (I will update the website on the fly ;)).
>
> MQTT IO is missing in the list. Gonna provide a website PR about that.
>
> I also created ExecIO while ago but we didn't get a consensus about what we should do with it.
>
> Regards
> JB
>
> On 04/12/2017 02:01 AM, Stephen Sisk wrote:
>> Hi!
>>
>> We occasionally get questions about whether or not an I/O is planned
>> to be added to Beam. I've added a list of known in-progress I/O
>> Transforms to the "Built-in Transforms" page (where in-progress is
>> defined as "has a JIRA issue"). The Built-In Transforms page is the
>> publicly visible list of I/O transforms, so it's a natural place to let users know what's also in-progress.
>>
>> You can find the current list here:
>> https://beam.apache.org/documentation/io/built-in/
>>
>> I may have missed one or two I/Os (there's no good way to find them
>> in JIRA as far as I can tell), so if you're working on an I/O
>> Transform that you intend to contribute to beam that's not listed, feel free to send a PR adding it to the list.
>>
>> Thanks,
>> Stephen
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: 答复: Public in-progress I/O Transform list
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi,
Yes, I saw that and already provided some info.
The RestIO I have works as follow:
- Producer Read (working)
pipeline.apply(RestIO.read().from("http://localhost:8080/rest"))
It provides a PCollection<String> where String is JSON or XML. It's also
possible to provide a data binding with withDataBinding() method.
It's basically a DoFn as it's not possible to do any splitting.
- Producer Write (working)
pipeline.apply()....apply(RestIO.write().to("http://localhost:8080/rest"))
Here, it's basically the same DoFn as producer using input PCollection as REST
request.
- Consumer Read (work in progress)
pipeline.apply(RestIO.read().bind("http://localhost:9090/foo"))
Here it's an unbounded source that bind a CXF connector (the initialSplit()
method defines where the connector will be bind).
I'm experiment this as it's more "ambitious" than the producer mode.
Regards
JB
On 04/12/2017 08:16 AM, Tang Jijun(\u4e0a\u6d77_\u6280\u672f\u90e8_\u6570\u636e\u5e73\u53f0_\u5510\u89ca\u96bd) wrote:
> Hi Stephen,
> We need restIO module. I am looking forward to your pr about it. For restIO ,I create a jira https://issues.apache.org/jira/browse/BEAM-1946.
>
> -----\u90ae\u4ef6\u539f\u4ef6-----
> \u53d1\u4ef6\u4eba: Jean-Baptiste Onofr� [mailto:jb@nanthrax.net]
> \u53d1\u9001\u65f6\u95f4: 2017\u5e744\u670812\u65e5 14:10
> \u6536\u4ef6\u4eba: user@beam.apache.org
> \u4e3b\u9898: Re: Public in-progress I/O Transform list
>
> Hi Stephen,
>
> Thanks !
>
> I would add Socket IO and REST IO that I have on local branches (I'm waiting to close some of my PRs before opening new ones). I also have (not fully ready for
> PR) CamelIO, IronMQIO, and others (I will update the website on the fly ;)).
>
> MQTT IO is missing in the list. Gonna provide a website PR about that.
>
> I also created ExecIO while ago but we didn't get a consensus about what we should do with it.
>
> Regards
> JB
>
> On 04/12/2017 02:01 AM, Stephen Sisk wrote:
>> Hi!
>>
>> We occasionally get questions about whether or not an I/O is planned
>> to be added to Beam. I've added a list of known in-progress I/O
>> Transforms to the "Built-in Transforms" page (where in-progress is
>> defined as "has a JIRA issue"). The Built-In Transforms page is the
>> publicly visible list of I/O transforms, so it's a natural place to let users know what's also in-progress.
>>
>> You can find the current list here:
>> https://beam.apache.org/documentation/io/built-in/
>>
>> I may have missed one or two I/Os (there's no good way to find them in
>> JIRA as far as I can tell), so if you're working on an I/O Transform
>> that you intend to contribute to beam that's not listed, feel free to send a PR adding it to the list.
>>
>> Thanks,
>> Stephen
>
> --
> Jean-Baptiste Onofr�
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
--
Jean-Baptiste Onofr�
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com