You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by Xin Wang <da...@gmail.com> on 2018/03/26 02:44:07 UTC
Re: The State of RocketMQ Streaming Integration
Hi folks,
I'm glad to say that the RocketMQ-Flink module has been merged into master.
https://github.com/apache/rocketmq-externals/tree/master/rocketmq-flink
Here is the state of rocketmq streaming integration:
Module Status
* RocketMQ-Storm *Available*
* RocketMQ-Spark *Available*, *The improvements will be available soon*.
* RocketMQ-Flink *Available*
* RocketMQ-Serializer(including JSON, Apache Avro) *Available*, *Not
merged yet*.
And, some new modules on the road:
* RocketMQ-SQL https://issues.apache.org/jira/browse/ROCKETMQ-357
* RocketMQ-Beam(Beam IO)
Anybody who is interested in these tasks too, please join me. Let's fight
together.
Thanks,
Xin Wang
2018-01-27 19:35 GMT+08:00 Xin Wang <da...@gmail.com>:
> Hi devs,
>
> I'd like to update the state of RocketMQ Streaming Integration:
> Now the task for RocketMQ-Flink integration is completed and patch is
> available here: https://github.com/apache/rocketmq-externals/pull/45
> Following is the brief changelog:
>
> - RocketMQSource - The RocketMQSource is based on RocketMQ pull
> consumer mode, and provides exactly once reliability guarantees when
> checkpoints are enabled.
> Otherwise, the source doesn't provide any reliability guarantees.
> - RocketMQSink - The RocketMQSink provides at-least-once reliability
> guarantees when checkpoints are enabled and withBatchFlushOnCheckpoint
> (true) is set.
> Otherwise, the sink reliability guarantees depends on rocketmq
> producer's retry policy, for this case, the messages sending way is sync by
> default,
> but you can change it by invoking withAsync(true).
> - KeyValueDeserializationSchema - The main API for deserializing topic
> and tags is the org.apache.rocketmq.flink.common.serialization.
> KeyValueDeserializationSchema interface.
> rocketmq-flink includes general purpose KeyValueDeserializationSchema implementations
> called SimpleKeyValueDeserializationSchema.
> - KeyValueSerializationSchema - The main API for serializing topic and
> tags is the org.apache.rocketmq.flink.common.serialization.
> KeyValueSerializationSchema interface.
> rocketmq-flink includes general purpose KeyValueSerializationSchema implementations
> called SimpleKeyValueSerializationSchema.
> - TopicSelector - The main API for selecting topic and tags is the
> org.apache.rocketmq.flink.common.selector.TopicSelector interface.
> rocketmq-flink includes general purpose TopicSelector implementations
> called DefaultTopicSelector and SimpleTopicSelector.
> - RocketMQFlinkExample - which receive messages from RocketMQ brokers
> and send messages to broker after processing.
>
> Any comments are welcome. And anybody who is also interested in these
> tasks, please join me. Let's fight together.
>
> Thanks,
> Xin Wang
>
>
> 2018-01-19 21:47 GMT+08:00 yukon <yu...@apache.org>:
>
>> Cool, thanks for your outstanding contribution, to help RocketMQ community
>> integrate with other streaming platforms.
>>
>> On Fri, Jan 19, 2018 at 7:52 PM, Xin Wang <da...@gmail.com> wrote:
>>
>> > Hi devs,
>> >
>> > I have finished the improvements for integrating RocketMQ with Apache
>> > Storm.
>> > Main changes are as following:
>> >
>> > - Upgraded RocketMQ version to 4.2.0 which brings improvements and
>> new
>> > features like batch sending
>> > - Imporved retry policy for RocketMQ consumer push mode to avoid data
>> > loss in some scenes
>> > - Batch sending supported for bolt and trident state
>> > - Allow running several consumer instances in one process, that is to
>> > say, different topics in one worker is possible
>> >
>> > PR: https://github.com/apache/storm/pull/2518
>> >
>> > And I submit the `RocketMQ-Serializer` patch several days ago. This
>> module
>> > includes several serialization formats, especially Apache Avro which I
>> > stated before[1].
>> >
>> > - Raw String
>> > - JSON
>> > - Avro Generic
>> > - Avro Specified
>> >
>> > PR: https://github.com/apache/rocketmq-externals/pull/42
>> >
>> > Any comments for these PRs are welcome. BTW here is the state of
>> rocketmq
>> > streaming integration:
>> > Module Status
>> > * RocketMQ-Storm Patch Available
>> > * RocketMQ-Spark To Refactor
>> > * RocketMQ-Flink Patch Available Soon
>> > * RocketMQ-Serializer Patch Available
>> >
>> > Anybody who is also interested in these tasks, please join me. Let's
>> fight
>> > together.
>> >
>> >
>> > [1] https://issues.apache.org/jira/browse/ROCKETMQ-157
>> >
>> >
>> > Thanks,
>> >
>> > Xin Wang (vesense)
>> >
>>
>
>
>
> --
> Thanks,
> Xin
>
--
Thanks,
Xin
Re: The State of RocketMQ Streaming Integration
Posted by Xin Wang <da...@gmail.com>.
Thanks very much. Glad to see that more users use and participate in the
bigdata eco-system connectors.
- Xin
2018-04-23 19:59 GMT+08:00 Von Gosling <vo...@apache.org>:
> HI Xing,
>
>
> Cool, some guys have looking for me to ask for help in the RocketMQ Data
> Connector, I am glad to introduce your works to them. Let’s put forward it
> together :-)
>
> BTW, I will merge some updates of the wiki which you have provided.
>
> Best Regards,
> Von Gosling
>
> > 在 2018年4月20日，14:47，Xin Wang <da...@gmail.com> 写道：
> >
> > Hi folks,
> >
> > Now the improvements for RocketMQ-Storm has been merged into master.
> >
> > Here is the latest state of rocketmq streaming integration:
> > Module Status
> > * RocketMQ-Storm Available
> > * RocketMQ-Spark Available, improvements patch to be merged.
> > * RocketMQ-Flink Available
> > * RocketMQ-Serializer(including JSON, Apache Avro) Available, to be
> merged.
> >
> > Some new modules on the road:
> > * RocketMQ-SQL https://issues.apache.org/jira/browse/ROCKETMQ-357 <
> https://issues.apache.org/jira/browse/ROCKETMQ-357>
> > * RocketMQ-Beam(Beam IO)
> >
> >
> > Recently, I'm working on the RocketMQ SQL. Hope the initial version can
> be released ASAP. Anybody who is interested in please join me.
> >
> >
> > Thanks,
> > Xin Wang
> >
>
>
--
Thanks,
Xin
Re: The State of RocketMQ Streaming Integration
Posted by Xin Wang <da...@gmail.com>.
Thanks very much. Glad to see that more users use and participate in the
bigdata eco-system connectors.
- Xin
2018-04-23 19:59 GMT+08:00 Von Gosling <vo...@apache.org>:
> HI Xing,
>
>
> Cool, some guys have looking for me to ask for help in the RocketMQ Data
> Connector, I am glad to introduce your works to them. Let’s put forward it
> together :-)
>
> BTW, I will merge some updates of the wiki which you have provided.
>
> Best Regards,
> Von Gosling
>
> > 在 2018年4月20日，14:47，Xin Wang <da...@gmail.com> 写道：
> >
> > Hi folks,
> >
> > Now the improvements for RocketMQ-Storm has been merged into master.
> >
> > Here is the latest state of rocketmq streaming integration:
> > Module Status
> > * RocketMQ-Storm Available
> > * RocketMQ-Spark Available, improvements patch to be merged.
> > * RocketMQ-Flink Available
> > * RocketMQ-Serializer(including JSON, Apache Avro) Available, to be
> merged.
> >
> > Some new modules on the road:
> > * RocketMQ-SQL https://issues.apache.org/jira/browse/ROCKETMQ-357 <
> https://issues.apache.org/jira/browse/ROCKETMQ-357>
> > * RocketMQ-Beam(Beam IO)
> >
> >
> > Recently, I'm working on the RocketMQ SQL. Hope the initial version can
> be released ASAP. Anybody who is interested in please join me.
> >
> >
> > Thanks,
> > Xin Wang
> >
>
>
--
Thanks,
Xin
Re: The State of RocketMQ Streaming Integration
Posted by Von Gosling <vo...@apache.org>.
HI Xing,
Cool, some guys have looking for me to ask for help in the RocketMQ Data Connector, I am glad to introduce your works to them. Let’s put forward it together :-)
BTW, I will merge some updates of the wiki which you have provided.
Best Regards,
Von Gosling
> 在 2018年4月20日，14:47，Xin Wang <da...@gmail.com> 写道：
>
> Hi folks,
>
> Now the improvements for RocketMQ-Storm has been merged into master.
>
> Here is the latest state of rocketmq streaming integration:
> Module Status
> * RocketMQ-Storm Available
> * RocketMQ-Spark Available, improvements patch to be merged.
> * RocketMQ-Flink Available
> * RocketMQ-Serializer(including JSON, Apache Avro) Available, to be merged.
>
> Some new modules on the road:
> * RocketMQ-SQL https://issues.apache.org/jira/browse/ROCKETMQ-357 <https://issues.apache.org/jira/browse/ROCKETMQ-357>
> * RocketMQ-Beam(Beam IO)
>
>
> Recently, I'm working on the RocketMQ SQL. Hope the initial version can be released ASAP. Anybody who is interested in please join me.
>
>
> Thanks,
> Xin Wang
>
Re: The State of RocketMQ Streaming Integration
Posted by Von Gosling <vo...@apache.org>.
HI Xing,
Cool, some guys have looking for me to ask for help in the RocketMQ Data Connector, I am glad to introduce your works to them. Let’s put forward it together :-)
BTW, I will merge some updates of the wiki which you have provided.
Best Regards,
Von Gosling
> 在 2018年4月20日，14:47，Xin Wang <da...@gmail.com> 写道：
>
> Hi folks,
>
> Now the improvements for RocketMQ-Storm has been merged into master.
>
> Here is the latest state of rocketmq streaming integration:
> Module Status
> * RocketMQ-Storm Available
> * RocketMQ-Spark Available, improvements patch to be merged.
> * RocketMQ-Flink Available
> * RocketMQ-Serializer(including JSON, Apache Avro) Available, to be merged.
>
> Some new modules on the road:
> * RocketMQ-SQL https://issues.apache.org/jira/browse/ROCKETMQ-357 <https://issues.apache.org/jira/browse/ROCKETMQ-357>
> * RocketMQ-Beam(Beam IO)
>
>
> Recently, I'm working on the RocketMQ SQL. Hope the initial version can be released ASAP. Anybody who is interested in please join me.
>
>
> Thanks,
> Xin Wang
>
Re: The State of RocketMQ Streaming Integration
Posted by Xin Wang <da...@gmail.com>.
Hi folks,
Now the improvements for RocketMQ-Storm has been merged into master.
Here is the latest state of rocketmq streaming integration:
Module Status
* RocketMQ-Storm *Available*
* RocketMQ-Spark *Available, improvements patch to be merged.*
* RocketMQ-Flink *Available*
* RocketMQ-Serializer(including JSON, Apache Avro) *Available, to be
merged.*
Some new modules on the road:
* RocketMQ-SQL https://issues.apache.org/jira/browse/ROCKETMQ-357
* RocketMQ-Beam(Beam IO)
Recently, I'm working on the RocketMQ *SQL*. Hope the initial version can
be released ASAP. Anybody who is interested in please join me.
Thanks,
Xin Wang
Re: The State of RocketMQ Streaming Integration
Posted by Xin Wang <da...@gmail.com>.
Hi folks,
Now the improvements for RocketMQ-Storm has been merged into master.
Here is the latest state of rocketmq streaming integration:
Module Status
* RocketMQ-Storm *Available*
* RocketMQ-Spark *Available, improvements patch to be merged.*
* RocketMQ-Flink *Available*
* RocketMQ-Serializer(including JSON, Apache Avro) *Available, to be
merged.*
Some new modules on the road:
* RocketMQ-SQL https://issues.apache.org/jira/browse/ROCKETMQ-357
* RocketMQ-Beam(Beam IO)
Recently, I'm working on the RocketMQ *SQL*. Hope the initial version can
be released ASAP. Anybody who is interested in please join me.
Thanks,
Xin Wang
Re: The State of RocketMQ Streaming Integration
Posted by yukon <yu...@apache.org>.
Thanks, Xin!
These plugins have greatly enriched the RocketMQ community, hope we could
get more feedback from the community about these plugins.
Regards,
yukon
On Mon, Mar 26, 2018 at 10:44 AM, Xin Wang <da...@gmail.com> wrote:
> Hi folks,
>
> I'm glad to say that the RocketMQ-Flink module has been merged into master.
> https://github.com/apache/rocketmq-externals/tree/master/rocketmq-flink
>
> Here is the state of rocketmq streaming integration:
> Module Status
> * RocketMQ-Storm *Available*
> * RocketMQ-Spark *Available*, *The improvements will be available soon*.
> * RocketMQ-Flink *Available*
> * RocketMQ-Serializer(including JSON, Apache Avro) *Available*, *Not
> merged yet*.
>
> And, some new modules on the road:
> * RocketMQ-SQL https://issues.apache.org/jira/browse/ROCKETMQ-357
> * RocketMQ-Beam(Beam IO)
>
>
> Anybody who is interested in these tasks too, please join me. Let's fight
> together.
>
>
> Thanks,
> Xin Wang
>
>
>
> 2018-01-27 19:35 GMT+08:00 Xin Wang <da...@gmail.com>:
>
> > Hi devs,
> >
> > I'd like to update the state of RocketMQ Streaming Integration:
> > Now the task for RocketMQ-Flink integration is completed and patch is
> > available here: https://github.com/apache/rocketmq-externals/pull/45
> > Following is the brief changelog:
> >
> > - RocketMQSource - The RocketMQSource is based on RocketMQ pull
> > consumer mode, and provides exactly once reliability guarantees when
> > checkpoints are enabled.
> > Otherwise, the source doesn't provide any reliability guarantees.
> > - RocketMQSink - The RocketMQSink provides at-least-once reliability
> > guarantees when checkpoints are enabled and withBatchFlushOnCheckpoint
> > (true) is set.
> > Otherwise, the sink reliability guarantees depends on rocketmq
> > producer's retry policy, for this case, the messages sending way is
> sync by
> > default,
> > but you can change it by invoking withAsync(true).
> > - KeyValueDeserializationSchema - The main API for deserializing topic
> > and tags is the org.apache.rocketmq.flink.common.serialization.
> > KeyValueDeserializationSchema interface.
> > rocketmq-flink includes general purpose KeyValueDeserializationSchema
> implementations
> > called SimpleKeyValueDeserializationSchema.
> > - KeyValueSerializationSchema - The main API for serializing topic and
> > tags is the org.apache.rocketmq.flink.common.serialization.
> > KeyValueSerializationSchema interface.
> > rocketmq-flink includes general purpose KeyValueSerializationSchema
> implementations
> > called SimpleKeyValueSerializationSchema.
> > - TopicSelector - The main API for selecting topic and tags is the
> > org.apache.rocketmq.flink.common.selector.TopicSelector interface.
> > rocketmq-flink includes general purpose TopicSelector implementations
> > called DefaultTopicSelector and SimpleTopicSelector.
> > - RocketMQFlinkExample - which receive messages from RocketMQ brokers
> > and send messages to broker after processing.
> >
> > Any comments are welcome. And anybody who is also interested in these
> > tasks, please join me. Let's fight together.
> >
> > Thanks,
> > Xin Wang
> >
> >
> > 2018-01-19 21:47 GMT+08:00 yukon <yu...@apache.org>:
> >
> >> Cool, thanks for your outstanding contribution, to help RocketMQ
> community
> >> integrate with other streaming platforms.
> >>
> >> On Fri, Jan 19, 2018 at 7:52 PM, Xin Wang <da...@gmail.com>
> wrote:
> >>
> >> > Hi devs,
> >> >
> >> > I have finished the improvements for integrating RocketMQ with Apache
> >> > Storm.
> >> > Main changes are as following:
> >> >
> >> > - Upgraded RocketMQ version to 4.2.0 which brings improvements and
> >> new
> >> > features like batch sending
> >> > - Imporved retry policy for RocketMQ consumer push mode to avoid
> data
> >> > loss in some scenes
> >> > - Batch sending supported for bolt and trident state
> >> > - Allow running several consumer instances in one process, that is
> to
> >> > say, different topics in one worker is possible
> >> >
> >> > PR: https://github.com/apache/storm/pull/2518
> >> >
> >> > And I submit the `RocketMQ-Serializer` patch several days ago. This
> >> module
> >> > includes several serialization formats, especially Apache Avro which I
> >> > stated before[1].
> >> >
> >> > - Raw String
> >> > - JSON
> >> > - Avro Generic
> >> > - Avro Specified
> >> >
> >> > PR: https://github.com/apache/rocketmq-externals/pull/42
> >> >
> >> > Any comments for these PRs are welcome. BTW here is the state of
> >> rocketmq
> >> > streaming integration:
> >> > Module Status
> >> > * RocketMQ-Storm Patch Available
> >> > * RocketMQ-Spark To Refactor
> >> > * RocketMQ-Flink Patch Available Soon
> >> > * RocketMQ-Serializer Patch Available
> >> >
> >> > Anybody who is also interested in these tasks, please join me. Let's
> >> fight
> >> > together.
> >> >
> >> >
> >> > [1] https://issues.apache.org/jira/browse/ROCKETMQ-157
> >> >
> >> >
> >> > Thanks,
> >> >
> >> > Xin Wang (vesense)
> >> >
> >>
> >
> >
> >
> > --
> > Thanks,
> > Xin
> >
>
>
>
> --
> Thanks,
> Xin
>
Re: The State of RocketMQ Streaming Integration
Posted by yukon <yu...@apache.org>.
Thanks, Xin!
These plugins have greatly enriched the RocketMQ community, hope we could
get more feedback from the community about these plugins.
Regards,
yukon
On Mon, Mar 26, 2018 at 10:44 AM, Xin Wang <da...@gmail.com> wrote:
> Hi folks,
>
> I'm glad to say that the RocketMQ-Flink module has been merged into master.
> https://github.com/apache/rocketmq-externals/tree/master/rocketmq-flink
>
> Here is the state of rocketmq streaming integration:
> Module Status
> * RocketMQ-Storm *Available*
> * RocketMQ-Spark *Available*, *The improvements will be available soon*.
> * RocketMQ-Flink *Available*
> * RocketMQ-Serializer(including JSON, Apache Avro) *Available*, *Not
> merged yet*.
>
> And, some new modules on the road:
> * RocketMQ-SQL https://issues.apache.org/jira/browse/ROCKETMQ-357
> * RocketMQ-Beam(Beam IO)
>
>
> Anybody who is interested in these tasks too, please join me. Let's fight
> together.
>
>
> Thanks,
> Xin Wang
>
>
>
> 2018-01-27 19:35 GMT+08:00 Xin Wang <da...@gmail.com>:
>
> > Hi devs,
> >
> > I'd like to update the state of RocketMQ Streaming Integration:
> > Now the task for RocketMQ-Flink integration is completed and patch is
> > available here: https://github.com/apache/rocketmq-externals/pull/45
> > Following is the brief changelog:
> >
> > - RocketMQSource - The RocketMQSource is based on RocketMQ pull
> > consumer mode, and provides exactly once reliability guarantees when
> > checkpoints are enabled.
> > Otherwise, the source doesn't provide any reliability guarantees.
> > - RocketMQSink - The RocketMQSink provides at-least-once reliability
> > guarantees when checkpoints are enabled and withBatchFlushOnCheckpoint
> > (true) is set.
> > Otherwise, the sink reliability guarantees depends on rocketmq
> > producer's retry policy, for this case, the messages sending way is
> sync by
> > default,
> > but you can change it by invoking withAsync(true).
> > - KeyValueDeserializationSchema - The main API for deserializing topic
> > and tags is the org.apache.rocketmq.flink.common.serialization.
> > KeyValueDeserializationSchema interface.
> > rocketmq-flink includes general purpose KeyValueDeserializationSchema
> implementations
> > called SimpleKeyValueDeserializationSchema.
> > - KeyValueSerializationSchema - The main API for serializing topic and
> > tags is the org.apache.rocketmq.flink.common.serialization.
> > KeyValueSerializationSchema interface.
> > rocketmq-flink includes general purpose KeyValueSerializationSchema
> implementations
> > called SimpleKeyValueSerializationSchema.
> > - TopicSelector - The main API for selecting topic and tags is the
> > org.apache.rocketmq.flink.common.selector.TopicSelector interface.
> > rocketmq-flink includes general purpose TopicSelector implementations
> > called DefaultTopicSelector and SimpleTopicSelector.
> > - RocketMQFlinkExample - which receive messages from RocketMQ brokers
> > and send messages to broker after processing.
> >
> > Any comments are welcome. And anybody who is also interested in these
> > tasks, please join me. Let's fight together.
> >
> > Thanks,
> > Xin Wang
> >
> >
> > 2018-01-19 21:47 GMT+08:00 yukon <yu...@apache.org>:
> >
> >> Cool, thanks for your outstanding contribution, to help RocketMQ
> community
> >> integrate with other streaming platforms.
> >>
> >> On Fri, Jan 19, 2018 at 7:52 PM, Xin Wang <da...@gmail.com>
> wrote:
> >>
> >> > Hi devs,
> >> >
> >> > I have finished the improvements for integrating RocketMQ with Apache
> >> > Storm.
> >> > Main changes are as following:
> >> >
> >> > - Upgraded RocketMQ version to 4.2.0 which brings improvements and
> >> new
> >> > features like batch sending
> >> > - Imporved retry policy for RocketMQ consumer push mode to avoid
> data
> >> > loss in some scenes
> >> > - Batch sending supported for bolt and trident state
> >> > - Allow running several consumer instances in one process, that is
> to
> >> > say, different topics in one worker is possible
> >> >
> >> > PR: https://github.com/apache/storm/pull/2518
> >> >
> >> > And I submit the `RocketMQ-Serializer` patch several days ago. This
> >> module
> >> > includes several serialization formats, especially Apache Avro which I
> >> > stated before[1].
> >> >
> >> > - Raw String
> >> > - JSON
> >> > - Avro Generic
> >> > - Avro Specified
> >> >
> >> > PR: https://github.com/apache/rocketmq-externals/pull/42
> >> >
> >> > Any comments for these PRs are welcome. BTW here is the state of
> >> rocketmq
> >> > streaming integration:
> >> > Module Status
> >> > * RocketMQ-Storm Patch Available
> >> > * RocketMQ-Spark To Refactor
> >> > * RocketMQ-Flink Patch Available Soon
> >> > * RocketMQ-Serializer Patch Available
> >> >
> >> > Anybody who is also interested in these tasks, please join me. Let's
> >> fight
> >> > together.
> >> >
> >> >
> >> > [1] https://issues.apache.org/jira/browse/ROCKETMQ-157
> >> >
> >> >
> >> > Thanks,
> >> >
> >> > Xin Wang (vesense)
> >> >
> >>
> >
> >
> >
> > --
> > Thanks,
> > Xin
> >
>
>
>
> --
> Thanks,
> Xin
>