You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by 倪泽 <un...@gmail.com> on 2022/02/09 09:38:03 UTC
[DISCUSS][RIP-35] Use rocketmq instead of mysql as state store
Hi, RocketMQ Community,
First of all, Happy Chinese New Year!
I want to start a RIP to replace mysql with RocketMQ as state store in
RocketMQ-streams. Base the RocketMQ storage, I will provide a exactly-once
supported and efficient state storage scheme.
I have written my proposal and you can click on the link below：
https://docs.google.com/document/d/1gmwU4rC5wyG07R55jW5oN4GzxHSq8QoFNdoNL61xT0M/edit?usp=sharing
Chinese version：
https://shimo.im/docs/pjDRKHqtdQC9RWCg
If you have any questions or suggestions, please reply to this email or
comment on the proposal.
Thanks
Nize
Re: [DISCUSS][RIP-35] Use rocketmq instead of mysql as state store
Posted by ze Ni <un...@gmail.com>.
please try again, my colleague can access shimo doc， maybe need to login
in it first.
dongeforever <do...@apache.org> 于2022年2月9日周三 18:44写道：
> The. Shimo doc cannot be accessed.
>
> Maybe the public permission is not granted.
>
> heng du <du...@apache.org> 于2022年2月9日周三 17:55写道：
>
> > Great, this RIP not only removes database dependencies, but also provides
> > best practices for state storage in RocketMQ
> >
> > 倪泽 <un...@gmail.com> 于2022年2月9日周三 17:38写道：
> >
> > > Hi, RocketMQ Community,
> > >
> > > First of all, Happy Chinese New Year!
> > > I want to start a RIP to replace mysql with RocketMQ as state store in
> > > RocketMQ-streams. Base the RocketMQ storage, I will provide a
> > exactly-once
> > > supported and efficient state storage scheme.
> > >
> > > I have written my proposal and you can click on the link below：
> > >
> > >
> > >
> >
> https://docs.google.com/document/d/1gmwU4rC5wyG07R55jW5oN4GzxHSq8QoFNdoNL61xT0M/edit?usp=sharing
> > >
> > > Chinese version：
> > >
> > > https://shimo.im/docs/pjDRKHqtdQC9RWCg
> > >
> > > If you have any questions or suggestions, please reply to this email or
> > > comment on the proposal.
> > >
> > >
> > > Thanks
> > > Nize
> > >
> >
>
Re: [DISCUSS][RIP-35] Use rocketmq instead of mysql as state store
Posted by dongeforever <do...@apache.org>.
The. Shimo doc cannot be accessed.
Maybe the public permission is not granted.
heng du <du...@apache.org> 于2022年2月9日周三 17:55写道：
> Great, this RIP not only removes database dependencies, but also provides
> best practices for state storage in RocketMQ
>
> 倪泽 <un...@gmail.com> 于2022年2月9日周三 17:38写道：
>
> > Hi, RocketMQ Community,
> >
> > First of all, Happy Chinese New Year!
> > I want to start a RIP to replace mysql with RocketMQ as state store in
> > RocketMQ-streams. Base the RocketMQ storage, I will provide a
> exactly-once
> > supported and efficient state storage scheme.
> >
> > I have written my proposal and you can click on the link below：
> >
> >
> >
> https://docs.google.com/document/d/1gmwU4rC5wyG07R55jW5oN4GzxHSq8QoFNdoNL61xT0M/edit?usp=sharing
> >
> > Chinese version：
> >
> > https://shimo.im/docs/pjDRKHqtdQC9RWCg
> >
> > If you have any questions or suggestions, please reply to this email or
> > comment on the proposal.
> >
> >
> > Thanks
> > Nize
> >
>
Re: [DISCUSS][RIP-35] Use rocketmq instead of mysql as state store
Posted by heng du <du...@apache.org>.
Great, this RIP not only removes database dependencies, but also provides
best practices for state storage in RocketMQ
倪泽 <un...@gmail.com> 于2022年2月9日周三 17:38写道：
> Hi, RocketMQ Community,
>
> First of all, Happy Chinese New Year!
> I want to start a RIP to replace mysql with RocketMQ as state store in
> RocketMQ-streams. Base the RocketMQ storage, I will provide a exactly-once
> supported and efficient state storage scheme.
>
> I have written my proposal and you can click on the link below：
>
>
> https://docs.google.com/document/d/1gmwU4rC5wyG07R55jW5oN4GzxHSq8QoFNdoNL61xT0M/edit?usp=sharing
>
> Chinese version：
>
> https://shimo.im/docs/pjDRKHqtdQC9RWCg
>
> If you have any questions or suggestions, please reply to this email or
> comment on the proposal.
>
>
> Thanks
> Nize
>
Re: [DISCUSS][RIP-35] Use rocketmq instead of mysql as state store
Posted by ze Ni <un...@gmail.com>.
It does have a connection with RIP-30, compact topic can speed up state
recovery.
I add some explanation to illustrate how compact topic work on state
recovery in the doc.
dongeforever <do...@apache.org> 于2022年2月10日周四 17:59写道：
> What is the relationship between this RIP and RIP-30 ?
>
> If this RIP is based on RIP-30, it is better to attach some description.
>
> If this RIP is independent, it needs to explain the detailed relationship.
>
>
> ze Ni <un...@gmail.com> 于2022年2月10日周四 14:42写道：
>
> > Thanks for your review, and I am glad to add you into shepherds list.
> >
> >
> > Exactly-once involves a larger range, such as when we should
> > submit consumption offset and store state data, and what it's behavior is
> > when RocketMQ-streams recover from crash. This RIP only discusses a part
> of
> > them, to achieve exactly-once how we should store and recover state with
> > RocketMQ.
> >
> > When a message is found from a new shard, RocketMQ-streams load it's
> > corresponding state.When shard removed because of rebalancing,
> > RocketMQ-streams save state to RocketMQ and remove local. When making a
> > checkpoint, firstly save state to RocketMQ, secondly commit state
> > consumption offset, and at last, poll new state into local.
> >
> > When RocketMQ-streams crash between checkpoints, state data in RocketMQ
> > exactly correspond to the last checkpoint, and state consumption offset
> > corresponds to the penultimate checkpoint.
> >
> >
> > That is to say, we can recover state from the last checkpoint when
> > RocketMQ-streams restart, which is state behavior in exactly-once
> > semantics.
> >
> > As for the complete exact-once implementation, I would like to discuss
> with
> > another RIP.
> >
> > vongosling <vo...@apache.org> 于2022年2月10日周四 08:55写道：
> >
> > > I like IStorage design, good to start. This is a good improvement for
> our
> > > streams architecture. If you need another shepherd, you could count me
> > :-)
> > >
> > > BTW, your design seems to not solve the exactly-once semantic in the
> > > streaming scenario, or say, we miss some detailed comments in the doc?
> > >
> > > 倪泽 <un...@gmail.com> 于2022年2月9日周三 17:38写道：
> > >
> > > > Hi, RocketMQ Community,
> > > >
> > > > First of all, Happy Chinese New Year!
> > > > I want to start a RIP to replace mysql with RocketMQ as state store
> in
> > > > RocketMQ-streams. Base the RocketMQ storage, I will provide a
> > > exactly-once
> > > > supported and efficient state storage scheme.
> > > >
> > > > I have written my proposal and you can click on the link below：
> > > >
> > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1gmwU4rC5wyG07R55jW5oN4GzxHSq8QoFNdoNL61xT0M/edit?usp=sharing
> > > >
> > > > Chinese version：
> > > >
> > > > https://shimo.im/docs/pjDRKHqtdQC9RWCg
> > > >
> > > > If you have any questions or suggestions, please reply to this email
> or
> > > > comment on the proposal.
> > > >
> > > >
> > > > Thanks
> > > > Nize
> > > >
> > >
> > >
> > > --
> > > Best Regards :-)
> > >
> >
>
Re: [DISCUSS][RIP-35] Use rocketmq instead of mysql as state store
Posted by dongeforever <do...@apache.org>.
What is the relationship between this RIP and RIP-30 ?
If this RIP is based on RIP-30, it is better to attach some description.
If this RIP is independent, it needs to explain the detailed relationship.
ze Ni <un...@gmail.com> 于2022年2月10日周四 14:42写道：
> Thanks for your review, and I am glad to add you into shepherds list.
>
>
> Exactly-once involves a larger range, such as when we should
> submit consumption offset and store state data, and what it's behavior is
> when RocketMQ-streams recover from crash. This RIP only discusses a part of
> them, to achieve exactly-once how we should store and recover state with
> RocketMQ.
>
> When a message is found from a new shard, RocketMQ-streams load it's
> corresponding state.When shard removed because of rebalancing,
> RocketMQ-streams save state to RocketMQ and remove local. When making a
> checkpoint, firstly save state to RocketMQ, secondly commit state
> consumption offset, and at last, poll new state into local.
>
> When RocketMQ-streams crash between checkpoints, state data in RocketMQ
> exactly correspond to the last checkpoint, and state consumption offset
> corresponds to the penultimate checkpoint.
>
>
> That is to say, we can recover state from the last checkpoint when
> RocketMQ-streams restart, which is state behavior in exactly-once
> semantics.
>
> As for the complete exact-once implementation, I would like to discuss with
> another RIP.
>
> vongosling <vo...@apache.org> 于2022年2月10日周四 08:55写道：
>
> > I like IStorage design, good to start. This is a good improvement for our
> > streams architecture. If you need another shepherd, you could count me
> :-)
> >
> > BTW, your design seems to not solve the exactly-once semantic in the
> > streaming scenario, or say, we miss some detailed comments in the doc?
> >
> > 倪泽 <un...@gmail.com> 于2022年2月9日周三 17:38写道：
> >
> > > Hi, RocketMQ Community,
> > >
> > > First of all, Happy Chinese New Year!
> > > I want to start a RIP to replace mysql with RocketMQ as state store in
> > > RocketMQ-streams. Base the RocketMQ storage, I will provide a
> > exactly-once
> > > supported and efficient state storage scheme.
> > >
> > > I have written my proposal and you can click on the link below：
> > >
> > >
> > >
> >
> https://docs.google.com/document/d/1gmwU4rC5wyG07R55jW5oN4GzxHSq8QoFNdoNL61xT0M/edit?usp=sharing
> > >
> > > Chinese version：
> > >
> > > https://shimo.im/docs/pjDRKHqtdQC9RWCg
> > >
> > > If you have any questions or suggestions, please reply to this email or
> > > comment on the proposal.
> > >
> > >
> > > Thanks
> > > Nize
> > >
> >
> >
> > --
> > Best Regards :-)
> >
>
Re: [DISCUSS][RIP-35] Use rocketmq instead of mysql as state store
Posted by ze Ni <un...@gmail.com>.
Thanks for your review, and I am glad to add you into shepherds list.
Exactly-once involves a larger range, such as when we should
submit consumption offset and store state data, and what it's behavior is
when RocketMQ-streams recover from crash. This RIP only discusses a part of
them, to achieve exactly-once how we should store and recover state with
RocketMQ.
When a message is found from a new shard, RocketMQ-streams load it's
corresponding state.When shard removed because of rebalancing,
RocketMQ-streams save state to RocketMQ and remove local. When making a
checkpoint, firstly save state to RocketMQ, secondly commit state
consumption offset, and at last, poll new state into local.
When RocketMQ-streams crash between checkpoints, state data in RocketMQ
exactly correspond to the last checkpoint, and state consumption offset
corresponds to the penultimate checkpoint.
That is to say, we can recover state from the last checkpoint when
RocketMQ-streams restart, which is state behavior in exactly-once
semantics.
As for the complete exact-once implementation, I would like to discuss with
another RIP.
vongosling <vo...@apache.org> 于2022年2月10日周四 08:55写道：
> I like IStorage design, good to start. This is a good improvement for our
> streams architecture. If you need another shepherd, you could count me :-)
>
> BTW, your design seems to not solve the exactly-once semantic in the
> streaming scenario, or say, we miss some detailed comments in the doc?
>
> 倪泽 <un...@gmail.com> 于2022年2月9日周三 17:38写道：
>
> > Hi, RocketMQ Community,
> >
> > First of all, Happy Chinese New Year!
> > I want to start a RIP to replace mysql with RocketMQ as state store in
> > RocketMQ-streams. Base the RocketMQ storage, I will provide a
> exactly-once
> > supported and efficient state storage scheme.
> >
> > I have written my proposal and you can click on the link below：
> >
> >
> >
> https://docs.google.com/document/d/1gmwU4rC5wyG07R55jW5oN4GzxHSq8QoFNdoNL61xT0M/edit?usp=sharing
> >
> > Chinese version：
> >
> > https://shimo.im/docs/pjDRKHqtdQC9RWCg
> >
> > If you have any questions or suggestions, please reply to this email or
> > comment on the proposal.
> >
> >
> > Thanks
> > Nize
> >
>
>
> --
> Best Regards :-)
>
Re: [DISCUSS][RIP-35] Use rocketmq instead of mysql as state store
Posted by vongosling <vo...@apache.org>.
I like IStorage design, good to start. This is a good improvement for our
streams architecture. If you need another shepherd, you could count me :-)
BTW, your design seems to not solve the exactly-once semantic in the
streaming scenario, or say, we miss some detailed comments in the doc?
倪泽 <un...@gmail.com> 于2022年2月9日周三 17:38写道：
> Hi, RocketMQ Community,
>
> First of all, Happy Chinese New Year!
> I want to start a RIP to replace mysql with RocketMQ as state store in
> RocketMQ-streams. Base the RocketMQ storage, I will provide a exactly-once
> supported and efficient state storage scheme.
>
> I have written my proposal and you can click on the link below：
>
>
> https://docs.google.com/document/d/1gmwU4rC5wyG07R55jW5oN4GzxHSq8QoFNdoNL61xT0M/edit?usp=sharing
>
> Chinese version：
>
> https://shimo.im/docs/pjDRKHqtdQC9RWCg
>
> If you have any questions or suggestions, please reply to this email or
> comment on the proposal.
>
>
> Thanks
> Nize
>
--
Best Regards :-)