You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by 李 德鑫 <de...@outlook.com> on 2018/03/16 14:11:38 UTC
答复: 答复: 答复: [GSOC|ROCKETMQ-123]Support Distributed Transactional Messaging
Hi yukon
I've wrote a design doc about this topic, please have a look.
Regards,
Dexin
________________________________
发件人: yukon <yu...@apache.org>
发送时间: 2018年3月9日 15:11:01
收件人: dev
主题: Re: 答复: 答复: [GSOC|ROCKETMQ-123]Support Distributed Transactional Messaging
Hi Dexin,
```
1. Does the transactional messaging need to be "exactly only once"
delivered?
```
Please check out https://issues.apache.org/jira/browse/ROCKETMQ-124
```
2. Where to store the transaction state? Since the state must be
persistent and also updated.
```
We could reuse the rocketmq-store model to store the transaction state.
And, don't worry about the dirty pages if you follow the rule: Append Only.
Regards,
yukon
On Wed, Mar 7, 2018 at 12:11 PM, 李 德鑫 <de...@outlook.com> wrote:
> Thank you, Yukon!
>
>
> I still have some question about transactional messaging
>
> 1. Does the transactional messaging need to be "exactly only once"
> delivered?
> 2. Where to store the transaction state? Since the state must be
> persistent and also updated.
>
>
> About the second question, I've considered about some methods
>
> 1. Inside of the message. But it is said that this would generate lots
> of dirty pages on the OS.
> 2. In a database, especially an SQL database. Since class
> JDBCTransactionStore already exists. But I want to know whether the DB will
> be the bottleneck of performance or scalability.
> 3. Like Kafka, use an internal topic called Transaction log. It will
> record all the operation about transaction states. And store the latest
> states in the memory. If a partition crashed, It will read the log to
> recover. But the recovery would be slow.
> 4. Similar to 3, store and the transaction state in the Transaction
> log. But if there's a lot of transaction message, (I guess)this would be
> like 1 to generate many dirty pages.
>
>
> Regards,
> Dexin
> ________________________________
> 发件人: yukon <yu...@apache.org>
> 发送时间: 2018年3月5日 12:28:47
> 收件人: dev
> 主题: Re: 答复: [GSOC|ROCKETMQ-123]Support Distributed Transactional Messaging
>
> Hi,
>
> The links you mentioned just are interfaces, we should implement these
> APIs.
>
> Regards,
> yukon
>
> On Tue, Feb 27, 2018 at 12:07 PM, 李 德鑫 <de...@outlook.com> wrote:
>
> > My previous email didn't clarify the question clearly, so I'll try to
> make
> > it in detail.
> >
> >
> > In this jira's(https://issues.apache.org/jira/browse/ROCKETMQ-123)
> > description:
> >
> >
> >
> >
> > Implement a TransactionProducer and LocalTransactionChecker, guarantee
> > message delivery and local transaction operations are atomic.
> >
> > The following is the simple transactional messaging flow:
> >
> > 1. TransactionProducer sends a half message to the broker. A half message
> > means it's not confirmed and can't be delivered to the consumer.
> >
> > 2. TransactionProducer executes the specific local transaction.
> >
> > 3. Commit or Rollback the half message according to the status of the
> > local transaction. A half message will be deleted if it's rollbacked and
> > delivered to the consumer if it's committed.
> >
> >
> >
> > The 1st step I found in https://github.com/apache/
> > rocketmq/blob/master/client/src/main/java/org/apache/
> > rocketmq/client/impl/producer/DefaultMQProducerImpl.java#L964
> >
> > The 2nd step I found in https://github.com/apache/
> > rocketmq/blob/master/client/src/main/java/org/apache/
> > rocketmq/client/impl/producer/DefaultMQProducerImpl.java#L977
> >
> > The 3rd step I found in https://github.com/apache/
> > rocketmq/blob/master/client/src/main/java/org/apache/
> > rocketmq/client/impl/producer/DefaultMQProducerImpl.java#L1026 for
> client
> > and https://github.com/apache/rocketmq/blob/master/broker/
> > src/main/java/org/apache/rocketmq/broker/processor/
> > EndTransactionProcessor.java#L49 for broker
> >
> >
> > And my question is what is the part to be implemented for this issue? Is
> > there anything I missed?
> >
> >
> >
> > ________________________________
> > 发件人: 李 德鑫 <De...@outlook.com>
> > 发送时间: 2018年2月23日 9:35:17
> > 收件人: dev@rocketmq.apache.org
> > 主题: [GSOC|ROCKETMQ-123]Support Distributed Transactional Messaging
> >
> > Hi,
> >
> > I am working on this issue
> > (https://issues.apache.org/jira/browse/ROCKETMQ-123) as my GSOC project
> > [ROCKETMQ-123] Support Distributed Transactional Messaging ...<
> > https://issues.apache.org/jira/browse/ROCKETMQ-123>
> > In most businesses, transactional messaging is frequently used to make a
> > series of operations as an atomic transaction. At present, RocketMQ
> doesn't
> > support ...
> > issues.apache.org
> >
> >
> >
> > However, there already have
> > TransactionMQProducer
> >
> > https://github.com/apache/rocketmq/blob/master/client/
> > src/main/java/org/apache/rocketmq/client/producer/
> > TransactionMQProducer.java<https://github.com/apache/
> > rocketmq/blob/master/client/src/main/java/org/apache/
> > rocketmq/client/producer/TransactionMQProducer.java#L23>
> > [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
> > https://github.com/apache/rocketmq/blob/master/client/src/main/java/org/
> > apache/rocketmq/client/producer/TransactionMQProducer.java#L23>
> >
> > apache/rocketmq<https://github.com/apache/rocketmq/
> > blob/master/client/src/main/java/org/apache/rocketmq/client/producer/
> > TransactionMQProducer.java#L23>
> > rocketmq - Mirror of Apache RocketMQ
> > github.com
> >
> > and TransactionCheckListener
> > https://github.com/apache/rocketmq/blob/master/client/
> > src/main/java/org/apache/rocketmq/client/producer/
> > TransactionCheckListener.java
> >
> > So, what is the part I should implement for this issue?
> >
> > Regards,
> > Dexin Li
> >
> >
>
Re: 答复: 答复: 答复: [GSOC|ROCKETMQ-123]Support Distributed Transactional Messaging
Posted by yukon <yu...@apache.org>.
+ riven
Hi Dexin,
Riven is a very senior engineer in RocketMQ community, especially in
distributed transaction area. He is enthusiastic and will help you finished
the design.
Regards,
yukon
On Fri, Mar 16, 2018 at 10:13 PM, 李 德鑫 <de...@outlook.com> wrote:
> 😓Sorry, I forgot to add the link
>
> https://docs.google.com/document/d/1HGQYPc3YtPga-jATx6whMYS7e11NQ6_
> r3tzC3yl4ulI/edit?usp=sharing
>
> ________________________________
> 发件人: 李 德鑫 <de...@outlook.com>
> 发送时间: 2018年3月16日 14:11:38
> 收件人: dev@rocketmq.apache.org
> 主题: 答复: 答复: 答复: [GSOC|ROCKETMQ-123]Support Distributed Transactional
> Messaging
>
> Hi yukon
>
> I've wrote a design doc about this topic, please have a look.
>
>
>
> Regards,
> Dexin
>
> ________________________________
> 发件人: yukon <yu...@apache.org>
> 发送时间: 2018年3月9日 15:11:01
> 收件人: dev
> 主题: Re: 答复: 答复: [GSOC|ROCKETMQ-123]Support Distributed Transactional
> Messaging
>
> Hi Dexin,
>
> ```
> 1. Does the transactional messaging need to be "exactly only once"
> delivered?
> ```
>
> Please check out https://issues.apache.org/jira/browse/ROCKETMQ-124
>
> ```
> 2. Where to store the transaction state? Since the state must be
> persistent and also updated.
> ```
>
> We could reuse the rocketmq-store model to store the transaction state.
>
> And, don't worry about the dirty pages if you follow the rule: Append Only.
>
> Regards,
> yukon
>
>
>
> On Wed, Mar 7, 2018 at 12:11 PM, 李 德鑫 <de...@outlook.com> wrote:
>
> > Thank you, Yukon!
> >
> >
> > I still have some question about transactional messaging
> >
> > 1. Does the transactional messaging need to be "exactly only once"
> > delivered?
> > 2. Where to store the transaction state? Since the state must be
> > persistent and also updated.
> >
> >
> > About the second question, I've considered about some methods
> >
> > 1. Inside of the message. But it is said that this would generate lots
> > of dirty pages on the OS.
> > 2. In a database, especially an SQL database. Since class
> > JDBCTransactionStore already exists. But I want to know whether the DB
> will
> > be the bottleneck of performance or scalability.
> > 3. Like Kafka, use an internal topic called Transaction log. It will
> > record all the operation about transaction states. And store the latest
> > states in the memory. If a partition crashed, It will read the log to
> > recover. But the recovery would be slow.
> > 4. Similar to 3, store and the transaction state in the Transaction
> > log. But if there's a lot of transaction message, (I guess)this would be
> > like 1 to generate many dirty pages.
> >
> >
> > Regards,
> > Dexin
> > ________________________________
> > 发件人: yukon <yu...@apache.org>
> > 发送时间: 2018年3月5日 12:28:47
> > 收件人: dev
> > 主题: Re: 答复: [GSOC|ROCKETMQ-123]Support Distributed Transactional
> Messaging
> >
> > Hi,
> >
> > The links you mentioned just are interfaces, we should implement these
> > APIs.
> >
> > Regards,
> > yukon
> >
> > On Tue, Feb 27, 2018 at 12:07 PM, 李 德鑫 <de...@outlook.com> wrote:
> >
> > > My previous email didn't clarify the question clearly, so I'll try to
> > make
> > > it in detail.
> > >
> > >
> > > In this jira's(https://issues.apache.org/jira/browse/ROCKETMQ-123)
> > > description:
> > >
> > >
> > >
> > >
> > > Implement a TransactionProducer and LocalTransactionChecker, guarantee
> > > message delivery and local transaction operations are atomic.
> > >
> > > The following is the simple transactional messaging flow:
> > >
> > > 1. TransactionProducer sends a half message to the broker. A half
> message
> > > means it's not confirmed and can't be delivered to the consumer.
> > >
> > > 2. TransactionProducer executes the specific local transaction.
> > >
> > > 3. Commit or Rollback the half message according to the status of the
> > > local transaction. A half message will be deleted if it's rollbacked
> and
> > > delivered to the consumer if it's committed.
> > >
> > >
> > >
> > > The 1st step I found in https://github.com/apache/
> > > rocketmq/blob/master/client/src/main/java/org/apache/
> > > rocketmq/client/impl/producer/DefaultMQProducerImpl.java#L964
> > >
> > > The 2nd step I found in https://github.com/apache/
> > > rocketmq/blob/master/client/src/main/java/org/apache/
> > > rocketmq/client/impl/producer/DefaultMQProducerImpl.java#L977
> > >
> > > The 3rd step I found in https://github.com/apache/
> > > rocketmq/blob/master/client/src/main/java/org/apache/
> > > rocketmq/client/impl/producer/DefaultMQProducerImpl.java#L1026 for
> > client
> > > and https://github.com/apache/rocketmq/blob/master/broker/
> > > src/main/java/org/apache/rocketmq/broker/processor/
> > > EndTransactionProcessor.java#L49 for broker
> > >
> > >
> > > And my question is what is the part to be implemented for this issue?
> Is
> > > there anything I missed?
> > >
> > >
> > >
> > > ________________________________
> > > 发件人: 李 德鑫 <De...@outlook.com>
> > > 发送时间: 2018年2月23日 9:35:17
> > > 收件人: dev@rocketmq.apache.org
> > > 主题: [GSOC|ROCKETMQ-123]Support Distributed Transactional Messaging
> > >
> > > Hi,
> > >
> > > I am working on this issue
> > > (https://issues.apache.org/jira/browse/ROCKETMQ-123) as my GSOC
> project
> > > [ROCKETMQ-123] Support Distributed Transactional Messaging ...<
> > > https://issues.apache.org/jira/browse/ROCKETMQ-123>
> > > In most businesses, transactional messaging is frequently used to make
> a
> > > series of operations as an atomic transaction. At present, RocketMQ
> > doesn't
> > > support ...
> > > issues.apache.org
> > >
> > >
> > >
> > > However, there already have
> > > TransactionMQProducer
> > >
> > > https://github.com/apache/rocketmq/blob/master/client/
> > > src/main/java/org/apache/rocketmq/client/producer/
> > > TransactionMQProducer.java<https://github.com/apache/
> > > rocketmq/blob/master/client/src/main/java/org/apache/
> > > rocketmq/client/producer/TransactionMQProducer.java#L23>
> > > [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
> > > https://github.com/apache/rocketmq/blob/master/client/
> src/main/java/org/
> > > apache/rocketmq/client/producer/TransactionMQProducer.java#L23>
> > >
> > > apache/rocketmq<https://github.com/apache/rocketmq/
> > > blob/master/client/src/main/java/org/apache/rocketmq/client/producer/
> > > TransactionMQProducer.java#L23>
> > > rocketmq - Mirror of Apache RocketMQ
> > > github.com
> > >
> > > and TransactionCheckListener
> > > https://github.com/apache/rocketmq/blob/master/client/
> > > src/main/java/org/apache/rocketmq/client/producer/
> > > TransactionCheckListener.java
> > >
> > > So, what is the part I should implement for this issue?
> > >
> > > Regards,
> > > Dexin Li
> > >
> > >
> >
>
答复: 答复: 答复: [GSOC|ROCKETMQ-123]Support Distributed Transactional Messaging
Posted by 李 德鑫 <de...@outlook.com>.
😓Sorry, I forgot to add the link
https://docs.google.com/document/d/1HGQYPc3YtPga-jATx6whMYS7e11NQ6_r3tzC3yl4ulI/edit?usp=sharing
________________________________
发件人: 李 德鑫 <de...@outlook.com>
发送时间: 2018年3月16日 14:11:38
收件人: dev@rocketmq.apache.org
主题: 答复: 答复: 答复: [GSOC|ROCKETMQ-123]Support Distributed Transactional Messaging
Hi yukon
I've wrote a design doc about this topic, please have a look.
Regards,
Dexin
________________________________
发件人: yukon <yu...@apache.org>
发送时间: 2018年3月9日 15:11:01
收件人: dev
主题: Re: 答复: 答复: [GSOC|ROCKETMQ-123]Support Distributed Transactional Messaging
Hi Dexin,
```
1. Does the transactional messaging need to be "exactly only once"
delivered?
```
Please check out https://issues.apache.org/jira/browse/ROCKETMQ-124
```
2. Where to store the transaction state? Since the state must be
persistent and also updated.
```
We could reuse the rocketmq-store model to store the transaction state.
And, don't worry about the dirty pages if you follow the rule: Append Only.
Regards,
yukon
On Wed, Mar 7, 2018 at 12:11 PM, 李 德鑫 <de...@outlook.com> wrote:
> Thank you, Yukon!
>
>
> I still have some question about transactional messaging
>
> 1. Does the transactional messaging need to be "exactly only once"
> delivered?
> 2. Where to store the transaction state? Since the state must be
> persistent and also updated.
>
>
> About the second question, I've considered about some methods
>
> 1. Inside of the message. But it is said that this would generate lots
> of dirty pages on the OS.
> 2. In a database, especially an SQL database. Since class
> JDBCTransactionStore already exists. But I want to know whether the DB will
> be the bottleneck of performance or scalability.
> 3. Like Kafka, use an internal topic called Transaction log. It will
> record all the operation about transaction states. And store the latest
> states in the memory. If a partition crashed, It will read the log to
> recover. But the recovery would be slow.
> 4. Similar to 3, store and the transaction state in the Transaction
> log. But if there's a lot of transaction message, (I guess)this would be
> like 1 to generate many dirty pages.
>
>
> Regards,
> Dexin
> ________________________________
> 发件人: yukon <yu...@apache.org>
> 发送时间: 2018年3月5日 12:28:47
> 收件人: dev
> 主题: Re: 答复: [GSOC|ROCKETMQ-123]Support Distributed Transactional Messaging
>
> Hi,
>
> The links you mentioned just are interfaces, we should implement these
> APIs.
>
> Regards,
> yukon
>
> On Tue, Feb 27, 2018 at 12:07 PM, 李 德鑫 <de...@outlook.com> wrote:
>
> > My previous email didn't clarify the question clearly, so I'll try to
> make
> > it in detail.
> >
> >
> > In this jira's(https://issues.apache.org/jira/browse/ROCKETMQ-123)
> > description:
> >
> >
> >
> >
> > Implement a TransactionProducer and LocalTransactionChecker, guarantee
> > message delivery and local transaction operations are atomic.
> >
> > The following is the simple transactional messaging flow:
> >
> > 1. TransactionProducer sends a half message to the broker. A half message
> > means it's not confirmed and can't be delivered to the consumer.
> >
> > 2. TransactionProducer executes the specific local transaction.
> >
> > 3. Commit or Rollback the half message according to the status of the
> > local transaction. A half message will be deleted if it's rollbacked and
> > delivered to the consumer if it's committed.
> >
> >
> >
> > The 1st step I found in https://github.com/apache/
> > rocketmq/blob/master/client/src/main/java/org/apache/
> > rocketmq/client/impl/producer/DefaultMQProducerImpl.java#L964
> >
> > The 2nd step I found in https://github.com/apache/
> > rocketmq/blob/master/client/src/main/java/org/apache/
> > rocketmq/client/impl/producer/DefaultMQProducerImpl.java#L977
> >
> > The 3rd step I found in https://github.com/apache/
> > rocketmq/blob/master/client/src/main/java/org/apache/
> > rocketmq/client/impl/producer/DefaultMQProducerImpl.java#L1026 for
> client
> > and https://github.com/apache/rocketmq/blob/master/broker/
> > src/main/java/org/apache/rocketmq/broker/processor/
> > EndTransactionProcessor.java#L49 for broker
> >
> >
> > And my question is what is the part to be implemented for this issue? Is
> > there anything I missed?
> >
> >
> >
> > ________________________________
> > 发件人: 李 德鑫 <De...@outlook.com>
> > 发送时间: 2018年2月23日 9:35:17
> > 收件人: dev@rocketmq.apache.org
> > 主题: [GSOC|ROCKETMQ-123]Support Distributed Transactional Messaging
> >
> > Hi,
> >
> > I am working on this issue
> > (https://issues.apache.org/jira/browse/ROCKETMQ-123) as my GSOC project
> > [ROCKETMQ-123] Support Distributed Transactional Messaging ...<
> > https://issues.apache.org/jira/browse/ROCKETMQ-123>
> > In most businesses, transactional messaging is frequently used to make a
> > series of operations as an atomic transaction. At present, RocketMQ
> doesn't
> > support ...
> > issues.apache.org
> >
> >
> >
> > However, there already have
> > TransactionMQProducer
> >
> > https://github.com/apache/rocketmq/blob/master/client/
> > src/main/java/org/apache/rocketmq/client/producer/
> > TransactionMQProducer.java<https://github.com/apache/
> > rocketmq/blob/master/client/src/main/java/org/apache/
> > rocketmq/client/producer/TransactionMQProducer.java#L23>
> > [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
> > https://github.com/apache/rocketmq/blob/master/client/src/main/java/org/
> > apache/rocketmq/client/producer/TransactionMQProducer.java#L23>
> >
> > apache/rocketmq<https://github.com/apache/rocketmq/
> > blob/master/client/src/main/java/org/apache/rocketmq/client/producer/
> > TransactionMQProducer.java#L23>
> > rocketmq - Mirror of Apache RocketMQ
> > github.com
> >
> > and TransactionCheckListener
> > https://github.com/apache/rocketmq/blob/master/client/
> > src/main/java/org/apache/rocketmq/client/producer/
> > TransactionCheckListener.java
> >
> > So, what is the part I should implement for this issue?
> >
> > Regards,
> > Dexin Li
> >
> >
>