You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by yukon <yu...@apache.org> on 2018/03/05 12:28:47 UTC
Re: 答复: [GSOC|ROCKETMQ-123]Support Distributed Transactional Messaging
Hi,
The links you mentioned just are interfaces, we should implement these APIs.
Regards,
yukon
On Tue, Feb 27, 2018 at 12:07 PM, 李 德鑫 <de...@outlook.com> wrote:
> My previous email didn't clarify the question clearly, so I'll try to make
> it in detail.
>
>
> In this jira's(https://issues.apache.org/jira/browse/ROCKETMQ-123)
> description:
>
>
>
>
> Implement a TransactionProducer and LocalTransactionChecker, guarantee
> message delivery and local transaction operations are atomic.
>
> The following is the simple transactional messaging flow:
>
> 1. TransactionProducer sends a half message to the broker. A half message
> means it's not confirmed and can't be delivered to the consumer.
>
> 2. TransactionProducer executes the specific local transaction.
>
> 3. Commit or Rollback the half message according to the status of the
> local transaction. A half message will be deleted if it's rollbacked and
> delivered to the consumer if it's committed.
>
>
>
> The 1st step I found in https://github.com/apache/
> rocketmq/blob/master/client/src/main/java/org/apache/
> rocketmq/client/impl/producer/DefaultMQProducerImpl.java#L964
>
> The 2nd step I found in https://github.com/apache/
> rocketmq/blob/master/client/src/main/java/org/apache/
> rocketmq/client/impl/producer/DefaultMQProducerImpl.java#L977
>
> The 3rd step I found in https://github.com/apache/
> rocketmq/blob/master/client/src/main/java/org/apache/
> rocketmq/client/impl/producer/DefaultMQProducerImpl.java#L1026 for client
> and https://github.com/apache/rocketmq/blob/master/broker/
> src/main/java/org/apache/rocketmq/broker/processor/
> EndTransactionProcessor.java#L49 for broker
>
>
> And my question is what is the part to be implemented for this issue? Is
> there anything I missed?
>
>
>
> ________________________________
> 发件人: 李 德鑫 <De...@outlook.com>
> 发送时间: 2018年2月23日 9:35:17
> 收件人: dev@rocketmq.apache.org
> 主题: [GSOC|ROCKETMQ-123]Support Distributed Transactional Messaging
>
> Hi,
>
> I am working on this issue
> (https://issues.apache.org/jira/browse/ROCKETMQ-123) as my GSOC project
> [ROCKETMQ-123] Support Distributed Transactional Messaging ...<
> https://issues.apache.org/jira/browse/ROCKETMQ-123>
> In most businesses, transactional messaging is frequently used to make a
> series of operations as an atomic transaction. At present, RocketMQ doesn't
> support ...
> issues.apache.org
>
>
>
> However, there already have
> TransactionMQProducer
>
> https://github.com/apache/rocketmq/blob/master/client/
> src/main/java/org/apache/rocketmq/client/producer/
> TransactionMQProducer.java<https://github.com/apache/
> rocketmq/blob/master/client/src/main/java/org/apache/
> rocketmq/client/producer/TransactionMQProducer.java#L23>
> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
> https://github.com/apache/rocketmq/blob/master/client/src/main/java/org/
> apache/rocketmq/client/producer/TransactionMQProducer.java#L23>
>
> apache/rocketmq<https://github.com/apache/rocketmq/
> blob/master/client/src/main/java/org/apache/rocketmq/client/producer/
> TransactionMQProducer.java#L23>
> rocketmq - Mirror of Apache RocketMQ
> github.com
>
> and TransactionCheckListener
> https://github.com/apache/rocketmq/blob/master/client/
> src/main/java/org/apache/rocketmq/client/producer/
> TransactionCheckListener.java
>
> So, what is the part I should implement for this issue?
>
> Regards,
> Dexin Li
>
>
Re: 答复: 答复: 答复: [GSOC|ROCKETMQ-123]Support Distributed Transactional Messaging
Posted by yukon <yu...@apache.org>.
+ riven
Hi Dexin,
Riven is a very senior engineer in RocketMQ community, especially in
distributed transaction area. He is enthusiastic and will help you finished
the design.
Regards,
yukon
On Fri, Mar 16, 2018 at 10:13 PM, 李 德鑫 <de...@outlook.com> wrote:
> 😓Sorry, I forgot to add the link
>
> https://docs.google.com/document/d/1HGQYPc3YtPga-jATx6whMYS7e11NQ6_
> r3tzC3yl4ulI/edit?usp=sharing
>
> ________________________________
> 发件人: 李 德鑫 <de...@outlook.com>
> 发送时间: 2018年3月16日 14:11:38
> 收件人: dev@rocketmq.apache.org
> 主题: 答复: 答复: 答复: [GSOC|ROCKETMQ-123]Support Distributed Transactional
> Messaging
>
> Hi yukon
>
> I've wrote a design doc about this topic, please have a look.
>
>
>
> Regards,
> Dexin
>
> ________________________________
> 发件人: yukon <yu...@apache.org>
> 发送时间: 2018年3月9日 15:11:01
> 收件人: dev
> 主题: Re: 答复: 答复: [GSOC|ROCKETMQ-123]Support Distributed Transactional
> Messaging
>
> Hi Dexin,
>
> ```
> 1. Does the transactional messaging need to be "exactly only once"
> delivered?
> ```
>
> Please check out https://issues.apache.org/jira/browse/ROCKETMQ-124
>
> ```
> 2. Where to store the transaction state? Since the state must be
> persistent and also updated.
> ```
>
> We could reuse the rocketmq-store model to store the transaction state.
>
> And, don't worry about the dirty pages if you follow the rule: Append Only.
>
> Regards,
> yukon
>
>
>
> On Wed, Mar 7, 2018 at 12:11 PM, 李 德鑫 <de...@outlook.com> wrote:
>
> > Thank you, Yukon!
> >
> >
> > I still have some question about transactional messaging
> >
> > 1. Does the transactional messaging need to be "exactly only once"
> > delivered?
> > 2. Where to store the transaction state? Since the state must be
> > persistent and also updated.
> >
> >
> > About the second question, I've considered about some methods
> >
> > 1. Inside of the message. But it is said that this would generate lots
> > of dirty pages on the OS.
> > 2. In a database, especially an SQL database. Since class
> > JDBCTransactionStore already exists. But I want to know whether the DB
> will
> > be the bottleneck of performance or scalability.
> > 3. Like Kafka, use an internal topic called Transaction log. It will
> > record all the operation about transaction states. And store the latest
> > states in the memory. If a partition crashed, It will read the log to
> > recover. But the recovery would be slow.
> > 4. Similar to 3, store and the transaction state in the Transaction
> > log. But if there's a lot of transaction message, (I guess)this would be
> > like 1 to generate many dirty pages.
> >
> >
> > Regards,
> > Dexin
> > ________________________________
> > 发件人: yukon <yu...@apache.org>
> > 发送时间: 2018年3月5日 12:28:47
> > 收件人: dev
> > 主题: Re: 答复: [GSOC|ROCKETMQ-123]Support Distributed Transactional
> Messaging
> >
> > Hi,
> >
> > The links you mentioned just are interfaces, we should implement these
> > APIs.
> >
> > Regards,
> > yukon
> >
> > On Tue, Feb 27, 2018 at 12:07 PM, 李 德鑫 <de...@outlook.com> wrote:
> >
> > > My previous email didn't clarify the question clearly, so I'll try to
> > make
> > > it in detail.
> > >
> > >
> > > In this jira's(https://issues.apache.org/jira/browse/ROCKETMQ-123)
> > > description:
> > >
> > >
> > >
> > >
> > > Implement a TransactionProducer and LocalTransactionChecker, guarantee
> > > message delivery and local transaction operations are atomic.
> > >
> > > The following is the simple transactional messaging flow:
> > >
> > > 1. TransactionProducer sends a half message to the broker. A half
> message
> > > means it's not confirmed and can't be delivered to the consumer.
> > >
> > > 2. TransactionProducer executes the specific local transaction.
> > >
> > > 3. Commit or Rollback the half message according to the status of the
> > > local transaction. A half message will be deleted if it's rollbacked
> and
> > > delivered to the consumer if it's committed.
> > >
> > >
> > >
> > > The 1st step I found in https://github.com/apache/
> > > rocketmq/blob/master/client/src/main/java/org/apache/
> > > rocketmq/client/impl/producer/DefaultMQProducerImpl.java#L964
> > >
> > > The 2nd step I found in https://github.com/apache/
> > > rocketmq/blob/master/client/src/main/java/org/apache/
> > > rocketmq/client/impl/producer/DefaultMQProducerImpl.java#L977
> > >
> > > The 3rd step I found in https://github.com/apache/
> > > rocketmq/blob/master/client/src/main/java/org/apache/
> > > rocketmq/client/impl/producer/DefaultMQProducerImpl.java#L1026 for
> > client
> > > and https://github.com/apache/rocketmq/blob/master/broker/
> > > src/main/java/org/apache/rocketmq/broker/processor/
> > > EndTransactionProcessor.java#L49 for broker
> > >
> > >
> > > And my question is what is the part to be implemented for this issue?
> Is
> > > there anything I missed?
> > >
> > >
> > >
> > > ________________________________
> > > 发件人: 李 德鑫 <De...@outlook.com>
> > > 发送时间: 2018年2月23日 9:35:17
> > > 收件人: dev@rocketmq.apache.org
> > > 主题: [GSOC|ROCKETMQ-123]Support Distributed Transactional Messaging
> > >
> > > Hi,
> > >
> > > I am working on this issue
> > > (https://issues.apache.org/jira/browse/ROCKETMQ-123) as my GSOC
> project
> > > [ROCKETMQ-123] Support Distributed Transactional Messaging ...<
> > > https://issues.apache.org/jira/browse/ROCKETMQ-123>
> > > In most businesses, transactional messaging is frequently used to make
> a
> > > series of operations as an atomic transaction. At present, RocketMQ
> > doesn't
> > > support ...
> > > issues.apache.org
> > >
> > >
> > >
> > > However, there already have
> > > TransactionMQProducer
> > >
> > > https://github.com/apache/rocketmq/blob/master/client/
> > > src/main/java/org/apache/rocketmq/client/producer/
> > > TransactionMQProducer.java<https://github.com/apache/
> > > rocketmq/blob/master/client/src/main/java/org/apache/
> > > rocketmq/client/producer/TransactionMQProducer.java#L23>
> > > [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
> > > https://github.com/apache/rocketmq/blob/master/client/
> src/main/java/org/
> > > apache/rocketmq/client/producer/TransactionMQProducer.java#L23>
> > >
> > > apache/rocketmq<https://github.com/apache/rocketmq/
> > > blob/master/client/src/main/java/org/apache/rocketmq/client/producer/
> > > TransactionMQProducer.java#L23>
> > > rocketmq - Mirror of Apache RocketMQ
> > > github.com
> > >
> > > and TransactionCheckListener
> > > https://github.com/apache/rocketmq/blob/master/client/
> > > src/main/java/org/apache/rocketmq/client/producer/
> > > TransactionCheckListener.java
> > >
> > > So, what is the part I should implement for this issue?
> > >
> > > Regards,
> > > Dexin Li
> > >
> > >
> >
>
答复: 答复: 答复: [GSOC|ROCKETMQ-123]Support Distributed Transactional Messaging
Posted by 李 德鑫 <de...@outlook.com>.
😓Sorry, I forgot to add the link
https://docs.google.com/document/d/1HGQYPc3YtPga-jATx6whMYS7e11NQ6_r3tzC3yl4ulI/edit?usp=sharing
________________________________
发件人: 李 德鑫 <de...@outlook.com>
发送时间: 2018年3月16日 14:11:38
收件人: dev@rocketmq.apache.org
主题: 答复: 答复: 答复: [GSOC|ROCKETMQ-123]Support Distributed Transactional Messaging
Hi yukon
I've wrote a design doc about this topic, please have a look.
Regards,
Dexin
________________________________
发件人: yukon <yu...@apache.org>
发送时间: 2018年3月9日 15:11:01
收件人: dev
主题: Re: 答复: 答复: [GSOC|ROCKETMQ-123]Support Distributed Transactional Messaging
Hi Dexin,
```
1. Does the transactional messaging need to be "exactly only once"
delivered?
```
Please check out https://issues.apache.org/jira/browse/ROCKETMQ-124
```
2. Where to store the transaction state? Since the state must be
persistent and also updated.
```
We could reuse the rocketmq-store model to store the transaction state.
And, don't worry about the dirty pages if you follow the rule: Append Only.
Regards,
yukon
On Wed, Mar 7, 2018 at 12:11 PM, 李 德鑫 <de...@outlook.com> wrote:
> Thank you, Yukon!
>
>
> I still have some question about transactional messaging
>
> 1. Does the transactional messaging need to be "exactly only once"
> delivered?
> 2. Where to store the transaction state? Since the state must be
> persistent and also updated.
>
>
> About the second question, I've considered about some methods
>
> 1. Inside of the message. But it is said that this would generate lots
> of dirty pages on the OS.
> 2. In a database, especially an SQL database. Since class
> JDBCTransactionStore already exists. But I want to know whether the DB will
> be the bottleneck of performance or scalability.
> 3. Like Kafka, use an internal topic called Transaction log. It will
> record all the operation about transaction states. And store the latest
> states in the memory. If a partition crashed, It will read the log to
> recover. But the recovery would be slow.
> 4. Similar to 3, store and the transaction state in the Transaction
> log. But if there's a lot of transaction message, (I guess)this would be
> like 1 to generate many dirty pages.
>
>
> Regards,
> Dexin
> ________________________________
> 发件人: yukon <yu...@apache.org>
> 发送时间: 2018年3月5日 12:28:47
> 收件人: dev
> 主题: Re: 答复: [GSOC|ROCKETMQ-123]Support Distributed Transactional Messaging
>
> Hi,
>
> The links you mentioned just are interfaces, we should implement these
> APIs.
>
> Regards,
> yukon
>
> On Tue, Feb 27, 2018 at 12:07 PM, 李 德鑫 <de...@outlook.com> wrote:
>
> > My previous email didn't clarify the question clearly, so I'll try to
> make
> > it in detail.
> >
> >
> > In this jira's(https://issues.apache.org/jira/browse/ROCKETMQ-123)
> > description:
> >
> >
> >
> >
> > Implement a TransactionProducer and LocalTransactionChecker, guarantee
> > message delivery and local transaction operations are atomic.
> >
> > The following is the simple transactional messaging flow:
> >
> > 1. TransactionProducer sends a half message to the broker. A half message
> > means it's not confirmed and can't be delivered to the consumer.
> >
> > 2. TransactionProducer executes the specific local transaction.
> >
> > 3. Commit or Rollback the half message according to the status of the
> > local transaction. A half message will be deleted if it's rollbacked and
> > delivered to the consumer if it's committed.
> >
> >
> >
> > The 1st step I found in https://github.com/apache/
> > rocketmq/blob/master/client/src/main/java/org/apache/
> > rocketmq/client/impl/producer/DefaultMQProducerImpl.java#L964
> >
> > The 2nd step I found in https://github.com/apache/
> > rocketmq/blob/master/client/src/main/java/org/apache/
> > rocketmq/client/impl/producer/DefaultMQProducerImpl.java#L977
> >
> > The 3rd step I found in https://github.com/apache/
> > rocketmq/blob/master/client/src/main/java/org/apache/
> > rocketmq/client/impl/producer/DefaultMQProducerImpl.java#L1026 for
> client
> > and https://github.com/apache/rocketmq/blob/master/broker/
> > src/main/java/org/apache/rocketmq/broker/processor/
> > EndTransactionProcessor.java#L49 for broker
> >
> >
> > And my question is what is the part to be implemented for this issue? Is
> > there anything I missed?
> >
> >
> >
> > ________________________________
> > 发件人: 李 德鑫 <De...@outlook.com>
> > 发送时间: 2018年2月23日 9:35:17
> > 收件人: dev@rocketmq.apache.org
> > 主题: [GSOC|ROCKETMQ-123]Support Distributed Transactional Messaging
> >
> > Hi,
> >
> > I am working on this issue
> > (https://issues.apache.org/jira/browse/ROCKETMQ-123) as my GSOC project
> > [ROCKETMQ-123] Support Distributed Transactional Messaging ...<
> > https://issues.apache.org/jira/browse/ROCKETMQ-123>
> > In most businesses, transactional messaging is frequently used to make a
> > series of operations as an atomic transaction. At present, RocketMQ
> doesn't
> > support ...
> > issues.apache.org
> >
> >
> >
> > However, there already have
> > TransactionMQProducer
> >
> > https://github.com/apache/rocketmq/blob/master/client/
> > src/main/java/org/apache/rocketmq/client/producer/
> > TransactionMQProducer.java<https://github.com/apache/
> > rocketmq/blob/master/client/src/main/java/org/apache/
> > rocketmq/client/producer/TransactionMQProducer.java#L23>
> > [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
> > https://github.com/apache/rocketmq/blob/master/client/src/main/java/org/
> > apache/rocketmq/client/producer/TransactionMQProducer.java#L23>
> >
> > apache/rocketmq<https://github.com/apache/rocketmq/
> > blob/master/client/src/main/java/org/apache/rocketmq/client/producer/
> > TransactionMQProducer.java#L23>
> > rocketmq - Mirror of Apache RocketMQ
> > github.com
> >
> > and TransactionCheckListener
> > https://github.com/apache/rocketmq/blob/master/client/
> > src/main/java/org/apache/rocketmq/client/producer/
> > TransactionCheckListener.java
> >
> > So, what is the part I should implement for this issue?
> >
> > Regards,
> > Dexin Li
> >
> >
>
答复: 答复: 答复: [GSOC|ROCKETMQ-123]Support Distributed Transactional Messaging
Posted by 李 德鑫 <de...@outlook.com>.
Hi yukon
I've wrote a design doc about this topic, please have a look.
Regards,
Dexin
________________________________
发件人: yukon <yu...@apache.org>
发送时间: 2018年3月9日 15:11:01
收件人: dev
主题: Re: 答复: 答复: [GSOC|ROCKETMQ-123]Support Distributed Transactional Messaging
Hi Dexin,
```
1. Does the transactional messaging need to be "exactly only once"
delivered?
```
Please check out https://issues.apache.org/jira/browse/ROCKETMQ-124
```
2. Where to store the transaction state? Since the state must be
persistent and also updated.
```
We could reuse the rocketmq-store model to store the transaction state.
And, don't worry about the dirty pages if you follow the rule: Append Only.
Regards,
yukon
On Wed, Mar 7, 2018 at 12:11 PM, 李 德鑫 <de...@outlook.com> wrote:
> Thank you, Yukon!
>
>
> I still have some question about transactional messaging
>
> 1. Does the transactional messaging need to be "exactly only once"
> delivered?
> 2. Where to store the transaction state? Since the state must be
> persistent and also updated.
>
>
> About the second question, I've considered about some methods
>
> 1. Inside of the message. But it is said that this would generate lots
> of dirty pages on the OS.
> 2. In a database, especially an SQL database. Since class
> JDBCTransactionStore already exists. But I want to know whether the DB will
> be the bottleneck of performance or scalability.
> 3. Like Kafka, use an internal topic called Transaction log. It will
> record all the operation about transaction states. And store the latest
> states in the memory. If a partition crashed, It will read the log to
> recover. But the recovery would be slow.
> 4. Similar to 3, store and the transaction state in the Transaction
> log. But if there's a lot of transaction message, (I guess)this would be
> like 1 to generate many dirty pages.
>
>
> Regards,
> Dexin
> ________________________________
> 发件人: yukon <yu...@apache.org>
> 发送时间: 2018年3月5日 12:28:47
> 收件人: dev
> 主题: Re: 答复: [GSOC|ROCKETMQ-123]Support Distributed Transactional Messaging
>
> Hi,
>
> The links you mentioned just are interfaces, we should implement these
> APIs.
>
> Regards,
> yukon
>
> On Tue, Feb 27, 2018 at 12:07 PM, 李 德鑫 <de...@outlook.com> wrote:
>
> > My previous email didn't clarify the question clearly, so I'll try to
> make
> > it in detail.
> >
> >
> > In this jira's(https://issues.apache.org/jira/browse/ROCKETMQ-123)
> > description:
> >
> >
> >
> >
> > Implement a TransactionProducer and LocalTransactionChecker, guarantee
> > message delivery and local transaction operations are atomic.
> >
> > The following is the simple transactional messaging flow:
> >
> > 1. TransactionProducer sends a half message to the broker. A half message
> > means it's not confirmed and can't be delivered to the consumer.
> >
> > 2. TransactionProducer executes the specific local transaction.
> >
> > 3. Commit or Rollback the half message according to the status of the
> > local transaction. A half message will be deleted if it's rollbacked and
> > delivered to the consumer if it's committed.
> >
> >
> >
> > The 1st step I found in https://github.com/apache/
> > rocketmq/blob/master/client/src/main/java/org/apache/
> > rocketmq/client/impl/producer/DefaultMQProducerImpl.java#L964
> >
> > The 2nd step I found in https://github.com/apache/
> > rocketmq/blob/master/client/src/main/java/org/apache/
> > rocketmq/client/impl/producer/DefaultMQProducerImpl.java#L977
> >
> > The 3rd step I found in https://github.com/apache/
> > rocketmq/blob/master/client/src/main/java/org/apache/
> > rocketmq/client/impl/producer/DefaultMQProducerImpl.java#L1026 for
> client
> > and https://github.com/apache/rocketmq/blob/master/broker/
> > src/main/java/org/apache/rocketmq/broker/processor/
> > EndTransactionProcessor.java#L49 for broker
> >
> >
> > And my question is what is the part to be implemented for this issue? Is
> > there anything I missed?
> >
> >
> >
> > ________________________________
> > 发件人: 李 德鑫 <De...@outlook.com>
> > 发送时间: 2018年2月23日 9:35:17
> > 收件人: dev@rocketmq.apache.org
> > 主题: [GSOC|ROCKETMQ-123]Support Distributed Transactional Messaging
> >
> > Hi,
> >
> > I am working on this issue
> > (https://issues.apache.org/jira/browse/ROCKETMQ-123) as my GSOC project
> > [ROCKETMQ-123] Support Distributed Transactional Messaging ...<
> > https://issues.apache.org/jira/browse/ROCKETMQ-123>
> > In most businesses, transactional messaging is frequently used to make a
> > series of operations as an atomic transaction. At present, RocketMQ
> doesn't
> > support ...
> > issues.apache.org
> >
> >
> >
> > However, there already have
> > TransactionMQProducer
> >
> > https://github.com/apache/rocketmq/blob/master/client/
> > src/main/java/org/apache/rocketmq/client/producer/
> > TransactionMQProducer.java<https://github.com/apache/
> > rocketmq/blob/master/client/src/main/java/org/apache/
> > rocketmq/client/producer/TransactionMQProducer.java#L23>
> > [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
> > https://github.com/apache/rocketmq/blob/master/client/src/main/java/org/
> > apache/rocketmq/client/producer/TransactionMQProducer.java#L23>
> >
> > apache/rocketmq<https://github.com/apache/rocketmq/
> > blob/master/client/src/main/java/org/apache/rocketmq/client/producer/
> > TransactionMQProducer.java#L23>
> > rocketmq - Mirror of Apache RocketMQ
> > github.com
> >
> > and TransactionCheckListener
> > https://github.com/apache/rocketmq/blob/master/client/
> > src/main/java/org/apache/rocketmq/client/producer/
> > TransactionCheckListener.java
> >
> > So, what is the part I should implement for this issue?
> >
> > Regards,
> > Dexin Li
> >
> >
>
Re: 答复: 答复: [GSOC|ROCKETMQ-123]Support Distributed Transactional Messaging
Posted by yukon <yu...@apache.org>.
Hi Dexin,
```
1. Does the transactional messaging need to be "exactly only once"
delivered?
```
Please check out https://issues.apache.org/jira/browse/ROCKETMQ-124
```
2. Where to store the transaction state? Since the state must be
persistent and also updated.
```
We could reuse the rocketmq-store model to store the transaction state.
And, don't worry about the dirty pages if you follow the rule: Append Only.
Regards,
yukon
On Wed, Mar 7, 2018 at 12:11 PM, 李 德鑫 <de...@outlook.com> wrote:
> Thank you, Yukon!
>
>
> I still have some question about transactional messaging
>
> 1. Does the transactional messaging need to be "exactly only once"
> delivered?
> 2. Where to store the transaction state? Since the state must be
> persistent and also updated.
>
>
> About the second question, I've considered about some methods
>
> 1. Inside of the message. But it is said that this would generate lots
> of dirty pages on the OS.
> 2. In a database, especially an SQL database. Since class
> JDBCTransactionStore already exists. But I want to know whether the DB will
> be the bottleneck of performance or scalability.
> 3. Like Kafka, use an internal topic called Transaction log. It will
> record all the operation about transaction states. And store the latest
> states in the memory. If a partition crashed, It will read the log to
> recover. But the recovery would be slow.
> 4. Similar to 3, store and the transaction state in the Transaction
> log. But if there's a lot of transaction message, (I guess)this would be
> like 1 to generate many dirty pages.
>
>
> Regards,
> Dexin
> ________________________________
> 发件人: yukon <yu...@apache.org>
> 发送时间: 2018年3月5日 12:28:47
> 收件人: dev
> 主题: Re: 答复: [GSOC|ROCKETMQ-123]Support Distributed Transactional Messaging
>
> Hi,
>
> The links you mentioned just are interfaces, we should implement these
> APIs.
>
> Regards,
> yukon
>
> On Tue, Feb 27, 2018 at 12:07 PM, 李 德鑫 <de...@outlook.com> wrote:
>
> > My previous email didn't clarify the question clearly, so I'll try to
> make
> > it in detail.
> >
> >
> > In this jira's(https://issues.apache.org/jira/browse/ROCKETMQ-123)
> > description:
> >
> >
> >
> >
> > Implement a TransactionProducer and LocalTransactionChecker, guarantee
> > message delivery and local transaction operations are atomic.
> >
> > The following is the simple transactional messaging flow:
> >
> > 1. TransactionProducer sends a half message to the broker. A half message
> > means it's not confirmed and can't be delivered to the consumer.
> >
> > 2. TransactionProducer executes the specific local transaction.
> >
> > 3. Commit or Rollback the half message according to the status of the
> > local transaction. A half message will be deleted if it's rollbacked and
> > delivered to the consumer if it's committed.
> >
> >
> >
> > The 1st step I found in https://github.com/apache/
> > rocketmq/blob/master/client/src/main/java/org/apache/
> > rocketmq/client/impl/producer/DefaultMQProducerImpl.java#L964
> >
> > The 2nd step I found in https://github.com/apache/
> > rocketmq/blob/master/client/src/main/java/org/apache/
> > rocketmq/client/impl/producer/DefaultMQProducerImpl.java#L977
> >
> > The 3rd step I found in https://github.com/apache/
> > rocketmq/blob/master/client/src/main/java/org/apache/
> > rocketmq/client/impl/producer/DefaultMQProducerImpl.java#L1026 for
> client
> > and https://github.com/apache/rocketmq/blob/master/broker/
> > src/main/java/org/apache/rocketmq/broker/processor/
> > EndTransactionProcessor.java#L49 for broker
> >
> >
> > And my question is what is the part to be implemented for this issue? Is
> > there anything I missed?
> >
> >
> >
> > ________________________________
> > 发件人: 李 德鑫 <De...@outlook.com>
> > 发送时间: 2018年2月23日 9:35:17
> > 收件人: dev@rocketmq.apache.org
> > 主题: [GSOC|ROCKETMQ-123]Support Distributed Transactional Messaging
> >
> > Hi,
> >
> > I am working on this issue
> > (https://issues.apache.org/jira/browse/ROCKETMQ-123) as my GSOC project
> > [ROCKETMQ-123] Support Distributed Transactional Messaging ...<
> > https://issues.apache.org/jira/browse/ROCKETMQ-123>
> > In most businesses, transactional messaging is frequently used to make a
> > series of operations as an atomic transaction. At present, RocketMQ
> doesn't
> > support ...
> > issues.apache.org
> >
> >
> >
> > However, there already have
> > TransactionMQProducer
> >
> > https://github.com/apache/rocketmq/blob/master/client/
> > src/main/java/org/apache/rocketmq/client/producer/
> > TransactionMQProducer.java<https://github.com/apache/
> > rocketmq/blob/master/client/src/main/java/org/apache/
> > rocketmq/client/producer/TransactionMQProducer.java#L23>
> > [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
> > https://github.com/apache/rocketmq/blob/master/client/src/main/java/org/
> > apache/rocketmq/client/producer/TransactionMQProducer.java#L23>
> >
> > apache/rocketmq<https://github.com/apache/rocketmq/
> > blob/master/client/src/main/java/org/apache/rocketmq/client/producer/
> > TransactionMQProducer.java#L23>
> > rocketmq - Mirror of Apache RocketMQ
> > github.com
> >
> > and TransactionCheckListener
> > https://github.com/apache/rocketmq/blob/master/client/
> > src/main/java/org/apache/rocketmq/client/producer/
> > TransactionCheckListener.java
> >
> > So, what is the part I should implement for this issue?
> >
> > Regards,
> > Dexin Li
> >
> >
>
答复: 答复: [GSOC|ROCKETMQ-123]Support Distributed Transactional Messaging
Posted by 李 德鑫 <de...@outlook.com>.
Thank you, Yukon!
I still have some question about transactional messaging
1. Does the transactional messaging need to be "exactly only once" delivered?
2. Where to store the transaction state? Since the state must be persistent and also updated.
About the second question, I've considered about some methods
1. Inside of the message. But it is said that this would generate lots of dirty pages on the OS.
2. In a database, especially an SQL database. Since class JDBCTransactionStore already exists. But I want to know whether the DB will be the bottleneck of performance or scalability.
3. Like Kafka, use an internal topic called Transaction log. It will record all the operation about transaction states. And store the latest states in the memory. If a partition crashed, It will read the log to recover. But the recovery would be slow.
4. Similar to 3, store and the transaction state in the Transaction log. But if there's a lot of transaction message, (I guess)this would be like 1 to generate many dirty pages.
Regards,
Dexin
________________________________
发件人: yukon <yu...@apache.org>
发送时间: 2018年3月5日 12:28:47
收件人: dev
主题: Re: 答复: [GSOC|ROCKETMQ-123]Support Distributed Transactional Messaging
Hi,
The links you mentioned just are interfaces, we should implement these APIs.
Regards,
yukon
On Tue, Feb 27, 2018 at 12:07 PM, 李 德鑫 <de...@outlook.com> wrote:
> My previous email didn't clarify the question clearly, so I'll try to make
> it in detail.
>
>
> In this jira's(https://issues.apache.org/jira/browse/ROCKETMQ-123)
> description:
>
>
>
>
> Implement a TransactionProducer and LocalTransactionChecker, guarantee
> message delivery and local transaction operations are atomic.
>
> The following is the simple transactional messaging flow:
>
> 1. TransactionProducer sends a half message to the broker. A half message
> means it's not confirmed and can't be delivered to the consumer.
>
> 2. TransactionProducer executes the specific local transaction.
>
> 3. Commit or Rollback the half message according to the status of the
> local transaction. A half message will be deleted if it's rollbacked and
> delivered to the consumer if it's committed.
>
>
>
> The 1st step I found in https://github.com/apache/
> rocketmq/blob/master/client/src/main/java/org/apache/
> rocketmq/client/impl/producer/DefaultMQProducerImpl.java#L964
>
> The 2nd step I found in https://github.com/apache/
> rocketmq/blob/master/client/src/main/java/org/apache/
> rocketmq/client/impl/producer/DefaultMQProducerImpl.java#L977
>
> The 3rd step I found in https://github.com/apache/
> rocketmq/blob/master/client/src/main/java/org/apache/
> rocketmq/client/impl/producer/DefaultMQProducerImpl.java#L1026 for client
> and https://github.com/apache/rocketmq/blob/master/broker/
> src/main/java/org/apache/rocketmq/broker/processor/
> EndTransactionProcessor.java#L49 for broker
>
>
> And my question is what is the part to be implemented for this issue? Is
> there anything I missed?
>
>
>
> ________________________________
> 发件人: 李 德鑫 <De...@outlook.com>
> 发送时间: 2018年2月23日 9:35:17
> 收件人: dev@rocketmq.apache.org
> 主题: [GSOC|ROCKETMQ-123]Support Distributed Transactional Messaging
>
> Hi,
>
> I am working on this issue
> (https://issues.apache.org/jira/browse/ROCKETMQ-123) as my GSOC project
> [ROCKETMQ-123] Support Distributed Transactional Messaging ...<
> https://issues.apache.org/jira/browse/ROCKETMQ-123>
> In most businesses, transactional messaging is frequently used to make a
> series of operations as an atomic transaction. At present, RocketMQ doesn't
> support ...
> issues.apache.org
>
>
>
> However, there already have
> TransactionMQProducer
>
> https://github.com/apache/rocketmq/blob/master/client/
> src/main/java/org/apache/rocketmq/client/producer/
> TransactionMQProducer.java<https://github.com/apache/
> rocketmq/blob/master/client/src/main/java/org/apache/
> rocketmq/client/producer/TransactionMQProducer.java#L23>
> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
> https://github.com/apache/rocketmq/blob/master/client/src/main/java/org/
> apache/rocketmq/client/producer/TransactionMQProducer.java#L23>
>
> apache/rocketmq<https://github.com/apache/rocketmq/
> blob/master/client/src/main/java/org/apache/rocketmq/client/producer/
> TransactionMQProducer.java#L23>
> rocketmq - Mirror of Apache RocketMQ
> github.com
>
> and TransactionCheckListener
> https://github.com/apache/rocketmq/blob/master/client/
> src/main/java/org/apache/rocketmq/client/producer/
> TransactionCheckListener.java
>
> So, what is the part I should implement for this issue?
>
> Regards,
> Dexin Li
>
>