You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Anindya Haldar <an...@oracle.com> on 2019/11/13 18:52:21 UTC
producer transaction and threading model
The producer API doc (https://kafka.apache.org/23/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html <https://kafka.apache.org/23/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html>) says that KafkaProducer is thread safe.
What I was trying to figure out is what the relationship is between a thread context and a transaction initiated in that thread. When multiple threads are using a single producer instance and initiating transactions are those transaction boundaries scoped to the thread context, or do they step on each others’ toes?
Suppose two threads are sharing the same producer instance the following way:
thread-1 begins a transaction T1
thread-1 starts sending messages in T!
thread-2 begins a transaction T2
thread-2 sends some messages in T2
thread-2 gets and error and rolls back its transaction T2
At this point does the rollback of transaction T2 by the thread-2 affect the transaction T1 remaining open and used by thread-1 in any fashion? It is not clear to me from the API documentation. Will very much appreciate some insights into the behavior here. Similar questions also arise for commits by threads independently of each other.
Sincerely,
Anindya Haldar
Oracle Responsys
Re: producer transaction and threading model
Posted by Edward Capriolo <ed...@gmail.com>.
Spring framework works around this by providing a method like
doInTransaction( listofstuff).
Behind the scenes it manages a pool of transactional producers with ids
like transaction_prefix + id.
So each call to dointransaction may initiate a transaction.
On Friday, November 15, 2019, Matthias J. Sax <ma...@confluent.io> wrote:
> That's correct. And each would need to use a different `transactional.id`.
>
>
> -Matthias
>
> On 11/14/19 11:17 AM, Anindya Haldar wrote:
> > Thanks for the information. Does that mean that each producer thread, in
> case it wants to have its own transactions, should use its own instance of
> KafkaProducer?
> >
> > Sincerely,
> > Anindya Haldar
> > Oracle Responsys
> >
> >
> >> On Nov 13, 2019, at 11:31 PM, Matthias J. Sax <ma...@confluent.io>
> wrote:
> >>
> >> That is not possible. A producer can only have a single open
> >> transaction. If your example, the call of thread-2 to start a new
> >> transactions would fail and an exception would be thrown.
> >>
> >> -Matthias
> >>
> >> On 11/13/19 10:52 AM, Anindya Haldar wrote:
> >>> The producer API doc (https://kafka.apache.org/23/
> javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html <
> https://kafka.apache.org/23/javadoc/index.html?org/apache/
> kafka/clients/producer/KafkaProducer.html>) says that KafkaProducer is
> thread safe.
> >>>
> >>> What I was trying to figure out is what the relationship is between a
> thread context and a transaction initiated in that thread. When multiple
> threads are using a single producer instance and initiating transactions
> are those transaction boundaries scoped to the thread context, or do they
> step on each others’ toes?
> >>>
> >>> Suppose two threads are sharing the same producer instance the
> following way:
> >>>
> >>> thread-1 begins a transaction T1
> >>> thread-1 starts sending messages in T!
> >>> thread-2 begins a transaction T2
> >>> thread-2 sends some messages in T2
> >>> thread-2 gets and error and rolls back its transaction T2
> >>>
> >>> At this point does the rollback of transaction T2 by the thread-2
> affect the transaction T1 remaining open and used by thread-1 in any
> fashion? It is not clear to me from the API documentation. Will very much
> appreciate some insights into the behavior here. Similar questions also
> arise for commits by threads independently of each other.
> >>>
> >>> Sincerely,
> >>> Anindya Haldar
> >>> Oracle Responsys
> >>>
> >>>
> >>
> >
>
>
--
Sorry this was sent from mobile. Will do less grammar and spell check than
usual.
Re: producer transaction and threading model
Posted by "Matthias J. Sax" <ma...@confluent.io>.
That's correct. And each would need to use a different `transactional.id`.
-Matthias
On 11/14/19 11:17 AM, Anindya Haldar wrote:
> Thanks for the information. Does that mean that each producer thread, in case it wants to have its own transactions, should use its own instance of KafkaProducer?
>
> Sincerely,
> Anindya Haldar
> Oracle Responsys
>
>
>> On Nov 13, 2019, at 11:31 PM, Matthias J. Sax <ma...@confluent.io> wrote:
>>
>> That is not possible. A producer can only have a single open
>> transaction. If your example, the call of thread-2 to start a new
>> transactions would fail and an exception would be thrown.
>>
>> -Matthias
>>
>> On 11/13/19 10:52 AM, Anindya Haldar wrote:
>>> The producer API doc (https://kafka.apache.org/23/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html <https://kafka.apache.org/23/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html>) says that KafkaProducer is thread safe.
>>>
>>> What I was trying to figure out is what the relationship is between a thread context and a transaction initiated in that thread. When multiple threads are using a single producer instance and initiating transactions are those transaction boundaries scoped to the thread context, or do they step on each others’ toes?
>>>
>>> Suppose two threads are sharing the same producer instance the following way:
>>>
>>> thread-1 begins a transaction T1
>>> thread-1 starts sending messages in T!
>>> thread-2 begins a transaction T2
>>> thread-2 sends some messages in T2
>>> thread-2 gets and error and rolls back its transaction T2
>>>
>>> At this point does the rollback of transaction T2 by the thread-2 affect the transaction T1 remaining open and used by thread-1 in any fashion? It is not clear to me from the API documentation. Will very much appreciate some insights into the behavior here. Similar questions also arise for commits by threads independently of each other.
>>>
>>> Sincerely,
>>> Anindya Haldar
>>> Oracle Responsys
>>>
>>>
>>
>
Re: producer transaction and threading model
Posted by Anindya Haldar <an...@oracle.com>.
Thanks for the information. Does that mean that each producer thread, in case it wants to have its own transactions, should use its own instance of KafkaProducer?
Sincerely,
Anindya Haldar
Oracle Responsys
> On Nov 13, 2019, at 11:31 PM, Matthias J. Sax <ma...@confluent.io> wrote:
>
> That is not possible. A producer can only have a single open
> transaction. If your example, the call of thread-2 to start a new
> transactions would fail and an exception would be thrown.
>
> -Matthias
>
> On 11/13/19 10:52 AM, Anindya Haldar wrote:
>> The producer API doc (https://kafka.apache.org/23/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html <https://kafka.apache.org/23/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html>) says that KafkaProducer is thread safe.
>>
>> What I was trying to figure out is what the relationship is between a thread context and a transaction initiated in that thread. When multiple threads are using a single producer instance and initiating transactions are those transaction boundaries scoped to the thread context, or do they step on each others’ toes?
>>
>> Suppose two threads are sharing the same producer instance the following way:
>>
>> thread-1 begins a transaction T1
>> thread-1 starts sending messages in T!
>> thread-2 begins a transaction T2
>> thread-2 sends some messages in T2
>> thread-2 gets and error and rolls back its transaction T2
>>
>> At this point does the rollback of transaction T2 by the thread-2 affect the transaction T1 remaining open and used by thread-1 in any fashion? It is not clear to me from the API documentation. Will very much appreciate some insights into the behavior here. Similar questions also arise for commits by threads independently of each other.
>>
>> Sincerely,
>> Anindya Haldar
>> Oracle Responsys
>>
>>
>
Re: producer transaction and threading model
Posted by "Matthias J. Sax" <ma...@confluent.io>.
That is not possible. A producer can only have a single open
transaction. If your example, the call of thread-2 to start a new
transactions would fail and an exception would be thrown.
-Matthias
On 11/13/19 10:52 AM, Anindya Haldar wrote:
> The producer API doc (https://kafka.apache.org/23/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html <https://kafka.apache.org/23/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html>) says that KafkaProducer is thread safe.
>
> What I was trying to figure out is what the relationship is between a thread context and a transaction initiated in that thread. When multiple threads are using a single producer instance and initiating transactions are those transaction boundaries scoped to the thread context, or do they step on each others’ toes?
>
> Suppose two threads are sharing the same producer instance the following way:
>
> thread-1 begins a transaction T1
> thread-1 starts sending messages in T!
> thread-2 begins a transaction T2
> thread-2 sends some messages in T2
> thread-2 gets and error and rolls back its transaction T2
>
> At this point does the rollback of transaction T2 by the thread-2 affect the transaction T1 remaining open and used by thread-1 in any fashion? It is not clear to me from the API documentation. Will very much appreciate some insights into the behavior here. Similar questions also arise for commits by threads independently of each other.
>
> Sincerely,
> Anindya Haldar
> Oracle Responsys
>
>