You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Bienek, Marcin" <mb...@team.mobile.de> on 2017/08/03 08:50:12 UTC
Kafka 0.11.0 problem with transactions.
Hi,
I’m trying to test the new exactly once transaction feature. Doing simple test like:
/opt/kafka/bin/kafka-console-producer.sh --request-required-acks "all" --producer-property "transactional.id=777" --producer-property="enable.idempotence=true" --broker-list broker1:9092 --topic bla
Fails with:
java.lang.IllegalStateException: Cannot perform a 'send' before completing a call to initTransactions when transactions are enabled.
at org.apache.kafka.clients.producer.internals.TransactionManager.failIfNotReadyForSend(TransactionManager.java:253)
at org.apache.kafka.clients.producer.internals.TransactionManager.maybeAddPartitionToTransaction(TransactionManager.java:233)
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:745)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:701)
at kafka.producer.NewShinyProducer.send(BaseProducer.scala:47)
at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:61)
at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)
I suspect that somehow the producer is not able to trigger the creation of the internal transaction topic ?
BR,
Marcin
Re: Kafka 0.11.0 problem with transactions.
Posted by "Bienek, Marcin" <mb...@team.mobile.de>.
Hi,
Many thanks for the explanation.
I just wanted to do some tests and look around, I think the perf tool will be OK for me.
Indeed, it’s a bit tricky with the console producer. To do a commit
after every new line (enter) sounds not right. Something like
begin; message, message… commit; would be better but how would that look like in a console producer ☺
or maybe simply a commit every n seconds.
BTW. Is there maybe any plan to implement the exactly once semantics in to the mirrormaker?
BR,
Marcin
On 03.08.17, 21:01, "Apurva Mehta" <ap...@confluent.io> wrote:
Ismael raises good questions about what transactions would mean for the
console producer.
However, the kafka-producer-perf-test script has transactions enabled. It
enables you to generate transactions of a certain duration (like 50ms,
100ms). It produces messages of specified size and commits them
transactionally in a periodic manner, enabling you to at least have a look
at the transaction log, etc.
Thanks,
Apurva
On Thu, Aug 3, 2017 at 6:22 AM, Ismael Juma <is...@juma.me.uk> wrote:
> Hi Marcin,
>
> The console producer hasn't been updated to invoke the appropriate methods
> if transactions are enabled. It also requires a bit of thinking on how it
> should work. Would there be a way to start and commit the transaction via
> the console or would the console producer do it periodically? What was your
> intent?
>
> Ismael
>
> On Thu, Aug 3, 2017 at 9:50 AM, Bienek, Marcin <mb...@team.mobile.de>
> wrote:
>
> > Hi,
> >
> > I’m trying to test the new exactly once transaction feature. Doing
> simple
> > test like:
> >
> > /opt/kafka/bin/kafka-console-producer.sh --request-required-acks "all"
> > --producer-property "transactional.id=777" --producer-property="enable.
> idempotence=true"
> > --broker-list broker1:9092 --topic bla
> >
> > Fails with:
> >
> > java.lang.IllegalStateException: Cannot perform a 'send' before
> > completing a call to initTransactions when transactions are enabled.
> > at org.apache.kafka.clients.producer.internals.
> TransactionManager.
> > failIfNotReadyForSend(TransactionManager.java:253)
> > at org.apache.kafka.clients.producer.internals.
> TransactionManager.
> > maybeAddPartitionToTransaction(TransactionManager.java:233)
> > at org.apache.kafka.clients.producer.KafkaProducer.doSend(
> > KafkaProducer.java:745)
> > at org.apache.kafka.clients.producer.KafkaProducer.send(
> > KafkaProducer.java:701)
> > at kafka.producer.NewShinyProducer.send(BaseProducer.scala:47)
> > at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:61)
> > at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)
> >
> > I suspect that somehow the producer is not able to trigger the creation
> of
> > the internal transaction topic ?
> >
> >
> > BR,
> > Marcin
> >
>
Re: Kafka 0.11.0 problem with transactions.
Posted by Apurva Mehta <ap...@confluent.io>.
Ismael raises good questions about what transactions would mean for the
console producer.
However, the kafka-producer-perf-test script has transactions enabled. It
enables you to generate transactions of a certain duration (like 50ms,
100ms). It produces messages of specified size and commits them
transactionally in a periodic manner, enabling you to at least have a look
at the transaction log, etc.
Thanks,
Apurva
On Thu, Aug 3, 2017 at 6:22 AM, Ismael Juma <is...@juma.me.uk> wrote:
> Hi Marcin,
>
> The console producer hasn't been updated to invoke the appropriate methods
> if transactions are enabled. It also requires a bit of thinking on how it
> should work. Would there be a way to start and commit the transaction via
> the console or would the console producer do it periodically? What was your
> intent?
>
> Ismael
>
> On Thu, Aug 3, 2017 at 9:50 AM, Bienek, Marcin <mb...@team.mobile.de>
> wrote:
>
> > Hi,
> >
> > I’m trying to test the new exactly once transaction feature. Doing
> simple
> > test like:
> >
> > /opt/kafka/bin/kafka-console-producer.sh --request-required-acks "all"
> > --producer-property "transactional.id=777" --producer-property="enable.
> idempotence=true"
> > --broker-list broker1:9092 --topic bla
> >
> > Fails with:
> >
> > java.lang.IllegalStateException: Cannot perform a 'send' before
> > completing a call to initTransactions when transactions are enabled.
> > at org.apache.kafka.clients.producer.internals.
> TransactionManager.
> > failIfNotReadyForSend(TransactionManager.java:253)
> > at org.apache.kafka.clients.producer.internals.
> TransactionManager.
> > maybeAddPartitionToTransaction(TransactionManager.java:233)
> > at org.apache.kafka.clients.producer.KafkaProducer.doSend(
> > KafkaProducer.java:745)
> > at org.apache.kafka.clients.producer.KafkaProducer.send(
> > KafkaProducer.java:701)
> > at kafka.producer.NewShinyProducer.send(BaseProducer.scala:47)
> > at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:61)
> > at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)
> >
> > I suspect that somehow the producer is not able to trigger the creation
> of
> > the internal transaction topic ?
> >
> >
> > BR,
> > Marcin
> >
>
Re: Kafka 0.11.0 problem with transactions.
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Marcin,
The console producer hasn't been updated to invoke the appropriate methods
if transactions are enabled. It also requires a bit of thinking on how it
should work. Would there be a way to start and commit the transaction via
the console or would the console producer do it periodically? What was your
intent?
Ismael
On Thu, Aug 3, 2017 at 9:50 AM, Bienek, Marcin <mb...@team.mobile.de>
wrote:
> Hi,
>
> I’m trying to test the new exactly once transaction feature. Doing simple
> test like:
>
> /opt/kafka/bin/kafka-console-producer.sh --request-required-acks "all"
> --producer-property "transactional.id=777" --producer-property="enable.idempotence=true"
> --broker-list broker1:9092 --topic bla
>
> Fails with:
>
> java.lang.IllegalStateException: Cannot perform a 'send' before
> completing a call to initTransactions when transactions are enabled.
> at org.apache.kafka.clients.producer.internals.TransactionManager.
> failIfNotReadyForSend(TransactionManager.java:253)
> at org.apache.kafka.clients.producer.internals.TransactionManager.
> maybeAddPartitionToTransaction(TransactionManager.java:233)
> at org.apache.kafka.clients.producer.KafkaProducer.doSend(
> KafkaProducer.java:745)
> at org.apache.kafka.clients.producer.KafkaProducer.send(
> KafkaProducer.java:701)
> at kafka.producer.NewShinyProducer.send(BaseProducer.scala:47)
> at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:61)
> at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)
>
> I suspect that somehow the producer is not able to trigger the creation of
> the internal transaction topic ?
>
>
> BR,
> Marcin
>