You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Anindya Haldar <an...@oracle.com> on 2019/10/25 21:51:34 UTC
Transaction error in Kafka producer perf test
We are evaluating Kafka for some of our use cases. As part of that effort I am trying to run an experiment with a cluster we have set up, and using the producer perf test tool supplied with the binaries.
Here’s the cluster info:
Runs in Kubernetes, with 4 CPUs, 32 GB RAM, 100 GB log space allocation for each node.
3 ZooKeeper nodes
5 Kafka nodes
Here is the topic description:
$ bin/kafka-topics.sh --describe --bootstrap-server localhost:9092 --topic testtopic3
Topic:testtopic3 PartitionCount:5 ReplicationFactor:3 Configs:min.insync.replicas=2,segment.bytes=1073741824,retention.ms <https://slack-redir.net/link?url=http%3A%2F%2Fretention.ms>=3600000,flush.messages=1,unclean.leader.election.enable=false
Topic: testtopic3 Partition: 0 Leader: 1 Replicas: 1,4,2 Isr: 1,4,2
Topic: testtopic3 Partition: 1 Leader: 4 Replicas: 4,2,3 Isr: 4,2,3
Topic: testtopic3 Partition: 2 Leader: 2 Replicas: 2,3,5 Isr: 2,3,5
Topic: testtopic3 Partition: 3 Leader: 3 Replicas: 3,5,1 Isr: 3,5,1
Topic: testtopic3 Partition: 4 Leader: 5 Replicas: 5,1,4 Isr: 5,1,4
And here is the producer test run command line and the result:
$ bin/kafka-producer-perf-test.sh --topic testtopic3 --num-records 1000000 --throughput -1 --record-size 256 --producer-props bootstrap.servers=kafka-a-0.ri:9092,kafka-b-0.ri:9092,kafka-c-0.ri:9092,kafka-d-0.ri:9092,kafka-e-0.ri:9092 acks=all batch.size=1 max.block.ms <https://slack-redir.net/link?url=http%3A%2F%2Fmax.block.ms>=3600000 enable.idempotence=true max.in.flight.requests.per.connection=1 retries=3 --transaction-duration-ms 3600000
4100 records sent, 819.7 records/sec (0.20 MB/sec), 2572.0 ms avg latency, 4892.0 ms max latency.
4261 records sent, 852.0 records/sec (0.21 MB/sec), 7397.2 ms avg latency, 9873.0 ms max latency.
4216 records sent, 843.0 records/sec (0.21 MB/sec), 12383.7 ms avg latency, 14849.0 ms max latency.
4400 records sent, 879.8 records/sec (0.21 MB/sec), 17332.0 ms avg latency, 19784.0 ms max latency.
4354 records sent, 870.8 records/sec (0.21 MB/sec), 22349.4 ms avg latency, 24763.0 ms max latency.
4477 records sent, 895.4 records/sec (0.22 MB/sec), 27241.1 ms avg latency, 29728.0 ms max latency.
4366 records sent, 873.2 records/sec (0.21 MB/sec), 32218.3 ms avg latency, 34703.0 ms max latency.
4408 records sent, 881.6 records/sec (0.22 MB/sec), 37190.6 ms avg latency, 39672.0 ms max latency.
4159 records sent, 831.5 records/sec (0.20 MB/sec), 42135.0 ms avg latency, 44640.0 ms max latency.
4260 records sent, 852.0 records/sec (0.21 MB/sec), 47098.0 ms avg latency, 49624.0 ms max latency.
4360 records sent, 872.0 records/sec (0.21 MB/sec), 52137.1 ms avg latency, 54574.0 ms max latency.
4514 records sent, 902.8 records/sec (0.22 MB/sec), 57038.1 ms avg latency, 59554.0 ms max latency.
4273 records sent, 854.3 records/sec (0.21 MB/sec), 62001.8 ms avg latency, 64524.0 ms max latency.
4348 records sent, 869.6 records/sec (0.21 MB/sec), 67037.8 ms avg latency, 69494.0 ms max latency.
4039 records sent, 807.5 records/sec (0.20 MB/sec), 72009.8 ms avg latency, 74481.0 ms max latency.
4327 records sent, 865.2 records/sec (0.21 MB/sec), 76993.8 ms avg latency, 79457.0 ms max latency.
4307 records sent, 861.4 records/sec (0.21 MB/sec), 82011.9 ms avg latency, 84449.0 ms max latency.
4506 records sent, 901.0 records/sec (0.22 MB/sec), 86922.6 ms avg latency, 89434.0 ms max latency.
4343 records sent, 868.6 records/sec (0.21 MB/sec), 91918.8 ms avg latency, 94394.0 ms max latency.
org.apache.kafka.common.errors.ProducerFencedException: Producer attempted an operation with an old epoch. Either there is a newer producer with the same transactionalId, or the producer's transaction has been expired by the broker.
Exception in thread "main" org.apache.kafka.common.KafkaException: Cannot perform send because at least one previous transactional or idempotent request has failed with errors.
at org.apache.kafka.clients.producer.internals.TransactionManager.failIfNotReadyForSend(TransactionManager.java:357)
at org.apache.kafka.clients.producer.internals.TransactionManager.maybeAddPartitionToTransaction(TransactionManager.java:341)
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:915)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:856)
at org.apache.kafka.tools.ProducerPerformance.main(ProducerPerformance.java:143)
Caused by: org.apache.kafka.common.errors.ProducerFencedException: Producer attempted an operation with an old epoch. Either there is a newer producer with the same transactionalId, or the producer's transaction has been expired by the broker.
[2019-10-25 21:15:05,183] ERROR [Producer clientId=producer-1, transactionalId=performance-producer-default-transactional-id] Aborting producer batches due to fatal error (org.apache.kafka.clients.producer.internals.Sender)
org.apache.kafka.common.errors.ProducerFencedException: Producer attempted an operation with an old epoch. Either there is a newer producer with the same transactionalId, or the producer's transaction has been expired by the broker.
It does not make any difference whether I add a transaction id flag like ‘—transactional-id mytxn’ to the command line.
Is there something I am missing here?
Sincerely,
Anindya Haldar
Oracle Responsys
Re: Transaction error in Kafka producer perf test
Posted by Anindya Haldar <an...@oracle.com>.
I understand that we could have used batch size as 0 here, if I am interpreting the documentation correctly.We essentially want no batching, and setting any value less than 256 in this particular case would achieve that, since the message size is 256 bytes for any message produced.
Sincerely,
Anindya Haldar
Oracle Responsys
> On Oct 28, 2019, at 12:30 PM, Anindya Haldar <an...@oracle.com> wrote:
>
> We are consciously choosing the batch size in the experiment to be 1 because that is what our producer use case will need to use.
>
> Regarding the request timeout that you have mentioned, how does one set that? The command line is already passing two timeout values: ‘max.block.ms’, and ‘transaction-duration-ms’. Both are set to 1 hour, and the failure we see happens within a few minutes. Is there a third timeout value that is expiring and causing this issue?
>
> Sincerely,
> Anindya Haldar
> Oracle Responsys
>
>
>> On Oct 28, 2019, at 11:52 AM, M. Manna <ma...@gmail.com> wrote:
>>
>> Hi,
>>
>> Per test is based on a set of tuning parameters e.g. batch.size, axes,
>> partitions, network latency etc. Your transactions are failing because your
>> batch has expired, (or at least, that’s what it shows on the log). You have
>> to tune your request timeout and batch.size correctly to improve on these.
>> I suggest you try and get this right first without a non-txn producer
>> setup. Then attempt with txn.
>>
>> Perhaps you want to recheck docs and understand what goal you want to
>> target (e.g. speed, consistency, balanced etc.).
>>
>> Regards,
>>
>>
>> On Mon, 28 Oct 2019 at 17:58, Anindya Haldar <an...@oracle.com>
>> wrote:
>>
>>> Anyone with a pointer on this? Do transactions work reliably with Kafka
>>> perf test tools? If yes, then is there a way to make it work in this
>>> scenario?
>>>
>>> Sincerely,
>>> Anindya Haldar
>>> Oracle Responsys
>>>
>>>
>>>> On Oct 25, 2019, at 2:51 PM, Anindya Haldar <an...@oracle.com>
>>> wrote:
>>>>
>>>> We are evaluating Kafka for some of our use cases. As part of that
>>> effort I am trying to run an experiment with a cluster we have set up, and
>>> using the producer perf test tool supplied with the binaries.
>>>>
>>>> Here’s the cluster info:
>>>>
>>>> Runs in Kubernetes, with 4 CPUs, 32 GB RAM, 100 GB log space allocation
>>> for each node.
>>>> 3 ZooKeeper nodes
>>>> 5 Kafka nodes
>>>>
>>>> Here is the topic description:
>>>>
>>>> $ bin/kafka-topics.sh --describe --bootstrap-server localhost:9092
>>> --topic testtopic3
>>>> Topic:testtopic3 PartitionCount:5 ReplicationFactor:3
>>> Configs:min.insync.replicas=2,segment.bytes=1073741824,retention.ms <
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__slack-2Dredir.net_link-3Furl-3Dhttp-253A-252F-252Fretention.ms&d=DwIFaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=vmJiAMDGSxNeZnFztNs5ITB_i_Z3h3VtLPGma9y7cKI&m=bcvrB2WHGYyMkDxuOiAX_kDm8dM_Gnhn9co-80eQpeM&s=d9QdJSgIZN7kFfuuRJDX3gi3J8Y7uTnC7UrLSFBdasI&e=
>>>> =3600000,flush.messages=1,unclean.leader.election.enable=false
>>>> Topic: testtopic3 Partition: 0 Leader: 1 Replicas: 1,4,2
>>> Isr: 1,4,2
>>>> Topic: testtopic3 Partition: 1 Leader: 4 Replicas: 4,2,3
>>> Isr: 4,2,3
>>>> Topic: testtopic3 Partition: 2 Leader: 2 Replicas: 2,3,5
>>> Isr: 2,3,5
>>>> Topic: testtopic3 Partition: 3 Leader: 3 Replicas: 3,5,1
>>> Isr: 3,5,1
>>>> Topic: testtopic3 Partition: 4 Leader: 5 Replicas: 5,1,4
>>> Isr: 5,1,4
>>>>
>>>>
>>>>
>>>> And here is the producer test run command line and the result:
>>>>
>>>> $ bin/kafka-producer-perf-test.sh --topic testtopic3 --num-records
>>> 1000000 --throughput -1 --record-size 256 --producer-props
>>> bootstrap.servers=kafka-a-0.ri:9092,kafka-b-0.ri:9092,kafka-c-0.ri:9092,kafka-d-0.ri:9092,kafka-e-0.ri:9092
>>> acks=all batch.size=1 max.block.ms <
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__slack-2Dredir.net_link-3Furl-3Dhttp-253A-252F-252Fmax.block.ms&d=DwIFaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=vmJiAMDGSxNeZnFztNs5ITB_i_Z3h3VtLPGma9y7cKI&m=bcvrB2WHGYyMkDxuOiAX_kDm8dM_Gnhn9co-80eQpeM&s=XP3cGgt3rD1lr8r0diTLpP4FP3BwZ8spcHNDNXc3i6I&e=
>>>> =3600000 enable.idempotence=true max.in.flight.requests.per.connection=1
>>> retries=3 --transaction-duration-ms 3600000
>>>> 4100 records sent, 819.7 records/sec (0.20 MB/sec), 2572.0 ms avg
>>> latency, 4892.0 ms max latency.
>>>> 4261 records sent, 852.0 records/sec (0.21 MB/sec), 7397.2 ms avg
>>> latency, 9873.0 ms max latency.
>>>> 4216 records sent, 843.0 records/sec (0.21 MB/sec), 12383.7 ms avg
>>> latency, 14849.0 ms max latency.
>>>> 4400 records sent, 879.8 records/sec (0.21 MB/sec), 17332.0 ms avg
>>> latency, 19784.0 ms max latency.
>>>> 4354 records sent, 870.8 records/sec (0.21 MB/sec), 22349.4 ms avg
>>> latency, 24763.0 ms max latency.
>>>> 4477 records sent, 895.4 records/sec (0.22 MB/sec), 27241.1 ms avg
>>> latency, 29728.0 ms max latency.
>>>> 4366 records sent, 873.2 records/sec (0.21 MB/sec), 32218.3 ms avg
>>> latency, 34703.0 ms max latency.
>>>> 4408 records sent, 881.6 records/sec (0.22 MB/sec), 37190.6 ms avg
>>> latency, 39672.0 ms max latency.
>>>> 4159 records sent, 831.5 records/sec (0.20 MB/sec), 42135.0 ms avg
>>> latency, 44640.0 ms max latency.
>>>> 4260 records sent, 852.0 records/sec (0.21 MB/sec), 47098.0 ms avg
>>> latency, 49624.0 ms max latency.
>>>> 4360 records sent, 872.0 records/sec (0.21 MB/sec), 52137.1 ms avg
>>> latency, 54574.0 ms max latency.
>>>> 4514 records sent, 902.8 records/sec (0.22 MB/sec), 57038.1 ms avg
>>> latency, 59554.0 ms max latency.
>>>> 4273 records sent, 854.3 records/sec (0.21 MB/sec), 62001.8 ms avg
>>> latency, 64524.0 ms max latency.
>>>> 4348 records sent, 869.6 records/sec (0.21 MB/sec), 67037.8 ms avg
>>> latency, 69494.0 ms max latency.
>>>> 4039 records sent, 807.5 records/sec (0.20 MB/sec), 72009.8 ms avg
>>> latency, 74481.0 ms max latency.
>>>> 4327 records sent, 865.2 records/sec (0.21 MB/sec), 76993.8 ms avg
>>> latency, 79457.0 ms max latency.
>>>> 4307 records sent, 861.4 records/sec (0.21 MB/sec), 82011.9 ms avg
>>> latency, 84449.0 ms max latency.
>>>> 4506 records sent, 901.0 records/sec (0.22 MB/sec), 86922.6 ms avg
>>> latency, 89434.0 ms max latency.
>>>> 4343 records sent, 868.6 records/sec (0.21 MB/sec), 91918.8 ms avg
>>> latency, 94394.0 ms max latency.
>>>> org.apache.kafka.common.errors.ProducerFencedException: Producer
>>> attempted an operation with an old epoch. Either there is a newer producer
>>> with the same transactionalId, or the producer's transaction has been
>>> expired by the broker.
>>>> Exception in thread "main" org.apache.kafka.common.KafkaException:
>>> Cannot perform send because at least one previous transactional or
>>> idempotent request has failed with errors.
>>>> at
>>> org.apache.kafka.clients.producer.internals.TransactionManager.failIfNotReadyForSend(TransactionManager.java:357)
>>>> at
>>> org.apache.kafka.clients.producer.internals.TransactionManager.maybeAddPartitionToTransaction(TransactionManager.java:341)
>>>> at
>>> org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:915)
>>>> at
>>> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:856)
>>>> at
>>> org.apache.kafka.tools.ProducerPerformance.main(ProducerPerformance.java:143)
>>>> Caused by: org.apache.kafka.common.errors.ProducerFencedException:
>>> Producer attempted an operation with an old epoch. Either there is a newer
>>> producer with the same transactionalId, or the producer's transaction has
>>> been expired by the broker.
>>>> [2019-10-25 21:15:05,183] ERROR [Producer clientId=producer-1,
>>> transactionalId=performance-producer-default-transactional-id] Aborting
>>> producer batches due to fatal error
>>> (org.apache.kafka.clients.producer.internals.Sender)
>>>> org.apache.kafka.common.errors.ProducerFencedException: Producer
>>> attempted an operation with an old epoch. Either there is a newer producer
>>> with the same transactionalId, or the producer's transaction has been
>>> expired by the broker.
>>>>
>>>> It does not make any difference whether I add a transaction id flag like
>>> ‘—transactional-id mytxn’ to the command line.
>>>>
>>>> Is there something I am missing here?
>>>>
>>>> Sincerely,
>>>> Anindya Haldar
>>>> Oracle Responsys
>>>>
>>>
>>>
>
Re: Transaction error in Kafka producer perf test
Posted by Anindya Haldar <an...@oracle.com>.
We are consciously choosing the batch size in the experiment to be 1 because that is what our producer use case will need to use.
Regarding the request timeout that you have mentioned, how does one set that? The command line is already passing two timeout values: ‘max.block.ms’, and ‘transaction-duration-ms’. Both are set to 1 hour, and the failure we see happens within a few minutes. Is there a third timeout value that is expiring and causing this issue?
Sincerely,
Anindya Haldar
Oracle Responsys
> On Oct 28, 2019, at 11:52 AM, M. Manna <ma...@gmail.com> wrote:
>
> Hi,
>
> Per test is based on a set of tuning parameters e.g. batch.size, axes,
> partitions, network latency etc. Your transactions are failing because your
> batch has expired, (or at least, that’s what it shows on the log). You have
> to tune your request timeout and batch.size correctly to improve on these.
> I suggest you try and get this right first without a non-txn producer
> setup. Then attempt with txn.
>
> Perhaps you want to recheck docs and understand what goal you want to
> target (e.g. speed, consistency, balanced etc.).
>
> Regards,
>
>
> On Mon, 28 Oct 2019 at 17:58, Anindya Haldar <an...@oracle.com>
> wrote:
>
>> Anyone with a pointer on this? Do transactions work reliably with Kafka
>> perf test tools? If yes, then is there a way to make it work in this
>> scenario?
>>
>> Sincerely,
>> Anindya Haldar
>> Oracle Responsys
>>
>>
>>> On Oct 25, 2019, at 2:51 PM, Anindya Haldar <an...@oracle.com>
>> wrote:
>>>
>>> We are evaluating Kafka for some of our use cases. As part of that
>> effort I am trying to run an experiment with a cluster we have set up, and
>> using the producer perf test tool supplied with the binaries.
>>>
>>> Here’s the cluster info:
>>>
>>> Runs in Kubernetes, with 4 CPUs, 32 GB RAM, 100 GB log space allocation
>> for each node.
>>> 3 ZooKeeper nodes
>>> 5 Kafka nodes
>>>
>>> Here is the topic description:
>>>
>>> $ bin/kafka-topics.sh --describe --bootstrap-server localhost:9092
>> --topic testtopic3
>>> Topic:testtopic3 PartitionCount:5 ReplicationFactor:3
>> Configs:min.insync.replicas=2,segment.bytes=1073741824,retention.ms <
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__slack-2Dredir.net_link-3Furl-3Dhttp-253A-252F-252Fretention.ms&d=DwIFaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=vmJiAMDGSxNeZnFztNs5ITB_i_Z3h3VtLPGma9y7cKI&m=bcvrB2WHGYyMkDxuOiAX_kDm8dM_Gnhn9co-80eQpeM&s=d9QdJSgIZN7kFfuuRJDX3gi3J8Y7uTnC7UrLSFBdasI&e=
>>> =3600000,flush.messages=1,unclean.leader.election.enable=false
>>> Topic: testtopic3 Partition: 0 Leader: 1 Replicas: 1,4,2
>> Isr: 1,4,2
>>> Topic: testtopic3 Partition: 1 Leader: 4 Replicas: 4,2,3
>> Isr: 4,2,3
>>> Topic: testtopic3 Partition: 2 Leader: 2 Replicas: 2,3,5
>> Isr: 2,3,5
>>> Topic: testtopic3 Partition: 3 Leader: 3 Replicas: 3,5,1
>> Isr: 3,5,1
>>> Topic: testtopic3 Partition: 4 Leader: 5 Replicas: 5,1,4
>> Isr: 5,1,4
>>>
>>>
>>>
>>> And here is the producer test run command line and the result:
>>>
>>> $ bin/kafka-producer-perf-test.sh --topic testtopic3 --num-records
>> 1000000 --throughput -1 --record-size 256 --producer-props
>> bootstrap.servers=kafka-a-0.ri:9092,kafka-b-0.ri:9092,kafka-c-0.ri:9092,kafka-d-0.ri:9092,kafka-e-0.ri:9092
>> acks=all batch.size=1 max.block.ms <
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__slack-2Dredir.net_link-3Furl-3Dhttp-253A-252F-252Fmax.block.ms&d=DwIFaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=vmJiAMDGSxNeZnFztNs5ITB_i_Z3h3VtLPGma9y7cKI&m=bcvrB2WHGYyMkDxuOiAX_kDm8dM_Gnhn9co-80eQpeM&s=XP3cGgt3rD1lr8r0diTLpP4FP3BwZ8spcHNDNXc3i6I&e=
>>> =3600000 enable.idempotence=true max.in.flight.requests.per.connection=1
>> retries=3 --transaction-duration-ms 3600000
>>> 4100 records sent, 819.7 records/sec (0.20 MB/sec), 2572.0 ms avg
>> latency, 4892.0 ms max latency.
>>> 4261 records sent, 852.0 records/sec (0.21 MB/sec), 7397.2 ms avg
>> latency, 9873.0 ms max latency.
>>> 4216 records sent, 843.0 records/sec (0.21 MB/sec), 12383.7 ms avg
>> latency, 14849.0 ms max latency.
>>> 4400 records sent, 879.8 records/sec (0.21 MB/sec), 17332.0 ms avg
>> latency, 19784.0 ms max latency.
>>> 4354 records sent, 870.8 records/sec (0.21 MB/sec), 22349.4 ms avg
>> latency, 24763.0 ms max latency.
>>> 4477 records sent, 895.4 records/sec (0.22 MB/sec), 27241.1 ms avg
>> latency, 29728.0 ms max latency.
>>> 4366 records sent, 873.2 records/sec (0.21 MB/sec), 32218.3 ms avg
>> latency, 34703.0 ms max latency.
>>> 4408 records sent, 881.6 records/sec (0.22 MB/sec), 37190.6 ms avg
>> latency, 39672.0 ms max latency.
>>> 4159 records sent, 831.5 records/sec (0.20 MB/sec), 42135.0 ms avg
>> latency, 44640.0 ms max latency.
>>> 4260 records sent, 852.0 records/sec (0.21 MB/sec), 47098.0 ms avg
>> latency, 49624.0 ms max latency.
>>> 4360 records sent, 872.0 records/sec (0.21 MB/sec), 52137.1 ms avg
>> latency, 54574.0 ms max latency.
>>> 4514 records sent, 902.8 records/sec (0.22 MB/sec), 57038.1 ms avg
>> latency, 59554.0 ms max latency.
>>> 4273 records sent, 854.3 records/sec (0.21 MB/sec), 62001.8 ms avg
>> latency, 64524.0 ms max latency.
>>> 4348 records sent, 869.6 records/sec (0.21 MB/sec), 67037.8 ms avg
>> latency, 69494.0 ms max latency.
>>> 4039 records sent, 807.5 records/sec (0.20 MB/sec), 72009.8 ms avg
>> latency, 74481.0 ms max latency.
>>> 4327 records sent, 865.2 records/sec (0.21 MB/sec), 76993.8 ms avg
>> latency, 79457.0 ms max latency.
>>> 4307 records sent, 861.4 records/sec (0.21 MB/sec), 82011.9 ms avg
>> latency, 84449.0 ms max latency.
>>> 4506 records sent, 901.0 records/sec (0.22 MB/sec), 86922.6 ms avg
>> latency, 89434.0 ms max latency.
>>> 4343 records sent, 868.6 records/sec (0.21 MB/sec), 91918.8 ms avg
>> latency, 94394.0 ms max latency.
>>> org.apache.kafka.common.errors.ProducerFencedException: Producer
>> attempted an operation with an old epoch. Either there is a newer producer
>> with the same transactionalId, or the producer's transaction has been
>> expired by the broker.
>>> Exception in thread "main" org.apache.kafka.common.KafkaException:
>> Cannot perform send because at least one previous transactional or
>> idempotent request has failed with errors.
>>> at
>> org.apache.kafka.clients.producer.internals.TransactionManager.failIfNotReadyForSend(TransactionManager.java:357)
>>> at
>> org.apache.kafka.clients.producer.internals.TransactionManager.maybeAddPartitionToTransaction(TransactionManager.java:341)
>>> at
>> org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:915)
>>> at
>> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:856)
>>> at
>> org.apache.kafka.tools.ProducerPerformance.main(ProducerPerformance.java:143)
>>> Caused by: org.apache.kafka.common.errors.ProducerFencedException:
>> Producer attempted an operation with an old epoch. Either there is a newer
>> producer with the same transactionalId, or the producer's transaction has
>> been expired by the broker.
>>> [2019-10-25 21:15:05,183] ERROR [Producer clientId=producer-1,
>> transactionalId=performance-producer-default-transactional-id] Aborting
>> producer batches due to fatal error
>> (org.apache.kafka.clients.producer.internals.Sender)
>>> org.apache.kafka.common.errors.ProducerFencedException: Producer
>> attempted an operation with an old epoch. Either there is a newer producer
>> with the same transactionalId, or the producer's transaction has been
>> expired by the broker.
>>>
>>> It does not make any difference whether I add a transaction id flag like
>> ‘—transactional-id mytxn’ to the command line.
>>>
>>> Is there something I am missing here?
>>>
>>> Sincerely,
>>> Anindya Haldar
>>> Oracle Responsys
>>>
>>
>>
Re: Transaction error in Kafka producer perf test
Posted by "M. Manna" <ma...@gmail.com>.
Hi,
Per test is based on a set of tuning parameters e.g. batch.size, axes,
partitions, network latency etc. Your transactions are failing because your
batch has expired, (or at least, that’s what it shows on the log). You have
to tune your request timeout and batch.size correctly to improve on these.
I suggest you try and get this right first without a non-txn producer
setup. Then attempt with txn.
Perhaps you want to recheck docs and understand what goal you want to
target (e.g. speed, consistency, balanced etc.).
Regards,
On Mon, 28 Oct 2019 at 17:58, Anindya Haldar <an...@oracle.com>
wrote:
> Anyone with a pointer on this? Do transactions work reliably with Kafka
> perf test tools? If yes, then is there a way to make it work in this
> scenario?
>
> Sincerely,
> Anindya Haldar
> Oracle Responsys
>
>
> > On Oct 25, 2019, at 2:51 PM, Anindya Haldar <an...@oracle.com>
> wrote:
> >
> > We are evaluating Kafka for some of our use cases. As part of that
> effort I am trying to run an experiment with a cluster we have set up, and
> using the producer perf test tool supplied with the binaries.
> >
> > Here’s the cluster info:
> >
> > Runs in Kubernetes, with 4 CPUs, 32 GB RAM, 100 GB log space allocation
> for each node.
> > 3 ZooKeeper nodes
> > 5 Kafka nodes
> >
> > Here is the topic description:
> >
> > $ bin/kafka-topics.sh --describe --bootstrap-server localhost:9092
> --topic testtopic3
> > Topic:testtopic3 PartitionCount:5 ReplicationFactor:3
> Configs:min.insync.replicas=2,segment.bytes=1073741824,retention.ms <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__slack-2Dredir.net_link-3Furl-3Dhttp-253A-252F-252Fretention.ms&d=DwIFaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=vmJiAMDGSxNeZnFztNs5ITB_i_Z3h3VtLPGma9y7cKI&m=bcvrB2WHGYyMkDxuOiAX_kDm8dM_Gnhn9co-80eQpeM&s=d9QdJSgIZN7kFfuuRJDX3gi3J8Y7uTnC7UrLSFBdasI&e=
> >=3600000,flush.messages=1,unclean.leader.election.enable=false
> > Topic: testtopic3 Partition: 0 Leader: 1 Replicas: 1,4,2
> Isr: 1,4,2
> > Topic: testtopic3 Partition: 1 Leader: 4 Replicas: 4,2,3
> Isr: 4,2,3
> > Topic: testtopic3 Partition: 2 Leader: 2 Replicas: 2,3,5
> Isr: 2,3,5
> > Topic: testtopic3 Partition: 3 Leader: 3 Replicas: 3,5,1
> Isr: 3,5,1
> > Topic: testtopic3 Partition: 4 Leader: 5 Replicas: 5,1,4
> Isr: 5,1,4
> >
> >
> >
> > And here is the producer test run command line and the result:
> >
> > $ bin/kafka-producer-perf-test.sh --topic testtopic3 --num-records
> 1000000 --throughput -1 --record-size 256 --producer-props
> bootstrap.servers=kafka-a-0.ri:9092,kafka-b-0.ri:9092,kafka-c-0.ri:9092,kafka-d-0.ri:9092,kafka-e-0.ri:9092
> acks=all batch.size=1 max.block.ms <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__slack-2Dredir.net_link-3Furl-3Dhttp-253A-252F-252Fmax.block.ms&d=DwIFaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=vmJiAMDGSxNeZnFztNs5ITB_i_Z3h3VtLPGma9y7cKI&m=bcvrB2WHGYyMkDxuOiAX_kDm8dM_Gnhn9co-80eQpeM&s=XP3cGgt3rD1lr8r0diTLpP4FP3BwZ8spcHNDNXc3i6I&e=
> >=3600000 enable.idempotence=true max.in.flight.requests.per.connection=1
> retries=3 --transaction-duration-ms 3600000
> > 4100 records sent, 819.7 records/sec (0.20 MB/sec), 2572.0 ms avg
> latency, 4892.0 ms max latency.
> > 4261 records sent, 852.0 records/sec (0.21 MB/sec), 7397.2 ms avg
> latency, 9873.0 ms max latency.
> > 4216 records sent, 843.0 records/sec (0.21 MB/sec), 12383.7 ms avg
> latency, 14849.0 ms max latency.
> > 4400 records sent, 879.8 records/sec (0.21 MB/sec), 17332.0 ms avg
> latency, 19784.0 ms max latency.
> > 4354 records sent, 870.8 records/sec (0.21 MB/sec), 22349.4 ms avg
> latency, 24763.0 ms max latency.
> > 4477 records sent, 895.4 records/sec (0.22 MB/sec), 27241.1 ms avg
> latency, 29728.0 ms max latency.
> > 4366 records sent, 873.2 records/sec (0.21 MB/sec), 32218.3 ms avg
> latency, 34703.0 ms max latency.
> > 4408 records sent, 881.6 records/sec (0.22 MB/sec), 37190.6 ms avg
> latency, 39672.0 ms max latency.
> > 4159 records sent, 831.5 records/sec (0.20 MB/sec), 42135.0 ms avg
> latency, 44640.0 ms max latency.
> > 4260 records sent, 852.0 records/sec (0.21 MB/sec), 47098.0 ms avg
> latency, 49624.0 ms max latency.
> > 4360 records sent, 872.0 records/sec (0.21 MB/sec), 52137.1 ms avg
> latency, 54574.0 ms max latency.
> > 4514 records sent, 902.8 records/sec (0.22 MB/sec), 57038.1 ms avg
> latency, 59554.0 ms max latency.
> > 4273 records sent, 854.3 records/sec (0.21 MB/sec), 62001.8 ms avg
> latency, 64524.0 ms max latency.
> > 4348 records sent, 869.6 records/sec (0.21 MB/sec), 67037.8 ms avg
> latency, 69494.0 ms max latency.
> > 4039 records sent, 807.5 records/sec (0.20 MB/sec), 72009.8 ms avg
> latency, 74481.0 ms max latency.
> > 4327 records sent, 865.2 records/sec (0.21 MB/sec), 76993.8 ms avg
> latency, 79457.0 ms max latency.
> > 4307 records sent, 861.4 records/sec (0.21 MB/sec), 82011.9 ms avg
> latency, 84449.0 ms max latency.
> > 4506 records sent, 901.0 records/sec (0.22 MB/sec), 86922.6 ms avg
> latency, 89434.0 ms max latency.
> > 4343 records sent, 868.6 records/sec (0.21 MB/sec), 91918.8 ms avg
> latency, 94394.0 ms max latency.
> > org.apache.kafka.common.errors.ProducerFencedException: Producer
> attempted an operation with an old epoch. Either there is a newer producer
> with the same transactionalId, or the producer's transaction has been
> expired by the broker.
> > Exception in thread "main" org.apache.kafka.common.KafkaException:
> Cannot perform send because at least one previous transactional or
> idempotent request has failed with errors.
> > at
> org.apache.kafka.clients.producer.internals.TransactionManager.failIfNotReadyForSend(TransactionManager.java:357)
> > at
> org.apache.kafka.clients.producer.internals.TransactionManager.maybeAddPartitionToTransaction(TransactionManager.java:341)
> > at
> org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:915)
> > at
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:856)
> > at
> org.apache.kafka.tools.ProducerPerformance.main(ProducerPerformance.java:143)
> > Caused by: org.apache.kafka.common.errors.ProducerFencedException:
> Producer attempted an operation with an old epoch. Either there is a newer
> producer with the same transactionalId, or the producer's transaction has
> been expired by the broker.
> > [2019-10-25 21:15:05,183] ERROR [Producer clientId=producer-1,
> transactionalId=performance-producer-default-transactional-id] Aborting
> producer batches due to fatal error
> (org.apache.kafka.clients.producer.internals.Sender)
> > org.apache.kafka.common.errors.ProducerFencedException: Producer
> attempted an operation with an old epoch. Either there is a newer producer
> with the same transactionalId, or the producer's transaction has been
> expired by the broker.
> >
> > It does not make any difference whether I add a transaction id flag like
> ‘—transactional-id mytxn’ to the command line.
> >
> > Is there something I am missing here?
> >
> > Sincerely,
> > Anindya Haldar
> > Oracle Responsys
> >
>
>
Re: Transaction error in Kafka producer perf test
Posted by Anindya Haldar <an...@oracle.com>.
Anyone with a pointer on this? Do transactions work reliably with Kafka perf test tools? If yes, then is there a way to make it work in this scenario?
Sincerely,
Anindya Haldar
Oracle Responsys
> On Oct 25, 2019, at 2:51 PM, Anindya Haldar <an...@oracle.com> wrote:
>
> We are evaluating Kafka for some of our use cases. As part of that effort I am trying to run an experiment with a cluster we have set up, and using the producer perf test tool supplied with the binaries.
>
> Here’s the cluster info:
>
> Runs in Kubernetes, with 4 CPUs, 32 GB RAM, 100 GB log space allocation for each node.
> 3 ZooKeeper nodes
> 5 Kafka nodes
>
> Here is the topic description:
>
> $ bin/kafka-topics.sh --describe --bootstrap-server localhost:9092 --topic testtopic3
> Topic:testtopic3 PartitionCount:5 ReplicationFactor:3 Configs:min.insync.replicas=2,segment.bytes=1073741824,retention.ms <https://urldefense.proofpoint.com/v2/url?u=https-3A__slack-2Dredir.net_link-3Furl-3Dhttp-253A-252F-252Fretention.ms&d=DwIFaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=vmJiAMDGSxNeZnFztNs5ITB_i_Z3h3VtLPGma9y7cKI&m=bcvrB2WHGYyMkDxuOiAX_kDm8dM_Gnhn9co-80eQpeM&s=d9QdJSgIZN7kFfuuRJDX3gi3J8Y7uTnC7UrLSFBdasI&e= >=3600000,flush.messages=1,unclean.leader.election.enable=false
> Topic: testtopic3 Partition: 0 Leader: 1 Replicas: 1,4,2 Isr: 1,4,2
> Topic: testtopic3 Partition: 1 Leader: 4 Replicas: 4,2,3 Isr: 4,2,3
> Topic: testtopic3 Partition: 2 Leader: 2 Replicas: 2,3,5 Isr: 2,3,5
> Topic: testtopic3 Partition: 3 Leader: 3 Replicas: 3,5,1 Isr: 3,5,1
> Topic: testtopic3 Partition: 4 Leader: 5 Replicas: 5,1,4 Isr: 5,1,4
>
>
>
> And here is the producer test run command line and the result:
>
> $ bin/kafka-producer-perf-test.sh --topic testtopic3 --num-records 1000000 --throughput -1 --record-size 256 --producer-props bootstrap.servers=kafka-a-0.ri:9092,kafka-b-0.ri:9092,kafka-c-0.ri:9092,kafka-d-0.ri:9092,kafka-e-0.ri:9092 acks=all batch.size=1 max.block.ms <https://urldefense.proofpoint.com/v2/url?u=https-3A__slack-2Dredir.net_link-3Furl-3Dhttp-253A-252F-252Fmax.block.ms&d=DwIFaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=vmJiAMDGSxNeZnFztNs5ITB_i_Z3h3VtLPGma9y7cKI&m=bcvrB2WHGYyMkDxuOiAX_kDm8dM_Gnhn9co-80eQpeM&s=XP3cGgt3rD1lr8r0diTLpP4FP3BwZ8spcHNDNXc3i6I&e= >=3600000 enable.idempotence=true max.in.flight.requests.per.connection=1 retries=3 --transaction-duration-ms 3600000
> 4100 records sent, 819.7 records/sec (0.20 MB/sec), 2572.0 ms avg latency, 4892.0 ms max latency.
> 4261 records sent, 852.0 records/sec (0.21 MB/sec), 7397.2 ms avg latency, 9873.0 ms max latency.
> 4216 records sent, 843.0 records/sec (0.21 MB/sec), 12383.7 ms avg latency, 14849.0 ms max latency.
> 4400 records sent, 879.8 records/sec (0.21 MB/sec), 17332.0 ms avg latency, 19784.0 ms max latency.
> 4354 records sent, 870.8 records/sec (0.21 MB/sec), 22349.4 ms avg latency, 24763.0 ms max latency.
> 4477 records sent, 895.4 records/sec (0.22 MB/sec), 27241.1 ms avg latency, 29728.0 ms max latency.
> 4366 records sent, 873.2 records/sec (0.21 MB/sec), 32218.3 ms avg latency, 34703.0 ms max latency.
> 4408 records sent, 881.6 records/sec (0.22 MB/sec), 37190.6 ms avg latency, 39672.0 ms max latency.
> 4159 records sent, 831.5 records/sec (0.20 MB/sec), 42135.0 ms avg latency, 44640.0 ms max latency.
> 4260 records sent, 852.0 records/sec (0.21 MB/sec), 47098.0 ms avg latency, 49624.0 ms max latency.
> 4360 records sent, 872.0 records/sec (0.21 MB/sec), 52137.1 ms avg latency, 54574.0 ms max latency.
> 4514 records sent, 902.8 records/sec (0.22 MB/sec), 57038.1 ms avg latency, 59554.0 ms max latency.
> 4273 records sent, 854.3 records/sec (0.21 MB/sec), 62001.8 ms avg latency, 64524.0 ms max latency.
> 4348 records sent, 869.6 records/sec (0.21 MB/sec), 67037.8 ms avg latency, 69494.0 ms max latency.
> 4039 records sent, 807.5 records/sec (0.20 MB/sec), 72009.8 ms avg latency, 74481.0 ms max latency.
> 4327 records sent, 865.2 records/sec (0.21 MB/sec), 76993.8 ms avg latency, 79457.0 ms max latency.
> 4307 records sent, 861.4 records/sec (0.21 MB/sec), 82011.9 ms avg latency, 84449.0 ms max latency.
> 4506 records sent, 901.0 records/sec (0.22 MB/sec), 86922.6 ms avg latency, 89434.0 ms max latency.
> 4343 records sent, 868.6 records/sec (0.21 MB/sec), 91918.8 ms avg latency, 94394.0 ms max latency.
> org.apache.kafka.common.errors.ProducerFencedException: Producer attempted an operation with an old epoch. Either there is a newer producer with the same transactionalId, or the producer's transaction has been expired by the broker.
> Exception in thread "main" org.apache.kafka.common.KafkaException: Cannot perform send because at least one previous transactional or idempotent request has failed with errors.
> at org.apache.kafka.clients.producer.internals.TransactionManager.failIfNotReadyForSend(TransactionManager.java:357)
> at org.apache.kafka.clients.producer.internals.TransactionManager.maybeAddPartitionToTransaction(TransactionManager.java:341)
> at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:915)
> at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:856)
> at org.apache.kafka.tools.ProducerPerformance.main(ProducerPerformance.java:143)
> Caused by: org.apache.kafka.common.errors.ProducerFencedException: Producer attempted an operation with an old epoch. Either there is a newer producer with the same transactionalId, or the producer's transaction has been expired by the broker.
> [2019-10-25 21:15:05,183] ERROR [Producer clientId=producer-1, transactionalId=performance-producer-default-transactional-id] Aborting producer batches due to fatal error (org.apache.kafka.clients.producer.internals.Sender)
> org.apache.kafka.common.errors.ProducerFencedException: Producer attempted an operation with an old epoch. Either there is a newer producer with the same transactionalId, or the producer's transaction has been expired by the broker.
>
> It does not make any difference whether I add a transaction id flag like ‘—transactional-id mytxn’ to the command line.
>
> Is there something I am missing here?
>
> Sincerely,
> Anindya Haldar
> Oracle Responsys
>