You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Justine Olshan (Jira)" <ji...@apache.org> on 2023/04/13 21:27:00 UTC
[jira] [Commented] (KAFKA-14904) Flaky Test kafka.api.TransactionsBounceTest.testWithGroupId()
[ https://issues.apache.org/jira/browse/KAFKA-14904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712090#comment-17712090 ]
Justine Olshan commented on KAFKA-14904:
----------------------------------------
Jenkins truncates this out since the bounced broker leads to a ton of aborted transaction errors, but after that we see:
{code:java}
[2023-04-13 14:15:44,487] WARN [Producer clientId=producer-test-txn, transactionalId=test-txn] Got error produce response with correlation id 16 on topic-partition output-topic-0, retrying (2147483646 attempts left). Error: CONCURRENT_TRANSACTIONS (org.apache.kafka.clients.producer.internals.Sender:630)
[2023-04-13 14:15:44,615] ERROR [ReplicaManager broker=1] Error processing append operation on partition output-topic-0 (kafka.server.ReplicaManager:76)
org.apache.kafka.common.errors.OutOfOrderSequenceException: Out of order sequence number for producer 1 at offset 11 in partition output-topic-0: 0 (incoming seq. number), 21 (current end sequence number)
[2023-04-13 14:15:44,615] WARN [Producer clientId=producer-test-txn, transactionalId=test-txn] Got error produce response with correlation id 20 on topic-partition output-topic-0, retrying (2147483645 attempts left). Error: OUT_OF_ORDER_SEQUENCE_NUMBER (org.apache.kafka.clients.producer.internals.Sender:630)
[2023-04-13 14:15:44,686] WARN [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Partition output-topic-0 marked as failed (kafka.server.ReplicaFetcherThread:70)
{code}
And a TON more out of sequence errors (probably most of the partitions). Will investigate further if this is a result of my change and if there are ways to fix this.
Other logs:
{code:java}
[2023-04-13 14:16:49,648] ERROR [ReplicaManager broker=1] Error processing append operation on partition output-topic-0 (kafka.server.ReplicaManager:76)
org.apache.kafka.common.errors.OutOfOrderSequenceException: Out of order sequence number for producer 1 at offset 489 in partition output-topic-0: 0 (incoming seq. number), 499 (current end sequence number)
[2023-04-13 14:16:49,649] WARN [Producer clientId=producer-test-txn, transactionalId=test-txn] Got error produce response with correlation id 774 on topic-partition output-topic-0, retrying (2147483019 attempts left). Error: OUT_OF_ORDER_SEQUENCE_NUMBER (org.apache.kafka.clients.producer.internals.Sender:630)
[2023-04-13 14:16:49,749] ERROR [ReplicaManager broker=1] Error processing append operation on partition output-topic-0 (kafka.server.ReplicaManager:76)
org.apache.kafka.common.errors.OutOfOrderSequenceException: Out of order sequence number for producer 1 at offset 489 in partition output-topic-0: 0 (incoming seq. number), 499 (current end sequence number)
[2023-04-13 14:16:49,750] WARN [Producer clientId=producer-test-txn, transactionalId=test-txn] Got error produce response with correlation id 775 on topic-partition output-topic-0, retrying (2147483018 attempts left). Error: OUT_OF_ORDER_SEQUENCE_NUMBER (org.apache.kafka.clients.producer.internals.Sender:630)
[2023-04-13 14:16:49,851] ERROR [ReplicaManager broker=1] Error processing append operation on partition output-topic-0 (kafka.server.ReplicaManager:76)
org.apache.kafka.common.errors.OutOfOrderSequenceException: Out of order sequence number for producer 1 at offset 489 in partition output-topic-0: 0 (incoming seq. number), 499 (current end sequence number)
[2023-04-13 14:16:49,851] WARN [Producer clientId=producer-test-txn, transactionalId=test-txn] Got error produce response with correlation id 776 on topic-partition output-topic-0, retrying (2147483017 attempts left). Error: OUT_OF_ORDER_SEQUENCE_NUMBER (org.apache.kafka.clients.producer.internals.Sender:630)
[2023-04-13 14:16:49,952] ERROR [ReplicaManager broker=1] Error processing append operation on partition output-topic-0 (kafka.server.ReplicaManager:76){code}
And
{code:java}
[2023-04-13 14:16:51,573] ERROR [ReplicaManager broker=1] Error processing append operation on partition output-topic-0 (kafka.server.ReplicaManager:76) org.apache.kafka.common.errors.OutOfOrderSequenceException: Out of order sequence number for producer 1 at offset 489 in partition output-topic-0: 0 (incoming seq. number), 499 (current end sequence number) [2023-04-13 14:16:51,573] WARN [Producer clientId=producer-test-txn, transactionalId=test-txn] Got error produce response with correlation id 793 on topic-partition output-topic-0, retrying (2147483000 attempts left). Error: OUT_OF_ORDER_SEQUENCE_NUMBER (org.apache.kafka.clients.producer.internals.Sender:630) [2023-04-13 14:16:51,622] ERROR Error when sending message to topic output-topic with key: 1, value: 1 with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52) org.apache.kafka.common.KafkaException: Producer is closed forcefully. at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083) at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283) at java.base/java.lang.Thread.run(Thread.java:833) [2023-04-13 14:16:51,622] ERROR Error when sending message to topic output-topic with key: 5, value: 5 with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52) org.apache.kafka.common.KafkaException: Producer is closed forcefully. at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083) at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283) at java.base/java.lang.Thread.run(Thread.java:833) [2023-04-13 14:16:51,622] ERROR Error when sending message to topic output-topic with key: 7, value: 7 with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52) org.apache.kafka.common.KafkaException: Producer is closed forcefully. at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083) at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283) at java.base/java.lang.Thread.run(Thread.java:833) [2023-04-13 14:16:51,622] ERROR Error when sending message to topic output-topic with key: 8, value: 8 with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52) org.apache.kafka.common.KafkaException: Producer is closed forcefully. at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083) at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283) at java.base/java.lang.Thread.run(Thread.java:833) [2023-04-13 14:16:51,622] ERROR Error when sending message to topic output-topic with key: 11, value: 11 with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52) org.apache.kafka.common.KafkaException: Producer is closed forcefully. at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083) at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283) at java.base/java.lang.Thread.run(Thread.java:833) [2023-04-13 14:16:51,622] ERROR Error when sending message to topic output-topic with key: 15, value: 15 with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52) org.apache.kafka.common.KafkaException: Producer is closed forcefully. at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083) at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283) at java.base/java.lang.Thread.run(Thread.java:833) [2023-04-13 14:16:51,622] ERROR Error when sending message to topic output-topic with key: 17, value: 17 with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52) org.apache.kafka.common.KafkaException: Producer is closed forcefully. at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083) at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283) at java.base/java.lang.Thread.run(Thread.java:833) [2023-04-13 14:16:51,622] ERROR Error when sending message to topic output-topic with key: 21, value: 21 with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52) org.apache.kafka.common.KafkaException: Producer is closed forcefully. at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083) at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283) at java.base/java.lang.Thread.run(Thread.java:833) [2023-04-13 14:16:51,622] ERROR Error when sending message to topic output-topic with key: 22, value: 22 with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52) org.apache.kafka.common.KafkaException: Producer is closed forcefully. at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083) at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283) at java.base/java.lang.Thread.run(Thread.java:833) [2023-04-13 14:16:51,622] ERROR Error when sending message to topic output-topic with key: 23, value: 23 with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52) org.apache.kafka.common.KafkaException: Producer is closed forcefully. at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083) at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283) at java.base/java.lang.Thread.run(Thread.java:833) [2023-04-13 14:16:51,622] ERROR Error when sending message to topic output-topic with key: 25, value: 25 with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52) org.apache.kafka.common.KafkaException: Producer is closed forcefully. at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083) at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283) at java.base/java.lang.Thread.run(Thread.java:833)[2023-04-13 14:16:51,573] ERROR [ReplicaManager broker=1] Error processing append operation on partition output-topic-0 (kafka.server.ReplicaManager:76)
org.apache.kafka.common.errors.OutOfOrderSequenceException: Out of order sequence number for producer 1 at offset 489 in partition output-topic-0: 0 (incoming seq. number), 499 (current end sequence number)
[2023-04-13 14:16:51,573] WARN [Producer clientId=producer-test-txn, transactionalId=test-txn] Got error produce response with correlation id 793 on topic-partition output-topic-0, retrying (2147483000 attempts left). Error: OUT_OF_ORDER_SEQUENCE_NUMBER (org.apache.kafka.clients.producer.internals.Sender:630)
[2023-04-13 14:16:51,622] ERROR Error when sending message to topic output-topic with key: 1, value: 1 with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully.
at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283)
at java.base/java.lang.Thread.run(Thread.java:833)
[2023-04-13 14:16:51,622] ERROR Error when sending message to topic output-topic with key: 5, value: 5 with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully.
at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283)
at java.base/java.lang.Thread.run(Thread.java:833)
[2023-04-13 14:16:51,622] ERROR Error when sending message to topic output-topic with key: 7, value: 7 with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully.
at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283)
at java.base/java.lang.Thread.run(Thread.java:833)
[2023-04-13 14:16:51,622] ERROR Error when sending message to topic output-topic with key: 8, value: 8 with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully.
at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283)
at java.base/java.lang.Thread.run(Thread.java:833)
[2023-04-13 14:16:51,622] ERROR Error when sending message to topic output-topic with key: 11, value: 11 with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully.
at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283)
at java.base/java.lang.Thread.run(Thread.java:833)
[2023-04-13 14:16:51,622] ERROR Error when sending message to topic output-topic with key: 15, value: 15 with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully.
at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283)
at java.base/java.lang.Thread.run(Thread.java:833)
[2023-04-13 14:16:51,622] ERROR Error when sending message to topic output-topic with key: 17, value: 17 with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully.
at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283)
at java.base/java.lang.Thread.run(Thread.java:833)
[2023-04-13 14:16:51,622] ERROR Error when sending message to topic output-topic with key: 21, value: 21 with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully.
at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283)
at java.base/java.lang.Thread.run(Thread.java:833)
[2023-04-13 14:16:51,622] ERROR Error when sending message to topic output-topic with key: 22, value: 22 with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully.
at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283)
at java.base/java.lang.Thread.run(Thread.java:833)
[2023-04-13 14:16:51,622] ERROR Error when sending message to topic output-topic with key: 23, value: 23 with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully.
at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283)
at java.base/java.lang.Thread.run(Thread.java:833)
[2023-04-13 14:16:51,622] ERROR Error when sending message to topic output-topic with key: 25, value: 25 with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback:52)
org.apache.kafka.common.KafkaException: Producer is closed forcefully.
at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:1083)
at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:1070)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:283)
at java.base/java.lang.Thread.run(Thread.java:833){code}
> Flaky Test kafka.api.TransactionsBounceTest.testWithGroupId()
> --------------------------------------------------------------
>
> Key: KAFKA-14904
> URL: https://issues.apache.org/jira/browse/KAFKA-14904
> Project: Kafka
> Issue Type: Test
> Reporter: Justine Olshan
> Assignee: Justine Olshan
> Priority: Major
>
> After merging KAFKA-14561 I noticed this test still occasionally failed via
> org.apache.kafka.common.errors.TimeoutException: Timeout expired after 60000ms while awaiting EndTxn(true)
> I will investigate the cause.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)