You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Matthias (Jira)" <ji...@apache.org> on 2021/10/11 09:17:00 UTC
[jira] [Comment Edited] (FLINK-24482)
KafkaShuffleTestBase.testMetrics fails on Azure
[ https://issues.apache.org/jira/browse/FLINK-24482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427027#comment-17427027 ]
Matthias edited comment on FLINK-24482 at 10/11/21, 9:16 AM:
-------------------------------------------------------------
This seems to be caused by failing to elect a leader for a partition:
{code}
19:59:11,665 [controller-event-thread] ERROR state.change.logger [] - [Controller id=2 epoch=2] Controller 2 epoch 2 failed to change state for partition testAbortTransactionAfterClosed-0 from OfflinePartition to
OnlinePartition
kafka.common.StateChangeFailedException: Failed to elect leader for partition testAbortTransactionAfterClosed-0 under strategy OfflinePartitionLeaderElectionStrategy(false)
at kafka.controller.ZkPartitionStateMachine$$anonfun$doElectLeaderForPartitions$2.apply(PartitionStateMachine.scala:427) ~[kafka_2.11-2.4.1.jar:?]
at kafka.controller.ZkPartitionStateMachine$$anonfun$doElectLeaderForPartitions$2.apply(PartitionStateMachine.scala:424) ~[kafka_2.11-2.4.1.jar:?]
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) ~[scala-library-2.11.12.jar:?]
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) ~[scala-library-2.11.12.jar:?]
at kafka.controller.ZkPartitionStateMachine.doElectLeaderForPartitions(PartitionStateMachine.scala:424) ~[kafka_2.11-2.4.1.jar:?]
at kafka.controller.ZkPartitionStateMachine.electLeaderForPartitions(PartitionStateMachine.scala:335) [kafka_2.11-2.4.1.jar:?]
at kafka.controller.ZkPartitionStateMachine.doHandleStateChanges(PartitionStateMachine.scala:233) [kafka_2.11-2.4.1.jar:?]
at kafka.controller.ZkPartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:154) [kafka_2.11-2.4.1.jar:?]
at kafka.controller.PartitionStateMachine.triggerOnlineStateChangeForPartitions(PartitionStateMachine.scala:73) [kafka_2.11-2.4.1.jar:?]
at kafka.controller.PartitionStateMachine.triggerOnlinePartitionStateChange(PartitionStateMachine.scala:58) [kafka_2.11-2.4.1.jar:?]
at kafka.controller.KafkaController.onReplicasBecomeOffline(KafkaController.scala:510) [kafka_2.11-2.4.1.jar:?]
at kafka.controller.KafkaController.onBrokerFailure(KafkaController.scala:478) [kafka_2.11-2.4.1.jar:?]
at kafka.controller.KafkaController.processBrokerChange(KafkaController.scala:1453) [kafka_2.11-2.4.1.jar:?]
at kafka.controller.KafkaController.process(KafkaController.scala:1878) [kafka_2.11-2.4.1.jar:?]
at kafka.controller.QueuedEvent.process(ControllerEventManager.scala:53) [kafka_2.11-2.4.1.jar:?]
at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply$mcV$sp(ControllerEventManager.scala:137) [kafka_2.11-2.4.1.jar:?]
at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:137) [kafka_2.11-2.4.1.jar:?]
at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:137) [kafka_2.11-2.4.1.jar:?]
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:31) [kafka_2.11-2.4.1.jar:?]
at kafka.controller.ControllerEventManager$ControllerEventThread.doWork(ControllerEventManager.scala:136) [kafka_2.11-2.4.1.jar:?]
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96) [kafka_2.11-2.4.1.jar:?]
{code}
There are some connection issues beforehand. I'm not sure whether that caused the failure mentioned above. I'm going to attached the build artifacts to this ticket for [this build failure|https://dev.azure.com/ververica-dev/daplatform-flink/_build/results?buildId=1565&view=logs&j=d543d572-9428-5803-a30c-e8e09bf70915&t=18a8058f-13c3-5857-f389-32cb0ee00ed2]. I guess, the builds are not accessible by everyone.
was (Author: mapohl):
This seems to be caused by failing to elect a leader for a partition:
```
19:59:11,665 [controller-event-thread] ERROR state.change.logger [] - [Controller id=2 epoch=2] Controller 2 epoch 2 failed to change state for partition testAbortTransactionAfterClosed-0 from OfflinePartition to
OnlinePartition
kafka.common.StateChangeFailedException: Failed to elect leader for partition testAbortTransactionAfterClosed-0 under strategy OfflinePartitionLeaderElectionStrategy(false)
at kafka.controller.ZkPartitionStateMachine$$anonfun$doElectLeaderForPartitions$2.apply(PartitionStateMachine.scala:427) ~[kafka_2.11-2.4.1.jar:?]
at kafka.controller.ZkPartitionStateMachine$$anonfun$doElectLeaderForPartitions$2.apply(PartitionStateMachine.scala:424) ~[kafka_2.11-2.4.1.jar:?]
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) ~[scala-library-2.11.12.jar:?]
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) ~[scala-library-2.11.12.jar:?]
at kafka.controller.ZkPartitionStateMachine.doElectLeaderForPartitions(PartitionStateMachine.scala:424) ~[kafka_2.11-2.4.1.jar:?]
at kafka.controller.ZkPartitionStateMachine.electLeaderForPartitions(PartitionStateMachine.scala:335) [kafka_2.11-2.4.1.jar:?]
at kafka.controller.ZkPartitionStateMachine.doHandleStateChanges(PartitionStateMachine.scala:233) [kafka_2.11-2.4.1.jar:?]
at kafka.controller.ZkPartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:154) [kafka_2.11-2.4.1.jar:?]
at kafka.controller.PartitionStateMachine.triggerOnlineStateChangeForPartitions(PartitionStateMachine.scala:73) [kafka_2.11-2.4.1.jar:?]
at kafka.controller.PartitionStateMachine.triggerOnlinePartitionStateChange(PartitionStateMachine.scala:58) [kafka_2.11-2.4.1.jar:?]
at kafka.controller.KafkaController.onReplicasBecomeOffline(KafkaController.scala:510) [kafka_2.11-2.4.1.jar:?]
at kafka.controller.KafkaController.onBrokerFailure(KafkaController.scala:478) [kafka_2.11-2.4.1.jar:?]
at kafka.controller.KafkaController.processBrokerChange(KafkaController.scala:1453) [kafka_2.11-2.4.1.jar:?]
at kafka.controller.KafkaController.process(KafkaController.scala:1878) [kafka_2.11-2.4.1.jar:?]
at kafka.controller.QueuedEvent.process(ControllerEventManager.scala:53) [kafka_2.11-2.4.1.jar:?]
at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply$mcV$sp(ControllerEventManager.scala:137) [kafka_2.11-2.4.1.jar:?]
at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:137) [kafka_2.11-2.4.1.jar:?]
at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:137) [kafka_2.11-2.4.1.jar:?]
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:31) [kafka_2.11-2.4.1.jar:?]
at kafka.controller.ControllerEventManager$ControllerEventThread.doWork(ControllerEventManager.scala:136) [kafka_2.11-2.4.1.jar:?]
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96) [kafka_2.11-2.4.1.jar:?]
```
There are some connection issues beforehand. I'm not sure whether that caused the failure mentioned above. I'm going to attached the build artifacts to this ticket for [this build failure|https://dev.azure.com/ververica-dev/daplatform-flink/_build/results?buildId=1565&view=logs&j=d543d572-9428-5803-a30c-e8e09bf70915&t=18a8058f-13c3-5857-f389-32cb0ee00ed2]. I guess, the builds are not accessible by everyone.
> KafkaShuffleTestBase.testMetrics fails on Azure
> -----------------------------------------------
>
> Key: FLINK-24482
> URL: https://issues.apache.org/jira/browse/FLINK-24482
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
> Affects Versions: 1.13.2
> Reporter: Till Rohrmann
> Priority: Critical
> Labels: test-stability
> Attachments: logs-ci_build-test_ci_build_kafka_gelly-1633289435.zip
>
>
> The test {{KafkaShuffleTestBase.testMetrics}} fails with
> {code}
> Oct 06 17:30:40 [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 33.966 s <<< FAILURE! - in org.apache.flink.streaming.connectors.kafka.shuffle.KafkaShuffleTestBase
> Oct 06 17:30:40 [ERROR] testMetrics(org.apache.flink.streaming.connectors.kafka.shuffle.KafkaShuffleTestBase) Time elapsed: 2.948 s <<< FAILURE!
> Oct 06 17:30:40 java.lang.AssertionError: expected:<14716> but was:<14206>
> Oct 06 17:30:40 at org.junit.Assert.fail(Assert.java:88)
> Oct 06 17:30:40 at org.junit.Assert.failNotEquals(Assert.java:834)
> Oct 06 17:30:40 at org.junit.Assert.assertEquals(Assert.java:645)
> Oct 06 17:30:40 at org.junit.Assert.assertEquals(Assert.java:631)
> Oct 06 17:30:40 at org.apache.flink.streaming.connectors.kafka.KafkaConsumerTestBase.runMetricsTest(KafkaConsumerTestBase.java:1892)
> Oct 06 17:30:40 at org.apache.flink.streaming.connectors.kafka.KafkaConsumerTestBase.testMetrics(KafkaConsumerTestBase.java:1808)
> Oct 06 17:30:40 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> Oct 06 17:30:40 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Oct 06 17:30:40 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Oct 06 17:30:40 at java.lang.reflect.Method.invoke(Method.java:498)
> Oct 06 17:30:40 at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> Oct 06 17:30:40 at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Oct 06 17:30:40 at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> Oct 06 17:30:40 at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Oct 06 17:30:40 at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> Oct 06 17:30:40 at org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> Oct 06 17:30:40 at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> Oct 06 17:30:40 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> Oct 06 17:30:40 at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> Oct 06 17:30:40 at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> Oct 06 17:30:40 at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> Oct 06 17:30:40 at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> Oct 06 17:30:40 at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> Oct 06 17:30:40 at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> Oct 06 17:30:40 at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> Oct 06 17:30:40 at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> Oct 06 17:30:40 at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> Oct 06 17:30:40 at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> Oct 06 17:30:40 at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> Oct 06 17:30:40 at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> Oct 06 17:30:40 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> Oct 06 17:30:40 at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> Oct 06 17:30:40 at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> Oct 06 17:30:40 at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> Oct 06 17:30:40 at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> Oct 06 17:30:40 at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> Oct 06 17:30:40 at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> Oct 06 17:30:40 at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> Oct 06 17:30:40 at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> Oct 06 17:30:40 at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
> https://dev.azure.com/ververica-dev/daplatform-flink/_build/results?buildId=1549&view=logs&j=d543d572-9428-5803-a30c-e8e09bf70915&t=18a8058f-13c3-5857-f389-32cb0ee00ed2&l=6507
--
This message was sent by Atlassian Jira
(v8.3.4#803005)