You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Anton Kalashnikov (Jira)" <ji...@apache.org> on 2022/01/04 11:26:00 UTC
[jira] [Resolved] (FLINK-25426) UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint fails on AZP because it cannot allocate enough network buffers
[ https://issues.apache.org/jira/browse/FLINK-25426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Anton Kalashnikov resolved FLINK-25426.
---------------------------------------
Resolution: Won't Fix
I close this ticket since the commit which introduced this problem was reverted(see https://issues.apache.org/jira/browse/FLINK-18356) and I don't see new failures after it.
> UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint fails on AZP because it cannot allocate enough network buffers
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-25426
> URL: https://issues.apache.org/jira/browse/FLINK-25426
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 1.15.0
> Reporter: Till Rohrmann
> Assignee: Anton Kalashnikov
> Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0
>
>
> The test {{UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint}} fails with
> {code}
> 2021-12-23T02:54:46.2862342Z Dec 23 02:54:46 [ERROR] UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint Time elapsed: 2.992 s <<< ERROR!
> 2021-12-23T02:54:46.2865774Z Dec 23 02:54:46 java.lang.OutOfMemoryError: Could not allocate enough memory segments for NetworkBufferPool (required (Mb): 64, allocated (Mb): 14, missing (Mb): 50). Cause: Direct buffer memory. The direct out-of-memory error has occurred. This can mean two things: either job(s) require(s) a larger size of JVM direct memory or there is a direct memory leak. The direct memory can be allocated by user code or some of its dependencies. In this case 'taskmanager.memory.task.off-heap.size' configuration option should be increased. Flink framework and its dependencies also consume the direct memory, mostly for network communication. The most of network memory is managed by Flink and should not result in out-of-memory error. In certain special cases, in particular for jobs with high parallelism, the framework may require more direct memory which is not managed by Flink. In this case 'taskmanager.memory.framework.off-heap.size' configuration option should be increased. If the error persists then there is probably a direct memory leak in user code or some of its dependencies which has to be investigated and fixed. The task executor has to be shutdown...
> 2021-12-23T02:54:46.2868239Z Dec 23 02:54:46 at org.apache.flink.runtime.io.network.buffer.NetworkBufferPool.<init>(NetworkBufferPool.java:138)
> 2021-12-23T02:54:46.2868975Z Dec 23 02:54:46 at org.apache.flink.runtime.io.network.NettyShuffleServiceFactory.createNettyShuffleEnvironment(NettyShuffleServiceFactory.java:140)
> 2021-12-23T02:54:46.2869771Z Dec 23 02:54:46 at org.apache.flink.runtime.io.network.NettyShuffleServiceFactory.createNettyShuffleEnvironment(NettyShuffleServiceFactory.java:94)
> 2021-12-23T02:54:46.2870550Z Dec 23 02:54:46 at org.apache.flink.runtime.io.network.NettyShuffleServiceFactory.createShuffleEnvironment(NettyShuffleServiceFactory.java:79)
> 2021-12-23T02:54:46.2871312Z Dec 23 02:54:46 at org.apache.flink.runtime.io.network.NettyShuffleServiceFactory.createShuffleEnvironment(NettyShuffleServiceFactory.java:58)
> 2021-12-23T02:54:46.2872062Z Dec 23 02:54:46 at org.apache.flink.runtime.taskexecutor.TaskManagerServices.createShuffleEnvironment(TaskManagerServices.java:414)
> 2021-12-23T02:54:46.2872767Z Dec 23 02:54:46 at org.apache.flink.runtime.taskexecutor.TaskManagerServices.fromConfiguration(TaskManagerServices.java:282)
> 2021-12-23T02:54:46.2873436Z Dec 23 02:54:46 at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.startTaskManager(TaskManagerRunner.java:523)
> 2021-12-23T02:54:46.2877615Z Dec 23 02:54:46 at org.apache.flink.runtime.minicluster.MiniCluster.startTaskManager(MiniCluster.java:645)
> 2021-12-23T02:54:46.2878247Z Dec 23 02:54:46 at org.apache.flink.runtime.minicluster.MiniCluster.startTaskManagers(MiniCluster.java:626)
> 2021-12-23T02:54:46.2878856Z Dec 23 02:54:46 at org.apache.flink.runtime.minicluster.MiniCluster.start(MiniCluster.java:379)
> 2021-12-23T02:54:46.2879487Z Dec 23 02:54:46 at org.apache.flink.runtime.testutils.MiniClusterResource.startMiniCluster(MiniClusterResource.java:209)
> 2021-12-23T02:54:46.2880152Z Dec 23 02:54:46 at org.apache.flink.runtime.testutils.MiniClusterResource.before(MiniClusterResource.java:95)
> 2021-12-23T02:54:46.2880821Z Dec 23 02:54:46 at org.apache.flink.test.util.MiniClusterWithClientResource.before(MiniClusterWithClientResource.java:64)
> 2021-12-23T02:54:46.2881519Z Dec 23 02:54:46 at org.apache.flink.test.checkpointing.UnalignedCheckpointTestBase.execute(UnalignedCheckpointTestBase.java:151)
> 2021-12-23T02:54:46.2882310Z Dec 23 02:54:46 at org.apache.flink.test.checkpointing.UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint(UnalignedCheckpointRescaleITCase.java:534)
> 2021-12-23T02:54:46.2882978Z Dec 23 02:54:46 at jdk.internal.reflect.GeneratedMethodAccessor123.invoke(Unknown Source)
> 2021-12-23T02:54:46.2883574Z Dec 23 02:54:46 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-12-23T02:54:46.2884171Z Dec 23 02:54:46 at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> 2021-12-23T02:54:46.2884732Z Dec 23 02:54:46 at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 2021-12-23T02:54:46.2885527Z Dec 23 02:54:46 at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2021-12-23T02:54:46.2886135Z Dec 23 02:54:46 at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 2021-12-23T02:54:46.2886755Z Dec 23 02:54:46 at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2021-12-23T02:54:46.2887387Z Dec 23 02:54:46 at org.junit.rules.Verifier$1.evaluate(Verifier.java:35)
> 2021-12-23T02:54:46.2887892Z Dec 23 02:54:46 at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2021-12-23T02:54:46.2888435Z Dec 23 02:54:46 at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2021-12-23T02:54:46.2889007Z Dec 23 02:54:46 at org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-12-23T02:54:46.2889568Z Dec 23 02:54:46 at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2021-12-23T02:54:46.2890104Z Dec 23 02:54:46 at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2021-12-23T02:54:46.2890686Z Dec 23 02:54:46 at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2021-12-23T02:54:46.2891259Z Dec 23 02:54:46 at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2021-12-23T02:54:46.2891819Z Dec 23 02:54:46 at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2021-12-23T02:54:46.2892421Z Dec 23 02:54:46 at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2021-12-23T02:54:46.2892978Z Dec 23 02:54:46 at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2021-12-23T02:54:46.2893508Z Dec 23 02:54:46 at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2021-12-23T02:54:46.2894049Z Dec 23 02:54:46 at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2021-12-23T02:54:46.2894588Z Dec 23 02:54:46 at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> 2021-12-23T02:54:46.2895203Z Dec 23 02:54:46 at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> 2021-12-23T02:54:46.2895721Z Dec 23 02:54:46 at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> 2021-12-23T02:54:46.2896304Z Dec 23 02:54:46 at org.junit.runners.Suite.runChild(Suite.java:128)
> 2021-12-23T02:54:46.2896781Z Dec 23 02:54:46 at org.junit.runners.Suite.runChild(Suite.java:27)
> 2021-12-23T02:54:46.2897359Z Dec 23 02:54:46 at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2021-12-23T02:54:46.2897892Z Dec 23 02:54:46 at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2021-12-23T02:54:46.2898429Z Dec 23 02:54:46 at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2021-12-23T02:54:46.2898968Z Dec 23 02:54:46 at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> 2021-12-23T02:54:46.2899487Z Dec 23 02:54:46 at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> 2021-12-23T02:54:46.2900025Z Dec 23 02:54:46 at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2021-12-23T02:54:46.2900542Z Dec 23 02:54:46 at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> 2021-12-23T02:54:46.2901044Z Dec 23 02:54:46 at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> 2021-12-23T02:54:46.2901540Z Dec 23 02:54:46 at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> 2021-12-23T02:54:46.2902086Z Dec 23 02:54:46 at org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
> 2021-12-23T02:54:46.2902702Z Dec 23 02:54:46 at org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
> 2021-12-23T02:54:46.2903297Z Dec 23 02:54:46 at org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:72)
> 2021-12-23T02:54:46.2903944Z Dec 23 02:54:46 at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
> 2021-12-23T02:54:46.2904712Z Dec 23 02:54:46 at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
> 2021-12-23T02:54:46.2905493Z Dec 23 02:54:46 at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
> 2021-12-23T02:54:46.2906245Z Dec 23 02:54:46 at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
> 2021-12-23T02:54:46.2906968Z Dec 23 02:54:46 at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
> 2021-12-23T02:54:46.2907692Z Dec 23 02:54:46 at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
> 2021-12-23T02:54:46.2908303Z Dec 23 02:54:46 at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
> 2021-12-23T02:54:46.2908971Z Dec 23 02:54:46 at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
> 2021-12-23T02:54:46.2909664Z Dec 23 02:54:46 at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
> 2021-12-23T02:54:46.2910347Z Dec 23 02:54:46 at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:188)
> 2021-12-23T02:54:46.2911042Z Dec 23 02:54:46 at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:154)
> 2021-12-23T02:54:46.2911743Z Dec 23 02:54:46 at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124)
> 2021-12-23T02:54:46.2912399Z Dec 23 02:54:46 at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
> 2021-12-23T02:54:46.2913009Z Dec 23 02:54:46 at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
> 2021-12-23T02:54:46.2913589Z Dec 23 02:54:46 at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
> 2021-12-23T02:54:46.2914162Z Dec 23 02:54:46 at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=28502&view=logs&j=2c3cbe13-dee0-5837-cf47-3053da9a8a78&t=b78d9d30-509a-5cea-1fef-db7abaa325ae&l=14634
> Maybe the test instability is caused by exceeding our available memory on the CI machines by running too many tests concurrently.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)