You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by “岁月飞扬” <16...@qq.com> on 2023/04/21 09:43:19 UTC
flink task submits error to k8s
Hello!
When we use flink version 1.14 to commit tasks to kubernetes, we get an error using Session mode. The following error is reported some time after the task is submitted .
2023-04-18 17:23:56,818 INFO org.apache.flink.client.program.rest.RestClusterClient [] - Submitting job 'FlinkStreamApp_Deploy_LPYZZM_0418' (0a5b3a4cc3c1957f4a26d8ce9dcbdec9).
2023-04-18 17:23:56,833 DEBUG org.apache.flink.runtime.rest.RestClient [] - Sending request of class class org.apache.flink.runtime.rest.messages.job.JobSubmitRequestBody to 10.213.116.48:31181/v1/jobs
2023-04-18 17:23:56,934 DEBUG org.apache.flink.shaded.netty4.io.netty.util.ResourceLeakDetector [] - -Dorg.apache.flink.shaded.netty4.io.netty.leakDetection.level: simple
2023-04-18 17:23:56,934 DEBUG org.apache.flink.shaded.netty4.io.netty.util.ResourceLeakDetector [] - -Dorg.apache.flink.shaded.netty4.io.netty.leakDetection.targetRecords: 4
2023-04-18 17:23:56,948 DEBUG org.apache.flink.shaded.netty4.io.netty.buffer.AbstractByteBuf [] - -Dorg.apache.flink.shaded.netty4.io.netty.buffer.checkAccessible: true
2023-04-18 17:23:56,948 DEBUG org.apache.flink.shaded.netty4.io.netty.buffer.AbstractByteBuf [] - -Dorg.apache.flink.shaded.netty4.io.netty.buffer.checkBounds: true
2023-04-18 17:23:56,950 DEBUG org.apache.flink.shaded.netty4.io.netty.util.ResourceLeakDetectorFactory [] - Loaded default ResourceLeakDetector: org.apache.flink.shaded.netty4.io.netty.util.ResourceLeakDetector@263c66c9
2023-04-18 17:23:57,003 DEBUG org.apache.flink.shaded.netty4.io.netty.buffer.PooledByteBufAllocator [] - -Dio.netty.allocator.numHeapArenas: 2
2023-04-18 17:23:57,004 DEBUG org.apache.flink.shaded.netty4.io.netty.buffer.PooledByteBufAllocator [] - -Dio.netty.allocator.numDirectArenas: 2
2023-04-18 17:23:57,004 DEBUG org.apache.flink.shaded.netty4.io.netty.buffer.PooledByteBufAllocator [] - -Dio.netty.allocator.pageSize: 8192
2023-04-18 17:23:57,004 DEBUG org.apache.flink.shaded.netty4.io.netty.buffer.PooledByteBufAllocator [] - -Dio.netty.allocator.maxOrder: 11
2023-04-18 17:23:57,004 DEBUG org.apache.flink.shaded.netty4.io.netty.buffer.PooledByteBufAllocator [] - -Dio.netty.allocator.chunkSize: 16777216
2023-04-18 17:23:57,004 DEBUG org.apache.flink.shaded.netty4.io.netty.buffer.PooledByteBufAllocator [] - -Dio.netty.allocator.smallCacheSize: 256
2023-04-18 17:23:57,004 DEBUG org.apache.flink.shaded.netty4.io.netty.buffer.PooledByteBufAllocator [] - -Dio.netty.allocator.normalCacheSize: 64
2023-04-18 17:23:57,004 DEBUG org.apache.flink.shaded.netty4.io.netty.buffer.PooledByteBufAllocator [] - -Dio.netty.allocator.maxCachedBufferCapacity: 32768
2023-04-18 17:23:57,004 DEBUG org.apache.flink.shaded.netty4.io.netty.buffer.PooledByteBufAllocator [] - -Dio.netty.allocator.cacheTrimInterval: 8192
2023-04-18 17:23:57,004 DEBUG org.apache.flink.shaded.netty4.io.netty.buffer.PooledByteBufAllocator [] - -Dio.netty.allocator.cacheTrimIntervalMillis: 0
2023-04-18 17:23:57,004 DEBUG org.apache.flink.shaded.netty4.io.netty.buffer.PooledByteBufAllocator [] - -Dio.netty.allocator.useCacheForAllThreads: true
2023-04-18 17:23:57,004 DEBUG org.apache.flink.shaded.netty4.io.netty.buffer.PooledByteBufAllocator [] - -Dio.netty.allocator.maxCachedByteBuffersPerChunk: 1023
2023-04-18 17:23:57,018 DEBUG org.apache.flink.shaded.netty4.io.netty.buffer.ByteBufUtil [] - -Dio.netty.allocator.type: pooled
2023-04-18 17:23:57,018 DEBUG org.apache.flink.shaded.netty4.io.netty.buffer.ByteBufUtil [] - -Dio.netty.threadLocalDirectBufferSize: 0
2023-04-18 17:23:57,018 DEBUG org.apache.flink.shaded.netty4.io.netty.buffer.ByteBufUtil [] - -Dio.netty.maxThreadLocalCharBufferSize: 16384
2023-04-18 17:23:57,048 DEBUG org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelId [] - -Dio.netty.processId: 3096 (auto-detected)
2023-04-18 17:23:57,051 DEBUG org.apache.flink.shaded.netty4.io.netty.util.NetUtil [] - -Djava.net.preferIPv4Stack: false
2023-04-18 17:23:57,052 DEBUG org.apache.flink.shaded.netty4.io.netty.util.NetUtil [] - -Djava.net.preferIPv6Addresses: false
2023-04-18 17:23:57,056 DEBUG org.apache.flink.shaded.netty4.io.netty.util.NetUtilInitializations [] - Loopback interface: lo (lo, 127.0.0.1)
2023-04-18 17:23:57,057 DEBUG org.apache.flink.shaded.netty4.io.netty.util.NetUtil [] - /proc/sys/net/core/somaxconn: 128
2023-04-18 17:23:57,059 DEBUG org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelId [] - -Dio.netty.machineId: 66:b9:79:ff:fe:8c:b9:d3 (auto-detected)
2023-04-18 17:23:57,175 DEBUG org.apache.flink.shaded.netty4.io.netty.util.Recycler [] - -Dio.netty.recycler.maxCapacityPerThread: 4096
2023-04-18 17:23:57,176 DEBUG org.apache.flink.shaded.netty4.io.netty.util.Recycler [] - -Dio.netty.recycler.maxSharedCapacityFactor: 2
2023-04-18 17:23:57,176 DEBUG org.apache.flink.shaded.netty4.io.netty.util.Recycler [] - -Dio.netty.recycler.linkCapacity: 16
2023-04-18 17:23:57,176 DEBUG org.apache.flink.shaded.netty4.io.netty.util.Recycler [] - -Dio.netty.recycler.ratio: 8
2023-04-18 17:23:57,176 DEBUG org.apache.flink.shaded.netty4.io.netty.util.Recycler [] - -Dio.netty.recycler.delayedQueue.ratio: 8
java.util.concurrent.CompletionException: org.apache.flink.runtime.rest.ConnectionIdleException: Channel became idle.
at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:326) ~[?:1.8.0_342]
at java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:338) ~[?:1.8.0_342]
at java.util.concurrent.CompletableFuture.uniRelay(CompletableFuture.java:925) ~[?:1.8.0_342]
at java.util.concurrent.CompletableFuture$UniRelay.tryFire(CompletableFuture.java:913) ~[?:1.8.0_342]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) [?:1.8.0_342]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) [?:1.8.0_342]
at org.apache.flink.runtime.rest.RestClient$ClientHandler.userEventTriggered(RestClient.java:642) [flink-dist_2.11-1.14.4.1.jar:1.14.4.1]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) [flink-dist_2.11-1.14.4.1.jar:1.14.4.1]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) [flink-dist_2.11-1.14.4.1.jar:1.14.4.1]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) [flink-dist_2.11-1.14.4.1.jar:1.14.4.1]
at org.apache.flink.shaded.netty4.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) [flink-dist_2.11-1.14.4.1.jar:1.14.4.1]
at org.apache.flink.shaded.netty4.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) [flink-dist_2.11-1.14.4.1.jar:1.14.4.1]
at org.apache.flink.shaded.netty4.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) [flink-dist_2.11-1.14.4.1.jar:1.14.4.1]
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) [flink-dist_2.11-1.14.4.1.jar:1.14.4.1]
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) [flink-dist_2.11-1.14.4.1.jar:1.14.4.1]
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [flink-dist_2.11-1.14.4.1.jar:1.14.4.1]
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [flink-dist_2.11-1.14.4.1.jar:1.14.4.1]
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [flink-dist_2.11-1.14.4.1.jar:1.14.4.1]
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [flink-dist_2.11-1.14.4.1.jar:1.14.4.1]
at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [flink-dist_2.11-1.14.4.1.jar:1.14.4.1]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_342]
Caused by: org.apache.flink.runtime.rest.ConnectionIdleException: Channel became idle.
... 15 more
2023-04-18 17:24:09,155 DEBUG org.apache.flink.runtime.rest.RestClient [] - Sending request of class class org.apache.flink.runtime.rest.messages.job.JobSubmitRequestBody to 10.213.116.48:31181/v1/jobs
“岁月飞扬”
1639467162@qq.com
Re: flink task submits error to k8s
Posted by Weihua Hu <hu...@gmail.com>.
Hi,
It's a DEBUG level log.
Does this submission failed?
Best,
Weihua
On Fri, Apr 21, 2023 at 5:44 PM “岁月飞扬” <16...@qq.com> wrote:
>
> Hello!
>
> When we use flink version 1.14 to commit tasks to kubernetes, we get an
> error using Session mode. The following error is reported some time after
> the task is submitted .
>
> 2023-04-18 17:23:56,818 INFO
> org.apache.flink.client.program.rest.RestClusterClient [] -
> Submitting job 'FlinkStreamApp_Deploy_LPYZZM_0418'
> (0a5b3a4cc3c1957f4a26d8ce9dcbdec9).
> 2023-04-18 17:23:56,833 DEBUG org.apache.flink.runtime.rest.RestClient
> [] - Sending request of class class
> org.apache.flink.runtime.rest.messages.job.JobSubmitRequestBody to
> 10.213.116.48:31181/v1/jobs
> 2023-04-18 17:23:56,934 DEBUG
> org.apache.flink.shaded.netty4.io.netty.util.ResourceLeakDetector [] -
> -Dorg.apache.flink.shaded.netty4.io.netty.leakDetection.level: simple
> 2023-04-18 17:23:56,934 DEBUG
> org.apache.flink.shaded.netty4.io.netty.util.ResourceLeakDetector [] -
> -Dorg.apache.flink.shaded.netty4.io.netty.leakDetection.targetRecords: 4
> 2023-04-18 17:23:56,948 DEBUG
> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractByteBuf [] -
> -Dorg.apache.flink.shaded.netty4.io.netty.buffer.checkAccessible: true
> 2023-04-18 17:23:56,948 DEBUG
> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractByteBuf [] -
> -Dorg.apache.flink.shaded.netty4.io.netty.buffer.checkBounds: true
> 2023-04-18 17:23:56,950 DEBUG
> org.apache.flink.shaded.netty4.io.netty.util.ResourceLeakDetectorFactory []
> - Loaded default ResourceLeakDetector:
> org.apache.flink.shaded.netty4.io.netty.util.ResourceLeakDetector@263c66c9
> 2023-04-18 17:23:57,003 DEBUG
> org.apache.flink.shaded.netty4.io.netty.buffer.PooledByteBufAllocator [] -
> -Dio.netty.allocator.numHeapArenas: 2
> 2023-04-18 17:23:57,004 DEBUG
> org.apache.flink.shaded.netty4.io.netty.buffer.PooledByteBufAllocator [] -
> -Dio.netty.allocator.numDirectArenas: 2
> 2023-04-18 17:23:57,004 DEBUG
> org.apache.flink.shaded.netty4.io.netty.buffer.PooledByteBufAllocator [] -
> -Dio.netty.allocator.pageSize: 8192
> 2023-04-18 17:23:57,004 DEBUG
> org.apache.flink.shaded.netty4.io.netty.buffer.PooledByteBufAllocator [] -
> -Dio.netty.allocator.maxOrder: 11
> 2023-04-18 17:23:57,004 DEBUG
> org.apache.flink.shaded.netty4.io.netty.buffer.PooledByteBufAllocator [] -
> -Dio.netty.allocator.chunkSize: 16777216
> 2023-04-18 17:23:57,004 DEBUG
> org.apache.flink.shaded.netty4.io.netty.buffer.PooledByteBufAllocator [] -
> -Dio.netty.allocator.smallCacheSize: 256
> 2023-04-18 17:23:57,004 DEBUG
> org.apache.flink.shaded.netty4.io.netty.buffer.PooledByteBufAllocator [] -
> -Dio.netty.allocator.normalCacheSize: 64
> 2023-04-18 17:23:57,004 DEBUG
> org.apache.flink.shaded.netty4.io.netty.buffer.PooledByteBufAllocator [] -
> -Dio.netty.allocator.maxCachedBufferCapacity: 32768
> 2023-04-18 17:23:57,004 DEBUG
> org.apache.flink.shaded.netty4.io.netty.buffer.PooledByteBufAllocator [] -
> -Dio.netty.allocator.cacheTrimInterval: 8192
> 2023-04-18 17:23:57,004 DEBUG
> org.apache.flink.shaded.netty4.io.netty.buffer.PooledByteBufAllocator [] -
> -Dio.netty.allocator.cacheTrimIntervalMillis: 0
> 2023-04-18 17:23:57,004 DEBUG
> org.apache.flink.shaded.netty4.io.netty.buffer.PooledByteBufAllocator [] -
> -Dio.netty.allocator.useCacheForAllThreads: true
> 2023-04-18 17:23:57,004 DEBUG
> org.apache.flink.shaded.netty4.io.netty.buffer.PooledByteBufAllocator [] -
> -Dio.netty.allocator.maxCachedByteBuffersPerChunk: 1023
> 2023-04-18 17:23:57,018 DEBUG
> org.apache.flink.shaded.netty4.io.netty.buffer.ByteBufUtil [] -
> -Dio.netty.allocator.type: pooled
> 2023-04-18 17:23:57,018 DEBUG
> org.apache.flink.shaded.netty4.io.netty.buffer.ByteBufUtil [] -
> -Dio.netty.threadLocalDirectBufferSize: 0
> 2023-04-18 17:23:57,018 DEBUG
> org.apache.flink.shaded.netty4.io.netty.buffer.ByteBufUtil [] -
> -Dio.netty.maxThreadLocalCharBufferSize: 16384
> 2023-04-18 17:23:57,048 DEBUG
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelId [] -
> -Dio.netty.processId: 3096 (auto-detected)
> 2023-04-18 17:23:57,051 DEBUG
> org.apache.flink.shaded.netty4.io.netty.util.NetUtil [] -
> -Djava.net.preferIPv4Stack: false
> 2023-04-18 17:23:57,052 DEBUG
> org.apache.flink.shaded.netty4.io.netty.util.NetUtil [] -
> -Djava.net.preferIPv6Addresses: false
> 2023-04-18 17:23:57,056 DEBUG
> org.apache.flink.shaded.netty4.io.netty.util.NetUtilInitializations [] -
> Loopback interface: lo (lo, 127.0.0.1)
> 2023-04-18 17:23:57,057 DEBUG
> org.apache.flink.shaded.netty4.io.netty.util.NetUtil [] -
> /proc/sys/net/core/somaxconn: 128
> 2023-04-18 17:23:57,059 DEBUG
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelId [] -
> -Dio.netty.machineId: 66:b9:79:ff:fe:8c:b9:d3 (auto-detected)
> 2023-04-18 17:23:57,175 DEBUG
> org.apache.flink.shaded.netty4.io.netty.util.Recycler [] -
> -Dio.netty.recycler.maxCapacityPerThread: 4096
> 2023-04-18 17:23:57,176 DEBUG
> org.apache.flink.shaded.netty4.io.netty.util.Recycler [] -
> -Dio.netty.recycler.maxSharedCapacityFactor: 2
> 2023-04-18 17:23:57,176 DEBUG
> org.apache.flink.shaded.netty4.io.netty.util.Recycler [] -
> -Dio.netty.recycler.linkCapacity: 16
> 2023-04-18 17:23:57,176 DEBUG
> org.apache.flink.shaded.netty4.io.netty.util.Recycler [] -
> -Dio.netty.recycler.ratio: 8
> 2023-04-18 17:23:57,176 DEBUG
> org.apache.flink.shaded.netty4.io.netty.util.Recycler [] -
> -Dio.netty.recycler.delayedQueue.ratio: 8
> java.util.concurrent.CompletionException:
> org.apache.flink.runtime.rest.ConnectionIdleException: Channel became idle.
> at
> java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:326)
> ~[?:1.8.0_342]
> at
> java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:338)
> ~[?:1.8.0_342]
> at
> java.util.concurrent.CompletableFuture.uniRelay(CompletableFuture.java:925)
> ~[?:1.8.0_342]
> at
> java.util.concurrent.CompletableFuture$UniRelay.tryFire(CompletableFuture.java:913)
> ~[?:1.8.0_342]
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> [?:1.8.0_342]
> at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
> [?:1.8.0_342]
> at
> org.apache.flink.runtime.rest.RestClient$ClientHandler.userEventTriggered(RestClient.java:642)
> [flink-dist_2.11-1.14.4.1.jar:1.14.4.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346)
> [flink-dist_2.11-1.14.4.1.jar:1.14.4.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332)
> [flink-dist_2.11-1.14.4.1.jar:1.14.4.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324)
> [flink-dist_2.11-1.14.4.1.jar:1.14.4.1]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371)
> [flink-dist_2.11-1.14.4.1.jar:1.14.4.1]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504)
> [flink-dist_2.11-1.14.4.1.jar:1.14.4.1]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476)
> [flink-dist_2.11-1.14.4.1.jar:1.14.4.1]
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
> [flink-dist_2.11-1.14.4.1.jar:1.14.4.1]
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170)
> [flink-dist_2.11-1.14.4.1.jar:1.14.4.1]
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
> [flink-dist_2.11-1.14.4.1.jar:1.14.4.1]
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
> [flink-dist_2.11-1.14.4.1.jar:1.14.4.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
> [flink-dist_2.11-1.14.4.1.jar:1.14.4.1]
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> [flink-dist_2.11-1.14.4.1.jar:1.14.4.1]
> at
> org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> [flink-dist_2.11-1.14.4.1.jar:1.14.4.1]
> at java.lang.Thread.run(Thread.java:750) [?:1.8.0_342]
> Caused by: org.apache.flink.runtime.rest.ConnectionIdleException: Channel
> became idle.
> ... 15 more
> 2023-04-18 17:24:09,155 DEBUG org.apache.flink.runtime.rest.RestClient
> [] - Sending request of class class
> org.apache.flink.runtime.rest.messages.job.JobSubmitRequestBody to
> 10.213.116.48:31181/v1/jobs
>
> ------------------------------
> “岁月飞扬”
> 1639467162@qq.com
>
> <https://wx.mail.qq.com/home/index?t=readmail_businesscard_midpage&nocheck=true&name=%E2%80%9C%E5%B2%81%E6%9C%88%E9%A3%9E%E6%89%AC%E2%80%9D&icon=https%3A%2F%2Fthirdqq.qlogo.cn%2Fg%3Fb%3Doidb%26k%3DWAwvXrzZM7y5iapqr0vpluQ%26s%3D0&mail=1639467162%40qq.com&code=OON-DNpqZvq-tbQ3-MyaGpguj0PstJKkf5Tk8s-l--bG7YaaLxcRGXCMOF68i-WzwdAWGAxlMVK1ma8bSPR1lQ>
>
>