You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Dian Fu (Jira)" <ji...@apache.org> on 2020/05/17 14:11:00 UTC
[jira] [Comment Edited] (FLINK-17771) "PyFlink end-to-end test"
fails with "The output result: [] is not as expected: [2, 3, 4]!" on Java11
[ https://issues.apache.org/jira/browse/FLINK-17771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109491#comment-17109491 ]
Dian Fu edited comment on FLINK-17771 at 5/17/20, 2:10 PM:
-----------------------------------------------------------
When I tried to reproduce this problem locally with JDK 11, it will throw exceptions as following:
{code}
Exception in thread "grpc-default-executor-0" java.lang.OutOfMemoryError: Direct buffer memory
at java.base/java.nio.Bits.reserveMemory(Bits.java:175)
at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118)
at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317)
at org.apache.beam.vendor.grpc.v1p21p0.io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:777)
at org.apache.beam.vendor.grpc.v1p21p0.io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:753)
at org.apache.beam.vendor.grpc.v1p21p0.io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:250)
at org.apache.beam.vendor.grpc.v1p21p0.io.netty.buffer.PoolArena.allocate(PoolArena.java:220)
at org.apache.beam.vendor.grpc.v1p21p0.io.netty.buffer.PoolArena.allocate(PoolArena.java:152)
at org.apache.beam.vendor.grpc.v1p21p0.io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:332)
at org.apache.beam.vendor.grpc.v1p21p0.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
at org.apache.beam.vendor.grpc.v1p21p0.io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:123)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.netty.NettyWritableBufferAllocator.allocate(NettyWritableBufferAllocator.java:51)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.MessageFramer.writeKnownLengthUncompressed(MessageFramer.java:226)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.MessageFramer.writeUncompressed(MessageFramer.java:167)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.MessageFramer.writePayload(MessageFramer.java:140)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.AbstractStream.writeMessage(AbstractStream.java:53)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.ServerCallImpl.sendMessageInternal(ServerCallImpl.java:163)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.ServerCallImpl.sendMessage(ServerCallImpl.java:145)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onNext(ServerCalls.java:349)
at org.apache.beam.runners.fnexecution.provisioning.StaticGrpcProvisionService.getProvisionInfo(StaticGrpcProvisionService.java:48)
at org.apache.beam.model.fnexecution.v1.ProvisionServiceGrpc$MethodHandlers.invoke(ProvisionServiceGrpc.java:246)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Contexts$ContextualizedServerCallListener.onHalfClose(Contexts.java:86)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:322)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:762)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
{code}
I guess this is the reason why the result is empty as it encounters OutOfMemoryError during running. (It could succeed when switching to JDK 8.)
was (Author: dian.fu):
When I tried to reproduce this problem locally with JDK 11, it will throw exceptions as following:
{code}
Exception in thread "grpc-default-executor-0" java.lang.OutOfMemoryError: Direct buffer memory
at java.base/java.nio.Bits.reserveMemory(Bits.java:175)
at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118)
at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317)
at org.apache.beam.vendor.grpc.v1p21p0.io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:777)
at org.apache.beam.vendor.grpc.v1p21p0.io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:753)
at org.apache.beam.vendor.grpc.v1p21p0.io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:250)
at org.apache.beam.vendor.grpc.v1p21p0.io.netty.buffer.PoolArena.allocate(PoolArena.java:220)
at org.apache.beam.vendor.grpc.v1p21p0.io.netty.buffer.PoolArena.allocate(PoolArena.java:152)
at org.apache.beam.vendor.grpc.v1p21p0.io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:332)
at org.apache.beam.vendor.grpc.v1p21p0.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
at org.apache.beam.vendor.grpc.v1p21p0.io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:123)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.netty.NettyWritableBufferAllocator.allocate(NettyWritableBufferAllocator.java:51)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.MessageFramer.writeKnownLengthUncompressed(MessageFramer.java:226)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.MessageFramer.writeUncompressed(MessageFramer.java:167)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.MessageFramer.writePayload(MessageFramer.java:140)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.AbstractStream.writeMessage(AbstractStream.java:53)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.ServerCallImpl.sendMessageInternal(ServerCallImpl.java:163)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.ServerCallImpl.sendMessage(ServerCallImpl.java:145)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onNext(ServerCalls.java:349)
at org.apache.beam.runners.fnexecution.provisioning.StaticGrpcProvisionService.getProvisionInfo(StaticGrpcProvisionService.java:48)
at org.apache.beam.model.fnexecution.v1.ProvisionServiceGrpc$MethodHandlers.invoke(ProvisionServiceGrpc.java:246)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Contexts$ContextualizedServerCallListener.onHalfClose(Contexts.java:86)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:322)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:762)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
{code}
I guess this is the reason why the result is empty as it encounters OutOfMemoryError during running.
> "PyFlink end-to-end test" fails with "The output result: [] is not as expected: [2, 3, 4]!" on Java11
> -----------------------------------------------------------------------------------------------------
>
> Key: FLINK-17771
> URL: https://issues.apache.org/jira/browse/FLINK-17771
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Tests
> Affects Versions: 1.11.0
> Reporter: Robert Metzger
> Priority: Major
> Labels: test-stability
> Fix For: 1.11.0
>
>
> Java 11 nightly profile: https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=1579&view=logs&j=6caf31d6-847a-526e-9624-468e053467d6&t=679407b1-ea2c-5965-2c8d-1467777fff88
> {code}
> Job has been submitted with JobID ef78030becb3bfd6415d3de2e06420b4
> java.lang.AssertionError: The output result: [] is not as expected: [2, 3, 4]!
> at org.apache.flink.python.tests.FlinkStreamPythonUdfSqlJob.main(FlinkStreamPythonUdfSqlJob.java:55)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:288)
> at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198)
> at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:148)
> at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:689)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:227)
> at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:906)
> at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:982)
> at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:982)
> Stopping taskexecutor daemon (pid: 2705) on host fv-az670.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)