You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user-zh@flink.apache.org by xiao cai <fl...@163.com> on 2021/11/08 14:20:21 UTC

flink1.12.4 写入hdfs报错 java.lang.OutOfMemoryError: Direct buffer memory

通过flink 1.12.4 streaming file sink 写入hdfs，运行过程中抛出以下异常：


2021-11-08 20:39:05
java.io.IOException: java.lang.OutOfMemoryError: Direct buffer memory
    at org.apache.hadoop.hdfs.DataStreamer$LastExceptionInStreamer.set(DataStreamer.java:299)
    at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:820)
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
    at java.nio.Bits.reserveMemory(Bits.java:694)
    at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
    at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
    at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:241)
    at sun.nio.ch.IOUtil.write(IOUtil.java:58)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:468)
    at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)
    at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
    at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
    at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
    at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
    at java.io.DataOutputStream.write(DataOutputStream.java:107)
    at org.apache.hadoop.hdfs.DFSPacket.writeTo(DFSPacket.java:180)
    at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:765)

Re: flink1.12.4 写入hdfs报错 java.lang.OutOfMemoryError: Direct buffer memory

Posted by Caizhi Weng <ts...@gmail.com>.

Hi！

可以通过配置 taskmanager.memory.task.off-heap.size 指定 direct memory 和 native
memory 的大小，详见 [1]。

[1]
https://nightlies.apache.org/flink/flink-docs-master/zh/docs/deployment/memory/mem_setup_tm/#%e9%85%8d%e7%bd%ae%e5%a0%86%e5%a4%96%e5%86%85%e5%ad%98%e7%9b%b4%e6%8e%a5%e5%86%85%e5%ad%98%e6%88%96%e6%9c%ac%e5%9c%b0%e5%86%85%e5%ad%98

xiao cai <fl...@163.com> 于2021年11月8日周一 下午10:20写道：

> 通过flink 1.12.4 streaming file sink 写入hdfs，运行过程中抛出以下异常：
>
>
> 2021-11-08 20:39:05
> java.io.IOException: java.lang.OutOfMemoryError: Direct buffer memory
>     at
> org.apache.hadoop.hdfs.DataStreamer$LastExceptionInStreamer.set(DataStreamer.java:299)
>     at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:820)
> Caused by: java.lang.OutOfMemoryError: Direct buffer memory
>     at java.nio.Bits.reserveMemory(Bits.java:694)
>     at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
>     at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
>     at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:241)
>     at sun.nio.ch.IOUtil.write(IOUtil.java:58)
>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:468)
>     at org.apache.hadoop.net
> .SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)
>     at org.apache.hadoop.net
> .SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>     at org.apache.hadoop.net
> .SocketOutputStream.write(SocketOutputStream.java:159)
>     at org.apache.hadoop.net
> .SocketOutputStream.write(SocketOutputStream.java:117)
>     at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
>     at java.io.DataOutputStream.write(DataOutputStream.java:107)
>     at org.apache.hadoop.hdfs.DFSPacket.writeTo(DFSPacket.java:180)
>     at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:765)