You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Xiaoyu Yao (JIRA)" <ji...@apache.org> on 2017/04/03 15:37:42 UTC
[jira] [Commented] (HDFS-11608) HDFS write crashed in the case of
huge block size
[ https://issues.apache.org/jira/browse/HDFS-11608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15953677#comment-15953677 ]
Xiaoyu Yao commented on HDFS-11608:
-----------------------------------
[~xiaobingo], the patch solved the 2nd overflow issue introduced by HDFS-7308. However, it changes the number of packets and the packet size for large block size with the code below.
{code}
final long psize = Math.min(blockSize - getStreamer().getBytesCurBlock(), dfsClient.getConf().getWritePacketSize());
final int ipsize = (int) Math.min(psize, Integer.MAX_VALUE);
computePacketChunkSize(psize, bytesPerChecksum);
{code}
After this change, a 3 GB block after patch v01 result in only 50 packets with average size of 60 MB. Before this change, a 3 GB block result in 2080930 packets with average size of 150 KB. I vaguely remember that DN has some limit on the maximum packet size (e.g., 16MB?). Can you check and ensure
1) large block size works end-to-end as it was before HDFS-7308?
2) performance is not degraded from /wo HDFS-7308 (2.6.0) to /w HDFS-7308+HDFS-11608.
> HDFS write crashed in the case of huge block size
> -------------------------------------------------
>
> Key: HDFS-11608
> URL: https://issues.apache.org/jira/browse/HDFS-11608
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
> Affects Versions: 2.8.0
> Reporter: Xiaobing Zhou
> Assignee: Xiaobing Zhou
> Priority: Critical
> Attachments: HDFS-11608.000.patch
>
>
> We've seen HDFS write crashes in the case of huge block size. For example, writing a 3G file using 3G block size, HDFS client throws out of memory exception. DataNode gives out IOException. After changing heap size limit, DFSOutputStream ResponseProcessor exception is seen followed by Broken pipe and pipeline recovery.
> Give below:
> DN exception,
> {noformat}
> 2017-03-30 16:34:33,828 ERROR datanode.DataNode (DataXceiver.java:run(278)) - c6401.ambari.apache.org:50010:DataXceiver error processing WRITE_BLOCK operation src: /192.168.64.101:47167 dst: /192.168.64.101:50010
> java.io.IOException: Incorrect value for packet payload size: 2147483128
> at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:159)
> at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:502)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:898)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:806)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:251)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org