[jira] [Commented] (HBASE-15709) Handle large edits for asynchronous WAL

["Duo Zhang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-15709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596763#comment-15596763 ] 

Duo Zhang commented on HBASE-15709:
-----------------------------------

I can add the support of entries larger than 16M, as you said, split it into several chunks. But here comes another limit, the block size. Our fan out output is 'one block'...

So there will always be a size limit for AsyncFSWAL. 16M or 256M. And the two WALs will break our serial replication feature. So if we decide to use AsyncFSWAL then we must reject the large edits...

If you guys think 16M is too small and 256M is OK then I could try to add chunking support for AsyncFSWAL.

Thanks.

> Handle large edits for asynchronous WAL
> ---------------------------------------
>
>                 Key: HBASE-15709
>                 URL: https://issues.apache.org/jira/browse/HBASE-15709
>             Project: HBase
>          Issue Type: Sub-task
>          Components: io, wal
>            Reporter: Duo Zhang
>            Priority: Critical
>
> First, FanOutOneBlockAsyncDFSOutput can not work if the buffered data is larger than PacketReceiver.MAX_PACKET_SIZE(16MB).
> Second, since we only allow one block here, we need to make sure we do not exceed the block size after writing a large chunk of data.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)