You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2021/11/30 06:34:59 UTC
[GitHub] [ozone] captainzmc commented on a change in pull request #2866: HDDS-5851. Define a PutBlock/maxBuffer fixed boundary for streaming writes.
captainzmc commented on a change in pull request #2866:
URL: https://github.com/apache/ozone/pull/2866#discussion_r758971760
##########
File path: hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java
##########
@@ -257,13 +257,29 @@ public void write(ByteBuffer b, int off, int len) throws IOException {
if (len == 0) {
return;
}
+ int curLen = len;
+ // set limit on the number of bytes that a ByteBuffer(StreamBuffer) can hold
+ int maxBufferLen = config.getDataStreamMaxBufferSize();
+ while (curLen > 0) {
+ int writeLen = Math.min(curLen, maxBufferLen);
+ final StreamBuffer buf = new StreamBuffer(b, off, writeLen);
+ off += writeLen;
+ bufferList.add(buf);
+ writeChunkToContainer(buf.duplicate());
+ curLen -= writeLen;
+ writtenDataLength += writeLen;
+ doFlushIfNeeded();
+ }
+ }
- final StreamBuffer buf = new StreamBuffer(b, off, len);
- bufferList.add(buf);
-
- writeChunkToContainer(buf.duplicate());
-
- writtenDataLength += len;
+ private void doFlushIfNeeded() throws IOException {
+ Preconditions.checkArgument(config.getDataStreamBufferFlushSize() > config
+ .getDataStreamMaxBufferSize());
+ long boundary = config.getDataStreamBufferFlushSize() / config
+ .getDataStreamMaxBufferSize();
+ if (bufferList.size() % boundary == 0) {
+ executePutBlock(false, false);
Review comment:
Did we need add updateFlushLength(), waitOnFlushFutures() and watchForCommit(false) here? Just as what we did in [handleFlush](https://github.com/apache/ozone/blob/HDDS-4454/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java#L435). Currently we only releaseBuffers after watchForCommit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org