[GitHub] [rocketmq] shengminw opened a new issue, #4468: Optimize broker buffer length initialization

[GitBox <gi...@apache.org>]

shengminw opened a new issue, #4468:
URL: https://github.com/apache/rocketmq/issues/4468

   ## Optimize broker buffer length initialization
   
   The format of the message written to the CommitLog,  except for the message body, also includes many other information.  
   
   CommitLog storage format：
   
   ```java
   // 1 TOTALSIZE
   this.byteBuf.writeInt(msgLen);
   // 2 MAGICCODE
   this.byteBuf.writeInt(CommitLog.MESSAGE_MAGIC_CODE);
   // 3 BODYCRC
   this.byteBuf.writeInt(msgInner.getBodyCRC());
   // 4 QUEUEID
   this.byteBuf.writeInt(msgInner.getQueueId());
   // 5 FLAG
   this.byteBuf.writeInt(msgInner.getFlag());
   // 6 QUEUEOFFSET, need update later
   this.byteBuf.writeLong(0);
   // 7 PHYSICALOFFSET, need update later
   this.byteBuf.writeLong(0);
   // 8 SYSFLAG
   this.byteBuf.writeInt(msgInner.getSysFlag());
   // 9 BORNTIMESTAMP
   this.byteBuf.writeLong(msgInner.getBornTimestamp());
   
   // 10 BORNHOST
   ByteBuffer bornHostBytes = msgInner.getBornHostBytes();
   this.byteBuf.writeBytes(bornHostBytes.array());
   
   // 11 STORETIMESTAMP
   this.byteBuf.writeLong(msgInner.getStoreTimestamp());
   
   // 12 STOREHOSTADDRESS
   ByteBuffer storeHostBytes = msgInner.getStoreHostBytes();
   this.byteBuf.writeBytes(storeHostBytes.array());
   
   // 13 RECONSUMETIMES
   this.byteBuf.writeInt(msgInner.getReconsumeTimes());
   // 14 Prepared Transaction Offset
   this.byteBuf.writeLong(msgInner.getPreparedTransactionOffset());
   // 15 BODY
   this.byteBuf.writeInt(bodyLength);
   if (bodyLength > 0)
       this.byteBuf.writeBytes(msgInner.getBody());
   // 16 TOPIC
   this.byteBuf.writeByte((byte) topicLength);
   this.byteBuf.writeBytes(topicData);
   // 17 PROPERTIES
   this.byteBuf.writeShort((short) propertiesLength);
   if (propertiesLength > 0)
       this.byteBuf.writeBytes(propertiesData);
   
   ```
   
   However, the length of the encdoer buffer is equal to the maximum length of the message body, which may cause the buffer to exceed the bounds during messageBodySize is on the boundry of maxMessageBodySize.
   
   ```java
   private final ByteBuf byteBuf;
   private final int maxMessageBodySize;
   
   MessageExtEncoder(final int maxMessageBodySize) {
       ByteBufAllocator alloc = UnpooledByteBufAllocator.DEFAULT;
       byteBuf = alloc.directBuffer(maxMessageBodySize);
       this.maxMessageBodySize = maxMessageBodySize;
   }
   ```
   
   I think the initial buffer length needs to be extended to ensure robustness when sending long messages.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] lizhanhui closed issue #4468: Optimize broker buffer length initialization

[GitBox <gi...@apache.org>]

lizhanhui closed issue #4468: Optimize broker buffer length initialization
URL: https://github.com/apache/rocketmq/issues/4468


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] ni-ze commented on issue #4468: Optimize broker buffer length initialization

[GitBox <gi...@apache.org>]

ni-ze commented on issue #4468:
URL: https://github.com/apache/rocketmq/issues/4468#issuecomment-1158400006

   Thanks for noticing this detail.
   But this ByteBufAllocator.directBuffer is a method of Netty, ByteBuf is a netty class instead of java NIO, which means it is a Scalable byteBuf.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] lizhanhui commented on issue #4468: Optimize broker buffer length initialization

[GitBox <gi...@apache.org>]

lizhanhui commented on issue #4468:
URL: https://github.com/apache/rocketmq/issues/4468#issuecomment-1165344826

   Close this issue as the associated pull request has been merged.
   https://github.com/apache/rocketmq/pull/4469


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org