[GitHub] [incubator-pinot] sunithabeeram commented on a change in pull request #4127: Declare byte array size when initializing ByteArrayOutputStream

[GitBox <gi...@apache.org>]

sunithabeeram commented on a change in pull request #4127: Declare byte array size when initializing ByteArrayOutputStream
URL: https://github.com/apache/incubator-pinot/pull/4127#discussion_r276315910
 
 

 ##########
 File path: pinot-core/src/main/java/org/apache/pinot/core/common/datatable/DataTableImplV2.java
 ##########
 @@ -346,6 +354,17 @@ public void addException(@Nonnull ProcessingException processingException) {
     return byteArrayOutputStream.toByteArray();
   }
 
+  private int getTotalByteArraySize(byte[] metadataBytes, byte[] dictionaryMapBytes, byte[] dataSchemaBytes) {
+    int metadataSize = metadataBytes.length + 1;
+    int dictionarySize = dictionaryMapBytes != null ? dictionaryMapBytes.length + 1 : 2;
+    int dataSchemaSize = dataSchemaBytes != null ? dataSchemaBytes.length + 1 : 2;
+    int fixedSizeDataSize = _fixedSizeDataBytes != null ? _fixedSizeDataBytes.length + 1 : 2;
+    int variableSizeDataSize = _variableSizeDataBytes != null ? _variableSizeDataBytes.length + 1 : 2;
+
+    // total size that byte array needs. 3 denotes version, number of rows and columns.
+    return 3 + metadataSize + dictionarySize + dataSchemaSize + fixedSizeDataSize + variableSizeDataSize;
 
 Review comment:
   Could we use static constants instead of magic numbers above? Its not immediately clear why something is a 2 vs 1 etc. Looking through the actual serialization gives a clue, but thats not evident here.
   
   I am assuming that even if the estimate here is off, the buffer will still get resized and we'll not encounter issues. If so, do the estimates need to be precise? Also, would it be an option to use a thread-local BAOS so the object is not allocated everytime?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org