You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "gortiz (via GitHub)" <gi...@apache.org> on 2024/08/07 10:20:00 UTC
Re: [PR] Improve performance of DataBlock serde [pinot]
gortiz commented on code in PR #13303:
URL: https://github.com/apache/pinot/pull/13303#discussion_r1706758696
##########
pinot-common/src/main/java/org/apache/pinot/common/datablock/BaseDataBlock.java:
##########
@@ -109,92 +114,27 @@ public BaseDataBlock(int numRows, @Nullable DataSchema dataSchema, String[] stri
_numColumns = dataSchema == null ? 0 : dataSchema.size();
_fixDataSize = 0;
_stringDictionary = stringDictionary;
- _fixedSizeDataBytes = fixedSizeDataBytes;
- _fixedSizeData = ByteBuffer.wrap(fixedSizeDataBytes);
- _variableSizeDataBytes = variableSizeDataBytes;
- _variableSizeData = ByteBuffer.wrap(variableSizeDataBytes);
+ _fixedSizeData = PinotByteBuffer.wrap(ByteBuffer.wrap(fixedSizeDataBytes));
+ _variableSizeData = PinotByteBuffer.wrap(ByteBuffer.wrap(variableSizeDataBytes));
_errCodeToExceptionMap = new HashMap<>();
}
- /**
- * Construct empty data table.
- */
- public BaseDataBlock() {
- _numRows = 0;
- _numColumns = 0;
+ public BaseDataBlock(int numRows, DataSchema dataSchema, String[] stringDictionary,
+ DataBuffer fixedSizeData, DataBuffer variableSizeData) {
+ Preconditions.checkArgument(fixedSizeData.size() <= Integer.MAX_VALUE, "Fixed size data too large ({} bytes",
+ fixedSizeData.size());
+ Preconditions.checkArgument(variableSizeData.size() <= Integer.MAX_VALUE, "Variable size data too large ({} bytes",
+ variableSizeData.size());
+ _numRows = numRows;
+ _dataSchema = dataSchema;
+ _numColumns = dataSchema.size();
_fixDataSize = 0;
Review Comment:
That is not easy to do because the value is calculated in columnar and row blocks in `computeBlockObjectConstants`. In order to calculate the value, this method needs some attributes from the super class like the schema.
To be clear, the order is:
1. ColumnarDataBlock and RowDataBlock constructor call super
2. BaseDataBlock constructor initializes some internal attributes reading the byte buffer
3. ColumnarDataBlock and RowDataBlock constructor use the internal attributes to calculate `_fixDataSize`.
What we can do is to remove the field in BaseDataBlock and call an abstract method. I'm going to modify the code to do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org