You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/05/19 01:40:04 UTC
[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #5400: Do not release the PinotDataBuffer when closing the index
mcvsubbu commented on a change in pull request #5400:
URL: https://github.com/apache/incubator-pinot/pull/5400#discussion_r426973388
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/v1/BaseChunkSingleValueReader.java
##########
@@ -75,7 +67,7 @@ public BaseChunkSingleValueReader(PinotDataBuffer pinotDataBuffer) {
headerOffset += Integer.BYTES;
int dataHeaderStart = headerOffset;
- if (_version > 1) {
+ if (version > 1) {
Review comment:
The version was made into a member variable so that when we add or bump the version it is available to derived classes. Not sure why you removed it.
Recently we made a version bump and it was painful and re-factoring had to be done. Some background in PR #5285
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/io/readerwriter/impl/FixedByteSingleColumnSingleValueReaderWriter.java
##########
@@ -176,18 +173,15 @@ private int getBufferId(int row) {
private void addBuffer() {
LOGGER.info("Allocating {} bytes for: {}", _chunkSizeInBytes, _allocationContext);
+ // NOTE: PinotDataBuffer is tracked in the PinotDataBufferMemoryManager. No need to track it inside the class.
PinotDataBuffer buffer = _memoryManager.allocate(_chunkSizeInBytes, _allocationContext);
- _dataBuffers.add(buffer);
+ _writers.add(
Review comment:
when are these closed?
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/io/readerwriter/impl/FixedByteSingleColumnMultiValueReaderWriter.java
##########
@@ -132,47 +129,40 @@ public FixedByteSingleColumnMultiValueReaderWriter(int maxNumberOfMultiValuesPer
_maxNumberOfMultiValuesPerRow = maxNumberOfMultiValuesPerRow;
_headerSize = rowCountPerChunk * SIZE_OF_INT * NUM_COLS_IN_HEADER;
_rowCountPerChunk = rowCountPerChunk;
- addHeaderBuffers();
+ addHeaderBuffer();
//at least create space for million entries, which for INT translates into 4mb buffer
_incrementalCapacity = incrementalCapacity;
- addDataBuffers(initialCapacity);
+ addDataBuffer(initialCapacity);
//init(_rowCountPerChunk, _columnSizeInBytes, _maxNumberOfMultiValuesPerRow, initialCapacity, _incrementalCapacity);
}
- private void addHeaderBuffers() {
+ private void addHeaderBuffer() {
LOGGER.info("Allocating header buffer of size {} for: {}", _headerSize, _context);
- _headerBuffer = _memoryManager.allocate(_headerSize, _context);
+ // NOTE: PinotDataBuffer is tracked in the PinotDataBufferMemoryManager. No need to track it inside the class.
+ PinotDataBuffer headerBuffer = _memoryManager.allocate(_headerSize, _context);
Review comment:
Not sure I understand this. is headerBuffer(s) alllocated here closed anywhere? Same q with databBuffers . The reader and writer classes we construct here do not close these buffers, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org