Apache Pinot Daily Email Digest (2021-01-23)

[Pinot Slack Email Digest <sn...@apache.org>]

### _#general_

  
 **@kennybastani:** @kennybastani set the channel topic: Welcome! Show your
support by starring our project repository on GitHub  
**@darek:** @darek has joined the channel  
 **@ddayan:** @ddayan has joined the channel  
 **@anirudhkrec:** @anirudhkrec has joined the channel  

###  _#random_

  
 **@darek:** @darek has joined the channel  
 **@ddayan:** @ddayan has joined the channel  
 **@anirudhkrec:** @anirudhkrec has joined the channel  

###  _#troubleshooting_

  
 **@ddayan:** @ddayan has joined the channel  
 **@anirudhkrec:** @anirudhkrec has joined the channel  
 **@ken:** We’ve run into an issue with loading segments on our server, where
we need more direct memory when building inverted indexes. The stack trace
looks like: ```Caused by: java.lang.OutOfMemoryError: Direct buffer memory at
java.nio.Bits.reserveMemory(Bits.java:694) ~[?:1.8.0_275] at
java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) ~[?:1.8.0_275] at
java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[?:1.8.0_275] at
org.apache.pinot.core.segment.memory.PinotByteBuffer.allocateDirect(PinotByteBuffer.java:38)
~[pinot-all-0.6.0-jar-with-
dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21] at
org.apache.pinot.core.segment.memory.PinotDataBuffer.allocateDirect(PinotDataBuffer.java:116)
~[pinot-all-0.6.0-jar-with-
dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21] at
org.apache.pinot.core.segment.creator.impl.inv.OffHeapBitmapInvertedIndexCreator.createTempBuffer(OffHeapBitmapInvertedIndexCreator.java:254)
~[pinot-all-0.6.0-jar-with-
dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21] at
org.apache.pinot.core.segment.creator.impl.inv.OffHeapBitmapInvertedIndexCreator.seal(OffHeapBitmapInvertedIndexCreator.java:152)
~[pinot-all-0.6.0-jar-with-
dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21] at
org.apache.pinot.core.segment.index.loader.invertedindex.InvertedIndexHandler.createInvertedIndexForColumn(InvertedIndexHandler.java:125)
~[pinot-all-0.6.0-jar-with-
dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21] at
org.apache.pinot.core.segment.index.loader.invertedindex.InvertedIndexHandler.createInvertedIndices(InvertedIndexHandler.java:73)
~[pinot-all-0.6.0-jar-with-
dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21] at
org.apache.pinot.core.segment.index.loader.SegmentPreProcessor.process(SegmentPreProcessor.java:109)
~[pinot-all-0.6.0-jar-with-
dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21] at
org.apache.pinot.core.indexsegment.immutable.ImmutableSegmentLoader.load(ImmutableSegmentLoader.java:99)
~[pinot-all-0.6.0-jar-with-
dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21] at
org.apache.pinot.core.data.manager.offline.OfflineTableDataManager.addSegment(OfflineTableDataManager.java:52)
~[pinot-all-0.6.0-jar-with-
dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21] at
org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addOfflineSegment(HelixInstanceDataManager.java:122)
~[pinot-all-0.6.0-jar-with-
dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21] at
org.apache.pinot.server.starter.helix.SegmentFetcherAndLoader.addOrReplaceOfflineSegment(SegmentFetcherAndLoader.java:116)
~[pinot-all-0.6.0-jar-with-
dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]``` We’ve
bumped direct memory to 5gb, but still hitting these exceptions. We can
continue increasing (or create the inverted index while building the
segments), but wanted to confirm that it’s expected to need a lot of direct
memory like this, thanks.  
**@fx19880617:** I feel we should have an option to make segment build with
bounded memory size. Issue tracking:  
**@mayanks:** Since your inv index size is huge, better to build offline  
**@mailtobuchi:** Hey folks, a bunch of segments are in BAD state in Pinot and
reloading all the segments from the UI didn’t help. Trying to debug it further
but didn’t see any warnings or errors in the logs. Any hints on where to
check? Any usual suspects?  
**@mayanks:** Server log should definitely have the error upon loading  

### _#presto-pinot-connector_

  
 **@ddayan:** @ddayan has joined the channel  
\--------------------------------------------------------------------- To
unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org For additional commands,
e-mail: dev-help@pinot.apache.org