You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pinot.apache.org by Pinot Slack Email Digest <sn...@apache.org> on 2021/06/09 02:00:25 UTC
Apache Pinot Daily Email Digest (2021-06-08)
### _#general_
**@santhosh:** Hi. We have a usecase to store the incoming user events. We
have multiple dimensions where we want to query on. We want to use S3 as deep
storage. We also have requirements like, the last half hour data will be
queried on frequently like a hot shard. Can we use pinot for this use case?
How can we model data optimally for this kind of use case? Do we have data
retention support where data older than that can be removed after some time?
**@mayanks:** Yes, this seems like a good fit for Pinot. and yes Pinot has
retention support.
**@ravi.ranjan:** @ravi.ranjan has joined the channel
**@jmeyer:** Hello :slightly_smiling_face: Is there an "official" / commonly
used Grafana Dashboard for Pinot ? (I've found this one -> \- thanks @dlavoie
^^)
**@kchavda:** @kchavda has joined the channel
**@karinwolok1:** Welcome new Apache Pinot community members! :wave: We're so
happy you're here and joining the movement of speeeeeed for user-facing
analytics. :wine_glass: Please take a moment, if you haven't already - to
introduce yourself and tell us what brought you here! :smiley: @kchavda
@ravi.ranjan @sheetalarun.kadam2 @tenghuanhe @santhosh @fritzbx @kaushikf9t
@ragiroux @ramabaratam @cgoddard @bryagd @mercyshans @kkmagic99 @saurabhd336
@bcbazevedo @ming.liu @kanleecarro @hari.prasanna @richard.hallier @bowenwan
@sharma.vinit @anusha.munukuntla @kylebanker @hongtaozhang @krishnalumia535
@kaustabhganguly
**@jai.patel856:** For an upsert table I have the order columns:
timeColumnName set to my updated_at timestamp. It used to be created_at when I
was using an offline-only table. I believe this is the correct change. My
question is for the sortedcolumn index, do I need to change it too? For my use
case I generally still want to be sorting on created_at. But does upsert
required the sortedcolumn be the same as the timecolumn?
**@jai.patel856:** @g.kishore @yupeng
**@jackie.jxt:** No, you can keep the current sort column
**@jai.patel856:** sweet
### _#random_
**@ravi.ranjan:** @ravi.ranjan has joined the channel
**@kchavda:** @kchavda has joined the channel
### _#troubleshooting_
**@patidar.rahul8392:** Hi, I have one hybrid table in pinot in
tablename_OFFLINE table I have 250000 records and tablename_realtime I have
total 2000 records. When I am doing select count from table it showing 252000
that is correct count. But when I am doing select * from tablename in superset
for the same query. It's below error message.
**@patidar.rahul8392:**
**@patidar.rahul8392:** @fx19880617 @ken @jackie.jxt kindly suggest is there
any row limit in pinot.?
**@patidar.rahul8392:** This is the complete log
**@jackie.jxt:** Pinot does not have this limit. Seems thrown from presto
connector @fx19880617
**@fx19880617:** the log says the limit is 50k for select * query:
**@fx19880617:** you can increase it from presto config
**@fx19880617:** default is 50k
**@ravi.ranjan:** @ravi.ranjan has joined the channel
**@patidar.rahul8392:** In prestro config file I am not giving any limit .I
have only these properties.
**@patidar.rahul8392:** If I am doing select from any other tables .i.e. hive
or druid I am able to select data .but in case of pinot in throwing this
error. Some problem with this prestro-pinot connector.kindly suggest if is
there any why to select data directly from pinot without prestro connector or
any way to increase this limit. @fx19880617 @jackie.jxt
**@fx19880617:** check pinot catalog config : ```pinot.limit-large-for-
segment=2147483647```
**@fx19880617:** wait, are you using prestodb or trino?
**@fx19880617:** for trino,set ```pinot.max-rows-per-split-for-segment-
queries=2147483647```
**@patidar.rahul8392:** It's prestrodb @fx19880617 version 350
**@fx19880617:** if 350, then it’s trino
**@patidar.rahul8392:** Ohh ok in logs everywhere I am getting prestrosql so I
thought it's prestrodb. Ok let me try with trino property then. Thanks alot
@fx19880617
**@patidar.rahul8392:** @fx19880617 thanks alot it worked. One update had to
decrease the max row limit from Int max size value i.e. 2147483647 otherwise
it was taking the minimum value of Int i.e. -2147483648 and again showing
limit error.
**@patidar.rahul8392:** In case of max value of integer , it was showing this
issue
**@fx19880617:** ok
**@fx19880617:** then you can make it a bit smaller I think
**@fx19880617:** I think 2Billion is fine as well
**@fx19880617:** As long as it’s bigger than your max docs of one single
segment
**@patidar.rahul8392:** Ok @fx19880617
**@pedro.cls93:** Hello, does Pinot support some auto-scaling of some sort to
deal with increasingly larger and heavier workloads? I have a single real-time
table consuming events from kafka (this is a wide table but not many fields,
currently there are 39, mostly strings, one of which has a max length of
2147483647 (INTEGER.MAX_VALUE) since it holds a json blob). My pinot cluster
is deployed in Kubernetes (hosted in azure) 2 pinot server instances with 5GB
heap + 3GB for direct memory, 100GB persistance volume (segment deepstorage is
configured) with a k8s memory limit of 10G. 1 controller instance with 1GB
heap for JVM, k8s memory limit 2G. 1 broker instance with 4GB heap, k8s memory
limit 5G. My servers are crashing with segment faults & OOM, as follows:
Server 1: ```# # A fatal error has been detected by the Java Runtime
Environment: # # SIGBUS (0x7) at pc=0x00007f4b79052422, pid=8,
tid=0x00007f4ae8739700 # # JRE version: OpenJDK Runtime Environment
(8.0_292-b10) (build 1.8.0_292-b10) # Java VM: OpenJDK 64-Bit Server VM
(25.292-b10 mixed mode linux-amd64 compressed oops) # Problematic frame: # v
~StubRoutines::jbyte_disjoint_arraycopy # # Core dump written. Default
location: /opt/pinot/core or core.8 # [thread 139959708407552 also had an
error] # An error report file with more information is saved as: #
/opt/pinot/hs_err_pid8.log # # If you would like to submit a bug report,
please visit: # # Aborted (core dumped)``` Server 2: ```Exception:
java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread
"Start a Pinot [SERVER]-SendThread(pinot-zookeeper:2181)" 2021/06/08
16:35:05.338 ERROR
[LLRealtimeSegmentDataManager_HitExecutionView_3mo__1__3__20210608T1552Z]
[HitExecutionView_3mo__1__3__20210608T1552Z] Could not build segment
java.lang.IllegalArgumentException: Self-suppression not permitted at
java.lang.Throwable.addSuppressed(Throwable.java:1072) ~[?:1.8.0_292] at
org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter.build(RealtimeSegmentConverter.java:132)
~[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-f15225f9c8abe8d9efa52c31c00f0d7418b368eb] at
org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentInternal(LLRealtimeSegmentDataManager.java:783)
[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-f15225f9c8abe8d9efa52c31c00f0d7418b368eb] at
org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentForCommit(LLRealtimeSegmentDataManager.java:717)
[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-f15225f9c8abe8d9efa52c31c00f0d7418b368eb] at
org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:628)
[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-f15225f9c8abe8d9efa52c31c00f0d7418b368eb] at
java.lang.Thread.run(Thread.java:748) [?:1.8.0_292] Caused by:
java.lang.OutOfMemoryError: Java heap space AsyncLogger error handling event
seq=1, value='null': java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space Exception in thread
"HitExecutionView_3mo__3__3__20210608T1552Z" java.lang.OutOfMemoryError: Java
heap space 2021/06/08 16:35:05.395 ERROR
[LLRealtimeSegmentDataManager_HitExecutionView_3mo__7__3__20210608T1553Z]
[HitExecutionView_3mo__7__3__20210608T1553Z] Could not build segment
java.lang.IllegalArgumentException: Self-suppression not permitted at
java.lang.Throwable.addSuppressed(Throwable.java:1072) ~[?:1.8.0_292] at
org.apache.pinot.segment.local.segment.index.converter.SegmentV1V2ToV3FormatConverter.copyIndexData(SegmentV1V2ToV3FormatConverter.java:160)
~[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-f15225f9c8abe8d9efa52c31c00f0d7418b368eb] at
org.apache.pinot.segment.local.segment.index.converter.SegmentV1V2ToV3FormatConverter.convert(SegmentV1V2ToV3FormatConverter.java:86)
~[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-f15225f9c8abe8d9efa52c31c00f0d7418b368eb] at
org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.convertFormatIfNecessary(SegmentIndexCreationDriverImpl.java:370)
~[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-f15225f9c8abe8d9efa52c31c00f0d7418b368eb] at
org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.handlePostCreation(SegmentIndexCreationDriverImpl.java:303)
~[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-f15225f9c8abe8d9efa52c31c00f0d7418b368eb] at
org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:256)
~[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-f15225f9c8abe8d9efa52c31c00f0d7418b368eb] at
org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter.build(RealtimeSegmentConverter.java:131)
~[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-f15225f9c8abe8d9efa52c31c00f0d7418b368eb] at
org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentInternal(LLRealtimeSegmentDataManager.java:783)
[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-f15225f9c8abe8d9efa52c31c00f0d7418b368eb] at
org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentForCommit(LLRealtimeSegmentDataManager.java:717)
[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-f15225f9c8abe8d9efa52c31c00f0d7418b368eb] at
org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:628)
[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-f15225f9c8abe8d9efa52c31c00f0d7418b368eb] at
java.lang.Thread.run(Thread.java:748) [?:1.8.0_292] Caused by:
java.lang.OutOfMemoryError: Java heap space```
**@mayanks:** Pinot does support scaling, but it is not fully auto at the
moment. You can add server nodes and invoke rebalance api
**@mayanks:** You seem to be running into `java.lang.OutOfMemoryError: Java
heap space`
**@pedro.cls93:** I'm injecting 33M records into the table, this is something
like 15GB in parquet. However the 100GB of disk in server 1 + 76GB in server 2
are used up. With 4.6M records, pinot reports 56GB in uncompressed table data.
Is this normal?
**@mayanks:** 56GB of heap?
**@pedro.cls93:** in `Reported Size` as reported by the table summary UI in
Pinot. I don't know if it is heap or disk.
**@mayanks:** I am guessing you used raw index for the string columns?
**@pedro.cls93:** The field with maxLength: 2147483647 is json indexed.
Everything else is the default
**@pedro.cls93:** This is the table indexing config: ```"tableIndexConfig": {
"invertedIndexColumns": [], "rangeIndexColumns": [], "jsonIndexColumns": [
"inputForUiControls" ], "autoGeneratedInvertedIndex": false,
"createInvertedIndexDuringSegmentGeneration": false, "bloomFilterColumns": [],
"loadMode": "MMAP", "streamConfigs": { "streamType": "kafka",
"stream.kafka.topic.name": "test.data.hitexecutionview_3mo",
"stream.kafka.broker.list": "dckafka.dc-kafka.svc.cluster.local:9092",
"stream.kafka.consumer.type": "lowlevel",
"stream.kafka.consumer.prop.auto.offset.reset": "smallest",
"stream.kafka.consumer.factory.class.name":
"org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
"stream.kafka.decoder.class.name":
"org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
"realtime.segment.flush.threshold.rows": "1600000" }, "noDictionaryColumns":
[], "onHeapDictionaryColumns": [], "varLengthDictionaryColumns": [],
"enableDefaultStarTree": false, "enableDynamicStarTreeCreation": false,
"segmentPartitionConfig": { "columnPartitionMap": { "externalHitExecutionId":
{ "functionName": "Murmur", "numPartitions": 16 } } }```
**@mayanks:** By default string columns are padded to make them equal length
in the dictionary. If you did not explicitly set no-dict column, then my guess
is that is where the space is being spent. Do you have access to
`metadata.properties` file inside the segment directory on the server? If so,
can you share?
**@mayanks:** yeah `"noDictionaryColumns": [],`
**@mayanks:** can you share metadata.properties file?
**@pedro.cls93:** let me check
**@pedro.cls93:** I can't seem to find the metadata.properties file, where
would it be found by default?
**@mayanks:** One the server's dataDir (where it stores segments), go inside
one of the segment dir
**@pedro.cls93:** any segment will do?
**@mayanks:** Sure. Try to pick biggest one
**@pedro.cls93:** Server 1 is crashed, can't recover it but in server 2 all
metadata.properties files are 35k, here is an example of the content:
**@mayanks:** which column has the INT_MAX length?
**@pedro.cls93:** inputForUiControls
**@mayanks:** `column.inputForUiControls.lengthOfEachEntry = 33655`
**@mayanks:** Your segment has 200k rows. With this padded string, the
dictionary for this column becomes 6,7GB
**@pedro.cls93:** This is the schema: Did I configure something wrong?
**@pedro.cls93:** Is that 6,7G per segment?
**@mayanks:** 6.7G for this one segment
**@mayanks:** what's the disk size you see for this segment (du -sh)
**@pedro.cls93:** 4.0G
**@mayanks:** hmm
**@pedro.cls93:** ```root@pinot-server-1:/var/pinot/server/data# du -sh
./index/HitExecutionView_3mo_REALTIME/HitExecutionView_3mo__5__3__20210608T1551Z/v3/
4.0G
./index/HitExecutionView_3mo_REALTIME/HitExecutionView_3mo__5__3__20210608T1551Z/v3/```
**@mayanks:** There are a few things to do, since you are running out of heap
**@pedro.cls93:** Also space in server 0: ```Filesystem Size Used Avail Use%
Mounted on /dev/sdb 93G 93G 99M 100% /var/pinot/server/data```
**@mayanks:** Where is the space being used? All in segments?
**@pedro.cls93:** Supposedly, yes, can't access the pod it's crashed
**@mayanks:** How did you get the metadata then?
**@pedro.cls93:** I accessed the other server, I have 2 (server 0 is crashed,
server 1 is ok)
**@pedro.cls93:** not the same volumes
**@ssubrama:** from the stack, it seems that they are runin gout of heap
during segment build
**@mayanks:** Closing the thread here - one string column had huge variations
in length of entries and almost 99% of the 4GB segment size was dictionary
padding. Converting it to no-dictionary column reduced the size by an order of
magnitude.
**@kchavda:** @kchavda has joined the channel
### _#pinot-dev_
**@ragiroux:** @ragiroux has joined the channel
\--------------------------------------------------------------------- To
unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org For additional commands,
e-mail: dev-help@pinot.apache.org