You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pinot.apache.org by Pinot Slack Email Digest <sn...@apache.org> on 2020/08/15 02:00:10 UTC
Apache Pinot Daily Email Digest (2020-08-14)
<h3><u>#general</u></h3><br><strong>@apsingekar.venkatesh: </strong>@apsingekar.venkatesh has joined the channel<br><strong>@sundar.djeabalane2: </strong>Hi Everyone ! I’m looking for options to expose presto (presto coordinator) outside kubernetes with some basic auth. we are currently running Pinot and presto inside kubernetes. We have a requirement where we want to expose presto outside as a service so clients can connect via presto consume the data. please let me know if anyone has implemented the authentication at the presto layer.<br><h3><u>#random</u></h3><br><strong>@apsingekar.venkatesh: </strong>@apsingekar.venkatesh has joined the channel<br><h3><u>#feat-text-search</u></h3><br><strong>@harrynet1989: </strong>@harrynet1989 has joined the channel<br><strong>@apsingekar.venkatesh: </strong>@apsingekar.venkatesh has joined the channel<br><h3><u>#feat-rt-seg-complete</u></h3><br><strong>@harrynet1989: </strong>@harrynet1989 has joined the channel<br><h3><u>#feat-presto-connector</u></h3><br><strong>@apsingekar.venkatesh: </strong>@apsingekar.venkatesh has joined the channel<br><h3><u>#troubleshooting</u></h3><br><strong>@apsingekar.venkatesh: </strong>@apsingekar.venkatesh has joined the channel<br><strong>@pradeepgv42: </strong>QQ on star tree index, does just specifying `dimensionsSplitOrder` would only include dimensions included in this list?
or would we have to also specify “skipStarNodeCreationForDimensions” to avoid other dimensions to get included in the start tree?<br><strong>@g.kishore: </strong>thats right<br><strong>@pradeepgv42: </strong>got it, thanks<br><strong>@andrew: </strong>@andrew has joined the channel<br><strong>@andrew: </strong>where i can find logs for each query? trying to troubleshoot slow queries
i tried this but not luck for any of the brokers: `$ kubectl -n pinot logs pinot-broker-6 broker -f`<br><strong>@sosyalmedya.oguzhan: </strong>In batch ingestion, table is created automatically if it is not exist right?<br><strong>@fx19880617: </strong>no<br><strong>@fx19880617: </strong>table is required for fetching required information e.g. schema table configs<br><strong>@mayanks: </strong>Yes, then you can check what was happening on that server at that time<br><strong>@pradeepgv42: </strong>some more questions on start-tree index, does reload create start-tree for old segments? and how do I verify that start-tree index is generated for the segments?
Basically I am not seeing improvement in query times after reload, so wondering if I am missing something<br><h3><u>#aggregators</u></h3><br><strong>@harrynet1989: </strong>@harrynet1989 has joined the channel<br><h3><u>#enable-generic-offsets</u></h3><br><strong>@harrynet1989: </strong>@harrynet1989 has joined the channel<br><h3><u>#pinot-dev</u></h3><br><strong>@harrynet1989: </strong>@harrynet1989 has joined the channel<br><strong>@jlli: </strong>Hey it seems the Swagger rest API console isn’t working. Can someone working on UI help take a look at it?<br><strong>@g.kishore: </strong>It works with QuickStart<br><h3><u>#announcements</u></h3><br><strong>@harrynet1989: </strong>@harrynet1989 has joined the channel<br><h3><u>#release-certifier</u></h3><br><strong>@mayanks: </strong>Hey guys, sorry if I am missing something obvious, is there a plan explaining the overall goal, with definition of 'done', along with short/medium/long-term milestones.<br><strong>@mayanks: </strong>We do have real issues that keep popping up in production today as new code gets checked in. It would be really helpful to ensure that at least our immediate/short term goals will indeed help address the issues we are actually seeing.<br><h3><u>#lp-pinot-poc</u></h3><br><strong>@andrew: </strong>@andrew has joined the channel<br><strong>@g.kishore: </strong>@g.kishore has joined the channel<br><strong>@mayanks: </strong>@mayanks has joined the channel<br><strong>@fx19880617: </strong>@fx19880617 has joined the channel<br><strong>@jackie.jxt: </strong>@jackie.jxt has joined the channel<br><strong>@andrew: </strong>@andrew set the channel purpose: Discuss Leanplum Pinot POC<br><strong>@g.kishore: </strong>@fx19880617 @mayanks Andrew is doing a perf benchmark<br><strong>@g.kishore: </strong>he has already enabled partitioning and segment relocator<br><strong>@mayanks: </strong>Nice<br><strong>@g.kishore: </strong>most of the queries take few ms<br><strong>@g.kishore: </strong>but the p99.9 shows some queries take 10sec<br><strong>@g.kishore: </strong>@andrew can you paste the graph<br><strong>@andrew: </strong><br><strong>@mayanks: </strong>How's the GC?<br><strong>@mayanks: </strong>Is replica-group based routing enabled?<br><strong>@mayanks: </strong>to reduce fanout?<br><strong>@g.kishore: </strong>thats what we suspect, either GC or its segment flush<br><strong>@g.kishore: </strong>this is real-time only<br><strong>@mayanks: </strong>ok<br><strong>@mayanks: </strong>how many nodes is the query fanning out to?<br><strong>@g.kishore: </strong>so it only queries 1 node<br><strong>@g.kishore: </strong>per partition<br><strong>@mayanks: </strong>ok good<br><strong>@g.kishore: </strong>this was before segment relocation<br><strong>@jackie.jxt: </strong>Is all the queries the same pattern?<br><strong>@g.kishore: </strong>not sure after relocation was enabled<br><strong>@g.kishore: </strong>yes<br><strong>@andrew: </strong>the queries are of the form `SELECT COUNT(*) FROM events WHERE timestampMillis > 0 AND userId = '14068106f053d26d101efef9' AND eventName = '100268a9e49b'` . The table is partitioned on userId<br><strong>@mayanks: </strong>is user id string?<br><strong>@andrew: </strong>yes<br><strong>@mayanks: </strong>ok, that's fine<br><strong>@mayanks: </strong>how many rows per user on average?<br><strong>@andrew: </strong>the data is generated randomly. the user IDs are distributed normally and the eventNames follow an exponential distribution<br><strong>@andrew: </strong>i’m starting my test with no data, so usually just a few rows<br><strong>@mayanks: </strong>@jackie.jxt does RT use off-heap by default?<br><strong>@mayanks: </strong>if GC on consuming nodes, we should check if off-heap is being used for RT<br><strong>@g.kishore: </strong>lets validate if its GC<br><strong>@mayanks: </strong>yeah<br><strong>@jackie.jxt: </strong>Seems not<br><strong>@jackie.jxt: </strong>`_isRealtimeOffheapAllocation` in `IndexLoadingConfig`<br><strong>@andrew: </strong>it’s easy to reproduce in one off queries too. here’s an example I just ran in the UI:
```{
"resultTable": {
"dataSchema": {
"columnDataTypes": [
"LONG"
],
"columnNames": [
"count(*)"
]
},
"rows": [
[
3
]
]
},
"exceptions": [],
"numServersQueried": 2,
"numServersResponded": 2,
"numSegmentsQueried": 2,
"numSegmentsProcessed": 2,
"numSegmentsMatched": 1,
"numConsumingSegmentsQueried": 1,
"numDocsScanned": 3,
"numEntriesScannedInFilter": 0,
"numEntriesScannedPostFilter": 0,
"numGroupsLimitReached": false,
"totalDocs": 149234,
"timeUsedMs": 1603,
"segmentStatistics": [],
"traceInfo": {},
"minConsumingFreshnessTimeMs": 1597434210591
}```<br><strong>@jackie.jxt: </strong>Do you have live queries when running this? @andrew<br><strong>@andrew: </strong>i am running a load test right now with 500-1000 qps<br><strong>@andrew: </strong>and ingesting 5-10k events/second at the same time<br><strong>@mayanks: </strong>Is there a way for you to check if latency spike is during segment commit/relocation?<br><strong>@mayanks: </strong>Segment commit will pause kafka consumption, so you can use that as a proxy to find the time of commit<br><strong>@jackie.jxt: </strong>Based on the graph, I feel the high latency queries are from the cold start<br><strong>@andrew: </strong>here’s the full graph<br><strong>@andrew: </strong><br><strong>@andrew: </strong>now some segments are in error state<br><strong>@andrew: </strong>the data freshness also seems to get bad<br><strong>@mayanks: </strong>the spike at 12:30 is likely due to segment commit<br><strong>@mayanks: </strong>server logs should have time stamps of commit<br><strong>@mayanks: </strong>try offheap setting<br><strong>@mayanks: </strong>``_isRealtimeOffheapAllocation` in `IndexLoadingConfig`<br><strong>@mayanks: </strong>Even if it does not fix all spikes, it is a good idea to reduce GC impact<br><strong>@andrew: </strong>is that by setting `pinot.server.instance.realtime.alloc.offheap` to true?<br><strong>@andrew: </strong>what is the best way to set this with a helm install?<br><strong>@mayanks: </strong>No index loading config in table config<br><strong>@mayanks: </strong>@jackie.jxt confirm ^^?<br><strong>@jackie.jxt: </strong>@andrew Yes, you need to set it as the server config<br><strong>@mayanks: </strong>Hmm, we should fix that. Or perhaps make it on by default<br><strong>@jackie.jxt: </strong>You can add that into the table config, and enable it if either table config or instance config has it as true<br><strong>@andrew: </strong>ok, maybe i’m missing something but i don’t see a field for server config in the Helm config file<br><strong>@andrew: </strong>i used the API to set it in the cluster config. should it take effect immediately? seems like the tail latency has dropped<br><strong>@mayanks: </strong>server config is read only at start, so i expect it would only take effect on restart<br><strong>@andrew: </strong>but it would read that config from zookeeper?<br><strong>@andrew: </strong>hm, it also looks like the cluster stopped ingesting data earlier, maybe related to segments in error state<br><strong>@andrew: </strong>i’m seeing logs like this now, and the server is unable to ingest data:
```2020/08/15 00:00:47.294 WARN [StateModel] [ZkClient-EventThread-20-pinot-zookeeper:2181] Default reset method invoked. Either because the process longer own this resource or session timedout
...
2020/08/14 23:57:47.584 WARN [events_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_STATE_TRANSITION] Skipping adding existing segment: events__65__0__20200814T2352Z for table: events_REALTIME with data manager class: LLRealtimeSegmentDataManager```<br><strong>@mayanks: </strong>This is probably the cause of the segments in error state that you already had <br><h3><u>#aggregate-metrics-change</u></h3><br><strong>@steotia: </strong>@jackie.jxt, what should be the correct fix here? I am wondering if the fix is fairly quick, I can create a hotfix instead of rolling back and waiting for the fix. I am guessing we have to get back to previous behavior to compute min/max and remove the if check for aggregateMetrics.<br><strong>@steotia: </strong>currently the min and max are null<br><strong>@jackie.jxt: </strong>I'll submit the fix soon<br><strong>@jackie.jxt: </strong>Trying to add the test now<br><strong>@steotia: </strong>ok will wait for now. thanks a lot<br><strong>@jackie.jxt: </strong><https://u17000708.ct.sendgrid.net/ls/click?upn=1BiFF0-2FtVRazUn1cLzaiMSfW2QiSG4bkQpnpkSL7FiK3MHb8libOHmhAW89nP5XKjr0tug9EOOmbvGLEos3pUw-3D-3D-Tjz_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTyI3XepWPf0zpQeVU94YsHyVBC-2FGf4L-2Fr8a23SmYbqTMfCucCXgJTV05t-2FKrK5GW5wrW217s5Y3wFG1TM2-2FOXbocLBtyvRZOaz9pPuFFU1Ox4dYiP1JPi1mGgI9umKbw-2FZipTxctb-2FYXc0q8t9jUdbqE4oSO3nMgvNNrSGtlQSUrR4x2gKleLDqtif1hRMQsSM-3D><br>