You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pinot.apache.org by Pinot Slack Email Digest <sn...@apache.org> on 2020/08/01 02:00:07 UTC
Apache Pinot Daily Email Digest (2020-07-31)
<h3><u>#general</u></h3><br><strong>@mayanks: </strong>Hello Pinot community, we at LinkedIn, are planning to host a Virtual Meetup for Apache Pinot on Aug 27. We would like to have one presenter from the community share their success-story/journey/etc with Pinot. Please reply to this thread if you would be interested in speaking, with a brief on what you would like to talk about.<br><strong>@christian: </strong>@christian has joined the channel<br><strong>@qiaochu: </strong>@qiaochu has joined the channel<br><h3><u>#random</u></h3><br><strong>@christian: </strong>@christian has joined the channel<br><strong>@qiaochu: </strong>@qiaochu has joined the channel<br><h3><u>#feat-text-search</u></h3><br><strong>@christian: </strong>@christian has joined the channel<br><h3><u>#troubleshooting</u></h3><br><strong>@sosyalmedya.oguzhan: </strong>can someone help me about maven enforcer rules? :disappointed:<br><strong>@yash.agarwal: </strong>I am getting the following exception when querying the controller with a specific sql query.
```Caught exception while sending query request
java.io.IOException: Failed : HTTP error code : 500
at org.apache.pinot.controller.api.resources.PinotQueryResource.sendPostRaw(PinotQueryResource.java:284) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-0fc081157c88734af0433c3cd30f310474d177f0]
at org.apache.pinot.controller.api.resources.PinotQueryResource.sendRequestRaw(PinotQueryResource.java:322) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-0fc081157c88734af0433c3cd30f310474d177f0]
at org.apache.pinot.controller.api.resources.PinotQueryResource.getQueryResponse(PinotQueryResource.java:204) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-0fc081157c88734af0433c3cd30f310474d177f0]
at org.apache.pinot.controller.api.resources.PinotQueryResource.handlePostSql(PinotQueryResource.java:131) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-0fc081157c88734af0433c3cd30f310474d177f0]```
If I directly query the broker I am getting
```<html>
<head>
<title>Grizzly 2.4.4</title>
<style>
<!--div.header {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#003300;font-size:22px;-moz-border-radius-topleft: 10px;border-top-left-radius: 10px;-moz-border-radius-topright: 10px;border-top-right-radius: 10px;padding-left: 5px}div.body {font-family:Tahoma,Arial,sans-serif;color:black;background-color:#FFFFCC;font-size:16px;padding-top:10px;padding-bottom:10px;padding-left:10px}div.footer {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#666633;font-size:14px;-moz-border-radius-bottomleft: 10px;border-bottom-left-radius: 10px;-moz-border-radius-bottomright: 10px;border-bottom-right-radius: 10px;padding-left: 5px}BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;}B {font-family:Tahoma,Arial,sans-serif;color:black;}A {color : black;}HR {color : #999966;}
-->
</style>
</head>
<body>
<div class="header">Request failed.</div>
<div class="body">Request failed.</div>
<div class="footer">Grizzly 2.4.4</div>
</body>
</html>```
I have recently added a config in broker
```pinot.broker.timeoutMs=120000```
Any Ideas ?<br><strong>@yash.agarwal: </strong>There are no logs in broker. Other queries are working fine, this query would scan 1.7 Bil recods and do a sum of 3 metrics and distinctcount of 1 dimension<br><strong>@yash.agarwal: </strong>It might be because of high memory usage on broker. as i am seeing usage spikes upto 60 - 80 %<br><strong>@mayanks: </strong>Yes, distinctcount on 1.7B records may the likely issue. Have you considered using HLL based approximate distinctcount?<br><strong>@yash.agarwal: </strong>We have tried HLL in the past. We are not looking for approximations.<br><strong>@yash.agarwal: </strong>Also seems like that is the problem, I tried without the distinctcount and it is returning successfully in 2 secs.<br><strong>@mailtobuchi: </strong>@mailtobuchi has left the channel<br><strong>@christian: </strong>@christian has joined the channel<br><h3><u>#pinot-dev</u></h3><br><strong>@mayanks: </strong>`mvn clean install -DskipTests -Pbin-dist` takes awfully long time, it seems to have creeped up quite a bit over the years. Any ideas on how to make the build faster?<br><strong>@dlavoie: </strong>~ 10 minutes on my own CI server.
```[INFO] Pinot .............................................. SUCCESS [ 22.428 s]
[INFO] Pinot Service Provider Interface ................... SUCCESS [ 7.931 s]
[INFO] Pinot Common ....................................... SUCCESS [ 32.668 s]
[INFO] Pinot Plugins ...................................... SUCCESS [ 0.563 s]
[INFO] Pinot Input Format ................................. SUCCESS [ 0.510 s]
[INFO] Pinot Avro Base .................................... SUCCESS [ 1.694 s]
[INFO] Pinot Avro ......................................... SUCCESS [ 5.284 s]
[INFO] Pinot Csv .......................................... SUCCESS [ 0.844 s]
[INFO] Pinot JSON ......................................... SUCCESS [ 0.738 s]
[INFO] Pinot Core ......................................... SUCCESS [ 22.112 s]
[INFO] Pinot Server ....................................... SUCCESS [ 9.760 s]
[INFO] Pinot Controller ................................... SUCCESS [01:25 min]
[INFO] Pinot Broker ....................................... SUCCESS [ 4.878 s]
[INFO] Pinot Clients ...................................... SUCCESS [ 0.230 s]
[INFO] Pinot Java Client .................................. SUCCESS [ 0.750 s]
[INFO] Pinot JDBC Client .................................. SUCCESS [ 0.975 s]
[INFO] Pinot Minion ....................................... SUCCESS [ 2.538 s]
[INFO] Pinot Confluent Avro ............................... SUCCESS [ 3.746 s]
[INFO] Pinot ORC .......................................... SUCCESS [ 4.028 s]
[INFO] Pinot Parquet ...................................... SUCCESS [ 6.670 s]
[INFO] Pinot Thrift ....................................... SUCCESS [ 1.268 s]
[INFO] Pinot Protocol Buffers ............................. SUCCESS [ 1.329 s]
[INFO] Pluggable Pinot file system ........................ SUCCESS [ 0.354 s]
[INFO] Pinot Azure Data Lake Storage ...................... SUCCESS [ 5.128 s]
[INFO] Pinot Hadoop Filesystem ............................ SUCCESS [ 1.115 s]
[INFO] Pinot Google Cloud Storage ......................... SUCCESS [ 5.645 s]
[INFO] Pinot Amazon S3 .................................... SUCCESS [ 11.483 s]
[INFO] Pinot Batch Ingestion .............................. SUCCESS [ 0.568 s]
[INFO] Pinot Batch Ingestion Common ....................... SUCCESS [ 0.785 s]
[INFO] Pinot Batch Ingestion for Spark .................... SUCCESS [ 11.263 s]
[INFO] Pinot Batch Ingestion for Hadoop ................... SUCCESS [ 3.663 s]
[INFO] Pinot Batch Ingestion Standalone ................... SUCCESS [ 0.985 s]
[INFO] Pinot Batch Ingestion .............................. SUCCESS [ 0.448 s]
[INFO] Pinot Ingestion Common ............................. SUCCESS [ 1.763 s]
[INFO] Pinot Hadoop ....................................... SUCCESS [ 34.624 s]
[INFO] Pinot Spark ........................................ SUCCESS [ 35.108 s]
[INFO] Pinot Stream Ingestion ............................. SUCCESS [ 0.581 s]
[INFO] Pinot Kafka Base ................................... SUCCESS [ 1.323 s]
[INFO] Pinot Kafka 0.9 .................................... SUCCESS [ 9.815 s]
[INFO] Pinot Kafka 2.0 .................................... SUCCESS [ 5.919 s]
[INFO] Pinot Tools ........................................ SUCCESS [01:47 min]
[INFO] Pinot Integration Tests ............................ SUCCESS [ 18.378 s]
[INFO] Pinot Perf ......................................... SUCCESS [ 7.437 s]
[INFO] Pinot Distribution ................................. SUCCESS [ 38.707 s]```
Overall Pinot Controller and Pinot tools are the suspects<br><strong>@dlavoie: </strong>Oh wait, don't mind that, it's a `mvn deploy` command to my personal artifactory.<br><strong>@g.kishore: </strong>shaded jars take time<br><strong>@dlavoie: </strong>If we had proper version definition and not placeholders, maybe you could rebuild module selectively and not the whole thing everytime something changes ? :slightly_smiling_face:<br><strong>@dlavoie: </strong>I'm personnaly maintaining a forked of pinot where I apply `mvn -f src/pom.xml --batch-mode release:update-versions -DdevelopmentVersion=0.5.0-SNAPSHOT` and it just makes life easier when recompiling.<br><strong>@mayanks: </strong>Don’t be so selfish @dlavoie, spread the kindness :stuck_out_tongue:<br><strong>@dlavoie: </strong>dhaha! It will impact the release proccess and I haven't taken the time to assert the impact of moving to fixed version in all our poms :wink: That's in my backlog<br><strong>@g.kishore: </strong>i also find it painful that we cant compile a single module anymore, I am sure we were able to do this<br><strong>@moradi.sajjad: </strong>A design doc for inconsistency problems around time boundary calculation in pinot brokers:
<https://u17000708.ct.sendgrid.net/ls/click?upn=1BiFF0-2FtVRazUn1cLzaiMc9VK8AZw4xfCWnhVjqO8F0hEXgV9LklcwhZkvCiU2HHynhCXcIWc42UTjQHNwGE8ZryeBZDuJXpGEVgCdbS-2FE2qp9xQpwGJFPCs3e6Uify3eGK2_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTwT-2FWrA-2FsGEELDvpFu6KY8-2FxdJWDLq5C-2FVUOv2WYSm6toZju5MCCiSZ65Q0yHD23EjRYErL28TdCm6MsOuVkKXhRY9tg8Cdl75stg4wIjLRTSH0T62zO9alwJVxjBW-2BjPZmo5-2B2R4tg02ZqoYhUJ6kHt8N-2BPMoKCkwJRMJGwYXkoLILEH-2BoDTHJUn7ESCVC1ME-3D><br><strong>@moradi.sajjad: </strong>please review and leave your comments.<br><strong>@mayanks: </strong>Thanks @moradi.sajjad<br>