You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pinot.apache.org by Pinot Slack Email Digest <sn...@apache.org> on 2021/06/11 02:00:17 UTC
Apache Pinot Daily Email Digest (2021-06-10)
### _#general_
**@burgalon:** In the `Pinot: Realtime OLAP for 530 Million Users` it says
```At Linkedin, business events are published in Kafka streams and are ETL'ed
onto HDFS. Pinot supports near-realtime data ingestion by reading events
directly from Kafka [19] as well as data pushes from offline systems like
Hadoop. As such, Pinot follows the lambda architecture [23], transparently
merging streaming data from Kafka and offline data from Hadoop. As data on
Hadoop is a global view of a single hour or day of data as opposed to a direct
stream of events, it allows for the generation of more optimal segments and
aggregation of records across the time window.``` Is there a general rule of
thumb of when should I keep raw events in Pinot vs aggregated data?
**@ken:** My rule of thumb would be “Use raw events until you are convinced
that won’t work”. Try to keep everything as raw events, with appropriate
indexes (especially star trees). If and only if you can’t get that to work
(latency is too high, storage requirements are too big, etc) then look at
doing pre-aggregations. I’m giving a Pinot talk on June 22nd, and one of the
items is how we worked around what looked like a pre-aggregation requirement.
**@gqian3:** Hi, does current Pinot python client support basic auth for
querying Pinot? Is there an example showing how to pass the auth header with
python client? Thanks.
**@g.kishore:** @slack1
**@slack1:** Hi Carl, afaik the python client doesn’t currently support auth.
An easy fallback here is hitting the REST apis of controller/broker directly
passing a standard HTTP `Authorization` header
**@gqian3:** Thanks I will give it a try.
**@karinwolok1:** :wave: Hi! Just wanted to share an exciting announcement
The original creators of Pinot just announced their product, ()! We're also
hosting a virtual event on June 24th to celebrate this milestone! Would love
to have you! :slightly_smiling_face:
**@patidar.rahul8392:** Is there any way to query Pinot table directly from
superset without using prestro as a middleware. I.e. To access pinot table
through superset I am using pinot prestro connector than in superset I am
using this catalog to connect from table so basically whenever I m firing some
queries from superset it's going to pinot with the help of prestro. Since I am
not using any joins in query so I believe I can also connect directly superset
and pinot without using prestro as a middleware. So I think this way queries
will be fast. @fx19880617 kindly suggest.
**@mayanks:** Have you looked at:
**@mayanks:** In general has a lot of good information and is also
searchable.
**@srini:** hi RK, I work on the Superset project and you can absolutely query
Pinot directly!
**@srini:** Ill also link to our docs page for Pinot-
**@patidar.rahul8392:** @srini @mayanks yes I checked this and in SqlAIchemy
URI I have mention /
**@patidar.rahul8392:** But for me it was showing this issue
**@srini:** ah, have you added the necessary drivers? I think this is a
missing driver issue
**@srini:** the `pinotdb` library () needs to be in the same python context as
Superset
**@srini:** if you’re using docker, we have some instructions here:
**@gqian3:** In default, Pinot return 10 rows when query select *, is there a
way to change and remove this default limit?
**@mayanks:** You can specify a limit of your choice in the query `limit xxx`
**@mayanks:**
**@gqian3:** Got it. Thanks
**@gqian3:** Just to confirm there is no way to change the default value of
limit 10 in cluster, right?
**@jackie.jxt:** Yes, the default limit of 10 is not configurable as of now
### _#troubleshooting_
**@phuchdh:** I got some problem with query result. I run the same query
statement ```select count(*) from RuleLogsUAT``` But got timeout & difference
result per request. Here is the request log
```requestId=43,table=RuleLogsUAT,timeMs=10001,docs=210279/210377,entries=0/0,segments(queried/processed/matched/consuming/unavailable):21/18/18/2/0,consumingFreshnessTimeMs=1623319344461,servers=4/5,groupLimitReached=false,brokerReduceTimeMs=0,exceptions=0,serverStats=(Server=SubmitDelayMs,ResponseDelayMs,ResponseSize,DeserializationTimeMs);pinot-
server-1_R=1,-1,0,0;pinot-server-2_R=2,6,250,0;pinot-
server-0_R=2,6,250,0;pinot-server-0_O=1,7,230,0;pinot-
server-1_O=2,11,230,0,query=select count(*) from
RuleLogsUAT,offlineThreadCpuTimeNs=1548682,realtimeThreadCpuTimeNs=1908620
requestId=44,table=RuleLogsUAT,timeMs=10002,docs=44807/44811,entries=0/0,segments(queried/processed/matched/consuming/unavailable):19/17/17/3/0,consumingFreshnessTimeMs=1623319158004,servers=4/5,groupLimitReached=false,brokerReduceTimeMs=0,exceptions=0,serverStats=(Server=SubmitDelayMs,ResponseDelayMs,ResponseSize,DeserializationTimeMs);pinot-
server-1_R=1,10,250,0;pinot-server-2_R=1,7,250,0;pinot-
server-0_R=1,8,250,0;pinot-server-2_O=1,-1,0,0;pinot-
server-1_O=1,11,230,0,query=select count(*) from
RuleLogsUAT,offlineThreadCpuTimeNs=1211610,realtimeThreadCpuTimeNs=1412400
requestId=45,table=RuleLogsUAT,timeMs=10002,docs=73633/73731,entries=0/0,segments(queried/processed/matched/consuming/unavailable):23/20/20/3/0,consumingFreshnessTimeMs=1623319158007,servers=4/5,groupLimitReached=false,brokerReduceTimeMs=0,exceptions=0,serverStats=(Server=SubmitDelayMs,ResponseDelayMs,ResponseSize,DeserializationTimeMs);pinot-
server-1_R=0,5,250,0;pinot-server-2_R=0,8,250,0;pinot-
server-0_R=0,8,250,0;pinot-server-0_O=0,7,230,0;pinot-
server-1_O=1,-1,0,0,query=select count(*) from
RuleLogsUAT,offlineThreadCpuTimeNs=655059,realtimeThreadCpuTimeNs=1141471
requestId=46,table=RuleLogsUAT,timeMs=10002,docs=44807/44811,entries=0/0,segments(queried/processed/matched/consuming/unavailable):19/17/17/3/0,consumingFreshnessTimeMs=1623319158004,servers=4/5,groupLimitReached=false,brokerReduceTimeMs=0,exceptions=0,serverStats=(Server=SubmitDelayMs,ResponseDelayMs,ResponseSize,DeserializationTimeMs);pinot-
server-1_R=1,5,250,0;pinot-server-2_R=1,7,250,0;pinot-
server-0_R=1,6,250,0;pinot-server-2_O=1,-1,0,0;pinot-
server-1_O=1,7,230,0,query=select count(*) from
RuleLogsUAT,offlineThreadCpuTimeNs=417430,realtimeThreadCpuTimeNs=1821230```
**@npawar:** are all your servers healthy? only 4 servers are responding
`servers=4/5`, hence the difference in result. You can check whats wrong with
1 server
**@npawar:** looks like a different server everytime. are they restarting?
**@shaileshjha061:** Hi @mayanks @npawar @dlavoie We have segments backed-up
in GCS. How can we reuse that segment files?? How backup and restore works??
Do we have any documentation for this? CC: @nadeemsadim @mohamed.sultan
**@g.kishore:** you dont need to do anything special.. back up/restore is
built into Pinot. You can add remove nodes and segments will be pulled from
deep store when needed
**@shaileshjha061:** Thanks for the response @g.kishore Can we deploy the
pinot in the new GKE cluster. and have the same data (which is already there
in GCS deepstore) in the new pinot cluster. If yes can you suggest what can we
done for this case. Thanks
**@mayanks:** I think I discussed this in detail with one of the team members?
**@shaileshjha061:** Yes Mayank I also had discussion with you related to
offline and realtime tables and came up with Hybrid table. But I am not able
to get the proper answer for reusing the GCS Backed-up segment file. In case
we have to move pinot to different GKE Cluster. CC: @nadeemsadim
**@dlavoie:** GCS segments files are managed by the Helix cluster of Pinot
(Zookeeper).
**@dlavoie:** You can’t share segments between 2 Pinot clusters.
**@dlavoie:** You could use segment download and upload apis to copy then
between clusters
**@shaileshjha061:** "You can’t share segments between 2 Pinot clusters."-->
Not at the same time. If one pinot or GKE cluster it self crashed. can't we
reuse the same segment files to create pinot in different GKE Cluster??
**@g.kishore:** Yes.. you can
**@shaileshjha061:** "You could use segment download and upload apis to copy
then between clusters"--> can you help me with download and upload apis. How
it could be done??
**@g.kishore:** You can simply call invoke the rest api calls to upload
segment from deep store into any cluster
**@dlavoie:** > You can’t share segments between 2 Pinot clusters. I meant in
active / active fashion.
**@nadeemsadim:** no .. we dont need active/active
**@shaileshjha061:** "You can simply call invoke the rest api calls to upload
segment from deep store into any cluster" @g.kishore Do we have any docs or
link for this??
**@mayanks:** Most questions have answers in and the search is also good. For
example,
**@mayanks:** I can summarize our previous discussion here, if you can help
create a FAQ for it?
**@mayanks:** @shaileshjha061
**@shaileshjha061:** Thanks Mayank will check on this link and update you.
**@nadeemsadim:** sure mayank .. I will create the FAQ .. please summarize
**@ravi.ranjan:** Hello, Setting up pinot on a docker-compose container.
Everthings runs as expected but when try to create a stream table by using the
below commands getting _*Connection refused*_ error. ```bin/pinot-admin.sh
AddTable \ -schemaFile examples/stream/airlineStats/airlineStats_schema.json \
-tableConfigFile
examples/stream/airlineStats/airlineStats_realtime_table_config.json \
-exec``` Error: ``` Got Exception to upload Pinot Schema: airlineStats
shaded.org.apache.http.conn.HttpHostConnectException: Connect to
172.26.0.5:9000 [/172.26.0.5] failed: Connection refused (Connection refused)
at
shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
org.apache.pinot.common.utils.FileUploadDownloadClient.sendRequest(FileUploadDownloadClient.java:383)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
org.apache.pinot.common.utils.FileUploadDownloadClient.addSchema(FileUploadDownloadClient.java:448)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
org.apache.pinot.tools.admin.command.AddTableCommand.uploadSchema(AddTableCommand.java:135)
[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
org.apache.pinot.tools.admin.command.AddTableCommand.execute(AddTableCommand.java:170)
[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:164)
[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:184)
[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] Caused by:
java.net.ConnectException: Connection refused (Connection refused) at
java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_282] at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
~[?:1.8.0_282] at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
~[?:1.8.0_282] at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
~[?:1.8.0_282] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
~[?:1.8.0_282] at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_282] at
shaded.org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] ... 15 more
Exception caught: shaded.org.apache.http.conn.HttpHostConnectException:
Connect to 172.26.0.5:9000 [/172.26.0.5] failed: Connection refused
(Connection refused) at
shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
org.apache.pinot.common.utils.FileUploadDownloadClient.sendRequest(FileUploadDownloadClient.java:383)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
org.apache.pinot.common.utils.FileUploadDownloadClient.addSchema(FileUploadDownloadClient.java:448)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
org.apache.pinot.tools.admin.command.AddTableCommand.uploadSchema(AddTableCommand.java:135)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
org.apache.pinot.tools.admin.command.AddTableCommand.execute(AddTableCommand.java:170)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:164)
[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:184)
[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] Caused by:
java.net.ConnectException: Connection refused (Connection refused) at
java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_282] at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
~[?:1.8.0_282] at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
~[?:1.8.0_282] at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
~[?:1.8.0_282] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
~[?:1.8.0_282] at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_282] at
shaded.org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] ... 15
more``` ```DOCKER_HOST=unix:///var/run/docker.sock docker ps ``` ```CONTAINER
ID IMAGE COMMAND CREATED STATUS PORTS NAMES 545bf572fd0d
apachepinot/pinot:release-0.7.1 "./bin/pinot-admin.s…" 2 hours ago Up 2 hours
8096-8097/tcp, 8099/tcp, 9000/tcp, 0.0.0.0:8098->8098/tcp, :::8098->8098/tcp
composetest_pinot-server_1 4033365f967e apachepinot/pinot:release-0.7.1
"./bin/pinot-admin.s…" 2 hours ago Up 2 hours 8096-8098/tcp, 9000/tcp,
0.0.0.0:8099->8099/tcp, :::8099->8099/tcp composetest_pinot-broker_1
1712fba17460 apachepinot/pinot:release-0.7.1 "./bin/pinot-admin.s…" 2 hours
ago Up 2 hours 8096-8099/tcp, 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp
composetest_pinot-controller_1 aeac35e847b6 confluentinc/cp-kafka:5.3.0
"/etc/confluent/dock…" 2 days ago Up 2 hours 0.0.0.0:9092->9092/tcp,
:::9092->9092/tcp composetest_kafka_1 050bf6511ab9 zookeeper:latest "/docker-
entrypoint.…" 2 days ago Up 2 hours 2888/tcp, 3888/tcp,
0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 8080/tcp composetest_zookeeper_1```
Can anyone kindly suggest how to proceed from here ?
**@fx19880617:** where are you running this addTable command?
**@fx19880617:** you need to run it inside pinot-controller container
**@fx19880617:** or ensure you can access pinot-controller:9000 from where you
want to issue the request
**@aaron:** I'm seeing that pinot-admin.sh will error out but not return a
nonzero return code, e.g. when data ingestion job specs are malformed
**@mayanks:** Should be a simple fix? Care to file a PR?
**@aaron:** Sure!
**@mayanks:** :thankyou:
**@aaron:**
**@mayanks:** Oh, this is an issue. Did you also want to file a PR for the
fix?
**@aaron:** I don't have a fix :smile:
**@ssubrama:** @aaron have you tried setting the system property
`pinot.admin.system.exit` to `true`?
**@aaron:** Oh I haven't, didn't know that was a thing
**@aaron:** Like, in JAVA_OPTS?
**@ssubrama:** I think it is via java opts. We look at it as
`System.getProperty()`
**@mayanks:** Oh yea I vaguely remember why we did this: ```"-Xms4G -Xmx4G
-Dpinot.admin.system.exit=true"```
**@aaron:** Great, I'll try that. Thank you!
**@mayanks:** Update the issue?
**@ssubrama:** @aaron did that work?
**@aaron:** Trying now
**@jmeyer:** Hello :slightly_smiling_face: Are subqueries supported in Pinot
? Such as 1\. `SELECT * FROM table WHERE x in (SELECT ...)` and 2\. `SELECT *
FROM (SELECT ...)` (i.e. working with the output of a subquery)
**@jmeyer:** My use case in particular right now is 2. Like: `SELECT
SUM(agg_value) FROM (SELECT dateString, AVG(value) as agg_value FROM table
GROUP BY dateString)` Or maybe there's another way to express this query ?
**@jmeyer:** The data looks like this ```date | 01 | 02 | 03 entity1 | 1 | 2 |
3 entity2 | 3 | 4 | 5``` The idea is to AVG over entity1 and entity2 per day,
then SUM those results over all days
**@mayanks:** Pinot does not support generic nested queries. There's some
limited use case like:
**@mayanks:** @jackie.jxt is there way to model this query using post-
aggregation?
**@jmeyer:** I've tried to find some info on post-aggregation (I've found a PR
about this), but couldn't find much ^^ Would really appreciate @jackie.jxt
input then :smile:
**@jackie.jxt:** Unfortunately this cannot be achieved with the post-
aggregation
**@jmeyer:** @mayanks @jackie.jxt Is there another approach to solving that
query ? Ideally as much as possible in realtime (i.e. no precomputation),
within Pinot
**@jackie.jxt:** In the subquery, we need to return the results for all groups
**@jmeyer:** Yes Can this subquery result be "exploited" further in any way ?
Without resorting to client level aggregations ?
**@jackie.jxt:** As a workaround, you may issue `SELECT AVG(value) FROM table
GROUP BY dateString LIMIT 10000` (assuming the cardinality of `dateString` is
smaller than 10000), then do the sum on the client side
**@jmeyer:** Yes, luckily dateString cardinality is low (<1000) I'll see how
performance fares in that case then Do you think any upcoming feature would
allow this query to be done in Pinot ?
**@jackie.jxt:** We don't have a feature already planned to support this
**@mayanks:** One option is to use Pinot + Presto if you have nested queries
like this.
**@jackie.jxt:** We can probably build a simple query engine on the broker to
support simple subquery like this
**@jackie.jxt:** Which takes a result table as the source
**@jackie.jxt:** Yes, using Presto might be an available option supported now
**@jmeyer:** > One option is to use Pinot + Presto if you have nested queries
like this. Sounds a bit overkill for that use case as it's not currently
available in my environment but thanks for the idea :slightly_smiling_face:
**@jmeyer:** @jackie.jxt I see, maybe some day then ! :smile:
**@jackie.jxt:** Can you please file an issue about this feature so that we
keep track of it
**@jmeyer:** Sure, will do
**@jmeyer:** Thank you very much for the feedback @mayanks & @jackie.jxt
**@mayanks:** :+1:
**@jmeyer:** I'm having a bit of trouble finding the appropriate terms to
describe the feature request, feel free to edit & add anything
:slightly_smiling_face:
**@luanmorenomaciel:** hi fellows, i'm trying to deploy the latest version of
pinot on my k8s but getting this funky issue, is there someone to shed some
lights? ```Starting a Pinot [SERVICE_MANAGER] at 0.08s since launch
Registering service status handler Skip Starting Pinot Service Manager admin
application Started Pinot [SERVICE_MANAGER] instance [ServiceManager_pinot-
broker-0.pinot-broker-headless.datastore.svc.cluster.local_-1] at 0.084s since
launch Starting a Pinot [BROKER] at 0.087s since launch Trying to start Pinot
Broker... Failed to initialize Pinot Broker Starter
java.lang.NumberFormatException: For input string: "" at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
~[?:1.8.0_292] at java.lang.Integer.parseInt(Integer.java:592) ~[?:1.8.0_292]
at java.lang.Integer.parseInt(Integer.java:615) ~[?:1.8.0_292] at
org.apache.pinot.core.util.ListenerConfigUtil.buildBrokerConfigs(ListenerConfigUtil.java:107)
~[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435] at
org.apache.pinot.broker.broker.helix.HelixBrokerStarter.<init>(HelixBrokerStarter.java:131)
~[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435] at
org.apache.pinot.tools.service.PinotServiceManager.startBroker(PinotServiceManager.java:133)
~[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435] at
org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:92)
~[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435] at
org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.lambda$run$0(StartServiceManagerCommand.java:259)
~[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435] at
org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:285)
[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435] at
org.apache.pinot.tools.admin.command.StartServiceManagerCommand.access$000(StartServiceManagerCommand.java:56)
[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435] at
org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.run(StartServiceManagerCommand.java:259)
[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-2de40fde8051c2c0281416c2da11c179c2190435]
Failed to start a Pinot [BROKER] at 0.114 since launch```
**@mayanks:** Seems you didn't specify query port in broker config?
**@luanmorenomaciel:** hmm really? i thought it would come as default let me
check thank you for getting back so rapidly!
**@luanmorenomaciel:** `service:` `annotations: {}` `clusterIP: "None"`
`externalIPs: []` `loadBalancerIP: ""` `loadBalancerSourceRanges: []` `type:
ClusterIP` `protocol: TCP` `gcpInternalLB: false` `port: 8099` `name: broker`
`nodePort: ""`
**@fx19880617:** hmm, which helm version are you using to start pinot?
**@fx19880617:** i think the issue is config refer:
**@fx19880617:** you can either add a new field `port:8099` in values.yaml
under broker section
**@fx19880617:** or modify the configmap template
**@luanmorenomaciel:** that's great i got this issue for all the 0.7.0 down to
0.6.0
**@fx19880617:** it’s not image issue
**@fx19880617:** it’s the helm chart template
**@fx19880617:** This pr should fix it:
**@luanmorenomaciel:** that's amazing i'll test it thank yoy
### _#pinot-dev_
**@ssubrama:** I am not sure what a good default is, since that is decided by
the way the ingestion stream behaves, and the hardware that is available. I
would leave it at zero and let installations tune it if they see a bottleneck
in this area.
\--------------------------------------------------------------------- To
unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org For additional commands,
e-mail: dev-help@pinot.apache.org