You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pinot.apache.org by Pinot Slack Email Digest <ap...@gmail.com> on 2022/02/22 02:00:26 UTC
Apache Pinot Daily Email Digest (2022-02-21)
### _#general_
**@juan:** @juan has joined the channel
**@praveen82:** @praveen82 has joined the channel
**@trustokoroego:** Hello! I need to connect presto to Pinot with basic Auth.
Could anyone point point me to how I can set this in the pinot.properties
Presto catalog configuration.
**@dadelcas:** This has been added to Trino recently. I'm not sure if presto
supports that, last time I checked the connector was a bit behind the latest
releases of pinot
**@mayanks:** I think that can only be done via HTTP headers in Presto.
@xiangfu0?
**@trustokoroego:** @mayanks I am not sure how to configure the HTTP header,
since I am using the Pinot connector config. How do I do it via HTTP header in
presto?
**@mayanks:** Ah I see. Will need to find, unless @xiangfu0 comes back sooner
**@mingleifly:** Hi, Why do we use DISTINCTCOUNT instead of using
COUNTDISTINCT here ?
**@mark.needham:** `distinctcount` is the name of the function that does exact
distinct counting?
**@mingleifly:** Yes. That’s right. Does it name should be COUNTDISTINCT ?
**@mark.needham:** Oh I see. yeh I'm not sure why the name is that way around,
@mayanks will probably know
**@mayanks:** It is there from PQL days. Now that we moved to SQL a while
back, you can also use the sql syntax of count(distinct, …).
**@dadelcas:** Hello, I've got a question about realtime tables. If I'm
correct the kafka consumer group ID is built in the code using the table name
and replica ID, however I'm not able to find a consumer group for the table in
my kafka cluster. Is there a way to list all the consumer groups that a
realtime table is using? I would look like those IDs are stored in ZK under
ideal states but I can't find them. Thanks
**@apte.kaivalya:** IIUC pinot doesn’t use consumer groups to consume from
kafka and it is `null`. This is done mainly to let multiple pinot consumers
consume from the same partition (for replication).
**@apte.kaivalya:** context:
**@dadelcas:** Thanks for forwarding that answer! I think that may relate to
an issue I see in one of our tables where we set `group.id`
**@dadelcas:** I'd still like to track offsets on each partition if this
possible at all
**@mark.needham:** Pinot does keep track of the offsets in individual segments
(which are also mapped to partitions, although the partition value is embedded
in the segment name, so it's maybe not exactly what you 're after
**@apte.kaivalya:** Is there a metric that Pinot exposes for tracking lag in
consuming?
**@dadelcas:** I'd like to know which offset it's been consumed at any time.
The information in the segment metadata only provides start and end offsets
for the segment. Alternatively I'd like to be able to store the offset and
partition with every ingested row
**@mark.needham:** how about the consuming segment info REST endpoint?
**@mark.needham:** e.g. ```curl -X GET "" -H "accept: application/json"```
```{ "_segmentToConsumingInfoMap": { "parkrun__0__142__20220219T2118Z": [ {
"serverName": "Server_172.21.0.6_8098", "consumerState": "CONSUMING",
"lastConsumedTimestamp": 1645452573998, "partitionToOffsetMap": { "0":
"13991875" } } ] } }```
**@dadelcas:** Thanks Mark, that may do the trick. Just one more question, if
I want to include consumer record metadata in my table (let's say headers,
offset, etc) is there an easy way to extract this? Any examples you can point
me at?
**@mark.needham:** not that I know of, but lemme have a quick look at the code
**@mark.needham:** If we can't find anything, you could always write it up as
a GH issue -->
**@dadelcas:** I've seen issue 7004 opens the discussion to store those values
as table metadata but the ticket has been opened for 8 months now
**@mark.needham:** @mayanks I guess got stuck back in July. Anything we
should do to revive it?
**@mayanks:** @dadelcas Can you comment on the issue describing your need and
+1, it will greatly help.
**@dadelcas:** Yup, done
**@naga.b:** @naga.b has joined the channel
**@jatink.5251:** @jatink.5251 has joined the channel
**@jatink.5251:** Hello Everyone, Can we ingest data from presto to
pinot(offline table). ? Is there any work around on this?
**@manish.jaiswal:** @manish.jaiswal has joined the channel
### _#random_
**@juan:** @juan has joined the channel
**@praveen82:** @praveen82 has joined the channel
**@naga.b:** @naga.b has joined the channel
**@jatink.5251:** @jatink.5251 has joined the channel
**@manish.jaiswal:** @manish.jaiswal has joined the channel
### _#troubleshooting_
**@4shivamsajwan:** Hi all, I am trying to run Pinot locally by providing
zookeeper url as it is recommended. But I am getting this: Exception while
reading External view from zookeeper
**@4shivamsajwan:**
**@mark.needham:** can you share what commands you've run that lead you to
that state? Did you start a controller/broker/server?
**@4shivamsajwan:** @mark.needham I have setup using these instructions
**@mark.needham:** oh, that's weird. It seems like the external view is empty
for some reason. Are you able to navigate to And then click
PinotCluster/EXTERNALVIEW and look for 'brokerResource'
**@4shivamsajwan:** Yes i am able to navigate to brokerResource. here is how
it looks
**@juan:** @juan has joined the channel
**@praveen82:** @praveen82 has joined the channel
**@apte.kaivalya:** Hey :wave: , I am facing an issue where after some time
(usually a few mins) servers start dying and segments go BAD. These servers
are able to join back on restarting, but are DEAD again. In the controller
logs I see some errors like: ```Caught 'java.net.SocketTimeoutException: Read
timed out' while executing GET on URL: Connection error
java.util.concurrent.ExecutionException: java.net.SocketTimeoutException: Read
timed out at java.util.concurrent.FutureTask.report(FutureTask.java:122)
~[?:?] at java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:?] at
org.apache.pinot.controller.util.CompletionServiceHelper.doMultiGetRequest(CompletionServiceHelper.java:79)
~[pinot-all-0.10.0-SNAPSHOT-jar-with-
dependencies.jar:0.10.0-SNAPSHOT-8bbf93aa4377dbdf597e7940670893330452b33f] at
org.apache.pinot.controller.api.resources.ServerTableSizeReader.getSegmentSizeInfoFromServers(ServerTableSizeReader.java:69)
~[pinot-all-0.10.0-SNAPSHOT-jar-with-
dependencies.jar:0.10.0-SNAPSHOT-8bbf93aa4377dbdf597e7940670893330452b33f] at
org.apache.pinot.controller.util.TableSizeReader.getTableSubtypeSize(TableSizeReader.java:181)
~[pinot-all-0.10.0-SNAPSHOT-jar-with-
dependencies.jar:0.10.0-SNAPSHOT-8bbf93aa4377dbdf597e7940670893330452b33f] at
org.apache.pinot.controller.util.TableSizeReader.getTableSizeDetails(TableSizeReader.java:101)
~[pinot-all-0.10.0-SNAPSHOT-jar-with-
dependencies.jar:0.10.0-SNAPSHOT-8bbf93aa4377dbdf597e7940670893330452b33f] at
org.apache.pinot.controller.api.resources.TableSize.getTableSize(TableSize.java:83)
~[pinot-all-0.10.0-SNAPSHOT-jar-with-
dependencies.jar:0.10.0-SNAPSHOT-8bbf93aa4377dbdf597e7940670893330452b33f] at
jdk.internal.reflect.GeneratedMethodAccessor818.invoke(Unknown Source) ~[?:?]
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
~[pinot-all-0.10.0-SNAPSHOT-jar-with-
dependencies.jar:0.10.0-SNAPSHOT-8bbf93aa4377dbdf597e7940670893330452b33f] at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
[pinot-all-0.10.0-SNAPSHOT-jar-with-
dependencies.jar:0.10.0-SNAPSHOT-8bbf93aa4377dbdf597e7940670893330452b33f] at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
[pinot-all-0.10.0-SNAPSHOT-jar-with-
dependencies.jar:0.10.0-SNAPSHOT-8bbf93aa4377dbdf597e7940670893330452b33f] at
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)
[pinot-all-0.10.0-SNAPSHOT-jar-with-
dependencies.jar:0.10.0-SNAPSHOT-8bbf93aa4377dbdf597e7940670893330452b33 f] at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
[pinot-all-0.10.0-SNAPSHOT-jar-with-
dependencies.jar:0.10.0-SNAPSHOT-8bbf93aa4377dbdf597e7940670893330452b33f] at
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)
[pinot-all-0.10.0-SNAPSHOT-jar-with-
dependencies.jar:0.10.0-SNAPSHOT-8bbf93aa4377dbdf597e7940670893330452b33f] at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)
[pinot-all-0.10.0-SNAPSHOT-jar-with-
dependencies.jar:0.10.0-SNAPSHOT-8bbf93aa4377dbdf597e7940670893330452b33f] at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)
[pinot-all-0.10.0-SNAPSHOT-jar-with-
dependencies.jar:0.10.0-SNAPSHOT-8bbf93aa4377dbdf597e7940670893330452b33f] at
org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)
[pinot-all-0.10.0-SNAPSHOT-jar-with-
dependencies.jar:0.10.0-SNAPSHOT-8bbf93aa4377dbdf597e7940670893330452b33f] at
org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [pinot-
all-0.10.0-SNAPSHOT-jar-with-
dependencies.jar:0.10.0-SNAPSHOT-8bbf93aa4377dbdf597e7940670893330452b33f] at
org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [pinot-
all-0.10.0-SNAPSHOT-jar-with-
dependencies.jar:0.10.0-SNAPSHOT-8bbf93aa4377dbdf597e7940670893330452b33f] at
org.glassfish.jersey.internal.Errors.process(Errors.java:292) [pinot-
all-0.10.0-SNAPSHOT-jar-with-
dependencies.jar:0.10.0-SNAPSHOT-8bbf93aa4377dbdf597e7940670893330452b33f] at
org.glassfish.jersey.internal.Errors.process(Errors.java:274) [pinot-
all-0.10.0-SNAPSHOT-jar-with-
dependencies.jar:0.10.0-SNAPSHOT-8bbf93aa4377dbdf597e7940670893330452b33f] at
org.glassfish.jersey.internal.Errors.process(Errors.java:244) [pinot-
all-0.10.0-SNAPSHOT-jar-with-
dependencies.jar:0.10.0-SNAPSHOT-8bbf93aa4377dbdf597e7940670893330452b33f] at
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
[pinot-all-0.10.0-SNAPSHOT-jar-with-
dependencies.jar:0.10.0-SNAPSHOT-8bbf93aa4377dbdf597e7940670893330452b33f] at
org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
[pinot-all-0.10.0-SNAPSHOT-jar-with-
dependencies.jar:0.10.0-SNAPSHOT-8bbf93aa4377dbdf597e7940670893330452b33f] at
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:679)
[pinot-all-0.10.0-SNAPSHOT-jar-with-
dependencies.jar:0.10.0-SNAPSHOT-8bbf93aa4377dbdf597e7940670893330452b33f] at
org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:353)
[pinot-all-0.10.0-SNAPSHOT-jar-with-
dependencies.jar:0.10.0-SNAPSHOT-8bbf93aa4377dbdf597e7940670893330452b33f] at
org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200)
[pinot-all-0.10.0-SNAPSHOT-jar-with-
dependencies.jar:0.10.0-SNAPSHOT-8bbf93aa4377dbdf597e7940670893330452b33f] at
org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569)
[pinot-all-0.10.0-SNAPSHOT-jar-with-
dependencies.jar:0.10.0-SNAPSHOT-8bbf93aa4377dbdf597e7940670893330452b33f] at
org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549)
[pinot-all-0.10.0-SNAPSHOT-jar-with-
dependencies.jar:0.10.0-SNAPSHOT-8bbf93aa4377dbdf597e7940670893330452b33f] at
java.lang.Thread.run(Thread.java:829) [?:?] Caused by:
java.net.SocketTimeoutException: Read timed out at
java.net.SocketInputStream.socketRead0(Native Method) ~[?:?] ``` Also I see
some zk disconnections: ```Consumed 0 events from (rate:0.0/s),
currentOffset=894238198, numRowsConsumedSoFar=555294,
numRowsIndexedSoFar=555294 [Consumer clientId=consumer-null-2, groupId=null]
Seeking to offset 894406661 for partition email-raw-events-62 [Consumer
clientId=consumer-null-53, groupId=null] Seeking to offset 894229286 for
partition email-raw-events-98 Consumed 0 events from (rate:0.0/s),
currentOffset=894328031, numRowsConsumedSoFar=645137,
numRowsIndexedSoFar=645137 [Consumer clientId=consumer-null-52, groupId=null]
Seeking to offset 894221230 for partition email-raw-events-86 [Consumer
clientId=consumer-null-58, groupId=null] Seeking to offset 894238198 for
partition email-raw-events-26 [Consumer clientId=consumer-null-43,
groupId=null] Seeking to offset 894338250 for partition email-raw-events-68
zookeeper state changed (Disconnected) ``` I don’t see any issues on the ZK
cluster. Any pointers ?
**@richard892:** hi I'll take a look and get back to you
**@apte.kaivalya:** Thanks @richard892
**@mayanks:** ZK disconnects usually happen if the servers are GC’ing. Can you
check on that @apte.kaivalya?
**@richard892:** @apte.kaivalya you can capture gc pause times without
restarting by running `jcmd <controller pid> JFR.start duration=60s
filename=controller.jfr settings=profile`
**@apte.kaivalya:** Thanks @mayanks and @richard892 let me use this and see
what I get.
**@walterddr:** also the error log comes from an external facing API
`/table/{tableName}/size` which is usually called by pinot UI if you open the
pinot controller link. this doesn't seem to be the root cause but rather an
indicator that the server has already gone dead.
**@walterddr:** were you able to check the server log?
**@aditya.patwal:** Hi Folks, I am experimenting with pinot merge and rollup
task It failed to download the segments from s3. Couldn't find and config
reference for minions similar to server
`pinot.server.segment.fetcher.protocols` Is it possible to use s3 deep store
with minions? What is the config for this?
**@richard892:** what's your configuration?
**@npawar:** It's on this page, scroll all the way to the bottom
**@aditya.patwal:** Thank you! The task was able to download from s3
**@aditya.patwal:** The merge and rollup worked but got another err
```2022/02/21 16:40:26.018 ERROR [PinotSegmentUploadDownloadRestletResource]
[grizzly-http-server-0] Any segments from 'segmentsTo' should not be available
in the table at this point``` data returned by brokers is as expected, rows
reduced after rollup but older segments still showed up in ideal state Tried
deleting all segments and re pushing old segments from s3, but deleting all
segments didn't cleared the segment lineage 2 of the old segments are
effectively hidden from brokers point of view I'll now try deleting the
segment lineage from zk
**@naga.b:** @naga.b has joined the channel
**@jatink.5251:** @jatink.5251 has joined the channel
**@manish.jaiswal:** @manish.jaiswal has joined the channel
**@mrpringle:** Think I have noticed a bug with how the controller (maybe
other components) allocates threads to http requests; it appears to be
unbounded and 1 thread per request? Found an issue in pen testing that we can
crash the controller by sending in thousands of http get requests to random
resources.
### _#pinot-dev_
**@jatink.5251:** @jatink.5251 has joined the channel
**@jatink.5251:** @g.kishore can you please suggest? Can we ingest data from
presto to pinot(offline table). ? Is there any work around on this?
**@g.kishore:** @elon.azoulay I know you have built this for trino-pinot, can
that be extended to presto-pinot?
### _#community_
**@juan:** @juan has joined the channel
### _#announcements_
**@juan:** @juan has joined the channel
### _#presto-pinot-connector_
**@juan:** @juan has joined the channel
### _#pinot-perf-tuning_
**@juan:** @juan has joined the channel
### _#getting-started_
**@juan:** @juan has joined the channel
**@trustokoroego:** Hello! I need to connect presto to Pinot with basic Auth.
Could anyone point point me to how I can set this in the pinot.properties
Presto catalog configuration.
**@xiangfu0:** you can set extra http headers
**@trustokoroego:** Any reference ? I see in Trino documentation, the pinot
connect has an explicit configuration. ```pinot.controller.authentication.type
pinot.controller.authentication.user pinot.controller.authentication.password
pinot.broker.authentication.user pinot.broker.authentication.password``` Just
to confirm if this apply to presto
**@trustokoroego:** @xiangfu0 could you please give more details on how to
apply your suggestion?
**@mayanks:** If I am guessing right, the HTTP headers to presto will be
passed to Pinot @xiangfu0? Or is there a different mechanism here?
**@xiangfu0:** you can use below config in presto to set any http headers
```pinot.extra-http-headers=k1:v1,k2:v2 ```
**@francois:** Hi :slightly_smiling_face: Even with the doc I cannot find a
proper answer ... Can a dimension table be an hybrid table ?
**@mark.needham:** a dimension table can only be an offline table. So I think
no to hybrid
**@francois:** Ok . thanks for you feedback
\--------------------------------------------------------------------- To
unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org For additional commands,
e-mail: dev-help@pinot.apache.org