You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pinot.apache.org by Pinot Slack Email Digest <ap...@gmail.com> on 2021/09/30 02:00:22 UTC
Apache Pinot Daily Email Digest (2021-09-29)
### _#general_
**@arunsundar4298:** @arunsundar4298 has joined the channel
**@ssandeepadas007:** @ssandeepadas007 has joined the channel
**@rhodges:** Dear Pinot Fans! This is a remainder OSA Con is coming up fast
and the call for proposals is closing soon. If you have an idea for a talk on
open source analytics (apps, projects, or both), we would love to hear from
you. Head over to the conference page to submit your talk. We'll be announcing
sessions and opening up attendee signup next week.
**@alinoorrahman:** @alinoorrahman has joined the channel
**@ssainz:** @ssainz has joined the channel
**@cebofil371:** @cebofil371 has joined the channel
### _#random_
**@arunsundar4298:** @arunsundar4298 has joined the channel
**@ssandeepadas007:** @ssandeepadas007 has joined the channel
**@alinoorrahman:** @alinoorrahman has joined the channel
**@ssainz:** @ssainz has joined the channel
**@cebofil371:** @cebofil371 has joined the channel
### _#troubleshooting_
**@gabuglc:** Hello guys, Any ideas how to troubleshoot this issue. My kafka
topic has 70M+ messages but pinot only consume 23M+
**@gabuglc:** I got this from my zookeeper log ```2021-09-28 06:26:58,746 -
WARN [NIOWorkerThread-3:NIOServerCnxn@371] - Unexpected exception zookeeper_1
| EndOfStreamException: Unable to read additional data from client, it
probably closed the socket: address = /172.31.5.244:36258, session =
0x1060148907c0003 zookeeper_1 | at
org.apache.zookeeper.server.NIOServerCnxn.handleFailedRead(NIOServerCnxn.java:170)
zookeeper_1 | at
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:333)
zookeeper_1 | at
org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:508)
zookeeper_1 | at
org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:154)
zookeeper_1 | at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
zookeeper_1 | at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
zookeeper_1 | at java.base/java.lang.Thread.run(Thread.java:829) zookeeper_1 |
2021-09-28 23:20:06,750 - WARN [NIOWorkerThread-2:NIOServerCnxn@371] -
Unexpected exception zookeeper_1 | EndOfStreamException: Unable to read
additional data from client, it probably closed the socket: address =
/172.31.12.84:51724, session = 0x1060148907c000c zookeeper_1 | at
org.apache.zookeeper.server.NIOServerCnxn.handleFailedRead(NIOServerCnxn.java:170)
zookeeper_1 | at
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:333)
zookeeper_1 | at
org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:508)
zookeeper_1 | at
org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:154)
zookeeper_1 | at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
zookeeper_1 | at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
zookeeper_1 | at java.base/java.lang.Thread.run(Thread.java:829)```
**@mayanks:** Any errors in server logs? Also check the debug endpoint for
any errors
**@gabuglc:** I'll try to check again but so far I haven't been able to find
error on server logs
**@gabuglc:**
```=10,scanInFilter=0,scanPostFilter=190,sched=fcfs,threadCpuTimeNs=1168435
pinot-server_1 | Consumed 100368 events from (rate:17162.791/s),
currentOffset=22965743, numRowsConsumedSoFar=3708712,
numRowsIndexedSoFar=3708712 pinot-server_1 | Client session timed out, have
not heard from server in 27990ms for sessionid 0x206014851500015 pinot-
server_1 | Client session timed out, have not heard from server in 27990ms for
sessionid 0x206014851500015, closing socket connection and attempting
reconnect pinot-server_1 | zookeeper state changed (Disconnected) pinot-
server_1 | KeeperState:Disconnected, SessionId: 206014851500015, instance:
Server_pinot-server-3_8001, type: PARTICIPANT pinot-server_1 | Opening socket
connection to server ip-172-31-3-50.ec2.internal/172.31.3.50:2181. Will not
attempt to authenticate using SASL (unknown error) pinot-server_1 | Socket
connection established, initiating session, client: /172.24.0.2:57324, server:
ip-172-31-3-50.ec2.internal/172.31.3.50:2181 pinot-server_1 | Session
establishment complete on server ip-172-31-3-50.ec2.internal/172.31.3.50:2181,
sessionid = 0x206014851500015, negotiated timeout = 30000 pinot-server_1 |
zookeeper state changed (SyncConnected) pinot-server_1 | KeeperState:
SyncConnected, instance: Server_pinot-server-3_8001, type: PARTICIPANT pinot-
server_1 | Client session timed out, have not heard from server in 22145ms for
sessionid 0x206014851500015 pinot-server_1 | Client session timed out, have
not heard from server in 22145ms for sessionid 0x206014851500015, closing
socket connection and attempting reconnect pinot-server_1 | zookeeper state
changed (Disconnected) pinot-server_1 | KeeperState:Disconnected, SessionId:
206014851500015, instance: Server_pinot-server-3_8001, type: PARTICIPANT
pinot-server_1 | Consumed 100395 events from (rate:2223.7852/s),
currentOffset=23066138, numRowsConsumedSoFar=3809107,
numRowsIndexedSoFar=3809107 pinot-server_1 | Opening socket connection to
server pinot-server-2/172.31.11.39:2181. Will not attempt to authenticate
using SASL (unknown error) pinot-server_1 | Socket connection established,
initiating session, client: /172.24.0.2:34540, server: pinot-
server-2/172.31.11.39:2181 pinot-server_1 | Session establishment complete on
server pinot-server-2/172.31.11.39:2181, sessionid = 0x206014851500015,
negotiated timeout = 30000 pinot-server_1 | zookeeper state changed
(SyncConnected) pinot-server_1 | KeeperState: SyncConnected, instance:
Server_pinot-server-3_8001, type: PARTICIPANT```
**@gabuglc:** this is what i got from server
**@mayanks:** Seems like server got disconnected from ZK? Is server GC’ing?
**@gabuglc:** No, the server is not GC'ing
**@mayanks:** Another thing to check if events confirm to schema in Pinot
**@gabuglc:** where can i check this?
**@arunsundar4298:** @arunsundar4298 has joined the channel
**@ssandeepadas007:** @ssandeepadas007 has joined the channel
**@ssandeepadas007:** Hi guys, I am trying to fetch data from pinot using
golang application. After fetching the data from table I have to iterate(for
loop) it in order to print the data. Since the efficiency of the application
is considerably going down in the above method. Is there any alternate method
to get the data without iterating? I am following this link for reference -
cc: @arunsundar4298
**@mayanks:** Is it the iteration that is slow, or could there be other issues
(eg huge response size, deserialization, etc)?
**@ssandeepadas007:** We have a huge amount of data so the iteration makes it
slow.
**@nadeemsadim:** it depends on the limit size in query how many rows you
fetch .. I guess you are concerned aboutr latency added because of pinot query
response json processing?
**@ssandeepadas007:** Yes. The same issue. Because the limit value is huge
**@g.kishore:** what is the use case and the query
**@ssandeepadas007:** UseCase - Select all the values from the table, store it
in a structure so that we can pass it as a json to the user. Query - General
select query but the limit is all the data in the table (If there is 1 lakh
record, we have to parse it all)
**@nadeemsadim:** you will have to limit query since pinot only returns 10
rows by default .. so you have to put your upper limit of count of rows you
want to fetch
**@nadeemsadim:** there is no way to fetch all records in table by default
**@ssandeepadas007:** Is there any other way we can handle the latency
**@nadeemsadim:** this is the processing latency ..you can calculate how many
milliseconds its taking to process one json record and then get an estimate
what wi;ll be the processing time for n no of records .. also check which line
no is conusming max amount of processsing time
**@nadeemsadim:** in your processing logic
**@zineb.raiiss:** Hello, I'm trying to load the data from my csv file so i
generated the .yml file but this error is showing me on terminal. pls help me
if you have any idea about this problem!!
**@xiangfu0:** is this table `tools` created ?
**@xiangfu0:** can you run `curl localhost:9000/tables/tools/schema` and see
what’s the output
**@alinoorrahman:** @alinoorrahman has joined the channel
**@ssainz:** @ssainz has joined the channel
**@alinoorrahman:** Hi, I'd like to run `bin/quick-start-hybrid.sh` with
`pinot.broker.enable.query.limit.override=true`
`pinot.broker.query.response.limit=10000`, how can I do this? I have tried
`bin/quick-start-hybrid.sh -configFileName conf/pinot-broker.conf` but
`-configFileName` is not accepted as a valid argument.
**@xiangfu0:** quickstart doesn’t have those options, you need to start pinot
components separately with config files
**@alinoorrahman:** Thanks
**@alinoorrahman:** The reason I wanted to set the query limit is because when
using the jdbc driver, by default every query without a limit has a `limit
2147483647` appended to it. This seems to cause issues with some queries e.g.
`SELECT airlineStats.Origin, SUM(airlineStats.AirTime) FROM airlineStats GROUP
BY airlineStats.Origin LIMIT 100` works but `SELECT airlineStats.Origin,
SUM(airlineStats.AirTime) FROM airlineStats GROUP BY airlineStats.Origin LIMIT
2147483647` doesn't.
**@alinoorrahman:** I'm connecting to pinot using tableau+jdbc and I don't see
anything way of setting a limit on the driver or its config.
**@alinoorrahman:** Anyway, this seems like a bug?
**@xiangfu0:** I think this is a bug. @jackie.jxt does Pinot preallocating
space based on the limit groups?
**@jackie.jxt:** @alinoorrahman Which version of pinot are you running?
**@jackie.jxt:** We fixed a bug related to large limit, but that only applies
to selection query:
**@jackie.jxt:** @alinoorrahman What exception did you see when running
`SELECT airlineStats.Origin, SUM(airlineStats.AirTime) FROM airlineStats GROUP
BY airlineStats.Origin LIMIT 2147483647`?
**@cebofil371:** @cebofil371 has joined the channel
**@gabuglc:** Hello, Can I use the same zookeeper service that I use with
kafka for pinot? Or is it recommended to have 2 zk service for each one (
pinot and kafka )
**@dlavoie:** I would recommend two seperate zookeeper given that Kafka and
Pinot have very different usage pattern.
**@dlavoie:** Now, technically you can use both.
**@g.kishore:** +1 to what Daniel mentioned. you can if you really have to but
note that ZK does not have inbuilt multitenancy.. for instance - If your kafka
consumers are using ZK for checkpointing then a bad consumer can bring down
entire ZK and impact both Kafka and Pinot
\--------------------------------------------------------------------- To
unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org For additional commands,
e-mail: dev-help@pinot.apache.org