Apache Pinot Daily Email Digest (2022-03-02)

[Pinot Slack Email Digest <ap...@gmail.com>]

### _#general_

  
 **@madison:** @madison has joined the channel  
 **@jaromir.hamala:** Hello, congratz on the tiered storage! I'm reading the
announcement and it says: _Note that this is not implemented as lazy-loading -
Pinot servers directly query data on the cloud and are never downloading the
entire segments locally._ May I ask how does it work? I know close to nothing
about S3, but I believe it's a dummy blob-store. You have to download blobs
with segments before querying them, don't you? Am I missing anything? Thanks
for any hint!  
**@mark.needham:** cc @npawar @chinmay.cerebro  
**@g.kishore:** We will publish another blog on the details but at a high
level S3 has api where you can read a part of the blob and we leverage that  
**@luisfernandez:** Where do we get this announcements?  
**@g.kishore:** This is on StarTree blog  
**@luisfernandez:** :open_mouth:  
**@luisfernandez:** this will definitely be beneficial for our use case thank
you!  
**@jaromir.hamala:** If I understand it right Apache Pinot won't copy the
whole blob, but just some parts of it. Thank you!  
**@g.kishore:** thats right, only the parts that are needed to process the
query.  
 **@kasifysf:** @kasifysf has joined the channel  
 **@kaashit26:** @kaashit26 has joined the channel  
 **@somasuntaram15:** @somasuntaram15 has joined the channel  
 **@vaibhav.goyal:** @vaibhav.goyal has joined the channel  

###  _#random_

  
 **@madison:** @madison has joined the channel  
 **@kasifysf:** @kasifysf has joined the channel  
 **@kaashit26:** @kaashit26 has joined the channel  
 **@somasuntaram15:** @somasuntaram15 has joined the channel  
 **@vaibhav.goyal:** @vaibhav.goyal has joined the channel  

###  _#troubleshooting_

  
 **@madison:** @madison has joined the channel  
 **@kasifysf:** @kasifysf has joined the channel  
 **@kaashit26:** @kaashit26 has joined the channel  
 **@luisfernandez:** hey friends, I have a question, in our dev pinot cluster
(built on top of k8s), we have been scaling up and down things like
servers/controllers and all, once we scale down something we can see in the
pinot UI that servers that we scale down from go to Dead status, when we add a
REALTIME table at that point, for some reason the status is Bad, we have fixed
this by removing those servers and then eventually we can add this, any reason
why Realtime tables complain even tho those servers are now gone? do we have
to run a rebalance before we even add a new realtime table?  
**@mayanks:** Perhaps you or used the same tag as the dead servers?  
**@luisfernandez:** that’s based on this basically, yes?  
**@luisfernandez:** ```"tenants": { "broker": "DefaultTenant", "server":
"DefaultTenant", "tagOverrideConfig": { "realtimeCompleted":
"DefaultTenant_OFFLINE" } },```  
**@npawar:** If you've taken that server down, you need to also untag it and
then drop instance  
**@luisfernandez:** sorry how do i drop the instance? I think I got the gist  
**@luisfernandez:** so the table is using all the servers that have the same
tag to distribute the data, so what i have to do is untag the dead servers and
i should be able to create the table afterwards yes?  
**@luisfernandez:** this is it yes  
**@luisfernandez:** why do i need to untag at all? can i just not drop the
instance?  
**@npawar:** i think you won’t be able to dropInstance, if the instance has a
tag that is being used by some table  
**@npawar:** but give it a shot  
**@luisfernandez:** it did let me drop instance  
**@luisfernandez:** i guess that since the table wasn’t yet in production  
**@luisfernandez:** it did let me  
**@luisfernandez:** as in we haven’t deployed the table*  
**@luisfernandez:** when we deployed the table we god status bad and we
deleted it  
**@luisfernandez:** deleted the instances and deployed table again and it
worked  
**@luisfernandez:** but if table was there then we had to untag, rebalance and
delete yes?  
**@npawar:** Yes that's right  
 **@somasuntaram15:** @somasuntaram15 has joined the channel  
 **@luisfernandez:** hey friends, another question, I’m writing a query with
aggregation functions, however if i do something like `SUM(impression_count)`
in the return types in the json, I see this ```"columnDataTypes": [ "DOUBLE"
]``` but impression_count is an int, why is the columnDataType a double? is
there any way to fix it in the query? thank you!  
**@richard892:** it gets aggregated as a double to avoid overflow  
 **@dadelcas:** hello, I've run into an issue where someone has scaled up our
pinot servers, left the new nodes for a while and then just removed the nodes
back from the cluster - the API to remove the instances was never invoked and
so was table rebalancing. Some segments were assigned to the new nodes and
after these nodes were removed the tables went into bad state. The current
state is that queries are not reading the segments in bad state. The table
external view only refers to active nodes but some segments point at the
removed nodes in the ideal state. The cluster still list the removed nodes
when the instances API is called. I've try few things but I can't get the
tables to be back in good state: I've rebalanced the table using different
options, I've disabled the removed nodes in pinot and rebalanced the tables
again, I've rebalanced the servers, but none of these have worked so far. I
wonder if someone could let me know the steps to fix the issue  
**@dadelcas:** Note this is happening to both realtime and offline tables. I
thought in a scenario like this another server would take over and the tables
would get repaired immediately but this doesn't seem to be the case  
**@dadelcas:** my controller is on pinot 0.9.0, all the other nodes are on
pinot 0.8.0  
**@mayanks:** Reading through your original issue. However, definitely
recommend not mixing and matching Pinot component versions  
**@mayanks:** Are you able to untag/remove the bad instances?  
**@dadelcas:** I was doing the upgrade as per the docs so we left the
controller running for a while when this happened  
**@dadelcas:** I can't remove the servers, the API returns 409 and it says the
server is used in ideal states  
**@dadelcas:** I haven't tried untagging, bear with me  
**@dadelcas:** I was able to update the tags  
**@dadelcas:** I've rebalanced the table again but nothing happens  
**@mayanks:** Hmm if tags are updated then rebalance would move them  
**@dadelcas:** Actually, I see more segments in bad state than before. I don't
think is related to what I've just done though  
**@mayanks:** Can you confirm the tag that you see on instances  
**@dadelcas:** Before untagging all were using DefaultTenant_OFFLINE and
DefaultTenant_ONLINE  
**@dadelcas:** I've replaced that with "removed" for all the dead servers  
**@mayanks:** Do you see the new tag on the instances though  
**@mayanks:** Also did you try rebalance with downtime  
**@dadelcas:** Yes the servers are tagged as expected  
**@dadelcas:** I've just tried with downtime and it works but I don't
understand why  
**@dadelcas:** I still have 3 servers running  
**@dadelcas:** Downtime is supposed to be used when a single instance is
running?  
**@dadelcas:** For offline tables that doesn't work  
**@dadelcas:** I've managed to removed the servers by calling the endpoint as
soon as I trigger rebalance on the table  
**@dadelcas:** One table would refresh its ideal state correctly but little
after it would refer to the dead nodes again  
**@dadelcas:** Some segments still report bad status, I'll leave like this for
a while  
**@dadelcas:** I just noticed that someone above had a similar issue  
**@dadelcas:** This is raising some questions, the fact that a node shuts down
should mean that the segments it holds should now be served by another node
transparently, is that not how it's supposed to work?  
**@mayanks:** Totally understand this is not good experience. Let me dm you to
help see what is going on  
**@dadelcas:** I've raised a github issue summarizing the problem at  please
let me know if it needs more details - I'll try to include some logs. Thank
you for the time you've spent looking at this problem  
**@mayanks:** Thanks @dadelcas. To close the loop, we mitigated the issue by
restarting controllers.  
 **@somasuntaram15:** Hi all. I am facing the below issue while creating
schema or table executing by CLI from pods. I have deployed Pinot in K8 using
helm charts. Kindly help me to fix this issue.  
**@somasuntaram15:** FYI, Pinot version - 0.7.1  
**@mayanks:** It says port is already in use? Check what process is using the
port that Pinot wants to use  
**@walterddr:** did you also deployed a non-k8s pinot cluster? if you are
trying to connect to a k8s pinot cluster you should start a port-forwarding
using kubectl control to access the localhost:9000 port directly (and that
might've also failed because address is already in used so check against the
process that occupies the port)  
 **@vaibhav.goyal:** @vaibhav.goyal has joined the channel  

###  _#pinot-dev_

  
 **@somasuntaram15:** @somasuntaram15 has joined the channel  
 **@slackbot:** This message was deleted.  
**@richard892:** hi, can you ask in <#C011C9JHN7R|troubleshooting> please?  
**@somasuntaram15:** sure. Thanks  
 **@moradi.sajjad:** FYI, I'm going to start the process for release 0.10.0
tomorrow based on the commit `fd9c58a` ().  
 **@npawar:** hey folks, we are missing all design docs here starting 2021: .
If you have anything that should go here, please take a moment and add it
:pray:  

###  _#announcements_

  
 **@francois:** @francois has joined the channel  

###  _#pinot-docsrus_

  
 **@luisfernandez:** @luisfernandez has joined the channel  
\--------------------------------------------------------------------- To
unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org For additional commands,
e-mail: dev-help@pinot.apache.org