Apache Pinot Daily Email Digest (2022-06-05)

[Pinot Slack Email Digest <ap...@gmail.com>]

### _#general_

  
 **@iamtherealdarknight:** @iamtherealdarknight has joined the channel  

###  _#random_

  
 **@iamtherealdarknight:** @iamtherealdarknight has joined the channel  

###  _#troubleshooting_

  
 **@ysuo:** Hi, what’s the common reason for errorCode 235? [ { “message”:
“ServerSegmentMissing:\n17 segments [table_name__1__22__20220527T1518Z,
table_name__1__10__20220516T1300Z, table_name__1__14__20220517T1900Z, missing
on server: Server_pinot-server-28.pinot-server-
headless.pinot.svc.cluster.local_8098", “errorCode”: 235 } ]  
**@mayanks:** Check ideal-state and external-view for the table for these
segments. This error message means that the server did not find these
segments.  
**@ysuo:** I checked the external-view, it said the segment is on another
server instance. The table status shows good.  
**@mayanks:** Hmm, that means the broker routing table is not updated (that
shouldn’t happen). Can you rebuild broker resource in swagger, that should
update the routing table.  
**@ysuo:** You mean this one?  
**@mayanks:** yes  
**@ysuo:** Thanks. I’ll have a try.  
**@ysuo:** ```{ "status": "Broker resource is not rebuilt because ideal state
is the same for table: table_test_REALTIME" }```  
**@mayanks:** I think there is api on broker to print the routing table, you
can check that to see why the broker thinks these segments are on this server  
**@mayanks:** `/debug/routingTable/{tableName}`  
**@mayanks:** Also, are you sure the IS/EV say these segments are in another
sever?  
**@ysuo:** yes, ```, "table_test__1__585__20220605T0324Z": { "Server_pinot-
server-27.pinot-server-headless.pinot.svc.cluster.local_8098": "ONLINE" },
"table_test__1__586__20220605T0354Z": { "Server_pinot-server-27.pinot-server-
headless.pinot.svc.cluster.local_8098": "CONSUMING" },```  
**@ysuo:** it’s on server 27. But query shows segment not on server 28.  
**@mayanks:** ok, then check the broker routing table  
**@ysuo:** There is only /debug/tables/{tableName}  
**@ysuo:** I can’t find /debug/routingTable/{tableName}  
**@mayanks:** :brokerPort/  
**@mayanks:** It is on broker  
**@ysuo:** It said this segment is on server 28.  
**@ysuo:** Is there anything I can do to solve this error?  
**@mayanks:** Hmm, this is not expected. What operations were done on the
cluster? For now, restarting the broker should fix this. But. I haven’t really
seen this behavior in the past. cc: @jackie.jxt  
**@ysuo:** I think I didn’t do any operation that may cause this
problem.:sweat_smile:  
**@ysuo:** edit server tag?  
**@mayanks:** Ah, did you rebalance after that?  
**@ysuo:** didn’t update any table config to use that new tag, so not
rebalance operation.  
**@ysuo:** restart broker didn’t help  
**@mayanks:** Please run rebalance  
**@ysuo:** rebalanced broker and server  
**@ysuo:** same error there.  
**@mayanks:** This does not make sense  
**@mayanks:** Broker routing table is built from EV. If EV is correct I don’t
understand why wrong server shows up in routing table  
**@mayanks:** Did you confirm that routing table shows incorrect server  
**@ysuo:** Three segments were missing from server 28. After restart broker,
rebalance broker and servers, there is only one segment has this error. It’s a
consuming status segment.  
**@mayanks:** When you did rebalance, did you including consuming segment?  
**@mayanks:** If not, do that.  
**@ysuo:** just after you told me to run rebalance. I did check consuming.  
**@ysuo:**  
**@ysuo:** Should reassign instance be checked?  
**@mayanks:** yeah try that  
**@ysuo:** It seems a new consuming status segment has this issue on server
28.  
**@mayanks:** Try `reset` segment api  
**@ysuo:** I tried reset. reset segment success returned. Then I query this
table, no error. But run the same query again, same error occurred.  
**@ysuo:** :sweat:  
**@mayanks:** After reset segment is the consuming segment CONSUMING or ERROR
in EV?  
**@mayanks:** And after reset + rebalance is it on server 27 or 28? And the
new error is complaning about the same consuming segment to be on incorrect
server?  
**@ysuo:** Hi, i just found server-27 and server-28 have the same hostName.  
**@ysuo:** I really don’t know how is that happening.  
**@mayanks:** I am confused, if they have 27 and 28 in the name, how is the
host name same?  
**@mayanks:** But sounds like that is likely the issue, Your hostnames are
messed up, but that is outside of Pinot  
**@ysuo:** restarted server 27. I’ll check if there’s still trouble.  
**@ysuo:** I’m confused too.  
**@ysuo:** Hi, it’s ok now. Thank you so much.  
**@mayanks:** Ok so restart or sever fixed it?  
**@ysuo:** restarted server 27 which has a wrong hostName.  
**@ysuo:** It’s strange. Edit server tag could cause this problem?  
**@mayanks:** I’ll need to know the exact events, but seems like it  
**@ysuo:** :sweat_smile:  
 **@iamtherealdarknight:** @iamtherealdarknight has joined the channel  

###  _#getting-started_

  
 **@iamtherealdarknight:** @iamtherealdarknight has joined the channel  

###  _#introductions_

  
 **@iamtherealdarknight:** @iamtherealdarknight has joined the channel  
\--------------------------------------------------------------------- To
unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org For additional commands,
e-mail: dev-help@pinot.apache.org