You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/11/14 15:08:41 UTC
[GitHub] [pinot] drawal1 opened a new issue, #9793: Dead server and broker exist in ideal state - stopping kafka topic ingestion and leaving all tables in bad state
drawal1 opened a new issue, #9793:
URL: https://github.com/apache/pinot/issues/9793
Repro steps:
1. kafka/pinot setup on EKS
2. kafka topic retention time is 1 day. In that 1 day, the topic accumulates a small amount of data (~5MB)
3. Wait the 1 day
4. See that server and broker are dead but they exist in ideal state. See that kafka topic ingestion has stopped. See that all tables are in a bad state
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] drawal1 commented on issue #9793: Dead server and broker exist in ideal state - stopping kafka topic ingestion and leaving all tables in bad state
Posted by GitBox <gi...@apache.org>.
drawal1 commented on issue #9793:
URL: https://github.com/apache/pinot/issues/9793#issuecomment-1316184100
This is not related to Kafka retention/cleanup, it happens any time servers/brokers/controllers die and new ones take their place. Two issues - why are the servers/controllers/brokers dying? And why are the tables not transitioning to the new ones taking their place?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mayankshriv commented on issue #9793: Dead server and broker exist in ideal state - stopping kafka topic ingestion and leaving all tables in bad state
Posted by GitBox <gi...@apache.org>.
mayankshriv commented on issue #9793:
URL: https://github.com/apache/pinot/issues/9793#issuecomment-1313910682
Thanks for reporting this.
@61yao please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #9793: Dead server and broker exist in ideal state - stopping kafka topic ingestion and leaving all tables in bad state
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #9793:
URL: https://github.com/apache/pinot/issues/9793#issuecomment-1322848403
The table won't automatically switch to use the new servers/brokers if the instance id doesn't match. Users will need to manually trigger a rebalance/rebuildBrokerResourceFromHelixTags to switch to the new servers/brokers because pinot won't know if the old instances are permanently gone. Usually the way to replace dead instances is to start new instances with the same instance id, so that the table is automatically served by the new instances.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] drawal1 commented on issue #9793: Dead server and broker exist in ideal state - stopping kafka topic ingestion and leaving all tables in bad state
Posted by GitBox <gi...@apache.org>.
drawal1 commented on issue #9793:
URL: https://github.com/apache/pinot/issues/9793#issuecomment-1325578883
Thanks to @mayankshriv for pointing me in the right direction. The key is setting pinot.set.instance.id.to.hostname to true.
Here is what I did to ensure that controller/broker/server id's were not based on IP address, but set tot the hostname:
1 Use config files for starting controller/broker/server. For example, here are my commands from the kube yaml:
- pinot-admin.sh StartController -configFileName /pinot-config/pinot-controller.conf
- pinot-admin.sh StartBroker -clusterName pinot -zkAddress zookeeper:2181 -configFileName /pinot-config/pinot-broker.conf
- pinot-admin.sh StartServer -clusterName pinot -zkAddress zookeeper:2181 -configFileName /pinot-config/pinot-server.conf
2. Set pinot.set.instance.id.to.hostname to true in pinot-controller.conf, pinot-broker.conf, and pinot-server.conf. For example, here is what my conf files look like:
**pinot-controller.conf:**
controller.helix.cluster.name=pinot
controller.port=9000
#controller.vip.host=pinot-controller
#controller.vip.port=9000
controller.access.protocols.http.port=9000
controller.data.dir=/data
controller.zk.str=zookeeper:2181
pinot.set.instance.id.to.hostname=true
**pinot-broker.conf:**
pinot.broker.client.queryPort=8099
pinot.broker.routing.table.builder.class=random
pinot.set.instance.id.to.hostname=true
**pinot-server.conf:**
pinot.server.netty.port=8098
pinot.server.adminapi.port=8097
pinot.server.instance.dataDir=/data
pinot.set.instance.id.to.hostname=true
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] drawal1 commented on issue #9793: Dead server and broker exist in ideal state - stopping kafka topic ingestion and leaving all tables in bad state
Posted by GitBox <gi...@apache.org>.
drawal1 commented on issue #9793:
URL: https://github.com/apache/pinot/issues/9793#issuecomment-1317173712
why are the servers/controllers/brokers dying? - This may be due to memory pressure and so not a pinot issue. So the only question is why the tables are not transitioning to the new servers/brokers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #9793: Dead server and broker exist in ideal state - stopping kafka topic ingestion and leaving all tables in bad state
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #9793:
URL: https://github.com/apache/pinot/issues/9793#issuecomment-1329813321
Thanks for providing the detailed solution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang closed issue #9793: Dead server and broker exist in ideal state - stopping kafka topic ingestion and leaving all tables in bad state
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang closed issue #9793: Dead server and broker exist in ideal state - stopping kafka topic ingestion and leaving all tables in bad state
URL: https://github.com/apache/pinot/issues/9793
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org