You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "ankitsultana (via GitHub)" <gi...@apache.org> on 2023/09/19 19:32:44 UTC
[GitHub] [pinot] ankitsultana opened a new issue, #11626: Helix Incorrectly Reports Pending Messages (potentially)
ankitsultana opened a new issue, #11626:
URL: https://github.com/apache/pinot/issues/11626
We use the following MBean to track Helix Pending messages in Pinot Servers:
```
CLMParticipantReport:MonitorType=ParticipantMessageMonitor,ParticipantName=some_instance_id:
```
We have often seen that this MBean reports that there are Helix pending messages, but there's no thread processing any Helix messages. There are also no messages in `INSTANCES/<instance-id>/MESSAGES` ZNode.
In this particular instance, I see the following things happened around the time we saw this issue:
* There was a big GC pause
* Zk client lost connection
* There's a Helix log message: `Tasks that never commenced execution after 200`. This is followed by a list of 238 FutureTask, and the MBean is set to exactly 238 Pending Messages
* There were failures in deleting Helix messages in `INSTANCES/<instance-id>/MESSAGES`. There were 9 such messages.
Some other signals can be seen in the attached Grafana screenshot.
<img width="1722" alt="image" src="https://github.com/apache/pinot/assets/8644710/9cc301bd-3d3f-4983-a810-88f5ff0abb62">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #11626: Helix Incorrectly Reports Pending Messages (potentially)
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #11626:
URL: https://github.com/apache/pinot/issues/11626#issuecomment-1732156420
Can you help file a ticket to Helix about this issue? Hopefully it is already fixed in their newer versions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Helix Incorrectly Reports Pending Messages (potentially) [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #11626:
URL: https://github.com/apache/pinot/issues/11626#issuecomment-6280341509
This issue has been closed because it has been inactive for 180 days. If you believe this issue is still relevant, please feel free to reopen it or create a new issue. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] ankitsultana commented on issue #11626: Helix Incorrectly Reports Pending Messages (potentially)
Posted by "ankitsultana (via GitHub)" <gi...@apache.org>.
ankitsultana commented on issue #11626:
URL: https://github.com/apache/pinot/issues/11626#issuecomment-1729961481
It is from Helix: https://github.com/apache/helix/blob/79cefa3ebe4059fce238256e0a12a364e290bd88/metrics-common/src/main/java/org/apache/helix/monitoring/mbeans/MonitorDomainNames.java#L31
I found there are some other scenarios as well where Helix can falsely report pending message. e.g. it happened again just now but there was no log like "Tasks that never commenced execution" this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #11626: Helix Incorrectly Reports Pending Messages (potentially)
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #11626:
URL: https://github.com/apache/pinot/issues/11626#issuecomment-1728502838
Is this metric emitted by Pinot or Helix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Helix Incorrectly Reports Pending Messages (potentially) [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #11626:
URL: https://github.com/apache/pinot/issues/11626#issuecomment-5737798398
This issue has been automatically marked as stale because it has not had any activity in the last 150 days. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Helix Incorrectly Reports Pending Messages (potentially) [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #11626: Helix Incorrectly Reports Pending Messages (potentially)
URL: https://github.com/apache/pinot/issues/11626
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org