You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "dtspence (via GitHub)" <gi...@apache.org> on 2024/04/17 22:39:38 UTC
[I] HostRegexTableLoadBalancer logging change for migrations [accumulo]
dtspence opened a new issue, #4469:
URL: https://github.com/apache/accumulo/issues/4469
**Is your feature request related to a problem? Please describe.**
The HostRegexTableLoadBalancer logs can be improved to help diagnose a tablet that is not migrating and migrations in-general.
**Describe the solution you'd like**
Within the HostRegexTableLoadBalancer:
1. Log the migrations that are not moving as DEBUG. The message is currently TRACE.
2. Log the number of migrations being performed separately from the log line of all migrations.
**Describe alternatives you've considered**
No other alternatives considered.
**Additional context**
The issue was related to a cluster start-up - a single tablet blocked migrations and were unable determine the specific tablet causing the issue. The cluster was using the HostRegexTableLoadBalancer.
Source reference:
https://github.com/apache/accumulo/blob/rel/2.1.2/core/src/main/java/org/apache/accumulo/core/spi/balancer/HostRegexTableLoadBalancer.java#L463
https://github.com/apache/accumulo/blob/rel/2.1.2/core/src/main/java/org/apache/accumulo/core/spi/balancer/HostRegexTableLoadBalancer.java#L529
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] HostRegexTableLoadBalancer logging change for migrations [accumulo]
Posted by "ctubbsii (via GitHub)" <gi...@apache.org>.
ctubbsii commented on issue #4469:
URL: https://github.com/apache/accumulo/issues/4469#issuecomment-2252196638
Closing due to lack of response, and it seems like this is fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] HostRegexTableLoadBalancer logging change for migrations [accumulo]
Posted by "EdColeman (via GitHub)" <gi...@apache.org>.
EdColeman commented on issue #4469:
URL: https://github.com/apache/accumulo/issues/4469#issuecomment-2063771291
There seems two parts to this request. Logging the number of migrations separately seems helpful.
As for the level change: The logging config is dynamic and can be changed at run-time. Why is it insufficient to set the HostRegexTableLoadBlancer to trace if that level of detail is necessary to troubleshoot? Maybe some debug level statements could be added that provide a summary (and maybe that's provided by logging the number?).
Often, we seem to ping-pong with logging in these cases. It's needed in a rare case so the level is elevated - then later a different issue occurs and it's noticed that the logging is excessive when things are operating as expected and are not providing useful information, so the log level is then reduced.
Rather than just changing the level, is there something that could provide a summary, maybe with counts of #moved, #to be moved, #seen before but not moved,... Something that would be a one-liner, but sufficient to show progress, amount of work remaining. Then, if that points to something not making progress, that would trigger modifying the log level if needed.
I am not saying that this change may be unnecessary or helpful in this situation - rather just asking the question to see if there is something that may add more value (even if it's more work).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] HostRegexTableLoadBalancer logging change for migrations [accumulo]
Posted by "dtspence (via GitHub)" <gi...@apache.org>.
dtspence commented on issue #4469:
URL: https://github.com/apache/accumulo/issues/4469#issuecomment-2065620709
@EdColeman
> As for the level change: The logging config is dynamic and can be changed at run-time. Why is it insufficient to set the HostRegexTableLoadBlancer to trace if that level of detail is necessary to troubleshoot? Maybe some debug level statements could be added that provide a summary (and maybe that's provided by logging the number?).
Thank you - in the particular issue we encountered - it took some time investigating (another team was performing the restart of the cluster) and I did not look at the code until later to see the TRACE statement. We encountered an issue again today and set the class to TRACE level to learn more about the problem.
@ddanielr
> We could make the change to add the summary count of moved, then if "moved" hasn't progressed in 5min, bump the "TRACE" message to "DEBUG"
We've done similar things in the code to help with "spammy" messages without requiring log4j2.property file changes.
f98c87c
Thank you - this sounds like a good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] HostRegexTableLoadBalancer logging change for migrations [accumulo]
Posted by "ddanielr (via GitHub)" <gi...@apache.org>.
ddanielr commented on issue #4469:
URL: https://github.com/apache/accumulo/issues/4469#issuecomment-2211389540
@dtspence are the changes in https://github.com/apache/accumulo/pull/4592 sufficient for this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] HostRegexTableLoadBalancer logging change for migrations [accumulo]
Posted by "ctubbsii (via GitHub)" <gi...@apache.org>.
ctubbsii closed issue #4469: HostRegexTableLoadBalancer logging change for migrations
URL: https://github.com/apache/accumulo/issues/4469
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] HostRegexTableLoadBalancer logging change for migrations [accumulo]
Posted by "ddanielr (via GitHub)" <gi...@apache.org>.
ddanielr commented on issue #4469:
URL: https://github.com/apache/accumulo/issues/4469#issuecomment-2063821160
> 1. Log the migrations that are not moving as DEBUG. The message is currently TRACE.
To avoid the logging option changes, we could probably take a look at this and do some caching.
> Rather than just changing the level, is there something that could provide a summary, maybe with counts of #moved, #to be moved, #seen before but not moved,... Something that would be a one-liner, but sufficient to show progress, amount of work remaining. Then, if that points to something not making progress, that would trigger modifying the log level if needed.
We could make the change to add the summary count of moved, then if "moved" hasn't progressed in 5min, bump the "TRACE" message to "DEBUG"
We've done similar things in the code to help with "spammy" messages without requiring log4j2.property file changes.
https://github.com/apache/accumulo/commit/f98c87cb5798c781364a7773d17b1325a73b9f5d
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] HostRegexTableLoadBalancer logging change for migrations [accumulo]
Posted by "ddanielr (via GitHub)" <gi...@apache.org>.
ddanielr commented on issue #4469:
URL: https://github.com/apache/accumulo/issues/4469#issuecomment-2063693600
Feature request is a logging change to assist with troubleshooting.
Should probably look at any other balancer code and update those as well.
Might be worth looking into if this is something that can be handled at a higher level in `3.1` vs in each individual balancer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org