You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2022/02/17 17:50:38 UTC
[GitHub] [accumulo] milleruntime opened a new issue #2506: SimpleLoadBalancer logging too many outstanding migrations
milleruntime opened a new issue #2506:
URL: https://github.com/apache/accumulo/issues/2506
The `SimpleLoadBalancer` logs too many outstanding migrations. I don't know why this is happening but anytime there is an outstanding migration, it will spam the Manager log with too many log messages.
<pre>
:17,605 [balancer.SimpleLoadBalancer] WARN : Not balancing due to 8 outstanding migrations.
2022-02-17T12:44:17,605 [balancer.SimpleLoadBalancer] DEBUG: Sample up to 10 outstanding migrations: 30;d;c, 30<;f, 30;e;d, 30;a;9, 30;f;e, 30;9;8, 30;b;a, 30;c;b
2022-02-17T12:44:17,605 [balancer.SimpleLoadBalancer] WARN : Not balancing due to 8 outstanding migrations.
2022-02-17T12:44:17,605 [balancer.SimpleLoadBalancer] DEBUG: Sample up to 10 outstanding migrations: 30;d;c, 30<;f, 30;e;d, 30;a;9, 30;f;e, 30;9;8, 30;b;a, 30;c;b
2022-02-17T12:44:17,605 [balancer.SimpleLoadBalancer] WARN : Not balancing due to 8 outstanding migrations.
2022-02-17T12:44:17,605 [balancer.SimpleLoadBalancer] DEBUG: Sample up to 10 outstanding migrations: 30;d;c, 30<;f, 30;e;d, 30;a;9, 30;f;e, 30;9;8, 30;b;a, 30;c;b
2022-02-17T12:44:17,605 [balancer.SimpleLoadBalancer] WARN : Not balancing due to 8 outstanding migrations.
2022-02-17T12:44:17,605 [balancer.SimpleLoadBalancer] DEBUG: Sample up to 10 outstanding migrations: 30;d;c, 30<;f, 30;e;d, 30;a;9, 30;f;e, 30;9;8, 30;b;a, 30;c;b
2022-02-17T12:44:17,605 [balancer.SimpleLoadBalancer] WARN : Not balancing due to 8 outstanding migrations.
2022-02-17T12:44:17,605 [balancer.SimpleLoadBalancer] DEBUG: Sample up to 10 outstanding migrations: 30;d;c, 30<;f, 30;e;d, 30;a;9, 30;f;e, 30;9;8, 30;b;a, 30;c;b
2022-02-17T12:44:17,605 [balancer.SimpleLoadBalancer] WARN : Not balancing due to 8 outstanding migrations.
2022-02-17T12:44:17,606 [balancer.SimpleLoadBalancer] DEBUG: Sample up to 10 outstanding migrations: 30;d;c, 30<;f, 30;e;d, 30;a;9, 30;f;e, 30;9;8, 30;b;a, 30;c;b
2022-02-17T12:44:17,606 [balancer.SimpleLoadBalancer] WARN : Not balancing due to 8 outstanding migrations.
2022-02-17T12:44:17,606 [balancer.SimpleLoadBalancer] DEBUG: Sample up to 10 outstanding migrations: 30;d;c, 30<;f, 30;e;d, 30;a;9, 30;f;e, 30;9;8, 30;b;a, 30;c;b
2022-02-17T12:44:17,606 [balancer.SimpleLoadBalancer] WARN : Not balancing due to 8 outstanding migrations.
2022-02-17T12:44:17,606 [balancer.SimpleLoadBalancer] DEBUG: Sample up to 10 outstanding migrations: 30;d;c, 30<;f, 30;e;d, 30;a;9, 30;f;e, 30;9;8, 30;b;a, 30;c;b
2022-02-17T12:44:17,606 [balancer.SimpleLoadBalancer] WARN : Not balancing due to 8 outstanding migrations.
2022-02-17T12:44:17,606 [balancer.SimpleLoadBalancer] DEBUG: Sample up to 10 outstanding migrations: 30;d;c, 30<;f, 30;e;d, 30;a;9, 30;f;e, 30;9;8, 30;b;a, 30;c;b
2022-02-17T12:44:17,606 [balancer.SimpleLoadBalancer] WARN : Not balancing due to 8 outstanding migrations.
2022-02-17T12:44:17,606 [balancer.SimpleLoadBalancer] DEBUG: Sample up to 10 outstanding migrations: 30;d;c, 30<;f, 30;e;d, 30;a;9, 30;f;e, 30;9;8, 30;b;a, 30;c;b
2022-02-17T12:44:17,606 [balancer.SimpleLoadBalancer] WARN : Not balancing due to 8 outstanding migrations.
2022-02-17T12:44:17,606 [balancer.SimpleLoadBalancer] DEBUG: Sample up to 10 outstanding migrations: 30;d;c, 30<;f, 30;e;d, 30;a;9, 30;f;e, 30;9;8, 30;b;a, 30;c;b
2022-02-17T12:44:17,606 [balancer.SimpleLoadBalancer] WARN : Not balancing due to 8 outstanding migrations.
2022-02-17T12:44:17,606 [balancer.SimpleLoadBalancer] DEBUG: Sample up to 10 outstanding migrations: 30;d;c, 30<;f, 30;e;d, 30;a;9, 30;f;e, 30;9;8, 30;b;a, 30;c;b
2022-02-17T12:44:17,606 [balancer.SimpleLoadBalancer] WARN : Not balancing due to 8 outstanding migrations.
2022-02-17T12:44:17,606 [balancer.SimpleLoadBalancer] DEBUG: Sample up to 10 outstanding migrations: 30;d;c, 30<;f, 30;e;d, 30;a;9, 30;f;e, 30;9;8, 30;b;a, 30;c;b
2022-02-17T12:44:17,606 [balancer.SimpleLoadBalancer] WARN : Not balancing due to 8 outstanding migrations.
2022-02-17T12:44:17,606 [balancer.SimpleLoadBalancer] DEBUG: Sample up to 10 outstanding migrations: 30;d;c, 30<;f, 30;e;d, 30;a;9, 30;f;e, 30;9;8, 30;b;a, 30;c;b
2022-02-17T12:44:17,606 [balancer.SimpleLoadBalancer] WARN : Not balancing due to 8 outstanding migrations.
2022-02-17T12:44:17,606 [balancer.SimpleLoadBalancer] DEBUG: Sample up to 10 outstanding migrations: 30;d;c, 30<;f, 30;e;d, 30;a;9, 30;f;e, 30;9;8, 30;b;a, 30;c;b
2022-02-17T12:44:17,606 [balancer.SimpleLoadBalancer] WARN : Not balancing due to 8 outstanding migrations.
2022-02-17T12:44:17,606 [balancer.SimpleLoadBalancer] DEBUG: Sample up to 10 outstanding migrations: 30;d;c, 30<;f, 30;e;d, 30;a;9, 30;f;e, 30;9;8, 30;b;a, 30;c;b
2022-02-17T12:44:17,606 [balancer.SimpleLoadBalancer] WARN : Not balancing due to 8 outstanding migrations.
2022-02-17T12:44:17,606 [balancer.SimpleLoadBalancer] DEBUG: Sample up to 10 outstanding migrations: 30;d;c, 30<;f, 30;e;d, 30;a;9, 30;f;e, 30;9;8, 30;b;a, 30;c;b
2022-02-17T12:44:17,606 [balancer.SimpleLoadBalancer] WARN : Not balancing due to 8 outstanding migrations.
2022-02-17T12:44:17,606 [balancer.SimpleLoadBalancer] DEBUG: Sample up to 10 outstanding migrations: 30;d;c, 30<;f, 30;e;d, 30;a;9, 30;f;e, 30;9;8, 30;b;a, 30;c;b
2022-02-17T12:44:17,606 [balancer.SimpleLoadBalancer] WARN : Not balancing due to 8 outstanding migrations.
2022-02-17T12:44:17,606 [balancer.SimpleLoadBalancer] DEBUG: Sample up to 10 outstanding migrations: 30;d;c, 30<;f, 30;e;d, 30;a;9, 30;f;e, 30;9;8, 30;b;a, 30;c;b
2022-02-17T12:44:17,606 [balancer.SimpleLoadBalancer] WARN : Not balancing due to 8 outstanding migrations.
</pre>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #2506: SimpleLoadBalancer logging too many outstanding migrations
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #2506:
URL: https://github.com/apache/accumulo/issues/2506#issuecomment-1044791181
I saw this TODO in the code. https://issues.apache.org/jira/browse/ACCUMULO-2938
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman edited a comment on issue #2506: SimpleLoadBalancer logging too many outstanding migrations
Posted by GitBox <gi...@apache.org>.
EdColeman edited a comment on issue #2506:
URL: https://github.com/apache/accumulo/issues/2506#issuecomment-1082213376
How does encryption handle keys - are they encrypted? If they are not, then anything that prints a key has the potential to reveal something that is sensitive. Not sure there is a way around it, but logs and commands that can reveal keys need to be treated as sensitive.
An example would be if someone (probabily against best practices) used SSNs as a key - any thing that printed that key would need to treat the output as containing personal identifying information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #2506: SimpleLoadBalancer logging too many outstanding migrations
Posted by GitBox <gi...@apache.org>.
dlmarion commented on issue #2506:
URL: https://github.com/apache/accumulo/issues/2506#issuecomment-1079044488
@milleruntime - I wonder if the issue is that the Problem key is being removed from the WeakHashMap in ThrottledBalancerProblemReporter.problemReportTimes by the garbage collector, so ThrottledBalancerProblemReporter.reportProblem is always going to report the problem. Changing the map type, as a test, should confirm whether or not this is the case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on issue #2506: SimpleLoadBalancer logging too many outstanding migrations
Posted by GitBox <gi...@apache.org>.
EdColeman commented on issue #2506:
URL: https://github.com/apache/accumulo/issues/2506#issuecomment-1082213376
How does encryption handle keys - are they encrypted? If they are not, then anything that prints a key has the potential to reveal something that is sensitive. Not sure there is a way around it, but logs and commands that can reveal keys need to be treated as sensitive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #2506: SimpleLoadBalancer logging too many outstanding migrations
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on issue #2506:
URL: https://github.com/apache/accumulo/issues/2506#issuecomment-1082104058
> I saw this TODO in the code. https://issues.apache.org/jira/browse/ACCUMULO-2938
Is it still applicable? If not, can just delete the TODO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #2506: SimpleLoadBalancer logging too many outstanding migrations
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #2506:
URL: https://github.com/apache/accumulo/issues/2506#issuecomment-1082110946
> > I saw this TODO in the code. https://issues.apache.org/jira/browse/ACCUMULO-2938
>
> Is it still applicable? If not, can just delete the TODO. Or at least, seems like a separate issue.
I think we could delete the TODO and close the JIRA issue. It is a little too generic and we haven't seen any leakage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #2506: SimpleLoadBalancer logging too many outstanding migrations
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #2506:
URL: https://github.com/apache/accumulo/issues/2506#issuecomment-1044469327
@brianloss any interest in taking a look at this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #2506: SimpleLoadBalancer logging too many outstanding migrations
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #2506:
URL: https://github.com/apache/accumulo/issues/2506#issuecomment-1044436944
It appears that the balancer is getting called for each tablet, hence why so many repetitive messages. This was supposed to be fixed with the creation of a new class `ThrottledBalancerProblemReporter` in #1891 but it doesn't look like the reporting is working properly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii edited a comment on issue #2506: SimpleLoadBalancer logging too many outstanding migrations
Posted by GitBox <gi...@apache.org>.
ctubbsii edited a comment on issue #2506:
URL: https://github.com/apache/accumulo/issues/2506#issuecomment-1082104058
> I saw this TODO in the code. https://issues.apache.org/jira/browse/ACCUMULO-2938
Is it still applicable? If not, can just delete the TODO. Or at least, seems like a separate issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org