You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2022/03/18 11:17:36 UTC

[GitHub] [accumulo] milleruntime opened a new issue #2577: Warn user of slow metadata scans

milleruntime opened a new issue #2577:
URL: https://github.com/apache/accumulo/issues/2577


   **Is your feature request related to a problem? Please describe.**
   If the metadata table takes a long time to scan, the overall performance of Accumulo will suffer dearly.
   
   **Describe the solution you'd like**
   Warn the user if scans of the metadata table are taking "too long". With the creation of the Ample (Accumulo metadata persistence layer) in 2.1, this should be easier then in past versions.
   
   **Describe alternatives you've considered**
   Take more drastic steps to help a cluster, like shutting things down, automatically merge tablets, etc. We could also warn if writes to the metadata are taking too long.
   
   **Additional context**
   I find that users can be blind to how scans of the metadata are performing. Does a cluster have too many tables? Does a cluster have too many tablets hosted per tserver? These are typical questions when the performance of a cluster is suffering. Warning the user of slow metadata scans will hopefully help to alleviate this behavior.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [accumulo] EdColeman commented on issue #2577: Warn user of slow metadata scans

Posted by GitBox <gi...@apache.org>.
EdColeman commented on issue #2577:
URL: https://github.com/apache/accumulo/issues/2577#issuecomment-1072524017


   Another option may be to add a scheduled thread in manager that does the scan - it is tangentially related to the monitor critical threads issue (https://github.com/apache/accumulo/issues/946) - mainly because being able to scan the metadata table is sort of critical.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [accumulo] milleruntime closed issue #2577: Warn user of slow metadata scans

Posted by GitBox <gi...@apache.org>.
milleruntime closed issue #2577:
URL: https://github.com/apache/accumulo/issues/2577


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [accumulo] EdColeman commented on issue #2577: Warn user of slow metadata scans

Posted by GitBox <gi...@apache.org>.
EdColeman commented on issue #2577:
URL: https://github.com/apache/accumulo/issues/2577#issuecomment-1072848670


   Micrometer has a FunctionTimer that provides a count and the total latency - that allows the back end metrics system to reason about the rate and latency.  By monitoring the count you can determine liveliness and also track the latency.
   
   Equivalent to using a counter incremented each metadata scan and a counter that sums the time for each scan.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [accumulo] ctubbsii commented on issue #2577: Warn user of slow metadata scans

Posted by GitBox <gi...@apache.org>.
ctubbsii commented on issue #2577:
URL: https://github.com/apache/accumulo/issues/2577#issuecomment-1072469256


   I like the idea of exposing as metrics. As a warning log, I'm not so sure. A lot of this depends on hardware, or varies significantly with user design/load. Trying to determine heuristically when to warn and provide hints to users about the possible causes seems like a lot of complexity that is very subjective, somewhat arbitrary, and likely flaky. I would definitely prefer the metrics route, though. Then users will be empowered with the tools they need to notice the impact on performance that any change they make in their deployments might cause.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [accumulo] EdColeman commented on issue #2577: Warn user of slow metadata scans

Posted by GitBox <gi...@apache.org>.
EdColeman commented on issue #2577:
URL: https://github.com/apache/accumulo/issues/2577#issuecomment-1072318444


   This would be useful if exposed as a metric as well as any logging.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [accumulo] ctubbsii commented on issue #2577: Warn user of slow metadata scans

Posted by GitBox <gi...@apache.org>.
ctubbsii commented on issue #2577:
URL: https://github.com/apache/accumulo/issues/2577#issuecomment-1072839802


   If you just want timing, wrapping with trace would probably be better than metrics. I don't know where the dividing line is, but I imagine metrics to be something like ops/sec, or counts of ops, or averages, rather than simple timing of calls.
   
   So, the above sample code might be suitable for metrics, as a starting point for pushing to a metric that averages. (but don't use `System.currentTimeMillis()`, because that's clock time, and not suitable for durations).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [accumulo] milleruntime commented on issue #2577: Warn user of slow metadata scans

Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #2577:
URL: https://github.com/apache/accumulo/issues/2577#issuecomment-1072518596


   I don't know how to expose metrics in Ample. Any ideas? I think the best place would be to do in `TabletsMetadata`. Something like:
   ```
   private TabletsMetadata buildNonRoot(AccumuloClient client) {
   Instant scanStart = Instant.ofEpochMilli(System.currentTimeMillis());
   try {
   ...
    } finally {
     Instant scanEnd = Instant.ofEpochMilli(System.currentTimeMillis());
    }
   }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [accumulo] EdColeman commented on issue #2577: Warn user of slow metadata scans

Posted by GitBox <gi...@apache.org>.
EdColeman commented on issue #2577:
URL: https://github.com/apache/accumulo/issues/2577#issuecomment-1072318444


   This would be useful if exposed as a metric as well as any logging.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org