You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/02/06 22:08:35 UTC
[GitHub] [hadoop-ozone] bharatviswa504 opened a new pull request #534:
HDDS-2987. Add metrics to OM DoubleBuffer.
bharatviswa504 opened a new pull request #534: HDDS-2987. Add metrics to OM DoubleBuffer.
URL: https://github.com/apache/hadoop-ozone/pull/534
## What changes were proposed in this pull request?
Add a metric that will help in understanding the Average flush time which will help in understanding how the flush time increases over time.
Add another metric to show the average number of flush transactions in an iteration. This will show how many transactions are flushed in a single iteration over time.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-2987
## How was this patch tested?
Added a UT and also deployed cluster to look into these new metrics.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request
#534: HDDS-2987. Add metrics to OM DoubleBuffer.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #534: HDDS-2987. Add metrics to OM DoubleBuffer.
URL: https://github.com/apache/hadoop-ozone/pull/534#discussion_r377186717
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##########
@@ -145,7 +146,10 @@ private void flushTransactions() {
}
});
+ long startTime = Time.monotonicNowNanos();
omMetadataManager.getStore().commitBatchOperation(batchOperation);
+ ozoneManagerDoubleBufferMetrics.updateFlushTime(
Review comment:
In case of exception, we miss calling updateFlushTime is to handle this scenario. I think that is fine right because anyway we got an exception, so missing updateFlushTime is fine I believe.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek closed pull request #534: HDDS-2987. Add
metrics to OM DoubleBuffer.
Posted by GitBox <gi...@apache.org>.
elek closed pull request #534: HDDS-2987. Add metrics to OM DoubleBuffer.
URL: https://github.com/apache/hadoop-ozone/pull/534
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request
#534: HDDS-2987. Add metrics to OM DoubleBuffer.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #534: HDDS-2987. Add metrics to OM DoubleBuffer.
URL: https://github.com/apache/hadoop-ozone/pull/534#discussion_r376150247
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/metrics/OzoneManagerDoubleBufferMetrics.java
##########
@@ -44,6 +47,15 @@
"number of transactions flushed in a single flush iteration till now.")
private MutableCounterLong maxNumberOfTransactionsFlushedInOneIteration;
+ @Metric(about = "DoubleBuffer flushTime. This metrics particularly captures" +
+ " rocksdb batch commit time.")
+ private MutableRate flushTime;
+
+ @Metric(about = "Average number of transactions flushed in a single " +
+ "iteration")
+ private MutableGaugeLong avgFlushTransactionsInOneIteration;
Review comment:
Thank You @avijayanhwx for the review. Address comments in the latest commit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on a change in pull request #534:
HDDS-2987. Add metrics to OM DoubleBuffer.
Posted by GitBox <gi...@apache.org>.
elek commented on a change in pull request #534: HDDS-2987. Add metrics to OM DoubleBuffer.
URL: https://github.com/apache/hadoop-ozone/pull/534#discussion_r377671317
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##########
@@ -145,7 +146,10 @@ private void flushTransactions() {
}
});
+ long startTime = Time.monotonicNowNanos();
omMetadataManager.getStore().commitBatchOperation(batchOperation);
+ ozoneManagerDoubleBufferMetrics.updateFlushTime(
Review comment:
> think that is fine right because anyway we got an exception, so missing updateFlushTime is fine I believe.
Fair enough. Thanks to help to understand it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request
#534: HDDS-2987. Add metrics to OM DoubleBuffer.
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on a change in pull request #534: HDDS-2987. Add metrics to OM DoubleBuffer.
URL: https://github.com/apache/hadoop-ozone/pull/534#discussion_r376114951
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/metrics/OzoneManagerDoubleBufferMetrics.java
##########
@@ -44,6 +47,15 @@
"number of transactions flushed in a single flush iteration till now.")
private MutableCounterLong maxNumberOfTransactionsFlushedInOneIteration;
+ @Metric(about = "DoubleBuffer flushTime. This metrics particularly captures" +
+ " rocksdb batch commit time.")
+ private MutableRate flushTime;
+
+ @Metric(about = "Average number of transactions flushed in a single " +
+ "iteration")
+ private MutableGaugeLong avgFlushTransactionsInOneIteration;
Review comment:
Can this be a MutableGaugeFloat instead of a MutableGaugeLong? We are losing some precision here because of the division and cast to Long.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on a change in pull request #534:
HDDS-2987. Add metrics to OM DoubleBuffer.
Posted by GitBox <gi...@apache.org>.
elek commented on a change in pull request #534: HDDS-2987. Add metrics to OM DoubleBuffer.
URL: https://github.com/apache/hadoop-ozone/pull/534#discussion_r377090569
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##########
@@ -145,7 +146,10 @@ private void flushTransactions() {
}
});
+ long startTime = Time.monotonicNowNanos();
omMetadataManager.getStore().commitBatchOperation(batchOperation);
+ ozoneManagerDoubleBufferMetrics.updateFlushTime(
Review comment:
Do we need a finally block here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on issue #534: HDDS-2987. Add metrics
to OM DoubleBuffer.
Posted by GitBox <gi...@apache.org>.
elek commented on issue #534: HDDS-2987. Add metrics to OM DoubleBuffer.
URL: https://github.com/apache/hadoop-ozone/pull/534#issuecomment-584673375
> let me know your thoughts
I misunderstood the average calculation previously therefore I used the wrong examples. This is a better one:
Let's say you have 100 day uptime and at each of day you had 2_000_000 flush transactions with 1_000_000 flush operations. The avgFlushTransactionsInOneIteration is 2_000_000 * 100 / 1_000_000 * 100 = 2.0
The 101st day is a very bad day as you have 2_000_000 transaction per day but with 1_500_000 operations. This is 50% bigger than before, but from the average it's very hard to notice: 2_000_000 * 100 + 2_000_000) / (1_000_000 * 100 + 1_500_000) = 1.99...
I think a generic average (which is calculated based on **all the events from the start-up**) is not so effective. It is very expressive at the beginning but later it couldn't show real problems.
With real moving averages (which can be easily calculated by prometheus or any other monitoring system) you can see the average of the **last day** or any period.
But this is not a blocking problem. I don't think that the average of this patch is a very useful metric for production but it might be useful for your performance tests (where you are interested only about the few hours after the startup). (But you can always use prometheus to calculate moving averages...)
Long story short: I will merge it now... Thanks the patch...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on issue #534: HDDS-2987.
Add metrics to OM DoubleBuffer.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on issue #534: HDDS-2987. Add metrics to OM DoubleBuffer.
URL: https://github.com/apache/hadoop-ozone/pull/534#issuecomment-584221061
> Thanks the patch @bharatviswa504
>
> Overall it looks good. I have a small question inline.
>
> And one note: I am not sure if calculating the average it the most effective way. External metrics tool (like prometheus) can do it in a much smarter way (instead of calculating an generic average, they can return with moving average).
>
> After a few days a global average couldn't say too much about the current state. (But I have no problem to commit it, if you really need it...)
Thank You @elek for the review. So here the suggestion is to use MutableRate, which will calculcate Average automatically? But anyway the current implementation also does the same right, totalNumberOfFlushedTransactions/totalFlushOps which gives the same value over time. let me know your thoughts?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org