You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "devmadhuu (via GitHub)" <gi...@apache.org> on 2024/08/19 10:52:23 UTC
[PR] HDDS-11339. Publishing hadoop metrics immediately in Prometheus sink fills up SinkQueue quickly. [ozone]
devmadhuu opened a new pull request, #7092:
URL: https://github.com/apache/ozone/pull/7092
## What changes were proposed in this pull request?
HDDS-11339. Publishing hadoop metrics immediately in Prometheus sink fills up SinkQueue quickly.
Two issues:
`PrometheusServlet` is being registered with `BaseHttpServer` when prometheus support is enabled and `PrometheusServlet` is being called every 15 secs by default as scraping interval and it publishes the hadoop metrics immediately. So if there are large number of metrics needs to be published in a very busy cluster, this makes SinkQueue gets filled up quickly and then sink cannot consume the given metrics and just dropped them outright.
A part from dropping , another issue is taking the object lock of `MetricsSystemImpl` class and before metrics actually being published, other threads keeps waiting to take the object lock. There was a recent issue came to highlight where in a busy cluster, there were ~ 190 threads BLOCKED just to acquire the lock of the `MetricsSystemImpl` class. This makes Recon role unresponsive and after sometime JVM couldn't allocate sufficient memory and crashes with OOM. This prometheus and OOM issue is not related to Recon directly as this can happen with any role which is going to use Prometheus service in a busy cluster.
`Solution:` We need not to publish the metrics immediately by calling
`DefaultMetricsSystem.instance().publishMetricsNow();`
because a prometheus sink already have a mechanism to publish metrics every 10 secs by default using call back with timer event. So we need to remove the above code to publish immediately.
Below is the onTimerEvent callback:
`org.apache.hadoop.metrics2.impl.MetricsSystemImpl#onTimerEvent`
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-11339
## How was this patch tested?
This patch is tested manually by running a docker cluster using Prometheus support enabled and validates the metrics being published to prometheus instance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11339. Publishing hadoop metrics immediately in Prometheus sink fills up SinkQueue quickly. [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on PR #7092:
URL: https://github.com/apache/ozone/pull/7092#issuecomment-2296380668
@sumitagrawl kindly review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11339. Publishing hadoop metrics immediately in Prometheus sink fills up SinkQueue quickly. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #7092:
URL: https://github.com/apache/ozone/pull/7092#issuecomment-2336455080
> confirm the apache project under which we should create JIRA
I think it should be a HADOOP issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11339. Publishing hadoop metrics immediately in Prometheus sink fills up SinkQueue quickly. [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on PR #7092:
URL: https://github.com/apache/ozone/pull/7092#issuecomment-2301302352
> Can you please elaborate on why the alternative will perform better.
>
> ```
> because a prometheus sink already have a mechanism to publish metrics every 10 secs by default using call back with timer event. So we need to remove the above code to publish immediately.
> ```
Thanks @kerneltime for review. So when we start the MetricsSystem, in `org.apache.hadoop.metrics2.impl.MetricsSystemImpl#start`, we also start a timer and while starting a timer, we also schedule a timerEvent which gets called every 10 secs default period.
<img width="914" alt="image" src="https://github.com/user-attachments/assets/89db3ada-55fa-4492-b5fb-cb2e7e8e4bfd">
and as we can see in above screenshot of code, we are scheduling to call `onTimerEvent` every 10 secs which publishes the metrics already with `immediate` arg as `false`.
So we need not to publish the metrics immediately as we are doing in `PrometheusServlet`. This puts too much load on queue.
<img width="601" alt="image" src="https://github.com/user-attachments/assets/869d2897-6e1f-4b54-b0be-61cd0b983f78">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11339. Publishing hadoop metrics immediately in Prometheus sink fills up SinkQueue quickly. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #7092:
URL: https://github.com/apache/ozone/pull/7092
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11339. Publishing hadoop metrics immediately in Prometheus sink fills up SinkQueue quickly. [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on PR #7092:
URL: https://github.com/apache/ozone/pull/7092#issuecomment-2324089582
> Thanks @devmadhuu for the patch. Verified that Prometheus gets updated metrics eventually. You might want to propose similar improvement in Hadoop:
>
> https://github.com/apache/hadoop/blob/e4ee3d560bddc27a495cc9a158278a9c18276dd0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/PrometheusServlet.java#L43
>
> Sidenote:
>
> Please share text as text, not screenshots. Plain text makes it easier to
>
> * read (especially for those with visual impairments)
> * reference (e.g. quote)
> * search
>
> It also reduces size to be stored by GitHub.
>
> Sharing code can be even better by using [permalink](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-a-permanent-link-to-a-code-snippet) URLs.
Thanks @adoroszlai for your review. For similar improvement in hadoop, can you kindly confirm the apache project under which we should create JIRA ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11339. Publishing hadoop metrics immediately in Prometheus sink fills up SinkQueue quickly. [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #7092:
URL: https://github.com/apache/ozone/pull/7092#issuecomment-2299785618
Can you please elaborate on
```
because a prometheus sink already have a mechanism to publish metrics every 10 secs by default using call back with timer event. So we need to remove the above code to publish immediately.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org