You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/12/05 12:22:42 UTC
[GitHub] [pulsar] Nicklee007 opened a new pull request, #18747: [fix][broker] Avoid OOM not trigger PulsarByteBufAllocator outOfMemoryListener when use ByteBufAllocator.DEFAULT.heapBuffer in PrometheusMetricsGeneratorUtils
Nicklee007 opened a new pull request, #18747:
URL: https://github.com/apache/pulsar/pull/18747
### Motivation
To avoid OOM exception not trigger `PulsarByteBufAllocator outOfMemoryListener`, we should use `PulsarByteBufAllocator.DEFAULT` to replace `ByteBufAllocator.DEFAULT` to allocate memory;
In our case, broker.conf set as `skipBrokerShutdownOnOOM=false`
and the prometheus-stats-39-1 throw OOM exception, but not trigger broker shutdown.
```
[prometheus-stats-39-1] ERROR org.apache.bookkeeper.common.util.SafeRunnable - Unexpected throwable caught
java.lang.Error: java.lang.reflect.InvocationTargetException
Caused by: java.lang.OutOfMemoryError: Java heap space
at jdk.internal.misc.Unsafe.allocateUninitializedArray(Unsafe.java:1269) ~[?:?]
at jdk.internal.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at io.netty.util.internal.PlatformDependent0.allocateUninitializedArray(PlatformDependent0.java:575)
at io.netty.util.internal.PlatformDependent.allocateUninitializedArray(PlatformDependent.java:287)
at io.netty.buffer.PoolArena$HeapArena.newByteArray(PoolArena.java:567)
at io.netty.buffer.PoolArena$HeapArena.newUnpooledChunk(PoolArena.java:583)
at io.netty.buffer.PoolArena.allocateHuge(PoolArena.java:214)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:141)
at io.netty.buffer.PoolArena.reallocate(PoolArena.java:287)
at io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:122)
at io.netty.buffer.AbstractByteBuf.ensureWritable0(AbstractByteBuf.java:305)
at io.netty.buffer.AbstractByteBuf.writeByte(AbstractByteBuf.java:984)
at org.apache.pulsar.common.util.SimpleTextOutputStream.write(SimpleTextOutputStream.java:57)
at org.apache.pulsar.broker.stats.prometheus.TopicStats.appendRequiredLabels(TopicStats.java:453)
at org.apache.pulsar.broker.stats.prometheus.TopicStats.metric(TopicStats.java:385)
at org.apache.pulsar.broker.stats.prometheus.TopicStats.printTopicStats(TopicStats.java:128)
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$generate$0(NamespaceStatsAggregator.java:84)
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator$$Lambda$1115/0x00000008009d5840.accept(Unknown Source) ~[?:?]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:544)
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:272)
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$generate$1(NamespaceStatsAggregator.java:75)
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator$$Lambda$1114/0x00000008009d4840.accept(Unknown Source) ~[?:?]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:544)
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:272)
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$generate$2(NamespaceStatsAggregator.java:74)
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator$$Lambda$1113/0x00000008009d3440.accept(Unknown Source) ~[?:?]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:544)
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:272)
at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.generate(NamespaceStatsAggregator.java:70)
at org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsGenerator.generate(PrometheusMetricsGenerator.java:110)
```
### Modifications
1. Use `PulsarByteBufAllocator.DEFAULT` replace `ByteBufAllocator.DEFAULT` in `PrometheusMetricsGenerator/PrometheusMetricsGeneratorUtils/SchemaUtils`.
### Documentation
- [X] `doc-not-needed` <!-- Your PR changes do not impact docs -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- closed pull request #18747: [fix][broker] Avoid OOM not trigger PulsarByteBufAllocator outOfMemoryListener when use ByteBufAllocator.DEFAULT.heapBuffer in PrometheusMetricsGeneratorUtils
Posted by GitBox <gi...@apache.org>.
Technoboy- closed pull request #18747: [fix][broker] Avoid OOM not trigger PulsarByteBufAllocator outOfMemoryListener when use ByteBufAllocator.DEFAULT.heapBuffer in PrometheusMetricsGeneratorUtils
URL: https://github.com/apache/pulsar/pull/18747
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Nicklee007 commented on pull request #18747: [fix][broker] Avoid OOM not trigger PulsarByteBufAllocator outOfMemoryListener when use ByteBufAllocator.DEFAULT.heapBuffer in PrometheusMetricsGeneratorUtils
Posted by GitBox <gi...@apache.org>.
Nicklee007 commented on PR #18747:
URL: https://github.com/apache/pulsar/pull/18747#issuecomment-1339394625
> @Nicklee007 hello, we are testing new version of pulsar. I want to know if this will happened in our environment. Can you tell me how many topic and subscription on this broker will cause OOM ？
There are some memory lack position caused the OOM exception which has fixed in master branch. Maybe you will not meet the error if you use the new version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- merged pull request #18747: [fix][broker] Avoid OOM not trigger PulsarByteBufAllocator outOfMemoryListener when use ByteBufAllocator.DEFAULT.heapBuffer in PrometheusMetricsGeneratorUtils
Posted by GitBox <gi...@apache.org>.
Technoboy- merged PR #18747:
URL: https://github.com/apache/pulsar/pull/18747
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lifepuzzlefun commented on pull request #18747: [fix][broker] Avoid OOM not trigger PulsarByteBufAllocator outOfMemoryListener when use ByteBufAllocator.DEFAULT.heapBuffer in PrometheusMetricsGeneratorUtils
Posted by GitBox <gi...@apache.org>.
lifepuzzlefun commented on PR #18747:
URL: https://github.com/apache/pulsar/pull/18747#issuecomment-1337329564
@Nicklee007 hello, we are testing new version of pulsar. I want to know if this will happened in our environment. Can you tell me how many topic and subscription on this broker will cause OOM ？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codecov-commenter commented on pull request #18747: [fix][broker] Avoid OOM not trigger PulsarByteBufAllocator outOfMemoryListener when use ByteBufAllocator.DEFAULT.heapBuffer in PrometheusMetricsGeneratorUtils
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #18747:
URL: https://github.com/apache/pulsar/pull/18747#issuecomment-1339272452
# [Codecov](https://codecov.io/gh/apache/pulsar/pull/18747?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#18747](https://codecov.io/gh/apache/pulsar/pull/18747?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (22992e9) into [master](https://codecov.io/gh/apache/pulsar/commit/99e26f5e943e3978bba135a2c17ff08dc9b9599a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (99e26f5) will **decrease** coverage by `4.48%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/pulsar/pull/18747?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #18747 +/- ##
============================================
- Coverage 44.79% 40.30% -4.49%
+ Complexity 10026 7504 -2522
============================================
Files 703 623 -80
Lines 68808 59088 -9720
Branches 7375 6147 -1228
============================================
- Hits 30822 23816 -7006
+ Misses 34378 32288 -2090
+ Partials 3608 2984 -624
```
| Flag | Coverage Δ | |
|---|---|---|
| unittests | `40.30% <0.00%> (-4.49%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pulsar/pull/18747?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...r/stats/prometheus/PrometheusMetricsGenerator.java](https://codecov.io/gh/apache/pulsar/pull/18747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zdGF0cy9wcm9tZXRoZXVzL1Byb21ldGhldXNNZXRyaWNzR2VuZXJhdG9yLmphdmE=) | `62.31% <0.00%> (+62.31%)` | :arrow_up: |
| [...ava/org/apache/pulsar/broker/admin/v1/Brokers.java](https://codecov.io/gh/apache/pulsar/pull/18747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi92MS9Ccm9rZXJzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...va/org/apache/pulsar/broker/admin/v1/Clusters.java](https://codecov.io/gh/apache/pulsar/pull/18747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi92MS9DbHVzdGVycy5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../org/apache/pulsar/broker/admin/v1/Properties.java](https://codecov.io/gh/apache/pulsar/pull/18747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi92MS9Qcm9wZXJ0aWVzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ar/common/naming/PartitionedManagedLedgerInfo.java](https://codecov.io/gh/apache/pulsar/pull/18747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NvbW1vbi9uYW1pbmcvUGFydGl0aW9uZWRNYW5hZ2VkTGVkZ2VySW5mby5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...e/pulsar/broker/admin/impl/ResourceQuotasBase.java](https://codecov.io/gh/apache/pulsar/pull/18747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi9pbXBsL1Jlc291cmNlUXVvdGFzQmFzZS5qYXZh) | `0.00% <0.00%> (-96.43%)` | :arrow_down: |
| [...he/pulsar/broker/service/AnalyzeBacklogResult.java](https://codecov.io/gh/apache/pulsar/pull/18747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zZXJ2aWNlL0FuYWx5emVCYWNrbG9nUmVzdWx0LmphdmE=) | `0.00% <0.00%> (-92.31%)` | :arrow_down: |
| [.../apache/pulsar/broker/admin/v1/ResourceQuotas.java](https://codecov.io/gh/apache/pulsar/pull/18747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi92MS9SZXNvdXJjZVF1b3Rhcy5qYXZh) | `0.00% <0.00%> (-91.43%)` | :arrow_down: |
| [...e/pulsar/broker/stats/AllocatorStatsGenerator.java](https://codecov.io/gh/apache/pulsar/pull/18747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zdGF0cy9BbGxvY2F0b3JTdGF0c0dlbmVyYXRvci5qYXZh) | `0.00% <0.00%> (-91.38%)` | :arrow_down: |
| [...pache/pulsar/broker/stats/MBeanStatsGenerator.java](https://codecov.io/gh/apache/pulsar/pull/18747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zdGF0cy9NQmVhblN0YXRzR2VuZXJhdG9yLmphdmE=) | `0.00% <0.00%> (-84.38%)` | :arrow_down: |
| ... and [272 more](https://codecov.io/gh/apache/pulsar/pull/18747/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org