You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/10/28 09:43:46 UTC
[GitHub] [pulsar] lhotari opened a new issue, #18243: Add observability metrics for CommandPartitionedTopicMetadata requests
lhotari opened a new issue, #18243:
URL: https://github.com/apache/pulsar/issues/18243
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Motivation
Currently, there's no way to track CommandPartitionedTopicMetadata requests. There's no metrics or logs that indicate that a broker is handling CommandPartitionedTopicMetadata requests.
Misconfigured clients might flood brokers with CommandPartitionedTopicMetadata requests and cause high CPU consumption.
One example of this is misconfiguration of splunk-otel-collector's Pulsar exporter. The example config configures pulsar-client-go's PartitionsAutoDiscoveryInterval setting to 1 nanosecond. I have sent a PR to fix the example config with https://github.com/signalfx/splunk-otel-collector/pull/2185 . This example shows that it's easy to mix the units and misconfigure a Pulsar client.
### Solution
Add observability metrics for CommandPartitionedTopicMetadata requests, similar to what there is for lookup requests added by #8272.
### Alternatives
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tjiuming commented on issue #18243: Add observability metrics for CommandPartitionedTopicMetadata requests
Posted by GitBox <gi...@apache.org>.
tjiuming commented on issue #18243:
URL: https://github.com/apache/pulsar/issues/18243#issuecomment-1295003076
currently, we have [metadata store metrics](https://pulsar.apache.org/docs/next/reference-metrics/#metadata-store-metrics), if it could meet your needs, I'd like to handle the issue.
@lhotari
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tjiuming commented on issue #18243: Add observability metrics for CommandPartitionedTopicMetadata requests
Posted by GitBox <gi...@apache.org>.
tjiuming commented on issue #18243:
URL: https://github.com/apache/pulsar/issues/18243#issuecomment-1298376854
@lhotari @codelipenghui PTAL https://github.com/apache/pulsar/pull/18281
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tjiuming commented on issue #18243: Add observability metrics for CommandPartitionedTopicMetadata requests
Posted by GitBox <gi...@apache.org>.
tjiuming commented on issue #18243:
URL: https://github.com/apache/pulsar/issues/18243#issuecomment-1296860181
@codelipenghui @lhotari There are 2 ways to get PartitionedTopicMetadata, one is `ServerCnx#handlePartitionMetadataRequest(CommandPartitionedTopicMetadata partitionMetadata)`, another one is `PersistentTopics#getPartitionedMetadata(Args ...)`
if we need to add metrics for them, please assign the issue to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #18243: Add observability metrics for CommandPartitionedTopicMetadata requests
Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #18243:
URL: https://github.com/apache/pulsar/issues/18243#issuecomment-1295142729
> currently, we have [metadata store metrics](https://pulsar.apache.org/docs/next/reference-metrics/#metadata-store-metrics), if it could meet your needs, I'd like to handle the issue.
> @lhotari
How are metadata store metrics used currently? I think it could be a breaking change if CommandPartitionedTopicMetadata requests are tracked as part of some other metric. I think it should be a new metric that is unique for CommandPartitionedTopicMetadata requests. @codelipenghui do you have a suggestion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tjiuming commented on issue #18243: Add observability metrics for CommandPartitionedTopicMetadata requests
Posted by GitBox <gi...@apache.org>.
tjiuming commented on issue #18243:
URL: https://github.com/apache/pulsar/issues/18243#issuecomment-1296981244
> > How are metadata store metrics used currently? I think it could be a breaking change if CommandPartitionedTopicMetadata requests are tracked as part of some other metric. I think it should be a new metric that is unique for CommandPartitionedTopicMetadata requests. @codelipenghui do you have a suggestion?
>
> The metadata store metrics are on the metadata store level which can provide the metastore operation latency. The REST API request metrics should be a separate part. The CommandPartitionedTopicMetadata requests metrics should not 100% equal to the metadata store operation. Maybe the jetty thread is blocked somewhere.
>
> I think maybe jetty already provides the ability to expose the metrics with the request path label?
I've checked jetty, seems there are no such ability.
if we want the ability, it's not easy. because we need to converge the request path. such as: `/api/v2/persistent/myTenant/myNamespace/partitioned` -> `/api/v2/persistent/{tenant}/{namespace}/partitioned`. it may takes some time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #18243: Add observability metrics for CommandPartitionedTopicMetadata requests
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #18243:
URL: https://github.com/apache/pulsar/issues/18243#issuecomment-1336625684
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] WJL3333 commented on issue #18243: Add observability metrics for CommandPartitionedTopicMetadata requests
Posted by GitBox <gi...@apache.org>.
WJL3333 commented on issue #18243:
URL: https://github.com/apache/pulsar/issues/18243#issuecomment-1296488936
i'd like to help if we need add some metric like `CommandLookup` request number and process time for `CommandPartitionedTopicMetadata`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #18243: Add observability metrics for CommandPartitionedTopicMetadata requests
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #18243:
URL: https://github.com/apache/pulsar/issues/18243#issuecomment-1296473080
> How are metadata store metrics used currently? I think it could be a breaking change if CommandPartitionedTopicMetadata requests are tracked as part of some other metric. I think it should be a new metric that is unique for CommandPartitionedTopicMetadata requests. @codelipenghui do you have a suggestion?
The metadata store metrics are on the metadata store level which can provide the metastore operation latency. The REST API request metrics should be a separate part. The CommandPartitionedTopicMetadata requests metrics should not 100% equal to the metadata store operation. Maybe the jetty thread is blocked somewhere.
I think maybe jetty already provides the ability to expose the metrics with the request path label?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tjiuming commented on issue #18243: Add observability metrics for CommandPartitionedTopicMetadata requests
Posted by GitBox <gi...@apache.org>.
tjiuming commented on issue #18243:
URL: https://github.com/apache/pulsar/issues/18243#issuecomment-1301901235
The PIP discuss thread: https://lists.apache.org/thread/sybl4nno4503w42hzt7b5lsyk6m2rbo6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org