[GitHub] [pulsar] rdhabalia opened a new issue, #20265: PIP-267: Add support of topic stats/stats-internal client api

["rdhabalia (via GitHub)" <gi...@apache.org>]

rdhabalia opened a new issue, #20265:
URL: https://github.com/apache/pulsar/issues/20265

   
   # Motivation
   
   Right now, Pulsar provides the topic's stats and stats-internal over HTTP admin API, and this stats data is used by user applications and also by Pulsar internal components such as Pulsar-functions to derive the certain states of the applications.
   for example, there are use cases where the application wants to check the topic's backlog, subscription's state (readPosition, list of subscriptions), numberOfEntriesSinceFirstNotAckedMessage, etc to bootstrap the application or handle the application’s resiliency and state dynamically. Applications can retrieve this stats information by using the broker’s admin HTTP APIs. 
   
   However, stats retrieval over HTTP API doesn’t work well in use cases when users would like to access this API at a higher scale when a large number of application nodes would like to use it over HTTP which could overload brokers and sometimes makes broker irresponsive and impact admin API performance. It’s also become difficult when Pulsar is deployed in the cloud behind the SNI proxy and applications also want to access large-scale stats information periodically over different HTTP port instead it would be better if applications can fetch stats over on same binary protocol for scalability and accessibility reasons. 
   
   Therefore, there are multiple use cases where producer/consumer applications need stats information for topics using the client library over binary protocol. Hence, this PIP introduces client API for producers and consumers to access topic stats/internal-stats information which can be used by applications as needed.
   
   
   # Client library changes
   
   The client library will allow both producer and consumer to retrieve stats using `TopicStatsProvider` that fetches stats/internal-stats for both persistent and non-persistent topics. Producer/consumer of a partition can use the same connection with the broker to access topic and retrieve topics stats. Pulsar also allows application to access multiple different partitions/topics using single  producer/consumer so, that producer/consumer should be able to provide generic API to access stats of single or multi topics/partitions served by them.
   
   Therefore, producer/consumer will have additional API that returns `TopicStatsProvider` which allows the application to fetch stats/internal-stats of all topics/partitions served by that producer/consumer.
   
   ```
   Producer.java / Consumer.java
   /**
   * Returns {@link TopicStatsProvider} to retrieve stats/internal-stats of the topic.
   */
   TopicStatsProvider getTopicStatsProvider();
   ```
   ```
   TopicStatsProvider.java
   /**
   * TopicStatsProvider provides API to access topic’s stats and internal-stat information.
   */
   public interface TopicStatsProvider {
   
       /**
        * @return the topic stats
        */
       CompletableFuture<TopicStatsInfo>  getStats();
   
       /**
        * @return the internal topic stats
        */
       CompletableFuture<TopicInternalStatsInfo>  getInternalStats();
   }
   ```
   
   ```
   TopicStatsInfo.java
   
   /**
   * TopicStatsInfo contains the stats information of all partitions of a given topic. It allows the client to access stats of each individual partition from the partition stats map.
   *
   public class TopicStatsInfo {
       private Map<String, TopicStats> partitions;
   }
   
   TopicInternalStatsInfo.java
   
   /**
   * TopicInternalStatsInfo contains internal-stats information of topic. It allows the client to access stats of partitioned and non-partitioned topics.
   */
   public class TopicInternalStatsInfo {
       private Map<String, PersistentTopicInternalStats> partitions;
   }
   ```
   
   We would like to create create a generic stats protocol between client and broker so, broker can send any type of stats (eg: stats or internal-stats) by serializing into specific format and client can deserialize it into appropriate format  and return back to the client application.
   Therefore, this PIP will introduce new wire-protocol for `TopicStats` where client sends stats-type and topic-partition to the broker and broker sends back serialized stats response which can be deserialized by client based on stats type format.
   
   ```
   message CommandTopicStats {
       enum StatsType {
           STATS = 0;
           STATS_INTERNAL = 1;
       }
       required uint64 request_id    = 1;
       required string topic_name    = 2;
       required StatsType stats_type = 3;
   }
   
   message CommandTopicStatsResponse {
       required uint64 request_id    = 1;
       optional ServerError error_code    = 2;
       optional string error_message = 3;
       optional string stats_json    = 4;
   }
   ```
   
   # Broker changes
   
   A broker will have support to handle the stats command for a partition and return stats/internal-stats response for a partition. Broker will authorize connected client if client is authorized to access topic stats and returns the stats once client is successfully authorized.
   
   This PIP will also restrict clients and brokers to handle a number of concurrent requests to protect brokers against a large number of such stats requests.
   
   [Sample Prototype with new API support](https://github.com/apache/pulsar/compare/master...rdhabalia:stats_proto?expand=1)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] github-actions[bot] commented on issue #20265: PIP-268: Add support of topic stats/stats-internal using client api

["github-actions[bot] (via GitHub)" <gi...@apache.org>]

github-actions[bot] commented on issue #20265:
URL: https://github.com/apache/pulsar/issues/20265#issuecomment-1583831868

   The issue had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org