[PR] Limit concurrent goroutines in cluster node probing [kvrocks-controller]

["Jitmisra (via GitHub)" <gi...@apache.org>]

Jitmisra opened a new pull request, #280:
URL: https://github.com/apache/kvrocks-controller/pull/280

   In the current implementation of parallelProbeNodes(), a new goroutine is spawned for every node in the cluster without any limit on concurrent operations. This can lead to resource exhaustion in large deployments with many nodes.
   
   `func (c *ClusterChecker) parallelProbeNodes(ctx context.Context, cluster *store.Cluster) {
       for i, shard := range cluster.Shards {
           for _, node := range shard.Nodes {
               go func(shardIdx int, n store.Node) {
                   // Node probing code...
               }(i, node)
           }
       }
       // No waiting mechanism or concurrency control
   }`
   
   Problems
   Resource Exhaustion: In large clusters (hundreds or thousands of nodes), this creates too many goroutines at once
   Network Floods: All probes happen simultaneously, potentially overwhelming network connections
   No Completion Guarantee: The function returns immediately without ensuring all probes complete
   System Resource Strain: Can exhaust system resources like file descriptors for network connections
   Solution
   This PR implements a semaphore pattern to limit the number of concurrent operations while still maintaining parallelism:
   
   This change makes the system more resilient and scalable for production deployments of all sizes.
   This PR implements a semaphore pattern to limit the number of concurrent operations while still maintaining parallelism:
   
   `func (c *ClusterChecker) parallelProbeNodes(ctx context.Context, cluster *store.Cluster) {
       // Limit concurrent operations to a reasonable number
       semaphore := make(chan struct{}, 20) // Configurable limit
       var wg sync.WaitGroup
   
       for i, shard := range cluster.Shards {
           for _, node := range shard.Nodes {
               wg.Add(1)
               go func(shardIdx int, n store.Node) {
                   defer wg.Done()
                   
                   // Acquire semaphore before proceeding
                   semaphore <- struct{}{}
                   defer func() { <-semaphore }()
                   
                   // Existing node probing code...
               }(i, node)
           }
       }
   
       // Wait for all probes to complete
       wg.Wait()
   }`
   
   This change makes the system more resilient and scalable for production deployments of all sizes.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Limit concurrent goroutines in cluster node probing [kvrocks-controller]

["Jitmisra (via GitHub)" <gi...@apache.org>]

Jitmisra closed pull request #280: Limit concurrent goroutines in cluster node probing
URL: https://github.com/apache/kvrocks-controller/pull/280


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org