[jira] [Comment Edited] (TINKERPOP-2288) Get ConnectionPoolBusyException and then ServerUnavailableExceptions

["Some One (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TINKERPOP-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16957660#comment-16957660 ] 

Some One edited comment on TINKERPOP-2288 at 10/23/19 8:42 AM:
---------------------------------------------------------------

For everyone who is waiting for a fix, the following work around with polly does the trick for us, although it is not ideal. We already handled the '429 Too Many Requests' exceptions before, but now also added the handling of ConnectionPoolBusyExceptions and ServerUnavailableExceptions where we immediately create a new client (by calling the SetupGremlinClient method below).

 

 

ResultSet<dynamic> response = await Polly.Policy
 .Handle<ResponseException>(exception => exception.CosmosDbStatusCode() == RetryAfterStatusCode)
 .Or<ConnectionPoolBusyException>()
 .Or<ServerUnavailableException>()
 .WaitAndRetryAsync(
 retryCount: 10,
 sleepDurationProvider: (attempt, exception, _) =>
 {
 if (exception is ResponseException responseException)
 return responseException.CosmosDbRetryAfter();

// In case this is a connection issue, we try again immediately
 return TimeSpan.Zero;
 },
 onRetryAsync: (exception, waitTime, attempt, _) =>
 {
 // In case this is a connection issue we reset the client
 if (exception is ConnectionPoolBusyException || exception is ServerUnavailableException)
 {
 _log.LogWarning($"Resetting the GremlinClient after experiencing an '\{exception.GetType()}'");
 SetupGremlinClient();
 }
 return Task.CompletedTask;
 })
 .ExecuteAsync(() => _gremlinClient.SubmitAsync<dynamic>(query));

 

private void SetupGremlinClient()
 {
 var stopWatch = new Stopwatch();
 stopWatch.Start();

var settings = new ConnectionPoolSettings\{MaxInProcessPerConnection = 64, PoolSize = 8};// Default MacInProcessPerConnection = 32, PoolSize = 4

_gremlinClient = new GremlinClient(_gremlinServer, new GraphSON2Reader(), new GraphSON2Writer(),
 GremlinClient.GraphSON2MimeType,settings
 ); 
 stopWatch.Stop();

_log.LogInformation($"Setting up GremlinClient with PoolSize of \{settings.PoolSize} and MaxInProcessPerConnection of \{settings.MaxInProcessPerConnection} took \{stopWatch.ElapsedMilliseconds}ms");
 }


was (Author: someoneelse):
For everyone who is waiting for a fix, the following work around with polly does the trick for us, although it is not ideal. We already handled the '429 Too Many Requests' exceptions before, but now also added the handling of ConnectionPoolBusyExceptions and ServerUnavailableExceptions where we immediately create a new client (by calling the SetupGremlinClient method below).

 

{{ResultSet<dynamic> response = await Polly.Policy}}
{{ .Handle<ResponseException>(exception => exception.CosmosDbStatusCode() == RetryAfterStatusCode)}}
{{ .Or<ConnectionPoolBusyException>()}}
{{ .Or<ServerUnavailableException>()}}
{{ .WaitAndRetryAsync(}}
{{ retryCount: 10,}}
{{ sleepDurationProvider: (attempt, exception, _) =>}}
{{ {}}
{{ if (exception is ResponseException responseException)}}
{{ return responseException.CosmosDbRetryAfter();}}{{// In case this is a connection issue, we try again immediately}}
{{ return TimeSpan.Zero;}}
{{ },}}
{{ onRetryAsync: (exception, waitTime, attempt, _) =>}}
{{ {}}
{{ // In case this is a connection issue we reset the client}}
{{ if (exception is ConnectionPoolBusyException || exception is ServerUnavailableException)}}
{{ {}}
{{ _log.LogWarning($"Resetting the GremlinClient after experiencing an '\{exception.GetType()}'");}}
{{ SetupGremlinClient();}}
{{ }}}
{{ return Task.CompletedTask;}}
{{ })}}
{{ .ExecuteAsync(() => _gremlinClient.SubmitAsync<dynamic>(query));}}

 

{{private void SetupGremlinClient()}}
{{ {}}
{{ var stopWatch = new Stopwatch();}}
{{ stopWatch.Start();}}{{var settings = new ConnectionPoolSettings\{MaxInProcessPerConnection = 64, PoolSize = 8};// Default MacInProcessPerConnection = 32, PoolSize = 4}}{{_gremlinClient = new GremlinClient(_gremlinServer, new GraphSON2Reader(), new GraphSON2Writer(),}}
{{ GremlinClient.GraphSON2MimeType,settings}}
{{ ); }}
{{ stopWatch.Stop();}}{{_log.LogInformation($"Setting up GremlinClient with PoolSize of \{settings.PoolSize} and MaxInProcessPerConnection of \{settings.MaxInProcessPerConnection} took \{stopWatch.ElapsedMilliseconds}ms");}}
{{ }}}

> Get ConnectionPoolBusyException and then ServerUnavailableExceptions
> --------------------------------------------------------------------
>
>                 Key: TINKERPOP-2288
>                 URL: https://issues.apache.org/jira/browse/TINKERPOP-2288
>             Project: TinkerPop
>          Issue Type: Bug
>          Components: dotnet
>    Affects Versions: 3.4.1
>         Environment: Gremlin.Net 3.4.1
> Microsoft.NetCore.App 2.2
> Azure Cosmos DB
>            Reporter: patrice huot
>            Priority: Critical
>
> I am using .Net core Gremlin API  query Cosmos DB.
> From time to time we are getting an error saying that no connection is available and then the server become unavailable. When this is occurring we need to restart the server. It looks like the connections are not released properly and become unavailable forever.
> We have configured the pool size to 50 and the MaxInProcessPerConnection to 32 (Which I guess should be sufficient).
> To diagnose the issue, Is there a way to access diagnostic information on the connection pool in order to know how many connections are open and how many processes are running in each connection?
> I would like to be able to monitor the connections usage to see if they are about to be exhausted and to see if the number of used connections is always increasing or of the connection lease is release when the queries completes?
> As a work around, Is there a way we can access this information from the code so that I can catch those scenario and create logic that re-initiate the connection pool?
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)