[jira] [Commented] (TINKERPOP-2352) Gremlin Python driver default pool size makes Gremlin keep-alive difficult

["Stephen Mallette (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TINKERPOP-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064737#comment-17064737 ] 

Stephen Mallette commented on TINKERPOP-2352:
---------------------------------------------

Thanks for your thoughts on this one. I don't think there's a problem with changing the default pool size to what you suggest as long as we have tests that continue to validate behavior of the larger pool size somehow. A pull request would be great especially one that included some more documentation of the type you describe, of course, I think a nicer pull request would be to solve the keep-alive problem more generally as described on TINKERPOP-1886. Fixing that would be a much more robust solution. If you have the opportunity to help there it would be appreciated. If you could solve that, then perhaps we should just close this ticket in favor of that one and continue our discussion there. 

> Gremlin Python driver default pool size makes Gremlin keep-alive difficult
> --------------------------------------------------------------------------
>
>                 Key: TINKERPOP-2352
>                 URL: https://issues.apache.org/jira/browse/TINKERPOP-2352
>             Project: TinkerPop
>          Issue Type: Bug
>          Components: python
>    Affects Versions: 3.3.5, 3.4.5
>         Environment: AWS Lambda, Python 3.7 runtime, AWS Neptune.
> (AWS Lambda functions can remain in memory and thus hold connections open for many minutes between invocations)
>            Reporter: Mark Br...e
>            Priority: Major
>
> I'm working with a Gremlin database that (like many) terminates connections if they don't execute any transactions with a timeout period.  When we want to run a traversal we first check our `GraphTraversalSource` by running `g.V().limit(1).count().next()` and if that raises an exception we know we need to reconnect before running the actual traversal.
> We've been very confused that this hasn't worked as expected: we intermittently see traversals fail with `WebSocketClosed` or other connection-related errors immediately after the "connection test" passes. 
> I've (finally) found the cause of this inconsistency is the default pool size in `gremlin_python.driver.client.Client` being 4.  This means there's no visiblity outside the `Client` of which connection in the pool is tested and/or used, and in fact no way for the application (`GraphTraversalSource`) to run keep-alive type traversals reliably.  Anytime an application passes in a pool size of `None` or a number > 1 there'll be no way to make sure that each and every connection in the pool actually sends keep-alive traversals to the remote, _except_ in the case of a single-threaded application where a tight loop could issue `pool_size` of them.  In that latter case as the application is single-threaded then a `pool_size` above 1 won't provide much benefit.
> I've raised this as a bug because I think a default `pool_size` of 1 would give much more predictable behaviour, and in the specific case of the Python driver is probably more appropriate because Python applications tend to run single-threaded by default, with multi-threading carefully added when performance requires it.  Perhaps it's a wish, but as the behaviour from the default option is quite confusing it feels more like a bug, at least.  If it would help I'm happy to raise a PR with some updated function header comments or maybe updated documentation about multi-threaded / multi-async-loop usage of gremlin-python.
> (This is my first issue here, apologies if it has some fields wrong.)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)