You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@phoenix.apache.org by "Chinmay Kulkarni (Jira)" <ji...@apache.org> on 2020/03/27 06:29:00 UTC
[jira] [Updated] (PHOENIX-5802) Connection leaks in UPSERT
SELECT/DELETE paths due to MutatingParallelIteratorFactory iterator not
being closed
[ https://issues.apache.org/jira/browse/PHOENIX-5802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chinmay Kulkarni updated PHOENIX-5802:
--------------------------------------
Description:
During an UPSERT SELECT query or client-side DELETE, we clone the existing Phoenix connection and call [mutate|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L59] which eventually calls either [upsertSelect|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/UpsertCompiler.java#L178] or [deleteRows|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java#L126].
When getting iterators, if we encounter any exception, then on calling [close|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1377], we attempt to cancel any queued work, while [accumulating already-started task futures|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1453].
Later we attempt to block on getting the result (iterator) of already-started tasks and [close each iterator|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1464-L1465].
Here however, if any of the upsert-select or client-side delete tasks had thrown an exception _E_, we wouldn't be able to close the underlying iterator (which in-turn [closes the cloned connection|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L101]). This manifests as [these logs|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1470]:
{code:java}
"Failed to execute task during cancel", <exception _E_ stack trace>
{code}
This is leading to a connection leak. Because we use CQSI to create the Phoenix Connection here, it also gets accounted for during open Phoenix connection throttling, thus leaving the client application with less connections that can be opened.
> Connection leaks in UPSERT SELECT/DELETE paths due to MutatingParallelIteratorFactory iterator not being closed
> ---------------------------------------------------------------------------------------------------------------
>
> Key: PHOENIX-5802
> URL: https://issues.apache.org/jira/browse/PHOENIX-5802
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 5.0.0, 4.15.0
> Reporter: Chinmay Kulkarni
> Assignee: Chinmay Kulkarni
> Priority: Major
> Fix For: 5.1.0, 4.16.0
>
>
> During an UPSERT SELECT query or client-side DELETE, we clone the existing Phoenix connection and call [mutate|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L59] which eventually calls either [upsertSelect|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/UpsertCompiler.java#L178] or [deleteRows|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java#L126].
> When getting iterators, if we encounter any exception, then on calling [close|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1377], we attempt to cancel any queued work, while [accumulating already-started task futures|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1453].
> Later we attempt to block on getting the result (iterator) of already-started tasks and [close each iterator|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1464-L1465].
> Here however, if any of the upsert-select or client-side delete tasks had thrown an exception _E_, we wouldn't be able to close the underlying iterator (which in-turn [closes the cloned connection|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L101]). This manifests as [these logs|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1470]:
> {code:java}
> "Failed to execute task during cancel", <exception _E_ stack trace>
> {code}
> This is leading to a connection leak. Because we use CQSI to create the Phoenix Connection here, it also gets accounted for during open Phoenix connection throttling, thus leaving the client application with less connections that can be opened.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)