[jira] [Updated] (PHOENIX-5802) Connection leaks in UPSERT SELECT/DELETE paths due to MutatingParallelIteratorFactory iterator not being closed

["Chinmay Kulkarni (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PHOENIX-5802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chinmay Kulkarni updated PHOENIX-5802:
--------------------------------------
    Affects Version/s: 4.14.3

> Connection leaks in UPSERT SELECT/DELETE paths due to MutatingParallelIteratorFactory iterator not being closed
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: PHOENIX-5802
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-5802
>             Project: Phoenix
>          Issue Type: Bug
>    Affects Versions: 5.0.0, 4.15.0, 4.14.3
>            Reporter: Chinmay Kulkarni
>            Assignee: Chinmay Kulkarni
>            Priority: Major
>              Labels: phoenix-hardening
>             Fix For: 5.1.0, 4.16.0
>
>         Attachments: PHOENIX-5802-4.x-v1.patch, PHOENIX-5802-4.x-v2.patch
>
>          Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> During an UPSERT SELECT query or client-side DELETE, we clone the existing Phoenix connection and call [mutate|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L59] which eventually calls either [upsertSelect|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/UpsertCompiler.java#L178] or [deleteRows|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java#L126].
> When getting iterators, if we encounter any exception, then on calling [close|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1377], we attempt to cancel any queued work, while [accumulating already-started task futures|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1453]. 
>  Later we attempt to block on getting the result (iterator) of already-started tasks and [close each iterator|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1464-L1465].
> Here however, if any of the upsert-select or client-side delete tasks had thrown an exception _E_, we wouldn't be able to close the underlying iterator (which in-turn would've [closed the cloned connection|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L101]). This manifests as [these logs|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1470]:
> {code:java}
> "Failed to execute task during cancel", <exception E stack trace>
> {code}
> This is leading to a connection leak. Because we use CQSI to create the Phoenix Connection here, it also gets accounted for during open Phoenix connection throttling, thus leaving the client application with less connections that can be opened.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)