[jira] [Commented] (ARROW-11569) Out-of-heap memory leaks in FlightClient.getStream

["David Li (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-11569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281819#comment-17281819 ] 

David Li commented on ARROW-11569:
----------------------------------

[~lulijun] Just to double-check, are you also closing the ArrowRecordBatch you generate? It's not clear from the snippet here + that's a common mistake.

> Out-of-heap memory leaks in FlightClient.getStream
> --------------------------------------------------
>
>                 Key: ARROW-11569
>                 URL: https://issues.apache.org/jira/browse/ARROW-11569
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: FlightRPC
>    Affects Versions: 3.0.0
>         Environment: Java version: 8
> Flight&Arrow version: 3.0.0
>            Reporter: lulijun
>            Priority: Major
>              Labels: FlightClient, MemoryLeak
>
> I'm trying to use Arrow and Flight in my Java project, I used FlightClient.getStream to request data from a remote node, but I found the allocator's allocated memory size grows with the request.
> Java version: 8
> Flight&Arrow version: 3.0.0
>  
> Here's a snippet of my code, I found the allocated memory of flithStream still exist after root.close:
> try (FlightStream flightStream = client.getStream(ticket, CallOptions.timeout(1, TimeUnit.MILLISECONDS))) {
>     VectorSchemaRoot root = flightStream.getRoot();
>     root.clear();
>     while (flightStream.next()) {
>         // transfer root to ArrowRecordBatch and execute
>         dispatchNext(shuffleChunk, root);
>         root.clear();
>     }
>     root.close();
> } catch (FlightRuntimeException fe) {
>     String errorMsg = "IndexReadOperator read index fail ={}  "+indexChunkDesc.toString();
>     LOGGER.error(errorMsg, fe);
>     throw new NetworkException(errorMsg, fe);
> } catch (Exception e) {
>     LOGGER.error("IndexReadOperator read index fail!", e);
>  throw new NetworkException("IndexReadOperator getStream fail! ", e);
> }
>  
> Is this a bug or am I using it incorrectly?
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)