[jira] [Created] (ARROW-16163) [Go] IPC FileReader leaks memory when used with ZSTD compression

["Nicolas MOREAU (Jira)" <ji...@apache.org>]

Nicolas MOREAU created ARROW-16163:
--------------------------------------

             Summary: [Go] IPC FileReader leaks memory when used with ZSTD compression
                 Key: ARROW-16163
                 URL: https://issues.apache.org/jira/browse/ARROW-16163
             Project: Apache Arrow
          Issue Type: Bug
          Components: Go
            Reporter: Nicolas MOREAU


I'll submit a PR :)

 

When using the IPC Reader to read data compressed in zstd, we observed what looked like a memory leak.

After further debugging, we observed that calling the IPC Reader created goroutines that were never stopped, and consumed memory.

 

After deeper investigation, we found the following root cause:

 

In arrow/ipc/file_reader.go, the newRecord function calls getDecompressor, that returns a zstd.Decoder object.

 

When used in its stream version (calling .Read()), this decoder makes use of a Goroutine with a channel to receive input. This channel is closed by the decoder's .Close() function, which ends the Goroutine and therefore the collection of its memory by the GC.

 

After using the zstd Decoder, the IPC FileReader should close it to prevent the leak.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)