You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@drill.apache.org by "James Turton (Jira)" <ji...@apache.org> on 2022/12/06 15:32:00 UTC

[jira] [Created] (DRILL-8366) Late release of compressor memory in the Parquet writer

James Turton created DRILL-8366:
-----------------------------------

             Summary: Late release of compressor memory in the Parquet writer
                 Key: DRILL-8366
                 URL: https://issues.apache.org/jira/browse/DRILL-8366
             Project: Apache Drill
          Issue Type: Bug
          Components: Storage - Parquet
    Affects Versions: 1.20.2
            Reporter: James Turton
            Assignee: James Turton
             Fix For: 1.20.3


The Parquet writer waits until the end of the entire write before releasing its compression codec factory. The factory in turn releases compressors which release direct memory buffers used during compression. This deferred release leads a build up of direct memory use and can cause large write jobs to fail. The Parquet writer can instead release the abovementioned each time that a file/row group is flushed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)