[jira] [Created] (DRILL-5267) Managed external sort spills too often with Parquet data

["Paul Rogers (JIRA)" <ji...@apache.org>]

Paul Rogers created DRILL-5267:
----------------------------------

             Summary: Managed external sort spills too often with Parquet data
                 Key: DRILL-5267
                 URL: https://issues.apache.org/jira/browse/DRILL-5267
             Project: Apache Drill
          Issue Type: Bug
    Affects Versions: 1.10
            Reporter: Paul Rogers
            Assignee: Paul Rogers
             Fix For: 1.10


DRILL-5266 describes how Parquet produces low-density record batches. The result of these batches is that the external sort spills more frequently than it should because it sizes spill files based on batch size, not data content of the batch. Since Parquet batches are 95% empty space, the spill files end up far too small.

Adjust the spill calculations based on actual data content, not the size of the overall record batch.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)