[jira] [Created] (IMPALA-12995) Don't always sort data for Iceberg partitioned inserts

["Zoltán Borók-Nagy (Jira)" <ji...@apache.org>]

Zoltán Borók-Nagy created IMPALA-12995:
------------------------------------------

             Summary: Don't always sort data for Iceberg partitioned inserts
                 Key: IMPALA-12995
                 URL: https://issues.apache.org/jira/browse/IMPALA-12995
             Project: IMPALA
          Issue Type: Bug
          Components: Frontend
            Reporter: Zoltán Borók-Nagy


Currently we always do a SHUFFLE and SORT before inserting data into a partitioned Iceberg table.
With SHUFFLE, we can guarantee that each partition is assigned to a single sink operator, hence we can minimize the number of files being created.
With SORT, we can write partitions one after the other, therefore we only need to write one file at a time (and only buffer data for that one file). This way we can avoid out of memory situations in the sink.

SORT does a total ordering of the incoming records, therefore it can be very expensive. And when SORT needs to spill to disk, its fragment cannot receive incoming records, blocking the execution of the whole query cluster-wide (because after some time every sender is blocked on the receiving SORT fragment).

In a lot of cases the SORT is not necessary because there's only a very few partitions assigned to each sink, especially in large clusters with high MT_DOP.

During planning Impala should decide whether to add the SORT node for such partitioned INSERTs. Also, it should respect the optimizer hints CLUSTERED/NOCLUSTERED.
https://impala.apache.org/docs/build/html/topics/impala_hints.html



--
This message was sent by Atlassian Jira
(v8.20.10#820010)