[jira] [Created] (IMPALA-11539) Intra-node skew for HDFS scans

["Zoltán Borók-Nagy (Jira)" <ji...@apache.org>]

Zoltán Borók-Nagy created IMPALA-11539:
------------------------------------------

             Summary: Intra-node skew for HDFS scans
                 Key: IMPALA-11539
                 URL: https://issues.apache.org/jira/browse/IMPALA-11539
             Project: IMPALA
          Issue Type: Bug
          Components: Backend
            Reporter: Zoltán Borók-Nagy


Before IMPALA-9655 scan ranges were statically assigned to intra-node fragment instances based on Least-Processing Time algorithm:
https://github.com/apache/impala/blame/a7866a94578be6289bbac31686de4d9032ad9261/be/src/scheduling/scheduler.cc#L499-L501

From IMPALA-9655 we use dynamic intra-node load balancing for HDFS scans. It means fragment instances have a shared queue of scan ranges and the fragment instances grab the next scan range to be read from this queue.

IMPALA-9655 got rid of the LPT-algorithm which means  the scan ranges are in a random order in the queue. This can lead to a skew if there are large scan ranges at the end.

We could mix the above two by using a priority queue for the scan ranges, so each fragment instance would grab the largest scan range in the queue. This could further mitigate intra-node skewing.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)