You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Michael Ho (Jira)" <ji...@apache.org> on 2019/10/22 20:47:00 UTC

[jira] [Created] (IMPALA-9083) Explore co-partitioning join strategies

Michael Ho created IMPALA-9083:
----------------------------------

             Summary: Explore co-partitioning join strategies
                 Key: IMPALA-9083
                 URL: https://issues.apache.org/jira/browse/IMPALA-9083
             Project: IMPALA
          Issue Type: Improvement
          Components: Frontend
    Affects Versions: Impala 3.4.0
            Reporter: Michael Ho


The idea of co-partitioning join is not new and it has been implemented in other systems (see [link|https://docs.oracle.com/en/database/oracle/oracle-database/12.2/vldbg/partition-wise-joins.html] and [link|http://amithora.com/understanding-co-partitions-and-co-grouping-in-spark/]). With HDFS, we may not have as much control over the block locations so it's not as easy to make assumption about the co-location of partitions of multiple tables.

With remote reads (e.g. S3), we don't have this constraint anymore as all data are remote. So, if we are joining on the partition key of two tables, we can create a scan range schedule to co-locate partitions with the same partition values on the same executor and do the join locally to avoid the subsequent shuffling step.

The potential downside with this idea is that if there are skews in the data distribution, we may overwhelm a few of the nodes in both the scans and join operators. Previously, if we have skew in data distribution, we may still overwhelm the join operator but the scans can be better parallelized. cc'ing [~drorke]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)