You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hive.apache.org by "Gopal V (JIRA)" <ji...@apache.org> on 2018/07/06 22:29:00 UTC

[jira] [Created] (HIVE-20113) Shuffle avoidance: Disable 1-1 edges for sorted shuffle

Gopal V created HIVE-20113:
------------------------------

             Summary: Shuffle avoidance: Disable 1-1 edges for sorted shuffle 
                 Key: HIVE-20113
                 URL: https://issues.apache.org/jira/browse/HIVE-20113
             Project: Hive
          Issue Type: Bug
          Components: Tez
            Reporter: Gopal V


The sorted shuffle avoidance can have some issues when the shuffle data gets broken up into multiple chunks on disk.

The 1-1 edge cannot skip the tez final merge - there's no reason for 1-1 to have a final merge at all, it should open a single compressed file and write a single index entry.

Until the shuffle issue is resolved & a lot more testing, it is prudent to disable the optimization for sorted shuffle edges and stop rewriting the RS(sorted) = = = RS(sorted) into RS(sorted) = = = RS(FORWARD).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)