[jira] [Created] (TAJO-1996) Too many index lookups during range shuffle

["Jihoon Son (JIRA)" <ji...@apache.org>]

Jihoon Son created TAJO-1996:
--------------------------------

             Summary: Too many index lookups during range shuffle
                 Key: TAJO-1996
                 URL: https://issues.apache.org/jira/browse/TAJO-1996
             Project: Tajo
          Issue Type: Improvement
          Components: Pull Server, Data Shuffle
            Reporter: Jihoon Son
            Priority: Critical
             Fix For: 0.12.0


During range shuffle, there are a huge number of index lookups, thereby taking at least the half time of the total shuffle time. This is because it is difficult to know which data is stored on which machine, so each machine needs to ask to all machines including itself. 

Unfortunately, this problem is difficult to resolved using statistics. With an assumption of even data distribution, every machine is expected to have the data of the whole range. 

So, we need to investigate an alternative solution to reduce the frequent index lookups.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)