You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tajo.apache.org by "Jihoon Son (JIRA)" <ji...@apache.org> on 2014/11/11 02:56:34 UTC
[jira] [Commented] (TAJO-982) Improve Fetcher to get multiple
shuffle outputs through a request
[ https://issues.apache.org/jira/browse/TAJO-982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14205798#comment-14205798 ]
Jihoon Son commented on TAJO-982:
---------------------------------
Great idea!
IMHO, this issue is a type of an improvement instead of a bug.
What do you think?
> Improve Fetcher to get multiple shuffle outputs through a request
> -----------------------------------------------------------------
>
> Key: TAJO-982
> URL: https://issues.apache.org/jira/browse/TAJO-982
> Project: Tajo
> Issue Type: Bug
> Components: data shuffle
> Reporter: Hyunsik Choi
> Assignee: Hyoungjun Kim
> Fix For: 0.9.1
>
>
> Currently, Fetcher only can request at most a fetch for one shuffle output at a time. The implementation can cause performance degradation even though intermediate data is actually small.
> For example, If an input data set of the first stage is big and the intermediate data is very small, QueryMaster will choose a few of nodes for next execution block. Since each node keeps limited threads for fetch, it will take a long time for the nodes for next stage to fetch all intermediate.
> If Fetcher can get multiple shuffle outputs through a request, it would solve the slowness which occurs in some cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)