You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Siddharth Seth (JIRA)" <ji...@apache.org> on 2013/08/19 20:46:48 UTC
[jira] [Updated] (TEZ-376) Reduce Processor can parallelize
I/O/Processor initialization
[ https://issues.apache.org/jira/browse/TEZ-376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated TEZ-376:
-------------------------------
Attachment: TEZ-376.txt
Attaching a patch to parallelize the initialization.
[~hitesh], could you please take a look.
Will update the patch with similar changes on the Map side as well if this looks reasonable.
> Reduce Processor can parallelize I/O/Processor initialization
> -------------------------------------------------------------
>
> Key: TEZ-376
> URL: https://issues.apache.org/jira/browse/TEZ-376
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Attachments: TEZ-376.txt
>
>
> Reported by [~gopalv]. Reduce initialization can take a fair amount of time after shuffle is complete - depending on how expensive the Reduce initialization is.
> Quote from Gopal's investigation
> 1.6 seconds is spent in ReflectionUtils.newInstance() -> ReflectionUtils.setConf() of which 1.2 seconds is Utilities.getMapRedWork().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira