[jira] [Resolved] (HBASE-8475) TableMapReduceUtils#addDependencyJars(Job) shouldn't ship duplicate entries in tmpjars

["Andrew Kyle Purtell (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Kyle Purtell resolved HBASE-8475.
----------------------------------------
    Resolution: Incomplete

> TableMapReduceUtils#addDependencyJars(Job) shouldn't ship duplicate entries in tmpjars
> --------------------------------------------------------------------------------------
>
>                 Key: HBASE-8475
>                 URL: https://issues.apache.org/jira/browse/HBASE-8475
>             Project: HBase
>          Issue Type: Bug
>          Components: mapreduce
>            Reporter: Nick Dimiduk
>            Priority: Major
>
> *note* this is a hypothesis that needs further investigation.
> Users running the above method in their jobs as a convenience for  packaging up dependencies will see duplicate entries for jars providing common functionality, ie hbase.jar, hadoop.jar, user application jar. For instance, if they specify a custom output format and custom partitioner, both in the same jar, that jar is resolved via classloader and added to tmpjars twice. This imposes a fair amount of pre-job IO (particularly if you jars are fat with packed dependencies) and delay job launch accordingly.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)