[jira] [Comment Edited] (TEZ-3240) Improvements to tez.lib.uris to allow for multiple tarballs and mixing tarballs and jars.

["Hitesh Shah (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15270937#comment-15270937 ] 

Hitesh Shah edited comment on TEZ-3240 at 5/4/16 4:27 PM:
----------------------------------------------------------

bq. This method would allow us to specify exactly what version we want to run with tez.

Sure - from a deployment perspective, this does let you select what version of the MR tarball you want to use. However, it has no correlation to what version of hadoop/MR modules the tez codebase was compiled against. 

bq. Instead, you would have to include the mapreduce jars in the single archive that you give to tez.lib.uris

The mapreduce jars that Tez needs are already bundled into the tez tarball today as compile time. If you are using the minimal tarball then that is not a recommended approach for a cluster that does rolling upgrades. 


was (Author: hitesh):
bq. This method would allow us to specify exactly what version we want to run with tez.

Sure - from a deployment perspective, this does let you select what version of the MR tarball you want to use. However, it has no correlation to what version of hadoop/MR modules the tez codebase was compiled against. 

bq. Instead, you would have to include the mapreduce jars in the single archive that you give to tez.lib.uris

The mapreduce jars that Tez needs are already bundled into the tez tarball today. If you are using the minimal tarball then that is not a recommended approach for a cluster that does rolling upgrades. 

> Improvements to tez.lib.uris to allow for multiple tarballs and mixing tarballs and jars. 
> ------------------------------------------------------------------------------------------
>
>                 Key: TEZ-3240
>                 URL: https://issues.apache.org/jira/browse/TEZ-3240
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Eric Badger
>            Assignee: Eric Badger
>         Attachments: TEZ-3240.001.patch
>
>
> Currently, tez.lib.uris only supports either a single archive or paths for multiple jars. You cannot mix and match between the two and you also cannot specify more than one archive. This means that you cannot specify both the tez and mapreduce archives. In the case where there is already a mapreduce archive in the distributed cache, you would not be able to use it when running tez. Instead, you would have to include the mapreduce jars in the single archive that you give to tez.lib.uris or use the mapreduce jars that are on the cluster node itself. This makes it very easy for the mapreduce versions to be out of sync with each other. 
> With the current implementation, during a rolling upgrade it is very easy to have jobs that do not get the same mapreduce jars across all of the containers, since some will start after the node's jars have been upgraded and some will start before. 
> If, instead, the job uses a archive that packages both tez and mapreduce together, then you will have 2 copies of the mapreduce jars in the distributed cache and will also have to upgrade both whenever you make a single upgrade to mapreduce. 
> I propose 2 improvements:
> 1) Allow tez.lib.uris to take an arbitrary number of archives and jars, while not being limited to only one or the other
> 2) Allow tez.lib.uris to specify a fragment following the '#' symbol (as is done in mapreduce) that will create a symlink with the name of the fragment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)