[jira] [Commented] (TEZ-3240) Improvements to tez.lib.uris to allow for multiple tarballs and mixing tarballs and jars.

["Hitesh Shah (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15270868#comment-15270868 ] 

Hitesh Shah commented on TEZ-3240:
----------------------------------

The main problem with allowing for a mapreduce tarball to be supplied is the classpath construction. Additionally, unlike today where all compile time dependencies are correctly setup in the single tez tarball, relying on externally supplied dependencies from a different tarball implies potential gotchas for mismatches of versions which will only be caught at runtime. 

That said, we could consider enhancing tez.aux.uris to support archive formats and let tez.lib.uris just be for the tez bits as it exists today.

> Improvements to tez.lib.uris to allow for multiple tarballs and mixing tarballs and jars. 
> ------------------------------------------------------------------------------------------
>
>                 Key: TEZ-3240
>                 URL: https://issues.apache.org/jira/browse/TEZ-3240
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Eric Badger
>            Assignee: Eric Badger
>
> Currently, tez.lib.uris only supports either a single archive or paths for multiple jars. You cannot mix and match between the two and you also cannot specify more than one archive. This means that you cannot specify both the tez and mapreduce archives. In the case where there is already a mapreduce archive in the distributed cache, you would not be able to use it when running tez. Instead, you would have to include the mapreduce jars in the single archive that you give to tez.lib.uris or use the mapreduce jars that are on the cluster node itself. This makes it very easy for the mapreduce versions to be out of sync with each other. 
> With the current implementation, during a rolling upgrade it is very easy to have jobs that do not get the same mapreduce jars across all of the containers, since some will start after the node's jars have been upgraded and some will start before. 
> If, instead, the job uses a archive that packages both tez and mapreduce together, then you will have 2 copies of the mapreduce jars in the distributed cache and will also have to upgrade both whenever you make a single upgrade to mapreduce. 
> I propose 2 improvements:
> 1) Allow tez.lib.uris to take an arbitrary number of archives and jars, while not being limited to only one or the other
> 2) Allow tez.lib.uris to specify a fragment following the '#' symbol (as is done in mapreduce) that will create a symlink with the name of the fragment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)