[jira] [Commented] (TEZ-2563) unable to pass environment/classpath to AM via TezClient

["Hitesh Shah (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-2563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14590831#comment-14590831 ] 

Hitesh Shah commented on TEZ-2563:
----------------------------------

[~fs111] Can you upload the launch_container.sh for the ApplicationMaster? It would exist in one of the local-dirs configured for the YARN Nodemanager on the node where the AM was launched. 

tez.am.launch.env seems to have been implement to only set the environment if there is no existing value. 

You can try setting "tez.cluster.additional.classpath.prefix" to get the classpath updated. By default, $PWD and $PWD/* are always in the classpath so normal jars would work if they have been added to LocalResources as type FILE. On the other hand, fat-jars at times do not work depending on whether they are unbundled on localization. Taking a look at launch_container.sh would help to see what is going on. Furthermore, you can also look at the container's working dir to see how the localized bits are laid out. 

You will need to set "yarn.nodemanager.delete.debug-delay-sec" to a large value and restart all nodemanagers to be able to find the above files. ( Dirs get deleted on container completion unless debug delay is configured ) 






> unable to pass environment/classpath to AM via TezClient
> --------------------------------------------------------
>
>                 Key: TEZ-2563
>                 URL: https://issues.apache.org/jira/browse/TEZ-2563
>             Project: Apache Tez
>          Issue Type: Bug
>    Affects Versions: 0.7.0, 0.6.1
>            Reporter: André Kelpe
>
> In Cascading we allow our users to submit applications as jar files. These jar files can contain custom Input- and Output-Formats, that are only in the application jar and not installed cluster side. We now run into the problem that the ApplicationMaster is unable to find the InputFormat during split calculation, if it is packaged in a hadoop style jar. (fat-jars seem to work). 
> On the Dag level we are able to pass an environment along with the LocalResources to be used so that we can tell YARN, what to put on the classpath. There is no obvious way to do the same for the TezClient so that we can fix the ClassNotFoundException.
> P.S.:
> I tried to "hack" around this problem by setting TezConfiguration.TEZ_AM_LAUNCH_ENV, but that does not work, since I have to be able to resolve all paths and cannot just pass the templates along.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)