[jira] [Comment Edited] (TEZ-1534) Make client side configs available to AM and tasks

["Hitesh Shah (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-1534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14129312#comment-14129312 ] 

Hitesh Shah edited comment on TEZ-1534 at 9/10/14 11:05 PM:
------------------------------------------------------------

Comments:

{code}
+    if (allowedPrefixes.length == 0) {
+      throw new TezUncheckedException("Configuration filter list should not be empty");
+    }
{code}
  - should this imply only Tez settings are allowed? 

{code}
+    if (allowedPrefixes.length == 1 &&
+        allowedPrefixes[0].equals(TezConfiguration.TEZ_ALLOWED_CONF_PREFIXES_ALLOW_ALL)) {
+      filterEnabled = false;
{code}
  - a trim may be needed before the equality check - dont believe getStrings() returns trimmed strings. 

  - Not sure whether "TEZ_ALLOWED_CONF_PREFIXES_DEFAULT" is useful from an end user point of view. When using this on a different ( non HDFS ) filesystem, there is no trivial way to pull in all the configs for that filesystem without modifying this config setting. Should we have _ALL_ as the default? 
  - Also, there is no check to ensure TEZ_PREFIX is always set. Users should not be allowed to remove tez settings via the whitelist filter. 

 



 


was (Author: hitesh):
Comments:

{code}
+    if (allowedPrefixes.length == 0) {
+      throw new TezUncheckedException("Configuration filter list should not be empty");
+    }
{code}
  - should this imply only Tez settings are allowed? 

{code}
+    if (allowedPrefixes.length == 1 &&
+        allowedPrefixes[0].equals(TezConfiguration.TEZ_ALLOWED_CONF_PREFIXES_ALLOW_ALL)) {
+      filterEnabled = false;
{code}
  - a trim may be needed before the equality check - dont believe getStrings() returns trimmed strings. 

  - Not sure whether "TEZ_ALLOWED_CONF_PREFIXES" is useful from an end user point of view. When using this on a different ( non HDFS ) filesystem, there is no trivial way to pull in all the configs for that filesystem without modifying this config setting. Should we have _ALL_ as the default? 
  - Also, there is no check to ensure TEZ_PREFIX is always set. Users should not be allowed to remove tez settings via the whitelist filter. 

 



 

> Make client side configs available to AM and tasks
> --------------------------------------------------
>
>                 Key: TEZ-1534
>                 URL: https://issues.apache.org/jira/browse/TEZ-1534
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Siddharth Seth
>            Assignee: Siddharth Seth
>         Attachments: TEZ-1534.1.txt
>
>
> Configs from the client (specifically the ones provided to TezClient, along with YARN additions) should be shipped over to the cluster (AM and tasks), instead of AM/tasks depending on configs present on cluster nodes.
> These configs will primarily be used for Tez components like RPC servers, clients etc - and not by the Processor / Input / Output - which should be sending over fully configured payloads in any case.
> Tez should continue to run without core-site, hdfs-site, yarn-site etc in the classpath on cluster nodes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)