You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Sergey Shelukhin (JIRA)" <ji...@apache.org> on 2017/10/18 01:39:00 UTC
[jira] [Updated] (HIVE-17827) refactor TezTask file management

     [ https://issues.apache.org/jira/browse/HIVE-17827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sergey Shelukhin updated HIVE-17827:
------------------------------------
    Description: 
There are about 5 different duplicate and intersecting structures used in TezTask to manage the file list (additionalLr, inputOutputX xN (array and list and/or map?), resourceMap; plus a list in the actual session object); multiple methods named addExtraResourcesBlah, localizeResourcesBlah; at least 2 places where TezClient is invoked to add resources to AM, on the same path.
All this mess needs to be changed to have exactly one place that figures out what needs to be localized, and exactly one place (in TezSessionState so we don't duplicate state and TezClient usage inside and outside of it) where we look at what's already there and localize things that are not.

  was:
There are about 5 different duplicate and intersecting structures used in TezTask so to manage the file list (additionalLr, inputOutputX xN (array and list and/or map?), resourceMap; plus a list in the actual session object); multiple methods named addExtraResourcesBlah, localizeResourcesBlah; at least 2 places where TezClient is invoked to add resources to AM, on the same path.
All this mess needs to be changed to have exactly one place that figures out what needs to be localized, and exactly one place (in TezSessionState so we don't duplicate state and TezClient usage inside and outside of it) where we look at what's already there and localize things that are not.


> refactor TezTask file management
> --------------------------------
>
>                 Key: HIVE-17827
>                 URL: https://issues.apache.org/jira/browse/HIVE-17827
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Sergey Shelukhin
>
> There are about 5 different duplicate and intersecting structures used in TezTask to manage the file list (additionalLr, inputOutputX xN (array and list and/or map?), resourceMap; plus a list in the actual session object); multiple methods named addExtraResourcesBlah, localizeResourcesBlah; at least 2 places where TezClient is invoked to add resources to AM, on the same path.
> All this mess needs to be changed to have exactly one place that figures out what needs to be localized, and exactly one place (in TezSessionState so we don't duplicate state and TezClient usage inside and outside of it) where we look at what's already there and localize things that are not.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)