[jira] [Commented] (FALCON-90) Propagate hive table through Process mapper

["Srikanth Sundarrajan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FALCON-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782119#comment-13782119 ] 

Srikanth Sundarrajan commented on FALCON-90:
--------------------------------------------

Should we add a "." between prefix and property name ? Also should we rename this method to propogateCatalogProperties instead since Storage is common to both FS & catalog. Perhaps we can keep pigFilter and other filter instead of filter for each action type. (

{code}
    private void propagateStorageProperties(Input input, CatalogStorage tableStorage,
                                            Map<String, String> props) {
        String prefix = input.getName();

        props.put(prefix + "StorageType", tableStorage.getType().name());
        props.put(prefix + "CatalogUrl", tableStorage.getCatalogUrl());
        props.put(prefix + "Database", tableStorage.getDatabase());
        props.put(prefix + "Table", tableStorage.getTable());

        props.put(prefix + "PigPartitionFilter", "${coord:dataInPartitionFilter('input', 'pig')}");
        props.put(prefix + "HivePartitionFilter", "${coord:dataInPartitionFilter('input', 'hive')}");
        props.put(prefix + "JavaPartitionFilter", "${coord:dataInPartitionFilter('input', 'java')}");
    }
{code}


LateRerunConsumer::detectLate assumes falconInputFeeds & falconInPaths are populated. however they are not populated in case hcat tables are used for process. Can we ensure that this code path is not executed if input is hcat till the late handling is wired up for hcat tables ?

Should propagateStorageProperties() also prefix + "." + property names ?

> Propagate hive table through Process mapper
> -------------------------------------------
>
>                 Key: FALCON-90
>                 URL: https://issues.apache.org/jira/browse/FALCON-90
>             Project: Falcon
>          Issue Type: Sub-task
>    Affects Versions: 0.3
>            Reporter: Venkatesh Seetharam
>            Assignee: Venkatesh Seetharam
>         Attachments: FALCON-90.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1#6144)