[jira] [Commented] (OOZIE-1741) Add new coord EL function to get input partitions value string

["Satish Mittal (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OOZIE-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14017657#comment-14017657 ] 

Satish Mittal commented on OOZIE-1741:
--------------------------------------

Thanks [~thiruvel] for your inputs!

- For 1), we already have an EL dataInPartitionFilter().
- For 2), the proposed EL in this jira dataInPartitions() can serve the purpose.
- For 3), I think we should add new ELs as per the requirement, since a large number of combinations are possible here (comparator:- =, !=, <, > etc and separator:- AND, OR). 

The use-case of 3) is closer to 1) since it aims to generate a filter criteria, as against 2) which aims to generate the value of a partition spec to be used in DDL/import/export queries.

I checked that in Hive, the format of partition spec in various DDL queries (alter, truncate, drop, show, describe), DML queries (load, insert) is: (partcol1=val1, partcol2=val2 ...). The format of partition spec for IMPORT/EXPORT queries is: (partcol1="val1", partcol2="val2" ...). I verified that the latter format is a super-set, i.e. specifying (partcol1="val1", partcol2="val2" ...) works for all these queries. That is the format generated by EL dataInPartitions() in this patch as well.

Regarding partial/full partition spec, even the existing EL dataInPartitionSpec writes all partition cols specified in HCat URI for dataset. The same approach is taken for EL dataInPartitions() introduced in this patch.

[~rohini], can you provide your comments here?

> Add new coord EL function to get input partitions value string
> --------------------------------------------------------------
>
>                 Key: OOZIE-1741
>                 URL: https://issues.apache.org/jira/browse/OOZIE-1741
>             Project: Oozie
>          Issue Type: Improvement
>            Reporter: Satish Mittal
>            Assignee: Satish Mittal
>         Attachments: OOZIE-1741-v2.patch, OOZIE-1741-v3.patch, OOZIE-1741.patch.txt
>
>
> Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND)  for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...). We need to add a new EL function that returns partition value in the above format.



--
This message was sent by Atlassian JIRA
(v6.2#6252)