[jira] [Comment Edited] (LENS-418) Design issues in add partitions(s)

["Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LENS-418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14367103#comment-14367103 ] 

Amareshwari Sriramadasu edited comment on LENS-418 at 3/18/15 1:11 PM:
-----------------------------------------------------------------------

On the metastore service we have partitions under {noformat}/facts/{factName}/storages/{storage}/partitions {noformat}and {noformat}/dimtables/{dimTableName}/storages/{storage}/partitions {noformat}- With this resource binding, partitions are always tied to fact in a storage, dimtable in a storage. If we go with this, even add/drop partitions will happen on fact/dimtable on a storage.

What is put in the jira is whether we allow add/partitions to be across facts/dimtables and storages. 

I would say no user will be registering partitions across facts/dimtables and storages because usually the ingestion pipelines will be adding partitions for one fact or one dimtable on one storage at a time. But open to look at other options here. 

bq. Fact name is mentioned in command as well as the xml. What will happen if both are different?
We can remove usage of redundant fact_or_dimension_table_name from XPartition with current registration model.


was (Author: amareshwari):
On the metastore service we have partitions under /facts/{factName}/storages/{storage}/partitions and /dimtables/{dimTableName}/storages/{storage}/partitions - With this resource binding, partitions are always tied to fact in a storage, dimtable in a storage. If we go with this, even add/drop partitions will happen on fact/dimtable on a storage.

What is put in the jira is whether we allow add/partitions to be across facts/dimtables and storages. 

I would say no user will be registering partitions across facts/dimtables and storages because usually the ingestion pipelines will be adding partitions for one fact or one dimtable on one storage at a time. But open to look at other options here. 

bq. Fact name is mentioned in command as well as the xml. What will happen if both are different?
We can remove usage of redundant fact_or_dimension_table_name from XPartition with current registration model.

> Design issues in add partitions(s)
> ----------------------------------
>
>                 Key: LENS-418
>                 URL: https://issues.apache.org/jira/browse/LENS-418
>             Project: Apache Lens
>          Issue Type: Bug
>            Reporter: Rajat Khandelwal
>
> cli command is: 
> {noformat}
> fact add partitions factname storagename filepath
> {noformat}
> example partition list is:
> {noformat}
> <x_partition_list xmlns="uri:lens:cube:0.1"
>   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="uri:lens:cube:0.1 cube-0.1.xsd ">
>   <partition fact_or_dimension_table_name="fact1" location="/tmp/fact1" update_period="HOURLY"
>     xmlns="uri:lens:cube:0.1"
>     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="uri:lens:cube:0.1 cube-0.1.xsd ">
>     <time_partition_spec>
>       <part_spec_element key="dt" value="2014-03-27T12:00:00"/>
>     </time_partition_spec>
>   </partition>
> </x_partition_list>
> {noformat}
> Fact name is mentioned in command as well as the xml. What will happen if both are different? 
> In my opinion, the design should be the following:
> command: 
> {noformat}
> add partitions filepath
> {noformat}
> file will contain x_partition_list. which will contain a lot of `partition`s which can belong to different fact/dimtable and storages. 
> Thoughts?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)