You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@falcon.apache.org by Margus Roo <ma...@roo.ee> on 2016/02/15 08:39:58 UTC

Hive table partition and process feed

Hi

Maybe it is a dumb question but in case I have table with partition in 
example date_feed and flume puts data in to the right directories.
Now I have feed_1 hive table (partition=date_feed) -> process -> feed_2 
hive table.
First I'd like that process creates partition like: ALTER TABLE 
tablename ADD PARTITION (date_feed="current date") LOCATION (right 
location in HDFS)
Is is possible? I am a little confused. Does process starts if there is 
no partition exists is hive table already?

-- 
Margus (margusja) Roo
http://margus.roo.ee
skype: margusja
+372 51 48 780

Re: Hive table partition and process feed

Posted by Ying Zheng <yz...@hortonworks.com>.
Margus: The current implementation doesn¹t support ³current date². It
needs to be relative to nominal time (e.g. data_feed in your example).

Thanks,
Ying


On 2/14/16, 11:39 PM, "Margus Roo" <ma...@roo.ee> wrote:

>Hi
>
>Maybe it is a dumb question but in case I have table with partition in
>example date_feed and flume puts data in to the right directories.
>Now I have feed_1 hive table (partition=date_feed) -> process -> feed_2
>hive table.
>First I'd like that process creates partition like: ALTER TABLE
>tablename ADD PARTITION (date_feed="current date") LOCATION (right
>location in HDFS)
>Is is possible? I am a little confused. Does process starts if there is
>no partition exists is hive table already?
>
>-- 
>Margus (margusja) Roo
>http://margus.roo.ee
>skype: margusja
>+372 51 48 780
>
>

Re: Hive table partition and process feed

Posted by Peeyush Bishnoi <bp...@yahoo.co.in>.
Margus,
Yes it is possible to create partition, for the location of data in HDFS from hive based process entity,  for the table.For this you have to define required hive statement in your hive script.
No, process will not start if partition for table does not exists. But ensure that in Falcon, table URI must be defined in Feed entity using HCatalog partition URI or partition must be added through HCatalog.
Thanks---Peeyush   

    On Monday, 15 February 2016 1:10 PM, Margus Roo <ma...@roo.ee> wrote:
 

 Hi

Maybe it is a dumb question but in case I have table with partition in 
example date_feed and flume puts data in to the right directories.
Now I have feed_1 hive table (partition=date_feed) -> process -> feed_2 
hive table.
First I'd like that process creates partition like: ALTER TABLE 
tablename ADD PARTITION (date_feed="current date") LOCATION (right 
location in HDFS)
Is is possible? I am a little confused. Does process starts if there is 
no partition exists is hive table already?

-- 
Margus (margusja) Roo
http://margus.roo.ee
skype: margusja
+372 51 48 780