You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by praveenesh kumar <pr...@gmail.com> on 2013/09/17 13:20:00 UTC
Oozie dynamic action
Hi,
I have a scenario in which I want to trigger a hive uploading script every
day. I have a set of folders created for a set of customer ids everyday. My
hive script will read the customer id from the path, checks whether the
table for the customer id exits and if not create a table and will create
partition based on date for a "set of unknown customer_ids".
I can get the set of unique customer_ids from a shell action. It can be
passed as a list or string.
My problem is how can I achieve this dynamic checking/creation of hive
tables and partitions from oozie.
Currently I am doing everything from the shell script and calling it as a
shell action in oozie, but I was wondering if these kind of checks or some
kind of for-loop kind of actions can be done in oozie.
Any thoughts/suggestions on how to tackle the above scenario in the best
way possible using oozie, would be highly helpful.
Regards
Praveenesh
Re: Oozie dynamic action
Posted by Peyman Mohajerian <mo...@gmail.com>.
If you want to see a simple example of what you are looking for:
https://github.com/cloudera/cdh-twitter-example
It is part of this article:
http://blog.cloudera.com/blog/2012/09/analyzing-twitter-data-with-hadoop/
On Tue, Sep 17, 2013 at 4:20 AM, praveenesh kumar <pr...@gmail.com>wrote:
> Hi,
>
> I have a scenario in which I want to trigger a hive uploading script every
> day. I have a set of folders created for a set of customer ids everyday. My
> hive script will read the customer id from the path, checks whether the
> table for the customer id exits and if not create a table and will create
> partition based on date for a "set of unknown customer_ids".
>
> I can get the set of unique customer_ids from a shell action. It can be
> passed as a list or string.
>
> My problem is how can I achieve this dynamic checking/creation of hive
> tables and partitions from oozie.
>
> Currently I am doing everything from the shell script and calling it as a
> shell action in oozie, but I was wondering if these kind of checks or some
> kind of for-loop kind of actions can be done in oozie.
>
> Any thoughts/suggestions on how to tackle the above scenario in the best
> way possible using oozie, would be highly helpful.
>
> Regards
> Praveenesh
>
>
Re: Oozie dynamic action
Posted by Peyman Mohajerian <mo...@gmail.com>.
If you want to see a simple example of what you are looking for:
https://github.com/cloudera/cdh-twitter-example
It is part of this article:
http://blog.cloudera.com/blog/2012/09/analyzing-twitter-data-with-hadoop/
On Tue, Sep 17, 2013 at 4:20 AM, praveenesh kumar <pr...@gmail.com>wrote:
> Hi,
>
> I have a scenario in which I want to trigger a hive uploading script every
> day. I have a set of folders created for a set of customer ids everyday. My
> hive script will read the customer id from the path, checks whether the
> table for the customer id exits and if not create a table and will create
> partition based on date for a "set of unknown customer_ids".
>
> I can get the set of unique customer_ids from a shell action. It can be
> passed as a list or string.
>
> My problem is how can I achieve this dynamic checking/creation of hive
> tables and partitions from oozie.
>
> Currently I am doing everything from the shell script and calling it as a
> shell action in oozie, but I was wondering if these kind of checks or some
> kind of for-loop kind of actions can be done in oozie.
>
> Any thoughts/suggestions on how to tackle the above scenario in the best
> way possible using oozie, would be highly helpful.
>
> Regards
> Praveenesh
>
>
Re: Oozie dynamic action
Posted by Peyman Mohajerian <mo...@gmail.com>.
If you want to see a simple example of what you are looking for:
https://github.com/cloudera/cdh-twitter-example
It is part of this article:
http://blog.cloudera.com/blog/2012/09/analyzing-twitter-data-with-hadoop/
On Tue, Sep 17, 2013 at 4:20 AM, praveenesh kumar <pr...@gmail.com>wrote:
> Hi,
>
> I have a scenario in which I want to trigger a hive uploading script every
> day. I have a set of folders created for a set of customer ids everyday. My
> hive script will read the customer id from the path, checks whether the
> table for the customer id exits and if not create a table and will create
> partition based on date for a "set of unknown customer_ids".
>
> I can get the set of unique customer_ids from a shell action. It can be
> passed as a list or string.
>
> My problem is how can I achieve this dynamic checking/creation of hive
> tables and partitions from oozie.
>
> Currently I am doing everything from the shell script and calling it as a
> shell action in oozie, but I was wondering if these kind of checks or some
> kind of for-loop kind of actions can be done in oozie.
>
> Any thoughts/suggestions on how to tackle the above scenario in the best
> way possible using oozie, would be highly helpful.
>
> Regards
> Praveenesh
>
>
Re: Oozie dynamic action
Posted by Peyman Mohajerian <mo...@gmail.com>.
If you want to see a simple example of what you are looking for:
https://github.com/cloudera/cdh-twitter-example
It is part of this article:
http://blog.cloudera.com/blog/2012/09/analyzing-twitter-data-with-hadoop/
On Tue, Sep 17, 2013 at 4:20 AM, praveenesh kumar <pr...@gmail.com>wrote:
> Hi,
>
> I have a scenario in which I want to trigger a hive uploading script every
> day. I have a set of folders created for a set of customer ids everyday. My
> hive script will read the customer id from the path, checks whether the
> table for the customer id exits and if not create a table and will create
> partition based on date for a "set of unknown customer_ids".
>
> I can get the set of unique customer_ids from a shell action. It can be
> passed as a list or string.
>
> My problem is how can I achieve this dynamic checking/creation of hive
> tables and partitions from oozie.
>
> Currently I am doing everything from the shell script and calling it as a
> shell action in oozie, but I was wondering if these kind of checks or some
> kind of for-loop kind of actions can be done in oozie.
>
> Any thoughts/suggestions on how to tackle the above scenario in the best
> way possible using oozie, would be highly helpful.
>
> Regards
> Praveenesh
>
>
Re: Oozie dynamic action
Posted by Peyman Mohajerian <mo...@gmail.com>.
If you want to see a simple example of what you are looking for:
https://github.com/cloudera/cdh-twitter-example
It is part of this article:
http://blog.cloudera.com/blog/2012/09/analyzing-twitter-data-with-hadoop/
On Tue, Sep 17, 2013 at 4:20 AM, praveenesh kumar <pr...@gmail.com>wrote:
> Hi,
>
> I have a scenario in which I want to trigger a hive uploading script every
> day. I have a set of folders created for a set of customer ids everyday. My
> hive script will read the customer id from the path, checks whether the
> table for the customer id exits and if not create a table and will create
> partition based on date for a "set of unknown customer_ids".
>
> I can get the set of unique customer_ids from a shell action. It can be
> passed as a list or string.
>
> My problem is how can I achieve this dynamic checking/creation of hive
> tables and partitions from oozie.
>
> Currently I am doing everything from the shell script and calling it as a
> shell action in oozie, but I was wondering if these kind of checks or some
> kind of for-loop kind of actions can be done in oozie.
>
> Any thoughts/suggestions on how to tackle the above scenario in the best
> way possible using oozie, would be highly helpful.
>
> Regards
> Praveenesh
>
>