Re: [Oozie-users] Re: periodic execution

[Carl Steinbach <ca...@cloudera.com>]

Hi Balaji,

Just wanted to add that you should put your Hive configuration settings in
hive-site.xml. hive-default.xml is intended for default property values and
should not be modified by anyone other than Hive developers. Additionally,
it's likely that hive-default.xml will be deprecated in a future version of
Hive, in which case any changes you have made to hive-default.xml will be
ignored.

Thanks.

Carl

On Thu, Feb 10, 2011 at 12:51 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

>
>
> Hi Balaji,
>
> The latest patch of the Hive action does not bundle hive-default.xml (got
> same feedback from Carl), you'll be responsible for bundling it in the WF
> directory until Hive JARs bundles it.
>
> I'll upload the new patch early next week and then ask Oozie it integrate
> it.
>
> Still the problem I have is that, AFAIK, not all Hadoop and Hive JARs are
> available in public Maven repositories currently used by Oozie build. I'll
> submit as part o the PR a separate commit that configures Oozie build to
> pull for Cloudera's Maven repositories where all JARs are available.
>
> Thanks.
>
> Alejandro
>
> On Thu, Feb 10, 2011 at 4:34 PM, Balaji Rajagopalan <
> balajirg@yahoo-inc.com> wrote:
>
>> Alejandro,
>>
>>    I have used your hive action patch from tucu’s forked branch in yahoo
>> github and it works fine, when will your patch be  available in the master
>> branch of yahoo github.  Also I have a small suggestion if I may,
>> hive-default.xml is bundled with the oozie-core.jar, instead can we have the
>> hive-default.xml is the same folder of workflow.xml in the hdfs, so when I
>> change the hive-default.xml I don’t have to bundle the jar again.
>>
>>
>>
>> Regards,
>>
>> Balaji
>>
>>
>>
>> *From:* Alejandro Abdelnur [mailto:tucu@cloudera.com]
>> *Sent:* Thursday, February 10, 2011 3:12 AM
>> *To:* user@hive.apache.org
>> *Subject:* Re: periodic execution
>>
>>
>>
>> Hi Cam,
>>
>>
>>
>> A bit of information that may be useful for you, Cloudera's Oozie has a
>> Hive action that you can use from workflow jobs.
>>
>>
>>
>> Cheers
>>
>>
>>
>> Alejandro
>>
>>
>>
>> On Wed, Feb 9, 2011 at 11:44 AM, Cam Bazz <ca...@gmail.com> wrote:
>>
>> Hello,
>>
>> I am looking over oozie's coordinator. But meanwhile, I managed to
>> write a simple java program to connect to hive using jdbc.
>>
>> I can import data and execute queries.
>>
>> I was wondering, somewhat for doing workflows, one needs to keep
>> metadata, i.e. which was the last file, partition processed etc.
>>
>> I could do this usually using a database like db4o, and keeping a static
>> file.
>>
>> Is the derby database that comes with hive is for this purpose? how do
>> people usually store state when using a hive application?
>>
>> best regards,
>> -C.B.
>>
>>
>> On Wed, Feb 9, 2011 at 5:23 AM, Jeff Hammerbacher <ha...@cloudera.com>
>> wrote:
>> > Hey Cam,
>> > You should use Oozie's
>> > Coordinator: https://github.com/yahoo/oozie/wiki/Oozie-Coord-Use-Cases.
>> > Regards,
>> > Jeff
>> >
>> > On Tue, Feb 8, 2011 at 4:29 PM, Cam Bazz <ca...@gmail.com> wrote:
>> >>
>> >> Hello,
>> >>
>> >> What kind of strategy must i follow, in order to periodically run
>> >> certain things.
>> >>
>> >> For example, each hour, i want to look up log files from certain dir,
>> >> and for new files, i need to run:
>> >>
>> >> load data local inpath '/home/cam/logs/log.2011310120' into table
>> >> item_view_raw partition (date_hour=2011310120);
>> >>
>> >> FROM item_view_raw ivr INSERT OVERWRITE TABLE item_view partition
>> >> (date_hour=2011310120) SELECT ivr.view_time, ivr.ip_number,
>> >> ivr.session_id, ivr.session_cookie, ivr.eser_sid, ivr.sale_status,
>> >> ivr.maker_name, ivr.title WHERE ivr.log_tag = 'PROD' and
>> >> ivr.date_hour='2011310120';
>> >>
>> >> obviously, i need to deduce which files are new, iterate over them,
>> >> and extract the time key, which will be used as a partition name, in
>> >> this case is: 2011310120
>> >>
>> >> It seems like i can write a java program to deal with the
>> >> syncronization of all these tasks, but i was wondering, what would you
>> >> guys suggest?
>> >>
>> >> Any ideas/recomendations/help greatly appreciated
>> >>
>> >> Best Regards,
>> >> C.B.
>> >
>> >
>>
>>
>>
>
>  __._,_.___
>   Reply to sender <tucu@cloudera.com?subject=Re%3A%20periodic%20execution>| Reply
> to group<Oozie-users@yahoogroups.com?subject=Re%3A%20periodic%20execution>| Reply
> via web post<http://groups.yahoo.com/group/Oozie-users/post;_ylc=X3oDMTJwYjE4bDRqBF9TAzk3MzU5NzE0BGdycElkAzU5MTgxODQzBGdycHNwSWQDMTcwNzI4MTk0MgRtc2dJZAM4ODEEc2VjA2Z0cgRzbGsDcnBseQRzdGltZQMxMjk3MzI3OTUz?act=reply&messageNum=881>| Start
> a New Topic<http://groups.yahoo.com/group/Oozie-users/post;_ylc=X3oDMTJmc29hMHVzBF9TAzk3MzU5NzE0BGdycElkAzU5MTgxODQzBGdycHNwSWQDMTcwNzI4MTk0MgRzZWMDZnRyBHNsawNudHBjBHN0aW1lAzEyOTczMjc5NTM->
> Messages in this topic<http://groups.yahoo.com/group/Oozie-users/message/881;_ylc=X3oDMTMzZXJsbDM2BF9TAzk3MzU5NzE0BGdycElkAzU5MTgxODQzBGdycHNwSWQDMTcwNzI4MTk0MgRtc2dJZAM4ODEEc2VjA2Z0cgRzbGsDdnRwYwRzdGltZQMxMjk3MzI3OTUzBHRwY0lkAzg4MQ-->(
> 1)
>  Recent Activity:
>
>    - New Members<http://groups.yahoo.com/group/Oozie-users/members;_ylc=X3oDMTJndTBmMTBsBF9TAzk3MzU5NzE0BGdycElkAzU5MTgxODQzBGdycHNwSWQDMTcwNzI4MTk0MgRzZWMDdnRsBHNsawN2bWJycwRzdGltZQMxMjk3MzI3OTUy?o=6>
>    13
>
>  Visit Your Group<http://groups.yahoo.com/group/Oozie-users;_ylc=X3oDMTJmZTFxMGUxBF9TAzk3MzU5NzE0BGdycElkAzU5MTgxODQzBGdycHNwSWQDMTcwNzI4MTk0MgRzZWMDdnRsBHNsawN2Z2hwBHN0aW1lAzEyOTczMjc5NTI->
>  MARKETPLACE
>
> Stay on top of your group activity without leaving the page you're on - Get
> the Yahoo! Toolbar now.<http://global.ard.yahoo.com/SIG=15oprc7tj/M=493064.14543979.14365478.13298430/D=groups/S=1707281942:MKP1/Y=YAHOO/EXP=1297335153/L=19d9f45e-34f3-11e0-b9d4-bfd2d0de8b7a/B=OvC9AGKImnw-/J=1297327953660495/K=D9EbHgc6bb6ZCnHoGfWjbQ/A=6060255/R=0/SIG=1194m4keh/*http://us.toolbar.yahoo.com/?.cpdl=grpj>
>   ------------------------------
>
> Find useful articles and helpful tips on living with Fibromyalgia. Visit
> the Fibromyalgia Zone today!<http://global.ard.yahoo.com/SIG=15oh54mjn/M=493064.14543977.14365476.10835568/D=groups/S=1707281942:MKP1/Y=YAHOO/EXP=1297335153/L=19d9f45e-34f3-11e0-b9d4-bfd2d0de8b7a/B=O_C9AGKImnw-/J=1297327953660495/K=D9EbHgc6bb6ZCnHoGfWjbQ/A=6260316/R=0/SIG=11bc8n8gt/*http://custom.yahoo.com/fibromyalgiazone/>
>   [image: Yahoo! Groups]<http://groups.yahoo.com/;_ylc=X3oDMTJlbzdqdWpiBF9TAzk3NDc2NTkwBGdycElkAzU5MTgxODQzBGdycHNwSWQDMTcwNzI4MTk0MgRzZWMDZnRyBHNsawNnZnAEc3RpbWUDMTI5NzMyNzk1Mw-->
> Switch to: Text-Only<Oozie-users-traditional@yahoogroups.com?subject=Change+Delivery+Format:+Traditional>,
> Daily Digest<Oozie-users-digest@yahoogroups.com?subject=Email+Delivery:+Digest>•
> Unsubscribe <Oozie-users-unsubscribe@yahoogroups.com?subject=Unsubscribe>• Terms
> of Use <http://docs.yahoo.com/info/terms/>
>    .
>
> __,_._,___
>