You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Manish Kothari <Ma...@vonage.com> on 2014/04/17 22:51:12 UTC
Oozie job with hive action completes successfully but does not
create table
Hi,
I have written an oozie job with hive action. It completes successfully but does not create a table in hive. When I execute the hive query independently it creates the table. Any help in this regard is appreciated.
Here is the oozie log file : -
2014-04-17 20:35:53,445 INFO ActionStartXCommand:539 - USER[root] GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf] JOB[0000016-140415213056450-oozie-oozi-W] ACTION[0000016-140415213056450-oozie-oozi-W@:start:] Start action [0000016-140415213056450-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2014-04-17 20:35:53,445 WARN ActionStartXCommand:542 - USER[root] GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf] JOB[0000016-140415213056450-oozie-oozi-W] ACTION[0000016-140415213056450-oozie-oozi-W@:start:] [***0000016-140415213056450-oozie-oozi-W@:start:***]Action status=DONE
2014-04-17 20:35:53,446 WARN ActionStartXCommand:542 - USER[root] GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf] JOB[0000016-140415213056450-oozie-oozi-W] ACTION[0000016-140415213056450-oozie-oozi-W@:start:] [***0000016-140415213056450-oozie-oozi-W@:start:***]Action updated in DB!
2014-04-17 20:35:53,601 INFO ActionStartXCommand:539 - USER[root] GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf] JOB[0000016-140415213056450-oozie-oozi-W] ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging] Start action [0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2014-04-17 20:35:54,619 WARN HiveActionExecutor:542 - USER[root] GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf] JOB[0000016-140415213056450-oozie-oozi-W] ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging] credentials is null for the action
2014-04-17 20:35:57,073 INFO HiveActionExecutor:539 - USER[root] GROUP[-] TOKEN[] APP[hive-gwcalls-insert-wf] JOB[0000016-140415213056450-oozie-oozi-W] ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging] checking action, external ID [job_1397592572894_0174] status [RUNNING]
2014-04-17 20:35:57,074 WARN ActionStartXCommand:542 - USER[root] GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf] JOB[0000016-140415213056450-oozie-oozi-W] ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging] [***0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging***]Action status=RUNNING
2014-04-17 20:35:57,075 WARN ActionStartXCommand:542 - USER[root] GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf] JOB[0000016-140415213056450-oozie-oozi-W] ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging] [***0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging***]Action updated in DB!
2014-04-17 20:36:29,057 INFO CallbackServlet:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000016-140415213056450-oozie-oozi-W] ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging] callback for action [0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
2014-04-17 20:36:29,509 INFO HiveActionExecutor:539 - USER[root] GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf] JOB[0000016-140415213056450-oozie-oozi-W] ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging] action completed, external ID [job_1397592572894_0174]
2014-04-17 20:36:29,542 INFO HiveActionExecutor:539 - USER[root] GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf] JOB[0000016-140415213056450-oozie-oozi-W] ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging] action produced output
2014-04-17 20:36:29,771 INFO ActionStartXCommand:539 - USER[root] GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf] JOB[0000016-140415213056450-oozie-oozi-W] ACTION[0000016-140415213056450-oozie-oozi-W@end] Start action [0000016-140415213056450-oozie-oozi-W@end] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2014-04-17 20:36:29,771 WARN ActionStartXCommand:542 - USER[root] GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf] JOB[0000016-140415213056450-oozie-oozi-W] ACTION[0000016-140415213056450-oozie-oozi-W@end] [***0000016-140415213056450-oozie-oozi-W@end***]Action status=DONE
2014-04-17 20:36:29,772 WARN ActionStartXCommand:542 - USER[root] GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf] JOB[0000016-140415213056450-oozie-oozi-W] ACTION[0000016-140415213056450-oozie-oozi-W@end] [***0000016-140415213056450-oozie-oozi-W@end***]Action updated in DB!
2014-04-17 20:36:29,887 WARN CoordActionUpdateXCommand:542 - USER[root] GROUP[-] TOKEN[] APP[hive-gwcalls-insert-wf] JOB[0000016-140415213056450-oozie-oozi-W] ACTION[-] E1100: Command precondition does not hold before execution, [, coord action is null], Error Code: E1100
Thanks,
Manish Kothari
Oracle Business Analyst
Vonage | 23 Main Street | Holmdel, NJ 07733-2136
t: +1.732.203.7613 | BB: +1.848.248.9615
e: manish.kothari@vonage.com<ma...@vonage.com>
w: www.vonage.com<http://www.vonage.com>
RE: Oozie job with hive action completes successfully but does not
create table
Posted by Manish Kothari <Ma...@vonage.com>.
Thanks Amit. That worked.
Thanks,
Manish Kothari
Oracle Business Analyst
Vonage | 23 Main Street | Holmdel, NJ 07733-2136
t: +1.732.203.7613 | BB: +1.848.248.9615
e: manish.kothari@vonage.com
w: www.vonage.com
-----Original Message-----
From: Amit Patil [mailto:arpatil@skyhighnetworks.com]
Sent: Thursday, April 17, 2014 5:40 PM
To: user@oozie.apache.org
Subject: Re: Oozie job with hive action completes successfully but does not create table
Try adding <job-xml>hive-site.xml</job-xml> to your hive action.
Also make sure that the metastore uri:port is reachable from all the machines on your hadoop cluster.
On Thu, Apr 17, 2014 at 2:31 PM, Manish Kothari
<Ma...@vonage.com>wrote:
> Thanks for responding. The hive metastore service is running and also
> the hive.metastore.uris is pointing to the remote metastore.
>
>
>
> Here are the necessary files : -
>
>
> Job property file
> nameNode=hdfs://hadoop-master-01.kewr0.m.vonagenetworks.net:8020/
> jobTracker=hadoop-master-02.kewr0.m.vonagenetworks.net:8050
>
> queueName=default
> oozie.use.system.libpath=true
>
> oozie.wf.application.path=${nameNode}user/nkgupta/oozie/workflow
>
> workflow Hive action:
>
> <action name="hive-gwcalls-staging">
> <hive xmlns="uri:oozie:hive-action:0.2">
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <configuration>
> <property>
> <name>mapred.compress.map.output</name>
> <value>true</value>
> </property>
> <property>
> <name>oozie.hive.defaults</name>
> <value>hive-site.xml</value>
> </property>
> </configuration>
> <script>create_gw_calls_staging.hive</script>
> </hive>
> <ok to="end"/>
> <error to="fail"/>
> </action>
>
> Create_gw_calls_staging.hive
>
> CREATE EXTERNAL TABLE default.gw_calls_staging_a (
> gateway varchar(20),
> type varchar(10),
> site varchar(10),
> call_type varchar(20),
> call_reference_id int,
> start_timestamp timestamp,
> disconnect_timestamp timestamp,
> disc_date date,
> disc_hour int,
> setup_to_connect float,
> record_type varchar(20),
> call_duration_secs float,
> first_release_source int,
> cause_code int,
> cause_location int,
> calling_number varchar(20),
> calling_number_prefix varchar(15),
> called_number varchar(20),
> called_number_prefix varchar(15),
> route_index varchar(1),
> route_name varchar(32),
> originating_line_info int,
> presentation_restriction int,
> incoming_calling_number varchar(20),
> t38_fax_call boolean,
> glare_encountered boolean,
> ingress_sip_call_id varchar(128),
> ingress_remote_sip_host varchar(64),
> ingress_trunk_group varchar(32),
> ingress_cic int,
> ingress_local_media_host varchar(64),
> ingress_local_media_port int,
> ingress_remote_media_host varchar(64),
> ingress_remote_media_port int,
> ingress_orig_point_code varchar(15),
> ingress_dest_point_code varchar(15),
> ingress_codec_type varchar(16),
> ingress_rtp_packetization int,
> ingress_num_bytes_sent int,
> ingress_num_packets_sent int,
> ingress_num_bytes_recv int,
> ingress_num_packets_recv int,
> ingress_num_packets_lost int,
> ingress_jitter int,
> ingress_latency int,
> ingress_max_packet_outage int,
> ingress_playout_quality varchar(40),
> ingress_echo_cancellation int,
> ingress_carrier varchar(32),
> egress_sip_call_id varchar(128),
> egress_remote_sip_host varchar(64),
> egress_trunk_group varchar(32),
> egress_cic int,
> egress_local_media_host varchar(64),
> egress_local_media_port int,
> egress_remote_media_host varchar(64),
> egress_remote_media_port int,
> egress_orig_point_code varchar(15),
> egress_dest_point_code varchar(15),
> egress_codec_type varchar(16),
> egress_rtp_packetization int,
> egress_num_bytes_sent int,
> egress_num_packets_sent int,
> egress_num_bytes_recv int,
> egress_num_packets_recv int,
> egress_num_packets_lost int,
> egress_jitter int,
> egress_latency int,
> egress_max_packet_outage int,
> egress_playout_quality varchar(40),
> egress_echo_cancellation int,
> egress_carrier varchar(32),
> session_id int,
> answer_filename varchar(64),
> release_filename varchar(64),
> inserted_timestamp timestamp,
> direction varchar(16),
> ingress_local_sip_host varchar(64),
> egress_local_sip_host varchar(64),
> calling_number_acct varchar(16),
> called_number_acct varchar(16),
> country_cd varchar(16),
> country varchar(64),
> lrn varchar(50),
> class_of_service varchar(32),
> route_source varchar(16),
> gateway_hostname varchar(255),
> global_call_id varchar(255),
> called_number_country_code varchar(15),
> called_number_country_iso varchar(2),
> called_number_state varchar(64),
> called_number_country_is_possible boolean,
> called_number_country_is_valid boolean,
> dnis int,
> rank int,
> sip_status_code_sent varchar(64),
> sip_status_code_received varchar(64),
> flow_type varchar(25),
> flow_originator varchar(255),
> flow_terminator varchar(255),
> transfer_id varchar(255),
> post_dial_delay int,
> calling_number_country_code varchar(15),
> calling_number_country_iso varchar(2),
> calling_number_state varchar(64),
> calling_number_country_is_possible boolean,
> calling_number_country_is_valid boolean,
> routing_type varchar(255),
> routing_prefix varchar(32)
>
> )
> COMMENT 'GW Calls Staging'
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '~' LINES TERMINATED BY '\n'
> STORED AS TEXTFILE LOCATION '/user/nkgupta/oozie/workflow/hiveinput';
>
> Hive-site.xml
>
> <!--Thu Apr 17 02:37:54 2014-->
> <configuration>
> <property>
> <name>hive.optimize.reducededuplication.min.reducer</name>
> <value>1</value>
> </property>
> <property>
> <name>javax.jdo.option.ConnectionURL</name>
> <value>jdbc:oracle:thin:@//
> db-gridcontrol-02.kewr0.s.vonagenetworks.net:1521/HADOOP_PROD</value>
> </property>
> <property>
> <name>hive.auto.convert.join.noconditionaltask</name>
> <value>true</value>
> </property>
> <property>
> <name>datanucleus.validateColumns</name>
> <value>false</value>
> </property>
> <property>
> <name>hive.optimize.index.filter</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.optimize.reducededuplication</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.mapjoin.bucket.cache.size</name>
> <value>10000</value>
> </property>
> <property>
> <name>hive.optimize.bucketmapjoin</name>
> <value>true</value>
> </property>
> <property>
> <name>javax.jdo.option.ConnectionUserName</name>
> <value>hive</value>
> </property>
> <property>
> <name>hive.enforce.bucketing</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.metastore.uris</name>
> <value>thrift://hadoop-master-02.kewr0.m.vonagenetworks.net:9083
> </value>
> </property>
> <property>
> <name>hive.semantic.analyzer.factory.impl</name>
> <value>org.apache.hivealog.cli.HCatSemanticAnalyzerFactory</value>
> </property>
> <property>
> <name>hive.auto.convert.join.noconditionaltask.size</name>
> <value>1000000000</value>
> </property>
> <property>
> <name>hive.optimize.bucketmapjoin.sortedmerge</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.map.aggr</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.auto.convert.sortmerge.join.noconditionaltask</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.auto.convert.join</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.auto.convert.sortmerge.join</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.metastore.client.socket.timeout</name>
> <value>60</value>
> </property>
> <property>
> <name>hive.optimize.mapjoin.mapreduce</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.security.authorization.enabled</name>
> <value>false</value>
> </property>
> <property>
> <name>javax.jdo.option.ConnectionDriverName</name>
> <value>oracle.jdbc.driver.OracleDriver</value>
> </property>
> <property>
> <name>hive.vectorized.execution.enabled</name>
> <value>false</value>
> </property>
> <property>
> <name>hive.enforce.sorting</name>
> <value>true</value>
> </property>
> <property>
> <name>fs.hdfs.impl.disable.cache</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.mapred.reduce.tasks.speculative.execution</name>
> <value>false</value>
> </property>
> <property>
> <name>fs.file.impl.disable.cache</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.security.authorization.manager</name>
>
> <value>org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider</value>
> </property>
> <property>
> <name>hive.metastore.execute.setugi</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.metastore.cache.pinobjtypes</name>
> <value>Table,Database,Type,FieldSchema,Order</value>
> </property>
> <property>
> <name>hive.metastore.warehouse.dir</name>
> <value>/apps/hive/warehouse</value>
> </property>
> <property>
> <name>hive.server2.enable.doAs</name>
> <value>true</value>
> </property>
> </configuration>
>
>
>
>
>
>
>
> Thanks,
>
> Manish Kothari
>
> Oracle Business Analyst
>
> Vonage | 23 Main Street | Holmdel, NJ 07733-2136
>
> t: +1.732.203.7613 | BB: +1.848.248.9615
>
> e: manish.kothari@vonage.com
>
> w: www.vonage.com
>
>
>
> -----Original Message-----
>
> From: Amit Patil [mailto:arpatil@skyhighnetworks.com]
>
> Sent: Thursday, April 17, 2014 4:56 PM
>
> To: user@oozie.apache.org
>
> Subject: Re: Oozie job with hive action completes successfully but
> does not create table
>
>
>
> Its because your oozie hive action is not connecting to the hive metastore.
>
> You need to start the hive metastore service. Refer
> https://cwiki.apache.org/confluence/display/Hive/AdminManual+Metastore
> Admin
>
> Once you start your remote metastore, make the required changes in the
> hive-site.xml which your oozie application is using. This includes
> changing the property hive.metastore.uris to point to the remote metastore.
>
> This should work.
>
>
>
>
>
> On Thu, Apr 17, 2014 at 1:51 PM, Manish Kothari
>
> <Ma...@vonage.com>wrote:
>
>
>
> > Hi,
>
> >
>
> > I have written an oozie job with hive action. It completes
>
> > successfully but does not create a table in hive. When I execute the
>
> > hive query independently it creates the table. Any help in this
> > regard
> is appreciated.
>
> >
>
> > Here is the oozie log file : -
>
> >
>
> > 2014-04-17 20:35:53,445 INFO ActionStartXCommand:539 - USER[root]
>
> > GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
>
> > JOB[0000016-140415213056450-oozie-oozi-W]
>
> > ACTION[0000016-140415213056450-oozie-oozi-W@:start:] Start action
>
> > [0000016-140415213056450-oozie-oozi-W@:start:] with user-retry state :
>
> > userRetryCount [0], userRetryMax [0], userRetryInterval [10]
>
> > 2014-04-17 20:35:53,445 WARN ActionStartXCommand:542 - USER[root]
>
> > GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
>
> > JOB[0000016-140415213056450-oozie-oozi-W]
>
> > ACTION[0000016-140415213056450-oozie-oozi-W@:start:]
>
> > [***0000016-140415213056450-oozie-oozi-W@:start:***]Action
> > status=DONE
>
> > 2014-04-17 20:35:53,446 WARN ActionStartXCommand:542 - USER[root]
>
> > GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
>
> > JOB[0000016-140415213056450-oozie-oozi-W]
>
> > ACTION[0000016-140415213056450-oozie-oozi-W@:start:]
>
> > [***0000016-140415213056450-oozie-oozi-W@:start:***]Action updated
> > in
> DB!
>
> > 2014-04-17 20:35:53,601 INFO ActionStartXCommand:539 - USER[root]
>
> > GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
>
> > JOB[0000016-140415213056450-oozie-oozi-W]
>
> > ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
>
> > Start action
>
> > [0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging] with
>
> > user-retry state : userRetryCount [0], userRetryMax [0],
>
> > userRetryInterval [10]
>
> > 2014-04-17 20:35:54,619 WARN HiveActionExecutor:542 - USER[root]
>
> > GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
>
> > JOB[0000016-140415213056450-oozie-oozi-W]
>
> > ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
>
> > credentials is null for the action
>
> > 2014-04-17 20:35:57,073 INFO HiveActionExecutor:539 - USER[root]
>
> > GROUP[-] TOKEN[] APP[hive-gwcalls-insert-wf]
>
> > JOB[0000016-140415213056450-oozie-oozi-W]
>
> > ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
>
> > checking action, external ID [job_1397592572894_0174] status
> > [RUNNING]
>
> > 2014-04-17 20:35:57,074 WARN ActionStartXCommand:542 - USER[root]
>
> > GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
>
> > JOB[0000016-140415213056450-oozie-oozi-W]
>
> > ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
>
> > [***0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging***]Act
> > io
>
> > n
>
> > status=RUNNING
>
> > 2014-04-17 20:35:57,075 WARN ActionStartXCommand:542 - USER[root]
>
> > GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
>
> > JOB[0000016-140415213056450-oozie-oozi-W]
>
> > ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
>
> > [***0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging***]Act
> > io
>
> > n
>
> > updated in DB!
>
> > 2014-04-17 20:36:29,057 INFO CallbackServlet:539 - USER[-] GROUP[-]
>
> > TOKEN[-] APP[-] JOB[0000016-140415213056450-oozie-oozi-W]
>
> > ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
>
> > callback for action
>
> > [0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
>
> > 2014-04-17 20:36:29,509 INFO HiveActionExecutor:539 - USER[root]
>
> > GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
>
> > JOB[0000016-140415213056450-oozie-oozi-W]
>
> > ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
>
> > action completed, external ID [job_1397592572894_0174]
>
> > 2014-04-17 20:36:29,542 INFO HiveActionExecutor:539 - USER[root]
>
> > GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
>
> > JOB[0000016-140415213056450-oozie-oozi-W]
>
> > ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
>
> > action produced output
>
> > 2014-04-17 20:36:29,771 INFO ActionStartXCommand:539 - USER[root]
>
> > GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
>
> > JOB[0000016-140415213056450-oozie-oozi-W]
>
> > ACTION[0000016-140415213056450-oozie-oozi-W@end] Start action
>
> > [0000016-140415213056450-oozie-oozi-W@end] with user-retry state :
>
> > userRetryCount [0], userRetryMax [0], userRetryInterval [10]
>
> > 2014-04-17 20:36:29,771 WARN ActionStartXCommand:542 - USER[root]
>
> > GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
>
> > JOB[0000016-140415213056450-oozie-oozi-W]
>
> > ACTION[0000016-140415213056450-oozie-oozi-W@end]
>
> > [***0000016-140415213056450-oozie-oozi-W@end***]Action status=DONE
>
> > 2014-04-17 20:36:29,772 WARN ActionStartXCommand:542 - USER[root]
>
> > GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
>
> > JOB[0000016-140415213056450-oozie-oozi-W]
>
> > ACTION[0000016-140415213056450-oozie-oozi-W@end]
>
> > [***0000016-140415213056450-oozie-oozi-W@end***]Action updated in DB!
>
> > 2014-04-17 20:36:29,887 WARN CoordActionUpdateXCommand:542 -
>
> > USER[root] GROUP[-] TOKEN[] APP[hive-gwcalls-insert-wf]
>
> > JOB[0000016-140415213056450-oozie-oozi-W] ACTION[-] E1100: Command
>
> > precondition does not hold before execution, [, coord action is
> > null],
>
> > Error Code: E1100
>
> >
>
> > Thanks,
>
> > Manish Kothari
>
> > Oracle Business Analyst
>
> > Vonage | 23 Main Street | Holmdel, NJ 07733-2136
>
> > t: +1.732.203.7613 | BB: +1.848.248.9615
>
> > e: manish.kothari@vonage.com<ma...@vonage.com>
>
> > w: www.vonage.com<http://www.vonage.com>
>
> >
>
> >
>
Re: Oozie job with hive action completes successfully but does not
create table
Posted by Amit Patil <ar...@skyhighnetworks.com>.
Try adding <job-xml>hive-site.xml</job-xml> to your hive action.
Also make sure that the metastore uri:port is reachable from all the
machines on your hadoop cluster.
On Thu, Apr 17, 2014 at 2:31 PM, Manish Kothari
<Ma...@vonage.com>wrote:
> Thanks for responding. The hive metastore service is running and also the
> hive.metastore.uris is pointing to the remote metastore.
>
>
>
> Here are the necessary files : -
>
>
> Job property file
> nameNode=hdfs://hadoop-master-01.kewr0.m.vonagenetworks.net:8020/
> jobTracker=hadoop-master-02.kewr0.m.vonagenetworks.net:8050
>
> queueName=default
> oozie.use.system.libpath=true
>
> oozie.wf.application.path=${nameNode}user/nkgupta/oozie/workflow
>
> workflow Hive action:
>
> <action name="hive-gwcalls-staging">
> <hive xmlns="uri:oozie:hive-action:0.2">
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <configuration>
> <property>
> <name>mapred.compress.map.output</name>
> <value>true</value>
> </property>
> <property>
> <name>oozie.hive.defaults</name>
> <value>hive-site.xml</value>
> </property>
> </configuration>
> <script>create_gw_calls_staging.hive</script>
> </hive>
> <ok to="end"/>
> <error to="fail"/>
> </action>
>
> Create_gw_calls_staging.hive
>
> CREATE EXTERNAL TABLE default.gw_calls_staging_a
> (
> gateway varchar(20),
> type varchar(10),
> site varchar(10),
> call_type varchar(20),
> call_reference_id int,
> start_timestamp timestamp,
> disconnect_timestamp timestamp,
> disc_date date,
> disc_hour int,
> setup_to_connect float,
> record_type varchar(20),
> call_duration_secs float,
> first_release_source int,
> cause_code int,
> cause_location int,
> calling_number varchar(20),
> calling_number_prefix varchar(15),
> called_number varchar(20),
> called_number_prefix varchar(15),
> route_index varchar(1),
> route_name varchar(32),
> originating_line_info int,
> presentation_restriction int,
> incoming_calling_number varchar(20),
> t38_fax_call boolean,
> glare_encountered boolean,
> ingress_sip_call_id varchar(128),
> ingress_remote_sip_host varchar(64),
> ingress_trunk_group varchar(32),
> ingress_cic int,
> ingress_local_media_host varchar(64),
> ingress_local_media_port int,
> ingress_remote_media_host varchar(64),
> ingress_remote_media_port int,
> ingress_orig_point_code varchar(15),
> ingress_dest_point_code varchar(15),
> ingress_codec_type varchar(16),
> ingress_rtp_packetization int,
> ingress_num_bytes_sent int,
> ingress_num_packets_sent int,
> ingress_num_bytes_recv int,
> ingress_num_packets_recv int,
> ingress_num_packets_lost int,
> ingress_jitter int,
> ingress_latency int,
> ingress_max_packet_outage int,
> ingress_playout_quality varchar(40),
> ingress_echo_cancellation int,
> ingress_carrier varchar(32),
> egress_sip_call_id varchar(128),
> egress_remote_sip_host varchar(64),
> egress_trunk_group varchar(32),
> egress_cic int,
> egress_local_media_host varchar(64),
> egress_local_media_port int,
> egress_remote_media_host varchar(64),
> egress_remote_media_port int,
> egress_orig_point_code varchar(15),
> egress_dest_point_code varchar(15),
> egress_codec_type varchar(16),
> egress_rtp_packetization int,
> egress_num_bytes_sent int,
> egress_num_packets_sent int,
> egress_num_bytes_recv int,
> egress_num_packets_recv int,
> egress_num_packets_lost int,
> egress_jitter int,
> egress_latency int,
> egress_max_packet_outage int,
> egress_playout_quality varchar(40),
> egress_echo_cancellation int,
> egress_carrier varchar(32),
> session_id int,
> answer_filename varchar(64),
> release_filename varchar(64),
> inserted_timestamp timestamp,
> direction varchar(16),
> ingress_local_sip_host varchar(64),
> egress_local_sip_host varchar(64),
> calling_number_acct varchar(16),
> called_number_acct varchar(16),
> country_cd varchar(16),
> country varchar(64),
> lrn varchar(50),
> class_of_service varchar(32),
> route_source varchar(16),
> gateway_hostname varchar(255),
> global_call_id varchar(255),
> called_number_country_code varchar(15),
> called_number_country_iso varchar(2),
> called_number_state varchar(64),
> called_number_country_is_possible boolean,
> called_number_country_is_valid boolean,
> dnis int,
> rank int,
> sip_status_code_sent varchar(64),
> sip_status_code_received varchar(64),
> flow_type varchar(25),
> flow_originator varchar(255),
> flow_terminator varchar(255),
> transfer_id varchar(255),
> post_dial_delay int,
> calling_number_country_code varchar(15),
> calling_number_country_iso varchar(2),
> calling_number_state varchar(64),
> calling_number_country_is_possible boolean,
> calling_number_country_is_valid boolean,
> routing_type varchar(255),
> routing_prefix varchar(32)
>
> )
> COMMENT 'GW Calls Staging'
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '~' LINES TERMINATED BY '\n'
> STORED AS TEXTFILE LOCATION '/user/nkgupta/oozie/workflow/hiveinput';
>
> Hive-site.xml
>
> <!--Thu Apr 17 02:37:54 2014-->
> <configuration>
> <property>
> <name>hive.optimize.reducededuplication.min.reducer</name>
> <value>1</value>
> </property>
> <property>
> <name>javax.jdo.option.ConnectionURL</name>
> <value>jdbc:oracle:thin:@//
> db-gridcontrol-02.kewr0.s.vonagenetworks.net:1521/HADOOP_PROD</value>
> </property>
> <property>
> <name>hive.auto.convert.join.noconditionaltask</name>
> <value>true</value>
> </property>
> <property>
> <name>datanucleus.validateColumns</name>
> <value>false</value>
> </property>
> <property>
> <name>hive.optimize.index.filter</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.optimize.reducededuplication</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.mapjoin.bucket.cache.size</name>
> <value>10000</value>
> </property>
> <property>
> <name>hive.optimize.bucketmapjoin</name>
> <value>true</value>
> </property>
> <property>
> <name>javax.jdo.option.ConnectionUserName</name>
> <value>hive</value>
> </property>
> <property>
> <name>hive.enforce.bucketing</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.metastore.uris</name>
> <value>thrift://hadoop-master-02.kewr0.m.vonagenetworks.net:9083
> </value>
> </property>
> <property>
> <name>hive.semantic.analyzer.factory.impl</name>
> <value>org.apache.hivealog.cli.HCatSemanticAnalyzerFactory</value>
> </property>
> <property>
> <name>hive.auto.convert.join.noconditionaltask.size</name>
> <value>1000000000</value>
> </property>
> <property>
> <name>hive.optimize.bucketmapjoin.sortedmerge</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.map.aggr</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.auto.convert.sortmerge.join.noconditionaltask</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.auto.convert.join</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.auto.convert.sortmerge.join</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.metastore.client.socket.timeout</name>
> <value>60</value>
> </property>
> <property>
> <name>hive.optimize.mapjoin.mapreduce</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.security.authorization.enabled</name>
> <value>false</value>
> </property>
> <property>
> <name>javax.jdo.option.ConnectionDriverName</name>
> <value>oracle.jdbc.driver.OracleDriver</value>
> </property>
> <property>
> <name>hive.vectorized.execution.enabled</name>
> <value>false</value>
> </property>
> <property>
> <name>hive.enforce.sorting</name>
> <value>true</value>
> </property>
> <property>
> <name>fs.hdfs.impl.disable.cache</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.mapred.reduce.tasks.speculative.execution</name>
> <value>false</value>
> </property>
> <property>
> <name>fs.file.impl.disable.cache</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.security.authorization.manager</name>
>
> <value>org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider</value>
> </property>
> <property>
> <name>hive.metastore.execute.setugi</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.metastore.cache.pinobjtypes</name>
> <value>Table,Database,Type,FieldSchema,Order</value>
> </property>
> <property>
> <name>hive.metastore.warehouse.dir</name>
> <value>/apps/hive/warehouse</value>
> </property>
> <property>
> <name>hive.server2.enable.doAs</name>
> <value>true</value>
> </property>
> </configuration>
>
>
>
>
>
>
>
> Thanks,
>
> Manish Kothari
>
> Oracle Business Analyst
>
> Vonage | 23 Main Street | Holmdel, NJ 07733-2136
>
> t: +1.732.203.7613 | BB: +1.848.248.9615
>
> e: manish.kothari@vonage.com
>
> w: www.vonage.com
>
>
>
> -----Original Message-----
>
> From: Amit Patil [mailto:arpatil@skyhighnetworks.com]
>
> Sent: Thursday, April 17, 2014 4:56 PM
>
> To: user@oozie.apache.org
>
> Subject: Re: Oozie job with hive action completes successfully but does
> not create table
>
>
>
> Its because your oozie hive action is not connecting to the hive metastore.
>
> You need to start the hive metastore service. Refer
> https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin
>
> Once you start your remote metastore, make the required changes in the
> hive-site.xml which your oozie application is using. This includes changing
> the property hive.metastore.uris to point to the remote metastore.
>
> This should work.
>
>
>
>
>
> On Thu, Apr 17, 2014 at 1:51 PM, Manish Kothari
>
> <Ma...@vonage.com>wrote:
>
>
>
> > Hi,
>
> >
>
> > I have written an oozie job with hive action. It completes
>
> > successfully but does not create a table in hive. When I execute the
>
> > hive query independently it creates the table. Any help in this regard
> is appreciated.
>
> >
>
> > Here is the oozie log file : -
>
> >
>
> > 2014-04-17 20:35:53,445 INFO ActionStartXCommand:539 - USER[root]
>
> > GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
>
> > JOB[0000016-140415213056450-oozie-oozi-W]
>
> > ACTION[0000016-140415213056450-oozie-oozi-W@:start:] Start action
>
> > [0000016-140415213056450-oozie-oozi-W@:start:] with user-retry state :
>
> > userRetryCount [0], userRetryMax [0], userRetryInterval [10]
>
> > 2014-04-17 20:35:53,445 WARN ActionStartXCommand:542 - USER[root]
>
> > GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
>
> > JOB[0000016-140415213056450-oozie-oozi-W]
>
> > ACTION[0000016-140415213056450-oozie-oozi-W@:start:]
>
> > [***0000016-140415213056450-oozie-oozi-W@:start:***]Action status=DONE
>
> > 2014-04-17 20:35:53,446 WARN ActionStartXCommand:542 - USER[root]
>
> > GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
>
> > JOB[0000016-140415213056450-oozie-oozi-W]
>
> > ACTION[0000016-140415213056450-oozie-oozi-W@:start:]
>
> > [***0000016-140415213056450-oozie-oozi-W@:start:***]Action updated in
> DB!
>
> > 2014-04-17 20:35:53,601 INFO ActionStartXCommand:539 - USER[root]
>
> > GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
>
> > JOB[0000016-140415213056450-oozie-oozi-W]
>
> > ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
>
> > Start action
>
> > [0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging] with
>
> > user-retry state : userRetryCount [0], userRetryMax [0],
>
> > userRetryInterval [10]
>
> > 2014-04-17 20:35:54,619 WARN HiveActionExecutor:542 - USER[root]
>
> > GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
>
> > JOB[0000016-140415213056450-oozie-oozi-W]
>
> > ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
>
> > credentials is null for the action
>
> > 2014-04-17 20:35:57,073 INFO HiveActionExecutor:539 - USER[root]
>
> > GROUP[-] TOKEN[] APP[hive-gwcalls-insert-wf]
>
> > JOB[0000016-140415213056450-oozie-oozi-W]
>
> > ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
>
> > checking action, external ID [job_1397592572894_0174] status [RUNNING]
>
> > 2014-04-17 20:35:57,074 WARN ActionStartXCommand:542 - USER[root]
>
> > GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
>
> > JOB[0000016-140415213056450-oozie-oozi-W]
>
> > ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
>
> > [***0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging***]Actio
>
> > n
>
> > status=RUNNING
>
> > 2014-04-17 20:35:57,075 WARN ActionStartXCommand:542 - USER[root]
>
> > GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
>
> > JOB[0000016-140415213056450-oozie-oozi-W]
>
> > ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
>
> > [***0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging***]Actio
>
> > n
>
> > updated in DB!
>
> > 2014-04-17 20:36:29,057 INFO CallbackServlet:539 - USER[-] GROUP[-]
>
> > TOKEN[-] APP[-] JOB[0000016-140415213056450-oozie-oozi-W]
>
> > ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
>
> > callback for action
>
> > [0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
>
> > 2014-04-17 20:36:29,509 INFO HiveActionExecutor:539 - USER[root]
>
> > GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
>
> > JOB[0000016-140415213056450-oozie-oozi-W]
>
> > ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
>
> > action completed, external ID [job_1397592572894_0174]
>
> > 2014-04-17 20:36:29,542 INFO HiveActionExecutor:539 - USER[root]
>
> > GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
>
> > JOB[0000016-140415213056450-oozie-oozi-W]
>
> > ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
>
> > action produced output
>
> > 2014-04-17 20:36:29,771 INFO ActionStartXCommand:539 - USER[root]
>
> > GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
>
> > JOB[0000016-140415213056450-oozie-oozi-W]
>
> > ACTION[0000016-140415213056450-oozie-oozi-W@end] Start action
>
> > [0000016-140415213056450-oozie-oozi-W@end] with user-retry state :
>
> > userRetryCount [0], userRetryMax [0], userRetryInterval [10]
>
> > 2014-04-17 20:36:29,771 WARN ActionStartXCommand:542 - USER[root]
>
> > GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
>
> > JOB[0000016-140415213056450-oozie-oozi-W]
>
> > ACTION[0000016-140415213056450-oozie-oozi-W@end]
>
> > [***0000016-140415213056450-oozie-oozi-W@end***]Action status=DONE
>
> > 2014-04-17 20:36:29,772 WARN ActionStartXCommand:542 - USER[root]
>
> > GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
>
> > JOB[0000016-140415213056450-oozie-oozi-W]
>
> > ACTION[0000016-140415213056450-oozie-oozi-W@end]
>
> > [***0000016-140415213056450-oozie-oozi-W@end***]Action updated in DB!
>
> > 2014-04-17 20:36:29,887 WARN CoordActionUpdateXCommand:542 -
>
> > USER[root] GROUP[-] TOKEN[] APP[hive-gwcalls-insert-wf]
>
> > JOB[0000016-140415213056450-oozie-oozi-W] ACTION[-] E1100: Command
>
> > precondition does not hold before execution, [, coord action is null],
>
> > Error Code: E1100
>
> >
>
> > Thanks,
>
> > Manish Kothari
>
> > Oracle Business Analyst
>
> > Vonage | 23 Main Street | Holmdel, NJ 07733-2136
>
> > t: +1.732.203.7613 | BB: +1.848.248.9615
>
> > e: manish.kothari@vonage.com<ma...@vonage.com>
>
> > w: www.vonage.com<http://www.vonage.com>
>
> >
>
> >
>
RE: Oozie job with hive action completes successfully but does not
create table
Posted by Manish Kothari <Ma...@vonage.com>.
Thanks for responding. The hive metastore service is running and also the hive.metastore.uris is pointing to the remote metastore.
Here are the necessary files : -
Job property file
nameNode=hdfs://hadoop-master-01.kewr0.m.vonagenetworks.net:8020/
jobTracker=hadoop-master-02.kewr0.m.vonagenetworks.net:8050
queueName=default
oozie.use.system.libpath=true
oozie.wf.application.path=${nameNode}user/nkgupta/oozie/workflow
workflow Hive action:
<action name="hive-gwcalls-staging">
<hive xmlns="uri:oozie:hive-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.compress.map.output</name>
<value>true</value>
</property>
<property>
<name>oozie.hive.defaults</name>
<value>hive-site.xml</value>
</property>
</configuration>
<script>create_gw_calls_staging.hive</script>
</hive>
<ok to="end"/>
<error to="fail"/>
</action>
Create_gw_calls_staging.hive
CREATE EXTERNAL TABLE default.gw_calls_staging_a
(
gateway varchar(20),
type varchar(10),
site varchar(10),
call_type varchar(20),
call_reference_id int,
start_timestamp timestamp,
disconnect_timestamp timestamp,
disc_date date,
disc_hour int,
setup_to_connect float,
record_type varchar(20),
call_duration_secs float,
first_release_source int,
cause_code int,
cause_location int,
calling_number varchar(20),
calling_number_prefix varchar(15),
called_number varchar(20),
called_number_prefix varchar(15),
route_index varchar(1),
route_name varchar(32),
originating_line_info int,
presentation_restriction int,
incoming_calling_number varchar(20),
t38_fax_call boolean,
glare_encountered boolean,
ingress_sip_call_id varchar(128),
ingress_remote_sip_host varchar(64),
ingress_trunk_group varchar(32),
ingress_cic int,
ingress_local_media_host varchar(64),
ingress_local_media_port int,
ingress_remote_media_host varchar(64),
ingress_remote_media_port int,
ingress_orig_point_code varchar(15),
ingress_dest_point_code varchar(15),
ingress_codec_type varchar(16),
ingress_rtp_packetization int,
ingress_num_bytes_sent int,
ingress_num_packets_sent int,
ingress_num_bytes_recv int,
ingress_num_packets_recv int,
ingress_num_packets_lost int,
ingress_jitter int,
ingress_latency int,
ingress_max_packet_outage int,
ingress_playout_quality varchar(40),
ingress_echo_cancellation int,
ingress_carrier varchar(32),
egress_sip_call_id varchar(128),
egress_remote_sip_host varchar(64),
egress_trunk_group varchar(32),
egress_cic int,
egress_local_media_host varchar(64),
egress_local_media_port int,
egress_remote_media_host varchar(64),
egress_remote_media_port int,
egress_orig_point_code varchar(15),
egress_dest_point_code varchar(15),
egress_codec_type varchar(16),
egress_rtp_packetization int,
egress_num_bytes_sent int,
egress_num_packets_sent int,
egress_num_bytes_recv int,
egress_num_packets_recv int,
egress_num_packets_lost int,
egress_jitter int,
egress_latency int,
egress_max_packet_outage int,
egress_playout_quality varchar(40),
egress_echo_cancellation int,
egress_carrier varchar(32),
session_id int,
answer_filename varchar(64),
release_filename varchar(64),
inserted_timestamp timestamp,
direction varchar(16),
ingress_local_sip_host varchar(64),
egress_local_sip_host varchar(64),
calling_number_acct varchar(16),
called_number_acct varchar(16),
country_cd varchar(16),
country varchar(64),
lrn varchar(50),
class_of_service varchar(32),
route_source varchar(16),
gateway_hostname varchar(255),
global_call_id varchar(255),
called_number_country_code varchar(15),
called_number_country_iso varchar(2),
called_number_state varchar(64),
called_number_country_is_possible boolean,
called_number_country_is_valid boolean,
dnis int,
rank int,
sip_status_code_sent varchar(64),
sip_status_code_received varchar(64),
flow_type varchar(25),
flow_originator varchar(255),
flow_terminator varchar(255),
transfer_id varchar(255),
post_dial_delay int,
calling_number_country_code varchar(15),
calling_number_country_iso varchar(2),
calling_number_state varchar(64),
calling_number_country_is_possible boolean,
calling_number_country_is_valid boolean,
routing_type varchar(255),
routing_prefix varchar(32)
)
COMMENT 'GW Calls Staging'
ROW FORMAT DELIMITED FIELDS TERMINATED BY '~' LINES TERMINATED BY '\n'
STORED AS TEXTFILE LOCATION '/user/nkgupta/oozie/workflow/hiveinput';
Hive-site.xml
<!--Thu Apr 17 02:37:54 2014-->
<configuration>
<property>
<name>hive.optimize.reducededuplication.min.reducer</name>
<value>1</value>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:oracle:thin:@//db-gridcontrol-02.kewr0.s.vonagenetworks.net:1521/HADOOP_PROD</value>
</property>
<property>
<name>hive.auto.convert.join.noconditionaltask</name>
<value>true</value>
</property>
<property>
<name>datanucleus.validateColumns</name>
<value>false</value>
</property>
<property>
<name>hive.optimize.index.filter</name>
<value>true</value>
</property>
<property>
<name>hive.optimize.reducededuplication</name>
<value>true</value>
</property>
<property>
<name>hive.mapjoin.bucket.cache.size</name>
<value>10000</value>
</property>
<property>
<name>hive.optimize.bucketmapjoin</name>
<value>true</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive</value>
</property>
<property>
<name>hive.enforce.bucketing</name>
<value>true</value>
</property>
<property>
<name>hive.metastore.uris</name>
<value>thrift://hadoop-master-02.kewr0.m.vonagenetworks.net:9083</value>
</property>
<property>
<name>hive.semantic.analyzer.factory.impl</name>
<value>org.apache.hivealog.cli.HCatSemanticAnalyzerFactory</value>
</property>
<property>
<name>hive.auto.convert.join.noconditionaltask.size</name>
<value>1000000000</value>
</property>
<property>
<name>hive.optimize.bucketmapjoin.sortedmerge</name>
<value>true</value>
</property>
<property>
<name>hive.map.aggr</name>
<value>true</value>
</property>
<property>
<name>hive.auto.convert.sortmerge.join.noconditionaltask</name>
<value>true</value>
</property>
<property>
<name>hive.auto.convert.join</name>
<value>true</value>
</property>
<property>
<name>hive.auto.convert.sortmerge.join</name>
<value>true</value>
</property>
<property>
<name>hive.metastore.client.socket.timeout</name>
<value>60</value>
</property>
<property>
<name>hive.optimize.mapjoin.mapreduce</name>
<value>true</value>
</property>
<property>
<name>hive.security.authorization.enabled</name>
<value>false</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>oracle.jdbc.driver.OracleDriver</value>
</property>
<property>
<name>hive.vectorized.execution.enabled</name>
<value>false</value>
</property>
<property>
<name>hive.enforce.sorting</name>
<value>true</value>
</property>
<property>
<name>fs.hdfs.impl.disable.cache</name>
<value>true</value>
</property>
<property>
<name>hive.mapred.reduce.tasks.speculative.execution</name>
<value>false</value>
</property>
<property>
<name>fs.file.impl.disable.cache</name>
<value>true</value>
</property>
<property>
<name>hive.security.authorization.manager</name>
<value>org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider</value>
</property>
<property>
<name>hive.metastore.execute.setugi</name>
<value>true</value>
</property>
<property>
<name>hive.metastore.cache.pinobjtypes</name>
<value>Table,Database,Type,FieldSchema,Order</value>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/apps/hive/warehouse</value>
</property>
<property>
<name>hive.server2.enable.doAs</name>
<value>true</value>
</property>
</configuration>
Thanks,
Manish Kothari
Oracle Business Analyst
Vonage | 23 Main Street | Holmdel, NJ 07733-2136
t: +1.732.203.7613 | BB: +1.848.248.9615
e: manish.kothari@vonage.com
w: www.vonage.com
-----Original Message-----
From: Amit Patil [mailto:arpatil@skyhighnetworks.com]
Sent: Thursday, April 17, 2014 4:56 PM
To: user@oozie.apache.org
Subject: Re: Oozie job with hive action completes successfully but does not create table
Its because your oozie hive action is not connecting to the hive metastore.
You need to start the hive metastore service. Refer https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin
Once you start your remote metastore, make the required changes in the hive-site.xml which your oozie application is using. This includes changing the property hive.metastore.uris to point to the remote metastore.
This should work.
On Thu, Apr 17, 2014 at 1:51 PM, Manish Kothari
<Ma...@vonage.com>wrote:
> Hi,
>
> I have written an oozie job with hive action. It completes
> successfully but does not create a table in hive. When I execute the
> hive query independently it creates the table. Any help in this regard is appreciated.
>
> Here is the oozie log file : -
>
> 2014-04-17 20:35:53,445 INFO ActionStartXCommand:539 - USER[root]
> GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
> JOB[0000016-140415213056450-oozie-oozi-W]
> ACTION[0000016-140415213056450-oozie-oozi-W@:start:] Start action
> [0000016-140415213056450-oozie-oozi-W@:start:] with user-retry state :
> userRetryCount [0], userRetryMax [0], userRetryInterval [10]
> 2014-04-17 20:35:53,445 WARN ActionStartXCommand:542 - USER[root]
> GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
> JOB[0000016-140415213056450-oozie-oozi-W]
> ACTION[0000016-140415213056450-oozie-oozi-W@:start:]
> [***0000016-140415213056450-oozie-oozi-W@:start:***]Action status=DONE
> 2014-04-17 20:35:53,446 WARN ActionStartXCommand:542 - USER[root]
> GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
> JOB[0000016-140415213056450-oozie-oozi-W]
> ACTION[0000016-140415213056450-oozie-oozi-W@:start:]
> [***0000016-140415213056450-oozie-oozi-W@:start:***]Action updated in DB!
> 2014-04-17 20:35:53,601 INFO ActionStartXCommand:539 - USER[root]
> GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
> JOB[0000016-140415213056450-oozie-oozi-W]
> ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
> Start action
> [0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging] with
> user-retry state : userRetryCount [0], userRetryMax [0],
> userRetryInterval [10]
> 2014-04-17 20:35:54,619 WARN HiveActionExecutor:542 - USER[root]
> GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
> JOB[0000016-140415213056450-oozie-oozi-W]
> ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
> credentials is null for the action
> 2014-04-17 20:35:57,073 INFO HiveActionExecutor:539 - USER[root]
> GROUP[-] TOKEN[] APP[hive-gwcalls-insert-wf]
> JOB[0000016-140415213056450-oozie-oozi-W]
> ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
> checking action, external ID [job_1397592572894_0174] status [RUNNING]
> 2014-04-17 20:35:57,074 WARN ActionStartXCommand:542 - USER[root]
> GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
> JOB[0000016-140415213056450-oozie-oozi-W]
> ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
> [***0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging***]Actio
> n
> status=RUNNING
> 2014-04-17 20:35:57,075 WARN ActionStartXCommand:542 - USER[root]
> GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
> JOB[0000016-140415213056450-oozie-oozi-W]
> ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
> [***0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging***]Actio
> n
> updated in DB!
> 2014-04-17 20:36:29,057 INFO CallbackServlet:539 - USER[-] GROUP[-]
> TOKEN[-] APP[-] JOB[0000016-140415213056450-oozie-oozi-W]
> ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
> callback for action
> [0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
> 2014-04-17 20:36:29,509 INFO HiveActionExecutor:539 - USER[root]
> GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
> JOB[0000016-140415213056450-oozie-oozi-W]
> ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
> action completed, external ID [job_1397592572894_0174]
> 2014-04-17 20:36:29,542 INFO HiveActionExecutor:539 - USER[root]
> GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
> JOB[0000016-140415213056450-oozie-oozi-W]
> ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
> action produced output
> 2014-04-17 20:36:29,771 INFO ActionStartXCommand:539 - USER[root]
> GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
> JOB[0000016-140415213056450-oozie-oozi-W]
> ACTION[0000016-140415213056450-oozie-oozi-W@end] Start action
> [0000016-140415213056450-oozie-oozi-W@end] with user-retry state :
> userRetryCount [0], userRetryMax [0], userRetryInterval [10]
> 2014-04-17 20:36:29,771 WARN ActionStartXCommand:542 - USER[root]
> GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
> JOB[0000016-140415213056450-oozie-oozi-W]
> ACTION[0000016-140415213056450-oozie-oozi-W@end]
> [***0000016-140415213056450-oozie-oozi-W@end***]Action status=DONE
> 2014-04-17 20:36:29,772 WARN ActionStartXCommand:542 - USER[root]
> GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
> JOB[0000016-140415213056450-oozie-oozi-W]
> ACTION[0000016-140415213056450-oozie-oozi-W@end]
> [***0000016-140415213056450-oozie-oozi-W@end***]Action updated in DB!
> 2014-04-17 20:36:29,887 WARN CoordActionUpdateXCommand:542 -
> USER[root] GROUP[-] TOKEN[] APP[hive-gwcalls-insert-wf]
> JOB[0000016-140415213056450-oozie-oozi-W] ACTION[-] E1100: Command
> precondition does not hold before execution, [, coord action is null],
> Error Code: E1100
>
> Thanks,
> Manish Kothari
> Oracle Business Analyst
> Vonage | 23 Main Street | Holmdel, NJ 07733-2136
> t: +1.732.203.7613 | BB: +1.848.248.9615
> e: manish.kothari@vonage.com<ma...@vonage.com>
> w: www.vonage.com<http://www.vonage.com>
>
>
Re: Oozie job with hive action completes successfully but does not
create table
Posted by Amit Patil <ar...@skyhighnetworks.com>.
Its because your oozie hive action is not connecting to the hive metastore.
You need to start the hive metastore service. Refer
https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin
Once you start your remote metastore, make the required changes in the
hive-site.xml which your oozie application is using. This includes changing
the property hive.metastore.uris to point to the remote metastore.
This should work.
On Thu, Apr 17, 2014 at 1:51 PM, Manish Kothari
<Ma...@vonage.com>wrote:
> Hi,
>
> I have written an oozie job with hive action. It completes successfully
> but does not create a table in hive. When I execute the hive query
> independently it creates the table. Any help in this regard is appreciated.
>
> Here is the oozie log file : -
>
> 2014-04-17 20:35:53,445 INFO ActionStartXCommand:539 - USER[root]
> GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
> JOB[0000016-140415213056450-oozie-oozi-W]
> ACTION[0000016-140415213056450-oozie-oozi-W@:start:] Start action
> [0000016-140415213056450-oozie-oozi-W@:start:] with user-retry state :
> userRetryCount [0], userRetryMax [0], userRetryInterval [10]
> 2014-04-17 20:35:53,445 WARN ActionStartXCommand:542 - USER[root]
> GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
> JOB[0000016-140415213056450-oozie-oozi-W]
> ACTION[0000016-140415213056450-oozie-oozi-W@:start:]
> [***0000016-140415213056450-oozie-oozi-W@:start:***]Action status=DONE
> 2014-04-17 20:35:53,446 WARN ActionStartXCommand:542 - USER[root]
> GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
> JOB[0000016-140415213056450-oozie-oozi-W]
> ACTION[0000016-140415213056450-oozie-oozi-W@:start:]
> [***0000016-140415213056450-oozie-oozi-W@:start:***]Action updated in DB!
> 2014-04-17 20:35:53,601 INFO ActionStartXCommand:539 - USER[root]
> GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
> JOB[0000016-140415213056450-oozie-oozi-W]
> ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging] Start
> action [0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging] with
> user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval
> [10]
> 2014-04-17 20:35:54,619 WARN HiveActionExecutor:542 - USER[root] GROUP[-]
> TOKEN[-] APP[hive-gwcalls-insert-wf]
> JOB[0000016-140415213056450-oozie-oozi-W]
> ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
> credentials is null for the action
> 2014-04-17 20:35:57,073 INFO HiveActionExecutor:539 - USER[root] GROUP[-]
> TOKEN[] APP[hive-gwcalls-insert-wf]
> JOB[0000016-140415213056450-oozie-oozi-W]
> ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
> checking action, external ID [job_1397592572894_0174] status [RUNNING]
> 2014-04-17 20:35:57,074 WARN ActionStartXCommand:542 - USER[root]
> GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
> JOB[0000016-140415213056450-oozie-oozi-W]
> ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
> [***0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging***]Action
> status=RUNNING
> 2014-04-17 20:35:57,075 WARN ActionStartXCommand:542 - USER[root]
> GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
> JOB[0000016-140415213056450-oozie-oozi-W]
> ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
> [***0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging***]Action
> updated in DB!
> 2014-04-17 20:36:29,057 INFO CallbackServlet:539 - USER[-] GROUP[-]
> TOKEN[-] APP[-] JOB[0000016-140415213056450-oozie-oozi-W]
> ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
> callback for action
> [0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging]
> 2014-04-17 20:36:29,509 INFO HiveActionExecutor:539 - USER[root] GROUP[-]
> TOKEN[-] APP[hive-gwcalls-insert-wf]
> JOB[0000016-140415213056450-oozie-oozi-W]
> ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging] action
> completed, external ID [job_1397592572894_0174]
> 2014-04-17 20:36:29,542 INFO HiveActionExecutor:539 - USER[root] GROUP[-]
> TOKEN[-] APP[hive-gwcalls-insert-wf]
> JOB[0000016-140415213056450-oozie-oozi-W]
> ACTION[0000016-140415213056450-oozie-oozi-W@hive-gwcalls-staging] action
> produced output
> 2014-04-17 20:36:29,771 INFO ActionStartXCommand:539 - USER[root]
> GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
> JOB[0000016-140415213056450-oozie-oozi-W]
> ACTION[0000016-140415213056450-oozie-oozi-W@end] Start action
> [0000016-140415213056450-oozie-oozi-W@end] with user-retry state :
> userRetryCount [0], userRetryMax [0], userRetryInterval [10]
> 2014-04-17 20:36:29,771 WARN ActionStartXCommand:542 - USER[root]
> GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
> JOB[0000016-140415213056450-oozie-oozi-W]
> ACTION[0000016-140415213056450-oozie-oozi-W@end]
> [***0000016-140415213056450-oozie-oozi-W@end***]Action status=DONE
> 2014-04-17 20:36:29,772 WARN ActionStartXCommand:542 - USER[root]
> GROUP[-] TOKEN[-] APP[hive-gwcalls-insert-wf]
> JOB[0000016-140415213056450-oozie-oozi-W]
> ACTION[0000016-140415213056450-oozie-oozi-W@end]
> [***0000016-140415213056450-oozie-oozi-W@end***]Action updated in DB!
> 2014-04-17 20:36:29,887 WARN CoordActionUpdateXCommand:542 - USER[root]
> GROUP[-] TOKEN[] APP[hive-gwcalls-insert-wf]
> JOB[0000016-140415213056450-oozie-oozi-W] ACTION[-] E1100: Command
> precondition does not hold before execution, [, coord action is null],
> Error Code: E1100
>
> Thanks,
> Manish Kothari
> Oracle Business Analyst
> Vonage | 23 Main Street | Holmdel, NJ 07733-2136
> t: +1.732.203.7613 | BB: +1.848.248.9615
> e: manish.kothari@vonage.com<ma...@vonage.com>
> w: www.vonage.com<http://www.vonage.com>
>
>