You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by "Bhagwan S. Soni" <bh...@gmail.com> on 2015/05/14 17:21:14 UTC
hive job not making progress due to Number of reduce tasks is set to
0 since there's no reduce operator
Hi Hive Users,
I'm using Cloudera distribution and Hive's 13th version on my cluster.
I came across a problem where job is not making any progress after writing
log line - "*Number of reduce tasks is set to 0 since there's no reduce
operator*"
Below is the log for the same, could you help me what kind of issue is this
because this is not a code issue as if i re-run the same job it completes
successfully.
Logging initialized using configuration in
jar:file:/opt/cloudera/parcels/CDH-5.2.1-1.cdh5.2.1.p0.12/jars/hive-common-0.13.1-cdh5.2.1.jar!/hive-log4j.properties
Total jobs = 5
Launching Job 1 out of 5
Launching Job 2 out of 5
Number of reduce tasks not specified. Defaulting to jobconf value of: 10
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
Number of reduce tasks not specified. Defaulting to jobconf value of: 10
set mapreduce.job.reduces=<number>
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1431159077692_1399, Tracking URL =
http://xyz.com:8088/proxy/application_1431159077692_1399/
Starting Job = job_1431159077692_1398, Tracking URL =
http://xyz.com:8088/proxy/application_1431159077692_1398/
Kill Command =
/opt/cloudera/parcels/CDH-5.2.1-1.cdh5.2.1.p0.12/lib/hadoop/bin/hadoop job
-kill job_1431159077692_1399
Kill Command =
/opt/cloudera/parcels/CDH-5.2.1-1.cdh5.2.1.p0.12/lib/hadoop/bin/hadoop job
-kill job_1431159077692_1398
Hadoop job information for Stage-12: number of mappers: 5; number of
reducers: 10
Hadoop job information for Stage-1: number of mappers: 5; number of
reducers: 10
2015-05-12 19:59:12,298 Stage-1 map = 0%, reduce = 0%
2015-05-12 19:59:12,298 Stage-12 map = 0%, reduce = 0%
2015-05-12 19:59:20,832 Stage-1 map = 20%, reduce = 0%, Cumulative CPU 2.5
sec
2015-05-12 19:59:20,832 Stage-12 map = 80%, reduce = 0%, Cumulative CPU
8.63 sec
2015-05-12 19:59:21,905 Stage-1 map = 60%, reduce = 0%, Cumulative CPU
7.06 sec
2015-05-12 19:59:22,968 Stage-1 map = 80%, reduce = 0%, Cumulative CPU
9.34 sec
2015-05-12 19:59:24,031 Stage-1 map = 100%, reduce = 0%, Cumulative CPU
11.46 sec
2015-05-12 19:59:26,265 Stage-12 map = 100%, reduce = 0%, Cumulative CPU
10.92 sec
2015-05-12 19:59:32,665 Stage-12 map = 100%, reduce = 30%, Cumulative CPU
24.51 sec
2015-05-12 19:59:33,726 Stage-12 map = 100%, reduce = 100%, Cumulative CPU
57.61 sec
2015-05-12 19:59:35,021 Stage-1 map = 100%, reduce = 30%, Cumulative CPU
20.99 sec
MapReduce Total cumulative CPU time: 57 seconds 610 msec
Ended Job = job_1431159077692_1399
2015-05-12 19:59:36,084 Stage-1 map = 100%, reduce = 80%, Cumulative CPU
39.24 sec
2015-05-12 19:59:37,146 Stage-1 map = 100%, reduce = 90%, Cumulative CPU
42.37 sec
2015-05-12 19:59:38,203 Stage-1 map = 100%, reduce = 100%, Cumulative CPU
45.97 sec
MapReduce Total cumulative CPU time: 45 seconds 970 msec
Ended Job = job_1431159077692_1398
2015-05-12 19:59:45,180 WARN [main] conf.Configuration
(Configuration.java:loadProperty(2510)) -
file:/tmp/srv-hdp-mkt-d/hive_2015-05-12_19-58-53_081_2145723752519383568-1/-local-10014/jobconf.xml:an
attempt to override final parameter: hadoop.ssl.require.client.cert;
Ignoring.
2015-05-12 19:59:45,193 WARN [main] conf.Configuration
(Configuration.java:loadProperty(2510)) -
file:/tmp/srv-hdp-mkt-d/hive_2015-05-12_19-58-53_081_2145723752519383568-1/-local-10014/jobconf.xml:an
attempt to override final parameter:
mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-05-12 19:59:45,196 WARN [main] conf.Configuration
(Configuration.java:loadProperty(2510)) -
file:/tmp/srv-hdp-mkt-d/hive_2015-05-12_19-58-53_081_2145723752519383568-1/-local-10014/jobconf.xml:an
attempt to override final parameter: hadoop.ssl.client.conf; Ignoring.
2015-05-12 19:59:45,201 WARN [main] conf.Configuration
(Configuration.java:loadProperty(2510)) -
file:/tmp/srv-hdp-mkt-d/hive_2015-05-12_19-58-53_081_2145723752519383568-1/-local-10014/jobconf.xml:an
attempt to override final parameter: hadoop.ssl.keystores.factory.class;
Ignoring.
2015-05-12 19:59:45,210 WARN [main] conf.Configuration
(Configuration.java:loadProperty(2510)) -
file:/tmp/srv-hdp-mkt-d/hive_2015-05-12_19-58-53_081_2145723752519383568-1/-local-10014/jobconf.xml:an
attempt to override final parameter: hadoop.ssl.server.conf; Ignoring.
2015-05-12 19:59:45,258 WARN [main] conf.Configuration
(Configuration.java:loadProperty(2510)) -
file:/tmp/srv-hdp-mkt-d/hive_2015-05-12_19-58-53_081_2145723752519383568-1/-local-10014/jobconf.xml:an
attempt to override final parameter:
mapreduce.job.end-notification.max.attempts; Ignoring.
2015-05-12 19:59:45,792 WARN [main] conf.HiveConf
(HiveConf.java:initialize(1491)) - DEPRECATED: Configuration property
hive.metastore.local no longer has any effect. Make sure to provide a valid
value for hive.metastore.uris if you are connecting to a remote metastore.
Execution log at:
/tmp/srv-hdp-mkt-d/srv-hdp-mkt-d_20150512195858_1b598453-78a8-4867-9402-d972e3c067f2.log
2015-05-12 07:59:46 Starting to launch local task to process map
join; maximum memory = 257949696
2015-05-12 07:59:47 Dump the side-table into file:
file:/tmp/srv-hdp-mkt-d/hive_2015-05-12_19-58-53_081_2145723752519383568-1/-local-10007/HashTable-Stage-4/MapJoin-mapfile10--.hashtable
2015-05-12 07:59:47 Uploaded 1 File to:
file:/tmp/srv-hdp-mkt-d/hive_2015-05-12_19-58-53_081_2145723752519383568-1/-local-10007/HashTable-Stage-4/MapJoin-mapfile10--.hashtable
(475 bytes)
2015-05-12 07:59:47 Dump the side-table into file:
file:/tmp/srv-hdp-mkt-d/hive_2015-05-12_19-58-53_081_2145723752519383568-1/-local-10007/HashTable-Stage-4/MapJoin-mapfile01--.hashtable
2015-05-12 07:59:47 Uploaded 1 File to:
file:/tmp/srv-hdp-mkt-d/hive_2015-05-12_19-58-53_081_2145723752519383568-1/-local-10007/HashTable-Stage-4/MapJoin-mapfile01--.hashtable
(388 bytes)
2015-05-12 07:59:47 End of local task; Time Taken: 1.209 sec.
Execution completed successfully
MapredLocal task succeeded
Launching Job 3 out of 5
Number of reduce tasks is set to 0 since there's no reduce operator