You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by sumit ghosh <su...@yahoo.com> on 2013/04/22 03:39:43 UTC
Re: Set JobClient properties from hive CLI
Hi,
Sending to a wider audience - How do I set jobclient.progress.monitor.poll.interval in Hive?
Thanks,
Sumit
________________________________
From: sumit ghosh <su...@yahoo.com>
To: "user@hive.apache.org" <us...@hive.apache.org>
Sent: Saturday, 20 April 2013 1:28 AM
Subject: Set JobClient properties from hive CLI
Hi,
I am trying to set the property to change the logging interval to more than 5 seconds. Hence I am using the following parameter: jobclient.progress.monitor.poll.interval=5000
It works fine when I use it on an Map Reduce -
$ hadoop jar /usr/lib/hadoop-0.20-mapreduce/hadoop-2.0.0-mr1-cdh4.1.3-examples.jar wordcount -Djobclient.progress.monitor.poll.interval=5000 Input wc
13/04/20 03:23:59 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/04/20 03:24:00 INFO input.FileInputFormat: Total input paths to process : 1
13/04/20 03:24:00 INFO mapred.JobClient: Running job: job_201304012052_0616
13/04/20 03:24:05 INFO mapred.JobClient: map 0% reduce 0%
13/04/20 03:24:15 INFO mapred.JobClient: map 18% reduce 0%
13/04/20 03:24:20 INFO mapred.JobClient: map 24% reduce 0%
13/04/20 03:24:25 INFO mapred.JobClient: map 25% reduce 0%
13/04/20 03:24:30 INFO mapred.JobClient: map 32% reduce 0%
13/04/20 03:24:35 INFO mapred.JobClient: map 40% reduce 8%
13/04/20 03:24:40 INFO mapred.JobClient: map 50% reduce 8%
13/04/20 03:24:45 INFO mapred.JobClient: map 50% reduce 14%
13/04/20 03:24:50 INFO mapred.JobClient: map 55% reduce 16%
But this property is not working for hive CLI using set or using -hiveconf
hive -e "set jobclient.progress.monitor.poll.interval=5000; select count(*) from Table"
OR
hive -hiveconf jobclient.progress.monitor.poll.interval=5000 -e "select count(*) from Table"
What am I doing wrong?
Thanks,
Sumit.
Re: Set JobClient properties from hive CLI
Posted by sumit ghosh <su...@yahoo.com>.
Thanks Mike.
________________________________
From: Mike Liddell <Mi...@microsoft.com>
To: "user@hive.apache.org" <us...@hive.apache.org>; sumit ghosh <su...@yahoo.com>
Sent: Monday, 22 April 2013 11:02 AM
Subject: RE: Set JobClient properties from hive CLI
Dropping user@hadoop.apache.org
try “set hive.exec.counters.pull.interval = 5000;”
Ref: ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java::progress()
-mike.
From:sumit ghosh [mailto:sumitkg@yahoo.com]
Sent: Sunday, April 21, 2013 6:40 PM
To: user@hive.apache.org; sumit ghosh; user@hadoop.apache.org
Subject: Re: Set JobClient properties from hive CLI
Hi,
Sending to a wider audience - How do I set jobclient.progress.monitor.poll.interval in Hive?
Thanks,
Sumit
From:sumit ghosh <su...@yahoo.com>
To: "user@hive.apache.org" <us...@hive.apache.org>
Sent: Saturday, 20 April 2013 1:28 AM
Subject: Set JobClient properties from hive CLI
Hi,
I am trying to set the property to change the logging interval to more than 5 seconds. Hence I am using the following parameter: jobclient.progress.monitor.poll.interval=5000
It works fine when I use it on an Map Reduce -
$ hadoop jar /usr/lib/hadoop-0.20-mapreduce/hadoop-2.0.0-mr1-cdh4.1.3-examples.jar wordcount -Djobclient.progress.monitor.poll.interval=5000 Input wc
13/04/20 03:23:59 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/04/20 03:24:00 INFO input.FileInputFormat: Total input paths to process : 1
13/04/20 03:24:00 INFO mapred.JobClient: Running job: job_201304012052_0616
13/04/20 03:24:05 INFO mapred.JobClient: map 0% reduce 0%
13/04/20 03:24:15 INFO mapred.JobClient: map 18% reduce 0%
13/04/20 03:24:20 INFO mapred.JobClient: map 24% reduce 0%
13/04/20 03:24:25 INFO mapred.JobClient: map 25% reduce 0%
13/04/20 03:24:30 INFO mapred.JobClient: map 32% reduce 0%
13/04/20 03:24:35 INFO mapred.JobClient: map 40% reduce 8%
13/04/20 03:24:40 INFO mapred.JobClient: map 50% reduce 8%
13/04/20 03:24:45 INFO mapred.JobClient: map 50% reduce 14%
13/04/20 03:24:50 INFO mapred.JobClient: map 55% reduce 16%
But this property is not working for hive CLI using set or using -hiveconf
hive -e "set jobclient.progress.monitor.poll.interval=5000; select count(*) from Table"
OR
hive -hiveconf jobclient.progress.monitor.poll.interval=5000 -e "select count(*) from Table"
What am I doing wrong?
Thanks,
Sumit.
RE: Set JobClient properties from hive CLI
Posted by Mike Liddell <Mi...@microsoft.com>.
Dropping user@hadoop.apache.org<ma...@hadoop.apache.org>
try "set hive.exec.counters.pull.interval = 5000;"
Ref: ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java::progress()
-mike.
From: sumit ghosh [mailto:sumitkg@yahoo.com]
Sent: Sunday, April 21, 2013 6:40 PM
To: user@hive.apache.org; sumit ghosh; user@hadoop.apache.org
Subject: Re: Set JobClient properties from hive CLI
Hi,
Sending to a wider audience - How do I set jobclient.progress.monitor.poll.interval in Hive?
Thanks,
Sumit
From: sumit ghosh <su...@yahoo.com>>
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Sent: Saturday, 20 April 2013 1:28 AM
Subject: Set JobClient properties from hive CLI
Hi,
I am trying to set the property to change the logging interval to more than 5 seconds. Hence I am using the following parameter: jobclient.progress.monitor.poll.interval=5000
It works fine when I use it on an Map Reduce -
$ hadoop jar /usr/lib/hadoop-0.20-mapreduce/hadoop-2.0.0-mr1-cdh4.1.3-examples.jar wordcount -Djobclient.progress.monitor.poll.interval=5000 Input wc
13/04/20 03:23:59 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/04/20 03:24:00 INFO input.FileInputFormat: Total input paths to process : 1
13/04/20 03:24:00 INFO mapred.JobClient: Running job: job_201304012052_0616
13/04/20 03:24:05 INFO mapred.JobClient: map 0% reduce 0%
13/04/20 03:24:15 INFO mapred.JobClient: map 18% reduce 0%
13/04/20 03:24:20 INFO mapred.JobClient: map 24% reduce 0%
13/04/20 03:24:25 INFO mapred.JobClient: map 25% reduce 0%
13/04/20 03:24:30 INFO mapred.JobClient: map 32% reduce 0%
13/04/20 03:24:35 INFO mapred.JobClient: map 40% reduce 8%
13/04/20 03:24:40 INFO mapred.JobClient: map 50% reduce 8%
13/04/20 03:24:45 INFO mapred.JobClient: map 50% reduce 14%
13/04/20 03:24:50 INFO mapred.JobClient: map 55% reduce 16%
But this property is not working for hive CLI using set or using -hiveconf
hive -e "set jobclient.progress.monitor.poll.interval=5000; select count(*) from Table"
OR
hive -hiveconf jobclient.progress.monitor.poll.interval=5000 -e "select count(*) from Table"
What am I doing wrong?
Thanks,
Sumit.