You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by "Michael G. Noll" <mi...@googlemail.com> on 2011/09/15 15:59:00 UTC
Configuring Hive to automatically create _SUCCESS files
Hello,
is there a way to configure Hive 0.7.1 so that it creates _SUCCESS files
similar to Hadoop MapReduce and Pig?
Hadoop has the property "mapreduce.fileoutputcommitter.marksuccessfuljobs"
(see FileOutputCommitter.java), which is as of Hadoop 0.20.203.0 enabled by
default. In Pig 0.9.0, you can enable success files by e.g. setting the
property on the command line when running a Pig script (see PIG-1433 [1]):
$ pig -Dmapreduce.fileoutputcommitter.marksuccessfuljobs=true ...
I have unsuccessfully tried to enable success files in Hive 0.7.1 in
combination with Hadoop 0.20.203.0:
# has no effect
$ hive -hiveconf mapreduce.fileoutputcommitter.marksuccessfuljobs=true
...
The job configuration XML file lists the property as being set to true, but
Hive will not create a success file.
Best,
Michael
[1] https://issues.apache.org/jira/browse/PIG-1433
Re: Configuring Hive to automatically create _SUCCESS files
Posted by "Michael G. Noll" <mi...@googlemail.com>.
Hi Siddharth,
thanks for your reply. I have followed your suggestion and tried it with
Hadoop 0.20.2 as well (and Hive 0.7.1). Unfortunately, this combination
does not work, too -- no _SUCCESS files are generated.
Anything else I could try?
Best,
Michael
On Thu, Sep 15, 2011 at 19:07, Siddharth Tiwari
<si...@live.com>wrote:
> Dear Mike
>
> Did u try the same using hive-0.7.1 and hadoop-0.20.2 ? try it out once
>
RE: Configuring Hive to automatically create _SUCCESS files
Posted by Siddharth Tiwari <si...@live.com>.
Dear Mike
Did u try the same using hive-0.7.1 and hadoop-0.20.2 ? try it out once
*------------------------*
Cheers !!!
Siddharth Tiwari
Have a refreshing day !!!
Date: Thu, 15 Sep 2011 15:59:00 +0200
Subject: Configuring Hive to automatically create _SUCCESS files
From: michael.g.noll@googlemail.com
To: user@hive.apache.org
Hello,
is there a way to configure Hive 0.7.1 so that it creates _SUCCESS files similar to Hadoop MapReduce and Pig?
Hadoop has the property "mapreduce.fileoutputcommitter.marksuccessfuljobs" (see FileOutputCommitter.java), which is as of Hadoop 0.20.203.0 enabled by default. In Pig 0.9.0, you can enable success files by e.g. setting the property on the command line when running a Pig script (see PIG-1433 [1]):
$ pig -Dmapreduce.fileoutputcommitter.marksuccessfuljobs=true ...
I have unsuccessfully tried to enable success files in Hive 0.7.1 in combination with Hadoop 0.20.203.0:
# has no effect
$ hive -hiveconf mapreduce.fileoutputcommitter.marksuccessfuljobs=true ...
The job configuration XML file lists the property as being set to true, but Hive will not create a success file.
Best,
Michael
[1] https://issues.apache.org/jira/browse/PIG-1433