You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by "Michael G. Noll" <mi...@googlemail.com> on 2011/09/15 15:59:00 UTC

Configuring Hive to automatically create _SUCCESS files

Hello,

is there a way to configure Hive 0.7.1 so that it creates _SUCCESS files
similar to Hadoop MapReduce and Pig?

Hadoop has the property "mapreduce.fileoutputcommitter.marksuccessfuljobs"
(see FileOutputCommitter.java), which is as of Hadoop 0.20.203.0 enabled by
default.  In Pig 0.9.0, you can enable success files by e.g. setting the
property on the command line when running a Pig script (see PIG-1433 [1]):

    $ pig -Dmapreduce.fileoutputcommitter.marksuccessfuljobs=true ...

I have unsuccessfully tried to enable success files in Hive 0.7.1 in
combination with Hadoop 0.20.203.0:

    # has no effect
    $ hive -hiveconf mapreduce.fileoutputcommitter.marksuccessfuljobs=true
...

The job configuration XML file lists the property as being set to true, but
Hive will not create a success file.

Best,
Michael


[1] https://issues.apache.org/jira/browse/PIG-1433

Re: Configuring Hive to automatically create _SUCCESS files

Posted by "Michael G. Noll" <mi...@googlemail.com>.
Hi Siddharth,

thanks for your reply.  I have followed your suggestion and tried it with
Hadoop 0.20.2 as well (and Hive 0.7.1).  Unfortunately, this combination
does not work, too -- no _SUCCESS files are generated.

Anything else I could try?

Best,
Michael



On Thu, Sep 15, 2011 at 19:07, Siddharth Tiwari
<si...@live.com>wrote:

>  Dear Mike
>
> Did u try the same using hive-0.7.1 and hadoop-0.20.2 ? try it out once
>

RE: Configuring Hive to automatically create _SUCCESS files

Posted by Siddharth Tiwari <si...@live.com>.
Dear Mike

Did u try the same using hive-0.7.1 and hadoop-0.20.2 ? try it out once



*------------------------*

Cheers !!!

Siddharth Tiwari

Have a refreshing day !!!


Date: Thu, 15 Sep 2011 15:59:00 +0200
Subject: Configuring Hive to automatically create _SUCCESS files
From: michael.g.noll@googlemail.com
To: user@hive.apache.org

Hello,

is there a way to configure Hive 0.7.1 so that it creates _SUCCESS files similar to Hadoop MapReduce and Pig?

Hadoop has the property "mapreduce.fileoutputcommitter.marksuccessfuljobs" (see FileOutputCommitter.java), which is as of Hadoop 0.20.203.0 enabled by default.  In Pig 0.9.0, you can enable success files by e.g. setting the property on the command line when running a Pig script (see PIG-1433 [1]):


    $ pig -Dmapreduce.fileoutputcommitter.marksuccessfuljobs=true ...

I have unsuccessfully tried to enable success files in Hive 0.7.1 in combination with Hadoop 0.20.203.0:


    # has no effect
    $ hive -hiveconf mapreduce.fileoutputcommitter.marksuccessfuljobs=true ...

The job configuration XML file lists the property as being set to true, but Hive will not create a success file.


Best,
Michael


[1] https://issues.apache.org/jira/browse/PIG-1433