_SUCCESS file -> _FAILURE file?

["Zebeljan, Nebojsa" <ne...@adtech.com>]

Hi,
Pig writes a 0 byte _SUCCESS file, when the STORE has been successfully done. Is there something like a _FAILURE file that indicates that the STORE failed?

I need to determine with a another application (java), if the pig run has finished and if it was successful or not.

Any approach is welcome – Thanks for your help!

Regards,
Nebo

Re: _SUCCESS file -> _FAILURE file?

[Gianmarco De Francisci Morales <gd...@apache.org>]

You could use ZooKeeper from your wrapper script to notify clients of
state changes.

Cheers,
--
Gianmarco


On Thu, Oct 18, 2012 at 3:59 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> that will only work if the interested party in the output fo the pig
> script is the same one running the pig script.
>
> On Thu, Oct 18, 2012 at 2:10 PM, Prashant Kommireddi
> <pr...@gmail.com> wrote:
>> One way could be to handle the failure of a Pig job in a wrapper script. Or
>> use the Java APIs to be able to see if the job completed successfully.
>>
>> http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/PigServer.html#store%28java.lang.String,%20java.lang.String%29
>> http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/PigServer.html#executeBatch%28%29
>>
>>
>> On Thu, Oct 18, 2012 at 2:03 PM, Dmitriy Ryaboy <dv...@gmail.com> wrote:
>>
>>> That's a Hadoop mapreduce feature, not a Pig feature, so that request
>>> should go there.
>>>
>>> Can't really do the _failure thing though, if you think about it --
>>> programs can fail by crashing, in which case they might not be able to
>>> write a file. Or maybe they are not crashing, but there is a problem
>>> talking to HDFS, still can't write the failure.
>>>
>>>
>>>
>>> D
>>>
>>>
>>> On Thu, Oct 18, 2012 at 12:43 AM, Zebeljan, Nebojsa
>>> <ne...@adtech.com> wrote:
>>> > Hi,
>>> > Pig writes a 0 byte _SUCCESS file, when the STORE has been successfully
>>> done. Is there something like a _FAILURE file that indicates that the STORE
>>> failed?
>>> >
>>> > I need to determine with a another application (java), if the pig run
>>> has finished and if it was successful or not.
>>> >
>>> > Any approach is welcome – Thanks for your help!
>>> >
>>> > Regards,
>>> > Nebo
>>>
>
>
>
> --
> Alejandro

Re: _SUCCESS file -> _FAILURE file?

[Alejandro Abdelnur <tu...@cloudera.com>]

that will only work if the interested party in the output fo the pig
script is the same one running the pig script.

On Thu, Oct 18, 2012 at 2:10 PM, Prashant Kommireddi
<pr...@gmail.com> wrote:
> One way could be to handle the failure of a Pig job in a wrapper script. Or
> use the Java APIs to be able to see if the job completed successfully.
>
> http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/PigServer.html#store%28java.lang.String,%20java.lang.String%29
> http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/PigServer.html#executeBatch%28%29
>
>
> On Thu, Oct 18, 2012 at 2:03 PM, Dmitriy Ryaboy <dv...@gmail.com> wrote:
>
>> That's a Hadoop mapreduce feature, not a Pig feature, so that request
>> should go there.
>>
>> Can't really do the _failure thing though, if you think about it --
>> programs can fail by crashing, in which case they might not be able to
>> write a file. Or maybe they are not crashing, but there is a problem
>> talking to HDFS, still can't write the failure.
>>
>>
>>
>> D
>>
>>
>> On Thu, Oct 18, 2012 at 12:43 AM, Zebeljan, Nebojsa
>> <ne...@adtech.com> wrote:
>> > Hi,
>> > Pig writes a 0 byte _SUCCESS file, when the STORE has been successfully
>> done. Is there something like a _FAILURE file that indicates that the STORE
>> failed?
>> >
>> > I need to determine with a another application (java), if the pig run
>> has finished and if it was successful or not.
>> >
>> > Any approach is welcome – Thanks for your help!
>> >
>> > Regards,
>> > Nebo
>>



-- 
Alejandro

Re: _SUCCESS file -> _FAILURE file?

[Prashant Kommireddi <pr...@gmail.com>]

One way could be to handle the failure of a Pig job in a wrapper script. Or
use the Java APIs to be able to see if the job completed successfully.

http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/PigServer.html#store%28java.lang.String,%20java.lang.String%29
http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/PigServer.html#executeBatch%28%29


On Thu, Oct 18, 2012 at 2:03 PM, Dmitriy Ryaboy <dv...@gmail.com> wrote:

> That's a Hadoop mapreduce feature, not a Pig feature, so that request
> should go there.
>
> Can't really do the _failure thing though, if you think about it --
> programs can fail by crashing, in which case they might not be able to
> write a file. Or maybe they are not crashing, but there is a problem
> talking to HDFS, still can't write the failure.
>
>
>
> D
>
>
> On Thu, Oct 18, 2012 at 12:43 AM, Zebeljan, Nebojsa
> <ne...@adtech.com> wrote:
> > Hi,
> > Pig writes a 0 byte _SUCCESS file, when the STORE has been successfully
> done. Is there something like a _FAILURE file that indicates that the STORE
> failed?
> >
> > I need to determine with a another application (java), if the pig run
> has finished and if it was successful or not.
> >
> > Any approach is welcome – Thanks for your help!
> >
> > Regards,
> > Nebo
>

Re: _SUCCESS file -> _FAILURE file?

[Dmitriy Ryaboy <dv...@gmail.com>]

That's a Hadoop mapreduce feature, not a Pig feature, so that request
should go there.

Can't really do the _failure thing though, if you think about it --
programs can fail by crashing, in which case they might not be able to
write a file. Or maybe they are not crashing, but there is a problem
talking to HDFS, still can't write the failure.



D


On Thu, Oct 18, 2012 at 12:43 AM, Zebeljan, Nebojsa
<ne...@adtech.com> wrote:
> Hi,
> Pig writes a 0 byte _SUCCESS file, when the STORE has been successfully done. Is there something like a _FAILURE file that indicates that the STORE failed?
>
> I need to determine with a another application (java), if the pig run has finished and if it was successful or not.
>
> Any approach is welcome – Thanks for your help!
>
> Regards,
> Nebo