You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by John Farrelly <Jo...@adaptivemobile.com> on 2013/04/09 15:28:45 UTC
Stopping after load with no tuples?
Hi there,
I have a pig script similar to the one below. When testing this on a cluster with an empty file, I see it taking ages to complete - It goes through all the commands and runs the jobs across the tracker. In our test environment, we will (for various reasons) have an input directory with a file which is empty. Is there any way to tell pig to exit the script if there is no data available (or a command like EXIT IF data EMPTY?). My UDF loader does return null as the tuple when it reads an empty file, so ideally I'd like it to stop processing. Is there any way to do this?
Thanks,
John.
define MyDataLoader com.example.test.MyDataLoader('config1');
raw = LOAD '$inputdir' USING MyDataLoader AS (date:chararray, values:bag{t:(policy:chararray)});
data1 = FOREACH raw GENERATE date, .....
data2 = GROUP data1 BY (date, .....
data3 = FOREACH data3 GENERATE group.date, .....
STORE data3 INTO 'MyDataStore'
</pre>****************************************************************************************<br>This email and any files transmitted with are confidential and intended solely for the<br>use of the individual or entity to whom they are addressed. If you have received this<br>email in error then please delete it and notify the sender. Do not make a copy or forward<br>it to anyone. This footnote also confirms that this email message has been swept for the<br>presence of computer viruses.<br><br>Adaptive Mobile Security Ltd, Ferry House, 48 Lower Mount Street, Dublin 2, Ireland<br>Directors: B. Collins, G. Maclachlan (UK), N. Grierson (UK), J. Ennis (UK), D. Summers (UK).<br>Registered in Ireland, Company No. 370343, VAT Reg.No.IE6390343O<br>****************************************************************************************</pre>
RE: Stopping after load with no tuples?
Posted by John Farrelly <Jo...@adaptivemobile.com>.
Thanks for the reply Ruslan. I was guessing it wouldn't be possible, but thought I would ask incase there was an easy way to do it.I hadn't read about Oozie - looks useful!
Regards,
John.
-----Original Message-----
From: Ruslan Al-Fakikh [mailto:metaruslan@gmail.com]
Sent: 17 April 2013 18:09
To: user@pig.apache.org
Subject: Re: Stopping after load with no tuples?
Hi
I haven't heard of such functionality in Pig, but it seems that you are looking for a functionality that is not what Pig is for. It lacks some control things like IF ELSE statements, because it is not a general-purpose language. There are a couple of ways you can do such things:
1) Launch Pig with a bash wrapper.
2) Embed Pig into a Java or Python, etc (just like you would embed SQL into a regular language). Pig supports it out of the box
3) Use Oozie or something similar for the jobs orchestration.
Hope that helps
Ruslan Al-Fakikh
On Tue, Apr 9, 2013 at 5:28 PM, John Farrelly < JohnFarrelly@adaptivemobile.com> wrote:
> Hi there,
>
> I have a pig script similar to the one below. When testing this on a
> cluster with an empty file, I see it taking ages to complete - It goes
> through all the commands and runs the jobs across the tracker. In our
> test environment, we will (for various reasons) have an input
> directory with a file which is empty. Is there any way to tell pig to
> exit the script if there is no data available (or a command like EXIT
> IF data EMPTY?). My UDF loader does return null as the tuple when it
> reads an empty file, so ideally I'd like it to stop processing. Is there any way to do this?
>
> Thanks,
> John.
>
> define MyDataLoader com.example.test.MyDataLoader('config1');
> raw = LOAD '$inputdir' USING MyDataLoader AS (date:chararray,
> values:bag{t:(policy:chararray)});
>
> data1 = FOREACH raw GENERATE date, .....
> data2 = GROUP data1 BY (date, .....
> data3 = FOREACH data3 GENERATE group.date, .....
>
> STORE data3 INTO 'MyDataStore'
> </pre>****************************************************************
> ************************<br>This email and any files transmitted with
> are confidential and intended solely for the<br>use of the individual
> or entity to whom they are addressed. If you have received
> this<br>email in error then please delete it and notify the sender. Do
> not make a copy or forward<br>it to anyone. This footnote also
> confirms that this email message has been swept for the<br>presence of
> computer viruses.<br><br>Adaptive Mobile Security Ltd, Ferry House, 48
> Lower Mount Street, Dublin 2, Ireland<br>Directors: B. Collins, G.
> Maclachlan (UK), N. Grierson (UK), J. Ennis (UK), D. Summers
> (UK).<br>Registered in Ireland, Company No. 370343, VAT
> Reg.No.IE6390343O<br>*************************************************
> ***************************************</pre>
>
>
</pre>****************************************************************************************<br>This email and any files transmitted with are confidential and intended solely for the<br>use of the individual or entity to whom they are addressed. If you have received this<br>email in error then please delete it and notify the sender. Do not make a copy or forward<br>it to anyone. This footnote also confirms that this email message has been swept for the<br>presence of computer viruses.<br><br>Adaptive Mobile Security Ltd, Ferry House, 48 Lower Mount Street, Dublin 2, Ireland<br>Directors: B. Collins, G. Maclachlan (UK), N. Grierson (UK), J. Ennis (UK), D. Summers (UK).<br>Registered in Ireland, Company No. 370343, VAT Reg.No.IE6390343O<br>****************************************************************************************</pre>
Re: Stopping after load with no tuples?
Posted by Ruslan Al-Fakikh <me...@gmail.com>.
Hi
I haven't heard of such functionality in Pig, but it seems that you are
looking for a functionality that is not what Pig is for. It lacks some
control things like IF ELSE statements, because it is not a general-purpose
language. There are a couple of ways you can do such things:
1) Launch Pig with a bash wrapper.
2) Embed Pig into a Java or Python, etc (just like you would embed SQL into
a regular language). Pig supports it out of the box
3) Use Oozie or something similar for the jobs orchestration.
Hope that helps
Ruslan Al-Fakikh
On Tue, Apr 9, 2013 at 5:28 PM, John Farrelly <
JohnFarrelly@adaptivemobile.com> wrote:
> Hi there,
>
> I have a pig script similar to the one below. When testing this on a
> cluster with an empty file, I see it taking ages to complete - It goes
> through all the commands and runs the jobs across the tracker. In our test
> environment, we will (for various reasons) have an input directory with a
> file which is empty. Is there any way to tell pig to exit the script if
> there is no data available (or a command like EXIT IF data EMPTY?). My UDF
> loader does return null as the tuple when it reads an empty file, so
> ideally I'd like it to stop processing. Is there any way to do this?
>
> Thanks,
> John.
>
> define MyDataLoader com.example.test.MyDataLoader('config1');
> raw = LOAD '$inputdir' USING MyDataLoader AS (date:chararray,
> values:bag{t:(policy:chararray)});
>
> data1 = FOREACH raw GENERATE date, .....
> data2 = GROUP data1 BY (date, .....
> data3 = FOREACH data3 GENERATE group.date, .....
>
> STORE data3 INTO 'MyDataStore'
> </pre>****************************************************************************************<br>This
> email and any files transmitted with are confidential and intended solely
> for the<br>use of the individual or entity to whom they are addressed. If
> you have received this<br>email in error then please delete it and notify
> the sender. Do not make a copy or forward<br>it to anyone. This footnote
> also confirms that this email message has been swept for the<br>presence of
> computer viruses.<br><br>Adaptive Mobile Security Ltd, Ferry House, 48
> Lower Mount Street, Dublin 2, Ireland<br>Directors: B. Collins, G.
> Maclachlan (UK), N. Grierson (UK), J. Ennis (UK), D. Summers
> (UK).<br>Registered in Ireland, Company No. 370343, VAT
> Reg.No.IE6390343O<br>****************************************************************************************</pre>
>
>