Newbie Questions - Enlighten me please

[Andre Philippi <an...@gmail.com>]

Hi,

I started learning and working with Pig 3 days ago and I have been able to
make it work with my EC2/HDFS environment and parse some files. Yay! :)

Now, after reading all of the docs on the wiki, I still have a few questions
that I was hoping someone here could help me with. I was also thinking that
the answers to these basic questions could be re-used on a HOWTO doc for new
Pig users.

1) Is there a way to know the number of positional fields returned after a
"foo = load 'bar' using PigStorage();" ?

Ex.: $#


2) Is there a way to test the existence of a particular positional field?

Ex.: "SPLIT A INTO X IF $3; Y if !$3" or something like that :)


3) How can I further split a field after a load?

Ex.:

foo = load "apache_logs" using PigStorage(' ') as (timestamp, ip, url.
query_string);
bar = split foo using ?????('&') as (anon_id, user_action, age, sex
location);

P.S.: So far I'm hacking this by doing the following to accomplish what I
want, but I'm sure (hope) there's a better way:

log = load '/data/logs/apache.log' using PigStorage(',') as
(time,ip,url,query);
queries = foreach log generate query;
store queries into '/data/out/pig/queries' using PigStorage();
split_queries = load '/data/out/pig/queries/*' using PigStorage('&');


4) Apache Logs:

I saw the UDF expandQuery function referenced on the sigmod08.pdf, but since
I'm not being able to compile the piggybank, I'm not sure if it's really
available.

So, my question is... Has anyone written any UDFs for parsing apache logs
yet?

I'm sure a lot of Pig users would be interested in such functions...


5) Pig Streaming:

I saw the following code snippet at the PigPerformance doc:

IP = load '/pig/in';
FILTERED_DATA = filter IP by $1 > '0';
OP = stream IP through `perl -ne 'print $_;'`;
store OP into '/pig/out';

However, I'm still a bit puzzled with the usage of the stream feature.

Could someone be kind enough to give a better (more practical) example (or
share some working code) of using the streaming functionality?

Ex.:

OP = stream IP through `perl -ne 'print $_ if =~
/(\d{3}).(\d{3}).(\d{3}).(\d{3})/;'`;

5.1) Would something like that work?
5.2) What would be the pros/cons of "transfering" some of the regex parsing
to perl (similar to the code above) taking in consideration Pig/MapReduce
and the distributed architecture?

Thank you in advance for any help on the isues above,

Best Regards,

Andre Philippi

RE: Newbie Questions - Enlighten me please

[Olga Natkovich <ol...@yahoo-inc.com>]

Hi Andre,

It is great that you are starting to use Pig! Welcome to the community
:).

My answers to your questions are inline. 

> -----Original Message-----
> From: Andre Philippi [mailto:andre.philippi@gmail.com] 
> Sent: Monday, July 14, 2008 1:47 PM
> To: pig-user@incubator.apache.org
> Subject: Newbie Questions - Enlighten me please
> 
> Hi,
> 
> I started learning and working with Pig 3 days ago and I have 
> been able to make it work with my EC2/HDFS environment and 
> parse some files. Yay! :)
> 
> Now, after reading all of the docs on the wiki, I still have 
> a few questions that I was hoping someone here could help me 
> with. I was also thinking that the answers to these basic 
> questions could be re-used on a HOWTO doc for new Pig users.
> 
> 1) Is there a way to know the number of positional fields 
> returned after a "foo = load 'bar' using PigStorage();" ?
> 
> Ex.: $#

You can use built-in ARITY function to find out how many fields a tuple
has. 

> 
> 
> 2) Is there a way to test the existence of a particular 
> positional field?
> 
> Ex.: "SPLIT A INTO X IF $3; Y if !$3" or something like that :)

You mean whether it is null or not? In the current version of pig, you
need to know what the null value in your data looks like. For instance,
you could say $3 eq ''. As part of pipeline redesign we are adding
proper support for nulls.

> 3) How can I further split a field after a load?
> 
> Ex.:
> 
> foo = load "apache_logs" using PigStorage(' ') as (timestamp, ip, url.
> query_string);
> bar = split foo using ?????('&') as (anon_id, user_action, 
> age, sex location);
> 
> P.S.: So far I'm hacking this by doing the following to 
> accomplish what I want, but I'm sure (hope) there's a better way:
> 
> log = load '/data/logs/apache.log' using PigStorage(',') as 
> (time,ip,url,query); queries = foreach log generate query; 
> store queries into '/data/out/pig/queries' using 
> PigStorage(); split_queries = load '/data/out/pig/queries/*' 
> using PigStorage('&');

You can either write a custom storage function that will fully parse
your data. Or if you only need this to do conditionally (like after a
filter), you could write a user defined function that parses the needed
field.

> 4) Apache Logs:
> 
> I saw the UDF expandQuery function referenced on the 
> sigmod08.pdf, but since I'm not being able to compile the 
> piggybank, I'm not sure if it's really available.
> 
> So, my question is... Has anyone written any UDFs for parsing 
> apache logs yet?
> 
> I'm sure a lot of Pig users would be interested in such functions...

Piggybank currently does not have it. One other user asked about it
recently and we encouraged him to contribute it if he finds/writes one.

> 
> 
> 5) Pig Streaming:
> 
> I saw the following code snippet at the PigPerformance doc:
> 
> IP = load '/pig/in';
> FILTERED_DATA = filter IP by $1 > '0';
> OP = stream IP through `perl -ne 'print $_;'`; store OP into 
> '/pig/out';
> 
> However, I'm still a bit puzzled with the usage of the stream feature.

I am not quite sure what your question is here. Any perl code should
work. In general things that can we done within pig itself like regex
matching should be done with pig. Streaming is generally for
functionality not available in pig as it has cost of copying data in and
out of a process. Streaming documentation is available at
http://wiki.apache.org/pig/PigStreamingFunctionalSpec

> 
> Could someone be kind enough to give a better (more 
> practical) example (or share some working code) of using the 
> streaming functionality?
> 
> Ex.:
> 
> OP = stream IP through `perl -ne 'print $_ if =~ 
> /(\d{3}).(\d{3}).(\d{3}).(\d{3})/;'`;
> 
> 5.1) Would something like that work?
> 5.2) What would be the pros/cons of "transfering" some of the 
> regex parsing to perl (similar to the code above) taking in 
> consideration Pig/MapReduce and the distributed architecture?
> 
> Thank you in advance for any help on the isues above,
> 
> Best Regards,
> 
> Andre Philippi
>