what happens under the hood

[jamal sasha <ja...@gmail.com>]

Hi,
  I am trying to dig deep on the workings of pig libraries.

So can someone help me understand what happens when someone does:

in = load 'in.txt' using PigStorage(',') as (foo:int);
dump in;

what happens behind the scenes..
How does it executes map reduce jobs..
where is this "load" defined in the pig code base .
I am just trying to see how  the backend code is implemented where this two
lines of code translates into the map reduce code.
Any pointers.
Thanks
Jamal

Re: what happens under the hood

[Jonathan Coveney <jc...@gmail.com>]

This is a very broad question. On the Pig website you can find some papers
on how Pig was implemented, and this should give you a high level view of
what is going on.

For this code, you can use the explain command (explain in; instead of dump
in;) to see the 3 plans that this code generates (logical, physical, mr).
If you want to be a real pro, put in a debug statement in your ide and
actually look at the steps as it builds the logical plan and the converts
to physical and mr.


2012/12/19 jamal sasha <ja...@gmail.com>

> Hi,
>   I am trying to dig deep on the workings of pig libraries.
>
> So can someone help me understand what happens when someone does:
>
> in = load 'in.txt' using PigStorage(',') as (foo:int);
> dump in;
>
> what happens behind the scenes..
> How does it executes map reduce jobs..
> where is this "load" defined in the pig code base .
> I am just trying to see how  the backend code is implemented where this two
> lines of code translates into the map reduce code.
> Any pointers.
> Thanks
> Jamal
>