You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@pig.apache.org by Sai Zhang <sz...@cs.washington.edu> on 2010/05/25 19:11:33 UTC

Generating example data for dataflow program

Hi all:

As indicated in the sigmod'09 paper, there is an "ILLUSTRATE" command 
from the Pig Shell to generate
example data for dataflow programs. But I did not find that either in 
the tutorial page nor in the pig -help output.

Could anyone give me some hint (or examples) on how to use this command? 
Also, I probably may want to
have a look at the source code data generating to full understand the 
algorithm, could any one point me out
where relevant source code resident (which packages)?

Thanks very much.

-Sai

Re: Generating example data for dataflow program

Posted by Sai Zhang <sz...@cs.washington.edu>.

Thanks Dmitriy and Aniket:

Probably what I need is to run that in the local mode for data 
generation, so the current implementation
should be sufficient (i am not clear which operators are not supported)!

 One more question, I important the whole pig project into eclipse, and 
would like to run (debug and trace in)
the data flow algorithm in the development environment (not in the 
deployment environment). Is there any
hints on which "main" method should I start, and how to set up the 
environment? In other words, could I
just run the code of data generation part instead of launching the whole 
pig shell?

thanks

-Sai

Dmitriy Ryaboy wrote:
> NB: Unfortunately Illustrate has not been well-maintained; it does not work
> for several pig operators, or in map-reduce mode. It would be great if
> someone updated it to address these issues.
>
> -Dmitriy
>
> On Tue, May 25, 2010 at 10:51 AM, Aniket Mokashi <am...@andrew.cmu.edu>wrote:
>
>   
>> Hi Sai,
>>
>> I recently read about it, most of the commands are explained on the Pig
>> Latin Reference Manual-2. Here is the link for ILLUSTRATE command-
>> http://hadoop.apache.org/pig/docs/r0.7.0/piglatin_ref2.html#ILLUSTRATE.
>>
>> Thanks,
>> Aniket
>>
>> On Tue, May 25, 2010 1:11 pm, Sai Zhang wrote:
>>     
>>> Hi all:
>>>
>>>
>>> As indicated in the sigmod'09 paper, there is an "ILLUSTRATE" command
>>> from the Pig Shell to generate example data for dataflow programs. But I
>>> did not find that either in the tutorial page nor in the pig -help
>>>       
>> output.
>>     
>>> Could anyone give me some hint (or examples) on how to use this command?
>>> Also, I probably may want to
>>> have a look at the source code data generating to full understand the
>>> algorithm, could any one point me out where relevant source code resident
>>> (which packages)?
>>>
>>>
>>> Thanks very much.
>>>
>>>
>>> -Sai
>>>
>>>
>>>
>>>       
>>
>>     
>
>

Re: Generating example data for dataflow program

Posted by Dmitriy Ryaboy <dv...@gmail.com>.

NB: Unfortunately Illustrate has not been well-maintained; it does not work
for several pig operators, or in map-reduce mode. It would be great if
someone updated it to address these issues.

-Dmitriy

On Tue, May 25, 2010 at 10:51 AM, Aniket Mokashi <am...@andrew.cmu.edu>wrote:

> Hi Sai,
>
> I recently read about it, most of the commands are explained on the Pig
> Latin Reference Manual-2. Here is the link for ILLUSTRATE command-
> http://hadoop.apache.org/pig/docs/r0.7.0/piglatin_ref2.html#ILLUSTRATE.
>
> Thanks,
> Aniket
>
> On Tue, May 25, 2010 1:11 pm, Sai Zhang wrote:
> > Hi all:
> >
> >
> > As indicated in the sigmod'09 paper, there is an "ILLUSTRATE" command
> > from the Pig Shell to generate example data for dataflow programs. But I
> > did not find that either in the tutorial page nor in the pig -help
> output.
> >
> >
> > Could anyone give me some hint (or examples) on how to use this command?
> > Also, I probably may want to
> > have a look at the source code data generating to full understand the
> > algorithm, could any one point me out where relevant source code resident
> > (which packages)?
> >
> >
> > Thanks very much.
> >
> >
> > -Sai
> >
> >
> >
>
>
>

Re: Generating example data for dataflow program

Posted by Aniket Mokashi <am...@andrew.cmu.edu>.

Hi Sai,

I recently read about it, most of the commands are explained on the Pig
Latin Reference Manual-2. Here is the link for ILLUSTRATE command-
http://hadoop.apache.org/pig/docs/r0.7.0/piglatin_ref2.html#ILLUSTRATE.

Thanks,
Aniket

On Tue, May 25, 2010 1:11 pm, Sai Zhang wrote:
> Hi all:
>
>
> As indicated in the sigmod'09 paper, there is an "ILLUSTRATE" command
> from the Pig Shell to generate example data for dataflow programs. But I
> did not find that either in the tutorial page nor in the pig -help output.
>
>
> Could anyone give me some hint (or examples) on how to use this command?
> Also, I probably may want to
> have a look at the source code data generating to full understand the
> algorithm, could any one point me out where relevant source code resident
> (which packages)?
>
>
> Thanks very much.
>
>
> -Sai
>
>
>