You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by Jon Coveney <jc...@gmail.com> on 2012/07/03 21:45:53 UTC

Are there any explanations of the implementation of illustrate?

I was curious at a level slightly higher than "dig through the code" how illustrate is so fast, and how it deals with joins effectively. Are there any resources on this (or does anyone at Hortonworks want to write a tech oriented blog post? :)

RE: Are there any explanations of the implementation of illustrate?

Posted by Doug Daniels <dd...@mortardata.com>.
Hey Jonathan,

We've done a bunch of updates to illustrate on a private 0.9 branch that we're looking to merge in as soon as we can, including making it much more testable and making far more of the pig operators work again.  

One thing to note on the algorithm is that the upstream 'data synthesis' process doesn't work on scripts that include UDFs, as it cannot invert data through the UDF.  This means that for many scripts, joins (and very selective filters) are still troublesome.

-Doug
________________________________________
From: Jonathan Coveney [jcoveney@gmail.com]
Sent: Tuesday, July 03, 2012 9:34 PM
To: dev@pig.apache.org
Subject: Re: Are there any explanations of the implementation of illustrate?

Jie, that's perfect, thanks. This doc, specifically:
http://i.stanford.edu/~olston/publications/sigmod09.pdf is exactly the
detailed explanation I was looking for.

2012/7/3 Jie Li <ji...@cs.duke.edu>

> Some document here: http://wiki.apache.org/pig/PigIllustrate
>
> I agree that more tests are needed for illustrate, otherwise it can be
> easily broken without notice.
>
> Jie
>
> On Tue, Jul 3, 2012 at 12:45 PM, Jon Coveney <jc...@gmail.com> wrote:
> > I was curious at a level slightly higher than "dig through the code" how
> illustrate is so fast, and how it deals with joins effectively. Are there
> any resources on this (or does anyone at Hortonworks want to write a tech
> oriented blog post? :)
> >
>

Re: Are there any explanations of the implementation of illustrate?

Posted by Jonathan Coveney <jc...@gmail.com>.
Doug,

Would definitely love for you guys to merge in your updates.

Thejas,

Thanks for providing more information. Good to know where to look. Between
the paper and some info on where to look, I have some solid leads!

Jon

2012/7/5 Thejas Nair <th...@hortonworks.com>

> Earlier implementation of illustrate used the pig local mode execution
> engine (which corresponds to the time when paper was published) .
>
> As part of illustrate reword in PIG-1712, Yan replaced the default Map and
> Reduce context objects with a IllustratorContext. Look for
> IllustratorContext and LocalMapReduceSimulator in
> https://issues.apache.org/**jira/secure/attachment/**
> 12459267/illustrator_2.patch<https://issues.apache.org/jira/secure/attachment/12459267/illustrator_2.patch>
> The context objects write their output and read input from memory.
>
> We can consider using this for pig local mode as well, by replacing the in
> memory list with something that can spill to disk.
>
> -Thejas
>
>
>
> On 7/3/12 6:34 PM, Jonathan Coveney wrote:
>
>> Jie, that's perfect, thanks. This doc, specifically:
>> http://i.stanford.edu/~olston/**publications/sigmod09.pdf<http://i.stanford.edu/~olston/publications/sigmod09.pdf>is exactly the
>> detailed explanation I was looking for.
>>
>> 2012/7/3 Jie Li <ji...@cs.duke.edu>
>>
>>  Some document here: http://wiki.apache.org/pig/**PigIllustrate<http://wiki.apache.org/pig/PigIllustrate>
>>>
>>> I agree that more tests are needed for illustrate, otherwise it can be
>>> easily broken without notice.
>>>
>>> Jie
>>>
>>> On Tue, Jul 3, 2012 at 12:45 PM, Jon Coveney <jc...@gmail.com> wrote:
>>>
>>>> I was curious at a level slightly higher than "dig through the code" how
>>>>
>>> illustrate is so fast, and how it deals with joins effectively. Are there
>>> any resources on this (or does anyone at Hortonworks want to write a tech
>>> oriented blog post? :)
>>>
>>>>
>>>>
>>>
>>
>
>

Re: Are there any explanations of the implementation of illustrate?

Posted by Thejas Nair <th...@hortonworks.com>.
Earlier implementation of illustrate used the pig local mode execution 
engine (which corresponds to the time when paper was published) .

As part of illustrate reword in PIG-1712, Yan replaced the default Map 
and Reduce context objects with a IllustratorContext. Look for 
IllustratorContext and LocalMapReduceSimulator in 
https://issues.apache.org/jira/secure/attachment/12459267/illustrator_2.patch
The context objects write their output and read input from memory.

We can consider using this for pig local mode as well, by replacing the 
in memory list with something that can spill to disk.

-Thejas


On 7/3/12 6:34 PM, Jonathan Coveney wrote:
> Jie, that's perfect, thanks. This doc, specifically:
> http://i.stanford.edu/~olston/publications/sigmod09.pdf is exactly the
> detailed explanation I was looking for.
>
> 2012/7/3 Jie Li <ji...@cs.duke.edu>
>
>> Some document here: http://wiki.apache.org/pig/PigIllustrate
>>
>> I agree that more tests are needed for illustrate, otherwise it can be
>> easily broken without notice.
>>
>> Jie
>>
>> On Tue, Jul 3, 2012 at 12:45 PM, Jon Coveney <jc...@gmail.com> wrote:
>>> I was curious at a level slightly higher than "dig through the code" how
>> illustrate is so fast, and how it deals with joins effectively. Are there
>> any resources on this (or does anyone at Hortonworks want to write a tech
>> oriented blog post? :)
>>>
>>
>



Re: Are there any explanations of the implementation of illustrate?

Posted by Jonathan Coveney <jc...@gmail.com>.
Jie, that's perfect, thanks. This doc, specifically:
http://i.stanford.edu/~olston/publications/sigmod09.pdf is exactly the
detailed explanation I was looking for.

2012/7/3 Jie Li <ji...@cs.duke.edu>

> Some document here: http://wiki.apache.org/pig/PigIllustrate
>
> I agree that more tests are needed for illustrate, otherwise it can be
> easily broken without notice.
>
> Jie
>
> On Tue, Jul 3, 2012 at 12:45 PM, Jon Coveney <jc...@gmail.com> wrote:
> > I was curious at a level slightly higher than "dig through the code" how
> illustrate is so fast, and how it deals with joins effectively. Are there
> any resources on this (or does anyone at Hortonworks want to write a tech
> oriented blog post? :)
> >
>

Re: Are there any explanations of the implementation of illustrate?

Posted by Jie Li <ji...@cs.duke.edu>.
Some document here: http://wiki.apache.org/pig/PigIllustrate

I agree that more tests are needed for illustrate, otherwise it can be
easily broken without notice.

Jie

On Tue, Jul 3, 2012 at 12:45 PM, Jon Coveney <jc...@gmail.com> wrote:
> I was curious at a level slightly higher than "dig through the code" how illustrate is so fast, and how it deals with joins effectively. Are there any resources on this (or does anyone at Hortonworks want to write a tech oriented blog post? :)
>