You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Rajesh Balamohan <ra...@gmail.com> on 2010/12/14 02:04:34 UTC
How do I provide > filter during cogroup
Hi Folks,
I have 2 datasets (T1, T2) to be joinned.
I need to join T1 with T2 based on some criteria. COGROUP does it based on
== condition.
ex: COGROUP T1 by f1, T2 by f2 (but I need to filter T2.f2 > T1.f1).
Is there a way to specify such conditions in PIG?.
~Rajesh.B
Re: How do I provide > filter during cogroup
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
Yes it's tough, and no it's not common :)
Scale brings limitations...
On Tue, Dec 14, 2010 at 4:05 AM, Rajesh Balamohan
<ra...@gmail.com> wrote:
> Thanks for the quick reply Dmitriy. Does it mean that its tough to have
> non-equi join type of joins between 2 datasets in PIG? Isn't it a common
> scenario in production systems?
>
> On Tue, Dec 14, 2010 at 6:59 AM, Dmitriy Ryaboy <dv...@gmail.com> wrote:
>
>> Rajesh, that's not a map-reduce friendly computation, as it is essentially
>> a
>> cross.
>> Which is how you would implement something like this -- and it would be
>> awfully slow or just not computable for very large datasets: cross, then
>> filter.
>>
>> -Dmitriy
>>
>> On Mon, Dec 13, 2010 at 5:04 PM, Rajesh Balamohan <
>> rajesh.balamohan@gmail.com> wrote:
>>
>> > Hi Folks,
>> >
>> > I have 2 datasets (T1, T2) to be joinned.
>> >
>> > I need to join T1 with T2 based on some criteria. COGROUP does it based
>> on
>> > == condition.
>> >
>> > ex: COGROUP T1 by f1, T2 by f2 (but I need to filter T2.f2 > T1.f1).
>> >
>> > Is there a way to specify such conditions in PIG?.
>> >
>> > ~Rajesh.B
>> >
>>
>
>
>
> --
> ~Rajesh.B
>
Re: How do I provide > filter during cogroup
Posted by Rajesh Balamohan <ra...@gmail.com>.
Thanks for the quick reply Dmitriy. Does it mean that its tough to have
non-equi join type of joins between 2 datasets in PIG? Isn't it a common
scenario in production systems?
On Tue, Dec 14, 2010 at 6:59 AM, Dmitriy Ryaboy <dv...@gmail.com> wrote:
> Rajesh, that's not a map-reduce friendly computation, as it is essentially
> a
> cross.
> Which is how you would implement something like this -- and it would be
> awfully slow or just not computable for very large datasets: cross, then
> filter.
>
> -Dmitriy
>
> On Mon, Dec 13, 2010 at 5:04 PM, Rajesh Balamohan <
> rajesh.balamohan@gmail.com> wrote:
>
> > Hi Folks,
> >
> > I have 2 datasets (T1, T2) to be joinned.
> >
> > I need to join T1 with T2 based on some criteria. COGROUP does it based
> on
> > == condition.
> >
> > ex: COGROUP T1 by f1, T2 by f2 (but I need to filter T2.f2 > T1.f1).
> >
> > Is there a way to specify such conditions in PIG?.
> >
> > ~Rajesh.B
> >
>
--
~Rajesh.B
Re: How do I provide > filter during cogroup
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
Rajesh, that's not a map-reduce friendly computation, as it is essentially a
cross.
Which is how you would implement something like this -- and it would be
awfully slow or just not computable for very large datasets: cross, then
filter.
-Dmitriy
On Mon, Dec 13, 2010 at 5:04 PM, Rajesh Balamohan <
rajesh.balamohan@gmail.com> wrote:
> Hi Folks,
>
> I have 2 datasets (T1, T2) to be joinned.
>
> I need to join T1 with T2 based on some criteria. COGROUP does it based on
> == condition.
>
> ex: COGROUP T1 by f1, T2 by f2 (but I need to filter T2.f2 > T1.f1).
>
> Is there a way to specify such conditions in PIG?.
>
> ~Rajesh.B
>