You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@crunch.apache.org by Chandan Biswas <cb...@gmail.com> on 2014/11/17 21:07:23 UTC

Query on Channels.split

Hi,
Currently Channels.split supports only Pair class. But, in our team we need
Tuple support.  Currently what we are doing is -

PCollection<Pair<Pair<A,B>,C> pair1=someDoFn....
Pair<PCollection<Pair<A,B>>,PCollection<C>> pair1AfterFirstSplit =
Channels.split(pair1);
Pair<PCollection<A>,PCollection<B>> pair1AfterSecondSplit =
Channels.split(pair1.first());

instead we can do if Tuple is supported -

PCollection<Tuple3<A,B,C> firstTuple3=someDoFn....
Tuple3<PCollection<A>,PCollection<B>,PCollection<C>>
firstTuple3AfterSplit=firstTuple3.split(...);

Please let me know is if the functionality is already present in Crunch. If
yes, it will be great if anybody directs me to it.

Following the existing pattern in Channels class, I have written separate
Channels classes for Tuple3 and Tuple4. If it's required I can send it.

Thanks,
Chandan

Re: Query on Channels.split

Posted by Chandan Biswas <cb...@gmail.com>.

Hi Josh,
Here is the code for Tuple3 and Tuple4-
https://gist.github.com/cbiswas1983/7018fda426e8b085c6c7
https://gist.github.com/cbiswas1983/b9140376e702876783b6

If it looks good and you all agree with design then I will write the unit
test and send it to you. If any changes are required then suggest me. I
will be happier to do it.

Thanks,
Chandan


On Mon, Nov 17, 2014 at 5:27 PM, Josh Wills <jo...@gmail.com> wrote:

> Hey Chandan,
>
> No, it doesn't exist right now, please do send it along.
>
> Josh
> On Mon, Nov 17, 2014 at 12:07 PM Chandan Biswas <cb...@gmail.com>
> wrote:
>
> > Hi,
> > Currently Channels.split supports only Pair class. But, in our team we
> need
> > Tuple support.  Currently what we are doing is -
> >
> > PCollection<Pair<Pair<A,B>,C> pair1=someDoFn....
> > Pair<PCollection<Pair<A,B>>,PCollection<C>> pair1AfterFirstSplit =
> > Channels.split(pair1);
> > Pair<PCollection<A>,PCollection<B>> pair1AfterSecondSplit =
> > Channels.split(pair1.first());
> >
> > instead we can do if Tuple is supported -
> >
> > PCollection<Tuple3<A,B,C> firstTuple3=someDoFn....
> > Tuple3<PCollection<A>,PCollection<B>,PCollection<C>>
> > firstTuple3AfterSplit=firstTuple3.split(...);
> >
> > Please let me know is if the functionality is already present in Crunch.
> If
> > yes, it will be great if anybody directs me to it.
> >
> > Following the existing pattern in Channels class, I have written separate
> > Channels classes for Tuple3 and Tuple4. If it's required I can send it.
> >
> > Thanks,
> > Chandan
> >
>

Re: Query on Channels.split

Posted by Josh Wills <jo...@gmail.com>.

Hey Chandan,

No, it doesn't exist right now, please do send it along.

Josh
On Mon, Nov 17, 2014 at 12:07 PM Chandan Biswas <cb...@gmail.com>
wrote:

> Hi,
> Currently Channels.split supports only Pair class. But, in our team we need
> Tuple support.  Currently what we are doing is -
>
> PCollection<Pair<Pair<A,B>,C> pair1=someDoFn....
> Pair<PCollection<Pair<A,B>>,PCollection<C>> pair1AfterFirstSplit =
> Channels.split(pair1);
> Pair<PCollection<A>,PCollection<B>> pair1AfterSecondSplit =
> Channels.split(pair1.first());
>
> instead we can do if Tuple is supported -
>
> PCollection<Tuple3<A,B,C> firstTuple3=someDoFn....
> Tuple3<PCollection<A>,PCollection<B>,PCollection<C>>
> firstTuple3AfterSplit=firstTuple3.split(...);
>
> Please let me know is if the functionality is already present in Crunch. If
> yes, it will be great if anybody directs me to it.
>
> Following the existing pattern in Channels class, I have written separate
> Channels classes for Tuple3 and Tuple4. If it's required I can send it.
>
> Thanks,
> Chandan
>