You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Stephan Ewen <se...@apache.org> on 2014/11/24 19:52:49 UTC
Custom Partitioners Syntax
Hi all!
Custom partitioners allow you manually define the assignment of keys to
partitions, for cases that have special constraints.
This is a call for opinion on the Syntax for custom partitioners, in the
case of Join and CoGroup.
Option 1:
input1
.join(input2)
.where("key1").equalTo("key2")
.withPartitioner( (key, num) -> key % num )
.with( (right, left) -> new Tuple3<>(right.key1, right.name,
left.sizeOfEyebrow) )
Option 2:
input1
.join(input2)
.where("key1").equalTo("key2")
.with( (right, left) -> new Tuple3<>(right.key1, right.name,
left.sizeOfEyebrow) )
.withPartitioner( (key, num) -> key % num )
(The UDF and the partitioner calls are swapped here) (CoGroup would be
analogous)
Greetings,
Stephan
Re: Custom Partitioners Syntax
Posted by Stephan Ewen <se...@apache.org>.
I actually prefer the first one as well...
Am 24.11.2014 20:56 schrieb "Fabian Hueske" <fh...@apache.org>:
> I prefer the first option where partitioning (assigning keys to partitions)
> follows key selection.
>
>
> 2014-11-24 19:52 GMT+01:00 Stephan Ewen <se...@apache.org>:
>
> > Hi all!
> >
> > Custom partitioners allow you manually define the assignment of keys to
> > partitions, for cases that have special constraints.
> >
> > This is a call for opinion on the Syntax for custom partitioners, in the
> > case of Join and CoGroup.
> >
> > Option 1:
> >
> > input1
> > .join(input2)
> > .where("key1").equalTo("key2")
> >
> > .withPartitioner( (key, num) -> key % num )
> >
> > .with( (right, left) -> new Tuple3<>(right.key1, right.name,
> > left.sizeOfEyebrow) )
> >
> > Option 2:
> >
> > input1
> > .join(input2)
> > .where("key1").equalTo("key2")
> > .with( (right, left) -> new Tuple3<>(right.key1, right.name,
> > left.sizeOfEyebrow) )
> >
> > .withPartitioner( (key, num) -> key % num )
> >
> >
> > (The UDF and the partitioner calls are swapped here) (CoGroup would be
> > analogous)
> >
> > Greetings,
> > Stephan
> >
>
Re: Custom Partitioners Syntax
Posted by Stephan Ewen <se...@apache.org>.
Ah, that is actually a good argument...
On Mon, Nov 24, 2014 at 11:50 PM, Till Rohrmann <ti...@gmail.com>
wrote:
> The latter version would allow to use the apply method in Scala
> without calling it directly, whereas in the first case the user would
> have to spell it out.
>
> On Mon, Nov 24, 2014 at 8:56 PM, Fabian Hueske <fh...@apache.org> wrote:
> > I prefer the first option where partitioning (assigning keys to
> partitions)
> > follows key selection.
> >
> >
> > 2014-11-24 19:52 GMT+01:00 Stephan Ewen <se...@apache.org>:
> >
> >> Hi all!
> >>
> >> Custom partitioners allow you manually define the assignment of keys to
> >> partitions, for cases that have special constraints.
> >>
> >> This is a call for opinion on the Syntax for custom partitioners, in the
> >> case of Join and CoGroup.
> >>
> >> Option 1:
> >>
> >> input1
> >> .join(input2)
> >> .where("key1").equalTo("key2")
> >>
> >> .withPartitioner( (key, num) -> key % num )
> >>
> >> .with( (right, left) -> new Tuple3<>(right.key1, right.name,
> >> left.sizeOfEyebrow) )
> >>
> >> Option 2:
> >>
> >> input1
> >> .join(input2)
> >> .where("key1").equalTo("key2")
> >> .with( (right, left) -> new Tuple3<>(right.key1, right.name,
> >> left.sizeOfEyebrow) )
> >>
> >> .withPartitioner( (key, num) -> key % num )
> >>
> >>
> >> (The UDF and the partitioner calls are swapped here) (CoGroup would be
> >> analogous)
> >>
> >> Greetings,
> >> Stephan
> >>
>
Re: Custom Partitioners Syntax
Posted by Till Rohrmann <ti...@gmail.com>.
The latter version would allow to use the apply method in Scala
without calling it directly, whereas in the first case the user would
have to spell it out.
On Mon, Nov 24, 2014 at 8:56 PM, Fabian Hueske <fh...@apache.org> wrote:
> I prefer the first option where partitioning (assigning keys to partitions)
> follows key selection.
>
>
> 2014-11-24 19:52 GMT+01:00 Stephan Ewen <se...@apache.org>:
>
>> Hi all!
>>
>> Custom partitioners allow you manually define the assignment of keys to
>> partitions, for cases that have special constraints.
>>
>> This is a call for opinion on the Syntax for custom partitioners, in the
>> case of Join and CoGroup.
>>
>> Option 1:
>>
>> input1
>> .join(input2)
>> .where("key1").equalTo("key2")
>>
>> .withPartitioner( (key, num) -> key % num )
>>
>> .with( (right, left) -> new Tuple3<>(right.key1, right.name,
>> left.sizeOfEyebrow) )
>>
>> Option 2:
>>
>> input1
>> .join(input2)
>> .where("key1").equalTo("key2")
>> .with( (right, left) -> new Tuple3<>(right.key1, right.name,
>> left.sizeOfEyebrow) )
>>
>> .withPartitioner( (key, num) -> key % num )
>>
>>
>> (The UDF and the partitioner calls are swapped here) (CoGroup would be
>> analogous)
>>
>> Greetings,
>> Stephan
>>
Re: Custom Partitioners Syntax
Posted by Fabian Hueske <fh...@apache.org>.
I prefer the first option where partitioning (assigning keys to partitions)
follows key selection.
2014-11-24 19:52 GMT+01:00 Stephan Ewen <se...@apache.org>:
> Hi all!
>
> Custom partitioners allow you manually define the assignment of keys to
> partitions, for cases that have special constraints.
>
> This is a call for opinion on the Syntax for custom partitioners, in the
> case of Join and CoGroup.
>
> Option 1:
>
> input1
> .join(input2)
> .where("key1").equalTo("key2")
>
> .withPartitioner( (key, num) -> key % num )
>
> .with( (right, left) -> new Tuple3<>(right.key1, right.name,
> left.sizeOfEyebrow) )
>
> Option 2:
>
> input1
> .join(input2)
> .where("key1").equalTo("key2")
> .with( (right, left) -> new Tuple3<>(right.key1, right.name,
> left.sizeOfEyebrow) )
>
> .withPartitioner( (key, num) -> key % num )
>
>
> (The UDF and the partitioner calls are swapped here) (CoGroup would be
> analogous)
>
> Greetings,
> Stephan
>