How to use MapElements.via in a nice way?

[Pawel Szczur <pa...@gmail.com>]

Hi,

I'm using Beam since Google Dataflow has been made public.
There used to be MapFn. Some time later it was removed.

Now we have MapElements helper which kind of is suppose to do similar
things.
Using it with new Java 8 API seems pretty ugly to me, especially when I
have something like:

PCollection<KV<Class1, Class2>> input = ...;
input.apply(
  MapElements.into(*?*).via(v -> KV.of(newKey(v.getKey()), v.getValue()));

I would need to replace *?* by KVCoder.of(KeyCoder.of(), ValueCoder.of()).
This is troublesome especially when in a longer transformation chains.

For me, it's became easier to write:
input.apply(DoFn(new ReplaceKey()))
where ReplaceKey is a implementation of SimpleFunction defined somewhere
and just replacing KV's key.

I'm wondering, am I missing something?

Cheers, Paweł Szczur

Re: How to use MapElements.via in a nice way?

[Kenneth Knowles <kl...@google.com>]

Hi Pawel,

I will ignore your mistakes and just mention that it is a known problem -
the way lambdas are typed in Java 8 is quite annoying for us. That is why
you have to specify the type. But if I recall, this should work to at least
be a little more concise:

   MapElements.via(new SimpleFunction(<lambda>){})

Kenn

On Wed, Apr 25, 2018 at 9:52 AM Pawel Szczur <pa...@gmail.com> wrote:

> This email has at least 3 mistakes, please ignore it. I need to rethink it.
>
> 2018-04-25 18:45 GMT+02:00 Pawel Szczur <pa...@gmail.com>:
>
>> Hi,
>>
>> I'm using Beam since Google Dataflow has been made public.
>> There used to be MapFn. Some time later it was removed.
>>
>> Now we have MapElements helper which kind of is suppose to do similar
>> things.
>> Using it with new Java 8 API seems pretty ugly to me, especially when I
>> have something like:
>>
>> PCollection<KV<Class1, Class2>> input = ...;
>> input.apply(
>>   MapElements.into(*?*).via(v -> KV.of(newKey(v.getKey()),
>> v.getValue()));
>>
>> I would need to replace *?* by KVCoder.of(KeyCoder.of(),
>> ValueCoder.of()).
>> This is troublesome especially when in a longer transformation chains.
>>
>> For me, it's became easier to write:
>> input.apply(DoFn(new ReplaceKey()))
>> where ReplaceKey is a implementation of SimpleFunction defined somewhere
>> and just replacing KV's key.
>>
>> I'm wondering, am I missing something?
>>
>> Cheers, Paweł Szczur
>>
>>
>>
>>
>

Re: How to use MapElements.via in a nice way?

[Pawel Szczur <pa...@gmail.com>]

This email has at least 3 mistakes, please ignore it. I need to rethink it.

2018-04-25 18:45 GMT+02:00 Pawel Szczur <pa...@gmail.com>:

> Hi,
>
> I'm using Beam since Google Dataflow has been made public.
> There used to be MapFn. Some time later it was removed.
>
> Now we have MapElements helper which kind of is suppose to do similar
> things.
> Using it with new Java 8 API seems pretty ugly to me, especially when I
> have something like:
>
> PCollection<KV<Class1, Class2>> input = ...;
> input.apply(
>   MapElements.into(*?*).via(v -> KV.of(newKey(v.getKey()), v.getValue()));
>
> I would need to replace *?* by KVCoder.of(KeyCoder.of(), ValueCoder.of()).
> This is troublesome especially when in a longer transformation chains.
>
> For me, it's became easier to write:
> input.apply(DoFn(new ReplaceKey()))
> where ReplaceKey is a implementation of SimpleFunction defined somewhere
> and just replacing KV's key.
>
> I'm wondering, am I missing something?
>
> Cheers, Paweł Szczur
>
>
>
>