DoFN Lamdba

[Jesse Anderson <je...@smokinghand.com>]

Resurrecting a thread from the users list of the same name. I hacked
together an example of what this code could look like. I created a modified
MapElements
<https://github.com/eljefe6a/beamexample/blob/master/DataflowTutorial/src/main/java/com/google/cloud/dataflow/examples/complete/game/utils/MapContextElements.java>
class to pass in the ProcessContext and BoundedWindow.

Here is what the resulting client code looked like
<https://github.com/eljefe6a/beamexample/blob/master/DataflowTutorial/src/main/java/com/google/cloud/dataflow/examples/complete/game/utils/Output.java#L96>
.

I wanted to prototype something to if this code is something others want to
do. I think this sort of functionality needs to be accessible via Lambdas.

I'm not happy that all of the objects are being passed in as a single
object. Now that the new DoFN splits the two objects, I had to include both.

Does anyone else see value in improving this idea before I spend more time
cleaning things up?

Thanks,

Jesse

Re: DoFN Lamdba

[Kenneth Knowles <kl...@google.com.INVALID>]

There are two bits here, I think.

1. "Map" from (InputT, W) -> OutputT makes sense to me. Likewise FlatMap.
2. ParDo.of(<lambda>) with capabilities analogous to DoFn could be useful
in some cases.

I'd start with #1.

Kenn

On Tue, Aug 9, 2016 at 8:23 AM, Lukasz Cwik <lc...@google.com.invalid>
wrote:

> I was going to suggest to dynamically map the parameters to the lambda
> depending on what the lambda is expecting to be called with similar to what
> the new DoFn is doing. Unfortunately this will be limited to concrete types
> which would work for ProcessContext and BoundedWindow but not for other
> envisioned uses which will rely on parameter annotations since lambda
> expressions and annotations still don't play well in JDK 8.
>
> This SO post better explains the issue with lambdas and annotations
> http://stackoverflow.com/questions/22375891/annotating-
> the-functional-interface-of-a-lambda-expression
> Much of the annotations and lambda support is slated for JDK 9
> https://bugs.openjdk.java.net/browse/JDK-8027181.
>
> On Mon, Aug 8, 2016 at 1:18 PM, Jesse Anderson <je...@smokinghand.com>
> wrote:
>
> > Resurrecting a thread from the users list of the same name. I hacked
> > together an example of what this code could look like. I created a
> modified
> > MapElements
> > <https://github.com/eljefe6a/beamexample/blob/master/
> > DataflowTutorial/src/main/java/com/google/cloud/
> > dataflow/examples/complete/game/utils/MapContextElements.java>
> > class to pass in the ProcessContext and BoundedWindow.
> >
> > Here is what the resulting client code looked like
> > <https://github.com/eljefe6a/beamexample/blob/master/
> > DataflowTutorial/src/main/java/com/google/cloud/
> > dataflow/examples/complete/game/utils/Output.java#L96>
> > .
> >
> > I wanted to prototype something to if this code is something others want
> to
> > do. I think this sort of functionality needs to be accessible via
> Lambdas.
> >
> > I'm not happy that all of the objects are being passed in as a single
> > object. Now that the new DoFN splits the two objects, I had to include
> > both.
> >
> > Does anyone else see value in improving this idea before I spend more
> time
> > cleaning things up?
> >
> > Thanks,
> >
> > Jesse
> >
>

Re: DoFN Lamdba

[Lukasz Cwik <lc...@google.com.INVALID>]

I was going to suggest to dynamically map the parameters to the lambda
depending on what the lambda is expecting to be called with similar to what
the new DoFn is doing. Unfortunately this will be limited to concrete types
which would work for ProcessContext and BoundedWindow but not for other
envisioned uses which will rely on parameter annotations since lambda
expressions and annotations still don't play well in JDK 8.

This SO post better explains the issue with lambdas and annotations
http://stackoverflow.com/questions/22375891/annotating-the-functional-interface-of-a-lambda-expression
Much of the annotations and lambda support is slated for JDK 9
https://bugs.openjdk.java.net/browse/JDK-8027181.

On Mon, Aug 8, 2016 at 1:18 PM, Jesse Anderson <je...@smokinghand.com>
wrote:

> Resurrecting a thread from the users list of the same name. I hacked
> together an example of what this code could look like. I created a modified
> MapElements
> <https://github.com/eljefe6a/beamexample/blob/master/
> DataflowTutorial/src/main/java/com/google/cloud/
> dataflow/examples/complete/game/utils/MapContextElements.java>
> class to pass in the ProcessContext and BoundedWindow.
>
> Here is what the resulting client code looked like
> <https://github.com/eljefe6a/beamexample/blob/master/
> DataflowTutorial/src/main/java/com/google/cloud/
> dataflow/examples/complete/game/utils/Output.java#L96>
> .
>
> I wanted to prototype something to if this code is something others want to
> do. I think this sort of functionality needs to be accessible via Lambdas.
>
> I'm not happy that all of the objects are being passed in as a single
> object. Now that the new DoFN splits the two objects, I had to include
> both.
>
> Does anyone else see value in improving this idea before I spend more time
> cleaning things up?
>
> Thanks,
>
> Jesse
>