how to get some configuration in user defined DoFn.setup() function?

[陈竞 <cj...@gmail.com>]

i want to get some configuration in my own DoFn.setup(), but i see that
setup() has no argument, i cant get anything. while in hadoop's api Mapper
has function setup(Context c).

Re: how to get some configuration in user defined DoFn.setup() function?

[Thomas Groh <tg...@google.com>]

We have an outstanding JIRA to add a LifecycleContext parameter (or
something similar) to setup/teardown, but unfortunately I can't link it to
you at the moment because JIRA isn't cooperating with me.

Currently, except for template Pipelines, the recommended way to pass data
to your DoFn is to set it as a property/field in the DoFn and allow Java
Serialization to handle the reconstruction. You can also use the
ValueProvider interface within PipelineOptions to extract an option during
Pipeline execution.

On Wed, Jan 11, 2017 at 4:41 AM, Alexey Demin <di...@gmail.com> wrote:

> Hi
>
> Lifecycle:
>
> 1) DoFn created over new DoFn
> 2) serialization
> 3) transfer to workers
> 4) deserialization
> 5) @Setup (preparing for work)
>
> for every part of data
> 6) @StartBundle (before every bundle)
> 7) @ProcessElement
> 8) @FinishBundle
>
>
> 9) @Teardown (shutdown function)
>
> You can't pass argument in setup() because this function called on remote
> machine after deserialization.
>
> Thanks,
> Alexey Diomin
>
> 2017-01-11 16:20 GMT+04:00 陈竞 <cj...@gmail.com>:
>
>> i want to get some configuration in my own DoFn.setup(), but i see that
>> setup() has no argument, i cant get anything. while in hadoop's api Mapper
>> has function setup(Context c).
>>
>
>

Re: how to get some configuration in user defined DoFn.setup() function?

[Alexey Demin <di...@gmail.com>]

Hi

Lifecycle:

1) DoFn created over new DoFn
2) serialization
3) transfer to workers
4) deserialization
5) @Setup (preparing for work)

for every part of data
6) @StartBundle (before every bundle)
7) @ProcessElement
8) @FinishBundle


9) @Teardown (shutdown function)

You can't pass argument in setup() because this function called on remote
machine after deserialization.

Thanks,
Alexey Diomin

2017-01-11 16:20 GMT+04:00 陈竞 <cj...@gmail.com>:

> i want to get some configuration in my own DoFn.setup(), but i see that
> setup() has no argument, i cant get anything. while in hadoop's api Mapper
> has function setup(Context c).
>