You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by "Aparup Banerjee (apbanerj)" <ap...@cisco.com> on 2016/07/25 15:04:23 UTC
Beam/Flink : State access
I am looking for a way to access streaming engine state (Flink) in my beam transformations. I understand this can be accessed only though the runtime context. Has any one tried accessing flink runtime context in beam? May be roll it up as a custom API of some sort. Might need some changes in FlinkRunner is my hunch.
Thanks,
Aparup
Re: Beam/Flink : State access
Posted by Lukasz Cwik <lc...@google.com.INVALID>.
There is no current state API supported for users, its in
design/development. BEAM-25/dev mailing list should be updated with details
soon.
On Tue, Jul 26, 2016 at 10:51 AM, Aparup Banerjee (apbanerj) <
apbanerj@cisco.com> wrote:
> Ok can you also point me to the current state API? I can’t seem to find it
>
>
>
>
> On 7/26/16, 7:15 AM, "Aparup Banerjee (apbanerj)" <ap...@cisco.com>
> wrote:
>
> >Yes, that’s what I meant. Ok I will go through it.
> >
> >
> >
> >
> >On 7/26/16, 6:17 AM, "Lukasz Cwik" <lc...@google.com.INVALID> wrote:
> >
> >>I think your interested in https://issues.apache.org/jira/browse/BEAM-25
> >>which is about exposing access to state in a runner independent way in
> the
> >>Beam model. You can watch that issue and comment on design / development
> as
> >>it is currently being worked on.
> >>
> >>On Tue, Jul 26, 2016 at 8:53 AM, Aparup Banerjee (apbanerj) <
> >>apbanerj@cisco.com> wrote:
> >>
> >>> Yes and no, I think state access is a valid use cases , and should
> surface
> >>> in beam layer , through the sdk abstraction, or through a separate
> function
> >>> like RichMapFunction of Flink. Hence the question.
> >>>
> >>> Aparup
> >>>
> >>>
> >>>
> >>>
> >>> On 7/26/16, 4:29 AM, "Aljoscha Krettek" <al...@apache.org> wrote:
> >>>
> >>> >Hi,
> >>> >the purpose of Beam is to abstract the user from the underlying
> execution
> >>> >engine. IMHO, allowing access to state of the underlying execution
> engine
> >>> >will never be a goal for the Beam project.
> >>> >
> >>> >If you want/need to access Flink state, I think this is a good
> indicator
> >>> >that you should use Flink directly because your programs would never
> run
> >>> on
> >>> >another runner anyways.
> >>> >
> >>> >Cheers,
> >>> >Aljoscha
> >>> >
> >>> >On Mon, 25 Jul 2016 at 17:04 Aparup Banerjee (apbanerj) <
> >>> apbanerj@cisco.com>
> >>> >wrote:
> >>> >
> >>> >> I am looking for a way to access streaming engine state (Flink) in
> my
> >>> beam
> >>> >> transformations. I understand this can be accessed only though the
> >>> runtime
> >>> >> context. Has any one tried accessing flink runtime context in beam?
> May
> >>> be
> >>> >> roll it up as a custom API of some sort. Might need some changes in
> >>> >> FlinkRunner is my hunch.
> >>> >>
> >>> >> Thanks,
> >>> >> Aparup
> >>> >>
> >>>
>
Re: Beam/Flink : State access
Posted by "Aparup Banerjee (apbanerj)" <ap...@cisco.com>.
Ok can you also point me to the current state API? I can’t seem to find it
On 7/26/16, 7:15 AM, "Aparup Banerjee (apbanerj)" <ap...@cisco.com> wrote:
>Yes, that’s what I meant. Ok I will go through it.
>
>
>
>
>On 7/26/16, 6:17 AM, "Lukasz Cwik" <lc...@google.com.INVALID> wrote:
>
>>I think your interested in https://issues.apache.org/jira/browse/BEAM-25
>>which is about exposing access to state in a runner independent way in the
>>Beam model. You can watch that issue and comment on design / development as
>>it is currently being worked on.
>>
>>On Tue, Jul 26, 2016 at 8:53 AM, Aparup Banerjee (apbanerj) <
>>apbanerj@cisco.com> wrote:
>>
>>> Yes and no, I think state access is a valid use cases , and should surface
>>> in beam layer , through the sdk abstraction, or through a separate function
>>> like RichMapFunction of Flink. Hence the question.
>>>
>>> Aparup
>>>
>>>
>>>
>>>
>>> On 7/26/16, 4:29 AM, "Aljoscha Krettek" <al...@apache.org> wrote:
>>>
>>> >Hi,
>>> >the purpose of Beam is to abstract the user from the underlying execution
>>> >engine. IMHO, allowing access to state of the underlying execution engine
>>> >will never be a goal for the Beam project.
>>> >
>>> >If you want/need to access Flink state, I think this is a good indicator
>>> >that you should use Flink directly because your programs would never run
>>> on
>>> >another runner anyways.
>>> >
>>> >Cheers,
>>> >Aljoscha
>>> >
>>> >On Mon, 25 Jul 2016 at 17:04 Aparup Banerjee (apbanerj) <
>>> apbanerj@cisco.com>
>>> >wrote:
>>> >
>>> >> I am looking for a way to access streaming engine state (Flink) in my
>>> beam
>>> >> transformations. I understand this can be accessed only though the
>>> runtime
>>> >> context. Has any one tried accessing flink runtime context in beam? May
>>> be
>>> >> roll it up as a custom API of some sort. Might need some changes in
>>> >> FlinkRunner is my hunch.
>>> >>
>>> >> Thanks,
>>> >> Aparup
>>> >>
>>>
Re: Beam/Flink : State access
Posted by "Aparup Banerjee (apbanerj)" <ap...@cisco.com>.
Yes, that’s what I meant. Ok I will go through it.
On 7/26/16, 6:17 AM, "Lukasz Cwik" <lc...@google.com.INVALID> wrote:
>I think your interested in https://issues.apache.org/jira/browse/BEAM-25
>which is about exposing access to state in a runner independent way in the
>Beam model. You can watch that issue and comment on design / development as
>it is currently being worked on.
>
>On Tue, Jul 26, 2016 at 8:53 AM, Aparup Banerjee (apbanerj) <
>apbanerj@cisco.com> wrote:
>
>> Yes and no, I think state access is a valid use cases , and should surface
>> in beam layer , through the sdk abstraction, or through a separate function
>> like RichMapFunction of Flink. Hence the question.
>>
>> Aparup
>>
>>
>>
>>
>> On 7/26/16, 4:29 AM, "Aljoscha Krettek" <al...@apache.org> wrote:
>>
>> >Hi,
>> >the purpose of Beam is to abstract the user from the underlying execution
>> >engine. IMHO, allowing access to state of the underlying execution engine
>> >will never be a goal for the Beam project.
>> >
>> >If you want/need to access Flink state, I think this is a good indicator
>> >that you should use Flink directly because your programs would never run
>> on
>> >another runner anyways.
>> >
>> >Cheers,
>> >Aljoscha
>> >
>> >On Mon, 25 Jul 2016 at 17:04 Aparup Banerjee (apbanerj) <
>> apbanerj@cisco.com>
>> >wrote:
>> >
>> >> I am looking for a way to access streaming engine state (Flink) in my
>> beam
>> >> transformations. I understand this can be accessed only though the
>> runtime
>> >> context. Has any one tried accessing flink runtime context in beam? May
>> be
>> >> roll it up as a custom API of some sort. Might need some changes in
>> >> FlinkRunner is my hunch.
>> >>
>> >> Thanks,
>> >> Aparup
>> >>
>>
Re: Beam/Flink : State access
Posted by Lukasz Cwik <lc...@google.com.INVALID>.
I think your interested in https://issues.apache.org/jira/browse/BEAM-25
which is about exposing access to state in a runner independent way in the
Beam model. You can watch that issue and comment on design / development as
it is currently being worked on.
On Tue, Jul 26, 2016 at 8:53 AM, Aparup Banerjee (apbanerj) <
apbanerj@cisco.com> wrote:
> Yes and no, I think state access is a valid use cases , and should surface
> in beam layer , through the sdk abstraction, or through a separate function
> like RichMapFunction of Flink. Hence the question.
>
> Aparup
>
>
>
>
> On 7/26/16, 4:29 AM, "Aljoscha Krettek" <al...@apache.org> wrote:
>
> >Hi,
> >the purpose of Beam is to abstract the user from the underlying execution
> >engine. IMHO, allowing access to state of the underlying execution engine
> >will never be a goal for the Beam project.
> >
> >If you want/need to access Flink state, I think this is a good indicator
> >that you should use Flink directly because your programs would never run
> on
> >another runner anyways.
> >
> >Cheers,
> >Aljoscha
> >
> >On Mon, 25 Jul 2016 at 17:04 Aparup Banerjee (apbanerj) <
> apbanerj@cisco.com>
> >wrote:
> >
> >> I am looking for a way to access streaming engine state (Flink) in my
> beam
> >> transformations. I understand this can be accessed only though the
> runtime
> >> context. Has any one tried accessing flink runtime context in beam? May
> be
> >> roll it up as a custom API of some sort. Might need some changes in
> >> FlinkRunner is my hunch.
> >>
> >> Thanks,
> >> Aparup
> >>
>
Re: Beam/Flink : State access
Posted by "Aparup Banerjee (apbanerj)" <ap...@cisco.com>.
Yes and no, I think state access is a valid use cases , and should surface in beam layer , through the sdk abstraction, or through a separate function like RichMapFunction of Flink. Hence the question.
Aparup
On 7/26/16, 4:29 AM, "Aljoscha Krettek" <al...@apache.org> wrote:
>Hi,
>the purpose of Beam is to abstract the user from the underlying execution
>engine. IMHO, allowing access to state of the underlying execution engine
>will never be a goal for the Beam project.
>
>If you want/need to access Flink state, I think this is a good indicator
>that you should use Flink directly because your programs would never run on
>another runner anyways.
>
>Cheers,
>Aljoscha
>
>On Mon, 25 Jul 2016 at 17:04 Aparup Banerjee (apbanerj) <ap...@cisco.com>
>wrote:
>
>> I am looking for a way to access streaming engine state (Flink) in my beam
>> transformations. I understand this can be accessed only though the runtime
>> context. Has any one tried accessing flink runtime context in beam? May be
>> roll it up as a custom API of some sort. Might need some changes in
>> FlinkRunner is my hunch.
>>
>> Thanks,
>> Aparup
>>
Re: Beam/Flink : State access
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
the purpose of Beam is to abstract the user from the underlying execution
engine. IMHO, allowing access to state of the underlying execution engine
will never be a goal for the Beam project.
If you want/need to access Flink state, I think this is a good indicator
that you should use Flink directly because your programs would never run on
another runner anyways.
Cheers,
Aljoscha
On Mon, 25 Jul 2016 at 17:04 Aparup Banerjee (apbanerj) <ap...@cisco.com>
wrote:
> I am looking for a way to access streaming engine state (Flink) in my beam
> transformations. I understand this can be accessed only though the runtime
> context. Has any one tried accessing flink runtime context in beam? May be
> roll it up as a custom API of some sort. Might need some changes in
> FlinkRunner is my hunch.
>
> Thanks,
> Aparup
>