Re: Making broadcast state queryable?

[Oytun Tez <oy...@motaword.com>]

Hi Yu,

Excuse my late reply... We simply want Flink to be our centralized stream
analysis platform, where we 1) consume data, 2) generate analysis, 3)
present analysis. I honestly don't want "stream analysis" to spill out to
other components in our ecosystem (e.g., sinking insights into a DB-like
place).

So the case for QS for us is centralization of input, output, presentation.
State Processor API for instance also counts as a presentation tool for us
(on top of migration tool).

This kind of all-in-one (in, out, ui) packaging helps with small teams.

---
Oytun Tez

*M O T A W O R D*
The World's Fastest Human Translation Platform.
oytun@motaword.com — www.motaword.com


On Wed, Aug 14, 2019 at 3:13 AM Yu Li <ca...@gmail.com> wrote:

> Good to know your thoughts and the coming talk in Flink Forward Berlin
> Oytun, interesting topic and great job! And it's great to hear the voice
> from application perspective.
>
> Could you share, if possible, the reason why you need to open the state to
> outside instead of writing the result to sink and directly query there? In
> another thread there's a case that sink writes to different kafka topics so
> state is the only place to get a global view, is this the same case you're
> facing? Or some different requirements? I believe more attention will be
> drawn to QS if more and more user requirements emerge (smile).
>
> Thanks.
>
> Best Regards,
> Yu
>
>
> On Wed, 14 Aug 2019 at 00:50, Oytun Tez <oy...@motaword.com> wrote:
>
>> Thank you for the honest response, Yu!
>>
>> There is so much that comes to mind when we look at Flink as a
>> "application framework" (my talk
>> <https://europe-2019.flink-forward.org/conference-program#not-so-big-%E2%80%93-flink-as-a-true-application-framework>
>> in Flink Forward in Berlin will be about this). QS is one of them
>> (need-wise, not QS itself necessarily). I opened the design doc Vino Yang
>> created, I'll try to contribute to that.
>>
>> Meanwhile, for the sake of opening our state to outside, we will put a
>> stupid simple operator in between to keep a *duplicate* of the state...
>>
>> Thanks again!
>>
>>
>>
>>
>>
>> ---
>> Oytun Tez
>>
>> *M O T A W O R D*
>> The World's Fastest Human Translation Platform.
>> oytun@motaword.com — www.motaword.com
>>
>>
>> On Tue, Aug 13, 2019 at 6:29 PM Yu Li <ca...@gmail.com> wrote:
>>
>>> Hi Oytun,
>>>
>>> Sorry but TBH such support will probably not be added in the foreseeable
>>> future due to lack of committer bandwidth (not only support queryable
>>> broadcast state but all about QueryableState module) as pointed out in
>>> other threads [1] [2].
>>>
>>> However, I think you could open a JIRA for this so when things changed
>>> it could be taken care of. Thanks.
>>>
>>> [1] https://s.apache.org/MaOl
>>> [2] https://s.apache.org/r8k8a
>>>
>>> Best Regards,
>>> Yu
>>>
>>>
>>> On Tue, 13 Aug 2019 at 20:34, Oytun Tez <oy...@motaword.com> wrote:
>>>
>>>> Hi there,
>>>>
>>>> Can we set a broadcast state as queryable? I've looked around, not much
>>>> to find about it. I am receiving UnknownKvStateLocation when I try to query
>>>> with the descriptor/state name I give to the broadcast state.
>>>>
>>>> If it doesn't work, what could be the alternative? My mind goes around
>>>> ctx.getBroadcastState and making it queryable in the operator level (I'd
>>>> rather not).
>>>>
>>>> ---
>>>> Oytun Tez
>>>>
>>>> *M O T A W O R D*
>>>> The World's Fastest Human Translation Platform.
>>>> oytun@motaword.com — www.motaword.com
>>>>
>>>

Re: Making broadcast state queryable?

[Yu Li <ca...@gmail.com>]

Thanks for the reply Oytun (and sorry for the late response, somehow just
noticed).

Requirement received, interesting one. Let's see whether this could draw
any attention from the committers (smile).

Best Regards,
Yu


On Fri, 6 Sep 2019 at 22:14, Oytun Tez <oy...@motaword.com> wrote:

> Hi Yu,
>
> Excuse my late reply... We simply want Flink to be our centralized stream
> analysis platform, where we 1) consume data, 2) generate analysis, 3)
> present analysis. I honestly don't want "stream analysis" to spill out to
> other components in our ecosystem (e.g., sinking insights into a DB-like
> place).
>
> So the case for QS for us is centralization of input, output,
> presentation. State Processor API for instance also counts as a
> presentation tool for us (on top of migration tool).
>
> This kind of all-in-one (in, out, ui) packaging helps with small teams.
>
> ---
> Oytun Tez
>
> *M O T A W O R D*
> The World's Fastest Human Translation Platform.
> oytun@motaword.com — www.motaword.com
>
>
> On Wed, Aug 14, 2019 at 3:13 AM Yu Li <ca...@gmail.com> wrote:
>
>> Good to know your thoughts and the coming talk in Flink Forward Berlin
>> Oytun, interesting topic and great job! And it's great to hear the voice
>> from application perspective.
>>
>> Could you share, if possible, the reason why you need to open the state
>> to outside instead of writing the result to sink and directly query there?
>> In another thread there's a case that sink writes to different kafka topics
>> so state is the only place to get a global view, is this the same case
>> you're facing? Or some different requirements? I believe more attention
>> will be drawn to QS if more and more user requirements emerge (smile).
>>
>> Thanks.
>>
>> Best Regards,
>> Yu
>>
>>
>> On Wed, 14 Aug 2019 at 00:50, Oytun Tez <oy...@motaword.com> wrote:
>>
>>> Thank you for the honest response, Yu!
>>>
>>> There is so much that comes to mind when we look at Flink as a
>>> "application framework" (my talk
>>> <https://europe-2019.flink-forward.org/conference-program#not-so-big-%E2%80%93-flink-as-a-true-application-framework>
>>> in Flink Forward in Berlin will be about this). QS is one of them
>>> (need-wise, not QS itself necessarily). I opened the design doc Vino Yang
>>> created, I'll try to contribute to that.
>>>
>>> Meanwhile, for the sake of opening our state to outside, we will put a
>>> stupid simple operator in between to keep a *duplicate* of the state...
>>>
>>> Thanks again!
>>>
>>>
>>>
>>>
>>>
>>> ---
>>> Oytun Tez
>>>
>>> *M O T A W O R D*
>>> The World's Fastest Human Translation Platform.
>>> oytun@motaword.com — www.motaword.com
>>>
>>>
>>> On Tue, Aug 13, 2019 at 6:29 PM Yu Li <ca...@gmail.com> wrote:
>>>
>>>> Hi Oytun,
>>>>
>>>> Sorry but TBH such support will probably not be added in the
>>>> foreseeable future due to lack of committer bandwidth (not only support
>>>> queryable broadcast state but all about QueryableState module) as pointed
>>>> out in other threads [1] [2].
>>>>
>>>> However, I think you could open a JIRA for this so when things changed
>>>> it could be taken care of. Thanks.
>>>>
>>>> [1] https://s.apache.org/MaOl
>>>> [2] https://s.apache.org/r8k8a
>>>>
>>>> Best Regards,
>>>> Yu
>>>>
>>>>
>>>> On Tue, 13 Aug 2019 at 20:34, Oytun Tez <oy...@motaword.com> wrote:
>>>>
>>>>> Hi there,
>>>>>
>>>>> Can we set a broadcast state as queryable? I've looked around, not
>>>>> much to find about it. I am receiving UnknownKvStateLocation when I try to
>>>>> query with the descriptor/state name I give to the broadcast state.
>>>>>
>>>>> If it doesn't work, what could be the alternative? My mind goes around
>>>>> ctx.getBroadcastState and making it queryable in the operator level (I'd
>>>>> rather not).
>>>>>
>>>>> ---
>>>>> Oytun Tez
>>>>>
>>>>> *M O T A W O R D*
>>>>> The World's Fastest Human Translation Platform.
>>>>> oytun@motaword.com — www.motaword.com
>>>>>
>>>>