Re: [DISCUSS] Improve Queryable State and introduce a QueryServerProxy component

[Aljoscha Krettek <al...@apache.org>]

Hi Everyone,

I think this is a good discussion and valuable ideas have come up. However, it seems none of the committers and/or PMCs currently have time to work on this subject. Till, who’s focusing on the distributed runtime side, which is touched quite a bit by queryable state, is currently focusing on refactorings that we need for better batch scheduling and resource management, among other things. I and other committers that work more on the API side are focusing on reworking the Table API to separate the concerns and facilitate the incorporation of the new Table API runner that is being developed by contributors at Alibaba.

I’m not saying that you shouldn’t discuss this topic or maybe even develop a proof of concept. It will probably not be added to Flink in the foreseeable future because of lack of committer bandwidth, though. I hope this is not too discouraging.

Aljoscha

> On 30. Apr 2019, at 04:09, vino yang <ya...@gmail.com> wrote:
> 
> Hi Elias,
> 
> OK, I think we do not need to agree on this point of view. In order to make
> the discussion more efficient, we need to focus a bit, let's talk about the
> query architecture's improvement.
> 
> Best,
> Vino
> 
> Elias Levy <fe...@gmail.com> 于2019年4月30日周二 上午1:06写道：
> 
>> On Fri, Apr 26, 2019 at 8:58 PM vino yang <ya...@gmail.com> wrote:
>> 
>>> I agree with your opinion that "*Flink jobs don't sufficiently meet these
>>> requirements to work as a replacement for a data store.*".  Actually, I
>>> think it's obviously not Flink's goal.
>>> 
>> 
>> I would not be so sure.  When data Artisans introduced
>> <https://www.ververica.com/blog/queryable-state-use-case-demo> Queryable
>> State in Flink, one of the use cases was explicitly removing the need for
>> external key-value stores. This mirrored Confluent's earlier
>> <
>> https://www.confluent.io/blog/unifying-stream-processing-and-interactive-queries-in-apache-kafka/
>>> 
>> introduction
>> of Interactive Queries in Kafka Streams, and they certainly saw querying of
>> streaming state as a possible alternative to traditional data stores.
>> 

Re: [DISCUSS] Improve Queryable State and introduce a QueryServerProxy component

[vino yang <ya...@gmail.com>]

Hi Aljoscha,

Thanks for agreeing this is a valuable idea. I know the committers and PMCs
are busy before Flink 1.9 even 2.0. We think it's a good improvement for
queryable state and even some interactive query scenarios.
I don't mind if the timeline will be pulled long.

Although, it could not be added to Flink now. However, IMO, we can do some
work to let it towards the goal. For example,


   - Write a design draft (I want to try this);
   - Discuss and review the design detail;
   - When we agree with the design, we can split it into several subtasks
   (I think there are other contributors want to implement together)

I will try to invite some committers to review the design and give
suggestions. What do you think?

Best,
Vino

Aljoscha Krettek <al...@apache.org> 于2019年5月15日周三 下午4:27写道：

> Hi Everyone,
>
> I think this is a good discussion and valuable ideas have come up.
> However, it seems none of the committers and/or PMCs currently have time to
> work on this subject. Till, who’s focusing on the distributed runtime side,
> which is touched quite a bit by queryable state, is currently focusing on
> refactorings that we need for better batch scheduling and resource
> management, among other things. I and other committers that work more on
> the API side are focusing on reworking the Table API to separate the
> concerns and facilitate the incorporation of the new Table API runner that
> is being developed by contributors at Alibaba.
>
> I’m not saying that you shouldn’t discuss this topic or maybe even develop
> a proof of concept. It will probably not be added to Flink in the
> foreseeable future because of lack of committer bandwidth, though. I hope
> this is not too discouraging.
>
> Aljoscha
>
> > On 30. Apr 2019, at 04:09, vino yang <ya...@gmail.com> wrote:
> >
> > Hi Elias,
> >
> > OK, I think we do not need to agree on this point of view. In order to
> make
> > the discussion more efficient, we need to focus a bit, let's talk about
> the
> > query architecture's improvement.
> >
> > Best,
> > Vino
> >
> > Elias Levy <fe...@gmail.com> 于2019年4月30日周二 上午1:06写道：
> >
> >> On Fri, Apr 26, 2019 at 8:58 PM vino yang <ya...@gmail.com>
> wrote:
> >>
> >>> I agree with your opinion that "*Flink jobs don't sufficiently meet
> these
> >>> requirements to work as a replacement for a data store.*".  Actually, I
> >>> think it's obviously not Flink's goal.
> >>>
> >>
> >> I would not be so sure.  When data Artisans introduced
> >> <https://www.ververica.com/blog/queryable-state-use-case-demo>
> Queryable
> >> State in Flink, one of the use cases was explicitly removing the need
> for
> >> external key-value stores. This mirrored Confluent's earlier
> >> <
> >>
> https://www.confluent.io/blog/unifying-stream-processing-and-interactive-queries-in-apache-kafka/
> >>>
> >> introduction
> >> of Interactive Queries in Kafka Streams, and they certainly saw
> querying of
> >> streaming state as a possible alternative to traditional data stores.
> >>
>
>