You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Aljoscha Krettek <al...@apache.org> on 2018/08/23 08:12:09 UTC
Flink 1.7 Development Priorities
Hi Everyone,
After the recent Flink 1.6 release the people working on Flink at data Artisans came together to talk about what we want to work on for Flink 1.7. The following is a list of high-level directions that we will be working on for the next couple of months. This doesn't mean that other things are not important or maybe more important, so please chime in.
That being said, here's the high-level list:
- make the Rest API versioned
- provide docker-compose based quickstarts for Flink SQL
- support writing to S3 in the new streaming file sink
- add a new type of join that allows "joining streams with tables"
- Scala 2.12 support
- improvements to resource scheduling, local recovery
- improved support for running Flink in containers, Flink dynamically reacting to changes in the container deployment
- automatic rescaling policies
- initial support for state migration, i.e. changing the schema/TypeSerializer of Flink State
This is also an invitation for others to post what they would like to work on and also to point out glaring omissions.
Best,
Aljoscha
Re: Flink 1.7 Development Priorities
Posted by vino yang <ya...@gmail.com>.
Hi,
I think Kafka 1.0/1.1/2.0 connector can be part of the Flink 1.7 release.
The current kafka 1.0 connector PR has been submitted.
I am refactoring the existing kafka connector test code to reduce the
amount of duplicate code.
Thanks, vino.
Coo1 min <gj...@gmail.com> 于2018年8月23日周四 下午5:59写道：
> Hi,
>
> I am concerned about the progress of CEP library development, Can the
> following two main feature be kicked off and be involed in Flink1.7?
>
> 1) integration of CEP & SQL
> 2) dynamic change of CEP pattern without the downtime
>
> And i am willing to contribute to this, thx.
>
> Aljoscha Krettek <al...@apache.org> 于2018年8月23日周四 下午4:12写道：
>
> > Hi Everyone,
> >
> > After the recent Flink 1.6 release the people working on Flink at data
> > Artisans came together to talk about what we want to work on for Flink
> 1.7.
> > The following is a list of high-level directions that we will be working
> on
> > for the next couple of months. This doesn't mean that other things are
> not
> > important or maybe more important, so please chime in.
> >
> > That being said, here's the high-level list:
> > - make the Rest API versioned
> > - provide docker-compose based quickstarts for Flink SQL
> > - support writing to S3 in the new streaming file sink
> > - add a new type of join that allows "joining streams with tables"
> > - Scala 2.12 support
> > - improvements to resource scheduling, local recovery
> > - improved support for running Flink in containers, Flink dynamically
> > reacting to changes in the container deployment
> > - automatic rescaling policies
> > - initial support for state migration, i.e. changing the
> > schema/TypeSerializer of Flink State
> >
> > This is also an invitation for others to post what they would like to
> work
> > on and also to point out glaring omissions.
> >
> > Best,
> > Aljoscha
>
Re: Flink 1.7 Development Priorities
Posted by Coo1 min <gj...@gmail.com>.
Hi,
I am concerned about the progress of CEP library development, Can the
following two main feature be kicked off and be involed in Flink1.7?
1) integration of CEP & SQL
2) dynamic change of CEP pattern without the downtime
And i am willing to contribute to this, thx.
Aljoscha Krettek <al...@apache.org> 于2018年8月23日周四 下午4:12写道：
> Hi Everyone,
>
> After the recent Flink 1.6 release the people working on Flink at data
> Artisans came together to talk about what we want to work on for Flink 1.7.
> The following is a list of high-level directions that we will be working on
> for the next couple of months. This doesn't mean that other things are not
> important or maybe more important, so please chime in.
>
> That being said, here's the high-level list:
> - make the Rest API versioned
> - provide docker-compose based quickstarts for Flink SQL
> - support writing to S3 in the new streaming file sink
> - add a new type of join that allows "joining streams with tables"
> - Scala 2.12 support
> - improvements to resource scheduling, local recovery
> - improved support for running Flink in containers, Flink dynamically
> reacting to changes in the container deployment
> - automatic rescaling policies
> - initial support for state migration, i.e. changing the
> schema/TypeSerializer of Flink State
>
> This is also an invitation for others to post what they would like to work
> on and also to point out glaring omissions.
>
> Best,
> Aljoscha
Re: Flink 1.7 Development Priorities
Posted by "yzhou@coupang.com" <yz...@coupang.com>.
I would like to have https://issues.apache.org/jira/browse/FLINK-7289 in the
release. It's problematic to control memory usage of rocksdb backend. And
many people have issue on it.
Best
Yan
Aljoscha Krettek-2 wrote
> Hi Everyone,
>
> After the recent Flink 1.6 release the people working on Flink at data
> Artisans came together to talk about what we want to work on for Flink
> 1.7. The following is a list of high-level directions that we will be
> working on for the next couple of months. This doesn't mean that other
> things are not important or maybe more important, so please chime in.
>
> That being said, here's the high-level list:
> - make the Rest API versioned
> - provide docker-compose based quickstarts for Flink SQL
> - support writing to S3 in the new streaming file sink
> - add a new type of join that allows "joining streams with tables"
> - Scala 2.12 support
> - improvements to resource scheduling, local recovery
> - improved support for running Flink in containers, Flink dynamically
> reacting to changes in the container deployment
> - automatic rescaling policies
> - initial support for state migration, i.e. changing the
> schema/TypeSerializer of Flink State
>
> This is also an invitation for others to post what they would like to work
> on and also to point out glaring omissions.
>
> Best,
> Aljoscha
--
Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
Re: Flink 1.7 Development Priorities
Posted by Thomas Weise <th...@apache.org>.
Any plans to advance the "splittable source" idea? It probably won't be
ready to implement, but important to get on the radar.
Thanks,
Thomas
On Thu, Aug 23, 2018 at 1:12 AM Aljoscha Krettek <al...@apache.org>
wrote:
> Hi Everyone,
>
> After the recent Flink 1.6 release the people working on Flink at data
> Artisans came together to talk about what we want to work on for Flink 1.7.
> The following is a list of high-level directions that we will be working on
> for the next couple of months. This doesn't mean that other things are not
> important or maybe more important, so please chime in.
>
> That being said, here's the high-level list:
> - make the Rest API versioned
> - provide docker-compose based quickstarts for Flink SQL
> - support writing to S3 in the new streaming file sink
> - add a new type of join that allows "joining streams with tables"
> - Scala 2.12 support
> - improvements to resource scheduling, local recovery
> - improved support for running Flink in containers, Flink dynamically
> reacting to changes in the container deployment
> - automatic rescaling policies
> - initial support for state migration, i.e. changing the
> schema/TypeSerializer of Flink State
>
> This is also an invitation for others to post what they would like to work
> on and also to point out glaring omissions.
>
> Best,
> Aljoscha
Re: Flink 1.7 Development Priorities
Posted by Renjie Liu <li...@gmail.com>.
Hi:
Can we include task manager isolation for jobs in 1.7.0? This is a feature
that blocks us from deploying flink more widely in our compan?
I've opened issues <https://issues.apache.org/jira/browse/FLINK-9662> for
that and want to help to contribute that, but lacks guidance and reviews
from community.
On Thu, Aug 23, 2018 at 4:12 PM Aljoscha Krettek <al...@apache.org>
wrote:
> Hi Everyone,
>
> After the recent Flink 1.6 release the people working on Flink at data
> Artisans came together to talk about what we want to work on for Flink 1.7.
> The following is a list of high-level directions that we will be working on
> for the next couple of months. This doesn't mean that other things are not
> important or maybe more important, so please chime in.
>
> That being said, here's the high-level list:
> - make the Rest API versioned
> - provide docker-compose based quickstarts for Flink SQL
> - support writing to S3 in the new streaming file sink
> - add a new type of join that allows "joining streams with tables"
> - Scala 2.12 support
> - improvements to resource scheduling, local recovery
> - improved support for running Flink in containers, Flink dynamically
> reacting to changes in the container deployment
> - automatic rescaling policies
> - initial support for state migration, i.e. changing the
> schema/TypeSerializer of Flink State
>
> This is also an invitation for others to post what they would like to work
> on and also to point out glaring omissions.
>
> Best,
> Aljoscha
--
Liu, Renjie
Software Engineer, MVAD
Re: Flink 1.7 Development Priorities
Posted by Elias Levy <fe...@gmail.com>.
I would be nice to see the state TTL work finished. At the moment state
will not expire if a key is not accessed while the job is running, which
means jobs that require state expiration must continue using timers to
expire data.
On Thu, Aug 23, 2018 at 1:12 AM Aljoscha Krettek <al...@apache.org>
wrote:
> Hi Everyone,
>
> After the recent Flink 1.6 release the people working on Flink at data
> Artisans came together to talk about what we want to work on for Flink 1.7.
> The following is a list of high-level directions that we will be working on
> for the next couple of months. This doesn't mean that other things are not
> important or maybe more important, so please chime in.
>
> That being said, here's the high-level list:
> - make the Rest API versioned
> - provide docker-compose based quickstarts for Flink SQL
> - support writing to S3 in the new streaming file sink
> - add a new type of join that allows "joining streams with tables"
> - Scala 2.12 support
> - improvements to resource scheduling, local recovery
> - improved support for running Flink in containers, Flink dynamically
> reacting to changes in the container deployment
> - automatic rescaling policies
> - initial support for state migration, i.e. changing the
> schema/TypeSerializer of Flink State
>
> This is also an invitation for others to post what they would like to work
> on and also to point out glaring omissions.
>
> Best,
> Aljoscha