You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Stephan Ewen <se...@apache.org> on 2017/07/12 14:43:21 UTC
[POLL] Dropping savepoint format compatibility for 1.1.x in the Flink
1.4.0 release
Hi users!
Flink currently maintains backwards compatibility for savepoint formats,
which means that savepoints taken with Flink version 1.1.x and 1.2.x can be
resumed in Flink 1.3.x
We are discussing how many versions back to support. The proposition is the
following:
* Suggestion: Flink 1.4.0 will be able to resume savepoints taken with
version 1.3.x and 1.2.x, but not savepoints from version 1.1.x and 1.0.x*
The reason for that is that there is a lot of code mapping between the
completely different legacy format (1.1.x, not re-scalable) and the
key-group-oriented format (1.2.x onwards, re-scalable). It would greatly
help the development of state and checkpointing features to drop that old
code.
Please let us know if you have concerns about that.
Best,
Stephan
Re: [POLL] Dropping savepoint format compatibility for 1.1.x in the
Flink 1.4.0 release
Posted by Ufuk Celebi <uc...@apache.org>.
On Fri, Jul 28, 2017 at 4:03 PM, Stephan Ewen <se...@apache.org> wrote:
> Seems like no one raised a concern so far about dropping the savepoint
> format compatibility for 1.1 in 1.4.
>
> Leaving this thread open for some more days, but from the sentiment, it
> seems like we should go ahead?
+1
Re: [POLL] Dropping savepoint format compatibility for 1.1.x in the
Flink 1.4.0 release
Posted by Stephan Ewen <se...@apache.org>.
@Greg - One benefit I can clearly see is the following:
If we keep that old 1.1-style state code, than we want to guarantee its
correctness in the face of the changes that have been made (consolidate
state code to be per-operator rather than per-task in the runtime as well)
and the changes that are WIP, for example for state evolution (like eager
state) or a better failover (don't reload state from DFS if the node did
not actually crash).
Guaranteeing that correctness is a lot of work. I think that not ensuring
the correctness as thoroughly would be worse that removing that code.
The CEP library had not really reached stable state in 1.2 anyways, it was
broken and reworked to introduce features like 'loops' or quantifiers. So
on that side, there would be no 1.2 compatibility anyways.
Do you see any concrete case were dropping 1.1 compatibility breaks any
setups? I personally know no user that is still on 1.1, most were very
eager about 1.2 due to rescalable state.
As someone that has also worked on the state code, I completely understand
Stefan's desire to simplify things there - it really slows down
developments in that component big time right now...
On Thu, Aug 17, 2017 at 1:34 PM, Stefan Richter <s.richter@data-artisans.com
> wrote:
> I think we are still doing changes for which this is relevant. Also I
> cannot really see a benefit in delaying this because the whole discussion
> will apply in exactly the same way to 1.5.
>
> > Am 17.08.2017 um 13:29 schrieb Greg Hogan <co...@greghogan.com>:
> >
> > There’s an argument for delaying this change to 1.5 since the feature
> freeze is two weeks away. There is little time to realize benefits from
> removing this code.
> >
> > "The reason for that is that there is a lot of code mapping between the
> completely different legacy format (1.1.x, not re-scalable) and the
> key-group-oriented format (1.2.x onwards, re-scalable). It would greatly
> help the development of state and checkpointing features to drop that old
> code.”
> >
> > Greg
> >
> >
> >> On Aug 17, 2017, at 5:36 AM, Stefan Richter <
> s.richter@data-artisans.com> wrote:
> >>
> >> One more comment about the consequences of this PR, as pointed out in
> the comments on Github: this will also break direct compatibility for the
> CEP library between Flink 1.2 and 1.4. There is still a way to migrate via
> Flink 1.3: Flink 1.1/2 -> savepoint -> Flink 1.3 -> savepoint -> Flink 1.4.
> >>
> >>> Am 16.08.2017 um 17:31 schrieb Stefan Richter <
> s.richter@data-artisans.com>:
> >>>
> >>> Hi,
> >>>
> >>> after there have been no objections since a long time, I took the next
> step and created a PR that implements this change in commit
> 95e44099784c9deaf2ca422b8dfc11c3d67d7f82 of https://github.com/apache/
> flink/pull/4550 <https://github.com/apache/flink/pull/4550> . Announcing
> this here as a last opportunity for further discussions. FYI, this will
> decrease the code base by almost 12K LOC.
> >>>
> >>> Best,
> >>> Stefan
> >>>
> >>>
> >>>> Am 02.08.2017 um 15:26 schrieb Kostas Kloudas <
> k.kloudas@data-artisans.com <ma...@data-artisans.com>>:
> >>>>
> >>>> +1
> >>>>
> >>>>> On Aug 2, 2017, at 3:16 PM, Till Rohrmann <trohrmann@apache.org
> <ma...@apache.org>> wrote:
> >>>>>
> >>>>> +1
> >>>>>
> >>>>> On Wed, Aug 2, 2017 at 9:12 AM, Stefan Richter <
> s.richter@data-artisans.com <ma...@data-artisans.com>>
> >>>>> wrote:
> >>>>>
> >>>>>> +1
> >>>>>>
> >>>>>> Am 28.07.2017 um 16:03 schrieb Stephan Ewen <sewen@apache.org
> <ma...@apache.org>>:
> >>>>>>
> >>>>>> Seems like no one raised a concern so far about dropping the
> savepoint
> >>>>>> format compatibility for 1.1 in 1.4.
> >>>>>>
> >>>>>> Leaving this thread open for some more days, but from the
> sentiment, it
> >>>>>> seems like we should go ahead?
> >>>>>>
> >>>>>> On Wed, Jul 12, 2017 at 4:43 PM, Stephan Ewen <sewen@apache.org
> <ma...@apache.org>> wrote:
> >>>>>>
> >>>>>>> Hi users!
> >>>>>>>
> >>>>>>> Flink currently maintains backwards compatibility for savepoint
> formats,
> >>>>>>> which means that savepoints taken with Flink version 1.1.x and
> 1.2.x can be
> >>>>>>> resumed in Flink 1.3.x
> >>>>>>>
> >>>>>>> We are discussing how many versions back to support. The
> proposition is
> >>>>>>> the following:
> >>>>>>>
> >>>>>>> * Suggestion: Flink 1.4.0 will be able to resume savepoints
> taken with
> >>>>>>> version 1.3.x and 1.2.x, but not savepoints from version 1.1.x and
> 1.0.x*
> >>>>>>>
> >>>>>>>
> >>>>>>> The reason for that is that there is a lot of code mapping between
> the
> >>>>>>> completely different legacy format (1.1.x, not re-scalable) and the
> >>>>>>> key-group-oriented format (1.2.x onwards, re-scalable). It would
> greatly
> >>>>>>> help the development of state and checkpointing features to drop
> that old
> >>>>>>> code.
> >>>>>>>
> >>>>>>> Please let us know if you have concerns about that.
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Stephan
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>
> >>
> >
>
>
Re: [POLL] Dropping savepoint format compatibility for 1.1.x in the
Flink 1.4.0 release
Posted by Stefan Richter <s....@data-artisans.com>.
I think we are still doing changes for which this is relevant. Also I cannot really see a benefit in delaying this because the whole discussion will apply in exactly the same way to 1.5.
> Am 17.08.2017 um 13:29 schrieb Greg Hogan <co...@greghogan.com>:
>
> There’s an argument for delaying this change to 1.5 since the feature freeze is two weeks away. There is little time to realize benefits from removing this code.
>
> "The reason for that is that there is a lot of code mapping between the completely different legacy format (1.1.x, not re-scalable) and the key-group-oriented format (1.2.x onwards, re-scalable). It would greatly help the development of state and checkpointing features to drop that old code.”
>
> Greg
>
>
>> On Aug 17, 2017, at 5:36 AM, Stefan Richter <s....@data-artisans.com> wrote:
>>
>> One more comment about the consequences of this PR, as pointed out in the comments on Github: this will also break direct compatibility for the CEP library between Flink 1.2 and 1.4. There is still a way to migrate via Flink 1.3: Flink 1.1/2 -> savepoint -> Flink 1.3 -> savepoint -> Flink 1.4.
>>
>>> Am 16.08.2017 um 17:31 schrieb Stefan Richter <s....@data-artisans.com>:
>>>
>>> Hi,
>>>
>>> after there have been no objections since a long time, I took the next step and created a PR that implements this change in commit 95e44099784c9deaf2ca422b8dfc11c3d67d7f82 of https://github.com/apache/flink/pull/4550 <https://github.com/apache/flink/pull/4550> . Announcing this here as a last opportunity for further discussions. FYI, this will decrease the code base by almost 12K LOC.
>>>
>>> Best,
>>> Stefan
>>>
>>>
>>>> Am 02.08.2017 um 15:26 schrieb Kostas Kloudas <k.kloudas@data-artisans.com <ma...@data-artisans.com>>:
>>>>
>>>> +1
>>>>
>>>>> On Aug 2, 2017, at 3:16 PM, Till Rohrmann <trohrmann@apache.org <ma...@apache.org>> wrote:
>>>>>
>>>>> +1
>>>>>
>>>>> On Wed, Aug 2, 2017 at 9:12 AM, Stefan Richter <s.richter@data-artisans.com <ma...@data-artisans.com>>
>>>>> wrote:
>>>>>
>>>>>> +1
>>>>>>
>>>>>> Am 28.07.2017 um 16:03 schrieb Stephan Ewen <sewen@apache.org <ma...@apache.org>>:
>>>>>>
>>>>>> Seems like no one raised a concern so far about dropping the savepoint
>>>>>> format compatibility for 1.1 in 1.4.
>>>>>>
>>>>>> Leaving this thread open for some more days, but from the sentiment, it
>>>>>> seems like we should go ahead?
>>>>>>
>>>>>> On Wed, Jul 12, 2017 at 4:43 PM, Stephan Ewen <sewen@apache.org <ma...@apache.org>> wrote:
>>>>>>
>>>>>>> Hi users!
>>>>>>>
>>>>>>> Flink currently maintains backwards compatibility for savepoint formats,
>>>>>>> which means that savepoints taken with Flink version 1.1.x and 1.2.x can be
>>>>>>> resumed in Flink 1.3.x
>>>>>>>
>>>>>>> We are discussing how many versions back to support. The proposition is
>>>>>>> the following:
>>>>>>>
>>>>>>> * Suggestion: Flink 1.4.0 will be able to resume savepoints taken with
>>>>>>> version 1.3.x and 1.2.x, but not savepoints from version 1.1.x and 1.0.x*
>>>>>>>
>>>>>>>
>>>>>>> The reason for that is that there is a lot of code mapping between the
>>>>>>> completely different legacy format (1.1.x, not re-scalable) and the
>>>>>>> key-group-oriented format (1.2.x onwards, re-scalable). It would greatly
>>>>>>> help the development of state and checkpointing features to drop that old
>>>>>>> code.
>>>>>>>
>>>>>>> Please let us know if you have concerns about that.
>>>>>>>
>>>>>>> Best,
>>>>>>> Stephan
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>
>>
>
Re: [POLL] Dropping savepoint format compatibility for 1.1.x in the
Flink 1.4.0 release
Posted by Greg Hogan <co...@greghogan.com>.
There’s an argument for delaying this change to 1.5 since the feature freeze is two weeks away. There is little time to realize benefits from removing this code.
"The reason for that is that there is a lot of code mapping between the completely different legacy format (1.1.x, not re-scalable) and the key-group-oriented format (1.2.x onwards, re-scalable). It would greatly help the development of state and checkpointing features to drop that old code.”
Greg
> On Aug 17, 2017, at 5:36 AM, Stefan Richter <s....@data-artisans.com> wrote:
>
> One more comment about the consequences of this PR, as pointed out in the comments on Github: this will also break direct compatibility for the CEP library between Flink 1.2 and 1.4. There is still a way to migrate via Flink 1.3: Flink 1.1/2 -> savepoint -> Flink 1.3 -> savepoint -> Flink 1.4.
>
>> Am 16.08.2017 um 17:31 schrieb Stefan Richter <s....@data-artisans.com>:
>>
>> Hi,
>>
>> after there have been no objections since a long time, I took the next step and created a PR that implements this change in commit 95e44099784c9deaf2ca422b8dfc11c3d67d7f82 of https://github.com/apache/flink/pull/4550 <https://github.com/apache/flink/pull/4550> . Announcing this here as a last opportunity for further discussions. FYI, this will decrease the code base by almost 12K LOC.
>>
>> Best,
>> Stefan
>>
>>
>>> Am 02.08.2017 um 15:26 schrieb Kostas Kloudas <k.kloudas@data-artisans.com <ma...@data-artisans.com>>:
>>>
>>> +1
>>>
>>>> On Aug 2, 2017, at 3:16 PM, Till Rohrmann <trohrmann@apache.org <ma...@apache.org>> wrote:
>>>>
>>>> +1
>>>>
>>>> On Wed, Aug 2, 2017 at 9:12 AM, Stefan Richter <s.richter@data-artisans.com <ma...@data-artisans.com>>
>>>> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> Am 28.07.2017 um 16:03 schrieb Stephan Ewen <sewen@apache.org <ma...@apache.org>>:
>>>>>
>>>>> Seems like no one raised a concern so far about dropping the savepoint
>>>>> format compatibility for 1.1 in 1.4.
>>>>>
>>>>> Leaving this thread open for some more days, but from the sentiment, it
>>>>> seems like we should go ahead?
>>>>>
>>>>> On Wed, Jul 12, 2017 at 4:43 PM, Stephan Ewen <sewen@apache.org <ma...@apache.org>> wrote:
>>>>>
>>>>>> Hi users!
>>>>>>
>>>>>> Flink currently maintains backwards compatibility for savepoint formats,
>>>>>> which means that savepoints taken with Flink version 1.1.x and 1.2.x can be
>>>>>> resumed in Flink 1.3.x
>>>>>>
>>>>>> We are discussing how many versions back to support. The proposition is
>>>>>> the following:
>>>>>>
>>>>>> * Suggestion: Flink 1.4.0 will be able to resume savepoints taken with
>>>>>> version 1.3.x and 1.2.x, but not savepoints from version 1.1.x and 1.0.x*
>>>>>>
>>>>>>
>>>>>> The reason for that is that there is a lot of code mapping between the
>>>>>> completely different legacy format (1.1.x, not re-scalable) and the
>>>>>> key-group-oriented format (1.2.x onwards, re-scalable). It would greatly
>>>>>> help the development of state and checkpointing features to drop that old
>>>>>> code.
>>>>>>
>>>>>> Please let us know if you have concerns about that.
>>>>>>
>>>>>> Best,
>>>>>> Stephan
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>
>>
>
Re: [POLL] Dropping savepoint format compatibility for 1.1.x in the
Flink 1.4.0 release
Posted by Stefan Richter <s....@data-artisans.com>.
One more comment about the consequences of this PR, as pointed out in the comments on Github: this will also break direct compatibility for the CEP library between Flink 1.2 and 1.4. There is still a way to migrate via Flink 1.3: Flink 1.1/2 -> savepoint -> Flink 1.3 -> savepoint -> Flink 1.4.
> Am 16.08.2017 um 17:31 schrieb Stefan Richter <s....@data-artisans.com>:
>
> Hi,
>
> after there have been no objections since a long time, I took the next step and created a PR that implements this change in commit 95e44099784c9deaf2ca422b8dfc11c3d67d7f82 of https://github.com/apache/flink/pull/4550 <https://github.com/apache/flink/pull/4550> . Announcing this here as a last opportunity for further discussions. FYI, this will decrease the code base by almost 12K LOC.
>
> Best,
> Stefan
>
>
>> Am 02.08.2017 um 15:26 schrieb Kostas Kloudas <k.kloudas@data-artisans.com <ma...@data-artisans.com>>:
>>
>> +1
>>
>>> On Aug 2, 2017, at 3:16 PM, Till Rohrmann <trohrmann@apache.org <ma...@apache.org>> wrote:
>>>
>>> +1
>>>
>>> On Wed, Aug 2, 2017 at 9:12 AM, Stefan Richter <s.richter@data-artisans.com <ma...@data-artisans.com>>
>>> wrote:
>>>
>>>> +1
>>>>
>>>> Am 28.07.2017 um 16:03 schrieb Stephan Ewen <sewen@apache.org <ma...@apache.org>>:
>>>>
>>>> Seems like no one raised a concern so far about dropping the savepoint
>>>> format compatibility for 1.1 in 1.4.
>>>>
>>>> Leaving this thread open for some more days, but from the sentiment, it
>>>> seems like we should go ahead?
>>>>
>>>> On Wed, Jul 12, 2017 at 4:43 PM, Stephan Ewen <sewen@apache.org <ma...@apache.org>> wrote:
>>>>
>>>>> Hi users!
>>>>>
>>>>> Flink currently maintains backwards compatibility for savepoint formats,
>>>>> which means that savepoints taken with Flink version 1.1.x and 1.2.x can be
>>>>> resumed in Flink 1.3.x
>>>>>
>>>>> We are discussing how many versions back to support. The proposition is
>>>>> the following:
>>>>>
>>>>> * Suggestion: Flink 1.4.0 will be able to resume savepoints taken with
>>>>> version 1.3.x and 1.2.x, but not savepoints from version 1.1.x and 1.0.x*
>>>>>
>>>>>
>>>>> The reason for that is that there is a lot of code mapping between the
>>>>> completely different legacy format (1.1.x, not re-scalable) and the
>>>>> key-group-oriented format (1.2.x onwards, re-scalable). It would greatly
>>>>> help the development of state and checkpointing features to drop that old
>>>>> code.
>>>>>
>>>>> Please let us know if you have concerns about that.
>>>>>
>>>>> Best,
>>>>> Stephan
>>>>>
>>>>>
>>>>
>>>>
>>
>
Re: [POLL] Dropping savepoint format compatibility for 1.1.x in the
Flink 1.4.0 release
Posted by Stefan Richter <s....@data-artisans.com>.
Hi,
after there have been no objections since a long time, I took the next step and created a PR that implements this change in commit 95e44099784c9deaf2ca422b8dfc11c3d67d7f82 of https://github.com/apache/flink/pull/4550 <https://github.com/apache/flink/pull/4550> . Announcing this here as a last opportunity for further discussions. FYI, this will decrease the code base by almost 12K LOC.
Best,
Stefan
> Am 02.08.2017 um 15:26 schrieb Kostas Kloudas <k....@data-artisans.com>:
>
> +1
>
>> On Aug 2, 2017, at 3:16 PM, Till Rohrmann <tr...@apache.org> wrote:
>>
>> +1
>>
>> On Wed, Aug 2, 2017 at 9:12 AM, Stefan Richter <s....@data-artisans.com>
>> wrote:
>>
>>> +1
>>>
>>> Am 28.07.2017 um 16:03 schrieb Stephan Ewen <se...@apache.org>:
>>>
>>> Seems like no one raised a concern so far about dropping the savepoint
>>> format compatibility for 1.1 in 1.4.
>>>
>>> Leaving this thread open for some more days, but from the sentiment, it
>>> seems like we should go ahead?
>>>
>>> On Wed, Jul 12, 2017 at 4:43 PM, Stephan Ewen <se...@apache.org> wrote:
>>>
>>>> Hi users!
>>>>
>>>> Flink currently maintains backwards compatibility for savepoint formats,
>>>> which means that savepoints taken with Flink version 1.1.x and 1.2.x can be
>>>> resumed in Flink 1.3.x
>>>>
>>>> We are discussing how many versions back to support. The proposition is
>>>> the following:
>>>>
>>>> * Suggestion: Flink 1.4.0 will be able to resume savepoints taken with
>>>> version 1.3.x and 1.2.x, but not savepoints from version 1.1.x and 1.0.x*
>>>>
>>>>
>>>> The reason for that is that there is a lot of code mapping between the
>>>> completely different legacy format (1.1.x, not re-scalable) and the
>>>> key-group-oriented format (1.2.x onwards, re-scalable). It would greatly
>>>> help the development of state and checkpointing features to drop that old
>>>> code.
>>>>
>>>> Please let us know if you have concerns about that.
>>>>
>>>> Best,
>>>> Stephan
>>>>
>>>>
>>>
>>>
>
Re: [POLL] Dropping savepoint format compatibility for 1.1.x in the
Flink 1.4.0 release
Posted by Kostas Kloudas <k....@data-artisans.com>.
+1
> On Aug 2, 2017, at 3:16 PM, Till Rohrmann <tr...@apache.org> wrote:
>
> +1
>
> On Wed, Aug 2, 2017 at 9:12 AM, Stefan Richter <s....@data-artisans.com>
> wrote:
>
>> +1
>>
>> Am 28.07.2017 um 16:03 schrieb Stephan Ewen <se...@apache.org>:
>>
>> Seems like no one raised a concern so far about dropping the savepoint
>> format compatibility for 1.1 in 1.4.
>>
>> Leaving this thread open for some more days, but from the sentiment, it
>> seems like we should go ahead?
>>
>> On Wed, Jul 12, 2017 at 4:43 PM, Stephan Ewen <se...@apache.org> wrote:
>>
>>> Hi users!
>>>
>>> Flink currently maintains backwards compatibility for savepoint formats,
>>> which means that savepoints taken with Flink version 1.1.x and 1.2.x can be
>>> resumed in Flink 1.3.x
>>>
>>> We are discussing how many versions back to support. The proposition is
>>> the following:
>>>
>>> * Suggestion: Flink 1.4.0 will be able to resume savepoints taken with
>>> version 1.3.x and 1.2.x, but not savepoints from version 1.1.x and 1.0.x*
>>>
>>>
>>> The reason for that is that there is a lot of code mapping between the
>>> completely different legacy format (1.1.x, not re-scalable) and the
>>> key-group-oriented format (1.2.x onwards, re-scalable). It would greatly
>>> help the development of state and checkpointing features to drop that old
>>> code.
>>>
>>> Please let us know if you have concerns about that.
>>>
>>> Best,
>>> Stephan
>>>
>>>
>>
>>
Re: [POLL] Dropping savepoint format compatibility for 1.1.x in the
Flink 1.4.0 release
Posted by Kostas Kloudas <k....@data-artisans.com>.
+1
> On Aug 2, 2017, at 3:16 PM, Till Rohrmann <tr...@apache.org> wrote:
>
> +1
>
> On Wed, Aug 2, 2017 at 9:12 AM, Stefan Richter <s....@data-artisans.com>
> wrote:
>
>> +1
>>
>> Am 28.07.2017 um 16:03 schrieb Stephan Ewen <se...@apache.org>:
>>
>> Seems like no one raised a concern so far about dropping the savepoint
>> format compatibility for 1.1 in 1.4.
>>
>> Leaving this thread open for some more days, but from the sentiment, it
>> seems like we should go ahead?
>>
>> On Wed, Jul 12, 2017 at 4:43 PM, Stephan Ewen <se...@apache.org> wrote:
>>
>>> Hi users!
>>>
>>> Flink currently maintains backwards compatibility for savepoint formats,
>>> which means that savepoints taken with Flink version 1.1.x and 1.2.x can be
>>> resumed in Flink 1.3.x
>>>
>>> We are discussing how many versions back to support. The proposition is
>>> the following:
>>>
>>> * Suggestion: Flink 1.4.0 will be able to resume savepoints taken with
>>> version 1.3.x and 1.2.x, but not savepoints from version 1.1.x and 1.0.x*
>>>
>>>
>>> The reason for that is that there is a lot of code mapping between the
>>> completely different legacy format (1.1.x, not re-scalable) and the
>>> key-group-oriented format (1.2.x onwards, re-scalable). It would greatly
>>> help the development of state and checkpointing features to drop that old
>>> code.
>>>
>>> Please let us know if you have concerns about that.
>>>
>>> Best,
>>> Stephan
>>>
>>>
>>
>>
Re: [POLL] Dropping savepoint format compatibility for 1.1.x in the
Flink 1.4.0 release
Posted by Till Rohrmann <tr...@apache.org>.
+1
On Wed, Aug 2, 2017 at 9:12 AM, Stefan Richter <s....@data-artisans.com>
wrote:
> +1
>
> Am 28.07.2017 um 16:03 schrieb Stephan Ewen <se...@apache.org>:
>
> Seems like no one raised a concern so far about dropping the savepoint
> format compatibility for 1.1 in 1.4.
>
> Leaving this thread open for some more days, but from the sentiment, it
> seems like we should go ahead?
>
> On Wed, Jul 12, 2017 at 4:43 PM, Stephan Ewen <se...@apache.org> wrote:
>
>> Hi users!
>>
>> Flink currently maintains backwards compatibility for savepoint formats,
>> which means that savepoints taken with Flink version 1.1.x and 1.2.x can be
>> resumed in Flink 1.3.x
>>
>> We are discussing how many versions back to support. The proposition is
>> the following:
>>
>> * Suggestion: Flink 1.4.0 will be able to resume savepoints taken with
>> version 1.3.x and 1.2.x, but not savepoints from version 1.1.x and 1.0.x*
>>
>>
>> The reason for that is that there is a lot of code mapping between the
>> completely different legacy format (1.1.x, not re-scalable) and the
>> key-group-oriented format (1.2.x onwards, re-scalable). It would greatly
>> help the development of state and checkpointing features to drop that old
>> code.
>>
>> Please let us know if you have concerns about that.
>>
>> Best,
>> Stephan
>>
>>
>
>
Re: [POLL] Dropping savepoint format compatibility for 1.1.x in the
Flink 1.4.0 release
Posted by Till Rohrmann <tr...@apache.org>.
+1
On Wed, Aug 2, 2017 at 9:12 AM, Stefan Richter <s....@data-artisans.com>
wrote:
> +1
>
> Am 28.07.2017 um 16:03 schrieb Stephan Ewen <se...@apache.org>:
>
> Seems like no one raised a concern so far about dropping the savepoint
> format compatibility for 1.1 in 1.4.
>
> Leaving this thread open for some more days, but from the sentiment, it
> seems like we should go ahead?
>
> On Wed, Jul 12, 2017 at 4:43 PM, Stephan Ewen <se...@apache.org> wrote:
>
>> Hi users!
>>
>> Flink currently maintains backwards compatibility for savepoint formats,
>> which means that savepoints taken with Flink version 1.1.x and 1.2.x can be
>> resumed in Flink 1.3.x
>>
>> We are discussing how many versions back to support. The proposition is
>> the following:
>>
>> * Suggestion: Flink 1.4.0 will be able to resume savepoints taken with
>> version 1.3.x and 1.2.x, but not savepoints from version 1.1.x and 1.0.x*
>>
>>
>> The reason for that is that there is a lot of code mapping between the
>> completely different legacy format (1.1.x, not re-scalable) and the
>> key-group-oriented format (1.2.x onwards, re-scalable). It would greatly
>> help the development of state and checkpointing features to drop that old
>> code.
>>
>> Please let us know if you have concerns about that.
>>
>> Best,
>> Stephan
>>
>>
>
>
Re: [POLL] Dropping savepoint format compatibility for 1.1.x in the
Flink 1.4.0 release
Posted by Stefan Richter <s....@data-artisans.com>.
+1
> Am 28.07.2017 um 16:03 schrieb Stephan Ewen <se...@apache.org>:
>
> Seems like no one raised a concern so far about dropping the savepoint format compatibility for 1.1 in 1.4.
>
> Leaving this thread open for some more days, but from the sentiment, it seems like we should go ahead?
>
> On Wed, Jul 12, 2017 at 4:43 PM, Stephan Ewen <sewen@apache.org <ma...@apache.org>> wrote:
> Hi users!
>
> Flink currently maintains backwards compatibility for savepoint formats, which means that savepoints taken with Flink version 1.1.x and 1.2.x can be resumed in Flink 1.3.x
>
> We are discussing how many versions back to support. The proposition is the following:
>
> Suggestion: Flink 1.4.0 will be able to resume savepoints taken with version 1.3.x and 1.2.x, but not savepoints from version 1.1.x and 1.0.x
>
>
> The reason for that is that there is a lot of code mapping between the completely different legacy format (1.1.x, not re-scalable) and the key-group-oriented format (1.2.x onwards, re-scalable). It would greatly help the development of state and checkpointing features to drop that old code.
>
> Please let us know if you have concerns about that.
>
> Best,
> Stephan
>
>
Re: [POLL] Dropping savepoint format compatibility for 1.1.x in the
Flink 1.4.0 release
Posted by Stefan Richter <s....@data-artisans.com>.
+1
> Am 28.07.2017 um 16:03 schrieb Stephan Ewen <se...@apache.org>:
>
> Seems like no one raised a concern so far about dropping the savepoint format compatibility for 1.1 in 1.4.
>
> Leaving this thread open for some more days, but from the sentiment, it seems like we should go ahead?
>
> On Wed, Jul 12, 2017 at 4:43 PM, Stephan Ewen <sewen@apache.org <ma...@apache.org>> wrote:
> Hi users!
>
> Flink currently maintains backwards compatibility for savepoint formats, which means that savepoints taken with Flink version 1.1.x and 1.2.x can be resumed in Flink 1.3.x
>
> We are discussing how many versions back to support. The proposition is the following:
>
> Suggestion: Flink 1.4.0 will be able to resume savepoints taken with version 1.3.x and 1.2.x, but not savepoints from version 1.1.x and 1.0.x
>
>
> The reason for that is that there is a lot of code mapping between the completely different legacy format (1.1.x, not re-scalable) and the key-group-oriented format (1.2.x onwards, re-scalable). It would greatly help the development of state and checkpointing features to drop that old code.
>
> Please let us know if you have concerns about that.
>
> Best,
> Stephan
>
>
Re: [POLL] Dropping savepoint format compatibility for 1.1.x in the
Flink 1.4.0 release
Posted by Ufuk Celebi <uc...@apache.org>.
On Fri, Jul 28, 2017 at 4:03 PM, Stephan Ewen <se...@apache.org> wrote:
> Seems like no one raised a concern so far about dropping the savepoint
> format compatibility for 1.1 in 1.4.
>
> Leaving this thread open for some more days, but from the sentiment, it
> seems like we should go ahead?
+1
Re: [POLL] Dropping savepoint format compatibility for 1.1.x in the
Flink 1.4.0 release
Posted by Stephan Ewen <se...@apache.org>.
Seems like no one raised a concern so far about dropping the savepoint
format compatibility for 1.1 in 1.4.
Leaving this thread open for some more days, but from the sentiment, it
seems like we should go ahead?
On Wed, Jul 12, 2017 at 4:43 PM, Stephan Ewen <se...@apache.org> wrote:
> Hi users!
>
> Flink currently maintains backwards compatibility for savepoint formats,
> which means that savepoints taken with Flink version 1.1.x and 1.2.x can be
> resumed in Flink 1.3.x
>
> We are discussing how many versions back to support. The proposition is
> the following:
>
> * Suggestion: Flink 1.4.0 will be able to resume savepoints taken with
> version 1.3.x and 1.2.x, but not savepoints from version 1.1.x and 1.0.x*
>
>
> The reason for that is that there is a lot of code mapping between the
> completely different legacy format (1.1.x, not re-scalable) and the
> key-group-oriented format (1.2.x onwards, re-scalable). It would greatly
> help the development of state and checkpointing features to drop that old
> code.
>
> Please let us know if you have concerns about that.
>
> Best,
> Stephan
>
>
Re: [POLL] Dropping savepoint format compatibility for 1.1.x in the
Flink 1.4.0 release
Posted by Stephan Ewen <se...@apache.org>.
Seems like no one raised a concern so far about dropping the savepoint
format compatibility for 1.1 in 1.4.
Leaving this thread open for some more days, but from the sentiment, it
seems like we should go ahead?
On Wed, Jul 12, 2017 at 4:43 PM, Stephan Ewen <se...@apache.org> wrote:
> Hi users!
>
> Flink currently maintains backwards compatibility for savepoint formats,
> which means that savepoints taken with Flink version 1.1.x and 1.2.x can be
> resumed in Flink 1.3.x
>
> We are discussing how many versions back to support. The proposition is
> the following:
>
> * Suggestion: Flink 1.4.0 will be able to resume savepoints taken with
> version 1.3.x and 1.2.x, but not savepoints from version 1.1.x and 1.0.x*
>
>
> The reason for that is that there is a lot of code mapping between the
> completely different legacy format (1.1.x, not re-scalable) and the
> key-group-oriented format (1.2.x onwards, re-scalable). It would greatly
> help the development of state and checkpointing features to drop that old
> code.
>
> Please let us know if you have concerns about that.
>
> Best,
> Stephan
>
>
Re: [POLL] Dropping savepoint format compatibility for 1.1.x in the
Flink 1.4.0 release
Posted by Günter Hipler <gu...@unibas.ch>.
+1 to drop Java 7 support
On 12.07.2017 16:43, Stephan Ewen wrote:
> Hi users!
>
> Flink currently maintains backwards compatibility for savepoint
> formats, which means that savepoints taken with Flink version 1.1.x
> and 1.2.x can be resumed in Flink 1.3.x
>
> We are discussing how many versions back to support. The proposition
> is the following:
>
> * Suggestion: Flink 1.4.0 will be able to resume savepoints taken
> with version 1.3.x and 1.2.x, but not savepoints from version 1.1.x
> and 1.0.x*
>
>
> The reason for that is that there is a lot of code mapping between the
> completely different legacy format (1.1.x, not re-scalable) and the
> key-group-oriented format (1.2.x onwards, re-scalable). It would
> greatly help the development of state and checkpointing features to
> drop that old code.
>
> Please let us know if you have concerns about that.
>
> Best,
> Stephan
>
--
Universität Basel
Universitätsbibliothek
Günter Hipler
Projekt SwissBib
Schoenbeinstrasse 18-20
4056 Basel, Schweiz
Tel.: + 41 (0)61 267 31 12 Fax: ++41 61 267 3103
E-Mail guenter.hipler@unibas.ch
URL: www.swissbib.org / http://www.ub.unibas.ch/
Re: [POLL] Dropping savepoint format compatibility for 1.1.x in the
Flink 1.4.0 release
Posted by Stephan Ewen <se...@apache.org>.
Hi Kanstantsin!
Looking at other mail thread, I think you refer to the CEP library API,
correct?
The CEP library has not been labeled as stable yet. The stability
guarantees that we introduced refer to the APIs marked with the `@Public`
interface.
The CEP library was still too much in change/improvement mode to freeze it
already and mark it as stable.
Have you seen parts marked as `@Public` where you had to adjust code?
Best,
Stephan
On Wed, Jul 12, 2017 at 6:12 PM, Kanstantsin Kamkou <kk...@gmail.com>
wrote:
> Hi users!
>
> I can't upgrade from 1.2.x to 1.3.x without code adaptations.
> Upgrading from 1.(0|1).x to 1.2.x produces configuration mess.
> Maybe you can discuss changing the release plan, speed it up a little
> bit and use the major.minor.patch versions as advantage to organize
> the release process more accurate. As the result, backwards
> compatibility might be guarantied among a single major number only. It
> also sounds like a new version will be 2.0.0. This way flink might get
> more trust as I will see the complexity of changing the version
> number.
>
> > MINOR version when you add functionality in a backwards-compatible manner
> That is exactly my expectation.
>
> Best.
>
>
> On Wed, Jul 12, 2017 at 4:43 PM, Stephan Ewen <se...@apache.org> wrote:
> > Hi users!
> >
> > Flink currently maintains backwards compatibility for savepoint formats,
> > which means that savepoints taken with Flink version 1.1.x and 1.2.x can
> be
> > resumed in Flink 1.3.x
> >
> > We are discussing how many versions back to support. The proposition is
> the
> > following:
> >
> > Suggestion: Flink 1.4.0 will be able to resume savepoints taken with
> > version 1.3.x and 1.2.x, but not savepoints from version 1.1.x and 1.0.x
> >
> >
> > The reason for that is that there is a lot of code mapping between the
> > completely different legacy format (1.1.x, not re-scalable) and the
> > key-group-oriented format (1.2.x onwards, re-scalable). It would greatly
> > help the development of state and checkpointing features to drop that old
> > code.
> >
> > Please let us know if you have concerns about that.
> >
> > Best,
> > Stephan
> >
>
Re: [POLL] Dropping savepoint format compatibility for 1.1.x in the
Flink 1.4.0 release
Posted by Kanstantsin Kamkou <kk...@gmail.com>.
Hi users!
I can't upgrade from 1.2.x to 1.3.x without code adaptations.
Upgrading from 1.(0|1).x to 1.2.x produces configuration mess.
Maybe you can discuss changing the release plan, speed it up a little
bit and use the major.minor.patch versions as advantage to organize
the release process more accurate. As the result, backwards
compatibility might be guarantied among a single major number only. It
also sounds like a new version will be 2.0.0. This way flink might get
more trust as I will see the complexity of changing the version
number.
> MINOR version when you add functionality in a backwards-compatible manner
That is exactly my expectation.
Best.
On Wed, Jul 12, 2017 at 4:43 PM, Stephan Ewen <se...@apache.org> wrote:
> Hi users!
>
> Flink currently maintains backwards compatibility for savepoint formats,
> which means that savepoints taken with Flink version 1.1.x and 1.2.x can be
> resumed in Flink 1.3.x
>
> We are discussing how many versions back to support. The proposition is the
> following:
>
> Suggestion: Flink 1.4.0 will be able to resume savepoints taken with
> version 1.3.x and 1.2.x, but not savepoints from version 1.1.x and 1.0.x
>
>
> The reason for that is that there is a lot of code mapping between the
> completely different legacy format (1.1.x, not re-scalable) and the
> key-group-oriented format (1.2.x onwards, re-scalable). It would greatly
> help the development of state and checkpointing features to drop that old
> code.
>
> Please let us know if you have concerns about that.
>
> Best,
> Stephan
>