You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by nicktoker <gi...@git.apache.org> on 2018/01/17 09:05:06 UTC
[GitHub] flink pull request #5305: Release 1.3
GitHub user nicktoker opened a pull request:
https://github.com/apache/flink/pull/5305
Release 1.3
when asyncIO have timeout all operator fail and break the all stream
need to add option to control when timeout happen and choose what to do in this case
ignore this stream element and continue to next one ........ or beak the flow
one timeout don't need break the stream
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/flink release-1.3
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5305.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5305
----
commit 0544b44808d3680fa22695f23d69e341820afde9
Author: Tzu-Li (Gordon) Tai <tz...@...>
Date: 2017-06-03T21:45:22Z
[FLINK-6830] [fileSink] Port BucketingSinkFrom12MigrationTest for Flink 1.3
commit 74bb9a83a923a76443426e15ad4b4285bcfeb69d
Author: Tzu-Li (Gordon) Tai <tz...@...>
Date: 2017-06-03T22:18:26Z
[FLINK-6830] [kafka] Port migration tests for FlinkKafkaConsumerBase to Flink 1.3
This commit also consolidates all tests for migration from Flink 1.1 and
1.2 for the FlinkKafkaConsumerBase to a single class
FlinkKafkaConsumerBaseMigrationTest. Parameterization is used to test
migration from different Flink version savepoints.
commit e5a435b2ddb555f625a419897ed56f792f7372b3
Author: Tzu-Li (Gordon) Tai <tz...@...>
Date: 2017-06-03T22:55:48Z
[FLINK-6830] [DataStream] Port window operator migration tests for Flink 1.3
This commit also consolidates all Flink 1.1 and 1.2 window operator
migration tests to a single WindowOperatorMigrationTest class.
Parameterization is used to test restoring from different previous Flink
version snapshots.
commit d4a646a035366918a100f64428c471464870b8d0
Author: Tzu-Li (Gordon) Tai <tz...@...>
Date: 2017-06-03T23:42:04Z
[FLINK-6830] [fileSource] Port continuous file reader migration tests for Flink 1.3
This commit also consolidates all Flink 1.1 and 1.2 migration tests into
a single ContinuousFileProcessingMigrationTest class. Parameterization
is used to test restore from different previous Flink versions.
This closes #4059.
commit f72eff7fcb1f41128ac9ad85c541b0a347ecc3dc
Author: Robert Metzger <rm...@...>
Date: 2017-06-02T10:04:54Z
[FLINK-6815] Fix javadocs by upgrading genjavadoc-plugin_2.10.6 to 0.10
commit b2d6dc1d4914f29f6ed1120daff645b3c8073716
Author: Dawid Wysakowicz <da...@...>
Date: 2017-06-01T11:17:25Z
[FLINK-6783] Changed passing index of type argument while extracting return type.
commit d24515fee29fc8924359ab6493e7c8e05ac9b173
Author: kkloudas <kk...@...>
Date: 2017-06-08T14:24:35Z
[FLINK-6772] [cep] Fix ordering (by timestamp) of matched events.
commit c35b1dafd8cc83d2929e0ebcc140942ab981001e
Author: kkloudas <kk...@...>
Date: 2017-05-23T17:07:42Z
[FLINK-6198] [cep] [doc] Update CEP documentation.
commit 261bb38260163daa1b07f9809355641a99fc2d63
Author: David Anderson <da...@...>
Date: 2017-06-07T12:23:51Z
[FLINK-6198] [cep] [doc] Update CEP documentation.
commit 9cb728e2021132528f4f40e891eb46656de8bd60
Author: Tzu-Li (Gordon) Tai <tz...@...>
Date: 2017-06-08T06:52:04Z
[hotfix] [scala] Fix instantiation of Scala serializers' config snapshot classes
Prior to this commit, the configuration snapshot classes of Scala
serializers did not have the proper default empty constructor that is
used for deserializing the configuration snapshot.
Since some Scala serializers' config snapshots extend the Java
CompositeTypeSerializerConfigSnapshot, their config snapshot classes are
also changed to be implemented in Java since in Scala we can only call a
single base class constructor from subclasses.
commit 534d28c469c8be9a0da0ee4c48744de9386f2e26
Author: Tzu-Li (Gordon) Tai <tz...@...>
Date: 2017-06-08T13:29:45Z
[FLINK-6869] [scala] Specify serialVersionUID for all Scala serializers
Previously, Scala serializers did not specify the serialVersionUID, and
therefore prohibited restore from previous Flink version snapshots
because the serializers' implementations changed.
The serialVersionUIDs added in this commit are identical to what they
were (as generated by Java) in Flink 1.2, so that we can at least
restore state that were written with the Scala serializers as of 1.2.
commit 7aafaf6aa549c30257cf6176fb220ce5150ee9fe
Author: Tzu-Li (Gordon) Tai <tz...@...>
Date: 2017-06-10T20:41:35Z
[FLINK-6869] [core] Tolerate serialVersionUID mismatches for Scala and anonymous serializers
This commit lets the TypeSerializerSerializationProxy be tolerable for
serialVersionUID mismatches when reading anonymous classed serializers
or our Scala serializers.
Our Scala serializers require this since they use Scala macros to be
generated at compile time, and therefore is not possible to fix a
certain serialVersionUID for them. For non-generated Scala serializers,
we still also need this because their serialVersionUIDs pre-1.3 may
vary depending on the Scala version used.
This can be seen as a workaround, and should be reverted once 1.2
savepoint compatibility is no longer maintained.
This commit also updates the streaming state docs to educate the user to
avoid using anonymous classes for their state serializers.
This closes #4090.
commit 57421f904382a7b8d7f177c688192a012378bb85
Author: Tzu-Li (Gordon) Tai <tz...@...>
Date: 2017-06-11T09:02:38Z
[hotfix] [cep] Fix incorrect CompatibilityResult.requiresMigration calls in CEP
commit 39c8270d39684765484fa4b6b2711e5714b81b64
Author: Tzu-Li (Gordon) Tai <tz...@...>
Date: 2017-06-11T13:30:36Z
[FLINK-6883] [core] Refactor TypeSerializer to not implement TypeDeserializer
The separation of the TypeDeserializer interface from the TypeSerializer
base class is due to the fact that additionally implementing the
TypeDeserializer interface alters the generation order of anonymos
serializer classes for Scala case classes and collections.
Instead, the TypeDeserializer is now used as a mixin on the
TypeDeserializerAdapter utility, which now serves as a bridge for
both directions (i.e. TypeSerializer to TypeDeserializer, and vice
versa). No user interfaces are broken due to this change.
commit 6ae09955000783109c526005aab7e8a85f27dddc
Author: Tzu-Li (Gordon) Tai <tz...@...>
Date: 2017-06-11T13:31:42Z
[FLINK-6883] [tests] Add migration tests for Scala jobs
This commit adds migration ITCases for jobs written using the Scala API.
An extra concern for migration of Scala jobs is that Scala case classes
and collections use anonymous generated serializers, which may affect
state restore.
This closes #4103.
commit 546796602e9b8a42f98b76c8d34219eeeb44f33c
Author: Till Rohrmann <tr...@...>
Date: 2017-05-31T13:14:11Z
[FLINK-6796] [tests] Use Environment's class loader in AbstractStreamOperatorTestHarness
Generalize KeyedOneInputStreamOperatorTestHarness
Generalize AbstractStreamOperatorTestHarness
commit 1c2361e25044a3d73e64763d3566cb001103087e
Author: Till Rohrmann <tr...@...>
Date: 2017-05-31T16:37:12Z
[FLINK-6803] [tests] Add test for PojoSerializer state upgrade
The added PojoSerializerUpgradeTest tests the state migration behaviour when the
underlying pojo type changes and one tries to recover from old state. Currently
not all tests could be activated, because there still some pending issues to be
fixed first. We should arm these tests once the issues have been fixed.
commit b8f239d723aa4dd145835f6f8de33b89b4bded92
Author: Tzu-Li (Gordon) Tai <tz...@...>
Date: 2017-06-04T10:30:58Z
[FLINK-6801] [core] Relax missing fields check when reading PojoSerializerConfigSnapshot
Prior to this commit, when reading the PojoSerializerConfigSnapshot, if
the underlying POJO type has a missing field, then the read would fail.
Failing the deserialization of the config snapshot is too severe,
because that would leave no oppurtunity to restore the checkpoint at
all, whereas we should be able to restore the config and provide it to
the new PojoSerializer for the change of getting a convert deserializer.
This commit changes this by only restoring the field names when reading
the PojoSerializerConfigSnapshot. In PojoSerializer.ensureCompatibility,
the field name is used to lookup the fields of the new PojoSerializer.
This change does not change the serialization format of the
PojoSerializerConfigSnapshot.
commit 34ff16e9f6ef45ae7eab4334c45e43046e2ea514
Author: Tzu-Li (Gordon) Tai <tz...@...>
Date: 2017-06-04T13:51:12Z
[hotfix] [tests] Fix failing tests in AsyncWaitOperatorTest and StateBackendTestBase
commit deaf4bfd5bc578239b0f67ba034e7ea328a30963
Author: Tzu-Li (Gordon) Tai <tz...@...>
Date: 2017-06-04T17:32:53Z
[FLINK-6803] [tests] Enhancements to PojoSerializerUpgradeTest
1. Allow tests to ignore missing fields.
2. Add equivalent tests which use POJOs as managed operator state.
For 2, all tests have to be ignored for now until FLINK-6804 is fixed.
commit 8a0276af6f65814d945bef1372c199f96f265bb5
Author: Tzu-Li (Gordon) Tai <tz...@...>
Date: 2017-06-04T18:41:59Z
[FLINK-6801] [core] Allow deserialized PojoSerializer to have removed fields
Prior to this commit, deserializing the PojoSerializer would fail when
we encounter a missing field that existed in the POJO type before. It is
actually perfectly fine to have a missing field; the deserialized
PojoSerializer should simply skip reading the removed field's previously
serialized values, i.e. much like how Java Object Serialization works.
This commit relaxes the deserialization of the PojoSerializer, so that a
null will be used as a placeholder value to indicate a removed field
that previously existed. De-/serialization and copying methods on the
PojoSerializer will respect null Fields and simply skip them.
commit db975260cfba8b089d4f477a0ebfc011d0bda130
Author: Fokko Driesprong <fo...@...>
Date: 2017-06-04T14:08:44Z
[FLINK-6848] [doc] Update managed state docs to include Scala snippets
Add an example of how to work with managed state in Scala.
This closes #4072.
commit 379be13b67948d28be66e071072412c870d6e1f8
Author: Tzu-Li (Gordon) Tai <tz...@...>
Date: 2017-06-04T20:40:26Z
[FLINK-6804] [state] Consistent state migration behaviour across state backends
Prior to this commit, memory and non-memory state backends behaved
differently w.r.t. state migration. For the memory backends, we did
not require the new serializer to be compatible in order for the job to
proceed after restore, because all state have already been deserialized
to objects and the new serializer can always just be used as is.
Therefore, the compatibility checks were not performed for the memory
backends, resulting in different code paths between the different state
backends.
However, this inconsistent behaviour across backends will be confusing
for users. This commit adds the code path to check the newly registered
serializer's compatibility in the memory backends (even though it isn't
required), and deliberately fails the job if the new serializer is
incompatible.
Note that the compatibiilty code paths will be truly unified and
required for all backends once we have eager state registration.
This closes #4073.
commit 1bdd19d044d18e81b4d2c4016ff38e6d86c6f609
Author: Tzu-Li (Gordon) Tai <tz...@...>
Date: 2017-06-04T20:48:45Z
[FLINK-6803] [tests] Fully enable PojoSerializerUpgradeTests for all state backends
With the fixes for the PojoSerializer in, this commit fully enables all
tests for upgrading the PojoSerializer for all state backends, which
otherwise could not pass before.
This closes #4044.
commit a3103c2ddd6b8a7566b8fd27f3acd695b4b345c7
Author: Till Rohrmann <tr...@...>
Date: 2017-06-02T13:48:54Z
[FLINK-6833] [task] Fail StreamTask only due to async exception if it is running
In order to resolve a race condition between a properly terminated StreamTask which
cleans up its resources (stopping asynchronous operations, etc.) and a cancelled
asynchronous operation (e.g. asynchronous checkpointing operation), we check whether
the StreamTask is still running before failing it externally.
This closes #4058.
commit a1d70f5730c7186dd4aa02ec5628a19d8c56f6e8
Author: Till Rohrmann <tr...@...>
Date: 2017-06-12T13:21:47Z
[FLINK-6899] [state] Create correctly sized state array in NestedMapsStateTable
This closes #4107.
commit 06dfb57b86d5397b4fd9864f59998af16b8f73e7
Author: Till Rohrmann <tr...@...>
Date: 2017-06-13T08:56:25Z
[FLINK-6744] [tests] Harden ExecutionGraphSchedulingTest.testDeployPipelinedConnectedComponentsTogether
Increase the timeout for the verification check that the TaskManagerGateway#submitTask
method has been called.
commit 8de51f953421a30073d2ff57bf37daa81bdaa991
Author: Stefan Richter <s....@...>
Date: 2017-06-12T09:48:15Z
[FLINK-6685] Adjust scopes of SafetyNetCloseableRegistry usages
This closes #4108.
commit 5281dd6598f17c8dfe0c7b091c90c8721d305375
Author: Tzu-Li (Gordon) Tai <tz...@...>
Date: 2017-06-13T09:47:44Z
[hotfix] [tests] Fix failing Scala StatefulJobSavepointMigrationITCase
commit 88fbe2ac83f08074467c33b48e981b7ba9d1032f
Author: Jark Wu <wu...@...>
Date: 2017-06-02T13:28:45Z
[FLINK-6817] [table] Add OverWindowWithPreceding class to guide users apply preceding in over window
----
---
[GitHub] flink issue #5305: Release 1.3
Posted by GJL <gi...@git.apache.org>.
Github user GJL commented on the issue:
https://github.com/apache/flink/pull/5305
Thanks for your contribution, @nicktoker. Unfortunately with your pull request you are trying to merge the `release-1.3` branch into the `master`. You probably want to create a feature branch from a release or the master first and then target your pull request to the specific release. Also make sure that for non-trivial changes, there is a JIRA ticket. For more information, see the contribution guidelines: http://flink.apache.org/contribute-code.html
---
[GitHub] flink pull request #5305: Release 1.3
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/5305
---