You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by tzulitai <gi...@git.apache.org> on 2017/07/24 08:48:08 UTC

[GitHub] flink pull request #4387: [FLINK-7143] [kafka] Forward ports of new Kafka te...

GitHub user tzulitai opened a pull request:

    https://github.com/apache/flink/pull/4387

    [FLINK-7143] [kafka] Forward ports of new Kafka tests to master

    This PR forward ports all new tests added in #4357 to `master`, so that the behaviors is correctly guarded there also.
    
    ## Changes
    1. Introduce `KafkaTopicPartitionAssigner` class to master branch, which strictly defines the partition assignment contract (discussed in #4301).
    2. Port rescaling unit test. Note that some tested behaviors needed to be changed due to the differences between 1.3 and 1.4 for the Kafka consumer.
    3. Make checkpoint methods final.
    4. (new change, not a port) Remove invalid `checkRestoredNullCheckpointWhenFetcherNotReady` test, which was testing a legacy behavior of the consumer

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/tzulitai/flink kafka-forward-ports

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/4387.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #4387
    
----
commit 7295777984084fc470edaee44e1bc32881409665
Author: Tzu-Li (Gordon) Tai <tz...@apache.org>
Date:   2017-07-24T07:00:16Z

    [FLINK-7143] [kafka] Introduce KafkaTopicPartitionAssigner with stricter assignment contracts
    
    This commit refactors the local partition assignment logic to be located
    in a strict contract-defining method, to make it explicit of the
    expected partition to subtask assignment without relying solely on
    hashCode's of kafka partitions.

commit 72a8c42505ba791f51001129a08744b104a171d7
Author: Aljoscha Krettek <al...@gmail.com>
Date:   2017-07-18T09:57:46Z

    [FLINK-7143] [kafka] Add test for Kafka Consumer rescaling
    
    This verifies that the consumer always correctly knows whether it is
    restored or not and is not affected by changes in the partitions as
    reported by Kafka.
    
    Previously, operator state reshuffling could lead to partitions being
    subscribed to multiple times.

commit 100936c7ab0b7bca4dab10143aa184dc31e2fd46
Author: Aljoscha Krettek <al...@gmail.com>
Date:   2017-07-18T08:35:54Z

    [hotfix] [kafka] Make checkpoint methods final in KafkaConsumerBase
    
    This prevents concrete Kafka Source implementations from accidentally
    overriding the checkpointing methods. This would be problematic when not
    providing tests. We test the checkpoint methods of the ConsumerBase but
    derived methods would not be tested.

commit b0cf8779b76b5fe94beb4ffb6ba9adad16280be6
Author: Tzu-Li (Gordon) Tai <tz...@apache.org>
Date:   2017-07-24T08:34:37Z

    [FLINK-7248] [kafka, tests] Remove invalid checkRestoredNullCheckpointWhenFetcherNotReady test
    
    This test is an invalid remnant from recent major Kafka consumer
    refactorings. The actual behaviour is covered by
    checkRestoredCheckpointWhenFetcherNotReady. When the fetcher is not yet
    ready and exposed and a checkpoint happens, we fallback to using any
    restored state as the checkpoint.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4387: [FLINK-7143] [kafka] Forward ports of new Kafka te...

Posted by asfgit <gi...@git.apache.org>.

Github user asfgit closed the pull request at:

    https://github.com/apache/flink/pull/4387


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #4387: [FLINK-7143] [kafka] Forward ports of new Kafka tests to ...

Posted by tzulitai <gi...@git.apache.org>.

Github user tzulitai commented on the issue:

    https://github.com/apache/flink/pull/4387
  
    Merging this now, as the changes were already reviewed when applying them onto `release-1.3`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---