You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by majorendre <gi...@git.apache.org> on 2018/11/12 14:30:43 UTC

[GitHub] flume pull request #235: Flume 3281 Update to Kafka 2.0

GitHub user majorendre opened a pull request:

    https://github.com/apache/flume/pull/235

    Flume 3281 Update to Kafka 2.0

    This has been tested with unit tests. The main difference that caused the most problems is the consumer.poll(Duration) change. This does not block even when it fetches meta data whereas the previous poll(long timeout) blocked indefinitely for meta data fetching.
    This has resulted in many test timing issues. I tried to do minimal changes at the tests, just enough to make them pass.
    
    Kafka 2.0 requires a higher version for slf4j, I had to update it to 1.7.25.
    
    Option migrateZookeeperOffsets is deprecated in this PR. This will allow us to get rid of Kafka server libraries in Flume.
    
    Compatibility testing. 
    Modified the TestUtil to be able to use external servers. This I could test against a variety of Kafka Server versions using the normal unit tests. 
    Channel tests using 2.0.1 client:
    Kafka_2.11-0.11.0.3 - timeouts in TestPartitions when creating topics
    Kafka_2.11-1.0.2 - passed
    Kafka_2.11-1.1.1 - passed
    Kafka_2.11-2.0.1 - passed 
    
    I will publish further results here, today or tomorrow.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/majorendre/flume FLUME-3281

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flume/pull/235.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #235
    
----
commit 2b6818ad2f8c9d8367ba5526f800583f29967464
Author: Endre Major <em...@...>
Date:   2018-11-08T14:36:47Z

    FLUME-3281 Update to Kafka 2.0

commit e1a98bf98bdfa2fc3f524e94ed5e5603477f7820
Author: Endre Major <em...@...>
Date:   2018-11-10T16:14:06Z

    FLUME-3281 TestUtil improvements external servers

commit 34ce07c9ba3cc388630a6fd2e5cb14f94b665ca5
Author: Endre Major <em...@...>
Date:   2018-11-12T13:31:34Z

    FLUME-3281 Deprecating migrateZookeeperOffsets

----


---

[GitHub] flume pull request #235: FLUME-3281 Update to Kafka 2.0

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/flume/pull/235


---