[GitHub] flink issue #4015: [FLINK-6301] [flink-connector-kafka-0.10] Upgrading Kafka...

[vidhu5269 <gi...@git.apache.org>]

Github user vidhu5269 commented on the issue:

    https://github.com/apache/flink/pull/4015
  
    Hi @tzulitai 
    
    Apologies for such a long delay. It took me quite a while to come back to this.
    
    I ran the updated connector on the cluster and didn't see any dependency conflicts. The job using it was reading from a gzipped avro topic and producing into two different topics: a text and an avro topic. Both the consumption and production worked as expected.
    
    It was done on a standalone cluster with 2 workers and 1 master with 8 slots/worker. The job was using 10 slots and was running on the both the workers. Each worker had 8 vCPUs and 8 GB of RAM. 
    
    With this job, I also verified that there wasn't any memory leak with the `kafka-clients` version change.
    
    Following is the diff between the master and version change commit from `mvn dependency:tree`:
    ```
    20c20
    < [INFO] +- org.apache.kafka:kafka-clients:jar:0.10.0.1:compile
    ---
    > [INFO] +- org.apache.kafka:kafka-clients:jar:0.10.2.0:compile
    27,28c27,28
    < [INFO] +- org.apache.kafka:kafka_2.11:jar:0.10.0.1:test
    < [INFO] |  +- com.101tec:zkclient:jar:0.8:test
    ---
    > [INFO] +- org.apache.kafka:kafka_2.11:jar:0.10.2.0:test
    > [INFO] |  +- net.sf.jopt-simple:jopt-simple:jar:5.0.3:test
    31,35c31,33
    < [INFO] |  +- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.4:test
    < [INFO] |  +- net.sf.jopt-simple:jopt-simple:jar:4.9:test
    < [INFO] |  \- org.apache.zookeeper:zookeeper:jar:3.4.6:provided
    < [INFO] |     +- jline:jline:jar:0.9.94:provided
    < [INFO] |     \- io.netty:netty:jar:3.7.0.Final:provided
    ---
    > [INFO] |  +- com.101tec:zkclient:jar:0.10:test
    > [INFO] |  +- org.apache.zookeeper:zookeeper:jar:3.4.6:provided
    > [INFO] |  \- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.4:test
    70a69
    > [INFO] |  |  +- io.netty:netty:jar:3.8.0.Final:provided
    ```
    From what we can see here, apart from the new kafka-clients library, there are a few changes coming from its tests-jar as well.
    1. The tests-jar has new version of `net.sf.jopt-simple:jopt-simple` and `com.101tec:zkclient:jar:0.10` in `test` scope. 
    2. `jline:jline` is not there in the new version.
    3. `3.8.0-Final` of `io.netty:netty` is being pulled in (from `flakka-remote`) instead of `3.7.0-Final`. 
    
    Since the job and the unit tests worked as expected, I am assuming that these dependency changes are not breaking anything. Do tell me if I am missing something here.
    
    Although, these are not new dependencies but I still verified that `zkclient` and `netty` both have ASL 2.0 whereas, jopt-simple has `MIT` which is also compatible with ASL 2.0. So, we should be good as far as licensing goes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---