[GitHub] kafka pull request #2984: KAFKA-4667 Connect uses AdminClient to create inte...

[rhauch <gi...@git.apache.org>]

GitHub user rhauch opened a pull request:

    https://github.com/apache/kafka/pull/2984

    KAFKA-4667 Connect uses AdminClient to create internal topics when needed

    The backing store for offsets, status, and configs now attempts to use the new AdminClient to look up the internal topics and create them if they don’t yet exist. If the necessary APIs are not available in the connected broker, the stores fall back to the old behavior of relying upon auto-created topics. Kafka Connect requires a minimum of Apache Kafka 0.10.0.1-cp1, and the AdminClient can work with all versions since 0.10.0.0.
    
    All three of Connect’s internal topics are created as compacted topics, and new distributed worker configuration properties control the replication factor for all three topics and the number of partitions for the offsets and status topics; the config topic requires a single partition and does not allow it to be set via configuration. All of these new configuration properties have sensible defaults, meaning users can upgrade without having to change any of the existing configurations. In most situations, existing Connect deployments will have already created the storage topics prior to upgrading.
    
    The replication factor defaults to 3, so anyone running Kafka clusters with fewer nodes than 3 will receive an error unless they explicitly set the replication factor for the three internal topics. This is actually desired behavior, since it signals the users that they should be aware they are not using sufficient replication for production use.
    
    The integration tests use a cluster with a single broker, so they were changed to explicitly specify a replication factor of 1 and a single partition.
    
    The `KafkaAdminClientTest` was refactored to extract a utility for setting up a `KafkaAdminClient` with a `MockClient` for unit tests.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/rhauch/kafka kafka-4667

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/kafka/pull/2984.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2984
    
----
commit ea181ba29a678e47a5ff88ca0ad4c995aa4e017b
Author: Randall Hauch <rh...@gmail.com>
Date:   2017-05-05T17:14:07Z

    KAFKA-4667 Connect can create internal topics using AdminClient
    
    The backing store for offsets, status, and configs now attempts to use the new AdminClient to look up the
    internal topics and create them if they don’t yet exist. If the necessary APIs are not available in the connected
    broker, the stores fall back to the old behavior of relying upon auto-created topics. Kafka Connect requires
    a minimum of Apache Kafka 0.10.0.1-cp1, and the AdminClient can work with all versions since 0.10.0.0.
    
    All three of Connect’s internal topics are created as compacted topics, and new distributed worker configuration
    properties control the replication factor for all three topics and the number of partitions for the offsets
    and status topics; the config topic requires a single partition and does not allow it to be set via configuration.
    All of these new configuration properties have sensible defaults, meaning users can upgrade without having to
    change any of the existing configurations. In most situations, existing Connect deployments will have already
    created the storage topics prior to upgrading.
    
    The replication factor defaults to 3, so anyone running Kafka clusters with fewer nodes than 3 will receive an
    error unless they explicitly set the replication factor for the three internal topics. This is actually desired behavior,
    since it signals the users that they should be aware they are not using sufficient replication for production use.
    
    The integration tests use a cluster with a single broker, so they were changed to explictly specify a replication factor of 1
    and a single partition.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2984: KIP-154 KAFKA-4667 Connect uses AdminClient to cre...

[asfgit <gi...@git.apache.org>]

Github user asfgit closed the pull request at:

    https://github.com/apache/kafka/pull/2984


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---