You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by vahidhashemian <gi...@git.apache.org> on 2017/01/21 04:11:17 UTC
[GitHub] kafka pull request #2415: KAFKA-4547 (0.10.1 hotfix): Avoid unnecessary offs...
GitHub user vahidhashemian opened a pull request:
https://github.com/apache/kafka/pull/2415
KAFKA-4547 (0.10.1 hotfix): Avoid unnecessary offset commit that could lead to an invalid offset position if partition is paused
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vahidhashemian/kafka KAFKA-4547-0.10.1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/2415.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2415
----
commit 6acbf27aebbe5b6b4f230ce44ea5151c4c53ffbd
Author: Jason Gustafson <ja...@confluent.io>
Date: 2016-09-20T02:59:03Z
MINOR: Bump to version 0.10.2
commit f396fdac197409fb955f00a6f642f04e4926ba41
Author: Ben Stopford <be...@gmail.com>
Date: 2016-09-20T06:17:23Z
KAFKA-4193; Fix for intermittent failure in FetcherTest
Author: Ben Stopford <be...@gmail.com>
Reviewers: Jason Gustafson <ja...@confluent.io>
Closes #1881 from benstopford/KAFKA-4193
commit c195003cb6e05f2d8c49285ff7e77b1cb3aa4361
Author: Eno Thereska <en...@gmail.com>
Date: 2016-09-20T10:33:50Z
HOTFIX: Added check for metadata unavailable
Author: Eno Thereska <en...@gmail.com>
Reviewers: Damian Guy <da...@gmail.com>, Ismael Juma <is...@juma.me.uk>
Closes #1887 from enothereska/hotfix-metadata-unavailable
commit 3663275cf066b7715cc11b26fd9c144bbff1c373
Author: Ben Stopford <be...@gmail.com>
Date: 2016-09-20T13:53:48Z
KAFKA-4184; Intermittent failures in ReplicationQuotasTest.shouldBootstrapTwoBrokersWithFollowerThrottle
Build is unstable, so it's hard to validate this change. Of the various builds up until 11am BST the test ran twice and passed twice.
Author: Ben Stopford <be...@gmail.com>
Reviewers: Ismael Juma <is...@juma.me.uk>
Closes #1873 from benstopford/KAFKA-4184
commit 4f821830bc6b726cddf90999fff76006745b1a3f
Author: Ben Stopford <be...@gmail.com>
Date: 2016-09-20T14:41:14Z
KAFKA-4197; Make ReassignPartitionsTest System Test move data
The ReassignPartitionsTest system tests doesn't reassign any replicas (i.e. move data).
This is a simple issue. It uses a 3 node cluster with replication factor of 3, so whilst the replicas are jumbled around, nothing actually is moved from machine to machine when the assignment is executed.
This fix just ups the number of nodes to 4 so things move.
Tests pass locally.
There are runs pending on the two branch builders
Passes:
https://jenkins.confluent.io/job/system-test-kafka-branch-builder/551/
https://jenkins.confluent.io/job/system-test-kafka-branch-builder-2/94/
https://jenkins.confluent.io/job/system-test-kafka-branch-builder/553/
https://jenkins.confluent.io/job/system-test-kafka-branch-builder/554/
https://jenkins.confluent.io/job/system-test-kafka-branch-builder-2/95
Failures:
https://jenkins.confluent.io/job/system-test-kafka-branch-builder/552 => _RuntimeError: There aren't enough available nodes to satisfy the resource request. Total cluster size: 1, Requested: 4, Already allocated: 1, Available: 0._ Which I assume to do with the test env.
Author: Ben Stopford <be...@gmail.com>
Reviewers: Ismael Juma <is...@juma.me.uk>
Closes #1892 from benstopford/fix_reassignment_test
commit 24f81ea764a493b4422b6a3ef6b3e771d0e4d63b
Author: Damian Guy <da...@gmail.com>
Date: 2016-09-21T18:11:12Z
MINOR: add javadoc comment to PersistenKeyValueFactory.enableCaching
missing javadoc on public API method PersistenKeyValueFactory.enableCaching
Author: Damian Guy <da...@gmail.com>
Reviewers: Eno Thereska, Guozhang Wang
Closes #1891 from dguy/minor-java-doc
commit a632716a3c9a871f325c6f13aefa9aed0add4b82
Author: Damian Guy <da...@gmail.com>
Date: 2016-09-21T18:13:39Z
MINOR: remove unused code from InternalTopicManager
Remove isValidCleanupPolicy and related fields as they are never used.
Author: Damian Guy <da...@gmail.com>
Reviewers: Eno Thereska, Guozhang Wang
Closes #1888 from dguy/minor-remove-unused
commit 732fabf94ebc9631d31f2feb2116ee8b63beabef
Author: Jason Gustafson <ja...@confluent.io>
Date: 2016-09-22T17:07:50Z
KAFKA-3782: Ensure heartbeat thread restarted after rebalance woken up
Author: Jason Gustafson <ja...@confluent.io>
Reviewers: Guozhang Wang
Closes #1898 from hachikuji/KAFKA-3782
commit 27e3edc791760dea7ff4d048f87d1585f9e235d7
Author: Elias Levy <fe...@gmail.com>
Date: 2016-09-22T17:33:23Z
MINOR: Fix comments in KStreamKStreamJoinTest
Minor comment fixes.
Author: Elias Levy <fe...@gmail.com>
Reviewers: Guozhang Wang
Closes #1885 from eliaslevy/fix-test-comments
commit 4112bc3af7bec63ae1248c1b9db745ba8ac4c63c
Author: Eno Thereska <en...@gmail.com>
Date: 2016-09-22T19:45:43Z
HOTFIX: Decrease commit interval
The original commit interval of 30 seconds might be too large in some cases, e.g., when the verifier finishes before those 30 seconds have elapsed.
Author: Eno Thereska <en...@gmail.com>
Reviewers: Damian Guy, Guozhang Wang
Closes #1899 from enothereska/hotfix-smoke-test-commit-interval
commit d32f3f2828c199a5a09bbba779d07430e43fb190
Author: Ismael Juma <is...@juma.me.uk>
Date: 2016-09-22T21:53:40Z
MINOR: Increase `zkConnectionTimeout` and timeout in `testReachableServer`
We had a number of failures recently due to these timeouts being too low. It's a particular problem if multiple forks are used while running the tests.
Author: Ismael Juma <is...@juma.me.uk>
Reviewers: Jason Gustafson <ja...@confluent.io>
Closes #1889 from ijuma/increase-zk-timeout-in-tests
commit 106a7456060750ab0604d290b8c1e33a57adbf20
Author: Jason Gustafson <ja...@confluent.io>
Date: 2016-09-22T22:03:49Z
MINOR: Add test cases for delays in consumer rebalance listener
Author: Jason Gustafson <ja...@confluent.io>
Reviewers: Ismael Juma <is...@juma.me.uk>
Closes #1866 from hachikuji/rebalance-delay-test-cases
commit 6a13a3dbaddf99850b2583007577fa2a6e1e6d3a
Author: Jason Gustafson <ja...@confluent.io>
Date: 2016-09-23T20:13:29Z
KAFKA-3590; Handle not-enough-replicas errors when writing to offsets topic
Author: Jason Gustafson <ja...@confluent.io>
Reviewers: Ismael Juma <is...@juma.me.uk>, Guozhang Wang <wa...@gmail.com>
Closes #1859 from hachikuji/KAFKA-3590
commit 0de807357be4d776747e960b5fd17ddeae9a6cad
Author: Ryan Pridgeon <ry...@gmail.com>
Date: 2016-09-23T21:20:41Z
KAFKA-3719; Allow underscores in hostname
Technically this does not strictly adhere to RFC-952 however it is valid for domain names, urls and uris so we should loosen the requirements a tad.
Author: Ryan Pridgeon <ry...@gmail.com>
Reviewers: Ismael Juma <is...@juma.me.uk>
Closes #1856 from rnpridgeon/KAFKA-3719
commit 36242b846a42b33d7d4c1931f2dae93ebe1547c7
Author: Sumit Arrawatia <su...@gmail.com>
Date: 2016-09-24T09:16:49Z
KAFKA-4151; Update public docs for Cluster Id (KIP-78)
- Updated implementation docs with details on Cluster Id generation.
- Mention cluster id in "noteworthy changes for 0.10.1.0" in upgrade docs.
Author: Sumit Arrawatia <su...@gmail.com>
Author: arrawatia <su...@gmail.com>
Reviewers: Ismael Juma <is...@juma.me.uk>
Closes #1895 from arrawatia/kip-78-docs
commit 1d055f7551d138324d2540095a1cfc1c8f74d76f
Author: Arun Mahadevan <ai...@hortonworks.com>
Date: 2016-09-25T07:44:56Z
KAFKA-3282; Change tools to use new consumer if zookeeper is not specified
Author: Arun Mahadevan <ai...@hortonworks.com>
Reviewers: Jason Gustafson <ja...@confluent.io>, Ismael Juma <is...@juma.me.uk>
Closes #1376 from arunmahadevan/cons-consumer-fix
commit c0a62b70a8eadc550c937bb18e0203ab691618f5
Author: Rajini Sivaram <ra...@googlemail.com>
Date: 2016-09-26T00:01:45Z
KAFKA-4055; System tests for secure quotas
Fix existing client-id quota test which currently don't configure quota overrides correctly. Add new tests for user and (user, client-id) quota overrides and default quotas.
Author: Rajini Sivaram <ra...@googlemail.com>
Reviewers: Jun Rao <ju...@gmail.com>
Closes #1860 from rajinisivaram/KAFKA-4055
commit b75245cfbbefc712103b9329da0f27a205baa6aa
Author: Jason Gustafson <ja...@confluent.io>
Date: 2016-09-26T21:54:01Z
MINOR: Wakeups propagated from commitOffsets in WorkerSinkTask should be caught
Author: Jason Gustafson <ja...@confluent.io>
Reviewers: Ewen Cheslack-Postava <ew...@confluent.io>
Closes #1907 from hachikuji/catch-wakeup-worker-sink-task
commit be6056abc9970600347c95c4c8658799b76dbe6b
Author: Apurva Mehta <ap...@gmail.com>
Date: 2016-09-27T00:18:18Z
KAFKA-4214; kafka-reassign-partitions fails all the time when brokers are bounced during reassignment
There is a corner case bug, where during partition reassignment, if the
controller and a broker receiving a new replica are bounced at the same
time, the partition reassignment is failed.
The cause of this bug is a block of code in the KafkaController which
fails the reassignment if the aliveNewReplicas != newReplicas, ie. if
some of the new replicas are offline at the time a controller fails
over.
The fix is to have the controller listen for ISR change events even for
new replicas which are not alive when the controller boots up. Once the
said replicas come online, they will be in the ISR set, and the new
controller will detect this, and then mark the reassignment as
successful.
Interestingly, the block of code in question was introduced in
KAFKA-990, where a concern about this exact scenario was raised :)
This bug was revealed in the system tests in https://github.com/apache/kafka/pull/1904.
The relevant tests will be enabled in either this or a followup PR when PR-1904 is merged.
Thanks to junrao identifying the issue and providing the patch.
Author: Apurva Mehta <ap...@gmail.com>
Reviewers: Jun Rao <ju...@gmail.com>
Closes #1910 from apurvam/KAFKA-4214
commit 60ad6d727861a87fa756918a7be7547e9b1f4c3d
Author: Vahid Hashemian <va...@us.ibm.com>
Date: 2016-09-27T02:04:22Z
KAFKA-3831; Prepare for updating new-consumer-based Mirror Maker's default partition assignment strategy to round robin
This patch adds proper warning message and necessary doc updates for updating the default partition assignment strategy of Mirror Maker from range to round robin. The actual switch would occur as part of a major release cycle (to be scheduled).
Author: Vahid Hashemian <va...@us.ibm.com>
Reviewers: Jason Gustafson <ja...@confluent.io>
Closes #1499 from vahidhashemian/KAFKA-3831
commit a208478f37f216324a87b530f18f30ecd79601c8
Author: Magnus Reftel <ma...@skatteetaten.no>
Date: 2016-09-27T04:15:25Z
MINOR: Fix Javadoc of @throws AuthorizationException for KafkaConsumer.poll
Author: Magnus Reftel <ma...@skatteetaten.no>
Reviewers: Jason Gustafson <ja...@confluent.io>
Closes #1901 from reftel/feature/poll_javadoc
commit 5d6408f6cfda3f8ab366195f69e90de048cde25d
Author: Ben Stopford <be...@gmail.com>
Date: 2016-09-27T13:00:44Z
KAFKA-4200; Fix throttle argument in kafka-reassign-partitions.sh
Simple jira which alters two things:
1. kafka-reassign-partitions --verify prints Throttle was removed regardless of whether a throttle was applied. It should only print this if the value was actually changed.
2. --verify should exception if the \u2014throttle argument. (check generate too)
To test this I extracted all validation logic into a separate method and added a test which covers the majority of combinations. The validation logic was retained as is, other than implementing (2) and adding validation to the --broker-list option which you can currently apply to any of hte main actions (where it is ignored). Requirement 1 was tested manually (as it's just println).
Testing:
- Build passes locally.
- System test reassign_partitions_test.py also passes.
Author: Ben Stopford <be...@gmail.com>
Reviewers: Jun Rao <ju...@gmail.com>
Closes #1896 from benstopford/KAFKA-4200
commit b8ed4a51134ca8b98d1445871a55ed33b6ad5b92
Author: Ben Stopford <be...@gmail.com>
Date: 2016-09-27T19:27:20Z
KAFKA-4177; Remove ThrottledReplicationRateLimit from Server Config
This small PR pulls ThrottledReplicationRateLimit out of KafkaConfig and puts it in a class that defines Dynamic Configs. Client configs are also placed in this class and validation added.
Author: Ben Stopford <be...@gmail.com>
Reviewers: Jun Rao <ju...@gmail.com>, Ismael Juma <is...@juma.me.uk>
Closes #1864 from benstopford/KAFKA-4177
commit 3db752a565071c78e4b11eaafa739844fa785b04
Author: Jason Gustafson <ja...@confluent.io>
Date: 2016-09-27T20:35:29Z
MINOR: Make new consumer default for Mirror Maker
Author: Jason Gustafson <ja...@confluent.io>
Reviewers: Ismael Juma <is...@juma.me.uk>
Closes #1914 from hachikuji/mm-default-new-consumer
commit 61d3378bc84914a521a65cdfffb7299928fa8671
Author: Ismael Juma <is...@juma.me.uk>
Date: 2016-09-27T23:49:46Z
MINOR: Remove no longer required --new-consumer switch in docs
Author: Ismael Juma <is...@juma.me.uk>
Reviewers: Jason Gustafson <ja...@confluent.io>
Closes #1905 from ijuma/no-new-consumer-switch-in-examples
commit 0c25c73782e6e70b8f37e3dda2fa2a5b0b1c8c65
Author: Damian Guy <da...@gmail.com>
Date: 2016-09-28T00:35:24Z
HOTFIX: fix npe in StreamsMetadataState when onChange has not been called
If some StreamsMetadataState methods are called before the onChange method is called a NullPointerException was being thrown. Added null check for cluster in isInitialized method
Author: Damian Guy <da...@gmail.com>
Reviewers: Guozhang Wang <wa...@gmail.com>
Closes #1920 from dguy/fix-npe-streamsmetadata
commit 0c4cc5a4466d0dfce01903c29d97af3e428597f3
Author: Guozhang Wang <wa...@gmail.com>
Date: 2016-09-28T00:37:12Z
MINOR: fixes a few error logging formats
Author: Guozhang Wang <wa...@gmail.com>
Reviewers: Eno Thereska <en...@gmail.com>
Closes #1919 from guozhangwang/minor-error-message-fixes
commit c526c0c3f6f153ef8f84a3e66d9f997d8c31e20c
Author: Damian Guy <da...@gmail.com>
Date: 2016-09-28T00:43:36Z
MINOR: add test to make sure ProcessorStateManager can handle State Stores with logging disabled
Adding the test so we know that the State Stores with logging disabled or without a topic don't throw any exceptions.
Author: Damian Guy <da...@gmail.com>
Reviewers: Guozhang Wang <wa...@gmail.com>
Closes #1916 from dguy/state-store-logging-disabled
commit d83cde7cabe4e86951c6760e68e65b99752cfe0e
Author: Damian Guy <da...@gmail.com>
Date: 2016-09-28T18:07:44Z
KAFKA-3708: Better exception handling in Kafka Streams
KafkaExceptions currently thrown from within StreamThread/StreamTask currently bubble up without any additional context. This makes it hard to figure out where something went wrong, i.e, which topic had the serialization exception etc
Author: Damian Guy <da...@gmail.com>
Reviewers: Guozhang Wang <wa...@gmail.com>
Closes #1819 from dguy/kafka-3708 and squashes the following commits:
d6feaa8 [Damian Guy] address comments
15b89e7 [Damian Guy] merge trunk
6b8a8af [Damian Guy] catch exceptions in various places and throw more informative versions
c86eeda [Damian Guy] fix conflicts
8f37e2c [Damian Guy] add some context to exceptions
commit 281fac9ed626b16190d56e609c9bf04a3188374a
Author: Rajini Sivaram <ra...@googlemail.com>
Date: 2016-09-28T19:18:20Z
KAFKA-4227; Shutdown AdminManager when KafkaServer is shutdown
Terminate topic purgatory thread in AdminManager during server shutdown to avoid threads being left around in unit tests.
Author: Rajini Sivaram <ra...@googlemail.com>
Reviewers: Ismael Juma <is...@juma.me.uk>
Closes #1927 from rajinisivaram/KAFKA-4227
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] kafka pull request #2415: KAFKA-4547 (0.10.1 hotfix): Avoid unnecessary offs...
Posted by vahidhashemian <gi...@git.apache.org>.
Github user vahidhashemian closed the pull request at:
https://github.com/apache/kafka/pull/2415
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---