You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/06/02 13:26:10 UTC
[jira] [Commented] (KAFKA-5919) Adding checks on "version" field
for tools using it
[ https://issues.apache.org/jira/browse/KAFKA-5919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16499062#comment-16499062 ]
ASF GitHub Bot commented on KAFKA-5919:
---------------------------------------
ppatierno closed pull request #3887: KAFKA-5919: Adding checks on "version" field for tools using it
URL: https://github.com/apache/kafka/pull/3887
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/.gitignore b/.gitignore
index 60883492f31..04f8feed0ad 100644
--- a/.gitignore
+++ b/.gitignore
@@ -47,3 +47,7 @@ tests/venv
docs/generated/
.release-settings.json
+
+kafkatest.egg-info/
+systest/
+*.swp
diff --git a/README.md b/README.md
index 9371e482169..18cb03e9880 100644
--- a/README.md
+++ b/README.md
@@ -4,9 +4,9 @@ See our [web site](http://kafka.apache.org) for details on the project.
You need to have [Gradle](http://www.gradle.org/installation) and [Java](http://www.oracle.com/technetwork/java/javase/downloads/index.html) installed.
-Kafka requires Gradle 3.0 or higher.
+Kafka requires Gradle 4.5 or higher.
-Java 7 should be used for building in order to support both Java 7 and Java 8 at runtime.
+Java 8 should be used for building in order to support both Java 8 and Java 10 at runtime.
### First bootstrap and download the wrapper ###
cd kafka_source_dir
@@ -85,8 +85,6 @@ You can pass either the major version (eg 2.11) or the full version (eg 2.11.12)
./gradlew -PscalaVersion=2.11 test
./gradlew -PscalaVersion=2.11 releaseTarGz
-Scala 2.12.x requires Java 8.
-
### Running a task for a specific project ###
This is for `core`, `examples` and `clients`
diff --git a/TROGDOR.md b/TROGDOR.md
new file mode 100644
index 00000000000..b8c94d4bc7c
--- /dev/null
+++ b/TROGDOR.md
@@ -0,0 +1,154 @@
+Trogdor
+========================================
+Trogdor is a test framework for Apache Kafka.
+
+Trogdor can run benchmarks and other workloads. Trogdor can also inject faults in order to stress test the system.
+
+Quickstart
+=========================================================
+First, we want to start a single-node Kafka cluster with a ZooKeeper and a broker.
+
+Running ZooKeeper:
+
+ > ./bin/zookeeper-server-start.sh ./config/zookeeper.properties &> /tmp/zookeeper.log &
+
+Running Kafka:
+
+ > ./bin/kafka-server-start.sh ./config/server.properties &> /tmp/kafka.log &
+
+Then, we want to run a Trogdor Agent, plus a Trogdor broker.
+
+To run the Trogdor Agent:
+
+ > ./bin/trogdor.sh agent -c ./config/trogdor.conf -n node0 &> /tmp/trogdor-agent.log &
+
+To run the Trogdor Coordinator:
+
+ > ./bin/trogdor.sh coordinator -c ./config/trogdor.conf -n node0 &> /tmp/trogdor-coordinator.log &
+
+Let's confirm that all of the daemons are running:
+
+ > jps
+ 116212 Coordinator
+ 115188 QuorumPeerMain
+ 116571 Jps
+ 115420 Kafka
+ 115694 Agent
+
+Now, we can submit a test job to Trogdor. Here's an example of a short bash script which makes it easier.
+
+ > ./tests/bin/trogdor-run-produce-bench.sh
+ [2018-04-12 10:32:04,055] DEBUG Sending POST with input {"id":"produce_bench_22137","spec":{"class":"org.apache.kafka.trogdor.workload.ProduceBenchSpec","startMs":0,"durationMs":10000000,"producerNode":"node0","bootstrapServers":"localhost:9092","targetMessagesPerSec":10,"maxMessages":100,"keyGenerator":{"type":"sequential","size":4,"startOffset":0},"valueGenerator":{"type":"constant","size":512,"value":"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="},"totalTopics":10,"activeTopics":5,"topicPrefix":"foo","replicationFactor":1,"classLoader":{},"numPartitions":1}} to http://localhost:8889/coordinator/task/create (org.apache.kafka.trogdor.coordinator.CoordinatorClient)
+ Created task.
+ $TASK_ID = produce_bench_20462
+
+To get the test results, we run --show-tasks:
+
+ ./tests/bin/trogdor.sh client --show-tasks localhost:8889
+ Got coordinator tasks: {
+ "tasks" : {
+ "produce_bench_20462" : {
+ "state" : "DONE",
+ "spec" : {
+ "class" : "org.apache.kafka.trogdor.workload.ProduceBenchSpec",
+ "startMs" : 0,
+ "durationMs" : 10000000,
+ "producerNode" : "node0",
+ "bootstrapServers" : "localhost:9092",
+ "targetMessagesPerSec" : 10,
+ "maxMessages" : 100,
+ "keyGenerator" : {
+ "type" : "sequential",
+ "size" : 4,
+ "startOffset" : 0
+ },
+ "valueGenerator" : {
+ "type" : "constant",
+ "size" : 512,
+ "value" : "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="
+ },
+ "totalTopics" : 10,
+ "activeTopics" : 5,
+ "topicPrefix" : "foo",
+ "replicationFactor" : 1,
+ "classLoader" : { },
+ "numPartitions" : 1
+ },
+ "startedMs" : 1523552769850,
+ "doneMs" : 1523552780878,
+ "cancelled" : false,
+ "status" : {
+ "totalSent" : 500,
+ "averageLatencyMs" : 4.972,
+ "p50LatencyMs" : 4,
+ "p95LatencyMs" : 6,
+ "p99LatencyMs" : 12
+ }
+ }
+ }
+ }
+
+Trogdor Architecture
+========================================
+Trogdor has a single coordinator process which manages multiple agent processes. Each agent process is responsible for a single cluster node.
+
+The Trogdor coordinator manages tasks. A task is anything we might want to do on a cluster, such as running a benchmark, injecting a fault, or running a workload. In order to implement each task, the coordinator creates workers on one or more agent nodes.
+
+The Trogdor agent process implements the tasks. For example, when running a workload, the agent process is the process which produces and consumes messages.
+
+Both the coordinator and the agent expose a REST interface that accepts objects serialized via JSON. There is also a command-line program which makes it easy to send messages to either one without manually crafting the JSON message body.
+
+All Trogdor RPCs are idempotent except the shutdown requests. Sending an idempotent RPC twice in a row has the same effect as sending the RPC once.
+
+Tasks
+========================================
+Tasks are described by specifications containing:
+
+* A "class" field describing the task type. This contains a full Java class name.
+* A "startMs" field describing when the task should start. This is given in terms of milliseconds since the UNIX epoch.
+* A "durationMs" field describing how long the task should last. This is given in terms of milliseconds.
+* Other fields which are task-specific.
+
+The task specification is usually written as JSON. For example, this task specification describes a network partition between nodes 1 and 2, and 3:
+
+ {
+ "class": "org.apache.kafka.trogdor.fault.NetworkPartitionFaultSpec",
+ "startMs": 1000,
+ "durationMs": 30000,
+ "partitions": [["node1", "node2"], ["node3"]]
+ }
+
+Tasks are submitted to the coordinator. Once the coordinator determines that it is time for the task to start, it creates workers on agent processes. The workers run until the task is done.
+
+Task specifications are immutable; they do not change after the task has been created.
+
+Tasks can be in several states:
+* PENDING, when task is waiting to execute,
+* RUNNING, when the task is running,
+* STOPPING, when the task is in the process of stopping,
+* DONE, when the task is done.
+
+Tasks that are DONE also have an error field which will be set if the task failed.
+
+Workloads
+========================================
+Trogdor can run several workloads. Workloads perform operations on the cluster and measure their performance. Workloads fail when the operations cannot be performed.
+
+### ProduceBench
+ProduceBench starts a Kafka producer on a single agent node, producing to several partitions. The workload measures the average produce latency, as well as the median, 95th percentile, and 99th percentile latency.
+
+### RoundTripWorkload
+RoundTripWorkload tests both production and consumption. The workload starts a Kafka producer and consumer on a single node. The consumer will read back the messages that were produced by the producer.
+
+### ConsumeBench
+ConsumeBench starts a Kafka consumer on a single agent node. The workload measures the average produce latency, as well as the median, 95th percentile, and 99th percentile latency.
+
+Faults
+========================================
+Trogdor can run several faults which deliberately break something in the cluster.
+
+### ProcessStopFault
+ProcessStopFault stops a process by sending it a SIGSTOP signal. When the fault ends, the process is resumed with SIGCONT.
+
+### NetworkPartitionFault
+NetworkPartitionFault sets up an artificial network partition between one or more sets of nodes. Currently, this is implemented using iptables. The iptables rules are set up on the outbound traffic from the affected nodes. Therefore, the affected nodes should still be reachable from outside the cluster.
diff --git a/Vagrantfile b/Vagrantfile
index 2179a3c37ec..363607691fd 100644
--- a/Vagrantfile
+++ b/Vagrantfile
@@ -40,7 +40,7 @@ ec2_keypair_file = nil
ec2_region = "us-east-1"
ec2_az = nil # Uses set by AWS
-ec2_ami = "ami-9eaa1cf6"
+ec2_ami = "ami-905730e8"
ec2_instance_type = "m3.medium"
ec2_user = "ubuntu"
ec2_instance_name_prefix = "kafka-vagrant"
@@ -80,7 +80,7 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
# share to a temporary location and the provisioning scripts symlink data
# to the right location.
override.cache.enable :generic, {
- "oracle-jdk7" => { cache_dir: "/tmp/oracle-jdk7-installer-cache" },
+ "oracle-jdk8" => { cache_dir: "/tmp/oracle-jdk8-installer-cache" },
}
end
end
@@ -141,7 +141,10 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
def name_node(node, name, ec2_instance_name_prefix)
node.vm.hostname = name
node.vm.provider :aws do |aws|
- aws.tags = { 'Name' => ec2_instance_name_prefix + "-" + Socket.gethostname + "-" + name }
+ aws.tags = {
+ 'Name' => ec2_instance_name_prefix + "-" + Socket.gethostname + "-" + name,
+ 'JenkinsBuildUrl' => ENV['BUILD_URL']
+ }
end
end
diff --git a/bin/kafka-dump-log.sh b/bin/kafka-dump-log.sh
new file mode 100755
index 00000000000..a97ea7d3d9f
--- /dev/null
+++ b/bin/kafka-dump-log.sh
@@ -0,0 +1,17 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+exec $(dirname $0)/kafka-run-class.sh kafka.tools.DumpLogSegments "$@"
diff --git a/bin/kafka-run-class.sh b/bin/kafka-run-class.sh
index 6676d0d2156..0ea8b831946 100755
--- a/bin/kafka-run-class.sh
+++ b/bin/kafka-run-class.sh
@@ -69,28 +69,50 @@ do
fi
done
-for file in "$base_dir"/clients/build/libs/kafka-clients*.jar;
-do
- if should_include_file "$file"; then
- CLASSPATH="$CLASSPATH":"$file"
- fi
-done
+if [ -z "$UPGRADE_KAFKA_STREAMS_TEST_VERSION" ]; then
+ clients_lib_dir=$(dirname $0)/../clients/build/libs
+ streams_lib_dir=$(dirname $0)/../streams/build/libs
+ rocksdb_lib_dir=$(dirname $0)/../streams/build/dependant-libs-${SCALA_VERSION}
+else
+ clients_lib_dir=/opt/kafka-$UPGRADE_KAFKA_STREAMS_TEST_VERSION/libs
+ streams_lib_dir=$clients_lib_dir
+ rocksdb_lib_dir=$streams_lib_dir
+fi
+
-for file in "$base_dir"/streams/build/libs/kafka-streams*.jar;
+for file in "$clients_lib_dir"/kafka-clients*.jar;
do
if should_include_file "$file"; then
CLASSPATH="$CLASSPATH":"$file"
fi
done
-for file in "$base_dir"/streams/examples/build/libs/kafka-streams-examples*.jar;
+for file in "$streams_lib_dir"/kafka-streams*.jar;
do
if should_include_file "$file"; then
CLASSPATH="$CLASSPATH":"$file"
fi
done
-for file in "$base_dir"/streams/build/dependant-libs-${SCALA_VERSION}/rocksdb*.jar;
+if [ -z "$UPGRADE_KAFKA_STREAMS_TEST_VERSION" ]; then
+ for file in "$base_dir"/streams/examples/build/libs/kafka-streams-examples*.jar;
+ do
+ if should_include_file "$file"; then
+ CLASSPATH="$CLASSPATH":"$file"
+ fi
+ done
+else
+ VERSION_NO_DOTS=`echo $UPGRADE_KAFKA_STREAMS_TEST_VERSION | sed 's/\.//g'`
+ SHORT_VERSION_NO_DOTS=${VERSION_NO_DOTS:0:((${#VERSION_NO_DOTS} - 1))} # remove last char, ie, bug-fix number
+ for file in "$base_dir"/streams/upgrade-system-tests-$SHORT_VERSION_NO_DOTS/build/libs/kafka-streams-upgrade-system-tests*.jar;
+ do
+ if should_include_file "$file"; then
+ CLASSPATH="$CLASSPATH":"$file"
+ fi
+ done
+fi
+
+for file in "$rocksdb_lib_dir"/rocksdb*.jar;
do
CLASSPATH="$CLASSPATH":"$file"
done
@@ -137,7 +159,7 @@ done
shopt -u nullglob
if [ -z "$CLASSPATH" ] ; then
- echo "Classpath is empty. Please build the project first e.g. by running './gradlew jar -Pscala_version=$SCALA_VERSION'"
+ echo "Classpath is empty. Please build the project first e.g. by running './gradlew jar -PscalaVersion=$SCALA_VERSION'"
exit 1
fi
@@ -216,6 +238,12 @@ if [ -z "$KAFKA_JVM_PERFORMANCE_OPTS" ]; then
KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true"
fi
+# version option
+for args in "$@" ; do
+ if [ "$args" = "--version" ]; then
+ exec $JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "kafka.utils.VersionInfo"
+ fi
+done
while [ $# -gt 0 ]; do
COMMAND=$1
@@ -246,9 +274,16 @@ GC_FILE_SUFFIX='-gc.log'
GC_LOG_FILE_NAME=''
if [ "x$GC_LOG_ENABLED" = "xtrue" ]; then
GC_LOG_FILE_NAME=$DAEMON_NAME$GC_FILE_SUFFIX
- # the first segment of the version number, which is '1' for releases before Java 9
+
+ # The first segment of the version number, which is '1' for releases before Java 9
# it then becomes '9', '10', ...
- JAVA_MAJOR_VERSION=$($JAVA -version 2>&1 | sed -E -n 's/.* version "([^.-]*).*"/\1/p')
+ # Some examples of the first line of `java --version`:
+ # 8 -> java version "1.8.0_152"
+ # 9.0.4 -> java version "9.0.4"
+ # 10 -> java version "10" 2018-03-20
+ # 10.0.1 -> java version "10.0.1" 2018-04-17
+ # We need to match to the end of the line to prevent sed from printing the characters that do not match
+ JAVA_MAJOR_VERSION=$($JAVA -version 2>&1 | sed -E -n 's/.* version "([0-9]*).*$/\1/p')
if [[ "$JAVA_MAJOR_VERSION" -ge "9" ]] ; then
KAFKA_GC_LOG_OPTS="-Xlog:gc*:file=$LOG_DIR/$GC_LOG_FILE_NAME:time,tags:filecount=10,filesize=102400"
else
diff --git a/bin/windows/kafka-consumer-offset-checker.bat b/bin/windows/kafka-dump-log.bat
similarity index 92%
rename from bin/windows/kafka-consumer-offset-checker.bat
rename to bin/windows/kafka-dump-log.bat
index 2baa1b84061..3a1473dc61b 100644
--- a/bin/windows/kafka-consumer-offset-checker.bat
+++ b/bin/windows/kafka-dump-log.bat
@@ -14,4 +14,4 @@ rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
rem See the License for the specific language governing permissions and
rem limitations under the License.
-"%~dp0kafka-run-class.bat" kafka.tools.ConsumerOffsetChecker %*
+"%~dp0kafka-run-class.bat" kafka.tools.DumpLogSegments %*
diff --git a/build.gradle b/build.gradle
index 5e4c35643c2..4f3fd770b78 100644
--- a/build.gradle
+++ b/build.gradle
@@ -26,9 +26,9 @@ buildscript {
// For Apache Rat plugin to ignore non-Git files
classpath "org.ajoberstar:grgit:1.9.3"
classpath 'com.github.ben-manes:gradle-versions-plugin:0.17.0'
- classpath 'org.scoverage:gradle-scoverage:2.1.0'
- classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.2'
- classpath 'org.owasp:dependency-check-gradle:3.0.2'
+ classpath 'org.scoverage:gradle-scoverage:2.3.0'
+ classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.4'
+ classpath 'org.owasp:dependency-check-gradle:3.1.2'
}
}
@@ -41,7 +41,6 @@ allprojects {
}
apply plugin: 'idea'
- apply plugin: "jacoco"
apply plugin: 'org.owasp.dependencycheck'
apply plugin: 'com.github.ben-manes.versions'
@@ -68,24 +67,19 @@ allprojects {
}
}
- if (JavaVersion.current().isJava8Compatible()) {
- tasks.withType(Javadoc) {
- // disable the crazy super-strict doclint tool in Java 8
- // noinspection SpellCheckingInspection
- options.addStringOption('Xdoclint:none', '-quiet')
- }
+ tasks.withType(Javadoc) {
+ // disable the crazy super-strict doclint tool in Java 8
+ // noinspection SpellCheckingInspection
+ options.addStringOption('Xdoclint:none', '-quiet')
}
}
ext {
- gradleVersion = "4.5.1"
+ gradleVersion = "4.7"
+ minJavaVersion = "8"
buildVersionFileName = "kafka-version.properties"
- maxPermSizeArgs = []
- if (!JavaVersion.current().isJava8Compatible())
- maxPermSizeArgs += '-XX:MaxPermSize=512m'
-
userMaxForks = project.hasProperty('maxParallelForks') ? maxParallelForks.toInteger() : null
skipSigning = project.hasProperty('skipSigning') && skipSigning.toBoolean()
@@ -121,11 +115,13 @@ if (new File('.git').exists()) {
'PULL_REQUEST_TEMPLATE.md',
'gradlew',
'gradlew.bat',
+ 'TROGDOR.md',
'**/README.md',
'**/id_rsa',
'**/id_rsa.pub',
'checkstyle/suppressions.xml',
- 'streams/quickstart/java/src/test/resources/projects/basic/goal.txt'
+ 'streams/quickstart/java/src/test/resources/projects/basic/goal.txt',
+ 'streams/streams-scala/logs/*'
])
}
}
@@ -144,20 +140,17 @@ subprojects {
if (!JavaVersion.current().isJava9Compatible())
apply plugin: 'findbugs'
- sourceCompatibility = 1.7
- targetCompatibility = 1.7
+ sourceCompatibility = minJavaVersion
+ targetCompatibility = minJavaVersion
compileJava {
options.encoding = 'UTF-8'
- options.compilerArgs << "-Xlint:deprecation"
- // -Xlint:unchecked is too buggy in Java 7, so we only enable for Java 8 or higher
- if (JavaVersion.current().isJava8Compatible())
- options.compilerArgs << "-Xlint:unchecked"
+ options.compilerArgs << "-Xlint:deprecation,unchecked"
// --release is the recommended way to select the target release, but it's only supported in Java 9 so we also
// set --source and --target via `sourceCompatibility` and `targetCompatibility`. If/when Gradle supports `--release`
// natively (https://github.com/gradle/gradle/issues/2510), we should switch to that.
if (JavaVersion.current().isJava9Compatible())
- options.compilerArgs << "--release" << "7"
+ options.compilerArgs << "--release" << minJavaVersion
}
uploadArchives {
@@ -201,7 +194,6 @@ subprojects {
minHeapSize = "256m"
maxHeapSize = "2048m"
- jvmArgs = maxPermSizeArgs
testLogging {
events = userTestLoggingEvents ?: testLoggingEvents
@@ -216,7 +208,6 @@ subprojects {
minHeapSize = "256m"
maxHeapSize = "2048m"
- jvmArgs = maxPermSizeArgs
testLogging {
events = userTestLoggingEvents ?: testLoggingEvents
@@ -235,7 +226,6 @@ subprojects {
minHeapSize = "256m"
maxHeapSize = "2048m"
- jvmArgs = maxPermSizeArgs
testLogging {
events = userTestLoggingEvents ?: testLoggingEvents
@@ -342,7 +332,7 @@ subprojects {
"-Xlint:private-shadow",
"-Xlint:stars-align",
"-Xlint:type-parameter-shadow",
- "-Xlint:unsound-match",
+ "-Xlint:unsound-match"
]
if (versions.baseScala != '2.11') {
@@ -354,15 +344,14 @@ subprojects {
configure(scalaCompileOptions.forkOptions) {
memoryMaximumSize = '1g'
- jvmArgs = ['-Xss2m'] + maxPermSizeArgs
+ jvmArgs = ['-Xss2m']
}
}
checkstyle {
configFile = new File(rootDir, "checkstyle/checkstyle.xml")
configProperties = [importControlFile: "$rootDir/checkstyle/import-control.xml"]
- // version 7.x requires Java 8
- toolVersion = '6.19'
+ toolVersion = '8.10'
}
test.dependsOn('checkstyleMain', 'checkstyleTest')
@@ -384,9 +373,10 @@ subprojects {
// Ignore core since its a scala project
if (it.path != ':core') {
+ apply plugin: "jacoco"
jacoco {
- toolVersion = "0.8.0"
+ toolVersion = "0.8.1"
}
// NOTE: Jacoco Gradle plugin does not support "offline instrumentation" this means that classes mocked by PowerMock
@@ -518,7 +508,7 @@ for ( sv in availableScalaVersions ) {
}
def connectPkgs = ['connect:api', 'connect:runtime', 'connect:transforms', 'connect:json', 'connect:file']
-def pkgs = ['clients', 'examples', 'log4j-appender', 'tools', 'streams', 'streams:test-utils', 'streams:examples'] + connectPkgs
+def pkgs = ['clients', 'examples', 'log4j-appender', 'tools', 'streams', 'streams:streams-scala', 'streams:test-utils', 'streams:examples'] + connectPkgs
/** Create one task per default Scala version */
def withDefScalaVersions(taskName) {
@@ -591,8 +581,7 @@ project(':core') {
scoverage libs.scoveragePlugin
scoverage libs.scoverageRuntime
}
-
- jacocoTestReport.enabled = false
+
scoverage {
reportDir = file("${rootProject.buildDir}/scoverage")
highlighting = false
@@ -634,6 +623,13 @@ project(':core') {
standardOutput = new File(generatedDocsDir, "protocol_errors.html").newOutputStream()
}
+ task genProtocolTypesDocs(type: JavaExec) {
+ classpath = sourceSets.main.runtimeClasspath
+ main = 'org.apache.kafka.common.protocol.types.Type'
+ if( !generatedDocsDir.exists() ) { generatedDocsDir.mkdirs() }
+ standardOutput = new File(generatedDocsDir, "protocol_types.html").newOutputStream()
+ }
+
task genProtocolApiKeyDocs(type: JavaExec) {
classpath = sourceSets.main.runtimeClasspath
main = 'org.apache.kafka.common.protocol.ApiKeys'
@@ -697,7 +693,7 @@ project(':core') {
standardOutput = new File(generatedDocsDir, "producer_metrics.html").newOutputStream()
}
- task siteDocsTar(dependsOn: ['genProtocolErrorDocs', 'genProtocolApiKeyDocs', 'genProtocolMessageDocs',
+ task siteDocsTar(dependsOn: ['genProtocolErrorDocs', 'genProtocolTypesDocs', 'genProtocolApiKeyDocs', 'genProtocolMessageDocs',
'genAdminClientConfigDocs', 'genProducerConfigDocs', 'genConsumerConfigDocs',
'genKafkaConfigDocs', 'genTopicConfigDocs',
':connect:runtime:genConnectConfigDocs', ':connect:runtime:genConnectTransformationDocs',
@@ -734,6 +730,8 @@ project(':core') {
from(project(':connect:file').configurations.runtime) { into("libs/") }
from(project(':streams').jar) { into("libs/") }
from(project(':streams').configurations.runtime) { into("libs/") }
+ from(project(':streams:streams-scala').jar) { into("libs/") }
+ from(project(':streams:streams-scala').configurations.runtime) { into("libs/") }
from(project(':streams:test-utils').jar) { into("libs/") }
from(project(':streams:test-utils').configurations.runtime) { into("libs/") }
from(project(':streams:examples').jar) { into("libs/") }
@@ -789,6 +787,7 @@ project(':clients') {
compile libs.lz4
compile libs.snappy
compile libs.slf4jApi
+ compile libs.jacksonDatabind // for SASL/OAUTHBEARER bearer token parsing
testCompile libs.bcpkix
testCompile libs.junit
@@ -859,6 +858,8 @@ project(':clients') {
include "**/org/apache/kafka/common/config/*"
include "**/org/apache/kafka/common/security/auth/*"
include "**/org/apache/kafka/server/policy/*"
+ include "**/org/apache/kafka/common/security/token/delegation/*"
+ include "**/org/apache/kafka/common/security/oauthbearer/*"
}
}
@@ -874,6 +875,9 @@ project(':tools') {
compile libs.jacksonJaxrsJsonProvider
compile libs.jerseyContainerServlet
+ compile libs.jerseyHk2
+ compile libs.jaxbApi // Jersey dependency that was available in the JDK before Java 9
+ compile libs.activation // Jersey dependency that was available in the JDK before Java 9
compile libs.jettyServer
compile libs.jettyServlet
compile libs.jettyServlets
@@ -918,13 +922,17 @@ project(':streams') {
compile libs.slf4jApi
compile libs.rocksDBJni
+ // testCompileOnly prevents streams from exporting a dependency on test-utils, which would cause a dependency cycle
+ testCompileOnly project(':streams:test-utils')
testCompile project(':clients').sourceSets.test.output
testCompile project(':core')
testCompile project(':core').sourceSets.test.output
+ testCompile libs.log4j
testCompile libs.junit
testCompile libs.easymock
testCompile libs.bcpkix
+ testRuntimeOnly project(':streams:test-utils')
testRuntime libs.slf4jlog4j
}
@@ -961,6 +969,45 @@ project(':streams') {
}
}
+project(':streams:streams-scala') {
+ println "Building project 'streams-scala' with Scala version ${versions.scala}"
+ apply plugin: 'scala'
+ archivesBaseName = "kafka-streams-scala_${versions.baseScala}"
+
+ dependencies {
+ compile project(':streams')
+
+ compile libs.scalaLibrary
+
+ testCompile project(':core')
+ testCompile project(':core').sourceSets.test.output
+ testCompile project(':streams').sourceSets.test.output
+ testCompile project(':clients').sourceSets.test.output
+
+ testCompile libs.junit
+ testCompile libs.scalatest
+
+ testRuntime libs.slf4jlog4j
+ }
+
+ javadoc {
+ include "**/org/apache/kafka/streams/scala/**"
+ }
+
+ tasks.create(name: "copyDependantLibs", type: Copy) {
+ from (configurations.runtime) {
+ exclude('kafka-streams*')
+ }
+ into "$buildDir/dependant-libs-${versions.scala}"
+ duplicatesStrategy 'exclude'
+ }
+
+ jar {
+ dependsOn 'copyDependantLibs'
+ }
+
+}
+
project(':streams:test-utils') {
archivesBaseName = "kafka-streams-test-utils"
@@ -1000,6 +1047,10 @@ project(':streams:examples') {
compile project(':streams')
compile project(':connect:json') // this dependency should be removed after we unify data API
compile libs.slf4jlog4j
+
+ testCompile project(':streams:test-utils')
+ testCompile project(':clients').sourceSets.test.output // for org.apache.kafka.test.IntegrationTest
+ testCompile libs.junit
}
javadoc {
@@ -1019,6 +1070,78 @@ project(':streams:examples') {
}
}
+project(':streams:upgrade-system-tests-0100') {
+ archivesBaseName = "kafka-streams-upgrade-system-tests-0100"
+
+ dependencies {
+ testCompile libs.kafkaStreams_0100
+ }
+
+ systemTestLibs {
+ dependsOn testJar
+ }
+}
+
+project(':streams:upgrade-system-tests-0101') {
+ archivesBaseName = "kafka-streams-upgrade-system-tests-0101"
+
+ dependencies {
+ testCompile libs.kafkaStreams_0101
+ }
+
+ systemTestLibs {
+ dependsOn testJar
+ }
+}
+
+project(':streams:upgrade-system-tests-0102') {
+ archivesBaseName = "kafka-streams-upgrade-system-tests-0102"
+
+ dependencies {
+ testCompile libs.kafkaStreams_0102
+ }
+
+ systemTestLibs {
+ dependsOn testJar
+ }
+}
+
+project(':streams:upgrade-system-tests-0110') {
+ archivesBaseName = "kafka-streams-upgrade-system-tests-0110"
+
+ dependencies {
+ testCompile libs.kafkaStreams_0110
+ }
+
+ systemTestLibs {
+ dependsOn testJar
+ }
+}
+
+project(':streams:upgrade-system-tests-10') {
+ archivesBaseName = "kafka-streams-upgrade-system-tests-10"
+
+ dependencies {
+ testCompile libs.kafkaStreams_10
+ }
+
+ systemTestLibs {
+ dependsOn testJar
+ }
+}
+
+project(':streams:upgrade-system-tests-11') {
+ archivesBaseName = "kafka-streams-upgrade-system-tests-11"
+
+ dependencies {
+ testCompile libs.kafkaStreams_11
+ }
+
+ systemTestLibs {
+ dependsOn testJar
+ }
+}
+
project(':jmh-benchmarks') {
apply plugin: 'com.github.johnrengelman.shadow'
@@ -1084,6 +1207,7 @@ project(':connect:api') {
dependencies {
compile project(':clients')
compile libs.slf4jApi
+ compile libs.jerseyContainerServlet
testCompile libs.junit
@@ -1195,14 +1319,19 @@ project(':connect:runtime') {
archivesBaseName = "connect-runtime"
dependencies {
+
compile project(':connect:api')
- compile project(":connect:transforms")
compile project(':clients')
compile project(':tools')
- compile libs.slf4jApi
+ compile project(':connect:json')
+ compile project(':connect:transforms')
+ compile libs.slf4jApi
compile libs.jacksonJaxrsJsonProvider
compile libs.jerseyContainerServlet
+ compile libs.jerseyHk2
+ compile libs.jaxbApi // Jersey dependency that was available in the JDK before Java 9
+ compile libs.activation // Jersey dependency that was available in the JDK before Java 9
compile libs.jettyServer
compile libs.jettyServlet
compile libs.jettyServlets
@@ -1216,7 +1345,6 @@ project(':connect:runtime') {
testCompile libs.powermockJunit4
testCompile libs.powermockEasymock
- testCompile project(":connect:json")
testCompile project(':clients').sourceSets.test.output
testRuntime libs.slf4jlog4j
@@ -1304,6 +1432,45 @@ project(':connect:file') {
}
}
+project(':connect:basic-auth-extension') {
+ archivesBaseName = "connect-basic-auth-extension"
+
+ dependencies {
+ compile project(':connect:api')
+ compile libs.slf4jApi
+
+ testCompile libs.bcpkix
+ testCompile libs.easymock
+ testCompile libs.junit
+ testCompile libs.powermockJunit4
+ testCompile libs.powermockEasymock
+ testCompile project(':clients').sourceSets.test.output
+
+ testRuntime libs.slf4jlog4j
+ }
+
+ javadoc {
+ enabled = false
+ }
+
+ tasks.create(name: "copyDependantLibs", type: Copy) {
+ from (configurations.testRuntime) {
+ include('slf4j-log4j12*')
+ include('log4j*jar')
+ }
+ from (configurations.runtime) {
+ exclude('kafka-clients*')
+ exclude('connect-*')
+ }
+ into "$buildDir/dependant-libs"
+ duplicatesStrategy 'exclude'
+ }
+
+ jar {
+ dependsOn copyDependantLibs
+ }
+}
+
task aggregatedJavadoc(type: Javadoc) {
def projectsWithJavadoc = subprojects.findAll { it.javadoc.enabled }
source = projectsWithJavadoc.collect { it.sourceSets.main.allJava }
diff --git a/checkstyle/checkstyle.xml b/checkstyle/checkstyle.xml
index ccab85ce4fe..ad8545027a0 100644
--- a/checkstyle/checkstyle.xml
+++ b/checkstyle/checkstyle.xml
@@ -105,7 +105,7 @@
</module>
<module name="ClassDataAbstractionCoupling">
<!-- default is 7 -->
- <property name="max" value="20"/>
+ <property name="max" value="25"/>
</module>
<module name="BooleanExpressionComplexity">
<!-- default is 3 -->
@@ -114,7 +114,7 @@
<module name="ClassFanOutComplexity">
<!-- default is 20 -->
- <property name="max" value="40"/>
+ <property name="max" value="50"/>
</module>
<module name="CyclomaticComplexity">
<!-- default is 10-->
diff --git a/checkstyle/import-control.xml b/checkstyle/import-control.xml
index 000acc38463..5549205bd13 100644
--- a/checkstyle/import-control.xml
+++ b/checkstyle/import-control.xml
@@ -101,6 +101,9 @@
<subpackage name="scram">
<allow pkg="javax.crypto" />
</subpackage>
+ <subpackage name="oauthbearer">
+ <allow pkg="com.fasterxml.jackson.databind" />
+ </subpackage>
</subpackage>
<subpackage name="protocol">
@@ -164,6 +167,8 @@
<subpackage name="admin">
<allow pkg="org.apache.kafka.clients.admin" />
+ <allow pkg="org.apache.kafka.clients.consumer.internals" />
+ <allow pkg="org.apache.kafka.clients.consumer" />
</subpackage>
</subpackage>
@@ -205,6 +210,8 @@
<allow pkg="org.apache.kafka.clients"/>
<allow pkg="org.apache.kafka.clients.producer" exact-match="true"/>
<allow pkg="org.apache.kafka.clients.consumer" exact-match="true"/>
+ <!-- Temporary until EosTestDriver migrates to the Java AdminClient -->
+ <allow pkg="kafka.admin" exact-match="true"/>
<allow pkg="org.apache.kafka.streams"/>
@@ -247,6 +254,10 @@
<allow pkg="org.I0Itec.zkclient" />
<allow pkg="com.fasterxml.jackson" />
<allow pkg="org.apache.zookeeper" />
+ <allow pkg="org.apache.zookeeper" />
+ <subpackage name="testutil">
+ <allow pkg="org.apache.log4j" />
+ </subpackage>
</subpackage>
</subpackage>
</subpackage>
@@ -279,6 +290,7 @@
<allow pkg="org.apache.kafka.connect.data" />
<allow pkg="org.apache.kafka.connect.errors" />
<allow pkg="org.apache.kafka.connect.header" />
+ <allow pkg="org.apache.kafka.connect.components"/>
<allow pkg="org.apache.kafka.clients" />
<allow pkg="org.apache.kafka.test"/>
@@ -296,7 +308,16 @@
<subpackage name="converters">
<allow pkg="org.apache.kafka.connect.storage" />
</subpackage>
-
+
+ <subpackage name="rest">
+ <allow pkg="org.apache.kafka.connect.health" />
+ <allow pkg="javax.ws.rs" />
+ <allow pkg= "javax.security.auth"/>
+ <subpackage name="basic">
+ <allow pkg="org.apache.kafka.connect.rest"/>
+ </subpackage>
+ </subpackage>
+
<subpackage name="runtime">
<allow pkg="org.apache.kafka.connect" />
<allow pkg="org.reflections"/>
@@ -316,6 +337,8 @@
</subpackage>
</subpackage>
+
+
<subpackage name="cli">
<allow pkg="org.apache.kafka.connect.runtime" />
<allow pkg="org.apache.kafka.connect.storage" />
diff --git a/checkstyle/suppressions.xml b/checkstyle/suppressions.xml
index 45ee4e60a7c..5bf69b6b65f 100644
--- a/checkstyle/suppressions.xml
+++ b/checkstyle/suppressions.xml
@@ -10,7 +10,7 @@
<!-- Clients -->
<suppress checks="ClassFanOutComplexity"
- files="(Fetcher|Sender|SenderTest|ConsumerCoordinator|KafkaConsumer|KafkaProducer|Utils|TransactionManagerTest|KafkaAdminClient|NetworkClient).java"/>
+ files="(Fetcher|Sender|SenderTest|ConsumerCoordinator|KafkaConsumer|KafkaProducer|Utils|TransactionManagerTest|KafkaAdminClient|NetworkClient|AdminClient).java"/>
<suppress checks="ClassFanOutComplexity"
files="(SaslServerAuthenticator|SaslAuthenticatorTest).java"/>
<suppress checks="ClassFanOutComplexity"
@@ -45,7 +45,7 @@
<suppress checks="ClassDataAbstractionCoupling"
files="(KafkaConsumer|ConsumerCoordinator|Fetcher|KafkaProducer|AbstractRequest|AbstractResponse|TransactionManager|KafkaAdminClient).java"/>
<suppress checks="ClassDataAbstractionCoupling"
- files="(Errors|SaslAuthenticatorTest|AgentTest).java"/>
+ files="(Errors|SaslAuthenticatorTest|AgentTest|CoordinatorTest).java"/>
<suppress checks="BooleanExpressionComplexity"
files="(Utils|Topic|KafkaLZ4BlockOutputStream|AclData).java"/>
@@ -57,7 +57,7 @@
files="AbstractRequest.java|KerberosLogin.java|WorkerSinkTaskTest.java|TransactionManagerTest.java"/>
<suppress checks="NPathComplexity"
- files="(BufferPool|MetricName|Node|ConfigDef|SslTransportLayer|MetadataResponse|KerberosLogin|Selector|Sender|Serdes|Agent|Values|PluginUtils).java"/>
+ files="(BufferPool|Fetcher|MetricName|Node|ConfigDef|RecordBatch|SslFactory|SslTransportLayer|MetadataResponse|KerberosLogin|Selector|Sender|Serdes|TokenInformation|Agent|Values|PluginUtils|MiniTrogdorCluster).java"/>
<!-- clients tests -->
<suppress checks="ClassDataAbstractionCoupling"
@@ -72,6 +72,9 @@
<suppress checks="JavaNCSS"
files="RequestResponseTest.java"/>
+ <suppress checks="NPathComplexity"
+ files="MemoryRecordsTest.java"/>
+
<!-- Connect -->
<suppress checks="ClassFanOutComplexity"
files="DistributedHerder(|Test).java"/>
@@ -80,7 +83,7 @@
files="(KafkaConfigBackingStore|RequestResponseTest|WorkerSinkTaskTest).java"/>
<suppress checks="ParameterNumber"
- files="WorkerSourceTask.java"/>
+ files="(WorkerSinkTask|WorkerSourceTask).java"/>
<suppress checks="ParameterNumber"
files="WorkerCoordinator.java"/>
<suppress checks="ParameterNumber"
@@ -111,17 +114,8 @@
files="Values.java"/>
<suppress checks="NPathComplexity"
- files="ConnectRecord.java"/>
- <suppress checks="NPathComplexity"
- files="ConnectSchema.java"/>
- <suppress checks="NPathComplexity"
- files="FileStreamSourceTask.java"/>
- <suppress checks="NPathComplexity"
- files="JsonConverter.java"/>
- <suppress checks="NPathComplexity"
- files="DistributedHerder.java"/>
- <suppress checks="NPathComplexity"
- files="ConnectHeaders.java"/>
+ files="(AbstractStatus|ConnectHeaders|ConnectRecord|ConnectSchema|DistributedHerder|FileStreamSourceTask|JsonConverter|KafkaConfigBackingStore).java"/>
+
<suppress checks="MethodLength"
files="Values.java"/>
@@ -155,11 +149,7 @@
files="StreamsPartitionAssignor.java"/>
<suppress checks="NPathComplexity"
- files="ProcessorStateManager.java"/>
- <suppress checks="NPathComplexity"
- files="StreamsPartitionAssignor.java"/>
- <suppress checks="NPathComplexity"
- files="StreamThread.java"/>
+ files="(ProcessorStateManager|InternalTopologyBuilder|StreamsPartitionAssignor|StreamThread).java"/>
<!-- Streams tests -->
<suppress checks="ClassFanOutComplexity"
@@ -189,7 +179,7 @@
files="SmokeTestDriver.java"/>
<suppress checks="NPathComplexity"
- files="KStreamKStreamJoinTest.java"/>
+ files="EosTestDriver|KStreamKStreamJoinTest.java|SmokeTestDriver.java"/>
<suppress checks="NPathComplexity"
files="KStreamKStreamLeftJoinTest.java"/>
@@ -212,6 +202,8 @@
files="SignalLogger.java"/>
<suppress checks="IllegalImport"
files="SignalLogger.java"/>
+ <suppress checks="ParameterNumber"
+ files="ProduceBenchSpec.java"/>
<!-- Log4J-Appender -->
<suppress checks="CyclomaticComplexity"
diff --git a/clients/src/main/java/org/apache/kafka/clients/ClusterConnectionStates.java b/clients/src/main/java/org/apache/kafka/clients/ClusterConnectionStates.java
index 89e958d7c46..c3a2856f4ad 100644
--- a/clients/src/main/java/org/apache/kafka/clients/ClusterConnectionStates.java
+++ b/clients/src/main/java/org/apache/kafka/clients/ClusterConnectionStates.java
@@ -128,6 +128,49 @@ public void disconnected(String id, long now) {
updateReconnectBackoff(nodeState);
}
+ /**
+ * Indicate that the connection is throttled until the specified deadline.
+ * @param id the connection to be throttled
+ * @param throttleUntilTimeMs the throttle deadline in milliseconds
+ */
+ public void throttle(String id, long throttleUntilTimeMs) {
+ NodeConnectionState state = nodeState.get(id);
+ // The throttle deadline should never regress.
+ if (state != null && state.throttleUntilTimeMs < throttleUntilTimeMs) {
+ state.throttleUntilTimeMs = throttleUntilTimeMs;
+ }
+ }
+
+ /**
+ * Return the remaining throttling delay in milliseconds if throttling is in progress. Return 0, otherwise.
+ * @param id the connection to check
+ * @param now the current time in ms
+ */
+ public long throttleDelayMs(String id, long now) {
+ NodeConnectionState state = nodeState.get(id);
+ if (state != null && state.throttleUntilTimeMs > now) {
+ return state.throttleUntilTimeMs - now;
+ } else {
+ return 0;
+ }
+ }
+
+ /**
+ * Return the number of milliseconds to wait, based on the connection state and the throttle time, before
+ * attempting to send data. If the connection has been established but being throttled, return throttle delay.
+ * Otherwise, return connection delay.
+ * @param id the connection to check
+ * @param now the current time in ms
+ */
+ public long pollDelayMs(String id, long now) {
+ long throttleDelayMs = throttleDelayMs(id, now);
+ if (isConnected(id) && throttleDelayMs > 0) {
+ return throttleDelayMs;
+ } else {
+ return connectionDelay(id, now);
+ }
+ }
+
/**
* Enter the checking_api_versions state for the given node.
* @param id the connection identifier
@@ -144,6 +187,7 @@ public void checkingApiVersions(String id) {
public void ready(String id) {
NodeConnectionState nodeState = nodeState(id);
nodeState.state = ConnectionState.READY;
+ nodeState.authenticationException = null;
resetReconnectBackoff(nodeState);
}
@@ -162,26 +206,43 @@ public void authenticationFailed(String id, long now, AuthenticationException ex
}
/**
- * Return true if the connection is ready.
+ * Return true if the connection is in the READY state and currently not throttled.
+ *
* @param id the connection identifier
+ * @param now the current time
*/
- public boolean isReady(String id) {
- NodeConnectionState state = nodeState.get(id);
- return state != null && state.state == ConnectionState.READY;
+ public boolean isReady(String id, long now) {
+ return isReady(nodeState.get(id), now);
+ }
+
+ private boolean isReady(NodeConnectionState state, long now) {
+ return state != null && state.state == ConnectionState.READY && state.throttleUntilTimeMs <= now;
}
/**
- * Return true if there is at least one node with connection in ready state and false otherwise.
+ * Return true if there is at least one node with connection in the READY state and not throttled. Returns false
+ * otherwise.
+ *
+ * @param now the current time
*/
- public boolean hasReadyNodes() {
+ public boolean hasReadyNodes(long now) {
for (Map.Entry<String, NodeConnectionState> entry : nodeState.entrySet()) {
- NodeConnectionState state = entry.getValue();
- if (state != null && state.state == ConnectionState.READY)
+ if (isReady(entry.getValue(), now)) {
return true;
+ }
}
return false;
}
+ /**
+ * Return true if the connection has been established
+ * @param id The id of the node to check
+ */
+ public boolean isConnected(String id) {
+ NodeConnectionState state = nodeState.get(id);
+ return state != null && state.state.isConnected();
+ }
+
/**
* Return true if the connection has been disconnected
* @param id The id of the node to check
@@ -271,6 +332,8 @@ private NodeConnectionState nodeState(String id) {
long lastConnectAttemptMs;
long failedAttempts;
long reconnectBackoffMs;
+ // Connection is being throttled if current time < throttleUntilTimeMs.
+ long throttleUntilTimeMs;
public NodeConnectionState(ConnectionState state, long lastConnectAttempt, long reconnectBackoffMs) {
this.state = state;
@@ -278,10 +341,11 @@ public NodeConnectionState(ConnectionState state, long lastConnectAttempt, long
this.lastConnectAttemptMs = lastConnectAttempt;
this.failedAttempts = 0;
this.reconnectBackoffMs = reconnectBackoffMs;
+ this.throttleUntilTimeMs = 0;
}
public String toString() {
- return "NodeState(" + state + ", " + lastConnectAttemptMs + ", " + failedAttempts + ")";
+ return "NodeState(" + state + ", " + lastConnectAttemptMs + ", " + failedAttempts + ", " + throttleUntilTimeMs + ")";
}
}
}
diff --git a/clients/src/main/java/org/apache/kafka/clients/ConnectionState.java b/clients/src/main/java/org/apache/kafka/clients/ConnectionState.java
index 28b43d67fb1..f92c7fa037b 100644
--- a/clients/src/main/java/org/apache/kafka/clients/ConnectionState.java
+++ b/clients/src/main/java/org/apache/kafka/clients/ConnectionState.java
@@ -31,4 +31,8 @@
public boolean isDisconnected() {
return this == AUTHENTICATION_FAILED || this == DISCONNECTED;
}
+
+ public boolean isConnected() {
+ return this == CHECKING_API_VERSIONS || this == READY;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java b/clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java
index a062818e3b2..5caee2d4c87 100644
--- a/clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java
+++ b/clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java
@@ -20,12 +20,12 @@
import java.util.Collections;
import java.util.Deque;
import java.util.HashMap;
+import java.util.Iterator;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;
import java.util.concurrent.atomic.AtomicInteger;
-
/**
* The set of requests which have been sent or are being sent but haven't yet received a response
*/
@@ -151,9 +151,14 @@ public boolean isEmpty() {
if (reqs == null) {
return Collections.emptyList();
} else {
- Deque<NetworkClient.InFlightRequest> clearedRequests = requests.remove(node);
+ final Deque<NetworkClient.InFlightRequest> clearedRequests = requests.remove(node);
inFlightRequestCount.getAndAdd(-clearedRequests.size());
- return clearedRequests;
+ return new Iterable<NetworkClient.InFlightRequest>() {
+ @Override
+ public Iterator<NetworkClient.InFlightRequest> iterator() {
+ return clearedRequests.descendingIterator();
+ }
+ };
}
}
diff --git a/clients/src/main/java/org/apache/kafka/clients/KafkaClient.java b/clients/src/main/java/org/apache/kafka/clients/KafkaClient.java
index 0a9b51981e7..49bf3a3eab9 100644
--- a/clients/src/main/java/org/apache/kafka/clients/KafkaClient.java
+++ b/clients/src/main/java/org/apache/kafka/clients/KafkaClient.java
@@ -48,7 +48,7 @@
boolean ready(Node node, long now);
/**
- * Returns the number of milliseconds to wait, based on the connection state, before attempting to send data. When
+ * Return the number of milliseconds to wait, based on the connection state, before attempting to send data. When
* disconnected, this respects the reconnect backoff time. When connecting or connected, this handles slow/stalled
* connections.
*
@@ -58,6 +58,16 @@
*/
long connectionDelay(Node node, long now);
+ /**
+ * Return the number of milliseconds to wait, based on the connection state and the throttle time, before
+ * attempting to send data. If the connection has been established but being throttled, return throttle delay.
+ * Otherwise, return connection delay.
+ *
+ * @param node the connection to check
+ * @param now the current time in ms
+ */
+ long pollDelayMs(Node node, long now);
+
/**
* Check if the connection of the node has failed, based on the connection state. Such connection failure are
* usually transient and can be resumed in the next {@link #ready(org.apache.kafka.common.Node, long)} }
@@ -145,9 +155,12 @@
boolean hasInFlightRequests(String nodeId);
/**
- * Return true if there is at least one node with connection in ready state and false otherwise.
+ * Return true if there is at least one node with connection in the READY state and not throttled. Returns false
+ * otherwise.
+ *
+ * @param now the current time
*/
- boolean hasReadyNodes();
+ boolean hasReadyNodes(long now);
/**
* Wake up the client if it is currently blocked waiting for I/O
diff --git a/clients/src/main/java/org/apache/kafka/clients/MetadataUpdater.java b/clients/src/main/java/org/apache/kafka/clients/MetadataUpdater.java
index 126728342d4..09ed995d14c 100644
--- a/clients/src/main/java/org/apache/kafka/clients/MetadataUpdater.java
+++ b/clients/src/main/java/org/apache/kafka/clients/MetadataUpdater.java
@@ -29,7 +29,7 @@
* <p>
* This class is not thread-safe!
*/
-interface MetadataUpdater {
+public interface MetadataUpdater {
/**
* Gets the current cluster info without blocking.
diff --git a/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java b/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java
index 4b291b5eb67..26fa467f81a 100644
--- a/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java
+++ b/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java
@@ -300,6 +300,22 @@ public long connectionDelay(Node node, long now) {
return connectionStates.connectionDelay(node.idString(), now);
}
+ // Return the remaining throttling delay in milliseconds if throttling is in progress. Return 0, otherwise.
+ // This is for testing.
+ public long throttleDelayMs(Node node, long now) {
+ return connectionStates.throttleDelayMs(node.idString(), now);
+ }
+
+ /**
+ * Return the poll delay in milliseconds based on both connection and throttle delay.
+ * @param node the connection to check
+ * @param now the current time in ms
+ */
+ @Override
+ public long pollDelayMs(Node node, long now) {
+ return connectionStates.pollDelayMs(node.idString(), now);
+ }
+
/**
* Check if the connection of the node has failed, based on the connection state. Such connection failure are
* usually transient and can be resumed in the next {@link #ready(org.apache.kafka.common.Node, long)} }
@@ -336,16 +352,18 @@ public AuthenticationException authenticationException(Node node) {
public boolean isReady(Node node, long now) {
// if we need to update our metadata now declare all requests unready to make metadata requests first
// priority
- return !metadataUpdater.isUpdateDue(now) && canSendRequest(node.idString());
+ return !metadataUpdater.isUpdateDue(now) && canSendRequest(node.idString(), now);
}
/**
* Are we connected and ready and able to send more requests to the given connection?
*
* @param node The node
+ * @param now the current timestamp
*/
- private boolean canSendRequest(String node) {
- return connectionStates.isReady(node) && selector.isChannelReady(node) && inFlightRequests.canSendMore(node);
+ private boolean canSendRequest(String node, long now) {
+ return connectionStates.isReady(node, now) && selector.isChannelReady(node) &&
+ inFlightRequests.canSendMore(node);
}
/**
@@ -373,7 +391,7 @@ private void doSend(ClientRequest clientRequest, boolean isInternalRequest, long
// will be slightly different for some internal requests (for
// example, ApiVersionsRequests can be sent prior to being in
// READY state.)
- if (!canSendRequest(nodeId))
+ if (!canSendRequest(nodeId, now))
throw new IllegalStateException("Attempt to send a request to node " + nodeId + " which is not ready.");
}
AbstractRequest.Builder<?> builder = clientRequest.requestBuilder();
@@ -513,8 +531,8 @@ public boolean hasInFlightRequests(String node) {
}
@Override
- public boolean hasReadyNodes() {
- return connectionStates.hasReadyNodes();
+ public boolean hasReadyNodes(long now) {
+ return connectionStates.hasReadyNodes(now);
}
/**
@@ -603,11 +621,13 @@ private void processDisconnection(List<ClientResponse> responses, String nodeId,
nodesNeedingApiVersionsFetch.remove(nodeId);
switch (disconnectState.state()) {
case AUTHENTICATION_FAILED:
- connectionStates.authenticationFailed(nodeId, now, disconnectState.exception());
- log.error("Connection to node {} failed authentication due to: {}", nodeId, disconnectState.exception().getMessage());
+ AuthenticationException exception = disconnectState.exception();
+ connectionStates.authenticationFailed(nodeId, now, exception);
+ metadataUpdater.handleAuthenticationFailure(exception);
+ log.error("Connection to node {} failed authentication due to: {}", nodeId, exception.getMessage());
break;
case AUTHENTICATE:
- // This warning applies to older brokers which dont provide feedback on authentication failures
+ // This warning applies to older brokers which don't provide feedback on authentication failures
log.warn("Connection to node {} terminated during authentication. This may indicate " +
"that authentication failed due to invalid credentials.", nodeId);
break;
@@ -625,9 +645,6 @@ private void processDisconnection(List<ClientResponse> responses, String nodeId,
else if (request.header.apiKey() == ApiKeys.METADATA)
metadataUpdater.handleDisconnection(request.destination);
}
- AuthenticationException authenticationException = connectionStates.authenticationException(nodeId);
- if (authenticationException != null)
- metadataUpdater.handleAuthenticationFailure(authenticationException);
}
/**
@@ -673,6 +690,24 @@ private void handleCompletedSends(List<ClientResponse> responses, long now) {
}
}
+ /**
+ * If a response from a node includes a non-zero throttle delay and client-side throttling has been enabled for
+ * the connection to the node, throttle the connection for the specified delay.
+ *
+ * @param response the response
+ * @param apiVersion the API version of the response
+ * @param nodeId the id of the node
+ * @param now The current time
+ */
+ private void maybeThrottle(AbstractResponse response, short apiVersion, String nodeId, long now) {
+ int throttleTimeMs = response.throttleTimeMs();
+ if (throttleTimeMs > 0 && response.shouldClientThrottle(apiVersion)) {
+ connectionStates.throttle(nodeId, now + throttleTimeMs);
+ log.trace("Connection to node {} is throttled for {} ms until timestamp {}", nodeId, throttleTimeMs,
+ now + throttleTimeMs);
+ }
+ }
+
/**
* Handle any completed receives and update the response list with the responses received.
*
@@ -689,7 +724,9 @@ private void handleCompletedReceives(List<ClientResponse> responses, long now) {
log.trace("Completed receive from node {} for {} with correlation id {}, received {}", req.destination,
req.header.apiKey(), req.header.correlationId(), responseStruct);
}
+ // If the received response includes a throttle delay, throttle the connection.
AbstractResponse body = AbstractResponse.parseResponse(req.header.apiKey(), responseStruct);
+ maybeThrottle(body, req.header.apiVersion(), req.destination, now);
if (req.isInternalRequest && body instanceof MetadataResponse)
metadataUpdater.handleCompletedMetadataResponse(req.header, now, (MetadataResponse) body);
else if (req.isInternalRequest && body instanceof ApiVersionsResponse)
@@ -716,9 +753,7 @@ private void handleApiVersionsResponse(List<ClientResponse> responses,
NodeApiVersions nodeVersionInfo = new NodeApiVersions(apiVersionsResponse.apiVersions());
apiVersions.update(node, nodeVersionInfo);
this.connectionStates.ready(node);
- if (log.isDebugEnabled()) {
- log.debug("Recorded API versions for node {}: {}", node, nodeVersionInfo);
- }
+ log.debug("Recorded API versions for node {}: {}", node, nodeVersionInfo);
}
/**
@@ -833,6 +868,7 @@ public long maybeUpdate(long now) {
long waitForMetadataFetch = this.metadataFetchInProgress ? requestTimeoutMs : 0;
long metadataTimeout = Math.max(timeToNextMetadataUpdate, waitForMetadataFetch);
+
if (metadataTimeout > 0) {
return metadataTimeout;
}
@@ -914,7 +950,7 @@ private boolean isAnyNodeConnecting() {
private long maybeUpdate(long now, Node node) {
String nodeConnectionId = node.idString();
- if (canSendRequest(nodeConnectionId)) {
+ if (canSendRequest(nodeConnectionId, now)) {
this.metadataFetchInProgress = true;
MetadataRequest.Builder metadataRequest;
if (metadata.needMetadataForAllTopics())
diff --git a/clients/src/main/java/org/apache/kafka/clients/consumer/internals/StaleMetadataException.java b/clients/src/main/java/org/apache/kafka/clients/StaleMetadataException.java
similarity index 74%
rename from clients/src/main/java/org/apache/kafka/clients/consumer/internals/StaleMetadataException.java
rename to clients/src/main/java/org/apache/kafka/clients/StaleMetadataException.java
index 53110d38387..dafc2d5483c 100644
--- a/clients/src/main/java/org/apache/kafka/clients/consumer/internals/StaleMetadataException.java
+++ b/clients/src/main/java/org/apache/kafka/clients/StaleMetadataException.java
@@ -14,13 +14,22 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.kafka.clients.consumer.internals;
+package org.apache.kafka.clients;
import org.apache.kafka.common.errors.InvalidMetadataException;
/**
- * Thrown when metadata is old and needs to be refreshed.
+ * Thrown when current metadata cannot be used. This is often used as a way to trigger a metadata
+ * update before retrying another operation.
+ *
+ * Note: this is not a public API.
*/
public class StaleMetadataException extends InvalidMetadataException {
private static final long serialVersionUID = 1L;
+
+ public StaleMetadataException() {}
+
+ public StaleMetadataException(String message) {
+ super(message);
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/clients/admin/AdminClient.java b/clients/src/main/java/org/apache/kafka/clients/admin/AdminClient.java
index 897e127d557..0171b617e7c 100644
--- a/clients/src/main/java/org/apache/kafka/clients/admin/AdminClient.java
+++ b/clients/src/main/java/org/apache/kafka/clients/admin/AdminClient.java
@@ -535,4 +535,237 @@ public DeleteRecordsResult deleteRecords(Map<TopicPartition, RecordsToDelete> re
*/
public abstract DeleteRecordsResult deleteRecords(Map<TopicPartition, RecordsToDelete> recordsToDelete,
DeleteRecordsOptions options);
+
+ /**
+ * <p>Create a Delegation Token.</p>
+ *
+ * <p>This is a convenience method for {@link #createDelegationToken(CreateDelegationTokenOptions)} with default options.
+ * See the overload for more details.</p>
+ *
+ * @return The CreateDelegationTokenResult.
+ */
+ public CreateDelegationTokenResult createDelegationToken() {
+ return createDelegationToken(new CreateDelegationTokenOptions());
+ }
+
+
+ /**
+ * <p>Create a Delegation Token.</p>
+ *
+ * <p>This operation is supported by brokers with version 1.1.0 or higher.</p>
+ *
+ * <p>The following exceptions can be anticipated when calling {@code get()} on the futures obtained from the
+ * {@link CreateDelegationTokenResult#delegationToken() delegationToken()} method of the returned {@code CreateDelegationTokenResult}</p>
+ * <ul>
+ * <li>{@link org.apache.kafka.common.errors.UnsupportedByAuthenticationException}
+ * If the request sent on PLAINTEXT/1-way SSL channels or delegation token authenticated channels.</li>
+ * <li>{@link org.apache.kafka.common.errors.InvalidPrincipalTypeException}
+ * if the renewers principal type is not supported.</li>
+ * <li>{@link org.apache.kafka.common.errors.DelegationTokenDisabledException}
+ * if the delegation token feature is disabled.</li>
+ * <li>{@link org.apache.kafka.common.errors.TimeoutException}
+ * if the request was not completed in within the given {@link CreateDelegationTokenOptions#timeoutMs()}.</li>
+ * </ul>
+ *
+ * @param options The options to use when creating delegation token.
+ * @return The DeleteRecordsResult.
+ */
+ public abstract CreateDelegationTokenResult createDelegationToken(CreateDelegationTokenOptions options);
+
+
+ /**
+ * <p>Renew a Delegation Token.</p>
+ *
+ * <p>This is a convenience method for {@link #renewDelegationToken(byte[], RenewDelegationTokenOptions)} with default options.
+ * See the overload for more details.</p>
+ *
+ *
+ * @param hmac HMAC of the Delegation token
+ * @return The RenewDelegationTokenResult.
+ */
+ public RenewDelegationTokenResult renewDelegationToken(byte[] hmac) {
+ return renewDelegationToken(hmac, new RenewDelegationTokenOptions());
+ }
+
+ /**
+ * <p> Renew a Delegation Token.</p>
+ *
+ * <p>This operation is supported by brokers with version 1.1.0 or higher.</p>
+ *
+ * <p>The following exceptions can be anticipated when calling {@code get()} on the futures obtained from the
+ * {@link RenewDelegationTokenResult#expiryTimestamp() expiryTimestamp()} method of the returned {@code RenewDelegationTokenResult}</p>
+ * <ul>
+ * <li>{@link org.apache.kafka.common.errors.UnsupportedByAuthenticationException}
+ * If the request sent on PLAINTEXT/1-way SSL channels or delegation token authenticated channels.</li>
+ * <li>{@link org.apache.kafka.common.errors.DelegationTokenDisabledException}
+ * if the delegation token feature is disabled.</li>
+ * <li>{@link org.apache.kafka.common.errors.DelegationTokenNotFoundException}
+ * if the delegation token is not found on server.</li>
+ * <li>{@link org.apache.kafka.common.errors.DelegationTokenOwnerMismatchException}
+ * if the authenticated user is not owner/renewer of the token.</li>
+ * <li>{@link org.apache.kafka.common.errors.DelegationTokenExpiredException}
+ * if the delegation token is expired.</li>
+ * <li>{@link org.apache.kafka.common.errors.TimeoutException}
+ * if the request was not completed in within the given {@link RenewDelegationTokenOptions#timeoutMs()}.</li>
+ * </ul>
+ *
+ * @param hmac HMAC of the Delegation token
+ * @param options The options to use when renewing delegation token.
+ * @return The RenewDelegationTokenResult.
+ */
+ public abstract RenewDelegationTokenResult renewDelegationToken(byte[] hmac, RenewDelegationTokenOptions options);
+
+ /**
+ * <p>Expire a Delegation Token.</p>
+ *
+ * <p>This is a convenience method for {@link #expireDelegationToken(byte[], ExpireDelegationTokenOptions)} with default options.
+ * This will expire the token immediately. See the overload for more details.</p>
+ *
+ * @param hmac HMAC of the Delegation token
+ * @return The ExpireDelegationTokenResult.
+ */
+ public ExpireDelegationTokenResult expireDelegationToken(byte[] hmac) {
+ return expireDelegationToken(hmac, new ExpireDelegationTokenOptions());
+ }
+
+ /**
+ * <p>Expire a Delegation Token.</p>
+ *
+ * <p>This operation is supported by brokers with version 1.1.0 or higher.</p>
+ *
+ * <p>The following exceptions can be anticipated when calling {@code get()} on the futures obtained from the
+ * {@link ExpireDelegationTokenResult#expiryTimestamp() expiryTimestamp()} method of the returned {@code ExpireDelegationTokenResult}</p>
+ * <ul>
+ * <li>{@link org.apache.kafka.common.errors.UnsupportedByAuthenticationException}
+ * If the request sent on PLAINTEXT/1-way SSL channels or delegation token authenticated channels.</li>
+ * <li>{@link org.apache.kafka.common.errors.DelegationTokenDisabledException}
+ * if the delegation token feature is disabled.</li>
+ * <li>{@link org.apache.kafka.common.errors.DelegationTokenNotFoundException}
+ * if the delegation token is not found on server.</li>
+ * <li>{@link org.apache.kafka.common.errors.DelegationTokenOwnerMismatchException}
+ * if the authenticated user is not owner/renewer of the requested token.</li>
+ * <li>{@link org.apache.kafka.common.errors.DelegationTokenExpiredException}
+ * if the delegation token is expired.</li>
+ * <li>{@link org.apache.kafka.common.errors.TimeoutException}
+ * if the request was not completed in within the given {@link ExpireDelegationTokenOptions#timeoutMs()}.</li>
+ * </ul>
+ *
+ * @param hmac HMAC of the Delegation token
+ * @param options The options to use when expiring delegation token.
+ * @return The ExpireDelegationTokenResult.
+ */
+ public abstract ExpireDelegationTokenResult expireDelegationToken(byte[] hmac, ExpireDelegationTokenOptions options);
+
+ /**
+ *<p>Describe the Delegation Tokens.</p>
+ *
+ * <p>This is a convenience method for {@link #describeDelegationToken(DescribeDelegationTokenOptions)} with default options.
+ * This will return all the user owned tokens and tokens where user have Describe permission. See the overload for more details.</p>
+ *
+ * @return The DescribeDelegationTokenResult.
+ */
+ public DescribeDelegationTokenResult describeDelegationToken() {
+ return describeDelegationToken(new DescribeDelegationTokenOptions());
+ }
+
+ /**
+ * <p>Describe the Delegation Tokens.</p>
+ *
+ * <p>This operation is supported by brokers with version 1.1.0 or higher.</p>
+ *
+ * <p>The following exceptions can be anticipated when calling {@code get()} on the futures obtained from the
+ * {@link DescribeDelegationTokenResult#delegationTokens() delegationTokens()} method of the returned {@code DescribeDelegationTokenResult}</p>
+ * <ul>
+ * <li>{@link org.apache.kafka.common.errors.UnsupportedByAuthenticationException}
+ * If the request sent on PLAINTEXT/1-way SSL channels or delegation token authenticated channels.</li>
+ * <li>{@link org.apache.kafka.common.errors.DelegationTokenDisabledException}
+ * if the delegation token feature is disabled.</li>
+ * <li>{@link org.apache.kafka.common.errors.TimeoutException}
+ * if the request was not completed in within the given {@link DescribeDelegationTokenOptions#timeoutMs()}.</li>
+ * </ul>
+ *
+ * @param options The options to use when describing delegation tokens.
+ * @return The DescribeDelegationTokenResult.
+ */
+ public abstract DescribeDelegationTokenResult describeDelegationToken(DescribeDelegationTokenOptions options);
+
+ /**
+ * Describe some group IDs in the cluster.
+ *
+ * @param groupIds The IDs of the groups to describe.
+ * @param options The options to use when describing the groups.
+ * @return The DescribeConsumerGroupResult.
+ */
+ public abstract DescribeConsumerGroupsResult describeConsumerGroups(Collection<String> groupIds,
+ DescribeConsumerGroupsOptions options);
+
+ /**
+ * Describe some group IDs in the cluster, with the default options.
+ * <p>
+ * This is a convenience method for
+ * #{@link AdminClient#describeConsumerGroups(Collection, DescribeConsumerGroupsOptions)} with
+ * default options. See the overload for more details.
+ *
+ * @param groupIds The IDs of the groups to describe.
+ * @return The DescribeConsumerGroupResult.
+ */
+ public DescribeConsumerGroupsResult describeConsumerGroups(Collection<String> groupIds) {
+ return describeConsumerGroups(groupIds, new DescribeConsumerGroupsOptions());
+ }
+
+ /**
+ * List the consumer groups available in the cluster.
+ *
+ * @param options The options to use when listing the consumer groups.
+ * @return The ListGroupsResult.
+ */
+ public abstract ListConsumerGroupsResult listConsumerGroups(ListConsumerGroupsOptions options);
+
+ /**
+ * List the consumer groups available in the cluster with the default options.
+ *
+ * This is a convenience method for #{@link AdminClient#listConsumerGroups(ListConsumerGroupsOptions)} with default options.
+ * See the overload for more details.
+ *
+ * @return The ListGroupsResult.
+ */
+ public ListConsumerGroupsResult listConsumerGroups() {
+ return listConsumerGroups(new ListConsumerGroupsOptions());
+ }
+
+ /**
+ * List the consumer group offsets available in the cluster.
+ *
+ * @param options The options to use when listing the consumer group offsets.
+ * @return The ListGroupOffsetsResult
+ */
+ public abstract ListConsumerGroupOffsetsResult listConsumerGroupOffsets(String groupId, ListConsumerGroupOffsetsOptions options);
+
+ /**
+ * List the consumer group offsets available in the cluster with the default options.
+ *
+ * This is a convenience method for #{@link AdminClient#listConsumerGroupOffsets(String, ListConsumerGroupOffsetsOptions)} with default options.
+ *
+ * @return The ListGroupOffsetsResult.
+ */
+ public ListConsumerGroupOffsetsResult listConsumerGroupOffsets(String groupId) {
+ return listConsumerGroupOffsets(groupId, new ListConsumerGroupOffsetsOptions());
+ }
+
+ /**
+ * Delete consumer groups from the cluster.
+ *
+ * @param options The options to use when deleting a consumer group.
+ * @return The DeletConsumerGroupResult.
+ */
+ public abstract DeleteConsumerGroupsResult deleteConsumerGroups(Collection<String> groupIds, DeleteConsumerGroupsOptions options);
+
+ /**
+ * Delete consumer groups from the cluster with the default options.
+ *
+ * @return The DeleteConsumerGroupResult.
+ */
+ public DeleteConsumerGroupsResult deleteConsumerGroups(Collection<String> groupIds) {
+ return deleteConsumerGroups(groupIds, new DeleteConsumerGroupsOptions());
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/clients/admin/Config.java b/clients/src/main/java/org/apache/kafka/clients/admin/Config.java
index 56f9c270792..c81c0b60274 100644
--- a/clients/src/main/java/org/apache/kafka/clients/admin/Config.java
+++ b/clients/src/main/java/org/apache/kafka/clients/admin/Config.java
@@ -21,39 +21,40 @@
import java.util.Collection;
import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
/**
* A configuration object containing the configuration entries for a resource.
- *
+ * <p>
* The API of this class is evolving, see {@link AdminClient} for details.
*/
@InterfaceStability.Evolving
public class Config {
- private final Collection<ConfigEntry> entries;
+ private final Map<String, ConfigEntry> entries = new HashMap<>();
/**
* Create a configuration instance with the provided entries.
*/
public Config(Collection<ConfigEntry> entries) {
- this.entries = Collections.unmodifiableCollection(entries);
+ for (ConfigEntry entry : entries) {
+ this.entries.put(entry.name(), entry);
+ }
}
/**
* Configuration entries for a resource.
*/
public Collection<ConfigEntry> entries() {
- return entries;
+ return Collections.unmodifiableCollection(entries.values());
}
/**
* Get the configuration entry with the provided name or null if there isn't one.
*/
public ConfigEntry get(String name) {
- for (ConfigEntry entry : entries)
- if (entry.name().equals(name))
- return entry;
- return null;
+ return entries.get(name);
}
@Override
@@ -75,6 +76,6 @@ public int hashCode() {
@Override
public String toString() {
- return "Config(entries=" + entries + ")";
+ return "Config(entries=" + entries.values() + ")";
}
}
diff --git a/clients/src/main/java/org/apache/kafka/clients/admin/ConsumerGroupDescription.java b/clients/src/main/java/org/apache/kafka/clients/admin/ConsumerGroupDescription.java
new file mode 100644
index 00000000000..933d66ca4b7
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/clients/admin/ConsumerGroupDescription.java
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.clients.admin;
+
+import org.apache.kafka.common.ConsumerGroupState;
+import org.apache.kafka.common.Node;
+import org.apache.kafka.common.utils.Utils;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Objects;
+
+/**
+ * A detailed description of a single consumer group in the cluster.
+ */
+public class ConsumerGroupDescription {
+ private final String groupId;
+ private final boolean isSimpleConsumerGroup;
+ private final Collection<MemberDescription> members;
+ private final String partitionAssignor;
+ private final ConsumerGroupState state;
+ private final Node coordinator;
+
+ public ConsumerGroupDescription(String groupId,
+ boolean isSimpleConsumerGroup,
+ Collection<MemberDescription> members,
+ String partitionAssignor,
+ ConsumerGroupState state,
+ Node coordinator) {
+ this.groupId = groupId == null ? "" : groupId;
+ this.isSimpleConsumerGroup = isSimpleConsumerGroup;
+ this.members = members == null ? Collections.<MemberDescription>emptyList() :
+ Collections.unmodifiableList(new ArrayList<>(members));
+ this.partitionAssignor = partitionAssignor == null ? "" : partitionAssignor;
+ this.state = state;
+ this.coordinator = coordinator;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
+ if (o == null || getClass() != o.getClass()) return false;
+ ConsumerGroupDescription that = (ConsumerGroupDescription) o;
+ return isSimpleConsumerGroup == that.isSimpleConsumerGroup &&
+ groupId.equals(that.groupId) &&
+ members.equals(that.members) &&
+ partitionAssignor.equals(that.partitionAssignor) &&
+ state.equals(that.state);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(isSimpleConsumerGroup, groupId, members, partitionAssignor, state);
+ }
+
+ /**
+ * The id of the consumer group.
+ */
+ public String groupId() {
+ return groupId;
+ }
+
+ /**
+ * If consumer group is simple or not.
+ */
+ public boolean isSimpleConsumerGroup() {
+ return isSimpleConsumerGroup;
+ }
+
+ /**
+ * A list of the members of the consumer group.
+ */
+ public Collection<MemberDescription> members() {
+ return members;
+ }
+
+ /**
+ * The consumer group partition assignor.
+ */
+ public String partitionAssignor() {
+ return partitionAssignor;
+ }
+
+ /**
+ * The consumer group state, or UNKNOWN if the state is too new for us to parse.
+ */
+ public ConsumerGroupState state() {
+ return state;
+ }
+
+ /**
+ * The consumer group coordinator, or null if the coordinator is not known.
+ */
+ public Node coordinator() {
+ return coordinator;
+ }
+
+ @Override
+ public String toString() {
+ return "(groupId=" + groupId +
+ ", isSimpleConsumerGroup=" + isSimpleConsumerGroup +
+ ", members=" + Utils.join(members, ",") +
+ ", partitionAssignor=" + partitionAssignor +
+ ", state=" + state +
+ ", coordinator=" + coordinator +
+ ")";
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/clients/admin/ConsumerGroupListing.java b/clients/src/main/java/org/apache/kafka/clients/admin/ConsumerGroupListing.java
new file mode 100644
index 00000000000..46da9628010
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/clients/admin/ConsumerGroupListing.java
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.clients.admin;
+
+/**
+ * A listing of a consumer group in the cluster.
+ */
+public class ConsumerGroupListing {
+ private final String groupId;
+ private final boolean isSimpleConsumerGroup;
+
+ /**
+ * Create an instance with the specified parameters.
+ *
+ * @param groupId Group Id
+ * @param isSimpleConsumerGroup If consumer group is simple or not.
+ */
+ public ConsumerGroupListing(String groupId, boolean isSimpleConsumerGroup) {
+ this.groupId = groupId;
+ this.isSimpleConsumerGroup = isSimpleConsumerGroup;
+ }
+
+ /**
+ * Consumer Group Id
+ */
+ public String groupId() {
+ return groupId;
+ }
+
+ /**
+ * If Consumer Group is simple or not.
+ */
+ public boolean isSimpleConsumerGroup() {
+ return isSimpleConsumerGroup;
+ }
+
+ @Override
+ public String toString() {
+ return "(" +
+ "groupId='" + groupId + '\'' +
+ ", isSimpleConsumerGroup=" + isSimpleConsumerGroup +
+ ')';
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/clients/admin/CreateDelegationTokenOptions.java b/clients/src/main/java/org/apache/kafka/clients/admin/CreateDelegationTokenOptions.java
new file mode 100644
index 00000000000..1b77b943800
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/clients/admin/CreateDelegationTokenOptions.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.clients.admin;
+
+import java.util.LinkedList;
+import java.util.List;
+
+import org.apache.kafka.common.annotation.InterfaceStability;
+import org.apache.kafka.common.security.auth.KafkaPrincipal;
+
+/**
+ * Options for {@link AdminClient#createDelegationToken(CreateDelegationTokenOptions)}.
+ *
+ * The API of this class is evolving, see {@link AdminClient} for details.
+ */
+@InterfaceStability.Evolving
+public class CreateDelegationTokenOptions extends AbstractOptions<CreateDelegationTokenOptions> {
+ private long maxLifeTimeMs = -1;
+ private List<KafkaPrincipal> renewers = new LinkedList<>();
+
+ public CreateDelegationTokenOptions renewers(List<KafkaPrincipal> renewers) {
+ this.renewers = renewers;
+ return this;
+ }
+
+ public List<KafkaPrincipal> renewers() {
+ return renewers;
+ }
+
+ public CreateDelegationTokenOptions maxlifeTimeMs(long maxLifeTimeMs) {
+ this.maxLifeTimeMs = maxLifeTimeMs;
+ return this;
+ }
+
+ public long maxlifeTimeMs() {
+ return maxLifeTimeMs;
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/clients/admin/CreateDelegationTokenResult.java b/clients/src/main/java/org/apache/kafka/clients/admin/CreateDelegationTokenResult.java
new file mode 100644
index 00000000000..043cbe87fef
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/clients/admin/CreateDelegationTokenResult.java
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.clients.admin;
+
+import org.apache.kafka.common.KafkaFuture;
+import org.apache.kafka.common.annotation.InterfaceStability;
+import org.apache.kafka.common.security.token.delegation.DelegationToken;
+
+/**
+ * The result of the {@link KafkaAdminClient#createDelegationToken(CreateDelegationTokenOptions)} call.
+ *
+ * The API of this class is evolving, see {@link AdminClient} for details.
+ */
+@InterfaceStability.Evolving
+public class CreateDelegationTokenResult {
+ private final KafkaFuture<DelegationToken> delegationToken;
+
+ CreateDelegationTokenResult(KafkaFuture<DelegationToken> delegationToken) {
+ this.delegationToken = delegationToken;
+ }
+
+ /**
+ * Returns a future which yields a delegation token
+ */
+ public KafkaFuture<DelegationToken> delegationToken() {
+ return delegationToken;
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/clients/admin/DeleteConsumerGroupsOptions.java b/clients/src/main/java/org/apache/kafka/clients/admin/DeleteConsumerGroupsOptions.java
new file mode 100644
index 00000000000..cd505f4c111
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/clients/admin/DeleteConsumerGroupsOptions.java
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.admin;
+
+import org.apache.kafka.common.annotation.InterfaceStability;
+
+import java.util.Collection;
+
+/**
+ * Options for the {@link AdminClient#deleteConsumerGroups(Collection)} call.
+ *
+ * The API of this class is evolving, see {@link AdminClient} for details.
+ */
+@InterfaceStability.Evolving
+public class DeleteConsumerGroupsOptions extends AbstractOptions<DeleteConsumerGroupsOptions> {
+
+}
diff --git a/clients/src/main/java/org/apache/kafka/clients/admin/DeleteConsumerGroupsResult.java b/clients/src/main/java/org/apache/kafka/clients/admin/DeleteConsumerGroupsResult.java
new file mode 100644
index 00000000000..dd6835cf10c
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/clients/admin/DeleteConsumerGroupsResult.java
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.admin;
+
+import org.apache.kafka.common.KafkaFuture;
+import org.apache.kafka.common.annotation.InterfaceStability;
+
+import java.util.Collection;
+import java.util.Map;
+
+/**
+ * The result of the {@link AdminClient#deleteConsumerGroups(Collection)} call.
+ *
+ * The API of this class is evolving, see {@link AdminClient} for details.
+ */
+@InterfaceStability.Evolving
+public class DeleteConsumerGroupsResult {
+ private final Map<String, KafkaFuture<Void>> futures;
+
+ DeleteConsumerGroupsResult(final Map<String, KafkaFuture<Void>> futures) {
+ this.futures = futures;
+ }
+
+ /**
+ * Return a map from group id to futures which can be used to check the status of
+ * individual deletions.
+ */
+ public Map<String, KafkaFuture<Void>> deletedGroups() {
+ return futures;
+ }
+
+ /**
+ * Return a future which succeeds only if all the consumer group deletions succeed.
+ */
+ public KafkaFuture<Void> all() {
+ return KafkaFuture.allOf(futures.values().toArray(new KafkaFuture[0]));
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/clients/admin/DescribeConsumerGroupsOptions.java b/clients/src/main/java/org/apache/kafka/clients/admin/DescribeConsumerGroupsOptions.java
new file mode 100644
index 00000000000..7daff1a483b
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/clients/admin/DescribeConsumerGroupsOptions.java
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.clients.admin;
+
+import org.apache.kafka.common.annotation.InterfaceStability;
+
+import java.util.Collection;
+
+/**
+ * Options for {@link AdminClient#describeConsumerGroups(Collection, DescribeConsumerGroupsOptions)}.
+ * <p>
+ * The API of this class is evolving, see {@link AdminClient} for details.
+ */
+@InterfaceStability.Evolving
+public class DescribeConsumerGroupsOptions extends AbstractOptions<DescribeConsumerGroupsOptions> {
+}
\ No newline at end of file
diff --git a/clients/src/main/java/org/apache/kafka/clients/admin/DescribeConsumerGroupsResult.java b/clients/src/main/java/org/apache/kafka/clients/admin/DescribeConsumerGroupsResult.java
new file mode 100644
index 00000000000..8f0ebad4f7a
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/clients/admin/DescribeConsumerGroupsResult.java
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.clients.admin;
+
+import org.apache.kafka.common.KafkaFuture;
+import org.apache.kafka.common.annotation.InterfaceStability;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ExecutionException;
+
+
+/**
+ * The result of the {@link KafkaAdminClient#describeConsumerGroups(Collection, DescribeConsumerGroupsOptions)}} call.
+ *
+ * The API of this class is evolving, see {@link AdminClient} for details.
+ */
+@InterfaceStability.Evolving
+public class DescribeConsumerGroupsResult {
+
+ private final Map<String, KafkaFuture<ConsumerGroupDescription>> futures;
+
+ public DescribeConsumerGroupsResult(final Map<String, KafkaFuture<ConsumerGroupDescription>> futures) {
+ this.futures = futures;
+ }
+
+ /**
+ * Return a map from group id to futures which yield group descriptions.
+ */
+ public Map<String, KafkaFuture<ConsumerGroupDescription>> describedGroups() {
+ return futures;
+ }
+
+ /**
+ * Return a future which yields all ConsumerGroupDescription objects, if all the describes succeed.
+ */
+ public KafkaFuture<Map<String, ConsumerGroupDescription>> all() {
+ return KafkaFuture.allOf(futures.values().toArray(new KafkaFuture[0])).thenApply(
+ new KafkaFuture.BaseFunction<Void, Map<String, ConsumerGroupDescription>>() {
+ @Override
+ public Map<String, ConsumerGroupDescription> apply(Void v) {
+ try {
+ Map<String, ConsumerGroupDescription> descriptions = new HashMap<>(futures.size());
+ for (Map.Entry<String, KafkaFuture<ConsumerGroupDescription>> entry : futures.entrySet()) {
+ descriptions.put(entry.getKey(), entry.getValue().get());
+ }
+ return descriptions;
+ } catch (InterruptedException | ExecutionException e) {
+ // This should be unreachable, since the KafkaFuture#allOf already ensured
+ // that all of the futures completed successfully.
+ throw new RuntimeException(e);
+ }
+ }
+ });
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/clients/admin/DescribeDelegationTokenOptions.java b/clients/src/main/java/org/apache/kafka/clients/admin/DescribeDelegationTokenOptions.java
new file mode 100644
index 00000000000..60b99354e35
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/clients/admin/DescribeDelegationTokenOptions.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.clients.admin;
+
+import java.util.List;
+
+import org.apache.kafka.common.annotation.InterfaceStability;
+import org.apache.kafka.common.security.auth.KafkaPrincipal;
+
+/**
+ * Options for {@link AdminClient#describeDelegationToken(DescribeDelegationTokenOptions)}.
+ *
+ * The API of this class is evolving, see {@link AdminClient} for details.
+ */
+@InterfaceStability.Evolving
+public class DescribeDelegationTokenOptions extends AbstractOptions<DescribeDelegationTokenOptions> {
+ private List<KafkaPrincipal> owners;
+
+ /**
+ * if owners is null, all the user owned tokens and tokens where user have Describe permission
+ * will be returned.
+ * @param owners
+ * @return this instance
+ */
+ public DescribeDelegationTokenOptions owners(List<KafkaPrincipal> owners) {
+ this.owners = owners;
+ return this;
+ }
+
+ public List<KafkaPrincipal> owners() {
+ return owners;
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/clients/admin/DescribeDelegationTokenResult.java b/clients/src/main/java/org/apache/kafka/clients/admin/DescribeDelegationTokenResult.java
new file mode 100644
index 00000000000..7a9d4b9dd97
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/clients/admin/DescribeDelegationTokenResult.java
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.clients.admin;
+
+import java.util.List;
+
+import org.apache.kafka.common.KafkaFuture;
+import org.apache.kafka.common.annotation.InterfaceStability;
+import org.apache.kafka.common.security.token.delegation.DelegationToken;
+
+/**
+ * The result of the {@link KafkaAdminClient#describeDelegationToken(DescribeDelegationTokenOptions)} call.
+ *
+ * The API of this class is evolving, see {@link AdminClient} for details.
+ */
+@InterfaceStability.Evolving
+public class DescribeDelegationTokenResult {
+ private final KafkaFuture<List<DelegationToken>> delegationTokens;
+
+ DescribeDelegationTokenResult(KafkaFuture<List<DelegationToken>> delegationTokens) {
+ this.delegationTokens = delegationTokens;
+ }
+
+ /**
+ * Returns a future which yields list of delegation tokens
+ */
+ public KafkaFuture<List<DelegationToken>> delegationTokens() {
+ return delegationTokens;
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/clients/admin/ExpireDelegationTokenOptions.java b/clients/src/main/java/org/apache/kafka/clients/admin/ExpireDelegationTokenOptions.java
new file mode 100644
index 00000000000..138cd4e69e4
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/clients/admin/ExpireDelegationTokenOptions.java
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.clients.admin;
+
+import org.apache.kafka.common.annotation.InterfaceStability;
+
+/**
+ * Options for {@link AdminClient#expireDelegationToken(byte[], ExpireDelegationTokenOptions)}.
+ *
+ * The API of this class is evolving, see {@link AdminClient} for details.
+ */
+@InterfaceStability.Evolving
+public class ExpireDelegationTokenOptions extends AbstractOptions<ExpireDelegationTokenOptions> {
+ private long expiryTimePeriodMs = -1L;
+
+ public ExpireDelegationTokenOptions expiryTimePeriodMs(long expiryTimePeriodMs) {
+ this.expiryTimePeriodMs = expiryTimePeriodMs;
+ return this;
+ }
+
+ public long expiryTimePeriodMs() {
+ return expiryTimePeriodMs;
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/clients/admin/ExpireDelegationTokenResult.java b/clients/src/main/java/org/apache/kafka/clients/admin/ExpireDelegationTokenResult.java
new file mode 100644
index 00000000000..41782bdcb5c
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/clients/admin/ExpireDelegationTokenResult.java
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.clients.admin;
+
+import org.apache.kafka.common.KafkaFuture;
+import org.apache.kafka.common.annotation.InterfaceStability;
+
+/**
+ * The result of the {@link KafkaAdminClient#expireDelegationToken(byte[], ExpireDelegationTokenOptions)} call.
+ *
+ * The API of this class is evolving, see {@link AdminClient} for details.
+ */
+@InterfaceStability.Evolving
+public class ExpireDelegationTokenResult {
+ private final KafkaFuture<Long> expiryTimestamp;
+
+ ExpireDelegationTokenResult(KafkaFuture<Long> expiryTimestamp) {
+ this.expiryTimestamp = expiryTimestamp;
+ }
+
+ /**
+ * Returns a future which yields expiry timestamp
+ */
+ public KafkaFuture<Long> expiryTimestamp() {
+ return expiryTimestamp;
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java b/clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java
index 36cbe6cad00..6b02e11e9bb 100644
--- a/clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java
+++ b/clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java
@@ -22,12 +22,17 @@
import org.apache.kafka.clients.ClientResponse;
import org.apache.kafka.clients.ClientUtils;
import org.apache.kafka.clients.KafkaClient;
-import org.apache.kafka.clients.Metadata;
import org.apache.kafka.clients.NetworkClient;
+import org.apache.kafka.clients.StaleMetadataException;
import org.apache.kafka.clients.admin.DeleteAclsResult.FilterResult;
import org.apache.kafka.clients.admin.DeleteAclsResult.FilterResults;
import org.apache.kafka.clients.admin.DescribeReplicaLogDirsResult.ReplicaLogDirInfo;
+import org.apache.kafka.clients.admin.internals.AdminMetadataManager;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.clients.consumer.internals.ConsumerProtocol;
+import org.apache.kafka.clients.consumer.internals.PartitionAssignor;
import org.apache.kafka.common.Cluster;
+import org.apache.kafka.common.ConsumerGroupState;
import org.apache.kafka.common.KafkaException;
import org.apache.kafka.common.KafkaFuture;
import org.apache.kafka.common.Node;
@@ -41,8 +46,8 @@
import org.apache.kafka.common.config.ConfigResource;
import org.apache.kafka.common.errors.ApiException;
import org.apache.kafka.common.errors.AuthenticationException;
-import org.apache.kafka.common.errors.BrokerNotAvailableException;
import org.apache.kafka.common.errors.DisconnectException;
+import org.apache.kafka.common.errors.InvalidGroupIdException;
import org.apache.kafka.common.errors.InvalidRequestException;
import org.apache.kafka.common.errors.InvalidTopicException;
import org.apache.kafka.common.errors.RetriableException;
@@ -69,6 +74,8 @@
import org.apache.kafka.common.requests.CreateAclsRequest.AclCreation;
import org.apache.kafka.common.requests.CreateAclsResponse;
import org.apache.kafka.common.requests.CreateAclsResponse.AclCreationResponse;
+import org.apache.kafka.common.requests.CreateDelegationTokenRequest;
+import org.apache.kafka.common.requests.CreateDelegationTokenResponse;
import org.apache.kafka.common.requests.CreatePartitionsRequest;
import org.apache.kafka.common.requests.CreatePartitionsResponse;
import org.apache.kafka.common.requests.CreateTopicsRequest;
@@ -77,6 +84,8 @@
import org.apache.kafka.common.requests.DeleteAclsResponse;
import org.apache.kafka.common.requests.DeleteAclsResponse.AclDeletionResult;
import org.apache.kafka.common.requests.DeleteAclsResponse.AclFilterResponse;
+import org.apache.kafka.common.requests.DeleteGroupsRequest;
+import org.apache.kafka.common.requests.DeleteGroupsResponse;
import org.apache.kafka.common.requests.DeleteRecordsRequest;
import org.apache.kafka.common.requests.DeleteRecordsResponse;
import org.apache.kafka.common.requests.DeleteTopicsRequest;
@@ -85,12 +94,28 @@
import org.apache.kafka.common.requests.DescribeAclsResponse;
import org.apache.kafka.common.requests.DescribeConfigsRequest;
import org.apache.kafka.common.requests.DescribeConfigsResponse;
+import org.apache.kafka.common.requests.DescribeDelegationTokenRequest;
+import org.apache.kafka.common.requests.DescribeDelegationTokenResponse;
+import org.apache.kafka.common.requests.DescribeGroupsRequest;
+import org.apache.kafka.common.requests.DescribeGroupsResponse;
import org.apache.kafka.common.requests.DescribeLogDirsRequest;
import org.apache.kafka.common.requests.DescribeLogDirsResponse;
+import org.apache.kafka.common.requests.ExpireDelegationTokenRequest;
+import org.apache.kafka.common.requests.ExpireDelegationTokenResponse;
+import org.apache.kafka.common.requests.FindCoordinatorRequest;
+import org.apache.kafka.common.requests.FindCoordinatorResponse;
+import org.apache.kafka.common.requests.ListGroupsRequest;
+import org.apache.kafka.common.requests.ListGroupsResponse;
import org.apache.kafka.common.requests.MetadataRequest;
import org.apache.kafka.common.requests.MetadataResponse;
+import org.apache.kafka.common.requests.OffsetFetchRequest;
+import org.apache.kafka.common.requests.OffsetFetchResponse;
+import org.apache.kafka.common.requests.RenewDelegationTokenRequest;
+import org.apache.kafka.common.requests.RenewDelegationTokenResponse;
import org.apache.kafka.common.requests.Resource;
import org.apache.kafka.common.requests.ResourceType;
+import org.apache.kafka.common.security.token.delegation.DelegationToken;
+import org.apache.kafka.common.security.token.delegation.TokenInformation;
import org.apache.kafka.common.utils.AppInfoParser;
import org.apache.kafka.common.utils.KafkaThread;
import org.apache.kafka.common.utils.LogContext;
@@ -158,9 +183,9 @@
private final Time time;
/**
- * The cluster metadata used by the KafkaClient.
+ * The cluster metadata manager used by the KafkaClient.
*/
- private final Metadata metadata;
+ private final AdminMetadataManager metadataManager;
/**
* The metrics for this KafkaAdminClient.
@@ -195,6 +220,8 @@
private final int maxRetries;
+ private final long retryBackoffMs;
+
/**
* Get or create a list value from a map.
*
@@ -299,8 +326,9 @@ static KafkaAdminClient createInternal(AdminClientConfig config, TimeoutProcesso
try {
// Since we only request node information, it's safe to pass true for allowAutoTopicCreation (and it
// simplifies communication with older brokers)
- Metadata metadata = new Metadata(config.getLong(AdminClientConfig.RETRY_BACKOFF_MS_CONFIG),
- config.getLong(AdminClientConfig.METADATA_MAX_AGE_CONFIG), true);
+ AdminMetadataManager metadataManager = new AdminMetadataManager(logContext,
+ config.getLong(AdminClientConfig.RETRY_BACKOFF_MS_CONFIG),
+ config.getLong(AdminClientConfig.METADATA_MAX_AGE_CONFIG));
List<MetricsReporter> reporters = config.getConfiguredInstances(AdminClientConfig.METRIC_REPORTER_CLASSES_CONFIG,
MetricsReporter.class);
Map<String, String> metricTags = Collections.singletonMap("client-id", clientId);
@@ -316,7 +344,7 @@ static KafkaAdminClient createInternal(AdminClientConfig config, TimeoutProcesso
metrics, time, metricGrpPrefix, channelBuilder, logContext);
networkClient = new NetworkClient(
selector,
- metadata,
+ metadataManager.updater(),
clientId,
1,
config.getLong(AdminClientConfig.RECONNECT_BACKOFF_MS_CONFIG),
@@ -328,7 +356,7 @@ static KafkaAdminClient createInternal(AdminClientConfig config, TimeoutProcesso
true,
apiVersions,
logContext);
- return new KafkaAdminClient(config, clientId, time, metadata, metrics, networkClient,
+ return new KafkaAdminClient(config, clientId, time, metadataManager, metrics, networkClient,
timeoutProcessorFactory, logContext);
} catch (Throwable exc) {
closeQuietly(metrics, "Metrics");
@@ -339,35 +367,44 @@ static KafkaAdminClient createInternal(AdminClientConfig config, TimeoutProcesso
}
}
- static KafkaAdminClient createInternal(AdminClientConfig config, KafkaClient client, Metadata metadata, Time time) {
+ static KafkaAdminClient createInternal(AdminClientConfig config, KafkaClient client, Time time) {
Metrics metrics = null;
String clientId = generateClientId(config);
try {
metrics = new Metrics(new MetricConfig(), new LinkedList<MetricsReporter>(), time);
- return new KafkaAdminClient(config, clientId, time, metadata, metrics, client, null,
- createLogContext(clientId));
+ LogContext logContext = createLogContext(clientId);
+ AdminMetadataManager metadataManager = new AdminMetadataManager(logContext,
+ config.getLong(AdminClientConfig.RETRY_BACKOFF_MS_CONFIG),
+ config.getLong(AdminClientConfig.METADATA_MAX_AGE_CONFIG));
+ return new KafkaAdminClient(config, clientId, time, metadataManager, metrics,
+ client, null, logContext);
} catch (Throwable exc) {
closeQuietly(metrics, "Metrics");
throw new KafkaException("Failed create new KafkaAdminClient", exc);
}
}
- private static LogContext createLogContext(String clientId) {
+ static LogContext createLogContext(String clientId) {
return new LogContext("[AdminClient clientId=" + clientId + "] ");
}
- private KafkaAdminClient(AdminClientConfig config, String clientId, Time time, Metadata metadata,
- Metrics metrics, KafkaClient client, TimeoutProcessorFactory timeoutProcessorFactory,
- LogContext logContext) {
+ private KafkaAdminClient(AdminClientConfig config,
+ String clientId,
+ Time time,
+ AdminMetadataManager metadataManager,
+ Metrics metrics,
+ KafkaClient client,
+ TimeoutProcessorFactory timeoutProcessorFactory,
+ LogContext logContext) {
this.defaultTimeoutMs = config.getInt(AdminClientConfig.REQUEST_TIMEOUT_MS_CONFIG);
this.clientId = clientId;
this.log = logContext.logger(KafkaAdminClient.class);
this.time = time;
- this.metadata = metadata;
+ this.metadataManager = metadataManager;
List<InetSocketAddress> addresses = ClientUtils.parseAndValidateAddresses(
config.getList(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG));
- this.metadata.update(Cluster.bootstrap(addresses), Collections.<String>emptySet(), time.milliseconds());
+ metadataManager.update(Cluster.bootstrap(addresses), time.milliseconds());
this.metrics = metrics;
this.client = client;
this.runnable = new AdminClientRunnable();
@@ -376,6 +413,7 @@ private KafkaAdminClient(AdminClientConfig config, String clientId, Time time, M
this.timeoutProcessorFactory = (timeoutProcessorFactory == null) ?
new TimeoutProcessorFactory() : timeoutProcessorFactory;
this.maxRetries = config.getInt(AdminClientConfig.RETRIES_CONFIG);
+ this.retryBackoffMs = config.getLong(AdminClientConfig.RETRY_BACKOFF_MS_CONFIG);
config.logUnused();
AppInfoParser.registerAppInfo(JMX_PREFIX, clientId, metrics);
log.debug("Kafka admin client initialized");
@@ -434,6 +472,13 @@ public void close(long duration, TimeUnit unit) {
Node provide();
}
+ private class MetadataUpdateNodeIdProvider implements NodeProvider {
+ @Override
+ public Node provide() {
+ return client.leastLoadedNode(time.milliseconds());
+ }
+ }
+
private class ConstantNodeIdProvider implements NodeProvider {
private final int nodeId;
@@ -443,7 +488,16 @@ public void close(long duration, TimeUnit unit) {
@Override
public Node provide() {
- return metadata.fetch().nodeById(nodeId);
+ if (metadataManager.isReady() &&
+ (metadataManager.nodeById(nodeId) != null)) {
+ return metadataManager.nodeById(nodeId);
+ }
+ // If we can't find the node with the given constant ID, we schedule a
+ // metadata update and hope it appears. This behavior is useful for avoiding
+ // flaky behavior in tests when the cluster is starting up and not all nodes
+ // have appeared.
+ metadataManager.requestUpdate();
+ return null;
}
}
@@ -453,7 +507,12 @@ public Node provide() {
private class ControllerNodeProvider implements NodeProvider {
@Override
public Node provide() {
- return metadata.fetch().controller();
+ if (metadataManager.isReady() &&
+ (metadataManager.controller() != null)) {
+ return metadataManager.controller();
+ }
+ metadataManager.requestUpdate();
+ return null;
}
}
@@ -463,23 +522,41 @@ public Node provide() {
private class LeastLoadedNodeProvider implements NodeProvider {
@Override
public Node provide() {
- return client.leastLoadedNode(time.milliseconds());
+ if (metadataManager.isReady()) {
+ // This may return null if all nodes are busy.
+ // In that case, we will postpone node assignment.
+ return client.leastLoadedNode(time.milliseconds());
+ }
+ metadataManager.requestUpdate();
+ return null;
}
}
abstract class Call {
+ private final boolean internal;
private final String callName;
private final long deadlineMs;
private final NodeProvider nodeProvider;
private int tries = 0;
private boolean aborted = false;
+ private Node curNode = null;
+ private long nextAllowedTryMs = 0;
- Call(String callName, long deadlineMs, NodeProvider nodeProvider) {
+ Call(boolean internal, String callName, long deadlineMs, NodeProvider nodeProvider) {
+ this.internal = internal;
this.callName = callName;
this.deadlineMs = deadlineMs;
this.nodeProvider = nodeProvider;
}
+ Call(String callName, long deadlineMs, NodeProvider nodeProvider) {
+ this(false, callName, deadlineMs, nodeProvider);
+ }
+
+ protected Node curNode() {
+ return curNode;
+ }
+
/**
* Handle a failure.
*
@@ -508,11 +585,13 @@ final void fail(long now, Throwable throwable) {
// this RPC. That is why 'tries' is not incremented.
if ((throwable instanceof UnsupportedVersionException) &&
handleUnsupportedVersionException((UnsupportedVersionException) throwable)) {
- log.trace("{} attempting protocol downgrade.", this);
+ log.debug("{} attempting protocol downgrade and then retry.", this);
runnable.enqueue(this, now);
return;
}
tries++;
+ nextAllowedTryMs = now + retryBackoffMs;
+
// If the call has timed out, fail.
if (calcTimeoutMsRemainingAsInt(now, deadlineMs) < 0) {
if (log.isDebugEnabled()) {
@@ -587,6 +666,10 @@ boolean handleUnsupportedVersionException(UnsupportedVersionException exception)
public String toString() {
return "Call(callName=" + callName + ", deadlineMs=" + deadlineMs + ")";
}
+
+ public boolean isInternal() {
+ return internal;
+ }
}
static class TimeoutProcessorFactory {
@@ -664,58 +747,52 @@ int nextTimeoutMs() {
private final class AdminClientRunnable implements Runnable {
/**
- * Pending calls. Protected by the object monitor.
- * This will be null only if the thread has shut down.
+ * Calls which have not yet been assigned to a node.
+ * Only accessed from this thread.
*/
- private List<Call> newCalls = new LinkedList<>();
+ private final ArrayList<Call> pendingCalls = new ArrayList<>();
/**
- * Check if the AdminClient metadata is ready.
- * We need to know who the controller is, and have a non-empty view of the cluster.
- *
- * @param prevMetadataVersion The previous metadata version which wasn't usable.
- * @return null if the metadata is usable; the current metadata
- * version otherwise
+ * Maps nodes to calls that we want to send.
+ * Only accessed from this thread.
*/
- private Integer checkMetadataReady(Integer prevMetadataVersion) {
- if (prevMetadataVersion != null) {
- if (prevMetadataVersion == metadata.version())
- return prevMetadataVersion;
- }
- Cluster cluster = metadata.fetch();
- if (cluster.nodes().isEmpty()) {
- log.trace("Metadata is not ready yet. No cluster nodes found.");
- return metadata.requestUpdate();
- }
- if (cluster.controller() == null) {
- log.trace("Metadata is not ready yet. No controller found.");
- return metadata.requestUpdate();
- }
- if (prevMetadataVersion != null) {
- log.trace("Metadata is now ready.");
- }
- return null;
- }
+ private final Map<Node, List<Call>> callsToSend = new HashMap<>();
+
+ /**
+ * Maps node ID strings to calls that have been sent.
+ * Only accessed from this thread.
+ */
+ private final Map<String, List<Call>> callsInFlight = new HashMap<>();
+
+ /**
+ * Maps correlation IDs to calls that have been sent.
+ * Only accessed from this thread.
+ */
+ private final Map<Integer, Call> correlationIdToCalls = new HashMap<>();
+
+ /**
+ * Pending calls. Protected by the object monitor.
+ * This will be null only if the thread has shut down.
+ */
+ private List<Call> newCalls = new LinkedList<>();
/**
- * Time out the elements in the newCalls list which are expired.
+ * Time out the elements in the pendingCalls list which are expired.
*
* @param processor The timeout processor.
*/
- private synchronized void timeoutNewCalls(TimeoutProcessor processor) {
- int numTimedOut = processor.handleTimeouts(newCalls,
- "Timed out waiting for a node assignment.");
+ private void timeoutPendingCalls(TimeoutProcessor processor) {
+ int numTimedOut = processor.handleTimeouts(pendingCalls, "Timed out waiting for a node assignment.");
if (numTimedOut > 0)
- log.debug("Timed out {} new calls.", numTimedOut);
+ log.debug("Timed out {} pending calls.", numTimedOut);
}
/**
* Time out calls which have been assigned to nodes.
*
* @param processor The timeout processor.
- * @param callsToSend A map of nodes to the calls they need to handle.
*/
- private void timeoutCallsToSend(TimeoutProcessor processor, Map<Node, List<Call>> callsToSend) {
+ private int timeoutCallsToSend(TimeoutProcessor processor) {
int numTimedOut = 0;
for (List<Call> callList : callsToSend.values()) {
numTimedOut += processor.handleTimeouts(callList,
@@ -723,65 +800,70 @@ private void timeoutCallsToSend(TimeoutProcessor processor, Map<Node, List<Call>
}
if (numTimedOut > 0)
log.debug("Timed out {} call(s) with assigned nodes.", numTimedOut);
+ return numTimedOut;
}
/**
- * Choose nodes for the calls in the callsToSend list.
+ * Drain all the calls from newCalls into pendingCalls.
*
* This function holds the lock for the minimum amount of time, to avoid blocking
* users of AdminClient who will also take the lock to add new calls.
- *
- * @param now The current time in milliseconds.
- * @param callsToSend A map of nodes to the calls they need to handle.
- *
*/
- private void chooseNodesForNewCalls(long now, Map<Node, List<Call>> callsToSend) {
- List<Call> newCallsToAdd = null;
- synchronized (this) {
- if (newCalls.isEmpty()) {
- return;
- }
- newCallsToAdd = newCalls;
- newCalls = new LinkedList<>();
- }
- for (Call call : newCallsToAdd) {
- chooseNodeForNewCall(now, callsToSend, call);
+ private synchronized void drainNewCalls() {
+ if (!newCalls.isEmpty()) {
+ pendingCalls.addAll(newCalls);
+ newCalls.clear();
}
}
/**
- * Choose a node for a new call.
+ * Choose nodes for the calls in the pendingCalls list.
*
* @param now The current time in milliseconds.
- * @param callsToSend A map of nodes to the calls they need to handle.
- * @param call The call.
+ * @param pendingIter An iterator yielding pending calls.
*/
- private void chooseNodeForNewCall(long now, Map<Node, List<Call>> callsToSend, Call call) {
- Node node = call.nodeProvider.provide();
- if (node == null) {
- call.fail(now, new BrokerNotAvailableException(
- String.format("Error choosing node for %s: no node found.", call.callName)));
- return;
+ private long chooseNodesForPendingCalls(long now, Iterator<Call> pendingIter) {
+ long pollTimeout = Long.MAX_VALUE;
+ log.trace("Trying to choose nodes for {} at {}", pendingIter, now);
+ while (pendingIter.hasNext()) {
+ Call call = pendingIter.next();
+
+ // If the call is being retried, await the proper backoff before finding the node
+ if (now < call.nextAllowedTryMs) {
+ pollTimeout = Math.min(pollTimeout, call.nextAllowedTryMs - now);
+ continue;
+ }
+
+ Node node = null;
+ try {
+ node = call.nodeProvider.provide();
+ } catch (Throwable t) {
+ // Handle authentication errors while choosing nodes.
+ log.debug("Unable to choose node for {}", call, t);
+ pendingIter.remove();
+ call.fail(now, t);
+ }
+ if (node != null) {
+ log.trace("Assigned {} to node {}", call, node);
+ pendingIter.remove();
+ call.curNode = node;
+ getOrCreateListValue(callsToSend, node).add(call);
+ } else {
+ log.trace("Unable to assign {} to a node.", call);
+ }
}
- log.trace("Assigned {} to {}", call, node);
- getOrCreateListValue(callsToSend, node).add(call);
+ return pollTimeout;
}
/**
* Send the calls which are ready.
*
* @param now The current time in milliseconds.
- * @param callsToSend The calls to send, by node.
- * @param correlationIdToCalls A map of correlation IDs to calls.
- * @param callsInFlight A map of nodes to the calls they have in flight.
- *
* @return The minimum timeout we need for poll().
*/
- private long sendEligibleCalls(long now, Map<Node, List<Call>> callsToSend,
- Map<Integer, Call> correlationIdToCalls, Map<String, List<Call>> callsInFlight) {
+ private long sendEligibleCalls(long now) {
long pollTimeout = Long.MAX_VALUE;
- for (Iterator<Map.Entry<Node, List<Call>>> iter = callsToSend.entrySet().iterator();
- iter.hasNext(); ) {
+ for (Iterator<Map.Entry<Node, List<Call>>> iter = callsToSend.entrySet().iterator(); iter.hasNext(); ) {
Map.Entry<Node, List<Call>> entry = iter.next();
List<Call> calls = entry.getValue();
if (calls.isEmpty()) {
@@ -790,7 +872,7 @@ private long sendEligibleCalls(long now, Map<Node, List<Call>> callsToSend,
}
Node node = entry.getKey();
if (!client.ready(node, now)) {
- long nodeTimeout = client.connectionDelay(node, now);
+ long nodeTimeout = client.pollDelayMs(node, now);
pollTimeout = Math.min(pollTimeout, nodeTimeout);
log.trace("Client is not ready to send to {}. Must delay {} ms", node, nodeTimeout);
continue;
@@ -822,9 +904,8 @@ private long sendEligibleCalls(long now, Map<Node, List<Call>> callsToSend,
* to time them out is to close the entire connection.
*
* @param processor The timeout processor.
- * @param callsInFlight A map of nodes to the calls they have in flight.
*/
- private void timeoutCallsInFlight(TimeoutProcessor processor, Map<String, List<Call>> callsInFlight) {
+ private void timeoutCallsInFlight(TimeoutProcessor processor) {
int numTimedOut = 0;
for (Map.Entry<String, List<Call>> entry : callsInFlight.entrySet()) {
List<Call> contexts = entry.getValue();
@@ -852,51 +933,17 @@ private void timeoutCallsInFlight(TimeoutProcessor processor, Map<String, List<C
log.debug("Timed out {} call(s) in flight.", numTimedOut);
}
- /**
- * If an authentication exception is encountered with connection to any broker,
- * fail all pending requests.
- */
- private void handleAuthenticationException(long now, Map<Node, List<Call>> callsToSend) {
- AuthenticationException authenticationException = metadata.getAndClearAuthenticationException();
- if (authenticationException == null) {
- for (Node node : callsToSend.keySet()) {
- authenticationException = client.authenticationException(node);
- if (authenticationException != null)
- break;
- }
- }
- if (authenticationException != null) {
- synchronized (this) {
- failCalls(now, newCalls, authenticationException);
- }
- for (List<Call> calls : callsToSend.values()) {
- failCalls(now, calls, authenticationException);
- }
- callsToSend.clear();
- }
- }
-
- private void failCalls(long now, List<Call> calls, AuthenticationException authenticationException) {
- for (Call call : calls) {
- call.fail(now, authenticationException);
- }
- calls.clear();
- }
-
/**
* Handle responses from the server.
*
* @param now The current time in milliseconds.
* @param responses The latest responses from KafkaClient.
- * @param correlationIdToCall A map of correlation IDs to calls.
- * @param callsInFlight A map of nodes to the calls they have in flight.
- **/
- private void handleResponses(long now, List<ClientResponse> responses, Map<String, List<Call>> callsInFlight,
- Map<Integer, Call> correlationIdToCall) {
+ **/
+ private void handleResponses(long now, List<ClientResponse> responses) {
for (ClientResponse response : responses) {
int correlationId = response.requestHeader().correlationId();
- Call call = correlationIdToCall.get(correlationId);
+ Call call = correlationIdToCalls.get(correlationId);
if (call == null) {
// If the server returns information about a correlation ID we didn't use yet,
// an internal server error has occurred. Close the connection and log an error message.
@@ -908,7 +955,7 @@ private void handleResponses(long now, List<ClientResponse> responses, Map<Strin
}
// Stop tracking this call.
- correlationIdToCall.remove(correlationId);
+ correlationIdToCalls.remove(correlationId);
List<Call> calls = callsInFlight.get(response.destination());
if ((calls == null) || (!calls.remove(call))) {
log.error("Internal server error on {}: ignoring call {} in correlationIdToCall " +
@@ -921,9 +968,14 @@ private void handleResponses(long now, List<ClientResponse> responses, Map<Strin
if (response.versionMismatch() != null) {
call.fail(now, response.versionMismatch());
} else if (response.wasDisconnected()) {
- call.fail(now, new DisconnectException(String.format(
- "Cancelled %s request with correlation id %s due to node %s being disconnected",
- call.callName, correlationId, response.destination())));
+ AuthenticationException authException = client.authenticationException(call.curNode());
+ if (authException != null) {
+ call.fail(now, authException);
+ } else {
+ call.fail(now, new DisconnectException(String.format(
+ "Cancelled %s request with correlation id %s due to node %s being disconnected",
+ call.callName, correlationId, response.destination())));
+ }
} else {
try {
call.handleResponse(response.responseBody());
@@ -939,13 +991,59 @@ private void handleResponses(long now, List<ClientResponse> responses, Map<Strin
}
}
- private synchronized boolean threadShouldExit(long now, long curHardShutdownTimeMs,
- Map<Node, List<Call>> callsToSend, Map<Integer, Call> correlationIdToCalls) {
- if (newCalls.isEmpty() && callsToSend.isEmpty() && correlationIdToCalls.isEmpty()) {
+ /**
+ * Reassign calls that have not yet been sent. When metadata is refreshed,
+ * all unsent calls are reassigned to handle controller change and node changes.
+ * When a node is disconnected, all calls assigned to the node are reassigned.
+ *
+ * @param now The current time in milliseconds
+ * @param disconnectedOnly Reassign only calls to nodes that were disconnected
+ * in the last poll
+ */
+ private void reassignUnsentCalls(long now, boolean disconnectedOnly) {
+ ArrayList<Call> pendingCallsToSend = new ArrayList<>();
+ for (Iterator<Map.Entry<Node, List<Call>>> iter = callsToSend.entrySet().iterator(); iter.hasNext(); ) {
+ Map.Entry<Node, List<Call>> entry = iter.next();
+ if (!disconnectedOnly || client.connectionFailed(entry.getKey())) {
+ for (Call call : entry.getValue()) {
+ pendingCallsToSend.add(call);
+ }
+ iter.remove();
+ }
+ }
+ chooseNodesForPendingCalls(now, pendingCallsToSend.iterator());
+ }
+
+ private boolean hasActiveExternalCalls(Collection<Call> calls) {
+ for (Call call : calls) {
+ if (!call.isInternal()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Return true if there are currently active external calls.
+ */
+ private boolean hasActiveExternalCalls() {
+ if (hasActiveExternalCalls(pendingCalls)) {
+ return true;
+ }
+ for (List<Call> callList : callsToSend.values()) {
+ if (hasActiveExternalCalls(callList)) {
+ return true;
+ }
+ }
+ return hasActiveExternalCalls(correlationIdToCalls.values());
+ }
+
+ private boolean threadShouldExit(long now, long curHardShutdownTimeMs) {
+ if (!hasActiveExternalCalls()) {
log.trace("All work has been completed, and the I/O thread is now exiting.");
return true;
}
- if (now > curHardShutdownTimeMs) {
+ if (now >= curHardShutdownTimeMs) {
log.info("Forcing a hard I/O thread shutdown. Requests in progress will be aborted.");
return true;
}
@@ -955,52 +1053,42 @@ private synchronized boolean threadShouldExit(long now, long curHardShutdownTime
@Override
public void run() {
- /*
- * Maps nodes to calls that we want to send.
- */
- Map<Node, List<Call>> callsToSend = new HashMap<>();
-
- /*
- * Maps node ID strings to calls that have been sent.
- */
- Map<String, List<Call>> callsInFlight = new HashMap<>();
-
- /*
- * Maps correlation IDs to calls that have been sent.
- */
- Map<Integer, Call> correlationIdToCalls = new HashMap<>();
-
- /*
- * The previous metadata version which wasn't usable, or null if there is none.
- */
- Integer prevMetadataVersion = null;
-
long now = time.milliseconds();
log.trace("Thread starting");
while (true) {
+ // Copy newCalls into pendingCalls.
+ drainNewCalls();
+
// Check if the AdminClient thread should shut down.
long curHardShutdownTimeMs = hardShutdownTimeMs.get();
- if ((curHardShutdownTimeMs != INVALID_SHUTDOWN_TIME) &&
- threadShouldExit(now, curHardShutdownTimeMs, callsToSend, correlationIdToCalls))
+ if ((curHardShutdownTimeMs != INVALID_SHUTDOWN_TIME) && threadShouldExit(now, curHardShutdownTimeMs))
break;
// Handle timeouts.
TimeoutProcessor timeoutProcessor = timeoutProcessorFactory.create(now);
- timeoutNewCalls(timeoutProcessor);
- timeoutCallsToSend(timeoutProcessor, callsToSend);
- timeoutCallsInFlight(timeoutProcessor, callsInFlight);
+ timeoutPendingCalls(timeoutProcessor);
+ timeoutCallsToSend(timeoutProcessor);
+ timeoutCallsInFlight(timeoutProcessor);
long pollTimeout = Math.min(1200000, timeoutProcessor.nextTimeoutMs());
if (curHardShutdownTimeMs != INVALID_SHUTDOWN_TIME) {
pollTimeout = Math.min(pollTimeout, curHardShutdownTimeMs - now);
}
- // Handle new calls and metadata update requests.
- prevMetadataVersion = checkMetadataReady(prevMetadataVersion);
- if (prevMetadataVersion == null) {
- chooseNodesForNewCalls(now, callsToSend);
- pollTimeout = Math.min(pollTimeout,
- sendEligibleCalls(now, callsToSend, correlationIdToCalls, callsInFlight));
+ // Choose nodes for our pending calls.
+ pollTimeout = Math.min(pollTimeout, chooseNodesForPendingCalls(now, pendingCalls.iterator()));
+ long metadataFetchDelayMs = metadataManager.metadataFetchDelayMs(now);
+ if (metadataFetchDelayMs == 0) {
+ metadataManager.transitionToUpdatePending(now);
+ Call metadataCall = makeMetadataCall(now);
+ // Create a new metadata fetch call and add it to the end of pendingCalls.
+ // Assign a node for just the new call (we handled the other pending nodes above).
+ pendingCalls.add(metadataCall);
+ chooseNodesForPendingCalls(now, pendingCalls.listIterator(pendingCalls.size() - 1));
+ }
+ pollTimeout = Math.min(pollTimeout, sendEligibleCalls(now));
+ if (metadataFetchDelayMs > 0) {
+ pollTimeout = Math.min(pollTimeout, metadataFetchDelayMs);
}
// Wait for network responses.
@@ -1010,20 +1098,21 @@ public void run() {
// Update the current time and handle the latest responses.
now = time.milliseconds();
- handleAuthenticationException(now, callsToSend);
- handleResponses(now, responses, callsInFlight, correlationIdToCalls);
+ reassignUnsentCalls(now, true); // reassign calls to disconnected nodes
+ handleResponses(now, responses);
}
int numTimedOut = 0;
TimeoutProcessor timeoutProcessor = new TimeoutProcessor(Long.MAX_VALUE);
synchronized (this) {
- numTimedOut += timeoutProcessor.handleTimeouts(newCalls,
- "The AdminClient thread has exited.");
+ numTimedOut += timeoutProcessor.handleTimeouts(newCalls, "The AdminClient thread has exited.");
newCalls = null;
}
+ numTimedOut += timeoutProcessor.handleTimeouts(pendingCalls, "The AdminClient thread has exited.");
+ numTimedOut += timeoutCallsToSend(timeoutProcessor);
numTimedOut += timeoutProcessor.handleTimeouts(correlationIdToCalls.values(),
"The AdminClient thread has exited.");
if (numTimedOut > 0) {
- log.debug("Timed out {} remaining operations.", numTimedOut);
+ log.debug("Timed out {} remaining operation(s).", numTimedOut);
}
closeQuietly(client, "KafkaClient");
closeQuietly(metrics, "Metrics");
@@ -1075,6 +1164,35 @@ void call(Call call, long now) {
enqueue(call, now);
}
}
+
+ /**
+ * Create a new metadata call.
+ */
+ private Call makeMetadataCall(long now) {
+ return new Call(true, "fetchMetadata", calcDeadlineMs(now, defaultTimeoutMs),
+ new MetadataUpdateNodeIdProvider()) {
+ @Override
+ public AbstractRequest.Builder createRequest(int timeoutMs) {
+ // Since this only requests node information, it's safe to pass true
+ // for allowAutoTopicCreation (and it simplifies communication with
+ // older brokers)
+ return new MetadataRequest.Builder(Collections.<String>emptyList(), true);
+ }
+
+ @Override
+ public void handleResponse(AbstractResponse abstractResponse) {
+ MetadataResponse response = (MetadataResponse) abstractResponse;
+ long now = time.milliseconds();
+ metadataManager.update(response.cluster(), now);
+ reassignUnsentCalls(now, false);
+ }
+
+ @Override
+ public void handleFailure(Throwable e) {
+ metadataManager.updateFailed(e);
+ }
+ };
+ }
}
/**
@@ -1083,7 +1201,11 @@ void call(Call call, long now) {
* those policies on the server, so that they can be changed in the future if needed.
*/
private static boolean topicNameIsUnrepresentable(String topicName) {
- return (topicName == null) || topicName.isEmpty();
+ return topicName == null || topicName.isEmpty();
+ }
+
+ private static boolean groupIdIsUnrepresentable(String groupId) {
+ return groupId == null;
}
@Override
@@ -1114,6 +1236,14 @@ public CreateTopicsResult createTopics(final Collection<NewTopic> newTopics,
@Override
public void handleResponse(AbstractResponse abstractResponse) {
CreateTopicsResponse response = (CreateTopicsResponse) abstractResponse;
+ // Check for controller change
+ for (ApiError error : response.errors().values()) {
+ if (error.error() == Errors.NOT_CONTROLLER) {
+ metadataManager.clearController();
+ metadataManager.requestUpdate();
+ throw error.exception();
+ }
+ }
// Handle server responses for particular topics.
for (Map.Entry<String, ApiError> entry : response.errors().entrySet()) {
KafkaFutureImpl<Void> future = topicFutures.get(entry.getKey());
@@ -1150,9 +1280,10 @@ void handleFailure(Throwable throwable) {
}
@Override
- public DeleteTopicsResult deleteTopics(final Collection<String> topicNames,
+ public DeleteTopicsResult deleteTopics(Collection<String> topicNames,
DeleteTopicsOptions options) {
final Map<String, KafkaFutureImpl<Void>> topicFutures = new HashMap<>(topicNames.size());
+ final List<String> validTopicNames = new ArrayList<>(topicNames.size());
for (String topicName : topicNames) {
if (topicNameIsUnrepresentable(topicName)) {
KafkaFutureImpl<Void> future = new KafkaFutureImpl<>();
@@ -1161,6 +1292,7 @@ public DeleteTopicsResult deleteTopics(final Collection<String> topicNames,
topicFutures.put(topicName, future);
} else if (!topicFutures.containsKey(topicName)) {
topicFutures.put(topicName, new KafkaFutureImpl<Void>());
+ validTopicNames.add(topicName);
}
}
final long now = time.milliseconds();
@@ -1169,12 +1301,20 @@ public DeleteTopicsResult deleteTopics(final Collection<String> topicNames,
@Override
AbstractRequest.Builder createRequest(int timeoutMs) {
- return new DeleteTopicsRequest.Builder(new HashSet<>(topicNames), timeoutMs);
+ return new DeleteTopicsRequest.Builder(new HashSet<>(validTopicNames), timeoutMs);
}
@Override
void handleResponse(AbstractResponse abstractResponse) {
DeleteTopicsResponse response = (DeleteTopicsResponse) abstractResponse;
+ // Check for controller change
+ for (Errors error : response.errors().values()) {
+ if (error == Errors.NOT_CONTROLLER) {
+ metadataManager.clearController();
+ metadataManager.requestUpdate();
+ throw error.exception();
+ }
+ }
// Handle server responses for particular topics.
for (Map.Entry<String, Errors> entry : response.errors().entrySet()) {
KafkaFutureImpl<Void> future = topicFutures.get(entry.getKey());
@@ -1204,7 +1344,7 @@ void handleFailure(Throwable throwable) {
completeAllExceptionally(topicFutures.values(), throwable);
}
};
- if (!topicNames.isEmpty()) {
+ if (!validTopicNames.isEmpty()) {
runnable.call(call, now);
}
return new DeleteTopicsResult(new HashMap<String, KafkaFuture<Void>>(topicFutures));
@@ -1276,15 +1416,16 @@ public DescribeTopicsResult describeTopics(final Collection<String> topicNames,
void handleResponse(AbstractResponse abstractResponse) {
MetadataResponse response = (MetadataResponse) abstractResponse;
// Handle server responses for particular topics.
+ Cluster cluster = response.cluster();
+ Map<String, Errors> errors = response.errors();
for (Map.Entry<String, KafkaFutureImpl<TopicDescription>> entry : topicFutures.entrySet()) {
String topicName = entry.getKey();
KafkaFutureImpl<TopicDescription> future = entry.getValue();
- Errors topicError = response.errors().get(topicName);
+ Errors topicError = errors.get(topicName);
if (topicError != null) {
future.completeExceptionally(topicError.exception());
continue;
}
- Cluster cluster = response.cluster();
if (!cluster.topics().contains(topicName)) {
future.completeExceptionally(new InvalidTopicException("Topic " + topicName + " not found."));
continue;
@@ -1923,6 +2064,7 @@ void handleFailure(Throwable throwable) {
return new DescribeReplicaLogDirsResult(new HashMap<TopicPartitionReplica, KafkaFuture<ReplicaLogDirInfo>>(futures));
}
+ @Override
public CreatePartitionsResult createPartitions(Map<String, NewPartitions> newPartitions,
final CreatePartitionsOptions options) {
final Map<String, KafkaFutureImpl<Void>> futures = new HashMap<>(newPartitions.size());
@@ -1943,6 +2085,14 @@ public CreatePartitionsResult createPartitions(Map<String, NewPartitions> newPar
@Override
public void handleResponse(AbstractResponse abstractResponse) {
CreatePartitionsResponse response = (CreatePartitionsResponse) abstractResponse;
+ // Check for controller change
+ for (ApiError error : response.errors().values()) {
+ if (error.error() == Errors.NOT_CONTROLLER) {
+ metadataManager.clearController();
+ metadataManager.requestUpdate();
+ throw error.exception();
+ }
+ }
for (Map.Entry<String, ApiError> result : response.errors().entrySet()) {
KafkaFutureImpl<Void> future = futures.get(result.getKey());
if (result.getValue().isSuccess()) {
@@ -1961,6 +2111,7 @@ void handleFailure(Throwable throwable) {
return new CreatePartitionsResult(new HashMap<String, KafkaFuture<Void>>(futures));
}
+ @Override
public DeleteRecordsResult deleteRecords(final Map<TopicPartition, RecordsToDelete> recordsToDelete,
final DeleteRecordsOptions options) {
@@ -2069,4 +2220,513 @@ void handleFailure(Throwable throwable) {
return new DeleteRecordsResult(new HashMap<TopicPartition, KafkaFuture<DeletedRecords>>(futures));
}
+
+ @Override
+ public CreateDelegationTokenResult createDelegationToken(final CreateDelegationTokenOptions options) {
+ final KafkaFutureImpl<DelegationToken> delegationTokenFuture = new KafkaFutureImpl<>();
+ final long now = time.milliseconds();
+ runnable.call(new Call("createDelegationToken", calcDeadlineMs(now, options.timeoutMs()),
+ new LeastLoadedNodeProvider()) {
+
+ @Override
+ AbstractRequest.Builder createRequest(int timeoutMs) {
+ return new CreateDelegationTokenRequest.Builder(options.renewers(), options.maxlifeTimeMs());
+ }
+
+ @Override
+ void handleResponse(AbstractResponse abstractResponse) {
+ CreateDelegationTokenResponse response = (CreateDelegationTokenResponse) abstractResponse;
+ if (response.hasError()) {
+ delegationTokenFuture.completeExceptionally(response.error().exception());
+ } else {
+ TokenInformation tokenInfo = new TokenInformation(response.tokenId(), response.owner(),
+ options.renewers(), response.issueTimestamp(), response.maxTimestamp(), response.expiryTimestamp());
+ DelegationToken token = new DelegationToken(tokenInfo, response.hmacBytes());
+ delegationTokenFuture.complete(token);
+ }
+ }
+
+ @Override
+ void handleFailure(Throwable throwable) {
+ delegationTokenFuture.completeExceptionally(throwable);
+ }
+ }, now);
+
+ return new CreateDelegationTokenResult(delegationTokenFuture);
+ }
+
+ @Override
+ public RenewDelegationTokenResult renewDelegationToken(final byte[] hmac, final RenewDelegationTokenOptions options) {
+ final KafkaFutureImpl<Long> expiryTimeFuture = new KafkaFutureImpl<>();
+ final long now = time.milliseconds();
+ runnable.call(new Call("renewDelegationToken", calcDeadlineMs(now, options.timeoutMs()),
+ new LeastLoadedNodeProvider()) {
+
+ @Override
+ AbstractRequest.Builder createRequest(int timeoutMs) {
+ return new RenewDelegationTokenRequest.Builder(hmac, options.renewTimePeriodMs());
+ }
+
+ @Override
+ void handleResponse(AbstractResponse abstractResponse) {
+ RenewDelegationTokenResponse response = (RenewDelegationTokenResponse) abstractResponse;
+ if (response.hasError()) {
+ expiryTimeFuture.completeExceptionally(response.error().exception());
+ } else {
+ expiryTimeFuture.complete(response.expiryTimestamp());
+ }
+ }
+
+ @Override
+ void handleFailure(Throwable throwable) {
+ expiryTimeFuture.completeExceptionally(throwable);
+ }
+ }, now);
+
+ return new RenewDelegationTokenResult(expiryTimeFuture);
+ }
+
+ @Override
+ public ExpireDelegationTokenResult expireDelegationToken(final byte[] hmac, final ExpireDelegationTokenOptions options) {
+ final KafkaFutureImpl<Long> expiryTimeFuture = new KafkaFutureImpl<>();
+ final long now = time.milliseconds();
+ runnable.call(new Call("expireDelegationToken", calcDeadlineMs(now, options.timeoutMs()),
+ new LeastLoadedNodeProvider()) {
+
+ @Override
+ AbstractRequest.Builder createRequest(int timeoutMs) {
+ return new ExpireDelegationTokenRequest.Builder(hmac, options.expiryTimePeriodMs());
+ }
+
+ @Override
+ void handleResponse(AbstractResponse abstractResponse) {
+ ExpireDelegationTokenResponse response = (ExpireDelegationTokenResponse) abstractResponse;
+ if (response.hasError()) {
+ expiryTimeFuture.completeExceptionally(response.error().exception());
+ } else {
+ expiryTimeFuture.complete(response.expiryTimestamp());
+ }
+ }
+
+ @Override
+ void handleFailure(Throwable throwable) {
+ expiryTimeFuture.completeExceptionally(throwable);
+ }
+ }, now);
+
+ return new ExpireDelegationTokenResult(expiryTimeFuture);
+ }
+
+ @Override
+ public DescribeDelegationTokenResult describeDelegationToken(final DescribeDelegationTokenOptions options) {
+ final KafkaFutureImpl<List<DelegationToken>> tokensFuture = new KafkaFutureImpl<>();
+ final long now = time.milliseconds();
+ runnable.call(new Call("describeDelegationToken", calcDeadlineMs(now, options.timeoutMs()),
+ new LeastLoadedNodeProvider()) {
+
+ @Override
+ AbstractRequest.Builder createRequest(int timeoutMs) {
+ return new DescribeDelegationTokenRequest.Builder(options.owners());
+ }
+
+ @Override
+ void handleResponse(AbstractResponse abstractResponse) {
+ DescribeDelegationTokenResponse response = (DescribeDelegationTokenResponse) abstractResponse;
+ if (response.hasError()) {
+ tokensFuture.completeExceptionally(response.error().exception());
+ } else {
+ tokensFuture.complete(response.tokens());
+ }
+ }
+
+ @Override
+ void handleFailure(Throwable throwable) {
+ tokensFuture.completeExceptionally(throwable);
+ }
+ }, now);
+
+ return new DescribeDelegationTokenResult(tokensFuture);
+ }
+
+ @Override
+ public DescribeConsumerGroupsResult describeConsumerGroups(final Collection<String> groupIds,
+ final DescribeConsumerGroupsOptions options) {
+
+ final Map<String, KafkaFutureImpl<ConsumerGroupDescription>> futures = new HashMap<>(groupIds.size());
+ for (String groupId: groupIds) {
+ if (groupIdIsUnrepresentable(groupId)) {
+ KafkaFutureImpl<ConsumerGroupDescription> future = new KafkaFutureImpl<>();
+ future.completeExceptionally(new InvalidGroupIdException("The given group id '" +
+ groupId + "' cannot be represented in a request."));
+ futures.put(groupId, future);
+ } else if (!futures.containsKey(groupId)) {
+ futures.put(groupId, new KafkaFutureImpl<ConsumerGroupDescription>());
+ }
+ }
+
+ // TODO: KAFKA-6788, we should consider grouping the request per coordinator and send one request with a list of
+ // all consumer groups this coordinator host
+ for (final Map.Entry<String, KafkaFutureImpl<ConsumerGroupDescription>> entry : futures.entrySet()) {
+ // skip sending request for those futures that already failed.
+ if (entry.getValue().isCompletedExceptionally())
+ continue;
+
+ final String groupId = entry.getKey();
+
+ final long startFindCoordinatorMs = time.milliseconds();
+ final long deadline = calcDeadlineMs(startFindCoordinatorMs, options.timeoutMs());
+
+ runnable.call(new Call("findCoordinator", deadline, new LeastLoadedNodeProvider()) {
+ @Override
+ AbstractRequest.Builder createRequest(int timeoutMs) {
+ return new FindCoordinatorRequest.Builder(FindCoordinatorRequest.CoordinatorType.GROUP, groupId);
+ }
+
+ @Override
+ void handleResponse(AbstractResponse abstractResponse) {
+ final FindCoordinatorResponse fcResponse = (FindCoordinatorResponse) abstractResponse;
+ Errors error = fcResponse.error();
+ if (error == Errors.COORDINATOR_NOT_AVAILABLE) {
+ // Retry COORDINATOR_NOT_AVAILABLE, in case the error is temporary.
+ throw error.exception();
+ } else if (error != Errors.NONE) {
+ // All other errors are immediate failures.
+ KafkaFutureImpl<ConsumerGroupDescription> future = futures.get(groupId);
+ future.completeExceptionally(error.exception());
+ return;
+ }
+
+ final long nowDescribeConsumerGroups = time.milliseconds();
+ final int nodeId = fcResponse.node().id();
+ runnable.call(new Call("describeConsumerGroups", deadline, new ConstantNodeIdProvider(nodeId)) {
+ @Override
+ AbstractRequest.Builder createRequest(int timeoutMs) {
+ return new DescribeGroupsRequest.Builder(Collections.singletonList(groupId));
+ }
+
+ @Override
+ void handleResponse(AbstractResponse abstractResponse) {
+ final DescribeGroupsResponse response = (DescribeGroupsResponse) abstractResponse;
+
+ KafkaFutureImpl<ConsumerGroupDescription> future = futures.get(groupId);
+ final DescribeGroupsResponse.GroupMetadata groupMetadata = response.groups().get(groupId);
+
+ final Errors groupError = groupMetadata.error();
+ if (groupError != Errors.NONE) {
+ // TODO: KAFKA-6789, we can retry based on the error code
+ future.completeExceptionally(groupError.exception());
+ } else {
+ final String protocolType = groupMetadata.protocolType();
+ if (protocolType.equals(ConsumerProtocol.PROTOCOL_TYPE) || protocolType.isEmpty()) {
+ final List<DescribeGroupsResponse.GroupMember> members = groupMetadata.members();
+ final List<MemberDescription> memberDescriptions = new ArrayList<>(members.size());
+
+ for (DescribeGroupsResponse.GroupMember groupMember : members) {
+ Set<TopicPartition> partitions = Collections.emptySet();
+ if (groupMember.memberAssignment().remaining() > 0) {
+ final PartitionAssignor.Assignment assignment = ConsumerProtocol.
+ deserializeAssignment(groupMember.memberAssignment().duplicate());
+ partitions = new HashSet<>(assignment.partitions());
+ }
+ final MemberDescription memberDescription =
+ new MemberDescription(groupMember.memberId(),
+ groupMember.clientId(),
+ groupMember.clientHost(),
+ new MemberAssignment(partitions));
+ memberDescriptions.add(memberDescription);
+ }
+ final ConsumerGroupDescription consumerGroupDescription =
+ new ConsumerGroupDescription(groupId,
+ protocolType.isEmpty(),
+ memberDescriptions,
+ groupMetadata.protocol(),
+ ConsumerGroupState.parse(groupMetadata.state()),
+ fcResponse.node());
+ future.complete(consumerGroupDescription);
+ }
+ }
+ }
+
+ @Override
+ void handleFailure(Throwable throwable) {
+ KafkaFutureImpl<ConsumerGroupDescription> future = futures.get(groupId);
+ future.completeExceptionally(throwable);
+ }
+ }, nowDescribeConsumerGroups);
+ }
+
+ @Override
+ void handleFailure(Throwable throwable) {
+ KafkaFutureImpl<ConsumerGroupDescription> future = futures.get(groupId);
+ future.completeExceptionally(throwable);
+ }
+ }, startFindCoordinatorMs);
+ }
+
+ return new DescribeConsumerGroupsResult(new HashMap<String, KafkaFuture<ConsumerGroupDescription>>(futures));
+ }
+
+ private final static class ListConsumerGroupsResults {
+ private final List<Throwable> errors;
+ private final HashMap<String, ConsumerGroupListing> listings;
+ private final HashSet<Node> remaining;
+ private final KafkaFutureImpl<Collection<Object>> future;
+
+ ListConsumerGroupsResults(Collection<Node> leaders,
+ KafkaFutureImpl<Collection<Object>> future) {
+ this.errors = new ArrayList<>();
+ this.listings = new HashMap<>();
+ this.remaining = new HashSet<>(leaders);
+ this.future = future;
+ tryComplete();
+ }
+
+ synchronized void addError(Throwable throwable, Node node) {
+ ApiError error = ApiError.fromThrowable(throwable);
+ if (error.message() == null || error.message().isEmpty()) {
+ errors.add(error.error().exception("Error listing groups on " + node));
+ } else {
+ errors.add(error.error().exception("Error listing groups on " + node + ": " + error.message()));
+ }
+ }
+
+ synchronized void addListing(ConsumerGroupListing listing) {
+ listings.put(listing.groupId(), listing);
+ }
+
+ synchronized void tryComplete(Node leader) {
+ remaining.remove(leader);
+ tryComplete();
+ }
+
+ private synchronized void tryComplete() {
+ if (remaining.isEmpty()) {
+ ArrayList<Object> results = new ArrayList<Object>(listings.values());
+ results.addAll(errors);
+ future.complete(results);
+ }
+ }
+ };
+
+ @Override
+ public ListConsumerGroupsResult listConsumerGroups(ListConsumerGroupsOptions options) {
+ final KafkaFutureImpl<Collection<Object>> all = new KafkaFutureImpl<>();
+ final long nowMetadata = time.milliseconds();
+ final long deadline = calcDeadlineMs(nowMetadata, options.timeoutMs());
+ runnable.call(new Call("findAllBrokers", deadline, new LeastLoadedNodeProvider()) {
+ @Override
+ AbstractRequest.Builder createRequest(int timeoutMs) {
+ return new MetadataRequest.Builder(Collections.emptyList(), true);
+ }
+
+ @Override
+ void handleResponse(AbstractResponse abstractResponse) {
+ MetadataResponse metadataResponse = (MetadataResponse) abstractResponse;
+ Cluster cluster = metadataResponse.cluster();
+
+ if (cluster.nodes().isEmpty())
+ throw new StaleMetadataException("Metadata fetch failed due to missing broker list");
+
+ HashSet<Node> allNodes = new HashSet<>(cluster.nodes());
+ final ListConsumerGroupsResults results = new ListConsumerGroupsResults(allNodes, all);
+
+ for (final Node node : allNodes) {
+ final long nowList = time.milliseconds();
+ runnable.call(new Call("listConsumerGroups", deadline, new ConstantNodeIdProvider(node.id())) {
+ @Override
+ AbstractRequest.Builder createRequest(int timeoutMs) {
+ return new ListGroupsRequest.Builder();
+ }
+
+ private void maybeAddConsumerGroup(ListGroupsResponse.Group group) {
+ String protocolType = group.protocolType();
+ if (protocolType.equals(ConsumerProtocol.PROTOCOL_TYPE) || protocolType.isEmpty()) {
+ final String groupId = group.groupId();
+ final ConsumerGroupListing groupListing = new ConsumerGroupListing(groupId, protocolType.isEmpty());
+ results.addListing(groupListing);
+ }
+ }
+
+ @Override
+ void handleResponse(AbstractResponse abstractResponse) {
+ final ListGroupsResponse response = (ListGroupsResponse) abstractResponse;
+ synchronized (results) {
+ if (response.error() != Errors.NONE) {
+ results.addError(response.error().exception(), node);
+ } else {
+ for (ListGroupsResponse.Group group : response.groups()) {
+ maybeAddConsumerGroup(group);
+ }
+ }
+ results.tryComplete(node);
+ }
+ }
+
+ @Override
+ void handleFailure(Throwable throwable) {
+ synchronized (results) {
+ results.addError(throwable, node);
+ results.tryComplete(node);
+ }
+ }
+ }, nowList);
+ }
+ }
+
+ @Override
+ void handleFailure(Throwable throwable) {
+ KafkaException exception = new KafkaException("Failed to find brokers to send ListGroups", throwable);
+ all.complete(Collections.singletonList(exception));
+ }
+ }, nowMetadata);
+
+ return new ListConsumerGroupsResult(all);
+ }
+
+ @Override
+ public ListConsumerGroupOffsetsResult listConsumerGroupOffsets(final String groupId, final ListConsumerGroupOffsetsOptions options) {
+ final KafkaFutureImpl<Map<TopicPartition, OffsetAndMetadata>> groupOffsetListingFuture = new KafkaFutureImpl<>();
+
+ final long startFindCoordinatorMs = time.milliseconds();
+ final long deadline = calcDeadlineMs(startFindCoordinatorMs, options.timeoutMs());
+
+ runnable.call(new Call("findCoordinator", deadline, new LeastLoadedNodeProvider()) {
+ @Override
+ AbstractRequest.Builder createRequest(int timeoutMs) {
+ return new FindCoordinatorRequest.Builder(FindCoordinatorRequest.CoordinatorType.GROUP, groupId);
+ }
+
+ @Override
+ void handleResponse(AbstractResponse abstractResponse) {
+ final FindCoordinatorResponse response = (FindCoordinatorResponse) abstractResponse;
+
+ final long nowListConsumerGroupOffsets = time.milliseconds();
+
+ final int nodeId = response.node().id();
+
+ runnable.call(new Call("listConsumerGroupOffsets", deadline, new ConstantNodeIdProvider(nodeId)) {
+ @Override
+ AbstractRequest.Builder createRequest(int timeoutMs) {
+ return new OffsetFetchRequest.Builder(groupId, options.topicPartitions());
+ }
+
+ @Override
+ void handleResponse(AbstractResponse abstractResponse) {
+ final OffsetFetchResponse response = (OffsetFetchResponse) abstractResponse;
+ final Map<TopicPartition, OffsetAndMetadata> groupOffsetsListing = new HashMap<>();
+
+ if (response.hasError()) {
+ groupOffsetListingFuture.completeExceptionally(response.error().exception());
+ } else {
+ for (Map.Entry<TopicPartition, OffsetFetchResponse.PartitionData> entry :
+ response.responseData().entrySet()) {
+ final TopicPartition topicPartition = entry.getKey();
+ final Errors error = entry.getValue().error;
+
+ if (error == Errors.NONE) {
+ final Long offset = entry.getValue().offset;
+ final String metadata = entry.getValue().metadata;
+ groupOffsetsListing.put(topicPartition, new OffsetAndMetadata(offset, metadata));
+ } else {
+ log.warn("Skipping return offset for {} due to error {}.", topicPartition, error);
+ }
+ }
+ groupOffsetListingFuture.complete(groupOffsetsListing);
+ }
+ }
+
+ @Override
+ void handleFailure(Throwable throwable) {
+ groupOffsetListingFuture.completeExceptionally(throwable);
+ }
+ }, nowListConsumerGroupOffsets);
+ }
+
+ @Override
+ void handleFailure(Throwable throwable) {
+ groupOffsetListingFuture.completeExceptionally(throwable);
+ }
+ }, startFindCoordinatorMs);
+
+ return new ListConsumerGroupOffsetsResult(groupOffsetListingFuture);
+ }
+
+ @Override
+ public DeleteConsumerGroupsResult deleteConsumerGroups(Collection<String> groupIds, DeleteConsumerGroupsOptions options) {
+
+ final Map<String, KafkaFutureImpl<Void>> futures = new HashMap<>(groupIds.size());
+ for (String groupId: groupIds) {
+ if (groupIdIsUnrepresentable(groupId)) {
+ KafkaFutureImpl<Void> future = new KafkaFutureImpl<>();
+ future.completeExceptionally(new ApiException("The given group id '" +
+ groupId + "' cannot be represented in a request."));
+ futures.put(groupId, future);
+ } else if (!futures.containsKey(groupId)) {
+ futures.put(groupId, new KafkaFutureImpl<Void>());
+ }
+ }
+
+ // TODO: KAFKA-6788, we should consider grouping the request per coordinator and send one request with a list of
+ // all consumer groups this coordinator host
+ for (final String groupId : groupIds) {
+ // skip sending request for those futures that already failed.
+ if (futures.get(groupId).isCompletedExceptionally())
+ continue;
+
+ final long startFindCoordinatorMs = time.milliseconds();
+ final long deadline = calcDeadlineMs(startFindCoordinatorMs, options.timeoutMs());
+
+ runnable.call(new Call("findCoordinator", deadline, new LeastLoadedNodeProvider()) {
+ @Override
+ AbstractRequest.Builder createRequest(int timeoutMs) {
+ return new FindCoordinatorRequest.Builder(FindCoordinatorRequest.CoordinatorType.GROUP, groupId);
+ }
+
+ @Override
+ void handleResponse(AbstractResponse abstractResponse) {
+ final FindCoordinatorResponse response = (FindCoordinatorResponse) abstractResponse;
+
+ final long nowDeleteConsumerGroups = time.milliseconds();
+
+ final int nodeId = response.node().id();
+
+ runnable.call(new Call("deleteConsumerGroups", deadline, new ConstantNodeIdProvider(nodeId)) {
+
+ @Override
+ AbstractRequest.Builder createRequest(int timeoutMs) {
+ return new DeleteGroupsRequest.Builder(Collections.singleton(groupId));
+ }
+
+ @Override
+ void handleResponse(AbstractResponse abstractResponse) {
+ final DeleteGroupsResponse response = (DeleteGroupsResponse) abstractResponse;
+
+ KafkaFutureImpl<Void> future = futures.get(groupId);
+ final Errors groupError = response.get(groupId);
+
+ if (groupError != Errors.NONE) {
+ future.completeExceptionally(groupError.exception());
+ } else {
+ future.complete(null);
+ }
+ }
+
+ @Override
+ void handleFailure(Throwable throwable) {
+ KafkaFutureImpl<Void> future = futures.get(groupId);
+ future.completeExceptionally(throwable);
+ }
+ }, nowDeleteConsumerGroups);
+ }
+
+ @Override
+ void handleFailure(Throwable throwable) {
+ KafkaFutureImpl<Void> future = futures.get(groupId);
+ future.completeExceptionally(throwable);
+ }
+ }, startFindCoordinatorMs);
+ }
+
+ return new DeleteConsumerGroupsResult(new HashMap<String, KafkaFuture<Void>>(futures));
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/clients/admin/ListConsumerGroupOffsetsOptions.java b/clients/src/main/java/org/apache/kafka/clients/admin/ListConsumerGroupOffsetsOptions.java
new file mode 100644
index 00000000000..c6434ebb15c
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/clients/admin/ListConsumerGroupOffsetsOptions.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.clients.admin;
+
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.annotation.InterfaceStability;
+
+import java.util.List;
+
+/**
+ * Options for {@link AdminClient#listConsumerGroupOffsets(String)}.
+ * <p>
+ * The API of this class is evolving, see {@link AdminClient} for details.
+ */
+@InterfaceStability.Evolving
+public class ListConsumerGroupOffsetsOptions extends AbstractOptions<ListConsumerGroupOffsetsOptions> {
+
+ private List<TopicPartition> topicPartitions = null;
+
+ /**
+ * Set the topic partitions to list as part of the result.
+ * {@code null} includes all topic partitions.
+ *
+ * @param topicPartitions List of topic partitions to include
+ * @return This ListGroupOffsetsOptions
+ */
+ public ListConsumerGroupOffsetsOptions topicPartitions(List<TopicPartition> topicPartitions) {
+ this.topicPartitions = topicPartitions;
+ return this;
+ }
+
+ /**
+ * Returns a list of topic partitions to add as part of the result.
+ */
+ public List<TopicPartition> topicPartitions() {
+ return topicPartitions;
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/clients/admin/ListConsumerGroupOffsetsResult.java b/clients/src/main/java/org/apache/kafka/clients/admin/ListConsumerGroupOffsetsResult.java
new file mode 100644
index 00000000000..23657b5ad3c
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/clients/admin/ListConsumerGroupOffsetsResult.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.clients.admin;
+
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.KafkaFuture;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.annotation.InterfaceStability;
+
+import java.util.Map;
+
+/**
+ * The result of the {@link AdminClient#listConsumerGroupOffsets(String)} call.
+ * <p>
+ * The API of this class is evolving, see {@link AdminClient} for details.
+ */
+@InterfaceStability.Evolving
+public class ListConsumerGroupOffsetsResult {
+
+ final KafkaFuture<Map<TopicPartition, OffsetAndMetadata>> future;
+
+ ListConsumerGroupOffsetsResult(KafkaFuture<Map<TopicPartition, OffsetAndMetadata>> future) {
+ this.future = future;
+ }
+
+ /**
+ * Return a future which yields a map of topic partitions to OffsetAndMetadata objects.
+ */
+ public KafkaFuture<Map<TopicPartition, OffsetAndMetadata>> partitionsToOffsetAndMetadata() {
+ return future;
+ }
+
+}
diff --git a/clients/src/main/java/org/apache/kafka/clients/admin/ListConsumerGroupsOptions.java b/clients/src/main/java/org/apache/kafka/clients/admin/ListConsumerGroupsOptions.java
new file mode 100644
index 00000000000..86ca171b726
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/clients/admin/ListConsumerGroupsOptions.java
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.clients.admin;
+
+import org.apache.kafka.common.annotation.InterfaceStability;
+
+/**
+ * Options for {@link AdminClient#listConsumerGroups()}.
+ *
+ * The API of this class is evolving, see {@link AdminClient} for details.
+ */
+@InterfaceStability.Evolving
+public class ListConsumerGroupsOptions extends AbstractOptions<ListConsumerGroupsOptions> {
+}
diff --git a/clients/src/main/java/org/apache/kafka/clients/admin/ListConsumerGroupsResult.java b/clients/src/main/java/org/apache/kafka/clients/admin/ListConsumerGroupsResult.java
new file mode 100644
index 00000000000..7de485bd2b9
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/clients/admin/ListConsumerGroupsResult.java
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.clients.admin;
+
+import org.apache.kafka.common.KafkaFuture;
+import org.apache.kafka.common.annotation.InterfaceStability;
+import org.apache.kafka.common.internals.KafkaFutureImpl;
+
+import java.util.ArrayList;
+import java.util.Collection;
+
+/**
+ * The result of the {@link AdminClient#listConsumerGroups()} call.
+ * <p>
+ * The API of this class is evolving, see {@link AdminClient} for details.
+ */
+@InterfaceStability.Evolving
+public class ListConsumerGroupsResult {
+ private final KafkaFutureImpl<Collection<ConsumerGroupListing>> all;
+ private final KafkaFutureImpl<Collection<ConsumerGroupListing>> valid;
+ private final KafkaFutureImpl<Collection<Throwable>> errors;
+
+ ListConsumerGroupsResult(KafkaFutureImpl<Collection<Object>> future) {
+ this.all = new KafkaFutureImpl<>();
+ this.valid = new KafkaFutureImpl<>();
+ this.errors = new KafkaFutureImpl<>();
+ future.thenApply(new KafkaFuture.BaseFunction<Collection<Object>, Void>() {
+ @Override
+ public Void apply(Collection<Object> results) {
+ ArrayList<Throwable> curErrors = new ArrayList<>();
+ ArrayList<ConsumerGroupListing> curValid = new ArrayList<>();
+ for (Object resultObject : results) {
+ if (resultObject instanceof Throwable) {
+ curErrors.add((Throwable) resultObject);
+ } else {
+ curValid.add((ConsumerGroupListing) resultObject);
+ }
+ }
+ if (!curErrors.isEmpty()) {
+ all.completeExceptionally(curErrors.get(0));
+ } else {
+ all.complete(curValid);
+ }
+ valid.complete(curValid);
+ errors.complete(curErrors);
+ return null;
+ }
+ });
+ }
+
+ /**
+ * Returns a future that yields either an exception, or the full set of consumer group
+ * listings.
+ *
+ * In the event of a failure, the future yields nothing but the first exception which
+ * occurred.
+ */
+ public KafkaFuture<Collection<ConsumerGroupListing>> all() {
+ return all;
+ }
+
+ /**
+ * Returns a future which yields just the valid listings.
+ *
+ * This future never fails with an error, no matter what happens. Errors are completely
+ * ignored. If nothing can be fetched, an empty collection is yielded.
+ * If there is an error, but some results can be returned, this future will yield
+ * those partial results. When using this future, it is a good idea to also check
+ * the errors future so that errors can be displayed and handled.
+ */
+ public KafkaFuture<Collection<ConsumerGroupListing>> valid() {
+ return valid;
+ }
+
+ /**
+ * Returns a future which yields just the errors which occurred.
+ *
+ * If this future yields a non-empty collection, it is very likely that elements are
+ * missing from the valid() set.
+ *
+ * This future itself never fails with an error. In the event of an error, this future
+ * will successfully yield a collection containing at least one exception.
+ */
+ public KafkaFuture<Collection<Throwable>> errors() {
+ return errors;
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/clients/admin/MemberAssignment.java b/clients/src/main/java/org/apache/kafka/clients/admin/MemberAssignment.java
new file mode 100644
index 00000000000..8765c17d984
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/clients/admin/MemberAssignment.java
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.admin;
+
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.utils.Utils;
+
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+
+/**
+ * A description of the assignments of a specific group member.
+ */
+public class MemberAssignment {
+ private final Set<TopicPartition> topicPartitions;
+
+ /**
+ * Creates an instance with the specified parameters.
+ *
+ * @param topicPartitions List of topic partitions
+ */
+ public MemberAssignment(Set<TopicPartition> topicPartitions) {
+ this.topicPartitions = topicPartitions == null ? Collections.<TopicPartition>emptySet() :
+ Collections.unmodifiableSet(new HashSet<>(topicPartitions));
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
+ if (o == null || getClass() != o.getClass()) return false;
+
+ MemberAssignment that = (MemberAssignment) o;
+
+ return topicPartitions != null ? topicPartitions.equals(that.topicPartitions) : that.topicPartitions == null;
+ }
+
+ @Override
+ public int hashCode() {
+ return topicPartitions != null ? topicPartitions.hashCode() : 0;
+ }
+
+ /**
+ * The topic partitions assigned to a group member.
+ */
+ public Set<TopicPartition> topicPartitions() {
+ return topicPartitions;
+ }
+
+ @Override
+ public String toString() {
+ return "(topicPartitions=" + Utils.join(topicPartitions, ",") + ")";
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/clients/admin/MemberDescription.java b/clients/src/main/java/org/apache/kafka/clients/admin/MemberDescription.java
new file mode 100644
index 00000000000..a8865bed0ee
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/clients/admin/MemberDescription.java
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.clients.admin;
+
+import org.apache.kafka.common.TopicPartition;
+
+import java.util.Collections;
+import java.util.Objects;
+
+/**
+ * A detailed description of a single group instance in the cluster.
+ */
+public class MemberDescription {
+ private final String memberId;
+ private final String clientId;
+ private final String host;
+ private final MemberAssignment assignment;
+
+ public MemberDescription(String memberId, String clientId, String host, MemberAssignment assignment) {
+ this.memberId = memberId == null ? "" : memberId;
+ this.clientId = clientId == null ? "" : clientId;
+ this.host = host == null ? "" : host;
+ this.assignment = assignment == null ?
+ new MemberAssignment(Collections.<TopicPartition>emptySet()) : assignment;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
+ if (o == null || getClass() != o.getClass()) return false;
+ MemberDescription that = (MemberDescription) o;
+ return memberId.equals(that.memberId) &&
+ clientId.equals(that.clientId) &&
+ host.equals(that.host) &&
+ assignment.equals(that.assignment);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(memberId, clientId, host, assignment);
+ }
+
+ /**
+ * The consumer id of the group member.
+ */
+ public String consumerId() {
+ return memberId;
+ }
+
+ /**
+ * The client id of the group member.
+ */
+ public String clientId() {
+ return clientId;
+ }
+
+ /**
+ * The host where the group member is running.
+ */
+ public String host() {
+ return host;
+ }
+
+ /**
+ * The assignment of the group member.
+ */
+ public MemberAssignment assignment() {
+ return assignment;
+ }
+
+ @Override
+ public String toString() {
+ return "(memberId=" + memberId +
+ ", clientId=" + clientId +
+ ", host=" + host +
+ ", assignment=" + assignment + ")";
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/clients/admin/RenewDelegationTokenOptions.java b/clients/src/main/java/org/apache/kafka/clients/admin/RenewDelegationTokenOptions.java
new file mode 100644
index 00000000000..238dc4a3494
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/clients/admin/RenewDelegationTokenOptions.java
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.clients.admin;
+
+import org.apache.kafka.common.annotation.InterfaceStability;
+
+/**
+ * Options for {@link AdminClient#renewDelegationToken(byte[], RenewDelegationTokenOptions)}.
+ *
+ * The API of this class is evolving, see {@link AdminClient} for details.
+ */
+@InterfaceStability.Evolving
+public class RenewDelegationTokenOptions extends AbstractOptions<RenewDelegationTokenOptions> {
+ private long renewTimePeriodMs = -1;
+
+ public RenewDelegationTokenOptions renewTimePeriodMs(long renewTimePeriodMs) {
+ this.renewTimePeriodMs = renewTimePeriodMs;
+ return this;
+ }
+
+ public long renewTimePeriodMs() {
+ return renewTimePeriodMs;
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/clients/admin/RenewDelegationTokenResult.java b/clients/src/main/java/org/apache/kafka/clients/admin/RenewDelegationTokenResult.java
new file mode 100644
index 00000000000..38cdf1ae1b2
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/clients/admin/RenewDelegationTokenResult.java
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.clients.admin;
+
+import org.apache.kafka.common.KafkaFuture;
+import org.apache.kafka.common.annotation.InterfaceStability;
+
+/**
+ * The result of the {@link KafkaAdminClient#expireDelegationToken(byte[], ExpireDelegationTokenOptions)} call.
+ *
+ * The API of this class is evolving, see {@link AdminClient} for details.
+ */
+@InterfaceStability.Evolving
+public class RenewDelegationTokenResult {
+ private final KafkaFuture<Long> expiryTimestamp;
+
+ RenewDelegationTokenResult(KafkaFuture<Long> expiryTimestamp) {
+ this.expiryTimestamp = expiryTimestamp;
+ }
+
+ /**
+ * Returns a future which yields expiry timestamp
+ */
+ public KafkaFuture<Long> expiryTimestamp() {
+ return expiryTimestamp;
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/clients/admin/internals/AdminMetadataManager.java b/clients/src/main/java/org/apache/kafka/clients/admin/internals/AdminMetadataManager.java
new file mode 100644
index 00000000000..85d3c28e8df
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/clients/admin/internals/AdminMetadataManager.java
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.clients.admin.internals;
+
+import org.apache.kafka.clients.MetadataUpdater;
+import org.apache.kafka.common.Cluster;
+import org.apache.kafka.common.Node;
+import org.apache.kafka.common.PartitionInfo;
+import org.apache.kafka.common.errors.AuthenticationException;
+import org.apache.kafka.common.requests.MetadataResponse;
+import org.apache.kafka.common.requests.RequestHeader;
+import org.apache.kafka.common.utils.LogContext;
+import org.slf4j.Logger;
+
+import java.util.Collections;
+import java.util.List;
+
+/**
+ * Manages the metadata for KafkaAdminClient.
+ *
+ * This class is not thread-safe. It is only accessed from the AdminClient
+ * service thread (which also uses the NetworkClient).
+ */
+public class AdminMetadataManager {
+ private Logger log;
+
+ /**
+ * The minimum amount of time that we should wait between subsequent
+ * retries, when fetching metadata.
+ */
+ private final long refreshBackoffMs;
+
+ /**
+ * The minimum amount of time that we should wait before triggering an
+ * automatic metadata refresh.
+ */
+ private final long metadataExpireMs;
+
+ /**
+ * Used to update the NetworkClient metadata.
+ */
+ private final AdminMetadataUpdater updater;
+
+ /**
+ * The current metadata state.
+ */
+ private State state = State.QUIESCENT;
+
+ /**
+ * The time in wall-clock milliseconds when we last updated the metadata.
+ */
+ private long lastMetadataUpdateMs = 0;
+
+ /**
+ * The time in wall-clock milliseconds when we last attempted to fetch new
+ * metadata.
+ */
+ private long lastMetadataFetchAttemptMs = 0;
+
+ /**
+ * The current cluster information.
+ */
+ private Cluster cluster = Cluster.empty();
+
+ /**
+ * If we got an authorization exception when we last attempted to fetch
+ * metadata, this is it; null, otherwise.
+ */
+ private AuthenticationException authException = null;
+
+ public class AdminMetadataUpdater implements MetadataUpdater {
+ @Override
+ public List<Node> fetchNodes() {
+ return cluster.nodes();
+ }
+
+ @Override
+ public boolean isUpdateDue(long now) {
+ return false;
+ }
+
+ @Override
+ public long maybeUpdate(long now) {
+ return Long.MAX_VALUE;
+ }
+
+ @Override
+ public void handleDisconnection(String destination) {
+ // Do nothing
+ }
+
+ @Override
+ public void handleAuthenticationFailure(AuthenticationException e) {
+ updateFailed(e);
+ }
+
+ @Override
+ public void handleCompletedMetadataResponse(RequestHeader requestHeader, long now, MetadataResponse metadataResponse) {
+ // Do nothing
+ }
+
+ @Override
+ public void requestUpdate() {
+ AdminMetadataManager.this.requestUpdate();
+ }
+ }
+
+ /**
+ * The current AdminMetadataManager state.
+ */
+ enum State {
+ QUIESCENT,
+ UPDATE_REQUESTED,
+ UPDATE_PENDING
+ }
+
+ public AdminMetadataManager(LogContext logContext, long refreshBackoffMs, long metadataExpireMs) {
+ this.log = logContext.logger(AdminMetadataManager.class);
+ this.refreshBackoffMs = refreshBackoffMs;
+ this.metadataExpireMs = metadataExpireMs;
+ this.updater = new AdminMetadataUpdater();
+ }
+
+ public AdminMetadataUpdater updater() {
+ return updater;
+ }
+
+ public boolean isReady() {
+ if (authException != null) {
+ log.debug("Metadata is not usable: failed to get metadata.", authException);
+ throw authException;
+ }
+ if (cluster.nodes().isEmpty()) {
+ log.trace("Metadata is not ready: bootstrap nodes have not been " +
+ "initialized yet.");
+ return false;
+ }
+ if (cluster.isBootstrapConfigured()) {
+ log.trace("Metadata is not ready: we have not fetched metadata from " +
+ "the bootstrap nodes yet.");
+ return false;
+ }
+ log.trace("Metadata is ready to use.");
+ return true;
+ }
+
+ public Node controller() {
+ return cluster.controller();
+ }
+
+ public Node nodeById(int nodeId) {
+ return cluster.nodeById(nodeId);
+ }
+
+ public void requestUpdate() {
+ if (state == State.QUIESCENT) {
+ state = State.UPDATE_REQUESTED;
+ log.debug("Requesting metadata update.");
+ }
+ }
+
+ public void clearController() {
+ if (cluster.controller() != null) {
+ log.trace("Clearing cached controller node {}.", cluster.controller());
+ this.cluster = new Cluster(cluster.clusterResource().clusterId(),
+ cluster.nodes(),
+ Collections.<PartitionInfo>emptySet(),
+ Collections.<String>emptySet(),
+ Collections.<String>emptySet(),
+ null);
+ }
+ }
+
+ /**
+ * Determine if the AdminClient should fetch new metadata.
+ */
+ public long metadataFetchDelayMs(long now) {
+ switch (state) {
+ case QUIESCENT:
+ // Calculate the time remaining until the next periodic update.
+ // We want to avoid making many metadata requests in a short amount of time,
+ // so there is a metadata refresh backoff period.
+ return Math.max(delayBeforeNextAttemptMs(now), delayBeforeNextExpireMs(now));
+ case UPDATE_REQUESTED:
+ // Respect the backoff, even if an update has been requested
+ return delayBeforeNextAttemptMs(now);
+ default:
+ // An update is already pending, so we don't need to initiate another one.
+ return Long.MAX_VALUE;
+ }
+ }
+
+ private long delayBeforeNextExpireMs(long now) {
+ long timeSinceUpdate = now - lastMetadataUpdateMs;
+ return Math.max(0, metadataExpireMs - timeSinceUpdate);
+ }
+
+ private long delayBeforeNextAttemptMs(long now) {
+ long timeSinceAttempt = now - lastMetadataFetchAttemptMs;
+ return Math.max(0, refreshBackoffMs - timeSinceAttempt);
+ }
+
+ /**
+ * Transition into the UPDATE_PENDING state. Updates lastMetadataFetchAttemptMs.
+ */
+ public void transitionToUpdatePending(long now) {
+ this.state = State.UPDATE_PENDING;
+ this.lastMetadataFetchAttemptMs = now;
+ }
+
+ public void updateFailed(Throwable exception) {
+ // We depend on pending calls to request another metadata update
+ this.state = State.QUIESCENT;
+
+ if (exception instanceof AuthenticationException) {
+ log.warn("Metadata update failed due to authentication error", exception);
+ this.authException = (AuthenticationException) exception;
+ } else {
+ log.info("Metadata update failed", exception);
+ }
+ }
+
+ /**
+ * Receive new metadata, and transition into the QUIESCENT state.
+ * Updates lastMetadataUpdateMs, cluster, and authException.
+ */
+ public void update(Cluster cluster, long now) {
+ if (cluster.isBootstrapConfigured()) {
+ log.debug("Setting bootstrap cluster metadata {}.", cluster);
+ } else {
+ log.debug("Updating cluster metadata to {}", cluster);
+ this.lastMetadataUpdateMs = now;
+ }
+
+ this.state = State.QUIESCENT;
+ this.authException = null;
+
+ if (!cluster.nodes().isEmpty()) {
+ this.cluster = cluster;
+ }
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/clients/consumer/Consumer.java b/clients/src/main/java/org/apache/kafka/clients/consumer/Consumer.java
index 0e27e1f0383..07a6fb48a2a 100644
--- a/clients/src/main/java/org/apache/kafka/clients/consumer/Consumer.java
+++ b/clients/src/main/java/org/apache/kafka/clients/consumer/Consumer.java
@@ -22,6 +22,7 @@
import org.apache.kafka.common.TopicPartition;
import java.io.Closeable;
+import java.time.Duration;
import java.util.Collection;
import java.util.List;
import java.util.Map;
@@ -38,156 +39,212 @@
/**
* @see KafkaConsumer#assignment()
*/
- public Set<TopicPartition> assignment();
+ Set<TopicPartition> assignment();
/**
* @see KafkaConsumer#subscription()
*/
- public Set<String> subscription();
+ Set<String> subscription();
/**
* @see KafkaConsumer#subscribe(Collection)
*/
- public void subscribe(Collection<String> topics);
+ void subscribe(Collection<String> topics);
/**
* @see KafkaConsumer#subscribe(Collection, ConsumerRebalanceListener)
*/
- public void subscribe(Collection<String> topics, ConsumerRebalanceListener callback);
+ void subscribe(Collection<String> topics, ConsumerRebalanceListener callback);
/**
* @see KafkaConsumer#assign(Collection)
*/
- public void assign(Collection<TopicPartition> partitions);
+ void assign(Collection<TopicPartition> partitions);
/**
* @see KafkaConsumer#subscribe(Pattern, ConsumerRebalanceListener)
*/
- public void subscribe(Pattern pattern, ConsumerRebalanceListener callback);
+ void subscribe(Pattern pattern, ConsumerRebalanceListener callback);
/**
* @see KafkaConsumer#subscribe(Pattern)
*/
- public void subscribe(Pattern pattern);
+ void subscribe(Pattern pattern);
/**
* @see KafkaConsumer#unsubscribe()
*/
- public void unsubscribe();
+ void unsubscribe();
/**
* @see KafkaConsumer#poll(long)
*/
- public ConsumerRecords<K, V> poll(long timeout);
+ @Deprecated
+ ConsumerRecords<K, V> poll(long timeout);
+
+ /**
+ * @see KafkaConsumer#poll(Duration)
+ */
+ ConsumerRecords<K, V> poll(Duration timeout);
/**
* @see KafkaConsumer#commitSync()
*/
- public void commitSync();
+ void commitSync();
+
+ /**
+ * @see KafkaConsumer#commitSync(Duration)
+ */
+ void commitSync(Duration timeout);
/**
* @see KafkaConsumer#commitSync(Map)
*/
- public void commitSync(Map<TopicPartition, OffsetAndMetadata> offsets);
+ void commitSync(Map<TopicPartition, OffsetAndMetadata> offsets);
+ /**
+ * @see KafkaConsumer#commitSync(Map, Duration)
+ */
+ void commitSync(final Map<TopicPartition, OffsetAndMetadata> offsets, final Duration timeout);
/**
* @see KafkaConsumer#commitAsync()
*/
- public void commitAsync();
+ void commitAsync();
/**
* @see KafkaConsumer#commitAsync(OffsetCommitCallback)
*/
- public void commitAsync(OffsetCommitCallback callback);
+ void commitAsync(OffsetCommitCallback callback);
/**
* @see KafkaConsumer#commitAsync(Map, OffsetCommitCallback)
*/
- public void commitAsync(Map<TopicPartition, OffsetAndMetadata> offsets, OffsetCommitCallback callback);
+ void commitAsync(Map<TopicPartition, OffsetAndMetadata> offsets, OffsetCommitCallback callback);
/**
* @see KafkaConsumer#seek(TopicPartition, long)
*/
- public void seek(TopicPartition partition, long offset);
+ void seek(TopicPartition partition, long offset);
/**
* @see KafkaConsumer#seekToBeginning(Collection)
*/
- public void seekToBeginning(Collection<TopicPartition> partitions);
+ void seekToBeginning(Collection<TopicPartition> partitions);
/**
* @see KafkaConsumer#seekToEnd(Collection)
*/
- public void seekToEnd(Collection<TopicPartition> partitions);
+ void seekToEnd(Collection<TopicPartition> partitions);
/**
* @see KafkaConsumer#position(TopicPartition)
*/
- public long position(TopicPartition partition);
+ long position(TopicPartition partition);
+
+ /**
+ * @see KafkaConsumer#position(TopicPartition, Duration)
+ */
+ long position(TopicPartition partition, final Duration timeout);
/**
* @see KafkaConsumer#committed(TopicPartition)
*/
- public OffsetAndMetadata committed(TopicPartition partition);
+ OffsetAndMetadata committed(TopicPartition partition);
+
+ /**
+ * @see KafkaConsumer#committed(TopicPartition, Duration)
+ */
+ OffsetAndMetadata committed(TopicPartition partition, final Duration timeout);
/**
* @see KafkaConsumer#metrics()
*/
- public Map<MetricName, ? extends Metric> metrics();
+ Map<MetricName, ? extends Metric> metrics();
/**
* @see KafkaConsumer#partitionsFor(String)
*/
- public List<PartitionInfo> partitionsFor(String topic);
+ List<PartitionInfo> partitionsFor(String topic);
+
+ /**
+ * @see KafkaConsumer#partitionsFor(String, Duration)
+ */
+ List<PartitionInfo> partitionsFor(String topic, Duration timeout);
/**
* @see KafkaConsumer#listTopics()
*/
- public Map<String, List<PartitionInfo>> listTopics();
+ Map<String, List<PartitionInfo>> listTopics();
+
+ /**
+ * @see KafkaConsumer#listTopics(Duration)
+ */
+ Map<String, List<PartitionInfo>> listTopics(Duration timeout);
/**
* @see KafkaConsumer#paused()
*/
- public Set<TopicPartition> paused();
+ Set<TopicPartition> paused();
/**
* @see KafkaConsumer#pause(Collection)
*/
- public void pause(Collection<TopicPartition> partitions);
+ void pause(Collection<TopicPartition> partitions);
/**
* @see KafkaConsumer#resume(Collection)
*/
- public void resume(Collection<TopicPartition> partitions);
+ void resume(Collection<TopicPartition> partitions);
+
+ /**
+ * @see KafkaConsumer#offsetsForTimes(Map)
+ */
+ Map<TopicPartition, OffsetAndTimestamp> offsetsForTimes(Map<TopicPartition, Long> timestampsToSearch);
/**
- * @see KafkaConsumer#offsetsForTimes(java.util.Map)
+ * @see KafkaConsumer#offsetsForTimes(Map, Duration)
*/
- public Map<TopicPartition, OffsetAndTimestamp> offsetsForTimes(Map<TopicPartition, Long> timestampsToSearch);
+ Map<TopicPartition, OffsetAndTimestamp> offsetsForTimes(Map<TopicPartition, Long> timestampsToSearch, Duration timeout);
/**
- * @see KafkaConsumer#beginningOffsets(java.util.Collection)
+ * @see KafkaConsumer#beginningOffsets(Collection)
*/
- public Map<TopicPartition, Long> beginningOffsets(Collection<TopicPartition> partitions);
+ Map<TopicPartition, Long> beginningOffsets(Collection<TopicPartition> partitions);
/**
- * @see KafkaConsumer#endOffsets(java.util.Collection)
+ * @see KafkaConsumer#beginningOffsets(Collection, Duration)
*/
- public Map<TopicPartition, Long> endOffsets(Collection<TopicPartition> partitions);
+ Map<TopicPartition, Long> beginningOffsets(Collection<TopicPartition> partitions, Duration timeout);
+
+ /**
+ * @see KafkaConsumer#endOffsets(Collection)
+ */
+ Map<TopicPartition, Long> endOffsets(Collection<TopicPartition> partitions);
+
+ /**
+ * @see KafkaConsumer#endOffsets(Collection, Duration)
+ */
+ Map<TopicPartition, Long> endOffsets(Collection<TopicPartition> partitions, Duration timeout);
/**
* @see KafkaConsumer#close()
*/
- public void close();
+ void close();
/**
* @see KafkaConsumer#close(long, TimeUnit)
*/
- public void close(long timeout, TimeUnit unit);
+ @Deprecated
+ void close(long timeout, TimeUnit unit);
+
+ /**
+ * @see KafkaConsumer#close(Duration)
+ */
+ void close(Duration timeout);
/**
* @see KafkaConsumer#wakeup()
*/
- public void wakeup();
+ void wakeup();
}
diff --git a/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerInterceptor.java b/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerInterceptor.java
index 2f4e310f702..763fe512217 100644
--- a/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerInterceptor.java
+++ b/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerInterceptor.java
@@ -35,14 +35,16 @@
* the user configures the interceptor with the wrong key and value type parameters, the consumer will not throw an exception,
* just log the errors.
* <p>
- * ConsumerInterceptor callbacks are called from the same thread that invokes {@link org.apache.kafka.clients.consumer.KafkaConsumer#poll(long)}.
+ * ConsumerInterceptor callbacks are called from the same thread that invokes
+ * {@link org.apache.kafka.clients.consumer.KafkaConsumer#poll(java.time.Duration)}.
* <p>
* Implement {@link org.apache.kafka.common.ClusterResourceListener} to receive cluster metadata once it's available. Please see the class documentation for ClusterResourceListener for more information.
*/
public interface ConsumerInterceptor<K, V> extends Configurable {
/**
- * This is called just before the records are returned by {@link org.apache.kafka.clients.consumer.KafkaConsumer#poll(long)}
+ * This is called just before the records are returned by
+ * {@link org.apache.kafka.clients.consumer.KafkaConsumer#poll(java.time.Duration)}
* <p>
* This method is allowed to modify consumer records, in which case the new records will be
* returned. There is no limitation on number of records that could be returned from this
diff --git a/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerRebalanceListener.java b/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerRebalanceListener.java
index a37e1661079..74e8b060c73 100644
--- a/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerRebalanceListener.java
+++ b/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerRebalanceListener.java
@@ -44,7 +44,8 @@
* partition is reassigned it may want to automatically trigger a flush of this cache, before the new owner takes over
* consumption.
* <p>
- * This callback will only execute in the user thread as part of the {@link Consumer#poll(long) poll(long)} call whenever partition assignment changes.
+ * This callback will only execute in the user thread as part of the {@link Consumer#poll(java.time.Duration) poll(long)} call
+ * whenever partition assignment changes.
* <p>
* It is guaranteed that all consumer processes will invoke {@link #onPartitionsRevoked(Collection) onPartitionsRevoked} prior to
* any process invoking {@link #onPartitionsAssigned(Collection) onPartitionsAssigned}. So if offsets or other state is saved in the
@@ -91,7 +92,7 @@
* It is common for the revocation callback to use the consumer instance in order to commit offsets. It is possible
* for a {@link org.apache.kafka.common.errors.WakeupException} or {@link org.apache.kafka.common.errors.InterruptException}
* to be raised from one these nested invocations. In this case, the exception will be propagated to the current
- * invocation of {@link KafkaConsumer#poll(long)} in which this callback is being executed. This means it is not
+ * invocation of {@link KafkaConsumer#poll(java.time.Duration)} in which this callback is being executed. This means it is not
* necessary to catch these exceptions and re-attempt to wakeup or interrupt the consumer thread.
*
* @param partitions The list of partitions that were assigned to the consumer on the last rebalance
@@ -102,8 +103,8 @@
/**
* A callback method the user can implement to provide handling of customized offsets on completion of a successful
- * partition re-assignment. This method will be called after an offset re-assignment completes and before the
- * consumer starts fetching data, and only as the result of a {@link Consumer#poll(long) poll(long)} call.
+ * partition re-assignment. This method will be called after the partition re-assignment completes and before the
+ * consumer starts fetching data, and only as the result of a {@link Consumer#poll(java.time.Duration) poll(long)} call.
* <p>
* It is guaranteed that all the processes in a consumer group will execute their
* {@link #onPartitionsRevoked(Collection)} callback before any instance executes its
@@ -112,7 +113,7 @@
* It is common for the assignment callback to use the consumer instance in order to query offsets. It is possible
* for a {@link org.apache.kafka.common.errors.WakeupException} or {@link org.apache.kafka.common.errors.InterruptException}
* to be raised from one these nested invocations. In this case, the exception will be propagated to the current
- * invocation of {@link KafkaConsumer#poll(long)} in which this callback is being executed. This means it is not
+ * invocation of {@link KafkaConsumer#poll(java.time.Duration)} in which this callback is being executed. This means it is not
* necessary to catch these exceptions and re-attempt to wakeup or interrupt the consumer thread.
*
* @param partitions The list of partitions that are now assigned to the consumer (may include partitions previously
diff --git a/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerRecords.java b/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerRecords.java
index f2dc9bbc2ba..4d0f62c3a1d 100644
--- a/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerRecords.java
+++ b/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerRecords.java
@@ -29,7 +29,7 @@
/**
* A container that holds the list {@link ConsumerRecord} per partition for a
* particular topic. There is one {@link ConsumerRecord} list for every topic
- * partition returned by a {@link Consumer#poll(long)} operation.
+ * partition returned by a {@link Consumer#poll(java.time.Duration)} operation.
*/
public class ConsumerRecords<K, V> implements Iterable<ConsumerRecord<K, V>> {
diff --git a/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java b/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java
index 3cd034eff76..5bd6b935b39 100644
--- a/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java
+++ b/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java
@@ -24,7 +24,6 @@
import org.apache.kafka.clients.consumer.internals.ConsumerInterceptors;
import org.apache.kafka.clients.consumer.internals.ConsumerMetrics;
import org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient;
-import org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.PollCondition;
import org.apache.kafka.clients.consumer.internals.Fetcher;
import org.apache.kafka.clients.consumer.internals.NoOpConsumerRebalanceListener;
import org.apache.kafka.clients.consumer.internals.PartitionAssignor;
@@ -37,6 +36,7 @@
import org.apache.kafka.common.TopicPartition;
import org.apache.kafka.common.config.ConfigException;
import org.apache.kafka.common.errors.InterruptException;
+import org.apache.kafka.common.errors.TimeoutException;
import org.apache.kafka.common.internals.ClusterResourceListeners;
import org.apache.kafka.common.metrics.JmxReporter;
import org.apache.kafka.common.metrics.MetricConfig;
@@ -55,6 +55,7 @@
import org.slf4j.Logger;
import java.net.InetSocketAddress;
+import java.time.Duration;
import java.util.Collection;
import java.util.Collections;
import java.util.ConcurrentModificationException;
@@ -98,7 +99,7 @@
* <p>
* The {@link #position(TopicPartition) position} of the consumer gives the offset of the next record that will be given
* out. It will be one larger than the highest offset the consumer has seen in that partition. It automatically advances
- * every time the consumer receives messages in a call to {@link #poll(long)}.
+ * every time the consumer receives messages in a call to {@link #poll(Duration)}.
* <p>
* The {@link #commitSync() committed position} is the last offset that has been stored securely. Should the
* process fail and restart, this is the offset that the consumer will recover to. The consumer can either automatically commit
@@ -149,7 +150,7 @@
*
* <h3><a name="failuredetection">Detecting Consumer Failures</a></h3>
*
- * After subscribing to a set of topics, the consumer will automatically join the group when {@link #poll(long)} is
+ * After subscribing to a set of topics, the consumer will automatically join the group when {@link #poll(Duration)} is
* invoked. The poll API is designed to ensure consumer liveness. As long as you continue to call poll, the consumer
* will stay in the group and continue to receive messages from the partitions it was assigned. Underneath the covers,
* the consumer sends periodic heartbeats to the server. If the consumer crashes or is unable to send heartbeats for
@@ -168,10 +169,10 @@
* The consumer provides two configuration settings to control the behavior of the poll loop:
* <ol>
* <li><code>max.poll.interval.ms</code>: By increasing the interval between expected polls, you can give
- * the consumer more time to handle a batch of records returned from {@link #poll(long)}. The drawback
+ * the consumer more time to handle a batch of records returned from {@link #poll(Duration)}. The drawback
* is that increasing this value may delay a group rebalance since the consumer will only join the rebalance
* inside the call to poll. You can use this setting to bound the time to finish a rebalance, but
- * you risk slower progress if the consumer cannot actually call {@link #poll(long) poll} often enough.</li>
+ * you risk slower progress if the consumer cannot actually call {@link #poll(Duration) poll} often enough.</li>
* <li><code>max.poll.records</code>: Use this setting to limit the total records returned from a single
* call to poll. This can make it easier to predict the maximum that must be handled within each poll
* interval. By tuning this value, you may be able to reduce the poll interval, which will reduce the
@@ -180,12 +181,12 @@
* <p>
* For use cases where message processing time varies unpredictably, neither of these options may be sufficient.
* The recommended way to handle these cases is to move message processing to another thread, which allows
- * the consumer to continue calling {@link #poll(long) poll} while the processor is still working. Some care must be taken
- * to ensure that committed offsets do not get ahead of the actual position. Typically, you must disable automatic
- * commits and manually commit processed offsets for records only after the thread has finished handling them
- * (depending on the delivery semantics you need). Note also that you will need to {@link #pause(Collection) pause}
- * the partition so that no new records are received from poll until after thread has finished handling those
- * previously returned.
+ * the consumer to continue calling {@link #poll(Duration) poll} while the processor is still working.
+ * Some care must be taken to ensure that committed offsets do not get ahead of the actual position.
+ * Typically, you must disable automatic commits and manually commit processed offsets for records only after the
+ * thread has finished handling them (depending on the delivery semantics you need).
+ * Note also that you will need to {@link #pause(Collection) pause} the partition so that no new records are received
+ * from poll until after thread has finished handling those previously returned.
*
* <h3>Usage Examples</h3>
* The consumer APIs offer flexibility to cover a variety of consumption use cases. Here are some examples to
@@ -258,7 +259,8 @@
*
* In this example we will consume a batch of records and batch them up in memory. When we have enough records
* batched, we will insert them into a database. If we allowed offsets to auto commit as in the previous example, records
- * would be considered consumed after they were returned to the user in {@link #poll(long) poll}. It would then be possible
+ * would be considered consumed after they were returned to the user in {@link #poll(Duration) poll}. It would then be
+ * possible
* for our process to fail after batching the records, but before they had been inserted into the database.
* <p>
* To avoid this, we will manually commit the offsets only after the corresponding records have been inserted into the
@@ -270,7 +272,7 @@
* time but in failure cases could be duplicated.
* <p>
* <b>Note: Using automatic offset commits can also give you "at-least-once" delivery, but the requirement is that
- * you must consume all data returned from each call to {@link #poll(long)} before any subsequent calls, or before
+ * you must consume all data returned from each call to {@link #poll(Duration)} before any subsequent calls, or before
* {@link #close() closing} the consumer. If you fail to do either of these, it is possible for the committed offset
* to get ahead of the consumed position, which results in missing records. The advantage of using manual offset
* control is that you have direct control over when a record is considered "consumed."</b>
@@ -325,7 +327,7 @@
* consumer.assign(Arrays.asList(partition0, partition1));
* </pre>
*
- * Once assigned, you can call {@link #poll(long) poll} in a loop, just as in the preceding examples to consume
+ * Once assigned, you can call {@link #poll(Duration) poll} in a loop, just as in the preceding examples to consume
* records. The group that the consumer specifies is still used for committing offsets, but now the set of partitions
* will only change with another call to {@link #assign(Collection) assign}. Manual partition assignment does
* not use group coordination, so consumer failures will not cause assigned partitions to be rebalanced. Each consumer
@@ -417,7 +419,7 @@
* <p>
* Kafka supports dynamic controlling of consumption flows by using {@link #pause(Collection)} and {@link #resume(Collection)}
* to pause the consumption on the specified assigned partitions and resume the consumption
- * on the specified paused partitions respectively in the future {@link #poll(long)} calls.
+ * on the specified paused partitions respectively in the future {@link #poll(Duration)} calls.
*
* <h3>Reading Transactional Messages</h3>
*
@@ -468,7 +470,7 @@
* try {
* consumer.subscribe(Arrays.asList("topic"));
* while (!closed.get()) {
- * ConsumerRecords records = consumer.poll(10000);
+ * ConsumerRecords records = consumer.poll(Duration.ofMillis(10000));
* // Handle new records
* }
* } catch (WakeupException e) {
@@ -574,6 +576,9 @@
// refcount is used to allow reentrant access by the thread who has acquired currentThread
private final AtomicInteger refcount = new AtomicInteger(0);
+ // to keep from repeatedly scanning subscriptions in poll(), cache the result during metadata updates
+ private boolean cachedSubscriptionHashAllFetchPositions;
+
/**
* A consumer is instantiated by providing a set of key-value pairs as configuration. Valid configuration strings
* are documented <a href="http://kafka.apache.org/documentation.html#consumerconfigs" >here</a>. Values can be
@@ -878,7 +883,7 @@ private KafkaConsumer(ConsumerConfig config,
* <p>
* When any of these events are triggered, the provided listener will be invoked first to indicate that
* the consumer's assignment has been revoked, and then again when the new assignment has been received.
- * Note that rebalances will only occur during an active call to {@link #poll(long)}, so callbacks will
+ * Note that rebalances will only occur during an active call to {@link #poll(Duration)}, so callbacks will
* also only be invoked during that time.
*
* The provided listener will immediately override any listener set in a previous call to subscribe.
@@ -954,7 +959,7 @@ public void subscribe(Collection<String> topics) {
* See {@link #subscribe(Collection, ConsumerRebalanceListener)} for details on the
* use of the {@link ConsumerRebalanceListener}. Generally rebalances are triggered when there
* is a change to the topics matching the provided pattern and when consumer group membership changes.
- * Group rebalances only take place during an active call to {@link #poll(long)}.
+ * Group rebalances only take place during an active call to {@link #poll(Duration)}.
*
* @param pattern Pattern to subscribe to
* @param listener Non-null listener instance to get notifications on partition assignment/revocation for the
@@ -966,13 +971,12 @@ public void subscribe(Collection<String> topics) {
*/
@Override
public void subscribe(Pattern pattern, ConsumerRebalanceListener listener) {
+ if (pattern == null)
+ throw new IllegalArgumentException("Topic pattern to subscribe to cannot be null");
+
acquireAndEnsureOpen();
try {
- if (pattern == null)
- throw new IllegalArgumentException("Topic pattern to subscribe to cannot be null");
-
throwIfNoAssignorsConfigured();
-
log.debug("Subscribed to pattern: {}", pattern);
this.subscriptions.subscribe(pattern, listener);
this.metadata.needMetadataForAllTopics(true);
@@ -1097,22 +1101,88 @@ public void assign(Collection<TopicPartition> partitions) {
* @throws java.lang.IllegalArgumentException if the timeout value is negative
* @throws java.lang.IllegalStateException if the consumer is not subscribed to any topics or manually assigned any
* partitions to consume from
+ *
+ *
+ * @deprecated Since 2.0. Use {@link #poll(Duration)}, which does not block beyond the timeout awaiting partition
+ * assignment. See <a href="https://cwiki.apache.org/confluence/x/5kiHB">KIP-266</a> for more information.
*/
+ @Deprecated
@Override
- public ConsumerRecords<K, V> poll(long timeout) {
+ public ConsumerRecords<K, V> poll(final long timeout) {
+ return poll(timeout, false);
+ }
+
+ /**
+ * Fetch data for the topics or partitions specified using one of the subscribe/assign APIs. It is an error to not have
+ * subscribed to any topics or partitions before polling for data.
+ * <p>
+ * On each poll, consumer will try to use the last consumed offset as the starting offset and fetch sequentially. The last
+ * consumed offset can be manually set through {@link #seek(TopicPartition, long)} or automatically set as the last committed
+ * offset for the subscribed list of partitions
+ *
+ * <p>
+ * This method returns immediately if there are records available. Otherwise, it will await the passed timeout.
+ * If the timeout expires, an empty record set will be returned. Note that this method may block beyond the
+ * timeout in order to execute custom {@link ConsumerRebalanceListener} callbacks.
+ *
+ *
+ * @param timeout The maximum time to block (must not be greater than {@link Long#MAX_VALUE} milliseconds)
+ *
+ * @return map of topic to records since the last fetch for the subscribed list of topics and partitions
+ *
+ * @throws org.apache.kafka.clients.consumer.InvalidOffsetException if the offset for a partition or set of
+ * partitions is undefined or out of range and no offset reset policy has been configured
+ * @throws org.apache.kafka.common.errors.WakeupException if {@link #wakeup()} is called before or while this
+ * function is called
+ * @throws org.apache.kafka.common.errors.InterruptException if the calling thread is interrupted before or while
+ * this function is called
+ * @throws org.apache.kafka.common.errors.AuthenticationException if authentication fails. See the exception for more details
+ * @throws org.apache.kafka.common.errors.AuthorizationException if caller lacks Read access to any of the subscribed
+ * topics or to the configured groupId. See the exception for more details
+ * @throws org.apache.kafka.common.KafkaException for any other unrecoverable errors (e.g. invalid groupId or
+ * session timeout, errors deserializing key/value pairs, or any new error cases in future versions)
+ * @throws java.lang.IllegalArgumentException if the timeout value is negative
+ * @throws java.lang.IllegalStateException if the consumer is not subscribed to any topics or manually assigned any
+ * partitions to consume from
+ * @throws java.lang.ArithmeticException if the timeout is greater than {@link Long#MAX_VALUE} milliseconds.
+ */
+ @Override
+ public ConsumerRecords<K, V> poll(final Duration timeout) {
+ return poll(timeout.toMillis(), true);
+ }
+
+ private ConsumerRecords<K, V> poll(final long timeoutMs, final boolean includeMetadataInTimeout) {
acquireAndEnsureOpen();
try {
- if (timeout < 0)
- throw new IllegalArgumentException("Timeout must not be negative");
+ if (timeoutMs < 0) throw new IllegalArgumentException("Timeout must not be negative");
- if (this.subscriptions.hasNoSubscriptionOrUserAssignment())
+ if (this.subscriptions.hasNoSubscriptionOrUserAssignment()) {
throw new IllegalStateException("Consumer is not subscribed to any topics or assigned any partitions");
+ }
// poll for new data until the timeout expires
- long start = time.milliseconds();
- long remaining = timeout;
+ long elapsedTime = 0L;
do {
- Map<TopicPartition, List<ConsumerRecord<K, V>>> records = pollOnce(remaining);
+
+ client.maybeTriggerWakeup();
+
+ final long metadataEnd;
+ if (includeMetadataInTimeout) {
+ final long metadataStart = time.milliseconds();
+ if (!updateAssignmentMetadataIfNeeded(remainingTimeAtLeastZero(timeoutMs, elapsedTime))) {
+ return ConsumerRecords.empty();
+ }
+ metadataEnd = time.milliseconds();
+ elapsedTime += metadataEnd - metadataStart;
+ } else {
+ while (!updateAssignmentMetadataIfNeeded(Long.MAX_VALUE)) {
+ log.warn("Still waiting for metadata");
+ }
+ metadataEnd = time.milliseconds();
+ }
+
+ final Map<TopicPartition, List<ConsumerRecord<K, V>>> records = pollForFetches(remainingTimeAtLeastZero(timeoutMs, elapsedTime));
+
if (!records.isEmpty()) {
// before returning the fetched records, we can send off the next round of fetches
// and avoid block waiting for their responses to enable pipelining while the user
@@ -1120,15 +1190,16 @@ public void assign(Collection<TopicPartition> partitions) {
//
// NOTE: since the consumed position has already been updated, we must not allow
// wakeups or any other errors to be triggered prior to returning the fetched records.
- if (fetcher.sendFetches() > 0 || client.hasPendingRequests())
+ if (fetcher.sendFetches() > 0 || client.hasPendingRequests()) {
client.pollNoWakeup();
+ }
return this.interceptors.onConsume(new ConsumerRecords<>(records));
}
+ final long fetchEnd = time.milliseconds();
+ elapsedTime += fetchEnd - metadataEnd;
- long elapsed = time.milliseconds() - start;
- remaining = timeout - elapsed;
- } while (remaining > 0);
+ } while (elapsedTime < timeoutMs);
return ConsumerRecords.empty();
} finally {
@@ -1137,56 +1208,61 @@ public void assign(Collection<TopicPartition> partitions) {
}
/**
- * Do one round of polling. In addition to checking for new data, this does any needed offset commits
- * (if auto-commit is enabled), and offset resets (if an offset reset policy is defined).
- * @param timeout The maximum time to block in the underlying call to {@link ConsumerNetworkClient#poll(long)}.
- * @return The fetched records (may be empty)
+ * Visible for testing
*/
- private Map<TopicPartition, List<ConsumerRecord<K, V>>> pollOnce(long timeout) {
- client.maybeTriggerWakeup();
+ boolean updateAssignmentMetadataIfNeeded(final long timeoutMs) {
+ final long startMs = time.milliseconds();
+ if (!coordinator.poll(timeoutMs)) {
+ return false;
+ }
- long startMs = time.milliseconds();
- coordinator.poll(startMs, timeout);
+ return updateFetchPositions(remainingTimeAtLeastZero(timeoutMs, time.milliseconds() - startMs));
+ }
- // Lookup positions of assigned partitions
- boolean hasAllFetchPositions = updateFetchPositions();
+ private Map<TopicPartition, List<ConsumerRecord<K, V>>> pollForFetches(final long timeoutMs) {
+ final long startMs = time.milliseconds();
+ long pollTimeout = Math.min(coordinator.timeToNextPoll(startMs), timeoutMs);
// if data is available already, return it immediately
- Map<TopicPartition, List<ConsumerRecord<K, V>>> records = fetcher.fetchedRecords();
- if (!records.isEmpty())
+ final Map<TopicPartition, List<ConsumerRecord<K, V>>> records = fetcher.fetchedRecords();
+ if (!records.isEmpty()) {
return records;
+ }
// send any new fetches (won't resend pending fetches)
fetcher.sendFetches();
- long nowMs = time.milliseconds();
- long remainingTimeMs = Math.max(0, timeout - (nowMs - startMs));
- long pollTimeout = Math.min(coordinator.timeToNextPoll(nowMs), remainingTimeMs);
-
// We do not want to be stuck blocking in poll if we are missing some positions
// since the offset lookup may be backing off after a failure
- if (!hasAllFetchPositions && pollTimeout > retryBackoffMs)
+
+ // NOTE: the use of cachedSubscriptionHashAllFetchPositions means we MUST call
+ // updateAssignmentMetadataIfNeeded before this method.
+ if (!cachedSubscriptionHashAllFetchPositions && pollTimeout > retryBackoffMs) {
pollTimeout = retryBackoffMs;
+ }
- client.poll(pollTimeout, nowMs, new PollCondition() {
- @Override
- public boolean shouldBlock() {
- // since a fetch might be completed by the background thread, we need this poll condition
- // to ensure that we do not block unnecessarily in poll()
- return !fetcher.hasCompletedFetches();
- }
+ client.poll(pollTimeout, startMs, () -> {
+ // since a fetch might be completed by the background thread, we need this poll condition
+ // to ensure that we do not block unnecessarily in poll()
+ return !fetcher.hasCompletedFetches();
});
// after the long poll, we should check whether the group needs to rebalance
// prior to returning data so that the group can stabilize faster
- if (coordinator.needRejoin())
+ if (coordinator.rejoinNeededOrPending()) {
return Collections.emptyMap();
+ }
return fetcher.fetchedRecords();
}
+ private long remainingTimeAtLeastZero(final long timeoutMs, final long elapsedTime) {
+ return Math.max(0, timeoutMs - elapsedTime);
+ }
+
/**
- * Commit offsets returned on the last {@link #poll(long) poll()} for all the subscribed list of topics and partitions.
+ * Commit offsets returned on the last {@link #poll(Duration) poll()} for all the subscribed list of topics and
+ * partitions.
* <p>
* This commits offsets only to Kafka. The offsets committed using this API will be used on the first fetch after
* every rebalance and also on startup. As such, if you need to store offsets in anything other than Kafka, this API
@@ -1213,9 +1289,46 @@ public boolean shouldBlock() {
*/
@Override
public void commitSync() {
+ commitSync(Duration.ofMillis(Long.MAX_VALUE));
+ }
+
+ /**
+ * Commit offsets returned on the last {@link #poll(Duration) poll()} for all the subscribed list of topics and
+ * partitions.
+ * <p>
+ * This commits offsets only to Kafka. The offsets committed using this API will be used on the first fetch after
+ * every rebalance and also on startup. As such, if you need to store offsets in anything other than Kafka, this API
+ * should not be used.
+ * <p>
+ * This is a synchronous commits and will block until either the commit succeeds, an unrecoverable error is
+ * encountered (in which case it is thrown to the caller), or the passed timeout expires.
+ * <p>
+ * Note that asynchronous offset commits sent previously with the {@link #commitAsync(OffsetCommitCallback)}
+ * (or similar) are guaranteed to have their callbacks invoked prior to completion of this method.
+ *
+ * @throws org.apache.kafka.clients.consumer.CommitFailedException if the commit failed and cannot be retried.
+ * This can only occur if you are using automatic group management with {@link #subscribe(Collection)},
+ * or if there is an active group with the same groupId which is using group management.
+ * @throws org.apache.kafka.common.errors.WakeupException if {@link #wakeup()} is called before or while this
+ * function is called
+ * @throws org.apache.kafka.common.errors.InterruptException if the calling thread is interrupted before or while
+ * this function is called
+ * @throws org.apache.kafka.common.errors.AuthenticationException if authentication fails. See the exception for more details
+ * @throws org.apache.kafka.common.errors.AuthorizationException if not authorized to the topic or to the
+ * configured groupId. See the exception for more details
+ * @throws org.apache.kafka.common.KafkaException for any other unrecoverable errors (e.g. if offset metadata
+ * is too large or if the topic does not exist).
+ * @throws org.apache.kafka.common.errors.TimeoutException if the timeout expires before successful completion
+ * of the offset commit
+ */
+ @Override
+ public void commitSync(Duration timeout) {
acquireAndEnsureOpen();
try {
- coordinator.commitOffsetsSync(subscriptions.allConsumed(), Long.MAX_VALUE);
+ if (!coordinator.commitOffsetsSync(subscriptions.allConsumed(), timeout.toMillis())) {
+ throw new TimeoutException("Timeout of " + timeout.toMillis() + "ms expired before successfully " +
+ "committing the current consumed offsets");
+ }
} finally {
release();
}
@@ -1252,16 +1365,56 @@ public void commitSync() {
*/
@Override
public void commitSync(final Map<TopicPartition, OffsetAndMetadata> offsets) {
+ commitSync(offsets, Duration.ofMillis(Long.MAX_VALUE));
+ }
+
+ /**
+ * Commit the specified offsets for the specified list of topics and partitions.
+ * <p>
+ * This commits offsets to Kafka. The offsets committed using this API will be used on the first fetch after every
+ * rebalance and also on startup. As such, if you need to store offsets in anything other than Kafka, this API
+ * should not be used. The committed offset should be the next message your application will consume,
+ * i.e. lastProcessedMessageOffset + 1.
+ * <p>
+ * This is a synchronous commits and will block until either the commit succeeds, an unrecoverable error is
+ * encountered (in which case it is thrown to the caller), or the timeout expires.
+ * <p>
+ * Note that asynchronous offset commits sent previously with the {@link #commitAsync(OffsetCommitCallback)}
+ * (or similar) are guaranteed to have their callbacks invoked prior to completion of this method.
+ *
+ * @param offsets A map of offsets by partition with associated metadata
+ * @param timeout The maximum amount of time to await completion of the offset commit
+ * @throws org.apache.kafka.clients.consumer.CommitFailedException if the commit failed and cannot be retried.
+ * This can only occur if you are using automatic group management with {@link #subscribe(Collection)},
+ * or if there is an active group with the same groupId which is using group management.
+ * @throws org.apache.kafka.common.errors.WakeupException if {@link #wakeup()} is called before or while this
+ * function is called
+ * @throws org.apache.kafka.common.errors.InterruptException if the calling thread is interrupted before or while
+ * this function is called
+ * @throws org.apache.kafka.common.errors.AuthenticationException if authentication fails. See the exception for more details
+ * @throws org.apache.kafka.common.errors.AuthorizationException if not authorized to the topic or to the
+ * configured groupId. See the exception for more details
+ * @throws java.lang.IllegalArgumentException if the committed offset is negative
+ * @throws org.apache.kafka.common.KafkaException for any other unrecoverable errors (e.g. if offset metadata
+ * is too large or if the topic does not exist).
+ * @throws org.apache.kafka.common.errors.TimeoutException if the timeout expires before successful completion
+ * of the offset commit
+ */
+ @Override
+ public void commitSync(final Map<TopicPartition, OffsetAndMetadata> offsets, final Duration timeout) {
acquireAndEnsureOpen();
try {
- coordinator.commitOffsetsSync(new HashMap<>(offsets), Long.MAX_VALUE);
+ if (!coordinator.commitOffsetsSync(new HashMap<>(offsets), timeout.toMillis())) {
+ throw new TimeoutException("Timeout of " + timeout.toMillis() + "ms expired before successfully " +
+ "committing offsets " + offsets);
+ }
} finally {
release();
}
}
/**
- * Commit offsets returned on the last {@link #poll(long) poll()} for all the subscribed list of topics and partition.
+ * Commit offsets returned on the last {@link #poll(Duration)} for all the subscribed list of topics and partition.
* Same as {@link #commitAsync(OffsetCommitCallback) commitAsync(null)}
*/
@Override
@@ -1270,7 +1423,7 @@ public void commitAsync() {
}
/**
- * Commit offsets returned on the last {@link #poll(long) poll()} for the subscribed list of topics and partitions.
+ * Commit offsets returned on the last {@link #poll(Duration) poll()} for the subscribed list of topics and partitions.
* <p>
* This commits offsets only to Kafka. The offsets committed using this API will be used on the first fetch after
* every rebalance and also on startup. As such, if you need to store offsets in anything other than Kafka, this API
@@ -1328,20 +1481,20 @@ public void commitAsync(final Map<TopicPartition, OffsetAndMetadata> offsets, Of
}
/**
- * Overrides the fetch offsets that the consumer will use on the next {@link #poll(long) poll(timeout)}. If this API
+ * Overrides the fetch offsets that the consumer will use on the next {@link #poll(Duration) poll(timeout)}. If this API
* is invoked for the same partition more than once, the latest offset will be used on the next poll(). Note that
* you may lose data if this API is arbitrarily used in the middle of consumption, to reset the fetch offsets
*
- * @throws IllegalArgumentException if the provided TopicPartition is not assigned to this consumer
- * or if provided offset is negative
+ * @throws IllegalArgumentException if the provided offset is negative
+ * @throws IllegalStateException if the provided TopicPartition is not assigned to this consumer
*/
@Override
public void seek(TopicPartition partition, long offset) {
+ if (offset < 0)
+ throw new IllegalArgumentException("seek offset must not be a negative number");
+
acquireAndEnsureOpen();
try {
- if (offset < 0)
- throw new IllegalArgumentException("seek offset must not be a negative number");
-
log.debug("Seeking to offset {} for partition {}", offset, partition);
this.subscriptions.seek(partition, offset);
} finally {
@@ -1351,17 +1504,19 @@ public void seek(TopicPartition partition, long offset) {
/**
* Seek to the first offset for each of the given partitions. This function evaluates lazily, seeking to the
- * first offset in all partitions only when {@link #poll(long)} or {@link #position(TopicPartition)} are called.
+ * first offset in all partitions only when {@link #poll(Duration)} or {@link #position(TopicPartition)} are called.
* If no partitions are provided, seek to the first offset for all of the currently assigned partitions.
*
- * @throws IllegalArgumentException if {@code partitions} is {@code null} or the provided TopicPartition is not assigned to this consumer
+ * @throws IllegalArgumentException if {@code partitions} is {@code null}
+ * @throws IllegalStateException if any of the provided partitions are not currently assigned to this consumer
*/
+ @Override
public void seekToBeginning(Collection<TopicPartition> partitions) {
+ if (partitions == null)
+ throw new IllegalArgumentException("Partitions collection cannot be null");
+
acquireAndEnsureOpen();
try {
- if (partitions == null) {
- throw new IllegalArgumentException("Partitions collection cannot be null");
- }
Collection<TopicPartition> parts = partitions.size() == 0 ? this.subscriptions.assignedPartitions() : partitions;
for (TopicPartition tp : parts) {
log.debug("Seeking to beginning of partition {}", tp);
@@ -1374,20 +1529,22 @@ public void seekToBeginning(Collection<TopicPartition> partitions) {
/**
* Seek to the last offset for each of the given partitions. This function evaluates lazily, seeking to the
- * final offset in all partitions only when {@link #poll(long)} or {@link #position(TopicPartition)} are called.
+ * final offset in all partitions only when {@link #poll(Duration)} or {@link #position(TopicPartition)} are called.
* If no partitions are provided, seek to the final offset for all of the currently assigned partitions.
* <p>
* If {@code isolation.level=read_committed}, the end offset will be the Last Stable Offset, i.e., the offset
* of the first message with an open transaction.
*
- * @throws IllegalArgumentException if {@code partitions} is {@code null} or the provided TopicPartition is not assigned to this consumer
+ * @throws IllegalArgumentException if {@code partitions} is {@code null}
+ * @throws IllegalStateException if any of the provided partitions are not currently assigned to this consumer
*/
+ @Override
public void seekToEnd(Collection<TopicPartition> partitions) {
+ if (partitions == null)
+ throw new IllegalArgumentException("Partitions collection cannot be null");
+
acquireAndEnsureOpen();
try {
- if (partitions == null) {
- throw new IllegalArgumentException("Partitions collection cannot be null");
- }
Collection<TopicPartition> parts = partitions.size() == 0 ? this.subscriptions.assignedPartitions() : partitions;
for (TopicPartition tp : parts) {
log.debug("Seeking to end of partition {}", tp);
@@ -1407,7 +1564,7 @@ public void seekToEnd(Collection<TopicPartition> partitions) {
*
* @param partition The partition to get the position for
* @return The current position of the consumer (that is, the offset of the next record to be fetched)
- * @throws IllegalArgumentException if the provided TopicPartition is not assigned to this consumer
+ * @throws IllegalStateException if the provided TopicPartition is not assigned to this consumer
* @throws org.apache.kafka.clients.consumer.InvalidOffsetException if no offset is currently defined for
* the partition
* @throws org.apache.kafka.common.errors.WakeupException if {@link #wakeup()} is called before or while this
@@ -1419,18 +1576,61 @@ public void seekToEnd(Collection<TopicPartition> partitions) {
* configured groupId. See the exception for more details
* @throws org.apache.kafka.common.KafkaException for any other unrecoverable errors
*/
+ @Override
public long position(TopicPartition partition) {
+ return position(partition, Duration.ofMillis(Long.MAX_VALUE));
+ }
+
+ /**
+ * Get the offset of the <i>next record</i> that will be fetched (if a record with that offset exists).
+ * This method may issue a remote call to the server if there is no current position
+ * for the given partition.
+ * <p>
+ * This call will block until the position can be determined, an unrecoverable error is
+ * encountered (in which case it is thrown to the caller), or the timeout expires.
+ *
+ * @param partition The partition to get the position for
+ * @param timeout The maximum amount of time to await determination of the current position
+ * @return The current position of the consumer (that is, the offset of the next record to be fetched)
+ * @throws IllegalArgumentException if the provided TopicPartition is not assigned to this consumer
+ * @throws org.apache.kafka.clients.consumer.InvalidOffsetException if no offset is currently defined for
+ * the partition
+ * @throws org.apache.kafka.common.errors.WakeupException if {@link #wakeup()} is called before or while this
+ * function is called
+ * @throws org.apache.kafka.common.errors.InterruptException if the calling thread is interrupted before or while
+ * this function is called
+ * @throws org.apache.kafka.common.errors.TimeoutException if the position cannot be determined before the
+ * passed timeout expires
+ * @throws org.apache.kafka.common.errors.AuthenticationException if authentication fails. See the exception for more details
+ * @throws org.apache.kafka.common.errors.AuthorizationException if not authorized to the topic or to the
+ * configured groupId. See the exception for more details
+ * @throws org.apache.kafka.common.KafkaException for any other unrecoverable errors
+ */
+ @Override
+ public long position(TopicPartition partition, final Duration timeout) {
+ final long timeoutMs = timeout.toMillis();
acquireAndEnsureOpen();
try {
if (!this.subscriptions.isAssigned(partition))
- throw new IllegalArgumentException("You can only check the position for partitions assigned to this consumer.");
+ throw new IllegalStateException("You can only check the position for partitions assigned to this consumer.");
Long offset = this.subscriptions.position(partition);
- while (offset == null) {
+ final long startMs = time.milliseconds();
+ long finishMs = startMs;
+
+ while (offset == null && finishMs - startMs < timeoutMs) {
// batch update fetch positions for any partitions without a valid position
- updateFetchPositions();
- client.poll(retryBackoffMs);
+ if (!updateFetchPositions(remainingTimeAtLeastZero(timeoutMs, time.milliseconds() - startMs))) {
+ break;
+ }
+ finishMs = time.milliseconds();
+
+ client.poll(remainingTimeAtLeastZero(timeoutMs, finishMs - startMs));
offset = this.subscriptions.position(partition);
+ finishMs = time.milliseconds();
}
+ if (offset == null)
+ throw new TimeoutException("Timeout of " + timeout.toMillis() + "ms expired before the position " +
+ "for partition " + partition + " could be determined");
return offset;
} finally {
release();
@@ -1456,9 +1656,39 @@ public long position(TopicPartition partition) {
*/
@Override
public OffsetAndMetadata committed(TopicPartition partition) {
+ return committed(partition, Duration.ofMillis(Long.MAX_VALUE));
+ }
+
+ /**
+ * Get the last committed offset for the given partition (whether the commit happened by this process or
+ * another). This offset will be used as the position for the consumer in the event of a failure.
+ * <p>
+ * This call will block to do a remote call to get the latest committed offsets from the server.
+ *
+ * @param partition The partition to check
+ * @param timeout The maximum amount of time to await the current committed offset
+ * @return The last committed offset and metadata or null if there was no prior commit
+ * @throws org.apache.kafka.common.errors.WakeupException if {@link #wakeup()} is called before or while this
+ * function is called
+ * @throws org.apache.kafka.common.errors.InterruptException if the calling thread is interrupted before or while
+ * this function is called
+ * @throws org.apache.kafka.common.errors.AuthenticationException if authentication fails. See the exception for more details
+ * @throws org.apache.kafka.common.errors.AuthorizationException if not authorized to the topic or to the
+ * configured groupId. See the exception for more details
+ * @throws org.apache.kafka.common.KafkaException for any other unrecoverable errors
+ * @throws org.apache.kafka.common.errors.TimeoutException if the committed offset cannot be found before
+ * expiration of the timeout
+ */
+ @Override
+ public OffsetAndMetadata committed(TopicPartition partition, final Duration timeout) {
acquireAndEnsureOpen();
try {
- Map<TopicPartition, OffsetAndMetadata> offsets = coordinator.fetchCommittedOffsets(Collections.singleton(partition));
+ Map<TopicPartition, OffsetAndMetadata> offsets = coordinator.fetchCommittedOffsets(
+ Collections.singleton(partition), timeout.toMillis());
+ if (offsets == null) {
+ throw new TimeoutException("Timeout of " + timeout.toMillis() + "ms expired before the last " +
+ "committed offset for partition " + partition + " could be determined");
+ }
return offsets.get(partition);
} finally {
release();
@@ -1478,6 +1708,7 @@ public OffsetAndMetadata committed(TopicPartition partition) {
* does not already have any metadata about the given topic.
*
* @param topic The topic to get partition metadata for
+ *
* @return The list of partitions
* @throws org.apache.kafka.common.errors.WakeupException if {@link #wakeup()} is called before or while this
* function is called
@@ -1485,13 +1716,38 @@ public OffsetAndMetadata committed(TopicPartition partition) {
* this function is called
* @throws org.apache.kafka.common.errors.AuthenticationException if authentication fails. See the exception for more details
* @throws org.apache.kafka.common.errors.AuthorizationException if not authorized to the specified topic. See the exception for more details
- * @throws org.apache.kafka.common.errors.TimeoutException if the topic metadata could not be fetched before
- * expiration of the configured request timeout
* @throws org.apache.kafka.common.KafkaException for any other unrecoverable errors
+ * @throws org.apache.kafka.common.errors.TimeoutException if the offset metadata could not be fetched before
+ * the amount of time allocated by {@code request.timeout.ms} expires.
*/
@Override
public List<PartitionInfo> partitionsFor(String topic) {
+ return partitionsFor(topic, Duration.ofMillis(requestTimeoutMs));
+ }
+
+ /**
+ * Get metadata about the partitions for a given topic. This method will issue a remote call to the server if it
+ * does not already have any metadata about the given topic.
+ *
+ * @param topic The topic to get partition metadata for
+ * @param timeout The maximum of time to await topic metadata
+ *
+ * @return The list of partitions
+ * @throws org.apache.kafka.common.errors.WakeupException if {@link #wakeup()} is called before or while this
+ * function is called
+ * @throws org.apache.kafka.common.errors.InterruptException if the calling thread is interrupted before or while
+ * this function is called
+ * @throws org.apache.kafka.common.errors.AuthenticationException if authentication fails. See the exception for more details
+ * @throws org.apache.kafka.common.errors.AuthorizationException if not authorized to the specified topic. See
+ * the exception for more details
+ * @throws org.apache.kafka.common.errors.TimeoutException if topic metadata cannot be fetched before expiration
+ * of the passed timeout
+ * @throws org.apache.kafka.common.KafkaException for any other unrecoverable errors
+ */
+ @Override
+ public List<PartitionInfo> partitionsFor(String topic, Duration timeout) {
acquireAndEnsureOpen();
+ long timeoutMs = timeout.toMillis();
try {
Cluster cluster = this.metadata.fetch();
List<PartitionInfo> parts = cluster.partitionsForTopic(topic);
@@ -1499,7 +1755,7 @@ public OffsetAndMetadata committed(TopicPartition partition) {
return parts;
Map<String, List<PartitionInfo>> topicMetadata = fetcher.getTopicMetadata(
- new MetadataRequest.Builder(Collections.singletonList(topic), true), requestTimeoutMs);
+ new MetadataRequest.Builder(Collections.singletonList(topic), true), timeoutMs);
return topicMetadata.get(topic);
} finally {
release();
@@ -1511,31 +1767,52 @@ public OffsetAndMetadata committed(TopicPartition partition) {
* remote call to the server.
* @return The map of topics and its partitions
+ *
* @throws org.apache.kafka.common.errors.WakeupException if {@link #wakeup()} is called before or while this
* function is called
* @throws org.apache.kafka.common.errors.InterruptException if the calling thread is interrupted before or while
* this function is called
- * @throws org.apache.kafka.common.errors.TimeoutException if the topic metadata could not be fetched before
- * expiration of the configured request timeout
* @throws org.apache.kafka.common.KafkaException for any other unrecoverable errors
+ * @throws org.apache.kafka.common.errors.TimeoutException if the offset metadata could not be fetched before
+ * the amount of time allocated by {@code request.timeout.ms} expires.
*/
@Override
public Map<String, List<PartitionInfo>> listTopics() {
+ return listTopics(Duration.ofMillis(requestTimeoutMs));
+ }
+
+ /**
+ * Get metadata about partitions for all topics that the user is authorized to view. This method will issue a
+ * remote call to the server.
+ *
+ * @param timeout The maximum time this operation will block to fetch topic metadata
+ *
+ * @return The map of topics and its partitions
+ * @throws org.apache.kafka.common.errors.WakeupException if {@link #wakeup()} is called before or while this
+ * function is called
+ * @throws org.apache.kafka.common.errors.InterruptException if the calling thread is interrupted before or while
+ * this function is called
+ * @throws org.apache.kafka.common.errors.TimeoutException if the topic metadata could not be fetched before
+ * expiration of the passed timeout
+ * @throws org.apache.kafka.common.KafkaException for any other unrecoverable errors
+ */
+ @Override
+ public Map<String, List<PartitionInfo>> listTopics(Duration timeout) {
acquireAndEnsureOpen();
try {
- return fetcher.getAllTopicMetadata(requestTimeoutMs);
+ return fetcher.getAllTopicMetadata(timeout.toMillis());
} finally {
release();
}
}
/**
- * Suspend fetching from the requested partitions. Future calls to {@link #poll(long)} will not return
+ * Suspend fetching from the requested partitions. Future calls to {@link #poll(Duration)} will not return
* any records from these partitions until they have been resumed using {@link #resume(Collection)}.
* Note that this method does not affect partition subscription. In particular, it does not cause a group
* rebalance when automatic assignment is used.
* @param partitions The partitions which should be paused
- * @throws IllegalStateException if one of the provided partitions is not assigned to this consumer
+ * @throws IllegalStateException if any of the provided partitions are not currently assigned to this consumer
*/
@Override
public void pause(Collection<TopicPartition> partitions) {
@@ -1552,10 +1829,10 @@ public void pause(Collection<TopicPartition> partitions) {
/**
* Resume specified partitions which have been paused with {@link #pause(Collection)}. New calls to
- * {@link #poll(long)} will return records from these partitions if there are any to be fetched.
+ * {@link #poll(Duration)} will return records from these partitions if there are any to be fetched.
* If the partitions were not previously paused, this method is a no-op.
* @param partitions The partitions which should be resumed
- * @throws IllegalStateException if one of the provided partitions is not assigned to this consumer
+ * @throws IllegalStateException if any of the provided partitions are not currently assigned to this consumer
*/
@Override
public void resume(Collection<TopicPartition> partitions) {
@@ -1594,6 +1871,7 @@ public void resume(Collection<TopicPartition> partitions) {
* will be returned for that partition.
*
* @param timestampsToSearch the mapping from partition to the timestamp to look up.
+ *
* @return a mapping from partition to the timestamp and offset of the first message with timestamp greater
* than or equal to the target timestamp. {@code null} will be returned for the partition if there is no
* such message.
@@ -1601,12 +1879,39 @@ public void resume(Collection<TopicPartition> partitions) {
* @throws org.apache.kafka.common.errors.AuthorizationException if not authorized to the topic(s). See the exception for more details
* @throws IllegalArgumentException if the target timestamp is negative
* @throws org.apache.kafka.common.errors.TimeoutException if the offset metadata could not be fetched before
- * expiration of the configured {@code request.timeout.ms}
+ * the amount of time allocated by {@code request.timeout.ms} expires.
* @throws org.apache.kafka.common.errors.UnsupportedVersionException if the broker does not support looking up
* the offsets by timestamp
*/
@Override
public Map<TopicPartition, OffsetAndTimestamp> offsetsForTimes(Map<TopicPartition, Long> timestampsToSearch) {
+ return offsetsForTimes(timestampsToSearch, Duration.ofMillis(requestTimeoutMs));
+ }
+
+ /**
+ * Look up the offsets for the given partitions by timestamp. The returned offset for each partition is the
+ * earliest offset whose timestamp is greater than or equal to the given timestamp in the corresponding partition.
+ *
+ * This is a blocking call. The consumer does not have to be assigned the partitions.
+ * If the message format version in a partition is before 0.10.0, i.e. the messages do not have timestamps, null
+ * will be returned for that partition.
+ *
+ * @param timestampsToSearch the mapping from partition to the timestamp to look up.
+ * @param timeout The maximum amount of time to await retrieval of the offsets
+ *
+ * @return a mapping from partition to the timestamp and offset of the first message with timestamp greater
+ * than or equal to the target timestamp. {@code null} will be returned for the partition if there is no
+ * such message.
+ * @throws org.apache.kafka.common.errors.AuthenticationException if authentication fails. See the exception for more details
+ * @throws org.apache.kafka.common.errors.AuthorizationException if not authorized to the topic(s). See the exception for more details
+ * @throws IllegalArgumentException if the target timestamp is negative
+ * @throws org.apache.kafka.common.errors.TimeoutException if the offset metadata could not be fetched before
+ * expiration of the passed timeout
+ * @throws org.apache.kafka.common.errors.UnsupportedVersionException if the broker does not support looking up
+ * the offsets by timestamp
+ */
+ @Override
+ public Map<TopicPartition, OffsetAndTimestamp> offsetsForTimes(Map<TopicPartition, Long> timestampsToSearch, Duration timeout) {
acquireAndEnsureOpen();
try {
for (Map.Entry<TopicPartition, Long> entry : timestampsToSearch.entrySet()) {
@@ -1616,7 +1921,7 @@ public void resume(Collection<TopicPartition> partitions) {
throw new IllegalArgumentException("The target time for partition " + entry.getKey() + " is " +
entry.getValue() + ". The target time cannot be negative.");
}
- return fetcher.offsetsByTimes(timestampsToSearch, requestTimeoutMs);
+ return fetcher.offsetsByTimes(timestampsToSearch, timeout.toMillis());
} finally {
release();
}
@@ -1633,45 +1938,90 @@ public void resume(Collection<TopicPartition> partitions) {
* @return The earliest available offsets for the given partitions
* @throws org.apache.kafka.common.errors.AuthenticationException if authentication fails. See the exception for more details
* @throws org.apache.kafka.common.errors.AuthorizationException if not authorized to the topic(s). See the exception for more details
- * @throws org.apache.kafka.common.errors.TimeoutException if the offsets could not be fetched before
+ * @throws org.apache.kafka.common.errors.TimeoutException if the offset metadata could not be fetched before
* expiration of the configured {@code request.timeout.ms}
*/
@Override
public Map<TopicPartition, Long> beginningOffsets(Collection<TopicPartition> partitions) {
+ return beginningOffsets(partitions, Duration.ofMillis(requestTimeoutMs));
+ }
+
+ /**
+ * Get the first offset for the given partitions.
+ * <p>
+ * This method does not change the current consumer position of the partitions.
+ *
+ * @see #seekToBeginning(Collection)
+ *
+ * @param partitions the partitions to get the earliest offsets
+ * @param timeout The maximum amount of time to await retrieval of the beginning offsets
+ *
+ * @return The earliest available offsets for the given partitions
+ * @throws org.apache.kafka.common.errors.AuthenticationException if authentication fails. See the exception for more details
+ * @throws org.apache.kafka.common.errors.AuthorizationException if not authorized to the topic(s). See the exception for more details
+ * @throws org.apache.kafka.common.errors.TimeoutException if the offset metadata could not be fetched before
+ * expiration of the passed timeout
+ */
+ @Override
+ public Map<TopicPartition, Long> beginningOffsets(Collection<TopicPartition> partitions, Duration timeout) {
acquireAndEnsureOpen();
try {
- return fetcher.beginningOffsets(partitions, requestTimeoutMs);
+ return fetcher.beginningOffsets(partitions, timeout.toMillis());
} finally {
release();
}
}
/**
- * Get the last offset for the given partitions. The last offset of a partition is the offset of the upcoming
- * message, i.e. the offset of the last available message + 1. If messages have never been written
- * to the the partition, the offset returned will be 0.
+ * Get the end offsets for the given partitions. In the default {@code read_uncommitted} isolation level, the end
+ * offset is the high watermark (that is, the offset of the last successfully replicated message plus one). For
+ * {@code read_committed} consumers, the end offset is the last stable offset (LSO), which is the minimum of
+ * the high watermark and the smallest offset of any open transaction. Finally, if the partition has never been
+ * written to, the end offset is 0.
*
* <p>
* This method does not change the current consumer position of the partitions.
+ *
+ * @see #seekToEnd(Collection)
+ *
+ * @param partitions the partitions to get the end offsets.
+ * @return The end offsets for the given partitions.
+ * @throws org.apache.kafka.common.errors.AuthenticationException if authentication fails. See the exception for more details
+ * @throws org.apache.kafka.common.errors.AuthorizationException if not authorized to the topic(s). See the exception for more details
+ * @throws org.apache.kafka.common.errors.TimeoutException if the offset metadata could not be fetched before
+ * the amount of time allocated by {@code request.timeout.ms} expires
+ */
+ @Override
+ public Map<TopicPartition, Long> endOffsets(Collection<TopicPartition> partitions) {
+ return endOffsets(partitions, Duration.ofMillis(requestTimeoutMs));
+ }
+
+ /**
+ * Get the end offsets for the given partitions. In the default {@code read_uncommitted} isolation level, the end
+ * offset is the high watermark (that is, the offset of the last successfully replicated message plus one). For
+ * {@code read_committed} consumers, the end offset is the last stable offset (LSO), which is the minimum of
+ * the high watermark and the smallest offset of any open transaction. Finally, if the partition has never been
+ * written to, the end offset is 0.
+ *
* <p>
- * When {@code isolation.level=read_committed} the last offset will be the Last Stable Offset (LSO).
- * This is the offset of the first message with an open transaction. The LSO moves forward as transactions
- * are completed.
+ * This method does not change the current consumer position of the partitions.
*
* @see #seekToEnd(Collection)
*
* @param partitions the partitions to get the end offsets.
+ * @param timeout The maximum amount of time to await retrieval of the end offsets
+ *
* @return The end offsets for the given partitions.
* @throws org.apache.kafka.common.errors.AuthenticationException if authentication fails. See the exception for more details
* @throws org.apache.kafka.common.errors.AuthorizationException if not authorized to the topic(s). See the exception for more details
* @throws org.apache.kafka.common.errors.TimeoutException if the offsets could not be fetched before
- * expiration of the configured {@code request.timeout.ms}
+ * expiration of the passed timeout
*/
@Override
- public Map<TopicPartition, Long> endOffsets(Collection<TopicPartition> partitions) {
+ public Map<TopicPartition, Long> endOffsets(Collection<TopicPartition> partitions, Duration timeout) {
acquireAndEnsureOpen();
try {
- return fetcher.endOffsets(partitions, requestTimeoutMs);
+ return fetcher.endOffsets(partitions, timeout.toMillis());
} finally {
release();
}
@@ -1680,7 +2030,7 @@ public void resume(Collection<TopicPartition> partitions) {
/**
* Close the consumer, waiting for up to the default timeout of 30 seconds for any needed cleanup.
* If auto-commit is enabled, this will commit the current offsets if possible within the default
- * timeout. See {@link #close(long, TimeUnit)} for details. Note that {@link #wakeup()}
+ * timeout. See {@link #close(Duration)} for details. Note that {@link #wakeup()}
* cannot be used to interrupt close.
*
* @throws org.apache.kafka.common.errors.InterruptException if the calling thread is interrupted
@@ -1689,7 +2039,7 @@ public void resume(Collection<TopicPartition> partitions) {
*/
@Override
public void close() {
- close(DEFAULT_CLOSE_TIMEOUT_MS, TimeUnit.MILLISECONDS);
+ close(Duration.ofMillis(DEFAULT_CLOSE_TIMEOUT_MS));
}
/**
@@ -1706,15 +2056,39 @@ public void close() {
* @throws IllegalArgumentException If the {@code timeout} is negative.
* @throws InterruptException If the thread is interrupted before or while this function is called
* @throws org.apache.kafka.common.KafkaException for any other error during close
+ *
+ * @deprecated Since 2.0. Use {@link #close(Duration)} or {@link #close()}.
*/
+ @Deprecated
+ @Override
public void close(long timeout, TimeUnit timeUnit) {
- if (timeout < 0)
+ close(Duration.ofMillis(TimeUnit.MILLISECONDS.toMillis(timeout)));
+ }
+
+ /**
+ * Tries to close the consumer cleanly within the specified timeout. This method waits up to
+ * {@code timeout} for the consumer to complete pending commits and leave the group.
+ * If auto-commit is enabled, this will commit the current offsets if possible within the
+ * timeout. If the consumer is unable to complete offset commits and gracefully leave the group
+ * before the timeout expires, the consumer is force closed. Note that {@link #wakeup()} cannot be
+ * used to interrupt close.
+ *
+ * @param timeout The maximum time to wait for consumer to close gracefully. The value must be
+ * non-negative. Specifying a timeout of zero means do not wait for pending requests to complete.
+ *
+ * @throws IllegalArgumentException If the {@code timeout} is negative.
+ * @throws InterruptException If the thread is interrupted before or while this function is called
+ * @throws org.apache.kafka.common.KafkaException for any other error during close
+ */
+ @Override
+ public void close(Duration timeout) {
+ if (timeout.toMillis() < 0)
throw new IllegalArgumentException("The timeout cannot be negative.");
acquire();
try {
if (!closed) {
closed = true;
- close(timeUnit.toMillis(timeout), false);
+ close(timeout.toMillis(), false);
}
} finally {
release();
@@ -1775,18 +2149,18 @@ private void close(long timeoutMs, boolean swallowException) {
* @throws org.apache.kafka.common.errors.AuthenticationException if authentication fails. See the exception for more details
* @throws NoOffsetForPartitionException If no offset is stored for a given partition and no offset reset policy is
* defined
- * @return true if all assigned positions have a position, false otherwise
+ * @return true iff the operation completed without timing out
*/
- private boolean updateFetchPositions() {
- if (subscriptions.hasAllFetchPositions())
- return true;
+ private boolean updateFetchPositions(final long timeoutMs) {
+ cachedSubscriptionHashAllFetchPositions = subscriptions.hasAllFetchPositions();
+ if (cachedSubscriptionHashAllFetchPositions) return true;
// If there are any partitions which do not have a valid position and are not
// awaiting reset, then we need to fetch committed offsets. We will only do a
// coordinator lookup if there are partitions which have missing positions, so
// a consumer with manually assigned partitions can avoid a coordinator dependence
// by always ensuring that assigned partitions have an initial position.
- coordinator.refreshCommittedOffsetsIfNeeded();
+ if (!coordinator.refreshCommittedOffsetsIfNeeded(timeoutMs)) return false;
// If there are partitions still needing a position and a reset policy is defined,
// request reset using the default policy. If no reset strategy is defined and there
@@ -1797,7 +2171,7 @@ private boolean updateFetchPositions() {
// partitions which are awaiting reset.
fetcher.resetOffsetsIfNeeded();
- return false;
+ return true;
}
/**
@@ -1838,4 +2212,5 @@ private void throwIfNoAssignorsConfigured() {
throw new IllegalStateException("Must configure at least one partition assigner class name to " +
ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG + " configuration property");
}
+
}
diff --git a/clients/src/main/java/org/apache/kafka/clients/consumer/MockConsumer.java b/clients/src/main/java/org/apache/kafka/clients/consumer/MockConsumer.java
index ceb7024b97b..cf1b07fabe2 100644
--- a/clients/src/main/java/org/apache/kafka/clients/consumer/MockConsumer.java
+++ b/clients/src/main/java/org/apache/kafka/clients/consumer/MockConsumer.java
@@ -25,6 +25,7 @@
import org.apache.kafka.common.TopicPartition;
import org.apache.kafka.common.errors.WakeupException;
+import java.time.Duration;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
@@ -43,7 +44,7 @@
/**
* A mock of the {@link Consumer} interface you can use for testing code that uses Kafka. This class is <i> not
* threadsafe </i>. However, you can use the {@link #schedulePollTask(Runnable)} method to write multithreaded tests
- * where a driver thread waits for {@link #poll(long)} to be called by a background thread and then can safely perform
+ * where a driver thread waits for {@link #poll(Duration)} to be called by a background thread and then can safely perform
* operations during a callback.
*/
public class MockConsumer<K, V> implements Consumer<K, V> {
@@ -146,8 +147,14 @@ public synchronized void unsubscribe() {
subscriptions.unsubscribe();
}
+ @Deprecated
@Override
public synchronized ConsumerRecords<K, V> poll(long timeout) {
+ return poll(Duration.ZERO);
+ }
+
+ @Override
+ public synchronized ConsumerRecords<K, V> poll(final Duration timeout) {
ensureNotClosed();
// Synchronize around the entire execution so new tasks to be triggered on subsequent poll calls can be added in
@@ -244,6 +251,16 @@ public synchronized void commitSync() {
commitSync(this.subscriptions.allConsumed());
}
+ @Override
+ public synchronized void commitSync(Duration timeout) {
+ commitSync(this.subscriptions.allConsumed());
+ }
+
+ @Override
+ public void commitSync(Map<TopicPartition, OffsetAndMetadata> offsets, final Duration timeout) {
+ commitSync(offsets);
+ }
+
@Override
public synchronized void seek(TopicPartition partition, long offset) {
ensureNotClosed();
@@ -259,6 +276,11 @@ public synchronized OffsetAndMetadata committed(TopicPartition partition) {
return new OffsetAndMetadata(0);
}
+ @Override
+ public OffsetAndMetadata committed(TopicPartition partition, final Duration timeout) {
+ return committed(partition);
+ }
+
@Override
public synchronized long position(TopicPartition partition) {
ensureNotClosed();
@@ -272,6 +294,11 @@ public synchronized long position(TopicPartition partition) {
return offset;
}
+ @Override
+ public synchronized long position(TopicPartition partition, final Duration timeout) {
+ return position(partition);
+ }
+
@Override
public synchronized void seekToBeginning(Collection<TopicPartition> partitions) {
ensureNotClosed();
@@ -401,7 +428,7 @@ public synchronized void wakeup() {
}
/**
- * Schedule a task to be executed during a poll(). One enqueued task will be executed per {@link #poll(long)}
+ * Schedule a task to be executed during a poll(). One enqueued task will be executed per {@link #poll(Duration)}
* invocation. You can use this repeatedly to mock out multiple responses to poll invocations.
* @param task the task to be executed
*/
@@ -463,4 +490,35 @@ private Long getEndOffset(List<Long> offsets) {
}
return offsets.size() > 1 ? offsets.remove(0) : offsets.get(0);
}
+
+ @Override
+ public List<PartitionInfo> partitionsFor(String topic, Duration timeout) {
+ return partitionsFor(topic);
+ }
+
+ @Override
+ public Map<String, List<PartitionInfo>> listTopics(Duration timeout) {
+ return listTopics();
+ }
+
+ @Override
+ public Map<TopicPartition, OffsetAndTimestamp> offsetsForTimes(Map<TopicPartition, Long> timestampsToSearch,
+ Duration timeout) {
+ return offsetsForTimes(timestampsToSearch);
+ }
+
+ @Override
+ public Map<TopicPartition, Long> beginningOffsets(Collection<TopicPartition> partitions, Duration timeout) {
+ return beginningOffsets(partitions);
+ }
+
+ @Override
+ public Map<TopicPartition, Long> endOffsets(Collection<TopicPartition> partitions, Duration timeout) {
+ return endOffsets(partitions);
+ }
+
+ @Override
+ public void close(Duration timeout) {
+ close();
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/clients/consumer/OffsetCommitCallback.java b/clients/src/main/java/org/apache/kafka/clients/consumer/OffsetCommitCallback.java
index b217a632574..383c1c82bce 100644
--- a/clients/src/main/java/org/apache/kafka/clients/consumer/OffsetCommitCallback.java
+++ b/clients/src/main/java/org/apache/kafka/clients/consumer/OffsetCommitCallback.java
@@ -23,7 +23,7 @@
/**
* A callback interface that the user can implement to trigger custom actions when a commit request completes. The callback
- * may be executed in any thread calling {@link Consumer#poll(long) poll()}.
+ * may be executed in any thread calling {@link Consumer#poll(java.time.Duration) poll()}.
*/
public interface OffsetCommitCallback {
diff --git a/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractCoordinator.java b/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractCoordinator.java
index c3edaa77534..adbaae776ab 100644
--- a/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractCoordinator.java
+++ b/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractCoordinator.java
@@ -58,6 +58,7 @@
import java.util.Collections;
import java.util.List;
import java.util.Map;
+import java.util.Objects;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicReference;
@@ -198,46 +199,44 @@ protected abstract void onJoinComplete(int generation,
ByteBuffer memberAssignment);
/**
- * Block until the coordinator for this group is known and is ready to receive requests.
- */
- public synchronized void ensureCoordinatorReady() {
- // Using zero as current time since timeout is effectively infinite
- ensureCoordinatorReady(0, Long.MAX_VALUE);
- }
-
- /**
+ * Visible for testing.
+ *
* Ensure that the coordinator is ready to receive requests.
- * @param startTimeMs Current time in milliseconds
+ *
* @param timeoutMs Maximum time to wait to discover the coordinator
* @return true If coordinator discovery and initial connection succeeded, false otherwise
*/
- protected synchronized boolean ensureCoordinatorReady(long startTimeMs, long timeoutMs) {
- long remainingMs = timeoutMs;
+ protected synchronized boolean ensureCoordinatorReady(final long timeoutMs) {
+ final long startTimeMs = time.milliseconds();
+ long elapsedTime = 0L;
while (coordinatorUnknown()) {
- RequestFuture<Void> future = lookupCoordinator();
- client.poll(future, remainingMs);
+ final RequestFuture<Void> future = lookupCoordinator();
+ client.poll(future, remainingTimeAtLeastZero(timeoutMs, elapsedTime));
+ if (!future.isDone()) {
+ // ran out of time
+ break;
+ }
if (future.failed()) {
if (future.isRetriable()) {
- remainingMs = timeoutMs - (time.milliseconds() - startTimeMs);
- if (remainingMs <= 0)
- break;
+ elapsedTime = time.milliseconds() - startTimeMs;
+
+ if (elapsedTime >= timeoutMs) break;
log.debug("Coordinator discovery failed, refreshing metadata");
- client.awaitMetadataUpdate(remainingMs);
+ client.awaitMetadataUpdate(remainingTimeAtLeastZero(timeoutMs, elapsedTime));
+ elapsedTime = time.milliseconds() - startTimeMs;
} else
throw future.exception();
- } else if (coordinator != null && client.connectionFailed(coordinator)) {
+ } else if (coordinator != null && client.isUnavailable(coordinator)) {
// we found the coordinator, but the connection has failed, so mark
// it dead and backoff before retrying discovery
markCoordinatorUnknown();
- time.sleep(retryBackoffMs);
+ final long sleepTime = Math.min(retryBackoffMs, remainingTimeAtLeastZero(timeoutMs, elapsedTime));
+ time.sleep(sleepTime);
+ elapsedTime += sleepTime;
}
-
- remainingMs = timeoutMs - (time.milliseconds() - startTimeMs);
- if (remainingMs <= 0)
- break;
}
return !coordinatorUnknown();
@@ -261,15 +260,14 @@ private synchronized void clearFindCoordinatorFuture() {
}
/**
- * Check whether the group should be rejoined (e.g. if metadata changes)
+ * Check whether the group should be rejoined (e.g. if metadata changes) or whether a
+ * rejoin request is already in flight and needs to be completed.
+ *
* @return true if it should, false otherwise
*/
- protected synchronized boolean needRejoin() {
- return rejoinNeeded;
- }
-
- private synchronized boolean rejoinIncomplete() {
- return joinFuture != null;
+ protected synchronized boolean rejoinNeededOrPending() {
+ // if there's a pending joinFuture, we should try to complete handling it.
+ return rejoinNeeded || joinFuture != null;
}
/**
@@ -309,11 +307,28 @@ protected synchronized long timeToNextHeartbeat(long now) {
* Ensure that the group is active (i.e. joined and synced)
*/
public void ensureActiveGroup() {
+ while (!ensureActiveGroup(Long.MAX_VALUE)) {
+ log.warn("still waiting to ensure active group");
+ }
+ }
+
+ /**
+ * Ensure the group is active (i.e., joined and synced)
+ *
+ * @param timeoutMs A time budget for ensuring the group is active
+ * @return true iff the group is active
+ */
+ boolean ensureActiveGroup(final long timeoutMs) {
+ final long startTime = time.milliseconds();
// always ensure that the coordinator is ready because we may have been disconnected
// when sending heartbeats and does not necessarily require us to rejoin the group.
- ensureCoordinatorReady();
+ if (!ensureCoordinatorReady(timeoutMs)) {
+ return false;
+ }
+
startHeartbeatThreadIfNeeded();
- joinGroupIfNeeded();
+
+ return joinGroupIfNeeded(remainingTimeAtLeastZero(timeoutMs, time.milliseconds() - startTime));
}
private synchronized void startHeartbeatThreadIfNeeded() {
@@ -345,10 +360,23 @@ private void closeHeartbeatThread() {
}
}
- // visible for testing. Joins the group without starting the heartbeat thread.
- void joinGroupIfNeeded() {
- while (needRejoin() || rejoinIncomplete()) {
- ensureCoordinatorReady();
+ /**
+ * Joins the group without starting the heartbeat thread.
+ *
+ * Visible for testing.
+ *
+ * @param timeoutMs Time to complete this action
+ * @return true iff the operation succeeded
+ */
+ boolean joinGroupIfNeeded(final long timeoutMs) {
+ final long startTime = time.milliseconds();
+ long elapsedTime = 0L;
+
+ while (rejoinNeededOrPending()) {
+ if (!ensureCoordinatorReady(remainingTimeAtLeastZero(timeoutMs, elapsedTime))) {
+ return false;
+ }
+ elapsedTime = time.milliseconds() - startTime;
// call onJoinPrepare if needed. We set a flag to make sure that we do not call it a second
// time if the client is woken up before a pending rebalance completes. This must be called
@@ -360,8 +388,12 @@ void joinGroupIfNeeded() {
needsJoinPrepare = false;
}
- RequestFuture<ByteBuffer> future = initiateJoinGroup();
- client.poll(future);
+ final RequestFuture<ByteBuffer> future = initiateJoinGroup();
+ client.poll(future, remainingTimeAtLeastZero(timeoutMs, elapsedTime));
+ if (!future.isDone()) {
+ // we ran out of time
+ return false;
+ }
if (future.succeeded()) {
onJoinComplete(generation.generationId, generation.memberId, generation.protocol, future.value());
@@ -372,7 +404,7 @@ void joinGroupIfNeeded() {
needsJoinPrepare = true;
} else {
resetJoinGroupFuture();
- RuntimeException exception = future.exception();
+ final RuntimeException exception = future.exception();
if (exception instanceof UnknownMemberIdException ||
exception instanceof RebalanceInProgressException ||
exception instanceof IllegalGenerationException)
@@ -381,7 +413,16 @@ else if (!future.isRetriable())
throw exception;
time.sleep(retryBackoffMs);
}
+
+ if (rejoinNeededOrPending()) {
+ elapsedTime = time.milliseconds() - startTime;
+ }
}
+ return true;
+ }
+
+ private long remainingTimeAtLeastZero(final long timeout, final long elapsedTime) {
+ return Math.max(0, timeout - elapsedTime);
}
private synchronized void resetJoinGroupFuture() {
@@ -637,7 +678,7 @@ public boolean coordinatorUnknown() {
* @return the current coordinator or null if it is unknown
*/
protected synchronized Node checkAndGetCoordinator() {
- if (coordinator != null && client.connectionFailed(coordinator)) {
+ if (coordinator != null && client.isUnavailable(coordinator)) {
markCoordinatorUnknown(true);
return null;
}
@@ -1035,6 +1076,21 @@ public Generation(int generationId, String memberId, String protocol) {
this.memberId = memberId;
this.protocol = protocol;
}
+
+ @Override
+ public boolean equals(final Object o) {
+ if (this == o) return true;
+ if (o == null || getClass() != o.getClass()) return false;
+ final Generation that = (Generation) o;
+ return generationId == that.generationId &&
+ Objects.equals(memberId, that.memberId) &&
+ Objects.equals(protocol, that.protocol);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(generationId, memberId, protocol);
+ }
}
private static class UnjoinedGroupException extends RetriableException {
diff --git a/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java b/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java
index 2afa1ff9236..9c19af17037 100644
--- a/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java
+++ b/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java
@@ -31,6 +31,7 @@
import org.apache.kafka.common.errors.GroupAuthorizationException;
import org.apache.kafka.common.errors.InterruptException;
import org.apache.kafka.common.errors.RetriableException;
+import org.apache.kafka.common.errors.TimeoutException;
import org.apache.kafka.common.errors.TopicAuthorizationException;
import org.apache.kafka.common.errors.WakeupException;
import org.apache.kafka.common.metrics.Measurable;
@@ -56,6 +57,7 @@
import java.util.HashSet;
import java.util.List;
import java.util.Map;
+import java.util.Objects;
import java.util.Set;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.atomic.AtomicInteger;
@@ -86,6 +88,28 @@
private MetadataSnapshot assignmentSnapshot;
private long nextAutoCommitDeadline;
+ // hold onto request&future for committed offset requests to enable async calls.
+ private PendingCommittedOffsetRequest pendingCommittedOffsetRequest = null;
+
+ private static class PendingCommittedOffsetRequest {
+ private final Set<TopicPartition> requestedPartitions;
+ private final Generation requestedGeneration;
+ private final RequestFuture<Map<TopicPartition, OffsetAndMetadata>> response;
+
+ private PendingCommittedOffsetRequest(final Set<TopicPartition> requestedPartitions,
+ final Generation generationAtRequestTime,
+ final RequestFuture<Map<TopicPartition, OffsetAndMetadata>> response) {
+ this.requestedPartitions = Objects.requireNonNull(requestedPartitions);
+ this.response = Objects.requireNonNull(response);
+ this.requestedGeneration = generationAtRequestTime;
+ }
+
+ private boolean sameRequest(final Set<TopicPartition> currentRequest, final Generation currentGeneration) {
+ return (requestedGeneration == null ? currentGeneration == null : requestedGeneration.equals(currentGeneration))
+ && requestedPartitions.equals(currentRequest);
+ }
+ }
+
/**
* Initialize the coordination manager.
*/
@@ -243,7 +267,7 @@ protected void onJoinComplete(int generation,
// update the metadata and enforce a refresh to make sure the fetcher can start
// fetching data in the next iteration
this.metadata.setTopics(subscriptions.groupSubscription());
- client.ensureFreshMetadata();
+ if (!client.ensureFreshMetadata(Long.MAX_VALUE)) throw new TimeoutException();
// give the assignor a chance to update internal state based on the received assignment
assignor.onAssignment(assignment);
@@ -268,30 +292,49 @@ protected void onJoinComplete(int generation,
* Poll for coordinator events. This ensures that the coordinator is known and that the consumer
* has joined the group (if it is using group management). This also handles periodic offset commits
* if they are enabled.
+ * <p>
+ * Returns early if the timeout expires
*
- * @param now current time in milliseconds
+ * @param timeoutMs The amount of time, in ms, allotted for this operation.
+ * @return true iff the operation succeeded
*/
- public void poll(long now, long remainingMs) {
+ public boolean poll(final long timeoutMs) {
+ final long startTime = time.milliseconds();
+ long currentTime = startTime;
+ long elapsed = 0L;
+
invokeCompletedOffsetCommitCallbacks();
if (subscriptions.partitionsAutoAssigned()) {
if (coordinatorUnknown()) {
- ensureCoordinatorReady();
- now = time.milliseconds();
+ if (!ensureCoordinatorReady(remainingTimeAtLeastZero(timeoutMs, elapsed))) {
+ return false;
+ }
+ currentTime = time.milliseconds();
+ elapsed = currentTime - startTime;
+
}
- if (needRejoin()) {
+ if (rejoinNeededOrPending()) {
// due to a race condition between the initial metadata fetch and the initial rebalance,
// we need to ensure that the metadata is fresh before joining initially. This ensures
// that we have matched the pattern against the cluster's topics at least once before joining.
- if (subscriptions.hasPatternSubscription())
- client.ensureFreshMetadata();
+ if (subscriptions.hasPatternSubscription()) {
+ if (!client.ensureFreshMetadata(remainingTimeAtLeastZero(timeoutMs, elapsed))) {
+ return false;
+ }
+ currentTime = time.milliseconds();
+ elapsed = currentTime - startTime;
+ }
- ensureActiveGroup();
- now = time.milliseconds();
+ if (!ensureActiveGroup(remainingTimeAtLeastZero(timeoutMs, elapsed))) {
+ return false;
+ }
+
+ currentTime = time.milliseconds();
}
- pollHeartbeat(now);
+ pollHeartbeat(currentTime);
} else {
// For manually assigned partitions, if there are no ready nodes, await metadata.
// If connections to all nodes fail, wakeups triggered while attempting to send fetch
@@ -300,19 +343,26 @@ public void poll(long now, long remainingMs) {
// awaitMetadataUpdate() initiates new connections with configured backoff and avoids the busy loop.
// When group management is used, metadata wait is already performed for this scenario as
// coordinator is unknown, hence this check is not required.
- if (metadata.updateRequested() && !client.hasReadyNodes()) {
- boolean metadataUpdated = client.awaitMetadataUpdate(remainingMs);
- if (!metadataUpdated && !client.hasReadyNodes())
- return;
- now = time.milliseconds();
+ if (metadata.updateRequested() && !client.hasReadyNodes(startTime)) {
+ final boolean metadataUpdated = client.awaitMetadataUpdate(remainingTimeAtLeastZero(timeoutMs, elapsed));
+ if (!metadataUpdated && !client.hasReadyNodes(time.milliseconds())) {
+ return false;
+ }
+
+ currentTime = time.milliseconds();
}
}
- maybeAutoCommitOffsetsAsync(now);
+ maybeAutoCommitOffsetsAsync(currentTime);
+ return true;
+ }
+
+ private long remainingTimeAtLeastZero(final long timeoutMs, final long elapsed) {
+ return Math.max(0, timeoutMs - elapsed);
}
/**
- * Return the time to the next needed invocation of {@link #poll(long, long)}.
+ * Return the time to the next needed invocation of {@link #poll(long)}.
* @param now current time in milliseconds
* @return the maximum time in milliseconds the caller should wait before the next invocation of poll()
*/
@@ -349,7 +399,7 @@ public long timeToNextPoll(long now) {
// update metadata (if needed) and keep track of the metadata used for assignment so that
// we can check after rebalance completion whether anything has changed
- client.ensureFreshMetadata();
+ if (!client.ensureFreshMetadata(Long.MAX_VALUE)) throw new TimeoutException();
isLeader = true;
@@ -385,7 +435,7 @@ public long timeToNextPoll(long now) {
allSubscribedTopics.addAll(assignedTopics);
this.subscriptions.groupSubscribe(allSubscribedTopics);
metadata.setTopics(this.subscriptions.groupSubscription());
- client.ensureFreshMetadata();
+ if (!client.ensureFreshMetadata(Long.MAX_VALUE)) throw new TimeoutException();
}
assignmentSnapshot = metadataSnapshot;
@@ -423,7 +473,7 @@ protected void onJoinPrepare(int generation, String memberId) {
}
@Override
- public boolean needRejoin() {
+ public boolean rejoinNeededOrPending() {
if (!subscriptions.partitionsAutoAssigned())
return false;
@@ -435,60 +485,94 @@ public boolean needRejoin() {
if (joinedSubscription != null && !joinedSubscription.equals(subscriptions.subscription()))
return true;
- return super.needRejoin();
+ return super.rejoinNeededOrPending();
}
/**
* Refresh the committed offsets for provided partitions.
+ *
+ * @param timeoutMs A time limit for this operation
+ * @return true iff the operation completed within the timeout
*/
- public void refreshCommittedOffsetsIfNeeded() {
- Set<TopicPartition> missingFetchPositions = subscriptions.missingFetchPositions();
- Map<TopicPartition, OffsetAndMetadata> offsets = fetchCommittedOffsets(missingFetchPositions);
- for (Map.Entry<TopicPartition, OffsetAndMetadata> entry : offsets.entrySet()) {
- TopicPartition tp = entry.getKey();
- long offset = entry.getValue().offset();
+ public boolean refreshCommittedOffsetsIfNeeded(final long timeoutMs) {
+ final Set<TopicPartition> missingFetchPositions = subscriptions.missingFetchPositions();
+
+ final Map<TopicPartition, OffsetAndMetadata> offsets = fetchCommittedOffsets(missingFetchPositions, timeoutMs);
+ if (offsets == null) return false;
+
+ for (final Map.Entry<TopicPartition, OffsetAndMetadata> entry : offsets.entrySet()) {
+ final TopicPartition tp = entry.getKey();
+ final long offset = entry.getValue().offset();
log.debug("Setting offset for partition {} to the committed offset {}", tp, offset);
this.subscriptions.seek(tp, offset);
}
+ return true;
}
/**
* Fetch the current committed offsets from the coordinator for a set of partitions.
+ *
* @param partitions The partitions to fetch offsets for
- * @return A map from partition to the committed offset
+ * @return A map from partition to the committed offset or null if the operation timed out
*/
- public Map<TopicPartition, OffsetAndMetadata> fetchCommittedOffsets(Set<TopicPartition> partitions) {
- if (partitions.isEmpty())
- return Collections.emptyMap();
+ public Map<TopicPartition, OffsetAndMetadata> fetchCommittedOffsets(final Set<TopicPartition> partitions,
+ final long timeoutMs) {
+ if (partitions.isEmpty()) return Collections.emptyMap();
+
+ final Generation generation = generation();
+ if (pendingCommittedOffsetRequest != null && !pendingCommittedOffsetRequest.sameRequest(partitions, generation)) {
+ // if we were waiting for a different request, then just clear it.
+ pendingCommittedOffsetRequest = null;
+ }
+
+ final long startMs = time.milliseconds();
+ long elapsedTime = 0L;
while (true) {
- ensureCoordinatorReady();
+ if (!ensureCoordinatorReady(remainingTimeAtLeastZero(timeoutMs, elapsedTime))) return null;
+ elapsedTime = time.milliseconds() - startMs;
// contact coordinator to fetch committed offsets
- RequestFuture<Map<TopicPartition, OffsetAndMetadata>> future = sendOffsetFetchRequest(partitions);
- client.poll(future);
+ final RequestFuture<Map<TopicPartition, OffsetAndMetadata>> future;
+ if (pendingCommittedOffsetRequest != null) {
+ future = pendingCommittedOffsetRequest.response;
+ } else {
+ future = sendOffsetFetchRequest(partitions);
+ pendingCommittedOffsetRequest = new PendingCommittedOffsetRequest(partitions, generation, future);
- if (future.succeeded())
- return future.value();
+ }
+ client.poll(future, remainingTimeAtLeastZero(timeoutMs, elapsedTime));
- if (!future.isRetriable())
- throw future.exception();
+ if (future.isDone()) {
+ pendingCommittedOffsetRequest = null;
- time.sleep(retryBackoffMs);
+ if (future.succeeded()) {
+ return future.value();
+ } else if (!future.isRetriable()) {
+ throw future.exception();
+ } else {
+ elapsedTime = time.milliseconds() - startMs;
+ final long sleepTime = Math.min(retryBackoffMs, remainingTimeAtLeastZero(startMs, elapsedTime));
+ time.sleep(sleepTime);
+ elapsedTime += sleepTime;
+ }
+ } else {
+ return null;
+ }
}
}
- public void close(long timeoutMs) {
+ public void close(final long timeoutMs) {
// we do not need to re-enable wakeups since we are closing already
client.disableWakeups();
long now = time.milliseconds();
- long endTimeMs = now + timeoutMs;
+ final long endTimeMs = now + timeoutMs;
try {
maybeAutoCommitOffsetsSync(timeoutMs);
now = time.milliseconds();
if (pendingAsyncCommits.get() > 0 && endTimeMs > now) {
- ensureCoordinatorReady(now, endTimeMs - now);
+ ensureCoordinatorReady(endTimeMs - now);
now = time.milliseconds();
}
} finally {
@@ -587,7 +671,7 @@ public boolean commitOffsetsSync(Map<TopicPartition, OffsetAndMetadata> offsets,
long remainingMs = timeoutMs;
do {
if (coordinatorUnknown()) {
- if (!ensureCoordinatorReady(now, remainingMs))
+ if (!ensureCoordinatorReady(remainingMs))
return false;
remainingMs = timeoutMs - (time.milliseconds() - startMs);
@@ -621,6 +705,7 @@ public boolean commitOffsetsSync(Map<TopicPartition, OffsetAndMetadata> offsets,
public void maybeAutoCommitOffsetsAsync(long now) {
if (autoCommitEnabled && now >= nextAutoCommitDeadline) {
+ this.nextAutoCommitDeadline = now + autoCommitIntervalMs;
doAutoCommitOffsetsAsync();
}
}
@@ -633,14 +718,15 @@ private void doAutoCommitOffsetsAsync() {
@Override
public void onComplete(Map<TopicPartition, OffsetAndMetadata> offsets, Exception exception) {
if (exception != null) {
- log.warn("Asynchronous auto-commit of offsets {} failed: {}", offsets, exception.getMessage());
- if (exception instanceof RetriableException)
+ if (exception instanceof RetriableException) {
+ log.debug("Asynchronous auto-commit of offsets {} failed due to retriable error: {}", offsets,
+ exception);
nextAutoCommitDeadline = Math.min(time.milliseconds() + retryBackoffMs, nextAutoCommitDeadline);
- else
- nextAutoCommitDeadline = time.milliseconds() + autoCommitIntervalMs;
+ } else {
+ log.warn("Asynchronous auto-commit of offsets {} failed: {}", offsets, exception.getMessage());
+ }
} else {
log.debug("Completed asynchronous auto-commit of offsets {}", offsets);
- nextAutoCommitDeadline = time.milliseconds() + autoCommitIntervalMs;
}
}
});
@@ -755,7 +841,8 @@ public void handle(OffsetCommitResponse commitResponse, RequestFuture<Void> futu
// raise the error to the user
future.raise(error);
return;
- } else if (error == Errors.COORDINATOR_LOAD_IN_PROGRESS) {
+ } else if (error == Errors.COORDINATOR_LOAD_IN_PROGRESS
+ || error == Errors.UNKNOWN_TOPIC_OR_PARTITION) {
// just retry
future.raise(error);
return;
@@ -772,9 +859,6 @@ public void handle(OffsetCommitResponse commitResponse, RequestFuture<Void> futu
resetGeneration();
future.raise(new CommitFailedException());
return;
- } else if (error == Errors.UNKNOWN_TOPIC_OR_PARTITION) {
- future.raise(new KafkaException("Topic or Partition " + tp + " does not exist"));
- return;
} else {
future.raise(new KafkaException("Unexpected error in commit: " + error.message()));
return;
diff --git a/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java b/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
index fb393a54c7e..b718d63c542 100644
--- a/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
+++ b/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
@@ -28,7 +28,6 @@
import org.apache.kafka.common.errors.TimeoutException;
import org.apache.kafka.common.errors.WakeupException;
import org.apache.kafka.common.requests.AbstractRequest;
-import org.apache.kafka.common.requests.RequestHeader;
import org.apache.kafka.common.utils.LogContext;
import org.apache.kafka.common.utils.Time;
import org.slf4j.Logger;
@@ -128,10 +127,10 @@ public Node leastLoadedNode() {
}
}
- public boolean hasReadyNodes() {
+ public boolean hasReadyNodes(long now) {
lock.lock();
try {
- return client.hasReadyNodes();
+ return client.hasReadyNodes(now);
} finally {
lock.unlock();
}
@@ -165,9 +164,13 @@ public boolean awaitMetadataUpdate(long timeout) {
* Ensure our metadata is fresh (if an update is expected, this will block
* until it has completed).
*/
- public void ensureFreshMetadata() {
- if (this.metadata.updateRequested() || this.metadata.timeToNextUpdate(time.milliseconds()) == 0)
- awaitMetadataUpdate();
+ boolean ensureFreshMetadata(final long timeout) {
+ if (this.metadata.updateRequested() || this.metadata.timeToNextUpdate(time.milliseconds()) == 0) {
+ return awaitMetadataUpdate(timeout);
+ } else {
+ // the metadata is already fresh
+ return true;
+ }
}
/**
@@ -249,7 +252,8 @@ public void poll(long timeout, long now, PollCondition pollCondition, boolean di
handlePendingDisconnects();
// send all the requests we can send now
- trySend(now);
+ long pollDelayMs = trySend(now);
+ timeout = Math.min(timeout, pollDelayMs);
// check whether the poll is still needed by the caller. Note that if the expected completion
// condition becomes satisfied after the call to shouldBlock() (because of a fired completion
@@ -464,22 +468,24 @@ private void failUnsentRequests(Node node, RuntimeException e) {
}
}
- private boolean trySend(long now) {
- // send any requests that can be sent now
- boolean requestsSent = false;
+ private long trySend(long now) {
+ long pollDelayMs = Long.MAX_VALUE;
+ // send any requests that can be sent now
for (Node node : unsent.nodes()) {
Iterator<ClientRequest> iterator = unsent.requestIterator(node);
+ if (iterator.hasNext())
+ pollDelayMs = Math.min(pollDelayMs, client.pollDelayMs(node, now));
+
while (iterator.hasNext()) {
ClientRequest request = iterator.next();
if (client.ready(node, now)) {
client.send(request, now);
iterator.remove();
- requestsSent = true;
}
}
}
- return requestsSent;
+ return pollDelayMs;
}
public void maybeTriggerWakeup() {
@@ -510,15 +516,29 @@ public void close() throws IOException {
}
}
+
+ /**
+ * Check if the code is disconnected and unavailable for immediate reconnection (i.e. if it is in
+ * reconnect backoff window following the disconnect).
+ */
+ public boolean isUnavailable(Node node) {
+ lock.lock();
+ try {
+ return client.connectionFailed(node) && client.connectionDelay(node, time.milliseconds()) > 0;
+ } finally {
+ lock.unlock();
+ }
+ }
+
/**
- * Find whether a previous connection has failed. Note that the failure state will persist until either
- * {@link #tryConnect(Node)} or {@link #send(Node, AbstractRequest.Builder)} has been called.
- * @param node Node to connect to if possible
+ * Check for an authentication error on a given node and raise the exception if there is one.
*/
- public boolean connectionFailed(Node node) {
+ public void maybeThrowAuthFailure(Node node) {
lock.lock();
try {
- return client.connectionFailed(node);
+ AuthenticationException exception = client.authenticationException(node);
+ if (exception != null)
+ throw exception;
} finally {
lock.unlock();
}
@@ -552,10 +572,8 @@ public void fireCompletion() {
if (e != null) {
future.raise(e);
} else if (response.wasDisconnected()) {
- RequestHeader requestHeader = response.requestHeader();
- int correlation = requestHeader.correlationId();
- log.debug("Cancelled {} request {} with correlation id {} due to node {} being disconnected",
- requestHeader.apiKey(), requestHeader, correlation, response.destination());
+ log.debug("Cancelled request with header {} due to node {} being disconnected",
+ response.requestHeader(), response.destination());
future.raise(DisconnectException.INSTANCE);
} else if (response.versionMismatch() != null) {
future.raise(response.versionMismatch());
diff --git a/clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java b/clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java
index b46a3a29380..ca8e0d26c81 100644
--- a/clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java
+++ b/clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java
@@ -19,6 +19,7 @@
import org.apache.kafka.clients.ClientResponse;
import org.apache.kafka.clients.FetchSessionHandler;
import org.apache.kafka.clients.Metadata;
+import org.apache.kafka.clients.StaleMetadataException;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.OffsetAndTimestamp;
@@ -51,6 +52,7 @@
import org.apache.kafka.common.record.InvalidRecordException;
import org.apache.kafka.common.record.Record;
import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.Records;
import org.apache.kafka.common.record.TimestampType;
import org.apache.kafka.common.requests.FetchRequest;
import org.apache.kafka.common.requests.FetchResponse;
@@ -203,7 +205,7 @@ public int sendFetches() {
.addListener(new RequestFutureListener<ClientResponse>() {
@Override
public void onSuccess(ClientResponse resp) {
- FetchResponse response = (FetchResponse) resp.responseBody();
+ FetchResponse<Records> response = (FetchResponse<Records>) resp.responseBody();
FetchSessionHandler handler = sessionHandlers.get(fetchTarget.id());
if (handler == null) {
log.error("Unable to find FetchSessionHandler for node {}. Ignoring fetch response.",
@@ -217,7 +219,7 @@ public void onSuccess(ClientResponse resp) {
Set<TopicPartition> partitions = new HashSet<>(response.responseData().keySet());
FetchResponseMetricAggregator metricAggregator = new FetchResponseMetricAggregator(sensors, partitions);
- for (Map.Entry<TopicPartition, FetchResponse.PartitionData> entry : response.responseData().entrySet()) {
+ for (Map.Entry<TopicPartition, FetchResponse.PartitionData<Records>> entry : response.responseData().entrySet()) {
TopicPartition partition = entry.getKey();
long fetchOffset = data.sessionPartitions().get(partition).fetchOffset;
FetchResponse.PartitionData fetchData = entry.getValue();
@@ -472,6 +474,7 @@ private ListOffsetResult fetchOffsetsByTimes(Map<TopicPartition, Long> timestamp
* @return The fetched records per partition
* @throws OffsetOutOfRangeException If there is OffsetOutOfRange error in fetchResponse and
* the defaultResetPolicy is NONE
+ * @throws TopicAuthorizationException If there is TopicAuthorization error in fetchResponse.
*/
public Map<TopicPartition, List<ConsumerRecord<K, V>>> fetchedRecords() {
Map<TopicPartition, List<ConsumerRecord<K, V>>> fetched = new HashMap<>();
@@ -483,7 +486,20 @@ private ListOffsetResult fetchOffsetsByTimes(Map<TopicPartition, Long> timestamp
CompletedFetch completedFetch = completedFetches.peek();
if (completedFetch == null) break;
- nextInLineRecords = parseCompletedFetch(completedFetch);
+ try {
+ nextInLineRecords = parseCompletedFetch(completedFetch);
+ } catch (Exception e) {
+ // Remove a completedFetch upon a parse with exception if (1) it contains no records, and
+ // (2) there are no fetched records with actual content preceding this exception.
+ // The first condition ensures that the completedFetches is not stuck with the same completedFetch
+ // in cases such as the TopicAuthorizationException, and the second condition ensures that no
+ // potential data loss due to an exception in a following record.
+ FetchResponse.PartitionData partition = completedFetch.partitionData;
+ if (fetched.isEmpty() && (partition.records == null || partition.records.sizeInBytes() == 0)) {
+ completedFetches.poll();
+ }
+ throw e;
+ }
completedFetches.poll();
} else {
List<ConsumerRecord<K, V>> records = fetchRecords(nextInLineRecords, recordsRemaining);
@@ -576,10 +592,11 @@ private void resetOffsetsAsync(Map<TopicPartition, Long> partitionResetTimestamp
Map<Node, Map<TopicPartition, Long>> timestampsToSearchByNode = groupListOffsetRequests(partitionResetTimestamps);
for (Map.Entry<Node, Map<TopicPartition, Long>> entry : timestampsToSearchByNode.entrySet()) {
+ Node node = entry.getKey();
final Map<TopicPartition, Long> resetTimestamps = entry.getValue();
subscriptions.setResetPending(resetTimestamps.keySet(), time.milliseconds() + requestTimeoutMs);
- RequestFuture<ListOffsetResult> future = sendListOffsetRequest(entry.getKey(), resetTimestamps, false);
+ RequestFuture<ListOffsetResult> future = sendListOffsetRequest(node, resetTimestamps, false);
future.addListener(new RequestFutureListener<ListOffsetResult>() {
@Override
public void onSuccess(ListOffsetResult result) {
@@ -667,11 +684,19 @@ public void onFailure(RuntimeException e) {
PartitionInfo info = metadata.fetch().partition(tp);
if (info == null) {
metadata.add(tp.topic());
- log.debug("Partition {} is unknown for fetching offset", tp);
+ log.debug("Leader for partition {} is unknown for fetching offset", tp);
metadata.requestUpdate();
} else if (info.leader() == null) {
- log.debug("Leader for partition {} unavailable for fetching offset", tp);
+ log.debug("Leader for partition {} is unavailable for fetching offset", tp);
metadata.requestUpdate();
+ } else if (client.isUnavailable(info.leader())) {
+ client.maybeThrowAuthFailure(info.leader());
+
+ // The connection has failed and we need to await the blackout period before we can
+ // try again. No need to request a metadata update since the disconnect will have
+ // done so already.
+ log.debug("Leader {} for partition {} is unavailable for fetching offset until reconnect backoff expires",
+ info.leader(), tp);
} else {
Node node = info.leader();
Map<TopicPartition, Long> topicData = timestampsToSearchByNode.get(node);
@@ -823,14 +848,21 @@ public ListOffsetResult() {
* that have no existing requests in flight.
*/
private Map<Node, FetchSessionHandler.FetchRequestData> prepareFetchRequests() {
- // create the fetch info
Cluster cluster = metadata.fetch();
Map<Node, FetchSessionHandler.Builder> fetchable = new LinkedHashMap<>();
for (TopicPartition partition : fetchablePartitions()) {
Node node = cluster.leaderFor(partition);
if (node == null) {
metadata.requestUpdate();
- } else if (!this.client.hasPendingRequests(node)) {
+ } else if (client.isUnavailable(node)) {
+ client.maybeThrowAuthFailure(node);
+
+ // If we try to send during the reconnect blackout window, then the request is just
+ // going to be failed anyway before being sent, so skip the send for now
+ log.trace("Skipping fetch for partition {} because node {} is awaiting reconnect backoff", partition, node);
+ } else if (client.hasPendingRequests(node)) {
+ log.trace("Skipping fetch for partition {} because there is an in-flight request to {}", partition, node);
+ } else {
// if there is a leader and no in-flight requests, issue a new fetch
FetchSessionHandler.Builder builder = fetchable.get(node);
if (builder == null) {
@@ -846,10 +878,9 @@ public ListOffsetResult() {
long position = this.subscriptions.position(partition);
builder.add(partition, new FetchRequest.PartitionData(position, FetchRequest.INVALID_LOG_START_OFFSET,
this.fetchSize));
+
log.debug("Added {} fetch request for partition {} at offset {} to node {}", isolationLevel,
partition, position, node);
- } else {
- log.trace("Skipping fetch for partition {} because there is an in-flight request to {}", partition, node);
}
}
Map<Node, FetchSessionHandler.FetchRequestData> reqs = new LinkedHashMap<>();
@@ -864,7 +895,7 @@ public ListOffsetResult() {
*/
private PartitionRecords parseCompletedFetch(CompletedFetch completedFetch) {
TopicPartition tp = completedFetch.partition;
- FetchResponse.PartitionData partition = completedFetch.partitionData;
+ FetchResponse.PartitionData<Records> partition = completedFetch.partitionData;
long fetchOffset = completedFetch.fetchedOffset;
PartitionRecords partitionRecords = null;
Errors error = partition.error;
@@ -930,7 +961,7 @@ private PartitionRecords parseCompletedFetch(CompletedFetch completedFetch) {
this.metadata.requestUpdate();
} else if (error == Errors.OFFSET_OUT_OF_RANGE) {
if (fetchOffset != subscriptions.position(tp)) {
- log.debug("Discarding stale fetch response for partition {} since the fetched offset {}" +
+ log.debug("Discarding stale fetch response for partition {} since the fetched offset {} " +
"does not match the current offset {}", tp, fetchOffset, subscriptions.position(tp));
} else if (subscriptions.hasDefaultOffsetResetPolicy()) {
log.info("Fetch offset {} is out of range for partition {}, resetting offset", fetchOffset, tp);
@@ -1222,13 +1253,13 @@ private boolean containsAbortMarker(RecordBatch batch) {
private static class CompletedFetch {
private final TopicPartition partition;
private final long fetchedOffset;
- private final FetchResponse.PartitionData partitionData;
+ private final FetchResponse.PartitionData<Records> partitionData;
private final FetchResponseMetricAggregator metricAggregator;
private final short responseVersion;
private CompletedFetch(TopicPartition partition,
long fetchedOffset,
- FetchResponse.PartitionData partitionData,
+ FetchResponse.PartitionData<Records> partitionData,
FetchResponseMetricAggregator metricAggregator,
short responseVersion) {
this.partition = partition;
diff --git a/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java b/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java
index 5fc9a1b9b38..a5af5b60093 100644
--- a/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java
+++ b/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java
@@ -256,6 +256,7 @@
private final ProducerInterceptors<K, V> interceptors;
private final ApiVersions apiVersions;
private final TransactionManager transactionManager;
+ private TransactionalRequestResult initTransactionsResult;
/**
* A producer is instantiated by providing a set of key-value pairs as configuration. Valid configuration strings
@@ -555,18 +556,36 @@ private static int parseAcks(String acksString) {
* 2. Gets the internal producer id and epoch, used in all future transactional
* messages issued by the producer.
*
+ * Note that this method will raise {@link TimeoutException} if the transactional state cannot
+ * be initialized before expiration of {@code max.block.ms}. Additionally, it will raise {@link InterruptException}
+ * if interrupted. It is safe to retry in either case, but once the transactional state has been successfully
+ * initialized, this method should no longer be used.
+ *
* @throws IllegalStateException if no transactional.id has been configured
* @throws org.apache.kafka.common.errors.UnsupportedVersionException fatal error indicating the broker
* does not support transactions (i.e. if its version is lower than 0.11.0.0)
* @throws org.apache.kafka.common.errors.AuthorizationException fatal error indicating that the configured
* transactional.id is not authorized. See the exception for more details
* @throws KafkaException if the producer has encountered a previous fatal error or for any other unexpected error
+ * @throws TimeoutException if the time taken for initialize the transaction has surpassed <code>max.block.ms</code>.
+ * @throws InterruptException if the thread is interrupted while blocked
*/
public void initTransactions() {
throwIfNoTransactionManager();
- TransactionalRequestResult result = transactionManager.initializeTransactions();
- sender.wakeup();
- result.await();
+ if (initTransactionsResult == null) {
+ initTransactionsResult = transactionManager.initializeTransactions();
+ sender.wakeup();
+ }
+
+ try {
+ if (initTransactionsResult.await(maxBlockTimeMs, TimeUnit.MILLISECONDS)) {
+ initTransactionsResult = null;
+ } else {
+ throw new TimeoutException("Timeout expired while initializing transactional state in " + maxBlockTimeMs + "ms.");
+ }
+ } catch (InterruptedException e) {
+ throw new InterruptException("Initialize transactions interrupted.", e);
+ }
}
/**
diff --git a/clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java b/clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java
index ddf5b887205..9e9869a7e48 100644
--- a/clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java
+++ b/clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java
@@ -32,7 +32,6 @@
import java.util.ArrayDeque;
import java.util.ArrayList;
-import java.util.Collections;
import java.util.Deque;
import java.util.HashMap;
import java.util.List;
@@ -70,6 +69,7 @@
private boolean producerFenced;
private boolean sentOffsets;
private long commitCount = 0L;
+ private Map<MetricName, Metric> mockMetrics;
/**
* Create a mock producer
@@ -99,6 +99,7 @@ public MockProducer(final Cluster cluster,
this.consumerGroupOffsets = new ArrayList<>();
this.uncommittedConsumerGroupOffsets = new HashMap<>();
this.completions = new ArrayDeque<>();
+ this.mockMetrics = new HashMap<>();
}
/**
@@ -293,7 +294,14 @@ public synchronized void flush() {
}
public Map<MetricName, Metric> metrics() {
- return Collections.emptyMap();
+ return mockMetrics;
+ }
+
+ /**
+ * Set a mock metric for testing purpose
+ */
+ public void setMockMetrics(MetricName name, Metric metric) {
+ mockMetrics.put(name, metric);
}
@Override
diff --git a/clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java b/clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java
index aa3147b28b8..0c484dfae05 100644
--- a/clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java
+++ b/clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java
@@ -212,7 +212,7 @@
/** <code> transaction.timeout.ms </code> */
public static final String TRANSACTION_TIMEOUT_CONFIG = "transaction.timeout.ms";
public static final String TRANSACTION_TIMEOUT_DOC = "The maximum amount of time in ms that the transaction coordinator will wait for a transaction status update from the producer before proactively aborting the ongoing transaction." +
- "If this value is larger than the max.transaction.timeout.ms setting in the broker, the request will fail with a `InvalidTransactionTimeout` error.";
+ "If this value is larger than the transaction.max.timeout.ms setting in the broker, the request will fail with a `InvalidTransactionTimeout` error.";
/** <code> transactional.id </code> */
public static final String TRANSACTIONAL_ID_CONFIG = "transactional.id";
diff --git a/clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java b/clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
index ba8c28ece27..e2b58448dc9 100644
--- a/clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
+++ b/clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
@@ -81,7 +81,7 @@
private final ConcurrentMap<TopicPartition, Deque<ProducerBatch>> batches;
private final IncompleteBatches incomplete;
// The following variables are only accessed by the sender thread, so we don't need to protect them.
- private final Set<TopicPartition> muted;
+ private final Map<TopicPartition, Long> muted;
private int drainIndex;
private final TransactionManager transactionManager;
@@ -126,7 +126,7 @@ public RecordAccumulator(LogContext logContext,
String metricGrpName = "producer-metrics";
this.free = new BufferPool(totalSize, batchSize, metrics, time, metricGrpName);
this.incomplete = new IncompleteBatches();
- this.muted = new HashSet<>();
+ this.muted = new HashMap<>();
this.time = time;
this.apiVersions = apiVersions;
this.transactionManager = transactionManager;
@@ -265,6 +265,13 @@ private RecordAppendResult tryAppend(long timestamp, byte[] key, byte[] value, H
return null;
}
+ private boolean isMuted(TopicPartition tp, long now) {
+ boolean result = muted.containsKey(tp) && muted.get(tp) > now;
+ if (!result)
+ muted.remove(tp);
+ return result;
+ }
+
/**
* Get a list of batches which have been sitting in the accumulator too long and need to be expired.
*/
@@ -277,7 +284,7 @@ private RecordAppendResult tryAppend(long timestamp, byte[] key, byte[] value, H
// This is to prevent later batches from being expired while an earlier batch is still in progress.
// Note that `muted` is only ever populated if `max.in.flight.request.per.connection=1` so this protection
// is only active in this case. Otherwise the expiration order is not guaranteed.
- if (!muted.contains(tp)) {
+ if (!isMuted(tp, now)) {
synchronized (dq) {
// iterate over the batches and expire them if they have been in the accumulator for more than requestTimeOut
ProducerBatch lastBatch = dq.peekLast();
@@ -436,7 +443,7 @@ public ReadyCheckResult ready(Cluster cluster, long nowMs) {
// This is a partition for which leader is not known, but messages are available to send.
// Note that entries are currently not removed from batches when deque is empty.
unknownLeaderTopics.add(part.topic());
- } else if (!readyNodes.contains(leader) && !muted.contains(part)) {
+ } else if (!readyNodes.contains(leader) && !isMuted(part, nowMs)) {
ProducerBatch batch = deque.peekFirst();
if (batch != null) {
long waitedTimeMs = batch.waitedTimeMs(nowMs);
@@ -504,7 +511,7 @@ public boolean hasUndrained() {
PartitionInfo part = parts.get(drainIndex);
TopicPartition tp = new TopicPartition(part.topic(), part.partition());
// Only proceed if the partition has no in-flight batches.
- if (!muted.contains(tp)) {
+ if (!isMuted(tp, now)) {
Deque<ProducerBatch> deque = getDeque(tp);
if (deque != null) {
synchronized (deque) {
@@ -733,11 +740,11 @@ void abortUndrainedBatches(RuntimeException reason) {
}
public void mutePartition(TopicPartition tp) {
- muted.add(tp);
+ muted.put(tp, Long.MAX_VALUE);
}
- public void unmutePartition(TopicPartition tp) {
- muted.remove(tp);
+ public void unmutePartition(TopicPartition tp, long throttleUntilTimeMs) {
+ muted.put(tp, throttleUntilTimeMs);
}
/**
diff --git a/clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java b/clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java
index 7eea4992b33..7c941796ce4 100644
--- a/clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java
+++ b/clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java
@@ -252,6 +252,9 @@ private long sendProducerData(long now) {
// and request metadata update, since there are messages to send to the topic.
for (String topic : result.unknownLeaderTopics)
this.metadata.add(topic);
+
+ log.debug("Requesting metadata update due to unknown leader topics from the batched records: {}", result.unknownLeaderTopics);
+
this.metadata.requestUpdate();
}
@@ -262,7 +265,7 @@ private long sendProducerData(long now) {
Node node = iter.next();
if (!this.client.ready(node, now)) {
iter.remove();
- notReadyTimeout = Math.min(notReadyTimeout, this.client.connectionDelay(node, now));
+ notReadyTimeout = Math.min(notReadyTimeout, this.client.pollDelayMs(node, now));
}
}
@@ -329,7 +332,7 @@ private boolean maybeSendTransactionalRequest(long now) {
return false;
AbstractRequest.Builder<?> requestBuilder = nextRequestHandler.requestBuilder();
- while (true) {
+ while (!forceClose) {
Node targetNode = null;
try {
if (nextRequestHandler.needsCoordinator()) {
@@ -458,17 +461,18 @@ private void maybeWaitForProducerId() {
*/
private void handleProduceResponse(ClientResponse response, Map<TopicPartition, ProducerBatch> batches, long now) {
RequestHeader requestHeader = response.requestHeader();
+ long receivedTimeMs = response.receivedTimeMs();
int correlationId = requestHeader.correlationId();
if (response.wasDisconnected()) {
log.trace("Cancelled request with header {} due to node {} being disconnected",
requestHeader, response.destination());
for (ProducerBatch batch : batches.values())
- completeBatch(batch, new ProduceResponse.PartitionResponse(Errors.NETWORK_EXCEPTION), correlationId, now);
+ completeBatch(batch, new ProduceResponse.PartitionResponse(Errors.NETWORK_EXCEPTION), correlationId, now, 0L);
} else if (response.versionMismatch() != null) {
log.warn("Cancelled request {} due to a version mismatch with node {}",
response, response.destination(), response.versionMismatch());
for (ProducerBatch batch : batches.values())
- completeBatch(batch, new ProduceResponse.PartitionResponse(Errors.UNSUPPORTED_VERSION), correlationId, now);
+ completeBatch(batch, new ProduceResponse.PartitionResponse(Errors.UNSUPPORTED_VERSION), correlationId, now, 0L);
} else {
log.trace("Received produce response from node {} with correlation id {}", response.destination(), correlationId);
// if we have a response, parse it
@@ -478,13 +482,13 @@ private void handleProduceResponse(ClientResponse response, Map<TopicPartition,
TopicPartition tp = entry.getKey();
ProduceResponse.PartitionResponse partResp = entry.getValue();
ProducerBatch batch = batches.get(tp);
- completeBatch(batch, partResp, correlationId, now);
+ completeBatch(batch, partResp, correlationId, now, receivedTimeMs + produceResponse.throttleTimeMs());
}
this.sensors.recordLatency(response.destination(), response.requestLatencyMs());
} else {
// this is the acks = 0 case, just complete all requests
for (ProducerBatch batch : batches.values()) {
- completeBatch(batch, new ProduceResponse.PartitionResponse(Errors.NONE), correlationId, now);
+ completeBatch(batch, new ProduceResponse.PartitionResponse(Errors.NONE), correlationId, now, 0L);
}
}
}
@@ -499,7 +503,7 @@ private void handleProduceResponse(ClientResponse response, Map<TopicPartition,
* @param now The current POSIX timestamp in milliseconds
*/
private void completeBatch(ProducerBatch batch, ProduceResponse.PartitionResponse response, long correlationId,
- long now) {
+ long now, long throttleUntilTimeMs) {
Errors error = response.error;
if (error == Errors.MESSAGE_TOO_LARGE && batch.recordCount > 1 &&
@@ -557,9 +561,13 @@ else if (error == Errors.CLUSTER_AUTHORIZATION_FAILED)
failBatch(batch, response, exception, batch.attempts() < this.retries);
}
if (error.exception() instanceof InvalidMetadataException) {
- if (error.exception() instanceof UnknownTopicOrPartitionException)
+ if (error.exception() instanceof UnknownTopicOrPartitionException) {
log.warn("Received unknown topic or partition error in produce request on partition {}. The " +
"topic/partition may not exist or the user may not have Describe access to it", batch.topicPartition);
+ } else {
+ log.warn("Received invalid metadata error in produce request on partition {} due to {}. Going " +
+ "to request metadata update now", batch.topicPartition, error.exception().toString());
+ }
metadata.requestUpdate();
}
@@ -569,7 +577,7 @@ else if (error == Errors.CLUSTER_AUTHORIZATION_FAILED)
// Unmute the completed partition.
if (guaranteeMessageOrder)
- this.accumulator.unmutePartition(batch.topicPartition);
+ this.accumulator.unmutePartition(batch.topicPartition, throttleUntilTimeMs);
}
private void reenqueueBatch(ProducerBatch batch, long currentTimeMs) {
diff --git a/clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionalRequestResult.java b/clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionalRequestResult.java
index ff93da872dc..9c02e94c045 100644
--- a/clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionalRequestResult.java
+++ b/clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionalRequestResult.java
@@ -59,7 +59,10 @@ public void await() {
}
public boolean await(long timeout, TimeUnit unit) throws InterruptedException {
- return latch.await(timeout, unit);
+ boolean success = latch.await(timeout, unit);
+ if (!isSuccessful())
+ throw error();
+ return success;
}
public RuntimeException error() {
diff --git a/clients/src/main/java/org/apache/kafka/common/Cluster.java b/clients/src/main/java/org/apache/kafka/common/Cluster.java
index 0c59f33824d..ccbaa306d48 100644
--- a/clients/src/main/java/org/apache/kafka/common/Cluster.java
+++ b/clients/src/main/java/org/apache/kafka/common/Cluster.java
@@ -274,7 +274,8 @@ public Node controller() {
@Override
public String toString() {
- return "Cluster(id = " + clusterResource.clusterId() + ", nodes = " + this.nodes + ", partitions = " + this.partitionsByTopicPartition.values() + ")";
+ return "Cluster(id = " + clusterResource.clusterId() + ", nodes = " + this.nodes +
+ ", partitions = " + this.partitionsByTopicPartition.values() + ", controller = " + controller + ")";
}
}
diff --git a/clients/src/main/java/org/apache/kafka/common/ClusterResourceListener.java b/clients/src/main/java/org/apache/kafka/common/ClusterResourceListener.java
index f8f99ece183..97245a393b0 100644
--- a/clients/src/main/java/org/apache/kafka/common/ClusterResourceListener.java
+++ b/clients/src/main/java/org/apache/kafka/common/ClusterResourceListener.java
@@ -36,7 +36,8 @@
* {@link org.apache.kafka.common.serialization.Deserializer} : The {@link ClusterResourceListener#onUpdate(ClusterResource)} method will be invoked before {@link org.apache.kafka.common.serialization.Deserializer#deserialize(String, byte[])}
* <p>
* {@link org.apache.kafka.common.metrics.MetricsReporter} : The {@link ClusterResourceListener#onUpdate(ClusterResource)} method will be invoked after first {@link org.apache.kafka.clients.producer.KafkaProducer#send(ProducerRecord)} invocation for Producer metrics reporter
- * and after first {@link org.apache.kafka.clients.consumer.KafkaConsumer#poll(long)} invocation for Consumer metrics reporters. The reporter may receive metric events from the network layer before this method is invoked.
+ * and after first {@link org.apache.kafka.clients.consumer.KafkaConsumer#poll(java.time.Duration)} invocation for Consumer metrics
+ * reporters. The reporter may receive metric events from the network layer before this method is invoked.
* <h4>Broker</h4>
* There is a single invocation {@link ClusterResourceListener#onUpdate(ClusterResource)} on broker start-up and the cluster metadata will never change.
* <p>
diff --git a/clients/src/main/java/org/apache/kafka/common/ConsumerGroupState.java b/clients/src/main/java/org/apache/kafka/common/ConsumerGroupState.java
new file mode 100644
index 00000000000..7f3d4f0883b
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/ConsumerGroupState.java
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.common;
+
+import java.util.HashMap;
+
+/**
+ * The consumer group state.
+ */
+public enum ConsumerGroupState {
+ UNKNOWN("Unknown"),
+ PREPARING_REBALANCE("PreparingRebalance"),
+ COMPLETING_REBALANCE("CompletingRebalance"),
+ STABLE("Stable"),
+ DEAD("Dead"),
+ EMPTY("Empty");
+
+ private final static HashMap<String, ConsumerGroupState> NAME_TO_ENUM;
+
+ static {
+ NAME_TO_ENUM = new HashMap<>();
+ for (ConsumerGroupState state : ConsumerGroupState.values()) {
+ NAME_TO_ENUM.put(state.name, state);
+ }
+ }
+
+ private final String name;
+
+ ConsumerGroupState(String name) {
+ this.name = name;
+ }
+
+
+ /**
+ * Parse a string into a consumer group state.
+ */
+ public static ConsumerGroupState parse(String name) {
+ ConsumerGroupState state = NAME_TO_ENUM.get(name);
+ return state == null ? UNKNOWN : state;
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/config/ConfigChangeCallback.java b/clients/src/main/java/org/apache/kafka/common/config/ConfigChangeCallback.java
new file mode 100644
index 00000000000..d4c9948bc93
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/config/ConfigChangeCallback.java
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.config;
+
+/**
+ * A callback passed to {@link ConfigProvider} for subscribing to changes.
+ */
+public interface ConfigChangeCallback {
+
+ /**
+ * Performs an action when configuration data changes.
+ *
+ * @param path the path at which the data resides
+ * @param data the configuration data
+ */
+ void onChange(String path, ConfigData data);
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/config/ConfigData.java b/clients/src/main/java/org/apache/kafka/common/config/ConfigData.java
new file mode 100644
index 00000000000..2bd0ff6b06a
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/config/ConfigData.java
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.config;
+
+import java.util.Map;
+
+/**
+ * Configuration data from a {@link ConfigProvider}.
+ */
+public class ConfigData {
+
+ private final Map<String, String> data;
+ private final Long ttl;
+
+ /**
+ * Creates a new ConfigData with the given data and TTL (in milliseconds).
+ *
+ * @param data a Map of key-value pairs
+ * @param ttl the time-to-live of the data in milliseconds, or null if there is no TTL
+ */
+ public ConfigData(Map<String, String> data, Long ttl) {
+ this.data = data;
+ this.ttl = ttl;
+ }
+
+ /**
+ * Creates a new ConfigData with the given data.
+ *
+ * @param data a Map of key-value pairs
+ */
+ public ConfigData(Map<String, String> data) {
+ this(data, null);
+ }
+
+ /**
+ * Returns the data.
+ *
+ * @return data a Map of key-value pairs
+ */
+ public Map<String, String> data() {
+ return data;
+ }
+
+ /**
+ * Returns the TTL (in milliseconds).
+ *
+ * @return ttl the time-to-live (in milliseconds) of the data, or null if there is no TTL
+ */
+ public Long ttl() {
+ return ttl;
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/config/ConfigProvider.java b/clients/src/main/java/org/apache/kafka/common/config/ConfigProvider.java
new file mode 100644
index 00000000000..7133baaebd0
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/config/ConfigProvider.java
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.config;
+
+import org.apache.kafka.common.Configurable;
+
+import java.io.Closeable;
+import java.util.Set;
+
+/**
+ * A provider of configuration data, which may optionally support subscriptions to configuration changes.
+ */
+public interface ConfigProvider extends Configurable, Closeable {
+
+ /**
+ * Retrieves the data at the given path.
+ *
+ * @param path the path where the data resides
+ * @return the configuration data
+ */
+ ConfigData get(String path);
+
+ /**
+ * Retrieves the data with the given keys at the given path.
+ *
+ * @param path the path where the data resides
+ * @param keys the keys whose values will be retrieved
+ * @return the configuration data
+ */
+ ConfigData get(String path, Set<String> keys);
+
+ /**
+ * Subscribes to changes for the given keys at the given path (optional operation).
+ *
+ * @param path the path where the data resides
+ * @param keys the keys whose values will be retrieved
+ * @param callback the callback to invoke upon change
+ * @throws {@link UnsupportedOperationException} if the subscribe operation is not supported
+ */
+ default void subscribe(String path, Set<String> keys, ConfigChangeCallback callback) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Unsubscribes to changes for the given keys at the given path (optional operation).
+ *
+ * @param path the path where the data resides
+ * @param keys the keys whose values will be retrieved
+ * @param callback the callback to be unsubscribed from changes
+ * @throws {@link UnsupportedOperationException} if the unsubscribe operation is not supported
+ */
+ default void unsubscribe(String path, Set<String> keys, ConfigChangeCallback callback) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Clears all subscribers (optional operation).
+ *
+ * @throws {@link UnsupportedOperationException} if the unsubscribeAll operation is not supported
+ */
+ default void unsubscribeAll() {
+ throw new UnsupportedOperationException();
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/config/ConfigResource.java b/clients/src/main/java/org/apache/kafka/common/config/ConfigResource.java
index 4402c260202..da718f54d54 100644
--- a/clients/src/main/java/org/apache/kafka/common/config/ConfigResource.java
+++ b/clients/src/main/java/org/apache/kafka/common/config/ConfigResource.java
@@ -90,6 +90,6 @@ public int hashCode() {
@Override
public String toString() {
- return "ConfigResource{type=" + type + ", name='" + name + "'}";
+ return "ConfigResource(type=" + type + ", name='" + name + "')";
}
}
diff --git a/clients/src/main/java/org/apache/kafka/common/config/ConfigTransformer.java b/clients/src/main/java/org/apache/kafka/common/config/ConfigTransformer.java
new file mode 100644
index 00000000000..7c3c516b073
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/config/ConfigTransformer.java
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.config;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * This class wraps a set of {@link ConfigProvider} instances and uses them to perform
+ * transformations.
+ *
+ * <p>The default variable pattern is of the form <code>${provider:[path:]key}</code>,
+ * where the <code>provider</code> corresponds to a {@link ConfigProvider} instance, as passed to
+ * {@link ConfigTransformer#ConfigTransformer(Map)}. The pattern will extract a set
+ * of paths (which are optional) and keys and then pass them to {@link ConfigProvider#get(String, Set)} to obtain the
+ * values with which to replace the variables.
+ *
+ * <p>For example, if a Map consisting of an entry with a provider name "file" and provider instance
+ * {@link FileConfigProvider} is passed to the {@link ConfigTransformer#ConfigTransformer(Map)}, and a Properties
+ * file with contents
+ * <pre>
+ * fileKey=someValue
+ * </pre>
+ * resides at the path "/tmp/properties.txt", then when a configuration Map which has an entry with a key "someKey" and
+ * a value "${file:/tmp/properties.txt:fileKey}" is passed to the {@link #transform(Map)} method, then the transformed
+ * Map will have an entry with key "someKey" and a value "someValue".
+ *
+ * <p>This class only depends on {@link ConfigProvider#get(String, Set)} and does not depend on subscription support
+ * in a {@link ConfigProvider}, such as the {@link ConfigProvider#subscribe(String, Set, ConfigChangeCallback)} and
+ * {@link ConfigProvider#unsubscribe(String, Set, ConfigChangeCallback)} methods.
+ */
+public class ConfigTransformer {
+ private static final Pattern DEFAULT_PATTERN = Pattern.compile("\\$\\{(.*?):((.*?):)?(.*?)\\}");
+ private static final String EMPTY_PATH = "";
+
+ private final Map<String, ConfigProvider> configProviders;
+
+ /**
+ * Creates a ConfigTransformer with the default pattern, of the form <code>${provider:[path:]key}</code>.
+ *
+ * @param configProviders a Map of provider names and {@link ConfigProvider} instances.
+ */
+ public ConfigTransformer(Map<String, ConfigProvider> configProviders) {
+ this.configProviders = configProviders;
+ }
+
+ /**
+ * Transforms the given configuration data by using the {@link ConfigProvider} instances to
+ * look up values to replace the variables in the pattern.
+ *
+ * @param configs the configuration values to be transformed
+ * @return an instance of {@link ConfigTransformerResult}
+ */
+ public ConfigTransformerResult transform(Map<String, String> configs) {
+ Map<String, Map<String, Set<String>>> keysByProvider = new HashMap<>();
+ Map<String, Map<String, Map<String, String>>> lookupsByProvider = new HashMap<>();
+
+ // Collect the variables from the given configs that need transformation
+ for (Map.Entry<String, String> config : configs.entrySet()) {
+ List<ConfigVariable> vars = getVars(config.getKey(), config.getValue(), DEFAULT_PATTERN);
+ for (ConfigVariable var : vars) {
+ Map<String, Set<String>> keysByPath = keysByProvider.computeIfAbsent(var.providerName, k -> new HashMap<>());
+ Set<String> keys = keysByPath.computeIfAbsent(var.path, k -> new HashSet<>());
+ keys.add(var.variable);
+ }
+ }
+
+ // Retrieve requested variables from the ConfigProviders
+ Map<String, Long> ttls = new HashMap<>();
+ for (Map.Entry<String, Map<String, Set<String>>> entry : keysByProvider.entrySet()) {
+ String providerName = entry.getKey();
+ ConfigProvider provider = configProviders.get(providerName);
+ Map<String, Set<String>> keysByPath = entry.getValue();
+ if (provider != null && keysByPath != null) {
+ for (Map.Entry<String, Set<String>> pathWithKeys : keysByPath.entrySet()) {
+ String path = pathWithKeys.getKey();
+ Set<String> keys = new HashSet<>(pathWithKeys.getValue());
+ ConfigData configData = provider.get(path, keys);
+ Map<String, String> data = configData.data();
+ Long ttl = configData.ttl();
+ if (ttl != null && ttl >= 0) {
+ ttls.put(path, ttl);
+ }
+ Map<String, Map<String, String>> keyValuesByPath =
+ lookupsByProvider.computeIfAbsent(providerName, k -> new HashMap<>());
+ keyValuesByPath.put(path, data);
+ }
+ }
+ }
+
+ // Perform the transformations by performing variable replacements
+ Map<String, String> data = new HashMap<>(configs);
+ for (Map.Entry<String, String> config : configs.entrySet()) {
+ data.put(config.getKey(), replace(lookupsByProvider, config.getValue(), DEFAULT_PATTERN));
+ }
+ return new ConfigTransformerResult(data, ttls);
+ }
+
+ private static List<ConfigVariable> getVars(String key, String value, Pattern pattern) {
+ List<ConfigVariable> configVars = new ArrayList<>();
+ Matcher matcher = pattern.matcher(value);
+ while (matcher.find()) {
+ configVars.add(new ConfigVariable(matcher));
+ }
+ return configVars;
+ }
+
+ private static String replace(Map<String, Map<String, Map<String, String>>> lookupsByProvider,
+ String value,
+ Pattern pattern) {
+ Matcher matcher = pattern.matcher(value);
+ StringBuilder builder = new StringBuilder();
+ int i = 0;
+ while (matcher.find()) {
+ ConfigVariable configVar = new ConfigVariable(matcher);
+ Map<String, Map<String, String>> lookupsByPath = lookupsByProvider.get(configVar.providerName);
+ if (lookupsByPath != null) {
+ Map<String, String> keyValues = lookupsByPath.get(configVar.path);
+ String replacement = keyValues.get(configVar.variable);
+ builder.append(value, i, matcher.start());
+ if (replacement == null) {
+ // No replacements will be performed; just return the original value
+ builder.append(matcher.group(0));
+ } else {
+ builder.append(replacement);
+ }
+ i = matcher.end();
+ }
+ }
+ builder.append(value, i, value.length());
+ return builder.toString();
+ }
+
+ private static class ConfigVariable {
+ final String providerName;
+ final String path;
+ final String variable;
+
+ ConfigVariable(Matcher matcher) {
+ this.providerName = matcher.group(1);
+ this.path = matcher.group(3) != null ? matcher.group(3) : EMPTY_PATH;
+ this.variable = matcher.group(4);
+ }
+
+ public String toString() {
+ return "(" + providerName + ":" + (path != null ? path + ":" : "") + variable + ")";
+ }
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/config/ConfigTransformerResult.java b/clients/src/main/java/org/apache/kafka/common/config/ConfigTransformerResult.java
new file mode 100644
index 00000000000..df7bea62f37
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/config/ConfigTransformerResult.java
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.config;
+
+import java.util.Map;
+
+/**
+ * The result of a transformation from {@link ConfigTransformer}.
+ */
+public class ConfigTransformerResult {
+
+ private Map<String, Long> ttls;
+ private Map<String, String> data;
+
+ /**
+ * Creates a new ConfigTransformerResult with the given data and TTL values for a set of paths.
+ *
+ * @param data a Map of key-value pairs
+ * @param ttls a Map of path and TTL values (in milliseconds)
+ */
+ public ConfigTransformerResult(Map<String, String> data, Map<String, Long> ttls) {
+ this.data = data;
+ this.ttls = ttls;
+ }
+
+ /**
+ * Returns the transformed data, with variables replaced with corresponding values from the
+ * ConfigProvider instances if found.
+ *
+ * <p>Modifying the transformed data that is returned does not affect the {@link ConfigProvider} nor the
+ * original data that was used as the source of the transformation.
+ *
+ * @return data a Map of key-value pairs
+ */
+ public Map<String, String> data() {
+ return data;
+ }
+
+ /**
+ * Returns the TTL values (in milliseconds) returned from the ConfigProvider instances for a given set of paths.
+ *
+ * @return data a Map of path and TTL values
+ */
+ public Map<String, Long> ttls() {
+ return ttls;
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/config/FileConfigProvider.java b/clients/src/main/java/org/apache/kafka/common/config/FileConfigProvider.java
new file mode 100644
index 00000000000..fefc93566f3
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/config/FileConfigProvider.java
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.config;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.Reader;
+import java.nio.charset.StandardCharsets;
+import java.util.Enumeration;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+
+/**
+ * An implementation of {@link ConfigProvider} that represents a Properties file.
+ * All property keys and values are stored as cleartext.
+ */
+public class FileConfigProvider implements ConfigProvider {
+
+ public void configure(Map<String, ?> configs) {
+ }
+
+ /**
+ * Retrieves the data at the given Properties file.
+ *
+ * @param path the file where the data resides
+ * @return the configuration data
+ */
+ public ConfigData get(String path) {
+ Map<String, String> data = new HashMap<>();
+ if (path == null || path.isEmpty()) {
+ return new ConfigData(data);
+ }
+ try (Reader reader = reader(path)) {
+ Properties properties = new Properties();
+ properties.load(reader);
+ Enumeration<Object> keys = properties.keys();
+ while (keys.hasMoreElements()) {
+ String key = keys.nextElement().toString();
+ String value = properties.getProperty(key);
+ if (value != null) {
+ data.put(key, value);
+ }
+ }
+ return new ConfigData(data);
+ } catch (IOException e) {
+ throw new ConfigException("Could not read properties from file " + path);
+ }
+ }
+
+ /**
+ * Retrieves the data with the given keys at the given Properties file.
+ *
+ * @param path the file where the data resides
+ * @param keys the keys whose values will be retrieved
+ * @return the configuration data
+ */
+ public ConfigData get(String path, Set<String> keys) {
+ Map<String, String> data = new HashMap<>();
+ if (path == null || path.isEmpty()) {
+ return new ConfigData(data);
+ }
+ try (Reader reader = reader(path)) {
+ Properties properties = new Properties();
+ properties.load(reader);
+ for (String key : keys) {
+ String value = properties.getProperty(key);
+ if (value != null) {
+ data.put(key, value);
+ }
+ }
+ return new ConfigData(data);
+ } catch (IOException e) {
+ throw new ConfigException("Could not read properties from file " + path);
+ }
+ }
+
+ // visible for testing
+ protected Reader reader(String path) throws IOException {
+ return new InputStreamReader(new FileInputStream(path), StandardCharsets.UTF_8);
+ }
+
+ public void close() {
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/config/SaslConfigs.java b/clients/src/main/java/org/apache/kafka/common/config/SaslConfigs.java
index f61b7dd5e6c..db93ea41647 100644
--- a/clients/src/main/java/org/apache/kafka/common/config/SaslConfigs.java
+++ b/clients/src/main/java/org/apache/kafka/common/config/SaslConfigs.java
@@ -16,6 +16,7 @@
*/
package org.apache.kafka.common.config;
+import org.apache.kafka.common.config.ConfigDef.Range;
import org.apache.kafka.common.config.internals.BrokerSecurityConfigs;
import java.util.List;
@@ -49,7 +50,24 @@
public static final String SASL_JAAS_CONFIG = "sasl.jaas.config";
public static final String SASL_JAAS_CONFIG_DOC = "JAAS login context parameters for SASL connections in the format used by JAAS configuration files. "
+ "JAAS configuration file format is described <a href=\"http://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/tutorials/LoginConfigFile.html\">here</a>. "
- + "The format for the value is: '<loginModuleClass> <controlFlag> (<optionName>=<optionValue>)*;'";
+ + "The format for the value is: '<code>loginModuleClass controlFlag (optionName=optionValue)*;</code>'. For brokers, "
+ + "the config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, "
+ + "listener.name.sasl_ssl.scram-sha-256.sasl.jaas.config=com.example.ScramLoginModule required;";
+
+ public static final String SASL_CLIENT_CALLBACK_HANDLER_CLASS = "sasl.client.callback.handler.class";
+ public static final String SASL_CLIENT_CALLBACK_HANDLER_CLASS_DOC = "The fully qualified name of a SASL client callback handler class "
+ + "that implements the AuthenticateCallbackHandler interface.";
+
+ public static final String SASL_LOGIN_CALLBACK_HANDLER_CLASS = "sasl.login.callback.handler.class";
+ public static final String SASL_LOGIN_CALLBACK_HANDLER_CLASS_DOC = "The fully qualified name of a SASL login callback handler class "
+ + "that implements the AuthenticateCallbackHandler interface. For brokers, login callback handler config must be prefixed with "
+ + "listener prefix and SASL mechanism name in lower-case. For example, "
+ + "listener.name.sasl_ssl.scram-sha-256.sasl.login.callback.handler.class=com.example.CustomScramLoginCallbackHandler";
+
+ public static final String SASL_LOGIN_CLASS = "sasl.login.class";
+ public static final String SASL_LOGIN_CLASS_DOC = "The fully qualified name of a class that implements the Login interface. "
+ + "For brokers, login config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, "
+ + "listener.name.sasl_ssl.scram-sha-256.sasl.login.class=com.example.CustomScramLogin";
public static final String SASL_KERBEROS_SERVICE_NAME = "sasl.kerberos.service.name";
public static final String SASL_KERBEROS_SERVICE_NAME_DOC = "The Kerberos principal name that Kafka runs as. "
@@ -72,6 +90,34 @@
public static final String SASL_KERBEROS_MIN_TIME_BEFORE_RELOGIN_DOC = "Login thread sleep time between refresh attempts.";
public static final long DEFAULT_KERBEROS_MIN_TIME_BEFORE_RELOGIN = 1 * 60 * 1000L;
+ public static final String SASL_LOGIN_REFRESH_WINDOW_FACTOR = "sasl.login.refresh.window.factor";
+ public static final String SASL_LOGIN_REFRESH_WINDOW_FACTOR_DOC = "Login refresh thread will sleep until the specified window factor relative to the"
+ + " credential's lifetime has been reached, at which time it will try to refresh the credential."
+ + " Legal values are between 0.5 (50%) and 1.0 (100%) inclusive; a default value of 0.8 (80%) is used"
+ + " if no value is specified. Currently applies only to OAUTHBEARER.";
+ public static final double DEFAULT_LOGIN_REFRESH_WINDOW_FACTOR = 0.80;
+
+ public static final String SASL_LOGIN_REFRESH_WINDOW_JITTER = "sasl.login.refresh.window.jitter";
+ public static final String SASL_LOGIN_REFRESH_WINDOW_JITTER_DOC = "The maximum amount of random jitter relative to the credential's lifetime"
+ + " that is added to the login refresh thread's sleep time. Legal values are between 0 and 0.25 (25%) inclusive;"
+ + " a default value of 0.05 (5%) is used if no value is specified. Currently applies only to OAUTHBEARER.";
+ public static final double DEFAULT_LOGIN_REFRESH_WINDOW_JITTER = 0.05;
+
+ public static final String SASL_LOGIN_REFRESH_MIN_PERIOD_SECONDS = "sasl.login.refresh.min.period.seconds";
+ public static final String SASL_LOGIN_REFRESH_MIN_PERIOD_SECONDS_DOC = "The desired minimum time for the login refresh thread to wait before refreshing a credential,"
+ + " in seconds. Legal values are between 0 and 900 (15 minutes); a default value of 60 (1 minute) is used if no value is specified. This value and "
+ + " sasl.login.refresh.buffer.seconds are both ignored if their sum exceeds the remaining lifetime of a credential."
+ + " Currently applies only to OAUTHBEARER.";
+ public static final short DEFAULT_LOGIN_REFRESH_MIN_PERIOD_SECONDS = 60;
+
+ public static final String SASL_LOGIN_REFRESH_BUFFER_SECONDS = "sasl.login.refresh.buffer.seconds";
+ public static final String SASL_LOGIN_REFRESH_BUFFER_SECONDS_DOC = "The amount of buffer time before credential expiration to maintain when refreshing a credential,"
+ + " in seconds. If a refresh would otherwise occur closer to expiration than the number of buffer seconds then the refresh will be moved up to maintain"
+ + " as much of the buffer time as possible. Legal values are between 0 and 3600 (1 hour); a default value of 300 (5 minutes) is used if no value is specified."
+ + " This value and sasl.login.refresh.min.period.seconds are both ignored if their sum exceeds the remaining lifetime of a credential."
+ + " Currently applies only to OAUTHBEARER.";
+ public static final short DEFAULT_LOGIN_REFRESH_BUFFER_SECONDS = 300;
+
/**
* @deprecated As of 1.0.0. This field will be removed in a future major release.
*/
@@ -94,7 +140,14 @@ public static void addClientSaslSupport(ConfigDef config) {
.define(SaslConfigs.SASL_KERBEROS_TICKET_RENEW_WINDOW_FACTOR, ConfigDef.Type.DOUBLE, SaslConfigs.DEFAULT_KERBEROS_TICKET_RENEW_WINDOW_FACTOR, ConfigDef.Importance.LOW, SaslConfigs.SASL_KERBEROS_TICKET_RENEW_WINDOW_FACTOR_DOC)
.define(SaslConfigs.SASL_KERBEROS_TICKET_RENEW_JITTER, ConfigDef.Type.DOUBLE, SaslConfigs.DEFAULT_KERBEROS_TICKET_RENEW_JITTER, ConfigDef.Importance.LOW, SaslConfigs.SASL_KERBEROS_TICKET_RENEW_JITTER_DOC)
.define(SaslConfigs.SASL_KERBEROS_MIN_TIME_BEFORE_RELOGIN, ConfigDef.Type.LONG, SaslConfigs.DEFAULT_KERBEROS_MIN_TIME_BEFORE_RELOGIN, ConfigDef.Importance.LOW, SaslConfigs.SASL_KERBEROS_MIN_TIME_BEFORE_RELOGIN_DOC)
+ .define(SaslConfigs.SASL_LOGIN_REFRESH_WINDOW_FACTOR, ConfigDef.Type.DOUBLE, SaslConfigs.DEFAULT_LOGIN_REFRESH_WINDOW_FACTOR, Range.between(0.5, 1.0), ConfigDef.Importance.LOW, SaslConfigs.SASL_LOGIN_REFRESH_WINDOW_FACTOR_DOC)
+ .define(SaslConfigs.SASL_LOGIN_REFRESH_WINDOW_JITTER, ConfigDef.Type.DOUBLE, SaslConfigs.DEFAULT_LOGIN_REFRESH_WINDOW_JITTER, Range.between(0.0, 0.25), ConfigDef.Importance.LOW, SaslConfigs.SASL_LOGIN_REFRESH_WINDOW_JITTER_DOC)
+ .define(SaslConfigs.SASL_LOGIN_REFRESH_MIN_PERIOD_SECONDS, ConfigDef.Type.SHORT, SaslConfigs.DEFAULT_LOGIN_REFRESH_MIN_PERIOD_SECONDS, Range.between(0, 900), ConfigDef.Importance.LOW, SaslConfigs.SASL_LOGIN_REFRESH_MIN_PERIOD_SECONDS_DOC)
+ .define(SaslConfigs.SASL_LOGIN_REFRESH_BUFFER_SECONDS, ConfigDef.Type.SHORT, SaslConfigs.DEFAULT_LOGIN_REFRESH_BUFFER_SECONDS, Range.between(0, 3600), ConfigDef.Importance.LOW, SaslConfigs.SASL_LOGIN_REFRESH_BUFFER_SECONDS_DOC)
.define(SaslConfigs.SASL_MECHANISM, ConfigDef.Type.STRING, SaslConfigs.DEFAULT_SASL_MECHANISM, ConfigDef.Importance.MEDIUM, SaslConfigs.SASL_MECHANISM_DOC)
- .define(SaslConfigs.SASL_JAAS_CONFIG, ConfigDef.Type.PASSWORD, null, ConfigDef.Importance.MEDIUM, SaslConfigs.SASL_JAAS_CONFIG_DOC);
+ .define(SaslConfigs.SASL_JAAS_CONFIG, ConfigDef.Type.PASSWORD, null, ConfigDef.Importance.MEDIUM, SaslConfigs.SASL_JAAS_CONFIG_DOC)
+ .define(SaslConfigs.SASL_CLIENT_CALLBACK_HANDLER_CLASS, ConfigDef.Type.CLASS, null, ConfigDef.Importance.MEDIUM, SaslConfigs.SASL_CLIENT_CALLBACK_HANDLER_CLASS_DOC)
+ .define(SaslConfigs.SASL_LOGIN_CALLBACK_HANDLER_CLASS, ConfigDef.Type.CLASS, null, ConfigDef.Importance.MEDIUM, SaslConfigs.SASL_LOGIN_CALLBACK_HANDLER_CLASS_DOC)
+ .define(SaslConfigs.SASL_LOGIN_CLASS, ConfigDef.Type.CLASS, null, ConfigDef.Importance.MEDIUM, SaslConfigs.SASL_LOGIN_CLASS_DOC);
}
}
diff --git a/clients/src/main/java/org/apache/kafka/common/config/SslConfigs.java b/clients/src/main/java/org/apache/kafka/common/config/SslConfigs.java
index fd4d39e51c7..9a3215f7a50 100644
--- a/clients/src/main/java/org/apache/kafka/common/config/SslConfigs.java
+++ b/clients/src/main/java/org/apache/kafka/common/config/SslConfigs.java
@@ -138,8 +138,12 @@ public static void addClientSslSupport(ConfigDef config) {
.define(SslConfigs.SSL_SECURE_RANDOM_IMPLEMENTATION_CONFIG, ConfigDef.Type.STRING, null, ConfigDef.Importance.LOW, SslConfigs.SSL_SECURE_RANDOM_IMPLEMENTATION_DOC);
}
- public static final Set<String> RECONFIGURABLE_CONFIGS = Utils.mkSet(SslConfigs.SSL_KEYSTORE_TYPE_CONFIG,
+ public static final Set<String> RECONFIGURABLE_CONFIGS = Utils.mkSet(
+ SslConfigs.SSL_KEYSTORE_TYPE_CONFIG,
SslConfigs.SSL_KEYSTORE_LOCATION_CONFIG,
SslConfigs.SSL_KEYSTORE_PASSWORD_CONFIG,
- SslConfigs.SSL_KEY_PASSWORD_CONFIG);
+ SslConfigs.SSL_KEY_PASSWORD_CONFIG,
+ SslConfigs.SSL_TRUSTSTORE_TYPE_CONFIG,
+ SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG,
+ SslConfigs.SSL_TRUSTSTORE_PASSWORD_CONFIG);
}
diff --git a/clients/src/main/java/org/apache/kafka/common/config/TopicConfig.java b/clients/src/main/java/org/apache/kafka/common/config/TopicConfig.java
index 168dd253593..d6b70032626 100755
--- a/clients/src/main/java/org/apache/kafka/common/config/TopicConfig.java
+++ b/clients/src/main/java/org/apache/kafka/common/config/TopicConfig.java
@@ -73,7 +73,7 @@
public static final String RETENTION_MS_DOC = "This configuration controls the maximum time we will retain a " +
"log before we will discard old log segments to free up space if we are using the " +
"\"delete\" retention policy. This represents an SLA on how soon consumers must read " +
- "their data.";
+ "their data. If set to -1, no time limit is applied.";
public static final String MAX_MESSAGE_BYTES_CONFIG = "max.message.bytes";
public static final String MAX_MESSAGE_BYTES_DOC = "<p>The largest record batch size allowed by Kafka. If this " +
diff --git a/clients/src/main/java/org/apache/kafka/common/config/internals/BrokerSecurityConfigs.java b/clients/src/main/java/org/apache/kafka/common/config/internals/BrokerSecurityConfigs.java
index 18616ec74e5..a29d8069b99 100644
--- a/clients/src/main/java/org/apache/kafka/common/config/internals/BrokerSecurityConfigs.java
+++ b/clients/src/main/java/org/apache/kafka/common/config/internals/BrokerSecurityConfigs.java
@@ -33,6 +33,7 @@
public static final String SASL_KERBEROS_PRINCIPAL_TO_LOCAL_RULES_CONFIG = "sasl.kerberos.principal.to.local.rules";
public static final String SSL_CLIENT_AUTH_CONFIG = "ssl.client.auth";
public static final String SASL_ENABLED_MECHANISMS_CONFIG = "sasl.enabled.mechanisms";
+ public static final String SASL_SERVER_CALLBACK_HANDLER_CLASS = "sasl.server.callback.handler.class";
public static final String PRINCIPAL_BUILDER_CLASS_DOC = "The fully qualified name of a class that implements the " +
"KafkaPrincipalBuilder interface, which is used to build the KafkaPrincipal object used during " +
@@ -67,4 +68,9 @@
+ "Only GSSAPI is enabled by default.";
public static final List<String> DEFAULT_SASL_ENABLED_MECHANISMS = Collections.singletonList(SaslConfigs.GSSAPI_MECHANISM);
+ public static final String SASL_SERVER_CALLBACK_HANDLER_CLASS_DOC = "The fully qualified name of a SASL server callback handler "
+ + "class that implements the AuthenticateCallbackHandler interface. Server callback handlers must be prefixed with "
+ + "listener prefix and SASL mechanism name in lower-case. For example, "
+ + "listener.name.sasl_ssl.plain.sasl.server.callback.handler.class=com.example.CustomPlainCallbackHandler.";
+
}
diff --git a/clients/src/main/java/org/apache/kafka/common/errors/GroupIdNotFoundException.java b/clients/src/main/java/org/apache/kafka/common/errors/GroupIdNotFoundException.java
index 1ff30f14fc0..a4d509d3a20 100644
--- a/clients/src/main/java/org/apache/kafka/common/errors/GroupIdNotFoundException.java
+++ b/clients/src/main/java/org/apache/kafka/common/errors/GroupIdNotFoundException.java
@@ -17,15 +17,7 @@
package org.apache.kafka.common.errors;
public class GroupIdNotFoundException extends ApiException {
- private final String groupId;
-
- public GroupIdNotFoundException(String groupId) {
- super("The group id " + groupId + " was not found");
- this.groupId = groupId;
- }
-
- public String groupId() {
- return groupId;
+ public GroupIdNotFoundException(String message) {
+ super(message);
}
-
}
diff --git a/clients/src/main/java/org/apache/kafka/common/errors/GroupNotEmptyException.java b/clients/src/main/java/org/apache/kafka/common/errors/GroupNotEmptyException.java
index 264e613719c..e15b3e6d57f 100644
--- a/clients/src/main/java/org/apache/kafka/common/errors/GroupNotEmptyException.java
+++ b/clients/src/main/java/org/apache/kafka/common/errors/GroupNotEmptyException.java
@@ -17,15 +17,7 @@
package org.apache.kafka.common.errors;
public class GroupNotEmptyException extends ApiException {
- private final String groupId;
-
- public GroupNotEmptyException(String groupId) {
- super("The group " + groupId + " is not empty");
- this.groupId = groupId;
- }
-
- public String groupId() {
- return groupId;
+ public GroupNotEmptyException(String message) {
+ super(message);
}
-
}
diff --git a/clients/src/main/java/org/apache/kafka/common/errors/InvalidTxnTimeoutException.java b/clients/src/main/java/org/apache/kafka/common/errors/InvalidTxnTimeoutException.java
index c751bc4553d..f16af66da7c 100644
--- a/clients/src/main/java/org/apache/kafka/common/errors/InvalidTxnTimeoutException.java
+++ b/clients/src/main/java/org/apache/kafka/common/errors/InvalidTxnTimeoutException.java
@@ -18,7 +18,7 @@
/**
* The transaction coordinator returns this error code if the timeout received via the InitProducerIdRequest is larger than
- * the `max.transaction.timeout.ms` config value.
+ * the `transaction.max.timeout.ms` config value.
*/
public class InvalidTxnTimeoutException extends ApiException {
private static final long serialVersionUID = 1L;
diff --git a/clients/src/main/java/org/apache/kafka/common/errors/WakeupException.java b/clients/src/main/java/org/apache/kafka/common/errors/WakeupException.java
index 4726ec13cca..f8ae8403a5d 100644
--- a/clients/src/main/java/org/apache/kafka/common/errors/WakeupException.java
+++ b/clients/src/main/java/org/apache/kafka/common/errors/WakeupException.java
@@ -21,7 +21,7 @@
/**
* Exception used to indicate preemption of a blocking operation by an external thread.
* For example, {@link org.apache.kafka.clients.consumer.KafkaConsumer#wakeup}
- * can be used to break out of an active {@link org.apache.kafka.clients.consumer.KafkaConsumer#poll(long)},
+ * can be used to break out of an active {@link org.apache.kafka.clients.consumer.KafkaConsumer#poll(java.time.Duration)},
* which would raise an instance of this exception.
*/
public class WakeupException extends KafkaException {
diff --git a/clients/src/main/java/org/apache/kafka/common/internals/KafkaFutureImpl.java b/clients/src/main/java/org/apache/kafka/common/internals/KafkaFutureImpl.java
index 049ed491ab4..33916ac952a 100644
--- a/clients/src/main/java/org/apache/kafka/common/internals/KafkaFutureImpl.java
+++ b/clients/src/main/java/org/apache/kafka/common/internals/KafkaFutureImpl.java
@@ -141,11 +141,15 @@ R await(long timeout, TimeUnit unit)
*/
@Override
public <R> KafkaFuture<R> thenApply(BaseFunction<T, R> function) {
- KafkaFutureImpl<R> future = new KafkaFutureImpl<R>();
+ KafkaFutureImpl<R> future = new KafkaFutureImpl<>();
addWaiter(new Applicant<>(function, future));
return future;
}
+ public <R> void copyWith(KafkaFuture<R> future, BaseFunction<R, T> function) {
+ KafkaFutureImpl<R> futureImpl = (KafkaFutureImpl<R>) future;
+ futureImpl.addWaiter(new Applicant<>(function, this));
+ }
/**
* @See KafkaFutureImpl#thenApply(BaseFunction)
@@ -192,7 +196,7 @@ public void accept(T val, Throwable exception) {
return future;
}
- public synchronized void addWaiter(BiConsumer<? super T, ? super Throwable> action) {
+ protected synchronized void addWaiter(BiConsumer<? super T, ? super Throwable> action) {
if (exception != null) {
action.accept(null, exception);
} else if (done) {
diff --git a/clients/src/main/java/org/apache/kafka/common/metrics/JmxReporter.java b/clients/src/main/java/org/apache/kafka/common/metrics/JmxReporter.java
index 063fb3b9338..17d15bea345 100644
--- a/clients/src/main/java/org/apache/kafka/common/metrics/JmxReporter.java
+++ b/clients/src/main/java/org/apache/kafka/common/metrics/JmxReporter.java
@@ -214,20 +214,25 @@ public KafkaMetric removeAttribute(String name) {
@Override
public MBeanInfo getMBeanInfo() {
- MBeanAttributeInfo[] attrs = new MBeanAttributeInfo[metrics.size()];
- int i = 0;
- for (Map.Entry<String, KafkaMetric> entry : this.metrics.entrySet()) {
- String attribute = entry.getKey();
- KafkaMetric metric = entry.getValue();
- attrs[i] = new MBeanAttributeInfo(attribute,
- double.class.getName(),
- metric.metricName().description(),
- true,
- false,
- false);
- i += 1;
- }
- return new MBeanInfo(this.getClass().getName(), "", attrs, null, null, null);
+ return new LazyMBeanInfo(this.getClass().getName(), "", null, null, null) {
+ @Override
+ protected MBeanAttributeInfo[] buildAttributes() {
+ MBeanAttributeInfo[] attrs = new MBeanAttributeInfo[metrics.size()];
+ int i = 0;
+ for (Map.Entry<String, KafkaMetric> entry : metrics.entrySet()) {
+ String attribute = entry.getKey();
+ KafkaMetric metric = entry.getValue();
+ attrs[i] = new MBeanAttributeInfo(attribute,
+ double.class.getName(),
+ metric.metricName().description(),
+ true,
+ false,
+ false);
+ i += 1;
+ }
+ return attrs;
+ }
+ };
}
@Override
diff --git a/clients/src/main/java/org/apache/kafka/common/metrics/KafkaMetric.java b/clients/src/main/java/org/apache/kafka/common/metrics/KafkaMetric.java
index f04981aee39..48999e1057d 100644
--- a/clients/src/main/java/org/apache/kafka/common/metrics/KafkaMetric.java
+++ b/clients/src/main/java/org/apache/kafka/common/metrics/KafkaMetric.java
@@ -55,9 +55,7 @@ public MetricName metricName() {
@Override
@Deprecated
public double value() {
- synchronized (this.lock) {
- return measurableValue(time.milliseconds());
- }
+ return measurableValue(time.milliseconds());
}
@Override
@@ -81,10 +79,12 @@ public Measurable measurable() {
}
double measurableValue(long timeMs) {
- if (this.metricValueProvider instanceof Measurable)
- return ((Measurable) metricValueProvider).measure(config, timeMs);
- else
- return 0;
+ synchronized (this.lock) {
+ if (this.metricValueProvider instanceof Measurable)
+ return ((Measurable) metricValueProvider).measure(config, timeMs);
+ else
+ return 0;
+ }
}
public void config(MetricConfig config) {
diff --git a/clients/src/main/java/org/apache/kafka/common/metrics/LazyMBeanInfo.java b/clients/src/main/java/org/apache/kafka/common/metrics/LazyMBeanInfo.java
new file mode 100644
index 00000000000..382734969cc
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/metrics/LazyMBeanInfo.java
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.metrics;
+
+import javax.management.MBeanAttributeInfo;
+import javax.management.MBeanConstructorInfo;
+import javax.management.MBeanInfo;
+import javax.management.MBeanNotificationInfo;
+import javax.management.MBeanOperationInfo;
+
+
+/**
+ * an MBeanInfo subclass that lazily calculates attributes
+ */
+public abstract class LazyMBeanInfo extends MBeanInfo {
+ private volatile MBeanAttributeInfo[] lazyAttrs = null;
+
+ public LazyMBeanInfo(
+ String className,
+ String description,
+ MBeanConstructorInfo[] constructors,
+ MBeanOperationInfo[] operations,
+ MBeanNotificationInfo[] notifications
+ ) throws IllegalArgumentException {
+ super(className, description, null, constructors, operations, notifications);
+ }
+
+ @Override
+ public MBeanAttributeInfo[] getAttributes() {
+ MBeanAttributeInfo[] val = lazyAttrs;
+ if (val != null) {
+ return val.clone(); //match upstream behaviour
+ }
+ val = buildAttributes();
+ if (val == null) {
+ val = new MBeanAttributeInfo[0];
+ }
+ lazyAttrs = val;
+ return val.clone();
+ }
+
+ protected abstract MBeanAttributeInfo[] buildAttributes();
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/metrics/Metrics.java b/clients/src/main/java/org/apache/kafka/common/metrics/Metrics.java
index ea18cd3fef8..a6da9f90397 100644
--- a/clients/src/main/java/org/apache/kafka/common/metrics/Metrics.java
+++ b/clients/src/main/java/org/apache/kafka/common/metrics/Metrics.java
@@ -96,7 +96,7 @@ public Metrics(Time time) {
* Expiration of Sensors is disabled.
*/
public Metrics(MetricConfig defaultConfig, Time time) {
- this(defaultConfig, new ArrayList<MetricsReporter>(0), time);
+ this(defaultConfig, new ArrayList<MetricsReporter>(0), time);
}
@@ -487,7 +487,8 @@ public void addMetric(MetricName metricName, MetricConfig config, Measurable mea
/**
* Add a metric to monitor an object that implements MetricValueProvider. This metric won't be associated with any
- * sensor. This is a way to expose existing values as metrics.
+ * sensor. This is a way to expose existing values as metrics. User is expected to add any additional
+ * synchronization to update and access metric values, if required.
*
* @param metricName The name of the metric
* @param metricValueProvider The metric value provider associated with this metric
@@ -503,7 +504,8 @@ public void addMetric(MetricName metricName, MetricConfig config, MetricValuePro
/**
* Add a metric to monitor an object that implements MetricValueProvider. This metric won't be associated with any
- * sensor. This is a way to expose existing values as metrics.
+ * sensor. This is a way to expose existing values as metrics. User is expected to add any additional
+ * synchronization to update and access metric values, if required.
*
* @param metricName The name of the metric
* @param metricValueProvider The metric value provider associated with this metric
@@ -522,8 +524,14 @@ public void addMetric(MetricName metricName, MetricValueProvider<?> metricValueP
public synchronized KafkaMetric removeMetric(MetricName metricName) {
KafkaMetric metric = this.metrics.remove(metricName);
if (metric != null) {
- for (MetricsReporter reporter : reporters)
- reporter.metricRemoval(metric);
+ for (MetricsReporter reporter : reporters) {
+ try {
+ reporter.metricRemoval(metric);
+ } catch (Exception e) {
+ log.error("Error when removing metric from " + reporter.getClass().getName(), e);
+ }
+ }
+ log.trace("Removed metric named {}", metricName);
}
return metric;
}
@@ -550,8 +558,14 @@ synchronized void registerMetric(KafkaMetric metric) {
if (this.metrics.containsKey(metricName))
throw new IllegalArgumentException("A metric named '" + metricName + "' already exists, can't register another one.");
this.metrics.put(metricName, metric);
- for (MetricsReporter reporter : reporters)
- reporter.metricChange(metric);
+ for (MetricsReporter reporter : reporters) {
+ try {
+ reporter.metricChange(metric);
+ } catch (Exception e) {
+ log.error("Error when registering metric on " + reporter.getClass().getName(), e);
+ }
+ }
+ log.trace("Registered metric named {}", metricName);
}
/**
@@ -632,8 +646,13 @@ public void close() {
}
}
- for (MetricsReporter reporter : this.reporters)
- reporter.close();
+ for (MetricsReporter reporter : reporters) {
+ try {
+ reporter.close();
+ } catch (Exception e) {
+ log.error("Error when closing " + reporter.getClass().getName(), e);
+ }
+ }
}
}
diff --git a/clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java b/clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java
index 06c8c7f362b..e4bf1aeee69 100644
--- a/clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java
+++ b/clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java
@@ -24,8 +24,11 @@
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.LinkedList;
import java.util.List;
import java.util.Locale;
+import java.util.Map;
import java.util.Set;
import java.util.concurrent.TimeUnit;
@@ -40,7 +43,7 @@
private final String name;
private final Sensor[] parents;
private final List<Stat> stats;
- private final List<KafkaMetric> metrics;
+ private final Map<MetricName, KafkaMetric> metrics;
private final MetricConfig config;
private final Time time;
private volatile long lastRecordTime;
@@ -103,7 +106,7 @@ public boolean shouldRecord(final int configId) {
this.registry = registry;
this.name = Utils.notNull(name);
this.parents = parents == null ? new Sensor[0] : parents;
- this.metrics = new ArrayList<>();
+ this.metrics = new LinkedHashMap<>();
this.stats = new ArrayList<>();
this.config = config;
this.time = time;
@@ -190,7 +193,7 @@ public void checkQuotas() {
}
public void checkQuotas(long timeMs) {
- for (KafkaMetric metric : this.metrics) {
+ for (KafkaMetric metric : this.metrics.values()) {
MetricConfig config = metric.config();
if (config != null) {
Quota quota = config.quota();
@@ -226,11 +229,13 @@ public synchronized boolean add(CompoundStat stat, MetricConfig config) {
return false;
this.stats.add(Utils.notNull(stat));
- Object lock = new Object();
+ Object lock = metricLock(stat);
for (NamedMeasurable m : stat.stats()) {
- KafkaMetric metric = new KafkaMetric(lock, m.name(), m.stat(), config == null ? this.config : config, time);
- this.registry.registerMetric(metric);
- this.metrics.add(metric);
+ final KafkaMetric metric = new KafkaMetric(lock, m.name(), m.stat(), config == null ? this.config : config, time);
+ if (!metrics.containsKey(metric.metricName())) {
+ registry.registerMetric(metric);
+ metrics.put(metric.metricName(), metric);
+ }
}
return true;
}
@@ -247,24 +252,30 @@ public boolean add(MetricName metricName, MeasurableStat stat) {
/**
* Register a metric with this sensor
+ *
* @param metricName The name of the metric
- * @param stat The statistic to keep
- * @param config A special configuration for this metric. If null use the sensor default configuration.
+ * @param stat The statistic to keep
+ * @param config A special configuration for this metric. If null use the sensor default configuration.
* @return true if metric is added to sensor, false if sensor is expired
*/
- public synchronized boolean add(MetricName metricName, MeasurableStat stat, MetricConfig config) {
- if (hasExpired())
+ public synchronized boolean add(final MetricName metricName, final MeasurableStat stat, final MetricConfig config) {
+ if (hasExpired()) {
return false;
-
- KafkaMetric metric = new KafkaMetric(new Object(),
- Utils.notNull(metricName),
- Utils.notNull(stat),
- config == null ? this.config : config,
- time);
- this.registry.registerMetric(metric);
- this.metrics.add(metric);
- this.stats.add(stat);
- return true;
+ } else if (metrics.containsKey(metricName)) {
+ return true;
+ } else {
+ final KafkaMetric metric = new KafkaMetric(
+ metricLock(stat),
+ Utils.notNull(metricName),
+ Utils.notNull(stat),
+ config == null ? this.config : config,
+ time
+ );
+ registry.registerMetric(metric);
+ metrics.put(metric.metricName(), metric);
+ stats.add(stat);
+ return true;
+ }
}
/**
@@ -276,6 +287,14 @@ public boolean hasExpired() {
}
synchronized List<KafkaMetric> metrics() {
- return Collections.unmodifiableList(this.metrics);
+ return Collections.unmodifiableList(new LinkedList<>(this.metrics.values()));
+ }
+
+ /**
+ * KafkaMetrics of sensors which use SampledStat should be synchronized on the Sensor object
+ * to allow concurrent reads and updates. For simplicity, all sensors are synchronized on Sensor.
+ */
+ private Object metricLock(Stat stat) {
+ return this;
}
}
diff --git a/clients/src/main/java/org/apache/kafka/common/network/ChannelBuilders.java b/clients/src/main/java/org/apache/kafka/common/network/ChannelBuilders.java
index 80ccb7e1382..078d844e85f 100644
--- a/clients/src/main/java/org/apache/kafka/common/network/ChannelBuilders.java
+++ b/clients/src/main/java/org/apache/kafka/common/network/ChannelBuilders.java
@@ -26,7 +26,7 @@
import org.apache.kafka.common.security.auth.KafkaPrincipalBuilder;
import org.apache.kafka.common.security.authenticator.CredentialCache;
import org.apache.kafka.common.security.kerberos.KerberosShortNamer;
-import org.apache.kafka.common.security.token.delegation.DelegationTokenCache;
+import org.apache.kafka.common.security.token.delegation.internal.DelegationTokenCache;
import org.apache.kafka.common.utils.Utils;
import java.util.Collections;
diff --git a/clients/src/main/java/org/apache/kafka/common/network/KafkaChannel.java b/clients/src/main/java/org/apache/kafka/common/network/KafkaChannel.java
index e125bbca10e..1839729f2e7 100644
--- a/clients/src/main/java/org/apache/kafka/common/network/KafkaChannel.java
+++ b/clients/src/main/java/org/apache/kafka/common/network/KafkaChannel.java
@@ -28,6 +28,52 @@
import java.util.Objects;
public class KafkaChannel {
+ /**
+ * Mute States for KafkaChannel:
+ * <ul>
+ * <li> NOT_MUTED: Channel is not muted. This is the default state. </li>
+ * <li> MUTED: Channel is muted. Channel must be in this state to be unmuted. </li>
+ * <li> MUTED_AND_RESPONSE_PENDING: (SocketServer only) Channel is muted and SocketServer has not sent a response
+ * back to the client yet (acks != 0) or is currently waiting to receive a
+ * response from the API layer (acks == 0). </li>
+ * <li> MUTED_AND_THROTTLED: (SocketServer only) Channel is muted and throttling is in progress due to quota
+ * violation. </li>
+ * <li> MUTED_AND_THROTTLED_AND_RESPONSE_PENDING: (SocketServer only) Channel is muted, throttling is in progress,
+ * and a response is currently pending. </li>
+ * </ul>
+ */
+ public enum ChannelMuteState {
+ NOT_MUTED,
+ MUTED,
+ MUTED_AND_RESPONSE_PENDING,
+ MUTED_AND_THROTTLED,
+ MUTED_AND_THROTTLED_AND_RESPONSE_PENDING
+ };
+
+ /** Socket server events that will change the mute state:
+ * <ul>
+ * <li> REQUEST_RECEIVED: A request has been received from the client. </li>
+ * <li> RESPONSE_SENT: A response has been sent out to the client (ack != 0) or SocketServer has heard back from
+ * the API layer (acks = 0) </li>
+ * <li> THROTTLE_STARTED: Throttling started due to quota violation. </li>
+ * <li> THROTTLE_ENDED: Throttling ended. </li>
+ * </ul>
+ *
+ * Valid transitions on each event are:
+ * <ul>
+ * <li> REQUEST_RECEIVED: MUTED => MUTED_AND_RESPONSE_PENDING </li>
+ * <li> RESPONSE_SENT: MUTED_AND_RESPONSE_PENDING => MUTED, MUTED_AND_THROTTLED_AND_RESPONSE_PENDING => MUTED_AND_THROTTLED </li>
+ * <li> THROTTLE_STARTED: MUTED_AND_RESPONSE_PENDING => MUTED_AND_THROTTLED_AND_RESPONSE_PENDING </li>
+ * <li> THROTTLE_ENDED: MUTED_AND_THROTTLED => MUTED, MUTED_AND_THROTTLED_AND_RESPONSE_PENDING => MUTED_AND_RESPONSE_PENDING </li>
+ * </ul>
+ */
+ public enum ChannelMuteEvent {
+ REQUEST_RECEIVED,
+ RESPONSE_SENT,
+ THROTTLE_STARTED,
+ THROTTLE_ENDED
+ };
+
private final String id;
private final TransportLayer transportLayer;
private final Authenticator authenticator;
@@ -41,7 +87,7 @@
// Track connection and mute state of channels to enable outstanding requests on channels to be
// processed after the channel is disconnected.
private boolean disconnected;
- private boolean muted;
+ private ChannelMuteState muteState;
private ChannelState state;
public KafkaChannel(String id, TransportLayer transportLayer, Authenticator authenticator, int maxReceiveSize, MemoryPool memoryPool) throws IOException {
@@ -52,7 +98,7 @@ public KafkaChannel(String id, TransportLayer transportLayer, Authenticator auth
this.maxReceiveSize = maxReceiveSize;
this.memoryPool = memoryPool;
this.disconnected = false;
- this.muted = false;
+ this.muteState = ChannelMuteState.NOT_MUTED;
this.state = ChannelState.NOT_CONNECTED;
}
@@ -125,22 +171,76 @@ public SelectionKey selectionKey() {
* externally muting a channel should be done via selector to ensure proper state handling
*/
void mute() {
- if (!disconnected)
- transportLayer.removeInterestOps(SelectionKey.OP_READ);
- muted = true;
+ if (muteState == ChannelMuteState.NOT_MUTED) {
+ if (!disconnected) transportLayer.removeInterestOps(SelectionKey.OP_READ);
+ muteState = ChannelMuteState.MUTED;
+ }
+ }
+
+ /**
+ * Unmute the channel. The channel can be unmuted only if it is in the MUTED state. For other muted states
+ * (MUTED_AND_*), this is a no-op.
+ *
+ * @return Whether or not the channel is in the NOT_MUTED state after the call
+ */
+ boolean maybeUnmute() {
+ if (muteState == ChannelMuteState.MUTED) {
+ if (!disconnected) transportLayer.addInterestOps(SelectionKey.OP_READ);
+ muteState = ChannelMuteState.NOT_MUTED;
+ }
+ return muteState == ChannelMuteState.NOT_MUTED;
+ }
+
+ // Handle the specified channel mute-related event and transition the mute state according to the state machine.
+ public void handleChannelMuteEvent(ChannelMuteEvent event) {
+ boolean stateChanged = false;
+ switch (event) {
+ case REQUEST_RECEIVED:
+ if (muteState == ChannelMuteState.MUTED) {
+ muteState = ChannelMuteState.MUTED_AND_RESPONSE_PENDING;
+ stateChanged = true;
+ }
+ break;
+ case RESPONSE_SENT:
+ if (muteState == ChannelMuteState.MUTED_AND_RESPONSE_PENDING) {
+ muteState = ChannelMuteState.MUTED;
+ stateChanged = true;
+ }
+ if (muteState == ChannelMuteState.MUTED_AND_THROTTLED_AND_RESPONSE_PENDING) {
+ muteState = ChannelMuteState.MUTED_AND_THROTTLED;
+ stateChanged = true;
+ }
+ break;
+ case THROTTLE_STARTED:
+ if (muteState == ChannelMuteState.MUTED_AND_RESPONSE_PENDING) {
+ muteState = ChannelMuteState.MUTED_AND_THROTTLED_AND_RESPONSE_PENDING;
+ stateChanged = true;
+ }
+ break;
+ case THROTTLE_ENDED:
+ if (muteState == ChannelMuteState.MUTED_AND_THROTTLED) {
+ muteState = ChannelMuteState.MUTED;
+ stateChanged = true;
+ }
+ if (muteState == ChannelMuteState.MUTED_AND_THROTTLED_AND_RESPONSE_PENDING) {
+ muteState = ChannelMuteState.MUTED_AND_RESPONSE_PENDING;
+ stateChanged = true;
+ }
+ }
+ if (!stateChanged) {
+ throw new IllegalStateException("Cannot transition from " + muteState.name() + " for " + event.name());
+ }
}
- void unmute() {
- if (!disconnected)
- transportLayer.addInterestOps(SelectionKey.OP_READ);
- muted = false;
+ public ChannelMuteState muteState() {
+ return muteState;
}
/**
* Returns true if this channel has been explicitly muted using {@link KafkaChannel#mute()}
*/
public boolean isMute() {
- return muted;
+ return muteState != ChannelMuteState.NOT_MUTED;
}
public boolean isInMutableState() {
diff --git a/clients/src/main/java/org/apache/kafka/common/network/ListenerName.java b/clients/src/main/java/org/apache/kafka/common/network/ListenerName.java
index fc0cb14925b..2decccbc506 100644
--- a/clients/src/main/java/org/apache/kafka/common/network/ListenerName.java
+++ b/clients/src/main/java/org/apache/kafka/common/network/ListenerName.java
@@ -73,6 +73,10 @@ public String configPrefix() {
}
public String saslMechanismConfigPrefix(String saslMechanism) {
- return configPrefix() + saslMechanism.toLowerCase(Locale.ROOT) + ".";
+ return configPrefix() + saslMechanismPrefix(saslMechanism);
+ }
+
+ public static String saslMechanismPrefix(String saslMechanism) {
+ return saslMechanism.toLowerCase(Locale.ROOT) + ".";
}
}
diff --git a/clients/src/main/java/org/apache/kafka/common/network/SaslChannelBuilder.java b/clients/src/main/java/org/apache/kafka/common/network/SaslChannelBuilder.java
index 095f826ccba..15df089343d 100644
--- a/clients/src/main/java/org/apache/kafka/common/network/SaslChannelBuilder.java
+++ b/clients/src/main/java/org/apache/kafka/common/network/SaslChannelBuilder.java
@@ -21,22 +21,39 @@
import org.apache.kafka.common.config.SslConfigs;
import org.apache.kafka.common.config.internals.BrokerSecurityConfigs;
import org.apache.kafka.common.memory.MemoryPool;
-import org.apache.kafka.common.security.auth.SecurityProtocol;
import org.apache.kafka.common.security.JaasContext;
-import org.apache.kafka.common.security.token.delegation.DelegationTokenCache;
-import org.apache.kafka.common.security.kerberos.KerberosShortNamer;
+import org.apache.kafka.common.security.auth.AuthenticateCallbackHandler;
+import org.apache.kafka.common.security.auth.Login;
+import org.apache.kafka.common.security.auth.SecurityProtocol;
import org.apache.kafka.common.security.authenticator.CredentialCache;
+import org.apache.kafka.common.security.authenticator.DefaultLogin;
import org.apache.kafka.common.security.authenticator.LoginManager;
import org.apache.kafka.common.security.authenticator.SaslClientAuthenticator;
+import org.apache.kafka.common.security.authenticator.SaslClientCallbackHandler;
import org.apache.kafka.common.security.authenticator.SaslServerAuthenticator;
+import org.apache.kafka.common.security.authenticator.SaslServerCallbackHandler;
+import org.apache.kafka.common.security.kerberos.KerberosClientCallbackHandler;
+import org.apache.kafka.common.security.kerberos.KerberosLogin;
+import org.apache.kafka.common.security.kerberos.KerberosShortNamer;
+import org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule;
+import org.apache.kafka.common.security.oauthbearer.internal.OAuthBearerRefreshingLogin;
+import org.apache.kafka.common.security.oauthbearer.internal.OAuthBearerSaslClientCallbackHandler;
+import org.apache.kafka.common.security.oauthbearer.internal.unsecured.OAuthBearerUnsecuredValidatorCallbackHandler;
+import org.apache.kafka.common.security.plain.internal.PlainSaslServer;
+import org.apache.kafka.common.security.plain.internal.PlainServerCallbackHandler;
+import org.apache.kafka.common.security.scram.ScramCredential;
+import org.apache.kafka.common.security.scram.internal.ScramMechanism;
+import org.apache.kafka.common.security.scram.internal.ScramServerCallbackHandler;
import org.apache.kafka.common.security.ssl.SslFactory;
+import org.apache.kafka.common.security.token.delegation.internal.DelegationTokenCache;
import org.apache.kafka.common.utils.Java;
+import org.apache.kafka.common.utils.Utils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import java.io.IOException;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
+import java.io.IOException;
import java.net.Socket;
import java.nio.channels.SelectionKey;
import java.nio.channels.SocketChannel;
@@ -66,6 +83,7 @@
private SslFactory sslFactory;
private Map<String, ?> configs;
private KerberosShortNamer kerberosShortNamer;
+ private Map<String, AuthenticateCallbackHandler> saslCallbackHandlers;
public SaslChannelBuilder(Mode mode,
Map<String, JaasContext> jaasContexts,
@@ -87,15 +105,25 @@ public SaslChannelBuilder(Mode mode,
this.clientSaslMechanism = clientSaslMechanism;
this.credentialCache = credentialCache;
this.tokenCache = tokenCache;
+ this.saslCallbackHandlers = new HashMap<>();
}
+ @SuppressWarnings("unchecked")
@Override
public void configure(Map<String, ?> configs) throws KafkaException {
try {
this.configs = configs;
- boolean hasKerberos = jaasContexts.containsKey(SaslConfigs.GSSAPI_MECHANISM);
+ if (mode == Mode.SERVER)
+ createServerCallbackHandlers(configs);
+ else
+ createClientCallbackHandler(configs);
+ for (Map.Entry<String, AuthenticateCallbackHandler> entry : saslCallbackHandlers.entrySet()) {
+ String mechanism = entry.getKey();
+ entry.getValue().configure(configs, mechanism, jaasContexts.get(mechanism).configurationEntries());
+ }
- if (hasKerberos) {
+ Class<? extends Login> defaultLoginClass = defaultLoginClass(configs);
+ if (jaasContexts.containsKey(SaslConfigs.GSSAPI_MECHANISM)) {
String defaultRealm;
try {
defaultRealm = defaultKerberosRealm();
@@ -111,7 +139,7 @@ public void configure(Map<String, ?> configs) throws KafkaException {
String mechanism = entry.getKey();
// With static JAAS configuration, use KerberosLogin if Kerberos is enabled. With dynamic JAAS configuration,
// use KerberosLogin only for the LoginContext corresponding to GSSAPI
- LoginManager loginManager = LoginManager.acquireLoginManager(entry.getValue(), mechanism, hasKerberos, configs);
+ LoginManager loginManager = LoginManager.acquireLoginManager(entry.getValue(), mechanism, defaultLoginClass, configs);
loginManagers.put(mechanism, loginManager);
subjects.put(mechanism, loginManager.subject());
}
@@ -120,7 +148,7 @@ public void configure(Map<String, ?> configs) throws KafkaException {
this.sslFactory = new SslFactory(mode, "none", isInterBrokerListener);
this.sslFactory.configure(configs);
}
- } catch (Exception e) {
+ } catch (Throwable e) {
close();
throw new KafkaException(e);
}
@@ -156,11 +184,20 @@ public KafkaChannel buildChannel(String id, SelectionKey key, int maxReceiveSize
TransportLayer transportLayer = buildTransportLayer(id, key, socketChannel);
Authenticator authenticator;
if (mode == Mode.SERVER) {
- authenticator = buildServerAuthenticator(configs, id, transportLayer, subjects);
+ authenticator = buildServerAuthenticator(configs,
+ saslCallbackHandlers,
+ id,
+ transportLayer,
+ subjects);
} else {
LoginManager loginManager = loginManagers.get(clientSaslMechanism);
- authenticator = buildClientAuthenticator(configs, id, socket.getInetAddress().getHostName(),
- loginManager.serviceName(), transportLayer, loginManager.subject());
+ authenticator = buildClientAuthenticator(configs,
+ saslCallbackHandlers.get(clientSaslMechanism),
+ id,
+ socket.getInetAddress().getHostName(),
+ loginManager.serviceName(),
+ transportLayer,
+ subjects.get(clientSaslMechanism));
}
return new KafkaChannel(id, transportLayer, authenticator, maxReceiveSize, memoryPool != null ? memoryPool : MemoryPool.NONE);
} catch (Exception e) {
@@ -174,6 +211,8 @@ public void close() {
for (LoginManager loginManager : loginManagers.values())
loginManager.release();
loginManagers.clear();
+ for (AuthenticateCallbackHandler handler : saslCallbackHandlers.values())
+ handler.close();
}
private TransportLayer buildTransportLayer(String id, SelectionKey key, SocketChannel socketChannel) throws IOException {
@@ -186,16 +225,23 @@ private TransportLayer buildTransportLayer(String id, SelectionKey key, SocketCh
}
// Visible to override for testing
- protected SaslServerAuthenticator buildServerAuthenticator(Map<String, ?> configs, String id,
- TransportLayer transportLayer, Map<String, Subject> subjects) throws IOException {
- return new SaslServerAuthenticator(configs, id, jaasContexts, subjects,
- kerberosShortNamer, credentialCache, listenerName, securityProtocol, transportLayer, tokenCache);
+ protected SaslServerAuthenticator buildServerAuthenticator(Map<String, ?> configs,
+ Map<String, AuthenticateCallbackHandler> callbackHandlers,
+ String id,
+ TransportLayer transportLayer,
+ Map<String, Subject> subjects) throws IOException {
+ return new SaslServerAuthenticator(configs, callbackHandlers, id, subjects,
+ kerberosShortNamer, listenerName, securityProtocol, transportLayer);
}
// Visible to override for testing
- protected SaslClientAuthenticator buildClientAuthenticator(Map<String, ?> configs, String id,
- String serverHost, String servicePrincipal, TransportLayer transportLayer, Subject subject) throws IOException {
- return new SaslClientAuthenticator(configs, id, subject, servicePrincipal,
+ protected SaslClientAuthenticator buildClientAuthenticator(Map<String, ?> configs,
+ AuthenticateCallbackHandler callbackHandler,
+ String id,
+ String serverHost,
+ String servicePrincipal,
+ TransportLayer transportLayer, Subject subject) throws IOException {
+ return new SaslClientAuthenticator(configs, callbackHandler, id, subject, servicePrincipal,
serverHost, clientSaslMechanism, handshakeRequestEnable, transportLayer);
}
@@ -224,4 +270,51 @@ private static String defaultKerberosRealm() throws ClassNotFoundException, NoSu
getDefaultRealmMethod = classRef.getDeclaredMethod("getDefaultRealm", new Class[0]);
return (String) getDefaultRealmMethod.invoke(kerbConf, new Object[0]);
}
+
+ private void createClientCallbackHandler(Map<String, ?> configs) {
+ Class<? extends AuthenticateCallbackHandler> clazz = (Class<? extends AuthenticateCallbackHandler>) configs.get(SaslConfigs.SASL_CLIENT_CALLBACK_HANDLER_CLASS);
+ if (clazz == null)
+ clazz = clientCallbackHandlerClass();
+ AuthenticateCallbackHandler callbackHandler = Utils.newInstance(clazz);
+ saslCallbackHandlers.put(clientSaslMechanism, callbackHandler);
+ }
+
+ private void createServerCallbackHandlers(Map<String, ?> configs) throws ClassNotFoundException {
+ for (String mechanism : jaasContexts.keySet()) {
+ AuthenticateCallbackHandler callbackHandler;
+ String prefix = ListenerName.saslMechanismPrefix(mechanism);
+ Class<? extends AuthenticateCallbackHandler> clazz =
+ (Class<? extends AuthenticateCallbackHandler>) configs.get(prefix + BrokerSecurityConfigs.SASL_SERVER_CALLBACK_HANDLER_CLASS);
+ if (clazz != null)
+ callbackHandler = Utils.newInstance(clazz);
+ else if (mechanism.equals(PlainSaslServer.PLAIN_MECHANISM))
+ callbackHandler = new PlainServerCallbackHandler();
+ else if (ScramMechanism.isScram(mechanism))
+ callbackHandler = new ScramServerCallbackHandler(credentialCache.cache(mechanism, ScramCredential.class), tokenCache);
+ else if (mechanism.equals(OAuthBearerLoginModule.OAUTHBEARER_MECHANISM))
+ callbackHandler = new OAuthBearerUnsecuredValidatorCallbackHandler();
+ else
+ callbackHandler = new SaslServerCallbackHandler();
+ saslCallbackHandlers.put(mechanism, callbackHandler);
+ }
+ }
+
+ private Class<? extends Login> defaultLoginClass(Map<String, ?> configs) {
+ if (jaasContexts.containsKey(SaslConfigs.GSSAPI_MECHANISM))
+ return KerberosLogin.class;
+ if (OAuthBearerLoginModule.OAUTHBEARER_MECHANISM.equals(clientSaslMechanism))
+ return OAuthBearerRefreshingLogin.class;
+ return DefaultLogin.class;
+ }
+
+ private Class<? extends AuthenticateCallbackHandler> clientCallbackHandlerClass() {
+ switch (clientSaslMechanism) {
+ case SaslConfigs.GSSAPI_MECHANISM:
+ return KerberosClientCallbackHandler.class;
+ case OAuthBearerLoginModule.OAUTHBEARER_MECHANISM:
+ return OAuthBearerSaslClientCallbackHandler.class;
+ default:
+ return SaslClientCallbackHandler.class;
+ }
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/network/Selector.java b/clients/src/main/java/org/apache/kafka/common/network/Selector.java
index 1b2d1a21bc0..334ca79f035 100644
--- a/clients/src/main/java/org/apache/kafka/common/network/Selector.java
+++ b/clients/src/main/java/org/apache/kafka/common/network/Selector.java
@@ -397,7 +397,7 @@ public void poll(long timeout) throws IOException {
log.trace("Broker no longer low on memory - unmuting incoming sockets");
for (KafkaChannel channel : channels.values()) {
if (channel.isInMutableState() && !explicitlyMutedChannels.contains(channel)) {
- channel.unmute();
+ channel.maybeUnmute();
}
}
outOfMemory = false;
@@ -610,8 +610,10 @@ public void unmute(String id) {
}
private void unmute(KafkaChannel channel) {
- explicitlyMutedChannels.remove(channel);
- channel.unmute();
+ // Remove the channel from explicitlyMutedChannels only if the channel has been actually unmuted.
+ if (channel.maybeUnmute()) {
+ explicitlyMutedChannels.remove(channel);
+ }
}
@Override
diff --git a/clients/src/main/java/org/apache/kafka/common/protocol/CommonFields.java b/clients/src/main/java/org/apache/kafka/common/protocol/CommonFields.java
index a436dff1d03..7f43caf8696 100644
--- a/clients/src/main/java/org/apache/kafka/common/protocol/CommonFields.java
+++ b/clients/src/main/java/org/apache/kafka/common/protocol/CommonFields.java
@@ -26,6 +26,7 @@
public static final Field.Int32 PARTITION_ID = new Field.Int32("partition", "Topic partition id");
public static final Field.Int16 ERROR_CODE = new Field.Int16("error_code", "Response error code");
public static final Field.NullableStr ERROR_MESSAGE = new Field.NullableStr("error_message", "Response error message");
+ public static final Field.Int32 LEADER_EPOCH = new Field.Int32("leader_epoch", "The epoch");
// Group APIs
public static final Field.Str GROUP_ID = new Field.Str("group_id", "The unique group identifier");
diff --git a/clients/src/main/java/org/apache/kafka/common/protocol/Errors.java b/clients/src/main/java/org/apache/kafka/common/protocol/Errors.java
index 941ce0b9e3d..5db1d314be3 100644
--- a/clients/src/main/java/org/apache/kafka/common/protocol/Errors.java
+++ b/clients/src/main/java/org/apache/kafka/common/protocol/Errors.java
@@ -474,7 +474,7 @@ public ApiException build(String message) {
}
}),
INVALID_TRANSACTION_TIMEOUT(50, "The transaction timeout is larger than the maximum value allowed by " +
- "the broker (as configured by max.transaction.timeout.ms).",
+ "the broker (as configured by transaction.max.timeout.ms).",
new ApiExceptionBuilder() {
@Override
public ApiException build(String message) {
diff --git a/clients/src/main/java/org/apache/kafka/common/protocol/types/ArrayOf.java b/clients/src/main/java/org/apache/kafka/common/protocol/types/ArrayOf.java
index 4213ecd08f0..6609dfd5196 100644
--- a/clients/src/main/java/org/apache/kafka/common/protocol/types/ArrayOf.java
+++ b/clients/src/main/java/org/apache/kafka/common/protocol/types/ArrayOf.java
@@ -16,12 +16,16 @@
*/
package org.apache.kafka.common.protocol.types;
+import org.apache.kafka.common.protocol.types.Type.DocumentedType;
+
import java.nio.ByteBuffer;
/**
* Represents a type for an array of a particular type
*/
-public class ArrayOf extends Type {
+public class ArrayOf extends DocumentedType {
+
+ private static final String ARRAY_TYPE_NAME = "ARRAY";
private final Type type;
private final boolean nullable;
@@ -93,7 +97,7 @@ public Type type() {
@Override
public String toString() {
- return "ARRAY(" + type + ")";
+ return ARRAY_TYPE_NAME + "(" + type + ")";
}
@Override
@@ -110,4 +114,18 @@ public String toString() {
throw new SchemaException("Not an Object[].");
}
}
+
+ @Override
+ public String typeName() {
+ return ARRAY_TYPE_NAME;
+ }
+
+ @Override
+ public String documentation() {
+ return "Represents a sequence of objects of a given type T. " +
+ "Type T can be either a primitive type (e.g. " + STRING + ") or a structure. " +
+ "First, the length N is given as an " + INT32 + ". Then N instances of type T follow. " +
+ "A null array is represented with a length of -1. " +
+ "In protocol documentation an array of T instances is referred to as [T].";
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/protocol/types/Schema.java b/clients/src/main/java/org/apache/kafka/common/protocol/types/Schema.java
index faa1540b499..cbcd4491d37 100644
--- a/clients/src/main/java/org/apache/kafka/common/protocol/types/Schema.java
+++ b/clients/src/main/java/org/apache/kafka/common/protocol/types/Schema.java
@@ -193,6 +193,4 @@ public void visit(Schema schema) {}
public void visit(ArrayOf array) {}
public void visit(Type field) {}
}
-
-
}
diff --git a/clients/src/main/java/org/apache/kafka/common/protocol/types/Struct.java b/clients/src/main/java/org/apache/kafka/common/protocol/types/Struct.java
index ac24a1b69b2..7dccc1015aa 100644
--- a/clients/src/main/java/org/apache/kafka/common/protocol/types/Struct.java
+++ b/clients/src/main/java/org/apache/kafka/common/protocol/types/Struct.java
@@ -16,7 +16,7 @@
*/
package org.apache.kafka.common.protocol.types;
-import org.apache.kafka.common.record.Records;
+import org.apache.kafka.common.record.BaseRecords;
import java.nio.ByteBuffer;
import java.util.Arrays;
@@ -172,8 +172,8 @@ public byte getByte(String name) {
return (Byte) get(name);
}
- public Records getRecords(String name) {
- return (Records) get(name);
+ public BaseRecords getRecords(String name) {
+ return (BaseRecords) get(name);
}
public Short getShort(BoundField field) {
diff --git a/clients/src/main/java/org/apache/kafka/common/protocol/types/Type.java b/clients/src/main/java/org/apache/kafka/common/protocol/types/Type.java
index 57d31f459fb..4bd508b7798 100644
--- a/clients/src/main/java/org/apache/kafka/common/protocol/types/Type.java
+++ b/clients/src/main/java/org/apache/kafka/common/protocol/types/Type.java
@@ -16,7 +16,7 @@
*/
package org.apache.kafka.common.protocol.types;
-import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.BaseRecords;
import org.apache.kafka.common.record.MemoryRecords;
import org.apache.kafka.common.record.Records;
import org.apache.kafka.common.utils.ByteUtils;
@@ -63,6 +63,29 @@ public boolean isNullable() {
return false;
}
+ /**
+ * A Type that can return its description for documentation purposes.
+ */
+ public static abstract class DocumentedType extends Type {
+
+ /**
+ * Short name of the type to identify it in documentation;
+ * @return the name of the type
+ */
+ public abstract String typeName();
+
+ /**
+ * Documentation of the Type.
+ *
+ * @return details about valid values, representation
+ */
+ public abstract String documentation();
+
+ @Override
+ public String toString() {
+ return typeName();
+ }
+ }
/**
* The Boolean type represents a boolean value in a byte by using
* the value of 0 to represent false, and 1 to represent true.
@@ -70,7 +93,7 @@ public boolean isNullable() {
* If for some reason a value that is not 0 or 1 is read,
* then any non-zero value will return true.
*/
- public static final Type BOOLEAN = new Type() {
+ public static final DocumentedType BOOLEAN = new DocumentedType() {
@Override
public void write(ByteBuffer buffer, Object o) {
if ((Boolean) o)
@@ -91,7 +114,7 @@ public int sizeOf(Object o) {
}
@Override
- public String toString() {
+ public String typeName() {
return "BOOLEAN";
}
@@ -102,9 +125,16 @@ public Boolean validate(Object item) {
else
throw new SchemaException(item + " is not a Boolean.");
}
+
+ @Override
+ public String documentation() {
+ return "Represents a boolean value in a byte. " +
+ "Values 0 and 1 are used to represent false and true respectively. " +
+ "When reading a boolean value, any non-zero value is considered true.";
+ }
};
- public static final Type INT8 = new Type() {
+ public static final DocumentedType INT8 = new DocumentedType() {
@Override
public void write(ByteBuffer buffer, Object o) {
buffer.put((Byte) o);
@@ -121,7 +151,7 @@ public int sizeOf(Object o) {
}
@Override
- public String toString() {
+ public String typeName() {
return "INT8";
}
@@ -132,9 +162,14 @@ public Byte validate(Object item) {
else
throw new SchemaException(item + " is not a Byte.");
}
+
+ @Override
+ public String documentation() {
+ return "Represents an integer between -2<sup>7</sup> and 2<sup>7</sup>-1 inclusive.";
+ }
};
- public static final Type INT16 = new Type() {
+ public static final DocumentedType INT16 = new DocumentedType() {
@Override
public void write(ByteBuffer buffer, Object o) {
buffer.putShort((Short) o);
@@ -151,7 +186,7 @@ public int sizeOf(Object o) {
}
@Override
- public String toString() {
+ public String typeName() {
return "INT16";
}
@@ -162,9 +197,15 @@ public Short validate(Object item) {
else
throw new SchemaException(item + " is not a Short.");
}
+
+ @Override
+ public String documentation() {
+ return "Represents an integer between -2<sup>15</sup> and 2<sup>15</sup>-1 inclusive. " +
+ "The values are encoded using two bytes in network byte order (big-endian).";
+ }
};
- public static final Type INT32 = new Type() {
+ public static final DocumentedType INT32 = new DocumentedType() {
@Override
public void write(ByteBuffer buffer, Object o) {
buffer.putInt((Integer) o);
@@ -181,7 +222,7 @@ public int sizeOf(Object o) {
}
@Override
- public String toString() {
+ public String typeName() {
return "INT32";
}
@@ -192,9 +233,15 @@ public Integer validate(Object item) {
else
throw new SchemaException(item + " is not an Integer.");
}
+
+ @Override
+ public String documentation() {
+ return "Represents an integer between -2<sup>31</sup> and 2<sup>31</sup>-1 inclusive. " +
+ "The values are encoded using four bytes in network byte order (big-endian).";
+ }
};
- public static final Type UNSIGNED_INT32 = new Type() {
+ public static final DocumentedType UNSIGNED_INT32 = new DocumentedType() {
@Override
public void write(ByteBuffer buffer, Object o) {
ByteUtils.writeUnsignedInt(buffer, (long) o);
@@ -211,7 +258,7 @@ public int sizeOf(Object o) {
}
@Override
- public String toString() {
+ public String typeName() {
return "UINT32";
}
@@ -222,9 +269,15 @@ public Long validate(Object item) {
else
throw new SchemaException(item + " is not a Long.");
}
+
+ @Override
+ public String documentation() {
+ return "Represents an integer between 0 and 2<sup>32</sup>-1 inclusive. " +
+ "The values are encoded using four bytes in network byte order (big-endian).";
+ }
};
- public static final Type INT64 = new Type() {
+ public static final DocumentedType INT64 = new DocumentedType() {
@Override
public void write(ByteBuffer buffer, Object o) {
buffer.putLong((Long) o);
@@ -241,7 +294,7 @@ public int sizeOf(Object o) {
}
@Override
- public String toString() {
+ public String typeName() {
return "INT64";
}
@@ -252,9 +305,15 @@ public Long validate(Object item) {
else
throw new SchemaException(item + " is not a Long.");
}
+
+ @Override
+ public String documentation() {
+ return "Represents an integer between -2<sup>63</sup> and 2<sup>63</sup>-1 inclusive. " +
+ "The values are encoded using eight bytes in network byte order (big-endian).";
+ }
};
- public static final Type STRING = new Type() {
+ public static final DocumentedType STRING = new DocumentedType() {
@Override
public void write(ByteBuffer buffer, Object o) {
byte[] bytes = Utils.utf8((String) o);
@@ -282,7 +341,7 @@ public int sizeOf(Object o) {
}
@Override
- public String toString() {
+ public String typeName() {
return "STRING";
}
@@ -293,9 +352,16 @@ public String validate(Object item) {
else
throw new SchemaException(item + " is not a String.");
}
+
+ @Override
+ public String documentation() {
+ return "Represents a sequence of characters. First the length N is given as an " + INT16 +
+ ". Then N bytes follow which are the UTF-8 encoding of the character sequence. " +
+ "Length must not be negative.";
+ }
};
- public static final Type NULLABLE_STRING = new Type() {
+ public static final DocumentedType NULLABLE_STRING = new DocumentedType() {
@Override
public boolean isNullable() {
return true;
@@ -336,7 +402,7 @@ public int sizeOf(Object o) {
}
@Override
- public String toString() {
+ public String typeName() {
return "NULLABLE_STRING";
}
@@ -350,9 +416,16 @@ public String validate(Object item) {
else
throw new SchemaException(item + " is not a String.");
}
+
+ @Override
+ public String documentation() {
+ return "Represents a sequence of characters or null. For non-null strings, first the length N is given as an " + INT16 +
+ ". Then N bytes follow which are the UTF-8 encoding of the character sequence. " +
+ "A null value is encoded with length of -1 and there are no following bytes.";
+ }
};
- public static final Type BYTES = new Type() {
+ public static final DocumentedType BYTES = new DocumentedType() {
@Override
public void write(ByteBuffer buffer, Object o) {
ByteBuffer arg = (ByteBuffer) o;
@@ -383,7 +456,7 @@ public int sizeOf(Object o) {
}
@Override
- public String toString() {
+ public String typeName() {
return "BYTES";
}
@@ -394,9 +467,15 @@ public ByteBuffer validate(Object item) {
else
throw new SchemaException(item + " is not a java.nio.ByteBuffer.");
}
+
+ @Override
+ public String documentation() {
+ return "Represents a raw sequence of bytes. First the length N is given as an " + INT32 +
+ ". Then N bytes follow.";
+ }
};
- public static final Type NULLABLE_BYTES = new Type() {
+ public static final DocumentedType NULLABLE_BYTES = new DocumentedType() {
@Override
public boolean isNullable() {
return true;
@@ -440,7 +519,7 @@ public int sizeOf(Object o) {
}
@Override
- public String toString() {
+ public String typeName() {
return "NULLABLE_BYTES";
}
@@ -454,9 +533,15 @@ public ByteBuffer validate(Object item) {
throw new SchemaException(item + " is not a java.nio.ByteBuffer.");
}
+
+ @Override
+ public String documentation() {
+ return "Represents a raw sequence of bytes or null. For non-null values, first the length N is given as an " + INT32 +
+ ". Then N bytes follow. A null value is encoded with length of -1 and there are no following bytes.";
+ }
};
- public static final Type RECORDS = new Type() {
+ public static final DocumentedType RECORDS = new DocumentedType() {
@Override
public boolean isNullable() {
return true;
@@ -464,14 +549,14 @@ public boolean isNullable() {
@Override
public void write(ByteBuffer buffer, Object o) {
- if (o instanceof FileRecords)
- throw new IllegalArgumentException("FileRecords must be written to the channel directly");
+ if (!(o instanceof MemoryRecords))
+ throw new IllegalArgumentException("Unexpected record type: " + o.getClass());
MemoryRecords records = (MemoryRecords) o;
NULLABLE_BYTES.write(buffer, records.buffer().duplicate());
}
@Override
- public Records read(ByteBuffer buffer) {
+ public MemoryRecords read(ByteBuffer buffer) {
ByteBuffer recordsBuffer = (ByteBuffer) NULLABLE_BYTES.read(buffer);
return MemoryRecords.readableRecords(recordsBuffer);
}
@@ -481,28 +566,35 @@ public int sizeOf(Object o) {
if (o == null)
return 4;
- Records records = (Records) o;
+ BaseRecords records = (BaseRecords) o;
return 4 + records.sizeInBytes();
}
@Override
- public String toString() {
+ public String typeName() {
return "RECORDS";
}
@Override
- public Records validate(Object item) {
+ public BaseRecords validate(Object item) {
if (item == null)
return null;
- if (item instanceof Records)
- return (Records) item;
+ if (item instanceof BaseRecords)
+ return (BaseRecords) item;
throw new SchemaException(item + " is not an instance of " + Records.class.getName());
}
+
+ @Override
+ public String documentation() {
+ return "Represents a sequence of Kafka records as " + NULLABLE_BYTES + ". " +
+ "For a detailed description of records see " +
+ "<a href=\"/documentation/#messageformat\">Message Sets</a>.";
+ }
};
- public static final Type VARINT = new Type() {
+ public static final DocumentedType VARINT = new DocumentedType() {
@Override
public void write(ByteBuffer buffer, Object o) {
ByteUtils.writeVarint((Integer) o, buffer);
@@ -520,7 +612,7 @@ public Integer validate(Object item) {
throw new SchemaException(item + " is not an integer");
}
- public String toString() {
+ public String typeName() {
return "VARINT";
}
@@ -528,9 +620,16 @@ public String toString() {
public int sizeOf(Object o) {
return ByteUtils.sizeOfVarint((Integer) o);
}
+
+ @Override
+ public String documentation() {
+ return "Represents an integer between -2<sup>31</sup> and 2<sup>31</sup>-1 inclusive. " +
+ "Encoding follows the variable-length zig-zag encoding from " +
+ " <a href=\"http://code.google.com/apis/protocolbuffers/docs/encoding.html\"> Google Protocol Buffers</a>.";
+ }
};
- public static final Type VARLONG = new Type() {
+ public static final DocumentedType VARLONG = new DocumentedType() {
@Override
public void write(ByteBuffer buffer, Object o) {
ByteUtils.writeVarlong((Long) o, buffer);
@@ -548,7 +647,7 @@ public Long validate(Object item) {
throw new SchemaException(item + " is not a long");
}
- public String toString() {
+ public String typeName() {
return "VARLONG";
}
@@ -556,6 +655,43 @@ public String toString() {
public int sizeOf(Object o) {
return ByteUtils.sizeOfVarlong((Long) o);
}
+
+ @Override
+ public String documentation() {
+ return "Represents an integer between -2<sup>63</sup> and 2<sup>63</sup>-1 inclusive. " +
+ "Encoding follows the variable-length zig-zag encoding from " +
+ " <a href=\"http://code.google.com/apis/protocolbuffers/docs/encoding.html\"> Google Protocol Buffers</a>.";
+ }
};
+ private static String toHtml() {
+
+ DocumentedType[] types = {
+ BOOLEAN, INT8, INT16, INT32, INT64,
+ UNSIGNED_INT32, VARINT, VARLONG,
+ STRING, NULLABLE_STRING, BYTES, NULLABLE_BYTES,
+ RECORDS, new ArrayOf(STRING)};
+ final StringBuilder b = new StringBuilder();
+ b.append("<table class=\"data-table\"><tbody>\n");
+ b.append("<tr>");
+ b.append("<th>Type</th>\n");
+ b.append("<th>Description</th>\n");
+ b.append("</tr>\n");
+ for (DocumentedType type : types) {
+ b.append("<tr>");
+ b.append("<td>");
+ b.append(type.typeName());
+ b.append("</td>");
+ b.append("<td>");
+ b.append(type.documentation());
+ b.append("</td>");
+ b.append("</tr>\n");
+ }
+ b.append("</table>\n");
+ return b.toString();
+ }
+
+ public static void main(String[] args) {
+ System.out.println(toHtml());
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/record/AbstractRecords.java b/clients/src/main/java/org/apache/kafka/common/record/AbstractRecords.java
index 2452798d485..5e41901fb88 100644
--- a/clients/src/main/java/org/apache/kafka/common/record/AbstractRecords.java
+++ b/clients/src/main/java/org/apache/kafka/common/record/AbstractRecords.java
@@ -18,13 +18,10 @@
import org.apache.kafka.common.header.Header;
import org.apache.kafka.common.utils.AbstractIterator;
-import org.apache.kafka.common.utils.Time;
import org.apache.kafka.common.utils.Utils;
import java.nio.ByteBuffer;
-import java.util.ArrayList;
import java.util.Iterator;
-import java.util.List;
public abstract class AbstractRecords implements Records {
@@ -51,92 +48,6 @@ public boolean hasCompatibleMagic(byte magic) {
return true;
}
- /**
- * Down convert batches to the provided message format version. The first offset parameter is only relevant in the
- * conversion from uncompressed v2 or higher to v1 or lower. The reason is that uncompressed records in v0 and v1
- * are not batched (put another way, each batch always has 1 record).
- *
- * If a client requests records in v1 format starting from the middle of an uncompressed batch in v2 format, we
- * need to drop records from the batch during the conversion. Some versions of librdkafka rely on this for
- * correctness.
- *
- * The temporaryMemoryBytes computation assumes that the batches are not loaded into the heap
- * (via classes like FileChannelRecordBatch) before this method is called. This is the case in the broker (we
- * only load records into the heap when down converting), but it's not for the producer. However, down converting
- * in the producer is very uncommon and the extra complexity to handle that case is not worth it.
- */
- protected ConvertedRecords<MemoryRecords> downConvert(Iterable<? extends RecordBatch> batches, byte toMagic,
- long firstOffset, Time time) {
- // maintain the batch along with the decompressed records to avoid the need to decompress again
- List<RecordBatchAndRecords> recordBatchAndRecordsList = new ArrayList<>();
- int totalSizeEstimate = 0;
- long startNanos = time.nanoseconds();
-
- for (RecordBatch batch : batches) {
- if (toMagic < RecordBatch.MAGIC_VALUE_V2 && batch.isControlBatch())
- continue;
-
- if (batch.magic() <= toMagic) {
- totalSizeEstimate += batch.sizeInBytes();
- recordBatchAndRecordsList.add(new RecordBatchAndRecords(batch, null, null));
- } else {
- List<Record> records = new ArrayList<>();
- for (Record record : batch) {
- // See the method javadoc for an explanation
- if (toMagic > RecordBatch.MAGIC_VALUE_V1 || batch.isCompressed() || record.offset() >= firstOffset)
- records.add(record);
- }
- if (records.isEmpty())
- continue;
- final long baseOffset;
- if (batch.magic() >= RecordBatch.MAGIC_VALUE_V2 && toMagic >= RecordBatch.MAGIC_VALUE_V2)
- baseOffset = batch.baseOffset();
- else
- baseOffset = records.get(0).offset();
- totalSizeEstimate += estimateSizeInBytes(toMagic, baseOffset, batch.compressionType(), records);
- recordBatchAndRecordsList.add(new RecordBatchAndRecords(batch, records, baseOffset));
- }
- }
-
- ByteBuffer buffer = ByteBuffer.allocate(totalSizeEstimate);
- long temporaryMemoryBytes = 0;
- int numRecordsConverted = 0;
- for (RecordBatchAndRecords recordBatchAndRecords : recordBatchAndRecordsList) {
- temporaryMemoryBytes += recordBatchAndRecords.batch.sizeInBytes();
- if (recordBatchAndRecords.batch.magic() <= toMagic) {
- recordBatchAndRecords.batch.writeTo(buffer);
- } else {
- MemoryRecordsBuilder builder = convertRecordBatch(toMagic, buffer, recordBatchAndRecords);
- buffer = builder.buffer();
- temporaryMemoryBytes += builder.uncompressedBytesWritten();
- numRecordsConverted += builder.numRecords();
- }
- }
-
- buffer.flip();
- RecordsProcessingStats stats = new RecordsProcessingStats(temporaryMemoryBytes, numRecordsConverted,
- time.nanoseconds() - startNanos);
- return new ConvertedRecords<>(MemoryRecords.readableRecords(buffer), stats);
- }
-
- /**
- * Return a buffer containing the converted record batches. The returned buffer may not be the same as the received
- * one (e.g. it may require expansion).
- */
- private MemoryRecordsBuilder convertRecordBatch(byte magic, ByteBuffer buffer, RecordBatchAndRecords recordBatchAndRecords) {
- RecordBatch batch = recordBatchAndRecords.batch;
- final TimestampType timestampType = batch.timestampType();
- long logAppendTime = timestampType == TimestampType.LOG_APPEND_TIME ? batch.maxTimestamp() : RecordBatch.NO_TIMESTAMP;
-
- MemoryRecordsBuilder builder = MemoryRecords.builder(buffer, magic, batch.compressionType(),
- timestampType, recordBatchAndRecords.baseOffset, logAppendTime);
- for (Record record : recordBatchAndRecords.records)
- builder.append(record);
-
- builder.close();
- return builder;
- }
-
/**
* Get an iterator over the deep records.
* @return An iterator over the records
@@ -146,6 +57,11 @@ private MemoryRecordsBuilder convertRecordBatch(byte magic, ByteBuffer buffer, R
return records;
}
+ @Override
+ public RecordsSend toSend(String destination) {
+ return new DefaultRecordsSend(destination, this);
+ }
+
private Iterator<Record> recordsIterator() {
return new AbstractIterator<Record>() {
private final Iterator<? extends RecordBatch> batches = batches().iterator();
@@ -236,16 +152,5 @@ public static int recordBatchHeaderSizeInBytes(byte magic, CompressionType compr
}
}
- private static class RecordBatchAndRecords {
- private final RecordBatch batch;
- private final List<Record> records;
- private final Long baseOffset;
-
- private RecordBatchAndRecords(RecordBatch batch, List<Record> records, Long baseOffset) {
- this.batch = batch;
- this.records = records;
- this.baseOffset = baseOffset;
- }
- }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/record/RecordFormat.java b/clients/src/main/java/org/apache/kafka/common/record/BaseRecords.java
similarity index 62%
rename from clients/src/main/java/org/apache/kafka/common/record/RecordFormat.java
rename to clients/src/main/java/org/apache/kafka/common/record/BaseRecords.java
index e71ec599f41..3ebaf792a8e 100644
--- a/clients/src/main/java/org/apache/kafka/common/record/RecordFormat.java
+++ b/clients/src/main/java/org/apache/kafka/common/record/BaseRecords.java
@@ -16,26 +16,19 @@
*/
package org.apache.kafka.common.record;
-public enum RecordFormat {
- V0(0), V1(1), V2(2);
-
- public final byte value;
-
- RecordFormat(int value) {
- this.value = (byte) value;
- }
-
- public static RecordFormat lookup(byte version) {
- switch (version) {
- case 0: return V0;
- case 1: return V1;
- case 2: return V2;
- default: throw new IllegalArgumentException("Unknown format version: " + version);
- }
- }
-
- public static RecordFormat current() {
- return V2;
- }
+/**
+ * Base interface for accessing records which could be contained in the log, or an in-memory materialization of log records.
+ */
+public interface BaseRecords {
+ /**
+ * The size of these records in bytes.
+ * @return The size in bytes of the records
+ */
+ int sizeInBytes();
+ /**
+ * Encapsulate this {@link BaseRecords} object into {@link RecordsSend}
+ * @return Initialized {@link RecordsSend} object
+ */
+ RecordsSend toSend(String destination);
}
diff --git a/clients/src/main/java/org/apache/kafka/common/record/ByteBufferLogInputStream.java b/clients/src/main/java/org/apache/kafka/common/record/ByteBufferLogInputStream.java
index bd54e40a457..7f91f266158 100644
--- a/clients/src/main/java/org/apache/kafka/common/record/ByteBufferLogInputStream.java
+++ b/clients/src/main/java/org/apache/kafka/common/record/ByteBufferLogInputStream.java
@@ -21,6 +21,7 @@
import java.io.IOException;
import java.nio.ByteBuffer;
+import static org.apache.kafka.common.record.Records.HEADER_SIZE_UP_TO_MAGIC;
import static org.apache.kafka.common.record.Records.LOG_OVERHEAD;
import static org.apache.kafka.common.record.Records.MAGIC_OFFSET;
import static org.apache.kafka.common.record.Records.SIZE_OFFSET;
@@ -40,18 +41,9 @@
public MutableRecordBatch nextBatch() throws IOException {
int remaining = buffer.remaining();
- if (remaining < LOG_OVERHEAD)
- return null;
-
- int recordSize = buffer.getInt(buffer.position() + SIZE_OFFSET);
- // V0 has the smallest overhead, stricter checking is done later
- if (recordSize < LegacyRecord.RECORD_OVERHEAD_V0)
- throw new CorruptRecordException(String.format("Record size is less than the minimum record overhead (%d)", LegacyRecord.RECORD_OVERHEAD_V0));
- if (recordSize > maxMessageSize)
- throw new CorruptRecordException(String.format("Record size exceeds the largest allowable message size (%d).", maxMessageSize));
- int batchSize = recordSize + LOG_OVERHEAD;
- if (remaining < batchSize)
+ Integer batchSize = nextBatchSize();
+ if (batchSize == null || remaining < batchSize)
return null;
byte magic = buffer.get(buffer.position() + MAGIC_OFFSET);
@@ -60,13 +52,38 @@ public MutableRecordBatch nextBatch() throws IOException {
batchSlice.limit(batchSize);
buffer.position(buffer.position() + batchSize);
- if (magic < 0 || magic > RecordBatch.CURRENT_MAGIC_VALUE)
- throw new CorruptRecordException("Invalid magic found in record: " + magic);
-
if (magic > RecordBatch.MAGIC_VALUE_V1)
return new DefaultRecordBatch(batchSlice);
else
return new AbstractLegacyRecordBatch.ByteBufferLegacyRecordBatch(batchSlice);
}
+ /**
+ * Validates the header of the next batch and returns batch size.
+ * @return next batch size including LOG_OVERHEAD if buffer contains header up to
+ * magic byte, null otherwise
+ * @throws CorruptRecordException if record size or magic is invalid
+ */
+ Integer nextBatchSize() throws CorruptRecordException {
+ int remaining = buffer.remaining();
+ if (remaining < LOG_OVERHEAD)
+ return null;
+ int recordSize = buffer.getInt(buffer.position() + SIZE_OFFSET);
+ // V0 has the smallest overhead, stricter checking is done later
+ if (recordSize < LegacyRecord.RECORD_OVERHEAD_V0)
+ throw new CorruptRecordException(String.format("Record size %d is less than the minimum record overhead (%d)",
+ recordSize, LegacyRecord.RECORD_OVERHEAD_V0));
+ if (recordSize > maxMessageSize)
+ throw new CorruptRecordException(String.format("Record size %d exceeds the largest allowable message size (%d).",
+ recordSize, maxMessageSize));
+
+ if (remaining < HEADER_SIZE_UP_TO_MAGIC)
+ return null;
+
+ byte magic = buffer.get(buffer.position() + MAGIC_OFFSET);
+ if (magic < 0 || magic > RecordBatch.CURRENT_MAGIC_VALUE)
+ throw new CorruptRecordException("Invalid magic found in record: " + magic);
+
+ return recordSize + LOG_OVERHEAD;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/record/ConvertedRecords.java b/clients/src/main/java/org/apache/kafka/common/record/ConvertedRecords.java
index fe37c48976f..d9150e5044d 100644
--- a/clients/src/main/java/org/apache/kafka/common/record/ConvertedRecords.java
+++ b/clients/src/main/java/org/apache/kafka/common/record/ConvertedRecords.java
@@ -19,18 +19,18 @@
public class ConvertedRecords<T extends Records> {
private final T records;
- private final RecordsProcessingStats recordsProcessingStats;
+ private final RecordConversionStats recordConversionStats;
- public ConvertedRecords(T records, RecordsProcessingStats recordsProcessingStats) {
+ public ConvertedRecords(T records, RecordConversionStats recordConversionStats) {
this.records = records;
- this.recordsProcessingStats = recordsProcessingStats;
+ this.recordConversionStats = recordConversionStats;
}
public T records() {
return records;
}
- public RecordsProcessingStats recordsProcessingStats() {
- return recordsProcessingStats;
+ public RecordConversionStats recordConversionStats() {
+ return recordConversionStats;
}
}
diff --git a/clients/src/main/java/org/apache/kafka/common/record/DefaultRecordsSend.java b/clients/src/main/java/org/apache/kafka/common/record/DefaultRecordsSend.java
new file mode 100644
index 00000000000..aa715ea4dd4
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/record/DefaultRecordsSend.java
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.record;
+
+import java.io.IOException;
+import java.nio.channels.GatheringByteChannel;
+
+public class DefaultRecordsSend extends RecordsSend<Records> {
+ public DefaultRecordsSend(String destination, Records records) {
+ this(destination, records, records.sizeInBytes());
+ }
+
+ public DefaultRecordsSend(String destination, Records records, int maxBytesToWrite) {
+ super(destination, records, maxBytesToWrite);
+ }
+
+ @Override
+ protected long writeTo(GatheringByteChannel channel, long previouslyWritten, int remaining) throws IOException {
+ return records().writeTo(channel, previouslyWritten, remaining);
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/record/FileLogInputStream.java b/clients/src/main/java/org/apache/kafka/common/record/FileLogInputStream.java
index 75eb1b3e354..92e8864a183 100644
--- a/clients/src/main/java/org/apache/kafka/common/record/FileLogInputStream.java
+++ b/clients/src/main/java/org/apache/kafka/common/record/FileLogInputStream.java
@@ -40,26 +40,27 @@
public class FileLogInputStream implements LogInputStream<FileLogInputStream.FileChannelRecordBatch> {
private int position;
private final int end;
- private final FileChannel channel;
+ private final FileRecords fileRecords;
private final ByteBuffer logHeaderBuffer = ByteBuffer.allocate(HEADER_SIZE_UP_TO_MAGIC);
/**
* Create a new log input stream over the FileChannel
- * @param channel Underlying FileChannel
+ * @param records Underlying FileRecords instance
* @param start Position in the file channel to start from
* @param end Position in the file channel not to read past
*/
- FileLogInputStream(FileChannel channel,
+ FileLogInputStream(FileRecords records,
int start,
int end) {
- this.channel = channel;
+ this.fileRecords = records;
this.position = start;
this.end = end;
}
@Override
public FileChannelRecordBatch nextBatch() throws IOException {
- if (position + HEADER_SIZE_UP_TO_MAGIC >= end)
+ FileChannel channel = fileRecords.channel();
+ if (position >= end - HEADER_SIZE_UP_TO_MAGIC)
return null;
logHeaderBuffer.rewind();
@@ -71,9 +72,10 @@ public FileChannelRecordBatch nextBatch() throws IOException {
// V0 has the smallest overhead, stricter checking is done later
if (size < LegacyRecord.RECORD_OVERHEAD_V0)
- throw new CorruptRecordException(String.format("Record size is smaller than minimum record overhead (%d).", LegacyRecord.RECORD_OVERHEAD_V0));
+ throw new CorruptRecordException(String.format("Found record size %d smaller than minimum record " +
+ "overhead (%d) in file %s.", size, LegacyRecord.RECORD_OVERHEAD_V0, fileRecords.file()));
- if (position + LOG_OVERHEAD + size > end)
+ if (position > end - LOG_OVERHEAD - size)
return null;
byte magic = logHeaderBuffer.get(MAGIC_OFFSET);
diff --git a/clients/src/main/java/org/apache/kafka/common/record/FileRecords.java b/clients/src/main/java/org/apache/kafka/common/record/FileRecords.java
index fd9b4beab43..e44d5d93be8 100644
--- a/clients/src/main/java/org/apache/kafka/common/record/FileRecords.java
+++ b/clients/src/main/java/org/apache/kafka/common/record/FileRecords.java
@@ -19,6 +19,7 @@
import org.apache.kafka.common.KafkaException;
import org.apache.kafka.common.network.TransportLayer;
import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.utils.AbstractIterator;
import org.apache.kafka.common.utils.Time;
import org.apache.kafka.common.utils.Utils;
@@ -132,9 +133,9 @@ public ByteBuffer readInto(ByteBuffer buffer, int position) throws IOException {
*/
public FileRecords read(int position, int size) throws IOException {
if (position < 0)
- throw new IllegalArgumentException("Invalid position: " + position);
+ throw new IllegalArgumentException("Invalid position: " + position + " in read from " + file);
if (size < 0)
- throw new IllegalArgumentException("Invalid size: " + size);
+ throw new IllegalArgumentException("Invalid size: " + size + " in read from " + file);
int end = this.start + position + size;
// handle integer overflow or if end is beyond the end of the file
@@ -228,7 +229,7 @@ public void renameTo(File f) throws IOException {
public int truncateTo(int targetSize) throws IOException {
int originalSize = sizeInBytes();
if (targetSize > originalSize || targetSize < 0)
- throw new KafkaException("Attempt to truncate log segment to " + targetSize + " bytes failed, " +
+ throw new KafkaException("Attempt to truncate log segment " + file + " to " + targetSize + " bytes failed, " +
" size of this log segment is " + originalSize + " bytes.");
if (targetSize < (int) channel.size()) {
channel.truncate(targetSize);
@@ -239,8 +240,8 @@ public int truncateTo(int targetSize) throws IOException {
@Override
public ConvertedRecords<? extends Records> downConvert(byte toMagic, long firstOffset, Time time) {
- ConvertedRecords<MemoryRecords> convertedRecords = downConvert(batches, toMagic, firstOffset, time);
- if (convertedRecords.recordsProcessingStats().numRecordsConverted() == 0) {
+ ConvertedRecords<MemoryRecords> convertedRecords = RecordsUtil.downConvert(batches, toMagic, firstOffset, time);
+ if (convertedRecords.recordConversionStats().numRecordsConverted() == 0) {
// This indicates that the message is too large, which means that the buffer is not large
// enough to hold a full record batch. We just return all the bytes in this instance.
// Even though the record batch does not have the right format version, we expect old clients
@@ -248,7 +249,7 @@ public int truncateTo(int targetSize) throws IOException {
// are not enough available bytes in the response to read it fully. Note that this is
// only possible prior to KIP-74, after which the broker was changed to always return at least
// one full record batch, even if it requires exceeding the max fetch size requested by the client.
- return new ConvertedRecords<>(this, RecordsProcessingStats.EMPTY);
+ return new ConvertedRecords<>(this, RecordConversionStats.EMPTY);
} else {
return convertedRecords;
}
@@ -347,6 +348,14 @@ public TimestampAndOffset largestTimestampAfter(int startingPosition) {
return batches;
}
+ @Override
+ public String toString() {
+ return "FileRecords(file= " + file +
+ ", start=" + start +
+ ", end=" + end +
+ ")";
+ }
+
private Iterable<FileChannelRecordBatch> batchesFrom(final int start) {
return new Iterable<FileChannelRecordBatch>() {
@Override
@@ -356,13 +365,18 @@ public TimestampAndOffset largestTimestampAfter(int startingPosition) {
};
}
- private Iterator<FileChannelRecordBatch> batchIterator(int start) {
+ @Override
+ public AbstractIterator<FileChannelRecordBatch> batchIterator() {
+ return batchIterator(start);
+ }
+
+ private AbstractIterator<FileChannelRecordBatch> batchIterator(int start) {
final int end;
if (isSlice)
end = this.end;
else
end = this.sizeInBytes();
- FileLogInputStream inputStream = new FileLogInputStream(channel, start, end);
+ FileLogInputStream inputStream = new FileLogInputStream(this, start, end);
return new RecordBatchIterator<>(inputStream);
}
@@ -502,5 +516,4 @@ public String toString() {
')';
}
}
-
}
diff --git a/clients/src/main/java/org/apache/kafka/common/record/LazyDownConversionRecords.java b/clients/src/main/java/org/apache/kafka/common/record/LazyDownConversionRecords.java
new file mode 100644
index 00000000000..da14b5b494f
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/record/LazyDownConversionRecords.java
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.record;
+
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.utils.AbstractIterator;
+import org.apache.kafka.common.utils.Time;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+
+/**
+ * Encapsulation for holding records that require down-conversion in a lazy, chunked manner (KIP-283). See
+ * {@link LazyDownConversionRecordsSend} for the actual chunked send implementation.
+ */
+public class LazyDownConversionRecords implements BaseRecords {
+ private final TopicPartition topicPartition;
+ private final Records records;
+ private final byte toMagic;
+ private final long firstOffset;
+ private ConvertedRecords firstConvertedBatch;
+ private final int sizeInBytes;
+ private final Time time;
+
+ /**
+ * @param topicPartition The topic-partition to which records belong
+ * @param records Records to lazily down-convert
+ * @param toMagic Magic version to down-convert to
+ * @param firstOffset The starting offset for down-converted records. This only impacts some cases. See
+ * {@link RecordsUtil#downConvert(Iterable, byte, long, Time)} for an explanation.
+ * @param time The time instance to use
+ */
+ public LazyDownConversionRecords(TopicPartition topicPartition, Records records, byte toMagic, long firstOffset, Time time) {
+ this.topicPartition = Objects.requireNonNull(topicPartition);
+ this.records = Objects.requireNonNull(records);
+ this.toMagic = toMagic;
+ this.firstOffset = firstOffset;
+ this.time = Objects.requireNonNull(time);
+
+ // Kafka consumers expect at least one full batch of messages for every topic-partition. To guarantee this, we
+ // need to make sure that we are able to accommodate one full batch of down-converted messages. The way we achieve
+ // this is by having sizeInBytes method factor in the size of the first down-converted batch and return at least
+ // its size.
+ AbstractIterator<? extends RecordBatch> it = records.batchIterator();
+ if (it.hasNext()) {
+ firstConvertedBatch = RecordsUtil.downConvert(Collections.singletonList(it.peek()), toMagic, firstOffset, time);
+ sizeInBytes = Math.max(records.sizeInBytes(), firstConvertedBatch.records().sizeInBytes());
+ } else {
+ firstConvertedBatch = null;
+ sizeInBytes = 0;
+ }
+ }
+
+ @Override
+ public int sizeInBytes() {
+ return sizeInBytes;
+ }
+
+ @Override
+ public LazyDownConversionRecordsSend toSend(String destination) {
+ return new LazyDownConversionRecordsSend(destination, this);
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (o instanceof LazyDownConversionRecords) {
+ LazyDownConversionRecords that = (LazyDownConversionRecords) o;
+ return toMagic == that.toMagic &&
+ firstOffset == that.firstOffset &&
+ topicPartition.equals(that.topicPartition) &&
+ records.equals(that.records);
+ }
+ return false;
+ }
+
+ @Override
+ public int hashCode() {
+ int result = toMagic;
+ result = 31 * result + (int) (firstOffset ^ (firstOffset >>> 32));
+ result = 31 * result + topicPartition.hashCode();
+ result = 31 * result + records.hashCode();
+ return result;
+ }
+
+ public java.util.Iterator<ConvertedRecords> iterator(long maximumReadSize) {
+ // We typically expect only one iterator instance to be created, so null out the first converted batch after
+ // first use to make it available for GC.
+ ConvertedRecords firstBatch = firstConvertedBatch;
+ firstConvertedBatch = null;
+ return new Iterator(records, maximumReadSize, firstBatch);
+ }
+
+ /**
+ * Implementation for being able to iterate over down-converted records. Goal of this implementation is to keep
+ * it as memory-efficient as possible by not having to maintain all down-converted records in-memory. Maintains
+ * a view into batches of down-converted records.
+ */
+ private class Iterator extends AbstractIterator<ConvertedRecords> {
+ private final AbstractIterator<? extends RecordBatch> batchIterator;
+ private final long maximumReadSize;
+ private ConvertedRecords firstConvertedBatch;
+
+ /**
+ * @param recordsToDownConvert Records that require down-conversion
+ * @param maximumReadSize Maximum possible size of underlying records that will be down-converted in each call to
+ * {@link #makeNext()}. This is a soft limit as {@link #makeNext()} will always convert
+ * and return at least one full message batch.
+ */
+ private Iterator(Records recordsToDownConvert, long maximumReadSize, ConvertedRecords firstConvertedBatch) {
+ this.batchIterator = recordsToDownConvert.batchIterator();
+ this.maximumReadSize = maximumReadSize;
+ this.firstConvertedBatch = firstConvertedBatch;
+ // If we already have the first down-converted batch, advance the underlying records iterator to next batch
+ if (firstConvertedBatch != null)
+ this.batchIterator.next();
+ }
+
+ /**
+ * Make next set of down-converted records
+ * @return Down-converted records
+ */
+ @Override
+ protected ConvertedRecords makeNext() {
+ // If we have cached the first down-converted batch, return that now
+ if (firstConvertedBatch != null) {
+ ConvertedRecords convertedBatch = firstConvertedBatch;
+ firstConvertedBatch = null;
+ return convertedBatch;
+ }
+
+ if (!batchIterator.hasNext())
+ return allDone();
+
+ // Figure out batches we should down-convert based on the size constraints
+ List<RecordBatch> batches = new ArrayList<>();
+ boolean isFirstBatch = true;
+ long sizeSoFar = 0;
+ while (batchIterator.hasNext() &&
+ (isFirstBatch || (batchIterator.peek().sizeInBytes() + sizeSoFar) <= maximumReadSize)) {
+ RecordBatch currentBatch = batchIterator.next();
+ batches.add(currentBatch);
+ sizeSoFar += currentBatch.sizeInBytes();
+ isFirstBatch = false;
+ }
+ return RecordsUtil.downConvert(batches, toMagic, firstOffset, time);
+ }
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/record/LazyDownConversionRecordsSend.java b/clients/src/main/java/org/apache/kafka/common/record/LazyDownConversionRecordsSend.java
new file mode 100644
index 00000000000..b78211418ac
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/record/LazyDownConversionRecordsSend.java
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.record;
+
+import org.apache.kafka.common.TopicPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.channels.GatheringByteChannel;
+import java.util.Iterator;
+
+/**
+ * Encapsulation for {@link RecordsSend} for {@link LazyDownConversionRecords}. Records are down-converted in batches and
+ * on-demand when {@link #writeTo} method is called.
+ */
+public final class LazyDownConversionRecordsSend extends RecordsSend<LazyDownConversionRecords> {
+ private static final Logger log = LoggerFactory.getLogger(LazyDownConversionRecordsSend.class);
+ private static final int MAX_READ_SIZE = 128 * 1024;
+
+ private RecordConversionStats recordConversionStats;
+ private RecordsSend convertedRecordsWriter;
+ private Iterator<ConvertedRecords> convertedRecordsIterator;
+
+ public LazyDownConversionRecordsSend(String destination, LazyDownConversionRecords records) {
+ super(destination, records, records.sizeInBytes());
+ convertedRecordsWriter = null;
+ recordConversionStats = new RecordConversionStats();
+ convertedRecordsIterator = records().iterator(MAX_READ_SIZE);
+ }
+
+ @Override
+ public long writeTo(GatheringByteChannel channel, long previouslyWritten, int remaining) throws IOException {
+ if (convertedRecordsWriter == null || convertedRecordsWriter.completed()) {
+ MemoryRecords convertedRecords;
+
+ // Check if we have more chunks left to down-convert
+ if (convertedRecordsIterator.hasNext()) {
+ // Get next chunk of down-converted messages
+ ConvertedRecords<MemoryRecords> recordsAndStats = convertedRecordsIterator.next();
+ convertedRecords = recordsAndStats.records();
+
+ int sizeOfFirstConvertedBatch = convertedRecords.batchIterator().next().sizeInBytes();
+ if (previouslyWritten == 0 && sizeOfFirstConvertedBatch > size())
+ throw new EOFException("Unable to send first batch completely." +
+ " maximum_size: " + size() +
+ " converted_records_size: " + sizeOfFirstConvertedBatch);
+
+ recordConversionStats.add(recordsAndStats.recordConversionStats());
+ log.debug("Got lazy converted records for {" + topicPartition() + "} with length=" + convertedRecords.sizeInBytes());
+ } else {
+ if (previouslyWritten == 0)
+ throw new EOFException("Unable to get the first batch of down-converted records");
+
+ // We do not have any records left to down-convert. Construct a "fake" message for the length remaining.
+ // This message will be ignored by the consumer because its length will be past the length of maximum
+ // possible response size.
+ // DefaultRecordBatch =>
+ // BaseOffset => Int64
+ // Length => Int32
+ // ...
+ // TODO: check if there is a better way to encapsulate this logic, perhaps in DefaultRecordBatch
+ log.debug("Constructing fake message batch for topic-partition {" + topicPartition() + "} for remaining length " + remaining);
+ int minLength = (Long.SIZE / Byte.SIZE) + (Integer.SIZE / Byte.SIZE);
+ ByteBuffer fakeMessageBatch = ByteBuffer.allocate(Math.max(minLength, Math.min(remaining + 1, MAX_READ_SIZE)));
+ fakeMessageBatch.putLong(-1L);
+ fakeMessageBatch.putInt(remaining + 1);
+ convertedRecords = MemoryRecords.readableRecords(fakeMessageBatch);
+ }
+
+ convertedRecordsWriter = new DefaultRecordsSend(destination(), convertedRecords, Math.min(convertedRecords.sizeInBytes(), remaining));
+ }
+ return convertedRecordsWriter.writeTo(channel);
+ }
+
+ public RecordConversionStats recordConversionStats() {
+ return recordConversionStats;
+ }
+
+ public TopicPartition topicPartition() {
+ return records().topicPartition();
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java b/clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java
index 932d4b69c22..55a471149c6 100644
--- a/clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java
+++ b/clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java
@@ -18,7 +18,9 @@
import org.apache.kafka.common.KafkaException;
import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.CorruptRecordException;
import org.apache.kafka.common.record.MemoryRecords.RecordFilter.BatchRetention;
+import org.apache.kafka.common.utils.AbstractIterator;
import org.apache.kafka.common.utils.ByteBufferOutputStream;
import org.apache.kafka.common.utils.CloseableIterator;
import org.apache.kafka.common.utils.Time;
@@ -48,7 +50,7 @@
private final Iterable<MutableRecordBatch> batches = new Iterable<MutableRecordBatch>() {
@Override
public Iterator<MutableRecordBatch> iterator() {
- return new RecordBatchIterator<>(new ByteBufferLogInputStream(buffer.duplicate(), Integer.MAX_VALUE));
+ return batchIterator();
}
};
@@ -114,7 +116,24 @@ public int validBytes() {
@Override
public ConvertedRecords<MemoryRecords> downConvert(byte toMagic, long firstOffset, Time time) {
- return downConvert(batches(), toMagic, firstOffset, time);
+ return RecordsUtil.downConvert(batches(), toMagic, firstOffset, time);
+ }
+
+ @Override
+ public AbstractIterator<MutableRecordBatch> batchIterator() {
+ return new RecordBatchIterator<>(new ByteBufferLogInputStream(buffer.duplicate(), Integer.MAX_VALUE));
+ }
+
+ /**
+ * Validates the header of the first batch and returns batch size.
+ * @return first batch size including LOG_OVERHEAD if buffer contains header up to
+ * magic byte, null otherwise
+ * @throws CorruptRecordException if record size or magic is invalid
+ */
+ public Integer firstBatchSize() {
+ if (buffer.remaining() < HEADER_SIZE_UP_TO_MAGIC)
+ return null;
+ return new ByteBufferLogInputStream(buffer, Integer.MAX_VALUE).nextBatchSize();
}
/**
@@ -145,7 +164,7 @@ private static FilterResult filterTo(TopicPartition partition, Iterable<MutableR
long maxOffset = -1L;
long shallowOffsetOfMaxTimestamp = -1L;
int messagesRead = 0;
- int bytesRead = 0;
+ int bytesRead = 0; // bytes processed from `batches`
int messagesRetained = 0;
int bytesRetained = 0;
@@ -359,6 +378,8 @@ public int hashCode() {
public final long maxTimestamp;
public final long shallowOffsetOfMaxTimestamp;
+ // Note that `bytesRead` should contain only bytes from batches that have been processed,
+ // i.e. bytes from `messagesRead` and any discarded batches.
public FilterResult(ByteBuffer output,
int messagesRead,
int bytesRead,
@@ -510,6 +531,10 @@ public static MemoryRecords withRecords(long initialOffset, CompressionType comp
records);
}
+ public static MemoryRecords withRecords(byte magic, long initialOffset, CompressionType compressionType, SimpleRecord... records) {
+ return withRecords(magic, initialOffset, compressionType, TimestampType.CREATE_TIME, records);
+ }
+
public static MemoryRecords withRecords(long initialOffset, CompressionType compressionType, Integer partitionLeaderEpoch, SimpleRecord... records) {
return withRecords(RecordBatch.CURRENT_MAGIC_VALUE, initialOffset, compressionType, TimestampType.CREATE_TIME, RecordBatch.NO_PRODUCER_ID,
RecordBatch.NO_PRODUCER_EPOCH, RecordBatch.NO_SEQUENCE, partitionLeaderEpoch, false, records);
@@ -565,7 +590,7 @@ public static MemoryRecords withRecords(byte magic, long initialOffset, Compress
public static MemoryRecords withRecords(byte magic, long initialOffset, CompressionType compressionType,
TimestampType timestampType, long producerId, short producerEpoch,
int baseSequence, int partitionLeaderEpoch, boolean isTransactional,
- SimpleRecord ... records) {
+ SimpleRecord... records) {
if (records.length == 0)
return MemoryRecords.EMPTY;
int sizeEstimate = AbstractRecords.estimateSizeInBytes(magic, compressionType, Arrays.asList(records));
diff --git a/clients/src/main/java/org/apache/kafka/common/network/MultiSend.java b/clients/src/main/java/org/apache/kafka/common/record/MultiRecordsSend.java
similarity index 59%
rename from clients/src/main/java/org/apache/kafka/common/network/MultiSend.java
rename to clients/src/main/java/org/apache/kafka/common/record/MultiRecordsSend.java
index 6b663609305..2bc8d1c5519 100644
--- a/clients/src/main/java/org/apache/kafka/common/network/MultiSend.java
+++ b/clients/src/main/java/org/apache/kafka/common/record/MultiRecordsSend.java
@@ -14,34 +14,39 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.kafka.common.network;
+package org.apache.kafka.common.record;
import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.network.Send;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.nio.channels.GatheringByteChannel;
+import java.util.HashMap;
+import java.util.Map;
import java.util.Queue;
/**
- * A set of composite sends, sent one after another
+ * A set of composite sends with nested {@link RecordsSend}, sent one after another
*/
-public class MultiSend implements Send {
- private static final Logger log = LoggerFactory.getLogger(MultiSend.class);
+public class MultiRecordsSend implements Send {
+ private static final Logger log = LoggerFactory.getLogger(MultiRecordsSend.class);
private final String dest;
private final Queue<Send> sendQueue;
private final long size;
+ private Map<TopicPartition, RecordConversionStats> recordConversionStats;
private long totalWritten = 0;
private Send current;
/**
- * Construct a MultiSend for the given destination from a queue of Send objects. The queue will be
- * consumed as the MultiSend progresses (on completion, it will be empty).
+ * Construct a MultiRecordsSend for the given destination from a queue of Send objects. The queue will be
+ * consumed as the MultiRecordsSend progresses (on completion, it will be empty).
*/
- public MultiSend(String dest, Queue<Send> sends) {
+ public MultiRecordsSend(String dest, Queue<Send> sends) {
this.dest = dest;
this.sendQueue = sends;
@@ -88,8 +93,10 @@ public long writeTo(GatheringByteChannel channel) throws IOException {
long written = current.writeTo(channel);
totalWrittenPerCall += written;
sendComplete = current.completed();
- if (sendComplete)
+ if (sendComplete) {
+ updateRecordConversionStats(current);
current = sendQueue.poll();
+ }
} while (!completed() && sendComplete);
totalWritten += totalWrittenPerCall;
@@ -103,4 +110,24 @@ public long writeTo(GatheringByteChannel channel) throws IOException {
return totalWrittenPerCall;
}
+ /**
+ * Get any statistics that were recorded as part of executing this {@link MultiRecordsSend}.
+ * @return Records processing statistics (could be null if no statistics were collected)
+ */
+ public Map<TopicPartition, RecordConversionStats> recordConversionStats() {
+ return recordConversionStats;
+ }
+
+ private void updateRecordConversionStats(Send completedSend) {
+ // The underlying send might have accumulated statistics that need to be recorded. For example,
+ // LazyDownConversionRecordsSend accumulates statistics related to the number of bytes down-converted, the amount
+ // of temporary memory used for down-conversion, etc. Pull out any such statistics from the underlying send
+ // and fold it up appropriately.
+ if (completedSend instanceof LazyDownConversionRecordsSend) {
+ if (recordConversionStats == null)
+ recordConversionStats = new HashMap<>();
+ LazyDownConversionRecordsSend lazyRecordsSend = (LazyDownConversionRecordsSend) completedSend;
+ recordConversionStats.put(lazyRecordsSend.topicPartition(), lazyRecordsSend.recordConversionStats());
+ }
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/record/RecordsProcessingStats.java b/clients/src/main/java/org/apache/kafka/common/record/RecordConversionStats.java
similarity index 71%
rename from clients/src/main/java/org/apache/kafka/common/record/RecordsProcessingStats.java
rename to clients/src/main/java/org/apache/kafka/common/record/RecordConversionStats.java
index e104bc8189f..4f0bca527fb 100644
--- a/clients/src/main/java/org/apache/kafka/common/record/RecordsProcessingStats.java
+++ b/clients/src/main/java/org/apache/kafka/common/record/RecordConversionStats.java
@@ -16,20 +16,30 @@
*/
package org.apache.kafka.common.record;
-public class RecordsProcessingStats {
+public class RecordConversionStats {
- public static final RecordsProcessingStats EMPTY = new RecordsProcessingStats(0L, 0, -1);
+ public static final RecordConversionStats EMPTY = new RecordConversionStats();
- private final long temporaryMemoryBytes;
- private final int numRecordsConverted;
- private final long conversionTimeNanos;
+ private long temporaryMemoryBytes;
+ private int numRecordsConverted;
+ private long conversionTimeNanos;
- public RecordsProcessingStats(long temporaryMemoryBytes, int numRecordsConverted, long conversionTimeNanos) {
+ public RecordConversionStats(long temporaryMemoryBytes, int numRecordsConverted, long conversionTimeNanos) {
this.temporaryMemoryBytes = temporaryMemoryBytes;
this.numRecordsConverted = numRecordsConverted;
this.conversionTimeNanos = conversionTimeNanos;
}
+ public RecordConversionStats() {
+ this(0, 0, 0);
+ }
+
+ public void add(RecordConversionStats stats) {
+ temporaryMemoryBytes += stats.temporaryMemoryBytes;
+ numRecordsConverted += stats.numRecordsConverted;
+ conversionTimeNanos += stats.conversionTimeNanos;
+ }
+
/**
* Returns the number of temporary memory bytes allocated to process the records.
* This size depends on whether the records need decompression and/or conversion:
@@ -54,7 +64,7 @@ public long conversionTimeNanos() {
@Override
public String toString() {
- return String.format("RecordsProcessingStats(temporaryMemoryBytes=%d, numRecordsConverted=%d, conversionTimeNanos=%d)",
+ return String.format("RecordConversionStats(temporaryMemoryBytes=%d, numRecordsConverted=%d, conversionTimeNanos=%d)",
temporaryMemoryBytes, numRecordsConverted, conversionTimeNanos);
}
}
diff --git a/clients/src/main/java/org/apache/kafka/common/record/RecordVersion.java b/clients/src/main/java/org/apache/kafka/common/record/RecordVersion.java
new file mode 100644
index 00000000000..1f80d623ca9
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/record/RecordVersion.java
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.record;
+
+/**
+ * Defines the record format versions supported by Kafka.
+ *
+ * For historical reasons, the record format version is also known as `magic` and `message format version`. Note that
+ * the version actually applies to the {@link RecordBatch} (instead of the {@link Record}). Finally, the
+ * `message.format.version` topic config confusingly expects an ApiVersion instead of a RecordVersion.
+ */
+public enum RecordVersion {
+ V0(0), V1(1), V2(2);
+
+ private static final RecordVersion[] VALUES = values();
+
+ public final byte value;
+
+ RecordVersion(int value) {
+ this.value = (byte) value;
+ }
+
+ public static RecordVersion lookup(byte value) {
+ if (value < 0 || value >= VALUES.length)
+ throw new IllegalArgumentException("Unknown record version: " + value);
+ return VALUES[value];
+ }
+
+ public static RecordVersion current() {
+ return V2;
+ }
+
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/record/Records.java b/clients/src/main/java/org/apache/kafka/common/record/Records.java
index 19152bae267..23607b46171 100644
--- a/clients/src/main/java/org/apache/kafka/common/record/Records.java
+++ b/clients/src/main/java/org/apache/kafka/common/record/Records.java
@@ -16,10 +16,13 @@
*/
package org.apache.kafka.common.record;
+import org.apache.kafka.common.utils.AbstractIterator;
+import org.apache.kafka.common.utils.Time;
+
import java.io.IOException;
import java.nio.channels.GatheringByteChannel;
+import java.util.Iterator;
-import org.apache.kafka.common.utils.Time;
/**
* Interface for accessing the records contained in a log. The log itself is represented as a sequence of record
@@ -28,20 +31,19 @@
* For magic versions 1 and below, each batch consists of an 8 byte offset, a 4 byte record size, and a "shallow"
* {@link Record record}. If the batch is not compressed, then each batch will have only the shallow record contained
* inside it. If it is compressed, the batch contains "deep" records, which are packed into the value field of the
- * shallow record. To iterate over the shallow batches, use {@link #batches()}; for the deep records, use
- * {@link #records()}. Note that the deep iterator handles both compressed and non-compressed batches: if the batch is
- * not compressed, the shallow record is returned; otherwise, the shallow batch is decompressed and the deep records
- * are returned.
+ * shallow record. To iterate over the shallow batches, use {@link Records#batches()}; for the deep records, use
+ * {@link Records#records()}. Note that the deep iterator handles both compressed and non-compressed batches:
+ * if the batch is not compressed, the shallow record is returned; otherwise, the shallow batch is decompressed and the
+ * deep records are returned.
*
* For magic version 2, every batch contains 1 or more log record, regardless of compression. You can iterate
- * over the batches directly using {@link #batches()}. Records can be iterated either directly from an individual
- * batch or through {@link #records()}. Just as in previous versions, iterating over the records typically involves
+ * over the batches directly using {@link Records#batches()}. Records can be iterated either directly from an individual
+ * batch or through {@link Records#records()}. Just as in previous versions, iterating over the records typically involves
* decompression and should therefore be used with caution.
*
* See {@link MemoryRecords} for the in-memory representation and {@link FileRecords} for the on-disk representation.
*/
-public interface Records {
-
+public interface Records extends BaseRecords {
int OFFSET_OFFSET = 0;
int OFFSET_LENGTH = 8;
int SIZE_OFFSET = OFFSET_OFFSET + OFFSET_LENGTH;
@@ -54,12 +56,6 @@
int MAGIC_LENGTH = 1;
int HEADER_SIZE_UP_TO_MAGIC = MAGIC_OFFSET + MAGIC_LENGTH;
- /**
- * The size of these records in bytes.
- * @return The size in bytes of the records
- */
- int sizeInBytes();
-
/**
* Attempts to write the contents of this buffer to a channel.
* @param channel The channel to write to
@@ -79,6 +75,13 @@
*/
Iterable<? extends RecordBatch> batches();
+ /**
+ * Get an iterator over the record batches. This is similar to {@link #batches()} but returns an {@link AbstractIterator}
+ * instead of {@link Iterator}, so that clients can use methods like {@link AbstractIterator#peek() peek}.
+ * @return An iterator over the record batches of the log
+ */
+ AbstractIterator<? extends RecordBatch> batchIterator();
+
/**
* Check whether all batches in this buffer have a certain magic value.
* @param magic The magic value to check
@@ -99,7 +102,7 @@
* deep iteration since all of the deep records must also be converted to the desired format.
* @param toMagic The magic value to convert to
* @param firstOffset The starting offset for returned records. This only impacts some cases. See
- * {@link AbstractRecords#downConvert(Iterable, byte, long, Time) for an explanation.
+ * {@link RecordsUtil#downConvert(Iterable, byte, long, Time)} for an explanation.
* @param time instance used for reporting stats
* @return A ConvertedRecords instance which may or may not contain the same instance in its records field.
*/
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/RecordsSend.java b/clients/src/main/java/org/apache/kafka/common/record/RecordsSend.java
similarity index 55%
rename from clients/src/main/java/org/apache/kafka/common/requests/RecordsSend.java
rename to clients/src/main/java/org/apache/kafka/common/record/RecordsSend.java
index 6608e9b1218..b40d6e6a6b8 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/RecordsSend.java
+++ b/clients/src/main/java/org/apache/kafka/common/record/RecordsSend.java
@@ -14,29 +14,30 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.kafka.common.requests;
+package org.apache.kafka.common.record;
import org.apache.kafka.common.network.Send;
import org.apache.kafka.common.network.TransportLayers;
-import org.apache.kafka.common.record.Records;
import java.io.EOFException;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.channels.GatheringByteChannel;
-public class RecordsSend implements Send {
+public abstract class RecordsSend<T extends BaseRecords> implements Send {
private static final ByteBuffer EMPTY_BYTE_BUFFER = ByteBuffer.allocate(0);
private final String destination;
- private final Records records;
+ private final T records;
+ private final int maxBytesToWrite;
private int remaining;
private boolean pending = false;
- public RecordsSend(String destination, Records records) {
+ protected RecordsSend(String destination, T records, int maxBytesToWrite) {
this.destination = destination;
this.records = records;
- this.remaining = records.sizeInBytes();
+ this.maxBytesToWrite = maxBytesToWrite;
+ this.remaining = maxBytesToWrite;
}
@Override
@@ -50,11 +51,11 @@ public boolean completed() {
}
@Override
- public long writeTo(GatheringByteChannel channel) throws IOException {
+ public final long writeTo(GatheringByteChannel channel) throws IOException {
long written = 0;
if (remaining > 0) {
- written = records.writeTo(channel, size() - remaining, remaining);
+ written = writeTo(channel, size() - remaining, remaining);
if (written < 0)
throw new EOFException("Wrote negative bytes to channel. This shouldn't happen.");
remaining -= written;
@@ -69,6 +70,23 @@ public long writeTo(GatheringByteChannel channel) throws IOException {
@Override
public long size() {
- return records.sizeInBytes();
+ return maxBytesToWrite;
}
+
+ protected T records() {
+ return records;
+ }
+
+ /**
+ * Write records up to `remaining` bytes to `channel`. The implementation is allowed to be stateful. The contract
+ * from the caller is that the first invocation will be with `previouslyWritten` equal to 0, and `remaining` equal to
+ * the to maximum bytes we want to write the to `channel`. `previouslyWritten` and `remaining` will be adjusted
+ * appropriately for every subsequent invocation. See {@link #writeTo} for example expected usage.
+ * @param channel The channel to write to
+ * @param previouslyWritten Bytes written in previous calls to {@link #writeTo(GatheringByteChannel, long, int)}; 0 if being called for the first time
+ * @param remaining Number of bytes remaining to be written
+ * @return The number of bytes actually written
+ * @throws IOException For any IO errors
+ */
+ protected abstract long writeTo(GatheringByteChannel channel, long previouslyWritten, int remaining) throws IOException;
}
diff --git a/clients/src/main/java/org/apache/kafka/common/record/RecordsUtil.java b/clients/src/main/java/org/apache/kafka/common/record/RecordsUtil.java
new file mode 100644
index 00000000000..c9b73941317
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/record/RecordsUtil.java
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.record;
+
+import org.apache.kafka.common.utils.Time;
+
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+
+public class RecordsUtil {
+ /**
+ * Down convert batches to the provided message format version. The first offset parameter is only relevant in the
+ * conversion from uncompressed v2 or higher to v1 or lower. The reason is that uncompressed records in v0 and v1
+ * are not batched (put another way, each batch always has 1 record).
+ *
+ * If a client requests records in v1 format starting from the middle of an uncompressed batch in v2 format, we
+ * need to drop records from the batch during the conversion. Some versions of librdkafka rely on this for
+ * correctness.
+ *
+ * The temporaryMemoryBytes computation assumes that the batches are not loaded into the heap
+ * (via classes like FileChannelRecordBatch) before this method is called. This is the case in the broker (we
+ * only load records into the heap when down converting), but it's not for the producer. However, down converting
+ * in the producer is very uncommon and the extra complexity to handle that case is not worth it.
+ */
+ protected static ConvertedRecords<MemoryRecords> downConvert(Iterable<? extends RecordBatch> batches, byte toMagic,
+ long firstOffset, Time time) {
+ // maintain the batch along with the decompressed records to avoid the need to decompress again
+ List<RecordBatchAndRecords> recordBatchAndRecordsList = new ArrayList<>();
+ int totalSizeEstimate = 0;
+ long startNanos = time.nanoseconds();
+
+ for (RecordBatch batch : batches) {
+ if (toMagic < RecordBatch.MAGIC_VALUE_V2 && batch.isControlBatch())
+ continue;
+
+ if (batch.magic() <= toMagic) {
+ totalSizeEstimate += batch.sizeInBytes();
+ recordBatchAndRecordsList.add(new RecordBatchAndRecords(batch, null, null));
+ } else {
+ List<Record> records = new ArrayList<>();
+ for (Record record : batch) {
+ // See the method javadoc for an explanation
+ if (toMagic > RecordBatch.MAGIC_VALUE_V1 || batch.isCompressed() || record.offset() >= firstOffset)
+ records.add(record);
+ }
+ if (records.isEmpty())
+ continue;
+ final long baseOffset;
+ if (batch.magic() >= RecordBatch.MAGIC_VALUE_V2 && toMagic >= RecordBatch.MAGIC_VALUE_V2)
+ baseOffset = batch.baseOffset();
+ else
+ baseOffset = records.get(0).offset();
+ totalSizeEstimate += AbstractRecords.estimateSizeInBytes(toMagic, baseOffset, batch.compressionType(), records);
+ recordBatchAndRecordsList.add(new RecordBatchAndRecords(batch, records, baseOffset));
+ }
+ }
+
+ ByteBuffer buffer = ByteBuffer.allocate(totalSizeEstimate);
+ long temporaryMemoryBytes = 0;
+ int numRecordsConverted = 0;
+ for (RecordBatchAndRecords recordBatchAndRecords : recordBatchAndRecordsList) {
+ temporaryMemoryBytes += recordBatchAndRecords.batch.sizeInBytes();
+ if (recordBatchAndRecords.batch.magic() <= toMagic) {
+ recordBatchAndRecords.batch.writeTo(buffer);
+ } else {
+ MemoryRecordsBuilder builder = convertRecordBatch(toMagic, buffer, recordBatchAndRecords);
+ buffer = builder.buffer();
+ temporaryMemoryBytes += builder.uncompressedBytesWritten();
+ numRecordsConverted += builder.numRecords();
+ }
+ }
+
+ buffer.flip();
+ RecordConversionStats stats = new RecordConversionStats(temporaryMemoryBytes, numRecordsConverted,
+ time.nanoseconds() - startNanos);
+ return new ConvertedRecords<>(MemoryRecords.readableRecords(buffer), stats);
+ }
+
+ /**
+ * Return a buffer containing the converted record batches. The returned buffer may not be the same as the received
+ * one (e.g. it may require expansion).
+ */
+ private static MemoryRecordsBuilder convertRecordBatch(byte magic, ByteBuffer buffer, RecordBatchAndRecords recordBatchAndRecords) {
+ RecordBatch batch = recordBatchAndRecords.batch;
+ final TimestampType timestampType = batch.timestampType();
+ long logAppendTime = timestampType == TimestampType.LOG_APPEND_TIME ? batch.maxTimestamp() : RecordBatch.NO_TIMESTAMP;
+
+ MemoryRecordsBuilder builder = MemoryRecords.builder(buffer, magic, batch.compressionType(),
+ timestampType, recordBatchAndRecords.baseOffset, logAppendTime);
+ for (Record record : recordBatchAndRecords.records) {
+ // Down-convert this record. Ignore headers when down-converting to V0 and V1 since they are not supported
+ if (magic > RecordBatch.MAGIC_VALUE_V1)
+ builder.append(record);
+ else
+ builder.appendWithOffset(record.offset(), record.timestamp(), record.key(), record.value());
+ }
+
+ builder.close();
+ return builder;
+ }
+
+
+ private static class RecordBatchAndRecords {
+ private final RecordBatch batch;
+ private final List<Record> records;
+ private final Long baseOffset;
+
+ private RecordBatchAndRecords(RecordBatch batch, List<Record> records, Long baseOffset) {
+ this.batch = batch;
+ this.records = records;
+ this.baseOffset = baseOffset;
+ }
+ }
+
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/AbstractResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/AbstractResponse.java
index 608f6c33746..c0ebef1d967 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/AbstractResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/AbstractResponse.java
@@ -73,7 +73,7 @@ public static AbstractResponse parseResponse(ApiKeys apiKey, Struct struct) {
case PRODUCE:
return new ProduceResponse(struct);
case FETCH:
- return new FetchResponse(struct);
+ return FetchResponse.parse(struct);
case LIST_OFFSETS:
return new ListOffsetResponse(struct);
case METADATA:
@@ -162,6 +162,19 @@ public static AbstractResponse parseResponse(ApiKeys apiKey, Struct struct) {
}
}
+ /**
+ * Returns whether or not client should throttle upon receiving a response of the specified version with a non-zero
+ * throttle time. Client-side throttling is needed when communicating with a newer version of broker which, on
+ * quota violation, sends out responses before throttling.
+ */
+ public boolean shouldClientThrottle(short version) {
+ return false;
+ }
+
+ public int throttleTimeMs() {
+ return DEFAULT_THROTTLE_TIME;
+ }
+
public String toString(short version) {
return toStruct(version).toString();
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/AddOffsetsToTxnRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/AddOffsetsToTxnRequest.java
index f6a1722112d..6fb9441ace6 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/AddOffsetsToTxnRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/AddOffsetsToTxnRequest.java
@@ -35,8 +35,13 @@
PRODUCER_EPOCH,
GROUP_ID);
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema ADD_OFFSETS_TO_TXN_REQUEST_V1 = ADD_OFFSETS_TO_TXN_REQUEST_V0;
+
public static Schema[] schemaVersions() {
- return new Schema[]{ADD_OFFSETS_TO_TXN_REQUEST_V0};
+ return new Schema[]{ADD_OFFSETS_TO_TXN_REQUEST_V0, ADD_OFFSETS_TO_TXN_REQUEST_V1};
}
public static class Builder extends AbstractRequest.Builder<AddOffsetsToTxnRequest> {
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/AddOffsetsToTxnResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/AddOffsetsToTxnResponse.java
index 69b8ad4feae..867ca6afd0e 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/AddOffsetsToTxnResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/AddOffsetsToTxnResponse.java
@@ -32,8 +32,13 @@
THROTTLE_TIME_MS,
ERROR_CODE);
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema ADD_OFFSETS_TO_TXN_RESPONSE_V1 = ADD_OFFSETS_TO_TXN_RESPONSE_V0;
+
public static Schema[] schemaVersions() {
- return new Schema[]{ADD_OFFSETS_TO_TXN_RESPONSE_V0};
+ return new Schema[]{ADD_OFFSETS_TO_TXN_RESPONSE_V0, ADD_OFFSETS_TO_TXN_RESPONSE_V1};
}
// Possible error codes:
@@ -59,6 +64,7 @@ public AddOffsetsToTxnResponse(Struct struct) {
this.error = Errors.forCode(struct.get(ERROR_CODE));
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -92,4 +98,8 @@ public String toString() {
')';
}
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 1;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/AddPartitionsToTxnRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/AddPartitionsToTxnRequest.java
index 0ca32beed1d..4a87289bbb0 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/AddPartitionsToTxnRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/AddPartitionsToTxnRequest.java
@@ -50,8 +50,13 @@
new Field(PARTITIONS_KEY_NAME, new ArrayOf(INT32)))),
"The partitions to add to the transaction."));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema ADD_PARTITIONS_TO_TXN_REQUEST_V1 = ADD_PARTITIONS_TO_TXN_REQUEST_V0;
+
public static Schema[] schemaVersions() {
- return new Schema[]{ADD_PARTITIONS_TO_TXN_REQUEST_V0};
+ return new Schema[]{ADD_PARTITIONS_TO_TXN_REQUEST_V0, ADD_PARTITIONS_TO_TXN_REQUEST_V1};
}
public static class Builder extends AbstractRequest.Builder<AddPartitionsToTxnRequest> {
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/AddPartitionsToTxnResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/AddPartitionsToTxnResponse.java
index 44724499636..977bd595b27 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/AddPartitionsToTxnResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/AddPartitionsToTxnResponse.java
@@ -48,8 +48,13 @@
PARTITION_ID,
ERROR_CODE)))))));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema ADD_PARTITIONS_TO_TXN_RESPONSE_V1 = ADD_PARTITIONS_TO_TXN_RESPONSE_V0;
+
public static Schema[] schemaVersions() {
- return new Schema[]{ADD_PARTITIONS_TO_TXN_RESPONSE_V0};
+ return new Schema[]{ADD_PARTITIONS_TO_TXN_RESPONSE_V0, ADD_PARTITIONS_TO_TXN_RESPONSE_V1};
}
private final int throttleTimeMs;
@@ -86,6 +91,7 @@ public AddPartitionsToTxnResponse(Struct struct) {
}
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -136,4 +142,8 @@ public String toString() {
')';
}
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 1;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/AlterConfigsRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/AlterConfigsRequest.java
index 14b39ae9dac..ed93b150bde 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/AlterConfigsRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/AlterConfigsRequest.java
@@ -60,8 +60,13 @@
"An array of resources to update with the provided configs."),
new Field(VALIDATE_ONLY_KEY_NAME, BOOLEAN));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema ALTER_CONFIGS_REQUEST_V1 = ALTER_CONFIGS_REQUEST_V0;
+
public static Schema[] schemaVersions() {
- return new Schema[] {ALTER_CONFIGS_REQUEST_V0};
+ return new Schema[] {ALTER_CONFIGS_REQUEST_V0, ALTER_CONFIGS_REQUEST_V1};
}
public static class Config {
@@ -187,6 +192,7 @@ public AbstractResponse getErrorResponse(int throttleTimeMs, Throwable e) {
short version = version();
switch (version) {
case 0:
+ case 1:
ApiError error = ApiError.fromThrowable(e);
Map<Resource, ApiError> errors = new HashMap<>(configs.size());
for (Resource resource : configs.keySet())
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/AlterConfigsResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/AlterConfigsResponse.java
index f292ef6731b..feb694bf077 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/AlterConfigsResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/AlterConfigsResponse.java
@@ -52,8 +52,13 @@
THROTTLE_TIME_MS,
new Field(RESOURCES_KEY_NAME, new ArrayOf(ALTER_CONFIGS_RESPONSE_ENTITY_V0)));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema ALTER_CONFIGS_RESPONSE_V1 = ALTER_CONFIGS_RESPONSE_V0;
+
public static Schema[] schemaVersions() {
- return new Schema[]{ALTER_CONFIGS_RESPONSE_V0};
+ return new Schema[]{ALTER_CONFIGS_RESPONSE_V0, ALTER_CONFIGS_RESPONSE_V1};
}
private final int throttleTimeMs;
@@ -87,6 +92,7 @@ public AlterConfigsResponse(Struct struct) {
return apiErrorCounts(errors);
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -112,4 +118,8 @@ public static AlterConfigsResponse parse(ByteBuffer buffer, short version) {
return new AlterConfigsResponse(ApiKeys.ALTER_CONFIGS.parseResponse(version, buffer));
}
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 1;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/AlterReplicaLogDirsRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/AlterReplicaLogDirsRequest.java
index ba217598efb..0bc7cd0c615 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/AlterReplicaLogDirsRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/AlterReplicaLogDirsRequest.java
@@ -55,8 +55,13 @@
TOPIC_NAME,
new Field("partitions", new ArrayOf(INT32), "List of partition ids of the topic."))))))));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema ALTER_REPLICA_LOG_DIRS_REQUEST_V1 = ALTER_REPLICA_LOG_DIRS_REQUEST_V0;
+
public static Schema[] schemaVersions() {
- return new Schema[]{ALTER_REPLICA_LOG_DIRS_REQUEST_V0};
+ return new Schema[]{ALTER_REPLICA_LOG_DIRS_REQUEST_V0, ALTER_REPLICA_LOG_DIRS_REQUEST_V1};
}
private final Map<TopicPartition, String> partitionDirs;
@@ -147,6 +152,7 @@ public AbstractResponse getErrorResponse(int throttleTimeMs, Throwable e) {
short versionId = version();
switch (versionId) {
case 0:
+ case 1:
return new AlterReplicaLogDirsResponse(throttleTimeMs, responseMap);
default:
throw new IllegalArgumentException(
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/AlterReplicaLogDirsResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/AlterReplicaLogDirsResponse.java
index f8d15466d62..c8f6e4df20f 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/AlterReplicaLogDirsResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/AlterReplicaLogDirsResponse.java
@@ -54,8 +54,13 @@
PARTITION_ID,
ERROR_CODE)))))));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema ALTER_REPLICA_LOG_DIRS_RESPONSE_V1 = ALTER_REPLICA_LOG_DIRS_RESPONSE_V0;
+
public static Schema[] schemaVersions() {
- return new Schema[]{ALTER_REPLICA_LOG_DIRS_RESPONSE_V0};
+ return new Schema[]{ALTER_REPLICA_LOG_DIRS_RESPONSE_V0, ALTER_REPLICA_LOG_DIRS_RESPONSE_V1};
}
/**
@@ -116,6 +121,7 @@ protected Struct toStruct(short version) {
return struct;
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -132,4 +138,9 @@ public int throttleTimeMs() {
public static AlterReplicaLogDirsResponse parse(ByteBuffer buffer, short version) {
return new AlterReplicaLogDirsResponse(ApiKeys.ALTER_REPLICA_LOG_DIRS.responseSchema(version).read(buffer));
}
+
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 1;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/ApiVersionsRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/ApiVersionsRequest.java
index 22daf6ce2c0..347e3558fd2 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/ApiVersionsRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/ApiVersionsRequest.java
@@ -30,8 +30,13 @@
/* v1 request is the same as v0. Throttle time has been added to response */
private static final Schema API_VERSIONS_REQUEST_V1 = API_VERSIONS_REQUEST_V0;
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema API_VERSIONS_REQUEST_V2 = API_VERSIONS_REQUEST_V1;
+
public static Schema[] schemaVersions() {
- return new Schema[]{API_VERSIONS_REQUEST_V0, API_VERSIONS_REQUEST_V1};
+ return new Schema[]{API_VERSIONS_REQUEST_V0, API_VERSIONS_REQUEST_V1, API_VERSIONS_REQUEST_V2};
}
public static class Builder extends AbstractRequest.Builder<ApiVersionsRequest> {
@@ -92,6 +97,7 @@ public ApiVersionsResponse getErrorResponse(int throttleTimeMs, Throwable e) {
case 0:
return new ApiVersionsResponse(Errors.forException(e), Collections.<ApiVersionsResponse.ApiVersion>emptyList());
case 1:
+ case 2:
return new ApiVersionsResponse(throttleTimeMs, Errors.forException(e), Collections.<ApiVersionsResponse.ApiVersion>emptyList());
default:
throw new IllegalArgumentException(String.format("Version %d is not valid. Valid versions for %s are 0 to %d",
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/ApiVersionsResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/ApiVersionsResponse.java
index 753db9e94bd..a37b6da0b72 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/ApiVersionsResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/ApiVersionsResponse.java
@@ -54,11 +54,16 @@
new Field(API_VERSIONS_KEY_NAME, new ArrayOf(API_VERSIONS_V0), "API versions supported by the broker."),
THROTTLE_TIME_MS);
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema API_VERSIONS_RESPONSE_V2 = API_VERSIONS_RESPONSE_V1;
+
// initialized lazily to avoid circular initialization dependence with ApiKeys
private static volatile ApiVersionsResponse defaultApiVersionsResponse;
public static Schema[] schemaVersions() {
- return new Schema[]{API_VERSIONS_RESPONSE_V0, API_VERSIONS_RESPONSE_V1};
+ return new Schema[]{API_VERSIONS_RESPONSE_V0, API_VERSIONS_RESPONSE_V1, API_VERSIONS_RESPONSE_V2};
}
/**
@@ -143,6 +148,7 @@ public static ApiVersionsResponse apiVersionsResponse(int throttleTimeMs, byte m
return createApiVersionsResponse(throttleTimeMs, maxMagic);
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -192,4 +198,8 @@ public static ApiVersionsResponse defaultApiVersionsResponse() {
return defaultApiVersionsResponse;
}
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 2;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/CreateAclsRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/CreateAclsRequest.java
index d281b3ba4af..ef4cba25f77 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/CreateAclsRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/CreateAclsRequest.java
@@ -50,8 +50,13 @@
OPERATION,
PERMISSION_TYPE))));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema CREATE_ACLS_REQUEST_V1 = CREATE_ACLS_REQUEST_V0;
+
public static Schema[] schemaVersions() {
- return new Schema[]{CREATE_ACLS_REQUEST_V0};
+ return new Schema[]{CREATE_ACLS_REQUEST_V0, CREATE_ACLS_REQUEST_V1};
}
public static class AclCreation {
@@ -139,6 +144,7 @@ public AbstractResponse getErrorResponse(int throttleTimeMs, Throwable throwable
short versionId = version();
switch (versionId) {
case 0:
+ case 1:
List<CreateAclsResponse.AclCreationResponse> responses = new ArrayList<>();
for (int i = 0; i < aclCreations.size(); i++)
responses.add(new CreateAclsResponse.AclCreationResponse(ApiError.fromThrowable(throwable)));
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/CreateAclsResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/CreateAclsResponse.java
index 6c3579893ce..787ad7a558c 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/CreateAclsResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/CreateAclsResponse.java
@@ -42,8 +42,13 @@
ERROR_CODE,
ERROR_MESSAGE))));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema CREATE_ACLS_RESPONSE_V1 = CREATE_ACLS_RESPONSE_V0;
+
public static Schema[] schemaVersions() {
- return new Schema[]{CREATE_ACLS_RESPONSE_V0};
+ return new Schema[]{CREATE_ACLS_RESPONSE_V0, CREATE_ACLS_RESPONSE_V1};
}
public static class AclCreationResponse {
@@ -96,6 +101,7 @@ protected Struct toStruct(short version) {
return struct;
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -115,4 +121,9 @@ public int throttleTimeMs() {
public static CreateAclsResponse parse(ByteBuffer buffer, short version) {
return new CreateAclsResponse(ApiKeys.CREATE_ACLS.responseSchema(version).read(buffer));
}
+
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 1;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/CreateDelegationTokenRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/CreateDelegationTokenRequest.java
index 7c48205b22b..68e480faba8 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/CreateDelegationTokenRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/CreateDelegationTokenRequest.java
@@ -44,6 +44,11 @@
"Max lifetime period for token in milli seconds. if value is -1, then max lifetime" +
" will default to a server side config value."));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema TOKEN_CREATE_REQUEST_V1 = TOKEN_CREATE_REQUEST_V0;
+
private final List<KafkaPrincipal> renewers;
private final long maxLifeTime;
@@ -73,7 +78,7 @@ public static CreateDelegationTokenRequest parse(ByteBuffer buffer, short versio
}
public static Schema[] schemaVersions() {
- return new Schema[]{TOKEN_CREATE_REQUEST_V0};
+ return new Schema[]{TOKEN_CREATE_REQUEST_V0, TOKEN_CREATE_REQUEST_V1};
}
@Override
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/CreateDelegationTokenResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/CreateDelegationTokenResponse.java
index c718cf0c706..8bb6631140b 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/CreateDelegationTokenResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/CreateDelegationTokenResponse.java
@@ -62,6 +62,11 @@
new Field(HMAC_KEY_NAME, BYTES, "HMAC of the delegation token."),
THROTTLE_TIME_MS);
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema TOKEN_CREATE_RESPONSE_V1 = TOKEN_CREATE_RESPONSE_V0;
+
public CreateDelegationTokenResponse(int throttleTimeMs,
Errors error,
KafkaPrincipal owner,
@@ -103,7 +108,7 @@ public static CreateDelegationTokenResponse parse(ByteBuffer buffer, short versi
}
public static Schema[] schemaVersions() {
- return new Schema[] {TOKEN_CREATE_RESPONSE_V0};
+ return new Schema[] {TOKEN_CREATE_RESPONSE_V0, TOKEN_CREATE_RESPONSE_V1};
}
@Override
@@ -158,6 +163,7 @@ public String tokenId() {
return byteArray;
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -165,4 +171,9 @@ public int throttleTimeMs() {
public boolean hasError() {
return this.error != Errors.NONE;
}
+
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 1;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/CreatePartitionsRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/CreatePartitionsRequest.java
index 41c5327189c..5e776bb4dd3 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/CreatePartitionsRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/CreatePartitionsRequest.java
@@ -60,8 +60,13 @@
new Field(VALIDATE_ONLY_KEY_NAME, BOOLEAN,
"If true then validate the request, but don't actually increase the number of partitions."));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema CREATE_PARTITIONS_REQUEST_V1 = CREATE_PARTITIONS_REQUEST_V0;
+
public static Schema[] schemaVersions() {
- return new Schema[]{CREATE_PARTITIONS_REQUEST_V0};
+ return new Schema[]{CREATE_PARTITIONS_REQUEST_V0, CREATE_PARTITIONS_REQUEST_V1};
}
// It is an error for duplicate topics to be present in the request,
@@ -201,6 +206,7 @@ public AbstractResponse getErrorResponse(int throttleTimeMs, Throwable e) {
short versionId = version();
switch (versionId) {
case 0:
+ case 1:
return new CreatePartitionsResponse(throttleTimeMs, topicErrors);
default:
throw new IllegalArgumentException(String.format("Version %d is not valid. Valid versions for %s are 0 to %d",
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/CreatePartitionsResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/CreatePartitionsResponse.java
index 73a6b565666..744cc32e7b3 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/CreatePartitionsResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/CreatePartitionsResponse.java
@@ -50,8 +50,13 @@
)), "Per topic results for the create partitions request")
);
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema CREATE_PARTITIONS_RESPONSE_V1 = CREATE_PARTITIONS_RESPONSE_V0;
+
public static Schema[] schemaVersions() {
- return new Schema[]{CREATE_PARTITIONS_RESPONSE_V0};
+ return new Schema[]{CREATE_PARTITIONS_RESPONSE_V0, CREATE_PARTITIONS_RESPONSE_V1};
}
private final int throttleTimeMs;
@@ -100,6 +105,7 @@ protected Struct toStruct(short version) {
return apiErrorCounts(errors);
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -108,4 +114,8 @@ public static CreatePartitionsResponse parse(ByteBuffer buffer, short version) {
return new CreatePartitionsResponse(ApiKeys.CREATE_PARTITIONS.parseResponse(version, buffer));
}
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 1;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/CreateTopicsRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/CreateTopicsRequest.java
index 3a1de51f8f2..aa346f5a260 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/CreateTopicsRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/CreateTopicsRequest.java
@@ -91,8 +91,14 @@
/* v2 request is the same as v1. Throttle time has been added to the response */
private static final Schema CREATE_TOPICS_REQUEST_V2 = CREATE_TOPICS_REQUEST_V1;
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema CREATE_TOPICS_REQUEST_V3 = CREATE_TOPICS_REQUEST_V2;
+
public static Schema[] schemaVersions() {
- return new Schema[]{CREATE_TOPICS_REQUEST_V0, CREATE_TOPICS_REQUEST_V1, CREATE_TOPICS_REQUEST_V2};
+ return new Schema[]{CREATE_TOPICS_REQUEST_V0, CREATE_TOPICS_REQUEST_V1, CREATE_TOPICS_REQUEST_V2,
+ CREATE_TOPICS_REQUEST_V3};
}
public static final class TopicDetails {
@@ -270,6 +276,7 @@ public AbstractResponse getErrorResponse(int throttleTimeMs, Throwable e) {
case 1:
return new CreateTopicsResponse(topicErrors);
case 2:
+ case 3:
return new CreateTopicsResponse(throttleTimeMs, topicErrors);
default:
throw new IllegalArgumentException(String.format("Version %d is not valid. Valid versions for %s are 0 to %d",
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/CreateTopicsResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/CreateTopicsResponse.java
index 1f73edac1da..b1504b1acdf 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/CreateTopicsResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/CreateTopicsResponse.java
@@ -58,8 +58,14 @@
THROTTLE_TIME_MS,
new Field(TOPIC_ERRORS_KEY_NAME, new ArrayOf(TOPIC_ERROR), "An array of per topic errors."));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema CREATE_TOPICS_RESPONSE_V3 = CREATE_TOPICS_RESPONSE_V2;
+
public static Schema[] schemaVersions() {
- return new Schema[]{CREATE_TOPICS_RESPONSE_V0, CREATE_TOPICS_RESPONSE_V1, CREATE_TOPICS_RESPONSE_V2};
+ return new Schema[]{CREATE_TOPICS_RESPONSE_V0, CREATE_TOPICS_RESPONSE_V1, CREATE_TOPICS_RESPONSE_V2,
+ CREATE_TOPICS_RESPONSE_V3};
}
/**
@@ -118,6 +124,7 @@ protected Struct toStruct(short version) {
return struct;
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -134,4 +141,9 @@ public int throttleTimeMs() {
public static CreateTopicsResponse parse(ByteBuffer buffer, short version) {
return new CreateTopicsResponse(ApiKeys.CREATE_TOPICS.responseSchema(version).read(buffer));
}
+
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 3;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/DeleteAclsRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/DeleteAclsRequest.java
index 2d50ea65cda..7f53ab59787 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/DeleteAclsRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/DeleteAclsRequest.java
@@ -51,8 +51,13 @@
OPERATION,
PERMISSION_TYPE))));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema DELETE_ACLS_REQUEST_V1 = DELETE_ACLS_REQUEST_V0;
+
public static Schema[] schemaVersions() {
- return new Schema[]{DELETE_ACLS_REQUEST_V0};
+ return new Schema[]{DELETE_ACLS_REQUEST_V0, DELETE_ACLS_REQUEST_V1};
}
public static class Builder extends AbstractRequest.Builder<DeleteAclsRequest> {
@@ -115,6 +120,7 @@ public AbstractResponse getErrorResponse(int throttleTimeMs, Throwable throwable
short versionId = version();
switch (versionId) {
case 0:
+ case 1:
List<DeleteAclsResponse.AclFilterResponse> responses = new ArrayList<>();
for (int i = 0; i < filters.size(); i++) {
responses.add(new DeleteAclsResponse.AclFilterResponse(
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/DeleteAclsResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/DeleteAclsResponse.java
index 7ae25da2c62..58dbb93bdd9 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/DeleteAclsResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/DeleteAclsResponse.java
@@ -69,8 +69,13 @@
ERROR_MESSAGE,
new Field(MATCHING_ACLS_KEY_NAME, new ArrayOf(MATCHING_ACL), "The matching ACLs")))));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema DELETE_ACLS_RESPONSE_V1 = DELETE_ACLS_RESPONSE_V0;
+
public static Schema[] schemaVersions() {
- return new Schema[]{DELETE_ACLS_RESPONSE_V0};
+ return new Schema[]{DELETE_ACLS_RESPONSE_V0, DELETE_ACLS_RESPONSE_V1};
}
public static class AclDeletionResult {
@@ -177,6 +182,7 @@ protected Struct toStruct(short version) {
return struct;
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -201,4 +207,8 @@ public String toString() {
return "(responses=" + Utils.join(responses, ",") + ")";
}
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 1;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/DeleteGroupsRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/DeleteGroupsRequest.java
index bda6b33a18b..29604a505f8 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/DeleteGroupsRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/DeleteGroupsRequest.java
@@ -39,8 +39,13 @@
private static final Schema DELETE_GROUPS_REQUEST_V0 = new Schema(
new Field(GROUPS_KEY_NAME, new ArrayOf(STRING), "An array of groups to be deleted."));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema DELETE_GROUPS_REQUEST_V1 = DELETE_GROUPS_REQUEST_V0;
+
public static Schema[] schemaVersions() {
- return new Schema[]{DELETE_GROUPS_REQUEST_V0};
+ return new Schema[]{DELETE_GROUPS_REQUEST_V0, DELETE_GROUPS_REQUEST_V1};
}
private final Set<String> groups;
@@ -99,6 +104,7 @@ public AbstractResponse getErrorResponse(int throttleTimeMs, Throwable e) {
switch (version()) {
case 0:
+ case 1:
return new DeleteGroupsResponse(throttleTimeMs, groupErrors);
default:
throw new IllegalArgumentException(String.format("Version %d is not valid. Valid versions for %s are 0 to %d",
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/DeleteGroupsResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/DeleteGroupsResponse.java
index d97bb0d1156..032f15de37a 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/DeleteGroupsResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/DeleteGroupsResponse.java
@@ -44,8 +44,13 @@
THROTTLE_TIME_MS,
new Field(GROUP_ERROR_CODES_KEY_NAME, new ArrayOf(GROUP_ERROR_CODE), "An array of per group error codes."));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema DELETE_GROUPS_RESPONSE_V1 = DELETE_GROUPS_RESPONSE_V0;
+
public static Schema[] schemaVersions() {
- return new Schema[]{DELETE_GROUPS_RESPONSE_V0};
+ return new Schema[]{DELETE_GROUPS_RESPONSE_V0, DELETE_GROUPS_RESPONSE_V1};
}
@@ -102,6 +107,7 @@ protected Struct toStruct(short version) {
return struct;
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -126,4 +132,9 @@ public Errors get(String group) {
public static DeleteGroupsResponse parse(ByteBuffer buffer, short version) {
return new DeleteGroupsResponse(ApiKeys.DELETE_GROUPS.responseSchema(version).read(buffer));
}
+
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 1;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/DeleteRecordsRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/DeleteRecordsRequest.java
index 6467e06a218..ad3db607886 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/DeleteRecordsRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/DeleteRecordsRequest.java
@@ -64,8 +64,13 @@
new Field(TOPICS_KEY_NAME, new ArrayOf(DELETE_RECORDS_REQUEST_TOPIC_V0)),
new Field(TIMEOUT_KEY_NAME, INT32, "The maximum time to await a response in ms."));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema DELETE_RECORDS_REQUEST_V1 = DELETE_RECORDS_REQUEST_V0;
+
public static Schema[] schemaVersions() {
- return new Schema[]{DELETE_RECORDS_REQUEST_V0};
+ return new Schema[]{DELETE_RECORDS_REQUEST_V0, DELETE_RECORDS_REQUEST_V1};
}
private final int timeout;
@@ -153,6 +158,7 @@ public AbstractResponse getErrorResponse(int throttleTimeMs, Throwable e) {
short versionId = version();
switch (versionId) {
case 0:
+ case 1:
return new DeleteRecordsResponse(throttleTimeMs, responseMap);
default:
throw new IllegalArgumentException(String.format("Version %d is not valid. Valid versions for %s are 0 to %d",
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/DeleteRecordsResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/DeleteRecordsResponse.java
index 4b4551738ef..0b494ba8c30 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/DeleteRecordsResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/DeleteRecordsResponse.java
@@ -64,8 +64,13 @@
THROTTLE_TIME_MS,
new Field(TOPICS_KEY_NAME, new ArrayOf(DELETE_RECORDS_RESPONSE_TOPIC_V0)));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema DELETE_RECORDS_RESPONSE_V1 = DELETE_RECORDS_RESPONSE_V0;
+
public static Schema[] schemaVersions() {
- return new Schema[]{DELETE_RECORDS_RESPONSE_V0};
+ return new Schema[]{DELETE_RECORDS_RESPONSE_V0, DELETE_RECORDS_RESPONSE_V1};
}
private final int throttleTimeMs;
@@ -152,6 +157,7 @@ protected Struct toStruct(short version) {
return struct;
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -171,4 +177,9 @@ public int throttleTimeMs() {
public static DeleteRecordsResponse parse(ByteBuffer buffer, short version) {
return new DeleteRecordsResponse(ApiKeys.DELETE_RECORDS.responseSchema(version).read(buffer));
}
+
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 1;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/DeleteTopicsRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/DeleteTopicsRequest.java
index 4696b503147..dbcc25d37c6 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/DeleteTopicsRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/DeleteTopicsRequest.java
@@ -46,8 +46,14 @@
/* v1 request is the same as v0. Throttle time has been added to the response */
private static final Schema DELETE_TOPICS_REQUEST_V1 = DELETE_TOPICS_REQUEST_V0;
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema DELETE_TOPICS_REQUEST_V2 = DELETE_TOPICS_REQUEST_V1;
+
public static Schema[] schemaVersions() {
- return new Schema[]{DELETE_TOPICS_REQUEST_V0, DELETE_TOPICS_REQUEST_V1};
+ return new Schema[]{DELETE_TOPICS_REQUEST_V0, DELETE_TOPICS_REQUEST_V1,
+ DELETE_TOPICS_REQUEST_V2};
}
private final Set<String> topics;
@@ -114,6 +120,7 @@ public AbstractResponse getErrorResponse(int throttleTimeMs, Throwable e) {
case 0:
return new DeleteTopicsResponse(topicErrors);
case 1:
+ case 2:
return new DeleteTopicsResponse(throttleTimeMs, topicErrors);
default:
throw new IllegalArgumentException(String.format("Version %d is not valid. Valid versions for %s are 0 to %d",
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/DeleteTopicsResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/DeleteTopicsResponse.java
index d13e4412b6c..db1c4343f9e 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/DeleteTopicsResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/DeleteTopicsResponse.java
@@ -48,8 +48,13 @@
THROTTLE_TIME_MS,
new Field(TOPIC_ERROR_CODES_KEY_NAME, new ArrayOf(TOPIC_ERROR_CODE), "An array of per topic error codes."));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema DELETE_TOPICS_RESPONSE_V2 = DELETE_TOPICS_RESPONSE_V1;
+
public static Schema[] schemaVersions() {
- return new Schema[]{DELETE_TOPICS_RESPONSE_V0, DELETE_TOPICS_RESPONSE_V1};
+ return new Schema[]{DELETE_TOPICS_RESPONSE_V0, DELETE_TOPICS_RESPONSE_V1, DELETE_TOPICS_RESPONSE_V2};
}
@@ -102,6 +107,7 @@ protected Struct toStruct(short version) {
return struct;
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -118,4 +124,9 @@ public int throttleTimeMs() {
public static DeleteTopicsResponse parse(ByteBuffer buffer, short version) {
return new DeleteTopicsResponse(ApiKeys.DELETE_TOPICS.responseSchema(version).read(buffer));
}
+
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 2;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/DescribeAclsRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/DescribeAclsRequest.java
index 1bacac7c7f5..4d6ec60575d 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/DescribeAclsRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/DescribeAclsRequest.java
@@ -43,8 +43,13 @@
OPERATION,
PERMISSION_TYPE);
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema DESCRIBE_ACLS_REQUEST_V1 = DESCRIBE_ACLS_REQUEST_V0;
+
public static Schema[] schemaVersions() {
- return new Schema[]{DESCRIBE_ACLS_REQUEST_V0};
+ return new Schema[]{DESCRIBE_ACLS_REQUEST_V0, DESCRIBE_ACLS_REQUEST_V1};
}
public static class Builder extends AbstractRequest.Builder<DescribeAclsRequest> {
@@ -93,6 +98,7 @@ public AbstractResponse getErrorResponse(int throttleTimeMs, Throwable throwable
short versionId = version();
switch (versionId) {
case 0:
+ case 1:
return new DescribeAclsResponse(throttleTimeMs, ApiError.fromThrowable(throwable),
Collections.<AclBinding>emptySet());
default:
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/DescribeAclsResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/DescribeAclsResponse.java
index a21230bfc39..c705c7116cf 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/DescribeAclsResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/DescribeAclsResponse.java
@@ -63,8 +63,13 @@
ERROR_MESSAGE,
new Field(RESOURCES_KEY_NAME, new ArrayOf(DESCRIBE_ACLS_RESOURCE), "The resources and their associated ACLs."));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema DESCRIBE_ACLS_RESPONSE_V1 = DESCRIBE_ACLS_RESPONSE_V0;
+
public static Schema[] schemaVersions() {
- return new Schema[]{DESCRIBE_ACLS_RESPONSE_V0};
+ return new Schema[]{DESCRIBE_ACLS_RESPONSE_V0, DESCRIBE_ACLS_RESPONSE_V1};
}
private final int throttleTimeMs;
@@ -126,6 +131,7 @@ protected Struct toStruct(short version) {
return struct;
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -146,4 +152,9 @@ public ApiError error() {
public static DescribeAclsResponse parse(ByteBuffer buffer, short version) {
return new DescribeAclsResponse(ApiKeys.DESCRIBE_ACLS.responseSchema(version).read(buffer));
}
+
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 1;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/DescribeConfigsRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/DescribeConfigsRequest.java
index 41563381151..72bb112f497 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/DescribeConfigsRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/DescribeConfigsRequest.java
@@ -54,9 +54,13 @@
new Field(RESOURCES_KEY_NAME, new ArrayOf(DESCRIBE_CONFIGS_REQUEST_RESOURCE_V0), "An array of config resources to be returned."),
new Field(INCLUDE_SYNONYMS, BOOLEAN));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema DESCRIBE_CONFIGS_REQUEST_V2 = DESCRIBE_CONFIGS_REQUEST_V1;
public static Schema[] schemaVersions() {
- return new Schema[]{DESCRIBE_CONFIGS_REQUEST_V0, DESCRIBE_CONFIGS_REQUEST_V1};
+ return new Schema[]{DESCRIBE_CONFIGS_REQUEST_V0, DESCRIBE_CONFIGS_REQUEST_V1, DESCRIBE_CONFIGS_REQUEST_V2};
}
public static class Builder extends AbstractRequest.Builder {
@@ -162,6 +166,7 @@ public DescribeConfigsResponse getErrorResponse(int throttleTimeMs, Throwable e)
switch (version) {
case 0:
case 1:
+ case 2:
ApiError error = ApiError.fromThrowable(e);
Map<Resource, DescribeConfigsResponse.Config> errors = new HashMap<>(resources().size());
DescribeConfigsResponse.Config config = new DescribeConfigsResponse.Config(error,
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/DescribeConfigsResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/DescribeConfigsResponse.java
index e463618e71b..9ae1b5e1c7b 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/DescribeConfigsResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/DescribeConfigsResponse.java
@@ -100,8 +100,13 @@
THROTTLE_TIME_MS,
new Field(RESOURCES_KEY_NAME, new ArrayOf(DESCRIBE_CONFIGS_RESPONSE_ENTITY_V1)));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema DESCRIBE_CONFIGS_RESPONSE_V2 = DESCRIBE_CONFIGS_RESPONSE_V1;
+
public static Schema[] schemaVersions() {
- return new Schema[]{DESCRIBE_CONFIGS_RESPONSE_V0, DESCRIBE_CONFIGS_RESPONSE_V1};
+ return new Schema[]{DESCRIBE_CONFIGS_RESPONSE_V0, DESCRIBE_CONFIGS_RESPONSE_V1, DESCRIBE_CONFIGS_RESPONSE_V2};
}
public static class Config {
@@ -290,6 +295,7 @@ public Config config(Resource resource) {
return configs.get(resource);
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -340,7 +346,7 @@ protected Struct toStruct(short version) {
}
}
resourceStruct.set(CONFIG_ENTRIES_KEY_NAME, configEntryStructs.toArray(new Struct[0]));
-
+
resourceStructs.add(resourceStruct);
}
struct.set(RESOURCES_KEY_NAME, resourceStructs.toArray(new Struct[0]));
@@ -351,4 +357,8 @@ public static DescribeConfigsResponse parse(ByteBuffer buffer, short version) {
return new DescribeConfigsResponse(ApiKeys.DESCRIBE_CONFIGS.parseResponse(version, buffer));
}
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 2;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/DescribeDelegationTokenRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/DescribeDelegationTokenRequest.java
index 61c0055bc5d..574bbcc6460 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/DescribeDelegationTokenRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/DescribeDelegationTokenRequest.java
@@ -39,6 +39,11 @@
public static final Schema TOKEN_DESCRIBE_REQUEST_V0 = new Schema(
new Field(OWNER_KEY_NAME, ArrayOf.nullable(new Schema(PRINCIPAL_TYPE, PRINCIPAL_NAME)), "An array of token owners."));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ public static final Schema TOKEN_DESCRIBE_REQUEST_V1 = TOKEN_DESCRIBE_REQUEST_V0;
+
public static class Builder extends AbstractRequest.Builder<DescribeDelegationTokenRequest> {
// describe token for the given list of owners, or null if we want to describe all tokens.
private final List<KafkaPrincipal> owners;
@@ -86,7 +91,7 @@ public DescribeDelegationTokenRequest(Struct struct, short versionId) {
}
public static Schema[] schemaVersions() {
- return new Schema[]{TOKEN_DESCRIBE_REQUEST_V0};
+ return new Schema[]{TOKEN_DESCRIBE_REQUEST_V0, TOKEN_DESCRIBE_REQUEST_V1};
}
@Override
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/DescribeDelegationTokenResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/DescribeDelegationTokenResponse.java
index dba29eafe99..c86b1413193 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/DescribeDelegationTokenResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/DescribeDelegationTokenResponse.java
@@ -70,6 +70,11 @@
new Field(TOKEN_DETAILS_KEY_NAME, new ArrayOf(TOKEN_DETAILS_V0)),
THROTTLE_TIME_MS);
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema TOKEN_DESCRIBE_RESPONSE_V1 = TOKEN_DESCRIBE_RESPONSE_V0;
+
public DescribeDelegationTokenResponse(int throttleTimeMs, Errors error, List<DelegationToken> tokens) {
this.throttleTimeMs = throttleTimeMs;
this.error = error;
@@ -170,9 +175,10 @@ protected Struct toStruct(short version) {
}
public static Schema[] schemaVersions() {
- return new Schema[]{TOKEN_DESCRIBE_RESPONSE_V0};
+ return new Schema[]{TOKEN_DESCRIBE_RESPONSE_V0, TOKEN_DESCRIBE_RESPONSE_V1};
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -184,4 +190,13 @@ public Errors error() {
public List<DelegationToken> tokens() {
return tokens;
}
+
+ public boolean hasError() {
+ return this.error != Errors.NONE;
+ }
+
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 1;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/DescribeGroupsRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/DescribeGroupsRequest.java
index 56117dad530..8ea4a8c2176 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/DescribeGroupsRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/DescribeGroupsRequest.java
@@ -41,8 +41,13 @@
/* v1 request is the same as v0. Throttle time has been added to response */
private static final Schema DESCRIBE_GROUPS_REQUEST_V1 = DESCRIBE_GROUPS_REQUEST_V0;
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema DESCRIBE_GROUPS_REQUEST_V2 = DESCRIBE_GROUPS_REQUEST_V1;
+
public static Schema[] schemaVersions() {
- return new Schema[]{DESCRIBE_GROUPS_REQUEST_V0, DESCRIBE_GROUPS_REQUEST_V1};
+ return new Schema[]{DESCRIBE_GROUPS_REQUEST_V0, DESCRIBE_GROUPS_REQUEST_V1, DESCRIBE_GROUPS_REQUEST_V2};
}
public static class Builder extends AbstractRequest.Builder<DescribeGroupsRequest> {
@@ -96,6 +101,7 @@ public AbstractResponse getErrorResponse(int throttleTimeMs, Throwable e) {
case 0:
return DescribeGroupsResponse.fromError(Errors.forException(e), groupIds);
case 1:
+ case 2:
return DescribeGroupsResponse.fromError(throttleTimeMs, Errors.forException(e), groupIds);
default:
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/DescribeGroupsResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/DescribeGroupsResponse.java
index 174b14bc744..fbdfaa3200a 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/DescribeGroupsResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/DescribeGroupsResponse.java
@@ -77,8 +77,13 @@
THROTTLE_TIME_MS,
new Field(GROUPS_KEY_NAME, new ArrayOf(DESCRIBE_GROUPS_RESPONSE_GROUP_METADATA_V0)));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema DESCRIBE_GROUPS_RESPONSE_V2 = DESCRIBE_GROUPS_RESPONSE_V1;
+
public static Schema[] schemaVersions() {
- return new Schema[] {DESCRIBE_GROUPS_RESPONSE_V0, DESCRIBE_GROUPS_RESPONSE_V1};
+ return new Schema[] {DESCRIBE_GROUPS_RESPONSE_V0, DESCRIBE_GROUPS_RESPONSE_V1, DESCRIBE_GROUPS_RESPONSE_V2};
}
public static final String UNKNOWN_STATE = "";
@@ -133,6 +138,7 @@ public DescribeGroupsResponse(Struct struct) {
}
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -285,4 +291,9 @@ protected Struct toStruct(short version) {
public static DescribeGroupsResponse parse(ByteBuffer buffer, short version) {
return new DescribeGroupsResponse(ApiKeys.DESCRIBE_GROUPS.parseResponse(version, buffer));
}
+
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 2;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/DescribeLogDirsRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/DescribeLogDirsRequest.java
index 5f35c436ec7..3728991040e 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/DescribeLogDirsRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/DescribeLogDirsRequest.java
@@ -49,8 +49,13 @@
TOPIC_NAME,
new Field(PARTITIONS_KEY_NAME, new ArrayOf(INT32), "List of partition ids of the topic.")))));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema DESCRIBE_LOG_DIRS_REQUEST_V1 = DESCRIBE_LOG_DIRS_REQUEST_V0;
+
public static Schema[] schemaVersions() {
- return new Schema[]{DESCRIBE_LOG_DIRS_REQUEST_V0};
+ return new Schema[]{DESCRIBE_LOG_DIRS_REQUEST_V0, DESCRIBE_LOG_DIRS_REQUEST_V1};
}
private final Set<TopicPartition> topicPartitions;
@@ -137,6 +142,7 @@ public AbstractResponse getErrorResponse(int throttleTimeMs, Throwable e) {
short versionId = version();
switch (versionId) {
case 0:
+ case 1:
return new DescribeLogDirsResponse(throttleTimeMs, new HashMap<String, LogDirInfo>());
default:
throw new IllegalArgumentException(
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/DescribeLogDirsResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/DescribeLogDirsResponse.java
index 9b68a9e7c45..41c26173e3e 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/DescribeLogDirsResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/DescribeLogDirsResponse.java
@@ -77,8 +77,13 @@
"AlterReplicaLogDirsRequest and will replace the current log of the replica " +
"in the future.")))))))))));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema DESCRIBE_LOG_DIRS_RESPONSE_V1 = DESCRIBE_LOG_DIRS_RESPONSE_V0;
+
public static Schema[] schemaVersions() {
- return new Schema[]{DESCRIBE_LOG_DIRS_RESPONSE_V0};
+ return new Schema[]{DESCRIBE_LOG_DIRS_RESPONSE_V0, DESCRIBE_LOG_DIRS_RESPONSE_V1};
}
private final int throttleTimeMs;
@@ -158,6 +163,7 @@ protected Struct toStruct(short version) {
return struct;
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -230,4 +236,9 @@ public String toString() {
return builder.toString();
}
}
+
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 1;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/EndTxnRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/EndTxnRequest.java
index 9118d6a5a08..0ec22dc2831 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/EndTxnRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/EndTxnRequest.java
@@ -38,8 +38,13 @@
PRODUCER_EPOCH,
new Field(TRANSACTION_RESULT_KEY_NAME, BOOLEAN, "The result of the transaction (0 = ABORT, 1 = COMMIT)"));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema END_TXN_REQUEST_V1 = END_TXN_REQUEST_V0;
+
public static Schema[] schemaVersions() {
- return new Schema[]{END_TXN_REQUEST_V0};
+ return new Schema[]{END_TXN_REQUEST_V0, END_TXN_REQUEST_V1};
}
public static class Builder extends AbstractRequest.Builder<EndTxnRequest> {
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/EndTxnResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/EndTxnResponse.java
index a0a453d2f58..cf499966f80 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/EndTxnResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/EndTxnResponse.java
@@ -32,8 +32,13 @@
THROTTLE_TIME_MS,
ERROR_CODE);
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema END_TXN_RESPONSE_V1 = END_TXN_RESPONSE_V0;
+
public static Schema[] schemaVersions() {
- return new Schema[]{END_TXN_RESPONSE_V0};
+ return new Schema[]{END_TXN_RESPONSE_V0, END_TXN_RESPONSE_V1};
}
// Possible error codes:
@@ -58,6 +63,7 @@ public EndTxnResponse(Struct struct) {
this.error = Errors.forCode(struct.get(ERROR_CODE));
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -90,4 +96,9 @@ public String toString() {
", throttleTimeMs=" + throttleTimeMs +
')';
}
+
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 1;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/EpochEndOffset.java b/clients/src/main/java/org/apache/kafka/common/requests/EpochEndOffset.java
index 0965e3612d8..ce938aad4f1 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/EpochEndOffset.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/EpochEndOffset.java
@@ -20,24 +20,29 @@
import static org.apache.kafka.common.record.RecordBatch.NO_PARTITION_LEADER_EPOCH;
+import java.util.Objects;
+
/**
* The offset, fetched from a leader, for a particular partition.
*/
public class EpochEndOffset {
public static final long UNDEFINED_EPOCH_OFFSET = NO_PARTITION_LEADER_EPOCH;
- public static final int UNDEFINED_EPOCH = -1;
+ public static final int UNDEFINED_EPOCH = NO_PARTITION_LEADER_EPOCH;
private Errors error;
+ private int leaderEpoch; // introduced in V1
private long endOffset;
- public EpochEndOffset(Errors error, long endOffset) {
+ public EpochEndOffset(Errors error, int leaderEpoch, long endOffset) {
this.error = error;
+ this.leaderEpoch = leaderEpoch;
this.endOffset = endOffset;
}
- public EpochEndOffset(long endOffset) {
+ public EpochEndOffset(int leaderEpoch, long endOffset) {
this.error = Errors.NONE;
+ this.leaderEpoch = leaderEpoch;
this.endOffset = endOffset;
}
@@ -53,10 +58,15 @@ public long endOffset() {
return endOffset;
}
+ public int leaderEpoch() {
+ return leaderEpoch;
+ }
+
@Override
public String toString() {
return "EpochEndOffset{" +
"error=" + error +
+ ", leaderEpoch=" + leaderEpoch +
", endOffset=" + endOffset +
'}';
}
@@ -68,14 +78,13 @@ public boolean equals(Object o) {
EpochEndOffset that = (EpochEndOffset) o;
- if (error != that.error) return false;
- return endOffset == that.endOffset;
+ return Objects.equals(error, that.error)
+ && Objects.equals(leaderEpoch, that.leaderEpoch)
+ && Objects.equals(endOffset, that.endOffset);
}
@Override
public int hashCode() {
- int result = (int) error.code();
- result = 31 * result + (int) (endOffset ^ (endOffset >>> 32));
- return result;
+ return Objects.hash(error, leaderEpoch, endOffset);
}
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/ExpireDelegationTokenRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/ExpireDelegationTokenRequest.java
index 0d43440d329..21edb5e2f30 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/ExpireDelegationTokenRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/ExpireDelegationTokenRequest.java
@@ -38,6 +38,11 @@
new Field(HMAC_KEY_NAME, BYTES, "HMAC of the delegation token to be expired."),
new Field(EXPIRY_TIME_PERIOD_KEY_NAME, INT64, "expiry time period in milli seconds."));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema TOKEN_EXPIRE_REQUEST_V1 = TOKEN_EXPIRE_REQUEST_V0;
+
private ExpireDelegationTokenRequest(short version, ByteBuffer hmac, long renewTimePeriod) {
super(version);
@@ -57,7 +62,7 @@ public static ExpireDelegationTokenRequest parse(ByteBuffer buffer, short versio
}
public static Schema[] schemaVersions() {
- return new Schema[] {TOKEN_EXPIRE_REQUEST_V0};
+ return new Schema[] {TOKEN_EXPIRE_REQUEST_V0, TOKEN_EXPIRE_REQUEST_V1};
}
@Override
@@ -88,9 +93,9 @@ public long expiryTimePeriod() {
private final ByteBuffer hmac;
private final long expiryTimePeriod;
- public Builder(ByteBuffer hmac, long expiryTimePeriod) {
+ public Builder(byte[] hmac, long expiryTimePeriod) {
super(ApiKeys.EXPIRE_DELEGATION_TOKEN);
- this.hmac = hmac;
+ this.hmac = ByteBuffer.wrap(hmac);
this.expiryTimePeriod = expiryTimePeriod;
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/ExpireDelegationTokenResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/ExpireDelegationTokenResponse.java
index f7e0ec44168..9491a3546a1 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/ExpireDelegationTokenResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/ExpireDelegationTokenResponse.java
@@ -42,6 +42,11 @@
new Field(EXPIRY_TIMESTAMP_KEY_NAME, INT64, "timestamp (in msec) at which this token expires.."),
THROTTLE_TIME_MS);
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema TOKEN_EXPIRE_RESPONSE_V1 = TOKEN_EXPIRE_RESPONSE_V0;
+
public ExpireDelegationTokenResponse(int throttleTimeMs, Errors error, long expiryTimestamp) {
this.throttleTimeMs = throttleTimeMs;
this.error = error;
@@ -63,7 +68,7 @@ public static ExpireDelegationTokenResponse parse(ByteBuffer buffer, short versi
}
public static Schema[] schemaVersions() {
- return new Schema[] {TOKEN_EXPIRE_RESPONSE_V0};
+ return new Schema[] {TOKEN_EXPIRE_RESPONSE_V0, TOKEN_EXPIRE_RESPONSE_V1};
}
public Errors error() {
@@ -90,7 +95,17 @@ protected Struct toStruct(short version) {
return struct;
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
+
+ public boolean hasError() {
+ return this.error != Errors.NONE;
+ }
+
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 1;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/FetchRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/FetchRequest.java
index 65cf7fea6b1..6e25f7cef66 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/FetchRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/FetchRequest.java
@@ -52,7 +52,7 @@
private static final String MIN_BYTES_KEY_NAME = "min_bytes";
private static final String ISOLATION_LEVEL_KEY_NAME = "isolation_level";
private static final String TOPICS_KEY_NAME = "topics";
- private static final String FORGOTTEN_TOPICS_DATA = "forgetten_topics_data";
+ private static final String FORGOTTEN_TOPICS_DATA = "forgotten_topics_data";
// request and partition level name
private static final String MAX_BYTES_KEY_NAME = "max_bytes";
@@ -175,9 +175,14 @@
new Field(TOPICS_KEY_NAME, new ArrayOf(FETCH_REQUEST_TOPIC_V5), "Topics to fetch in the order provided."),
new Field(FORGOTTEN_TOPICS_DATA, new ArrayOf(FORGOTTEN_TOPIC_DATA), "Topics to remove from the fetch session."));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema FETCH_REQUEST_V8 = FETCH_REQUEST_V7;
+
public static Schema[] schemaVersions() {
return new Schema[]{FETCH_REQUEST_V0, FETCH_REQUEST_V1, FETCH_REQUEST_V2, FETCH_REQUEST_V3, FETCH_REQUEST_V4,
- FETCH_REQUEST_V5, FETCH_REQUEST_V6, FETCH_REQUEST_V7};
+ FETCH_REQUEST_V5, FETCH_REQUEST_V6, FETCH_REQUEST_V7, FETCH_REQUEST_V8};
};
// default values for older versions where a request level limit did not exist
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/FetchResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/FetchResponse.java
index ed0f5a347dd..16e33965e9e 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/FetchResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/FetchResponse.java
@@ -18,7 +18,7 @@
import org.apache.kafka.common.TopicPartition;
import org.apache.kafka.common.network.ByteBufferSend;
-import org.apache.kafka.common.network.MultiSend;
+import org.apache.kafka.common.record.MultiRecordsSend;
import org.apache.kafka.common.network.Send;
import org.apache.kafka.common.protocol.ApiKeys;
import org.apache.kafka.common.protocol.Errors;
@@ -26,7 +26,8 @@
import org.apache.kafka.common.protocol.types.Field;
import org.apache.kafka.common.protocol.types.Schema;
import org.apache.kafka.common.protocol.types.Struct;
-import org.apache.kafka.common.record.Records;
+import org.apache.kafka.common.record.BaseRecords;
+import org.apache.kafka.common.record.MemoryRecords;
import java.nio.ByteBuffer;
import java.util.ArrayDeque;
@@ -50,7 +51,7 @@
/**
* This wrapper supports all versions of the Fetch API
*/
-public class FetchResponse extends AbstractResponse {
+public class FetchResponse<T extends BaseRecords> extends AbstractResponse {
private static final String RESPONSES_KEY_NAME = "responses";
@@ -152,19 +153,24 @@
*/
private static final Schema FETCH_RESPONSE_V6 = FETCH_RESPONSE_V5;
- // FETCH_REESPONSE_V7 added incremental fetch responses and a top-level error code.
+ // FETCH_RESPONSE_V7 added incremental fetch responses and a top-level error code.
public static final Field.Int32 SESSION_ID = new Field.Int32("session_id", "The fetch session ID");
private static final Schema FETCH_RESPONSE_V7 = new Schema(
- THROTTLE_TIME_MS,
- ERROR_CODE,
- SESSION_ID,
- new Field(RESPONSES_KEY_NAME, new ArrayOf(FETCH_RESPONSE_TOPIC_V5)));
+ THROTTLE_TIME_MS,
+ ERROR_CODE,
+ SESSION_ID,
+ new Field(RESPONSES_KEY_NAME, new ArrayOf(FETCH_RESPONSE_TOPIC_V5)));
+
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema FETCH_RESPONSE_V8 = FETCH_RESPONSE_V7;
public static Schema[] schemaVersions() {
return new Schema[] {FETCH_RESPONSE_V0, FETCH_RESPONSE_V1, FETCH_RESPONSE_V2,
FETCH_RESPONSE_V3, FETCH_RESPONSE_V4, FETCH_RESPONSE_V5, FETCH_RESPONSE_V6,
- FETCH_RESPONSE_V7};
+ FETCH_RESPONSE_V7, FETCH_RESPONSE_V8};
}
@@ -185,7 +191,7 @@
private final int throttleTimeMs;
private final Errors error;
private final int sessionId;
- private final LinkedHashMap<TopicPartition, PartitionData> responseData;
+ private final LinkedHashMap<TopicPartition, PartitionData<T>> responseData;
public static final class AbortedTransaction {
public final long producerId;
@@ -221,20 +227,20 @@ public String toString() {
}
}
- public static final class PartitionData {
+ public static final class PartitionData<T extends BaseRecords> {
public final Errors error;
public final long highWatermark;
public final long lastStableOffset;
public final long logStartOffset;
public final List<AbortedTransaction> abortedTransactions;
- public final Records records;
+ public final T records;
public PartitionData(Errors error,
long highWatermark,
long lastStableOffset,
long logStartOffset,
List<AbortedTransaction> abortedTransactions,
- Records records) {
+ T records) {
this.error = error;
this.highWatermark = highWatermark;
this.lastStableOffset = lastStableOffset;
@@ -292,16 +298,18 @@ public String toString() {
* @param throttleTimeMs The time in milliseconds that the response was throttled
* @param sessionId The fetch session id.
*/
- public FetchResponse(Errors error, LinkedHashMap<TopicPartition, PartitionData> responseData,
- int throttleTimeMs, int sessionId) {
+ public FetchResponse(Errors error,
+ LinkedHashMap<TopicPartition, PartitionData<T>> responseData,
+ int throttleTimeMs,
+ int sessionId) {
this.error = error;
this.responseData = responseData;
this.throttleTimeMs = throttleTimeMs;
this.sessionId = sessionId;
}
- public FetchResponse(Struct struct) {
- LinkedHashMap<TopicPartition, PartitionData> responseData = new LinkedHashMap<>();
+ public static FetchResponse<MemoryRecords> parse(Struct struct) {
+ LinkedHashMap<TopicPartition, PartitionData<MemoryRecords>> responseData = new LinkedHashMap<>();
for (Object topicResponseObj : struct.getArray(RESPONSES_KEY_NAME)) {
Struct topicResponse = (Struct) topicResponseObj;
String topic = topicResponse.get(TOPIC_NAME);
@@ -318,7 +326,10 @@ public FetchResponse(Struct struct) {
if (partitionResponseHeader.hasField(LOG_START_OFFSET_KEY_NAME))
logStartOffset = partitionResponseHeader.getLong(LOG_START_OFFSET_KEY_NAME);
- Records records = partitionResponse.getRecords(RECORD_SET_KEY_NAME);
+ BaseRecords baseRecords = partitionResponse.getRecords(RECORD_SET_KEY_NAME);
+ if (!(baseRecords instanceof MemoryRecords))
+ throw new IllegalStateException("Unknown records type found: " + baseRecords.getClass());
+ MemoryRecords records = (MemoryRecords) baseRecords;
List<AbortedTransaction> abortedTransactions = null;
if (partitionResponseHeader.hasField(ABORTED_TRANSACTIONS_KEY_NAME)) {
@@ -334,15 +345,13 @@ public FetchResponse(Struct struct) {
}
}
- PartitionData partitionData = new PartitionData(error, highWatermark, lastStableOffset, logStartOffset,
- abortedTransactions, records);
+ PartitionData<MemoryRecords> partitionData = new PartitionData<>(error, highWatermark, lastStableOffset,
+ logStartOffset, abortedTransactions, records);
responseData.put(new TopicPartition(topic, partition), partitionData);
}
}
- this.responseData = responseData;
- this.throttleTimeMs = struct.getOrElse(THROTTLE_TIME_MS, DEFAULT_THROTTLE_TIME);
- this.error = Errors.forCode(struct.getOrElse(ERROR_CODE, (short) 0));
- this.sessionId = struct.getOrElse(SESSION_ID, INVALID_SESSION_ID);
+ return new FetchResponse<>(Errors.forCode(struct.getOrElse(ERROR_CODE, (short) 0)), responseData,
+ struct.getOrElse(THROTTLE_TIME_MS, DEFAULT_THROTTLE_TIME), struct.getOrElse(SESSION_ID, INVALID_SESSION_ID));
}
@Override
@@ -364,17 +373,18 @@ protected Send toSend(String dest, ResponseHeader responseHeader, short apiVersi
Queue<Send> sends = new ArrayDeque<>();
sends.add(new ByteBufferSend(dest, buffer));
addResponseData(responseBodyStruct, throttleTimeMs, dest, sends);
- return new MultiSend(dest, sends);
+ return new MultiRecordsSend(dest, sends);
}
public Errors error() {
return error;
}
- public LinkedHashMap<TopicPartition, PartitionData> responseData() {
+ public LinkedHashMap<TopicPartition, PartitionData<T>> responseData() {
return responseData;
}
+ @Override
public int throttleTimeMs() {
return this.throttleTimeMs;
}
@@ -391,8 +401,8 @@ public int sessionId() {
return errorCounts;
}
- public static FetchResponse parse(ByteBuffer buffer, short version) {
- return new FetchResponse(ApiKeys.FETCH.responseSchema(version).read(buffer));
+ public static FetchResponse<MemoryRecords> parse(ByteBuffer buffer, short version) {
+ return parse(ApiKeys.FETCH.responseSchema(version).read(buffer));
}
private static void addResponseData(Struct struct, int throttleTimeMs, String dest, Queue<Send> sends) {
@@ -440,7 +450,7 @@ private static void addTopicData(String dest, Queue<Send> sends, Struct topicDat
private static void addPartitionData(String dest, Queue<Send> sends, Struct partitionData) {
Struct header = partitionData.getStruct(PARTITION_HEADER_KEY_NAME);
- Records records = partitionData.getRecords(RECORD_SET_KEY_NAME);
+ BaseRecords records = partitionData.getRecords(RECORD_SET_KEY_NAME);
// include the partition header and the size of the record set
ByteBuffer buffer = ByteBuffer.allocate(header.sizeOf() + 4);
@@ -450,24 +460,25 @@ private static void addPartitionData(String dest, Queue<Send> sends, Struct part
sends.add(new ByteBufferSend(dest, buffer));
// finally the send for the record set itself
- sends.add(new RecordsSend(dest, records));
+ sends.add(records.toSend(dest));
}
- private static Struct toStruct(short version, int throttleTimeMs, Errors error,
- Iterator<Map.Entry<TopicPartition, PartitionData>> partIterator, int sessionId) {
+ private static <T extends BaseRecords> Struct toStruct(short version, int throttleTimeMs, Errors error,
+ Iterator<Map.Entry<TopicPartition, PartitionData<T>>> partIterator,
+ int sessionId) {
Struct struct = new Struct(ApiKeys.FETCH.responseSchema(version));
struct.setIfExists(THROTTLE_TIME_MS, throttleTimeMs);
struct.setIfExists(ERROR_CODE, error.code());
struct.setIfExists(SESSION_ID, sessionId);
- List<FetchRequest.TopicAndPartitionData<PartitionData>> topicsData =
- FetchRequest.TopicAndPartitionData.batchByTopic(partIterator);
+ List<FetchRequest.TopicAndPartitionData<PartitionData<T>>> topicsData =
+ FetchRequest.TopicAndPartitionData.batchByTopic(partIterator);
List<Struct> topicArray = new ArrayList<>();
- for (FetchRequest.TopicAndPartitionData<PartitionData> topicEntry: topicsData) {
+ for (FetchRequest.TopicAndPartitionData<PartitionData<T>> topicEntry: topicsData) {
Struct topicData = struct.instance(RESPONSES_KEY_NAME);
topicData.set(TOPIC_NAME, topicEntry.topic);
List<Struct> partitionArray = new ArrayList<>();
- for (Map.Entry<Integer, PartitionData> partitionEntry : topicEntry.partitions.entrySet()) {
- PartitionData fetchPartitionData = partitionEntry.getValue();
+ for (Map.Entry<Integer, PartitionData<T>> partitionEntry : topicEntry.partitions.entrySet()) {
+ PartitionData<T> fetchPartitionData = partitionEntry.getValue();
short errorCode = fetchPartitionData.error.code();
// If consumer sends FetchRequest V5 or earlier, the client library is not guaranteed to recognize the error code
// for KafkaStorageException. In this case the client library will translate KafkaStorageException to
@@ -518,10 +529,15 @@ private static Struct toStruct(short version, int throttleTimeMs, Errors error,
* @param partIterator The partition iterator.
* @return The response size in bytes.
*/
- public static int sizeOf(short version, Iterator<Map.Entry<TopicPartition, PartitionData>> partIterator) {
+ public static <T extends BaseRecords> int sizeOf(short version,
+ Iterator<Map.Entry<TopicPartition, PartitionData<T>>> partIterator) {
// Since the throttleTimeMs and metadata field sizes are constant and fixed, we can
// use arbitrary values here without affecting the result.
return 4 + toStruct(version, 0, Errors.NONE, partIterator, INVALID_SESSION_ID).sizeOf();
}
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 8;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/FindCoordinatorRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/FindCoordinatorRequest.java
index 9bfc968d55f..8fb71a87c07 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/FindCoordinatorRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/FindCoordinatorRequest.java
@@ -41,8 +41,13 @@
"for transactional producers, this is the transactional id)"),
new Field("coordinator_type", INT8, "The type of coordinator to find (0 = group, 1 = transaction)"));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema FIND_COORDINATOR_REQUEST_V2 = FIND_COORDINATOR_REQUEST_V1;
+
public static Schema[] schemaVersions() {
- return new Schema[] {FIND_COORDINATOR_REQUEST_V0, FIND_COORDINATOR_REQUEST_V1};
+ return new Schema[] {FIND_COORDINATOR_REQUEST_V0, FIND_COORDINATOR_REQUEST_V1, FIND_COORDINATOR_REQUEST_V2};
}
public static class Builder extends AbstractRequest.Builder<FindCoordinatorRequest> {
@@ -114,6 +119,7 @@ public AbstractResponse getErrorResponse(int throttleTimeMs, Throwable e) {
case 0:
return new FindCoordinatorResponse(Errors.forException(e), Node.noNode());
case 1:
+ case 2:
return new FindCoordinatorResponse(throttleTimeMs, Errors.forException(e), Node.noNode());
default:
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/FindCoordinatorResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/FindCoordinatorResponse.java
index bddc41f770a..39726dac6b9 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/FindCoordinatorResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/FindCoordinatorResponse.java
@@ -56,8 +56,13 @@
ERROR_MESSAGE,
new Field(COORDINATOR_KEY_NAME, FIND_COORDINATOR_BROKER_V0, "Host and port information for the coordinator"));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema FIND_COORDINATOR_RESPONSE_V2 = FIND_COORDINATOR_RESPONSE_V1;
+
public static Schema[] schemaVersions() {
- return new Schema[] {FIND_COORDINATOR_RESPONSE_V0, FIND_COORDINATOR_RESPONSE_V1};
+ return new Schema[] {FIND_COORDINATOR_RESPONSE_V0, FIND_COORDINATOR_RESPONSE_V1, FIND_COORDINATOR_RESPONSE_V2};
}
/**
@@ -97,6 +102,7 @@ public FindCoordinatorResponse(Struct struct) {
node = new Node(nodeId, host, port);
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -142,4 +148,9 @@ public String toString() {
", node=" + node +
')';
}
+
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 2;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/HeartbeatRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/HeartbeatRequest.java
index 7d84918d3f4..5a70ed87dcd 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/HeartbeatRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/HeartbeatRequest.java
@@ -36,8 +36,14 @@
/* v1 request is the same as v0. Throttle time has been added to response */
private static final Schema HEARTBEAT_REQUEST_V1 = HEARTBEAT_REQUEST_V0;
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema HEARTBEAT_REQUEST_V2 = HEARTBEAT_REQUEST_V1;
+
public static Schema[] schemaVersions() {
- return new Schema[] {HEARTBEAT_REQUEST_V0, HEARTBEAT_REQUEST_V1};
+ return new Schema[] {HEARTBEAT_REQUEST_V0, HEARTBEAT_REQUEST_V1,
+ HEARTBEAT_REQUEST_V2};
}
public static class Builder extends AbstractRequest.Builder<HeartbeatRequest> {
@@ -94,6 +100,7 @@ public AbstractResponse getErrorResponse(int throttleTimeMs, Throwable e) {
case 0:
return new HeartbeatResponse(Errors.forException(e));
case 1:
+ case 2:
return new HeartbeatResponse(throttleTimeMs, Errors.forException(e));
default:
throw new IllegalArgumentException(String.format("Version %d is not valid. Valid versions for %s are 0 to %d",
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/HeartbeatResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/HeartbeatResponse.java
index b52a9936872..efe2ed89f29 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/HeartbeatResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/HeartbeatResponse.java
@@ -35,8 +35,14 @@
THROTTLE_TIME_MS,
ERROR_CODE);
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema HEARTBEAT_RESPONSE_V2 = HEARTBEAT_RESPONSE_V1;
+
public static Schema[] schemaVersions() {
- return new Schema[] {HEARTBEAT_RESPONSE_V0, HEARTBEAT_RESPONSE_V1};
+ return new Schema[] {HEARTBEAT_RESPONSE_V0, HEARTBEAT_RESPONSE_V1,
+ HEARTBEAT_RESPONSE_V2};
}
/**
@@ -66,6 +72,7 @@ public HeartbeatResponse(Struct struct) {
error = Errors.forCode(struct.get(ERROR_CODE));
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -90,4 +97,9 @@ protected Struct toStruct(short version) {
public static HeartbeatResponse parse(ByteBuffer buffer, short version) {
return new HeartbeatResponse(ApiKeys.HEARTBEAT.parseResponse(version, buffer));
}
+
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 2;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/InitProducerIdRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/InitProducerIdRequest.java
index 6c659ff207a..c3505997102 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/InitProducerIdRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/InitProducerIdRequest.java
@@ -36,8 +36,13 @@
NULLABLE_TRANSACTIONAL_ID,
new Field(TRANSACTION_TIMEOUT_KEY_NAME, INT32, "The time in ms to wait for before aborting idle transactions sent by this producer."));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema INIT_PRODUCER_ID_REQUEST_V1 = INIT_PRODUCER_ID_REQUEST_V0;
+
public static Schema[] schemaVersions() {
- return new Schema[]{INIT_PRODUCER_ID_REQUEST_V0};
+ return new Schema[]{INIT_PRODUCER_ID_REQUEST_V0, INIT_PRODUCER_ID_REQUEST_V1};
}
private final String transactionalId;
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/InitProducerIdResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/InitProducerIdResponse.java
index 7a988ca806a..9a1e0f744cd 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/InitProducerIdResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/InitProducerIdResponse.java
@@ -44,8 +44,13 @@
PRODUCER_ID,
PRODUCER_EPOCH);
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema INIT_PRODUCER_ID_RESPONSE_V1 = INIT_PRODUCER_ID_RESPONSE_V0;
+
public static Schema[] schemaVersions() {
- return new Schema[]{INIT_PRODUCER_ID_RESPONSE_V0};
+ return new Schema[]{INIT_PRODUCER_ID_RESPONSE_V0, INIT_PRODUCER_ID_RESPONSE_V1};
}
private final int throttleTimeMs;
@@ -71,6 +76,7 @@ public InitProducerIdResponse(int throttleTimeMs, Errors errors) {
this(throttleTimeMs, errors, RecordBatch.NO_PRODUCER_ID, (short) 0);
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -115,4 +121,9 @@ public String toString() {
", throttleTimeMs=" + throttleTimeMs +
')';
}
+
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 1;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/JoinGroupRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/JoinGroupRequest.java
index a7b62a98660..ba009a1ae2b 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/JoinGroupRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/JoinGroupRequest.java
@@ -71,8 +71,14 @@
/* v2 request is the same as v1. Throttle time has been added to response */
private static final Schema JOIN_GROUP_REQUEST_V2 = JOIN_GROUP_REQUEST_V1;
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema JOIN_GROUP_REQUEST_V3 = JOIN_GROUP_REQUEST_V2;
+
public static Schema[] schemaVersions() {
- return new Schema[] {JOIN_GROUP_REQUEST_V0, JOIN_GROUP_REQUEST_V1, JOIN_GROUP_REQUEST_V2};
+ return new Schema[] {JOIN_GROUP_REQUEST_V0, JOIN_GROUP_REQUEST_V1, JOIN_GROUP_REQUEST_V2,
+ JOIN_GROUP_REQUEST_V3};
}
public static final String UNKNOWN_MEMBER_ID = "";
@@ -202,6 +208,7 @@ public AbstractResponse getErrorResponse(int throttleTimeMs, Throwable e) {
JoinGroupResponse.UNKNOWN_MEMBER_ID, // leaderId
Collections.<String, ByteBuffer>emptyMap());
case 2:
+ case 3:
return new JoinGroupResponse(
throttleTimeMs,
Errors.forException(e),
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/JoinGroupResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/JoinGroupResponse.java
index 4bcd6e6d65f..e3efe86defd 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/JoinGroupResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/JoinGroupResponse.java
@@ -68,9 +68,14 @@
MEMBER_ID,
new Field(MEMBERS_KEY_NAME, new ArrayOf(JOIN_GROUP_RESPONSE_MEMBER_V0)));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema JOIN_GROUP_RESPONSE_V3 = JOIN_GROUP_RESPONSE_V2;
public static Schema[] schemaVersions() {
- return new Schema[] {JOIN_GROUP_RESPONSE_V0, JOIN_GROUP_RESPONSE_V1, JOIN_GROUP_RESPONSE_V2};
+ return new Schema[] {JOIN_GROUP_RESPONSE_V0, JOIN_GROUP_RESPONSE_V1, JOIN_GROUP_RESPONSE_V2,
+ JOIN_GROUP_RESPONSE_V3};
}
public static final String UNKNOWN_PROTOCOL = "";
@@ -139,6 +144,7 @@ public JoinGroupResponse(Struct struct) {
leaderId = struct.getString(LEADER_ID_KEY_NAME);
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -215,4 +221,9 @@ public String toString() {
", members=" + ((members == null) ? "null" :
Utils.join(members.keySet(), ",")) + ")";
}
+
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 3;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/LeaderAndIsrResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/LeaderAndIsrResponse.java
index c21f9a783b3..8e6324589df 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/LeaderAndIsrResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/LeaderAndIsrResponse.java
@@ -119,4 +119,13 @@ protected Struct toStruct(short version) {
return struct;
}
+
+ @Override
+ public String toString() {
+ return "LeaderAndIsrResponse(" +
+ "responses=" + responses +
+ ", error=" + error +
+ ")";
+ }
+
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/LeaveGroupRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/LeaveGroupRequest.java
index b0d0ad6d685..2b4acf876ff 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/LeaveGroupRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/LeaveGroupRequest.java
@@ -34,8 +34,14 @@
/* v1 request is the same as v0. Throttle time has been added to response */
private static final Schema LEAVE_GROUP_REQUEST_V1 = LEAVE_GROUP_REQUEST_V0;
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema LEAVE_GROUP_REQUEST_V2 = LEAVE_GROUP_REQUEST_V1;
+
public static Schema[] schemaVersions() {
- return new Schema[] {LEAVE_GROUP_REQUEST_V0, LEAVE_GROUP_REQUEST_V1};
+ return new Schema[] {LEAVE_GROUP_REQUEST_V0, LEAVE_GROUP_REQUEST_V1,
+ LEAVE_GROUP_REQUEST_V2};
}
public static class Builder extends AbstractRequest.Builder<LeaveGroupRequest> {
@@ -86,6 +92,7 @@ public AbstractResponse getErrorResponse(int throttleTimeMs, Throwable e) {
case 0:
return new LeaveGroupResponse(Errors.forException(e));
case 1:
+ case 2:
return new LeaveGroupResponse(throttleTimeMs, Errors.forException(e));
default:
throw new IllegalArgumentException(String.format("Version %d is not valid. Valid versions for %s are 0 to %d",
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/LeaveGroupResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/LeaveGroupResponse.java
index f8682ecde71..c2656f2489c 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/LeaveGroupResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/LeaveGroupResponse.java
@@ -35,8 +35,14 @@
THROTTLE_TIME_MS,
ERROR_CODE);
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema LEAVE_GROUP_RESPONSE_V2 = LEAVE_GROUP_RESPONSE_V1;
+
public static Schema[] schemaVersions() {
- return new Schema[] {LEAVE_GROUP_RESPONSE_V0, LEAVE_GROUP_RESPONSE_V1};
+ return new Schema[] {LEAVE_GROUP_RESPONSE_V0, LEAVE_GROUP_RESPONSE_V1,
+ LEAVE_GROUP_RESPONSE_V2};
}
/**
@@ -65,6 +71,7 @@ public LeaveGroupResponse(Struct struct) {
this.error = Errors.forCode(struct.get(ERROR_CODE));
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -90,4 +97,8 @@ public static LeaveGroupResponse parse(ByteBuffer buffer, short versionId) {
return new LeaveGroupResponse(ApiKeys.LEAVE_GROUP.parseResponse(versionId, buffer));
}
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 2;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/ListGroupsRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/ListGroupsRequest.java
index f279b4c618f..f64f71a56e3 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/ListGroupsRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/ListGroupsRequest.java
@@ -32,8 +32,14 @@
/* v1 request is the same as v0. Throttle time has been added to response */
private static final Schema LIST_GROUPS_REQUEST_V1 = LIST_GROUPS_REQUEST_V0;
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema LIST_GROUPS_REQUEST_V2 = LIST_GROUPS_REQUEST_V1;
+
public static Schema[] schemaVersions() {
- return new Schema[] {LIST_GROUPS_REQUEST_V0, LIST_GROUPS_REQUEST_V1};
+ return new Schema[] {LIST_GROUPS_REQUEST_V0, LIST_GROUPS_REQUEST_V1,
+ LIST_GROUPS_REQUEST_V2};
}
public static class Builder extends AbstractRequest.Builder<ListGroupsRequest> {
@@ -67,6 +73,7 @@ public ListGroupsResponse getErrorResponse(int throttleTimeMs, Throwable e) {
case 0:
return new ListGroupsResponse(Errors.forException(e), Collections.<ListGroupsResponse.Group>emptyList());
case 1:
+ case 2:
return new ListGroupsResponse(throttleTimeMs, Errors.forException(e), Collections.<ListGroupsResponse.Group>emptyList());
default:
throw new IllegalArgumentException(String.format("Version %d is not valid. Valid versions for %s are 0 to %d",
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/ListGroupsResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/ListGroupsResponse.java
index 9c82ae06bd6..b108803590f 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/ListGroupsResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/ListGroupsResponse.java
@@ -49,8 +49,14 @@
ERROR_CODE,
new Field(GROUPS_KEY_NAME, new ArrayOf(LIST_GROUPS_RESPONSE_GROUP_V0)));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema LIST_GROUPS_RESPONSE_V2 = LIST_GROUPS_RESPONSE_V1;
+
public static Schema[] schemaVersions() {
- return new Schema[] {LIST_GROUPS_RESPONSE_V0, LIST_GROUPS_RESPONSE_V1};
+ return new Schema[] {LIST_GROUPS_RESPONSE_V0, LIST_GROUPS_RESPONSE_V1,
+ LIST_GROUPS_RESPONSE_V2};
}
/**
@@ -86,6 +92,7 @@ public ListGroupsResponse(Struct struct) {
}
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -142,4 +149,8 @@ public static ListGroupsResponse parse(ByteBuffer buffer, short version) {
return new ListGroupsResponse(ApiKeys.LIST_GROUPS.parseResponse(version, buffer));
}
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 2;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/ListOffsetRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/ListOffsetRequest.java
index 98f53bdfc44..be094feec55 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/ListOffsetRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/ListOffsetRequest.java
@@ -90,9 +90,14 @@
"result, which allows consumers to discard ABORTED transactional records"),
new Field(TOPICS_KEY_NAME, new ArrayOf(LIST_OFFSET_REQUEST_TOPIC_V1), "Topics to list offsets."));;
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema LIST_OFFSET_REQUEST_V3 = LIST_OFFSET_REQUEST_V2;
public static Schema[] schemaVersions() {
- return new Schema[] {LIST_OFFSET_REQUEST_V0, LIST_OFFSET_REQUEST_V1, LIST_OFFSET_REQUEST_V2};
+ return new Schema[] {LIST_OFFSET_REQUEST_V0, LIST_OFFSET_REQUEST_V1, LIST_OFFSET_REQUEST_V2,
+ LIST_OFFSET_REQUEST_V3};
}
private final int replicaId;
@@ -272,6 +277,7 @@ public AbstractResponse getErrorResponse(int throttleTimeMs, Throwable e) {
case 0:
case 1:
case 2:
+ case 3:
return new ListOffsetResponse(throttleTimeMs, responseData);
default:
throw new IllegalArgumentException(String.format("Version %d is not valid. Valid versions for %s are 0 to %d",
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/ListOffsetResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/ListOffsetResponse.java
index 13f2dfb7106..e719dbbdbfd 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/ListOffsetResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/ListOffsetResponse.java
@@ -90,8 +90,14 @@
THROTTLE_TIME_MS,
new Field(RESPONSES_KEY_NAME, new ArrayOf(LIST_OFFSET_RESPONSE_TOPIC_V1)));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema LIST_OFFSET_RESPONSE_V3 = LIST_OFFSET_RESPONSE_V2;
+
public static Schema[] schemaVersions() {
- return new Schema[] {LIST_OFFSET_RESPONSE_V0, LIST_OFFSET_RESPONSE_V1, LIST_OFFSET_RESPONSE_V2};
+ return new Schema[] {LIST_OFFSET_RESPONSE_V0, LIST_OFFSET_RESPONSE_V1, LIST_OFFSET_RESPONSE_V2,
+ LIST_OFFSET_RESPONSE_V3};
}
public static final class PartitionData {
@@ -183,6 +189,7 @@ public ListOffsetResponse(Struct struct) {
}
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -244,4 +251,9 @@ public String toString() {
.append(")");
return bld.toString();
}
+
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 3;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/MetadataRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/MetadataRequest.java
index 934b0ed06d1..67dbe94af40 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/MetadataRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/MetadataRequest.java
@@ -62,9 +62,14 @@
/* The v5 metadata request is the same as v4. An additional field for offline_replicas has been added to the v5 metadata response */
private static final Schema METADATA_REQUEST_V5 = METADATA_REQUEST_V4;
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema METADATA_REQUEST_V6 = METADATA_REQUEST_V5;
+
public static Schema[] schemaVersions() {
return new Schema[] {METADATA_REQUEST_V0, METADATA_REQUEST_V1, METADATA_REQUEST_V2, METADATA_REQUEST_V3,
- METADATA_REQUEST_V4, METADATA_REQUEST_V5};
+ METADATA_REQUEST_V4, METADATA_REQUEST_V5, METADATA_REQUEST_V6};
}
public static class Builder extends AbstractRequest.Builder<MetadataRequest> {
@@ -170,6 +175,7 @@ public AbstractResponse getErrorResponse(int throttleTimeMs, Throwable e) {
case 3:
case 4:
case 5:
+ case 6:
return new MetadataResponse(throttleTimeMs, Collections.<Node>emptyList(), null, MetadataResponse.NO_CONTROLLER_ID, topicMetadatas);
default:
throw new IllegalArgumentException(String.format("Version %d is not valid. Valid versions for %s are 0 to %d",
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/MetadataResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/MetadataResponse.java
index cda3c0793b9..28a412df800 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/MetadataResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/MetadataResponse.java
@@ -164,9 +164,14 @@
new Field(CONTROLLER_ID_KEY_NAME, INT32, "The broker id of the controller broker."),
new Field(TOPIC_METADATA_KEY_NAME, new ArrayOf(TOPIC_METADATA_V2)));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema METADATA_RESPONSE_V6 = METADATA_RESPONSE_V5;
+
public static Schema[] schemaVersions() {
return new Schema[] {METADATA_RESPONSE_V0, METADATA_RESPONSE_V1, METADATA_RESPONSE_V2, METADATA_RESPONSE_V3,
- METADATA_RESPONSE_V4, METADATA_RESPONSE_V5};
+ METADATA_RESPONSE_V4, METADATA_RESPONSE_V5, METADATA_RESPONSE_V6};
}
private final int throttleTimeMs;
@@ -277,6 +282,7 @@ private Node getControllerNode(int controllerId, Collection<Node> brokers) {
return null;
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -431,6 +437,14 @@ public boolean isInternal() {
return partitionMetadata;
}
+ @Override
+ public String toString() {
+ return "(type=TopicMetadata" +
+ ", error=" + error +
+ ", topic=" + topic +
+ ", isInternal=" + isInternal +
+ ", partitionMetadata=" + partitionMetadata + ')';
+ }
}
// This is used to describe per-partition state in the MetadataResponse
@@ -559,4 +573,9 @@ protected Struct toStruct(short version) {
struct.set(TOPIC_METADATA_KEY_NAME, topicMetadataArray.toArray());
return struct;
}
+
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 6;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitRequest.java
index 4686c3bc258..570c4d5a66a 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitRequest.java
@@ -106,9 +106,14 @@
/* v3 request is same as v2. Throttle time has been added to response */
private static final Schema OFFSET_COMMIT_REQUEST_V3 = OFFSET_COMMIT_REQUEST_V2;
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema OFFSET_COMMIT_REQUEST_V4 = OFFSET_COMMIT_REQUEST_V3;
+
public static Schema[] schemaVersions() {
return new Schema[] {OFFSET_COMMIT_REQUEST_V0, OFFSET_COMMIT_REQUEST_V1, OFFSET_COMMIT_REQUEST_V2,
- OFFSET_COMMIT_REQUEST_V3};
+ OFFSET_COMMIT_REQUEST_V3, OFFSET_COMMIT_REQUEST_V4};
}
// default values for the current version
@@ -193,6 +198,7 @@ public OffsetCommitRequest build(short version) {
case 1:
case 2:
case 3:
+ case 4:
long retentionTime = version == 1 ? DEFAULT_RETENTION_TIME : this.retentionTime;
return new OffsetCommitRequest(groupId, generationId, memberId, retentionTime, offsetData, version);
default:
@@ -309,6 +315,7 @@ public AbstractResponse getErrorResponse(int throttleTimeMs, Throwable e) {
case 2:
return new OffsetCommitResponse(responseData);
case 3:
+ case 4:
return new OffsetCommitResponse(throttleTimeMs, responseData);
default:
throw new IllegalArgumentException(String.format("Version %d is not valid. Valid versions for %s are 0 to %d",
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitResponse.java
index b439034e1af..0b0b2283818 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitResponse.java
@@ -80,9 +80,14 @@
THROTTLE_TIME_MS,
new Field(RESPONSES_KEY_NAME, new ArrayOf(OFFSET_COMMIT_RESPONSE_TOPIC_V0)));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema OFFSET_COMMIT_RESPONSE_V4 = OFFSET_COMMIT_RESPONSE_V3;
+
public static Schema[] schemaVersions() {
return new Schema[] {OFFSET_COMMIT_RESPONSE_V0, OFFSET_COMMIT_RESPONSE_V1, OFFSET_COMMIT_RESPONSE_V2,
- OFFSET_COMMIT_RESPONSE_V3};
+ OFFSET_COMMIT_RESPONSE_V3, OFFSET_COMMIT_RESPONSE_V4};
}
private final Map<TopicPartition, Errors> responseData;
@@ -137,6 +142,7 @@ public Struct toStruct(short version) {
return struct;
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -154,4 +160,8 @@ public static OffsetCommitResponse parse(ByteBuffer buffer, short version) {
return new OffsetCommitResponse(ApiKeys.OFFSET_COMMIT.parseResponse(version, buffer));
}
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 4;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/OffsetFetchRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/OffsetFetchRequest.java
index 0db1c50ca57..e90c6ebbe91 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/OffsetFetchRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/OffsetFetchRequest.java
@@ -75,9 +75,11 @@
/* v3 request is the same as v2. Throttle time has been added to v3 response */
private static final Schema OFFSET_FETCH_REQUEST_V3 = OFFSET_FETCH_REQUEST_V2;
+ private static final Schema OFFSET_FETCH_REQUEST_V4 = OFFSET_FETCH_REQUEST_V3;
+
public static Schema[] schemaVersions() {
return new Schema[] {OFFSET_FETCH_REQUEST_V0, OFFSET_FETCH_REQUEST_V1, OFFSET_FETCH_REQUEST_V2,
- OFFSET_FETCH_REQUEST_V3};
+ OFFSET_FETCH_REQUEST_V3, OFFSET_FETCH_REQUEST_V4};
}
public static class Builder extends AbstractRequest.Builder<OffsetFetchRequest> {
@@ -178,6 +180,7 @@ public OffsetFetchResponse getErrorResponse(int throttleTimeMs, Errors error) {
case 2:
return new OffsetFetchResponse(error, responsePartitions);
case 3:
+ case 4:
return new OffsetFetchResponse(throttleTimeMs, error, responsePartitions);
default:
throw new IllegalArgumentException(String.format("Version %d is not valid. Valid versions for %s are 0 to %d",
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/OffsetFetchResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/OffsetFetchResponse.java
index e398442e847..613695bf09d 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/OffsetFetchResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/OffsetFetchResponse.java
@@ -76,9 +76,14 @@
new Field(RESPONSES_KEY_NAME, new ArrayOf(OFFSET_FETCH_RESPONSE_TOPIC_V0)),
ERROR_CODE);
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema OFFSET_FETCH_RESPONSE_V4 = OFFSET_FETCH_RESPONSE_V3;
+
public static Schema[] schemaVersions() {
return new Schema[] {OFFSET_FETCH_RESPONSE_V0, OFFSET_FETCH_RESPONSE_V1, OFFSET_FETCH_RESPONSE_V2,
- OFFSET_FETCH_RESPONSE_V3};
+ OFFSET_FETCH_RESPONSE_V3, OFFSET_FETCH_RESPONSE_V4};
}
public static final long INVALID_OFFSET = -1L;
@@ -179,6 +184,7 @@ public void maybeThrowFirstPartitionError() {
}
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -234,4 +240,9 @@ protected Struct toStruct(short version) {
return struct;
}
+
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 4;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/OffsetsForLeaderEpochRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/OffsetsForLeaderEpochRequest.java
index d0585bed6d5..651416d97a7 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/OffsetsForLeaderEpochRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/OffsetsForLeaderEpochRequest.java
@@ -50,8 +50,11 @@
private static final Schema OFFSET_FOR_LEADER_EPOCH_REQUEST_V0 = new Schema(
new Field(TOPICS_KEY_NAME, new ArrayOf(OFFSET_FOR_LEADER_EPOCH_REQUEST_TOPIC_V0), "An array of topics to get epochs for"));
+ /* v1 request is the same as v0. Per-partition leader epoch has been added to response */
+ private static final Schema OFFSET_FOR_LEADER_EPOCH_REQUEST_V1 = OFFSET_FOR_LEADER_EPOCH_REQUEST_V0;
+
public static Schema[] schemaVersions() {
- return new Schema[]{OFFSET_FOR_LEADER_EPOCH_REQUEST_V0};
+ return new Schema[]{OFFSET_FOR_LEADER_EPOCH_REQUEST_V0, OFFSET_FOR_LEADER_EPOCH_REQUEST_V1};
}
private Map<TopicPartition, Integer> epochsByPartition;
@@ -63,12 +66,12 @@
public static class Builder extends AbstractRequest.Builder<OffsetsForLeaderEpochRequest> {
private Map<TopicPartition, Integer> epochsByPartition = new HashMap<>();
- public Builder() {
- super(ApiKeys.OFFSET_FOR_LEADER_EPOCH);
+ public Builder(short version) {
+ super(ApiKeys.OFFSET_FOR_LEADER_EPOCH, version);
}
- public Builder(Map<TopicPartition, Integer> epochsByPartition) {
- super(ApiKeys.OFFSET_FOR_LEADER_EPOCH);
+ public Builder(short version, Map<TopicPartition, Integer> epochsByPartition) {
+ super(ApiKeys.OFFSET_FOR_LEADER_EPOCH, version);
this.epochsByPartition = epochsByPartition;
}
@@ -150,7 +153,8 @@ public AbstractResponse getErrorResponse(int throttleTimeMs, Throwable e) {
Errors error = Errors.forException(e);
Map<TopicPartition, EpochEndOffset> errorResponse = new HashMap<>();
for (TopicPartition tp : epochsByPartition.keySet()) {
- errorResponse.put(tp, new EpochEndOffset(error, EpochEndOffset.UNDEFINED_EPOCH_OFFSET));
+ errorResponse.put(tp, new EpochEndOffset(
+ error, EpochEndOffset.UNDEFINED_EPOCH, EpochEndOffset.UNDEFINED_EPOCH_OFFSET));
}
return new OffsetsForLeaderEpochResponse(errorResponse);
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/OffsetsForLeaderEpochResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/OffsetsForLeaderEpochResponse.java
index 4a91533938d..4da876704b7 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/OffsetsForLeaderEpochResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/OffsetsForLeaderEpochResponse.java
@@ -23,6 +23,7 @@
import org.apache.kafka.common.protocol.types.Field;
import org.apache.kafka.common.protocol.types.Schema;
import org.apache.kafka.common.protocol.types.Struct;
+import org.apache.kafka.common.record.RecordBatch;
import org.apache.kafka.common.utils.CollectionUtils;
import java.nio.ByteBuffer;
@@ -34,6 +35,7 @@
import static org.apache.kafka.common.protocol.CommonFields.ERROR_CODE;
import static org.apache.kafka.common.protocol.CommonFields.PARTITION_ID;
import static org.apache.kafka.common.protocol.CommonFields.TOPIC_NAME;
+import static org.apache.kafka.common.protocol.CommonFields.LEADER_EPOCH;
import static org.apache.kafka.common.protocol.types.Type.INT64;
public class OffsetsForLeaderEpochResponse extends AbstractResponse {
@@ -52,8 +54,22 @@
new Field(TOPICS_KEY_NAME, new ArrayOf(OFFSET_FOR_LEADER_EPOCH_RESPONSE_TOPIC_V0),
"An array of topics for which we have leader offsets for some requested Partition Leader Epoch"));
+ // OFFSET_FOR_LEADER_EPOCH_RESPONSE_PARTITION_V1 added a per-partition leader epoch field,
+ // which specifies which leader epoch the end offset belongs to
+ private static final Schema OFFSET_FOR_LEADER_EPOCH_RESPONSE_PARTITION_V1 = new Schema(
+ ERROR_CODE,
+ PARTITION_ID,
+ LEADER_EPOCH,
+ new Field(END_OFFSET_KEY_NAME, INT64, "The end offset"));
+ private static final Schema OFFSET_FOR_LEADER_EPOCH_RESPONSE_TOPIC_V1 = new Schema(
+ TOPIC_NAME,
+ new Field(PARTITIONS_KEY_NAME, new ArrayOf(OFFSET_FOR_LEADER_EPOCH_RESPONSE_PARTITION_V1)));
+ private static final Schema OFFSET_FOR_LEADER_EPOCH_RESPONSE_V1 = new Schema(
+ new Field(TOPICS_KEY_NAME, new ArrayOf(OFFSET_FOR_LEADER_EPOCH_RESPONSE_TOPIC_V1),
+ "An array of topics for which we have leader offsets for some requested Partition Leader Epoch"));
+
public static Schema[] schemaVersions() {
- return new Schema[]{OFFSET_FOR_LEADER_EPOCH_RESPONSE_V0};
+ return new Schema[]{OFFSET_FOR_LEADER_EPOCH_RESPONSE_V0, OFFSET_FOR_LEADER_EPOCH_RESPONSE_V1};
}
private Map<TopicPartition, EpochEndOffset> epochEndOffsetsByPartition;
@@ -68,8 +84,9 @@ public OffsetsForLeaderEpochResponse(Struct struct) {
Errors error = Errors.forCode(partitionAndEpoch.get(ERROR_CODE));
int partitionId = partitionAndEpoch.get(PARTITION_ID);
TopicPartition tp = new TopicPartition(topic, partitionId);
+ int leaderEpoch = partitionAndEpoch.getOrElse(LEADER_EPOCH, RecordBatch.NO_PARTITION_LEADER_EPOCH);
long endOffset = partitionAndEpoch.getLong(END_OFFSET_KEY_NAME);
- epochEndOffsetsByPartition.put(tp, new EpochEndOffset(error, endOffset));
+ epochEndOffsetsByPartition.put(tp, new EpochEndOffset(error, leaderEpoch, endOffset));
}
}
}
@@ -110,6 +127,7 @@ protected Struct toStruct(short version) {
Struct partitionStruct = topicStruct.instance(PARTITIONS_KEY_NAME);
partitionStruct.set(ERROR_CODE, partitionEndOffset.getValue().error().code());
partitionStruct.set(PARTITION_ID, partitionEndOffset.getKey());
+ partitionStruct.setIfExists(LEADER_EPOCH, partitionEndOffset.getValue().leaderEpoch());
partitionStruct.set(END_OFFSET_KEY_NAME, partitionEndOffset.getValue().endOffset());
partitions.add(partitionStruct);
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java
index 91e3aebcf3f..67745cbb4cc 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java
@@ -108,10 +108,14 @@
*/
private static final Schema PRODUCE_REQUEST_V5 = PRODUCE_REQUEST_V4;
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema PRODUCE_REQUEST_V6 = PRODUCE_REQUEST_V5;
public static Schema[] schemaVersions() {
return new Schema[] {PRODUCE_REQUEST_V0, PRODUCE_REQUEST_V1, PRODUCE_REQUEST_V2, PRODUCE_REQUEST_V3,
- PRODUCE_REQUEST_V4, PRODUCE_REQUEST_V5};
+ PRODUCE_REQUEST_V4, PRODUCE_REQUEST_V5, PRODUCE_REQUEST_V6};
}
public static class Builder extends AbstractRequest.Builder<ProduceRequest> {
@@ -325,6 +329,7 @@ public ProduceResponse getErrorResponse(int throttleTimeMs, Throwable e) {
case 3:
case 4:
case 5:
+ case 6:
return new ProduceResponse(responseMap, throttleTimeMs);
default:
throw new IllegalArgumentException(String.format("Version %d is not valid. Valid versions for %s are 0 to %d",
@@ -394,6 +399,7 @@ public static byte requiredMagicForVersion(short produceRequestVersion) {
case 3:
case 4:
case 5:
+ case 6:
return RecordBatch.MAGIC_VALUE_V2;
default:
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/ProduceResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/ProduceResponse.java
index afedc9d6e88..467c9804c3c 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/ProduceResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/ProduceResponse.java
@@ -139,14 +139,18 @@
LOG_START_OFFSET_FIELD)))))),
THROTTLE_TIME_MS);
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema PRODUCE_RESPONSE_V6 = PRODUCE_RESPONSE_V5;
public static Schema[] schemaVersions() {
return new Schema[]{PRODUCE_RESPONSE_V0, PRODUCE_RESPONSE_V1, PRODUCE_RESPONSE_V2, PRODUCE_RESPONSE_V3,
- PRODUCE_RESPONSE_V4, PRODUCE_RESPONSE_V5};
+ PRODUCE_RESPONSE_V4, PRODUCE_RESPONSE_V5, PRODUCE_RESPONSE_V6};
}
private final Map<TopicPartition, PartitionResponse> responses;
- private final int throttleTime;
+ private final int throttleTimeMs;
/**
* Constructor for Version 0
@@ -159,11 +163,11 @@ public ProduceResponse(Map<TopicPartition, PartitionResponse> responses) {
/**
* Constructor for the latest version
* @param responses Produced data grouped by topic-partition
- * @param throttleTime Time in milliseconds the response was throttled
+ * @param throttleTimeMs Time in milliseconds the response was throttled
*/
- public ProduceResponse(Map<TopicPartition, PartitionResponse> responses, int throttleTime) {
+ public ProduceResponse(Map<TopicPartition, PartitionResponse> responses, int throttleTimeMs) {
this.responses = responses;
- this.throttleTime = throttleTime;
+ this.throttleTimeMs = throttleTimeMs;
}
/**
@@ -185,7 +189,7 @@ public ProduceResponse(Struct struct) {
responses.put(tp, new PartitionResponse(error, offset, logAppendTime, logStartOffset));
}
}
- this.throttleTime = struct.getOrElse(THROTTLE_TIME_MS, DEFAULT_THROTTLE_TIME);
+ this.throttleTimeMs = struct.getOrElse(THROTTLE_TIME_MS, DEFAULT_THROTTLE_TIME);
}
@Override
@@ -220,7 +224,7 @@ protected Struct toStruct(short version) {
topicDatas.add(topicData);
}
struct.set(RESPONSES_KEY_NAME, topicDatas.toArray());
- struct.setIfExists(THROTTLE_TIME_MS, throttleTime);
+ struct.setIfExists(THROTTLE_TIME_MS, throttleTimeMs);
return struct;
}
@@ -229,8 +233,9 @@ protected Struct toStruct(short version) {
return this.responses;
}
- public int getThrottleTime() {
- return this.throttleTime;
+ @Override
+ public int throttleTimeMs() {
+ return this.throttleTimeMs;
}
@Override
@@ -279,4 +284,8 @@ public static ProduceResponse parse(ByteBuffer buffer, short version) {
return new ProduceResponse(ApiKeys.PRODUCE.responseSchema(version).read(buffer));
}
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 6;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/RenewDelegationTokenRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/RenewDelegationTokenRequest.java
index 4a4b762a72a..57c310014f5 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/RenewDelegationTokenRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/RenewDelegationTokenRequest.java
@@ -38,6 +38,11 @@
new Field(HMAC_KEY_NAME, BYTES, "HMAC of the delegation token to be renewed."),
new Field(RENEW_TIME_PERIOD_KEY_NAME, INT64, "Renew time period in milli seconds."));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ public static final Schema TOKEN_RENEW_REQUEST_V1 = TOKEN_RENEW_REQUEST_V0;
+
private RenewDelegationTokenRequest(short version, ByteBuffer hmac, long renewTimePeriod) {
super(version);
@@ -57,7 +62,7 @@ public static RenewDelegationTokenRequest parse(ByteBuffer buffer, short version
}
public static Schema[] schemaVersions() {
- return new Schema[] {TOKEN_RENEW_REQUEST_V0};
+ return new Schema[] {TOKEN_RENEW_REQUEST_V0, TOKEN_RENEW_REQUEST_V1};
}
@Override
@@ -88,9 +93,9 @@ public long renewTimePeriod() {
private final ByteBuffer hmac;
private final long renewTimePeriod;
- public Builder(ByteBuffer hmac, long renewTimePeriod) {
+ public Builder(byte[] hmac, long renewTimePeriod) {
super(ApiKeys.RENEW_DELEGATION_TOKEN);
- this.hmac = hmac;
+ this.hmac = ByteBuffer.wrap(hmac);
this.renewTimePeriod = renewTimePeriod;
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/RenewDelegationTokenResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/RenewDelegationTokenResponse.java
index 1885b480a96..dc961e176df 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/RenewDelegationTokenResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/RenewDelegationTokenResponse.java
@@ -42,6 +42,11 @@
new Field(EXPIRY_TIMESTAMP_KEY_NAME, INT64, "timestamp (in msec) at which this token expires.."),
THROTTLE_TIME_MS);
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema TOKEN_RENEW_RESPONSE_V1 = TOKEN_RENEW_RESPONSE_V0;
+
public RenewDelegationTokenResponse(int throttleTimeMs, Errors error, long expiryTimestamp) {
this.throttleTimeMs = throttleTimeMs;
this.error = error;
@@ -63,7 +68,7 @@ public static RenewDelegationTokenResponse parse(ByteBuffer buffer, short versio
}
public static Schema[] schemaVersions() {
- return new Schema[] {TOKEN_RENEW_RESPONSE_V0};
+ return new Schema[] {TOKEN_RENEW_RESPONSE_V0, TOKEN_RENEW_RESPONSE_V1};
}
@Override
@@ -82,6 +87,7 @@ protected Struct toStruct(short version) {
return struct;
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -93,4 +99,13 @@ public Errors error() {
public long expiryTimestamp() {
return expiryTimestamp;
}
+
+ public boolean hasError() {
+ return this.error != Errors.NONE;
+ }
+
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 1;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/Resource.java b/clients/src/main/java/org/apache/kafka/common/requests/Resource.java
index 6a360a5900a..bd814661ae3 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/Resource.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/Resource.java
@@ -55,6 +55,6 @@ public int hashCode() {
@Override
public String toString() {
- return "Resource(type=" + type + ", name='" + name + "'}";
+ return "Resource(type=" + type + ", name='" + name + "')";
}
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/SaslAuthenticateResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/SaslAuthenticateResponse.java
index e2441825b57..21ce83ed556 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/SaslAuthenticateResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/SaslAuthenticateResponse.java
@@ -102,5 +102,4 @@ public Struct toStruct(short version) {
public static SaslAuthenticateResponse parse(ByteBuffer buffer, short version) {
return new SaslAuthenticateResponse(ApiKeys.SASL_AUTHENTICATE.parseResponse(version, buffer));
}
-}
-
+}
\ No newline at end of file
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/SaslHandshakeResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/SaslHandshakeResponse.java
index 252c62dd230..9faa36cddea 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/SaslHandshakeResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/SaslHandshakeResponse.java
@@ -96,5 +96,4 @@ public Struct toStruct(short version) {
public static SaslHandshakeResponse parse(ByteBuffer buffer, short version) {
return new SaslHandshakeResponse(ApiKeys.SASL_HANDSHAKE.parseResponse(version, buffer));
}
-}
-
+}
\ No newline at end of file
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/StopReplicaResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/StopReplicaResponse.java
index 777416d1758..4e8efe9547a 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/StopReplicaResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/StopReplicaResponse.java
@@ -115,4 +115,13 @@ protected Struct toStruct(short version) {
struct.set(ERROR_CODE, error.code());
return struct;
}
+
+ @Override
+ public String toString() {
+ return "StopReplicaResponse(" +
+ "responses=" + responses +
+ ", error=" + error +
+ ")";
+ }
+
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/SyncGroupRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/SyncGroupRequest.java
index 14ed2625d5a..962bc77884d 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/SyncGroupRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/SyncGroupRequest.java
@@ -51,8 +51,14 @@
/* v1 request is the same as v0. Throttle time has been added to response */
private static final Schema SYNC_GROUP_REQUEST_V1 = SYNC_GROUP_REQUEST_V0;
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema SYNC_GROUP_REQUEST_V2 = SYNC_GROUP_REQUEST_V1;
+
public static Schema[] schemaVersions() {
- return new Schema[] {SYNC_GROUP_REQUEST_V0, SYNC_GROUP_REQUEST_V1};
+ return new Schema[] {SYNC_GROUP_REQUEST_V0, SYNC_GROUP_REQUEST_V1,
+ SYNC_GROUP_REQUEST_V2};
}
public static class Builder extends AbstractRequest.Builder<SyncGroupRequest> {
@@ -127,6 +133,7 @@ public AbstractResponse getErrorResponse(int throttleTimeMs, Throwable e) {
Errors.forException(e),
ByteBuffer.wrap(new byte[]{}));
case 1:
+ case 2:
return new SyncGroupResponse(
throttleTimeMs,
Errors.forException(e),
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/SyncGroupResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/SyncGroupResponse.java
index 77f951251fc..2b2fc6fbc33 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/SyncGroupResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/SyncGroupResponse.java
@@ -40,19 +40,29 @@
ERROR_CODE,
new Field(MEMBER_ASSIGNMENT_KEY_NAME, BYTES));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema SYNC_GROUP_RESPONSE_V2 = SYNC_GROUP_RESPONSE_V1;
+
public static Schema[] schemaVersions() {
- return new Schema[] {SYNC_GROUP_RESPONSE_V0, SYNC_GROUP_RESPONSE_V1};
+ return new Schema[] {SYNC_GROUP_RESPONSE_V0, SYNC_GROUP_RESPONSE_V1,
+ SYNC_GROUP_RESPONSE_V2};
}
/**
* Possible error codes:
*
- * GROUP_COORDINATOR_NOT_AVAILABLE (15)
+ * COORDINATOR_NOT_AVAILABLE (15)
* NOT_COORDINATOR (16)
* ILLEGAL_GENERATION (22)
* UNKNOWN_MEMBER_ID (25)
* REBALANCE_IN_PROGRESS (27)
* GROUP_AUTHORIZATION_FAILED (30)
+ *
+ * NOTE: Currently the coordinator returns REBALANCE_IN_PROGRESS while the coordinator is
+ * loading. On the next protocol bump, we should consider using COORDINATOR_LOAD_IN_PROGRESS
+ * to be consistent with the other APIs.
*/
private final Errors error;
@@ -75,6 +85,7 @@ public SyncGroupResponse(Struct struct) {
this.memberState = struct.getBytes(MEMBER_ASSIGNMENT_KEY_NAME);
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -105,4 +116,8 @@ public static SyncGroupResponse parse(ByteBuffer buffer, short version) {
return new SyncGroupResponse(ApiKeys.SYNC_GROUP.parseResponse(version, buffer));
}
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 2;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/TxnOffsetCommitRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/TxnOffsetCommitRequest.java
index 9787c2df0a8..25245be814f 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/TxnOffsetCommitRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/TxnOffsetCommitRequest.java
@@ -59,8 +59,13 @@
new Field(PARTITIONS_KEY_NAME, new ArrayOf(TXN_OFFSET_COMMIT_PARTITION_OFFSET_METADATA_REQUEST_V0)))),
"The partitions to write markers for."));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema TXN_OFFSET_COMMIT_REQUEST_V1 = TXN_OFFSET_COMMIT_REQUEST_V0;
+
public static Schema[] schemaVersions() {
- return new Schema[]{TXN_OFFSET_COMMIT_REQUEST_V0};
+ return new Schema[]{TXN_OFFSET_COMMIT_REQUEST_V0, TXN_OFFSET_COMMIT_REQUEST_V1};
}
public static class Builder extends AbstractRequest.Builder<TxnOffsetCommitRequest> {
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/TxnOffsetCommitResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/TxnOffsetCommitResponse.java
index ff0f8cefeef..c34fd400232 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/TxnOffsetCommitResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/TxnOffsetCommitResponse.java
@@ -49,8 +49,13 @@
new Field(PARTITIONS_KEY_NAME, new ArrayOf(TXN_OFFSET_COMMIT_PARTITION_ERROR_RESPONSE_V0)))),
"Errors per partition from writing markers."));
+ /**
+ * The version number is bumped to indicate that on quota violation brokers send out responses before throttling.
+ */
+ private static final Schema TXN_OFFSET_COMMIT_RESPONSE_V1 = TXN_OFFSET_COMMIT_RESPONSE_V0;
+
public static Schema[] schemaVersions() {
- return new Schema[]{TXN_OFFSET_COMMIT_RESPONSE_V0};
+ return new Schema[]{TXN_OFFSET_COMMIT_RESPONSE_V0, TXN_OFFSET_COMMIT_RESPONSE_V1};
}
// Possible error codes:
@@ -117,6 +122,7 @@ protected Struct toStruct(short version) {
return struct;
}
+ @Override
public int throttleTimeMs() {
return throttleTimeMs;
}
@@ -142,4 +148,8 @@ public String toString() {
')';
}
+ @Override
+ public boolean shouldClientThrottle(short version) {
+ return version >= 1;
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/UpdateMetadataRequest.java b/clients/src/main/java/org/apache/kafka/common/requests/UpdateMetadataRequest.java
index da48e9fa85b..a273765704d 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/UpdateMetadataRequest.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/UpdateMetadataRequest.java
@@ -435,7 +435,7 @@ public AbstractResponse getErrorResponse(int throttleTimeMs, Throwable e) {
return new UpdateMetadataResponse(Errors.forException(e));
else
throw new IllegalArgumentException(String.format("Version %d is not valid. Valid versions for %s are 0 to %d",
- versionId, this.getClass().getSimpleName(), ApiKeys.UPDATE_METADATA.latestVersion()));
+ versionId, this.getClass().getSimpleName(), ApiKeys.UPDATE_METADATA.latestVersion()));
}
public int controllerId() {
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/UpdateMetadataResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/UpdateMetadataResponse.java
index 4c21cdeaccc..bae816dc15c 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/UpdateMetadataResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/UpdateMetadataResponse.java
@@ -72,4 +72,4 @@ protected Struct toStruct(short version) {
struct.set(ERROR_CODE, error.code());
return struct;
}
-}
+}
\ No newline at end of file
diff --git a/clients/src/main/java/org/apache/kafka/common/requests/WriteTxnMarkersResponse.java b/clients/src/main/java/org/apache/kafka/common/requests/WriteTxnMarkersResponse.java
index f4bf157adaf..f307760e123 100644
--- a/clients/src/main/java/org/apache/kafka/common/requests/WriteTxnMarkersResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/requests/WriteTxnMarkersResponse.java
@@ -163,5 +163,4 @@ protected Struct toStruct(short version) {
public static WriteTxnMarkersResponse parse(ByteBuffer buffer, short version) {
return new WriteTxnMarkersResponse(ApiKeys.WRITE_TXN_MARKERS.parseResponse(version, buffer));
}
-
}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/JaasConfig.java b/clients/src/main/java/org/apache/kafka/common/security/JaasConfig.java
index 24bdac23787..5e837a69c16 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/JaasConfig.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/JaasConfig.java
@@ -81,6 +81,9 @@ public JaasConfig(String loginContextName, String jaasConfigParams) {
}
private LoginModuleControlFlag loginModuleControlFlag(String flag) {
+ if (flag == null)
+ throw new IllegalArgumentException("Login module control flag is not available in the JAAS config");
+
LoginModuleControlFlag controlFlag;
switch (flag.toUpperCase(Locale.ROOT)) {
case "REQUIRED":
diff --git a/clients/src/main/java/org/apache/kafka/common/security/JaasContext.java b/clients/src/main/java/org/apache/kafka/common/security/JaasContext.java
index d72f00dc590..849a97894db 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/JaasContext.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/JaasContext.java
@@ -33,7 +33,7 @@
public class JaasContext {
- private static final Logger LOG = LoggerFactory.getLogger(JaasUtils.class);
+ private static final Logger LOG = LoggerFactory.getLogger(JaasContext.class);
private static final String GLOBAL_CONTEXT_NAME_SERVER = "KafkaServer";
private static final String GLOBAL_CONTEXT_NAME_CLIENT = "KafkaClient";
@@ -183,7 +183,7 @@ public Password dynamicJaasConfig() {
* Returns the configuration option for <code>key</code> from this context.
* If login module name is specified, return option value only from that module.
*/
- public String configEntryOption(String key, String loginModuleName) {
+ public static String configEntryOption(List<AppConfigurationEntry> configurationEntries, String key, String loginModuleName) {
for (AppConfigurationEntry entry : configurationEntries) {
if (loginModuleName != null && !loginModuleName.equals(entry.getLoginModuleName()))
continue;
diff --git a/clients/src/main/java/org/apache/kafka/common/security/auth/AuthenticateCallbackHandler.java b/clients/src/main/java/org/apache/kafka/common/security/auth/AuthenticateCallbackHandler.java
new file mode 100644
index 00000000000..8951d3a5893
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/security/auth/AuthenticateCallbackHandler.java
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.common.security.auth;
+
+import java.util.List;
+import java.util.Map;
+
+import javax.security.auth.callback.CallbackHandler;
+import javax.security.auth.login.AppConfigurationEntry;
+
+/*
+ * Callback handler for SASL-based authentication
+ */
+public interface AuthenticateCallbackHandler extends CallbackHandler {
+
+ /**
+ * Configures this callback handler for the specified SASL mechanism.
+ *
+ * @param configs Key-value pairs containing the parsed configuration options of
+ * the client or broker. Note that these are the Kafka configuration options
+ * and not the JAAS configuration options. JAAS config options may be obtained
+ * from `jaasConfigEntries` for callbacks which obtain some configs from the
+ * JAAS configuration. For configs that may be specified as both Kafka config
+ * as well as JAAS config (e.g. sasl.kerberos.service.name), the configuration
+ * is treated as invalid if conflicting values are provided.
+ * @param saslMechanism Negotiated SASL mechanism. For clients, this is the SASL
+ * mechanism configured for the client. For brokers, this is the mechanism
+ * negotiated with the client and is one of the mechanisms enabled on the broker.
+ * @param jaasConfigEntries JAAS configuration entries from the JAAS login context.
+ * This list contains a single entry for clients and may contain more than
+ * one entry for brokers if multiple mechanisms are enabled on a listener using
+ * static JAAS configuration where there is no mapping between mechanisms and
+ * login module entries. In this case, callback handlers can use the login module in
+ * `jaasConfigEntries` to identify the entry corresponding to `saslMechanism`.
+ * Alternatively, dynamic JAAS configuration option
+ * {@link org.apache.kafka.common.config.SaslConfigs#SASL_JAAS_CONFIG} may be
+ * configured on brokers with listener and mechanism prefix, in which case
+ * only the configuration entry corresponding to `saslMechanism` will be provided
+ * in `jaasConfigEntries`.
+ */
+ void configure(Map<String, ?> configs, String saslMechanism, List<AppConfigurationEntry> jaasConfigEntries);
+
+ /**
+ * Closes this instance.
+ */
+ void close();
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/authenticator/Login.java b/clients/src/main/java/org/apache/kafka/common/security/auth/Login.java
similarity index 55%
rename from clients/src/main/java/org/apache/kafka/common/security/authenticator/Login.java
rename to clients/src/main/java/org/apache/kafka/common/security/auth/Login.java
index b41d1b2572f..eda5e7a225a 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/authenticator/Login.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/auth/Login.java
@@ -14,13 +14,12 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.kafka.common.security.authenticator;
-
-import org.apache.kafka.common.security.JaasContext;
+package org.apache.kafka.common.security.auth;
import java.util.Map;
import javax.security.auth.Subject;
+import javax.security.auth.login.Configuration;
import javax.security.auth.login.LoginContext;
import javax.security.auth.login.LoginException;
@@ -31,8 +30,21 @@
/**
* Configures this login instance.
+ * @param configs Key-value pairs containing the parsed configuration options of
+ * the client or broker. Note that these are the Kafka configuration options
+ * and not the JAAS configuration options. The JAAS options may be obtained
+ * from `jaasConfiguration`.
+ * @param contextName JAAS context name for this login which may be used to obtain
+ * the login context from `jaasConfiguration`.
+ * @param jaasConfiguration JAAS configuration containing the login context named
+ * `contextName`. If static JAAS configuration is used, this `Configuration`
+ * may also contain other login contexts.
+ * @param loginCallbackHandler Login callback handler instance to use for this Login.
+ * Login callback handler class may be configured using
+ * {@link org.apache.kafka.common.config.SaslConfigs#SASL_LOGIN_CALLBACK_HANDLER_CLASS}.
*/
- void configure(Map<String, ?> configs, JaasContext jaasContext);
+ void configure(Map<String, ?> configs, String contextName, Configuration jaasConfiguration,
+ AuthenticateCallbackHandler loginCallbackHandler);
/**
* Performs login for each login module specified for the login context of this instance.
@@ -54,4 +66,3 @@
*/
void close();
}
-
diff --git a/clients/src/main/java/org/apache/kafka/common/security/authenticator/AbstractLogin.java b/clients/src/main/java/org/apache/kafka/common/security/authenticator/AbstractLogin.java
index 643f859e829..7e1350864e4 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/authenticator/AbstractLogin.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/authenticator/AbstractLogin.java
@@ -16,20 +16,23 @@
*/
package org.apache.kafka.common.security.authenticator;
+import javax.security.auth.login.AppConfigurationEntry;
+import javax.security.auth.login.Configuration;
import javax.security.auth.login.LoginContext;
import javax.security.auth.login.LoginException;
import javax.security.sasl.RealmCallback;
import javax.security.auth.callback.Callback;
-import javax.security.auth.callback.CallbackHandler;
import javax.security.auth.callback.NameCallback;
import javax.security.auth.callback.PasswordCallback;
import javax.security.auth.callback.UnsupportedCallbackException;
import javax.security.auth.Subject;
-import org.apache.kafka.common.security.JaasContext;
+import org.apache.kafka.common.security.auth.AuthenticateCallbackHandler;
+import org.apache.kafka.common.security.auth.Login;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import java.util.List;
import java.util.Map;
/**
@@ -38,17 +41,22 @@
public abstract class AbstractLogin implements Login {
private static final Logger log = LoggerFactory.getLogger(AbstractLogin.class);
- private JaasContext jaasContext;
+ private String contextName;
+ private Configuration configuration;
private LoginContext loginContext;
+ private AuthenticateCallbackHandler loginCallbackHandler;
@Override
- public void configure(Map<String, ?> configs, JaasContext jaasContext) {
- this.jaasContext = jaasContext;
+ public void configure(Map<String, ?> configs, String contextName, Configuration configuration,
+ AuthenticateCallbackHandler loginCallbackHandler) {
+ this.contextName = contextName;
+ this.configuration = configuration;
+ this.loginCallbackHandler = loginCallbackHandler;
}
@Override
public LoginContext login() throws LoginException {
- loginContext = new LoginContext(jaasContext.name(), null, new LoginCallbackHandler(), jaasContext.configuration());
+ loginContext = new LoginContext(contextName, null, loginCallbackHandler, configuration);
loginContext.login();
log.info("Successfully logged in.");
return loginContext;
@@ -59,8 +67,12 @@ public Subject subject() {
return loginContext.getSubject();
}
- protected JaasContext jaasContext() {
- return jaasContext;
+ protected String contextName() {
+ return contextName;
+ }
+
+ protected Configuration configuration() {
+ return configuration;
}
/**
@@ -70,7 +82,11 @@ protected JaasContext jaasContext() {
* callback handlers which require additional user input.
*
*/
- public static class LoginCallbackHandler implements CallbackHandler {
+ public static class DefaultLoginCallbackHandler implements AuthenticateCallbackHandler {
+
+ @Override
+ public void configure(Map<String, ?> configs, String saslMechanism, List<AppConfigurationEntry> jaasConfigEntries) {
+ }
@Override
public void handle(Callback[] callbacks) throws UnsupportedCallbackException {
@@ -90,6 +106,10 @@ public void handle(Callback[] callbacks) throws UnsupportedCallbackException {
}
}
}
+
+ @Override
+ public void close() {
+ }
}
}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/authenticator/LoginManager.java b/clients/src/main/java/org/apache/kafka/common/security/authenticator/LoginManager.java
index 81dc063c743..69ed7b167d8 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/authenticator/LoginManager.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/authenticator/LoginManager.java
@@ -23,11 +23,19 @@
import java.util.ArrayList;
import java.util.HashMap;
import java.util.Map;
+import java.util.Objects;
+
+import org.apache.kafka.common.config.ConfigException;
import org.apache.kafka.common.config.SaslConfigs;
import org.apache.kafka.common.config.types.Password;
+import org.apache.kafka.common.network.ListenerName;
import org.apache.kafka.common.security.JaasContext;
-import org.apache.kafka.common.security.kerberos.KerberosLogin;
+import org.apache.kafka.common.security.auth.AuthenticateCallbackHandler;
+import org.apache.kafka.common.security.auth.Login;
+import org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule;
+import org.apache.kafka.common.security.oauthbearer.internal.unsecured.OAuthBearerUnsecuredLoginCallbackHandler;
+import org.apache.kafka.common.utils.Utils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -36,20 +44,23 @@
private static final Logger LOGGER = LoggerFactory.getLogger(LoginManager.class);
// static configs (broker or client)
- private static final Map<String, LoginManager> STATIC_INSTANCES = new HashMap<>();
+ private static final Map<LoginMetadata<String>, LoginManager> STATIC_INSTANCES = new HashMap<>();
- // dynamic configs (client-only)
- private static final Map<Password, LoginManager> DYNAMIC_INSTANCES = new HashMap<>();
+ // dynamic configs (broker or client)
+ private static final Map<LoginMetadata<Password>, LoginManager> DYNAMIC_INSTANCES = new HashMap<>();
private final Login login;
- private final Object cacheKey;
+ private final LoginMetadata<?> loginMetadata;
+ private final AuthenticateCallbackHandler loginCallbackHandler;
private int refCount;
- private LoginManager(JaasContext jaasContext, boolean hasKerberos, Map<String, ?> configs,
- Object cacheKey) throws IOException, LoginException {
- this.cacheKey = cacheKey;
- login = hasKerberos ? new KerberosLogin() : new DefaultLogin();
- login.configure(configs, jaasContext);
+ private LoginManager(JaasContext jaasContext, String saslMechanism, Map<String, ?> configs,
+ LoginMetadata<?> loginMetadata) throws IOException, LoginException {
+ this.loginMetadata = loginMetadata;
+ this.login = Utils.newInstance(loginMetadata.loginClass);
+ loginCallbackHandler = Utils.newInstance(loginMetadata.loginCallbackClass);
+ loginCallbackHandler.configure(configs, saslMechanism, jaasContext.configurationEntries());
+ login.configure(configs, jaasContext.name(), jaasContext.configuration(), loginCallbackHandler);
login.login();
}
@@ -72,28 +83,36 @@ private LoginManager(JaasContext jaasContext, boolean hasKerberos, Map<String, ?
* @param saslMechanism SASL mechanism for which login manager is being acquired. For dynamic contexts, the single
* login module in `jaasContext` corresponds to this SASL mechanism. Hence `Login` class is
* chosen based on this mechanism.
- * @param hasKerberos Boolean flag that indicates if Kerberos is enabled for the server listener or client. Since
- * static broker configuration may contain multiple login modules in a login context, KerberosLogin
- * must be used if Kerberos is enabled on the listener, even if `saslMechanism` is not GSSAPI.
+ * @param defaultLoginClass Default login class to use if an override is not specified in `configs`
* @param configs Config options used to configure `Login` if a new login manager is created.
*
*/
- public static LoginManager acquireLoginManager(JaasContext jaasContext, String saslMechanism, boolean hasKerberos,
+ public static LoginManager acquireLoginManager(JaasContext jaasContext, String saslMechanism,
+ Class<? extends Login> defaultLoginClass,
Map<String, ?> configs) throws IOException, LoginException {
+ Class<? extends Login> loginClass = configuredClassOrDefault(configs, jaasContext,
+ saslMechanism, SaslConfigs.SASL_LOGIN_CLASS, defaultLoginClass);
+ Class<? extends AuthenticateCallbackHandler> defaultLoginCallbackHandlerClass = OAuthBearerLoginModule.OAUTHBEARER_MECHANISM
+ .equals(saslMechanism) ? OAuthBearerUnsecuredLoginCallbackHandler.class
+ : AbstractLogin.DefaultLoginCallbackHandler.class;
+ Class<? extends AuthenticateCallbackHandler> loginCallbackClass = configuredClassOrDefault(configs, jaasContext,
+ saslMechanism, SaslConfigs.SASL_LOGIN_CALLBACK_HANDLER_CLASS, defaultLoginCallbackHandlerClass);
synchronized (LoginManager.class) {
LoginManager loginManager;
Password jaasConfigValue = jaasContext.dynamicJaasConfig();
if (jaasConfigValue != null) {
- loginManager = DYNAMIC_INSTANCES.get(jaasConfigValue);
+ LoginMetadata<Password> loginMetadata = new LoginMetadata<>(jaasConfigValue, loginClass, loginCallbackClass);
+ loginManager = DYNAMIC_INSTANCES.get(loginMetadata);
if (loginManager == null) {
- loginManager = new LoginManager(jaasContext, saslMechanism.equals(SaslConfigs.GSSAPI_MECHANISM), configs, jaasConfigValue);
- DYNAMIC_INSTANCES.put(jaasConfigValue, loginManager);
+ loginManager = new LoginManager(jaasContext, saslMechanism, configs, loginMetadata);
+ DYNAMIC_INSTANCES.put(loginMetadata, loginManager);
}
} else {
- loginManager = STATIC_INSTANCES.get(jaasContext.name());
+ LoginMetadata<String> loginMetadata = new LoginMetadata<>(jaasContext.name(), loginClass, loginCallbackClass);
+ loginManager = STATIC_INSTANCES.get(loginMetadata);
if (loginManager == null) {
- loginManager = new LoginManager(jaasContext, hasKerberos, configs, jaasContext.name());
- STATIC_INSTANCES.put(jaasContext.name(), loginManager);
+ loginManager = new LoginManager(jaasContext, saslMechanism, configs, loginMetadata);
+ STATIC_INSTANCES.put(loginMetadata, loginManager);
}
}
return loginManager.acquire();
@@ -110,7 +129,7 @@ public String serviceName() {
// Only for testing
Object cacheKey() {
- return cacheKey;
+ return loginMetadata.configInfo;
}
private LoginManager acquire() {
@@ -127,12 +146,13 @@ public void release() {
if (refCount == 0)
throw new IllegalStateException("release() called on disposed " + this);
else if (refCount == 1) {
- if (cacheKey instanceof Password) {
- DYNAMIC_INSTANCES.remove(cacheKey);
+ if (loginMetadata.configInfo instanceof Password) {
+ DYNAMIC_INSTANCES.remove(loginMetadata);
} else {
- STATIC_INSTANCES.remove(cacheKey);
+ STATIC_INSTANCES.remove(loginMetadata);
}
login.close();
+ loginCallbackHandler.close();
}
--refCount;
LOGGER.trace("{} released", this);
@@ -150,10 +170,56 @@ public String toString() {
/* Should only be used in tests. */
public static void closeAll() {
synchronized (LoginManager.class) {
- for (String key : new ArrayList<>(STATIC_INSTANCES.keySet()))
+ for (LoginMetadata<String> key : new ArrayList<>(STATIC_INSTANCES.keySet()))
STATIC_INSTANCES.remove(key).login.close();
- for (Password key : new ArrayList<>(DYNAMIC_INSTANCES.keySet()))
+ for (LoginMetadata<Password> key : new ArrayList<>(DYNAMIC_INSTANCES.keySet()))
DYNAMIC_INSTANCES.remove(key).login.close();
}
}
+
+ private static <T> Class<? extends T> configuredClassOrDefault(Map<String, ?> configs,
+ JaasContext jaasContext,
+ String saslMechanism,
+ String configName,
+ Class<? extends T> defaultClass) {
+ String prefix = jaasContext.type() == JaasContext.Type.SERVER ? ListenerName.saslMechanismPrefix(saslMechanism) : "";
+ Class<? extends T> clazz = (Class<? extends T>) configs.get(prefix + configName);
+ if (clazz != null && jaasContext.configurationEntries().size() != 1) {
+ String errorMessage = configName + " cannot be specified with multiple login modules in the JAAS context. " +
+ SaslConfigs.SASL_JAAS_CONFIG + " must be configured to override mechanism-specific configs.";
+ throw new ConfigException(errorMessage);
+ }
+ if (clazz == null)
+ clazz = defaultClass;
+ return clazz;
+ }
+
+ private static class LoginMetadata<T> {
+ final T configInfo;
+ final Class<? extends Login> loginClass;
+ final Class<? extends AuthenticateCallbackHandler> loginCallbackClass;
+
+ LoginMetadata(T configInfo, Class<? extends Login> loginClass,
+ Class<? extends AuthenticateCallbackHandler> loginCallbackClass) {
+ this.configInfo = configInfo;
+ this.loginClass = loginClass;
+ this.loginCallbackClass = loginCallbackClass;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(configInfo, loginClass, loginCallbackClass);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
+ if (o == null || getClass() != o.getClass()) return false;
+
+ LoginMetadata<?> loginMetadata = (LoginMetadata<?>) o;
+ return Objects.equals(configInfo, loginMetadata.configInfo) &&
+ Objects.equals(loginClass, loginMetadata.loginClass) &&
+ Objects.equals(loginCallbackClass, loginMetadata.loginCallbackClass);
+ }
+ }
}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/authenticator/SaslClientAuthenticator.java b/clients/src/main/java/org/apache/kafka/common/security/authenticator/SaslClientAuthenticator.java
index 8b0116563d8..2ef6d77f13f 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/authenticator/SaslClientAuthenticator.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/authenticator/SaslClientAuthenticator.java
@@ -24,9 +24,8 @@
import org.apache.kafka.common.errors.SaslAuthenticationException;
import org.apache.kafka.common.errors.UnsupportedSaslMechanismException;
import org.apache.kafka.common.network.Authenticator;
-import org.apache.kafka.common.network.Mode;
-import org.apache.kafka.common.network.NetworkReceive;
import org.apache.kafka.common.network.NetworkSend;
+import org.apache.kafka.common.network.NetworkReceive;
import org.apache.kafka.common.network.Send;
import org.apache.kafka.common.network.TransportLayer;
import org.apache.kafka.common.protocol.ApiKeys;
@@ -41,6 +40,7 @@
import org.apache.kafka.common.requests.SaslAuthenticateResponse;
import org.apache.kafka.common.requests.SaslHandshakeRequest;
import org.apache.kafka.common.requests.SaslHandshakeResponse;
+import org.apache.kafka.common.security.auth.AuthenticateCallbackHandler;
import org.apache.kafka.common.security.auth.KafkaPrincipal;
import org.apache.kafka.common.utils.Utils;
import org.slf4j.Logger;
@@ -87,7 +87,7 @@
private final SaslClient saslClient;
private final Map<String, ?> configs;
private final String clientPrincipalName;
- private final AuthCallbackHandler callbackHandler;
+ private final AuthenticateCallbackHandler callbackHandler;
// buffers used in `authenticate`
private NetworkReceive netInBuffer;
@@ -105,6 +105,7 @@
private short saslAuthenticateVersion;
public SaslClientAuthenticator(Map<String, ?> configs,
+ AuthenticateCallbackHandler callbackHandler,
String node,
Subject subject,
String servicePrincipal,
@@ -114,6 +115,7 @@ public SaslClientAuthenticator(Map<String, ?> configs,
TransportLayer transportLayer) throws IOException {
this.node = node;
this.subject = subject;
+ this.callbackHandler = callbackHandler;
this.host = host;
this.servicePrincipal = servicePrincipal;
this.mechanism = mechanism;
@@ -133,9 +135,6 @@ public SaslClientAuthenticator(Map<String, ?> configs,
else
this.clientPrincipalName = null;
- callbackHandler = new SaslClientCallbackHandler();
- callbackHandler.configure(configs, Mode.CLIENT, subject, mechanism);
-
saslClient = createSaslClient();
} catch (Exception e) {
throw new SaslAuthenticationException("Failed to configure SaslClientAuthenticator", e);
@@ -325,8 +324,6 @@ public boolean complete() {
public void close() throws IOException {
if (saslClient != null)
saslClient.dispose();
- if (callbackHandler != null)
- callbackHandler.close();
}
private byte[] receiveToken() throws IOException {
diff --git a/clients/src/main/java/org/apache/kafka/common/security/authenticator/SaslClientCallbackHandler.java b/clients/src/main/java/org/apache/kafka/common/security/authenticator/SaslClientCallbackHandler.java
index 31c51c22ca3..5b2a28181cd 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/authenticator/SaslClientCallbackHandler.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/authenticator/SaslClientCallbackHandler.java
@@ -16,6 +16,8 @@
*/
package org.apache.kafka.common.security.authenticator;
+import java.security.AccessController;
+import java.util.List;
import java.util.Map;
import javax.security.auth.Subject;
@@ -23,52 +25,46 @@
import javax.security.auth.callback.NameCallback;
import javax.security.auth.callback.PasswordCallback;
import javax.security.auth.callback.UnsupportedCallbackException;
+import javax.security.auth.login.AppConfigurationEntry;
import javax.security.sasl.AuthorizeCallback;
import javax.security.sasl.RealmCallback;
import org.apache.kafka.common.config.SaslConfigs;
-import org.apache.kafka.common.network.Mode;
import org.apache.kafka.common.security.scram.ScramExtensionsCallback;
+import org.apache.kafka.common.security.auth.AuthenticateCallbackHandler;
/**
- * Callback handler for Sasl clients. The callbacks required for the SASL mechanism
+ * Default callback handler for Sasl clients. The callbacks required for the SASL mechanism
* configured for the client should be supported by this callback handler. See
* <a href="https://docs.oracle.com/javase/8/docs/technotes/guides/security/sasl/sasl-refguide.html">Java SASL API</a>
* for the list of SASL callback handlers required for each SASL mechanism.
*/
-public class SaslClientCallbackHandler implements AuthCallbackHandler {
+public class SaslClientCallbackHandler implements AuthenticateCallbackHandler {
- private boolean isKerberos;
- private Subject subject;
+ private String mechanism;
@Override
- public void configure(Map<String, ?> configs, Mode mode, Subject subject, String mechanism) {
- this.isKerberos = mechanism.equals(SaslConfigs.GSSAPI_MECHANISM);
- this.subject = subject;
+ public void configure(Map<String, ?> configs, String saslMechanism, List<AppConfigurationEntry> jaasConfigEntries) {
+ this.mechanism = saslMechanism;
}
@Override
public void handle(Callback[] callbacks) throws UnsupportedCallbackException {
+ Subject subject = Subject.getSubject(AccessController.getContext());
for (Callback callback : callbacks) {
if (callback instanceof NameCallback) {
NameCallback nc = (NameCallback) callback;
- if (!isKerberos && subject != null && !subject.getPublicCredentials(String.class).isEmpty()) {
+ if (subject != null && !subject.getPublicCredentials(String.class).isEmpty()) {
nc.setName(subject.getPublicCredentials(String.class).iterator().next());
} else
nc.setName(nc.getDefaultName());
} else if (callback instanceof PasswordCallback) {
- if (!isKerberos && subject != null && !subject.getPrivateCredentials(String.class).isEmpty()) {
+ if (subject != null && !subject.getPrivateCredentials(String.class).isEmpty()) {
char[] password = subject.getPrivateCredentials(String.class).iterator().next().toCharArray();
((PasswordCallback) callback).setPassword(password);
} else {
String errorMessage = "Could not login: the client is being asked for a password, but the Kafka" +
" client code does not currently support obtaining a password from the user.";
- if (isKerberos) {
- errorMessage += " Make sure -Djava.security.auth.login.config property passed to JVM and" +
- " the client is configured to use a ticket cache (using" +
- " the JAAS configuration setting 'useTicketCache=true)'. Make sure you are using" +
- " FQDN of the Kafka broker you are trying to connect to.";
- }
throw new UnsupportedCallbackException(callback, errorMessage);
}
} else if (callback instanceof RealmCallback) {
@@ -83,7 +79,7 @@ public void handle(Callback[] callbacks) throws UnsupportedCallbackException {
ac.setAuthorizedID(authzId);
} else if (callback instanceof ScramExtensionsCallback) {
ScramExtensionsCallback sc = (ScramExtensionsCallback) callback;
- if (!isKerberos && subject != null && !subject.getPublicCredentials(Map.class).isEmpty()) {
+ if (!SaslConfigs.GSSAPI_MECHANISM.equals(mechanism) && subject != null && !subject.getPublicCredentials(Map.class).isEmpty()) {
sc.extensions((Map<String, String>) subject.getPublicCredentials(Map.class).iterator().next());
}
} else {
diff --git a/clients/src/main/java/org/apache/kafka/common/security/authenticator/SaslServerAuthenticator.java b/clients/src/main/java/org/apache/kafka/common/security/authenticator/SaslServerAuthenticator.java
index 2a80e5bc0e4..5140afb196d 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/authenticator/SaslServerAuthenticator.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/authenticator/SaslServerAuthenticator.java
@@ -28,7 +28,6 @@
import org.apache.kafka.common.network.Authenticator;
import org.apache.kafka.common.network.ChannelBuilders;
import org.apache.kafka.common.network.ListenerName;
-import org.apache.kafka.common.network.Mode;
import org.apache.kafka.common.network.NetworkReceive;
import org.apache.kafka.common.network.NetworkSend;
import org.apache.kafka.common.network.Send;
@@ -46,18 +45,15 @@
import org.apache.kafka.common.requests.SaslAuthenticateResponse;
import org.apache.kafka.common.requests.SaslHandshakeRequest;
import org.apache.kafka.common.requests.SaslHandshakeResponse;
-import org.apache.kafka.common.security.JaasContext;
+import org.apache.kafka.common.security.auth.AuthenticateCallbackHandler;
import org.apache.kafka.common.security.auth.KafkaPrincipal;
import org.apache.kafka.common.security.auth.KafkaPrincipalBuilder;
import org.apache.kafka.common.security.auth.SaslAuthenticationContext;
import org.apache.kafka.common.security.kerberos.KerberosName;
import org.apache.kafka.common.security.kerberos.KerberosShortNamer;
-import org.apache.kafka.common.security.scram.ScramCredential;
import org.apache.kafka.common.security.scram.ScramLoginModule;
-import org.apache.kafka.common.security.scram.ScramMechanism;
-import org.apache.kafka.common.security.scram.ScramServerCallbackHandler;
+import org.apache.kafka.common.security.scram.internal.ScramMechanism;
import org.apache.kafka.common.utils.Utils;
-import org.apache.kafka.common.security.token.delegation.DelegationTokenCache;
import org.ietf.jgss.GSSContext;
import org.ietf.jgss.GSSCredential;
import org.ietf.jgss.GSSException;
@@ -102,14 +98,12 @@
private final SecurityProtocol securityProtocol;
private final ListenerName listenerName;
private final String connectionId;
- private final Map<String, JaasContext> jaasContexts;
private final Map<String, Subject> subjects;
- private final CredentialCache credentialCache;
private final TransportLayer transportLayer;
private final Set<String> enabledMechanisms;
private final Map<String, ?> configs;
private final KafkaPrincipalBuilder principalBuilder;
- private final DelegationTokenCache tokenCache;
+ private final Map<String, AuthenticateCallbackHandler> callbackHandlers;
// Current SASL state
private SaslState saslState = SaslState.INITIAL_REQUEST;
@@ -119,7 +113,6 @@
private AuthenticationException pendingException = null;
private SaslServer saslServer;
private String saslMechanism;
- private AuthCallbackHandler callbackHandler;
// buffers used in `authenticate`
private NetworkReceive netInBuffer;
@@ -128,23 +121,19 @@
private boolean enableKafkaSaslAuthenticateHeaders;
public SaslServerAuthenticator(Map<String, ?> configs,
+ Map<String, AuthenticateCallbackHandler> callbackHandlers,
String connectionId,
- Map<String, JaasContext> jaasContexts,
Map<String, Subject> subjects,
KerberosShortNamer kerberosNameParser,
- CredentialCache credentialCache,
ListenerName listenerName,
SecurityProtocol securityProtocol,
- TransportLayer transportLayer,
- DelegationTokenCache tokenCache) throws IOException {
+ TransportLayer transportLayer) throws IOException {
+ this.callbackHandlers = callbackHandlers;
this.connectionId = connectionId;
- this.jaasContexts = jaasContexts;
this.subjects = subjects;
- this.credentialCache = credentialCache;
this.listenerName = listenerName;
this.securityProtocol = securityProtocol;
this.enableKafkaSaslAuthenticateHeaders = false;
- this.tokenCache = tokenCache;
this.transportLayer = transportLayer;
this.configs = configs;
@@ -154,8 +143,8 @@ public SaslServerAuthenticator(Map<String, ?> configs,
throw new IllegalArgumentException("No SASL mechanisms are enabled");
this.enabledMechanisms = new HashSet<>(enabledMechanisms);
for (String mechanism : enabledMechanisms) {
- if (!jaasContexts.containsKey(mechanism))
- throw new IllegalArgumentException("Jaas context not specified for SASL mechanism " + mechanism);
+ if (!callbackHandlers.containsKey(mechanism))
+ throw new IllegalArgumentException("Callback handler not specified for SASL mechanism " + mechanism);
if (!subjects.containsKey(mechanism))
throw new IllegalArgumentException("Subject cannot be null for SASL mechanism " + mechanism);
}
@@ -168,11 +157,7 @@ public SaslServerAuthenticator(Map<String, ?> configs,
private void createSaslServer(String mechanism) throws IOException {
this.saslMechanism = mechanism;
Subject subject = subjects.get(mechanism);
- if (!ScramMechanism.isScram(mechanism))
- callbackHandler = new SaslServerCallbackHandler(jaasContexts.get(mechanism));
- else
- callbackHandler = new ScramServerCallbackHandler(credentialCache.cache(mechanism, ScramCredential.class), tokenCache);
- callbackHandler.configure(configs, Mode.SERVER, subject, saslMechanism);
+ final AuthenticateCallbackHandler callbackHandler = callbackHandlers.get(mechanism);
if (mechanism.equals(SaslConfigs.GSSAPI_MECHANISM)) {
saslServer = createSaslKerberosServer(callbackHandler, configs, subject);
} else {
@@ -189,7 +174,7 @@ public SaslServer run() throws SaslException {
}
}
- private SaslServer createSaslKerberosServer(final AuthCallbackHandler saslServerCallbackHandler, final Map<String, ?> configs, Subject subject) throws IOException {
+ private SaslServer createSaslKerberosServer(final AuthenticateCallbackHandler saslServerCallbackHandler, final Map<String, ?> configs, Subject subject) throws IOException {
// server is using a JAAS-authenticated subject: determine service principal name and hostname from kafka server's subject.
final String servicePrincipal = SaslClientAuthenticator.firstPrincipal(subject);
KerberosName kerberosName;
@@ -316,8 +301,6 @@ public void close() throws IOException {
Utils.closeQuietly((Closeable) principalBuilder, "principal builder");
if (saslServer != null)
saslServer.dispose();
- if (callbackHandler != null)
- callbackHandler.close();
}
private void setSaslState(SaslState saslState) throws IOException {
diff --git a/clients/src/main/java/org/apache/kafka/common/security/authenticator/SaslServerCallbackHandler.java b/clients/src/main/java/org/apache/kafka/common/security/authenticator/SaslServerCallbackHandler.java
index 7d5372db977..d3d43cbfa26 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/authenticator/SaslServerCallbackHandler.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/authenticator/SaslServerCallbackHandler.java
@@ -16,51 +16,46 @@
*/
package org.apache.kafka.common.security.authenticator;
-import java.io.IOException;
+import java.util.List;
import java.util.Map;
-import org.apache.kafka.common.security.JaasContext;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import javax.security.auth.Subject;
import javax.security.auth.callback.Callback;
import javax.security.auth.callback.UnsupportedCallbackException;
+import javax.security.auth.login.AppConfigurationEntry;
import javax.security.sasl.AuthorizeCallback;
import javax.security.sasl.RealmCallback;
-import org.apache.kafka.common.network.Mode;
+import org.apache.kafka.common.config.SaslConfigs;
+import org.apache.kafka.common.security.auth.AuthenticateCallbackHandler;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
/**
- * Callback handler for Sasl servers. The callbacks required for all the SASL
+ * Default callback handler for Sasl servers. The callbacks required for all the SASL
* mechanisms enabled in the server should be supported by this callback handler. See
* <a href="https://docs.oracle.com/javase/8/docs/technotes/guides/security/sasl/sasl-refguide.html">Java SASL API</a>
* for the list of SASL callback handlers required for each SASL mechanism.
*/
-public class SaslServerCallbackHandler implements AuthCallbackHandler {
+public class SaslServerCallbackHandler implements AuthenticateCallbackHandler {
private static final Logger LOG = LoggerFactory.getLogger(SaslServerCallbackHandler.class);
- private final JaasContext jaasContext;
- public SaslServerCallbackHandler(JaasContext jaasContext) throws IOException {
- this.jaasContext = jaasContext;
- }
+ private String mechanism;
@Override
- public void configure(Map<String, ?> configs, Mode mode, Subject subject, String saslMechanism) {
- }
-
- public JaasContext jaasContext() {
- return jaasContext;
+ public void configure(Map<String, ?> configs, String mechanism, List<AppConfigurationEntry> jaasConfigEntries) {
+ this.mechanism = mechanism;
}
@Override
public void handle(Callback[] callbacks) throws UnsupportedCallbackException {
for (Callback callback : callbacks) {
- if (callback instanceof RealmCallback) {
+ if (callback instanceof RealmCallback)
handleRealmCallback((RealmCallback) callback);
- } else if (callback instanceof AuthorizeCallback) {
+ else if (callback instanceof AuthorizeCallback && mechanism.equals(SaslConfigs.GSSAPI_MECHANISM))
handleAuthorizeCallback((AuthorizeCallback) callback);
- }
+ else
+ throw new UnsupportedCallbackException(callback);
}
}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/kerberos/KerberosClientCallbackHandler.java b/clients/src/main/java/org/apache/kafka/common/security/kerberos/KerberosClientCallbackHandler.java
new file mode 100644
index 00000000000..fa9cad261c5
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/security/kerberos/KerberosClientCallbackHandler.java
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.security.kerberos;
+
+import org.apache.kafka.common.config.SaslConfigs;
+import org.apache.kafka.common.security.auth.AuthenticateCallbackHandler;
+
+import javax.security.auth.callback.Callback;
+import javax.security.auth.callback.NameCallback;
+import javax.security.auth.callback.PasswordCallback;
+import javax.security.auth.callback.UnsupportedCallbackException;
+import javax.security.auth.login.AppConfigurationEntry;
+import javax.security.sasl.AuthorizeCallback;
+import javax.security.sasl.RealmCallback;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Callback handler for SASL/GSSAPI clients.
+ */
+public class KerberosClientCallbackHandler implements AuthenticateCallbackHandler {
+
+ @Override
+ public void configure(Map<String, ?> configs, String saslMechanism, List<AppConfigurationEntry> jaasConfigEntries) {
+ if (!saslMechanism.equals(SaslConfigs.GSSAPI_MECHANISM))
+ throw new IllegalStateException("Kerberos callback handler should only be used with GSSAPI");
+ }
+
+ @Override
+ public void handle(Callback[] callbacks) throws UnsupportedCallbackException {
+ for (Callback callback : callbacks) {
+ if (callback instanceof NameCallback) {
+ NameCallback nc = (NameCallback) callback;
+ nc.setName(nc.getDefaultName());
+ } else if (callback instanceof PasswordCallback) {
+ String errorMessage = "Could not login: the client is being asked for a password, but the Kafka" +
+ " client code does not currently support obtaining a password from the user.";
+ errorMessage += " Make sure -Djava.security.auth.login.config property passed to JVM and" +
+ " the client is configured to use a ticket cache (using" +
+ " the JAAS configuration setting 'useTicketCache=true)'. Make sure you are using" +
+ " FQDN of the Kafka broker you are trying to connect to.";
+ throw new UnsupportedCallbackException(callback, errorMessage);
+ } else if (callback instanceof RealmCallback) {
+ RealmCallback rc = (RealmCallback) callback;
+ rc.setText(rc.getDefaultText());
+ } else if (callback instanceof AuthorizeCallback) {
+ AuthorizeCallback ac = (AuthorizeCallback) callback;
+ String authId = ac.getAuthenticationID();
+ String authzId = ac.getAuthorizationID();
+ ac.setAuthorized(authId.equals(authzId));
+ if (ac.isAuthorized())
+ ac.setAuthorizedID(authzId);
+ } else {
+ throw new UnsupportedCallbackException(callback, "Unrecognized SASL ClientCallback");
+ }
+ }
+ }
+
+ @Override
+ public void close() {
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/kerberos/KerberosLogin.java b/clients/src/main/java/org/apache/kafka/common/security/kerberos/KerberosLogin.java
index 65c3b1cbe44..ec996a8ccc6 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/kerberos/KerberosLogin.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/kerberos/KerberosLogin.java
@@ -18,6 +18,7 @@
import javax.security.auth.kerberos.KerberosPrincipal;
import javax.security.auth.login.AppConfigurationEntry;
+import javax.security.auth.login.Configuration;
import javax.security.auth.login.LoginContext;
import javax.security.auth.login.LoginException;
import javax.security.auth.kerberos.KerberosTicket;
@@ -25,6 +26,7 @@
import org.apache.kafka.common.security.JaasContext;
import org.apache.kafka.common.security.JaasUtils;
+import org.apache.kafka.common.security.auth.AuthenticateCallbackHandler;
import org.apache.kafka.common.security.authenticator.AbstractLogin;
import org.apache.kafka.common.config.SaslConfigs;
import org.apache.kafka.common.utils.KafkaThread;
@@ -33,11 +35,12 @@
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import java.util.Arrays;
import java.util.Date;
import java.util.List;
+import java.util.Map;
import java.util.Random;
import java.util.Set;
-import java.util.Map;
/**
* This class is responsible for refreshing Kerberos credentials for
@@ -78,13 +81,15 @@
private String serviceName;
private long lastLogin;
- public void configure(Map<String, ?> configs, JaasContext jaasContext) {
- super.configure(configs, jaasContext);
+ @Override
+ public void configure(Map<String, ?> configs, String contextName, Configuration configuration,
+ AuthenticateCallbackHandler callbackHandler) {
+ super.configure(configs, contextName, configuration, callbackHandler);
this.ticketRenewWindowFactor = (Double) configs.get(SaslConfigs.SASL_KERBEROS_TICKET_RENEW_WINDOW_FACTOR);
this.ticketRenewJitter = (Double) configs.get(SaslConfigs.SASL_KERBEROS_TICKET_RENEW_JITTER);
this.minTimeBeforeRelogin = (Long) configs.get(SaslConfigs.SASL_KERBEROS_MIN_TIME_BEFORE_RELOGIN);
this.kinitCmd = (String) configs.get(SaslConfigs.SASL_KERBEROS_KINIT_CMD);
- this.serviceName = getServiceName(configs, jaasContext);
+ this.serviceName = getServiceName(configs, contextName, configuration);
}
/**
@@ -99,13 +104,13 @@ public LoginContext login() throws LoginException {
subject = loginContext.getSubject();
isKrbTicket = !subject.getPrivateCredentials(KerberosTicket.class).isEmpty();
- List<AppConfigurationEntry> entries = jaasContext().configurationEntries();
- if (entries.isEmpty()) {
+ AppConfigurationEntry[] entries = configuration().getAppConfigurationEntry(contextName());
+ if (entries.length == 0) {
isUsingTicketCache = false;
principal = null;
} else {
// there will only be a single entry
- AppConfigurationEntry entry = entries.get(0);
+ AppConfigurationEntry entry = entries[0];
if (entry.getOptions().get("useTicketCache") != null) {
String val = (String) entry.getOptions().get("useTicketCache");
isUsingTicketCache = val.equals("true");
@@ -280,8 +285,9 @@ public String serviceName() {
return serviceName;
}
- private static String getServiceName(Map<String, ?> configs, JaasContext jaasContext) {
- String jaasServiceName = jaasContext.configEntryOption(JaasUtils.SERVICE_NAME, null);
+ private static String getServiceName(Map<String, ?> configs, String contextName, Configuration configuration) {
+ List<AppConfigurationEntry> configEntries = Arrays.asList(configuration.getAppConfigurationEntry(contextName));
+ String jaasServiceName = JaasContext.configEntryOption(configEntries, JaasUtils.SERVICE_NAME, null);
String configServiceName = (String) configs.get(SaslConfigs.SASL_KERBEROS_SERVICE_NAME);
if (jaasServiceName != null && configServiceName != null && !jaasServiceName.equals(configServiceName)) {
String message = String.format("Conflicting serviceName values found in JAAS and Kafka configs " +
@@ -360,7 +366,7 @@ private void reLogin() throws LoginException {
loginContext.logout();
//login and also update the subject field of this instance to
//have the new credentials (pass it to the LoginContext constructor)
- loginContext = new LoginContext(jaasContext().name(), subject, null, jaasContext().configuration());
+ loginContext = new LoginContext(contextName(), subject, null, configuration());
log.info("Initiating re-login for {}", principal);
loginContext.login();
}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/OAuthBearerLoginModule.java b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/OAuthBearerLoginModule.java
new file mode 100644
index 00000000000..ddadb99ea3c
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/OAuthBearerLoginModule.java
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.security.oauthbearer;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Objects;
+
+import javax.security.auth.Subject;
+import javax.security.auth.callback.Callback;
+import javax.security.auth.callback.CallbackHandler;
+import javax.security.auth.callback.UnsupportedCallbackException;
+import javax.security.auth.login.LoginException;
+import javax.security.auth.spi.LoginModule;
+
+import org.apache.kafka.common.config.SaslConfigs;
+import org.apache.kafka.common.security.auth.AuthenticateCallbackHandler;
+import org.apache.kafka.common.security.auth.Login;
+import org.apache.kafka.common.security.oauthbearer.internal.OAuthBearerSaslClientProvider;
+import org.apache.kafka.common.security.oauthbearer.internal.OAuthBearerSaslServerProvider;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The {@code LoginModule} for the SASL/OAUTHBEARER mechanism. When a client
+ * (whether a non-broker client or a broker when SASL/OAUTHBEARER is the
+ * inter-broker protocol) connects to Kafka the {@code OAuthBearerLoginModule}
+ * instance asks its configured {@link AuthenticateCallbackHandler}
+ * implementation to handle an instance of {@link OAuthBearerTokenCallback} and
+ * return an instance of {@link OAuthBearerToken}. A default, builtin
+ * {@link AuthenticateCallbackHandler} implementation creates an unsecured token
+ * as defined by these JAAS module options:
+ * <p>
+ * <table>
+ * <tr>
+ * <th>JAAS Module Option for Unsecured Token Retrieval</th>
+ * <th>Documentation</th>
+ * </tr>
+ * <tr>
+ * <td>{@code unsecuredLoginStringClaim_<claimname>="value"}</td>
+ * <td>Creates a {@code String} claim with the given name and value. Any valid
+ * claim name can be specified except '{@code iat}' and '{@code exp}' (these are
+ * automatically generated).</td>
+ * </tr>
+ * <tr>
+ * <td>{@code unsecuredLoginNumberClaim_<claimname>="value"}</td>
+ * <td>Creates a {@code Number} claim with the given name and value. Any valid
+ * claim name can be specified except '{@code iat}' and '{@code exp}' (these are
+ * automatically generated).</td>
+ * </tr>
+ * <tr>
+ * <td>{@code unsecuredLoginListClaim_<claimname>="value"}</td>
+ * <td>Creates a {@code String List} claim with the given name and values parsed
+ * from the given value where the first character is taken as the delimiter. For
+ * example: {@code unsecuredLoginListClaim_fubar="|value1|value2"}. Any valid
+ * claim name can be specified except '{@code iat}' and '{@code exp}' (these are
+ * automatically generated).</td>
+ * </tr>
+ * <tr>
+ * <td>{@code unsecuredLoginPrincipalClaimName}</td>
+ * <td>Set to a custom claim name if you wish the name of the {@code String}
+ * claim holding the principal name to be something other than
+ * '{@code sub}'.</td>
+ * </tr>
+ * <tr>
+ * <td>{@code unsecuredLoginLifetimeSeconds}</td>
+ * <td>Set to an integer value if the token expiration is to be set to something
+ * other than the default value of 3600 seconds (which is 1 hour). The
+ * '{@code exp}' claim will be set to reflect the expiration time.</td>
+ * </tr>
+ * <tr>
+ * <td>{@code unsecuredLoginScopeClaimName}</td>
+ * <td>Set to a custom claim name if you wish the name of the {@code String} or
+ * {@code String List} claim holding any token scope to be something other than
+ * '{@code scope}'.</td>
+ * </tr>
+ * </table>
+ * <p>
+ * Production use cases will require writing an implementation of
+ * {@link AuthenticateCallbackHandler} that can handle an instance of
+ * {@link OAuthBearerTokenCallback} and declaring it via either the
+ * {@code sasl.login.callback.handler.class} configuration option for a
+ * non-broker client or via the
+ * {@code listener.name.sasl_ssl.oauthbearer.sasl.login.callback.handler.class}
+ * configuration option for brokers (when SASL/OAUTHBEARER is the inter-broker
+ * protocol).
+ * <p>
+ * This class stores the retrieved {@link OAuthBearerToken} in the
+ * {@code Subject}'s private credentials where the {@code SaslClient} can
+ * retrieve it. An appropriate, builtin {@code SaslClient} implementation is
+ * automatically used and configured such that it can perform that retrieval.
+ * <p>
+ * Here is a typical, basic JAAS configuration for a client leveraging unsecured
+ * SASL/OAUTHBEARER authentication:
+ *
+ * <pre>
+ * KafkaClient {
+ * org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule Required
+ * unsecuredLoginStringClaim_sub="thePrincipalName";
+ * };
+ * </pre>
+ *
+ * An implementation of the {@link Login} interface specific to the
+ * {@code OAUTHBEARER} mechanism is automatically applied; it periodically
+ * refreshes any token before it expires so that the client can continue to make
+ * connections to brokers. The parameters that impact how the refresh algorithm
+ * operates are specified as part of the producer/consumer/broker configuration
+ * and are as follows. See the documentation for these properties elsewhere for
+ * details.
+ * <p>
+ * <table>
+ * <tr>
+ * <th>Producer/Consumer/Broker Configuration Property</th>
+ * </tr>
+ * <tr>
+ * <td>{@code sasl.login.refresh.window.factor}</td>
+ * </tr>
+ * <tr>
+ * <td>{@code sasl.login.refresh.window.jitter}</td>
+ * </tr>
+ * <tr>
+ * <td>{@code sasl.login.refresh.min.period.seconds}</td>
+ * </tr>
+ * <tr>
+ * <td>{@code sasl.login.refresh.min.buffer.seconds}</td>
+ * </tr>
+ * </table>
+ * <p>
+ * When a broker accepts a SASL/OAUTHBEARER connection the instance of the
+ * builtin {@code SaslServer} implementation asks its configured
+ * {@link AuthenticateCallbackHandler} implementation to handle an instance of
+ * {@link OAuthBearerValidatorCallback} constructed with the OAuth 2 Bearer
+ * Token's compact serialization and return an instance of
+ * {@link OAuthBearerToken} if the value validates. A default, builtin
+ * {@link AuthenticateCallbackHandler} implementation validates an unsecured
+ * token as defined by these JAAS module options:
+ * <p>
+ * <table>
+ * <tr>
+ * <th>JAAS Module Option for Unsecured Token Validation</th>
+ * <th>Documentation</th>
+ * </tr>
+ * <tr>
+ * <td>{@code unsecuredValidatorPrincipalClaimName="value"}</td>
+ * <td>Set to a non-empty value if you wish a particular {@code String} claim
+ * holding a principal name to be checked for existence; the default is to check
+ * for the existence of the '{@code sub}' claim.</td>
+ * </tr>
+ * <tr>
+ * <td>{@code unsecuredValidatorScopeClaimName="value"}</td>
+ * <td>Set to a custom claim name if you wish the name of the {@code String} or
+ * {@code String List} claim holding any token scope to be something other than
+ * '{@code scope}'.</td>
+ * </tr>
+ * <tr>
+ * <td>{@code unsecuredValidatorRequiredScope="value"}</td>
+ * <td>Set to a space-delimited list of scope values if you wish the
+ * {@code String/String List} claim holding the token scope to be checked to
+ * make sure it contains certain values.</td>
+ * </tr>
+ * <tr>
+ * <td>{@code unsecuredValidatorAllowableClockSkewMs="value"}</td>
+ * <td>Set to a positive integer value if you wish to allow up to some number of
+ * positive milliseconds of clock skew (the default is 0).</td>
+ * </tr>
+ * </table>
+ * <p>
+ * Here is a typical, basic JAAS configuration for a broker leveraging unsecured
+ * SASL/OAUTHBEARER validation:
+ *
+ * <pre>
+ * KafkaServer {
+ * org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule Required
+ * unsecuredLoginStringClaim_sub="thePrincipalName";
+ * };
+ * </pre>
+ *
+ * Production use cases will require writing an implementation of
+ * {@link AuthenticateCallbackHandler} that can handle an instance of
+ * {@link OAuthBearerValidatorCallback} and declaring it via the
+ * {@code listener.name.sasl_ssl.oauthbearer.sasl.server.callback.handler.class}
+ * broker configuration option.
+ * <p>
+ * The builtin {@code SaslServer} implementation for SASL/OAUTHBEARER in Kafka
+ * makes the instance of {@link OAuthBearerToken} available upon successful
+ * authentication via the negotiated property "{@code OAUTHBEARER.token}"; the
+ * token could be used in a custom authorizer (to authorize based on JWT claims
+ * rather than ACLs, for example).
+ * <p>
+ * This implementation's {@code logout()} method will logout the specific token
+ * that this instance logged in if it's {@code Subject} instance is shared
+ * across multiple {@code LoginContext}s and there happen to be multiple tokens
+ * on the {@code Subject}. This functionality is useful because it means a new
+ * token with a longer lifetime can be created before a soon-to-expire token is
+ * actually logged out. Otherwise, if multiple simultaneous tokens were not
+ * supported like this, the soon-to-be expired token would have to be logged out
+ * first, and then if the new token could not be retrieved (maybe the
+ * authorization server is temporarily unavailable, for example) the client
+ * would be left without a token and would be unable to create new connections.
+ * Better to mitigate this possibility by leaving the existing token (which
+ * still has some lifetime left) in place until a new replacement token is
+ * actually retrieved. This implementation supports this.
+ *
+ * @see SaslConfigs#SASL_LOGIN_REFRESH_WINDOW_FACTOR_DOC
+ * @see SaslConfigs#SASL_LOGIN_REFRESH_WINDOW_JITTER_DOC
+ * @see SaslConfigs#SASL_LOGIN_REFRESH_MIN_PERIOD_SECONDS_DOC
+ * @see SaslConfigs#SASL_LOGIN_REFRESH_BUFFER_SECONDS_DOC
+ */
+public class OAuthBearerLoginModule implements LoginModule {
+ /**
+ * The SASL Mechanism name for OAuth 2: {@code OAUTHBEARER}
+ */
+ public static final String OAUTHBEARER_MECHANISM = "OAUTHBEARER";
+ private static final Logger log = LoggerFactory.getLogger(OAuthBearerLoginModule.class);
+ private Subject subject = null;
+ private AuthenticateCallbackHandler callbackHandler = null;
+ private OAuthBearerToken tokenRequiringCommit = null;
+ private OAuthBearerToken myCommittedToken = null;
+
+ static {
+ OAuthBearerSaslClientProvider.initialize(); // not part of public API
+ OAuthBearerSaslServerProvider.initialize(); // not part of public API
+ }
+
+ @Override
+ public void initialize(Subject subject, CallbackHandler callbackHandler, Map<String, ?> sharedState,
+ Map<String, ?> options) {
+ this.subject = Objects.requireNonNull(subject);
+ if (!(Objects.requireNonNull(callbackHandler) instanceof AuthenticateCallbackHandler))
+ throw new IllegalArgumentException(String.format("Callback handler must be castable to %s: %s",
+ AuthenticateCallbackHandler.class.getName(), callbackHandler.getClass().getName()));
+ this.callbackHandler = (AuthenticateCallbackHandler) callbackHandler;
+ }
+
+ @Override
+ public boolean login() throws LoginException {
+ if (tokenRequiringCommit != null)
+ throw new IllegalStateException(String.format(
+ "Already have an uncommitted token with private credential token count=%d", committedTokenCount()));
+ if (myCommittedToken != null)
+ throw new IllegalStateException(String.format(
+ "Already have a committed token with private credential token count=%d; must login on another login context or logout here first before reusing the same login context",
+ committedTokenCount()));
+ OAuthBearerTokenCallback callback = new OAuthBearerTokenCallback();
+ try {
+ callbackHandler.handle(new Callback[] {callback});
+ } catch (IOException | UnsupportedCallbackException e) {
+ log.error(e.getMessage(), e);
+ throw new LoginException("An internal error occurred");
+ }
+ tokenRequiringCommit = callback.token();
+ if (tokenRequiringCommit == null) {
+ log.info(String.format("Login failed: %s : %s (URI=%s)", callback.errorCode(), callback.errorDescription(),
+ callback.errorUri()));
+ throw new LoginException(callback.errorDescription());
+ }
+ log.info("Login succeeded; invoke commit() to commit it; current committed token count={}",
+ committedTokenCount());
+ return true;
+ }
+
+ @Override
+ public boolean logout() {
+ if (tokenRequiringCommit != null)
+ throw new IllegalStateException(
+ "Cannot call logout() immediately after login(); need to first invoke commit() or abort()");
+ if (myCommittedToken == null) {
+ if (log.isDebugEnabled())
+ log.debug("Nothing here to log out");
+ return false;
+ }
+ log.info("Logging out my token; current committed token count = {}", committedTokenCount());
+ for (Iterator<Object> iterator = subject.getPrivateCredentials().iterator(); iterator.hasNext();) {
+ Object privateCredential = iterator.next();
+ if (privateCredential == myCommittedToken) {
+ iterator.remove();
+ myCommittedToken = null;
+ break;
+ }
+ }
+ log.info("Done logging out my token; committed token count is now {}", committedTokenCount());
+ return true;
+ }
+
+ @Override
+ public boolean commit() throws LoginException {
+ if (tokenRequiringCommit == null) {
+ if (log.isDebugEnabled())
+ log.debug("Nothing here to commit");
+ return false;
+ }
+ log.info("Committing my token; current committed token count = {}", committedTokenCount());
+ subject.getPrivateCredentials().add(tokenRequiringCommit);
+ myCommittedToken = tokenRequiringCommit;
+ tokenRequiringCommit = null;
+ log.info("Done committing my token; committed token count is now {}", committedTokenCount());
+ return true;
+ }
+
+ @Override
+ public boolean abort() throws LoginException {
+ if (tokenRequiringCommit != null) {
+ log.info("Login aborted");
+ tokenRequiringCommit = null;
+ return true;
+ }
+ if (log.isDebugEnabled())
+ log.debug("Nothing here to abort");
+ return false;
+ }
+
+ private int committedTokenCount() {
+ return subject.getPrivateCredentials(OAuthBearerToken.class).size();
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/OAuthBearerToken.java b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/OAuthBearerToken.java
new file mode 100644
index 00000000000..ee443ede7a2
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/OAuthBearerToken.java
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.security.oauthbearer;
+
+import java.util.Set;
+
+import org.apache.kafka.common.annotation.InterfaceStability;
+
+/**
+ * The <code>b64token</code> value as defined in
+ * <a href="https://tools.ietf.org/html/rfc6750#section-2.1">RFC 6750 Section
+ * 2.1</a> along with the token's specific scope and lifetime and principal
+ * name.
+ * <p>
+ * A network request would be required to re-hydrate an opaque token, and that
+ * could result in (for example) an {@code IOException}, but retrievers for
+ * various attributes ({@link #scope()}, {@link #lifetimeMs()}, etc.) declare no
+ * exceptions. Therefore, if a network request is required for any of these
+ * retriever methods, that request could be performed at construction time so
+ * that the various attributes can be reliably provided thereafter. For example,
+ * a constructor might declare {@code throws IOException} in such a case.
+ * Alternatively, the retrievers could throw unchecked exceptions.
+ * <p>
+ * This interface was introduced in 2.0.0 and, while it feels stable, it could
+ * evolve. We will try to evolve the API in a compatible manner (easier now that
+ * Java 7 and its lack of default methods doesn't have to be supported), but we
+ * reserve the right to make breaking changes in minor releases, if necessary.
+ * We will update the {@code InterfaceStability} annotation and this notice once
+ * the API is considered stable.
+ *
+ * @see <a href="https://tools.ietf.org/html/rfc6749#section-1.4">RFC 6749
+ * Section 1.4</a> and
+ * <a href="https://tools.ietf.org/html/rfc6750#section-2.1">RFC 6750
+ * Section 2.1</a>
+ */
+@InterfaceStability.Evolving
+public interface OAuthBearerToken {
+ /**
+ * The <code>b64token</code> value as defined in
+ * <a href="https://tools.ietf.org/html/rfc6750#section-2.1">RFC 6750 Section
+ * 2.1</a>
+ *
+ * @return <code>b64token</code> value as defined in
+ * <a href="https://tools.ietf.org/html/rfc6750#section-2.1">RFC 6750
+ * Section 2.1</a>
+ */
+ String value();
+
+ /**
+ * The token's scope of access, as per
+ * <a href="https://tools.ietf.org/html/rfc6749#section-1.4">RFC 6749 Section
+ * 1.4</a>
+ *
+ * @return the token's (always non-null but potentially empty) scope of access,
+ * as per <a href="https://tools.ietf.org/html/rfc6749#section-1.4">RFC
+ * 6749 Section 1.4</a>. Note that all values in the returned set will
+ * be trimmed of preceding and trailing whitespace, and the result will
+ * never contain the empty string.
+ */
+ Set<String> scope();
+
+ /**
+ * The token's lifetime, expressed as the number of milliseconds since the
+ * epoch, as per <a href="https://tools.ietf.org/html/rfc6749#section-1.4">RFC
+ * 6749 Section 1.4</a>
+ *
+ * @return the token'slifetime, expressed as the number of milliseconds since
+ * the epoch, as per
+ * <a href="https://tools.ietf.org/html/rfc6749#section-1.4">RFC 6749
+ * Section 1.4</a>.
+ */
+ long lifetimeMs();
+
+ /**
+ * The name of the principal to which this credential applies
+ *
+ * @return the always non-null/non-empty principal name
+ */
+ String principalName();
+
+ /**
+ * When the credential became valid, in terms of the number of milliseconds
+ * since the epoch, if known, otherwise null. An expiring credential may not
+ * necessarily indicate when it was created -- just when it expires -- so we
+ * need to support a null return value here.
+ *
+ * @return the time when the credential became valid, in terms of the number of
+ * milliseconds since the epoch, if known, otherwise null
+ */
+ Long startTimeMs();
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/OAuthBearerTokenCallback.java b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/OAuthBearerTokenCallback.java
new file mode 100644
index 00000000000..62ce492eb91
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/OAuthBearerTokenCallback.java
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.security.oauthbearer;
+
+import java.util.Objects;
+
+import javax.security.auth.callback.Callback;
+
+import org.apache.kafka.common.annotation.InterfaceStability;
+
+/**
+ * A {@code Callback} for use by the {@code SaslClient} and {@code Login}
+ * implementations when they require an OAuth 2 bearer token. Callback handlers
+ * should use the {@link #error(String, String, String)} method to communicate
+ * errors returned by the authorization server as per
+ * <a href="https://tools.ietf.org/html/rfc6749#section-5.2">RFC 6749: The OAuth
+ * 2.0 Authorization Framework</a>. Callback handlers should communicate other
+ * problems by raising an {@code IOException}.
+ * <p>
+ * This class was introduced in 2.0.0 and, while it feels stable, it could
+ * evolve. We will try to evolve the API in a compatible manner, but we reserve
+ * the right to make breaking changes in minor releases, if necessary. We will
+ * update the {@code InterfaceStability} annotation and this notice once the API
+ * is considered stable.
+ */
+@InterfaceStability.Evolving
+public class OAuthBearerTokenCallback implements Callback {
+ private OAuthBearerToken token = null;
+ private String errorCode = null;
+ private String errorDescription = null;
+ private String errorUri = null;
+
+ /**
+ * Return the (potentially null) token
+ *
+ * @return the (potentially null) token
+ */
+ public OAuthBearerToken token() {
+ return token;
+ }
+
+ /**
+ * Return the optional (but always non-empty if not null) error code as per
+ * <a href="https://tools.ietf.org/html/rfc6749#section-5.2">RFC 6749: The OAuth
+ * 2.0 Authorization Framework</a>.
+ *
+ * @return the optional (but always non-empty if not null) error code
+ */
+ public String errorCode() {
+ return errorCode;
+ }
+
+ /**
+ * Return the (potentially null) error description as per
+ * <a href="https://tools.ietf.org/html/rfc6749#section-5.2">RFC 6749: The OAuth
+ * 2.0 Authorization Framework</a>.
+ *
+ * @return the (potentially null) error description
+ */
+ public String errorDescription() {
+ return errorDescription;
+ }
+
+ /**
+ * Return the (potentially null) error URI as per
+ * <a href="https://tools.ietf.org/html/rfc6749#section-5.2">RFC 6749: The OAuth
+ * 2.0 Authorization Framework</a>.
+ *
+ * @return the (potentially null) error URI
+ */
+ public String errorUri() {
+ return errorUri;
+ }
+
+ /**
+ * Set the token. All error-related values are cleared.
+ *
+ * @param token
+ * the mandatory token to set
+ */
+ public void token(OAuthBearerToken token) {
+ this.token = Objects.requireNonNull(token);
+ this.errorCode = null;
+ this.errorDescription = null;
+ this.errorUri = null;
+ }
+
+ /**
+ * Set the error values as per
+ * <a href="https://tools.ietf.org/html/rfc6749#section-5.2">RFC 6749: The OAuth
+ * 2.0 Authorization Framework</a>. Any token is cleared.
+ *
+ * @param errorCode
+ * the mandatory error code to set
+ * @param errorDescription
+ * the optional error description to set
+ * @param errorUri
+ * the optional error URI to set
+ */
+ public void error(String errorCode, String errorDescription, String errorUri) {
+ if (Objects.requireNonNull(errorCode).isEmpty())
+ throw new IllegalArgumentException("error code must not be empty");
+ this.errorCode = errorCode;
+ this.errorDescription = errorDescription;
+ this.errorUri = errorUri;
+ this.token = null;
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/OAuthBearerValidatorCallback.java b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/OAuthBearerValidatorCallback.java
new file mode 100644
index 00000000000..36bcf089ac3
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/OAuthBearerValidatorCallback.java
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.security.oauthbearer;
+
+import java.util.Objects;
+
+import javax.security.auth.callback.Callback;
+
+import org.apache.kafka.common.annotation.InterfaceStability;
+
+/**
+ * A {@code Callback} for use by the {@code SaslServer} implementation when it
+ * needs to provide an OAuth 2 bearer token compact serialization for
+ * validation. Callback handlers should use the
+ * {@link #error(String, String, String)} method to communicate errors back to
+ * the SASL Client as per
+ * <a href="https://tools.ietf.org/html/rfc6749#section-5.2">RFC 6749: The OAuth
+ * 2.0 Authorization Framework</a> and the <a href=
+ * "https://www.iana.org/assignments/oauth-parameters/oauth-parameters.xhtml#extensions-error">IANA
+ * OAuth Extensions Error Registry</a>. Callback handlers should communicate
+ * other problems by raising an {@code IOException}.
+ * <p>
+ * This class was introduced in 2.0.0 and, while it feels stable, it could
+ * evolve. We will try to evolve the API in a compatible manner, but we reserve
+ * the right to make breaking changes in minor releases, if necessary. We will
+ * update the {@code InterfaceStability} annotation and this notice once the API
+ * is considered stable.
+ */
+@InterfaceStability.Evolving
+public class OAuthBearerValidatorCallback implements Callback {
+ private final String tokenValue;
+ private OAuthBearerToken token = null;
+ private String errorStatus = null;
+ private String errorScope = null;
+ private String errorOpenIDConfiguration = null;
+
+ /**
+ * Constructor
+ *
+ * @param tokenValue
+ * the mandatory/non-blank token value
+ */
+ public OAuthBearerValidatorCallback(String tokenValue) {
+ if (Objects.requireNonNull(tokenValue).isEmpty())
+ throw new IllegalArgumentException("token value must not be empty");
+ this.tokenValue = tokenValue;
+ }
+
+ /**
+ * Return the (always non-null) token value
+ *
+ * @return the (always non-null) token value
+ */
+ public String tokenValue() {
+ return tokenValue;
+ }
+
+ /**
+ * Return the (potentially null) token
+ *
+ * @return the (potentially null) token
+ */
+ public OAuthBearerToken token() {
+ return token;
+ }
+
+ /**
+ * Return the (potentially null) error status value as per
+ * <a href="https://tools.ietf.org/html/rfc7628#section-3.2.2">RFC 7628: A Set
+ * of Simple Authentication and Security Layer (SASL) Mechanisms for OAuth</a>
+ * and the <a href=
+ * "https://www.iana.org/assignments/oauth-parameters/oauth-parameters.xhtml#extensions-error">IANA
+ * OAuth Extensions Error Registry</a>.
+ *
+ * @return the (potentially null) error status value
+ */
+ public String errorStatus() {
+ return errorStatus;
+ }
+
+ /**
+ * Return the (potentially null) error scope value as per
+ * <a href="https://tools.ietf.org/html/rfc7628#section-3.2.2">RFC 7628: A Set
+ * of Simple Authentication and Security Layer (SASL) Mechanisms for OAuth</a>.
+ *
+ * @return the (potentially null) error scope value
+ */
+ public String errorScope() {
+ return errorScope;
+ }
+
+ /**
+ * Return the (potentially null) error openid-configuration value as per
+ * <a href="https://tools.ietf.org/html/rfc7628#section-3.2.2">RFC 7628: A Set
+ * of Simple Authentication and Security Layer (SASL) Mechanisms for OAuth</a>.
+ *
+ * @return the (potentially null) error openid-configuration value
+ */
+ public String errorOpenIDConfiguration() {
+ return errorOpenIDConfiguration;
+ }
+
+ /**
+ * Set the token. The token value is unchanged and is expected to match the
+ * provided token's value. All error values are cleared.
+ *
+ * @param token
+ * the mandatory token to set
+ */
+ public void token(OAuthBearerToken token) {
+ this.token = Objects.requireNonNull(token);
+ this.errorStatus = null;
+ this.errorScope = null;
+ this.errorOpenIDConfiguration = null;
+ }
+
+ /**
+ * Set the error values as per
+ * <a href="https://tools.ietf.org/html/rfc7628#section-3.2.2">RFC 7628: A Set
+ * of Simple Authentication and Security Layer (SASL) Mechanisms for OAuth</a>.
+ * Any token is cleared.
+ *
+ * @param errorStatus
+ * the mandatory error status value from the <a href=
+ * "https://www.iana.org/assignments/oauth-parameters/oauth-parameters.xhtml#extensions-error">IANA
+ * OAuth Extensions Error Registry</a> to set
+ * @param errorScope
+ * the optional error scope value to set
+ * @param errorOpenIDConfiguration
+ * the optional error openid-configuration value to set
+ */
+ public void error(String errorStatus, String errorScope, String errorOpenIDConfiguration) {
+ if (Objects.requireNonNull(errorStatus).isEmpty())
+ throw new IllegalArgumentException("error status must not be empty");
+ this.errorStatus = errorStatus;
+ this.errorScope = errorScope;
+ this.errorOpenIDConfiguration = errorOpenIDConfiguration;
+ this.token = null;
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/OAuthBearerRefreshingLogin.java b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/OAuthBearerRefreshingLogin.java
new file mode 100644
index 00000000000..303188cc10c
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/OAuthBearerRefreshingLogin.java
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.security.oauthbearer.internal;
+
+import java.util.Map;
+import java.util.Set;
+
+import javax.security.auth.Subject;
+import javax.security.auth.login.Configuration;
+import javax.security.auth.login.LoginContext;
+import javax.security.auth.login.LoginException;
+
+import org.apache.kafka.common.config.SaslConfigs;
+import org.apache.kafka.common.security.auth.AuthenticateCallbackHandler;
+import org.apache.kafka.common.security.auth.Login;
+import org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule;
+import org.apache.kafka.common.security.oauthbearer.OAuthBearerToken;
+import org.apache.kafka.common.security.oauthbearer.internal.expiring.ExpiringCredential;
+import org.apache.kafka.common.security.oauthbearer.internal.expiring.ExpiringCredentialRefreshConfig;
+import org.apache.kafka.common.security.oauthbearer.internal.expiring.ExpiringCredentialRefreshingLogin;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * This class is responsible for refreshing logins for both Kafka client and
+ * server when the credential is an OAuth 2 bearer token communicated over
+ * SASL/OAUTHBEARER. An OAuth 2 bearer token has a limited lifetime, and an
+ * instance of this class periodically refreshes it so that the client can
+ * create new connections to brokers on an ongoing basis.
+ * <p>
+ * This class does not need to be explicitly set via the
+ * {@code sasl.login.class} client configuration property or the
+ * {@code listener.name.sasl_[plaintext|ssl].oauthbearer.sasl.login.class}
+ * broker configuration property when the SASL mechanism is OAUTHBEARER; it is
+ * automatically set by default in that case.
+ * <p>
+ * The parameters that impact how the refresh algorithm operates are specified
+ * as part of the producer/consumer/broker configuration and are as follows. See
+ * the documentation for these properties elsewhere for details.
+ * <table>
+ * <tr>
+ * <th>Producer/Consumer/Broker Configuration Property</th>
+ * </tr>
+ * <tr>
+ * <td>{@code sasl.login.refresh.window.factor}</td>
+ * </tr>
+ * <tr>
+ * <td>{@code sasl.login.refresh.window.jitter}</td>
+ * </tr>
+ * <tr>
+ * <td>{@code sasl.login.refresh.min.period.seconds}</td>
+ * </tr>
+ * <tr>
+ * <td>{@code sasl.login.refresh.min.buffer.seconds}</td>
+ * </tr>
+ * </table>
+ *
+ * @see OAuthBearerLoginModule
+ * @see SaslConfigs#SASL_LOGIN_REFRESH_WINDOW_FACTOR_DOC
+ * @see SaslConfigs#SASL_LOGIN_REFRESH_WINDOW_JITTER_DOC
+ * @see SaslConfigs#SASL_LOGIN_REFRESH_MIN_PERIOD_SECONDS_DOC
+ * @see SaslConfigs#SASL_LOGIN_REFRESH_BUFFER_SECONDS_DOC
+ */
+public class OAuthBearerRefreshingLogin implements Login {
+ private static final Logger log = LoggerFactory.getLogger(OAuthBearerRefreshingLogin.class);
+ private ExpiringCredentialRefreshingLogin expiringCredentialRefreshingLogin = null;
+
+ @Override
+ public void configure(Map<String, ?> configs, String contextName, Configuration configuration,
+ AuthenticateCallbackHandler loginCallbackHandler) {
+ /*
+ * Specify this class as the one to synchronize on so that only one OAuth 2
+ * Bearer Token is refreshed at a given time. Specify null if we don't mind
+ * multiple simultaneously refreshes. Refreshes happen on the order of minutes
+ * rather than seconds or milliseconds, and there are typically minutes of
+ * lifetime remaining when the refresh occurs, so serializing them seems
+ * reasonable.
+ */
+ Class<OAuthBearerRefreshingLogin> classToSynchronizeOnPriorToRefresh = OAuthBearerRefreshingLogin.class;
+ expiringCredentialRefreshingLogin = new ExpiringCredentialRefreshingLogin(contextName, configuration,
+ new ExpiringCredentialRefreshConfig(configs, true), loginCallbackHandler,
+ classToSynchronizeOnPriorToRefresh) {
+ @Override
+ public ExpiringCredential expiringCredential() {
+ Set<OAuthBearerToken> privateCredentialTokens = expiringCredentialRefreshingLogin.subject()
+ .getPrivateCredentials(OAuthBearerToken.class);
+ if (privateCredentialTokens.isEmpty())
+ return null;
+ final OAuthBearerToken token = privateCredentialTokens.iterator().next();
+ if (log.isDebugEnabled())
+ log.debug("Found expiring credential with principal '{}'.", token.principalName());
+ return new ExpiringCredential() {
+ @Override
+ public String principalName() {
+ return token.principalName();
+ }
+
+ @Override
+ public Long startTimeMs() {
+ return token.startTimeMs();
+ }
+
+ @Override
+ public long expireTimeMs() {
+ return token.lifetimeMs();
+ }
+
+ @Override
+ public Long absoluteLastRefreshTimeMs() {
+ return null;
+ }
+ };
+ }
+ };
+ }
+
+ @Override
+ public void close() {
+ if (expiringCredentialRefreshingLogin != null)
+ expiringCredentialRefreshingLogin.close();
+ }
+
+ @Override
+ public Subject subject() {
+ return expiringCredentialRefreshingLogin != null ? expiringCredentialRefreshingLogin.subject() : null;
+ }
+
+ @Override
+ public String serviceName() {
+ return expiringCredentialRefreshingLogin != null ? expiringCredentialRefreshingLogin.serviceName() : null;
+ }
+
+ @Override
+ public synchronized LoginContext login() throws LoginException {
+ if (expiringCredentialRefreshingLogin != null)
+ return expiringCredentialRefreshingLogin.login();
+ throw new LoginException("Login was not configured properly");
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/OAuthBearerSaslClient.java b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/OAuthBearerSaslClient.java
new file mode 100644
index 00000000000..bb6ff0a8ae5
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/OAuthBearerSaslClient.java
@@ -0,0 +1,178 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.security.oauthbearer.internal;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+
+import javax.security.auth.callback.Callback;
+import javax.security.auth.callback.CallbackHandler;
+import javax.security.auth.callback.UnsupportedCallbackException;
+import javax.security.sasl.SaslClient;
+import javax.security.sasl.SaslClientFactory;
+import javax.security.sasl.SaslException;
+
+import org.apache.kafka.common.errors.IllegalSaslStateException;
+import org.apache.kafka.common.security.auth.AuthenticateCallbackHandler;
+import org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule;
+import org.apache.kafka.common.security.oauthbearer.OAuthBearerToken;
+import org.apache.kafka.common.security.oauthbearer.OAuthBearerTokenCallback;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * {@code SaslClient} implementation for SASL/OAUTHBEARER in Kafka. This
+ * implementation requires an instance of {@code AuthenticateCallbackHandler}
+ * that can handle an instance of {@link OAuthBearerTokenCallback} and return
+ * the {@link OAuthBearerToken} generated by the {@code login()} event on the
+ * {@code LoginContext}.
+ *
+ * @see <a href="https://tools.ietf.org/html/rfc6750#section-2.1">RFC 6750,
+ * Section 2.1</a>
+ *
+ */
+public class OAuthBearerSaslClient implements SaslClient {
+ static final byte BYTE_CONTROL_A = (byte) 0x01;
+ private static final Logger log = LoggerFactory.getLogger(OAuthBearerSaslClient.class);
+ private final CallbackHandler callbackHandler;
+
+ enum State {
+ SEND_CLIENT_FIRST_MESSAGE, RECEIVE_SERVER_FIRST_MESSAGE, RECEIVE_SERVER_MESSAGE_AFTER_FAILURE, COMPLETE, FAILED
+ };
+
+ private State state;
+
+ public OAuthBearerSaslClient(AuthenticateCallbackHandler callbackHandler) {
+ this.callbackHandler = Objects.requireNonNull(callbackHandler);
+ setState(State.SEND_CLIENT_FIRST_MESSAGE);
+ }
+
+ public CallbackHandler callbackHandler() {
+ return callbackHandler;
+ }
+
+ @Override
+ public String getMechanismName() {
+ return OAuthBearerLoginModule.OAUTHBEARER_MECHANISM;
+ }
+
+ @Override
+ public boolean hasInitialResponse() {
+ return true;
+ }
+
+ @Override
+ public byte[] evaluateChallenge(byte[] challenge) throws SaslException {
+ try {
+ OAuthBearerTokenCallback callback = new OAuthBearerTokenCallback();
+ switch (state) {
+ case SEND_CLIENT_FIRST_MESSAGE:
+ if (challenge != null && challenge.length != 0)
+ throw new SaslException("Expected empty challenge");
+ callbackHandler().handle(new Callback[] {callback});
+ setState(State.RECEIVE_SERVER_FIRST_MESSAGE);
+ return String.format("n,,auth=Bearer %s", callback.token().value())
+ .getBytes(StandardCharsets.UTF_8);
+ case RECEIVE_SERVER_FIRST_MESSAGE:
+ if (challenge != null && challenge.length != 0) {
+ String jsonErrorResponse = new String(challenge, StandardCharsets.UTF_8);
+ if (log.isDebugEnabled())
+ log.debug("Sending %%x01 response to server after receiving an error: {}",
+ jsonErrorResponse);
+ setState(State.RECEIVE_SERVER_MESSAGE_AFTER_FAILURE);
+ return new byte[] {BYTE_CONTROL_A};
+ }
+ callbackHandler().handle(new Callback[] {callback});
+ if (log.isDebugEnabled())
+ log.debug("Successfully authenticated as {}", callback.token().principalName());
+ setState(State.COMPLETE);
+ return null;
+ default:
+ throw new IllegalSaslStateException("Unexpected challenge in Sasl client state " + state);
+ }
+ } catch (SaslException e) {
+ setState(State.FAILED);
+ throw e;
+ } catch (IOException | UnsupportedCallbackException e) {
+ setState(State.FAILED);
+ throw new SaslException(e.getMessage(), e);
+ }
+ }
+
+ @Override
+ public boolean isComplete() {
+ return state == State.COMPLETE;
+ }
+
+ @Override
+ public byte[] unwrap(byte[] incoming, int offset, int len) throws SaslException {
+ if (!isComplete())
+ throw new IllegalStateException("Authentication exchange has not completed");
+ return Arrays.copyOfRange(incoming, offset, offset + len);
+ }
+
+ @Override
+ public byte[] wrap(byte[] outgoing, int offset, int len) throws SaslException {
+ if (!isComplete())
+ throw new IllegalStateException("Authentication exchange has not completed");
+ return Arrays.copyOfRange(outgoing, offset, offset + len);
+ }
+
+ @Override
+ public Object getNegotiatedProperty(String propName) {
+ if (!isComplete())
+ throw new IllegalStateException("Authentication exchange has not completed");
+ return null;
+ }
+
+ @Override
+ public void dispose() throws SaslException {
+ }
+
+ private void setState(State state) {
+ log.debug("Setting SASL/{} client state to {}", OAuthBearerLoginModule.OAUTHBEARER_MECHANISM, state);
+ this.state = state;
+ }
+
+ public static class OAuthBearerSaslClientFactory implements SaslClientFactory {
+ @Override
+ public SaslClient createSaslClient(String[] mechanisms, String authorizationId, String protocol,
+ String serverName, Map<String, ?> props, CallbackHandler callbackHandler) throws SaslException {
+ String[] mechanismNamesCompatibleWithPolicy = getMechanismNames(props);
+ for (String mechanism : mechanisms) {
+ for (int i = 0; i < mechanismNamesCompatibleWithPolicy.length; i++) {
+ if (mechanismNamesCompatibleWithPolicy[i].equals(mechanism)) {
+ if (!(Objects.requireNonNull(callbackHandler) instanceof AuthenticateCallbackHandler))
+ throw new IllegalArgumentException(String.format(
+ "Callback handler must be castable to %s: %s",
+ AuthenticateCallbackHandler.class.getName(), callbackHandler.getClass().getName()));
+ return new OAuthBearerSaslClient((AuthenticateCallbackHandler) callbackHandler);
+ }
+ }
+ }
+ return null;
+ }
+
+ @Override
+ public String[] getMechanismNames(Map<String, ?> props) {
+ return OAuthBearerSaslServer.mechanismNamesCompatibleWithPolicy(props);
+ }
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/OAuthBearerSaslClientCallbackHandler.java b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/OAuthBearerSaslClientCallbackHandler.java
new file mode 100644
index 00000000000..d406a8af703
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/OAuthBearerSaslClientCallbackHandler.java
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.security.oauthbearer.internal;
+
+import java.io.IOException;
+import java.security.AccessController;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import javax.security.auth.Subject;
+import javax.security.auth.callback.Callback;
+import javax.security.auth.callback.UnsupportedCallbackException;
+import javax.security.auth.login.AppConfigurationEntry;
+
+import org.apache.kafka.common.security.auth.AuthenticateCallbackHandler;
+import org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule;
+import org.apache.kafka.common.security.oauthbearer.OAuthBearerToken;
+import org.apache.kafka.common.security.oauthbearer.OAuthBearerTokenCallback;
+
+/**
+ * An implementation of {@code AuthenticateCallbackHandler} that recognizes
+ * {@link OAuthBearerTokenCallback} and retrieves OAuth 2 Bearer Token that was
+ * created when the {@code OAuthBearerLoginModule} logged in by looking for an
+ * instance of {@link OAuthBearerToken} in the {@code Subject}'s private
+ * credentials.
+ * <p>
+ * Use of this class is configured automatically and does not need to be
+ * explicitly set via the {@code sasl.client.callback.handler.class}
+ * configuration property.
+ */
+public class OAuthBearerSaslClientCallbackHandler implements AuthenticateCallbackHandler {
+ private boolean configured = false;
+
+ /**
+ * Return true if this instance has been configured, otherwise false
+ *
+ * @return true if this instance has been configured, otherwise false
+ */
+ public boolean configured() {
+ return configured;
+ }
+
+ @Override
+ public void configure(Map<String, ?> configs, String saslMechanism, List<AppConfigurationEntry> jaasConfigEntries) {
+ if (!OAuthBearerLoginModule.OAUTHBEARER_MECHANISM.equals(saslMechanism))
+ throw new IllegalArgumentException(String.format("Unexpected SASL mechanism: %s", saslMechanism));
+ configured = true;
+ }
+
+ @Override
+ public void handle(Callback[] callbacks) throws IOException, UnsupportedCallbackException {
+ if (!configured())
+ throw new IllegalStateException("Callback handler not configured");
+ for (Callback callback : callbacks) {
+ if (callback instanceof OAuthBearerTokenCallback)
+ handleCallback((OAuthBearerTokenCallback) callback);
+ else
+ throw new UnsupportedCallbackException(callback);
+ }
+ }
+
+ @Override
+ public void close() {
+ // empty
+ }
+
+ private void handleCallback(OAuthBearerTokenCallback callback) throws IOException {
+ if (callback.token() != null)
+ throw new IllegalArgumentException("Callback had a token already");
+ Subject subject = Subject.getSubject(AccessController.getContext());
+ Set<OAuthBearerToken> privateCredentials = subject != null
+ ? subject.getPrivateCredentials(OAuthBearerToken.class)
+ : Collections.<OAuthBearerToken>emptySet();
+ if (privateCredentials.size() != 1)
+ throw new IOException(
+ String.format("Unable to find OAuth Bearer token in Subject's private credentials (size=%d)",
+ privateCredentials.size()));
+ callback.token(privateCredentials.iterator().next());
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/OAuthBearerSaslClientProvider.java b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/OAuthBearerSaslClientProvider.java
new file mode 100644
index 00000000000..b292b31167e
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/OAuthBearerSaslClientProvider.java
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.security.oauthbearer.internal;
+
+import java.security.Provider;
+import java.security.Security;
+
+import org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule;
+import org.apache.kafka.common.security.oauthbearer.internal.OAuthBearerSaslClient.OAuthBearerSaslClientFactory;
+
+public class OAuthBearerSaslClientProvider extends Provider {
+ private static final long serialVersionUID = 1L;
+
+ protected OAuthBearerSaslClientProvider() {
+ super("SASL/OAUTHBEARER Client Provider", 1.0, "SASL/OAUTHBEARER Client Provider for Kafka");
+ put("SaslClientFactory." + OAuthBearerLoginModule.OAUTHBEARER_MECHANISM,
+ OAuthBearerSaslClientFactory.class.getName());
+ }
+
+ public static void initialize() {
+ Security.addProvider(new OAuthBearerSaslClientProvider());
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/OAuthBearerSaslServer.java b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/OAuthBearerSaslServer.java
new file mode 100644
index 00000000000..f84517d15b1
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/OAuthBearerSaslServer.java
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.security.oauthbearer.internal;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import javax.security.auth.callback.Callback;
+import javax.security.auth.callback.CallbackHandler;
+import javax.security.auth.callback.UnsupportedCallbackException;
+import javax.security.sasl.Sasl;
+import javax.security.sasl.SaslException;
+import javax.security.sasl.SaslServer;
+import javax.security.sasl.SaslServerFactory;
+
+import org.apache.kafka.common.errors.SaslAuthenticationException;
+import org.apache.kafka.common.security.auth.AuthenticateCallbackHandler;
+import org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule;
+import org.apache.kafka.common.security.oauthbearer.OAuthBearerToken;
+import org.apache.kafka.common.security.oauthbearer.OAuthBearerValidatorCallback;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * {@code SaslServer} implementation for SASL/OAUTHBEARER in Kafka. An instance
+ * of {@link OAuthBearerToken} is available upon successful authentication via
+ * the negotiated property "{@code OAUTHBEARER.token}"; the token could be used
+ * in a custom authorizer (to authorize based on JWT claims rather than ACLs,
+ * for example).
+ */
+public class OAuthBearerSaslServer implements SaslServer {
+ private static final String INVALID_OAUTHBEARER_CLIENT_FIRST_MESSAGE = "Invalid OAUTHBEARER client first message";
+ private static final Logger log = LoggerFactory.getLogger(OAuthBearerSaslServer.class);
+ private static final String NEGOTIATED_PROPERTY_KEY_TOKEN = OAuthBearerLoginModule.OAUTHBEARER_MECHANISM + ".token";
+ private static final String INTERNAL_ERROR_ON_SERVER = "Authentication could not be performed due to an internal error on the server";
+ private static final String SASLNAME = "(?:[\\x01-\\x7F&&[^=,]]|=2C|=3D)+";
+ private static final Pattern CLIENT_INITIAL_RESPONSE_PATTERN = Pattern.compile(
+ String.format("n,(a=(?<authzid>%s))?,auth=(?<scheme>[\\w]+)[ ]+(?<token>[-_\\.a-zA-Z0-9]+)", SASLNAME));
+
+ private final AuthenticateCallbackHandler callbackHandler;
+
+ private boolean complete;
+ private OAuthBearerToken tokenForNegotiatedProperty = null;
+ private String errorMessage = null;
+
+ public OAuthBearerSaslServer(CallbackHandler callbackHandler) {
+ if (!(Objects.requireNonNull(callbackHandler) instanceof AuthenticateCallbackHandler))
+ throw new IllegalArgumentException(String.format("Callback handler must be castable to %s: %s",
+ AuthenticateCallbackHandler.class.getName(), callbackHandler.getClass().getName()));
+ this.callbackHandler = (AuthenticateCallbackHandler) callbackHandler;
+ }
+
+ /**
+ * @throws SaslAuthenticationException
+ * if access token cannot be validated
+ * <p>
+ * <b>Note:</b> This method may throw
+ * {@link SaslAuthenticationException} to provide custom error
+ * messages to clients. But care should be taken to avoid including
+ * any information in the exception message that should not be
+ * leaked to unauthenticated clients. It may be safer to throw
+ * {@link SaslException} in some cases so that a standard error
+ * message is returned to clients.
+ * </p>
+ */
+ @Override
+ public byte[] evaluateResponse(byte[] response) throws SaslException, SaslAuthenticationException {
+ if (response.length == 1 && response[0] == OAuthBearerSaslClient.BYTE_CONTROL_A && errorMessage != null) {
+ if (log.isDebugEnabled())
+ log.debug("Received %x01 response from client after it received our error");
+ throw new SaslAuthenticationException(errorMessage);
+ }
+ errorMessage = null;
+ String responseMsg = new String(response, StandardCharsets.UTF_8);
+ Matcher matcher = CLIENT_INITIAL_RESPONSE_PATTERN.matcher(responseMsg);
+ if (!matcher.matches()) {
+ if (log.isDebugEnabled())
+ log.debug(INVALID_OAUTHBEARER_CLIENT_FIRST_MESSAGE);
+ throw new SaslException(INVALID_OAUTHBEARER_CLIENT_FIRST_MESSAGE);
+ }
+ String authzid = matcher.group("authzid");
+ String authorizationId = authzid != null ? authzid : "";
+ if (!"bearer".equalsIgnoreCase(matcher.group("scheme"))) {
+ String msg = String.format("Invalid scheme in OAUTHBEARER client first message: %s",
+ matcher.group("scheme"));
+ if (log.isDebugEnabled())
+ log.debug(msg);
+ throw new SaslException(msg);
+ }
+ String tokenValue = matcher.group("token");
+ return process(tokenValue, authorizationId);
+ }
+
+ @Override
+ public String getAuthorizationID() {
+ if (!complete)
+ throw new IllegalStateException("Authentication exchange has not completed");
+ return tokenForNegotiatedProperty.principalName();
+ }
+
+ @Override
+ public String getMechanismName() {
+ return OAuthBearerLoginModule.OAUTHBEARER_MECHANISM;
+ }
+
+ @Override
+ public Object getNegotiatedProperty(String propName) {
+ if (!complete)
+ throw new IllegalStateException("Authentication exchange has not completed");
+ return NEGOTIATED_PROPERTY_KEY_TOKEN.equals(propName) ? tokenForNegotiatedProperty : null;
+ }
+
+ @Override
+ public boolean isComplete() {
+ return complete;
+ }
+
+ @Override
+ public byte[] unwrap(byte[] incoming, int offset, int len) throws SaslException {
+ if (!complete)
+ throw new IllegalStateException("Authentication exchange has not completed");
+ return Arrays.copyOfRange(incoming, offset, offset + len);
+ }
+
+ @Override
+ public byte[] wrap(byte[] outgoing, int offset, int len) throws SaslException {
+ if (!complete)
+ throw new IllegalStateException("Authentication exchange has not completed");
+ return Arrays.copyOfRange(outgoing, offset, offset + len);
+ }
+
+ @Override
+ public void dispose() throws SaslException {
+ complete = false;
+ tokenForNegotiatedProperty = null;
+ }
+
+ private byte[] process(String tokenValue, String authorizationId) throws SaslException {
+ OAuthBearerValidatorCallback callback = new OAuthBearerValidatorCallback(tokenValue);
+ try {
+ callbackHandler.handle(new Callback[] {callback});
+ } catch (IOException | UnsupportedCallbackException e) {
+ String msg = String.format("%s: %s", INTERNAL_ERROR_ON_SERVER, e.getMessage());
+ if (log.isDebugEnabled())
+ log.debug(msg, e);
+ throw new SaslException(msg);
+ }
+ OAuthBearerToken token = callback.token();
+ if (token == null) {
+ errorMessage = jsonErrorResponse(callback.errorStatus(), callback.errorScope(),
+ callback.errorOpenIDConfiguration());
+ if (log.isDebugEnabled())
+ log.debug(errorMessage);
+ return errorMessage.getBytes(StandardCharsets.UTF_8);
+ }
+ /*
+ * We support the client specifying an authorization ID as per the SASL
+ * specification, but it must match the principal name if it is specified.
+ */
+ if (!authorizationId.isEmpty() && !authorizationId.equals(token.principalName()))
+ throw new SaslAuthenticationException(String.format(
+ "Authentication failed: Client requested an authorization id (%s) that is different from the token's principal name (%s)",
+ authorizationId, token.principalName()));
+ tokenForNegotiatedProperty = token;
+ complete = true;
+ if (log.isDebugEnabled())
+ log.debug("Successfully authenticate User={}", token.principalName());
+ return new byte[0];
+ }
+
+ private static String jsonErrorResponse(String errorStatus, String errorScope, String errorOpenIDConfiguration) {
+ String jsonErrorResponse = String.format("{\"status\":\"%s\"", errorStatus);
+ if (errorScope != null)
+ jsonErrorResponse = String.format("%s, \"scope\":\"%s\"", jsonErrorResponse, errorScope);
+ if (errorOpenIDConfiguration != null)
+ jsonErrorResponse = String.format("%s, \"openid-configuration\":\"%s\"", jsonErrorResponse,
+ errorOpenIDConfiguration);
+ jsonErrorResponse = String.format("%s}", jsonErrorResponse);
+ return jsonErrorResponse;
+ }
+
+ public static String[] mechanismNamesCompatibleWithPolicy(Map<String, ?> props) {
+ return props != null && "true".equals(String.valueOf(props.get(Sasl.POLICY_NOPLAINTEXT))) ? new String[] {}
+ : new String[] {OAuthBearerLoginModule.OAUTHBEARER_MECHANISM};
+ }
+
+ public static class OAuthBearerSaslServerFactory implements SaslServerFactory {
+ @Override
+ public SaslServer createSaslServer(String mechanism, String protocol, String serverName, Map<String, ?> props,
+ CallbackHandler callbackHandler) throws SaslException {
+ String[] mechanismNamesCompatibleWithPolicy = getMechanismNames(props);
+ for (int i = 0; i < mechanismNamesCompatibleWithPolicy.length; i++) {
+ if (mechanismNamesCompatibleWithPolicy[i].equals(mechanism)) {
+ return new OAuthBearerSaslServer(callbackHandler);
+ }
+ }
+ return null;
+ }
+
+ @Override
+ public String[] getMechanismNames(Map<String, ?> props) {
+ return OAuthBearerSaslServer.mechanismNamesCompatibleWithPolicy(props);
+ }
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/OAuthBearerSaslServerProvider.java b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/OAuthBearerSaslServerProvider.java
new file mode 100644
index 00000000000..cb6a94da27e
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/OAuthBearerSaslServerProvider.java
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.security.oauthbearer.internal;
+
+import java.security.Provider;
+import java.security.Security;
+
+import org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule;
+import org.apache.kafka.common.security.oauthbearer.internal.OAuthBearerSaslServer.OAuthBearerSaslServerFactory;
+
+public class OAuthBearerSaslServerProvider extends Provider {
+ private static final long serialVersionUID = 1L;
+
+ protected OAuthBearerSaslServerProvider() {
+ super("SASL/OAUTHBEARER Server Provider", 1.0, "SASL/OAUTHBEARER Server Provider for Kafka");
+ put("SaslServerFactory." + OAuthBearerLoginModule.OAUTHBEARER_MECHANISM,
+ OAuthBearerSaslServerFactory.class.getName());
+ }
+
+ public static void initialize() {
+ Security.addProvider(new OAuthBearerSaslServerProvider());
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/expiring/ExpiringCredential.java b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/expiring/ExpiringCredential.java
new file mode 100644
index 00000000000..472e980eb9d
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/expiring/ExpiringCredential.java
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.security.oauthbearer.internal.expiring;
+
+/**
+ * A credential that expires and that can potentially be refreshed
+ *
+ * @see ExpiringCredentialRefreshingLogin
+ */
+public interface ExpiringCredential {
+ /**
+ * The name of the principal to which this credential applies (used only for
+ * logging)
+ *
+ * @return the always non-null/non-empty principal name
+ */
+ String principalName();
+
+ /**
+ * When the credential became valid, in terms of the number of milliseconds
+ * since the epoch, if known, otherwise null. An expiring credential may not
+ * necessarily indicate when it was created -- just when it expires -- so we
+ * need to support a null return value here.
+ *
+ * @return the time when the credential became valid, in terms of the number of
+ * milliseconds since the epoch, if known, otherwise null
+ */
+ Long startTimeMs();
+
+ /**
+ * When the credential expires, in terms of the number of milliseconds since the
+ * epoch. All expiring credentials by definition must indicate their expiration
+ * time -- thus, unlike other methods, we do not support a null return value
+ * here.
+ *
+ * @return the time when the credential expires, in terms of the number of
+ * milliseconds since the epoch
+ */
+ long expireTimeMs();
+
+ /**
+ * The point after which the credential can no longer be refreshed, in terms of
+ * the number of milliseconds since the epoch, if any, otherwise null. Some
+ * expiring credentials can be refreshed over and over again without limit, so
+ * we support a null return value here.
+ *
+ * @return the point after which the credential can no longer be refreshed, in
+ * terms of the number of milliseconds since the epoch, if any,
+ * otherwise null
+ */
+ Long absoluteLastRefreshTimeMs();
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/expiring/ExpiringCredentialRefreshConfig.java b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/expiring/ExpiringCredentialRefreshConfig.java
new file mode 100644
index 00000000000..02c264b17de
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/expiring/ExpiringCredentialRefreshConfig.java
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.security.oauthbearer.internal.expiring;
+
+import java.util.Map;
+import java.util.Objects;
+
+import org.apache.kafka.common.config.SaslConfigs;
+
+/**
+ * Immutable refresh-related configuration for expiring credentials that can be
+ * parsed from a producer/consumer/broker config.
+ */
+public class ExpiringCredentialRefreshConfig {
+ private final double loginRefreshWindowFactor;
+ private final double loginRefreshWindowJitter;
+ private final short loginRefreshMinPeriodSeconds;
+ private final short loginRefreshBufferSeconds;
+ private final boolean loginRefreshReloginAllowedBeforeLogout;
+
+ /**
+ * Constructor based on producer/consumer/broker configs and the indicated value
+ * for whether or not client relogin is allowed before logout
+ *
+ * @param configs
+ * the mandatory (but possibly empty) producer/consumer/broker
+ * configs upon which to build this instance
+ * @param clientReloginAllowedBeforeLogout
+ * if the {@code LoginModule} and {@code SaslClient} implementations
+ * support multiple simultaneous login contexts on a single
+ * {@code Subject} at the same time. If true, then upon refresh,
+ * logout will only be invoked on the original {@code LoginContext}
+ * after a new one successfully logs in. This can be helpful if the
+ * original credential still has some lifetime left when an attempt
+ * to refresh the credential fails; the client will still be able to
+ * create new connections as long as the original credential remains
+ * valid. Otherwise, if logout is immediately invoked prior to
+ * relogin, a relogin failure leaves the client without the ability
+ * to connect until relogin does in fact succeed.
+ */
+ public ExpiringCredentialRefreshConfig(Map<String, ?> configs, boolean clientReloginAllowedBeforeLogout) {
+ Objects.requireNonNull(configs);
+ this.loginRefreshWindowFactor = (Double) configs.get(SaslConfigs.SASL_LOGIN_REFRESH_WINDOW_FACTOR);
+ this.loginRefreshWindowJitter = (Double) configs.get(SaslConfigs.SASL_LOGIN_REFRESH_WINDOW_JITTER);
+ this.loginRefreshMinPeriodSeconds = (Short) configs.get(SaslConfigs.SASL_LOGIN_REFRESH_MIN_PERIOD_SECONDS);
+ this.loginRefreshBufferSeconds = (Short) configs.get(SaslConfigs.SASL_LOGIN_REFRESH_BUFFER_SECONDS);
+ this.loginRefreshReloginAllowedBeforeLogout = clientReloginAllowedBeforeLogout;
+ }
+
+ /**
+ * Background login refresh thread will sleep until the specified window factor
+ * relative to the credential's total lifetime has been reached, at which time
+ * it will try to refresh the credential.
+ *
+ * @return the login refresh window factor
+ */
+ public double loginRefreshWindowFactor() {
+ return loginRefreshWindowFactor;
+ }
+
+ /**
+ * Amount of random jitter added to the background login refresh thread's sleep
+ * time.
+ *
+ * @return the login refresh window jitter
+ */
+ public double loginRefreshWindowJitter() {
+ return loginRefreshWindowJitter;
+ }
+
+ /**
+ * The desired minimum time between checks by the background login refresh
+ * thread, in seconds
+ *
+ * @return the desired minimum refresh period, in seconds
+ */
+ public short loginRefreshMinPeriodSeconds() {
+ return loginRefreshMinPeriodSeconds;
+ }
+
+ /**
+ * The amount of buffer time before expiration to maintain when refreshing. If a
+ * refresh is scheduled to occur closer to expiration than the number of seconds
+ * defined here then the refresh will be moved up to maintain as much of the
+ * desired buffer as possible.
+ *
+ * @return the refresh buffer, in seconds
+ */
+ public short loginRefreshBufferSeconds() {
+ return loginRefreshBufferSeconds;
+ }
+
+ /**
+ * If the LoginModule and SaslClient implementations support multiple
+ * simultaneous login contexts on a single Subject at the same time. If true,
+ * then upon refresh, logout will only be invoked on the original LoginContext
+ * after a new one successfully logs in. This can be helpful if the original
+ * credential still has some lifetime left when an attempt to refresh the
+ * credential fails; the client will still be able to create new connections as
+ * long as the original credential remains valid. Otherwise, if logout is
+ * immediately invoked prior to relogin, a relogin failure leaves the client
+ * without the ability to connect until relogin does in fact succeed.
+ *
+ * @return true if relogin is allowed prior to discarding an existing
+ * (presumably unexpired) credential, otherwise false
+ */
+ public boolean loginRefreshReloginAllowedBeforeLogout() {
+ return loginRefreshReloginAllowedBeforeLogout;
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/expiring/ExpiringCredentialRefreshingLogin.java b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/expiring/ExpiringCredentialRefreshingLogin.java
new file mode 100644
index 00000000000..5bd889e223a
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/expiring/ExpiringCredentialRefreshingLogin.java
@@ -0,0 +1,429 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.security.oauthbearer.internal.expiring;
+
+import java.util.Date;
+import java.util.Objects;
+import java.util.Random;
+
+import javax.security.auth.Subject;
+import javax.security.auth.login.Configuration;
+import javax.security.auth.login.LoginContext;
+import javax.security.auth.login.LoginException;
+
+import org.apache.kafka.common.security.auth.AuthenticateCallbackHandler;
+import org.apache.kafka.common.security.auth.Login;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.Time;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * This class is responsible for refreshing logins for both Kafka client and
+ * server when the login is a type that has a limited lifetime/will expire. The
+ * credentials for the login must implement {@link ExpiringCredential}.
+ */
+public abstract class ExpiringCredentialRefreshingLogin {
+ /**
+ * Class that can be overridden for testing
+ */
+ static class LoginContextFactory {
+ public LoginContext createLoginContext(ExpiringCredentialRefreshingLogin expiringCredentialRefreshingLogin)
+ throws LoginException {
+ return new LoginContext(expiringCredentialRefreshingLogin.contextName(),
+ expiringCredentialRefreshingLogin.subject(), expiringCredentialRefreshingLogin.callbackHandler(),
+ expiringCredentialRefreshingLogin.configuration());
+ }
+
+ public void refresherThreadStarted() {
+ // empty
+ }
+
+ public void refresherThreadDone() {
+ // empty
+ }
+ }
+
+ private static class ExitRefresherThreadDueToIllegalStateException extends Exception {
+ private static final long serialVersionUID = -6108495378411920380L;
+
+ public ExitRefresherThreadDueToIllegalStateException(String message) {
+ super(message);
+ }
+ }
+
+ private class Refresher implements Runnable {
+ @Override
+ public void run() {
+ log.info("[Principal={}]: Expiring credential re-login thread started.", principalLogText());
+ while (true) {
+ /*
+ * Refresh thread's main loop. Each expiring credential lives for one iteration
+ * of the loop. Thread will exit if the loop exits from here.
+ */
+ long nowMs = currentMs();
+ Long nextRefreshMs = refreshMs(nowMs);
+ if (nextRefreshMs == null) {
+ loginContextFactory.refresherThreadDone();
+ return;
+ }
+ log.info("[Principal={}]: Expiring credential re-login sleeping until: {}", principalLogText(),
+ new Date(nextRefreshMs));
+ time.sleep(nextRefreshMs - nowMs);
+ if (Thread.currentThread().isInterrupted()) {
+ log.info("[Principal={}]: Expiring credential re-login thread has been interrupted and will exit.",
+ principalLogText());
+ loginContextFactory.refresherThreadDone();
+ return;
+ }
+ while (true) {
+ /*
+ * Perform a re-login over and over again with some intervening delay
+ * unless/until either the refresh succeeds or we are interrupted.
+ */
+ try {
+ reLogin();
+ break; // success
+ } catch (ExitRefresherThreadDueToIllegalStateException e) {
+ log.error(e.getMessage(), e);
+ loginContextFactory.refresherThreadDone();
+ return;
+ } catch (LoginException loginException) {
+ log.warn(String.format(
+ "[Principal=%s]: LoginException during login retry; will sleep %d seconds before trying again.",
+ principalLogText(), DELAY_SECONDS_BEFORE_NEXT_RETRY_WHEN_RELOGIN_FAILS),
+ loginException);
+ // Sleep and allow loop to run/try again unless interrupted
+ time.sleep(DELAY_SECONDS_BEFORE_NEXT_RETRY_WHEN_RELOGIN_FAILS * 1000);
+ if (Thread.currentThread().isInterrupted()) {
+ log.error(
+ "[Principal={}]: Interrupted while trying to perform a subsequent expiring credential re-login after one or more initial re-login failures: re-login thread exiting now: {}",
+ principalLogText(), String.valueOf(loginException.getMessage()));
+ loginContextFactory.refresherThreadDone();
+ return;
+ }
+ }
+ }
+ }
+ }
+ }
+
+ private static final Logger log = LoggerFactory.getLogger(ExpiringCredentialRefreshingLogin.class);
+ private static final long DELAY_SECONDS_BEFORE_NEXT_RETRY_WHEN_RELOGIN_FAILS = 10L;
+ private static final Random RNG = new Random();
+ private final Time time;
+ private Thread refresherThread;
+
+ private final LoginContextFactory loginContextFactory;
+ private final String contextName;
+ private final Configuration configuration;
+ private final ExpiringCredentialRefreshConfig expiringCredentialRefreshConfig;
+ private final AuthenticateCallbackHandler callbackHandler;
+
+ // mark volatile due to existence of public subject() method
+ private volatile Subject subject = null;
+ private boolean hasExpiringCredential = false;
+ private String principalName = null;
+ private LoginContext loginContext = null;
+ private ExpiringCredential expiringCredential = null;
+ private final Class<?> mandatoryClassToSynchronizeOnPriorToRefresh;
+
+ public ExpiringCredentialRefreshingLogin(String contextName, Configuration configuration,
+ ExpiringCredentialRefreshConfig expiringCredentialRefreshConfig,
+ AuthenticateCallbackHandler callbackHandler, Class<?> mandatoryClassToSynchronizeOnPriorToRefresh) {
+ this(contextName, configuration, expiringCredentialRefreshConfig, callbackHandler,
+ mandatoryClassToSynchronizeOnPriorToRefresh, new LoginContextFactory(), Time.SYSTEM);
+ }
+
+ public ExpiringCredentialRefreshingLogin(String contextName, Configuration configuration,
+ ExpiringCredentialRefreshConfig expiringCredentialRefreshConfig,
+ AuthenticateCallbackHandler callbackHandler, Class<?> mandatoryClassToSynchronizeOnPriorToRefresh,
+ LoginContextFactory loginContextFactory, Time time) {
+ this.contextName = Objects.requireNonNull(contextName);
+ this.configuration = Objects.requireNonNull(configuration);
+ this.expiringCredentialRefreshConfig = Objects.requireNonNull(expiringCredentialRefreshConfig);
+ this.callbackHandler = callbackHandler;
+ this.mandatoryClassToSynchronizeOnPriorToRefresh = Objects
+ .requireNonNull(mandatoryClassToSynchronizeOnPriorToRefresh);
+ this.loginContextFactory = loginContextFactory;
+ this.time = Objects.requireNonNull(time);
+ }
+
+ public Subject subject() {
+ return subject; // field requires volatile keyword
+ }
+
+ public String contextName() {
+ return contextName;
+ }
+
+ public Configuration configuration() {
+ return configuration;
+ }
+
+ public AuthenticateCallbackHandler callbackHandler() {
+ return callbackHandler;
+ }
+
+ public String serviceName() {
+ return "kafka";
+ }
+
+ /**
+ * Performs login for each login module specified for the login context of this
+ * instance and starts the thread used to periodically re-login.
+ * <p>
+ * The synchronized keyword is not necessary because an implementation of
+ * {@link Login} will delegate to this code (e.g. OAuthBearerRefreshingLogin},
+ * and the {@code login()} method on the delegating class will itself be
+ * synchronized if necessary.
+ */
+ public LoginContext login() throws LoginException {
+ LoginContext tmpLoginContext = loginContextFactory.createLoginContext(this);
+ tmpLoginContext.login();
+ log.info("Successfully logged in.");
+ loginContext = tmpLoginContext;
+ subject = loginContext.getSubject();
+ expiringCredential = expiringCredential();
+ hasExpiringCredential = expiringCredential != null;
+ if (!hasExpiringCredential) {
+ // do not bother with re-logins.
+ log.debug("No Expiring Credential");
+ principalName = null;
+ refresherThread = null;
+ return loginContext;
+ }
+
+ principalName = expiringCredential.principalName();
+
+ // Check for a clock skew problem
+ long expireTimeMs = expiringCredential.expireTimeMs();
+ long nowMs = currentMs();
+ if (nowMs > expireTimeMs) {
+ log.error(
+ "[Principal={}]: Current clock: {} is later than expiry {}. This may indicate a clock skew problem."
+ + " Check that this host's and remote host's clocks are in sync. Not starting refresh thread."
+ + " This process is likely unable to authenticate SASL connections (for example, it is unlikely"
+ + " to be able to authenticate a connection with a Kafka Broker).",
+ principalLogText(), new Date(nowMs), new Date(expireTimeMs));
+ return loginContext;
+ }
+
+ if (log.isDebugEnabled())
+ log.debug("[Principal={}]: It is an expiring credential", principalLogText());
+
+ /*
+ * Re-login periodically. How often is determined by the expiration date of the
+ * credential and refresh-related configuration values.
+ */
+ refresherThread = KafkaThread.daemon(String.format("kafka-expiring-relogin-thread-%s", principalName),
+ new Refresher());
+ refresherThread.start();
+ loginContextFactory.refresherThreadStarted();
+ return loginContext;
+ }
+
+ public void close() {
+ if (refresherThread != null && refresherThread.isAlive()) {
+ refresherThread.interrupt();
+ try {
+ refresherThread.join();
+ } catch (InterruptedException e) {
+ log.warn("[Principal={}]: Interrupted while waiting for re-login thread to shutdown.",
+ principalLogText(), e);
+ Thread.currentThread().interrupt();
+ }
+ }
+ }
+
+ public abstract ExpiringCredential expiringCredential();
+
+ /**
+ * Determine when to sleep until before performing a refresh
+ *
+ * @param relativeToMs
+ * the point (in terms of number of milliseconds since the epoch) at
+ * which to perform the calculation
+ * @return null if no refresh should occur, otherwise the time to sleep until
+ * (in terms of the number of milliseconds since the epoch) before
+ * performing a refresh
+ */
+ private Long refreshMs(long relativeToMs) {
+ if (expiringCredential == null) {
+ /*
+ * Re-login failed because our login() invocation did not generate a credential
+ * but also did not generate an exception. Try logging in again after some delay
+ * (it seems likely to be a bug, but it doesn't hurt to keep trying to refresh).
+ */
+ long retvalNextRefreshMs = relativeToMs + DELAY_SECONDS_BEFORE_NEXT_RETRY_WHEN_RELOGIN_FAILS * 1000L;
+ log.warn("[Principal={}]: No Expiring credential found: will try again at {}", principalLogText(),
+ new Date(retvalNextRefreshMs));
+ return retvalNextRefreshMs;
+ }
+ long expireTimeMs = expiringCredential.expireTimeMs();
+ if (relativeToMs > expireTimeMs) {
+ boolean logoutRequiredBeforeLoggingBackIn = isLogoutRequiredBeforeLoggingBackIn();
+ if (logoutRequiredBeforeLoggingBackIn) {
+ log.error(
+ "[Principal={}]: Current clock: {} is later than expiry {}. This may indicate a clock skew problem."
+ + " Check that this host's and remote host's clocks are in sync. Exiting refresh thread.",
+ principalLogText(), new Date(relativeToMs), new Date(expireTimeMs));
+ return null;
+ } else {
+ /*
+ * Since the current soon-to-expire credential isn't logged out until we have a
+ * new credential with a refreshed lifetime, it is possible that the current
+ * credential could expire if the re-login continually fails over and over again
+ * making us unable to get the new credential. Therefore keep trying rather than
+ * exiting.
+ */
+ long retvalNextRefreshMs = relativeToMs + DELAY_SECONDS_BEFORE_NEXT_RETRY_WHEN_RELOGIN_FAILS * 1000L;
+ log.warn("[Principal={}]: Expiring credential already expired at {}: will try to refresh again at {}",
+ principalLogText(), new Date(expireTimeMs), new Date(retvalNextRefreshMs));
+ return retvalNextRefreshMs;
+ }
+ }
+ Long absoluteLastRefreshTimeMs = expiringCredential.absoluteLastRefreshTimeMs();
+ if (absoluteLastRefreshTimeMs != null && absoluteLastRefreshTimeMs.longValue() < expireTimeMs) {
+ log.warn("[Principal={}]: Expiring credential refresh thread exiting because the"
+ + " expiring credential's current expiration time ({}) exceeds the latest possible refresh time ({})."
+ + " This process will not be able to authenticate new SASL connections after that"
+ + " time (for example, it will not be able to authenticate a new connection with a Kafka Broker).",
+ principalLogText(), new Date(expireTimeMs), new Date(absoluteLastRefreshTimeMs.longValue()));
+ return null;
+ }
+ Long optionalStartTime = expiringCredential.startTimeMs();
+ long startMs = optionalStartTime != null ? optionalStartTime.longValue() : currentMs();
+ log.info("[Principal={}]: Expiring credential valid from {} to {}", expiringCredential.principalName(),
+ new java.util.Date(startMs), new java.util.Date(expireTimeMs));
+
+ double pct = expiringCredentialRefreshConfig.loginRefreshWindowFactor()
+ + (expiringCredentialRefreshConfig.loginRefreshWindowJitter() * RNG.nextDouble());
+ /*
+ * Ignore buffer times if the credential's remaining lifetime is less than their
+ * sum.
+ */
+ long refreshMinPeriodSeconds = expiringCredentialRefreshConfig.loginRefreshMinPeriodSeconds();
+ long clientRefreshBufferSeconds = expiringCredentialRefreshConfig.loginRefreshBufferSeconds();
+ if (relativeToMs + 1000L * (refreshMinPeriodSeconds + clientRefreshBufferSeconds) > expireTimeMs) {
+ long retvalRefreshMs = startMs + (long) ((expireTimeMs - startMs) * pct);
+ log.warn(
+ "[Principal={}]: Expiring credential expires at {}, so buffer times of {} and {} seconds"
+ + " at the front and back, respectively, cannot be accommodated. We will refresh at {}.",
+ principalLogText(), new Date(expireTimeMs), refreshMinPeriodSeconds, clientRefreshBufferSeconds,
+ new Date(retvalRefreshMs));
+ return retvalRefreshMs;
+ }
+ long proposedRefreshMs = startMs + (long) ((expireTimeMs - startMs) * pct);
+ // Don't let it violate the requested end buffer time
+ long beginningOfEndBufferTimeMs = expireTimeMs - clientRefreshBufferSeconds * 1000;
+ if (proposedRefreshMs > beginningOfEndBufferTimeMs) {
+ log.info(
+ "[Principal={}]: Proposed refresh time of {} extends into the desired buffer time of {} seconds before expiration, so refresh it at the desired buffer begin point, at {}",
+ expiringCredential.principalName(), new Date(proposedRefreshMs), clientRefreshBufferSeconds,
+ new Date(beginningOfEndBufferTimeMs));
+ return beginningOfEndBufferTimeMs;
+ }
+ // Don't let it violate the minimum refresh period
+ long endOfMinRefreshBufferTime = relativeToMs + 1000 * refreshMinPeriodSeconds;
+ if (proposedRefreshMs < endOfMinRefreshBufferTime) {
+ log.info(
+ "[Principal={}]: Expiring credential re-login thread time adjusted from {} to {} since the former is sooner "
+ + "than the minimum refresh interval ({} seconds from now).",
+ principalLogText(), new Date(proposedRefreshMs), new Date(endOfMinRefreshBufferTime),
+ refreshMinPeriodSeconds);
+ return endOfMinRefreshBufferTime;
+ }
+ // Proposed refresh time doesn't violate any constraints
+ return proposedRefreshMs;
+ }
+
+ private void reLogin() throws LoginException, ExitRefresherThreadDueToIllegalStateException {
+ synchronized (mandatoryClassToSynchronizeOnPriorToRefresh) {
+ // Only perform one refresh of a particular type at a time
+ boolean logoutRequiredBeforeLoggingBackIn = isLogoutRequiredBeforeLoggingBackIn();
+ if (hasExpiringCredential && logoutRequiredBeforeLoggingBackIn) {
+ String principalLogTextPriorToLogout = principalLogText();
+ log.info("Initiating logout for {}", principalLogTextPriorToLogout);
+ loginContext.logout();
+ // Make absolutely sure we were logged out
+ expiringCredential = expiringCredential();
+ hasExpiringCredential = expiringCredential != null;
+ if (hasExpiringCredential)
+ // We can't force the removal because we don't know how to do it, so abort
+ throw new ExitRefresherThreadDueToIllegalStateException(String.format(
+ "Subject's private credentials still contains an instance of %s even though logout() was invoked; exiting refresh thread",
+ expiringCredential.getClass().getName()));
+ }
+ /*
+ * Perform a login, making note of any credential that might need a logout()
+ * afterwards
+ */
+ ExpiringCredential optionalCredentialToLogout = expiringCredential;
+ LoginContext optionalLoginContextToLogout = loginContext;
+ loginContext = loginContextFactory.createLoginContext(ExpiringCredentialRefreshingLogin.this);
+ log.info("Initiating re-login for {}, logout() still needs to be called on a previous login = {}",
+ principalName, optionalCredentialToLogout != null);
+ loginContext.login();
+ // Perform a logout() on any original credential if necessary
+ if (optionalCredentialToLogout != null)
+ optionalLoginContextToLogout.logout();
+ /*
+ * Get the new credential and make sure it is not any old one that required a
+ * logout() after the login()
+ */
+ expiringCredential = expiringCredential();
+ hasExpiringCredential = expiringCredential != null;
+ if (!hasExpiringCredential) {
+ /*
+ * Re-login has failed because our login() invocation has not generated a
+ * credential but has also not generated an exception. We won't exit here;
+ * instead we will allow login retries in case we can somehow fix the issue (it
+ * seems likely to be a bug, but it doesn't hurt to keep trying to refresh).
+ */
+ log.error("No Expiring Credential after a supposedly-successful re-login");
+ principalName = null;
+ } else {
+ if (expiringCredential == optionalCredentialToLogout)
+ /*
+ * The login() didn't identify a new credential; we still have the old one. We
+ * don't know how to fix this, so abort.
+ */
+ throw new ExitRefresherThreadDueToIllegalStateException(String.format(
+ "Subject's private credentials still contains the previous, soon-to-expire instance of %s even though login() followed by logout() was invoked; exiting refresh thread",
+ expiringCredential.getClass().getName()));
+ principalName = expiringCredential.principalName();
+ if (log.isDebugEnabled())
+ log.debug("[Principal={}]: It is an expiring credential after re-login as expected",
+ principalLogText());
+ }
+ }
+ }
+
+ private String principalLogText() {
+ return expiringCredential == null ? principalName
+ : expiringCredential.getClass().getSimpleName() + ":" + principalName;
+ }
+
+ private long currentMs() {
+ return time.milliseconds();
+ }
+
+ private boolean isLogoutRequiredBeforeLoggingBackIn() {
+ return !expiringCredentialRefreshConfig.loginRefreshReloginAllowedBeforeLogout();
+ }
+}
diff --git a/tools/src/main/java/org/apache/kafka/trogdor/rest/CreateTaskResponse.java b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/unsecured/OAuthBearerConfigException.java
similarity index 60%
rename from tools/src/main/java/org/apache/kafka/trogdor/rest/CreateTaskResponse.java
rename to clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/unsecured/OAuthBearerConfigException.java
index 54ea0f23c97..8fde861bb29 100644
--- a/tools/src/main/java/org/apache/kafka/trogdor/rest/CreateTaskResponse.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/unsecured/OAuthBearerConfigException.java
@@ -14,26 +14,22 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
+package org.apache.kafka.common.security.oauthbearer.internal.unsecured;
-package org.apache.kafka.trogdor.rest;
-
-import com.fasterxml.jackson.annotation.JsonCreator;
-import com.fasterxml.jackson.annotation.JsonProperty;
-import org.apache.kafka.trogdor.task.TaskSpec;
+import org.apache.kafka.common.KafkaException;
/**
- * A response from the Trogdor coordinator about creating a task.
+ * Exception thrown when there is a problem with the configuration (an invalid
+ * option in a JAAS config, for example).
*/
-public class CreateTaskResponse extends Message {
- private final TaskSpec spec;
+public class OAuthBearerConfigException extends KafkaException {
+ private static final long serialVersionUID = -8056105648062343518L;
- @JsonCreator
- public CreateTaskResponse(@JsonProperty("spec") TaskSpec spec) {
- this.spec = spec;
+ public OAuthBearerConfigException(String s) {
+ super(s);
}
- @JsonProperty
- public TaskSpec spec() {
- return spec;
+ public OAuthBearerConfigException(String message, Throwable cause) {
+ super(message, cause);
}
}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/unsecured/OAuthBearerIllegalTokenException.java b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/unsecured/OAuthBearerIllegalTokenException.java
new file mode 100644
index 00000000000..4a09bc94ab4
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/unsecured/OAuthBearerIllegalTokenException.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.security.oauthbearer.internal.unsecured;
+
+import java.util.Objects;
+
+import org.apache.kafka.common.KafkaException;
+
+/**
+ * Exception thrown when token validation fails due to a problem with the token
+ * itself (as opposed to a missing remote resource or a configuration problem)
+ */
+public class OAuthBearerIllegalTokenException extends KafkaException {
+ private static final long serialVersionUID = -5275276640051316350L;
+ private final OAuthBearerValidationResult reason;
+
+ /**
+ * Constructor
+ *
+ * @param reason
+ * the mandatory reason for the validation failure; it must indicate
+ * failure
+ */
+ public OAuthBearerIllegalTokenException(OAuthBearerValidationResult reason) {
+ super(Objects.requireNonNull(reason).failureDescription());
+ if (reason.success())
+ throw new IllegalArgumentException("The reason indicates success; it must instead indicate failure");
+ this.reason = reason;
+ }
+
+ /**
+ * Return the (always non-null) reason for the validation failure
+ *
+ * @return the reason for the validation failure
+ */
+ public OAuthBearerValidationResult reason() {
+ return reason;
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/unsecured/OAuthBearerScopeUtils.java b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/unsecured/OAuthBearerScopeUtils.java
new file mode 100644
index 00000000000..40a5f0b931a
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/unsecured/OAuthBearerScopeUtils.java
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.security.oauthbearer.internal.unsecured;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.regex.Pattern;
+
+/**
+ * Utility class for help dealing with
+ * <a href="https://tools.ietf.org/html/rfc6749#section-3.3">Access Token
+ * Scopes</a>
+ */
+public class OAuthBearerScopeUtils {
+ private static final Pattern INDIVIDUAL_SCOPE_ITEM_PATTERN = Pattern.compile("[\\x23-\\x5B\\x5D-\\x7E\\x21]+");
+
+ /**
+ * Return true if the given value meets the definition of a valid scope item as
+ * per <a href="https://tools.ietf.org/html/rfc6749#section-3.3">RFC 6749
+ * Section 3.3</a>, otherwise false
+ *
+ * @param scopeItem
+ * the mandatory scope item to check for validity
+ * @return true if the given value meets the definition of a valid scope item,
+ * otherwise false
+ */
+ public static boolean isValidScopeItem(String scopeItem) {
+ return INDIVIDUAL_SCOPE_ITEM_PATTERN.matcher(Objects.requireNonNull(scopeItem)).matches();
+ }
+
+ /**
+ * Convert a space-delimited list of scope values (for example,
+ * <code>"scope1 scope2"</code>) to a List containing the individual elements
+ * (<code>"scope1"</code> and <code>"scope2"</code>)
+ *
+ * @param spaceDelimitedScope
+ * the mandatory (but possibly empty) space-delimited scope values,
+ * each of which must be valid according to
+ * {@link #isValidScopeItem(String)}
+ * @return the list of the given (possibly empty) space-delimited values
+ * @throws OAuthBearerConfigException
+ * if any of the individual scope values are malformed/illegal
+ */
+ public static List<String> parseScope(String spaceDelimitedScope) throws OAuthBearerConfigException {
+ List<String> retval = new ArrayList<>();
+ for (String individualScopeItem : Objects.requireNonNull(spaceDelimitedScope).split(" ")) {
+ if (!individualScopeItem.isEmpty()) {
+ if (!isValidScopeItem(individualScopeItem))
+ throw new OAuthBearerConfigException(String.format("Invalid scope value: %s", individualScopeItem));
+ retval.add(individualScopeItem);
+ }
+ }
+ return Collections.unmodifiableList(retval);
+ }
+
+ private OAuthBearerScopeUtils() {
+ // empty
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/unsecured/OAuthBearerUnsecuredJws.java b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/unsecured/OAuthBearerUnsecuredJws.java
new file mode 100644
index 00000000000..aedde8afe81
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/unsecured/OAuthBearerUnsecuredJws.java
@@ -0,0 +1,371 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.security.oauthbearer.internal.unsecured;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Set;
+
+import org.apache.kafka.common.security.oauthbearer.OAuthBearerToken;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.JsonNodeType;
+import com.fasterxml.jackson.databind.node.NumericNode;
+
+/**
+ * A simple unsecured JWS implementation. The '{@code nbf}' claim is ignored if
+ * it is given because the related logic is not required for Kafka testing and
+ * development purposes.
+ *
+ * @see <a href="https://tools.ietf.org/html/rfc7515">RFC 7515</a>
+ */
+public class OAuthBearerUnsecuredJws implements OAuthBearerToken {
+ private final String compactSerialization;
+ private final List<String> splits;
+ private final Map<String, Object> header;
+ private final String principalClaimName;
+ private final String scopeClaimName;
+ private final Map<String, Object> claims;
+ private final Set<String> scope;
+ private final long lifetime;
+ private final String principalName;
+ private final Long startTimeMs;
+
+ /**
+ * Constructor with the given principal and scope claim names
+ *
+ * @param compactSerialization
+ * the compact serialization to parse as an unsecured JWS
+ * @param principalClaimName
+ * the required principal claim name
+ * @param scopeClaimName
+ * the required scope claim name
+ * @throws OAuthBearerIllegalTokenException
+ * if the compact serialization is not a valid unsecured JWS
+ * (meaning it did not have 3 dot-separated Base64URL sections
+ * without an empty digital signature; or the header or claims
+ * either are not valid Base 64 URL encoded values or are not JSON
+ * after decoding; or the mandatory '{@code alg}' header value is
+ * not "{@code none}")
+ */
+ public OAuthBearerUnsecuredJws(String compactSerialization, String principalClaimName, String scopeClaimName)
+ throws OAuthBearerIllegalTokenException {
+ this.compactSerialization = Objects.requireNonNull(compactSerialization);
+ if (compactSerialization.contains(".."))
+ throw new OAuthBearerIllegalTokenException(
+ OAuthBearerValidationResult.newFailure("Malformed compact serialization contains '..'"));
+ this.splits = extractCompactSerializationSplits();
+ this.header = toMap(splits().get(0));
+ String claimsSplit = splits.get(1);
+ this.claims = toMap(claimsSplit);
+ String alg = Objects.requireNonNull(header().get("alg"), "JWS header must have an Algorithm value").toString();
+ if (!"none".equals(alg))
+ throw new OAuthBearerIllegalTokenException(
+ OAuthBearerValidationResult.newFailure("Unsecured JWS must have 'none' for an algorithm"));
+ String digitalSignatureSplit = splits.get(2);
+ if (!digitalSignatureSplit.isEmpty())
+ throw new OAuthBearerIllegalTokenException(
+ OAuthBearerValidationResult.newFailure("Unsecured JWS must not contain a digital signature"));
+ this.principalClaimName = Objects.requireNonNull(principalClaimName).trim();
+ if (this.principalClaimName.isEmpty())
+ throw new IllegalArgumentException("Must specify a non-blank principal claim name");
+ this.scopeClaimName = Objects.requireNonNull(scopeClaimName).trim();
+ if (this.scopeClaimName.isEmpty())
+ throw new IllegalArgumentException("Must specify a non-blank scope claim name");
+ this.scope = calculateScope();
+ Number expirationTimeSeconds = expirationTime();
+ if (expirationTimeSeconds == null)
+ throw new OAuthBearerIllegalTokenException(
+ OAuthBearerValidationResult.newFailure("No expiration time in JWT"));
+ lifetime = convertClaimTimeInSecondsToMs(expirationTimeSeconds);
+ String principalName = claim(this.principalClaimName, String.class);
+ if (principalName == null || principalName.trim().isEmpty())
+ throw new OAuthBearerIllegalTokenException(OAuthBearerValidationResult
+ .newFailure("No principal name in JWT claim: " + this.principalClaimName));
+ this.principalName = principalName;
+ this.startTimeMs = calculateStartTimeMs();
+ }
+
+ @Override
+ public String value() {
+ return compactSerialization;
+ }
+
+ /**
+ * Return the 3 or 5 dot-separated sections of the JWT compact serialization
+ *
+ * @return the 3 or 5 dot-separated sections of the JWT compact serialization
+ */
+ public List<String> splits() {
+ return splits;
+ }
+
+ /**
+ * Return the JOSE Header as a {@code Map}
+ *
+ * @return the JOSE header
+ */
+ public Map<String, Object> header() {
+ return header;
+ }
+
+ @Override
+ public String principalName() {
+ return principalName;
+ }
+
+ @Override
+ public Long startTimeMs() {
+ return startTimeMs;
+ }
+
+ @Override
+ public long lifetimeMs() {
+ return lifetime;
+ }
+
+ @Override
+ public Set<String> scope() throws OAuthBearerIllegalTokenException {
+ return scope;
+ }
+
+ /**
+ * Return the JWT Claim Set as a {@code Map}
+ *
+ * @return the (always non-null but possibly empty) claims
+ */
+ public Map<String, Object> claims() {
+ return claims;
+ }
+
+ /**
+ * Return the (always non-null/non-empty) principal claim name
+ *
+ * @return the (always non-null/non-empty) principal claim name
+ */
+ public String principalClaimName() {
+ return principalClaimName;
+ }
+
+ /**
+ * Return the (always non-null/non-empty) scope claim name
+ *
+ * @return the (always non-null/non-empty) scope claim name
+ */
+ public String scopeClaimName() {
+ return scopeClaimName;
+ }
+
+ /**
+ * Indicate if the claim exists and is the given type
+ *
+ * @param claimName
+ * the mandatory JWT claim name
+ * @param type
+ * the mandatory type, which should either be String.class,
+ * Number.class, or List.class
+ * @return true if the claim exists and is the given type, otherwise false
+ */
+ public boolean isClaimType(String claimName, Class<?> type) {
+ Object value = rawClaim(claimName);
+ Objects.requireNonNull(type);
+ if (value == null)
+ return false;
+ if (type == String.class && value instanceof String)
+ return true;
+ if (type == Number.class && value instanceof Number)
+ return true;
+ return type == List.class && value instanceof List;
+ }
+
+ /**
+ * Extract a claim of the given type
+ *
+ * @param claimName
+ * the mandatory JWT claim name
+ * @param type
+ * the mandatory type, which must either be String.class,
+ * Number.class, or List.class
+ * @return the claim if it exists, otherwise null
+ * @throws OAuthBearerIllegalTokenException
+ * if the claim exists but is not the given type
+ */
+ public <T> T claim(String claimName, Class<T> type) throws OAuthBearerIllegalTokenException {
+ Object value = rawClaim(claimName);
+ try {
+ return Objects.requireNonNull(type).cast(value);
+ } catch (ClassCastException e) {
+ throw new OAuthBearerIllegalTokenException(
+ OAuthBearerValidationResult.newFailure(String.format("The '%s' claim was not of type %s: %s",
+ claimName, type.getSimpleName(), value.getClass().getSimpleName())));
+ }
+ }
+
+ /**
+ * Extract a claim in its raw form
+ *
+ * @param claimName
+ * the mandatory JWT claim name
+ * @return the raw claim value, if it exists, otherwise null
+ */
+ public Object rawClaim(String claimName) {
+ return claims().get(Objects.requireNonNull(claimName));
+ }
+
+ /**
+ * Return the
+ * <a href="https://tools.ietf.org/html/rfc7519#section-4.1.4">Expiration
+ * Time</a> claim
+ *
+ * @return the <a href=
+ * "https://tools.ietf.org/html/rfc7519#section-4.1.4">Expiration
+ * Time</a> claim if available, otherwise null
+ * @throws OAuthBearerIllegalTokenException
+ * if the claim value is the incorrect type
+ */
+ public Number expirationTime() throws OAuthBearerIllegalTokenException {
+ return claim("exp", Number.class);
+ }
+
+ /**
+ * Return the <a href="https://tools.ietf.org/html/rfc7519#section-4.1.6">Issued
+ * At</a> claim
+ *
+ * @return the
+ * <a href= "https://tools.ietf.org/html/rfc7519#section-4.1.6">Issued
+ * At</a> claim if available, otherwise null
+ * @throws OAuthBearerIllegalTokenException
+ * if the claim value is the incorrect type
+ */
+ public Number issuedAt() throws OAuthBearerIllegalTokenException {
+ return claim("iat", Number.class);
+ }
+
+ /**
+ * Return the
+ * <a href="https://tools.ietf.org/html/rfc7519#section-4.1.2">Subject</a> claim
+ *
+ * @return the <a href=
+ * "https://tools.ietf.org/html/rfc7519#section-4.1.2">Subject</a> claim
+ * if available, otherwise null
+ * @throws OAuthBearerIllegalTokenException
+ * if the claim value is the incorrect type
+ */
+ public String subject() throws OAuthBearerIllegalTokenException {
+ return claim("sub", String.class);
+ }
+
+ /**
+ * Decode the given Base64URL-encoded value, parse the resulting JSON as a JSON
+ * object, and return the map of member names to their values (each value being
+ * represented as either a String, a Number, or a List of Strings).
+ *
+ * @param split
+ * the value to decode and parse
+ * @return the map of JSON member names to their String, Number, or String List
+ * value
+ * @throws OAuthBearerIllegalTokenException
+ * if the given Base64URL-encoded value cannot be decoded or parsed
+ */
+ public static Map<String, Object> toMap(String split) throws OAuthBearerIllegalTokenException {
+ Map<String, Object> retval = new HashMap<>();
+ try {
+ byte[] decode = Base64.getDecoder().decode(split);
+ JsonNode jsonNode = new ObjectMapper().readTree(decode);
+ if (jsonNode == null)
+ throw new OAuthBearerIllegalTokenException(OAuthBearerValidationResult.newFailure("malformed JSON"));
+ for (Iterator<Entry<String, JsonNode>> iterator = jsonNode.fields(); iterator.hasNext();) {
+ Entry<String, JsonNode> entry = iterator.next();
+ retval.put(entry.getKey(), convert(entry.getValue()));
+ }
+ return Collections.unmodifiableMap(retval);
+ } catch (IllegalArgumentException e) {
+ // potentially thrown by java.util.Base64.Decoder implementations
+ throw new OAuthBearerIllegalTokenException(
+ OAuthBearerValidationResult.newFailure("malformed Base64 URL encoded value"));
+ } catch (IOException e) {
+ throw new OAuthBearerIllegalTokenException(OAuthBearerValidationResult.newFailure("malformed JSON"));
+ }
+ }
+
+ private List<String> extractCompactSerializationSplits() {
+ List<String> tmpSplits = new ArrayList<>(Arrays.asList(compactSerialization.split("\\.")));
+ if (compactSerialization.endsWith("."))
+ tmpSplits.add("");
+ if (tmpSplits.size() != 3)
+ throw new OAuthBearerIllegalTokenException(OAuthBearerValidationResult.newFailure(
+ "Unsecured JWS compact serializations must have 3 dot-separated Base64URL-encoded values"));
+ return Collections.unmodifiableList(tmpSplits);
+ }
+
+ private static Object convert(JsonNode value) {
+ if (value.isArray()) {
+ List<String> retvalList = new ArrayList<>();
+ for (JsonNode arrayElement : value) {
+ retvalList.add(arrayElement.asText());
+ }
+ return retvalList;
+ }
+ return value.getNodeType() == JsonNodeType.NUMBER ? ((NumericNode) value).numberValue() : value.asText();
+ }
+
+ private Long calculateStartTimeMs() throws OAuthBearerIllegalTokenException {
+ Number issuedAtSeconds = claim("iat", Number.class);
+ return issuedAtSeconds == null ? null : convertClaimTimeInSecondsToMs(issuedAtSeconds);
+ }
+
+ private static long convertClaimTimeInSecondsToMs(Number claimValue) {
+ return Math.round(claimValue.doubleValue() * 1000);
+ }
+
+ private Set<String> calculateScope() {
+ String scopeClaimName = scopeClaimName();
+ if (isClaimType(scopeClaimName, String.class)) {
+ String scopeClaimValue = claim(scopeClaimName, String.class);
+ if (scopeClaimValue.trim().isEmpty())
+ return Collections.emptySet();
+ else {
+ Set<String> retval = new HashSet<>();
+ retval.add(scopeClaimValue.trim());
+ return Collections.unmodifiableSet(retval);
+ }
+ }
+ List<?> scopeClaimValue = claim(scopeClaimName, List.class);
+ if (scopeClaimValue == null || scopeClaimValue.isEmpty())
+ return Collections.emptySet();
+ @SuppressWarnings("unchecked")
+ List<String> stringList = (List<String>) scopeClaimValue;
+ Set<String> retval = new HashSet<>();
+ for (String scope : stringList) {
+ if (scope != null && !scope.trim().isEmpty()) {
+ retval.add(scope.trim());
+ }
+ }
+ return Collections.unmodifiableSet(retval);
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/unsecured/OAuthBearerUnsecuredLoginCallbackHandler.java b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/unsecured/OAuthBearerUnsecuredLoginCallbackHandler.java
new file mode 100644
index 00000000000..183f587573d
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/unsecured/OAuthBearerUnsecuredLoginCallbackHandler.java
@@ -0,0 +1,288 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.security.oauthbearer.internal.unsecured;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.Base64.Encoder;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+
+import javax.security.auth.callback.Callback;
+import javax.security.auth.callback.UnsupportedCallbackException;
+import javax.security.auth.login.AppConfigurationEntry;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.security.auth.AuthenticateCallbackHandler;
+import org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule;
+import org.apache.kafka.common.security.oauthbearer.OAuthBearerTokenCallback;
+import org.apache.kafka.common.utils.Time;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@code CallbackHandler} that recognizes {@link OAuthBearerTokenCallback}
+ * and returns an unsecured OAuth 2 bearer token.
+ * <p>
+ * Claims and their values on the returned token can be specified using
+ * {@code unsecuredLoginStringClaim_<claimname>},
+ * {@code unsecuredLoginNumberClaim_<claimname>}, and
+ * {@code unsecuredLoginListClaim_<claimname>} options. The first character of
+ * the value is taken as the delimiter for list claims. You may define any claim
+ * name and value except '{@code iat}' and '{@code exp}', both of which are
+ * calculated automatically.
+ * <p>
+ * This implementation also accepts the following options:
+ * <ul>
+ * <li>{@code unsecuredLoginPrincipalClaimName} set to a custom claim name if
+ * you wish the name of the String claim holding the principal name to be
+ * something other than '{@code sub}'.</li>
+ * <li>{@code unsecuredLoginLifetimeSeconds} set to an integer value if the
+ * token expiration is to be set to something other than the default value of
+ * 3600 seconds (which is 1 hour). The '{@code exp}' claim reflects the
+ * expiration time.</li>
+ * <li>{@code unsecuredLoginScopeClaimName} set to a custom claim name if you
+ * wish the name of the String or String List claim holding any token scope to
+ * be something other than '{@code scope}'</li>
+ * </ul>
+ * For example:
+ *
+ * <pre>
+ * KafkaClient {
+ * org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule Required
+ * unsecuredLoginStringClaim_sub="thePrincipalName"
+ * unsecuredLoginListClaim_scope="|scopeValue1|scopeValue2"
+ * unsecuredLoginLifetimeSeconds="60";
+ * };
+ * </pre>
+ *
+ * This class is the default when the SASL mechanism is OAUTHBEARER and no value
+ * is explicitly set via either the {@code sasl.login.callback.handler.class}
+ * client configuration property or the
+ * {@code listener.name.sasl_[plaintext|ssl].oauthbearer.sasl.login.callback.handler.class}
+ * broker configuration property.
+ */
+public class OAuthBearerUnsecuredLoginCallbackHandler implements AuthenticateCallbackHandler {
+ private final Logger log = LoggerFactory.getLogger(OAuthBearerUnsecuredLoginCallbackHandler.class);
+ private static final String OPTION_PREFIX = "unsecuredLogin";
+ private static final String PRINCIPAL_CLAIM_NAME_OPTION = OPTION_PREFIX + "PrincipalClaimName";
+ private static final String LIFETIME_SECONDS_OPTION = OPTION_PREFIX + "LifetimeSeconds";
+ private static final String SCOPE_CLAIM_NAME_OPTION = OPTION_PREFIX + "ScopeClaimName";
+ private static final Set<String> RESERVED_CLAIMS = Collections
+ .unmodifiableSet(new HashSet<>(Arrays.asList("iat", "exp")));
+ private static final String DEFAULT_PRINCIPAL_CLAIM_NAME = "sub";
+ private static final String DEFAULT_LIFETIME_SECONDS_ONE_HOUR = "3600";
+ private static final String DEFAULT_SCOPE_CLAIM_NAME = "scope";
+ private static final String STRING_CLAIM_PREFIX = OPTION_PREFIX + "StringClaim_";
+ private static final String NUMBER_CLAIM_PREFIX = OPTION_PREFIX + "NumberClaim_";
+ private static final String LIST_CLAIM_PREFIX = OPTION_PREFIX + "ListClaim_";
+ private static final String QUOTE = "\"";
+ private Time time = Time.SYSTEM;
+ private Map<String, String> moduleOptions = null;
+ private boolean configured = false;
+
+ /**
+ * For testing
+ *
+ * @param time
+ * the mandatory time to set
+ */
+ void time(Time time) {
+ this.time = Objects.requireNonNull(time);
+ }
+
+ /**
+ * Return true if this instance has been configured, otherwise false
+ *
+ * @return true if this instance has been configured, otherwise false
+ */
+ public boolean configured() {
+ return configured;
+ }
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void configure(Map<String, ?> configs, String saslMechanism, List<AppConfigurationEntry> jaasConfigEntries) {
+ if (!OAuthBearerLoginModule.OAUTHBEARER_MECHANISM.equals(saslMechanism))
+ throw new IllegalArgumentException(String.format("Unexpected SASL mechanism: %s", saslMechanism));
+ if (Objects.requireNonNull(jaasConfigEntries).size() != 1 || jaasConfigEntries.get(0) == null)
+ throw new IllegalArgumentException(
+ String.format("Must supply exactly 1 non-null JAAS mechanism configuration (size was %d)",
+ jaasConfigEntries.size()));
+ this.moduleOptions = Collections.unmodifiableMap((Map<String, String>) jaasConfigEntries.get(0).getOptions());
+ configured = true;
+ }
+
+ @Override
+ public void handle(Callback[] callbacks) throws IOException, UnsupportedCallbackException {
+ if (!configured())
+ throw new IllegalStateException("Callback handler not configured");
+ for (Callback callback : callbacks) {
+ if (callback instanceof OAuthBearerTokenCallback)
+ try {
+ handleCallback((OAuthBearerTokenCallback) callback);
+ } catch (KafkaException e) {
+ throw new IOException(e.getMessage(), e);
+ }
+ else
+ throw new UnsupportedCallbackException(callback);
+ }
+ }
+
+ @Override
+ public void close() {
+ // empty
+ }
+
+ private void handleCallback(OAuthBearerTokenCallback callback) throws IOException {
+ if (callback.token() != null)
+ throw new IllegalArgumentException("Callback had a token already");
+ String principalClaimNameValue = optionValue(PRINCIPAL_CLAIM_NAME_OPTION);
+ String principalClaimName = principalClaimNameValue != null && !principalClaimNameValue.trim().isEmpty()
+ ? principalClaimNameValue.trim()
+ : DEFAULT_PRINCIPAL_CLAIM_NAME;
+ String scopeClaimNameValue = optionValue(SCOPE_CLAIM_NAME_OPTION);
+ String scopeClaimName = scopeClaimNameValue != null && !scopeClaimNameValue.trim().isEmpty()
+ ? scopeClaimNameValue.trim()
+ : DEFAULT_SCOPE_CLAIM_NAME;
+ String headerJson = "{" + claimOrHeaderJsonText("alg", "none") + "}";
+ String lifetimeSecondsValueToUse = optionValue(LIFETIME_SECONDS_OPTION, DEFAULT_LIFETIME_SECONDS_ONE_HOUR);
+ String claimsJson;
+ try {
+ claimsJson = String.format("{%s,%s%s}", expClaimText(Long.parseLong(lifetimeSecondsValueToUse)),
+ claimOrHeaderJsonText("iat", time.milliseconds() / 1000.0),
+ commaPrependedStringNumberAndListClaimsJsonText());
+ } catch (NumberFormatException e) {
+ throw new OAuthBearerConfigException(e.getMessage());
+ }
+ try {
+ Encoder urlEncoderNoPadding = Base64.getUrlEncoder().withoutPadding();
+ OAuthBearerUnsecuredJws jws = new OAuthBearerUnsecuredJws(
+ String.format("%s.%s.",
+ urlEncoderNoPadding.encodeToString(headerJson.getBytes(StandardCharsets.UTF_8)),
+ urlEncoderNoPadding.encodeToString(claimsJson.getBytes(StandardCharsets.UTF_8))),
+ principalClaimName, scopeClaimName);
+ log.info("Retrieved token with principal {}", jws.principalName());
+ callback.token(jws);
+ } catch (OAuthBearerIllegalTokenException e) {
+ // occurs if the principal claim doesn't exist or has an empty value
+ throw new OAuthBearerConfigException(e.getMessage(), e);
+ }
+ }
+
+ private String commaPrependedStringNumberAndListClaimsJsonText() throws OAuthBearerConfigException {
+ StringBuilder sb = new StringBuilder();
+ for (String key : moduleOptions.keySet()) {
+ if (key.startsWith(STRING_CLAIM_PREFIX) && key.length() > STRING_CLAIM_PREFIX.length())
+ sb.append(',').append(claimOrHeaderJsonText(
+ confirmNotReservedClaimName(key.substring(STRING_CLAIM_PREFIX.length())), optionValue(key)));
+ else if (key.startsWith(NUMBER_CLAIM_PREFIX) && key.length() > NUMBER_CLAIM_PREFIX.length())
+ sb.append(',')
+ .append(claimOrHeaderJsonText(
+ confirmNotReservedClaimName(key.substring(NUMBER_CLAIM_PREFIX.length())),
+ Double.valueOf(optionValue(key))));
+ else if (key.startsWith(LIST_CLAIM_PREFIX) && key.length() > LIST_CLAIM_PREFIX.length())
+ sb.append(',')
+ .append(claimOrHeaderJsonArrayText(
+ confirmNotReservedClaimName(key.substring(LIST_CLAIM_PREFIX.length())),
+ listJsonText(optionValue(key))));
+ }
+ return sb.toString();
+ }
+
+ private String confirmNotReservedClaimName(String claimName) throws OAuthBearerConfigException {
+ if (RESERVED_CLAIMS.contains(claimName))
+ throw new OAuthBearerConfigException(String.format("Cannot explicitly set the '%s' claim", claimName));
+ return claimName;
+ }
+
+ private String listJsonText(String value) {
+ if (value.isEmpty() || value.length() <= 1)
+ return "[]";
+ String delimiter;
+ String unescapedDelimiterChar = value.substring(0, 1);
+ switch (unescapedDelimiterChar) {
+ case "\\":
+ case ".":
+ case "[":
+ case "(":
+ case "{":
+ case "|":
+ case "^":
+ case "$":
+ delimiter = "\\" + unescapedDelimiterChar;
+ break;
+ default:
+ delimiter = unescapedDelimiterChar;
+ break;
+ }
+ String listText = value.substring(1);
+ String[] elements = listText.split(delimiter);
+ StringBuilder sb = new StringBuilder();
+ for (String element : elements) {
+ sb.append(sb.length() == 0 ? '[' : ',');
+ sb.append('"').append(escape(element)).append('"');
+ }
+ if (listText.startsWith(unescapedDelimiterChar) || listText.endsWith(unescapedDelimiterChar)
+ || listText.contains(unescapedDelimiterChar + unescapedDelimiterChar))
+ sb.append(",\"\"");
+ return sb.append(']').toString();
+ }
+
+ private String optionValue(String key) {
+ return optionValue(key, null);
+ }
+
+ private String optionValue(String key, String defaultValue) {
+ String explicitValue = option(key);
+ return explicitValue != null ? explicitValue : defaultValue;
+ }
+
+ private String option(String key) {
+ if (!configured)
+ throw new IllegalStateException("Callback handler not configured");
+ return moduleOptions.get(Objects.requireNonNull(key));
+ }
+
+ private String claimOrHeaderJsonText(String claimName, Number claimValue) {
+ return QUOTE + escape(claimName) + QUOTE + ":" + claimValue;
+ }
+
+ private String claimOrHeaderJsonText(String claimName, String claimValue) {
+ return QUOTE + escape(claimName) + QUOTE + ":" + QUOTE + escape(claimValue) + QUOTE;
+ }
+
+ private String claimOrHeaderJsonArrayText(String claimName, String escapedClaimValue) {
+ if (!escapedClaimValue.startsWith("[") || !escapedClaimValue.endsWith("]"))
+ throw new IllegalArgumentException(String.format("Illegal JSON array: %s", escapedClaimValue));
+ return QUOTE + escape(claimName) + QUOTE + ":" + escapedClaimValue;
+ }
+
+ private String escape(String jsonStringValue) {
+ return jsonStringValue.replace("\"", "\\\"").replace("\\", "\\\\");
+ }
+
+ private String expClaimText(long lifetimeSeconds) {
+ return claimOrHeaderJsonText("exp", time.milliseconds() / 1000.0 + lifetimeSeconds);
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/unsecured/OAuthBearerUnsecuredValidatorCallbackHandler.java b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/unsecured/OAuthBearerUnsecuredValidatorCallbackHandler.java
new file mode 100644
index 00000000000..0e617274222
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/unsecured/OAuthBearerUnsecuredValidatorCallbackHandler.java
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.security.oauthbearer.internal.unsecured;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+import javax.security.auth.callback.Callback;
+import javax.security.auth.callback.UnsupportedCallbackException;
+import javax.security.auth.login.AppConfigurationEntry;
+
+import org.apache.kafka.common.security.auth.AuthenticateCallbackHandler;
+import org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule;
+import org.apache.kafka.common.security.oauthbearer.OAuthBearerValidatorCallback;
+import org.apache.kafka.common.utils.Time;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@code CallbackHandler} that recognizes
+ * {@link OAuthBearerValidatorCallback} and validates an unsecured OAuth 2
+ * bearer token. It requires there to be an <code>"exp" (Expiration Time)</code>
+ * claim of type Number. If <code>"iat" (Issued At)</code> or
+ * <code>"nbf" (Not Before)</code> claims are present each must be a number that
+ * precedes the Expiration Time claim, and if both are present the Not Before
+ * claim must not precede the Issued At claim. It also accepts the following
+ * options, none of which are required:
+ * <ul>
+ * <li>{@code unsecuredValidatorPrincipalClaimName} set to a non-empty value if
+ * you wish a particular String claim holding a principal name to be checked for
+ * existence; the default is to check for the existence of the '{@code sub}'
+ * claim</li>
+ * <li>{@code unsecuredValidatorScopeClaimName} set to a custom claim name if
+ * you wish the name of the String or String List claim holding any token scope
+ * to be something other than '{@code scope}'</li>
+ * <li>{@code unsecuredValidatorRequiredScope} set to a space-delimited list of
+ * scope values if you wish the String/String List claim holding the token scope
+ * to be checked to make sure it contains certain values</li>
+ * <li>{@code unsecuredValidatorAllowableClockSkewMs} set to a positive integer
+ * value if you wish to allow up to some number of positive milliseconds of
+ * clock skew (the default is 0)</li>
+ * <ul>
+ * For example:
+ *
+ * <pre>
+ * KafkaServer {
+ * org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule Required
+ * unsecuredLoginStringClaim_sub="thePrincipalName"
+ * unsecuredLoginListClaim_scope=",KAFKA_BROKER,LOGIN_TO_KAFKA"
+ * unsecuredValidatorRequiredScope="LOGIN_TO_KAFKA"
+ * unsecuredValidatorAllowableClockSkewMs="3000";
+ * };
+ * </pre>
+ *
+ * This class is the default when the SASL mechanism is OAUTHBEARER and no value
+ * is explicitly set via the
+ * {@code listener.name.sasl_[plaintext|ssl].oauthbearer.sasl.server.callback.handler.class}
+ * broker configuration property.
+ */
+public class OAuthBearerUnsecuredValidatorCallbackHandler implements AuthenticateCallbackHandler {
+ private static final Logger log = LoggerFactory.getLogger(OAuthBearerUnsecuredValidatorCallbackHandler.class);
+ private static final String OPTION_PREFIX = "unsecuredValidator";
+ private static final String PRINCIPAL_CLAIM_NAME_OPTION = OPTION_PREFIX + "PrincipalClaimName";
+ private static final String SCOPE_CLAIM_NAME_OPTION = OPTION_PREFIX + "ScopeClaimName";
+ private static final String REQUIRED_SCOPE_OPTION = OPTION_PREFIX + "RequiredScope";
+ private static final String ALLOWABLE_CLOCK_SKEW_MILLIS_OPTION = OPTION_PREFIX + "AllowableClockSkewMs";
+ private Time time = Time.SYSTEM;
+ private Map<String, String> moduleOptions = null;
+ private boolean configured = false;
+
+ /**
+ * For testing
+ *
+ * @param time
+ * the mandatory time to set
+ */
+ void time(Time time) {
+ this.time = Objects.requireNonNull(time);
+ }
+
+ /**
+ * Return true if this instance has been configured, otherwise false
+ *
+ * @return true if this instance has been configured, otherwise false
+ */
+ public boolean configured() {
+ return configured;
+ }
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void configure(Map<String, ?> configs, String saslMechanism, List<AppConfigurationEntry> jaasConfigEntries) {
+ if (!OAuthBearerLoginModule.OAUTHBEARER_MECHANISM.equals(saslMechanism))
+ throw new IllegalArgumentException(String.format("Unexpected SASL mechanism: %s", saslMechanism));
+ if (Objects.requireNonNull(jaasConfigEntries).size() != 1 || jaasConfigEntries.get(0) == null)
+ throw new IllegalArgumentException(
+ String.format("Must supply exactly 1 non-null JAAS mechanism configuration (size was %d)",
+ jaasConfigEntries.size()));
+ final Map<String, String> unmodifiableModuleOptions = Collections
+ .unmodifiableMap((Map<String, String>) jaasConfigEntries.get(0).getOptions());
+ this.moduleOptions = unmodifiableModuleOptions;
+ configured = true;
+ }
+
+ @Override
+ public void handle(Callback[] callbacks) throws IOException, UnsupportedCallbackException {
+ if (!configured())
+ throw new IllegalStateException("Callback handler not configured");
+ for (Callback callback : callbacks) {
+ if (callback instanceof OAuthBearerValidatorCallback) {
+ OAuthBearerValidatorCallback validationCallback = (OAuthBearerValidatorCallback) callback;
+ try {
+ handleCallback(validationCallback);
+ } catch (OAuthBearerIllegalTokenException e) {
+ OAuthBearerValidationResult failureReason = e.reason();
+ String failureScope = failureReason.failureScope();
+ validationCallback.error(failureScope != null ? "insufficient_scope" : "invalid_token",
+ failureScope, failureReason.failureOpenIdConfig());
+ }
+ } else
+ throw new UnsupportedCallbackException(callback);
+ }
+ }
+
+ @Override
+ public void close() {
+ // empty
+ }
+
+ private void handleCallback(OAuthBearerValidatorCallback callback) {
+ String tokenValue = callback.tokenValue();
+ if (tokenValue == null)
+ throw new IllegalArgumentException("Callback missing required token value");
+ String principalClaimName = principalClaimName();
+ String scopeClaimName = scopeClaimName();
+ List<String> requiredScope = requiredScope();
+ int allowableClockSkewMs = allowableClockSkewMs();
+ OAuthBearerUnsecuredJws unsecuredJwt = new OAuthBearerUnsecuredJws(tokenValue, principalClaimName,
+ scopeClaimName);
+ long now = time.milliseconds();
+ OAuthBearerValidationUtils
+ .validateClaimForExistenceAndType(unsecuredJwt, true, principalClaimName, String.class)
+ .throwExceptionIfFailed();
+ OAuthBearerValidationUtils.validateIssuedAt(unsecuredJwt, false, now, allowableClockSkewMs)
+ .throwExceptionIfFailed();
+ OAuthBearerValidationUtils.validateExpirationTime(unsecuredJwt, now, allowableClockSkewMs)
+ .throwExceptionIfFailed();
+ OAuthBearerValidationUtils.validateTimeConsistency(unsecuredJwt).throwExceptionIfFailed();
+ OAuthBearerValidationUtils.validateScope(unsecuredJwt, requiredScope).throwExceptionIfFailed();
+ log.info("Successfully validated token with principal {}: {}", unsecuredJwt.principalName(),
+ unsecuredJwt.claims().toString());
+ callback.token(unsecuredJwt);
+ }
+
+ private String principalClaimName() {
+ String principalClaimNameValue = option(PRINCIPAL_CLAIM_NAME_OPTION);
+ String principalClaimName = principalClaimNameValue != null && !principalClaimNameValue.trim().isEmpty()
+ ? principalClaimNameValue.trim()
+ : "sub";
+ return principalClaimName;
+ }
+
+ private String scopeClaimName() {
+ String scopeClaimNameValue = option(SCOPE_CLAIM_NAME_OPTION);
+ String scopeClaimName = scopeClaimNameValue != null && !scopeClaimNameValue.trim().isEmpty()
+ ? scopeClaimNameValue.trim()
+ : "scope";
+ return scopeClaimName;
+ }
+
+ private List<String> requiredScope() {
+ String requiredSpaceDelimitedScope = option(REQUIRED_SCOPE_OPTION);
+ List<String> requiredScope = requiredSpaceDelimitedScope == null || requiredSpaceDelimitedScope.trim().isEmpty()
+ ? Collections.<String>emptyList()
+ : OAuthBearerScopeUtils.parseScope(requiredSpaceDelimitedScope.trim());
+ return requiredScope;
+ }
+
+ private int allowableClockSkewMs() {
+ String allowableClockSkewMsValue = option(ALLOWABLE_CLOCK_SKEW_MILLIS_OPTION);
+ int allowableClockSkewMs = 0;
+ try {
+ allowableClockSkewMs = allowableClockSkewMsValue == null || allowableClockSkewMsValue.trim().isEmpty() ? 0
+ : Integer.parseInt(allowableClockSkewMsValue.trim());
+ } catch (NumberFormatException e) {
+ throw new OAuthBearerConfigException(e.getMessage(), e);
+ }
+ if (allowableClockSkewMs < 0) {
+ throw new OAuthBearerConfigException(
+ String.format("Allowable clock skew millis must not be negative: %s", allowableClockSkewMsValue));
+ }
+ return allowableClockSkewMs;
+ }
+
+ private String option(String key) {
+ if (!configured)
+ throw new IllegalStateException("Callback handler not configured");
+ return moduleOptions.get(Objects.requireNonNull(key));
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/unsecured/OAuthBearerValidationResult.java b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/unsecured/OAuthBearerValidationResult.java
new file mode 100644
index 00000000000..36551e92804
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/unsecured/OAuthBearerValidationResult.java
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.security.oauthbearer.internal.unsecured;
+
+import java.io.Serializable;
+
+/**
+ * The result of some kind of token validation
+ */
+public class OAuthBearerValidationResult implements Serializable {
+ private static final long serialVersionUID = 5774669940899777373L;
+ private final boolean success;
+ private final String failureDescription;
+ private final String failureScope;
+ private final String failureOpenIdConfig;
+
+ /**
+ * Return an instance indicating success
+ *
+ * @return an instance indicating success
+ */
+ public static OAuthBearerValidationResult newSuccess() {
+ return new OAuthBearerValidationResult(true, null, null, null);
+ }
+
+ /**
+ * Return a new validation failure instance
+ *
+ * @param failureDescription
+ * optional description of the failure
+ * @return a new validation failure instance
+ */
+ public static OAuthBearerValidationResult newFailure(String failureDescription) {
+ return newFailure(failureDescription, null, null);
+ }
+
+ /**
+ * Return a new validation failure instance
+ *
+ * @param failureDescription
+ * optional description of the failure
+ * @param failureScope
+ * optional scope to be reported with the failure
+ * @param failureOpenIdConfig
+ * optional OpenID Connect configuration to be reported with the
+ * failure
+ * @return a new validation failure instance
+ */
+ public static OAuthBearerValidationResult newFailure(String failureDescription, String failureScope,
+ String failureOpenIdConfig) {
+ return new OAuthBearerValidationResult(false, failureDescription, failureScope, failureOpenIdConfig);
+ }
+
+ private OAuthBearerValidationResult(boolean success, String failureDescription, String failureScope,
+ String failureOpenIdConfig) {
+ if (success && (failureScope != null || failureOpenIdConfig != null))
+ throw new IllegalArgumentException("success was indicated but failure scope/OpenIdConfig were provided");
+ this.success = success;
+ this.failureDescription = failureDescription;
+ this.failureScope = failureScope;
+ this.failureOpenIdConfig = failureOpenIdConfig;
+ }
+
+ /**
+ * Return true if this instance indicates success, otherwise false
+ *
+ * @return true if this instance indicates success, otherwise false
+ */
+ public boolean success() {
+ return success;
+ }
+
+ /**
+ * Return the (potentially null) descriptive message for the failure
+ *
+ * @return the (potentially null) descriptive message for the failure
+ */
+ public String failureDescription() {
+ return failureDescription;
+ }
+
+ /**
+ * Return the (potentially null) scope to be reported with the failure
+ *
+ * @return the (potentially null) scope to be reported with the failure
+ */
+ public String failureScope() {
+ return failureScope;
+ }
+
+ /**
+ * Return the (potentially null) OpenID Connect configuration to be reported
+ * with the failure
+ *
+ * @return the (potentially null) OpenID Connect configuration to be reported
+ * with the failure
+ */
+ public String failureOpenIdConfig() {
+ return failureOpenIdConfig;
+ }
+
+ /**
+ * Raise an exception if this instance indicates failure, otherwise do nothing
+ *
+ * @throws OAuthBearerIllegalTokenException
+ * if this instance indicates failure
+ */
+ public void throwExceptionIfFailed() throws OAuthBearerIllegalTokenException {
+ if (!success())
+ throw new OAuthBearerIllegalTokenException(this);
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/unsecured/OAuthBearerValidationUtils.java b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/unsecured/OAuthBearerValidationUtils.java
new file mode 100644
index 00000000000..6f1b2fef4eb
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internal/unsecured/OAuthBearerValidationUtils.java
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.security.oauthbearer.internal.unsecured;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+
+import org.apache.kafka.common.security.oauthbearer.OAuthBearerToken;
+
+public class OAuthBearerValidationUtils {
+ /**
+ * Validate the given claim for existence and type. It can be required to exist
+ * in the given claims, and if it exists it must be one of the types indicated
+ *
+ * @param jwt
+ * the mandatory JWT to which the validation will be applied
+ * @param required
+ * true if the claim is required to exist
+ * @param claimName
+ * the required claim name identifying the claim to be checked
+ * @param allowedTypes
+ * one or more of {@code String.class}, {@code Number.class}, and
+ * {@code List.class} identifying the type(s) that the claim value is
+ * allowed to be if it exists
+ * @return the result of the validation
+ */
+ public static OAuthBearerValidationResult validateClaimForExistenceAndType(OAuthBearerUnsecuredJws jwt,
+ boolean required, String claimName, Class<?>... allowedTypes) {
+ Object rawClaim = Objects.requireNonNull(jwt).rawClaim(Objects.requireNonNull(claimName));
+ if (rawClaim == null)
+ return required
+ ? OAuthBearerValidationResult.newFailure(String.format("Required claim missing: %s", claimName))
+ : OAuthBearerValidationResult.newSuccess();
+ for (Class<?> allowedType : allowedTypes) {
+ if (allowedType != null && allowedType.isAssignableFrom(rawClaim.getClass()))
+ return OAuthBearerValidationResult.newSuccess();
+ }
+ return OAuthBearerValidationResult.newFailure(String.format("The %s claim had the incorrect type: %s",
+ claimName, rawClaim.getClass().getSimpleName()));
+ }
+
+ /**
+ * Validate the 'iat' (Issued At) claim. It can be required to exist in the
+ * given claims, and if it exists it must be a (potentially fractional) number
+ * of seconds since the epoch defining when the JWT was issued; it is a
+ * validation error if the Issued At time is after the time at which the check
+ * is being done (plus any allowable clock skew).
+ *
+ * @param jwt
+ * the mandatory JWT to which the validation will be applied
+ * @param required
+ * true if the claim is required to exist
+ * @param whenCheckTimeMs
+ * the time relative to which the validation is to occur
+ * @param allowableClockSkewMs
+ * non-negative number to take into account some potential clock skew
+ * @return the result of the validation
+ * @throws OAuthBearerConfigException
+ * if the given allowable clock skew is negative
+ */
+ public static OAuthBearerValidationResult validateIssuedAt(OAuthBearerUnsecuredJws jwt, boolean required,
+ long whenCheckTimeMs, int allowableClockSkewMs) throws OAuthBearerConfigException {
+ Number value;
+ try {
+ value = Objects.requireNonNull(jwt).issuedAt();
+ } catch (OAuthBearerIllegalTokenException e) {
+ return e.reason();
+ }
+ boolean exists = value != null;
+ if (!exists)
+ return doesNotExistResult(required, "iat");
+ double doubleValue = value.doubleValue();
+ return 1000 * doubleValue > whenCheckTimeMs + confirmNonNegative(allowableClockSkewMs)
+ ? OAuthBearerValidationResult.newFailure(String.format(
+ "The Issued At value (%f seconds) was after the indicated time (%d ms) plus allowable clock skew (%d ms)",
+ doubleValue, whenCheckTimeMs, allowableClockSkewMs))
+ : OAuthBearerValidationResult.newSuccess();
+ }
+
+ /**
+ * Validate the 'exp' (Expiration Time) claim. It must exist and it must be a
+ * (potentially fractional) number of seconds defining the point at which the
+ * JWT expires. It is a validation error if the time at which the check is being
+ * done (minus any allowable clock skew) is on or after the Expiration Time
+ * time.
+ *
+ * @param jwt
+ * the mandatory JWT to which the validation will be applied
+ * @param whenCheckTimeMs
+ * the time relative to which the validation is to occur
+ * @param allowableClockSkewMs
+ * non-negative number to take into account some potential clock skew
+ * @return the result of the validation
+ * @throws OAuthBearerConfigException
+ * if the given allowable clock skew is negative
+ */
+ public static OAuthBearerValidationResult validateExpirationTime(OAuthBearerUnsecuredJws jwt, long whenCheckTimeMs,
+ int allowableClockSkewMs) throws OAuthBearerConfigException {
+ Number value;
+ try {
+ value = Objects.requireNonNull(jwt).expirationTime();
+ } catch (OAuthBearerIllegalTokenException e) {
+ return e.reason();
+ }
+ boolean exists = value != null;
+ if (!exists)
+ return doesNotExistResult(true, "exp");
+ double doubleValue = value.doubleValue();
+ return whenCheckTimeMs - confirmNonNegative(allowableClockSkewMs) >= 1000 * doubleValue
+ ? OAuthBearerValidationResult.newFailure(String.format(
+ "The indicated time (%d ms) minus allowable clock skew (%d ms) was on or after the Expiration Time value (%f seconds)",
+ whenCheckTimeMs, allowableClockSkewMs, doubleValue))
+ : OAuthBearerValidationResult.newSuccess();
+ }
+
+ /**
+ * Validate the 'iat' (Issued At) and 'exp' (Expiration Time) claims for
+ * internal consistency. The following must be true if both claims exist:
+ *
+ * <pre>
+ * exp > iat
+ * </pre>
+ *
+ * @param jwt
+ * the mandatory JWT to which the validation will be applied
+ * @return the result of the validation
+ */
+ public static OAuthBearerValidationResult validateTimeConsistency(OAuthBearerUnsecuredJws jwt) {
+ Number issuedAt;
+ Number expirationTime;
+ try {
+ issuedAt = Objects.requireNonNull(jwt).issuedAt();
+ expirationTime = jwt.expirationTime();
+ } catch (OAuthBearerIllegalTokenException e) {
+ return e.reason();
+ }
+ if (expirationTime != null && issuedAt != null && expirationTime.doubleValue() <= issuedAt.doubleValue())
+ return OAuthBearerValidationResult.newFailure(
+ String.format("The Expiration Time time (%f seconds) was not after the Issued At time (%f seconds)",
+ expirationTime.doubleValue(), issuedAt.doubleValue()));
+ return OAuthBearerValidationResult.newSuccess();
+ }
+
+ /**
+ * Validate the given token's scope against the required scope. Every required
+ * scope element (if any) must exist in the provided token's scope for the
+ * validation to succeed.
+ *
+ * @param token
+ * the required token for which the scope will to validate
+ * @param requiredScope
+ * the optional required scope against which the given token's scope
+ * will be validated
+ * @return the result of the validation
+ */
+ public static OAuthBearerValidationResult validateScope(OAuthBearerToken token, List<String> requiredScope) {
+ final Set<String> tokenScope = token.scope();
+ if (requiredScope == null || requiredScope.isEmpty())
+ return OAuthBearerValidationResult.newSuccess();
+ for (String requiredScopeElement : requiredScope) {
+ if (!tokenScope.contains(requiredScopeElement))
+ return OAuthBearerValidationResult.newFailure(String.format(
+ "The provided scope (%s) was mising a required scope (%s). All required scope elements: %s",
+ String.valueOf(tokenScope), requiredScopeElement, requiredScope.toString()),
+ requiredScope.toString(), null);
+ }
+ return OAuthBearerValidationResult.newSuccess();
+ }
+
+ private static int confirmNonNegative(int allowableClockSkewMs) throws OAuthBearerConfigException {
+ if (allowableClockSkewMs < 0)
+ throw new OAuthBearerConfigException(
+ String.format("Allowable clock skew must not be negative: %d", allowableClockSkewMs));
+ return allowableClockSkewMs;
+ }
+
+ private static OAuthBearerValidationResult doesNotExistResult(boolean required, String claimName) {
+ return required ? OAuthBearerValidationResult.newFailure(String.format("Required claim missing: %s", claimName))
+ : OAuthBearerValidationResult.newSuccess();
+ }
+
+ private OAuthBearerValidationUtils() {
+ // empty
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/plain/PlainAuthenticateCallback.java b/clients/src/main/java/org/apache/kafka/common/security/plain/PlainAuthenticateCallback.java
new file mode 100644
index 00000000000..7f42645e487
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/security/plain/PlainAuthenticateCallback.java
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.common.security.plain;
+
+import javax.security.auth.callback.Callback;
+
+/*
+ * Authentication callback for SASL/PLAIN authentication. Callback handler must
+ * set authenticated flag to true if the client provided password in the callback
+ * matches the expected password.
+ */
+public class PlainAuthenticateCallback implements Callback {
+ private final char[] password;
+ private boolean authenticated;
+
+ /**
+ * Creates a callback with the password provided by the client
+ * @param password The password provided by the client during SASL/PLAIN authentication
+ */
+ public PlainAuthenticateCallback(char[] password) {
+ this.password = password;
+ }
+
+ /**
+ * Returns the password provided by the client during SASL/PLAIN authentication
+ */
+ public char[] password() {
+ return password;
+ }
+
+ /**
+ * Returns true if client password matches expected password, false otherwise.
+ * This state is set the server-side callback handler.
+ */
+ public boolean authenticated() {
+ return this.authenticated;
+ }
+
+ /**
+ * Sets the authenticated state. This is set by the server-side callback handler
+ * by matching the client provided password with the expected password.
+ *
+ * @param authenticated true indicates successful authentication
+ */
+ public void authenticated(boolean authenticated) {
+ this.authenticated = authenticated;
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/plain/PlainLoginModule.java b/clients/src/main/java/org/apache/kafka/common/security/plain/PlainLoginModule.java
index c8b29fc69a9..f0a5971ebeb 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/plain/PlainLoginModule.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/plain/PlainLoginModule.java
@@ -16,6 +16,8 @@
*/
package org.apache.kafka.common.security.plain;
+import org.apache.kafka.common.security.plain.internal.PlainSaslServerProvider;
+
import java.util.Map;
import javax.security.auth.Subject;
diff --git a/clients/src/main/java/org/apache/kafka/common/security/plain/PlainSaslServer.java b/clients/src/main/java/org/apache/kafka/common/security/plain/internal/PlainSaslServer.java
similarity index 87%
rename from clients/src/main/java/org/apache/kafka/common/security/plain/PlainSaslServer.java
rename to clients/src/main/java/org/apache/kafka/common/security/plain/internal/PlainSaslServer.java
index e54887f652f..811d9e94aca 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/plain/PlainSaslServer.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/plain/internal/PlainSaslServer.java
@@ -14,21 +14,22 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.kafka.common.security.plain;
+package org.apache.kafka.common.security.plain.internal;
import java.io.UnsupportedEncodingException;
import java.util.Arrays;
import java.util.Map;
+import javax.security.auth.callback.Callback;
import javax.security.auth.callback.CallbackHandler;
+import javax.security.auth.callback.NameCallback;
import javax.security.sasl.Sasl;
import javax.security.sasl.SaslException;
import javax.security.sasl.SaslServer;
import javax.security.sasl.SaslServerFactory;
import org.apache.kafka.common.errors.SaslAuthenticationException;
-import org.apache.kafka.common.security.JaasContext;
-import org.apache.kafka.common.security.authenticator.SaslServerCallbackHandler;
+import org.apache.kafka.common.security.plain.PlainAuthenticateCallback;
/**
* Simple SaslServer implementation for SASL/PLAIN. In order to make this implementation
@@ -46,15 +47,13 @@
public class PlainSaslServer implements SaslServer {
public static final String PLAIN_MECHANISM = "PLAIN";
- private static final String JAAS_USER_PREFIX = "user_";
-
- private final JaasContext jaasContext;
+ private final CallbackHandler callbackHandler;
private boolean complete;
private String authorizationId;
- public PlainSaslServer(JaasContext jaasContext) {
- this.jaasContext = jaasContext;
+ public PlainSaslServer(CallbackHandler callbackHandler) {
+ this.callbackHandler = callbackHandler;
}
/**
@@ -101,12 +100,15 @@ public PlainSaslServer(JaasContext jaasContext) {
throw new SaslException("Authentication failed: password not specified");
}
- String expectedPassword = jaasContext.configEntryOption(JAAS_USER_PREFIX + username,
- PlainLoginModule.class.getName());
- if (!password.equals(expectedPassword)) {
- throw new SaslAuthenticationException("Authentication failed: Invalid username or password");
+ NameCallback nameCallback = new NameCallback("username", username);
+ PlainAuthenticateCallback authenticateCallback = new PlainAuthenticateCallback(password.toCharArray());
+ try {
+ callbackHandler.handle(new Callback[]{nameCallback, authenticateCallback});
+ } catch (Throwable e) {
+ throw new SaslAuthenticationException("Authentication failed: credentials for user could not be verified", e);
}
-
+ if (!authenticateCallback.authenticated())
+ throw new SaslAuthenticationException("Authentication failed: Invalid username or password");
if (!authorizationIdFromClient.isEmpty() && !authorizationIdFromClient.equals(username))
throw new SaslAuthenticationException("Authentication failed: Client requested an authorization id that is different from username");
@@ -167,10 +169,7 @@ public SaslServer createSaslServer(String mechanism, String protocol, String ser
if (!PLAIN_MECHANISM.equals(mechanism))
throw new SaslException(String.format("Mechanism \'%s\' is not supported. Only PLAIN is supported.", mechanism));
- if (!(cbh instanceof SaslServerCallbackHandler))
- throw new SaslException("CallbackHandler must be of type SaslServerCallbackHandler, but it is: " + cbh.getClass());
-
- return new PlainSaslServer(((SaslServerCallbackHandler) cbh).jaasContext());
+ return new PlainSaslServer(cbh);
}
@Override
diff --git a/clients/src/main/java/org/apache/kafka/common/security/plain/PlainSaslServerProvider.java b/clients/src/main/java/org/apache/kafka/common/security/plain/internal/PlainSaslServerProvider.java
similarity index 90%
rename from clients/src/main/java/org/apache/kafka/common/security/plain/PlainSaslServerProvider.java
rename to clients/src/main/java/org/apache/kafka/common/security/plain/internal/PlainSaslServerProvider.java
index ae1424417f8..c2229532449 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/plain/PlainSaslServerProvider.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/plain/internal/PlainSaslServerProvider.java
@@ -14,12 +14,12 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.kafka.common.security.plain;
+package org.apache.kafka.common.security.plain.internal;
import java.security.Provider;
import java.security.Security;
-import org.apache.kafka.common.security.plain.PlainSaslServer.PlainSaslServerFactory;
+import org.apache.kafka.common.security.plain.internal.PlainSaslServer.PlainSaslServerFactory;
public class PlainSaslServerProvider extends Provider {
diff --git a/clients/src/main/java/org/apache/kafka/common/security/plain/internal/PlainServerCallbackHandler.java b/clients/src/main/java/org/apache/kafka/common/security/plain/internal/PlainServerCallbackHandler.java
new file mode 100644
index 00000000000..84fbdfd4c62
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/security/plain/internal/PlainServerCallbackHandler.java
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.common.security.plain.internal;
+
+import org.apache.kafka.common.security.JaasContext;
+import org.apache.kafka.common.security.auth.AuthenticateCallbackHandler;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.security.plain.PlainAuthenticateCallback;
+import org.apache.kafka.common.security.plain.PlainLoginModule;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+import javax.security.auth.callback.Callback;
+import javax.security.auth.callback.NameCallback;
+import javax.security.auth.callback.UnsupportedCallbackException;
+import javax.security.auth.login.AppConfigurationEntry;
+
+public class PlainServerCallbackHandler implements AuthenticateCallbackHandler {
+
+ private static final String JAAS_USER_PREFIX = "user_";
+ private List<AppConfigurationEntry> jaasConfigEntries;
+
+ @Override
+ public void configure(Map<String, ?> configs, String mechanism, List<AppConfigurationEntry> jaasConfigEntries) {
+ this.jaasConfigEntries = jaasConfigEntries;
+ }
+
+ @Override
+ public void handle(Callback[] callbacks) throws IOException, UnsupportedCallbackException {
+ String username = null;
+ for (Callback callback: callbacks) {
+ if (callback instanceof NameCallback)
+ username = ((NameCallback) callback).getDefaultName();
+ else if (callback instanceof PlainAuthenticateCallback) {
+ PlainAuthenticateCallback plainCallback = (PlainAuthenticateCallback) callback;
+ boolean authenticated = authenticate(username, plainCallback.password());
+ plainCallback.authenticated(authenticated);
+ } else
+ throw new UnsupportedCallbackException(callback);
+ }
+ }
+
+ protected boolean authenticate(String username, char[] password) throws IOException {
+ if (username == null)
+ return false;
+ else {
+ String expectedPassword = JaasContext.configEntryOption(jaasConfigEntries,
+ JAAS_USER_PREFIX + username,
+ PlainLoginModule.class.getName());
+ return expectedPassword != null && Arrays.equals(password, expectedPassword.toCharArray());
+ }
+ }
+
+ @Override
+ public void close() throws KafkaException {
+ }
+
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/scram/ScramCredential.java b/clients/src/main/java/org/apache/kafka/common/security/scram/ScramCredential.java
index 09ff0aacd3c..dfbfef15b4f 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/scram/ScramCredential.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/scram/ScramCredential.java
@@ -16,6 +16,11 @@
*/
package org.apache.kafka.common.security.scram;
+/**
+ * SCRAM credential class that encapsulates the credential data persisted for each user that is
+ * accessible to the server. See <a href="https://tools.ietf.org/html/rfc5802#section-5">RFC rfc5802</a>
+ * for details.
+ */
public class ScramCredential {
private final byte[] salt;
@@ -23,6 +28,9 @@
private final byte[] storedKey;
private final int iterations;
+ /**
+ * Constructs a new credential.
+ */
public ScramCredential(byte[] salt, byte[] storedKey, byte[] serverKey, int iterations) {
this.salt = salt;
this.serverKey = serverKey;
@@ -30,18 +38,30 @@ public ScramCredential(byte[] salt, byte[] storedKey, byte[] serverKey, int iter
this.iterations = iterations;
}
+ /**
+ * Returns the salt used to process this credential using the SCRAM algorithm.
+ */
public byte[] salt() {
return salt;
}
+ /**
+ * Server key computed from the client password using the SCRAM algorithm.
+ */
public byte[] serverKey() {
return serverKey;
}
+ /**
+ * Stored key computed from the client password using the SCRAM algorithm.
+ */
public byte[] storedKey() {
return storedKey;
}
+ /**
+ * Number of iterations used to process this credential using the SCRAM algorithm.
+ */
public int iterations() {
return iterations;
}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/scram/ScramCredentialCallback.java b/clients/src/main/java/org/apache/kafka/common/security/scram/ScramCredentialCallback.java
index 931210a0e07..d5988cbeb89 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/scram/ScramCredentialCallback.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/scram/ScramCredentialCallback.java
@@ -18,14 +18,23 @@
import javax.security.auth.callback.Callback;
+/**
+ * Callback used for SCRAM mechanisms.
+ */
public class ScramCredentialCallback implements Callback {
private ScramCredential scramCredential;
- public ScramCredential scramCredential() {
- return scramCredential;
- }
-
+ /**
+ * Sets the SCRAM credential for this instance.
+ */
public void scramCredential(ScramCredential scramCredential) {
this.scramCredential = scramCredential;
}
-}
\ No newline at end of file
+
+ /**
+ * Returns the SCRAM credential if set on this instance.
+ */
+ public ScramCredential scramCredential() {
+ return scramCredential;
+ }
+}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/scram/ScramExtensionsCallback.java b/clients/src/main/java/org/apache/kafka/common/security/scram/ScramExtensionsCallback.java
index b40468bd650..debe163e36b 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/scram/ScramExtensionsCallback.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/scram/ScramExtensionsCallback.java
@@ -21,13 +21,25 @@
import java.util.Collections;
import java.util.Map;
+/**
+ * Optional callback used for SCRAM mechanisms if any extensions need to be set
+ * in the SASL/SCRAM exchange.
+ */
public class ScramExtensionsCallback implements Callback {
private Map<String, String> extensions = Collections.emptyMap();
+ /**
+ * Returns the extension names and values that are sent by the client to
+ * the server in the initial client SCRAM authentication message.
+ * Default is an empty map.
+ */
public Map<String, String> extensions() {
return extensions;
}
+ /**
+ * Sets the SCRAM extensions on this callback.
+ */
public void extensions(Map<String, String> extensions) {
this.extensions = extensions;
}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/scram/ScramLoginModule.java b/clients/src/main/java/org/apache/kafka/common/security/scram/ScramLoginModule.java
index 43df515256f..20d1f221b1c 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/scram/ScramLoginModule.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/scram/ScramLoginModule.java
@@ -16,6 +16,9 @@
*/
package org.apache.kafka.common.security.scram;
+import org.apache.kafka.common.security.scram.internal.ScramSaslClientProvider;
+import org.apache.kafka.common.security.scram.internal.ScramSaslServerProvider;
+
import java.util.Collections;
import java.util.Map;
diff --git a/clients/src/main/java/org/apache/kafka/common/security/scram/ScramCredentialUtils.java b/clients/src/main/java/org/apache/kafka/common/security/scram/internal/ScramCredentialUtils.java
similarity index 83%
rename from clients/src/main/java/org/apache/kafka/common/security/scram/ScramCredentialUtils.java
rename to clients/src/main/java/org/apache/kafka/common/security/scram/internal/ScramCredentialUtils.java
index b4875d6f57e..880747f0d01 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/scram/ScramCredentialUtils.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/scram/internal/ScramCredentialUtils.java
@@ -14,13 +14,14 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.kafka.common.security.scram;
+package org.apache.kafka.common.security.scram.internal;
+import java.util.Base64;
import java.util.Collection;
import java.util.Properties;
import org.apache.kafka.common.security.authenticator.CredentialCache;
-import org.apache.kafka.common.utils.Base64;
+import org.apache.kafka.common.security.scram.ScramCredential;
/**
* SCRAM Credential persistence utility functions. Implements format conversion used
@@ -42,11 +43,11 @@ private ScramCredentialUtils() {}
public static String credentialToString(ScramCredential credential) {
return String.format("%s=%s,%s=%s,%s=%s,%s=%d",
SALT,
- Base64.encoder().encodeToString(credential.salt()),
+ Base64.getEncoder().encodeToString(credential.salt()),
STORED_KEY,
- Base64.encoder().encodeToString(credential.storedKey()),
+ Base64.getEncoder().encodeToString(credential.storedKey()),
SERVER_KEY,
- Base64.encoder().encodeToString(credential.serverKey()),
+ Base64.getEncoder().encodeToString(credential.serverKey()),
ITERATIONS,
credential.iterations());
}
@@ -57,9 +58,9 @@ public static ScramCredential credentialFromString(String str) {
!props.containsKey(SERVER_KEY) || !props.containsKey(ITERATIONS)) {
throw new IllegalArgumentException("Credentials not valid: " + str);
}
- byte[] salt = Base64.decoder().decode(props.getProperty(SALT));
- byte[] storedKey = Base64.decoder().decode(props.getProperty(STORED_KEY));
- byte[] serverKey = Base64.decoder().decode(props.getProperty(SERVER_KEY));
+ byte[] salt = Base64.getDecoder().decode(props.getProperty(SALT));
+ byte[] storedKey = Base64.getDecoder().decode(props.getProperty(STORED_KEY));
+ byte[] serverKey = Base64.getDecoder().decode(props.getProperty(SERVER_KEY));
int iterations = Integer.parseInt(props.getProperty(ITERATIONS));
return new ScramCredential(salt, storedKey, serverKey, iterations);
}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/scram/ScramExtensions.java b/clients/src/main/java/org/apache/kafka/common/security/scram/internal/ScramExtensions.java
similarity index 95%
rename from clients/src/main/java/org/apache/kafka/common/security/scram/ScramExtensions.java
rename to clients/src/main/java/org/apache/kafka/common/security/scram/internal/ScramExtensions.java
index 0f461c0e3c2..66d9362a3e6 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/scram/ScramExtensions.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/scram/internal/ScramExtensions.java
@@ -14,7 +14,9 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.kafka.common.security.scram;
+package org.apache.kafka.common.security.scram.internal;
+
+import org.apache.kafka.common.security.scram.ScramLoginModule;
import java.util.Collections;
import java.util.HashMap;
diff --git a/clients/src/main/java/org/apache/kafka/common/security/scram/ScramFormatter.java b/clients/src/main/java/org/apache/kafka/common/security/scram/internal/ScramFormatter.java
similarity index 94%
rename from clients/src/main/java/org/apache/kafka/common/security/scram/ScramFormatter.java
rename to clients/src/main/java/org/apache/kafka/common/security/scram/internal/ScramFormatter.java
index 406c28599b3..6fcb7a152db 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/scram/ScramFormatter.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/scram/internal/ScramFormatter.java
@@ -14,7 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.kafka.common.security.scram;
+package org.apache.kafka.common.security.scram.internal;
import java.math.BigInteger;
import java.nio.charset.StandardCharsets;
@@ -27,9 +27,10 @@
import javax.crypto.spec.SecretKeySpec;
import org.apache.kafka.common.KafkaException;
-import org.apache.kafka.common.security.scram.ScramMessages.ClientFinalMessage;
-import org.apache.kafka.common.security.scram.ScramMessages.ClientFirstMessage;
-import org.apache.kafka.common.security.scram.ScramMessages.ServerFirstMessage;
+import org.apache.kafka.common.security.scram.ScramCredential;
+import org.apache.kafka.common.security.scram.internal.ScramMessages.ClientFinalMessage;
+import org.apache.kafka.common.security.scram.internal.ScramMessages.ClientFirstMessage;
+import org.apache.kafka.common.security.scram.internal.ScramMessages.ServerFirstMessage;
/**
* Scram message salt and hash functions defined in <a href="https://tools.ietf.org/html/rfc5802">RFC 5802</a>.
diff --git a/clients/src/main/java/org/apache/kafka/common/security/scram/ScramMechanism.java b/clients/src/main/java/org/apache/kafka/common/security/scram/internal/ScramMechanism.java
similarity index 97%
rename from clients/src/main/java/org/apache/kafka/common/security/scram/ScramMechanism.java
rename to clients/src/main/java/org/apache/kafka/common/security/scram/internal/ScramMechanism.java
index d8c0c6d3d60..73be4cf24fb 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/scram/ScramMechanism.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/scram/internal/ScramMechanism.java
@@ -14,7 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.kafka.common.security.scram;
+package org.apache.kafka.common.security.scram.internal;
import java.util.Collection;
import java.util.Collections;
diff --git a/clients/src/main/java/org/apache/kafka/common/security/scram/ScramMessages.java b/clients/src/main/java/org/apache/kafka/common/security/scram/internal/ScramMessages.java
similarity index 93%
rename from clients/src/main/java/org/apache/kafka/common/security/scram/ScramMessages.java
rename to clients/src/main/java/org/apache/kafka/common/security/scram/internal/ScramMessages.java
index 05b3d775540..7e88c4482a4 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/scram/ScramMessages.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/scram/internal/ScramMessages.java
@@ -14,11 +14,10 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.kafka.common.security.scram;
-
-import org.apache.kafka.common.utils.Base64;
+package org.apache.kafka.common.security.scram.internal;
import java.nio.charset.StandardCharsets;
+import java.util.Base64;
import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
@@ -156,7 +155,7 @@ public ServerFirstMessage(byte[] messageBytes) throws SaslException {
}
this.nonce = matcher.group("nonce");
String salt = matcher.group("salt");
- this.salt = Base64.decoder().decode(salt);
+ this.salt = Base64.getDecoder().decode(salt);
}
public ServerFirstMessage(String clientNonce, String serverNonce, byte[] salt, int iterations) {
this.nonce = clientNonce + serverNonce;
@@ -173,7 +172,7 @@ public int iterations() {
return iterations;
}
String toMessage() {
- return String.format("r=%s,s=%s,i=%d", nonce, Base64.encoder().encodeToString(salt), iterations);
+ return String.format("r=%s,s=%s,i=%d", nonce, Base64.getEncoder().encodeToString(salt), iterations);
}
}
/**
@@ -200,9 +199,9 @@ public ClientFinalMessage(byte[] messageBytes) throws SaslException {
if (!matcher.matches())
throw new SaslException("Invalid SCRAM client final message format: " + message);
- this.channelBinding = Base64.decoder().decode(matcher.group("channel"));
+ this.channelBinding = Base64.getDecoder().decode(matcher.group("channel"));
this.nonce = matcher.group("nonce");
- this.proof = Base64.decoder().decode(matcher.group("proof"));
+ this.proof = Base64.getDecoder().decode(matcher.group("proof"));
}
public ClientFinalMessage(byte[] channelBinding, String nonce) {
this.channelBinding = channelBinding;
@@ -222,13 +221,13 @@ public void proof(byte[] proof) {
}
public String clientFinalMessageWithoutProof() {
return String.format("c=%s,r=%s",
- Base64.encoder().encodeToString(channelBinding),
+ Base64.getEncoder().encodeToString(channelBinding),
nonce);
}
String toMessage() {
return String.format("%s,p=%s",
clientFinalMessageWithoutProof(),
- Base64.encoder().encodeToString(proof));
+ Base64.getEncoder().encodeToString(proof));
}
}
/**
@@ -259,7 +258,7 @@ public ServerFinalMessage(byte[] messageBytes) throws SaslException {
// ignore
}
if (error == null) {
- this.serverSignature = Base64.decoder().decode(matcher.group("signature"));
+ this.serverSignature = Base64.getDecoder().decode(matcher.group("signature"));
this.error = null;
} else {
this.serverSignature = null;
@@ -280,7 +279,7 @@ String toMessage() {
if (error != null)
return "e=" + error;
else
- return "v=" + Base64.encoder().encodeToString(serverSignature);
+ return "v=" + Base64.getEncoder().encodeToString(serverSignature);
}
}
}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/scram/ScramSaslClient.java b/clients/src/main/java/org/apache/kafka/common/security/scram/internal/ScramSaslClient.java
similarity index 90%
rename from clients/src/main/java/org/apache/kafka/common/security/scram/ScramSaslClient.java
rename to clients/src/main/java/org/apache/kafka/common/security/scram/internal/ScramSaslClient.java
index 71109df2ddd..a98a86d61cb 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/scram/ScramSaslClient.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/scram/internal/ScramSaslClient.java
@@ -14,9 +14,8 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.kafka.common.security.scram;
+package org.apache.kafka.common.security.scram.internal;
-import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
@@ -34,9 +33,10 @@
import javax.security.sasl.SaslException;
import org.apache.kafka.common.errors.IllegalSaslStateException;
-import org.apache.kafka.common.security.scram.ScramMessages.ClientFinalMessage;
-import org.apache.kafka.common.security.scram.ScramMessages.ServerFinalMessage;
-import org.apache.kafka.common.security.scram.ScramMessages.ServerFirstMessage;
+import org.apache.kafka.common.security.scram.ScramExtensionsCallback;
+import org.apache.kafka.common.security.scram.internal.ScramMessages.ClientFinalMessage;
+import org.apache.kafka.common.security.scram.internal.ScramMessages.ServerFinalMessage;
+import org.apache.kafka.common.security.scram.internal.ScramMessages.ServerFirstMessage;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -100,9 +100,15 @@ public boolean hasInitialResponse() {
ScramExtensionsCallback extensionsCallback = new ScramExtensionsCallback();
try {
- callbackHandler.handle(new Callback[]{nameCallback, extensionsCallback});
- } catch (IOException | UnsupportedCallbackException e) {
- throw new SaslException("User name could not be obtained", e);
+ callbackHandler.handle(new Callback[]{nameCallback});
+ try {
+ callbackHandler.handle(new Callback[]{extensionsCallback});
+ } catch (UnsupportedCallbackException e) {
+ log.debug("Extensions callback is not supported by client callback handler {}, no extensions will be added",
+ callbackHandler);
+ }
+ } catch (Throwable e) {
+ throw new SaslException("User name or extensions could not be obtained", e);
}
String username = nameCallback.getName();
@@ -121,7 +127,7 @@ public boolean hasInitialResponse() {
PasswordCallback passwordCallback = new PasswordCallback("Password:", false);
try {
callbackHandler.handle(new Callback[]{passwordCallback});
- } catch (IOException | UnsupportedCallbackException e) {
+ } catch (Throwable e) {
throw new SaslException("User name could not be obtained", e);
}
this.clientFinalMessage = handleServerFirstMessage(passwordCallback.getPassword());
diff --git a/clients/src/main/java/org/apache/kafka/common/security/scram/ScramSaslClientProvider.java b/clients/src/main/java/org/apache/kafka/common/security/scram/internal/ScramSaslClientProvider.java
similarity index 90%
rename from clients/src/main/java/org/apache/kafka/common/security/scram/ScramSaslClientProvider.java
rename to clients/src/main/java/org/apache/kafka/common/security/scram/internal/ScramSaslClientProvider.java
index d389f044a99..4d9ff81309b 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/scram/ScramSaslClientProvider.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/scram/internal/ScramSaslClientProvider.java
@@ -14,12 +14,12 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.kafka.common.security.scram;
+package org.apache.kafka.common.security.scram.internal;
import java.security.Provider;
import java.security.Security;
-import org.apache.kafka.common.security.scram.ScramSaslClient.ScramSaslClientFactory;
+import org.apache.kafka.common.security.scram.internal.ScramSaslClient.ScramSaslClientFactory;
public class ScramSaslClientProvider extends Provider {
diff --git a/clients/src/main/java/org/apache/kafka/common/security/scram/ScramSaslServer.java b/clients/src/main/java/org/apache/kafka/common/security/scram/internal/ScramSaslServer.java
similarity index 92%
rename from clients/src/main/java/org/apache/kafka/common/security/scram/ScramSaslServer.java
rename to clients/src/main/java/org/apache/kafka/common/security/scram/internal/ScramSaslServer.java
index 314c1d413ba..60857279f59 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/scram/ScramSaslServer.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/scram/internal/ScramSaslServer.java
@@ -14,9 +14,8 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.kafka.common.security.scram;
+package org.apache.kafka.common.security.scram.internal;
-import java.io.IOException;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
import java.util.Arrays;
@@ -27,18 +26,21 @@
import javax.security.auth.callback.Callback;
import javax.security.auth.callback.CallbackHandler;
import javax.security.auth.callback.NameCallback;
-import javax.security.auth.callback.UnsupportedCallbackException;
import javax.security.sasl.SaslException;
import javax.security.sasl.SaslServer;
import javax.security.sasl.SaslServerFactory;
+import org.apache.kafka.common.errors.AuthenticationException;
import org.apache.kafka.common.errors.IllegalSaslStateException;
import org.apache.kafka.common.errors.SaslAuthenticationException;
-import org.apache.kafka.common.security.scram.ScramMessages.ClientFinalMessage;
-import org.apache.kafka.common.security.scram.ScramMessages.ClientFirstMessage;
-import org.apache.kafka.common.security.scram.ScramMessages.ServerFinalMessage;
-import org.apache.kafka.common.security.scram.ScramMessages.ServerFirstMessage;
-import org.apache.kafka.common.security.token.delegation.DelegationTokenCredentialCallback;
+import org.apache.kafka.common.security.scram.ScramCredential;
+import org.apache.kafka.common.security.scram.ScramCredentialCallback;
+import org.apache.kafka.common.security.scram.ScramLoginModule;
+import org.apache.kafka.common.security.scram.internal.ScramMessages.ClientFinalMessage;
+import org.apache.kafka.common.security.scram.internal.ScramMessages.ClientFirstMessage;
+import org.apache.kafka.common.security.scram.internal.ScramMessages.ServerFinalMessage;
+import org.apache.kafka.common.security.scram.internal.ScramMessages.ServerFirstMessage;
+import org.apache.kafka.common.security.token.delegation.internal.DelegationTokenCredentialCallback;
import org.apache.kafka.common.utils.Utils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -133,7 +135,9 @@ public ScramSaslServer(ScramMechanism mechanism, Map<String, ?> props, CallbackH
scramCredential.iterations());
setState(State.RECEIVE_CLIENT_FINAL_MESSAGE);
return serverFirstMessage.toBytes();
- } catch (IOException | NumberFormatException | UnsupportedCallbackException e) {
+ } catch (SaslException | AuthenticationException e) {
+ throw e;
+ } catch (Throwable e) {
throw new SaslException("Authentication failed: Credentials could not be obtained", e);
}
@@ -154,7 +158,7 @@ public ScramSaslServer(ScramMechanism mechanism, Map<String, ?> props, CallbackH
default:
throw new IllegalSaslStateException("Unexpected challenge in Sasl server state " + state);
}
- } catch (SaslException e) {
+ } catch (SaslException | AuthenticationException e) {
clearCredentials();
setState(State.FAILED);
throw e;
diff --git a/clients/src/main/java/org/apache/kafka/common/security/scram/ScramSaslServerProvider.java b/clients/src/main/java/org/apache/kafka/common/security/scram/internal/ScramSaslServerProvider.java
similarity index 90%
rename from clients/src/main/java/org/apache/kafka/common/security/scram/ScramSaslServerProvider.java
rename to clients/src/main/java/org/apache/kafka/common/security/scram/internal/ScramSaslServerProvider.java
index 9f2a6b3d256..099e50e19dd 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/scram/ScramSaslServerProvider.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/scram/internal/ScramSaslServerProvider.java
@@ -14,12 +14,12 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.kafka.common.security.scram;
+package org.apache.kafka.common.security.scram.internal;
import java.security.Provider;
import java.security.Security;
-import org.apache.kafka.common.security.scram.ScramSaslServer.ScramSaslServerFactory;
+import org.apache.kafka.common.security.scram.internal.ScramSaslServer.ScramSaslServerFactory;
public class ScramSaslServerProvider extends Provider {
diff --git a/clients/src/main/java/org/apache/kafka/common/security/scram/ScramServerCallbackHandler.java b/clients/src/main/java/org/apache/kafka/common/security/scram/internal/ScramServerCallbackHandler.java
similarity index 76%
rename from clients/src/main/java/org/apache/kafka/common/security/scram/ScramServerCallbackHandler.java
rename to clients/src/main/java/org/apache/kafka/common/security/scram/internal/ScramServerCallbackHandler.java
index 5e37eae9d4f..9a3f0dc66b7 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/scram/ScramServerCallbackHandler.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/scram/internal/ScramServerCallbackHandler.java
@@ -14,23 +14,25 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.kafka.common.security.scram;
+package org.apache.kafka.common.security.scram.internal;
import java.io.IOException;
+import java.util.List;
import java.util.Map;
-import javax.security.auth.Subject;
import javax.security.auth.callback.Callback;
import javax.security.auth.callback.NameCallback;
import javax.security.auth.callback.UnsupportedCallbackException;
+import javax.security.auth.login.AppConfigurationEntry;
-import org.apache.kafka.common.network.Mode;
-import org.apache.kafka.common.security.authenticator.AuthCallbackHandler;
+import org.apache.kafka.common.security.auth.AuthenticateCallbackHandler;
import org.apache.kafka.common.security.authenticator.CredentialCache;
-import org.apache.kafka.common.security.token.delegation.DelegationTokenCache;
-import org.apache.kafka.common.security.token.delegation.DelegationTokenCredentialCallback;
+import org.apache.kafka.common.security.scram.ScramCredential;
+import org.apache.kafka.common.security.scram.ScramCredentialCallback;
+import org.apache.kafka.common.security.token.delegation.internal.DelegationTokenCache;
+import org.apache.kafka.common.security.token.delegation.internal.DelegationTokenCredentialCallback;
-public class ScramServerCallbackHandler implements AuthCallbackHandler {
+public class ScramServerCallbackHandler implements AuthenticateCallbackHandler {
private final CredentialCache.Cache<ScramCredential> credentialCache;
private final DelegationTokenCache tokenCache;
@@ -42,6 +44,11 @@ public ScramServerCallbackHandler(CredentialCache.Cache<ScramCredential> credent
this.tokenCache = tokenCache;
}
+ @Override
+ public void configure(Map<String, ?> configs, String mechanism, List<AppConfigurationEntry> jaasConfigEntries) {
+ this.saslMechanism = mechanism;
+ }
+
@Override
public void handle(Callback[] callbacks) throws IOException, UnsupportedCallbackException {
String username = null;
@@ -60,11 +67,6 @@ else if (callback instanceof DelegationTokenCredentialCallback) {
}
}
- @Override
- public void configure(Map<String, ?> configs, Mode mode, Subject subject, String saslMechanism) {
- this.saslMechanism = saslMechanism;
- }
-
@Override
public void close() {
}
diff --git a/clients/src/main/java/org/apache/kafka/common/security/ssl/SslFactory.java b/clients/src/main/java/org/apache/kafka/common/security/ssl/SslFactory.java
index c54260d580f..055404caab4 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/ssl/SslFactory.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/ssl/SslFactory.java
@@ -132,7 +132,7 @@ else if (clientAuthConfig.equals("requested"))
(String) configs.get(SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG),
(Password) configs.get(SslConfigs.SSL_TRUSTSTORE_PASSWORD_CONFIG));
try {
- this.sslContext = createSSLContext(keystore);
+ this.sslContext = createSSLContext(keystore, truststore);
} catch (Exception e) {
throw new KafkaException(e);
}
@@ -147,8 +147,12 @@ else if (clientAuthConfig.equals("requested"))
public void validateReconfiguration(Map<String, ?> configs) {
try {
SecurityStore newKeystore = maybeCreateNewKeystore(configs);
- if (newKeystore != null)
- createSSLContext(newKeystore);
+ SecurityStore newTruststore = maybeCreateNewTruststore(configs);
+ if (newKeystore != null || newTruststore != null) {
+ SecurityStore keystore = newKeystore != null ? newKeystore : this.keystore;
+ SecurityStore truststore = newTruststore != null ? newTruststore : this.truststore;
+ createSSLContext(keystore, truststore);
+ }
} catch (Exception e) {
throw new ConfigException("Validation of dynamic config update failed", e);
}
@@ -157,21 +161,25 @@ public void validateReconfiguration(Map<String, ?> configs) {
@Override
public void reconfigure(Map<String, ?> configs) throws KafkaException {
SecurityStore newKeystore = maybeCreateNewKeystore(configs);
- if (newKeystore != null) {
+ SecurityStore newTruststore = maybeCreateNewTruststore(configs);
+ if (newKeystore != null || newTruststore != null) {
try {
- this.sslContext = createSSLContext(newKeystore);
- this.keystore = newKeystore;
+ SecurityStore keystore = newKeystore != null ? newKeystore : this.keystore;
+ SecurityStore truststore = newTruststore != null ? newTruststore : this.truststore;
+ this.sslContext = createSSLContext(keystore, truststore);
+ this.keystore = keystore;
+ this.truststore = truststore;
} catch (Exception e) {
- throw new ConfigException("Reconfiguration of SSL keystore failed", e);
+ throw new ConfigException("Reconfiguration of SSL keystore/truststore failed", e);
}
}
}
private SecurityStore maybeCreateNewKeystore(Map<String, ?> configs) {
- boolean keystoreChanged = Objects.equals(configs.get(SslConfigs.SSL_KEYSTORE_TYPE_CONFIG), keystore.type) ||
- Objects.equals(configs.get(SslConfigs.SSL_KEYSTORE_LOCATION_CONFIG), keystore.path) ||
- Objects.equals(configs.get(SslConfigs.SSL_KEYSTORE_PASSWORD_CONFIG), keystore.password) ||
- Objects.equals(configs.get(SslConfigs.SSL_KEY_PASSWORD_CONFIG), keystore.keyPassword);
+ boolean keystoreChanged = !Objects.equals(configs.get(SslConfigs.SSL_KEYSTORE_TYPE_CONFIG), keystore.type) ||
+ !Objects.equals(configs.get(SslConfigs.SSL_KEYSTORE_LOCATION_CONFIG), keystore.path) ||
+ !Objects.equals(configs.get(SslConfigs.SSL_KEYSTORE_PASSWORD_CONFIG), keystore.password) ||
+ !Objects.equals(configs.get(SslConfigs.SSL_KEY_PASSWORD_CONFIG), keystore.keyPassword);
if (keystoreChanged) {
return createKeystore((String) configs.get(SslConfigs.SSL_KEYSTORE_TYPE_CONFIG),
@@ -182,8 +190,21 @@ private SecurityStore maybeCreateNewKeystore(Map<String, ?> configs) {
return null;
}
+ private SecurityStore maybeCreateNewTruststore(Map<String, ?> configs) {
+ boolean truststoreChanged = !Objects.equals(configs.get(SslConfigs.SSL_TRUSTSTORE_TYPE_CONFIG), truststore.type) ||
+ !Objects.equals(configs.get(SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG), truststore.path) ||
+ !Objects.equals(configs.get(SslConfigs.SSL_TRUSTSTORE_PASSWORD_CONFIG), truststore.password);
+
+ if (truststoreChanged) {
+ return createTruststore((String) configs.get(SslConfigs.SSL_TRUSTSTORE_TYPE_CONFIG),
+ (String) configs.get(SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG),
+ (Password) configs.get(SslConfigs.SSL_TRUSTSTORE_PASSWORD_CONFIG));
+ } else
+ return null;
+ }
+
// package access for testing
- SSLContext createSSLContext(SecurityStore keystore) throws GeneralSecurityException, IOException {
+ SSLContext createSSLContext(SecurityStore keystore, SecurityStore truststore) throws GeneralSecurityException, IOException {
SSLContext sslContext;
if (provider != null)
sslContext = SSLContext.getInstance(protocol, provider);
@@ -206,12 +227,17 @@ SSLContext createSSLContext(SecurityStore keystore) throws GeneralSecurityExcept
tmf.init(ts);
sslContext.init(keyManagers, tmf.getTrustManagers(), this.secureRandomImplementation);
- if (keystore != null && keystore != this.keystore) {
+ boolean verifyKeystore = keystore != null && keystore != this.keystore;
+ boolean verifyTruststore = truststore != null && truststore != this.truststore;
+ if (verifyKeystore || verifyTruststore) {
if (this.keystore == null)
throw new ConfigException("Cannot add SSL keystore to an existing listener for which no keystore was configured.");
- if (keystoreVerifiableUsingTruststore)
- SSLConfigValidatorEngine.validate(this, sslContext);
- if (!CertificateEntries.create(this.keystore.load()).equals(CertificateEntries.create(keystore.load()))) {
+ if (keystoreVerifiableUsingTruststore) {
+ SSLConfigValidatorEngine.validate(this, sslContext, this.sslContext);
+ SSLConfigValidatorEngine.validate(this, this.sslContext, sslContext);
+ }
+ if (verifyKeystore &&
+ !CertificateEntries.create(this.keystore.load()).equals(CertificateEntries.create(keystore.load()))) {
throw new ConfigException("Keystore DistinguishedName or SubjectAltNames do not match");
}
}
@@ -287,17 +313,21 @@ private SecurityStore createTruststore(String type, String path, Password passwo
this.keyPassword = keyPassword;
}
- KeyStore load() throws GeneralSecurityException, IOException {
- FileInputStream in = null;
- try {
+ /**
+ * Loads this keystore
+ * @return the keystore
+ * @throws KafkaException if the file could not be read or if the keystore could not be loaded
+ * using the specified configs (e.g. if the password or keystore type is invalid)
+ */
+ KeyStore load() {
+ try (FileInputStream in = new FileInputStream(path)) {
KeyStore ks = KeyStore.getInstance(type);
- in = new FileInputStream(path);
// If a password is not set access to the truststore is still available, but integrity checking is disabled.
char[] passwordChars = password != null ? password.value().toCharArray() : null;
ks.load(in, passwordChars);
return ks;
- } finally {
- if (in != null) in.close();
+ } catch (GeneralSecurityException | IOException e) {
+ throw new KafkaException("Failed to load SSL keystore " + path + " of type " + type, e);
}
}
}
@@ -307,16 +337,16 @@ KeyStore load() throws GeneralSecurityException, IOException {
* The validator checks that a successful handshake can be performed using the keystore and
* truststore configured on this SslFactory.
*/
- static class SSLConfigValidatorEngine {
+ private static class SSLConfigValidatorEngine {
private static final ByteBuffer EMPTY_BUF = ByteBuffer.allocate(0);
private final SSLEngine sslEngine;
private SSLEngineResult handshakeResult;
private ByteBuffer appBuffer;
private ByteBuffer netBuffer;
- static void validate(SslFactory sslFactory, SSLContext sslContext) throws SSLException {
- SSLConfigValidatorEngine clientEngine = new SSLConfigValidatorEngine(sslFactory, sslContext, Mode.CLIENT);
- SSLConfigValidatorEngine serverEngine = new SSLConfigValidatorEngine(sslFactory, sslContext, Mode.SERVER);
+ static void validate(SslFactory sslFactory, SSLContext clientSslContext, SSLContext serverSslContext) throws SSLException {
+ SSLConfigValidatorEngine clientEngine = new SSLConfigValidatorEngine(sslFactory, clientSslContext, Mode.CLIENT);
+ SSLConfigValidatorEngine serverEngine = new SSLConfigValidatorEngine(sslFactory, serverSslContext, Mode.SERVER);
try {
clientEngine.beginHandshake();
serverEngine.beginHandshake();
diff --git a/clients/src/main/java/org/apache/kafka/common/security/token/delegation/DelegationToken.java b/clients/src/main/java/org/apache/kafka/common/security/token/delegation/DelegationToken.java
index 05ccbda2fe6..a1e2372d7a0 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/token/delegation/DelegationToken.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/token/delegation/DelegationToken.java
@@ -16,11 +16,16 @@
*/
package org.apache.kafka.common.security.token.delegation;
-import org.apache.kafka.common.utils.Base64;
+import org.apache.kafka.common.annotation.InterfaceStability;
-import java.nio.ByteBuffer;
import java.util.Arrays;
+import java.util.Base64;
+/**
+ * A class representing a delegation token.
+ *
+ */
+@InterfaceStability.Evolving
public class DelegationToken {
private TokenInformation tokenInformation;
private byte[] hmac;
@@ -39,11 +44,7 @@ public TokenInformation tokenInfo() {
}
public String hmacAsBase64String() {
- return Base64.encoder().encodeToString(hmac);
- }
-
- public ByteBuffer hmacBuffer() {
- return ByteBuffer.wrap(hmac);
+ return Base64.getEncoder().encodeToString(hmac);
}
@Override
diff --git a/clients/src/main/java/org/apache/kafka/common/security/token/delegation/TokenInformation.java b/clients/src/main/java/org/apache/kafka/common/security/token/delegation/TokenInformation.java
index 1d500d21eef..ffd2af3f1c2 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/token/delegation/TokenInformation.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/token/delegation/TokenInformation.java
@@ -16,11 +16,17 @@
*/
package org.apache.kafka.common.security.token.delegation;
+import org.apache.kafka.common.annotation.InterfaceStability;
import org.apache.kafka.common.security.auth.KafkaPrincipal;
import java.util.ArrayList;
import java.util.Collection;
+/**
+ * A class representing a delegation token details.
+ *
+ */
+@InterfaceStability.Evolving
public class TokenInformation {
private KafkaPrincipal owner;
diff --git a/clients/src/main/java/org/apache/kafka/common/security/token/delegation/DelegationTokenCache.java b/clients/src/main/java/org/apache/kafka/common/security/token/delegation/internal/DelegationTokenCache.java
similarity index 91%
rename from clients/src/main/java/org/apache/kafka/common/security/token/delegation/DelegationTokenCache.java
rename to clients/src/main/java/org/apache/kafka/common/security/token/delegation/internal/DelegationTokenCache.java
index 78575b81bb6..c05b7350b41 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/token/delegation/DelegationTokenCache.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/token/delegation/internal/DelegationTokenCache.java
@@ -15,12 +15,14 @@
* limitations under the License.
*/
-package org.apache.kafka.common.security.token.delegation;
+package org.apache.kafka.common.security.token.delegation.internal;
import org.apache.kafka.common.security.authenticator.CredentialCache;
import org.apache.kafka.common.security.scram.ScramCredential;
-import org.apache.kafka.common.security.scram.ScramCredentialUtils;
-import org.apache.kafka.common.security.scram.ScramMechanism;
+import org.apache.kafka.common.security.scram.internal.ScramCredentialUtils;
+import org.apache.kafka.common.security.scram.internal.ScramMechanism;
+import org.apache.kafka.common.security.token.delegation.DelegationToken;
+import org.apache.kafka.common.security.token.delegation.TokenInformation;
import java.util.Collection;
import java.util.HashMap;
diff --git a/clients/src/main/java/org/apache/kafka/common/security/token/delegation/DelegationTokenCredentialCallback.java b/clients/src/main/java/org/apache/kafka/common/security/token/delegation/internal/DelegationTokenCredentialCallback.java
similarity index 94%
rename from clients/src/main/java/org/apache/kafka/common/security/token/delegation/DelegationTokenCredentialCallback.java
rename to clients/src/main/java/org/apache/kafka/common/security/token/delegation/internal/DelegationTokenCredentialCallback.java
index 7490a3e91b1..294d7b1d3a0 100644
--- a/clients/src/main/java/org/apache/kafka/common/security/token/delegation/DelegationTokenCredentialCallback.java
+++ b/clients/src/main/java/org/apache/kafka/common/security/token/delegation/internal/DelegationTokenCredentialCallback.java
@@ -14,7 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.kafka.common.security.token.delegation;
+package org.apache.kafka.common.security.token.delegation.internal;
import org.apache.kafka.common.security.scram.ScramCredentialCallback;
diff --git a/clients/src/main/java/org/apache/kafka/common/serialization/Serdes.java b/clients/src/main/java/org/apache/kafka/common/serialization/Serdes.java
index d6b4d2da611..7825ad4e9c3 100644
--- a/clients/src/main/java/org/apache/kafka/common/serialization/Serdes.java
+++ b/clients/src/main/java/org/apache/kafka/common/serialization/Serdes.java
@@ -26,7 +26,7 @@
*/
public class Serdes {
- static protected class WrapperSerde<T> implements Serde<T> {
+ static public class WrapperSerde<T> implements Serde<T> {
final private Serializer<T> serializer;
final private Deserializer<T> deserializer;
diff --git a/clients/src/main/java/org/apache/kafka/common/serialization/StringDeserializer.java b/clients/src/main/java/org/apache/kafka/common/serialization/StringDeserializer.java
index c647c9b7332..0398a1b2cc1 100644
--- a/clients/src/main/java/org/apache/kafka/common/serialization/StringDeserializer.java
+++ b/clients/src/main/java/org/apache/kafka/common/serialization/StringDeserializer.java
@@ -34,7 +34,7 @@ public void configure(Map<String, ?> configs, boolean isKey) {
Object encodingValue = configs.get(propertyName);
if (encodingValue == null)
encodingValue = configs.get("deserializer.encoding");
- if (encodingValue != null && encodingValue instanceof String)
+ if (encodingValue instanceof String)
encoding = (String) encodingValue;
}
diff --git a/clients/src/main/java/org/apache/kafka/common/utils/Base64.java b/clients/src/main/java/org/apache/kafka/common/utils/Base64.java
deleted file mode 100644
index 3ab4900d79e..00000000000
--- a/clients/src/main/java/org/apache/kafka/common/utils/Base64.java
+++ /dev/null
@@ -1,320 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.kafka.common.utils;
-
-import java.lang.invoke.MethodHandle;
-import java.lang.invoke.MethodHandles;
-import java.lang.invoke.MethodType;
-
-/**
- * Temporary class in order to support Java 7 and Java 9. `DatatypeConverter` is not in the base module of Java 9
- * and `java.util.Base64` was only introduced in Java 8.
- */
-public final class Base64 {
-
- private static final Factory FACTORY;
-
- static {
- if (Java.IS_JAVA8_COMPATIBLE)
- FACTORY = new Java8Factory();
- else
- FACTORY = new Java7Factory();
- }
-
- private Base64() {}
-
- public static Encoder encoder() {
- return FACTORY.encoder();
- }
-
- public static Encoder urlEncoderNoPadding() {
- return FACTORY.urlEncoderNoPadding();
- }
-
- public static Decoder decoder() {
- return FACTORY.decoder();
- }
-
- public static Decoder urlDecoder() {
- return FACTORY.urlDecoder();
- }
-
- /* Contains a subset of methods from java.util.Base64.Encoder (introduced in Java 8) */
- public interface Encoder {
- String encodeToString(byte[] bytes);
- }
-
- /* Contains a subset of methods from java.util.Base64.Decoder (introduced in Java 8) */
- public interface Decoder {
- byte[] decode(String string);
- }
-
- private interface Factory {
- Encoder urlEncoderNoPadding();
- Encoder encoder();
- Decoder urlDecoder();
- Decoder decoder();
- }
-
- private static class Java8Factory implements Factory {
-
- // Static final MethodHandles are optimised better by HotSpot
- private static final MethodHandle URL_ENCODE_NO_PADDING;
- private static final MethodHandle ENCODE;
- private static final MethodHandle URL_DECODE;
- private static final MethodHandle DECODE;
-
- private static final Encoder URL_ENCODER_NO_PADDING;
- private static final Encoder ENCODER;
- private static final Decoder URL_DECODER;
- private static final Decoder DECODER;
-
- static {
- try {
- Class<?> base64Class = Class.forName("java.util.Base64");
-
- MethodHandles.Lookup lookup = MethodHandles.publicLookup();
-
- Class<?> juEncoderClass = Class.forName("java.util.Base64$Encoder");
-
- MethodHandle getEncoder = lookup.findStatic(base64Class, "getEncoder",
- MethodType.methodType(juEncoderClass));
- Object juEncoder;
- try {
- juEncoder = getEncoder.invoke();
- } catch (Throwable throwable) {
- // Invoked method doesn't throw checked exceptions, so safe to cast
- throw (RuntimeException) throwable;
- }
- MethodHandle encode = lookup.findVirtual(juEncoderClass, "encodeToString",
- MethodType.methodType(String.class, byte[].class));
- ENCODE = encode.bindTo(juEncoder);
-
-
- MethodHandle getUrlEncoder = lookup.findStatic(base64Class, "getUrlEncoder",
- MethodType.methodType(juEncoderClass));
- Object juUrlEncoderNoPassing;
- try {
- juUrlEncoderNoPassing = lookup.findVirtual(juEncoderClass, "withoutPadding",
- MethodType.methodType(juEncoderClass)).invoke(getUrlEncoder.invoke());
- } catch (Throwable throwable) {
- // Invoked method doesn't throw checked exceptions, so safe to cast
- throw (RuntimeException) throwable;
- }
- URL_ENCODE_NO_PADDING = encode.bindTo(juUrlEncoderNoPassing);
-
- Class<?> juDecoderClass = Class.forName("java.util.Base64$Decoder");
- MethodHandle getDecoder = lookup.findStatic(base64Class, "getDecoder",
- MethodType.methodType(juDecoderClass));
- MethodHandle decode = lookup.findVirtual(juDecoderClass, "decode",
- MethodType.methodType(byte[].class, String.class));
- try {
- DECODE = decode.bindTo(getDecoder.invoke());
- } catch (Throwable throwable) {
- // Invoked method doesn't throw checked exceptions, so safe to cast
- throw (RuntimeException) throwable;
- }
-
- MethodHandle getUrlDecoder = lookup.findStatic(base64Class, "getUrlDecoder",
- MethodType.methodType(juDecoderClass));
- MethodHandle urlDecode = lookup.findVirtual(juDecoderClass, "decode",
- MethodType.methodType(byte[].class, String.class));
- try {
- URL_DECODE = urlDecode.bindTo(getUrlDecoder.invoke());
- } catch (Throwable throwable) {
- // Invoked method doesn't throw checked exceptions, so safe to cast
- throw (RuntimeException) throwable;
- }
-
- URL_ENCODER_NO_PADDING = new Encoder() {
- @Override
- public String encodeToString(byte[] bytes) {
- try {
- return (String) URL_ENCODE_NO_PADDING.invokeExact(bytes);
- } catch (Throwable throwable) {
- // Invoked method doesn't throw checked exceptions, so safe to cast
- throw (RuntimeException) throwable;
- }
- }
- };
-
- ENCODER = new Encoder() {
- @Override
- public String encodeToString(byte[] bytes) {
- try {
- return (String) ENCODE.invokeExact(bytes);
- } catch (Throwable throwable) {
- // Invoked method doesn't throw checked exceptions, so safe to cast
- throw (RuntimeException) throwable;
- }
- }
- };
-
- URL_DECODER = new Decoder() {
- @Override
- public byte[] decode(String string) {
- try {
- return (byte[]) URL_DECODE.invokeExact(string);
- } catch (Throwable throwable) {
- // Invoked method doesn't throw checked exceptions, so safe to cast
- throw (RuntimeException) throwable;
- }
- }
- };
-
- DECODER = new Decoder() {
- @Override
- public byte[] decode(String string) {
- try {
- return (byte[]) DECODE.invokeExact(string);
- } catch (Throwable throwable) {
- // Invoked method doesn't throw checked exceptions, so safe to cast
- throw (RuntimeException) throwable;
- }
- }
- };
-
- } catch (ReflectiveOperationException e) {
- // Should never happen
- throw new RuntimeException(e);
- }
- }
-
- @Override
- public Encoder urlEncoderNoPadding() {
- return URL_ENCODER_NO_PADDING;
- }
-
- @Override
- public Encoder encoder() {
- return ENCODER;
- }
-
- @Override
- public Decoder decoder() {
- return DECODER;
- }
-
- @Override
- public Decoder urlDecoder() {
- return URL_DECODER;
- }
- }
-
- private static class Java7Factory implements Factory {
-
- // Static final MethodHandles are optimised better by HotSpot
- private static final MethodHandle PRINT;
- private static final MethodHandle PARSE;
-
- static {
- try {
- Class<?> cls = Class.forName("javax.xml.bind.DatatypeConverter");
- MethodHandles.Lookup lookup = MethodHandles.publicLookup();
- PRINT = lookup.findStatic(cls, "printBase64Binary", MethodType.methodType(String.class,
- byte[].class));
- PARSE = lookup.findStatic(cls, "parseBase64Binary", MethodType.methodType(byte[].class,
- String.class));
- } catch (ReflectiveOperationException e) {
- // Should never happen
- throw new RuntimeException(e);
- }
- }
-
- public static final Encoder URL_ENCODER_NO_PADDING = new Encoder() {
-
- @Override
- public String encodeToString(byte[] bytes) {
- if (bytes.length == 0)
- return "";
- String base64EncodedUUID = Java7Factory.encodeToString(bytes);
- // Convert to URL safe variant by replacing + and / with - and _ respectively.
- String urlSafeBase64EncodedUUID = base64EncodedUUID.replace("+", "-").replace("/", "_");
- // Remove any "=" or "==" padding at the end.
- // Note that length will be at least 4 here.
- int index = urlSafeBase64EncodedUUID.indexOf('=', urlSafeBase64EncodedUUID.length() - 2);
- return index > 0 ? urlSafeBase64EncodedUUID.substring(0, index) : urlSafeBase64EncodedUUID;
- }
-
- };
-
- public static final Encoder ENCODER = new Encoder() {
- @Override
- public String encodeToString(byte[] bytes) {
- return Java7Factory.encodeToString(bytes);
- }
- };
-
- public static final Decoder DECODER = new Decoder() {
- @Override
- public byte[] decode(String string) {
- try {
- return (byte[]) PARSE.invokeExact(string);
- } catch (Throwable throwable) {
- // Invoked method doesn't throw checked exceptions, so safe to cast
- throw (RuntimeException) throwable;
- }
- }
- };
-
- public static final Decoder URL_DECODER = new Decoder() {
- @Override
- public byte[] decode(String string) {
- try {
- // Convert from URL safe variant by replacing - and _ with + and / respectively,
- // and append "=" or "==" padding; then decode.
- String unpadded = string.replace("-", "+").replace("_", "/");
- int padLength = 4 - (unpadded.length() & 3);
- return (byte[]) PARSE.invokeExact(padLength > 2 ? unpadded : unpadded + "==".substring(0, padLength));
- } catch (Throwable throwable) {
- // Invoked method doesn't throw checked exceptions, so safe to cast
- throw (RuntimeException) throwable;
- }
- }
- };
-
- private static String encodeToString(byte[] bytes) {
- try {
- return (String) PRINT.invokeExact(bytes);
- } catch (Throwable throwable) {
- // Invoked method doesn't throw checked exceptions, so safe to cast
- throw (RuntimeException) throwable;
- }
- }
-
- @Override
- public Encoder urlEncoderNoPadding() {
- return URL_ENCODER_NO_PADDING;
- }
-
- @Override
- public Encoder encoder() {
- return ENCODER;
- }
-
- @Override
- public Decoder urlDecoder() {
- return URL_DECODER;
- }
-
- @Override
- public Decoder decoder() {
- return DECODER;
- }
- }
-}
diff --git a/clients/src/main/java/org/apache/kafka/common/utils/Java.java b/clients/src/main/java/org/apache/kafka/common/utils/Java.java
index fb8cafab4b7..d3ada57c21f 100644
--- a/clients/src/main/java/org/apache/kafka/common/utils/Java.java
+++ b/clients/src/main/java/org/apache/kafka/common/utils/Java.java
@@ -38,7 +38,6 @@ static Version parseVersion(String versionString) {
// Having these as static final provides the best opportunity for compilar optimization
public static final boolean IS_JAVA9_COMPATIBLE = VERSION.isJava9Compatible();
- public static final boolean IS_JAVA8_COMPATIBLE = VERSION.isJava8Compatible();
public static boolean isIbmJdk() {
return System.getProperty("java.vendor").contains("IBM");
@@ -65,10 +64,6 @@ boolean isJava9Compatible() {
return majorVersion >= 9;
}
- // Package private for testing
- boolean isJava8Compatible() {
- return majorVersion > 1 || (majorVersion == 1 && minorVersion >= 8);
- }
}
}
diff --git a/clients/src/main/java/org/apache/kafka/common/utils/LoggingSignalHandler.java b/clients/src/main/java/org/apache/kafka/common/utils/LoggingSignalHandler.java
new file mode 100644
index 00000000000..fbe67366ed3
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/utils/LoggingSignalHandler.java
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.utils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationHandler;
+import java.lang.reflect.Method;
+import java.lang.reflect.Proxy;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+public class LoggingSignalHandler {
+
+ private static final Logger log = LoggerFactory.getLogger(LoggingSignalHandler.class);
+
+ private final Constructor<?> signalConstructor;
+ private final Class<?> signalHandlerClass;
+ private final Class<?> signalClass;
+ private final Method signalHandleMethod;
+ private final Method signalGetNameMethod;
+ private final Method signalHandlerHandleMethod;
+
+ /**
+ * Create an instance of this class.
+ *
+ * @throws ReflectiveOperationException if the underlying API has changed in an incompatible manner.
+ */
+ public LoggingSignalHandler() throws ReflectiveOperationException {
+ signalClass = Class.forName("sun.misc.Signal");
+ signalConstructor = signalClass.getConstructor(String.class);
+ signalHandlerClass = Class.forName("sun.misc.SignalHandler");
+ signalHandlerHandleMethod = signalHandlerClass.getMethod("handle", signalClass);
+ signalHandleMethod = signalClass.getMethod("handle", signalClass, signalHandlerClass);
+ signalGetNameMethod = signalClass.getMethod("getName");
+ }
+
+ /**
+ * Register signal handler to log termination due to SIGTERM, SIGHUP and SIGINT (control-c). This method
+ * does not currently work on Windows.
+ *
+ * @implNote sun.misc.Signal and sun.misc.SignalHandler are described as "not encapsulated" in
+ * http://openjdk.java.net/jeps/260. However, they are not available in the compile classpath if the `--release`
+ * flag is used. As a workaround, we rely on reflection.
+ */
+ public void register() throws ReflectiveOperationException {
+ Map<String, Object> jvmSignalHandlers = new ConcurrentHashMap<>();
+ register("TERM", jvmSignalHandlers);
+ register("INT", jvmSignalHandlers);
+ register("HUP", jvmSignalHandlers);
+ }
+
+ private Object createSignalHandler(final Map<String, Object> jvmSignalHandlers) {
+ InvocationHandler invocationHandler = new InvocationHandler() {
+
+ private String getName(Object signal) throws ReflectiveOperationException {
+ return (String) signalGetNameMethod.invoke(signal);
+ }
+
+ private void handle(Object signalHandler, Object signal) throws ReflectiveOperationException {
+ signalHandlerHandleMethod.invoke(signalHandler, signal);
+ }
+
+ @Override
+ public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
+ Object signal = args[0];
+ log.info("Terminating process due to signal {}", signal);
+ Object handler = jvmSignalHandlers.get(getName(signal));
+ if (handler != null)
+ handle(handler, signal);
(This diff was longer than 20,000 lines, and has been truncated...)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
> Adding checks on "version" field for tools using it
> ---------------------------------------------------
>
> Key: KAFKA-5919
> URL: https://issues.apache.org/jira/browse/KAFKA-5919
> Project: Kafka
> Issue Type: Bug
> Components: tools
> Reporter: Paolo Patierno
> Assignee: Paolo Patierno
> Priority: Minor
> Fix For: 2.0.0
>
>
> Hi,
> the kafka-delete-records script allows user to pass information about records to delete through a JSON file. Such file, as described in the command help, is made by a "partitions" array and a "version" field. Reading [KIP-107|https://cwiki.apache.org/confluence/display/KAFKA/KIP-107%3A+Add+purgeDataBefore%28%29+API+in+AdminClient] and the DeleteRecords API (Key: 21) description it's not clear what such field is and even it's not used at all (in the current implementation).
> It turned out that the field is for having backward compatibility in the future where the JSON format could change. This JIRA is about adding more checks on the "version" field having it not mandatory but assuming the earliest version (current 1) if it's omitted from the JSON file.
> The same for the kafka-reassign-partitions which has a topics-to-move JSON file as input (used with --generate option) and the partitions-to-move.json (used with --execute option). In both cases the same logic can be applied as above.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)