You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Oleg Khoruzhenko (Jira)" <ji...@apache.org> on 2020/03/04 18:27:00 UTC
[jira] [Comment Edited] (KAFKA-9568) Kstreams
APPLICATION_SERVER_CONFIG is not updated with static membership
[ https://issues.apache.org/jira/browse/KAFKA-9568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17051503#comment-17051503 ]
Oleg Khoruzhenko edited comment on KAFKA-9568 at 3/4/20, 6:26 PM:
------------------------------------------------------------------
We run Kafka Streams application with _Static Membership_ and uses _Interactive Queries_ functionality. Say we have two instances A and B running on HOST_A and HOST_B correspondingly. Then A migrated to another HOST_C. The APPLICATION_SERVER_CONFIG changes to the IP of HOST_C. Issue appears when you need to query remote store on the instance A. In that case {{StreamsMetadata}} doesn't get updated and returns ip address of HOST_A, even though A is already migrated to HOST_C.
Let me try to explain the case we are dealing with on an example from kafka-streams-example project.
# I added static group membership to WordCountInteractiveQueriesExample.java see [https://github.com/o0oxid/kafka-streams-examples/commit/54cb74f40ef3c659588dbedc496bc456123db215]
# Started confluent-platform:
{noformat}
48a1ede4f588 confluentinc/cp-enterprise-kafka:5.4.0 "/etc/confluent/dock…" 2 hours ago Up 2 hours 0.0.0.0:9092->9092/tcp, 0.0.0.0:29092->29092/tcp kafka-streams-examples_kafka_1
bd6cf95c56b7 confluentinc/cp-zookeeper:5.4.0 "/etc/confluent/dock…" 2 hours ago Up 2 hours 2181/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:32181->32181/tcp kafka-streams-examples_zookeeper_1
{noformat}
# Run first application in static group membership mode - group.instance.id 'client-1' and port '7070'
{noformat}
> java -cp target/kafka-streams-examples-5.4.0-standalone.jar io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesExample 7070 localhost:9092 client-1
[2020-03-04 10:38:31,071] WARN [main] The configuration 'admin.retries' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2020-03-04 10:38:31,072] WARN [main] The configuration 'admin.retry.backoff.ms' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
Mar 04, 2020 10:38:31 AM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime
WARNING: A provider io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesRestService registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesRestService will be ignored.
[2020-03-04 10:38:49,717] WARN [interactive-queries-example-client-StreamThread-1] [Consumer instanceId=client-1-1, clientId=interactive-queries-example-client-StreamThread-1-consumer, groupId=interactive-queries-example] The following subscribed topics are not assigned to any members: [interactive-queries-example-word-count-repartition] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
{noformat}
# Run second application in different terminal - group.instance.id 'client-2' and port '7071'
{noformat}
> java -cp target/kafka-streams-examples-5.4.0-standalone.jar io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesExample 7071 localhost:9092 client-2
[2020-03-04 10:39:05,188] WARN [main] The configuration 'admin.retries' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2020-03-04 10:39:05,189] WARN [main] The configuration 'admin.retry.backoff.ms' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
Mar 04, 2020 10:39:05 AM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime
WARNING: A provider io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesRestService registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesRestService will be ignored.
{noformat}
Querying StreamsMetadata returns two instances 'localhost:7070' and 'localhost:7071':
{noformat}
curl -s localhost:7070/state/instances | jq
[
{
"host": "localhost",
"port": 7070,
"storeNames": [
"windowed-word-count"
]
},
{
"host": "localhost",
"port": 7071,
"storeNames": [
"word-count"
]
}
]
{noformat}
Now let's re-run the second application with same group.instance.id 'client-2', but different port '7072'
{noformat}
>java -cp target/kafka-streams-examples-5.4.0-standalone.jar io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesExample 7072 localhost:9092 client-2
[2020-03-04 10:39:05,188] WARN [main] The configuration 'admin.retries' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2020-03-04 10:39:05,189] WARN [main] The configuration 'admin.retry.backoff.ms' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
Mar 04, 2020 10:39:05 AM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime
WARNING: A provider io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesRestService registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesRestService will be ignored.
{noformat}
Querying instance metadata returns stale data. Note the port is the same '7071', while application is running on '7072':
{noformat}
> curl -s localhost:7070/state/instances | jq
[
{
"host": "localhost",
"port": 7070,
"storeNames": [
"windowed-word-count"
]
},
{
"host": "localhost",
"port": 7071,
"storeNames": [
"word-count"
]
}
]
{noformat}
So now it's impossible to perform interactive queries against remote machine because metadata points to the old port '7071'.
Issue is not reproducible with dynamic membership.
was (Author: o0oxid):
We run Kafka Streams application with _Static Membership_ and uses _Interactive Queries_ functionality. Say we have two instances A and B running on HOST_A and HOST_B correspondingly. Then A migrated to another HOST_C. The APPLICATION_SERVER_CONFIG changes to the IP of HOST_C. Issue appears when you need to query remote store on the instance A. In that case {{StreamsMetadata}} doesn't get updated and returns ip address of HOST_A, even though A is already migrated to HOST_C.
Let me try to explain the case we are dealing with on example from kafka-streams-example project.
# I added static group membership to WordCountInteractiveQueriesExample.java see [https://github.com/o0oxid/kafka-streams-examples/commit/54cb74f40ef3c659588dbedc496bc456123db215]
# Started confluent-platform: {noformat}
48a1ede4f588 confluentinc/cp-enterprise-kafka:5.4.0 "/etc/confluent/dock…" 2 hours ago Up 2 hours 0.0.0.0:9092->9092/tcp, 0.0.0.0:29092->29092/tcp kafka-streams-examples_kafka_1
bd6cf95c56b7 confluentinc/cp-zookeeper:5.4.0 "/etc/confluent/dock…" 2 hours ago Up 2 hours 2181/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:32181->32181/tcp kafka-streams-examples_zookeeper_1
{noformat}
# Run first application in static group membership mode - group.instance.id 'client-1' and port '7070'
{noformat}
> java -cp target/kafka-streams-examples-5.4.0-standalone.jar io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesExample 7070 localhost:9092 client-1
[2020-03-04 10:38:31,071] WARN [main] The configuration 'admin.retries' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2020-03-04 10:38:31,072] WARN [main] The configuration 'admin.retry.backoff.ms' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
Mar 04, 2020 10:38:31 AM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime
WARNING: A provider io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesRestService registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesRestService will be ignored.
[2020-03-04 10:38:49,717] WARN [interactive-queries-example-client-StreamThread-1] [Consumer instanceId=client-1-1, clientId=interactive-queries-example-client-StreamThread-1-consumer, groupId=interactive-queries-example] The following subscribed topics are not assigned to any members: [interactive-queries-example-word-count-repartition] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
{noformat}
# Run second application in different terminal - group.instance.id 'client-2' and port '7071'
{noformat}
> java -cp target/kafka-streams-examples-5.4.0-standalone.jar io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesExample 7071 localhost:9092 client-2
[2020-03-04 10:39:05,188] WARN [main] The configuration 'admin.retries' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2020-03-04 10:39:05,189] WARN [main] The configuration 'admin.retry.backoff.ms' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
Mar 04, 2020 10:39:05 AM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime
WARNING: A provider io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesRestService registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesRestService will be ignored.
{noformat}
Querying StreamsMetadata returns two instances 'localhost:7070' and 'localhost:7071':
{noformat}
curl -s localhost:7070/state/instances | jq
[
{
"host": "localhost",
"port": 7070,
"storeNames": [
"windowed-word-count"
]
},
{
"host": "localhost",
"port": 7071,
"storeNames": [
"word-count"
]
}
]
{noformat}
Now let's re-run the second application with same group.instance.id 'client-2', but different port '7072'
{noformat}
>java -cp target/kafka-streams-examples-5.4.0-standalone.jar io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesExample 7072 localhost:9092 client-2
[2020-03-04 10:39:05,188] WARN [main] The configuration 'admin.retries' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2020-03-04 10:39:05,189] WARN [main] The configuration 'admin.retry.backoff.ms' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
Mar 04, 2020 10:39:05 AM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime
WARNING: A provider io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesRestService registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesRestService will be ignored.
{noformat}
Querying instance metadata returns stale data. Note the port is the same '7071', while application is running on '7072':
{noformat}
> curl -s localhost:7070/state/instances | jq
[
{
"host": "localhost",
"port": 7070,
"storeNames": [
"windowed-word-count"
]
},
{
"host": "localhost",
"port": 7071,
"storeNames": [
"word-count"
]
}
]
{noformat}
So now it's impossible to perform interactive queries against remote machine because metadata points to the old port '7071'.
Issue is not reproducible with dynamic membership.
> Kstreams APPLICATION_SERVER_CONFIG is not updated with static membership
> ------------------------------------------------------------------------
>
> Key: KAFKA-9568
> URL: https://issues.apache.org/jira/browse/KAFKA-9568
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 2.4.0
> Reporter: David J. Garcia
> Priority: Major
>
> A kstreams application with static membership, and StreamsConfg.APPLICATION_SERVER_CONFIG set, will NOT update old server config upon restart of application on new host.
> Steps to reproduce:
>
> # start two kstreams applications (with same consumer group) and enable static membership (and set application server config to <ip-addr>:<port>)
> # kill one of the applications and restart it on a new host(with new ip) before timeout ends (so that rebalancing doesn't occur).
> # the other kstreams application will now have an invalid application_server_config
> Possible fix:
> If an application restarts with a new host/identity..etc, it could trigger a "light-rebalance" where the other applications in the consumer group don't change partition assignments ,but instead just get their configuration updated.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)