You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "mshijin-ksolves (via GitHub)" <gi...@apache.org> on 2026/01/15 10:14:52 UTC
[PR] Kafka 19979 : Improve Kafka Streams config files [kafka]
mshijin-ksolves opened a new pull request, #21308:
URL: https://github.com/apache/kafka/pull/21308
**Ticket:-** KAFKA:19979 : Improve Kafka Streams config files
**Description**: Create a Kafka Streams config file for the shipped Kafka binary.
As per the requirement of the ticket mentioned in its description, we have created a ** config/streams.properties** .
This file contains important configuration properties along with their default values.
It simplifies development by avoiding the need to repeatedly define the same Kafka Streams configurations across multiple stream files.
Developers can directly load this properties file in their code and override only the required configurations based on their use case.
Ticket link : https://issues.apache.org/jira/browse/KAFKA-19979
@Evan Zhou Please review this file if possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19979: Improve Kafka Streams config files [kafka]
Posted by "majialoong (via GitHub)" <gi...@apache.org>.
majialoong commented on PR #21308:
URL: https://github.com/apache/kafka/pull/21308#issuecomment-4988919095
@peterzdhuang No problem at all! I'm currently short on bandwidth for this anyway, so you taking the lead is perfect. Thanks for being so proactive!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19979: Improve Kafka Streams config files [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #21308:
URL: https://github.com/apache/kafka/pull/21308#issuecomment-3815194794
@ezhou413 -- Can you help reviewing this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19979: Improve Kafka Streams config files [kafka]
Posted by "peterzdhuang (via GitHub)" <gi...@apache.org>.
peterzdhuang commented on PR #21308:
URL: https://github.com/apache/kafka/pull/21308#issuecomment-4973905455
Hi @majialoong are you still working on this PR?
I would be happy to pick this up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19979: Improve Kafka Streams config files [kafka]
Posted by "peterzdhuang (via GitHub)" <gi...@apache.org>.
peterzdhuang commented on PR #21308:
URL: https://github.com/apache/kafka/pull/21308#issuecomment-4981345418
@majialoong Hi sorry about that, I got so hasty, I should've definitely asked first, that's my bad.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19979: Improve Kafka Streams config files [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #21308:
URL: https://github.com/apache/kafka/pull/21308#issuecomment-4271649536
@mshijin-ksolves -- Any updates on this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19979: Improve Kafka Streams config files [kafka]
Posted by "github-actions[bot] (via GitHub) via jira" <ji...@kafka.apache.org>.
github-actions[bot] closed pull request #21308: KAFKA-19979: Improve Kafka Streams config files
URL: https://github.com/apache/kafka/pull/21308
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19979: Improve Kafka Streams config files [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21308:
URL: https://github.com/apache/kafka/pull/21308#issuecomment-6183876477
This PR is being marked as stale since it has not had any activity in 90 days. If you
would like to keep this PR alive, please leave a comment asking for a review. If the PR has
merge conflicts, update it with the latest from the base branch.
<p>
If you are having difficulty finding a reviewer, please reach out on the
[mailing list](https://kafka.apache.org/contact).
<p>
If this PR is no longer valid or desired, please feel free to close it. If no activity
occurs in the next 30 days, it will be automatically closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19979: Improve Kafka Streams config files [kafka]
Posted by "github-actions[bot] (via GitHub) via jira" <ji...@kafka.apache.org>.
github-actions[bot] commented on PR #21308:
URL: https://github.com/apache/kafka/pull/21308#issuecomment-6888619577
This PR has been closed since it has not had any activity in 120 days. If you feel like this
was a mistake, or you would like to continue working on it, please feel free to re-open the
PR and ask for a review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19979: Improve Kafka Streams config files [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21308:
URL: https://github.com/apache/kafka/pull/21308#issuecomment-3788013298
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19979: Improve Kafka Streams config files [kafka]
Posted by "majialoong (via GitHub)" <gi...@apache.org>.
majialoong commented on PR #21308:
URL: https://github.com/apache/kafka/pull/21308#issuecomment-4577691622
Hi @mshijin-ksolves , thanks for working on this.
I noticed this PR has been inactive for a while after the review comments. Are you still planning to continue with it?
If not, I’d be happy to help by picking up KAFKA-19979, addressing the remaining comments, and crediting/referencing this PR in a new one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19979: Improve Kafka Streams config files [kafka]
Posted by "ezhou413 (via GitHub)" <gi...@apache.org>.
ezhou413 commented on code in PR #21308:
URL: https://github.com/apache/kafka/pull/21308#discussion_r2757161845
##########
config/streams.properties:
##########
@@ -0,0 +1,92 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+############################# Kafka Streams Configuration #############################
+
+# A unique name for the Kafka Streams application.
+# Used as consumer group id and for internal state topics.
+# REQUIRED (no default)
+application.id=my-streams-app
+
+# Comma-separated list of Kafka brokers.
+# REQUIRED (no default)
+bootstrap.servers=<localhost:9092>
+############################# Stream Threads #############################
+
+# Number of stream processing threads.
+# Default: 1
+num.stream.threads=1
+############################# Default SerDes #############################
Review Comment:
Serdes do not have default values. Could you comment this section out, and list possible values for these configs?
##########
config/streams.properties:
##########
@@ -0,0 +1,92 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+############################# Kafka Streams Configuration #############################
+
+# A unique name for the Kafka Streams application.
+# Used as consumer group id and for internal state topics.
+# REQUIRED (no default)
+application.id=my-streams-app
+
+# Comma-separated list of Kafka brokers.
+# REQUIRED (no default)
+bootstrap.servers=<localhost:9092>
Review Comment:
Could you remove the `<>` brackets?
##########
config/streams.properties:
##########
@@ -0,0 +1,92 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+############################# Kafka Streams Configuration #############################
+
+# A unique name for the Kafka Streams application.
+# Used as consumer group id and for internal state topics.
+# REQUIRED (no default)
+application.id=my-streams-app
+
+# Comma-separated list of Kafka brokers.
+# REQUIRED (no default)
+bootstrap.servers=<localhost:9092>
+############################# Stream Threads #############################
+
+# Number of stream processing threads.
+# Default: 1
+num.stream.threads=1
+############################# Default SerDes #############################
+
+# Default key serializer/deserializer.
+# Default: org.apache.kafka.common.serialization.ByteArraySerde
+default.key.serde=org.apache.kafka.common.serialization.StringSerde
+
+# Default value serializer/deserializer.
+# Default: org.apache.kafka.common.serialization.ByteArraySerde
+default.value.serde=org.apache.kafka.common.serialization.StringSerde
+############################# Processing Guarantees #############################
+
+# Processing guarantee.
+# Values: at_least_once, exactly_once_v2
+# Default: at_least_once
+processing.guarantee=at_least_once
+
+# Commit interval for state and offsets (ms).
+# Default: 30000
+commit.interval.ms=30000
+############################# State Store #############################
+
+# Directory for local state stores.
+# Default: /tmp/kafka-streams
+state.dir=/tmp/kafka-streams
+
+# Maximum number of standby replicas.
+# Default: 0
+num.standby.replicas=0
+############################# Caching #############################
+
+# Maximum memory used for buffering records (bytes).
+# Default: 10485760 (10 MB)
+cache.max.bytes.buffering=10485760
+############################# Internal Topics #############################
+
+# Replication factor for internal topics (changelog, repartition).
+# Default: -1 (uses broker default)
+replication.factor=-1
+############################# Topology Optimization #############################
+
+# Enable topology optimization.
+# Values: none, all
+# Default: none
+topology.optimization=none
Review Comment:
Please list the possible values for this in the comments above
##########
config/streams.properties:
##########
@@ -0,0 +1,92 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+############################# Kafka Streams Configuration #############################
+
+# A unique name for the Kafka Streams application.
+# Used as consumer group id and for internal state topics.
+# REQUIRED (no default)
+application.id=my-streams-app
+
+# Comma-separated list of Kafka brokers.
+# REQUIRED (no default)
+bootstrap.servers=<localhost:9092>
+############################# Stream Threads #############################
+
+# Number of stream processing threads.
+# Default: 1
+num.stream.threads=1
+############################# Default SerDes #############################
+
+# Default key serializer/deserializer.
+# Default: org.apache.kafka.common.serialization.ByteArraySerde
+default.key.serde=org.apache.kafka.common.serialization.StringSerde
+
+# Default value serializer/deserializer.
+# Default: org.apache.kafka.common.serialization.ByteArraySerde
+default.value.serde=org.apache.kafka.common.serialization.StringSerde
+############################# Processing Guarantees #############################
+
+# Processing guarantee.
+# Values: at_least_once, exactly_once_v2
+# Default: at_least_once
+processing.guarantee=at_least_once
+
+# Commit interval for state and offsets (ms).
+# Default: 30000
+commit.interval.ms=30000
Review Comment:
The documentation for this config specifies different defaults depending on the configured `processing.guarantee`. Could you mention this in the comments above the config?
https://github.com/apache/kafka/blob/5498eedf92a016e6838bce5383042bda1c798d76/streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java#L475-L478
##########
config/streams.properties:
##########
@@ -0,0 +1,92 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+############################# Kafka Streams Configuration #############################
+
+# A unique name for the Kafka Streams application.
+# Used as consumer group id and for internal state topics.
+# REQUIRED (no default)
+application.id=my-streams-app
+
+# Comma-separated list of Kafka brokers.
+# REQUIRED (no default)
+bootstrap.servers=<localhost:9092>
+############################# Stream Threads #############################
+
+# Number of stream processing threads.
+# Default: 1
+num.stream.threads=1
+############################# Default SerDes #############################
+
+# Default key serializer/deserializer.
+# Default: org.apache.kafka.common.serialization.ByteArraySerde
+default.key.serde=org.apache.kafka.common.serialization.StringSerde
+
+# Default value serializer/deserializer.
+# Default: org.apache.kafka.common.serialization.ByteArraySerde
+default.value.serde=org.apache.kafka.common.serialization.StringSerde
+############################# Processing Guarantees #############################
+
+# Processing guarantee.
+# Values: at_least_once, exactly_once_v2
+# Default: at_least_once
+processing.guarantee=at_least_once
+
+# Commit interval for state and offsets (ms).
+# Default: 30000
+commit.interval.ms=30000
+############################# State Store #############################
+
+# Directory for local state stores.
+# Default: /tmp/kafka-streams
+state.dir=/tmp/kafka-streams
+
+# Maximum number of standby replicas.
+# Default: 0
+num.standby.replicas=0
+############################# Caching #############################
+
+# Maximum memory used for buffering records (bytes).
+# Default: 10485760 (10 MB)
+cache.max.bytes.buffering=10485760
Review Comment:
This property has been deprecated since 3.4: https://github.com/apache/kafka/blob/5498eedf92a016e6838bce5383042bda1c798d76/streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java#L458
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19979: Improve Kafka Streams config files [kafka]
Posted by "ezhou413 (via GitHub)" <gi...@apache.org>.
ezhou413 commented on PR #21308:
URL: https://github.com/apache/kafka/pull/21308#issuecomment-4504267973
@mshijin-ksolves bumping this PR again, any updates?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19979: Improve Kafka Streams config files [kafka]
Posted by "majialoong (via GitHub)" <gi...@apache.org>.
majialoong commented on PR #21308:
URL: https://github.com/apache/kafka/pull/21308#issuecomment-4976428349
Hi @peterzdhuang ,
Thanks for checking! I haven't started working on this yet. I noticed you've already opened a PR for this, feel free to take it over if you'd like.
cc @ezhou413
Thanks again!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org