You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "lucasbru (via GitHub)" <gi...@apache.org> on 2024/12/04 10:22:52 UTC
[PR] Add group.protocol config handling for streams [kafka]
lucasbru opened a new pull request, #18033:
URL: https://github.com/apache/kafka/pull/18033
- accept new streams group.protocol value
- do not forward to consumer clients
- log message that not for production
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add group.protocol config handling for streams [kafka]
Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on code in PR #18033:
URL: https://github.com/apache/kafka/pull/18033#discussion_r1869398274
##########
streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java:
##########
@@ -841,6 +841,8 @@ public class StreamsConfig extends AbstractConfig {
public static final String WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG = "windowstore.changelog.additional.retention.ms";
private static final String WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_DOC = "Added to a windows maintainMs to ensure data is not deleted from the log prematurely. Allows for clock drift. Default is 1 day";
+ private static final String[] NOT_CONFIGURABLE_IN_GLOBAL_CONSUMER =
+ new String[] {ConsumerConfig.GROUP_PROTOCOL_CONFIG};
Review Comment:
sure, lets use unprefixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add group.protocol config handling for streams [kafka]
Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on code in PR #18033:
URL: https://github.com/apache/kafka/pull/18033#discussion_r1869229775
##########
streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java:
##########
@@ -841,6 +841,8 @@ public class StreamsConfig extends AbstractConfig {
public static final String WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG = "windowstore.changelog.additional.retention.ms";
private static final String WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_DOC = "Added to a windows maintainMs to ensure data is not deleted from the log prematurely. Allows for clock drift. Default is 1 day";
+ private static final String[] NOT_CONFIGURABLE_IN_GLOBAL_CONSUMER =
+ new String[] {ConsumerConfig.GROUP_PROTOCOL_CONFIG};
Review Comment:
not sure. all the other static use "NOT_CONFIGURABLE", why use "UNSETTABLE" or "NON_SETABLE", is it better?
Let me suggest `CONSUMER_CONFIGS_IGNORED_IN_ROOT_SCOPE`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add group.protocol config handling for streams [kafka]
Posted by "cadonna (via GitHub)" <gi...@apache.org>.
cadonna commented on code in PR #18033:
URL: https://github.com/apache/kafka/pull/18033#discussion_r1869373446
##########
streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java:
##########
@@ -841,6 +841,8 @@ public class StreamsConfig extends AbstractConfig {
public static final String WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG = "windowstore.changelog.additional.retention.ms";
private static final String WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_DOC = "Added to a windows maintainMs to ensure data is not deleted from the log prematurely. Allows for clock drift. Default is 1 day";
+ private static final String[] NOT_CONFIGURABLE_IN_GLOBAL_CONSUMER =
+ new String[] {ConsumerConfig.GROUP_PROTOCOL_CONFIG};
Review Comment:
I understood now, what you mean. This is quite confusing, because the Streams config shadows the consumer config. Although, the configs have the same name, they are used in different places. We do not want to pass `group.protocol` to the consumer. Other unprefixed consumer configs we want to pass into the consumer. 🤯
What about `IGNORED_UNPREFIXED_CONSUMER_CONFIGS`?
That is my last suggestion. Take it or use `CONSUMER_CONFIGS_IGNORED_IN_ROOT_SCOPE`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add group.protocol config handling for streams [kafka]
Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on code in PR #18033:
URL: https://github.com/apache/kafka/pull/18033#discussion_r1869192832
##########
streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java:
##########
@@ -841,6 +841,8 @@ public class StreamsConfig extends AbstractConfig {
public static final String WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG = "windowstore.changelog.additional.retention.ms";
private static final String WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_DOC = "Added to a windows maintainMs to ensure data is not deleted from the log prematurely. Allows for clock drift. Default is 1 day";
+ private static final String[] NOT_CONFIGURABLE_IN_GLOBAL_CONSUMER =
+ new String[] {ConsumerConfig.GROUP_PROTOCOL_CONFIG};
Review Comment:
yeah it was called "global" in another place, I was also wondering about the name. I agree global is confusing.
Howabout "ROOT"? the root of the property tree
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add group.protocol config handling for streams [kafka]
Posted by "cadonna (via GitHub)" <gi...@apache.org>.
cadonna commented on code in PR #18033:
URL: https://github.com/apache/kafka/pull/18033#discussion_r1869238142
##########
streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java:
##########
@@ -841,6 +841,8 @@ public class StreamsConfig extends AbstractConfig {
public static final String WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG = "windowstore.changelog.additional.retention.ms";
private static final String WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_DOC = "Added to a windows maintainMs to ensure data is not deleted from the log prematurely. Allows for clock drift. Default is 1 day";
+ private static final String[] NOT_CONFIGURABLE_IN_GLOBAL_CONSUMER =
+ new String[] {ConsumerConfig.GROUP_PROTOCOL_CONFIG};
Review Comment:
IMO, `ROOT_SCOPE` is really hard to understand. It does also not follow the `NOT_CONFIGURABLE` (actually it should be `NON_CONFIGURABLE`).
But I like `IGNORED`. What about `IGNORED_COMMON_CONSUMER_CONFIGS`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add group.protocol config handling for streams [kafka]
Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on code in PR #18033:
URL: https://github.com/apache/kafka/pull/18033#discussion_r1869244623
##########
streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java:
##########
@@ -841,6 +841,8 @@ public class StreamsConfig extends AbstractConfig {
public static final String WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG = "windowstore.changelog.additional.retention.ms";
private static final String WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_DOC = "Added to a windows maintainMs to ensure data is not deleted from the log prematurely. Allows for clock drift. Default is 1 day";
+ private static final String[] NOT_CONFIGURABLE_IN_GLOBAL_CONSUMER =
+ new String[] {ConsumerConfig.GROUP_PROTOCOL_CONFIG};
Review Comment:
It changes the meaning. as far as I understand this class, common consumer config are defined in the `consumer` prefix, so we'd be talking about. `consumer.group.protocol`, but what we are ignoring is just `group.protocol`. So I think this would be incorrect
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add group.protocol config handling for streams [kafka]
Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru merged PR #18033:
URL: https://github.com/apache/kafka/pull/18033
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add group.protocol config handling for streams [kafka]
Posted by "cadonna (via GitHub)" <gi...@apache.org>.
cadonna commented on code in PR #18033:
URL: https://github.com/apache/kafka/pull/18033#discussion_r1869187904
##########
streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java:
##########
@@ -841,6 +841,8 @@ public class StreamsConfig extends AbstractConfig {
public static final String WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG = "windowstore.changelog.additional.retention.ms";
private static final String WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_DOC = "Added to a windows maintainMs to ensure data is not deleted from the log prematurely. Allows for clock drift. Default is 1 day";
+ private static final String[] NOT_CONFIGURABLE_IN_GLOBAL_CONSUMER =
+ new String[] {ConsumerConfig.GROUP_PROTOCOL_CONFIG};
Review Comment:
If I indeed misunderstood, you should choose a different name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add group.protocol config handling for streams [kafka]
Posted by "cadonna (via GitHub)" <gi...@apache.org>.
cadonna commented on code in PR #18033:
URL: https://github.com/apache/kafka/pull/18033#discussion_r1869203087
##########
streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java:
##########
@@ -841,6 +841,8 @@ public class StreamsConfig extends AbstractConfig {
public static final String WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG = "windowstore.changelog.additional.retention.ms";
private static final String WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_DOC = "Added to a windows maintainMs to ensure data is not deleted from the log prematurely. Allows for clock drift. Default is 1 day";
+ private static final String[] NOT_CONFIGURABLE_IN_GLOBAL_CONSUMER =
+ new String[] {ConsumerConfig.GROUP_PROTOCOL_CONFIG};
Review Comment:
Or `NON_SETABLE_COMMON_CONSUMER_CONFIGS`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add group.protocol config handling for streams [kafka]
Posted by "cadonna (via GitHub)" <gi...@apache.org>.
cadonna commented on code in PR #18033:
URL: https://github.com/apache/kafka/pull/18033#discussion_r1869197580
##########
streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java:
##########
@@ -841,6 +841,8 @@ public class StreamsConfig extends AbstractConfig {
public static final String WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG = "windowstore.changelog.additional.retention.ms";
private static final String WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_DOC = "Added to a windows maintainMs to ensure data is not deleted from the log prematurely. Allows for clock drift. Default is 1 day";
+ private static final String[] NOT_CONFIGURABLE_IN_GLOBAL_CONSUMER =
+ new String[] {ConsumerConfig.GROUP_PROTOCOL_CONFIG};
Review Comment:
What about `UNSETTABLE_COMMON_CONSUMER_CONFIGS`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add group.protocol config handling for streams [kafka]
Posted by "cadonna (via GitHub)" <gi...@apache.org>.
cadonna commented on code in PR #18033:
URL: https://github.com/apache/kafka/pull/18033#discussion_r1869171812
##########
streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java:
##########
@@ -841,6 +841,8 @@ public class StreamsConfig extends AbstractConfig {
public static final String WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG = "windowstore.changelog.additional.retention.ms";
private static final String WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_DOC = "Added to a windows maintainMs to ensure data is not deleted from the log prematurely. Allows for clock drift. Default is 1 day";
+ private static final String[] NOT_CONFIGURABLE_IN_GLOBAL_CONSUMER =
+ new String[] {ConsumerConfig.GROUP_PROTOCOL_CONFIG};
Review Comment:
Why is this needed for the global consumer but not for the restore consumer? Both are manually assigned.
##########
streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java:
##########
@@ -841,6 +841,8 @@ public class StreamsConfig extends AbstractConfig {
public static final String WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG = "windowstore.changelog.additional.retention.ms";
private static final String WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_DOC = "Added to a windows maintainMs to ensure data is not deleted from the log prematurely. Allows for clock drift. Default is 1 day";
+ private static final String[] NOT_CONFIGURABLE_IN_GLOBAL_CONSUMER =
+ new String[] {ConsumerConfig.GROUP_PROTOCOL_CONFIG};
Review Comment:
Ah, I think I misunderstood. You did not mean the global consumer here, did you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add group.protocol config handling for streams [kafka]
Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on code in PR #18033:
URL: https://github.com/apache/kafka/pull/18033#discussion_r1869244623
##########
streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java:
##########
@@ -841,6 +841,8 @@ public class StreamsConfig extends AbstractConfig {
public static final String WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG = "windowstore.changelog.additional.retention.ms";
private static final String WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_DOC = "Added to a windows maintainMs to ensure data is not deleted from the log prematurely. Allows for clock drift. Default is 1 day";
+ private static final String[] NOT_CONFIGURABLE_IN_GLOBAL_CONSUMER =
+ new String[] {ConsumerConfig.GROUP_PROTOCOL_CONFIG};
Review Comment:
It changes the meaning. as far as I understand this class, common consumer config are defined in `consumer.group.protocol`, but what we are ignoring is just `group.protocol`. So I think this would be incorrect
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org