You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "ganesh-sadanala (via GitHub)" <gi...@apache.org> on 2024/05/24 22:31:10 UTC
[PR] KAFKA-15630 Improve documentation of offset.lag.max [kafka]
ganesh-sadanala opened a new pull request, #16080:
URL: https://github.com/apache/kafka/pull/16080
## Description
This pull request improves the documentation for the `offset.lag.max` configuration in Apache Kafka's MirrorMaker. The current documentation is brief and lacks context about the role of this configuration in offset translation and potential use cases.
The updated documentation provides a more comprehensive explanation of `offset.lag.max` and its impact on the synchronization between source and remote partitions. It clarifies that `offset.lag.max` determines the maximum allowed lag between the source and remote partitions before MirrorMaker initiates a resync operation to catch up the remote partition.
The documentation also covers the following aspects:
- How the lag is calculated (the difference between the latest offset in the source partition and the last committed offset in the remote partition)
- The resync operation process (reading records from the source partition starting from the last committed offset in the remote partition and writing them to the remote partition)
- Use cases for setting a lower `offset.lag.max` value (scenarios where records may not flow constantly or at a consistent rate, ensuring remote partitions stay more closely in sync during periods of low throughput or inactivity)
- Use cases for setting a higher `offset.lag.max` value (when the source topic has high throughput and remote partitions can tolerate a larger lag)
- The possibility of setting `offset.lag.max` to 0 (initiating a resync operation as soon as the remote partition falls behind the source partition, useful for strict synchronization requirements but may increase the load on the source cluster due to frequent resync operations)
### Committer Checklist (excluded from commit message)
- [x] Verify design and implementation
- [x] Verify test coverage and CI build status
- [x] Verify documentation (including upgrade notes)
Note: The contribution is my original work and I license the work to the project under the project's open source license.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15630 Improve documentation of offset.lag.max [kafka]
Posted by "ganesh-sadanala (via GitHub)" <gi...@apache.org>.
ganesh-sadanala commented on PR #16080:
URL: https://github.com/apache/kafka/pull/16080#issuecomment-2145862281
I have recently read about Kafka in online blogs and its official documentation, and I have skimmed through the official codebase to understand it better.
I don't have extensive experience apart from using Kafka in a few of my learning projects. However, I have started loving it and have been diving deep into it recently.
I am a new contributor and have begun contributing by solving bugs, implementing enhancements, and improving documentation to increase my knowledge and understanding and gain open-source contribution experience. I believe this would help improve my skills.
These are the references I have followed:
https://kafka.apache.org/documentation/
https://blog.cloudera.com/a-look-inside-kafka-mirrormaker-2/
https://blog.cloudera.com/kafka-replication-the-case-for-mirrormaker-2-0/
https://issues.apache.org/jira/browse/KAFKA-15906
https://github.com/apache/kafka
@gharris1727 Thank you for sharing the copyright policy. I did not know and I don't think any of the references I have followed are copyrighted content. Anyway, I will keep this in mind for future contributions. I apologize for making a lot of mistakes, however, I have learned a lot from it. Thank you! Will make the necessary changes and revert back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15630 Improve documentation of offset.lag.max [kafka]
Posted by "gharris1727 (via GitHub)" <gi...@apache.org>.
gharris1727 commented on PR #16080:
URL: https://github.com/apache/kafka/pull/16080#issuecomment-2145800484
And another meta-question: How did you learn about MM2? What sorts of materials did you use to learn about it, and/or what sort of practical experience do you have? I'm curious what other documentation needs to be improved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15630 Improve documentation of offset.lag.max [kafka]
Posted by "ganesh-sadanala (via GitHub)" <gi...@apache.org>.
ganesh-sadanala commented on code in PR #16080:
URL: https://github.com/apache/kafka/pull/16080#discussion_r1623138681
##########
connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConfig.java:
##########
@@ -101,7 +101,13 @@ public class MirrorSourceConfig extends MirrorConnectorConfig {
public static final Class<?> CONFIG_PROPERTY_FILTER_CLASS_DEFAULT = DefaultConfigPropertyFilter.class;
public static final String OFFSET_LAG_MAX = "offset.lag.max";
- private static final String OFFSET_LAG_MAX_DOC = "How out-of-sync a remote partition can be before it is resynced.";
+ private static final String OFFSET_LAG_MAX_DOC = "Determines the maximum allowed lag between the source and remote partitions before MirrorMaker initiates a resync operation to catch up the remote partition. The lag is calculated as the difference between the latest offset in the source partition and the last committed offset in the remote partition.\n" +
+ "\n" +
Review Comment:
new lines are not needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15630 Improve documentation of offset.lag.max [kafka]
Posted by "gharris1727 (via GitHub)" <gi...@apache.org>.
gharris1727 commented on code in PR #16080:
URL: https://github.com/apache/kafka/pull/16080#discussion_r1622763556
##########
connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConfig.java:
##########
@@ -101,7 +101,13 @@ public class MirrorSourceConfig extends MirrorConnectorConfig {
public static final Class<?> CONFIG_PROPERTY_FILTER_CLASS_DEFAULT = DefaultConfigPropertyFilter.class;
public static final String OFFSET_LAG_MAX = "offset.lag.max";
- private static final String OFFSET_LAG_MAX_DOC = "How out-of-sync a remote partition can be before it is resynced.";
+ private static final String OFFSET_LAG_MAX_DOC = "Determines the maximum allowed lag between the source and remote partitions before MirrorMaker initiates a resync operation to catch up the remote partition. The lag is calculated as the difference between the latest offset in the source partition and the last committed offset in the remote partition.\n" +
+ "\n" +
+ "When the lag for a remote partition exceeds the <code>offset.lag.max</code> value, MirrorMaker will initiate a resync operation to catch up the remote partition with the source partition. This involves reading records from the source partition starting from the last committed offset in the remote partition and writing them to the remote partition.\n" +
Review Comment:
> This involves reading records from the source partition starting from the last committed offset in the remote partition and writing them to the remote partition.
This is incorrect, and I don't know where this information came from.
When the lag for a remote partition exceeds offset.lag.max, it will emit an offset sync to the offset sync topic, which can then be used by the MirrorCheckpointTask to translate upstream and downstream offsets.
##########
connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConfig.java:
##########
@@ -101,7 +101,13 @@ public class MirrorSourceConfig extends MirrorConnectorConfig {
public static final Class<?> CONFIG_PROPERTY_FILTER_CLASS_DEFAULT = DefaultConfigPropertyFilter.class;
public static final String OFFSET_LAG_MAX = "offset.lag.max";
- private static final String OFFSET_LAG_MAX_DOC = "How out-of-sync a remote partition can be before it is resynced.";
+ private static final String OFFSET_LAG_MAX_DOC = "Determines the maximum allowed lag between the source and remote partitions before MirrorMaker initiates a resync operation to catch up the remote partition. The lag is calculated as the difference between the latest offset in the source partition and the last committed offset in the remote partition.\n" +
+ "\n" +
+ "When the lag for a remote partition exceeds the <code>offset.lag.max</code> value, MirrorMaker will initiate a resync operation to catch up the remote partition with the source partition. This involves reading records from the source partition starting from the last committed offset in the remote partition and writing them to the remote partition.\n" +
+ "\n" +
+ "Setting <code>offset.lag.max</code> to a lower value can be beneficial in scenarios where records may not flow constantly or at a consistent rate, as it ensures the remote partitions stay more closely in sync with the source partitions during periods of low throughput or inactivity. On the other hand, setting it to a higher value can be useful when the source topic has high throughput and the remote partitions can tolerate a larger lag.\n" +
Review Comment:
> Setting <code>offset.lag.max</code> to a lower value can be beneficial in scenarios where records may not flow constantly or at a consistent rate
Lowering the offset.lag.max to a non-zero value doesn't help when the flow is inconsistent. If a pause in the flow happens in-between syncs, the records after the last sync aren't translated. That was addressed in a separate improvement: https://issues.apache.org/jira/browse/KAFKA-15906
Lowering the offset.lag.max to a nonzero value is only necessary when the partition has a consistent low throughput, and a fixed sync every offset.flush.interval.ms isn't acceptable, which is a very contrived scenario.
##########
connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConfig.java:
##########
@@ -101,7 +101,13 @@ public class MirrorSourceConfig extends MirrorConnectorConfig {
public static final Class<?> CONFIG_PROPERTY_FILTER_CLASS_DEFAULT = DefaultConfigPropertyFilter.class;
public static final String OFFSET_LAG_MAX = "offset.lag.max";
- private static final String OFFSET_LAG_MAX_DOC = "How out-of-sync a remote partition can be before it is resynced.";
+ private static final String OFFSET_LAG_MAX_DOC = "Determines the maximum allowed lag between the source and remote partitions before MirrorMaker initiates a resync operation to catch up the remote partition. The lag is calculated as the difference between the latest offset in the source partition and the last committed offset in the remote partition.\n" +
Review Comment:
> The lag is calculated as the difference between the latest offset in the source partition and the last committed offset in the remote partition.
The lag is the difference between the latest offset in the source partition, and the last sync emitted to the offset syncs topic.
##########
connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConfig.java:
##########
@@ -101,7 +101,13 @@ public class MirrorSourceConfig extends MirrorConnectorConfig {
public static final Class<?> CONFIG_PROPERTY_FILTER_CLASS_DEFAULT = DefaultConfigPropertyFilter.class;
public static final String OFFSET_LAG_MAX = "offset.lag.max";
- private static final String OFFSET_LAG_MAX_DOC = "How out-of-sync a remote partition can be before it is resynced.";
+ private static final String OFFSET_LAG_MAX_DOC = "Determines the maximum allowed lag between the source and remote partitions before MirrorMaker initiates a resync operation to catch up the remote partition. The lag is calculated as the difference between the latest offset in the source partition and the last committed offset in the remote partition.\n" +
+ "\n" +
Review Comment:
Not sure about these double newlines and how they render, how are you previewing these changes?
##########
connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConfig.java:
##########
@@ -101,7 +101,13 @@ public class MirrorSourceConfig extends MirrorConnectorConfig {
public static final Class<?> CONFIG_PROPERTY_FILTER_CLASS_DEFAULT = DefaultConfigPropertyFilter.class;
public static final String OFFSET_LAG_MAX = "offset.lag.max";
- private static final String OFFSET_LAG_MAX_DOC = "How out-of-sync a remote partition can be before it is resynced.";
+ private static final String OFFSET_LAG_MAX_DOC = "Determines the maximum allowed lag between the source and remote partitions before MirrorMaker initiates a resync operation to catch up the remote partition. The lag is calculated as the difference between the latest offset in the source partition and the last committed offset in the remote partition.\n" +
+ "\n" +
+ "When the lag for a remote partition exceeds the <code>offset.lag.max</code> value, MirrorMaker will initiate a resync operation to catch up the remote partition with the source partition. This involves reading records from the source partition starting from the last committed offset in the remote partition and writing them to the remote partition.\n" +
+ "\n" +
+ "Setting <code>offset.lag.max</code> to a lower value can be beneficial in scenarios where records may not flow constantly or at a consistent rate, as it ensures the remote partitions stay more closely in sync with the source partitions during periods of low throughput or inactivity. On the other hand, setting it to a higher value can be useful when the source topic has high throughput and the remote partitions can tolerate a larger lag.\n" +
+ "\n" +
+ "It's also possible to set <code>offset.lag.max</code> to 0, which will cause MirrorMaker to initiate a resync operation for a remote partition as soon as it falls behind the source partition. This can be useful for strict synchronization requirements but may increase the load on the source cluster due to frequent resync operations.";
Review Comment:
Instead of saying "initate a resync operation" we can say "emit an offset sync for every source record", mentioning that the additional load and extra throughput on the offset sync topic specifically.
The comment about increased load on the source cluster can be inaccurate, because the syncs topic may be present on the source or target cluster depending on the value of offset-syncs.topic.location configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15630 Improve documentation of offset.lag.max [kafka]
Posted by "ganesh-sadanala (via GitHub)" <gi...@apache.org>.
ganesh-sadanala commented on code in PR #16080:
URL: https://github.com/apache/kafka/pull/16080#discussion_r1622872574
##########
connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConfig.java:
##########
@@ -101,7 +101,13 @@ public class MirrorSourceConfig extends MirrorConnectorConfig {
public static final Class<?> CONFIG_PROPERTY_FILTER_CLASS_DEFAULT = DefaultConfigPropertyFilter.class;
public static final String OFFSET_LAG_MAX = "offset.lag.max";
- private static final String OFFSET_LAG_MAX_DOC = "How out-of-sync a remote partition can be before it is resynced.";
+ private static final String OFFSET_LAG_MAX_DOC = "Determines the maximum allowed lag between the source and remote partitions before MirrorMaker initiates a resync operation to catch up the remote partition. The lag is calculated as the difference between the latest offset in the source partition and the last committed offset in the remote partition.\n" +
Review Comment:
This clearly explains the latest mapping record in the offset sync topic which clearly states the latest sync in the target/remote partition. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15630 Improve documentation of offset.lag.max [kafka]
Posted by "gharris1727 (via GitHub)" <gi...@apache.org>.
gharris1727 commented on code in PR #16080:
URL: https://github.com/apache/kafka/pull/16080#discussion_r1624808530
##########
connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConfig.java:
##########
@@ -101,7 +101,11 @@ public class MirrorSourceConfig extends MirrorConnectorConfig {
public static final Class<?> CONFIG_PROPERTY_FILTER_CLASS_DEFAULT = DefaultConfigPropertyFilter.class;
public static final String OFFSET_LAG_MAX = "offset.lag.max";
- private static final String OFFSET_LAG_MAX_DOC = "How out-of-sync a remote partition can be before it is resynced.";
+ private static final String OFFSET_LAG_MAX_DOC = "Determines the maximum allowed lag between the source and target partitions before MirrorMaker emits an 'offset sync' event to catch up the remote partition. The lag is calculated as the difference between the latest offset in the source partition, and the last sync emitted to the offset syncs topic. " +
+ "When the lag for a remote partition exceeds the <code>offset.lag.max</code> value, MirrorMaker emits an 'offset sync' event to the 'offset-syncs' topic, which is then used by the <code>MirrorCheckpointTask</code> to translate source and target partitions." +
Review Comment:
please edit this to combine redundant sentences.
##########
connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConfig.java:
##########
@@ -101,7 +101,11 @@ public class MirrorSourceConfig extends MirrorConnectorConfig {
public static final Class<?> CONFIG_PROPERTY_FILTER_CLASS_DEFAULT = DefaultConfigPropertyFilter.class;
public static final String OFFSET_LAG_MAX = "offset.lag.max";
- private static final String OFFSET_LAG_MAX_DOC = "How out-of-sync a remote partition can be before it is resynced.";
+ private static final String OFFSET_LAG_MAX_DOC = "Determines the maximum allowed lag between the source and target partitions before MirrorMaker emits an 'offset sync' event to catch up the remote partition. The lag is calculated as the difference between the latest offset in the source partition, and the last sync emitted to the offset syncs topic. " +
+ "When the lag for a remote partition exceeds the <code>offset.lag.max</code> value, MirrorMaker emits an 'offset sync' event to the 'offset-syncs' topic, which is then used by the <code>MirrorCheckpointTask</code> to translate source and target partitions." +
+ "Setting <code>offset.lag.max</code> to a lower value can be beneficial in scenarios where partitions have consistently low throughput, and the fixed sync interval ('offset.flush.interval.ms') is not acceptable. A lower value will trigger more frequent offset syncs, keeping the target partitions more closely in sync with the source for these low-throughput partitions. " +
+ "On the other hand, setting it to a higher value can be useful when the source topic has high throughput and the remote partitions can tolerate a larger lag without impacting target costumers." +
+ "It's also possible to set <code>offset.lag.max</code> to 0, which will cause MirrorMaker to emit an 'offset sync' for every source record to a target partition. This can be useful for strict synchronization requirements but may increase the load and extra throughput on the 'offset-syncs' topic, depending on the value of the 'offset-syncs.topic.location' configuration";
Review Comment:
Don't use "strict synchronization" here, as it could be confused for "synchronous", which is a capability that MM2 does not have.
Also the mention of the offset-syncs.topic.location needs to be rephrased. If you say "X may do Y, depending on the value of Z", it makes me think there is some value of Z avoids Y, when that isn't the case.
Lowering the value _will_ increase the load on the offset-syncs topic except in very specific scenarios, so using _may_ is more confusing than it's worth.
##########
connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConfig.java:
##########
@@ -101,7 +101,11 @@ public class MirrorSourceConfig extends MirrorConnectorConfig {
public static final Class<?> CONFIG_PROPERTY_FILTER_CLASS_DEFAULT = DefaultConfigPropertyFilter.class;
public static final String OFFSET_LAG_MAX = "offset.lag.max";
- private static final String OFFSET_LAG_MAX_DOC = "How out-of-sync a remote partition can be before it is resynced.";
+ private static final String OFFSET_LAG_MAX_DOC = "Determines the maximum allowed lag between the source and target partitions before MirrorMaker emits an 'offset sync' event to catch up the remote partition. The lag is calculated as the difference between the latest offset in the source partition, and the last sync emitted to the offset syncs topic. " +
+ "When the lag for a remote partition exceeds the <code>offset.lag.max</code> value, MirrorMaker emits an 'offset sync' event to the 'offset-syncs' topic, which is then used by the <code>MirrorCheckpointTask</code> to translate source and target partitions." +
+ "Setting <code>offset.lag.max</code> to a lower value can be beneficial in scenarios where partitions have consistently low throughput, and the fixed sync interval ('offset.flush.interval.ms') is not acceptable. A lower value will trigger more frequent offset syncs, keeping the target partitions more closely in sync with the source for these low-throughput partitions. " +
+ "On the other hand, setting it to a higher value can be useful when the source topic has high throughput and the remote partitions can tolerate a larger lag without impacting target costumers." +
Review Comment:
You don't need to say "without impacting...", plus there is a spelling mistake there. This can be combined with the previous discussion on lowering the value.
##########
connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConfig.java:
##########
@@ -101,7 +101,11 @@ public class MirrorSourceConfig extends MirrorConnectorConfig {
public static final Class<?> CONFIG_PROPERTY_FILTER_CLASS_DEFAULT = DefaultConfigPropertyFilter.class;
public static final String OFFSET_LAG_MAX = "offset.lag.max";
- private static final String OFFSET_LAG_MAX_DOC = "How out-of-sync a remote partition can be before it is resynced.";
+ private static final String OFFSET_LAG_MAX_DOC = "Determines the maximum allowed lag between the source and target partitions before MirrorMaker emits an 'offset sync' event to catch up the remote partition. The lag is calculated as the difference between the latest offset in the source partition, and the last sync emitted to the offset syncs topic. " +
+ "When the lag for a remote partition exceeds the <code>offset.lag.max</code> value, MirrorMaker emits an 'offset sync' event to the 'offset-syncs' topic, which is then used by the <code>MirrorCheckpointTask</code> to translate source and target partitions." +
+ "Setting <code>offset.lag.max</code> to a lower value can be beneficial in scenarios where partitions have consistently low throughput, and the fixed sync interval ('offset.flush.interval.ms') is not acceptable. A lower value will trigger more frequent offset syncs, keeping the target partitions more closely in sync with the source for these low-throughput partitions. " +
Review Comment:
The only part of this line that is worth including is "A lower value will trigger more frequent offset syncs".
What I meant in the previous comment was that 100 is already so close to 0 that nobody will realistically lower this value, so we don't need to describe it in much detail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15630 Improve documentation of offset.lag.max [kafka]
Posted by "gharris1727 (via GitHub)" <gi...@apache.org>.
gharris1727 closed pull request #16080: KAFKA-15630 Improve documentation of offset.lag.max
URL: https://github.com/apache/kafka/pull/16080
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15630 Improve documentation of offset.lag.max [kafka]
Posted by "ganesh-sadanala (via GitHub)" <gi...@apache.org>.
ganesh-sadanala commented on code in PR #16080:
URL: https://github.com/apache/kafka/pull/16080#discussion_r1623135976
##########
connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConfig.java:
##########
@@ -101,7 +101,13 @@ public class MirrorSourceConfig extends MirrorConnectorConfig {
public static final Class<?> CONFIG_PROPERTY_FILTER_CLASS_DEFAULT = DefaultConfigPropertyFilter.class;
public static final String OFFSET_LAG_MAX = "offset.lag.max";
- private static final String OFFSET_LAG_MAX_DOC = "How out-of-sync a remote partition can be before it is resynced.";
+ private static final String OFFSET_LAG_MAX_DOC = "Determines the maximum allowed lag between the source and remote partitions before MirrorMaker initiates a resync operation to catch up the remote partition. The lag is calculated as the difference between the latest offset in the source partition and the last committed offset in the remote partition.\n" +
+ "\n" +
+ "When the lag for a remote partition exceeds the <code>offset.lag.max</code> value, MirrorMaker will initiate a resync operation to catch up the remote partition with the source partition. This involves reading records from the source partition starting from the last committed offset in the remote partition and writing them to the remote partition.\n" +
+ "\n" +
+ "Setting <code>offset.lag.max</code> to a lower value can be beneficial in scenarios where records may not flow constantly or at a consistent rate, as it ensures the remote partitions stay more closely in sync with the source partitions during periods of low throughput or inactivity. On the other hand, setting it to a higher value can be useful when the source topic has high throughput and the remote partitions can tolerate a larger lag.\n" +
Review Comment:
Agree!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15630 Improve documentation of offset.lag.max [kafka]
Posted by "ganesh-sadanala (via GitHub)" <gi...@apache.org>.
ganesh-sadanala commented on PR #16080:
URL: https://github.com/apache/kafka/pull/16080#issuecomment-2159220165
@gharris1727 Thank you! I totally understand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15630 Improve documentation of offset.lag.max [kafka]
Posted by "ganesh-sadanala (via GitHub)" <gi...@apache.org>.
ganesh-sadanala commented on code in PR #16080:
URL: https://github.com/apache/kafka/pull/16080#discussion_r1622879301
##########
connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConfig.java:
##########
@@ -101,7 +101,13 @@ public class MirrorSourceConfig extends MirrorConnectorConfig {
public static final Class<?> CONFIG_PROPERTY_FILTER_CLASS_DEFAULT = DefaultConfigPropertyFilter.class;
public static final String OFFSET_LAG_MAX = "offset.lag.max";
- private static final String OFFSET_LAG_MAX_DOC = "How out-of-sync a remote partition can be before it is resynced.";
+ private static final String OFFSET_LAG_MAX_DOC = "Determines the maximum allowed lag between the source and remote partitions before MirrorMaker initiates a resync operation to catch up the remote partition. The lag is calculated as the difference between the latest offset in the source partition and the last committed offset in the remote partition.\n" +
+ "\n" +
+ "When the lag for a remote partition exceeds the <code>offset.lag.max</code> value, MirrorMaker will initiate a resync operation to catch up the remote partition with the source partition. This involves reading records from the source partition starting from the last committed offset in the remote partition and writing them to the remote partition.\n" +
Review Comment:
Thank you for pointing this out and providing the feedback. You're correct!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15630 Improve documentation of offset.lag.max [kafka]
Posted by "gharris1727 (via GitHub)" <gi...@apache.org>.
gharris1727 commented on PR #16080:
URL: https://github.com/apache/kafka/pull/16080#issuecomment-2159216114
Hi @ganesh-sadanala Unfortunately I don't think we can move forward with this PR.
You've linked and mis-represented two pieces of documentation that are not Apache-2.0 license compatible, can't be used to write this documentation. See the Cloudera Terms and Conditions: https://www.cloudera.com/legal/terms-and-conditions.html which is linked from the documentation you referenced.
Please understand that we must take license compatibility very seriously, and can't accept contributions that risk mis-licensing others' intellectual property. When you are contributing to Kafka or other open-source projects in the future, it is best that the contributions you propose are totally your own work, derived from other open-source works. This includes avoiding the use of generative tools that may be trained on incompatibly-licensed works.
I'm closing this PR, and letting another contributor pick this up instead. Thank you for your time, and I hope you can continue contributing to Kafka in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org