You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2020/12/09 06:33:12 UTC
[GitHub] [kafka] kowshik opened a new pull request #9718: MINOR: Use the correct ProducerStateManager instance when updating producers
kowshik opened a new pull request #9718:
URL: https://github.com/apache/kafka/pull/9718
I have fixed what looked like a potential bug to me. From within `Log.updateProducers(…)`, the code operates on the `producerStateManager` attribute of the `Log` object (see [this](https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/log/Log.scala#L1464) where we call `producerStateManager.prepareUpdate`). This looks unusual particularly for `Log.loadProducersFromLog(...)` path. For this path I believe we should be updating the producers into the instance passed to it, rather than the one in the `Log` object attribute. I have fixed the same in this PR.
**Tests:**
Rely on existing tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] kowshik commented on pull request #9718: MINOR: Use the correct ProducerStateManager instance when updating producers
Posted by GitBox <gi...@apache.org>.
kowshik commented on pull request #9718:
URL: https://github.com/apache/kafka/pull/9718#issuecomment-741565872
cc @dhruvilshah3 @junrao @gardnervickers for review
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao merged pull request #9718: KAFKA-10832: Fix Log to use the correct ProducerStateManager instance when updating producers
Posted by GitBox <gi...@apache.org>.
junrao merged pull request #9718:
URL: https://github.com/apache/kafka/pull/9718
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #9718: KAFKA-10832: Fix Log to use the correct ProducerStateManager instance when updating producers
Posted by GitBox <gi...@apache.org>.
junrao commented on a change in pull request #9718:
URL: https://github.com/apache/kafka/pull/9718#discussion_r541280750
##########
File path: core/src/main/scala/kafka/log/Log.scala
##########
@@ -2670,6 +2645,34 @@ object Log {
private def isLogFile(file: File): Boolean =
file.getPath.endsWith(LogFileSuffix)
+ private def loadProducersFromLog(producerStateManager: ProducerStateManager, records: Records): Unit = {
Review comment:
Perhaps this could be name loadProducersFromRecords?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #9718: KAFKA-10832: Fix Log to use the correct ProducerStateManager instance when updating producers
Posted by GitBox <gi...@apache.org>.
junrao commented on a change in pull request #9718:
URL: https://github.com/apache/kafka/pull/9718#discussion_r540571244
##########
File path: core/src/main/scala/kafka/log/Log.scala
##########
@@ -2670,6 +2573,112 @@ object Log {
private def isLogFile(file: File): Boolean =
file.getPath.endsWith(LogFileSuffix)
+ // Rebuild producer state until lastOffset. This method may be called from the recovery code path, and thus must be
+ // free of all side-effects, i.e. it must not update any log-specific state.
+ private def rebuildProducerState(log: Log,
Review comment:
This seems too large as a companion method. I am also not sure that we should pass along Log to this method since it exposes all internal states in Log, which make it harder to track usage.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] kowshik commented on pull request #9718: MINOR: Use the correct ProducerStateManager instance when updating producers
Posted by GitBox <gi...@apache.org>.
kowshik commented on pull request #9718:
URL: https://github.com/apache/kafka/pull/9718#issuecomment-742008769
@ijuma Good idea. I can move `Log.rebuildProducerState` into the `Log` companion object, and consequently the following methods will move too: `Log.loadProducersFromLog` and `Log.updateProducers`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] kowshik commented on a change in pull request #9718: KAFKA-10832: Fix Log to use the correct ProducerStateManager instance when updating producers
Posted by GitBox <gi...@apache.org>.
kowshik commented on a change in pull request #9718:
URL: https://github.com/apache/kafka/pull/9718#discussion_r540573082
##########
File path: core/src/main/scala/kafka/log/Log.scala
##########
@@ -2670,6 +2573,112 @@ object Log {
private def isLogFile(file: File): Boolean =
file.getPath.endsWith(LogFileSuffix)
+ // Rebuild producer state until lastOffset. This method may be called from the recovery code path, and thus must be
+ // free of all side-effects, i.e. it must not update any log-specific state.
+ private def rebuildProducerState(log: Log,
Review comment:
That is true. Do you feel the original approach in f0466b2420dc500112b4839678564996c0677d2e was better, where, we didn't move methods into a companion object? I'd be happy to revert to that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] kowshik edited a comment on pull request #9718: KAFKA-10832: Fix Log to use the correct ProducerStateManager instance when updating producers
Posted by GitBox <gi...@apache.org>.
kowshik edited a comment on pull request #9718:
URL: https://github.com/apache/kafka/pull/9718#issuecomment-743426585
Thanks for the review @junrao ! I've addressed the comment in ec00e9f. The JDK 15 test failure in `org.apache.kafka.streams.integration.PurgeRepartitionTopicIntegrationTest.shouldRestoreState` seems unrelated to this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #9718: KAFKA-10832: Fix Log to use the correct ProducerStateManager instance when updating producers
Posted by GitBox <gi...@apache.org>.
junrao commented on a change in pull request #9718:
URL: https://github.com/apache/kafka/pull/9718#discussion_r540588994
##########
File path: core/src/main/scala/kafka/log/Log.scala
##########
@@ -2670,6 +2573,112 @@ object Log {
private def isLogFile(file: File): Boolean =
file.getPath.endsWith(LogFileSuffix)
+ // Rebuild producer state until lastOffset. This method may be called from the recovery code path, and thus must be
+ // free of all side-effects, i.e. it must not update any log-specific state.
+ private def rebuildProducerState(log: Log,
Review comment:
We can probably keep the two other two simpler methods (without Log as the input) as the companion methods.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] kowshik commented on pull request #9718: KAFKA-10832: Fix Log to use the correct ProducerStateManager instance when updating producers
Posted by GitBox <gi...@apache.org>.
kowshik commented on pull request #9718:
URL: https://github.com/apache/kafka/pull/9718#issuecomment-743426585
Thanks for the review @junrao ! I've addressed the comment in ec00e9f.
The JDK 15 test failure in `org.apache.kafka.streams.integration.PurgeRepartitionTopicIntegrationTest.shouldRestoreState` seems unrelated to this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] kowshik commented on pull request #9718: KAFKA-10832: Fix Log to use the correct ProducerStateManager instance when updating producers
Posted by GitBox <gi...@apache.org>.
kowshik commented on pull request #9718:
URL: https://github.com/apache/kafka/pull/9718#issuecomment-742922667
Thanks for the review, @junrao! I've addressed the comments in 5d798b7d884bd11e43884318ae3fc8d12e635c40.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] kowshik commented on a change in pull request #9718: KAFKA-10832: Fix Log to use the correct ProducerStateManager instance when updating producers
Posted by GitBox <gi...@apache.org>.
kowshik commented on a change in pull request #9718:
URL: https://github.com/apache/kafka/pull/9718#discussion_r541288822
##########
File path: core/src/main/scala/kafka/log/Log.scala
##########
@@ -2670,6 +2645,34 @@ object Log {
private def isLogFile(file: File): Boolean =
file.getPath.endsWith(LogFileSuffix)
+ private def loadProducersFromLog(producerStateManager: ProducerStateManager, records: Records): Unit = {
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #9718: MINOR: Use the correct ProducerStateManager instance when updating producers
Posted by GitBox <gi...@apache.org>.
ijuma commented on pull request #9718:
URL: https://github.com/apache/kafka/pull/9718#issuecomment-741791486
Good catch. This looks a bit error prone. Should we be moving some of these methods to the companion object or something?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] kowshik commented on pull request #9718: KAFKA-10832: Use the correct ProducerStateManager instance when updating producers
Posted by GitBox <gi...@apache.org>.
kowshik commented on pull request #9718:
URL: https://github.com/apache/kafka/pull/9718#issuecomment-742803224
@ijuma Thanks for the review! I've addressed your comments in 6d348bcc12f9e8b2ddf0552c089945b4eda67538.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org