You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "hudeqi (via GitHub)" <gi...@apache.org> on 2023/03/10 03:26:52 UTC
[GitHub] [kafka] hudeqi opened a new pull request, #13348: MINOR: Fix ProducerPerformance still counting successful sending when sending failed
hudeqi opened a new pull request, #13348:
URL: https://github.com/apache/kafka/pull/13348
When using ProducerPerformance, I found that when the sending fails, it is still counted as successfully sent by stat and the metrics are printed. For example, when there is no write permission and cannot be written in, the sending success rate is still magically displayed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] chia7712 commented on pull request #13348: MINOR: Fix ProducerPerformance still counting successful sending when sending failed
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #13348:
URL: https://github.com/apache/kafka/pull/13348#issuecomment-1471470912
> and then report back the original problem that failed to send but still recorded success, and solve the current problem at the same time, right?
yep. #13400 is used to revert this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13348: MINOR: Fix ProducerPerformance still counting successful sending when sending failed
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi commented on PR #13348:
URL: https://github.com/apache/kafka/pull/13348#issuecomment-1463199231
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13348: MINOR: Fix ProducerPerformance still counting successful sending when sending failed
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi commented on PR #13348:
URL: https://github.com/apache/kafka/pull/13348#issuecomment-1471416257
>
In fact, I also have this question. The variables in stats are all thread-unsafe at present, but the printed data looks fine😂, and this is a probabilistic problem for the version I repaired.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13348: MINOR: Fix ProducerPerformance still counting successful sending when sending failed
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi commented on PR #13348:
URL: https://github.com/apache/kafka/pull/13348#issuecomment-1463200864
> @hudeqi , thanks for the patch. This change makes sense to me. Could you add tests for this change?
Do you want to add test screenshots/results for this case, or the test code corresponding to this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on pull request #13348: MINOR: Fix ProducerPerformance still counting successful sending when sending failed
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on PR #13348:
URL: https://github.com/apache/kafka/pull/13348#issuecomment-1463202757
The test code, I mean. We already have some unit tests for producerPerformance [here](https://github.com/apache/kafka/blob/trunk/tools/src/test/java/org/apache/kafka/tools/ProducerPerformanceTest.java). Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi commented on a diff in pull request #13348: MINOR: Fix ProducerPerformance still counting successful sending when sending failed
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi commented on code in PR #13348:
URL: https://github.com/apache/kafka/pull/13348#discussion_r1132477341
##########
tools/src/main/java/org/apache/kafka/tools/ProducerPerformance.java:
##########
@@ -113,7 +113,7 @@ void start(String[] args) throws IOException {
record = new ProducerRecord<>(topicName, payload);
long sendStartMs = System.currentTimeMillis();
- Callback cb = stats.nextCompletion(sendStartMs, payload.length, stats);
+ cb = stats.nextCompletion(sendStartMs, payload.length, stats);
Review Comment:
A great and careful code optimization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] chia7712 commented on pull request #13348: MINOR: Fix ProducerPerformance still counting successful sending when sending failed
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #13348:
URL: https://github.com/apache/kafka/pull/13348#issuecomment-1471465952
> In addition, does this repair need to open a new jira?
In fact, we should revert this PR first. The solution offered by this PR bring antoher issue (thanks to @robobario). The origin issue is worth having a JIRA since it is related to a true bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13348: MINOR: Fix ProducerPerformance still counting successful sending when sending failed
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi commented on PR #13348:
URL: https://github.com/apache/kafka/pull/13348#issuecomment-1463470303
>
Hello, the relevant test code has been submitted, please help to review, thank you. @showuon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] chia7712 commented on pull request #13348: MINOR: Fix ProducerPerformance still counting successful sending when sending failed
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #13348:
URL: https://github.com/apache/kafka/pull/13348#issuecomment-1471410292
@robobario thanks for feedback. According to your nice catch `Stats` is shared between all producer thread. Not only `iteration` but also other mutable variables should be thread-safe, or we may see the incorrect stats value, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13348: MINOR: Fix ProducerPerformance still counting successful sending when sending failed
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi commented on PR #13348:
URL: https://github.com/apache/kafka/pull/13348#issuecomment-1471460665
>
> > If all variables are volatile or atomic, will it affect the printing performance of this perf program?
>
> Or we just handle the `iteration` as it seems `iteration` is the only variable used by main thread and producer thread at the same time.
I understand what you mean, because this is a separate perf producer program, there should be no need to consider the scenario of multiple producer threads, because this perf program is sent by one producer, but this problem is indeed caused by the competition between the main thread and the producer thread , so only this variable needs to be fixed.
In addition, does this repair need to open a new jira?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13348: MINOR: Fix ProducerPerformance still counting successful sending when sending failed
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi commented on PR #13348:
URL: https://github.com/apache/kafka/pull/13348#issuecomment-1471428193
> > Maybe my thread safety explanation is incorrect, I'll look again tomorrow.
>
> the thread safety explanation is correct to me. My point was that we should not only fix `iteration` but also fix other mutable stat values.
If all variables are volatile or atomic, will it affect the printing performance of this perf program?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] chia7712 commented on pull request #13348: MINOR: Fix ProducerPerformance still counting successful sending when sending failed
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #13348:
URL: https://github.com/apache/kafka/pull/13348#issuecomment-1471448616
> If all variables are volatile or atomic, will it affect the printing performance of this perf program?
Or we just handle the `iteration` as it seems `iteration` is the only variable used by main thread and producer thread at the same time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13348: MINOR: Fix ProducerPerformance still counting successful sending when sending failed
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi commented on PR #13348:
URL: https://github.com/apache/kafka/pull/13348#issuecomment-1471469700
> > In addition, does this repair need to open a new jira?
>
> In fact, we should revert this PR first. The solution offered by this PR bring antoher issue (thanks to @robobario). The origin issue is worth having a JIRA since it is related to a true bug.
Ok, that is to say, I need to create a new jira, and then report back the original problem that failed to send but still recorded success, and solve the current problem at the same time, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13348: MINOR: Fix ProducerPerformance still counting successful sending when sending failed
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi commented on PR #13348:
URL: https://github.com/apache/kafka/pull/13348#issuecomment-1458087082
Can someone take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13348: MINOR: Fix ProducerPerformance still counting successful sending when sending failed
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi commented on PR #13348:
URL: https://github.com/apache/kafka/pull/13348#issuecomment-1470060354
Thx, Is this [KIP](https://cwiki.apache.org/confluence/display/KAFKA/KIP-842%3A+Add+richer+group+offset+reset+mechanisms) interested in understanding and discussing? @chia7712 @showuon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13348: MINOR: Fix ProducerPerformance still counting successful sending when sending failed
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi commented on PR #13348:
URL: https://github.com/apache/kafka/pull/13348#issuecomment-1471374895
> Hi, I think this introduced a race condition that can cause the producer-perf-test to log a lot of exceptions to console if num records > 500000. Since the `iteration` is incremented from the callback, multiple threads may see the same value when deciding whether to update `latencies`, so it can go out of bounds when updating `latencies`.
>
> ```
> bin/kafka-producer-perf-test.sh --topic perf-test --throughput -1 --num-records 1000000 --producer-props acks=all bootstrap.servers=localhost:9092 --record-size 50
> ```
>
> Sometimes it logs:
>
> ```
> [2023-03-16 15:30:57,432] ERROR Error executing user-provided callback on message for topic-partition 'perf-test-0' (org.apache.kafka.clients.producer.internals.ProducerBatch)
> java.lang.ArrayIndexOutOfBoundsException: Index 500001 out of bounds for length 500001
> at org.apache.kafka.tools.ProducerPerformance$Stats.record(ProducerPerformance.java:38
> ```
>
> It doesn't kill the test run but it looks alarming to see a couple of hundred errors in console.
nice catch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] chia7712 merged pull request #13348: MINOR: Fix ProducerPerformance still counting successful sending when sending failed
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 merged PR #13348:
URL: https://github.com/apache/kafka/pull/13348
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] robobario commented on pull request #13348: MINOR: Fix ProducerPerformance still counting successful sending when sending failed
Posted by "robobario (via GitHub)" <gi...@apache.org>.
robobario commented on PR #13348:
URL: https://github.com/apache/kafka/pull/13348#issuecomment-1471191098
Hi, I think this introduced a race condition that can cause the producer-perf-test to log a lot of exceptions to console if num records > 500000. Since the `iteration` is incremented from the callback, multiple threads may see the same value when deciding whether to update `latencies`, so it can go out of bounds when updating `latencies`.
```
bin/kafka-producer-perf-test.sh --topic perf-test --throughput -1 --num-records 1000000 --producer-props acks=all bootstrap.servers=localhost:9092 --record-size 50
```
Sometimes it logs:
```
[2023-03-16 15:30:57,432] ERROR Error executing user-provided callback on message for topic-partition 'perf-test-0' (org.apache.kafka.clients.producer.internals.ProducerBatch)
java.lang.ArrayIndexOutOfBoundsException: Index 500001 out of bounds for length 500001
at org.apache.kafka.tools.ProducerPerformance$Stats.record(ProducerPerformance.java:38
```
It doesn't kill the test run but it looks alarming to see a couple of hundred errors in console.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on pull request #13348: MINOR: Fix ProducerPerformance still counting successful sending when sending failed
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on PR #13348:
URL: https://github.com/apache/kafka/pull/13348#issuecomment-1471333242
@robobario , nice catch! Welcome to submit a PR to fix it. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi closed pull request #13348: MINOR: Fix ProducerPerformance still counting successful sending when sending failed
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi closed pull request #13348: MINOR: Fix ProducerPerformance still counting successful sending when sending failed
URL: https://github.com/apache/kafka/pull/13348
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] chia7712 commented on a diff in pull request #13348: MINOR: Fix ProducerPerformance still counting successful sending when sending failed
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #13348:
URL: https://github.com/apache/kafka/pull/13348#discussion_r1132455184
##########
tools/src/main/java/org/apache/kafka/tools/ProducerPerformance.java:
##########
@@ -113,7 +113,7 @@ void start(String[] args) throws IOException {
record = new ProducerRecord<>(topicName, payload);
long sendStartMs = System.currentTimeMillis();
- Callback cb = stats.nextCompletion(sendStartMs, payload.length, stats);
+ cb = stats.nextCompletion(sendStartMs, payload.length, stats);
Review Comment:
Does it need to pass itself (`stats`) to nextCompletion? It seems to me line#399 could be replaced by `new PerfCallback(this.iteration, start, bytes, this);`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13348: MINOR: Fix ProducerPerformance still counting successful sending when sending failed
Posted by "hudeqi (via GitHub)" <gi...@apache.org>.
hudeqi commented on PR #13348:
URL: https://github.com/apache/kafka/pull/13348#issuecomment-1463199545
> >
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] chia7712 commented on pull request #13348: MINOR: Fix ProducerPerformance still counting successful sending when sending failed
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #13348:
URL: https://github.com/apache/kafka/pull/13348#issuecomment-1471425767
> Maybe my thread safety explanation is incorrect, I'll look again tomorrow.
the thread safety explanation is correct to me. My point was that we should not only fix `iteration` but also fix other mutable stat values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] robobario commented on pull request #13348: MINOR: Fix ProducerPerformance still counting successful sending when sending failed
Posted by "robobario (via GitHub)" <gi...@apache.org>.
robobario commented on PR #13348:
URL: https://github.com/apache/kafka/pull/13348#issuecomment-1471424085
Maybe my thread safety explanation is incorrect, I'll look again tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org