You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Alexey Raga (JIRA)" <ji...@apache.org> on 2016/05/05 04:32:12 UTC
[jira] [Commented] (KAFKA-3656) Avoid stressing system more when
already under stress
[ https://issues.apache.org/jira/browse/KAFKA-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271876#comment-15271876 ]
Alexey Raga commented on KAFKA-3656:
------------------------------------
Thanks, are the binaries published somewhere?
> Avoid stressing system more when already under stress
> -----------------------------------------------------
>
> Key: KAFKA-3656
> URL: https://issues.apache.org/jira/browse/KAFKA-3656
> Project: Kafka
> Issue Type: Bug
> Reporter: Alexey Raga
> Assignee: Liquan Pei
> Fix For: 0.10.1.0, 0.10.0.0
>
>
> I am working with Kafka Connect now and I am having error messages like that:
> {code}
> [2016-05-04 03:11:28,226] ERROR Failed to flush WorkerSourceTask{id=geo-connector-0}, timed out while waiting for producer to flush outstanding messages, 151860 left ([FAILED toString()]) (org.apache.kafka.connect.runtime.WorkerSourceTask:237)
> [2016-05-04 03:11:28,227] ERROR Failed to commit offsets for WorkerSourceTask{id=geo-connector-0} (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:112)
> {code}
> I didn't figure out the reason why Connect would pull so many records into memory when it clearly can't produce that fast and I don't yet know why producing messages is slow.
> But the part of {{151860 left ([FAILED toString()]) }} is interesting and I looked at the code and found this:
> {code}
> if (timeoutMs <= 0) {
> log.error(
> "Failed to flush {}, timed out while waiting for producer to flush outstanding "
> + "messages, {} left ({})", this, outstandingMessages.size(), outstandingMessages);
> finishFailedFlush();
> return false;
> }
> {code}
> So when the connector is under stress and, assuming {{151860}} messages, under a heavy memory pressure the code choses to take pretty much {{4 * 151860}} byte arrays and to convert it to a java string.
> This not only eats more memory and adds to GC, but is also useless for logging because the actual string, if it wouldn't fail, would look like:
> {code}
> (topic=lamington--geo-connector, partition=null, key=null, value=[B@62c66f62=ProducerRecord(topic=lamington--geo-connector, partition=null, key=null, value=[B@62c66f62, ProducerRecord(topic=lamington--geo-connector, partition=null, key=null, .....
> {code}
> I think it is a bug and a string representation of the outstanding messages should be removed from the log.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)