[jira] [Updated] (TWILL-204) AM should keep Kafka broker up until log consumers finish

["Terence Yim (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/TWILL-204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Terence Yim updated TWILL-204:
------------------------------
    Affects Version/s: 0.11.0
                       0.10.0

> AM should keep Kafka broker up until log consumers finish
> ---------------------------------------------------------
>
>                 Key: TWILL-204
>                 URL: https://issues.apache.org/jira/browse/TWILL-204
>             Project: Apache Twill
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 0.9.0, 0.10.0, 0.11.0
>            Reporter: Martin Serrano
>             Fix For: 0.12.0
>
>
> I'm encountering a situation on a fast machine where the Kafka log aggregation topic is not empty when the system shuts down.  The scenario:
>      log consumer consumes all messages
>     consumer sleeps (500ms) due to empty queue
>     containers exit, posting final log messages about why
>     controller notices containers are down and terminates consumers.
>     consumer is interrupted from sleep and but has been canceled so it does not get the rest of the messages.
> This scenario can be really confusing during development because an error may be missed (as in my case) if it falls into the final log messages.  Before I file a ticket and fix this, I wanted to get some feedback.  Looking at org.apache.twill.internal.kafka.client.SimpleKafkaConsumer it seems this behavior could be intentional given this log message (line 384):
>             LOG.debug("Unable to fetch messages on {}, kafka consumer service shutdown is in progress.", topicPart);
> My opinion is that final messages logged by a container are likely to be critical in diagnosing errors and that twill should do whatever it can to forward them before shutting things down.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)