[jira] [Commented] (KAFKA-560) Garbage Collect obsolete topics

["Neha Narkhede (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13473829#comment-13473829 ] 

Neha Narkhede commented on KAFKA-560:
-------------------------------------

OffsetRequest will not give you the time, but the offsets since a time (approximately). 

However, the original proposal for TopicMetadataRequest had a field for last modified timestamp per partition - 


/**
 * topic (2 bytes + topic.length)
 * number of partitions (4 bytes)
 *
 * partition id (4 bytes)
 *
 * does leader exist (1 byte)
 * leader info (4 + creator.length + host.length + 4 (port) + 4 (id))
 * number of replicas (2 bytes)
 * replica info (4 + creator.length + host.length + 4 (port) + 4 (id))
 * number of in sync replicas (2 bytes)
 * replica info (4 + creator.length + host.length + 4 (port) + 4 (id))
 *
 * does log metadata exist (1 byte)
 * number of log segments (4 bytes)
 * total size of log in bytes (8 bytes)
 *
 * number of log segments (4 bytes)
 * beginning offset (8 bytes)
 * last modified timestamp (8 bytes)
 * size of log segment (8 bytes)
 *
 */

However, we haven't really implemented the last few fields of this request, simply because we couldn't think of a use case. But what you describe seems useful and maybe we should file a JIRA to get the log segment portion of TopicMetadatRequest. 

                
> Garbage Collect obsolete topics
> -------------------------------
>
>                 Key: KAFKA-560
>                 URL: https://issues.apache.org/jira/browse/KAFKA-560
>             Project: Kafka
>          Issue Type: New Feature
>            Reporter: Jay Kreps
>
> Old junk topics tend to accumulate over time. Code may migrate to use new topics leaving the old ones orphaned. Likewise there are some use cases for temporary transient topics. It would be good to have a tool that could delete any topic that had not been written to in a configurable period of time and had no active consumer groups. Something like
>    ./bin/delete-unused-topics.sh --last-write [date] --zookeeper [zk_connect]
> This requires API support to get the last update time. I think it may be possible to do this through the OffsetRequest now?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira