[jira] [Commented] (SAMZA-223) Add a cache for stream metadata

["Chris Riccomini (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SAMZA-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13960945#comment-13960945 ] 

Chris Riccomini commented on SAMZA-223:
---------------------------------------

I agree that we should keep this around. It's just a nice abstraction, and I *know* we will want to get new metadata at some point, so it's good to have in the back pocket.

One thing that I still think might be useful to this patch is a way to fetch the latest data and by-pass the cache. This is a slow operation, but if we have control over when it's done in the container, it might be handy. Not sure if I am over thinking it now. Just a thought.

> Add a cache for stream metadata
> -------------------------------
>
>                 Key: SAMZA-223
>                 URL: https://issues.apache.org/jira/browse/SAMZA-223
>             Project: Samza
>          Issue Type: Improvement
>            Reporter: Martin Kleppmann
>            Assignee: Martin Kleppmann
>         Attachments: SAMZA-223.1.patch
>
>
> As part of SAMZA-179 I wrote a cache to reduce the number of calls to SystemAdmin.getSystemStreamMetadata (which go to the broker). It's currently unclear whether we still want to use the stream metadata to determine whether a task is caught up, so I've split the StreamMetadataCache out of SAMZA-179, and made it a standalone patch. This patch doesn't contain any of the job shutdown stuff, only the metadata caching.
> Admittedly there's less of a need for this cache if we're not requesting stream metadata within tasks, so perhaps we should just throw this code away. But it may also be useful, and certainly shouldn't do any harm. What do you think?



--
This message was sent by Atlassian JIRA
(v6.2#6252)