You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by "Yan Fang (JIRA)" <ji...@apache.org> on 2015/09/17 00:06:45 UTC
[jira] [Commented] (SAMZA-775) Specify prefetch buffer in terms of
bytes rather than count using fetchThreshold
[ https://issues.apache.org/jira/browse/SAMZA-775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14791196#comment-14791196 ]
Yan Fang commented on SAMZA-775:
--------------------------------
Agree. [~nickmlee] mentioned the same requirement in SAMZA-772. If you can provide a patch for this, it will be great. Thank you!
> Specify prefetch buffer in terms of bytes rather than count using fetchThreshold
> --------------------------------------------------------------------------------
>
> Key: SAMZA-775
> URL: https://issues.apache.org/jira/browse/SAMZA-775
> Project: Samza
> Issue Type: Improvement
> Reporter: Monal Daxini
> Assignee: Monal Daxini
>
> Currently the only way to control the prefetch buffer is count based using the systems.system-name.samza.fetch.threshold configuration. However, in the presence of variable sized messages this make it very hard to allocate determisnistic memory resources to a SamzaContainer.
> This JIRA is a for an improvment to allow tuning the prefetch buffer based on bytes as well using a new config like systems.source.samza.fetch.bytes. When this is present the threshold could be safely ignored.
> This is an extremely important feature for us as it allowed us to stabilize our platform as we have deterministic allocations.
> I have a patch of this against 0.9.1 running in canaries at scale in prod and is looking promising so far.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)