You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@metron.apache.org by "Casey Stella (JIRA)" <ji...@apache.org> on 2017/06/27 09:14:00 UTC

[jira] [Updated] (METRON-955) Make the default sync policy for HDFS Writer be based on the batch size

     [ https://issues.apache.org/jira/browse/METRON-955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Casey Stella updated METRON-955:
--------------------------------
    Fix Version/s: 0.4.0

> Make the default sync policy for HDFS Writer be based on the batch size
> -----------------------------------------------------------------------
>
>                 Key: METRON-955
>                 URL: https://issues.apache.org/jira/browse/METRON-955
>             Project: Metron
>          Issue Type: Improvement
>            Reporter: Casey Stella
>            Assignee: Casey Stella
>             Fix For: 0.4.0
>
>
> Right now, we do a sync per record written in HDFS.  This has performance penalties as it requires a Namenode contact.  We get around this by letting the user define the sync policy in Flux, but that's less convenient and more sensible defaults could be created.  We should use the batch size as a sensible default for the sync policy (i.e. sync every batch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)