You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/10/02 16:14:00 UTC
[jira] [Commented] (APEXMALHAR-2223) Managed state should
parallelize WAL writes
[ https://issues.apache.org/jira/browse/APEXMALHAR-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16188368#comment-16188368 ]
ASF GitHub Bot commented on APEXMALHAR-2223:
--------------------------------------------
vrozov closed pull request #438: APEXMALHAR-2223 ability to buffer data up to a threshold and write it…
URL: https://github.com/apache/apex-malhar/pull/438
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
> Managed state should parallelize WAL writes
> -------------------------------------------
>
> Key: APEXMALHAR-2223
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2223
> Project: Apache Apex Malhar
> Issue Type: Improvement
> Affects Versions: 3.4.0
> Reporter: Thomas Weise
> Assignee: Chandni Singh
>
> Currently, data is accumulated in memory and written to the WAL on checkpoint only. This causes a write spike on checkpoint and does not utilize the HDFS write pipeline. The other extreme is writing to the WAL as soon as data arrives and then only flush in beforeCheckpoint. The downside of this is that when the same key is written many times, all duplicates will be in the WAL. Need to find a balances approach, that the user can potentially fine tune.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)