You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Sneha Vijayarajan (Jira)" <ji...@apache.org> on 2020/12/01 19:07:00 UTC
[jira] [Updated] (HADOOP-17404) ABFS: Piggyback flush on Append
calls for short writes
[ https://issues.apache.org/jira/browse/HADOOP-17404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sneha Vijayarajan updated HADOOP-17404:
---------------------------------------
Description:
When Hflush or Hsync APIs are called, a call is made to store backend to commit the data that was appended.
If the data size written by Hadoop app is small, i.e. data size :
* before any of HFlush/HSync call is made or
* between 2 HFlush/Hsync API calls
is less than write buffer size, 2 separate calls, one for append and another for flush is made,
Apps that do such small writes eventually end up with almost similar number of calls for flush and append.
This PR enables Flush to be piggybacked onto append call for such short write scenarios.
NOTE: The changes is guarded over a config, and is disabled by default until relevant supported changes is made available on all store production clusters.
New Config added: fs.azure.write.enableappendwithflush
was:
When Hflush or Hsync APIs are called, a call is made to store backend to commit the data that was appended.
If the data size written by Hadoop app is small, i.e. data size :
* before any of HFlush/HSync call is made or
* between 2 HFlush/Hsync API calls
is less than write buffer size, 2 separate calls, one for append and another for flush is made,
Apps that do such small writes eventually end up with almost similar number of calls for flush and append.
This PR enables Flush to be piggybacked onto append call for such short write scenarios.
NOTE: The changes is guarded over a config, and is disabled by default until relevant supported changes is made available on all store production clusters.
> ABFS: Piggyback flush on Append calls for short writes
> ------------------------------------------------------
>
> Key: HADOOP-17404
> URL: https://issues.apache.org/jira/browse/HADOOP-17404
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 3.3.0
> Reporter: Sneha Vijayarajan
> Assignee: Sneha Vijayarajan
> Priority: Major
> Fix For: 3.3.1
>
>
> When Hflush or Hsync APIs are called, a call is made to store backend to commit the data that was appended.
> If the data size written by Hadoop app is small, i.e. data size :
> * before any of HFlush/HSync call is made or
> * between 2 HFlush/Hsync API calls
> is less than write buffer size, 2 separate calls, one for append and another for flush is made,
> Apps that do such small writes eventually end up with almost similar number of calls for flush and append.
> This PR enables Flush to be piggybacked onto append call for such short write scenarios.
>
> NOTE: The changes is guarded over a config, and is disabled by default until relevant supported changes is made available on all store production clusters.
> New Config added: fs.azure.write.enableappendwithflush
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org