You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Haiyang Hu (Jira)" <ji...@apache.org> on 2023/02/07 11:25:00 UTC
[jira] [Updated] (HDFS-16910) Fix incorrectly initializing RandomAccessFile caused PRC performance decreased for JN
[ https://issues.apache.org/jira/browse/HDFS-16910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Haiyang Hu updated HDFS-16910:
------------------------------
Description:
At present, after our cluster backport patch HDFS-15882,
when set shouldSyncWritesAndSkipFsync to false, there will be PRC performance degradation caused by JN.
Root Cause：
when setting shouldSyncWritesAndSkipFsync to false, the mode of init RandomAccessFile will be `rws`.
even if fc.force(false) is executed when flushAndSync is executed (hopefully, only requires updates to the file's content to be written to storage and the metadata is not update),
but since the mode of RandomAccessFile is `rws`, It will requires updates to both the file's content and its metadata to be written,
there will be PRC performance degradation caused by JN.
Fix:
Need to update RandomAccessFile's mode from `rws` to `rwd`:
rwd: Open for reading and writing, as with "rw", and also require that every update to the file's content be written synchronously to the underlying storage device.
{code:java}
if (shouldSyncWritesAndSkipFsync) {
rp = new RandomAccessFile(name, "rwd");
} else {
rp = new RandomAccessFile(name, "rw");
}
{code}
In this way, when flushAndSync is executed,
if shouldSyncWritesAndSkipFsync is false and the mode of RandomAccessFile is 'rw', it will call fc.force(false) to execute,
otherwise should use `rwd` to perform the operation.
> Fix incorrectly initializing RandomAccessFile caused PRC performance decreased for JN
> -------------------------------------------------------------------------------------
>
> Key: HDFS-16910
> URL: https://issues.apache.org/jira/browse/HDFS-16910
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Haiyang Hu
> Assignee: Haiyang Hu
> Priority: Major
>
> At present, after our cluster backport patch HDFS-15882,
> when set shouldSyncWritesAndSkipFsync to false, there will be PRC performance degradation caused by JN.
> Root Cause：
> when setting shouldSyncWritesAndSkipFsync to false, the mode of init RandomAccessFile will be `rws`.
> even if fc.force(false) is executed when flushAndSync is executed (hopefully, only requires updates to the file's content to be written to storage and the metadata is not update),
> but since the mode of RandomAccessFile is `rws`, It will requires updates to both the file's content and its metadata to be written,
> there will be PRC performance degradation caused by JN.
> Fix:
> Need to update RandomAccessFile's mode from `rws` to `rwd`:
> rwd: Open for reading and writing, as with "rw", and also require that every update to the file's content be written synchronously to the underlying storage device.
> {code:java}
> if (shouldSyncWritesAndSkipFsync) {
> rp = new RandomAccessFile(name, "rwd");
> } else {
> rp = new RandomAccessFile(name, "rw");
> }
> {code}
> In this way, when flushAndSync is executed,
> if shouldSyncWritesAndSkipFsync is false and the mode of RandomAccessFile is 'rw', it will call fc.force(false) to execute,
> otherwise should use `rwd` to perform the operation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org