[jira] [Created] (HDFS-6365) slow performance when reading big file (~10G)

["Jifeng Yin (JIRA)" <ji...@apache.org>]

Jifeng Yin created HDFS-6365:
--------------------------------

             Summary: slow performance when reading big file (~10G)
                 Key: HDFS-6365
                 URL: https://issues.apache.org/jira/browse/HDFS-6365
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: nfs
    Affects Versions: 2.4.0
            Reporter: Jifeng Yin


Mount options:
{code}rw,noatime,nodiratime,vers=3,rsize=65536,wsize=65536,hard,nolock,proto=tcp,timeo=600,retrans=2,sec=sys{code}

Read a big file (~10G) ,
{code}
time dd if=/file/from/hdfs bs=4M | pv | dd of=/dev/null
{code}

Results:
First time:
1. ~160MB/s at first, without cached memory increased on the nfs server
2. ~40MB/s with cached memory increased ( cached memory is always kept).
logs show a ton of the following records:
{quote}
2014-05-09 15:36:13,819 DEBUG org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3: READ_RPC_CALL_END______786375175
2014-05-09 15:36:13,819 DEBUG org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3: READ_RPC_CALL_START____803152391
2014-05-09 15:36:13,820 DEBUG org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3: NFS READ fileId: 17258 offset: 3243958272 count: 65536
2014-05-09 15:36:13,820 DEBUG org.apache.hadoop.hdfs.nfs.nfs3.WriteManager: No opened stream for fileId:17258 commitOffset=3244023808. Return success in this case.
{quote}

Second time:
the same as first time 2 phrase.



--
This message was sent by Atlassian JIRA
(v6.2#6252)