You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Marc-Olivier Fleury (JIRA)" <ji...@apache.org> on 2008/12/03 15:49:44 UTC

[jira] Issue Comment Edited: (HADOOP-4752) Major performance drop on slower machines

    [ https://issues.apache.org/jira/browse/HADOOP-4752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12652813#action_12652813 ] 

mofleury edited comment on HADOOP-4752 at 12/3/08 6:48 AM:
----------------------------------------------------------------------

I am using the default buffer size, which actually seems to be 10MB. (in main(), line 1793). I forgot to mention that the files I am reading are not large (10 - 100 KB), which implies that a 10MB buffer should be able to contain each file.

I noticed a strange little piece of code, that is repeated in nearly all operations:

{code}
hdfsFS userFS;
// if not connected, try to connect and fail out if we can't.
if ((userFS = doConnectAsUser(dfs->nn_hostname,dfs->nn_port))== NULL) {
    syslog(LOG_ERR, "ERROR: could not connect to dfs %s:%d\n", __FILE__, __LINE__);
    return -EIO;
}
{code}

The comment states that the connection is attempted only if not yet connected, but the code actually reconnects in any case. This part of the code was probably changed when permissions were introduced, making it necessary to handle multiple different connections, instead of a single one, making the use of dfs_context.fs impossible. However, if the only reason for reconnecting is to handle multiple users, a better solution would probably be to store multiple filesystem handles, one per user, which would prevent from reconnecting at each operation. In any case, when permissions are disabled at compilation time (which is what I am using now), the reconnection should be avoided.

Assuming that the connection process is not a simple operation, this could probably have a real impact on the performance.

There is probably a detail that I did not catch which makes the current implementation needed, but any insight would be greatly appreciated.


      was (Author: mofleury):
    I am using the default buffer size, which actually seems to be 10MB. (in main(), line 1793). I forgot to mention that the files I am reading are not large (10 - 100 KB), which implies that a 10MB buffer should be able to contain each file.

I noticed a strange little piece of code, that is repeated in nearly all operations:

{code}
hdfsFS userFS;
// if not connected, try to connect and fail out if we can't.
if ((userFS = doConnectAsUser(dfs->nn_hostname,dfs->nn_port))== NULL) {
    syslog(LOG_ERR, "ERROR: could not connect to dfs %s:%d\n", __FILE__, __LINE__);
    return -EIO;
}
{/code}

The comment states that the connection is attempted only if not yet connected, but the code actually reconnects in any case. This part of the code was probably changed when permissions were introduced, making it necessary to handle multiple different connections, instead of a single one, making the use of dfs_context.fs impossible. However, if the only reason for reconnecting is to handle multiple users, a better solution would probably be to store multiple filesystem handles, one per user, which would prevent from reconnecting at each operation. In any case, when permissions are disabled at compilation time (which is what I am using now), the reconnection should be avoided.

Assuming that the connection process is not a simple operation, this could probably have a real impact on the performance.

There is probably a detail that I did not catch which makes the current implementation needed, but any insight would be greatly appreciated.

  
> Major performance drop on slower machines
> -----------------------------------------
>
>                 Key: HADOOP-4752
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4752
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: contrib/fuse-dfs
>    Affects Versions: 0.18.2
>            Reporter: Marc-Olivier Fleury
>
> When running fuse_dfs on machines that have different CPU characteristics, I noticed that the performance of fuse_dfs is very sensitive to the machine power. 
> The command I used was simply a cat over a rather large amount of data stored on HDFS. Here are the comparative times for the different types of machines:
> Intel(R) Pentium(R) 4 CPU 2.40GHz :                                2 min 40 s 
> Intel(R) Pentium(R) 4 CPU 3.06GHz:                                 1 min 50 s 
> 2 x Intel(R) Pentium(R) 4 CPU 3.00GHz:                           0 min 40 s 
> 2 x Intel(R) Xeon(TM) MP CPU 3.33GHz:                           0 min 28 s 
> Intel(R) Core(TM)2 Quad CPU    Q6600  @ 2.40GHz      0 min 15 s
> I tried to find other explanations for the drop in performance, such as network configuration, or data locality, but the faster machines are the ones that are "further away" from the others considering the network configuration, and that don't run datanodes.
> top shows that the CPU usage of fuse_dfs is between 80-90% on the slower machines, and about 40% on the fastest one.
> This leads me to the conclusion that fuse_dfs consumes a lot of CPU resources, much more than expected.
> Any help or insight concerning this issue will be greatly appreciated, since these difference actually result in days of computations for a given job.
> Thank you

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.