You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Azuryy Yu <az...@gmail.com> on 2014/03/09 13:15:02 UTC
Fwd: problem with HDFS caching in Hadoop 2.3
add dev.
---------- Forwarded message ----------
From: "hwpstorage" <hw...@gmail.com>
Date: Mar 7, 2014 11:38 PM
Subject: problem with HDFS caching in Hadoop 2.3
To: <us...@hadoop.apache.org>
Cc:
Hello,
It looks like the HDFS caching does not work well.
The cached log file is around 200MB. The hadoop cluster has 3 nodes, each
has 4GB memory.
-bash-4.1$ hdfs cacheadmin -addPool wptest1
Successfully added cache pool wptest1.
-bash-4.1$ /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listPools
Found 1 result.
NAME OWNER GROUP MODE LIMIT MAXTTL
wptest1 hdfs hdfs rwxr-xr-x unlimited never
-bash-4.1$ hdfs cacheadmin -addDirective -path hadoop003.log -pool wptest1
Added cache directive 1
-bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
real 0m2.796s
user 0m4.263s
sys 0m0.203s
-bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
real 0m3.050s
user 0m4.176s
sys 0m0.192s
It is weird that the cache status shows 0 byte cached:-bash-4.1$
/hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listDirectives -stats -path
hadoop003.log -pool wptest1
Found 1 entry
ID POOL REPL EXPIRY PATH BYTES_NEEDED
BYTES_CACHED FILES_NEEDED FILES_CACHED
1 wptest1 1 never /user/hdfs/hadoop003.log
209715206 0 1 0
-bash-4.1$ file /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0
/hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0: ELF 64-bit LSB shared
object, x86-64, version 1 (SYSV), dynamically linked, not stripped
I also tried the word count example with the same file. The execution time
is always 40 seconds. (The map/reduce job without cache is 42 seconds)
Is there anything wrong?
Thanks a lot
Re: problem with HDFS caching in Hadoop 2.3
Posted by Andrew Wang <an...@cloudera.com>.
How was this cluster configured? Run dfsadmin -report to see the aggregate
configured cache capacity as seen by the NN. You need to configure some GBs
of cache on each DN and also raise the ulimit for max locked memory.
I'll also note that you are unlikely to see speedups with most mapreduce
jobs, especially running against TextInputFormat. There are a lot of copies
and string splitting, so it's typically not I/O bound.
The fs -tail command is likely also spending a lot of time on startup
costs, so I wouldn't expect much end-to-end latency savings.
Best,
Andrew
On Sun, Mar 9, 2014 at 5:15 AM, Azuryy Yu <az...@gmail.com> wrote:
> add dev.
> ---------- Forwarded message ----------
> From: "hwpstorage" <hw...@gmail.com>
> Date: Mar 7, 2014 11:38 PM
> Subject: problem with HDFS caching in Hadoop 2.3
> To: <us...@hadoop.apache.org>
> Cc:
>
> Hello,
>
> It looks like the HDFS caching does not work well.
> The cached log file is around 200MB. The hadoop cluster has 3 nodes, each
> has 4GB memory.
>
> -bash-4.1$ hdfs cacheadmin -addPool wptest1
> Successfully added cache pool wptest1.
>
> -bash-4.1$ /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listPools
> Found 1 result.
> NAME OWNER GROUP MODE LIMIT MAXTTL
> wptest1 hdfs hdfs rwxr-xr-x unlimited never
>
> -bash-4.1$ hdfs cacheadmin -addDirective -path hadoop003.log -pool wptest1
> Added cache directive 1
>
> -bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> real 0m2.796s
> user 0m4.263s
> sys 0m0.203s
>
> -bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> real 0m3.050s
> user 0m4.176s
> sys 0m0.192s
>
> It is weird that the cache status shows 0 byte cached:-bash-4.1$
> /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listDirectives -stats -path
> hadoop003.log -pool wptest1
> Found 1 entry
> ID POOL REPL EXPIRY PATH BYTES_NEEDED
> BYTES_CACHED FILES_NEEDED FILES_CACHED
> 1 wptest1 1 never /user/hdfs/hadoop003.log
> 209715206 0 1 0
>
> -bash-4.1$ file /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0
> /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0: ELF 64-bit LSB shared
> object, x86-64, version 1 (SYSV), dynamically linked, not stripped
>
> I also tried the word count example with the same file. The execution time
> is always 40 seconds. (The map/reduce job without cache is 42 seconds)
> Is there anything wrong?
> Thanks a lot
>