You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by hwpstorage <hw...@gmail.com> on 2014/03/07 16:37:38 UTC
problem with HDFS caching in Hadoop 2.3
Hello,
It looks like the HDFS caching does not work well.
The cached log file is around 200MB. The hadoop cluster has 3 nodes, each
has 4GB memory.
-bash-4.1$ hdfs cacheadmin -addPool wptest1
Successfully added cache pool wptest1.
-bash-4.1$ /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listPools
Found 1 result.
NAME OWNER GROUP MODE LIMIT MAXTTL
wptest1 hdfs hdfs rwxr-xr-x unlimited never
-bash-4.1$ hdfs cacheadmin -addDirective -path hadoop003.log -pool wptest1
Added cache directive 1
-bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
real 0m2.796s
user 0m4.263s
sys 0m0.203s
-bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
real 0m3.050s
user 0m4.176s
sys 0m0.192s
It is weird that the cache status shows 0 byte cached:-bash-4.1$
/hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listDirectives -stats -path
hadoop003.log -pool wptest1
Found 1 entry
ID POOL REPL EXPIRY PATH BYTES_NEEDED
BYTES_CACHED FILES_NEEDED FILES_CACHED
1 wptest1 1 never /user/hdfs/hadoop003.log
209715206 0 1 0
-bash-4.1$ file /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0
/hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0: ELF 64-bit LSB shared
object, x86-64, version 1 (SYSV), dynamically linked, not stripped
I also tried the word count example with the same file. The execution time
is always 40 seconds. (The map/reduce job without cache is 42 seconds)
Is there anything wrong?
Thanks a lot
Re: problem with HDFS caching in Hadoop 2.3
Posted by hwpstorage <hw...@gmail.com>.
Thanks for the reply.
I used the cat command this time, the result is not great.
In my test, file hadoop003.log is cached while hadoop010.log is not cached.
-bash-4.1$ /hadoop/hadoop-2.3.0/bin/hadoop fs -ls
-rw-r--r-- 3 hdfs hadoop 209715206 2014-03-06 18:14 hadoop003.log
-rw-r--r-- 3 hdfs hadoop 209715272 2014-03-07 14:37 hadoop010.log
-bash-4.1$ hdfs cacheadmin -listDirectives -stats -path hadoop003.log
Found 1 entry
ID POOL REPL EXPIRY PATH BYTES_NEEDED
BYTES_CACHED FILES_NEEDED FILES_CACHED
5 wptest1 3 never /user/hdfs/hadoop003.log 629145618
629145618 1 1
run first time
-bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -cat hadoop003.log>
/tmp/aa
real 0m4.881s
user 0m4.805s
sys 0m1.468s
-bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -cat hadoop010.log>
/tmp/aa
real 0m6.479s
user 0m4.777s
sys 0m1.312s
run 2nd time.
-bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -cat hadoop003.log>
/tmp/aa
real 0m4.751s
user 0m4.685s
sys 0m1.313s
-bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -cat hadoop010.log>
/tmp/aa
real 0m4.916s
user 0m4.779s
sys 0m1.378s
I did not see much cache improvement.
please advice.
Thanks
On Tue, Mar 11, 2014 at 3:55 PM, Colin McCabe <cm...@alumni.cmu.edu>wrote:
> On Fri, Mar 7, 2014 at 7:37 AM, hwpstorage <hw...@gmail.com> wrote:
> > Hello,
> >
> > It looks like the HDFS caching does not work well.
> > The cached log file is around 200MB. The hadoop cluster has 3 nodes, each
> > has 4GB memory.
> >
> > -bash-4.1$ hdfs cacheadmin -addPool wptest1
> > Successfully added cache pool wptest1.
> >
> > -bash-4.1$ /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listPools
> > Found 1 result.
> > NAME OWNER GROUP MODE LIMIT MAXTTL
> > wptest1 hdfs hdfs rwxr-xr-x unlimited never
> >
> > -bash-4.1$ hdfs cacheadmin -addDirective -path hadoop003.log -pool
> wptest1
> > Added cache directive 1
> >
> > -bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> > real 0m2.796s
> > user 0m4.263s
> > sys 0m0.203s
> >
> > -bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> > real 0m3.050s
> > user 0m4.176s
> > sys 0m0.192s
> >
> > It is weird that the cache status shows 0 byte cached:-bash-4.1$
> > /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listDirectives -stats -path
> > hadoop003.log -pool wptest1
> > Found 1 entry
> > ID POOL REPL EXPIRY PATH BYTES_NEEDED
> > BYTES_CACHED FILES_NEEDED FILES_CACHED
> > 1 wptest1 1 never /user/hdfs/hadoop003.log 209715206
> > 0 1 0
>
> If you take a look at this output, you can see that nothing is actually
> cached.
>
> One way to figure out why this is is to look at the logs of the
> NameNode and DataNode. Some of the relevant logs are at DEBUG or
> TRACE level, so you may need to turn up the logs. The
> CacheReplicationMonitor and FsDatasetCache classes are good places to
> start.
>
> Also be sure to check that you have set dfs.datanode.max.locked.memory.
>
> As Andrew commented, "hadoop tail" is not a good command to use for
> measuring performance, since you have a few seconds of Java startup
> time, followed by any HDFS setup time, followed by reading a single
> kilobyte of data. If you want to use the shell, the simplest thing to
> do is to use cat and read a large file, so that those startup costs
> don't dominate the measurement.
>
> best,
> Colin
>
>
> >
> > -bash-4.1$ file /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0
> > /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0: ELF 64-bit LSB shared
> > object, x86-64, version 1 (SYSV), dynamically linked, not stripped
> >
> > I also tried the word count example with the same file. The execution
> time
> > is always 40 seconds. (The map/reduce job without cache is 42 seconds)
> > Is there anything wrong?
> > Thanks a lot
>
Re: problem with HDFS caching in Hadoop 2.3
Posted by hwpstorage <hw...@gmail.com>.
Thanks for the reply.
I used the cat command this time, the result is not great.
In my test, file hadoop003.log is cached while hadoop010.log is not cached.
-bash-4.1$ /hadoop/hadoop-2.3.0/bin/hadoop fs -ls
-rw-r--r-- 3 hdfs hadoop 209715206 2014-03-06 18:14 hadoop003.log
-rw-r--r-- 3 hdfs hadoop 209715272 2014-03-07 14:37 hadoop010.log
-bash-4.1$ hdfs cacheadmin -listDirectives -stats -path hadoop003.log
Found 1 entry
ID POOL REPL EXPIRY PATH BYTES_NEEDED
BYTES_CACHED FILES_NEEDED FILES_CACHED
5 wptest1 3 never /user/hdfs/hadoop003.log 629145618
629145618 1 1
run first time
-bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -cat hadoop003.log>
/tmp/aa
real 0m4.881s
user 0m4.805s
sys 0m1.468s
-bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -cat hadoop010.log>
/tmp/aa
real 0m6.479s
user 0m4.777s
sys 0m1.312s
run 2nd time.
-bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -cat hadoop003.log>
/tmp/aa
real 0m4.751s
user 0m4.685s
sys 0m1.313s
-bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -cat hadoop010.log>
/tmp/aa
real 0m4.916s
user 0m4.779s
sys 0m1.378s
I did not see much cache improvement.
please advice.
Thanks
On Tue, Mar 11, 2014 at 3:55 PM, Colin McCabe <cm...@alumni.cmu.edu>wrote:
> On Fri, Mar 7, 2014 at 7:37 AM, hwpstorage <hw...@gmail.com> wrote:
> > Hello,
> >
> > It looks like the HDFS caching does not work well.
> > The cached log file is around 200MB. The hadoop cluster has 3 nodes, each
> > has 4GB memory.
> >
> > -bash-4.1$ hdfs cacheadmin -addPool wptest1
> > Successfully added cache pool wptest1.
> >
> > -bash-4.1$ /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listPools
> > Found 1 result.
> > NAME OWNER GROUP MODE LIMIT MAXTTL
> > wptest1 hdfs hdfs rwxr-xr-x unlimited never
> >
> > -bash-4.1$ hdfs cacheadmin -addDirective -path hadoop003.log -pool
> wptest1
> > Added cache directive 1
> >
> > -bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> > real 0m2.796s
> > user 0m4.263s
> > sys 0m0.203s
> >
> > -bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> > real 0m3.050s
> > user 0m4.176s
> > sys 0m0.192s
> >
> > It is weird that the cache status shows 0 byte cached:-bash-4.1$
> > /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listDirectives -stats -path
> > hadoop003.log -pool wptest1
> > Found 1 entry
> > ID POOL REPL EXPIRY PATH BYTES_NEEDED
> > BYTES_CACHED FILES_NEEDED FILES_CACHED
> > 1 wptest1 1 never /user/hdfs/hadoop003.log 209715206
> > 0 1 0
>
> If you take a look at this output, you can see that nothing is actually
> cached.
>
> One way to figure out why this is is to look at the logs of the
> NameNode and DataNode. Some of the relevant logs are at DEBUG or
> TRACE level, so you may need to turn up the logs. The
> CacheReplicationMonitor and FsDatasetCache classes are good places to
> start.
>
> Also be sure to check that you have set dfs.datanode.max.locked.memory.
>
> As Andrew commented, "hadoop tail" is not a good command to use for
> measuring performance, since you have a few seconds of Java startup
> time, followed by any HDFS setup time, followed by reading a single
> kilobyte of data. If you want to use the shell, the simplest thing to
> do is to use cat and read a large file, so that those startup costs
> don't dominate the measurement.
>
> best,
> Colin
>
>
> >
> > -bash-4.1$ file /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0
> > /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0: ELF 64-bit LSB shared
> > object, x86-64, version 1 (SYSV), dynamically linked, not stripped
> >
> > I also tried the word count example with the same file. The execution
> time
> > is always 40 seconds. (The map/reduce job without cache is 42 seconds)
> > Is there anything wrong?
> > Thanks a lot
>
Re: problem with HDFS caching in Hadoop 2.3
Posted by hwpstorage <hw...@gmail.com>.
Thanks for the reply.
I used the cat command this time, the result is not great.
In my test, file hadoop003.log is cached while hadoop010.log is not cached.
-bash-4.1$ /hadoop/hadoop-2.3.0/bin/hadoop fs -ls
-rw-r--r-- 3 hdfs hadoop 209715206 2014-03-06 18:14 hadoop003.log
-rw-r--r-- 3 hdfs hadoop 209715272 2014-03-07 14:37 hadoop010.log
-bash-4.1$ hdfs cacheadmin -listDirectives -stats -path hadoop003.log
Found 1 entry
ID POOL REPL EXPIRY PATH BYTES_NEEDED
BYTES_CACHED FILES_NEEDED FILES_CACHED
5 wptest1 3 never /user/hdfs/hadoop003.log 629145618
629145618 1 1
run first time
-bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -cat hadoop003.log>
/tmp/aa
real 0m4.881s
user 0m4.805s
sys 0m1.468s
-bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -cat hadoop010.log>
/tmp/aa
real 0m6.479s
user 0m4.777s
sys 0m1.312s
run 2nd time.
-bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -cat hadoop003.log>
/tmp/aa
real 0m4.751s
user 0m4.685s
sys 0m1.313s
-bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -cat hadoop010.log>
/tmp/aa
real 0m4.916s
user 0m4.779s
sys 0m1.378s
I did not see much cache improvement.
please advice.
Thanks
On Tue, Mar 11, 2014 at 3:55 PM, Colin McCabe <cm...@alumni.cmu.edu>wrote:
> On Fri, Mar 7, 2014 at 7:37 AM, hwpstorage <hw...@gmail.com> wrote:
> > Hello,
> >
> > It looks like the HDFS caching does not work well.
> > The cached log file is around 200MB. The hadoop cluster has 3 nodes, each
> > has 4GB memory.
> >
> > -bash-4.1$ hdfs cacheadmin -addPool wptest1
> > Successfully added cache pool wptest1.
> >
> > -bash-4.1$ /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listPools
> > Found 1 result.
> > NAME OWNER GROUP MODE LIMIT MAXTTL
> > wptest1 hdfs hdfs rwxr-xr-x unlimited never
> >
> > -bash-4.1$ hdfs cacheadmin -addDirective -path hadoop003.log -pool
> wptest1
> > Added cache directive 1
> >
> > -bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> > real 0m2.796s
> > user 0m4.263s
> > sys 0m0.203s
> >
> > -bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> > real 0m3.050s
> > user 0m4.176s
> > sys 0m0.192s
> >
> > It is weird that the cache status shows 0 byte cached:-bash-4.1$
> > /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listDirectives -stats -path
> > hadoop003.log -pool wptest1
> > Found 1 entry
> > ID POOL REPL EXPIRY PATH BYTES_NEEDED
> > BYTES_CACHED FILES_NEEDED FILES_CACHED
> > 1 wptest1 1 never /user/hdfs/hadoop003.log 209715206
> > 0 1 0
>
> If you take a look at this output, you can see that nothing is actually
> cached.
>
> One way to figure out why this is is to look at the logs of the
> NameNode and DataNode. Some of the relevant logs are at DEBUG or
> TRACE level, so you may need to turn up the logs. The
> CacheReplicationMonitor and FsDatasetCache classes are good places to
> start.
>
> Also be sure to check that you have set dfs.datanode.max.locked.memory.
>
> As Andrew commented, "hadoop tail" is not a good command to use for
> measuring performance, since you have a few seconds of Java startup
> time, followed by any HDFS setup time, followed by reading a single
> kilobyte of data. If you want to use the shell, the simplest thing to
> do is to use cat and read a large file, so that those startup costs
> don't dominate the measurement.
>
> best,
> Colin
>
>
> >
> > -bash-4.1$ file /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0
> > /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0: ELF 64-bit LSB shared
> > object, x86-64, version 1 (SYSV), dynamically linked, not stripped
> >
> > I also tried the word count example with the same file. The execution
> time
> > is always 40 seconds. (The map/reduce job without cache is 42 seconds)
> > Is there anything wrong?
> > Thanks a lot
>
Re: problem with HDFS caching in Hadoop 2.3
Posted by hwpstorage <hw...@gmail.com>.
Thanks for the reply.
I used the cat command this time, the result is not great.
In my test, file hadoop003.log is cached while hadoop010.log is not cached.
-bash-4.1$ /hadoop/hadoop-2.3.0/bin/hadoop fs -ls
-rw-r--r-- 3 hdfs hadoop 209715206 2014-03-06 18:14 hadoop003.log
-rw-r--r-- 3 hdfs hadoop 209715272 2014-03-07 14:37 hadoop010.log
-bash-4.1$ hdfs cacheadmin -listDirectives -stats -path hadoop003.log
Found 1 entry
ID POOL REPL EXPIRY PATH BYTES_NEEDED
BYTES_CACHED FILES_NEEDED FILES_CACHED
5 wptest1 3 never /user/hdfs/hadoop003.log 629145618
629145618 1 1
run first time
-bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -cat hadoop003.log>
/tmp/aa
real 0m4.881s
user 0m4.805s
sys 0m1.468s
-bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -cat hadoop010.log>
/tmp/aa
real 0m6.479s
user 0m4.777s
sys 0m1.312s
run 2nd time.
-bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -cat hadoop003.log>
/tmp/aa
real 0m4.751s
user 0m4.685s
sys 0m1.313s
-bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -cat hadoop010.log>
/tmp/aa
real 0m4.916s
user 0m4.779s
sys 0m1.378s
I did not see much cache improvement.
please advice.
Thanks
On Tue, Mar 11, 2014 at 3:55 PM, Colin McCabe <cm...@alumni.cmu.edu>wrote:
> On Fri, Mar 7, 2014 at 7:37 AM, hwpstorage <hw...@gmail.com> wrote:
> > Hello,
> >
> > It looks like the HDFS caching does not work well.
> > The cached log file is around 200MB. The hadoop cluster has 3 nodes, each
> > has 4GB memory.
> >
> > -bash-4.1$ hdfs cacheadmin -addPool wptest1
> > Successfully added cache pool wptest1.
> >
> > -bash-4.1$ /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listPools
> > Found 1 result.
> > NAME OWNER GROUP MODE LIMIT MAXTTL
> > wptest1 hdfs hdfs rwxr-xr-x unlimited never
> >
> > -bash-4.1$ hdfs cacheadmin -addDirective -path hadoop003.log -pool
> wptest1
> > Added cache directive 1
> >
> > -bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> > real 0m2.796s
> > user 0m4.263s
> > sys 0m0.203s
> >
> > -bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> > real 0m3.050s
> > user 0m4.176s
> > sys 0m0.192s
> >
> > It is weird that the cache status shows 0 byte cached:-bash-4.1$
> > /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listDirectives -stats -path
> > hadoop003.log -pool wptest1
> > Found 1 entry
> > ID POOL REPL EXPIRY PATH BYTES_NEEDED
> > BYTES_CACHED FILES_NEEDED FILES_CACHED
> > 1 wptest1 1 never /user/hdfs/hadoop003.log 209715206
> > 0 1 0
>
> If you take a look at this output, you can see that nothing is actually
> cached.
>
> One way to figure out why this is is to look at the logs of the
> NameNode and DataNode. Some of the relevant logs are at DEBUG or
> TRACE level, so you may need to turn up the logs. The
> CacheReplicationMonitor and FsDatasetCache classes are good places to
> start.
>
> Also be sure to check that you have set dfs.datanode.max.locked.memory.
>
> As Andrew commented, "hadoop tail" is not a good command to use for
> measuring performance, since you have a few seconds of Java startup
> time, followed by any HDFS setup time, followed by reading a single
> kilobyte of data. If you want to use the shell, the simplest thing to
> do is to use cat and read a large file, so that those startup costs
> don't dominate the measurement.
>
> best,
> Colin
>
>
> >
> > -bash-4.1$ file /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0
> > /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0: ELF 64-bit LSB shared
> > object, x86-64, version 1 (SYSV), dynamically linked, not stripped
> >
> > I also tried the word count example with the same file. The execution
> time
> > is always 40 seconds. (The map/reduce job without cache is 42 seconds)
> > Is there anything wrong?
> > Thanks a lot
>
Re: problem with HDFS caching in Hadoop 2.3
Posted by Colin McCabe <cm...@alumni.cmu.edu>.
On Fri, Mar 7, 2014 at 7:37 AM, hwpstorage <hw...@gmail.com> wrote:
> Hello,
>
> It looks like the HDFS caching does not work well.
> The cached log file is around 200MB. The hadoop cluster has 3 nodes, each
> has 4GB memory.
>
> -bash-4.1$ hdfs cacheadmin -addPool wptest1
> Successfully added cache pool wptest1.
>
> -bash-4.1$ /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listPools
> Found 1 result.
> NAME OWNER GROUP MODE LIMIT MAXTTL
> wptest1 hdfs hdfs rwxr-xr-x unlimited never
>
> -bash-4.1$ hdfs cacheadmin -addDirective -path hadoop003.log -pool wptest1
> Added cache directive 1
>
> -bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> real 0m2.796s
> user 0m4.263s
> sys 0m0.203s
>
> -bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> real 0m3.050s
> user 0m4.176s
> sys 0m0.192s
>
> It is weird that the cache status shows 0 byte cached:-bash-4.1$
> /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listDirectives -stats -path
> hadoop003.log -pool wptest1
> Found 1 entry
> ID POOL REPL EXPIRY PATH BYTES_NEEDED
> BYTES_CACHED FILES_NEEDED FILES_CACHED
> 1 wptest1 1 never /user/hdfs/hadoop003.log 209715206
> 0 1 0
If you take a look at this output, you can see that nothing is actually cached.
One way to figure out why this is is to look at the logs of the
NameNode and DataNode. Some of the relevant logs are at DEBUG or
TRACE level, so you may need to turn up the logs. The
CacheReplicationMonitor and FsDatasetCache classes are good places to
start.
Also be sure to check that you have set dfs.datanode.max.locked.memory.
As Andrew commented, "hadoop tail" is not a good command to use for
measuring performance, since you have a few seconds of Java startup
time, followed by any HDFS setup time, followed by reading a single
kilobyte of data. If you want to use the shell, the simplest thing to
do is to use cat and read a large file, so that those startup costs
don't dominate the measurement.
best,
Colin
>
> -bash-4.1$ file /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0
> /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0: ELF 64-bit LSB shared
> object, x86-64, version 1 (SYSV), dynamically linked, not stripped
>
> I also tried the word count example with the same file. The execution time
> is always 40 seconds. (The map/reduce job without cache is 42 seconds)
> Is there anything wrong?
> Thanks a lot
Re: problem with HDFS caching in Hadoop 2.3
Posted by hequn cheng <ch...@gmail.com>.
hi, have you solved your problem? i have the same problem. it seems that
the cache behavior has not been triggered.
2014-03-07 23:37 GMT+08:00 hwpstorage <hw...@gmail.com>:
> Hello,
>
> It looks like the HDFS caching does not work well.
> The cached log file is around 200MB. The hadoop cluster has 3 nodes, each
> has 4GB memory.
>
> -bash-4.1$ hdfs cacheadmin -addPool wptest1
> Successfully added cache pool wptest1.
>
> -bash-4.1$ /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listPools
> Found 1 result.
> NAME OWNER GROUP MODE LIMIT MAXTTL
> wptest1 hdfs hdfs rwxr-xr-x unlimited never
>
> -bash-4.1$ hdfs cacheadmin -addDirective -path hadoop003.log -pool wptest1
> Added cache directive 1
>
> -bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> real 0m2.796s
> user 0m4.263s
> sys 0m0.203s
>
> -bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> real 0m3.050s
> user 0m4.176s
> sys 0m0.192s
>
> It is weird that the cache status shows 0 byte cached:-bash-4.1$
> /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listDirectives -stats -path
> hadoop003.log -pool wptest1
> Found 1 entry
> ID POOL REPL EXPIRY PATH BYTES_NEEDED
> BYTES_CACHED FILES_NEEDED FILES_CACHED
> 1 wptest1 1 never /user/hdfs/hadoop003.log
> 209715206 0 1 0
>
> -bash-4.1$ file /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0
> /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0: ELF 64-bit LSB shared
> object, x86-64, version 1 (SYSV), dynamically linked, not stripped
>
> I also tried the word count example with the same file. The execution time
> is always 40 seconds. (The map/reduce job without cache is 42 seconds)
> Is there anything wrong?
> Thanks a lot
Re: problem with HDFS caching in Hadoop 2.3
Posted by Colin McCabe <cm...@alumni.cmu.edu>.
On Fri, Mar 7, 2014 at 7:37 AM, hwpstorage <hw...@gmail.com> wrote:
> Hello,
>
> It looks like the HDFS caching does not work well.
> The cached log file is around 200MB. The hadoop cluster has 3 nodes, each
> has 4GB memory.
>
> -bash-4.1$ hdfs cacheadmin -addPool wptest1
> Successfully added cache pool wptest1.
>
> -bash-4.1$ /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listPools
> Found 1 result.
> NAME OWNER GROUP MODE LIMIT MAXTTL
> wptest1 hdfs hdfs rwxr-xr-x unlimited never
>
> -bash-4.1$ hdfs cacheadmin -addDirective -path hadoop003.log -pool wptest1
> Added cache directive 1
>
> -bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> real 0m2.796s
> user 0m4.263s
> sys 0m0.203s
>
> -bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> real 0m3.050s
> user 0m4.176s
> sys 0m0.192s
>
> It is weird that the cache status shows 0 byte cached:-bash-4.1$
> /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listDirectives -stats -path
> hadoop003.log -pool wptest1
> Found 1 entry
> ID POOL REPL EXPIRY PATH BYTES_NEEDED
> BYTES_CACHED FILES_NEEDED FILES_CACHED
> 1 wptest1 1 never /user/hdfs/hadoop003.log 209715206
> 0 1 0
If you take a look at this output, you can see that nothing is actually cached.
One way to figure out why this is is to look at the logs of the
NameNode and DataNode. Some of the relevant logs are at DEBUG or
TRACE level, so you may need to turn up the logs. The
CacheReplicationMonitor and FsDatasetCache classes are good places to
start.
Also be sure to check that you have set dfs.datanode.max.locked.memory.
As Andrew commented, "hadoop tail" is not a good command to use for
measuring performance, since you have a few seconds of Java startup
time, followed by any HDFS setup time, followed by reading a single
kilobyte of data. If you want to use the shell, the simplest thing to
do is to use cat and read a large file, so that those startup costs
don't dominate the measurement.
best,
Colin
>
> -bash-4.1$ file /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0
> /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0: ELF 64-bit LSB shared
> object, x86-64, version 1 (SYSV), dynamically linked, not stripped
>
> I also tried the word count example with the same file. The execution time
> is always 40 seconds. (The map/reduce job without cache is 42 seconds)
> Is there anything wrong?
> Thanks a lot
Re: problem with HDFS caching in Hadoop 2.3
Posted by hequn cheng <ch...@gmail.com>.
hi, have you solved your problem? i have the same problem. it seems that
the cache behavior has not been triggered.
2014-03-07 23:37 GMT+08:00 hwpstorage <hw...@gmail.com>:
> Hello,
>
> It looks like the HDFS caching does not work well.
> The cached log file is around 200MB. The hadoop cluster has 3 nodes, each
> has 4GB memory.
>
> -bash-4.1$ hdfs cacheadmin -addPool wptest1
> Successfully added cache pool wptest1.
>
> -bash-4.1$ /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listPools
> Found 1 result.
> NAME OWNER GROUP MODE LIMIT MAXTTL
> wptest1 hdfs hdfs rwxr-xr-x unlimited never
>
> -bash-4.1$ hdfs cacheadmin -addDirective -path hadoop003.log -pool wptest1
> Added cache directive 1
>
> -bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> real 0m2.796s
> user 0m4.263s
> sys 0m0.203s
>
> -bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> real 0m3.050s
> user 0m4.176s
> sys 0m0.192s
>
> It is weird that the cache status shows 0 byte cached:-bash-4.1$
> /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listDirectives -stats -path
> hadoop003.log -pool wptest1
> Found 1 entry
> ID POOL REPL EXPIRY PATH BYTES_NEEDED
> BYTES_CACHED FILES_NEEDED FILES_CACHED
> 1 wptest1 1 never /user/hdfs/hadoop003.log
> 209715206 0 1 0
>
> -bash-4.1$ file /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0
> /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0: ELF 64-bit LSB shared
> object, x86-64, version 1 (SYSV), dynamically linked, not stripped
>
> I also tried the word count example with the same file. The execution time
> is always 40 seconds. (The map/reduce job without cache is 42 seconds)
> Is there anything wrong?
> Thanks a lot
Re: problem with HDFS caching in Hadoop 2.3
Posted by Andrew Wang <an...@cloudera.com>.
How was this cluster configured? Run dfsadmin -report to see the aggregate
configured cache capacity as seen by the NN. You need to configure some GBs
of cache on each DN and also raise the ulimit for max locked memory.
I'll also note that you are unlikely to see speedups with most mapreduce
jobs, especially running against TextInputFormat. There are a lot of copies
and string splitting, so it's typically not I/O bound.
The fs -tail command is likely also spending a lot of time on startup
costs, so I wouldn't expect much end-to-end latency savings.
Best,
Andrew
On Sun, Mar 9, 2014 at 5:15 AM, Azuryy Yu <az...@gmail.com> wrote:
> add dev.
> ---------- Forwarded message ----------
> From: "hwpstorage" <hw...@gmail.com>
> Date: Mar 7, 2014 11:38 PM
> Subject: problem with HDFS caching in Hadoop 2.3
> To: <us...@hadoop.apache.org>
> Cc:
>
> Hello,
>
> It looks like the HDFS caching does not work well.
> The cached log file is around 200MB. The hadoop cluster has 3 nodes, each
> has 4GB memory.
>
> -bash-4.1$ hdfs cacheadmin -addPool wptest1
> Successfully added cache pool wptest1.
>
> -bash-4.1$ /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listPools
> Found 1 result.
> NAME OWNER GROUP MODE LIMIT MAXTTL
> wptest1 hdfs hdfs rwxr-xr-x unlimited never
>
> -bash-4.1$ hdfs cacheadmin -addDirective -path hadoop003.log -pool wptest1
> Added cache directive 1
>
> -bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> real 0m2.796s
> user 0m4.263s
> sys 0m0.203s
>
> -bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> real 0m3.050s
> user 0m4.176s
> sys 0m0.192s
>
> It is weird that the cache status shows 0 byte cached:-bash-4.1$
> /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listDirectives -stats -path
> hadoop003.log -pool wptest1
> Found 1 entry
> ID POOL REPL EXPIRY PATH BYTES_NEEDED
> BYTES_CACHED FILES_NEEDED FILES_CACHED
> 1 wptest1 1 never /user/hdfs/hadoop003.log
> 209715206 0 1 0
>
> -bash-4.1$ file /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0
> /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0: ELF 64-bit LSB shared
> object, x86-64, version 1 (SYSV), dynamically linked, not stripped
>
> I also tried the word count example with the same file. The execution time
> is always 40 seconds. (The map/reduce job without cache is 42 seconds)
> Is there anything wrong?
> Thanks a lot
>
Fwd: problem with HDFS caching in Hadoop 2.3
Posted by Azuryy Yu <az...@gmail.com>.
add dev.
---------- Forwarded message ----------
From: "hwpstorage" <hw...@gmail.com>
Date: Mar 7, 2014 11:38 PM
Subject: problem with HDFS caching in Hadoop 2.3
To: <us...@hadoop.apache.org>
Cc:
Hello,
It looks like the HDFS caching does not work well.
The cached log file is around 200MB. The hadoop cluster has 3 nodes, each
has 4GB memory.
-bash-4.1$ hdfs cacheadmin -addPool wptest1
Successfully added cache pool wptest1.
-bash-4.1$ /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listPools
Found 1 result.
NAME OWNER GROUP MODE LIMIT MAXTTL
wptest1 hdfs hdfs rwxr-xr-x unlimited never
-bash-4.1$ hdfs cacheadmin -addDirective -path hadoop003.log -pool wptest1
Added cache directive 1
-bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
real 0m2.796s
user 0m4.263s
sys 0m0.203s
-bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
real 0m3.050s
user 0m4.176s
sys 0m0.192s
It is weird that the cache status shows 0 byte cached:-bash-4.1$
/hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listDirectives -stats -path
hadoop003.log -pool wptest1
Found 1 entry
ID POOL REPL EXPIRY PATH BYTES_NEEDED
BYTES_CACHED FILES_NEEDED FILES_CACHED
1 wptest1 1 never /user/hdfs/hadoop003.log
209715206 0 1 0
-bash-4.1$ file /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0
/hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0: ELF 64-bit LSB shared
object, x86-64, version 1 (SYSV), dynamically linked, not stripped
I also tried the word count example with the same file. The execution time
is always 40 seconds. (The map/reduce job without cache is 42 seconds)
Is there anything wrong?
Thanks a lot
Re: problem with HDFS caching in Hadoop 2.3
Posted by hequn cheng <ch...@gmail.com>.
hi, have you solved your problem? i have the same problem. it seems that
the cache behavior has not been triggered.
2014-03-07 23:37 GMT+08:00 hwpstorage <hw...@gmail.com>:
> Hello,
>
> It looks like the HDFS caching does not work well.
> The cached log file is around 200MB. The hadoop cluster has 3 nodes, each
> has 4GB memory.
>
> -bash-4.1$ hdfs cacheadmin -addPool wptest1
> Successfully added cache pool wptest1.
>
> -bash-4.1$ /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listPools
> Found 1 result.
> NAME OWNER GROUP MODE LIMIT MAXTTL
> wptest1 hdfs hdfs rwxr-xr-x unlimited never
>
> -bash-4.1$ hdfs cacheadmin -addDirective -path hadoop003.log -pool wptest1
> Added cache directive 1
>
> -bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> real 0m2.796s
> user 0m4.263s
> sys 0m0.203s
>
> -bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> real 0m3.050s
> user 0m4.176s
> sys 0m0.192s
>
> It is weird that the cache status shows 0 byte cached:-bash-4.1$
> /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listDirectives -stats -path
> hadoop003.log -pool wptest1
> Found 1 entry
> ID POOL REPL EXPIRY PATH BYTES_NEEDED
> BYTES_CACHED FILES_NEEDED FILES_CACHED
> 1 wptest1 1 never /user/hdfs/hadoop003.log
> 209715206 0 1 0
>
> -bash-4.1$ file /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0
> /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0: ELF 64-bit LSB shared
> object, x86-64, version 1 (SYSV), dynamically linked, not stripped
>
> I also tried the word count example with the same file. The execution time
> is always 40 seconds. (The map/reduce job without cache is 42 seconds)
> Is there anything wrong?
> Thanks a lot
Re: problem with HDFS caching in Hadoop 2.3
Posted by Colin McCabe <cm...@alumni.cmu.edu>.
On Fri, Mar 7, 2014 at 7:37 AM, hwpstorage <hw...@gmail.com> wrote:
> Hello,
>
> It looks like the HDFS caching does not work well.
> The cached log file is around 200MB. The hadoop cluster has 3 nodes, each
> has 4GB memory.
>
> -bash-4.1$ hdfs cacheadmin -addPool wptest1
> Successfully added cache pool wptest1.
>
> -bash-4.1$ /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listPools
> Found 1 result.
> NAME OWNER GROUP MODE LIMIT MAXTTL
> wptest1 hdfs hdfs rwxr-xr-x unlimited never
>
> -bash-4.1$ hdfs cacheadmin -addDirective -path hadoop003.log -pool wptest1
> Added cache directive 1
>
> -bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> real 0m2.796s
> user 0m4.263s
> sys 0m0.203s
>
> -bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> real 0m3.050s
> user 0m4.176s
> sys 0m0.192s
>
> It is weird that the cache status shows 0 byte cached:-bash-4.1$
> /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listDirectives -stats -path
> hadoop003.log -pool wptest1
> Found 1 entry
> ID POOL REPL EXPIRY PATH BYTES_NEEDED
> BYTES_CACHED FILES_NEEDED FILES_CACHED
> 1 wptest1 1 never /user/hdfs/hadoop003.log 209715206
> 0 1 0
If you take a look at this output, you can see that nothing is actually cached.
One way to figure out why this is is to look at the logs of the
NameNode and DataNode. Some of the relevant logs are at DEBUG or
TRACE level, so you may need to turn up the logs. The
CacheReplicationMonitor and FsDatasetCache classes are good places to
start.
Also be sure to check that you have set dfs.datanode.max.locked.memory.
As Andrew commented, "hadoop tail" is not a good command to use for
measuring performance, since you have a few seconds of Java startup
time, followed by any HDFS setup time, followed by reading a single
kilobyte of data. If you want to use the shell, the simplest thing to
do is to use cat and read a large file, so that those startup costs
don't dominate the measurement.
best,
Colin
>
> -bash-4.1$ file /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0
> /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0: ELF 64-bit LSB shared
> object, x86-64, version 1 (SYSV), dynamically linked, not stripped
>
> I also tried the word count example with the same file. The execution time
> is always 40 seconds. (The map/reduce job without cache is 42 seconds)
> Is there anything wrong?
> Thanks a lot
Re: problem with HDFS caching in Hadoop 2.3
Posted by Colin McCabe <cm...@alumni.cmu.edu>.
On Fri, Mar 7, 2014 at 7:37 AM, hwpstorage <hw...@gmail.com> wrote:
> Hello,
>
> It looks like the HDFS caching does not work well.
> The cached log file is around 200MB. The hadoop cluster has 3 nodes, each
> has 4GB memory.
>
> -bash-4.1$ hdfs cacheadmin -addPool wptest1
> Successfully added cache pool wptest1.
>
> -bash-4.1$ /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listPools
> Found 1 result.
> NAME OWNER GROUP MODE LIMIT MAXTTL
> wptest1 hdfs hdfs rwxr-xr-x unlimited never
>
> -bash-4.1$ hdfs cacheadmin -addDirective -path hadoop003.log -pool wptest1
> Added cache directive 1
>
> -bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> real 0m2.796s
> user 0m4.263s
> sys 0m0.203s
>
> -bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> real 0m3.050s
> user 0m4.176s
> sys 0m0.192s
>
> It is weird that the cache status shows 0 byte cached:-bash-4.1$
> /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listDirectives -stats -path
> hadoop003.log -pool wptest1
> Found 1 entry
> ID POOL REPL EXPIRY PATH BYTES_NEEDED
> BYTES_CACHED FILES_NEEDED FILES_CACHED
> 1 wptest1 1 never /user/hdfs/hadoop003.log 209715206
> 0 1 0
If you take a look at this output, you can see that nothing is actually cached.
One way to figure out why this is is to look at the logs of the
NameNode and DataNode. Some of the relevant logs are at DEBUG or
TRACE level, so you may need to turn up the logs. The
CacheReplicationMonitor and FsDatasetCache classes are good places to
start.
Also be sure to check that you have set dfs.datanode.max.locked.memory.
As Andrew commented, "hadoop tail" is not a good command to use for
measuring performance, since you have a few seconds of Java startup
time, followed by any HDFS setup time, followed by reading a single
kilobyte of data. If you want to use the shell, the simplest thing to
do is to use cat and read a large file, so that those startup costs
don't dominate the measurement.
best,
Colin
>
> -bash-4.1$ file /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0
> /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0: ELF 64-bit LSB shared
> object, x86-64, version 1 (SYSV), dynamically linked, not stripped
>
> I also tried the word count example with the same file. The execution time
> is always 40 seconds. (The map/reduce job without cache is 42 seconds)
> Is there anything wrong?
> Thanks a lot
Re: problem with HDFS caching in Hadoop 2.3
Posted by hequn cheng <ch...@gmail.com>.
hi, have you solved your problem? i have the same problem. it seems that
the cache behavior has not been triggered.
2014-03-07 23:37 GMT+08:00 hwpstorage <hw...@gmail.com>:
> Hello,
>
> It looks like the HDFS caching does not work well.
> The cached log file is around 200MB. The hadoop cluster has 3 nodes, each
> has 4GB memory.
>
> -bash-4.1$ hdfs cacheadmin -addPool wptest1
> Successfully added cache pool wptest1.
>
> -bash-4.1$ /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listPools
> Found 1 result.
> NAME OWNER GROUP MODE LIMIT MAXTTL
> wptest1 hdfs hdfs rwxr-xr-x unlimited never
>
> -bash-4.1$ hdfs cacheadmin -addDirective -path hadoop003.log -pool wptest1
> Added cache directive 1
>
> -bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> real 0m2.796s
> user 0m4.263s
> sys 0m0.203s
>
> -bash-4.1$ time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> real 0m3.050s
> user 0m4.176s
> sys 0m0.192s
>
> It is weird that the cache status shows 0 byte cached:-bash-4.1$
> /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listDirectives -stats -path
> hadoop003.log -pool wptest1
> Found 1 entry
> ID POOL REPL EXPIRY PATH BYTES_NEEDED
> BYTES_CACHED FILES_NEEDED FILES_CACHED
> 1 wptest1 1 never /user/hdfs/hadoop003.log
> 209715206 0 1 0
>
> -bash-4.1$ file /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0
> /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0: ELF 64-bit LSB shared
> object, x86-64, version 1 (SYSV), dynamically linked, not stripped
>
> I also tried the word count example with the same file. The execution time
> is always 40 seconds. (The map/reduce job without cache is 42 seconds)
> Is there anything wrong?
> Thanks a lot