You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by EdwardKing <zh...@neusoft.com> on 2014/02/10 09:44:10 UTC
Which node is used
I have two node,one is master(172.11.12.6) and one is node1(172.11.12.7), I use following command to run a program under 172.11.12.6 ,like follows:
$ hadoop jar wc2.jar WordCountPredefined abc.txt output
[hadoop@master ~]$ hadoop jar wc2.jar WordCountPredefined abc.txt output
14/02/10 00:23:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/02/10 00:23:25 INFO client.RMProxy: Connecting to ResourceManager at master/172.11.12.6:8993
14/02/10 00:23:26 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
14/02/10 00:23:28 INFO input.FileInputFormat: Total input paths to process : 1
14/02/10 00:23:28 INFO mapreduce.JobSubmitter: number of splits:1
14/02/10 00:23:28 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
14/02/10 00:23:28 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
14/02/10 00:23:28 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
14/02/10 00:23:28 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
14/02/10 00:23:28 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
14/02/10 00:23:28 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
14/02/10 00:23:28 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
14/02/10 00:23:28 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/02/10 00:23:28 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
14/02/10 00:23:28 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
14/02/10 00:23:29 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
14/02/10 00:23:29 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1391993915383_0003
14/02/10 00:23:30 INFO impl.YarnClientImpl: Submitted application application_1391993915383_0003 to ResourceManager at master/172.11.12.6:8993
14/02/10 00:23:31 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1391993915383_0003/
14/02/10 00:23:31 INFO mapreduce.Job: Running job: job_1391993915383_0003
14/02/10 00:23:54 INFO mapreduce.Job: Job job_1391993915383_0003 running in uber mode : false
14/02/10 00:23:54 INFO mapreduce.Job: map 0% reduce 0%
14/02/10 00:26:09 INFO mapreduce.Job: map 67% reduce 0%
14/02/10 00:26:12 INFO mapreduce.Job: map 100% reduce 0%
14/02/10 00:26:43 INFO mapreduce.Job: map 100% reduce 100%
14/02/10 00:26:45 INFO mapreduce.Job: Job job_1391993915383_0003 completed successfully
14/02/10 00:26:46 INFO mapreduce.Job: Counters: 43
File System Counters
FILE: Number of bytes read=30
FILE: Number of bytes written=156945
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=115
HDFS: Number of bytes written=16
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=139690
Total time spent by all reduces in occupied slots (ms)=20127
Map-Reduce Framework
Map input records=1
Map output records=2
Map output bytes=20
Map output materialized bytes=30
Input split bytes=103
Combine input records=0
Combine output records=0
Reduce input groups=2
Reduce shuffle bytes=30
Reduce input records=2
Reduce output records=2
Spilled Records=4
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=396
CPU time spent (ms)=2130
Physical memory (bytes) snapshot=199540736
Virtual memory (bytes) snapshot=709742592
Total committed heap usage (bytes)=124510208
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=12
File Output Format Counters
Bytes Written=16
I can view http://172.11.12.6:9002/dfsnodelist.jsp?whatNodes=LIVE in firefox
NameNode 'master:9000'
Started: Sun Feb 09 16:57:50 PST 2014
Version: 2.2.0, 1529768
Compiled: 2013-10-07T06:28Z by hortonmu from branch-2.2.0
Cluster ID: CID-e47f6b2e-ca1f-4eda-b6f7-dd9d5314c251
Block Pool ID: BP-1227723652-172.11.12.6-1390358446283
Live Datanodes : 2
Node Transferring Address Last Contact Admin State Configured Capacity (GB) Used(GB) Non DFS Used (GB) Remaining
master 172.11.12.6:50010 1 In Service 16.15 0.00 2.76 13.39
node1 172.11.12.7:50010 1 In Service 16.15 0.00 2.75 13.40
How can I know which node will be called ,master(172.11.12.6) or node1(172.11.12.6) ?
Thanks.
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------