You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by lujinhong <lu...@yahoo.com> on 2015/03/01 15:09:58 UTC
about the jobid
Hi, all.
I run nutch in deploy mode at about 3pm, 02/28/2015, but the jobid is job_201502271057_0251.I found that 201502271057 is the time I start hadoop(by start-all.sh).
Is the jobid should be job_201502281500_**** ？ what is the problem?
system date:
[jediael@master history]$ date
Sat Feb 28 15:39:00 CST 2015
log files of hadoop:
/mnt/jediael/hadoop-1.2.1/logs/history
[jediael@master history]$ ls
done job_201502271057_0245_conf.xml job_201502271057_0248_conf.xml job_201502271057_0251_1425107493248_jediael_%5BFeb2815%5Dfetch
job_201502271057_0243_conf.xml job_201502271057_0246_conf.xml job_201502271057_0249_conf.xml job_201502271057_0251_conf.xml
job_201502271057_0244_conf.xml job_201502271057_0247_conf.xml job_201502271057_0250_conf.xml
stdout of fetcher job:
15/02/28 15:11:32 INFO zookeeper.ClientCnxn: EventThread shut down
15/02/28 15:11:32 INFO zookeeper.ZooKeeper: Session: 0x4bc8f7c30a031b closed
15/02/28 15:11:33 INFO mapred.JobClient: Running job: job_201502271057_0251
15/02/28 15:11:34 INFO mapred.JobClient: map 0% reduce 0%
15/02/28 15:11:51 INFO mapred.JobClient: map 100% reduce 0%
15/02/28 15:12:00 INFO mapred.JobClient: map 100% reduce 16%
15/02/28 15:12:03 INFO mapred.JobClient: map 100% reduce 53%
RE: about the jobid
Posted by Rohith Sharma K S <ro...@huawei.com>.
Hi
Yarn application id allocation based on the daemon ResourceManager start time(assuming cluster is MR2 else JobTracker start time). Say if you have 3 job client submitting jobs to Yarn, then application id are application_<rm-start-time>_0001, application_<rm-start-time>_0002, application_<rm-start-time>_0003 AND corresponding job id's are job_<rm-start-time>_0001, job_<rm-start-time>_0002 and job_<rm-start-time>_0003 respectively.
>>>> Is the jobid should be job_201502281500_**** ？ what is the problem?
No, this is behaviour. In your case, 201502271057 is the start time of ResourceManager. So all the applications submitted to Yarn start with application_201502271057_XXXX and corresponding job id is Job_201502271057_XXXX. The 'xxxx' is counter for every job submission.
Thanks & Regards
Rohith Sharma K S
-----Original Message-----
From: lujinhong [mailto:lujinhong@yahoo.com]
Sent: 01 March 2015 19:40
To: User Hadoop
Subject: about the jobid
Hi, all.
I run nutch in deploy mode at about 3pm, 02/28/2015, but the jobid is job_201502271057_0251.I found that 201502271057 is the time I start hadoop(by start-all.sh).
Is the jobid should be job_201502281500_**** ？ what is the problem?
system date:
[jediael@master history]$ date
Sat Feb 28 15:39:00 CST 2015
log files of hadoop:
/mnt/jediael/hadoop-1.2.1/logs/history
[jediael@master history]$ ls
done job_201502271057_0245_conf.xml job_201502271057_0248_conf.xml job_201502271057_0251_1425107493248_jediael_%5BFeb2815%5Dfetch
job_201502271057_0243_conf.xml job_201502271057_0246_conf.xml job_201502271057_0249_conf.xml job_201502271057_0251_conf.xml job_201502271057_0244_conf.xml job_201502271057_0247_conf.xml job_201502271057_0250_conf.xml
stdout of fetcher job:
15/02/28 15:11:32 INFO zookeeper.ClientCnxn: EventThread shut down
15/02/28 15:11:32 INFO zookeeper.ZooKeeper: Session: 0x4bc8f7c30a031b closed
15/02/28 15:11:33 INFO mapred.JobClient: Running job: job_201502271057_0251
15/02/28 15:11:34 INFO mapred.JobClient: map 0% reduce 0%
15/02/28 15:11:51 INFO mapred.JobClient: map 100% reduce 0%
15/02/28 15:12:00 INFO mapred.JobClient: map 100% reduce 16%
15/02/28 15:12:03 INFO mapred.JobClient: map 100% reduce 53%
RE: about the jobid
Posted by Rohith Sharma K S <ro...@huawei.com>.
Hi
Yarn application id allocation based on the daemon ResourceManager start time(assuming cluster is MR2 else JobTracker start time). Say if you have 3 job client submitting jobs to Yarn, then application id are application_<rm-start-time>_0001, application_<rm-start-time>_0002, application_<rm-start-time>_0003 AND corresponding job id's are job_<rm-start-time>_0001, job_<rm-start-time>_0002 and job_<rm-start-time>_0003 respectively.
>>>> Is the jobid should be job_201502281500_**** ？ what is the problem?
No, this is behaviour. In your case, 201502271057 is the start time of ResourceManager. So all the applications submitted to Yarn start with application_201502271057_XXXX and corresponding job id is Job_201502271057_XXXX. The 'xxxx' is counter for every job submission.
Thanks & Regards
Rohith Sharma K S
-----Original Message-----
From: lujinhong [mailto:lujinhong@yahoo.com]
Sent: 01 March 2015 19:40
To: User Hadoop
Subject: about the jobid
Hi, all.
I run nutch in deploy mode at about 3pm, 02/28/2015, but the jobid is job_201502271057_0251.I found that 201502271057 is the time I start hadoop(by start-all.sh).
Is the jobid should be job_201502281500_**** ？ what is the problem?
system date:
[jediael@master history]$ date
Sat Feb 28 15:39:00 CST 2015
log files of hadoop:
/mnt/jediael/hadoop-1.2.1/logs/history
[jediael@master history]$ ls
done job_201502271057_0245_conf.xml job_201502271057_0248_conf.xml job_201502271057_0251_1425107493248_jediael_%5BFeb2815%5Dfetch
job_201502271057_0243_conf.xml job_201502271057_0246_conf.xml job_201502271057_0249_conf.xml job_201502271057_0251_conf.xml job_201502271057_0244_conf.xml job_201502271057_0247_conf.xml job_201502271057_0250_conf.xml
stdout of fetcher job:
15/02/28 15:11:32 INFO zookeeper.ClientCnxn: EventThread shut down
15/02/28 15:11:32 INFO zookeeper.ZooKeeper: Session: 0x4bc8f7c30a031b closed
15/02/28 15:11:33 INFO mapred.JobClient: Running job: job_201502271057_0251
15/02/28 15:11:34 INFO mapred.JobClient: map 0% reduce 0%
15/02/28 15:11:51 INFO mapred.JobClient: map 100% reduce 0%
15/02/28 15:12:00 INFO mapred.JobClient: map 100% reduce 16%
15/02/28 15:12:03 INFO mapred.JobClient: map 100% reduce 53%
RE: about the jobid
Posted by Rohith Sharma K S <ro...@huawei.com>.
Hi
Yarn application id allocation based on the daemon ResourceManager start time(assuming cluster is MR2 else JobTracker start time). Say if you have 3 job client submitting jobs to Yarn, then application id are application_<rm-start-time>_0001, application_<rm-start-time>_0002, application_<rm-start-time>_0003 AND corresponding job id's are job_<rm-start-time>_0001, job_<rm-start-time>_0002 and job_<rm-start-time>_0003 respectively.
>>>> Is the jobid should be job_201502281500_**** ？ what is the problem?
No, this is behaviour. In your case, 201502271057 is the start time of ResourceManager. So all the applications submitted to Yarn start with application_201502271057_XXXX and corresponding job id is Job_201502271057_XXXX. The 'xxxx' is counter for every job submission.
Thanks & Regards
Rohith Sharma K S
-----Original Message-----
From: lujinhong [mailto:lujinhong@yahoo.com]
Sent: 01 March 2015 19:40
To: User Hadoop
Subject: about the jobid
Hi, all.
I run nutch in deploy mode at about 3pm, 02/28/2015, but the jobid is job_201502271057_0251.I found that 201502271057 is the time I start hadoop(by start-all.sh).
Is the jobid should be job_201502281500_**** ？ what is the problem?
system date:
[jediael@master history]$ date
Sat Feb 28 15:39:00 CST 2015
log files of hadoop:
/mnt/jediael/hadoop-1.2.1/logs/history
[jediael@master history]$ ls
done job_201502271057_0245_conf.xml job_201502271057_0248_conf.xml job_201502271057_0251_1425107493248_jediael_%5BFeb2815%5Dfetch
job_201502271057_0243_conf.xml job_201502271057_0246_conf.xml job_201502271057_0249_conf.xml job_201502271057_0251_conf.xml job_201502271057_0244_conf.xml job_201502271057_0247_conf.xml job_201502271057_0250_conf.xml
stdout of fetcher job:
15/02/28 15:11:32 INFO zookeeper.ClientCnxn: EventThread shut down
15/02/28 15:11:32 INFO zookeeper.ZooKeeper: Session: 0x4bc8f7c30a031b closed
15/02/28 15:11:33 INFO mapred.JobClient: Running job: job_201502271057_0251
15/02/28 15:11:34 INFO mapred.JobClient: map 0% reduce 0%
15/02/28 15:11:51 INFO mapred.JobClient: map 100% reduce 0%
15/02/28 15:12:00 INFO mapred.JobClient: map 100% reduce 16%
15/02/28 15:12:03 INFO mapred.JobClient: map 100% reduce 53%
RE: about the jobid
Posted by Rohith Sharma K S <ro...@huawei.com>.
Hi
Yarn application id allocation based on the daemon ResourceManager start time(assuming cluster is MR2 else JobTracker start time). Say if you have 3 job client submitting jobs to Yarn, then application id are application_<rm-start-time>_0001, application_<rm-start-time>_0002, application_<rm-start-time>_0003 AND corresponding job id's are job_<rm-start-time>_0001, job_<rm-start-time>_0002 and job_<rm-start-time>_0003 respectively.
>>>> Is the jobid should be job_201502281500_**** ？ what is the problem?
No, this is behaviour. In your case, 201502271057 is the start time of ResourceManager. So all the applications submitted to Yarn start with application_201502271057_XXXX and corresponding job id is Job_201502271057_XXXX. The 'xxxx' is counter for every job submission.
Thanks & Regards
Rohith Sharma K S
-----Original Message-----
From: lujinhong [mailto:lujinhong@yahoo.com]
Sent: 01 March 2015 19:40
To: User Hadoop
Subject: about the jobid
Hi, all.
I run nutch in deploy mode at about 3pm, 02/28/2015, but the jobid is job_201502271057_0251.I found that 201502271057 is the time I start hadoop(by start-all.sh).
Is the jobid should be job_201502281500_**** ？ what is the problem?
system date:
[jediael@master history]$ date
Sat Feb 28 15:39:00 CST 2015
log files of hadoop:
/mnt/jediael/hadoop-1.2.1/logs/history
[jediael@master history]$ ls
done job_201502271057_0245_conf.xml job_201502271057_0248_conf.xml job_201502271057_0251_1425107493248_jediael_%5BFeb2815%5Dfetch
job_201502271057_0243_conf.xml job_201502271057_0246_conf.xml job_201502271057_0249_conf.xml job_201502271057_0251_conf.xml job_201502271057_0244_conf.xml job_201502271057_0247_conf.xml job_201502271057_0250_conf.xml
stdout of fetcher job:
15/02/28 15:11:32 INFO zookeeper.ClientCnxn: EventThread shut down
15/02/28 15:11:32 INFO zookeeper.ZooKeeper: Session: 0x4bc8f7c30a031b closed
15/02/28 15:11:33 INFO mapred.JobClient: Running job: job_201502271057_0251
15/02/28 15:11:34 INFO mapred.JobClient: map 0% reduce 0%
15/02/28 15:11:51 INFO mapred.JobClient: map 100% reduce 0%
15/02/28 15:12:00 INFO mapred.JobClient: map 100% reduce 16%
15/02/28 15:12:03 INFO mapred.JobClient: map 100% reduce 53%