Yarn -> job.jar not found in yarn staging dir (works with TestDFSIO but not ImportTsv )

["Stanley.Burnitt" <St...@huawei.com>]

Hello,

I am trying to run HBase's ImportTsv against Yarn (Hadoop 2.2.0).

I can run the Hadoop TestDFSIO Yarn job with no problems:
hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0-tests.jar TestDFSIO -write -nrFiles 20 -fileSize 10

I cannot run the HBase ImportTsv job:

hadoop jar $HBASE_HOME/lib/hbase-server-0.96.0-hadoop2.jar importtsv |
 -Dimporttsv.separator=               \
 -Dimporttsv.columns=$IMPORTTSV_COLUMNS \
 -Dimporttsv.bulk.output=hdfs://$HADOOP_NAMENODE:54310/user/bihadoop/importtsv-output \
 bulkloaded-xdremail  \
 hdfs://$HADOOP_NAMENODE:54310/user/bihadoop/kv-data-files

... because the job.jar file is never successfully downloaded to any yarn node:

2014-05-05 19:44:06,641 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-bihadoop/nm-local-dir/nmPrivate/container_1399316513410_0007_01_000001.tokens. Credentials list:
2014-05-05 19:44:06,642 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user bihadoop
2014-05-05 19:44:06,690 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-bihadoop/nm-local-dir/nmPrivate/container_1399316513410_0007_01_000001.tokens to /tmp/hadoop-bihadoop/nm-local-dir/usercache/bihadoop/appcache/application_1399316513410_0007/container_1399316513410_0007_01_000001.tokens
2014-05-05 19:44:06,690 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: CWD set to /tmp/hadoop-bihadoop/nm-local-dir/usercache/bihadoop/appcache/application_1399316513410_0007 = file:/tmp/hadoop-bihadoop/nm-local-dir/usercache/bihadoop/appcache/application_1399316513410_0007
2014-05-05 19:44:06,791 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:bihadoop (auth:SIMPLE) cause:java.io.FileNotFoundException: File file:/tmp/hadoop-yarn/staging/bihadoop/.staging/job_1399316513410_0007/job.jar does not exist
2014-05-05 19:44:06,810 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: DEBUG: FAILED { file:/tmp/hadoop-yarn/staging/bihadoop/.staging/job_1399316513410_0007/job.jar, 1399319045000, PATTERN, (?:classes/|lib/).* }, File file:/tmp/hadoop-yarn/staging/bihadoop/.staging/job_1399316513410_0007/job.jar does not exist
2014-05-05 19:44:06,812 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource file:/tmp/hadoop-yarn/staging/bihadoop/.staging/job_1399316513410_0007/job.jar transitioned from DOWNLOADING to FAILED


I have also tried running importtsv from the hbase script, i.e.:
hbase  org.apache.hadoop.hbase.mapreduce.ImportTsv \
...
...

The end result is the same: job.jar not found in staging dir.  (The job.jar is found on the master node, where I run the job.)

There seems to be no problems distributing job.jar to the staging dirs when I run the TestDFSIO job.
But the job.jar containing ImportTsv is never distributed.

Very frustrating,  time eating... ;/

Does anyone have any suggestions?  I am runinng Hadoop 2.2.0 w/ HBase 0.96.0

Thanks,
Stan