You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by grabbler <tw...@gmail.com> on 2010/08/05 20:37:16 UTC
Problems running hadoop on Amazon Elastic MapReduce
I am a complete newbie to hadoop. I'm running a job on 19 Amazon Elastic
MapReduce servers and am trying to understand two separate issues.
1) The job is ending with an error "ERROR
org.apache.pig.tools.grunt.GruntParser - ERROR 6015: During execution,
encountered a Hadoop error." I do not have the hadoop log files and will
have to rerun the job with different settings to obtain them. Once I have
them, I'll add them to the posting.
2) The job seems to complete %19 within 1 minute and then takes 15 minutes
to complete another %20. It then takes over 40 minutes to complete the last
%60. Why the sudden slow down? Am I misunderstanding the messages?
The following is the ouput from the job:
10/08/05 16:39:30 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found
in the classpath. Usage of hadoop-site.xml is deprecated. Instead use
core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of
core-default.xml, mapred-default.xml and hdfs-default.xml respectively
2010-08-05 16:39:30,688 [main] INFO org.apache.pig.Main - Logging error
messages to: /mnt/var/lib/hadoop/steps/2/pig_1281026370670.log
2010-08-05 16:39:31,663 [main] INFO
org.apache.hadoop.fs.s3native.NativeS3FileSystem - Opening
'/scripts/calculateTrackingErrors.pig' for reading
2010-08-05 16:39:32,071 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
to hadoop file system at:
hdfs://domU-12-31-39-09-F1-D2.compute-1.internal:9000
2010-08-05 16:39:32,662 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
to map-reduce job tracker at: domU-12-31-39-09-F1-D2.compute-1.internal:9001
2010-08-05 16:39:33,360 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No column pruned
for covMatrix
2010-08-05 16:39:33,360 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned
for covMatrix
2010-08-05 16:39:33,362 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No column pruned
for fundExposures2
2010-08-05 16:39:33,362 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned
for fundExposures2
2010-08-05 16:39:33,363 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No column pruned
for fundExposures1
2010-08-05 16:39:33,363 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned
for fundExposures1
2010-08-05 16:39:33,469 [main] WARN
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Encountered Warning DID_NOT_FIND_LOAD_ONLY_MAP_PLAN 1 time(s).
2010-08-05 16:39:33,484 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler$LastInputStreamingOptimizer
- Rewrite: POPackage->POForEach to POJoinPackage
2010-08-05 16:39:33,485 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler$LastInputStreamingOptimizer
- Rewrite: POPackage->POForEach to POJoinPackage
2010-08-05 16:39:33,486 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler$LastInputStreamingOptimizer
- Rewrite: POPackage->POForEach to POJoinPackage
2010-08-05 16:39:33,495 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 5
2010-08-05 16:39:33,496 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- Merged 0 out of total 3 MR operators.
2010-08-05 16:39:33,496 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 5
2010-08-05 16:39:33,501 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.AccumulatorOptimizer
- Reducer is to run in accumulative mode.
2010-08-05 16:39:33,502 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.AccumulatorOptimizer
- Reducer is to run in accumulative mode.
2010-08-05 16:39:34,465 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2010-08-05 16:39:34,559 [Thread-12] WARN org.apache.hadoop.mapred.JobClient
- Use GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
2010-08-05 16:39:35,585 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Cannot get jobid for this job
2010-08-05 16:39:42,847 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0% complete
2010-08-05 16:39:55,415 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 5% complete
2010-08-05 16:39:56,464 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 10% complete
2010-08-05 16:40:08,548 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 11% complete
2010-08-05 16:40:11,561 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 14% complete
2010-08-05 16:40:26,650 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 19% complete
2010-08-05 16:40:48,382 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2010-08-05 16:40:48,415 [Thread-23] WARN org.apache.hadoop.mapred.JobClient
- Use GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
2010-08-05 16:40:49,444 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Cannot get jobid for this job
2010-08-05 16:55:41,640 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 39% complete
2010-08-05 16:57:17,243 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2010-08-05 16:57:17,275 [Thread-33] WARN org.apache.hadoop.mapred.JobClient
- Use GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
2010-08-05 16:57:18,274 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Cannot get jobid for this job
2010-08-05 17:32:47,304 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 54% complete
2010-08-05 17:35:02,343 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 56% complete
2010-08-05 17:38:33,767 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 60% complete
2010-08-05 17:38:37,285 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 100% complete
2010-08-05 17:38:37,286 [main] ERROR
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map reduce job(s) failed!
2010-08-05 17:38:37,772 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Some jobs have failed! Stop running all dependent jobs
2010-08-05 17:38:37,775 [main] ERROR org.apache.pig.tools.grunt.GruntParser
- ERROR 6015: During execution, encountered a Hadoop error.
Details at logfile: /mnt/var/lib/hadoop/steps/2/pig_1281026370670.log
Command exiting with ret '255'
Thanks for any enlightenment you can provide
--
View this message in context: http://old.nabble.com/Problems-running-hadoop-on-Amazon-Elastic-MapReduce-tp29356384p29356384.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.
RE: Problems running hadoop on Amazon Elastic MapReduce
Posted by Ankit Bhatnagar <ab...@vantage.com>.
Hi,
EMR has a live debug option in the panel, you will find the logs there as well.
Ankit
-----Original Message-----
From: grabbler [mailto:twizansk@gmail.com]
Sent: Thursday, August 05, 2010 2:37 PM
To: core-user@hadoop.apache.org
Subject: Problems running hadoop on Amazon Elastic MapReduce
I am a complete newbie to hadoop. I'm running a job on 19 Amazon Elastic
MapReduce servers and am trying to understand two separate issues.
1) The job is ending with an error "ERROR
org.apache.pig.tools.grunt.GruntParser - ERROR 6015: During execution,
encountered a Hadoop error." I do not have the hadoop log files and will
have to rerun the job with different settings to obtain them. Once I have
them, I'll add them to the posting.
2) The job seems to complete %19 within 1 minute and then takes 15 minutes
to complete another %20. It then takes over 40 minutes to complete the last
%60. Why the sudden slow down? Am I misunderstanding the messages?
The following is the ouput from the job:
10/08/05 16:39:30 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found
in the classpath. Usage of hadoop-site.xml is deprecated. Instead use
core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of
core-default.xml, mapred-default.xml and hdfs-default.xml respectively
2010-08-05 16:39:30,688 [main] INFO org.apache.pig.Main - Logging error
messages to: /mnt/var/lib/hadoop/steps/2/pig_1281026370670.log
2010-08-05 16:39:31,663 [main] INFO
org.apache.hadoop.fs.s3native.NativeS3FileSystem - Opening
'/scripts/calculateTrackingErrors.pig' for reading
2010-08-05 16:39:32,071 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
to hadoop file system at:
hdfs://domU-12-31-39-09-F1-D2.compute-1.internal:9000
2010-08-05 16:39:32,662 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
to map-reduce job tracker at: domU-12-31-39-09-F1-D2.compute-1.internal:9001
2010-08-05 16:39:33,360 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No column pruned
for covMatrix
2010-08-05 16:39:33,360 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned
for covMatrix
2010-08-05 16:39:33,362 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No column pruned
for fundExposures2
2010-08-05 16:39:33,362 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned
for fundExposures2
2010-08-05 16:39:33,363 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No column pruned
for fundExposures1
2010-08-05 16:39:33,363 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned
for fundExposures1
2010-08-05 16:39:33,469 [main] WARN
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Encountered Warning DID_NOT_FIND_LOAD_ONLY_MAP_PLAN 1 time(s).
2010-08-05 16:39:33,484 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler$LastInputStreamingOptimizer
- Rewrite: POPackage->POForEach to POJoinPackage
2010-08-05 16:39:33,485 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler$LastInputStreamingOptimizer
- Rewrite: POPackage->POForEach to POJoinPackage
2010-08-05 16:39:33,486 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler$LastInputStreamingOptimizer
- Rewrite: POPackage->POForEach to POJoinPackage
2010-08-05 16:39:33,495 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 5
2010-08-05 16:39:33,496 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- Merged 0 out of total 3 MR operators.
2010-08-05 16:39:33,496 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 5
2010-08-05 16:39:33,501 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.AccumulatorOptimizer
- Reducer is to run in accumulative mode.
2010-08-05 16:39:33,502 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.AccumulatorOptimizer
- Reducer is to run in accumulative mode.
2010-08-05 16:39:34,465 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2010-08-05 16:39:34,559 [Thread-12] WARN org.apache.hadoop.mapred.JobClient
- Use GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
2010-08-05 16:39:35,585 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Cannot get jobid for this job
2010-08-05 16:39:42,847 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0% complete
2010-08-05 16:39:55,415 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 5% complete
2010-08-05 16:39:56,464 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 10% complete
2010-08-05 16:40:08,548 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 11% complete
2010-08-05 16:40:11,561 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 14% complete
2010-08-05 16:40:26,650 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 19% complete
2010-08-05 16:40:48,382 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2010-08-05 16:40:48,415 [Thread-23] WARN org.apache.hadoop.mapred.JobClient
- Use GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
2010-08-05 16:40:49,444 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Cannot get jobid for this job
2010-08-05 16:55:41,640 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 39% complete
2010-08-05 16:57:17,243 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2010-08-05 16:57:17,275 [Thread-33] WARN org.apache.hadoop.mapred.JobClient
- Use GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
2010-08-05 16:57:18,274 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Cannot get jobid for this job
2010-08-05 17:32:47,304 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 54% complete
2010-08-05 17:35:02,343 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 56% complete
2010-08-05 17:38:33,767 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 60% complete
2010-08-05 17:38:37,285 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 100% complete
2010-08-05 17:38:37,286 [main] ERROR
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map reduce job(s) failed!
2010-08-05 17:38:37,772 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Some jobs have failed! Stop running all dependent jobs
2010-08-05 17:38:37,775 [main] ERROR org.apache.pig.tools.grunt.GruntParser
- ERROR 6015: During execution, encountered a Hadoop error.
Details at logfile: /mnt/var/lib/hadoop/steps/2/pig_1281026370670.log
Command exiting with ret '255'
Thanks for any enlightenment you can provide
--
View this message in context: http://old.nabble.com/Problems-running-hadoop-on-Amazon-Elastic-MapReduce-tp29356384p29356384.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.