You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by "M. Dale" <me...@yahoo.com> on 2014/07/03 03:04:18 UTC
Re: Partitioning and setup errors
Two error messages seem to stand out:
1) Hadoop command-line option parsing not performed. Implement the Tool
interface and execute your application with ToolRunner to remedy this.
YourDriver extends Configured implements Tool?
Configuration conf = getConf(); //gets the configuration as parsed by
GenericOptionsParser via ToolRunner.run
Job job = Job.getInstance(conf);
2) No job jar file set. User classes may not be found. See Job or
Job#setJar(String).
job.setJarByClass(YourDriver.class); //see
http://stackoverflow.com/questions/3912267/hadoop-query-regarding-setjarbyclass-method-of-job-class
Hope this helps,
Markus
On 06/29/2014 08:26 AM, Chris Mawata wrote:
>
> It is the part about implementing the tool interface that is the issue.
> Does your driver class implement Tool?
> Chris
>
> On Jun 29, 2014 8:22 AM, "Chris MacKenzie"
> <studio@chrismackenziephotography.co.uk
> <ma...@chrismackenziephotography.co.uk>> wrote:
>
> HI Vinod,
>
> Thanks for your support. I’m packaging my application in Eclipse
> (Kepler) as a runnable jar. I can get the application to debug
> there in local mode.
>
> The command line I am using for my app is:
>
> "/usr/local/hadoop-2.4.0/jar_files $ hadoop jar
> /usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar
> a.txt 10 10”
>
> the jar permissions are -"rwxr-xr-x"
>
> The three parameters on the end of the command line are arbitrary
> commands from my sequential program which I will use as input when
> I can get this to run.
>
> I have downloaded the source and run a 2.4 version of WordCount,
> it’s not getting past the map stage either.
>
> I’m pasting in the whole job output.
>
> Deleted demo_output
>
> /usr/local/hadoop-2.4.0/jar_files $ hadoop jar
> /usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar
> a.txt 10 10
>
> Start time = 1404044126713
>
> SLF4J: Class path contains multiple SLF4J bindings.
>
> SLF4J: Found binding in [rsrc:org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:rsrc:slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for
> an explanation.
>
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
> 14/06/29 13:16:06 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java
> classes where applicable
>
> 14/06/29 13:16:31 INFO client.RMProxy: Connecting to
> ResourceManager at /0.0.0.0:8032 <http://0.0.0.0:8032>
>
> 14/06/29 13:16:46 WARN mapreduce.JobSubmitter: Hadoop command-line
> option parsing not performed. Implement the Tool interface and
> execute your application with ToolRunner to remedy this.
>
> 14/06/29 13:16:46 WARN mapreduce.JobSubmitter: No job jar file
> set. User classes may not be found. See Job or Job#setJar(String).
>
> 14/06/29 13:16:46 INFO input.FileInputFormat: Total input paths to
> process : 0
>
> 14/06/29 13:17:10 INFO mapreduce.JobSubmitter: number of splits:0
>
> 14/06/29 13:17:13 INFO mapreduce.JobSubmitter: Submitting tokens
> for job: job_1403953980618_0009
>
> 14/06/29 13:17:13 INFO mapred.YARNRunner: Job jar is not present.
> Not adding any jar to the list of resources.
>
> 14/06/29 13:17:13 INFO impl.YarnClientImpl: Submitted application
> application_1403953980618_0009
>
> 14/06/29 13:17:13 INFO mapreduce.Job: The url to track the job:
> http://admins-MacBook-Pro.local:8088/proxy/application_1403953980618_0009/
>
> 14/06/29 13:17:13 INFO mapreduce.Job: Running job:
> job_1403953980618_0009
>
> 14/06/29 13:17:19 INFO mapreduce.Job: Job job_1403953980618_0009
> running in uber mode : false
>
> 14/06/29 13:17:19 INFO mapreduce.Job: map 0% reduce 0%
>
> 14/06/29 13:17:23 INFO mapreduce.Job: Task Id :
> attempt_1403953980618_0009_r_000000_0, Status : FAILED
>
> Error: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
> at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
> Caused by: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
> ... 8 more
>
>
> 14/06/29 13:17:28 INFO mapreduce.Job: Task Id :
> attempt_1403953980618_0009_r_000000_1, Status : FAILED
>
> Error: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
> at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
> Caused by: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
> ... 8 more
>
>
> 14/06/29 13:17:32 INFO mapreduce.Job: Task Id :
> attempt_1403953980618_0009_r_000000_2, Status : FAILED
>
> Error: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
> at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
> Caused by: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
> ... 8 more
>
>
> 14/06/29 13:17:37 INFO mapreduce.Job: map 0% reduce 100%
>
> 14/06/29 13:17:37 INFO mapreduce.Job: Job job_1403953980618_0009
> failed with state FAILED due to: Task failed
> task_1403953980618_0009_r_000000
>
> Job failed as tasks failed. failedMaps:0 failedReduces:1
>
>
> 14/06/29 13:18:04 INFO mapreduce.Job: Counters: 7
>
> Job Counters
>
> Failed reduce tasks=4
>
> Launched reduce tasks=4
>
> Total time spent by all maps in occupied slots (ms)=0
>
> Total time spent by all reduces in occupied slots (ms)=7786
>
> Total time spent by all reduce tasks (ms)=7786
>
> Total vcore-seconds taken by all reduce tasks=7786
>
> Total megabyte-seconds taken by all reduce tasks=7972864
>
> 14/06/29 13:18:05 INFO ipc.Client: Retrying connect to server:
> admins-MacBook-Pro.local/192.168.0.5:53193
> <http://192.168.0.5:53193>. Already tried 0 time(s); retry policy
> is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
> sleepTime=1000 MILLISECONDS)
>
> 14/06/29 13:18:06 INFO ipc.Client: Retrying connect to server:
> admins-MacBook-Pro.local/192.168.0.5:53193
> <http://192.168.0.5:53193>. Already tried 1 time(s); retry policy
> is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
> sleepTime=1000 MILLISECONDS)
>
> 14/06/29 13:18:07 INFO ipc.Client: Retrying connect to server:
> admins-MacBook-Pro.local/192.168.0.5:53193
> <http://192.168.0.5:53193>. Already tried 2 time(s); retry policy
> is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
> sleepTime=1000 MILLISECONDS)
>
> 14/06/29 13:18:07 INFO mapred.ClientServiceDelegate: Application
> state is completed. FinalApplicationStatus=FAILED. Redirecting to
> job history server
>
> End time = 1404044288031
>
> Elapsed time = 161318 ms
>
> Finished
>
> /usr/local/hadoop-2.4.0/jar_files $
>
> Warm regards.
>
> Chris
>
>
> From: Vinod Kumar Vavilapalli <vinodkv@hortonworks.com
> <ma...@hortonworks.com>>
> Reply-To: <user@hadoop.apache.org <ma...@hadoop.apache.org>>
> Date: Sunday, 29 June 2014 00:20
> To: <user@hadoop.apache.org <ma...@hadoop.apache.org>>
> Subject: Re: Partitioning and setup errors
>
> What is happening is the client is not able to pick up the right
> jar to push to the cluster. It looks in the class-path for the jar
> that contains the class ParallelGeneticAlignment.
>
> How are you packaging your code? How are your running your job -
> paste the command line?
>
> +Vinod
>
> On Jun 27, 2014, at 5:15 AM, Chris MacKenzie
> <studio@chrismackenziephotography.co.uk
> <ma...@chrismackenziephotography.co.uk>> wrote:
>
>> Hi,
>>
>> I realise my previous question may have been a bit naïve and I
>> also realise I am asking an awful lot here, any advice would be
>> greatly appreciated.
>>
>> * I have been using Hadoop 2.4 in local mode and am sticking to
>> the mapreduce.* side of the track.
>> * I am using a Custom Line reader to read each sequence into a Map
>> * I have a partitioner class which is testing the key from the
>> map class.
>> * I've tried debugging in eclipse with a breakpoint in the
>> partitioner class but getPartition(LongWritable mapKey, Text
>> sequenceString, intnumReduceTasks) is not being called.
>>
>> Could there be any reason for that ?
>>
>> Because my map and reduce code works in local mode within
>> eclipse, I wondered if I may get the partitioner to work if
>> I changed to Pseudo Distributed Mode exporting a runnable jar
>> from Eclipse (Kepler)
>>
>> I have several faults On my own computer Pseudo Distributed Mode
>> and the university clusters Pseudo Distributed Mode which I set
>> up. I’ve googled and read extensively but am not seeing a
>> solution to any of these issues.
>>
>> I have this line:
>> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file
>> set. User classes may not be found. See Job or Job#setJar(String).
>> My driver code is:
>> private void doParallelConcordance() throws Exception {
>>
>>
>> Path inDir= newPath("input_sequences/10_sequences.txt");
>> Path outDir = new Path("demo_output");
>>
>> Job job = Job.getInstance(new Configuration());
>> job.setJarByClass(ParallelGeneticAlignment.class);
>>
>> job.setOutputKeyClass(Text.class);
>> job.setOutputValueClass(IntWritable.class);
>>
>> job.setInputFormatClass(CustomFileInputFormat.class);
>> job.setMapperClass(ConcordanceMapper.class);
>> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
>> job.setReducerClass(ConcordanceReducer.class);
>>
>> FileInputFormat.addInputPath(job, inDir);
>> FileOutputFormat.setOutputPath(job, outDir);
>>
>> job.waitForCompletion(true)
>> }
>>
>> On the university server I am getting this error:
>> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
>> attempt_1403860966764_0003_m_000000_0, Status : FAILED
>> Error: java.lang.RuntimeException:
>> java.lang.ClassNotFoundException: Class
>> par.gene.align.concordance.ConcordanceMapper not found
>>
>> On my machine the error is:
>> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
>> attempt_1403864060032_0004_r_000000_2, Status : FAILED
>> Error: java.lang.RuntimeException:
>> java.lang.ClassNotFoundException: Class
>> par.gene.align.concordance.ConcordanceReducer not found
>>
>> On the university server I get total paths to process:
>> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths
>> to process : 1
>> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>>
>> On my machine I get total paths to process:
>> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths
>> to process : 0
>> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>>
>> Being new to this community, I thought it polite to introduce
>> myself. I’m planning to return to software development via an MSc
>> at Heriot Watt University in Edinburgh. My MSc project is based
>> on Fosters Genetic Sequence Alignment. I have written a
>> sequential version my goal is now to port it to Hadoop.
>>
>> Thanks in advance,
>> Regards,
>>
>> Chris MacKenzie
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or
> entity to which it is addressed and may contain information that
> is confidential, privileged and exempt from disclosure under
> applicable law. If the reader of this message is not the intended
> recipient, you are hereby notified that any printing, copying,
> dissemination, distribution, disclosure or forwarding of this
> communication is strictly prohibited. If you have received this
> communication in error, please contact the sender immediately and
> delete it from your system. Thank You.
>