You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Chris MacKenzie <st...@chrismackenziephotography.co.uk> on 2014/06/27 14:15:47 UTC
Partitioning and setup errors
Hi,
I realise my previous question may have been a bit naïve and I also realise
I am asking an awful lot here, any advice would be greatly appreciated.
* I have been using Hadoop 2.4 in local mode and am sticking to the
mapreduce.* side of the track.
* I am using a Custom Line reader to read each sequence into a Map
* I have a partitioner class which is testing the key from the map class.
* I've tried debugging in eclipse with a breakpoint in the partitioner class
but getPartition(LongWritable mapKey, Text sequenceString, int
numReduceTasks) is not being called.
Could there be any reason for that ?
Because my map and reduce code works in local mode within eclipse, I
wondered if I may get the partitioner to work if I changed to Pseudo
Distributed Mode exporting a runnable jar from Eclipse (Kepler)
I have several faults On my own computer Pseudo Distributed Mode and the
university clusters Pseudo Distributed Mode which I set up. I¹ve googled and
read extensively but am not seeing a solution to any of these issues.
I have this line:
14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User
classes may not be found. See Job or Job#setJar(String).
My driver code is:
private void doParallelConcordance() throws Exception {
Path inDir = new Path("input_sequences/10_sequences.txt");
Path outDir = new Path("demo_output");
Job job = Job.getInstance(new Configuration());
job.setJarByClass(ParallelGeneticAlignment.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setInputFormatClass(CustomFileInputFormat.class);
job.setMapperClass(ConcordanceMapper.class);
job.setPartitionerClass(ConcordanceSequencePartitioner.class);
job.setReducerClass(ConcordanceReducer.class);
FileInputFormat.addInputPath(job, inDir);
FileOutputFormat.setOutputPath(job, outDir);
job.waitForCompletion(true)
}
On the university server I am getting this error:
4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
attempt_1403860966764_0003_m_000000_0, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceMapper not found
On my machine the error is:
4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
attempt_1403864060032_0004_r_000000_2, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceReducer not found
On the university server I get total paths to process:
14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process :
1
14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
On my machine I get total paths to process:
14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process :
0
14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
Being new to this community, I thought it polite to introduce myself. I¹m
planning to return to software development via an MSc at Heriot Watt
University in Edinburgh. My MSc project is based on Fosters Genetic Sequence
Alignment. I have written a sequential version my goal is now to port it to
Hadoop.
Thanks in advance,
Regards,
Chris MacKenzie
Re: Partitioning and setup errors
Posted by Chris MacKenzie <st...@chrismackenziephotography.co.uk>.
HI Chris,
I¹m away from my books for the weekend. Is that call (extends Configurable
implements Tool) a Hadoop 2 call. Would that mean that I am better sticking
with Hadoop 1.x ?
Warm regards.
Chris
<http://www.chrismackenziephotography.co.uk/>
Expert in all aspects of photography
telephone: 0131 332 6967 <tel:0131 332 6967>
email: studio@chrismackenziephotography.co.uk
corporate: www.chrismackenziephotography.co.uk
<http://www.chrismackenziephotography.co.uk/>
<http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
<http://twitter.com/#!/MacKenzieStudio>
<http://www.facebook.com/pages/Chris-MacKenzie-Photography/145946284250>
<http://www.linkedin.com/in/chrismackenziephotography/>
<http://pinterest.com/ChrisMacKenzieP/>
From: Chris Mawata <ch...@gmail.com>
Reply-To: <us...@hadoop.apache.org>
Date: Friday, 27 June 2014 23:46
To: <us...@hadoop.apache.org>
Subject: Re: Partitioning and setup errors
Probably my fault. I was looking for the
extends Configurable implements Tool
part. I will double check when I get home rather than send you on a wild
goose chase.
Cheers
Chris
On Jun 27, 2014 8:16 AM, "Chris MacKenzie"
<st...@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> I realise my previous question may have been a bit naïve and I also realise I
> am asking an awful lot here, any advice would be greatly appreciated.
> * I have been using Hadoop 2.4 in local mode and am sticking to the
> mapreduce.* side of the track.
> * I am using a Custom Line reader to read each sequence into a Map
> * I have a partitioner class which is testing the key from the map class.
> * I've tried debugging in eclipse with a breakpoint in the partitioner class
> but getPartition(LongWritable mapKey, Text sequenceString, int numReduceTasks)
> is not being called.
> Could there be any reason for that ?
>
> Because my map and reduce code works in local mode within eclipse, I wondered
> if I may get the partitioner to work if I changed to Pseudo Distributed Mode
> exporting a runnable jar from Eclipse (Kepler)
>
> I have several faults On my own computer Pseudo Distributed Mode and the
> university clusters Pseudo Distributed Mode which I set up. I¹ve googled and
> read extensively but am not seeing a solution to any of these issues.
>
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User
> classes may not be found. See Job or Job#setJar(String).
> My driver code is:
> private void doParallelConcordance() throws Exception {
>
> Path inDir = new Path("input_sequences/10_sequences.txt");
>
> Path outDir = new Path("demo_output");
>
>
>
> Job job = Job.getInstance(new Configuration());
>
> job.setJarByClass(ParallelGeneticAlignment.class);
>
> job.setOutputKeyClass(Text.class);
>
> job.setOutputValueClass(IntWritable.class);
>
>
>
> job.setInputFormatClass(CustomFileInputFormat.class);
>
> job.setMapperClass(ConcordanceMapper.class);
>
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
>
> job.setReducerClass(ConcordanceReducer.class);
>
>
>
> FileInputFormat.addInputPath(job, inDir);
>
> FileOutputFormat.setOutputPath(job, outDir);
>
>
>
> job.waitForCompletion(true)
>
> }
>
>
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
> attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceMapper not found
>
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
> attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>
> Being new to this community, I thought it polite to introduce myself. I¹m
> planning to return to software development via an MSc at Heriot Watt
> University in Edinburgh. My MSc project is based on Fosters Genetic Sequence
> Alignment. I have written a sequential version my goal is now to port it to
> Hadoop.
>
> Thanks in advance,
> Regards,
>
> Chris MacKenzie
Re: Partitioning and setup errors
Posted by Chris MacKenzie <st...@chrismackenziephotography.co.uk>.
HI Chris,
I¹m away from my books for the weekend. Is that call (extends Configurable
implements Tool) a Hadoop 2 call. Would that mean that I am better sticking
with Hadoop 1.x ?
Warm regards.
Chris
<http://www.chrismackenziephotography.co.uk/>
Expert in all aspects of photography
telephone: 0131 332 6967 <tel:0131 332 6967>
email: studio@chrismackenziephotography.co.uk
corporate: www.chrismackenziephotography.co.uk
<http://www.chrismackenziephotography.co.uk/>
<http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
<http://twitter.com/#!/MacKenzieStudio>
<http://www.facebook.com/pages/Chris-MacKenzie-Photography/145946284250>
<http://www.linkedin.com/in/chrismackenziephotography/>
<http://pinterest.com/ChrisMacKenzieP/>
From: Chris Mawata <ch...@gmail.com>
Reply-To: <us...@hadoop.apache.org>
Date: Friday, 27 June 2014 23:46
To: <us...@hadoop.apache.org>
Subject: Re: Partitioning and setup errors
Probably my fault. I was looking for the
extends Configurable implements Tool
part. I will double check when I get home rather than send you on a wild
goose chase.
Cheers
Chris
On Jun 27, 2014 8:16 AM, "Chris MacKenzie"
<st...@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> I realise my previous question may have been a bit naïve and I also realise I
> am asking an awful lot here, any advice would be greatly appreciated.
> * I have been using Hadoop 2.4 in local mode and am sticking to the
> mapreduce.* side of the track.
> * I am using a Custom Line reader to read each sequence into a Map
> * I have a partitioner class which is testing the key from the map class.
> * I've tried debugging in eclipse with a breakpoint in the partitioner class
> but getPartition(LongWritable mapKey, Text sequenceString, int numReduceTasks)
> is not being called.
> Could there be any reason for that ?
>
> Because my map and reduce code works in local mode within eclipse, I wondered
> if I may get the partitioner to work if I changed to Pseudo Distributed Mode
> exporting a runnable jar from Eclipse (Kepler)
>
> I have several faults On my own computer Pseudo Distributed Mode and the
> university clusters Pseudo Distributed Mode which I set up. I¹ve googled and
> read extensively but am not seeing a solution to any of these issues.
>
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User
> classes may not be found. See Job or Job#setJar(String).
> My driver code is:
> private void doParallelConcordance() throws Exception {
>
> Path inDir = new Path("input_sequences/10_sequences.txt");
>
> Path outDir = new Path("demo_output");
>
>
>
> Job job = Job.getInstance(new Configuration());
>
> job.setJarByClass(ParallelGeneticAlignment.class);
>
> job.setOutputKeyClass(Text.class);
>
> job.setOutputValueClass(IntWritable.class);
>
>
>
> job.setInputFormatClass(CustomFileInputFormat.class);
>
> job.setMapperClass(ConcordanceMapper.class);
>
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
>
> job.setReducerClass(ConcordanceReducer.class);
>
>
>
> FileInputFormat.addInputPath(job, inDir);
>
> FileOutputFormat.setOutputPath(job, outDir);
>
>
>
> job.waitForCompletion(true)
>
> }
>
>
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
> attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceMapper not found
>
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
> attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>
> Being new to this community, I thought it polite to introduce myself. I¹m
> planning to return to software development via an MSc at Heriot Watt
> University in Edinburgh. My MSc project is based on Fosters Genetic Sequence
> Alignment. I have written a sequential version my goal is now to port it to
> Hadoop.
>
> Thanks in advance,
> Regards,
>
> Chris MacKenzie
Re: Partitioning and setup errors
Posted by Chris MacKenzie <st...@chrismackenziephotography.co.uk>.
HI Chris,
I¹m away from my books for the weekend. Is that call (extends Configurable
implements Tool) a Hadoop 2 call. Would that mean that I am better sticking
with Hadoop 1.x ?
Warm regards.
Chris
<http://www.chrismackenziephotography.co.uk/>
Expert in all aspects of photography
telephone: 0131 332 6967 <tel:0131 332 6967>
email: studio@chrismackenziephotography.co.uk
corporate: www.chrismackenziephotography.co.uk
<http://www.chrismackenziephotography.co.uk/>
<http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
<http://twitter.com/#!/MacKenzieStudio>
<http://www.facebook.com/pages/Chris-MacKenzie-Photography/145946284250>
<http://www.linkedin.com/in/chrismackenziephotography/>
<http://pinterest.com/ChrisMacKenzieP/>
From: Chris Mawata <ch...@gmail.com>
Reply-To: <us...@hadoop.apache.org>
Date: Friday, 27 June 2014 23:46
To: <us...@hadoop.apache.org>
Subject: Re: Partitioning and setup errors
Probably my fault. I was looking for the
extends Configurable implements Tool
part. I will double check when I get home rather than send you on a wild
goose chase.
Cheers
Chris
On Jun 27, 2014 8:16 AM, "Chris MacKenzie"
<st...@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> I realise my previous question may have been a bit naïve and I also realise I
> am asking an awful lot here, any advice would be greatly appreciated.
> * I have been using Hadoop 2.4 in local mode and am sticking to the
> mapreduce.* side of the track.
> * I am using a Custom Line reader to read each sequence into a Map
> * I have a partitioner class which is testing the key from the map class.
> * I've tried debugging in eclipse with a breakpoint in the partitioner class
> but getPartition(LongWritable mapKey, Text sequenceString, int numReduceTasks)
> is not being called.
> Could there be any reason for that ?
>
> Because my map and reduce code works in local mode within eclipse, I wondered
> if I may get the partitioner to work if I changed to Pseudo Distributed Mode
> exporting a runnable jar from Eclipse (Kepler)
>
> I have several faults On my own computer Pseudo Distributed Mode and the
> university clusters Pseudo Distributed Mode which I set up. I¹ve googled and
> read extensively but am not seeing a solution to any of these issues.
>
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User
> classes may not be found. See Job or Job#setJar(String).
> My driver code is:
> private void doParallelConcordance() throws Exception {
>
> Path inDir = new Path("input_sequences/10_sequences.txt");
>
> Path outDir = new Path("demo_output");
>
>
>
> Job job = Job.getInstance(new Configuration());
>
> job.setJarByClass(ParallelGeneticAlignment.class);
>
> job.setOutputKeyClass(Text.class);
>
> job.setOutputValueClass(IntWritable.class);
>
>
>
> job.setInputFormatClass(CustomFileInputFormat.class);
>
> job.setMapperClass(ConcordanceMapper.class);
>
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
>
> job.setReducerClass(ConcordanceReducer.class);
>
>
>
> FileInputFormat.addInputPath(job, inDir);
>
> FileOutputFormat.setOutputPath(job, outDir);
>
>
>
> job.waitForCompletion(true)
>
> }
>
>
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
> attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceMapper not found
>
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
> attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>
> Being new to this community, I thought it polite to introduce myself. I¹m
> planning to return to software development via an MSc at Heriot Watt
> University in Edinburgh. My MSc project is based on Fosters Genetic Sequence
> Alignment. I have written a sequential version my goal is now to port it to
> Hadoop.
>
> Thanks in advance,
> Regards,
>
> Chris MacKenzie
Re: Partitioning and setup errors
Posted by Chris MacKenzie <st...@chrismackenziephotography.co.uk>.
HI Chris,
I¹m away from my books for the weekend. Is that call (extends Configurable
implements Tool) a Hadoop 2 call. Would that mean that I am better sticking
with Hadoop 1.x ?
Warm regards.
Chris
<http://www.chrismackenziephotography.co.uk/>
Expert in all aspects of photography
telephone: 0131 332 6967 <tel:0131 332 6967>
email: studio@chrismackenziephotography.co.uk
corporate: www.chrismackenziephotography.co.uk
<http://www.chrismackenziephotography.co.uk/>
<http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
<http://twitter.com/#!/MacKenzieStudio>
<http://www.facebook.com/pages/Chris-MacKenzie-Photography/145946284250>
<http://www.linkedin.com/in/chrismackenziephotography/>
<http://pinterest.com/ChrisMacKenzieP/>
From: Chris Mawata <ch...@gmail.com>
Reply-To: <us...@hadoop.apache.org>
Date: Friday, 27 June 2014 23:46
To: <us...@hadoop.apache.org>
Subject: Re: Partitioning and setup errors
Probably my fault. I was looking for the
extends Configurable implements Tool
part. I will double check when I get home rather than send you on a wild
goose chase.
Cheers
Chris
On Jun 27, 2014 8:16 AM, "Chris MacKenzie"
<st...@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> I realise my previous question may have been a bit naïve and I also realise I
> am asking an awful lot here, any advice would be greatly appreciated.
> * I have been using Hadoop 2.4 in local mode and am sticking to the
> mapreduce.* side of the track.
> * I am using a Custom Line reader to read each sequence into a Map
> * I have a partitioner class which is testing the key from the map class.
> * I've tried debugging in eclipse with a breakpoint in the partitioner class
> but getPartition(LongWritable mapKey, Text sequenceString, int numReduceTasks)
> is not being called.
> Could there be any reason for that ?
>
> Because my map and reduce code works in local mode within eclipse, I wondered
> if I may get the partitioner to work if I changed to Pseudo Distributed Mode
> exporting a runnable jar from Eclipse (Kepler)
>
> I have several faults On my own computer Pseudo Distributed Mode and the
> university clusters Pseudo Distributed Mode which I set up. I¹ve googled and
> read extensively but am not seeing a solution to any of these issues.
>
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User
> classes may not be found. See Job or Job#setJar(String).
> My driver code is:
> private void doParallelConcordance() throws Exception {
>
> Path inDir = new Path("input_sequences/10_sequences.txt");
>
> Path outDir = new Path("demo_output");
>
>
>
> Job job = Job.getInstance(new Configuration());
>
> job.setJarByClass(ParallelGeneticAlignment.class);
>
> job.setOutputKeyClass(Text.class);
>
> job.setOutputValueClass(IntWritable.class);
>
>
>
> job.setInputFormatClass(CustomFileInputFormat.class);
>
> job.setMapperClass(ConcordanceMapper.class);
>
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
>
> job.setReducerClass(ConcordanceReducer.class);
>
>
>
> FileInputFormat.addInputPath(job, inDir);
>
> FileOutputFormat.setOutputPath(job, outDir);
>
>
>
> job.waitForCompletion(true)
>
> }
>
>
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
> attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceMapper not found
>
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
> attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>
> Being new to this community, I thought it polite to introduce myself. I¹m
> planning to return to software development via an MSc at Heriot Watt
> University in Edinburgh. My MSc project is based on Fosters Genetic Sequence
> Alignment. I have written a sequential version my goal is now to port it to
> Hadoop.
>
> Thanks in advance,
> Regards,
>
> Chris MacKenzie
Re: Partitioning and setup errors
Posted by Chris Mawata <ch...@gmail.com>.
Probably my fault. I was looking for the
extends Configurable implements Tool
part. I will double check when I get home rather than send you on a wild
goose chase.
Cheers
Chris
On Jun 27, 2014 8:16 AM, "Chris MacKenzie" <
studio@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> I realise my previous question may have been a bit naïve and I also
> realise I am asking an awful lot here, any advice would be greatly
> appreciated.
>
> - I have been using Hadoop 2.4 in local mode and am sticking to the mapreduce.*
> side of the track.
> - I am using a Custom Line reader to read each sequence into a Map
> - I have a partitioner class which is testing the key from the map
> class.
> - I've tried debugging in eclipse with a breakpoint in the partitioner
> class but getPartition(LongWritable mapKey, Text sequenceString, int
> numReduceTasks) is not being called.
>
> Could there be any reason for that ?
>
> Because my map and reduce code works in local mode within eclipse, I wondered
> if I may get the partitioner to work if I changed to Pseudo Distributed
> Mode exporting a runnable jar from Eclipse (Kepler)
>
> I have several faults On my own computer Pseudo Distributed Mode and the
> university clusters Pseudo Distributed Mode which I set up. I’ve googled
> and read extensively but am not seeing a solution to any of these issues.
>
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User
> classes may not be found. See Job or Job#setJar(String).
> My driver code is:
>
> private void doParallelConcordance() throws Exception {
>
> Path inDir = new Path("input_sequences/10_sequences.txt");
>
> Path outDir = new Path("demo_output");
>
>
> Job job = Job.getInstance(new Configuration());
>
> job.setJarByClass(ParallelGeneticAlignment.class);
>
> job.setOutputKeyClass(Text.class);
>
> job.setOutputValueClass(IntWritable.class);
>
>
> job.setInputFormatClass(CustomFileInputFormat.class);
>
> job.setMapperClass(ConcordanceMapper.class);
>
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
>
> job.setReducerClass(ConcordanceReducer.class);
>
>
> FileInputFormat.addInputPath(job, inDir);
>
> FileOutputFormat.setOutputPath(job, outDir);
>
>
> job.waitForCompletion(true)
>
> }
>
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
> attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceMapper not found
>
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
> attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process
> : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process
> : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>
> Being new to this community, I thought it polite to introduce myself. I’m
> planning to return to software development via an MSc at Heriot Watt
> University in Edinburgh. My MSc project is based on Fosters Genetic
> Sequence Alignment. I have written a sequential version my goal is now to
> port it to Hadoop.
>
> Thanks in advance,
> Regards,
>
> Chris MacKenzie
>
Re: Partitioning and setup errors
Posted by "M. Dale" <me...@yahoo.com>.
Two error messages seem to stand out:
1) Hadoop command-line option parsing not performed. Implement the Tool
interface and execute your application with ToolRunner to remedy this.
YourDriver extends Configured implements Tool?
Configuration conf = getConf(); //gets the configuration as parsed by
GenericOptionsParser via ToolRunner.run
Job job = Job.getInstance(conf);
2) No job jar file set. User classes may not be found. See Job or
Job#setJar(String).
job.setJarByClass(YourDriver.class); //see
http://stackoverflow.com/questions/3912267/hadoop-query-regarding-setjarbyclass-method-of-job-class
Hope this helps,
Markus
On 06/29/2014 08:26 AM, Chris Mawata wrote:
>
> It is the part about implementing the tool interface that is the issue.
> Does your driver class implement Tool?
> Chris
>
> On Jun 29, 2014 8:22 AM, "Chris MacKenzie"
> <studio@chrismackenziephotography.co.uk
> <ma...@chrismackenziephotography.co.uk>> wrote:
>
> HI Vinod,
>
> Thanks for your support. I’m packaging my application in Eclipse
> (Kepler) as a runnable jar. I can get the application to debug
> there in local mode.
>
> The command line I am using for my app is:
>
> "/usr/local/hadoop-2.4.0/jar_files $ hadoop jar
> /usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar
> a.txt 10 10”
>
> the jar permissions are -"rwxr-xr-x"
>
> The three parameters on the end of the command line are arbitrary
> commands from my sequential program which I will use as input when
> I can get this to run.
>
> I have downloaded the source and run a 2.4 version of WordCount,
> it’s not getting past the map stage either.
>
> I’m pasting in the whole job output.
>
> Deleted demo_output
>
> /usr/local/hadoop-2.4.0/jar_files $ hadoop jar
> /usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar
> a.txt 10 10
>
> Start time = 1404044126713
>
> SLF4J: Class path contains multiple SLF4J bindings.
>
> SLF4J: Found binding in [rsrc:org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:rsrc:slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for
> an explanation.
>
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
> 14/06/29 13:16:06 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java
> classes where applicable
>
> 14/06/29 13:16:31 INFO client.RMProxy: Connecting to
> ResourceManager at /0.0.0.0:8032 <http://0.0.0.0:8032>
>
> 14/06/29 13:16:46 WARN mapreduce.JobSubmitter: Hadoop command-line
> option parsing not performed. Implement the Tool interface and
> execute your application with ToolRunner to remedy this.
>
> 14/06/29 13:16:46 WARN mapreduce.JobSubmitter: No job jar file
> set. User classes may not be found. See Job or Job#setJar(String).
>
> 14/06/29 13:16:46 INFO input.FileInputFormat: Total input paths to
> process : 0
>
> 14/06/29 13:17:10 INFO mapreduce.JobSubmitter: number of splits:0
>
> 14/06/29 13:17:13 INFO mapreduce.JobSubmitter: Submitting tokens
> for job: job_1403953980618_0009
>
> 14/06/29 13:17:13 INFO mapred.YARNRunner: Job jar is not present.
> Not adding any jar to the list of resources.
>
> 14/06/29 13:17:13 INFO impl.YarnClientImpl: Submitted application
> application_1403953980618_0009
>
> 14/06/29 13:17:13 INFO mapreduce.Job: The url to track the job:
> http://admins-MacBook-Pro.local:8088/proxy/application_1403953980618_0009/
>
> 14/06/29 13:17:13 INFO mapreduce.Job: Running job:
> job_1403953980618_0009
>
> 14/06/29 13:17:19 INFO mapreduce.Job: Job job_1403953980618_0009
> running in uber mode : false
>
> 14/06/29 13:17:19 INFO mapreduce.Job: map 0% reduce 0%
>
> 14/06/29 13:17:23 INFO mapreduce.Job: Task Id :
> attempt_1403953980618_0009_r_000000_0, Status : FAILED
>
> Error: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
> at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
> Caused by: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
> ... 8 more
>
>
> 14/06/29 13:17:28 INFO mapreduce.Job: Task Id :
> attempt_1403953980618_0009_r_000000_1, Status : FAILED
>
> Error: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
> at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
> Caused by: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
> ... 8 more
>
>
> 14/06/29 13:17:32 INFO mapreduce.Job: Task Id :
> attempt_1403953980618_0009_r_000000_2, Status : FAILED
>
> Error: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
> at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
> Caused by: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
> ... 8 more
>
>
> 14/06/29 13:17:37 INFO mapreduce.Job: map 0% reduce 100%
>
> 14/06/29 13:17:37 INFO mapreduce.Job: Job job_1403953980618_0009
> failed with state FAILED due to: Task failed
> task_1403953980618_0009_r_000000
>
> Job failed as tasks failed. failedMaps:0 failedReduces:1
>
>
> 14/06/29 13:18:04 INFO mapreduce.Job: Counters: 7
>
> Job Counters
>
> Failed reduce tasks=4
>
> Launched reduce tasks=4
>
> Total time spent by all maps in occupied slots (ms)=0
>
> Total time spent by all reduces in occupied slots (ms)=7786
>
> Total time spent by all reduce tasks (ms)=7786
>
> Total vcore-seconds taken by all reduce tasks=7786
>
> Total megabyte-seconds taken by all reduce tasks=7972864
>
> 14/06/29 13:18:05 INFO ipc.Client: Retrying connect to server:
> admins-MacBook-Pro.local/192.168.0.5:53193
> <http://192.168.0.5:53193>. Already tried 0 time(s); retry policy
> is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
> sleepTime=1000 MILLISECONDS)
>
> 14/06/29 13:18:06 INFO ipc.Client: Retrying connect to server:
> admins-MacBook-Pro.local/192.168.0.5:53193
> <http://192.168.0.5:53193>. Already tried 1 time(s); retry policy
> is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
> sleepTime=1000 MILLISECONDS)
>
> 14/06/29 13:18:07 INFO ipc.Client: Retrying connect to server:
> admins-MacBook-Pro.local/192.168.0.5:53193
> <http://192.168.0.5:53193>. Already tried 2 time(s); retry policy
> is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
> sleepTime=1000 MILLISECONDS)
>
> 14/06/29 13:18:07 INFO mapred.ClientServiceDelegate: Application
> state is completed. FinalApplicationStatus=FAILED. Redirecting to
> job history server
>
> End time = 1404044288031
>
> Elapsed time = 161318 ms
>
> Finished
>
> /usr/local/hadoop-2.4.0/jar_files $
>
> Warm regards.
>
> Chris
>
>
> From: Vinod Kumar Vavilapalli <vinodkv@hortonworks.com
> <ma...@hortonworks.com>>
> Reply-To: <user@hadoop.apache.org <ma...@hadoop.apache.org>>
> Date: Sunday, 29 June 2014 00:20
> To: <user@hadoop.apache.org <ma...@hadoop.apache.org>>
> Subject: Re: Partitioning and setup errors
>
> What is happening is the client is not able to pick up the right
> jar to push to the cluster. It looks in the class-path for the jar
> that contains the class ParallelGeneticAlignment.
>
> How are you packaging your code? How are your running your job -
> paste the command line?
>
> +Vinod
>
> On Jun 27, 2014, at 5:15 AM, Chris MacKenzie
> <studio@chrismackenziephotography.co.uk
> <ma...@chrismackenziephotography.co.uk>> wrote:
>
>> Hi,
>>
>> I realise my previous question may have been a bit naïve and I
>> also realise I am asking an awful lot here, any advice would be
>> greatly appreciated.
>>
>> * I have been using Hadoop 2.4 in local mode and am sticking to
>> the mapreduce.* side of the track.
>> * I am using a Custom Line reader to read each sequence into a Map
>> * I have a partitioner class which is testing the key from the
>> map class.
>> * I've tried debugging in eclipse with a breakpoint in the
>> partitioner class but getPartition(LongWritable mapKey, Text
>> sequenceString, intnumReduceTasks) is not being called.
>>
>> Could there be any reason for that ?
>>
>> Because my map and reduce code works in local mode within
>> eclipse, I wondered if I may get the partitioner to work if
>> I changed to Pseudo Distributed Mode exporting a runnable jar
>> from Eclipse (Kepler)
>>
>> I have several faults On my own computer Pseudo Distributed Mode
>> and the university clusters Pseudo Distributed Mode which I set
>> up. I’ve googled and read extensively but am not seeing a
>> solution to any of these issues.
>>
>> I have this line:
>> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file
>> set. User classes may not be found. See Job or Job#setJar(String).
>> My driver code is:
>> private void doParallelConcordance() throws Exception {
>>
>>
>> Path inDir= newPath("input_sequences/10_sequences.txt");
>> Path outDir = new Path("demo_output");
>>
>> Job job = Job.getInstance(new Configuration());
>> job.setJarByClass(ParallelGeneticAlignment.class);
>>
>> job.setOutputKeyClass(Text.class);
>> job.setOutputValueClass(IntWritable.class);
>>
>> job.setInputFormatClass(CustomFileInputFormat.class);
>> job.setMapperClass(ConcordanceMapper.class);
>> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
>> job.setReducerClass(ConcordanceReducer.class);
>>
>> FileInputFormat.addInputPath(job, inDir);
>> FileOutputFormat.setOutputPath(job, outDir);
>>
>> job.waitForCompletion(true)
>> }
>>
>> On the university server I am getting this error:
>> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
>> attempt_1403860966764_0003_m_000000_0, Status : FAILED
>> Error: java.lang.RuntimeException:
>> java.lang.ClassNotFoundException: Class
>> par.gene.align.concordance.ConcordanceMapper not found
>>
>> On my machine the error is:
>> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
>> attempt_1403864060032_0004_r_000000_2, Status : FAILED
>> Error: java.lang.RuntimeException:
>> java.lang.ClassNotFoundException: Class
>> par.gene.align.concordance.ConcordanceReducer not found
>>
>> On the university server I get total paths to process:
>> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths
>> to process : 1
>> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>>
>> On my machine I get total paths to process:
>> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths
>> to process : 0
>> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>>
>> Being new to this community, I thought it polite to introduce
>> myself. I’m planning to return to software development via an MSc
>> at Heriot Watt University in Edinburgh. My MSc project is based
>> on Fosters Genetic Sequence Alignment. I have written a
>> sequential version my goal is now to port it to Hadoop.
>>
>> Thanks in advance,
>> Regards,
>>
>> Chris MacKenzie
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or
> entity to which it is addressed and may contain information that
> is confidential, privileged and exempt from disclosure under
> applicable law. If the reader of this message is not the intended
> recipient, you are hereby notified that any printing, copying,
> dissemination, distribution, disclosure or forwarding of this
> communication is strictly prohibited. If you have received this
> communication in error, please contact the sender immediately and
> delete it from your system. Thank You.
>
Re: Partitioning and setup errors
Posted by "M. Dale" <me...@yahoo.com>.
Two error messages seem to stand out:
1) Hadoop command-line option parsing not performed. Implement the Tool
interface and execute your application with ToolRunner to remedy this.
YourDriver extends Configured implements Tool?
Configuration conf = getConf(); //gets the configuration as parsed by
GenericOptionsParser via ToolRunner.run
Job job = Job.getInstance(conf);
2) No job jar file set. User classes may not be found. See Job or
Job#setJar(String).
job.setJarByClass(YourDriver.class); //see
http://stackoverflow.com/questions/3912267/hadoop-query-regarding-setjarbyclass-method-of-job-class
Hope this helps,
Markus
On 06/29/2014 08:26 AM, Chris Mawata wrote:
>
> It is the part about implementing the tool interface that is the issue.
> Does your driver class implement Tool?
> Chris
>
> On Jun 29, 2014 8:22 AM, "Chris MacKenzie"
> <studio@chrismackenziephotography.co.uk
> <ma...@chrismackenziephotography.co.uk>> wrote:
>
> HI Vinod,
>
> Thanks for your support. I’m packaging my application in Eclipse
> (Kepler) as a runnable jar. I can get the application to debug
> there in local mode.
>
> The command line I am using for my app is:
>
> "/usr/local/hadoop-2.4.0/jar_files $ hadoop jar
> /usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar
> a.txt 10 10”
>
> the jar permissions are -"rwxr-xr-x"
>
> The three parameters on the end of the command line are arbitrary
> commands from my sequential program which I will use as input when
> I can get this to run.
>
> I have downloaded the source and run a 2.4 version of WordCount,
> it’s not getting past the map stage either.
>
> I’m pasting in the whole job output.
>
> Deleted demo_output
>
> /usr/local/hadoop-2.4.0/jar_files $ hadoop jar
> /usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar
> a.txt 10 10
>
> Start time = 1404044126713
>
> SLF4J: Class path contains multiple SLF4J bindings.
>
> SLF4J: Found binding in [rsrc:org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:rsrc:slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for
> an explanation.
>
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
> 14/06/29 13:16:06 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java
> classes where applicable
>
> 14/06/29 13:16:31 INFO client.RMProxy: Connecting to
> ResourceManager at /0.0.0.0:8032 <http://0.0.0.0:8032>
>
> 14/06/29 13:16:46 WARN mapreduce.JobSubmitter: Hadoop command-line
> option parsing not performed. Implement the Tool interface and
> execute your application with ToolRunner to remedy this.
>
> 14/06/29 13:16:46 WARN mapreduce.JobSubmitter: No job jar file
> set. User classes may not be found. See Job or Job#setJar(String).
>
> 14/06/29 13:16:46 INFO input.FileInputFormat: Total input paths to
> process : 0
>
> 14/06/29 13:17:10 INFO mapreduce.JobSubmitter: number of splits:0
>
> 14/06/29 13:17:13 INFO mapreduce.JobSubmitter: Submitting tokens
> for job: job_1403953980618_0009
>
> 14/06/29 13:17:13 INFO mapred.YARNRunner: Job jar is not present.
> Not adding any jar to the list of resources.
>
> 14/06/29 13:17:13 INFO impl.YarnClientImpl: Submitted application
> application_1403953980618_0009
>
> 14/06/29 13:17:13 INFO mapreduce.Job: The url to track the job:
> http://admins-MacBook-Pro.local:8088/proxy/application_1403953980618_0009/
>
> 14/06/29 13:17:13 INFO mapreduce.Job: Running job:
> job_1403953980618_0009
>
> 14/06/29 13:17:19 INFO mapreduce.Job: Job job_1403953980618_0009
> running in uber mode : false
>
> 14/06/29 13:17:19 INFO mapreduce.Job: map 0% reduce 0%
>
> 14/06/29 13:17:23 INFO mapreduce.Job: Task Id :
> attempt_1403953980618_0009_r_000000_0, Status : FAILED
>
> Error: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
> at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
> Caused by: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
> ... 8 more
>
>
> 14/06/29 13:17:28 INFO mapreduce.Job: Task Id :
> attempt_1403953980618_0009_r_000000_1, Status : FAILED
>
> Error: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
> at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
> Caused by: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
> ... 8 more
>
>
> 14/06/29 13:17:32 INFO mapreduce.Job: Task Id :
> attempt_1403953980618_0009_r_000000_2, Status : FAILED
>
> Error: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
> at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
> Caused by: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
> ... 8 more
>
>
> 14/06/29 13:17:37 INFO mapreduce.Job: map 0% reduce 100%
>
> 14/06/29 13:17:37 INFO mapreduce.Job: Job job_1403953980618_0009
> failed with state FAILED due to: Task failed
> task_1403953980618_0009_r_000000
>
> Job failed as tasks failed. failedMaps:0 failedReduces:1
>
>
> 14/06/29 13:18:04 INFO mapreduce.Job: Counters: 7
>
> Job Counters
>
> Failed reduce tasks=4
>
> Launched reduce tasks=4
>
> Total time spent by all maps in occupied slots (ms)=0
>
> Total time spent by all reduces in occupied slots (ms)=7786
>
> Total time spent by all reduce tasks (ms)=7786
>
> Total vcore-seconds taken by all reduce tasks=7786
>
> Total megabyte-seconds taken by all reduce tasks=7972864
>
> 14/06/29 13:18:05 INFO ipc.Client: Retrying connect to server:
> admins-MacBook-Pro.local/192.168.0.5:53193
> <http://192.168.0.5:53193>. Already tried 0 time(s); retry policy
> is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
> sleepTime=1000 MILLISECONDS)
>
> 14/06/29 13:18:06 INFO ipc.Client: Retrying connect to server:
> admins-MacBook-Pro.local/192.168.0.5:53193
> <http://192.168.0.5:53193>. Already tried 1 time(s); retry policy
> is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
> sleepTime=1000 MILLISECONDS)
>
> 14/06/29 13:18:07 INFO ipc.Client: Retrying connect to server:
> admins-MacBook-Pro.local/192.168.0.5:53193
> <http://192.168.0.5:53193>. Already tried 2 time(s); retry policy
> is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
> sleepTime=1000 MILLISECONDS)
>
> 14/06/29 13:18:07 INFO mapred.ClientServiceDelegate: Application
> state is completed. FinalApplicationStatus=FAILED. Redirecting to
> job history server
>
> End time = 1404044288031
>
> Elapsed time = 161318 ms
>
> Finished
>
> /usr/local/hadoop-2.4.0/jar_files $
>
> Warm regards.
>
> Chris
>
>
> From: Vinod Kumar Vavilapalli <vinodkv@hortonworks.com
> <ma...@hortonworks.com>>
> Reply-To: <user@hadoop.apache.org <ma...@hadoop.apache.org>>
> Date: Sunday, 29 June 2014 00:20
> To: <user@hadoop.apache.org <ma...@hadoop.apache.org>>
> Subject: Re: Partitioning and setup errors
>
> What is happening is the client is not able to pick up the right
> jar to push to the cluster. It looks in the class-path for the jar
> that contains the class ParallelGeneticAlignment.
>
> How are you packaging your code? How are your running your job -
> paste the command line?
>
> +Vinod
>
> On Jun 27, 2014, at 5:15 AM, Chris MacKenzie
> <studio@chrismackenziephotography.co.uk
> <ma...@chrismackenziephotography.co.uk>> wrote:
>
>> Hi,
>>
>> I realise my previous question may have been a bit naïve and I
>> also realise I am asking an awful lot here, any advice would be
>> greatly appreciated.
>>
>> * I have been using Hadoop 2.4 in local mode and am sticking to
>> the mapreduce.* side of the track.
>> * I am using a Custom Line reader to read each sequence into a Map
>> * I have a partitioner class which is testing the key from the
>> map class.
>> * I've tried debugging in eclipse with a breakpoint in the
>> partitioner class but getPartition(LongWritable mapKey, Text
>> sequenceString, intnumReduceTasks) is not being called.
>>
>> Could there be any reason for that ?
>>
>> Because my map and reduce code works in local mode within
>> eclipse, I wondered if I may get the partitioner to work if
>> I changed to Pseudo Distributed Mode exporting a runnable jar
>> from Eclipse (Kepler)
>>
>> I have several faults On my own computer Pseudo Distributed Mode
>> and the university clusters Pseudo Distributed Mode which I set
>> up. I’ve googled and read extensively but am not seeing a
>> solution to any of these issues.
>>
>> I have this line:
>> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file
>> set. User classes may not be found. See Job or Job#setJar(String).
>> My driver code is:
>> private void doParallelConcordance() throws Exception {
>>
>>
>> Path inDir= newPath("input_sequences/10_sequences.txt");
>> Path outDir = new Path("demo_output");
>>
>> Job job = Job.getInstance(new Configuration());
>> job.setJarByClass(ParallelGeneticAlignment.class);
>>
>> job.setOutputKeyClass(Text.class);
>> job.setOutputValueClass(IntWritable.class);
>>
>> job.setInputFormatClass(CustomFileInputFormat.class);
>> job.setMapperClass(ConcordanceMapper.class);
>> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
>> job.setReducerClass(ConcordanceReducer.class);
>>
>> FileInputFormat.addInputPath(job, inDir);
>> FileOutputFormat.setOutputPath(job, outDir);
>>
>> job.waitForCompletion(true)
>> }
>>
>> On the university server I am getting this error:
>> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
>> attempt_1403860966764_0003_m_000000_0, Status : FAILED
>> Error: java.lang.RuntimeException:
>> java.lang.ClassNotFoundException: Class
>> par.gene.align.concordance.ConcordanceMapper not found
>>
>> On my machine the error is:
>> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
>> attempt_1403864060032_0004_r_000000_2, Status : FAILED
>> Error: java.lang.RuntimeException:
>> java.lang.ClassNotFoundException: Class
>> par.gene.align.concordance.ConcordanceReducer not found
>>
>> On the university server I get total paths to process:
>> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths
>> to process : 1
>> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>>
>> On my machine I get total paths to process:
>> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths
>> to process : 0
>> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>>
>> Being new to this community, I thought it polite to introduce
>> myself. I’m planning to return to software development via an MSc
>> at Heriot Watt University in Edinburgh. My MSc project is based
>> on Fosters Genetic Sequence Alignment. I have written a
>> sequential version my goal is now to port it to Hadoop.
>>
>> Thanks in advance,
>> Regards,
>>
>> Chris MacKenzie
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or
> entity to which it is addressed and may contain information that
> is confidential, privileged and exempt from disclosure under
> applicable law. If the reader of this message is not the intended
> recipient, you are hereby notified that any printing, copying,
> dissemination, distribution, disclosure or forwarding of this
> communication is strictly prohibited. If you have received this
> communication in error, please contact the sender immediately and
> delete it from your system. Thank You.
>
Re: Partitioning and setup errors
Posted by "M. Dale" <me...@yahoo.com>.
Two error messages seem to stand out:
1) Hadoop command-line option parsing not performed. Implement the Tool
interface and execute your application with ToolRunner to remedy this.
YourDriver extends Configured implements Tool?
Configuration conf = getConf(); //gets the configuration as parsed by
GenericOptionsParser via ToolRunner.run
Job job = Job.getInstance(conf);
2) No job jar file set. User classes may not be found. See Job or
Job#setJar(String).
job.setJarByClass(YourDriver.class); //see
http://stackoverflow.com/questions/3912267/hadoop-query-regarding-setjarbyclass-method-of-job-class
Hope this helps,
Markus
On 06/29/2014 08:26 AM, Chris Mawata wrote:
>
> It is the part about implementing the tool interface that is the issue.
> Does your driver class implement Tool?
> Chris
>
> On Jun 29, 2014 8:22 AM, "Chris MacKenzie"
> <studio@chrismackenziephotography.co.uk
> <ma...@chrismackenziephotography.co.uk>> wrote:
>
> HI Vinod,
>
> Thanks for your support. I’m packaging my application in Eclipse
> (Kepler) as a runnable jar. I can get the application to debug
> there in local mode.
>
> The command line I am using for my app is:
>
> "/usr/local/hadoop-2.4.0/jar_files $ hadoop jar
> /usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar
> a.txt 10 10”
>
> the jar permissions are -"rwxr-xr-x"
>
> The three parameters on the end of the command line are arbitrary
> commands from my sequential program which I will use as input when
> I can get this to run.
>
> I have downloaded the source and run a 2.4 version of WordCount,
> it’s not getting past the map stage either.
>
> I’m pasting in the whole job output.
>
> Deleted demo_output
>
> /usr/local/hadoop-2.4.0/jar_files $ hadoop jar
> /usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar
> a.txt 10 10
>
> Start time = 1404044126713
>
> SLF4J: Class path contains multiple SLF4J bindings.
>
> SLF4J: Found binding in [rsrc:org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:rsrc:slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for
> an explanation.
>
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
> 14/06/29 13:16:06 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java
> classes where applicable
>
> 14/06/29 13:16:31 INFO client.RMProxy: Connecting to
> ResourceManager at /0.0.0.0:8032 <http://0.0.0.0:8032>
>
> 14/06/29 13:16:46 WARN mapreduce.JobSubmitter: Hadoop command-line
> option parsing not performed. Implement the Tool interface and
> execute your application with ToolRunner to remedy this.
>
> 14/06/29 13:16:46 WARN mapreduce.JobSubmitter: No job jar file
> set. User classes may not be found. See Job or Job#setJar(String).
>
> 14/06/29 13:16:46 INFO input.FileInputFormat: Total input paths to
> process : 0
>
> 14/06/29 13:17:10 INFO mapreduce.JobSubmitter: number of splits:0
>
> 14/06/29 13:17:13 INFO mapreduce.JobSubmitter: Submitting tokens
> for job: job_1403953980618_0009
>
> 14/06/29 13:17:13 INFO mapred.YARNRunner: Job jar is not present.
> Not adding any jar to the list of resources.
>
> 14/06/29 13:17:13 INFO impl.YarnClientImpl: Submitted application
> application_1403953980618_0009
>
> 14/06/29 13:17:13 INFO mapreduce.Job: The url to track the job:
> http://admins-MacBook-Pro.local:8088/proxy/application_1403953980618_0009/
>
> 14/06/29 13:17:13 INFO mapreduce.Job: Running job:
> job_1403953980618_0009
>
> 14/06/29 13:17:19 INFO mapreduce.Job: Job job_1403953980618_0009
> running in uber mode : false
>
> 14/06/29 13:17:19 INFO mapreduce.Job: map 0% reduce 0%
>
> 14/06/29 13:17:23 INFO mapreduce.Job: Task Id :
> attempt_1403953980618_0009_r_000000_0, Status : FAILED
>
> Error: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
> at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
> Caused by: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
> ... 8 more
>
>
> 14/06/29 13:17:28 INFO mapreduce.Job: Task Id :
> attempt_1403953980618_0009_r_000000_1, Status : FAILED
>
> Error: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
> at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
> Caused by: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
> ... 8 more
>
>
> 14/06/29 13:17:32 INFO mapreduce.Job: Task Id :
> attempt_1403953980618_0009_r_000000_2, Status : FAILED
>
> Error: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
> at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
> Caused by: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
> ... 8 more
>
>
> 14/06/29 13:17:37 INFO mapreduce.Job: map 0% reduce 100%
>
> 14/06/29 13:17:37 INFO mapreduce.Job: Job job_1403953980618_0009
> failed with state FAILED due to: Task failed
> task_1403953980618_0009_r_000000
>
> Job failed as tasks failed. failedMaps:0 failedReduces:1
>
>
> 14/06/29 13:18:04 INFO mapreduce.Job: Counters: 7
>
> Job Counters
>
> Failed reduce tasks=4
>
> Launched reduce tasks=4
>
> Total time spent by all maps in occupied slots (ms)=0
>
> Total time spent by all reduces in occupied slots (ms)=7786
>
> Total time spent by all reduce tasks (ms)=7786
>
> Total vcore-seconds taken by all reduce tasks=7786
>
> Total megabyte-seconds taken by all reduce tasks=7972864
>
> 14/06/29 13:18:05 INFO ipc.Client: Retrying connect to server:
> admins-MacBook-Pro.local/192.168.0.5:53193
> <http://192.168.0.5:53193>. Already tried 0 time(s); retry policy
> is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
> sleepTime=1000 MILLISECONDS)
>
> 14/06/29 13:18:06 INFO ipc.Client: Retrying connect to server:
> admins-MacBook-Pro.local/192.168.0.5:53193
> <http://192.168.0.5:53193>. Already tried 1 time(s); retry policy
> is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
> sleepTime=1000 MILLISECONDS)
>
> 14/06/29 13:18:07 INFO ipc.Client: Retrying connect to server:
> admins-MacBook-Pro.local/192.168.0.5:53193
> <http://192.168.0.5:53193>. Already tried 2 time(s); retry policy
> is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
> sleepTime=1000 MILLISECONDS)
>
> 14/06/29 13:18:07 INFO mapred.ClientServiceDelegate: Application
> state is completed. FinalApplicationStatus=FAILED. Redirecting to
> job history server
>
> End time = 1404044288031
>
> Elapsed time = 161318 ms
>
> Finished
>
> /usr/local/hadoop-2.4.0/jar_files $
>
> Warm regards.
>
> Chris
>
>
> From: Vinod Kumar Vavilapalli <vinodkv@hortonworks.com
> <ma...@hortonworks.com>>
> Reply-To: <user@hadoop.apache.org <ma...@hadoop.apache.org>>
> Date: Sunday, 29 June 2014 00:20
> To: <user@hadoop.apache.org <ma...@hadoop.apache.org>>
> Subject: Re: Partitioning and setup errors
>
> What is happening is the client is not able to pick up the right
> jar to push to the cluster. It looks in the class-path for the jar
> that contains the class ParallelGeneticAlignment.
>
> How are you packaging your code? How are your running your job -
> paste the command line?
>
> +Vinod
>
> On Jun 27, 2014, at 5:15 AM, Chris MacKenzie
> <studio@chrismackenziephotography.co.uk
> <ma...@chrismackenziephotography.co.uk>> wrote:
>
>> Hi,
>>
>> I realise my previous question may have been a bit naïve and I
>> also realise I am asking an awful lot here, any advice would be
>> greatly appreciated.
>>
>> * I have been using Hadoop 2.4 in local mode and am sticking to
>> the mapreduce.* side of the track.
>> * I am using a Custom Line reader to read each sequence into a Map
>> * I have a partitioner class which is testing the key from the
>> map class.
>> * I've tried debugging in eclipse with a breakpoint in the
>> partitioner class but getPartition(LongWritable mapKey, Text
>> sequenceString, intnumReduceTasks) is not being called.
>>
>> Could there be any reason for that ?
>>
>> Because my map and reduce code works in local mode within
>> eclipse, I wondered if I may get the partitioner to work if
>> I changed to Pseudo Distributed Mode exporting a runnable jar
>> from Eclipse (Kepler)
>>
>> I have several faults On my own computer Pseudo Distributed Mode
>> and the university clusters Pseudo Distributed Mode which I set
>> up. I’ve googled and read extensively but am not seeing a
>> solution to any of these issues.
>>
>> I have this line:
>> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file
>> set. User classes may not be found. See Job or Job#setJar(String).
>> My driver code is:
>> private void doParallelConcordance() throws Exception {
>>
>>
>> Path inDir= newPath("input_sequences/10_sequences.txt");
>> Path outDir = new Path("demo_output");
>>
>> Job job = Job.getInstance(new Configuration());
>> job.setJarByClass(ParallelGeneticAlignment.class);
>>
>> job.setOutputKeyClass(Text.class);
>> job.setOutputValueClass(IntWritable.class);
>>
>> job.setInputFormatClass(CustomFileInputFormat.class);
>> job.setMapperClass(ConcordanceMapper.class);
>> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
>> job.setReducerClass(ConcordanceReducer.class);
>>
>> FileInputFormat.addInputPath(job, inDir);
>> FileOutputFormat.setOutputPath(job, outDir);
>>
>> job.waitForCompletion(true)
>> }
>>
>> On the university server I am getting this error:
>> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
>> attempt_1403860966764_0003_m_000000_0, Status : FAILED
>> Error: java.lang.RuntimeException:
>> java.lang.ClassNotFoundException: Class
>> par.gene.align.concordance.ConcordanceMapper not found
>>
>> On my machine the error is:
>> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
>> attempt_1403864060032_0004_r_000000_2, Status : FAILED
>> Error: java.lang.RuntimeException:
>> java.lang.ClassNotFoundException: Class
>> par.gene.align.concordance.ConcordanceReducer not found
>>
>> On the university server I get total paths to process:
>> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths
>> to process : 1
>> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>>
>> On my machine I get total paths to process:
>> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths
>> to process : 0
>> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>>
>> Being new to this community, I thought it polite to introduce
>> myself. I’m planning to return to software development via an MSc
>> at Heriot Watt University in Edinburgh. My MSc project is based
>> on Fosters Genetic Sequence Alignment. I have written a
>> sequential version my goal is now to port it to Hadoop.
>>
>> Thanks in advance,
>> Regards,
>>
>> Chris MacKenzie
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or
> entity to which it is addressed and may contain information that
> is confidential, privileged and exempt from disclosure under
> applicable law. If the reader of this message is not the intended
> recipient, you are hereby notified that any printing, copying,
> dissemination, distribution, disclosure or forwarding of this
> communication is strictly prohibited. If you have received this
> communication in error, please contact the sender immediately and
> delete it from your system. Thank You.
>
Re: Partitioning and setup errors
Posted by "M. Dale" <me...@yahoo.com>.
Two error messages seem to stand out:
1) Hadoop command-line option parsing not performed. Implement the Tool
interface and execute your application with ToolRunner to remedy this.
YourDriver extends Configured implements Tool?
Configuration conf = getConf(); //gets the configuration as parsed by
GenericOptionsParser via ToolRunner.run
Job job = Job.getInstance(conf);
2) No job jar file set. User classes may not be found. See Job or
Job#setJar(String).
job.setJarByClass(YourDriver.class); //see
http://stackoverflow.com/questions/3912267/hadoop-query-regarding-setjarbyclass-method-of-job-class
Hope this helps,
Markus
On 06/29/2014 08:26 AM, Chris Mawata wrote:
>
> It is the part about implementing the tool interface that is the issue.
> Does your driver class implement Tool?
> Chris
>
> On Jun 29, 2014 8:22 AM, "Chris MacKenzie"
> <studio@chrismackenziephotography.co.uk
> <ma...@chrismackenziephotography.co.uk>> wrote:
>
> HI Vinod,
>
> Thanks for your support. I’m packaging my application in Eclipse
> (Kepler) as a runnable jar. I can get the application to debug
> there in local mode.
>
> The command line I am using for my app is:
>
> "/usr/local/hadoop-2.4.0/jar_files $ hadoop jar
> /usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar
> a.txt 10 10”
>
> the jar permissions are -"rwxr-xr-x"
>
> The three parameters on the end of the command line are arbitrary
> commands from my sequential program which I will use as input when
> I can get this to run.
>
> I have downloaded the source and run a 2.4 version of WordCount,
> it’s not getting past the map stage either.
>
> I’m pasting in the whole job output.
>
> Deleted demo_output
>
> /usr/local/hadoop-2.4.0/jar_files $ hadoop jar
> /usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar
> a.txt 10 10
>
> Start time = 1404044126713
>
> SLF4J: Class path contains multiple SLF4J bindings.
>
> SLF4J: Found binding in [rsrc:org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:rsrc:slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for
> an explanation.
>
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
> 14/06/29 13:16:06 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java
> classes where applicable
>
> 14/06/29 13:16:31 INFO client.RMProxy: Connecting to
> ResourceManager at /0.0.0.0:8032 <http://0.0.0.0:8032>
>
> 14/06/29 13:16:46 WARN mapreduce.JobSubmitter: Hadoop command-line
> option parsing not performed. Implement the Tool interface and
> execute your application with ToolRunner to remedy this.
>
> 14/06/29 13:16:46 WARN mapreduce.JobSubmitter: No job jar file
> set. User classes may not be found. See Job or Job#setJar(String).
>
> 14/06/29 13:16:46 INFO input.FileInputFormat: Total input paths to
> process : 0
>
> 14/06/29 13:17:10 INFO mapreduce.JobSubmitter: number of splits:0
>
> 14/06/29 13:17:13 INFO mapreduce.JobSubmitter: Submitting tokens
> for job: job_1403953980618_0009
>
> 14/06/29 13:17:13 INFO mapred.YARNRunner: Job jar is not present.
> Not adding any jar to the list of resources.
>
> 14/06/29 13:17:13 INFO impl.YarnClientImpl: Submitted application
> application_1403953980618_0009
>
> 14/06/29 13:17:13 INFO mapreduce.Job: The url to track the job:
> http://admins-MacBook-Pro.local:8088/proxy/application_1403953980618_0009/
>
> 14/06/29 13:17:13 INFO mapreduce.Job: Running job:
> job_1403953980618_0009
>
> 14/06/29 13:17:19 INFO mapreduce.Job: Job job_1403953980618_0009
> running in uber mode : false
>
> 14/06/29 13:17:19 INFO mapreduce.Job: map 0% reduce 0%
>
> 14/06/29 13:17:23 INFO mapreduce.Job: Task Id :
> attempt_1403953980618_0009_r_000000_0, Status : FAILED
>
> Error: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
> at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
> Caused by: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
> ... 8 more
>
>
> 14/06/29 13:17:28 INFO mapreduce.Job: Task Id :
> attempt_1403953980618_0009_r_000000_1, Status : FAILED
>
> Error: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
> at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
> Caused by: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
> ... 8 more
>
>
> 14/06/29 13:17:32 INFO mapreduce.Job: Task Id :
> attempt_1403953980618_0009_r_000000_2, Status : FAILED
>
> Error: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
> at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
> Caused by: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
> ... 8 more
>
>
> 14/06/29 13:17:37 INFO mapreduce.Job: map 0% reduce 100%
>
> 14/06/29 13:17:37 INFO mapreduce.Job: Job job_1403953980618_0009
> failed with state FAILED due to: Task failed
> task_1403953980618_0009_r_000000
>
> Job failed as tasks failed. failedMaps:0 failedReduces:1
>
>
> 14/06/29 13:18:04 INFO mapreduce.Job: Counters: 7
>
> Job Counters
>
> Failed reduce tasks=4
>
> Launched reduce tasks=4
>
> Total time spent by all maps in occupied slots (ms)=0
>
> Total time spent by all reduces in occupied slots (ms)=7786
>
> Total time spent by all reduce tasks (ms)=7786
>
> Total vcore-seconds taken by all reduce tasks=7786
>
> Total megabyte-seconds taken by all reduce tasks=7972864
>
> 14/06/29 13:18:05 INFO ipc.Client: Retrying connect to server:
> admins-MacBook-Pro.local/192.168.0.5:53193
> <http://192.168.0.5:53193>. Already tried 0 time(s); retry policy
> is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
> sleepTime=1000 MILLISECONDS)
>
> 14/06/29 13:18:06 INFO ipc.Client: Retrying connect to server:
> admins-MacBook-Pro.local/192.168.0.5:53193
> <http://192.168.0.5:53193>. Already tried 1 time(s); retry policy
> is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
> sleepTime=1000 MILLISECONDS)
>
> 14/06/29 13:18:07 INFO ipc.Client: Retrying connect to server:
> admins-MacBook-Pro.local/192.168.0.5:53193
> <http://192.168.0.5:53193>. Already tried 2 time(s); retry policy
> is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
> sleepTime=1000 MILLISECONDS)
>
> 14/06/29 13:18:07 INFO mapred.ClientServiceDelegate: Application
> state is completed. FinalApplicationStatus=FAILED. Redirecting to
> job history server
>
> End time = 1404044288031
>
> Elapsed time = 161318 ms
>
> Finished
>
> /usr/local/hadoop-2.4.0/jar_files $
>
> Warm regards.
>
> Chris
>
>
> From: Vinod Kumar Vavilapalli <vinodkv@hortonworks.com
> <ma...@hortonworks.com>>
> Reply-To: <user@hadoop.apache.org <ma...@hadoop.apache.org>>
> Date: Sunday, 29 June 2014 00:20
> To: <user@hadoop.apache.org <ma...@hadoop.apache.org>>
> Subject: Re: Partitioning and setup errors
>
> What is happening is the client is not able to pick up the right
> jar to push to the cluster. It looks in the class-path for the jar
> that contains the class ParallelGeneticAlignment.
>
> How are you packaging your code? How are your running your job -
> paste the command line?
>
> +Vinod
>
> On Jun 27, 2014, at 5:15 AM, Chris MacKenzie
> <studio@chrismackenziephotography.co.uk
> <ma...@chrismackenziephotography.co.uk>> wrote:
>
>> Hi,
>>
>> I realise my previous question may have been a bit naïve and I
>> also realise I am asking an awful lot here, any advice would be
>> greatly appreciated.
>>
>> * I have been using Hadoop 2.4 in local mode and am sticking to
>> the mapreduce.* side of the track.
>> * I am using a Custom Line reader to read each sequence into a Map
>> * I have a partitioner class which is testing the key from the
>> map class.
>> * I've tried debugging in eclipse with a breakpoint in the
>> partitioner class but getPartition(LongWritable mapKey, Text
>> sequenceString, intnumReduceTasks) is not being called.
>>
>> Could there be any reason for that ?
>>
>> Because my map and reduce code works in local mode within
>> eclipse, I wondered if I may get the partitioner to work if
>> I changed to Pseudo Distributed Mode exporting a runnable jar
>> from Eclipse (Kepler)
>>
>> I have several faults On my own computer Pseudo Distributed Mode
>> and the university clusters Pseudo Distributed Mode which I set
>> up. I’ve googled and read extensively but am not seeing a
>> solution to any of these issues.
>>
>> I have this line:
>> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file
>> set. User classes may not be found. See Job or Job#setJar(String).
>> My driver code is:
>> private void doParallelConcordance() throws Exception {
>>
>>
>> Path inDir= newPath("input_sequences/10_sequences.txt");
>> Path outDir = new Path("demo_output");
>>
>> Job job = Job.getInstance(new Configuration());
>> job.setJarByClass(ParallelGeneticAlignment.class);
>>
>> job.setOutputKeyClass(Text.class);
>> job.setOutputValueClass(IntWritable.class);
>>
>> job.setInputFormatClass(CustomFileInputFormat.class);
>> job.setMapperClass(ConcordanceMapper.class);
>> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
>> job.setReducerClass(ConcordanceReducer.class);
>>
>> FileInputFormat.addInputPath(job, inDir);
>> FileOutputFormat.setOutputPath(job, outDir);
>>
>> job.waitForCompletion(true)
>> }
>>
>> On the university server I am getting this error:
>> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
>> attempt_1403860966764_0003_m_000000_0, Status : FAILED
>> Error: java.lang.RuntimeException:
>> java.lang.ClassNotFoundException: Class
>> par.gene.align.concordance.ConcordanceMapper not found
>>
>> On my machine the error is:
>> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
>> attempt_1403864060032_0004_r_000000_2, Status : FAILED
>> Error: java.lang.RuntimeException:
>> java.lang.ClassNotFoundException: Class
>> par.gene.align.concordance.ConcordanceReducer not found
>>
>> On the university server I get total paths to process:
>> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths
>> to process : 1
>> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>>
>> On my machine I get total paths to process:
>> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths
>> to process : 0
>> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>>
>> Being new to this community, I thought it polite to introduce
>> myself. I’m planning to return to software development via an MSc
>> at Heriot Watt University in Edinburgh. My MSc project is based
>> on Fosters Genetic Sequence Alignment. I have written a
>> sequential version my goal is now to port it to Hadoop.
>>
>> Thanks in advance,
>> Regards,
>>
>> Chris MacKenzie
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or
> entity to which it is addressed and may contain information that
> is confidential, privileged and exempt from disclosure under
> applicable law. If the reader of this message is not the intended
> recipient, you are hereby notified that any printing, copying,
> dissemination, distribution, disclosure or forwarding of this
> communication is strictly prohibited. If you have received this
> communication in error, please contact the sender immediately and
> delete it from your system. Thank You.
>
Re: Partitioning and setup errors
Posted by Chris Mawata <ch...@gmail.com>.
It is the part about implementing the tool interface that is the issue.
Does your driver class implement Tool?
Chris
On Jun 29, 2014 8:22 AM, "Chris MacKenzie" <
studio@chrismackenziephotography.co.uk> wrote:
> HI Vinod,
>
> Thanks for your support. I’m packaging my application in Eclipse (Kepler)
> as a runnable jar. I can get the application to debug there in local mode.
>
> The command line I am using for my app is:
>
> "/usr/local/hadoop-2.4.0/jar_files $ hadoop jar
> /usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar a.txt 10 10”
>
> the jar permissions are -"rwxr-xr-x"
>
> The three parameters on the end of the command line are arbitrary commands
> from my sequential program which I will use as input when I can get this to
> run.
>
> I have downloaded the source and run a 2.4 version of WordCount, it’s not
> getting past the map stage either.
>
> I’m pasting in the whole job output.
>
> Deleted demo_output
>
> /usr/local/hadoop-2.4.0/jar_files $ hadoop jar
> /usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar a.txt 10 10
>
> Start time = 1404044126713
>
> SLF4J: Class path contains multiple SLF4J bindings.
>
> SLF4J: Found binding in [rsrc:org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:rsrc:slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
>
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
> 14/06/29 13:16:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
>
> 14/06/29 13:16:31 INFO client.RMProxy: Connecting to ResourceManager at /
> 0.0.0.0:8032
>
> 14/06/29 13:16:46 WARN mapreduce.JobSubmitter: Hadoop command-line option
> parsing not performed. Implement the Tool interface and execute your
> application with ToolRunner to remedy this.
>
> 14/06/29 13:16:46 WARN mapreduce.JobSubmitter: No job jar file set. User
> classes may not be found. See Job or Job#setJar(String).
>
> 14/06/29 13:16:46 INFO input.FileInputFormat: Total input paths to process
> : 0
>
> 14/06/29 13:17:10 INFO mapreduce.JobSubmitter: number of splits:0
>
> 14/06/29 13:17:13 INFO mapreduce.JobSubmitter: Submitting tokens for job:
> job_1403953980618_0009
>
> 14/06/29 13:17:13 INFO mapred.YARNRunner: Job jar is not present. Not
> adding any jar to the list of resources.
>
> 14/06/29 13:17:13 INFO impl.YarnClientImpl: Submitted application
> application_1403953980618_0009
>
> 14/06/29 13:17:13 INFO mapreduce.Job: The url to track the job:
> http://admins-MacBook-Pro.local:8088/proxy/application_1403953980618_0009/
>
> 14/06/29 13:17:13 INFO mapreduce.Job: Running job: job_1403953980618_0009
>
> 14/06/29 13:17:19 INFO mapreduce.Job: Job job_1403953980618_0009 running
> in uber mode : false
>
> 14/06/29 13:17:19 INFO mapreduce.Job: map 0% reduce 0%
>
> 14/06/29 13:17:23 INFO mapreduce.Job: Task Id :
> attempt_1403953980618_0009_r_000000_0, Status : FAILED
>
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
> Caused by: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
> ... 8 more
>
>
> 14/06/29 13:17:28 INFO mapreduce.Job: Task Id :
> attempt_1403953980618_0009_r_000000_1, Status : FAILED
>
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
> Caused by: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
> ... 8 more
>
>
> 14/06/29 13:17:32 INFO mapreduce.Job: Task Id :
> attempt_1403953980618_0009_r_000000_2, Status : FAILED
>
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
> Caused by: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
> ... 8 more
>
>
> 14/06/29 13:17:37 INFO mapreduce.Job: map 0% reduce 100%
>
> 14/06/29 13:17:37 INFO mapreduce.Job: Job job_1403953980618_0009 failed
> with state FAILED due to: Task failed task_1403953980618_0009_r_000000
>
> Job failed as tasks failed. failedMaps:0 failedReduces:1
>
>
> 14/06/29 13:18:04 INFO mapreduce.Job: Counters: 7
>
> Job Counters
>
> Failed reduce tasks=4
>
> Launched reduce tasks=4
>
> Total time spent by all maps in occupied slots (ms)=0
>
> Total time spent by all reduces in occupied slots (ms)=7786
>
> Total time spent by all reduce tasks (ms)=7786
>
> Total vcore-seconds taken by all reduce tasks=7786
>
> Total megabyte-seconds taken by all reduce tasks=7972864
>
> 14/06/29 13:18:05 INFO ipc.Client: Retrying connect to server:
> admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 0 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
> sleepTime=1000 MILLISECONDS)
>
> 14/06/29 13:18:06 INFO ipc.Client: Retrying connect to server:
> admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 1 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
> sleepTime=1000 MILLISECONDS)
>
> 14/06/29 13:18:07 INFO ipc.Client: Retrying connect to server:
> admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 2 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
> sleepTime=1000 MILLISECONDS)
>
> 14/06/29 13:18:07 INFO mapred.ClientServiceDelegate: Application state is
> completed. FinalApplicationStatus=FAILED. Redirecting to job history server
>
> End time = 1404044288031
>
> Elapsed time = 161318 ms
>
> Finished
>
> /usr/local/hadoop-2.4.0/jar_files $
>
> Warm regards.
>
> Chris
>
> From: Vinod Kumar Vavilapalli <vi...@hortonworks.com>
> Reply-To: <us...@hadoop.apache.org>
> Date: Sunday, 29 June 2014 00:20
> To: <us...@hadoop.apache.org>
> Subject: Re: Partitioning and setup errors
>
> What is happening is the client is not able to pick up the right jar to
> push to the cluster. It looks in the class-path for the jar that contains
> the class ParallelGeneticAlignment.
>
> How are you packaging your code? How are your running your job - paste the
> command line?
>
> +Vinod
>
> On Jun 27, 2014, at 5:15 AM, Chris MacKenzie <
> studio@chrismackenziephotography.co.uk> wrote:
>
> Hi,
>
> I realise my previous question may have been a bit naïve and I also
> realise I am asking an awful lot here, any advice would be greatly
> appreciated.
>
> - I have been using Hadoop 2.4 in local mode and am sticking to the mapreduce.*
> side of the track.
> - I am using a Custom Line reader to read each sequence into a Map
> - I have a partitioner class which is testing the key from the map
> class.
> - I've tried debugging in eclipse with a breakpoint in the partitioner
> class but getPartition(LongWritable mapKey, Text sequenceString, int
> numReduceTasks) is not being called.
>
> Could there be any reason for that ?
>
> Because my map and reduce code works in local mode within eclipse, I wondered
> if I may get the partitioner to work if I changed to Pseudo Distributed
> Mode exporting a runnable jar from Eclipse (Kepler)
>
> I have several faults On my own computer Pseudo Distributed Mode and the
> university clusters Pseudo Distributed Mode which I set up. I’ve googled
> and read extensively but am not seeing a solution to any of these issues.
>
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User
> classes may not be found. See Job or Job#setJar(String).
> My driver code is:
> private void doParallelConcordance() throws Exception {
>
>
> Path inDir = new Path("input_sequences/10_sequences.txt");
> Path outDir = new Path("demo_output");
>
> Job job = Job.getInstance(new Configuration());
> job.setJarByClass(ParallelGeneticAlignment.class);
>
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(IntWritable.class);
>
> job.setInputFormatClass(CustomFileInputFormat.class);
> job.setMapperClass(ConcordanceMapper.class);
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
> job.setReducerClass(ConcordanceReducer.class);
>
> FileInputFormat.addInputPath(job, inDir);
> FileOutputFormat.setOutputPath(job, outDir);
>
> job.waitForCompletion(true)
> }
>
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
> attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceMapper not found
>
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
> attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process
> : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process
> : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>
> Being new to this community, I thought it polite to introduce myself. I’m
> planning to return to software development via an MSc at Heriot Watt
> University in Edinburgh. My MSc project is based on Fosters Genetic
> Sequence Alignment. I have written a sequential version my goal is now to
> port it to Hadoop.
>
> Thanks in advance,
> Regards,
>
> Chris MacKenzie
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Partitioning and setup errors
Posted by Chris Mawata <ch...@gmail.com>.
It is the part about implementing the tool interface that is the issue.
Does your driver class implement Tool?
Chris
On Jun 29, 2014 8:22 AM, "Chris MacKenzie" <
studio@chrismackenziephotography.co.uk> wrote:
> HI Vinod,
>
> Thanks for your support. I’m packaging my application in Eclipse (Kepler)
> as a runnable jar. I can get the application to debug there in local mode.
>
> The command line I am using for my app is:
>
> "/usr/local/hadoop-2.4.0/jar_files $ hadoop jar
> /usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar a.txt 10 10”
>
> the jar permissions are -"rwxr-xr-x"
>
> The three parameters on the end of the command line are arbitrary commands
> from my sequential program which I will use as input when I can get this to
> run.
>
> I have downloaded the source and run a 2.4 version of WordCount, it’s not
> getting past the map stage either.
>
> I’m pasting in the whole job output.
>
> Deleted demo_output
>
> /usr/local/hadoop-2.4.0/jar_files $ hadoop jar
> /usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar a.txt 10 10
>
> Start time = 1404044126713
>
> SLF4J: Class path contains multiple SLF4J bindings.
>
> SLF4J: Found binding in [rsrc:org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:rsrc:slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
>
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
> 14/06/29 13:16:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
>
> 14/06/29 13:16:31 INFO client.RMProxy: Connecting to ResourceManager at /
> 0.0.0.0:8032
>
> 14/06/29 13:16:46 WARN mapreduce.JobSubmitter: Hadoop command-line option
> parsing not performed. Implement the Tool interface and execute your
> application with ToolRunner to remedy this.
>
> 14/06/29 13:16:46 WARN mapreduce.JobSubmitter: No job jar file set. User
> classes may not be found. See Job or Job#setJar(String).
>
> 14/06/29 13:16:46 INFO input.FileInputFormat: Total input paths to process
> : 0
>
> 14/06/29 13:17:10 INFO mapreduce.JobSubmitter: number of splits:0
>
> 14/06/29 13:17:13 INFO mapreduce.JobSubmitter: Submitting tokens for job:
> job_1403953980618_0009
>
> 14/06/29 13:17:13 INFO mapred.YARNRunner: Job jar is not present. Not
> adding any jar to the list of resources.
>
> 14/06/29 13:17:13 INFO impl.YarnClientImpl: Submitted application
> application_1403953980618_0009
>
> 14/06/29 13:17:13 INFO mapreduce.Job: The url to track the job:
> http://admins-MacBook-Pro.local:8088/proxy/application_1403953980618_0009/
>
> 14/06/29 13:17:13 INFO mapreduce.Job: Running job: job_1403953980618_0009
>
> 14/06/29 13:17:19 INFO mapreduce.Job: Job job_1403953980618_0009 running
> in uber mode : false
>
> 14/06/29 13:17:19 INFO mapreduce.Job: map 0% reduce 0%
>
> 14/06/29 13:17:23 INFO mapreduce.Job: Task Id :
> attempt_1403953980618_0009_r_000000_0, Status : FAILED
>
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
> Caused by: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
> ... 8 more
>
>
> 14/06/29 13:17:28 INFO mapreduce.Job: Task Id :
> attempt_1403953980618_0009_r_000000_1, Status : FAILED
>
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
> Caused by: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
> ... 8 more
>
>
> 14/06/29 13:17:32 INFO mapreduce.Job: Task Id :
> attempt_1403953980618_0009_r_000000_2, Status : FAILED
>
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
> Caused by: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
> ... 8 more
>
>
> 14/06/29 13:17:37 INFO mapreduce.Job: map 0% reduce 100%
>
> 14/06/29 13:17:37 INFO mapreduce.Job: Job job_1403953980618_0009 failed
> with state FAILED due to: Task failed task_1403953980618_0009_r_000000
>
> Job failed as tasks failed. failedMaps:0 failedReduces:1
>
>
> 14/06/29 13:18:04 INFO mapreduce.Job: Counters: 7
>
> Job Counters
>
> Failed reduce tasks=4
>
> Launched reduce tasks=4
>
> Total time spent by all maps in occupied slots (ms)=0
>
> Total time spent by all reduces in occupied slots (ms)=7786
>
> Total time spent by all reduce tasks (ms)=7786
>
> Total vcore-seconds taken by all reduce tasks=7786
>
> Total megabyte-seconds taken by all reduce tasks=7972864
>
> 14/06/29 13:18:05 INFO ipc.Client: Retrying connect to server:
> admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 0 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
> sleepTime=1000 MILLISECONDS)
>
> 14/06/29 13:18:06 INFO ipc.Client: Retrying connect to server:
> admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 1 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
> sleepTime=1000 MILLISECONDS)
>
> 14/06/29 13:18:07 INFO ipc.Client: Retrying connect to server:
> admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 2 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
> sleepTime=1000 MILLISECONDS)
>
> 14/06/29 13:18:07 INFO mapred.ClientServiceDelegate: Application state is
> completed. FinalApplicationStatus=FAILED. Redirecting to job history server
>
> End time = 1404044288031
>
> Elapsed time = 161318 ms
>
> Finished
>
> /usr/local/hadoop-2.4.0/jar_files $
>
> Warm regards.
>
> Chris
>
> From: Vinod Kumar Vavilapalli <vi...@hortonworks.com>
> Reply-To: <us...@hadoop.apache.org>
> Date: Sunday, 29 June 2014 00:20
> To: <us...@hadoop.apache.org>
> Subject: Re: Partitioning and setup errors
>
> What is happening is the client is not able to pick up the right jar to
> push to the cluster. It looks in the class-path for the jar that contains
> the class ParallelGeneticAlignment.
>
> How are you packaging your code? How are your running your job - paste the
> command line?
>
> +Vinod
>
> On Jun 27, 2014, at 5:15 AM, Chris MacKenzie <
> studio@chrismackenziephotography.co.uk> wrote:
>
> Hi,
>
> I realise my previous question may have been a bit naïve and I also
> realise I am asking an awful lot here, any advice would be greatly
> appreciated.
>
> - I have been using Hadoop 2.4 in local mode and am sticking to the mapreduce.*
> side of the track.
> - I am using a Custom Line reader to read each sequence into a Map
> - I have a partitioner class which is testing the key from the map
> class.
> - I've tried debugging in eclipse with a breakpoint in the partitioner
> class but getPartition(LongWritable mapKey, Text sequenceString, int
> numReduceTasks) is not being called.
>
> Could there be any reason for that ?
>
> Because my map and reduce code works in local mode within eclipse, I wondered
> if I may get the partitioner to work if I changed to Pseudo Distributed
> Mode exporting a runnable jar from Eclipse (Kepler)
>
> I have several faults On my own computer Pseudo Distributed Mode and the
> university clusters Pseudo Distributed Mode which I set up. I’ve googled
> and read extensively but am not seeing a solution to any of these issues.
>
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User
> classes may not be found. See Job or Job#setJar(String).
> My driver code is:
> private void doParallelConcordance() throws Exception {
>
>
> Path inDir = new Path("input_sequences/10_sequences.txt");
> Path outDir = new Path("demo_output");
>
> Job job = Job.getInstance(new Configuration());
> job.setJarByClass(ParallelGeneticAlignment.class);
>
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(IntWritable.class);
>
> job.setInputFormatClass(CustomFileInputFormat.class);
> job.setMapperClass(ConcordanceMapper.class);
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
> job.setReducerClass(ConcordanceReducer.class);
>
> FileInputFormat.addInputPath(job, inDir);
> FileOutputFormat.setOutputPath(job, outDir);
>
> job.waitForCompletion(true)
> }
>
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
> attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceMapper not found
>
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
> attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process
> : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process
> : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>
> Being new to this community, I thought it polite to introduce myself. I’m
> planning to return to software development via an MSc at Heriot Watt
> University in Edinburgh. My MSc project is based on Fosters Genetic
> Sequence Alignment. I have written a sequential version my goal is now to
> port it to Hadoop.
>
> Thanks in advance,
> Regards,
>
> Chris MacKenzie
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Partitioning and setup errors
Posted by Chris Mawata <ch...@gmail.com>.
It is the part about implementing the tool interface that is the issue.
Does your driver class implement Tool?
Chris
On Jun 29, 2014 8:22 AM, "Chris MacKenzie" <
studio@chrismackenziephotography.co.uk> wrote:
> HI Vinod,
>
> Thanks for your support. I’m packaging my application in Eclipse (Kepler)
> as a runnable jar. I can get the application to debug there in local mode.
>
> The command line I am using for my app is:
>
> "/usr/local/hadoop-2.4.0/jar_files $ hadoop jar
> /usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar a.txt 10 10”
>
> the jar permissions are -"rwxr-xr-x"
>
> The three parameters on the end of the command line are arbitrary commands
> from my sequential program which I will use as input when I can get this to
> run.
>
> I have downloaded the source and run a 2.4 version of WordCount, it’s not
> getting past the map stage either.
>
> I’m pasting in the whole job output.
>
> Deleted demo_output
>
> /usr/local/hadoop-2.4.0/jar_files $ hadoop jar
> /usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar a.txt 10 10
>
> Start time = 1404044126713
>
> SLF4J: Class path contains multiple SLF4J bindings.
>
> SLF4J: Found binding in [rsrc:org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:rsrc:slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
>
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
> 14/06/29 13:16:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
>
> 14/06/29 13:16:31 INFO client.RMProxy: Connecting to ResourceManager at /
> 0.0.0.0:8032
>
> 14/06/29 13:16:46 WARN mapreduce.JobSubmitter: Hadoop command-line option
> parsing not performed. Implement the Tool interface and execute your
> application with ToolRunner to remedy this.
>
> 14/06/29 13:16:46 WARN mapreduce.JobSubmitter: No job jar file set. User
> classes may not be found. See Job or Job#setJar(String).
>
> 14/06/29 13:16:46 INFO input.FileInputFormat: Total input paths to process
> : 0
>
> 14/06/29 13:17:10 INFO mapreduce.JobSubmitter: number of splits:0
>
> 14/06/29 13:17:13 INFO mapreduce.JobSubmitter: Submitting tokens for job:
> job_1403953980618_0009
>
> 14/06/29 13:17:13 INFO mapred.YARNRunner: Job jar is not present. Not
> adding any jar to the list of resources.
>
> 14/06/29 13:17:13 INFO impl.YarnClientImpl: Submitted application
> application_1403953980618_0009
>
> 14/06/29 13:17:13 INFO mapreduce.Job: The url to track the job:
> http://admins-MacBook-Pro.local:8088/proxy/application_1403953980618_0009/
>
> 14/06/29 13:17:13 INFO mapreduce.Job: Running job: job_1403953980618_0009
>
> 14/06/29 13:17:19 INFO mapreduce.Job: Job job_1403953980618_0009 running
> in uber mode : false
>
> 14/06/29 13:17:19 INFO mapreduce.Job: map 0% reduce 0%
>
> 14/06/29 13:17:23 INFO mapreduce.Job: Task Id :
> attempt_1403953980618_0009_r_000000_0, Status : FAILED
>
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
> Caused by: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
> ... 8 more
>
>
> 14/06/29 13:17:28 INFO mapreduce.Job: Task Id :
> attempt_1403953980618_0009_r_000000_1, Status : FAILED
>
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
> Caused by: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
> ... 8 more
>
>
> 14/06/29 13:17:32 INFO mapreduce.Job: Task Id :
> attempt_1403953980618_0009_r_000000_2, Status : FAILED
>
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
> Caused by: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
> ... 8 more
>
>
> 14/06/29 13:17:37 INFO mapreduce.Job: map 0% reduce 100%
>
> 14/06/29 13:17:37 INFO mapreduce.Job: Job job_1403953980618_0009 failed
> with state FAILED due to: Task failed task_1403953980618_0009_r_000000
>
> Job failed as tasks failed. failedMaps:0 failedReduces:1
>
>
> 14/06/29 13:18:04 INFO mapreduce.Job: Counters: 7
>
> Job Counters
>
> Failed reduce tasks=4
>
> Launched reduce tasks=4
>
> Total time spent by all maps in occupied slots (ms)=0
>
> Total time spent by all reduces in occupied slots (ms)=7786
>
> Total time spent by all reduce tasks (ms)=7786
>
> Total vcore-seconds taken by all reduce tasks=7786
>
> Total megabyte-seconds taken by all reduce tasks=7972864
>
> 14/06/29 13:18:05 INFO ipc.Client: Retrying connect to server:
> admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 0 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
> sleepTime=1000 MILLISECONDS)
>
> 14/06/29 13:18:06 INFO ipc.Client: Retrying connect to server:
> admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 1 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
> sleepTime=1000 MILLISECONDS)
>
> 14/06/29 13:18:07 INFO ipc.Client: Retrying connect to server:
> admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 2 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
> sleepTime=1000 MILLISECONDS)
>
> 14/06/29 13:18:07 INFO mapred.ClientServiceDelegate: Application state is
> completed. FinalApplicationStatus=FAILED. Redirecting to job history server
>
> End time = 1404044288031
>
> Elapsed time = 161318 ms
>
> Finished
>
> /usr/local/hadoop-2.4.0/jar_files $
>
> Warm regards.
>
> Chris
>
> From: Vinod Kumar Vavilapalli <vi...@hortonworks.com>
> Reply-To: <us...@hadoop.apache.org>
> Date: Sunday, 29 June 2014 00:20
> To: <us...@hadoop.apache.org>
> Subject: Re: Partitioning and setup errors
>
> What is happening is the client is not able to pick up the right jar to
> push to the cluster. It looks in the class-path for the jar that contains
> the class ParallelGeneticAlignment.
>
> How are you packaging your code? How are your running your job - paste the
> command line?
>
> +Vinod
>
> On Jun 27, 2014, at 5:15 AM, Chris MacKenzie <
> studio@chrismackenziephotography.co.uk> wrote:
>
> Hi,
>
> I realise my previous question may have been a bit naïve and I also
> realise I am asking an awful lot here, any advice would be greatly
> appreciated.
>
> - I have been using Hadoop 2.4 in local mode and am sticking to the mapreduce.*
> side of the track.
> - I am using a Custom Line reader to read each sequence into a Map
> - I have a partitioner class which is testing the key from the map
> class.
> - I've tried debugging in eclipse with a breakpoint in the partitioner
> class but getPartition(LongWritable mapKey, Text sequenceString, int
> numReduceTasks) is not being called.
>
> Could there be any reason for that ?
>
> Because my map and reduce code works in local mode within eclipse, I wondered
> if I may get the partitioner to work if I changed to Pseudo Distributed
> Mode exporting a runnable jar from Eclipse (Kepler)
>
> I have several faults On my own computer Pseudo Distributed Mode and the
> university clusters Pseudo Distributed Mode which I set up. I’ve googled
> and read extensively but am not seeing a solution to any of these issues.
>
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User
> classes may not be found. See Job or Job#setJar(String).
> My driver code is:
> private void doParallelConcordance() throws Exception {
>
>
> Path inDir = new Path("input_sequences/10_sequences.txt");
> Path outDir = new Path("demo_output");
>
> Job job = Job.getInstance(new Configuration());
> job.setJarByClass(ParallelGeneticAlignment.class);
>
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(IntWritable.class);
>
> job.setInputFormatClass(CustomFileInputFormat.class);
> job.setMapperClass(ConcordanceMapper.class);
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
> job.setReducerClass(ConcordanceReducer.class);
>
> FileInputFormat.addInputPath(job, inDir);
> FileOutputFormat.setOutputPath(job, outDir);
>
> job.waitForCompletion(true)
> }
>
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
> attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceMapper not found
>
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
> attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process
> : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process
> : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>
> Being new to this community, I thought it polite to introduce myself. I’m
> planning to return to software development via an MSc at Heriot Watt
> University in Edinburgh. My MSc project is based on Fosters Genetic
> Sequence Alignment. I have written a sequential version my goal is now to
> port it to Hadoop.
>
> Thanks in advance,
> Regards,
>
> Chris MacKenzie
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Partitioning and setup errors
Posted by Chris Mawata <ch...@gmail.com>.
It is the part about implementing the tool interface that is the issue.
Does your driver class implement Tool?
Chris
On Jun 29, 2014 8:22 AM, "Chris MacKenzie" <
studio@chrismackenziephotography.co.uk> wrote:
> HI Vinod,
>
> Thanks for your support. I’m packaging my application in Eclipse (Kepler)
> as a runnable jar. I can get the application to debug there in local mode.
>
> The command line I am using for my app is:
>
> "/usr/local/hadoop-2.4.0/jar_files $ hadoop jar
> /usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar a.txt 10 10”
>
> the jar permissions are -"rwxr-xr-x"
>
> The three parameters on the end of the command line are arbitrary commands
> from my sequential program which I will use as input when I can get this to
> run.
>
> I have downloaded the source and run a 2.4 version of WordCount, it’s not
> getting past the map stage either.
>
> I’m pasting in the whole job output.
>
> Deleted demo_output
>
> /usr/local/hadoop-2.4.0/jar_files $ hadoop jar
> /usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar a.txt 10 10
>
> Start time = 1404044126713
>
> SLF4J: Class path contains multiple SLF4J bindings.
>
> SLF4J: Found binding in [rsrc:org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:rsrc:slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
>
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
> 14/06/29 13:16:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
>
> 14/06/29 13:16:31 INFO client.RMProxy: Connecting to ResourceManager at /
> 0.0.0.0:8032
>
> 14/06/29 13:16:46 WARN mapreduce.JobSubmitter: Hadoop command-line option
> parsing not performed. Implement the Tool interface and execute your
> application with ToolRunner to remedy this.
>
> 14/06/29 13:16:46 WARN mapreduce.JobSubmitter: No job jar file set. User
> classes may not be found. See Job or Job#setJar(String).
>
> 14/06/29 13:16:46 INFO input.FileInputFormat: Total input paths to process
> : 0
>
> 14/06/29 13:17:10 INFO mapreduce.JobSubmitter: number of splits:0
>
> 14/06/29 13:17:13 INFO mapreduce.JobSubmitter: Submitting tokens for job:
> job_1403953980618_0009
>
> 14/06/29 13:17:13 INFO mapred.YARNRunner: Job jar is not present. Not
> adding any jar to the list of resources.
>
> 14/06/29 13:17:13 INFO impl.YarnClientImpl: Submitted application
> application_1403953980618_0009
>
> 14/06/29 13:17:13 INFO mapreduce.Job: The url to track the job:
> http://admins-MacBook-Pro.local:8088/proxy/application_1403953980618_0009/
>
> 14/06/29 13:17:13 INFO mapreduce.Job: Running job: job_1403953980618_0009
>
> 14/06/29 13:17:19 INFO mapreduce.Job: Job job_1403953980618_0009 running
> in uber mode : false
>
> 14/06/29 13:17:19 INFO mapreduce.Job: map 0% reduce 0%
>
> 14/06/29 13:17:23 INFO mapreduce.Job: Task Id :
> attempt_1403953980618_0009_r_000000_0, Status : FAILED
>
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
> Caused by: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
> ... 8 more
>
>
> 14/06/29 13:17:28 INFO mapreduce.Job: Task Id :
> attempt_1403953980618_0009_r_000000_1, Status : FAILED
>
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
> Caused by: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
> ... 8 more
>
>
> 14/06/29 13:17:32 INFO mapreduce.Job: Task Id :
> attempt_1403953980618_0009_r_000000_2, Status : FAILED
>
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
> Caused by: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
> ... 8 more
>
>
> 14/06/29 13:17:37 INFO mapreduce.Job: map 0% reduce 100%
>
> 14/06/29 13:17:37 INFO mapreduce.Job: Job job_1403953980618_0009 failed
> with state FAILED due to: Task failed task_1403953980618_0009_r_000000
>
> Job failed as tasks failed. failedMaps:0 failedReduces:1
>
>
> 14/06/29 13:18:04 INFO mapreduce.Job: Counters: 7
>
> Job Counters
>
> Failed reduce tasks=4
>
> Launched reduce tasks=4
>
> Total time spent by all maps in occupied slots (ms)=0
>
> Total time spent by all reduces in occupied slots (ms)=7786
>
> Total time spent by all reduce tasks (ms)=7786
>
> Total vcore-seconds taken by all reduce tasks=7786
>
> Total megabyte-seconds taken by all reduce tasks=7972864
>
> 14/06/29 13:18:05 INFO ipc.Client: Retrying connect to server:
> admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 0 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
> sleepTime=1000 MILLISECONDS)
>
> 14/06/29 13:18:06 INFO ipc.Client: Retrying connect to server:
> admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 1 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
> sleepTime=1000 MILLISECONDS)
>
> 14/06/29 13:18:07 INFO ipc.Client: Retrying connect to server:
> admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 2 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
> sleepTime=1000 MILLISECONDS)
>
> 14/06/29 13:18:07 INFO mapred.ClientServiceDelegate: Application state is
> completed. FinalApplicationStatus=FAILED. Redirecting to job history server
>
> End time = 1404044288031
>
> Elapsed time = 161318 ms
>
> Finished
>
> /usr/local/hadoop-2.4.0/jar_files $
>
> Warm regards.
>
> Chris
>
> From: Vinod Kumar Vavilapalli <vi...@hortonworks.com>
> Reply-To: <us...@hadoop.apache.org>
> Date: Sunday, 29 June 2014 00:20
> To: <us...@hadoop.apache.org>
> Subject: Re: Partitioning and setup errors
>
> What is happening is the client is not able to pick up the right jar to
> push to the cluster. It looks in the class-path for the jar that contains
> the class ParallelGeneticAlignment.
>
> How are you packaging your code? How are your running your job - paste the
> command line?
>
> +Vinod
>
> On Jun 27, 2014, at 5:15 AM, Chris MacKenzie <
> studio@chrismackenziephotography.co.uk> wrote:
>
> Hi,
>
> I realise my previous question may have been a bit naïve and I also
> realise I am asking an awful lot here, any advice would be greatly
> appreciated.
>
> - I have been using Hadoop 2.4 in local mode and am sticking to the mapreduce.*
> side of the track.
> - I am using a Custom Line reader to read each sequence into a Map
> - I have a partitioner class which is testing the key from the map
> class.
> - I've tried debugging in eclipse with a breakpoint in the partitioner
> class but getPartition(LongWritable mapKey, Text sequenceString, int
> numReduceTasks) is not being called.
>
> Could there be any reason for that ?
>
> Because my map and reduce code works in local mode within eclipse, I wondered
> if I may get the partitioner to work if I changed to Pseudo Distributed
> Mode exporting a runnable jar from Eclipse (Kepler)
>
> I have several faults On my own computer Pseudo Distributed Mode and the
> university clusters Pseudo Distributed Mode which I set up. I’ve googled
> and read extensively but am not seeing a solution to any of these issues.
>
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User
> classes may not be found. See Job or Job#setJar(String).
> My driver code is:
> private void doParallelConcordance() throws Exception {
>
>
> Path inDir = new Path("input_sequences/10_sequences.txt");
> Path outDir = new Path("demo_output");
>
> Job job = Job.getInstance(new Configuration());
> job.setJarByClass(ParallelGeneticAlignment.class);
>
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(IntWritable.class);
>
> job.setInputFormatClass(CustomFileInputFormat.class);
> job.setMapperClass(ConcordanceMapper.class);
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
> job.setReducerClass(ConcordanceReducer.class);
>
> FileInputFormat.addInputPath(job, inDir);
> FileOutputFormat.setOutputPath(job, outDir);
>
> job.waitForCompletion(true)
> }
>
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
> attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceMapper not found
>
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
> attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process
> : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process
> : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>
> Being new to this community, I thought it polite to introduce myself. I’m
> planning to return to software development via an MSc at Heriot Watt
> University in Edinburgh. My MSc project is based on Fosters Genetic
> Sequence Alignment. I have written a sequential version my goal is now to
> port it to Hadoop.
>
> Thanks in advance,
> Regards,
>
> Chris MacKenzie
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Partitioning and setup errors
Posted by Chris MacKenzie <st...@chrismackenziephotography.co.uk>.
HI Vinod,
Thanks for your support. I¹m packaging my application in Eclipse (Kepler) as
a runnable jar. I can get the application to debug there in local mode.
The command line I am using for my app is:
"/usr/local/hadoop-2.4.0/jar_files $ hadoop jar
/usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar a.txt 10 10²
the jar permissions are -"rwxr-xr-x"
The three parameters on the end of the command line are arbitrary commands
from my sequential program which I will use as input when I can get this to
run.
I have downloaded the source and run a 2.4 version of WordCount, it¹s not
getting past the map stage either.
I¹m pasting in the whole job output.
Deleted demo_output
/usr/local/hadoop-2.4.0/jar_files $ hadoop jar
/usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar a.txt 10 10
Start time = 1404044126713
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [rsrc:org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:rsrc:slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
14/06/29 13:16:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/06/29 13:16:31 INFO client.RMProxy: Connecting to ResourceManager at
/0.0.0.0:8032
14/06/29 13:16:46 WARN mapreduce.JobSubmitter: Hadoop command-line option
parsing not performed. Implement the Tool interface and execute your
application with ToolRunner to remedy this.
14/06/29 13:16:46 WARN mapreduce.JobSubmitter: No job jar file set. User
classes may not be found. See Job or Job#setJar(String).
14/06/29 13:16:46 INFO input.FileInputFormat: Total input paths to process :
0
14/06/29 13:17:10 INFO mapreduce.JobSubmitter: number of splits:0
14/06/29 13:17:13 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1403953980618_0009
14/06/29 13:17:13 INFO mapred.YARNRunner: Job jar is not present. Not adding
any jar to the list of resources.
14/06/29 13:17:13 INFO impl.YarnClientImpl: Submitted application
application_1403953980618_0009
14/06/29 13:17:13 INFO mapreduce.Job: The url to track the job:
http://admins-MacBook-Pro.local:8088/proxy/application_1403953980618_0009/
14/06/29 13:17:13 INFO mapreduce.Job: Running job: job_1403953980618_0009
14/06/29 13:17:19 INFO mapreduce.Job: Job job_1403953980618_0009 running in
uber mode : false
14/06/29 13:17:19 INFO mapreduce.Job: map 0% reduce 0%
14/06/29 13:17:23 INFO mapreduce.Job: Task Id :
attempt_1403953980618_0009_r_000000_0, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceReducer not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextIm
pl.java:210)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceReducer not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 8 more
14/06/29 13:17:28 INFO mapreduce.Job: Task Id :
attempt_1403953980618_0009_r_000000_1, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceReducer not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextIm
pl.java:210)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceReducer not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 8 more
14/06/29 13:17:32 INFO mapreduce.Job: Task Id :
attempt_1403953980618_0009_r_000000_2, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceReducer not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextIm
pl.java:210)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceReducer not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 8 more
14/06/29 13:17:37 INFO mapreduce.Job: map 0% reduce 100%
14/06/29 13:17:37 INFO mapreduce.Job: Job job_1403953980618_0009 failed with
state FAILED due to: Task failed task_1403953980618_0009_r_000000
Job failed as tasks failed. failedMaps:0 failedReduces:1
14/06/29 13:18:04 INFO mapreduce.Job: Counters: 7
Job Counters
Failed reduce tasks=4
Launched reduce tasks=4
Total time spent by all maps in occupied slots (ms)=0
Total time spent by all reduces in occupied slots (ms)=7786
Total time spent by all reduce tasks (ms)=7786
Total vcore-seconds taken by all reduce tasks=7786
Total megabyte-seconds taken by all reduce tasks=7972864
14/06/29 13:18:05 INFO ipc.Client: Retrying connect to server:
admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 0 time(s); retry
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000
MILLISECONDS)
14/06/29 13:18:06 INFO ipc.Client: Retrying connect to server:
admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 1 time(s); retry
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000
MILLISECONDS)
14/06/29 13:18:07 INFO ipc.Client: Retrying connect to server:
admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 2 time(s); retry
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000
MILLISECONDS)
14/06/29 13:18:07 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=FAILED. Redirecting to job history server
End time = 1404044288031
Elapsed time = 161318 ms
Finished
/usr/local/hadoop-2.4.0/jar_files $
Warm regards.
Chris
From: Vinod Kumar Vavilapalli <vi...@hortonworks.com>
Reply-To: <us...@hadoop.apache.org>
Date: Sunday, 29 June 2014 00:20
To: <us...@hadoop.apache.org>
Subject: Re: Partitioning and setup errors
What is happening is the client is not able to pick up the right jar to push
to the cluster. It looks in the class-path for the jar that contains the
class ParallelGeneticAlignment.
How are you packaging your code? How are your running your job - paste the
command line?
+Vinod
On Jun 27, 2014, at 5:15 AM, Chris MacKenzie
<st...@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> I realise my previous question may have been a bit naïve and I also realise I
> am asking an awful lot here, any advice would be greatly appreciated.
> * I have been using Hadoop 2.4 in local mode and am sticking to the
> mapreduce.* side of the track.
> * I am using a Custom Line reader to read each sequence into a Map
> * I have a partitioner class which is testing the key from the map class.
> * I've tried debugging in eclipse with a breakpoint in the partitioner class
> but getPartition(LongWritable mapKey, Text sequenceString, int numReduceTasks)
> is not being called.
> Could there be any reason for that ?
>
> Because my map and reduce code works in local mode within eclipse, I wondered
> if I may get the partitioner to work if I changed to Pseudo Distributed Mode
> exporting a runnable jar from Eclipse (Kepler)
>
> I have several faults On my own computer Pseudo Distributed Mode and the
> university clusters Pseudo Distributed Mode which I set up. I¹ve googled and
> read extensively but am not seeing a solution to any of these issues.
>
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User
> classes may not be found. See Job or Job#setJar(String).
> My driver code is:
> private void doParallelConcordance() throws Exception {
>
> Path inDir = new Path("input_sequences/10_sequences.txt");
> Path outDir = new Path("demo_output");
>
> Job job = Job.getInstance(new Configuration());
> job.setJarByClass(ParallelGeneticAlignment.class);
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(IntWritable.class);
>
> job.setInputFormatClass(CustomFileInputFormat.class);
> job.setMapperClass(ConcordanceMapper.class);
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
> job.setReducerClass(ConcordanceReducer.class);
>
> FileInputFormat.addInputPath(job, inDir);
> FileOutputFormat.setOutputPath(job, outDir);
>
>
> job.waitForCompletion(true)
> }
>
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
> attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceMapper not found
>
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
> attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>
> Being new to this community, I thought it polite to introduce myself. I¹m
> planning to return to software development via an MSc at Heriot Watt
> University in Edinburgh. My MSc project is based on Fosters Genetic Sequence
> Alignment. I have written a sequential version my goal is now to port it to
> Hadoop.
>
> Thanks in advance,
> Regards,
>
> Chris MacKenzie
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader of
this message is not the intended recipient, you are hereby notified that any
printing, copying, dissemination, distribution, disclosure or forwarding of
this communication is strictly prohibited. If you have received this
communication in error, please contact the sender immediately and delete it
from your system. Thank You.
Re: Partitioning and setup errors
Posted by Chris MacKenzie <st...@chrismackenziephotography.co.uk>.
HI Vinod,
Thanks for your support. I¹m packaging my application in Eclipse (Kepler) as
a runnable jar. I can get the application to debug there in local mode.
The command line I am using for my app is:
"/usr/local/hadoop-2.4.0/jar_files $ hadoop jar
/usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar a.txt 10 10²
the jar permissions are -"rwxr-xr-x"
The three parameters on the end of the command line are arbitrary commands
from my sequential program which I will use as input when I can get this to
run.
I have downloaded the source and run a 2.4 version of WordCount, it¹s not
getting past the map stage either.
I¹m pasting in the whole job output.
Deleted demo_output
/usr/local/hadoop-2.4.0/jar_files $ hadoop jar
/usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar a.txt 10 10
Start time = 1404044126713
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [rsrc:org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:rsrc:slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
14/06/29 13:16:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/06/29 13:16:31 INFO client.RMProxy: Connecting to ResourceManager at
/0.0.0.0:8032
14/06/29 13:16:46 WARN mapreduce.JobSubmitter: Hadoop command-line option
parsing not performed. Implement the Tool interface and execute your
application with ToolRunner to remedy this.
14/06/29 13:16:46 WARN mapreduce.JobSubmitter: No job jar file set. User
classes may not be found. See Job or Job#setJar(String).
14/06/29 13:16:46 INFO input.FileInputFormat: Total input paths to process :
0
14/06/29 13:17:10 INFO mapreduce.JobSubmitter: number of splits:0
14/06/29 13:17:13 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1403953980618_0009
14/06/29 13:17:13 INFO mapred.YARNRunner: Job jar is not present. Not adding
any jar to the list of resources.
14/06/29 13:17:13 INFO impl.YarnClientImpl: Submitted application
application_1403953980618_0009
14/06/29 13:17:13 INFO mapreduce.Job: The url to track the job:
http://admins-MacBook-Pro.local:8088/proxy/application_1403953980618_0009/
14/06/29 13:17:13 INFO mapreduce.Job: Running job: job_1403953980618_0009
14/06/29 13:17:19 INFO mapreduce.Job: Job job_1403953980618_0009 running in
uber mode : false
14/06/29 13:17:19 INFO mapreduce.Job: map 0% reduce 0%
14/06/29 13:17:23 INFO mapreduce.Job: Task Id :
attempt_1403953980618_0009_r_000000_0, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceReducer not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextIm
pl.java:210)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceReducer not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 8 more
14/06/29 13:17:28 INFO mapreduce.Job: Task Id :
attempt_1403953980618_0009_r_000000_1, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceReducer not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextIm
pl.java:210)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceReducer not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 8 more
14/06/29 13:17:32 INFO mapreduce.Job: Task Id :
attempt_1403953980618_0009_r_000000_2, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceReducer not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextIm
pl.java:210)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceReducer not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 8 more
14/06/29 13:17:37 INFO mapreduce.Job: map 0% reduce 100%
14/06/29 13:17:37 INFO mapreduce.Job: Job job_1403953980618_0009 failed with
state FAILED due to: Task failed task_1403953980618_0009_r_000000
Job failed as tasks failed. failedMaps:0 failedReduces:1
14/06/29 13:18:04 INFO mapreduce.Job: Counters: 7
Job Counters
Failed reduce tasks=4
Launched reduce tasks=4
Total time spent by all maps in occupied slots (ms)=0
Total time spent by all reduces in occupied slots (ms)=7786
Total time spent by all reduce tasks (ms)=7786
Total vcore-seconds taken by all reduce tasks=7786
Total megabyte-seconds taken by all reduce tasks=7972864
14/06/29 13:18:05 INFO ipc.Client: Retrying connect to server:
admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 0 time(s); retry
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000
MILLISECONDS)
14/06/29 13:18:06 INFO ipc.Client: Retrying connect to server:
admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 1 time(s); retry
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000
MILLISECONDS)
14/06/29 13:18:07 INFO ipc.Client: Retrying connect to server:
admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 2 time(s); retry
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000
MILLISECONDS)
14/06/29 13:18:07 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=FAILED. Redirecting to job history server
End time = 1404044288031
Elapsed time = 161318 ms
Finished
/usr/local/hadoop-2.4.0/jar_files $
Warm regards.
Chris
From: Vinod Kumar Vavilapalli <vi...@hortonworks.com>
Reply-To: <us...@hadoop.apache.org>
Date: Sunday, 29 June 2014 00:20
To: <us...@hadoop.apache.org>
Subject: Re: Partitioning and setup errors
What is happening is the client is not able to pick up the right jar to push
to the cluster. It looks in the class-path for the jar that contains the
class ParallelGeneticAlignment.
How are you packaging your code? How are your running your job - paste the
command line?
+Vinod
On Jun 27, 2014, at 5:15 AM, Chris MacKenzie
<st...@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> I realise my previous question may have been a bit naïve and I also realise I
> am asking an awful lot here, any advice would be greatly appreciated.
> * I have been using Hadoop 2.4 in local mode and am sticking to the
> mapreduce.* side of the track.
> * I am using a Custom Line reader to read each sequence into a Map
> * I have a partitioner class which is testing the key from the map class.
> * I've tried debugging in eclipse with a breakpoint in the partitioner class
> but getPartition(LongWritable mapKey, Text sequenceString, int numReduceTasks)
> is not being called.
> Could there be any reason for that ?
>
> Because my map and reduce code works in local mode within eclipse, I wondered
> if I may get the partitioner to work if I changed to Pseudo Distributed Mode
> exporting a runnable jar from Eclipse (Kepler)
>
> I have several faults On my own computer Pseudo Distributed Mode and the
> university clusters Pseudo Distributed Mode which I set up. I¹ve googled and
> read extensively but am not seeing a solution to any of these issues.
>
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User
> classes may not be found. See Job or Job#setJar(String).
> My driver code is:
> private void doParallelConcordance() throws Exception {
>
> Path inDir = new Path("input_sequences/10_sequences.txt");
> Path outDir = new Path("demo_output");
>
> Job job = Job.getInstance(new Configuration());
> job.setJarByClass(ParallelGeneticAlignment.class);
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(IntWritable.class);
>
> job.setInputFormatClass(CustomFileInputFormat.class);
> job.setMapperClass(ConcordanceMapper.class);
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
> job.setReducerClass(ConcordanceReducer.class);
>
> FileInputFormat.addInputPath(job, inDir);
> FileOutputFormat.setOutputPath(job, outDir);
>
>
> job.waitForCompletion(true)
> }
>
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
> attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceMapper not found
>
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
> attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>
> Being new to this community, I thought it polite to introduce myself. I¹m
> planning to return to software development via an MSc at Heriot Watt
> University in Edinburgh. My MSc project is based on Fosters Genetic Sequence
> Alignment. I have written a sequential version my goal is now to port it to
> Hadoop.
>
> Thanks in advance,
> Regards,
>
> Chris MacKenzie
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader of
this message is not the intended recipient, you are hereby notified that any
printing, copying, dissemination, distribution, disclosure or forwarding of
this communication is strictly prohibited. If you have received this
communication in error, please contact the sender immediately and delete it
from your system. Thank You.
Re: Partitioning and setup errors
Posted by Chris MacKenzie <st...@chrismackenziephotography.co.uk>.
HI Vinod,
Thanks for your support. I¹m packaging my application in Eclipse (Kepler) as
a runnable jar. I can get the application to debug there in local mode.
The command line I am using for my app is:
"/usr/local/hadoop-2.4.0/jar_files $ hadoop jar
/usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar a.txt 10 10²
the jar permissions are -"rwxr-xr-x"
The three parameters on the end of the command line are arbitrary commands
from my sequential program which I will use as input when I can get this to
run.
I have downloaded the source and run a 2.4 version of WordCount, it¹s not
getting past the map stage either.
I¹m pasting in the whole job output.
Deleted demo_output
/usr/local/hadoop-2.4.0/jar_files $ hadoop jar
/usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar a.txt 10 10
Start time = 1404044126713
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [rsrc:org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:rsrc:slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
14/06/29 13:16:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/06/29 13:16:31 INFO client.RMProxy: Connecting to ResourceManager at
/0.0.0.0:8032
14/06/29 13:16:46 WARN mapreduce.JobSubmitter: Hadoop command-line option
parsing not performed. Implement the Tool interface and execute your
application with ToolRunner to remedy this.
14/06/29 13:16:46 WARN mapreduce.JobSubmitter: No job jar file set. User
classes may not be found. See Job or Job#setJar(String).
14/06/29 13:16:46 INFO input.FileInputFormat: Total input paths to process :
0
14/06/29 13:17:10 INFO mapreduce.JobSubmitter: number of splits:0
14/06/29 13:17:13 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1403953980618_0009
14/06/29 13:17:13 INFO mapred.YARNRunner: Job jar is not present. Not adding
any jar to the list of resources.
14/06/29 13:17:13 INFO impl.YarnClientImpl: Submitted application
application_1403953980618_0009
14/06/29 13:17:13 INFO mapreduce.Job: The url to track the job:
http://admins-MacBook-Pro.local:8088/proxy/application_1403953980618_0009/
14/06/29 13:17:13 INFO mapreduce.Job: Running job: job_1403953980618_0009
14/06/29 13:17:19 INFO mapreduce.Job: Job job_1403953980618_0009 running in
uber mode : false
14/06/29 13:17:19 INFO mapreduce.Job: map 0% reduce 0%
14/06/29 13:17:23 INFO mapreduce.Job: Task Id :
attempt_1403953980618_0009_r_000000_0, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceReducer not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextIm
pl.java:210)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceReducer not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 8 more
14/06/29 13:17:28 INFO mapreduce.Job: Task Id :
attempt_1403953980618_0009_r_000000_1, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceReducer not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextIm
pl.java:210)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceReducer not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 8 more
14/06/29 13:17:32 INFO mapreduce.Job: Task Id :
attempt_1403953980618_0009_r_000000_2, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceReducer not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextIm
pl.java:210)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceReducer not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 8 more
14/06/29 13:17:37 INFO mapreduce.Job: map 0% reduce 100%
14/06/29 13:17:37 INFO mapreduce.Job: Job job_1403953980618_0009 failed with
state FAILED due to: Task failed task_1403953980618_0009_r_000000
Job failed as tasks failed. failedMaps:0 failedReduces:1
14/06/29 13:18:04 INFO mapreduce.Job: Counters: 7
Job Counters
Failed reduce tasks=4
Launched reduce tasks=4
Total time spent by all maps in occupied slots (ms)=0
Total time spent by all reduces in occupied slots (ms)=7786
Total time spent by all reduce tasks (ms)=7786
Total vcore-seconds taken by all reduce tasks=7786
Total megabyte-seconds taken by all reduce tasks=7972864
14/06/29 13:18:05 INFO ipc.Client: Retrying connect to server:
admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 0 time(s); retry
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000
MILLISECONDS)
14/06/29 13:18:06 INFO ipc.Client: Retrying connect to server:
admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 1 time(s); retry
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000
MILLISECONDS)
14/06/29 13:18:07 INFO ipc.Client: Retrying connect to server:
admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 2 time(s); retry
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000
MILLISECONDS)
14/06/29 13:18:07 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=FAILED. Redirecting to job history server
End time = 1404044288031
Elapsed time = 161318 ms
Finished
/usr/local/hadoop-2.4.0/jar_files $
Warm regards.
Chris
From: Vinod Kumar Vavilapalli <vi...@hortonworks.com>
Reply-To: <us...@hadoop.apache.org>
Date: Sunday, 29 June 2014 00:20
To: <us...@hadoop.apache.org>
Subject: Re: Partitioning and setup errors
What is happening is the client is not able to pick up the right jar to push
to the cluster. It looks in the class-path for the jar that contains the
class ParallelGeneticAlignment.
How are you packaging your code? How are your running your job - paste the
command line?
+Vinod
On Jun 27, 2014, at 5:15 AM, Chris MacKenzie
<st...@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> I realise my previous question may have been a bit naïve and I also realise I
> am asking an awful lot here, any advice would be greatly appreciated.
> * I have been using Hadoop 2.4 in local mode and am sticking to the
> mapreduce.* side of the track.
> * I am using a Custom Line reader to read each sequence into a Map
> * I have a partitioner class which is testing the key from the map class.
> * I've tried debugging in eclipse with a breakpoint in the partitioner class
> but getPartition(LongWritable mapKey, Text sequenceString, int numReduceTasks)
> is not being called.
> Could there be any reason for that ?
>
> Because my map and reduce code works in local mode within eclipse, I wondered
> if I may get the partitioner to work if I changed to Pseudo Distributed Mode
> exporting a runnable jar from Eclipse (Kepler)
>
> I have several faults On my own computer Pseudo Distributed Mode and the
> university clusters Pseudo Distributed Mode which I set up. I¹ve googled and
> read extensively but am not seeing a solution to any of these issues.
>
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User
> classes may not be found. See Job or Job#setJar(String).
> My driver code is:
> private void doParallelConcordance() throws Exception {
>
> Path inDir = new Path("input_sequences/10_sequences.txt");
> Path outDir = new Path("demo_output");
>
> Job job = Job.getInstance(new Configuration());
> job.setJarByClass(ParallelGeneticAlignment.class);
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(IntWritable.class);
>
> job.setInputFormatClass(CustomFileInputFormat.class);
> job.setMapperClass(ConcordanceMapper.class);
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
> job.setReducerClass(ConcordanceReducer.class);
>
> FileInputFormat.addInputPath(job, inDir);
> FileOutputFormat.setOutputPath(job, outDir);
>
>
> job.waitForCompletion(true)
> }
>
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
> attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceMapper not found
>
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
> attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>
> Being new to this community, I thought it polite to introduce myself. I¹m
> planning to return to software development via an MSc at Heriot Watt
> University in Edinburgh. My MSc project is based on Fosters Genetic Sequence
> Alignment. I have written a sequential version my goal is now to port it to
> Hadoop.
>
> Thanks in advance,
> Regards,
>
> Chris MacKenzie
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader of
this message is not the intended recipient, you are hereby notified that any
printing, copying, dissemination, distribution, disclosure or forwarding of
this communication is strictly prohibited. If you have received this
communication in error, please contact the sender immediately and delete it
from your system. Thank You.
Re: Partitioning and setup errors
Posted by Chris MacKenzie <st...@chrismackenziephotography.co.uk>.
HI Vinod,
Thanks for your support. I¹m packaging my application in Eclipse (Kepler) as
a runnable jar. I can get the application to debug there in local mode.
The command line I am using for my app is:
"/usr/local/hadoop-2.4.0/jar_files $ hadoop jar
/usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar a.txt 10 10²
the jar permissions are -"rwxr-xr-x"
The three parameters on the end of the command line are arbitrary commands
from my sequential program which I will use as input when I can get this to
run.
I have downloaded the source and run a 2.4 version of WordCount, it¹s not
getting past the map stage either.
I¹m pasting in the whole job output.
Deleted demo_output
/usr/local/hadoop-2.4.0/jar_files $ hadoop jar
/usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar a.txt 10 10
Start time = 1404044126713
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [rsrc:org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:rsrc:slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
14/06/29 13:16:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/06/29 13:16:31 INFO client.RMProxy: Connecting to ResourceManager at
/0.0.0.0:8032
14/06/29 13:16:46 WARN mapreduce.JobSubmitter: Hadoop command-line option
parsing not performed. Implement the Tool interface and execute your
application with ToolRunner to remedy this.
14/06/29 13:16:46 WARN mapreduce.JobSubmitter: No job jar file set. User
classes may not be found. See Job or Job#setJar(String).
14/06/29 13:16:46 INFO input.FileInputFormat: Total input paths to process :
0
14/06/29 13:17:10 INFO mapreduce.JobSubmitter: number of splits:0
14/06/29 13:17:13 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1403953980618_0009
14/06/29 13:17:13 INFO mapred.YARNRunner: Job jar is not present. Not adding
any jar to the list of resources.
14/06/29 13:17:13 INFO impl.YarnClientImpl: Submitted application
application_1403953980618_0009
14/06/29 13:17:13 INFO mapreduce.Job: The url to track the job:
http://admins-MacBook-Pro.local:8088/proxy/application_1403953980618_0009/
14/06/29 13:17:13 INFO mapreduce.Job: Running job: job_1403953980618_0009
14/06/29 13:17:19 INFO mapreduce.Job: Job job_1403953980618_0009 running in
uber mode : false
14/06/29 13:17:19 INFO mapreduce.Job: map 0% reduce 0%
14/06/29 13:17:23 INFO mapreduce.Job: Task Id :
attempt_1403953980618_0009_r_000000_0, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceReducer not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextIm
pl.java:210)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceReducer not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 8 more
14/06/29 13:17:28 INFO mapreduce.Job: Task Id :
attempt_1403953980618_0009_r_000000_1, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceReducer not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextIm
pl.java:210)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceReducer not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 8 more
14/06/29 13:17:32 INFO mapreduce.Job: Task Id :
attempt_1403953980618_0009_r_000000_2, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceReducer not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextIm
pl.java:210)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceReducer not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 8 more
14/06/29 13:17:37 INFO mapreduce.Job: map 0% reduce 100%
14/06/29 13:17:37 INFO mapreduce.Job: Job job_1403953980618_0009 failed with
state FAILED due to: Task failed task_1403953980618_0009_r_000000
Job failed as tasks failed. failedMaps:0 failedReduces:1
14/06/29 13:18:04 INFO mapreduce.Job: Counters: 7
Job Counters
Failed reduce tasks=4
Launched reduce tasks=4
Total time spent by all maps in occupied slots (ms)=0
Total time spent by all reduces in occupied slots (ms)=7786
Total time spent by all reduce tasks (ms)=7786
Total vcore-seconds taken by all reduce tasks=7786
Total megabyte-seconds taken by all reduce tasks=7972864
14/06/29 13:18:05 INFO ipc.Client: Retrying connect to server:
admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 0 time(s); retry
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000
MILLISECONDS)
14/06/29 13:18:06 INFO ipc.Client: Retrying connect to server:
admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 1 time(s); retry
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000
MILLISECONDS)
14/06/29 13:18:07 INFO ipc.Client: Retrying connect to server:
admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 2 time(s); retry
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000
MILLISECONDS)
14/06/29 13:18:07 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=FAILED. Redirecting to job history server
End time = 1404044288031
Elapsed time = 161318 ms
Finished
/usr/local/hadoop-2.4.0/jar_files $
Warm regards.
Chris
From: Vinod Kumar Vavilapalli <vi...@hortonworks.com>
Reply-To: <us...@hadoop.apache.org>
Date: Sunday, 29 June 2014 00:20
To: <us...@hadoop.apache.org>
Subject: Re: Partitioning and setup errors
What is happening is the client is not able to pick up the right jar to push
to the cluster. It looks in the class-path for the jar that contains the
class ParallelGeneticAlignment.
How are you packaging your code? How are your running your job - paste the
command line?
+Vinod
On Jun 27, 2014, at 5:15 AM, Chris MacKenzie
<st...@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> I realise my previous question may have been a bit naïve and I also realise I
> am asking an awful lot here, any advice would be greatly appreciated.
> * I have been using Hadoop 2.4 in local mode and am sticking to the
> mapreduce.* side of the track.
> * I am using a Custom Line reader to read each sequence into a Map
> * I have a partitioner class which is testing the key from the map class.
> * I've tried debugging in eclipse with a breakpoint in the partitioner class
> but getPartition(LongWritable mapKey, Text sequenceString, int numReduceTasks)
> is not being called.
> Could there be any reason for that ?
>
> Because my map and reduce code works in local mode within eclipse, I wondered
> if I may get the partitioner to work if I changed to Pseudo Distributed Mode
> exporting a runnable jar from Eclipse (Kepler)
>
> I have several faults On my own computer Pseudo Distributed Mode and the
> university clusters Pseudo Distributed Mode which I set up. I¹ve googled and
> read extensively but am not seeing a solution to any of these issues.
>
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User
> classes may not be found. See Job or Job#setJar(String).
> My driver code is:
> private void doParallelConcordance() throws Exception {
>
> Path inDir = new Path("input_sequences/10_sequences.txt");
> Path outDir = new Path("demo_output");
>
> Job job = Job.getInstance(new Configuration());
> job.setJarByClass(ParallelGeneticAlignment.class);
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(IntWritable.class);
>
> job.setInputFormatClass(CustomFileInputFormat.class);
> job.setMapperClass(ConcordanceMapper.class);
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
> job.setReducerClass(ConcordanceReducer.class);
>
> FileInputFormat.addInputPath(job, inDir);
> FileOutputFormat.setOutputPath(job, outDir);
>
>
> job.waitForCompletion(true)
> }
>
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
> attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceMapper not found
>
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
> attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>
> Being new to this community, I thought it polite to introduce myself. I¹m
> planning to return to software development via an MSc at Heriot Watt
> University in Edinburgh. My MSc project is based on Fosters Genetic Sequence
> Alignment. I have written a sequential version my goal is now to port it to
> Hadoop.
>
> Thanks in advance,
> Regards,
>
> Chris MacKenzie
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader of
this message is not the intended recipient, you are hereby notified that any
printing, copying, dissemination, distribution, disclosure or forwarding of
this communication is strictly prohibited. If you have received this
communication in error, please contact the sender immediately and delete it
from your system. Thank You.
Re: Partitioning and setup errors
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
What is happening is the client is not able to pick up the right jar to push to the cluster. It looks in the class-path for the jar that contains the class ParallelGeneticAlignment.
How are you packaging your code? How are your running your job - paste the command line?
+Vinod
On Jun 27, 2014, at 5:15 AM, Chris MacKenzie <st...@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> I realise my previous question may have been a bit naïve and I also realise I am asking an awful lot here, any advice would be greatly appreciated.
> I have been using Hadoop 2.4 in local mode and am sticking to the mapreduce.* side of the track.
> I am using a Custom Line reader to read each sequence into a Map
> I have a partitioner class which is testing the key from the map class.
> I've tried debugging in eclipse with a breakpoint in the partitioner class but getPartition(LongWritable mapKey, Text sequenceString, int numReduceTasks) is not being called.
> Could there be any reason for that ?
>
> Because my map and reduce code works in local mode within eclipse, I wondered if I may get the partitioner to work if I changed to Pseudo Distributed Mode exporting a runnable jar from Eclipse (Kepler)
>
> I have several faults On my own computer Pseudo Distributed Mode and the university clusters Pseudo Distributed Mode which I set up. I’ve googled and read extensively but am not seeing a solution to any of these issues.
>
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String).
> My driver code is:
> private void doParallelConcordance() throws Exception {
>
> Path inDir = new Path("input_sequences/10_sequences.txt");
> Path outDir = new Path("demo_output");
>
> Job job = Job.getInstance(new Configuration());
> job.setJarByClass(ParallelGeneticAlignment.class);
>
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(IntWritable.class);
>
> job.setInputFormatClass(CustomFileInputFormat.class);
> job.setMapperClass(ConcordanceMapper.class);
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
> job.setReducerClass(ConcordanceReducer.class);
>
> FileInputFormat.addInputPath(job, inDir);
> FileOutputFormat.setOutputPath(job, outDir);
>
> job.waitForCompletion(true)
> }
>
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id : attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class par.gene.align.concordance.ConcordanceMapper not found
>
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id : attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class par.gene.align.concordance.ConcordanceReducer not found
>
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>
> Being new to this community, I thought it polite to introduce myself. I’m planning to return to software development via an MSc at Heriot Watt University in Edinburgh. My MSc project is based on Fosters Genetic Sequence Alignment. I have written a sequential version my goal is now to port it to Hadoop.
>
> Thanks in advance,
> Regards,
>
> Chris MacKenzie
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Partitioning and setup errors
Posted by Chris MacKenzie <st...@chrismackenziephotography.co.uk>.
HI Chris,
Thanks for your response. I deeply appreciate it.
I don¹t know what you mean by that question. I use configuration:
* In the driver Job job = Job.getInstance(new Configuration());
* In the CustomLineRecordReader Configuration job =
context.getConfiguration();
One of the biggest issues I have had is staying true to the mapreduce.*
format
Best wishes,
Chris MacKenzie
From: Chris Mawata <ch...@gmail.com>
Reply-To: <us...@hadoop.apache.org>
Date: Friday, 27 June 2014 14:11
To: <us...@hadoop.apache.org>
Subject: Re: Partitioning and setup errors
The new Configuration() is suspicious. Are you setting configuration
information manually?
Chris
On Jun 27, 2014 5:16 AM, "Chris MacKenzie"
<st...@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> I realise my previous question may have been a bit naïve and I also realise I
> am asking an awful lot here, any advice would be greatly appreciated.
> * I have been using Hadoop 2.4 in local mode and am sticking to the
> mapreduce.* side of the track.
> * I am using a Custom Line reader to read each sequence into a Map
> * I have a partitioner class which is testing the key from the map class.
> * I've tried debugging in eclipse with a breakpoint in the partitioner class
> but getPartition(LongWritable mapKey, Text sequenceString, int numReduceTasks)
> is not being called.
> Could there be any reason for that ?
>
> Because my map and reduce code works in local mode within eclipse, I wondered
> if I may get the partitioner to work if I changed to Pseudo Distributed Mode
> exporting a runnable jar from Eclipse (Kepler)
>
> I have several faults On my own computer Pseudo Distributed Mode and the
> university clusters Pseudo Distributed Mode which I set up. I¹ve googled and
> read extensively but am not seeing a solution to any of these issues.
>
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User
> classes may not be found. See Job or Job#setJar(String).
> My driver code is:
> private void doParallelConcordance() throws Exception {
>
> Path inDir = new Path("input_sequences/10_sequences.txt");
>
> Path outDir = new Path("demo_output");
>
>
>
> Job job = Job.getInstance(new Configuration());
>
> job.setJarByClass(ParallelGeneticAlignment.class);
>
> job.setOutputKeyClass(Text.class);
>
> job.setOutputValueClass(IntWritable.class);
>
>
>
> job.setInputFormatClass(CustomFileInputFormat.class);
>
> job.setMapperClass(ConcordanceMapper.class);
>
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
>
> job.setReducerClass(ConcordanceReducer.class);
>
>
>
> FileInputFormat.addInputPath(job, inDir);
>
> FileOutputFormat.setOutputPath(job, outDir);
>
>
>
> job.waitForCompletion(true)
>
> }
>
>
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
> attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceMapper not found
>
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
> attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>
> Being new to this community, I thought it polite to introduce myself. I¹m
> planning to return to software development via an MSc at Heriot Watt
> University in Edinburgh. My MSc project is based on Fosters Genetic Sequence
> Alignment. I have written a sequential version my goal is now to port it to
> Hadoop.
>
> Thanks in advance,
> Regards,
>
> Chris MacKenzie
Re: Partitioning and setup errors
Posted by Chris MacKenzie <st...@chrismackenziephotography.co.uk>.
HI Chris,
Thanks for your response. I deeply appreciate it.
I don¹t know what you mean by that question. I use configuration:
* In the driver Job job = Job.getInstance(new Configuration());
* In the CustomLineRecordReader Configuration job =
context.getConfiguration();
One of the biggest issues I have had is staying true to the mapreduce.*
format
Best wishes,
Chris MacKenzie
From: Chris Mawata <ch...@gmail.com>
Reply-To: <us...@hadoop.apache.org>
Date: Friday, 27 June 2014 14:11
To: <us...@hadoop.apache.org>
Subject: Re: Partitioning and setup errors
The new Configuration() is suspicious. Are you setting configuration
information manually?
Chris
On Jun 27, 2014 5:16 AM, "Chris MacKenzie"
<st...@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> I realise my previous question may have been a bit naïve and I also realise I
> am asking an awful lot here, any advice would be greatly appreciated.
> * I have been using Hadoop 2.4 in local mode and am sticking to the
> mapreduce.* side of the track.
> * I am using a Custom Line reader to read each sequence into a Map
> * I have a partitioner class which is testing the key from the map class.
> * I've tried debugging in eclipse with a breakpoint in the partitioner class
> but getPartition(LongWritable mapKey, Text sequenceString, int numReduceTasks)
> is not being called.
> Could there be any reason for that ?
>
> Because my map and reduce code works in local mode within eclipse, I wondered
> if I may get the partitioner to work if I changed to Pseudo Distributed Mode
> exporting a runnable jar from Eclipse (Kepler)
>
> I have several faults On my own computer Pseudo Distributed Mode and the
> university clusters Pseudo Distributed Mode which I set up. I¹ve googled and
> read extensively but am not seeing a solution to any of these issues.
>
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User
> classes may not be found. See Job or Job#setJar(String).
> My driver code is:
> private void doParallelConcordance() throws Exception {
>
> Path inDir = new Path("input_sequences/10_sequences.txt");
>
> Path outDir = new Path("demo_output");
>
>
>
> Job job = Job.getInstance(new Configuration());
>
> job.setJarByClass(ParallelGeneticAlignment.class);
>
> job.setOutputKeyClass(Text.class);
>
> job.setOutputValueClass(IntWritable.class);
>
>
>
> job.setInputFormatClass(CustomFileInputFormat.class);
>
> job.setMapperClass(ConcordanceMapper.class);
>
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
>
> job.setReducerClass(ConcordanceReducer.class);
>
>
>
> FileInputFormat.addInputPath(job, inDir);
>
> FileOutputFormat.setOutputPath(job, outDir);
>
>
>
> job.waitForCompletion(true)
>
> }
>
>
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
> attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceMapper not found
>
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
> attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>
> Being new to this community, I thought it polite to introduce myself. I¹m
> planning to return to software development via an MSc at Heriot Watt
> University in Edinburgh. My MSc project is based on Fosters Genetic Sequence
> Alignment. I have written a sequential version my goal is now to port it to
> Hadoop.
>
> Thanks in advance,
> Regards,
>
> Chris MacKenzie
Re: Partitioning and setup errors
Posted by Chris MacKenzie <st...@chrismackenziephotography.co.uk>.
HI Chris,
Thanks for your response. I deeply appreciate it.
I don¹t know what you mean by that question. I use configuration:
* In the driver Job job = Job.getInstance(new Configuration());
* In the CustomLineRecordReader Configuration job =
context.getConfiguration();
One of the biggest issues I have had is staying true to the mapreduce.*
format
Best wishes,
Chris MacKenzie
From: Chris Mawata <ch...@gmail.com>
Reply-To: <us...@hadoop.apache.org>
Date: Friday, 27 June 2014 14:11
To: <us...@hadoop.apache.org>
Subject: Re: Partitioning and setup errors
The new Configuration() is suspicious. Are you setting configuration
information manually?
Chris
On Jun 27, 2014 5:16 AM, "Chris MacKenzie"
<st...@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> I realise my previous question may have been a bit naïve and I also realise I
> am asking an awful lot here, any advice would be greatly appreciated.
> * I have been using Hadoop 2.4 in local mode and am sticking to the
> mapreduce.* side of the track.
> * I am using a Custom Line reader to read each sequence into a Map
> * I have a partitioner class which is testing the key from the map class.
> * I've tried debugging in eclipse with a breakpoint in the partitioner class
> but getPartition(LongWritable mapKey, Text sequenceString, int numReduceTasks)
> is not being called.
> Could there be any reason for that ?
>
> Because my map and reduce code works in local mode within eclipse, I wondered
> if I may get the partitioner to work if I changed to Pseudo Distributed Mode
> exporting a runnable jar from Eclipse (Kepler)
>
> I have several faults On my own computer Pseudo Distributed Mode and the
> university clusters Pseudo Distributed Mode which I set up. I¹ve googled and
> read extensively but am not seeing a solution to any of these issues.
>
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User
> classes may not be found. See Job or Job#setJar(String).
> My driver code is:
> private void doParallelConcordance() throws Exception {
>
> Path inDir = new Path("input_sequences/10_sequences.txt");
>
> Path outDir = new Path("demo_output");
>
>
>
> Job job = Job.getInstance(new Configuration());
>
> job.setJarByClass(ParallelGeneticAlignment.class);
>
> job.setOutputKeyClass(Text.class);
>
> job.setOutputValueClass(IntWritable.class);
>
>
>
> job.setInputFormatClass(CustomFileInputFormat.class);
>
> job.setMapperClass(ConcordanceMapper.class);
>
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
>
> job.setReducerClass(ConcordanceReducer.class);
>
>
>
> FileInputFormat.addInputPath(job, inDir);
>
> FileOutputFormat.setOutputPath(job, outDir);
>
>
>
> job.waitForCompletion(true)
>
> }
>
>
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
> attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceMapper not found
>
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
> attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>
> Being new to this community, I thought it polite to introduce myself. I¹m
> planning to return to software development via an MSc at Heriot Watt
> University in Edinburgh. My MSc project is based on Fosters Genetic Sequence
> Alignment. I have written a sequential version my goal is now to port it to
> Hadoop.
>
> Thanks in advance,
> Regards,
>
> Chris MacKenzie
Re: Partitioning and setup errors
Posted by Chris MacKenzie <st...@chrismackenziephotography.co.uk>.
HI Chris,
Thanks for your response. I deeply appreciate it.
I don¹t know what you mean by that question. I use configuration:
* In the driver Job job = Job.getInstance(new Configuration());
* In the CustomLineRecordReader Configuration job =
context.getConfiguration();
One of the biggest issues I have had is staying true to the mapreduce.*
format
Best wishes,
Chris MacKenzie
From: Chris Mawata <ch...@gmail.com>
Reply-To: <us...@hadoop.apache.org>
Date: Friday, 27 June 2014 14:11
To: <us...@hadoop.apache.org>
Subject: Re: Partitioning and setup errors
The new Configuration() is suspicious. Are you setting configuration
information manually?
Chris
On Jun 27, 2014 5:16 AM, "Chris MacKenzie"
<st...@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> I realise my previous question may have been a bit naïve and I also realise I
> am asking an awful lot here, any advice would be greatly appreciated.
> * I have been using Hadoop 2.4 in local mode and am sticking to the
> mapreduce.* side of the track.
> * I am using a Custom Line reader to read each sequence into a Map
> * I have a partitioner class which is testing the key from the map class.
> * I've tried debugging in eclipse with a breakpoint in the partitioner class
> but getPartition(LongWritable mapKey, Text sequenceString, int numReduceTasks)
> is not being called.
> Could there be any reason for that ?
>
> Because my map and reduce code works in local mode within eclipse, I wondered
> if I may get the partitioner to work if I changed to Pseudo Distributed Mode
> exporting a runnable jar from Eclipse (Kepler)
>
> I have several faults On my own computer Pseudo Distributed Mode and the
> university clusters Pseudo Distributed Mode which I set up. I¹ve googled and
> read extensively but am not seeing a solution to any of these issues.
>
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User
> classes may not be found. See Job or Job#setJar(String).
> My driver code is:
> private void doParallelConcordance() throws Exception {
>
> Path inDir = new Path("input_sequences/10_sequences.txt");
>
> Path outDir = new Path("demo_output");
>
>
>
> Job job = Job.getInstance(new Configuration());
>
> job.setJarByClass(ParallelGeneticAlignment.class);
>
> job.setOutputKeyClass(Text.class);
>
> job.setOutputValueClass(IntWritable.class);
>
>
>
> job.setInputFormatClass(CustomFileInputFormat.class);
>
> job.setMapperClass(ConcordanceMapper.class);
>
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
>
> job.setReducerClass(ConcordanceReducer.class);
>
>
>
> FileInputFormat.addInputPath(job, inDir);
>
> FileOutputFormat.setOutputPath(job, outDir);
>
>
>
> job.waitForCompletion(true)
>
> }
>
>
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
> attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceMapper not found
>
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
> attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>
> Being new to this community, I thought it polite to introduce myself. I¹m
> planning to return to software development via an MSc at Heriot Watt
> University in Edinburgh. My MSc project is based on Fosters Genetic Sequence
> Alignment. I have written a sequential version my goal is now to port it to
> Hadoop.
>
> Thanks in advance,
> Regards,
>
> Chris MacKenzie
Re: Partitioning and setup errors
Posted by Chris Mawata <ch...@gmail.com>.
The new Configuration() is suspicious. Are you setting configuration
information manually?
Chris
On Jun 27, 2014 5:16 AM, "Chris MacKenzie" <
studio@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> I realise my previous question may have been a bit naïve and I also
> realise I am asking an awful lot here, any advice would be greatly
> appreciated.
>
> - I have been using Hadoop 2.4 in local mode and am sticking to the mapreduce.*
> side of the track.
> - I am using a Custom Line reader to read each sequence into a Map
> - I have a partitioner class which is testing the key from the map
> class.
> - I've tried debugging in eclipse with a breakpoint in the partitioner
> class but getPartition(LongWritable mapKey, Text sequenceString, int
> numReduceTasks) is not being called.
>
> Could there be any reason for that ?
>
> Because my map and reduce code works in local mode within eclipse, I wondered
> if I may get the partitioner to work if I changed to Pseudo Distributed
> Mode exporting a runnable jar from Eclipse (Kepler)
>
> I have several faults On my own computer Pseudo Distributed Mode and the
> university clusters Pseudo Distributed Mode which I set up. I’ve googled
> and read extensively but am not seeing a solution to any of these issues.
>
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User
> classes may not be found. See Job or Job#setJar(String).
> My driver code is:
>
> private void doParallelConcordance() throws Exception {
>
> Path inDir = new Path("input_sequences/10_sequences.txt");
>
> Path outDir = new Path("demo_output");
>
>
> Job job = Job.getInstance(new Configuration());
>
> job.setJarByClass(ParallelGeneticAlignment.class);
>
> job.setOutputKeyClass(Text.class);
>
> job.setOutputValueClass(IntWritable.class);
>
>
> job.setInputFormatClass(CustomFileInputFormat.class);
>
> job.setMapperClass(ConcordanceMapper.class);
>
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
>
> job.setReducerClass(ConcordanceReducer.class);
>
>
> FileInputFormat.addInputPath(job, inDir);
>
> FileOutputFormat.setOutputPath(job, outDir);
>
>
> job.waitForCompletion(true)
>
> }
>
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
> attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceMapper not found
>
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
> attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process
> : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process
> : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>
> Being new to this community, I thought it polite to introduce myself. I’m
> planning to return to software development via an MSc at Heriot Watt
> University in Edinburgh. My MSc project is based on Fosters Genetic
> Sequence Alignment. I have written a sequential version my goal is now to
> port it to Hadoop.
>
> Thanks in advance,
> Regards,
>
> Chris MacKenzie
>
Re: Partitioning and setup errors
Posted by Chris Mawata <ch...@gmail.com>.
The new Configuration() is suspicious. Are you setting configuration
information manually?
Chris
On Jun 27, 2014 5:16 AM, "Chris MacKenzie" <
studio@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> I realise my previous question may have been a bit naïve and I also
> realise I am asking an awful lot here, any advice would be greatly
> appreciated.
>
> - I have been using Hadoop 2.4 in local mode and am sticking to the mapreduce.*
> side of the track.
> - I am using a Custom Line reader to read each sequence into a Map
> - I have a partitioner class which is testing the key from the map
> class.
> - I've tried debugging in eclipse with a breakpoint in the partitioner
> class but getPartition(LongWritable mapKey, Text sequenceString, int
> numReduceTasks) is not being called.
>
> Could there be any reason for that ?
>
> Because my map and reduce code works in local mode within eclipse, I wondered
> if I may get the partitioner to work if I changed to Pseudo Distributed
> Mode exporting a runnable jar from Eclipse (Kepler)
>
> I have several faults On my own computer Pseudo Distributed Mode and the
> university clusters Pseudo Distributed Mode which I set up. I’ve googled
> and read extensively but am not seeing a solution to any of these issues.
>
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User
> classes may not be found. See Job or Job#setJar(String).
> My driver code is:
>
> private void doParallelConcordance() throws Exception {
>
> Path inDir = new Path("input_sequences/10_sequences.txt");
>
> Path outDir = new Path("demo_output");
>
>
> Job job = Job.getInstance(new Configuration());
>
> job.setJarByClass(ParallelGeneticAlignment.class);
>
> job.setOutputKeyClass(Text.class);
>
> job.setOutputValueClass(IntWritable.class);
>
>
> job.setInputFormatClass(CustomFileInputFormat.class);
>
> job.setMapperClass(ConcordanceMapper.class);
>
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
>
> job.setReducerClass(ConcordanceReducer.class);
>
>
> FileInputFormat.addInputPath(job, inDir);
>
> FileOutputFormat.setOutputPath(job, outDir);
>
>
> job.waitForCompletion(true)
>
> }
>
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
> attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceMapper not found
>
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
> attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process
> : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process
> : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>
> Being new to this community, I thought it polite to introduce myself. I’m
> planning to return to software development via an MSc at Heriot Watt
> University in Edinburgh. My MSc project is based on Fosters Genetic
> Sequence Alignment. I have written a sequential version my goal is now to
> port it to Hadoop.
>
> Thanks in advance,
> Regards,
>
> Chris MacKenzie
>
Re: Partitioning and setup errors
Posted by Chris Mawata <ch...@gmail.com>.
The new Configuration() is suspicious. Are you setting configuration
information manually?
Chris
On Jun 27, 2014 5:16 AM, "Chris MacKenzie" <
studio@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> I realise my previous question may have been a bit naïve and I also
> realise I am asking an awful lot here, any advice would be greatly
> appreciated.
>
> - I have been using Hadoop 2.4 in local mode and am sticking to the mapreduce.*
> side of the track.
> - I am using a Custom Line reader to read each sequence into a Map
> - I have a partitioner class which is testing the key from the map
> class.
> - I've tried debugging in eclipse with a breakpoint in the partitioner
> class but getPartition(LongWritable mapKey, Text sequenceString, int
> numReduceTasks) is not being called.
>
> Could there be any reason for that ?
>
> Because my map and reduce code works in local mode within eclipse, I wondered
> if I may get the partitioner to work if I changed to Pseudo Distributed
> Mode exporting a runnable jar from Eclipse (Kepler)
>
> I have several faults On my own computer Pseudo Distributed Mode and the
> university clusters Pseudo Distributed Mode which I set up. I’ve googled
> and read extensively but am not seeing a solution to any of these issues.
>
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User
> classes may not be found. See Job or Job#setJar(String).
> My driver code is:
>
> private void doParallelConcordance() throws Exception {
>
> Path inDir = new Path("input_sequences/10_sequences.txt");
>
> Path outDir = new Path("demo_output");
>
>
> Job job = Job.getInstance(new Configuration());
>
> job.setJarByClass(ParallelGeneticAlignment.class);
>
> job.setOutputKeyClass(Text.class);
>
> job.setOutputValueClass(IntWritable.class);
>
>
> job.setInputFormatClass(CustomFileInputFormat.class);
>
> job.setMapperClass(ConcordanceMapper.class);
>
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
>
> job.setReducerClass(ConcordanceReducer.class);
>
>
> FileInputFormat.addInputPath(job, inDir);
>
> FileOutputFormat.setOutputPath(job, outDir);
>
>
> job.waitForCompletion(true)
>
> }
>
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
> attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceMapper not found
>
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
> attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process
> : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process
> : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>
> Being new to this community, I thought it polite to introduce myself. I’m
> planning to return to software development via an MSc at Heriot Watt
> University in Edinburgh. My MSc project is based on Fosters Genetic
> Sequence Alignment. I have written a sequential version my goal is now to
> port it to Hadoop.
>
> Thanks in advance,
> Regards,
>
> Chris MacKenzie
>
Re: Partitioning and setup errors
Posted by Chris Mawata <ch...@gmail.com>.
Probably my fault. I was looking for the
extends Configurable implements Tool
part. I will double check when I get home rather than send you on a wild
goose chase.
Cheers
Chris
On Jun 27, 2014 8:16 AM, "Chris MacKenzie" <
studio@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> I realise my previous question may have been a bit naïve and I also
> realise I am asking an awful lot here, any advice would be greatly
> appreciated.
>
> - I have been using Hadoop 2.4 in local mode and am sticking to the mapreduce.*
> side of the track.
> - I am using a Custom Line reader to read each sequence into a Map
> - I have a partitioner class which is testing the key from the map
> class.
> - I've tried debugging in eclipse with a breakpoint in the partitioner
> class but getPartition(LongWritable mapKey, Text sequenceString, int
> numReduceTasks) is not being called.
>
> Could there be any reason for that ?
>
> Because my map and reduce code works in local mode within eclipse, I wondered
> if I may get the partitioner to work if I changed to Pseudo Distributed
> Mode exporting a runnable jar from Eclipse (Kepler)
>
> I have several faults On my own computer Pseudo Distributed Mode and the
> university clusters Pseudo Distributed Mode which I set up. I’ve googled
> and read extensively but am not seeing a solution to any of these issues.
>
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User
> classes may not be found. See Job or Job#setJar(String).
> My driver code is:
>
> private void doParallelConcordance() throws Exception {
>
> Path inDir = new Path("input_sequences/10_sequences.txt");
>
> Path outDir = new Path("demo_output");
>
>
> Job job = Job.getInstance(new Configuration());
>
> job.setJarByClass(ParallelGeneticAlignment.class);
>
> job.setOutputKeyClass(Text.class);
>
> job.setOutputValueClass(IntWritable.class);
>
>
> job.setInputFormatClass(CustomFileInputFormat.class);
>
> job.setMapperClass(ConcordanceMapper.class);
>
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
>
> job.setReducerClass(ConcordanceReducer.class);
>
>
> FileInputFormat.addInputPath(job, inDir);
>
> FileOutputFormat.setOutputPath(job, outDir);
>
>
> job.waitForCompletion(true)
>
> }
>
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
> attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceMapper not found
>
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
> attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process
> : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process
> : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>
> Being new to this community, I thought it polite to introduce myself. I’m
> planning to return to software development via an MSc at Heriot Watt
> University in Edinburgh. My MSc project is based on Fosters Genetic
> Sequence Alignment. I have written a sequential version my goal is now to
> port it to Hadoop.
>
> Thanks in advance,
> Regards,
>
> Chris MacKenzie
>
Re: Partitioning and setup errors
Posted by Chris Mawata <ch...@gmail.com>.
Probably my fault. I was looking for the
extends Configurable implements Tool
part. I will double check when I get home rather than send you on a wild
goose chase.
Cheers
Chris
On Jun 27, 2014 8:16 AM, "Chris MacKenzie" <
studio@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> I realise my previous question may have been a bit naïve and I also
> realise I am asking an awful lot here, any advice would be greatly
> appreciated.
>
> - I have been using Hadoop 2.4 in local mode and am sticking to the mapreduce.*
> side of the track.
> - I am using a Custom Line reader to read each sequence into a Map
> - I have a partitioner class which is testing the key from the map
> class.
> - I've tried debugging in eclipse with a breakpoint in the partitioner
> class but getPartition(LongWritable mapKey, Text sequenceString, int
> numReduceTasks) is not being called.
>
> Could there be any reason for that ?
>
> Because my map and reduce code works in local mode within eclipse, I wondered
> if I may get the partitioner to work if I changed to Pseudo Distributed
> Mode exporting a runnable jar from Eclipse (Kepler)
>
> I have several faults On my own computer Pseudo Distributed Mode and the
> university clusters Pseudo Distributed Mode which I set up. I’ve googled
> and read extensively but am not seeing a solution to any of these issues.
>
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User
> classes may not be found. See Job or Job#setJar(String).
> My driver code is:
>
> private void doParallelConcordance() throws Exception {
>
> Path inDir = new Path("input_sequences/10_sequences.txt");
>
> Path outDir = new Path("demo_output");
>
>
> Job job = Job.getInstance(new Configuration());
>
> job.setJarByClass(ParallelGeneticAlignment.class);
>
> job.setOutputKeyClass(Text.class);
>
> job.setOutputValueClass(IntWritable.class);
>
>
> job.setInputFormatClass(CustomFileInputFormat.class);
>
> job.setMapperClass(ConcordanceMapper.class);
>
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
>
> job.setReducerClass(ConcordanceReducer.class);
>
>
> FileInputFormat.addInputPath(job, inDir);
>
> FileOutputFormat.setOutputPath(job, outDir);
>
>
> job.waitForCompletion(true)
>
> }
>
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
> attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceMapper not found
>
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
> attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process
> : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process
> : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>
> Being new to this community, I thought it polite to introduce myself. I’m
> planning to return to software development via an MSc at Heriot Watt
> University in Edinburgh. My MSc project is based on Fosters Genetic
> Sequence Alignment. I have written a sequential version my goal is now to
> port it to Hadoop.
>
> Thanks in advance,
> Regards,
>
> Chris MacKenzie
>
Re: Partitioning and setup errors
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
What is happening is the client is not able to pick up the right jar to push to the cluster. It looks in the class-path for the jar that contains the class ParallelGeneticAlignment.
How are you packaging your code? How are your running your job - paste the command line?
+Vinod
On Jun 27, 2014, at 5:15 AM, Chris MacKenzie <st...@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> I realise my previous question may have been a bit naïve and I also realise I am asking an awful lot here, any advice would be greatly appreciated.
> I have been using Hadoop 2.4 in local mode and am sticking to the mapreduce.* side of the track.
> I am using a Custom Line reader to read each sequence into a Map
> I have a partitioner class which is testing the key from the map class.
> I've tried debugging in eclipse with a breakpoint in the partitioner class but getPartition(LongWritable mapKey, Text sequenceString, int numReduceTasks) is not being called.
> Could there be any reason for that ?
>
> Because my map and reduce code works in local mode within eclipse, I wondered if I may get the partitioner to work if I changed to Pseudo Distributed Mode exporting a runnable jar from Eclipse (Kepler)
>
> I have several faults On my own computer Pseudo Distributed Mode and the university clusters Pseudo Distributed Mode which I set up. I’ve googled and read extensively but am not seeing a solution to any of these issues.
>
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String).
> My driver code is:
> private void doParallelConcordance() throws Exception {
>
> Path inDir = new Path("input_sequences/10_sequences.txt");
> Path outDir = new Path("demo_output");
>
> Job job = Job.getInstance(new Configuration());
> job.setJarByClass(ParallelGeneticAlignment.class);
>
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(IntWritable.class);
>
> job.setInputFormatClass(CustomFileInputFormat.class);
> job.setMapperClass(ConcordanceMapper.class);
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
> job.setReducerClass(ConcordanceReducer.class);
>
> FileInputFormat.addInputPath(job, inDir);
> FileOutputFormat.setOutputPath(job, outDir);
>
> job.waitForCompletion(true)
> }
>
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id : attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class par.gene.align.concordance.ConcordanceMapper not found
>
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id : attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class par.gene.align.concordance.ConcordanceReducer not found
>
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>
> Being new to this community, I thought it polite to introduce myself. I’m planning to return to software development via an MSc at Heriot Watt University in Edinburgh. My MSc project is based on Fosters Genetic Sequence Alignment. I have written a sequential version my goal is now to port it to Hadoop.
>
> Thanks in advance,
> Regards,
>
> Chris MacKenzie
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Partitioning and setup errors
Posted by Chris Mawata <ch...@gmail.com>.
The new Configuration() is suspicious. Are you setting configuration
information manually?
Chris
On Jun 27, 2014 5:16 AM, "Chris MacKenzie" <
studio@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> I realise my previous question may have been a bit naïve and I also
> realise I am asking an awful lot here, any advice would be greatly
> appreciated.
>
> - I have been using Hadoop 2.4 in local mode and am sticking to the mapreduce.*
> side of the track.
> - I am using a Custom Line reader to read each sequence into a Map
> - I have a partitioner class which is testing the key from the map
> class.
> - I've tried debugging in eclipse with a breakpoint in the partitioner
> class but getPartition(LongWritable mapKey, Text sequenceString, int
> numReduceTasks) is not being called.
>
> Could there be any reason for that ?
>
> Because my map and reduce code works in local mode within eclipse, I wondered
> if I may get the partitioner to work if I changed to Pseudo Distributed
> Mode exporting a runnable jar from Eclipse (Kepler)
>
> I have several faults On my own computer Pseudo Distributed Mode and the
> university clusters Pseudo Distributed Mode which I set up. I’ve googled
> and read extensively but am not seeing a solution to any of these issues.
>
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User
> classes may not be found. See Job or Job#setJar(String).
> My driver code is:
>
> private void doParallelConcordance() throws Exception {
>
> Path inDir = new Path("input_sequences/10_sequences.txt");
>
> Path outDir = new Path("demo_output");
>
>
> Job job = Job.getInstance(new Configuration());
>
> job.setJarByClass(ParallelGeneticAlignment.class);
>
> job.setOutputKeyClass(Text.class);
>
> job.setOutputValueClass(IntWritable.class);
>
>
> job.setInputFormatClass(CustomFileInputFormat.class);
>
> job.setMapperClass(ConcordanceMapper.class);
>
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
>
> job.setReducerClass(ConcordanceReducer.class);
>
>
> FileInputFormat.addInputPath(job, inDir);
>
> FileOutputFormat.setOutputPath(job, outDir);
>
>
> job.waitForCompletion(true)
>
> }
>
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
> attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceMapper not found
>
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
> attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process
> : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process
> : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>
> Being new to this community, I thought it polite to introduce myself. I’m
> planning to return to software development via an MSc at Heriot Watt
> University in Edinburgh. My MSc project is based on Fosters Genetic
> Sequence Alignment. I have written a sequential version my goal is now to
> port it to Hadoop.
>
> Thanks in advance,
> Regards,
>
> Chris MacKenzie
>
Re: Partitioning and setup errors
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
What is happening is the client is not able to pick up the right jar to push to the cluster. It looks in the class-path for the jar that contains the class ParallelGeneticAlignment.
How are you packaging your code? How are your running your job - paste the command line?
+Vinod
On Jun 27, 2014, at 5:15 AM, Chris MacKenzie <st...@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> I realise my previous question may have been a bit naïve and I also realise I am asking an awful lot here, any advice would be greatly appreciated.
> I have been using Hadoop 2.4 in local mode and am sticking to the mapreduce.* side of the track.
> I am using a Custom Line reader to read each sequence into a Map
> I have a partitioner class which is testing the key from the map class.
> I've tried debugging in eclipse with a breakpoint in the partitioner class but getPartition(LongWritable mapKey, Text sequenceString, int numReduceTasks) is not being called.
> Could there be any reason for that ?
>
> Because my map and reduce code works in local mode within eclipse, I wondered if I may get the partitioner to work if I changed to Pseudo Distributed Mode exporting a runnable jar from Eclipse (Kepler)
>
> I have several faults On my own computer Pseudo Distributed Mode and the university clusters Pseudo Distributed Mode which I set up. I’ve googled and read extensively but am not seeing a solution to any of these issues.
>
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String).
> My driver code is:
> private void doParallelConcordance() throws Exception {
>
> Path inDir = new Path("input_sequences/10_sequences.txt");
> Path outDir = new Path("demo_output");
>
> Job job = Job.getInstance(new Configuration());
> job.setJarByClass(ParallelGeneticAlignment.class);
>
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(IntWritable.class);
>
> job.setInputFormatClass(CustomFileInputFormat.class);
> job.setMapperClass(ConcordanceMapper.class);
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
> job.setReducerClass(ConcordanceReducer.class);
>
> FileInputFormat.addInputPath(job, inDir);
> FileOutputFormat.setOutputPath(job, outDir);
>
> job.waitForCompletion(true)
> }
>
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id : attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class par.gene.align.concordance.ConcordanceMapper not found
>
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id : attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class par.gene.align.concordance.ConcordanceReducer not found
>
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>
> Being new to this community, I thought it polite to introduce myself. I’m planning to return to software development via an MSc at Heriot Watt University in Edinburgh. My MSc project is based on Fosters Genetic Sequence Alignment. I have written a sequential version my goal is now to port it to Hadoop.
>
> Thanks in advance,
> Regards,
>
> Chris MacKenzie
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Partitioning and setup errors
Posted by Chris Mawata <ch...@gmail.com>.
Probably my fault. I was looking for the
extends Configurable implements Tool
part. I will double check when I get home rather than send you on a wild
goose chase.
Cheers
Chris
On Jun 27, 2014 8:16 AM, "Chris MacKenzie" <
studio@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> I realise my previous question may have been a bit naïve and I also
> realise I am asking an awful lot here, any advice would be greatly
> appreciated.
>
> - I have been using Hadoop 2.4 in local mode and am sticking to the mapreduce.*
> side of the track.
> - I am using a Custom Line reader to read each sequence into a Map
> - I have a partitioner class which is testing the key from the map
> class.
> - I've tried debugging in eclipse with a breakpoint in the partitioner
> class but getPartition(LongWritable mapKey, Text sequenceString, int
> numReduceTasks) is not being called.
>
> Could there be any reason for that ?
>
> Because my map and reduce code works in local mode within eclipse, I wondered
> if I may get the partitioner to work if I changed to Pseudo Distributed
> Mode exporting a runnable jar from Eclipse (Kepler)
>
> I have several faults On my own computer Pseudo Distributed Mode and the
> university clusters Pseudo Distributed Mode which I set up. I’ve googled
> and read extensively but am not seeing a solution to any of these issues.
>
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User
> classes may not be found. See Job or Job#setJar(String).
> My driver code is:
>
> private void doParallelConcordance() throws Exception {
>
> Path inDir = new Path("input_sequences/10_sequences.txt");
>
> Path outDir = new Path("demo_output");
>
>
> Job job = Job.getInstance(new Configuration());
>
> job.setJarByClass(ParallelGeneticAlignment.class);
>
> job.setOutputKeyClass(Text.class);
>
> job.setOutputValueClass(IntWritable.class);
>
>
> job.setInputFormatClass(CustomFileInputFormat.class);
>
> job.setMapperClass(ConcordanceMapper.class);
>
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
>
> job.setReducerClass(ConcordanceReducer.class);
>
>
> FileInputFormat.addInputPath(job, inDir);
>
> FileOutputFormat.setOutputPath(job, outDir);
>
>
> job.waitForCompletion(true)
>
> }
>
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
> attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceMapper not found
>
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
> attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process
> : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process
> : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>
> Being new to this community, I thought it polite to introduce myself. I’m
> planning to return to software development via an MSc at Heriot Watt
> University in Edinburgh. My MSc project is based on Fosters Genetic
> Sequence Alignment. I have written a sequential version my goal is now to
> port it to Hadoop.
>
> Thanks in advance,
> Regards,
>
> Chris MacKenzie
>
Re: Partitioning and setup errors
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
What is happening is the client is not able to pick up the right jar to push to the cluster. It looks in the class-path for the jar that contains the class ParallelGeneticAlignment.
How are you packaging your code? How are your running your job - paste the command line?
+Vinod
On Jun 27, 2014, at 5:15 AM, Chris MacKenzie <st...@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> I realise my previous question may have been a bit naïve and I also realise I am asking an awful lot here, any advice would be greatly appreciated.
> I have been using Hadoop 2.4 in local mode and am sticking to the mapreduce.* side of the track.
> I am using a Custom Line reader to read each sequence into a Map
> I have a partitioner class which is testing the key from the map class.
> I've tried debugging in eclipse with a breakpoint in the partitioner class but getPartition(LongWritable mapKey, Text sequenceString, int numReduceTasks) is not being called.
> Could there be any reason for that ?
>
> Because my map and reduce code works in local mode within eclipse, I wondered if I may get the partitioner to work if I changed to Pseudo Distributed Mode exporting a runnable jar from Eclipse (Kepler)
>
> I have several faults On my own computer Pseudo Distributed Mode and the university clusters Pseudo Distributed Mode which I set up. I’ve googled and read extensively but am not seeing a solution to any of these issues.
>
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String).
> My driver code is:
> private void doParallelConcordance() throws Exception {
>
> Path inDir = new Path("input_sequences/10_sequences.txt");
> Path outDir = new Path("demo_output");
>
> Job job = Job.getInstance(new Configuration());
> job.setJarByClass(ParallelGeneticAlignment.class);
>
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(IntWritable.class);
>
> job.setInputFormatClass(CustomFileInputFormat.class);
> job.setMapperClass(ConcordanceMapper.class);
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
> job.setReducerClass(ConcordanceReducer.class);
>
> FileInputFormat.addInputPath(job, inDir);
> FileOutputFormat.setOutputPath(job, outDir);
>
> job.waitForCompletion(true)
> }
>
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id : attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class par.gene.align.concordance.ConcordanceMapper not found
>
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id : attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class par.gene.align.concordance.ConcordanceReducer not found
>
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>
> Being new to this community, I thought it polite to introduce myself. I’m planning to return to software development via an MSc at Heriot Watt University in Edinburgh. My MSc project is based on Fosters Genetic Sequence Alignment. I have written a sequential version my goal is now to port it to Hadoop.
>
> Thanks in advance,
> Regards,
>
> Chris MacKenzie
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.