You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Alex Thieme <at...@athieme.com> on 2013/02/12 22:11:01 UTC
Java submit job to remote server
I apologize for asking what seems to be such a basic question, but I would use some help with submitting a job to a remote server.
I have downloaded and installed hadoop locally in pseudo-distributed mode. I have written some Java code to submit a job.
Here's the org.apache.hadoop.util.Tool and org.apache.hadoop.mapreduce.Mapper I've written.
If I enable the conf.set("mapred.job.tracker", "localhost:9001") line, then I get the exception included below.
If that line is disabled, then the job is completed. However, in reviewing the hadoop server administration page (http://localhost:50030/jobtracker.jsp) I don't see the job as processed by the server. Instead, I wonder if my Java code is simply running the necessary mapper Java code, bypassing the locally installed server.
Thanks in advance.
Alex
public class OfflineDataTool extends Configured implements Tool {
public int run(final String[] args) throws Exception {
final Configuration conf = getConf();
//conf.set("mapred.job.tracker", "localhost:9001");
final Job job = new Job(conf);
job.setJarByClass(getClass());
job.setJobName(getClass().getName());
job.setMapperClass(OfflineDataMapper.class);
job.setInputFormatClass(TextInputFormat.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(Text.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
FileInputFormat.addInputPath(job, new org.apache.hadoop.fs.Path(args[0]));
final org.apache.hadoop.fs.Path output = new org.apache.hadoop.fs.Path(args[1]);
FileSystem.get(conf).delete(output, true);
FileOutputFormat.setOutputPath(job, output);
return job.waitForCompletion(true) ? 0 : 1;
}
public static void main(final String[] args) {
try {
int result = ToolRunner.run(new Configuration(), new OfflineDataTool(), new String[]{"offline/input", "offline/output"});
log.error("result = {}", result);
} catch (final Exception e) {
throw new RuntimeException(e);
}
}
}
public class OfflineDataMapper extends Mapper<LongWritable, Text, Text, Text> {
public OfflineDataMapper() {
super();
}
@Override
protected void map(final LongWritable key, final Text value, final Context context) throws IOException, InterruptedException {
final String inputString = value.toString();
OfflineDataMapper.log.error("inputString = {}", inputString);
}
}
Re: Java submit job to remote server
Posted by Harsh J <ha...@cloudera.com>.
Hi Alex,
There can be two reasons here: (a) your client libs and the server
libs have a version mismatch that includes incompatible RPC protocol
changes, making it impossible for communication to happen, or (b) the
port you are connecting to in your app, is not really the JobTracker
port.
For (a), fixing dependencies in the client runtime/project to match
the version of hadoop deployed on the server would usually fix it.
For (b), inspecting the server's core-site.xml (For the
fs.default.name port, which is the NN's port) and mapred-site.xml (For
the mapred.job.tracker port, which is the JT's port), would help you
figure out what the deployment looks like and fix the port configs in
code as well, to connect to the right one.
Does either of this help?
On Fri, Feb 15, 2013 at 6:22 AM, Alex Thieme <at...@athieme.com> wrote:
> Any thoughts on why my connection to the hadoop server fails? An help
> provided would be greatly appreciated.
>
> Alex Thieme
> athieme@athieme.com
> 508-361-2788
>
>
> On Feb 13, 2013, at 1:41 PM, Alex Thieme <at...@athieme.com> wrote:
>
> It appears this is the full extent of the stack trace. Anything prior to the
> org.apache.hadoop calls are from my container where hadoop is called from.
>
> Caused by: java.io.IOException: Call to /127.0.0.1:9001 failed on local
> exception: java.io.EOFException
> at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
> at org.apache.hadoop.ipc.Client.call(Client.java:743)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at org.apache.hadoop.mapred.$Proxy55.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
> at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:429)
> at org.apache.hadoop.mapred.JobClient.init(JobClient.java:423)
> at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:410)
> at org.apache.hadoop.mapreduce.Job.<init>(Job.java:50)
> at com.allenabi.sherlock.graph.OfflineDataTool.run(OfflineDataTool.java:25)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at
> com.allenabi.sherlock.graph.OfflineDataComponent.submitJob(OfflineDataComponent.java:67)
> ... 64 more
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
>
> Alex Thieme
> athieme@athieme.com
> 508-361-2788
> In
>
> On Feb 12, 2013, at 8:16 PM, Hemanth Yamijala <yh...@thoughtworks.com>
> wrote:
>
> Can you please include the complete stack trace and not just the root. Also,
> have you set fs.default.name to a hdfs location like hdfs://localhost:9000 ?
>
> Thanks
> Hemanth
>
> On Wednesday, February 13, 2013, Alex Thieme wrote:
>>
>> Thanks for the prompt reply and I'm sorry I forgot to include the
>> exception. My bad. I've included it below. There certainly appears to be a
>> server running on localhost:9001. At least, I was able to telnet to that
>> address. While in development, I'm treating the server on localhost as the
>> remote server. Moving to production, there'd obviously be a different remote
>> server address configured.
>>
>> Root Exception stack trace:
>> java.io.EOFException
>> at java.io.DataInputStream.readInt(DataInputStream.java:375)
>> at
>> org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
>> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
>> + 3 more (set debug level logging or '-Dmule.verbose.exceptions=true'
>> for everything)
>>
>> ********************************************************************************
>>
>> On Feb 12, 2013, at 4:22 PM, Nitin Pawar <ni...@gmail.com> wrote:
>>
>> conf.set("mapred.job.tracker", "localhost:9001");
>>
>> this means that your jobtracker is on port 9001 on localhost
>>
>> if you change it to the remote host and thats the port its running on then
>> it should work as expected
>>
>> whats the exception you are getting?
>>
>>
>> On Wed, Feb 13, 2013 at 2:41 AM, Alex Thieme <at...@athieme.com> wrote:
>>
>> I apologize for asking what seems to be such a basic question, but I would
>> use some help with submitting a job to a remote server.
>>
>> I have downloaded and installed hadoop locally in pseudo-distributed mode.
>> I have written some Java code to submit a job.
>>
>> Here's the org.apache.hadoop.util.Tool and
>> org.apache.hadoop.mapreduce.Mapper I've written.
>>
>> If I enable the conf.set("mapred.job.tracker", "localhost:9001") line,
>> then I get the exception included below.
>>
>> If that line is disabled, then the job is completed. However, in reviewing
>> the hadoop server administration page
>> (http://localhost:50030/jobtracker.jsp) I don't see the job as processed by
>> the server. Instead, I wonder if my Java code is simply running the
>> necessary mapper Java code, bypassing the locally installed server.
>>
>> Thanks in advance.
>>
>> Alex
>>
>> public class OfflineDataTool extends Configured implements Tool {
>>
>> public int run(final String[] args) throws Exception {
>> final Configuration conf = getConf();
>> //conf.set("mapred.job.tracker", "localhost:9001");
>>
>> final Job job = new Job(conf);
>> job.setJarByClass(getClass());
>> job.setJobName(getClass().getName());
>>
>> job.setMapperClass(OfflineDataMapper.class);
>>
>> job.setInputFormatClass(TextInputFormat.class);
>>
>> job.setMapOutputKeyClass(Text.class);
>> job.setMapOutputValueClass(Text.class);
>>
>> job.setOutputKeyClass(Text.class);
>> job.setOutputValueClass(Text.class);
>>
>> FileInputFormat.addInputPath(job, new
>> org.apache.hadoop.fs.Path(args[0]));
>>
>> final org.apache.hadoop.fs.Path output = new org.a
>
>
--
Harsh J
Re: Java submit job to remote server
Posted by Harsh J <ha...@cloudera.com>.
Hi Alex,
There can be two reasons here: (a) your client libs and the server
libs have a version mismatch that includes incompatible RPC protocol
changes, making it impossible for communication to happen, or (b) the
port you are connecting to in your app, is not really the JobTracker
port.
For (a), fixing dependencies in the client runtime/project to match
the version of hadoop deployed on the server would usually fix it.
For (b), inspecting the server's core-site.xml (For the
fs.default.name port, which is the NN's port) and mapred-site.xml (For
the mapred.job.tracker port, which is the JT's port), would help you
figure out what the deployment looks like and fix the port configs in
code as well, to connect to the right one.
Does either of this help?
On Fri, Feb 15, 2013 at 6:22 AM, Alex Thieme <at...@athieme.com> wrote:
> Any thoughts on why my connection to the hadoop server fails? An help
> provided would be greatly appreciated.
>
> Alex Thieme
> athieme@athieme.com
> 508-361-2788
>
>
> On Feb 13, 2013, at 1:41 PM, Alex Thieme <at...@athieme.com> wrote:
>
> It appears this is the full extent of the stack trace. Anything prior to the
> org.apache.hadoop calls are from my container where hadoop is called from.
>
> Caused by: java.io.IOException: Call to /127.0.0.1:9001 failed on local
> exception: java.io.EOFException
> at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
> at org.apache.hadoop.ipc.Client.call(Client.java:743)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at org.apache.hadoop.mapred.$Proxy55.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
> at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:429)
> at org.apache.hadoop.mapred.JobClient.init(JobClient.java:423)
> at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:410)
> at org.apache.hadoop.mapreduce.Job.<init>(Job.java:50)
> at com.allenabi.sherlock.graph.OfflineDataTool.run(OfflineDataTool.java:25)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at
> com.allenabi.sherlock.graph.OfflineDataComponent.submitJob(OfflineDataComponent.java:67)
> ... 64 more
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
>
> Alex Thieme
> athieme@athieme.com
> 508-361-2788
> In
>
> On Feb 12, 2013, at 8:16 PM, Hemanth Yamijala <yh...@thoughtworks.com>
> wrote:
>
> Can you please include the complete stack trace and not just the root. Also,
> have you set fs.default.name to a hdfs location like hdfs://localhost:9000 ?
>
> Thanks
> Hemanth
>
> On Wednesday, February 13, 2013, Alex Thieme wrote:
>>
>> Thanks for the prompt reply and I'm sorry I forgot to include the
>> exception. My bad. I've included it below. There certainly appears to be a
>> server running on localhost:9001. At least, I was able to telnet to that
>> address. While in development, I'm treating the server on localhost as the
>> remote server. Moving to production, there'd obviously be a different remote
>> server address configured.
>>
>> Root Exception stack trace:
>> java.io.EOFException
>> at java.io.DataInputStream.readInt(DataInputStream.java:375)
>> at
>> org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
>> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
>> + 3 more (set debug level logging or '-Dmule.verbose.exceptions=true'
>> for everything)
>>
>> ********************************************************************************
>>
>> On Feb 12, 2013, at 4:22 PM, Nitin Pawar <ni...@gmail.com> wrote:
>>
>> conf.set("mapred.job.tracker", "localhost:9001");
>>
>> this means that your jobtracker is on port 9001 on localhost
>>
>> if you change it to the remote host and thats the port its running on then
>> it should work as expected
>>
>> whats the exception you are getting?
>>
>>
>> On Wed, Feb 13, 2013 at 2:41 AM, Alex Thieme <at...@athieme.com> wrote:
>>
>> I apologize for asking what seems to be such a basic question, but I would
>> use some help with submitting a job to a remote server.
>>
>> I have downloaded and installed hadoop locally in pseudo-distributed mode.
>> I have written some Java code to submit a job.
>>
>> Here's the org.apache.hadoop.util.Tool and
>> org.apache.hadoop.mapreduce.Mapper I've written.
>>
>> If I enable the conf.set("mapred.job.tracker", "localhost:9001") line,
>> then I get the exception included below.
>>
>> If that line is disabled, then the job is completed. However, in reviewing
>> the hadoop server administration page
>> (http://localhost:50030/jobtracker.jsp) I don't see the job as processed by
>> the server. Instead, I wonder if my Java code is simply running the
>> necessary mapper Java code, bypassing the locally installed server.
>>
>> Thanks in advance.
>>
>> Alex
>>
>> public class OfflineDataTool extends Configured implements Tool {
>>
>> public int run(final String[] args) throws Exception {
>> final Configuration conf = getConf();
>> //conf.set("mapred.job.tracker", "localhost:9001");
>>
>> final Job job = new Job(conf);
>> job.setJarByClass(getClass());
>> job.setJobName(getClass().getName());
>>
>> job.setMapperClass(OfflineDataMapper.class);
>>
>> job.setInputFormatClass(TextInputFormat.class);
>>
>> job.setMapOutputKeyClass(Text.class);
>> job.setMapOutputValueClass(Text.class);
>>
>> job.setOutputKeyClass(Text.class);
>> job.setOutputValueClass(Text.class);
>>
>> FileInputFormat.addInputPath(job, new
>> org.apache.hadoop.fs.Path(args[0]));
>>
>> final org.apache.hadoop.fs.Path output = new org.a
>
>
--
Harsh J
Re: Java submit job to remote server
Posted by Harsh J <ha...@cloudera.com>.
Hi Alex,
There can be two reasons here: (a) your client libs and the server
libs have a version mismatch that includes incompatible RPC protocol
changes, making it impossible for communication to happen, or (b) the
port you are connecting to in your app, is not really the JobTracker
port.
For (a), fixing dependencies in the client runtime/project to match
the version of hadoop deployed on the server would usually fix it.
For (b), inspecting the server's core-site.xml (For the
fs.default.name port, which is the NN's port) and mapred-site.xml (For
the mapred.job.tracker port, which is the JT's port), would help you
figure out what the deployment looks like and fix the port configs in
code as well, to connect to the right one.
Does either of this help?
On Fri, Feb 15, 2013 at 6:22 AM, Alex Thieme <at...@athieme.com> wrote:
> Any thoughts on why my connection to the hadoop server fails? An help
> provided would be greatly appreciated.
>
> Alex Thieme
> athieme@athieme.com
> 508-361-2788
>
>
> On Feb 13, 2013, at 1:41 PM, Alex Thieme <at...@athieme.com> wrote:
>
> It appears this is the full extent of the stack trace. Anything prior to the
> org.apache.hadoop calls are from my container where hadoop is called from.
>
> Caused by: java.io.IOException: Call to /127.0.0.1:9001 failed on local
> exception: java.io.EOFException
> at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
> at org.apache.hadoop.ipc.Client.call(Client.java:743)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at org.apache.hadoop.mapred.$Proxy55.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
> at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:429)
> at org.apache.hadoop.mapred.JobClient.init(JobClient.java:423)
> at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:410)
> at org.apache.hadoop.mapreduce.Job.<init>(Job.java:50)
> at com.allenabi.sherlock.graph.OfflineDataTool.run(OfflineDataTool.java:25)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at
> com.allenabi.sherlock.graph.OfflineDataComponent.submitJob(OfflineDataComponent.java:67)
> ... 64 more
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
>
> Alex Thieme
> athieme@athieme.com
> 508-361-2788
> In
>
> On Feb 12, 2013, at 8:16 PM, Hemanth Yamijala <yh...@thoughtworks.com>
> wrote:
>
> Can you please include the complete stack trace and not just the root. Also,
> have you set fs.default.name to a hdfs location like hdfs://localhost:9000 ?
>
> Thanks
> Hemanth
>
> On Wednesday, February 13, 2013, Alex Thieme wrote:
>>
>> Thanks for the prompt reply and I'm sorry I forgot to include the
>> exception. My bad. I've included it below. There certainly appears to be a
>> server running on localhost:9001. At least, I was able to telnet to that
>> address. While in development, I'm treating the server on localhost as the
>> remote server. Moving to production, there'd obviously be a different remote
>> server address configured.
>>
>> Root Exception stack trace:
>> java.io.EOFException
>> at java.io.DataInputStream.readInt(DataInputStream.java:375)
>> at
>> org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
>> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
>> + 3 more (set debug level logging or '-Dmule.verbose.exceptions=true'
>> for everything)
>>
>> ********************************************************************************
>>
>> On Feb 12, 2013, at 4:22 PM, Nitin Pawar <ni...@gmail.com> wrote:
>>
>> conf.set("mapred.job.tracker", "localhost:9001");
>>
>> this means that your jobtracker is on port 9001 on localhost
>>
>> if you change it to the remote host and thats the port its running on then
>> it should work as expected
>>
>> whats the exception you are getting?
>>
>>
>> On Wed, Feb 13, 2013 at 2:41 AM, Alex Thieme <at...@athieme.com> wrote:
>>
>> I apologize for asking what seems to be such a basic question, but I would
>> use some help with submitting a job to a remote server.
>>
>> I have downloaded and installed hadoop locally in pseudo-distributed mode.
>> I have written some Java code to submit a job.
>>
>> Here's the org.apache.hadoop.util.Tool and
>> org.apache.hadoop.mapreduce.Mapper I've written.
>>
>> If I enable the conf.set("mapred.job.tracker", "localhost:9001") line,
>> then I get the exception included below.
>>
>> If that line is disabled, then the job is completed. However, in reviewing
>> the hadoop server administration page
>> (http://localhost:50030/jobtracker.jsp) I don't see the job as processed by
>> the server. Instead, I wonder if my Java code is simply running the
>> necessary mapper Java code, bypassing the locally installed server.
>>
>> Thanks in advance.
>>
>> Alex
>>
>> public class OfflineDataTool extends Configured implements Tool {
>>
>> public int run(final String[] args) throws Exception {
>> final Configuration conf = getConf();
>> //conf.set("mapred.job.tracker", "localhost:9001");
>>
>> final Job job = new Job(conf);
>> job.setJarByClass(getClass());
>> job.setJobName(getClass().getName());
>>
>> job.setMapperClass(OfflineDataMapper.class);
>>
>> job.setInputFormatClass(TextInputFormat.class);
>>
>> job.setMapOutputKeyClass(Text.class);
>> job.setMapOutputValueClass(Text.class);
>>
>> job.setOutputKeyClass(Text.class);
>> job.setOutputValueClass(Text.class);
>>
>> FileInputFormat.addInputPath(job, new
>> org.apache.hadoop.fs.Path(args[0]));
>>
>> final org.apache.hadoop.fs.Path output = new org.a
>
>
--
Harsh J
Re: Java submit job to remote server
Posted by Harsh J <ha...@cloudera.com>.
Hi Alex,
There can be two reasons here: (a) your client libs and the server
libs have a version mismatch that includes incompatible RPC protocol
changes, making it impossible for communication to happen, or (b) the
port you are connecting to in your app, is not really the JobTracker
port.
For (a), fixing dependencies in the client runtime/project to match
the version of hadoop deployed on the server would usually fix it.
For (b), inspecting the server's core-site.xml (For the
fs.default.name port, which is the NN's port) and mapred-site.xml (For
the mapred.job.tracker port, which is the JT's port), would help you
figure out what the deployment looks like and fix the port configs in
code as well, to connect to the right one.
Does either of this help?
On Fri, Feb 15, 2013 at 6:22 AM, Alex Thieme <at...@athieme.com> wrote:
> Any thoughts on why my connection to the hadoop server fails? An help
> provided would be greatly appreciated.
>
> Alex Thieme
> athieme@athieme.com
> 508-361-2788
>
>
> On Feb 13, 2013, at 1:41 PM, Alex Thieme <at...@athieme.com> wrote:
>
> It appears this is the full extent of the stack trace. Anything prior to the
> org.apache.hadoop calls are from my container where hadoop is called from.
>
> Caused by: java.io.IOException: Call to /127.0.0.1:9001 failed on local
> exception: java.io.EOFException
> at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
> at org.apache.hadoop.ipc.Client.call(Client.java:743)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at org.apache.hadoop.mapred.$Proxy55.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
> at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:429)
> at org.apache.hadoop.mapred.JobClient.init(JobClient.java:423)
> at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:410)
> at org.apache.hadoop.mapreduce.Job.<init>(Job.java:50)
> at com.allenabi.sherlock.graph.OfflineDataTool.run(OfflineDataTool.java:25)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at
> com.allenabi.sherlock.graph.OfflineDataComponent.submitJob(OfflineDataComponent.java:67)
> ... 64 more
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
>
> Alex Thieme
> athieme@athieme.com
> 508-361-2788
> In
>
> On Feb 12, 2013, at 8:16 PM, Hemanth Yamijala <yh...@thoughtworks.com>
> wrote:
>
> Can you please include the complete stack trace and not just the root. Also,
> have you set fs.default.name to a hdfs location like hdfs://localhost:9000 ?
>
> Thanks
> Hemanth
>
> On Wednesday, February 13, 2013, Alex Thieme wrote:
>>
>> Thanks for the prompt reply and I'm sorry I forgot to include the
>> exception. My bad. I've included it below. There certainly appears to be a
>> server running on localhost:9001. At least, I was able to telnet to that
>> address. While in development, I'm treating the server on localhost as the
>> remote server. Moving to production, there'd obviously be a different remote
>> server address configured.
>>
>> Root Exception stack trace:
>> java.io.EOFException
>> at java.io.DataInputStream.readInt(DataInputStream.java:375)
>> at
>> org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
>> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
>> + 3 more (set debug level logging or '-Dmule.verbose.exceptions=true'
>> for everything)
>>
>> ********************************************************************************
>>
>> On Feb 12, 2013, at 4:22 PM, Nitin Pawar <ni...@gmail.com> wrote:
>>
>> conf.set("mapred.job.tracker", "localhost:9001");
>>
>> this means that your jobtracker is on port 9001 on localhost
>>
>> if you change it to the remote host and thats the port its running on then
>> it should work as expected
>>
>> whats the exception you are getting?
>>
>>
>> On Wed, Feb 13, 2013 at 2:41 AM, Alex Thieme <at...@athieme.com> wrote:
>>
>> I apologize for asking what seems to be such a basic question, but I would
>> use some help with submitting a job to a remote server.
>>
>> I have downloaded and installed hadoop locally in pseudo-distributed mode.
>> I have written some Java code to submit a job.
>>
>> Here's the org.apache.hadoop.util.Tool and
>> org.apache.hadoop.mapreduce.Mapper I've written.
>>
>> If I enable the conf.set("mapred.job.tracker", "localhost:9001") line,
>> then I get the exception included below.
>>
>> If that line is disabled, then the job is completed. However, in reviewing
>> the hadoop server administration page
>> (http://localhost:50030/jobtracker.jsp) I don't see the job as processed by
>> the server. Instead, I wonder if my Java code is simply running the
>> necessary mapper Java code, bypassing the locally installed server.
>>
>> Thanks in advance.
>>
>> Alex
>>
>> public class OfflineDataTool extends Configured implements Tool {
>>
>> public int run(final String[] args) throws Exception {
>> final Configuration conf = getConf();
>> //conf.set("mapred.job.tracker", "localhost:9001");
>>
>> final Job job = new Job(conf);
>> job.setJarByClass(getClass());
>> job.setJobName(getClass().getName());
>>
>> job.setMapperClass(OfflineDataMapper.class);
>>
>> job.setInputFormatClass(TextInputFormat.class);
>>
>> job.setMapOutputKeyClass(Text.class);
>> job.setMapOutputValueClass(Text.class);
>>
>> job.setOutputKeyClass(Text.class);
>> job.setOutputValueClass(Text.class);
>>
>> FileInputFormat.addInputPath(job, new
>> org.apache.hadoop.fs.Path(args[0]));
>>
>> final org.apache.hadoop.fs.Path output = new org.a
>
>
--
Harsh J
Re: Java submit job to remote server
Posted by Alex Thieme <at...@athieme.com>.
Any thoughts on why my connection to the hadoop server fails? An help provided would be greatly appreciated.
Alex Thieme
athieme@athieme.com
508-361-2788
On Feb 13, 2013, at 1:41 PM, Alex Thieme <at...@athieme.com> wrote:
> It appears this is the full extent of the stack trace. Anything prior to the org.apache.hadoop calls are from my container where hadoop is called from.
>
> Caused by: java.io.IOException: Call to /127.0.0.1:9001 failed on local exception: java.io.EOFException
> at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
> at org.apache.hadoop.ipc.Client.call(Client.java:743)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at org.apache.hadoop.mapred.$Proxy55.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
> at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:429)
> at org.apache.hadoop.mapred.JobClient.init(JobClient.java:423)
> at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:410)
> at org.apache.hadoop.mapreduce.Job.<init>(Job.java:50)
> at com.allenabi.sherlock.graph.OfflineDataTool.run(OfflineDataTool.java:25)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at com.allenabi.sherlock.graph.OfflineDataComponent.submitJob(OfflineDataComponent.java:67)
> ... 64 more
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
>
> Alex Thieme
> athieme@athieme.com
> 508-361-2788
> In
>
> On Feb 12, 2013, at 8:16 PM, Hemanth Yamijala <yh...@thoughtworks.com> wrote:
>
>> Can you please include the complete stack trace and not just the root. Also, have you set fs.default.name to a hdfs location like hdfs://localhost:9000 ?
>>
>> Thanks
>> Hemanth
>>
>> On Wednesday, February 13, 2013, Alex Thieme wrote:
>>> Thanks for the prompt reply and I'm sorry I forgot to include the exception. My bad. I've included it below. There certainly appears to be a server running on localhost:9001. At least, I was able to telnet to that address. While in development, I'm treating the server on localhost as the remote server. Moving to production, there'd obviously be a different remote server address configured.
>>>
>>> Root Exception stack trace:
>>> java.io.EOFException
>>> at java.io.DataInputStream.readInt(DataInputStream.java:375)
>>> at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
>>> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
>>> + 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
>>> ********************************************************************************
>>>
>>> On Feb 12, 2013, at 4:22 PM, Nitin Pawar <ni...@gmail.com> wrote:
>>>
>>>> conf.set("mapred.job.tracker", "localhost:9001");
>>>>
>>>> this means that your jobtracker is on port 9001 on localhost
>>>>
>>>> if you change it to the remote host and thats the port its running on then it should work as expected
>>>>
>>>> whats the exception you are getting?
>>>>
>>>>
>>>> On Wed, Feb 13, 2013 at 2:41 AM, Alex Thieme <at...@athieme.com> wrote:
>>>> I apologize for asking what seems to be such a basic question, but I would use some help with submitting a job to a remote server.
>>>>
>>>> I have downloaded and installed hadoop locally in pseudo-distributed mode. I have written some Java code to submit a job.
>>>>
>>>> Here's the org.apache.hadoop.util.Tool and org.apache.hadoop.mapreduce.Mapper I've written.
>>>>
>>>> If I enable the conf.set("mapred.job.tracker", "localhost:9001") line, then I get the exception included below.
>>>>
>>>> If that line is disabled, then the job is completed. However, in reviewing the hadoop server administration page (http://localhost:50030/jobtracker.jsp) I don't see the job as processed by the server. Instead, I wonder if my Java code is simply running the necessary mapper Java code, bypassing the locally installed server.
>>>>
>>>> Thanks in advance.
>>>>
>>>> Alex
>>>>
>>>> public class OfflineDataTool extends Configured implements Tool {
>>>>
>>>> public int run(final String[] args) throws Exception {
>>>> final Configuration conf = getConf();
>>>> //conf.set("mapred.job.tracker", "localhost:9001");
>>>>
>>>> final Job job = new Job(conf);
>>>> job.setJarByClass(getClass());
>>>> job.setJobName(getClass().getName());
>>>>
>>>> job.setMapperClass(OfflineDataMapper.class);
>>>>
>>>> job.setInputFormatClass(TextInputFormat.class);
>>>>
>>>> job.setMapOutputKeyClass(Text.class);
>>>> job.setMapOutputValueClass(Text.class);
>>>>
>>>> job.setOutputKeyClass(Text.class);
>>>> job.setOutputValueClass(Text.class);
>>>>
>>>> FileInputFormat.addInputPath(job, new org.apache.hadoop.fs.Path(args[0]));
>>>>
>>>> final org.apache.hadoop.fs.Path output = new org.a
>
Re: Java submit job to remote server
Posted by Alex Thieme <at...@athieme.com>.
Any thoughts on why my connection to the hadoop server fails? An help provided would be greatly appreciated.
Alex Thieme
athieme@athieme.com
508-361-2788
On Feb 13, 2013, at 1:41 PM, Alex Thieme <at...@athieme.com> wrote:
> It appears this is the full extent of the stack trace. Anything prior to the org.apache.hadoop calls are from my container where hadoop is called from.
>
> Caused by: java.io.IOException: Call to /127.0.0.1:9001 failed on local exception: java.io.EOFException
> at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
> at org.apache.hadoop.ipc.Client.call(Client.java:743)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at org.apache.hadoop.mapred.$Proxy55.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
> at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:429)
> at org.apache.hadoop.mapred.JobClient.init(JobClient.java:423)
> at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:410)
> at org.apache.hadoop.mapreduce.Job.<init>(Job.java:50)
> at com.allenabi.sherlock.graph.OfflineDataTool.run(OfflineDataTool.java:25)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at com.allenabi.sherlock.graph.OfflineDataComponent.submitJob(OfflineDataComponent.java:67)
> ... 64 more
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
>
> Alex Thieme
> athieme@athieme.com
> 508-361-2788
> In
>
> On Feb 12, 2013, at 8:16 PM, Hemanth Yamijala <yh...@thoughtworks.com> wrote:
>
>> Can you please include the complete stack trace and not just the root. Also, have you set fs.default.name to a hdfs location like hdfs://localhost:9000 ?
>>
>> Thanks
>> Hemanth
>>
>> On Wednesday, February 13, 2013, Alex Thieme wrote:
>>> Thanks for the prompt reply and I'm sorry I forgot to include the exception. My bad. I've included it below. There certainly appears to be a server running on localhost:9001. At least, I was able to telnet to that address. While in development, I'm treating the server on localhost as the remote server. Moving to production, there'd obviously be a different remote server address configured.
>>>
>>> Root Exception stack trace:
>>> java.io.EOFException
>>> at java.io.DataInputStream.readInt(DataInputStream.java:375)
>>> at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
>>> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
>>> + 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
>>> ********************************************************************************
>>>
>>> On Feb 12, 2013, at 4:22 PM, Nitin Pawar <ni...@gmail.com> wrote:
>>>
>>>> conf.set("mapred.job.tracker", "localhost:9001");
>>>>
>>>> this means that your jobtracker is on port 9001 on localhost
>>>>
>>>> if you change it to the remote host and thats the port its running on then it should work as expected
>>>>
>>>> whats the exception you are getting?
>>>>
>>>>
>>>> On Wed, Feb 13, 2013 at 2:41 AM, Alex Thieme <at...@athieme.com> wrote:
>>>> I apologize for asking what seems to be such a basic question, but I would use some help with submitting a job to a remote server.
>>>>
>>>> I have downloaded and installed hadoop locally in pseudo-distributed mode. I have written some Java code to submit a job.
>>>>
>>>> Here's the org.apache.hadoop.util.Tool and org.apache.hadoop.mapreduce.Mapper I've written.
>>>>
>>>> If I enable the conf.set("mapred.job.tracker", "localhost:9001") line, then I get the exception included below.
>>>>
>>>> If that line is disabled, then the job is completed. However, in reviewing the hadoop server administration page (http://localhost:50030/jobtracker.jsp) I don't see the job as processed by the server. Instead, I wonder if my Java code is simply running the necessary mapper Java code, bypassing the locally installed server.
>>>>
>>>> Thanks in advance.
>>>>
>>>> Alex
>>>>
>>>> public class OfflineDataTool extends Configured implements Tool {
>>>>
>>>> public int run(final String[] args) throws Exception {
>>>> final Configuration conf = getConf();
>>>> //conf.set("mapred.job.tracker", "localhost:9001");
>>>>
>>>> final Job job = new Job(conf);
>>>> job.setJarByClass(getClass());
>>>> job.setJobName(getClass().getName());
>>>>
>>>> job.setMapperClass(OfflineDataMapper.class);
>>>>
>>>> job.setInputFormatClass(TextInputFormat.class);
>>>>
>>>> job.setMapOutputKeyClass(Text.class);
>>>> job.setMapOutputValueClass(Text.class);
>>>>
>>>> job.setOutputKeyClass(Text.class);
>>>> job.setOutputValueClass(Text.class);
>>>>
>>>> FileInputFormat.addInputPath(job, new org.apache.hadoop.fs.Path(args[0]));
>>>>
>>>> final org.apache.hadoop.fs.Path output = new org.a
>
Re: Java submit job to remote server
Posted by Alex Thieme <at...@athieme.com>.
Any thoughts on why my connection to the hadoop server fails? An help provided would be greatly appreciated.
Alex Thieme
athieme@athieme.com
508-361-2788
On Feb 13, 2013, at 1:41 PM, Alex Thieme <at...@athieme.com> wrote:
> It appears this is the full extent of the stack trace. Anything prior to the org.apache.hadoop calls are from my container where hadoop is called from.
>
> Caused by: java.io.IOException: Call to /127.0.0.1:9001 failed on local exception: java.io.EOFException
> at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
> at org.apache.hadoop.ipc.Client.call(Client.java:743)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at org.apache.hadoop.mapred.$Proxy55.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
> at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:429)
> at org.apache.hadoop.mapred.JobClient.init(JobClient.java:423)
> at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:410)
> at org.apache.hadoop.mapreduce.Job.<init>(Job.java:50)
> at com.allenabi.sherlock.graph.OfflineDataTool.run(OfflineDataTool.java:25)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at com.allenabi.sherlock.graph.OfflineDataComponent.submitJob(OfflineDataComponent.java:67)
> ... 64 more
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
>
> Alex Thieme
> athieme@athieme.com
> 508-361-2788
> In
>
> On Feb 12, 2013, at 8:16 PM, Hemanth Yamijala <yh...@thoughtworks.com> wrote:
>
>> Can you please include the complete stack trace and not just the root. Also, have you set fs.default.name to a hdfs location like hdfs://localhost:9000 ?
>>
>> Thanks
>> Hemanth
>>
>> On Wednesday, February 13, 2013, Alex Thieme wrote:
>>> Thanks for the prompt reply and I'm sorry I forgot to include the exception. My bad. I've included it below. There certainly appears to be a server running on localhost:9001. At least, I was able to telnet to that address. While in development, I'm treating the server on localhost as the remote server. Moving to production, there'd obviously be a different remote server address configured.
>>>
>>> Root Exception stack trace:
>>> java.io.EOFException
>>> at java.io.DataInputStream.readInt(DataInputStream.java:375)
>>> at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
>>> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
>>> + 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
>>> ********************************************************************************
>>>
>>> On Feb 12, 2013, at 4:22 PM, Nitin Pawar <ni...@gmail.com> wrote:
>>>
>>>> conf.set("mapred.job.tracker", "localhost:9001");
>>>>
>>>> this means that your jobtracker is on port 9001 on localhost
>>>>
>>>> if you change it to the remote host and thats the port its running on then it should work as expected
>>>>
>>>> whats the exception you are getting?
>>>>
>>>>
>>>> On Wed, Feb 13, 2013 at 2:41 AM, Alex Thieme <at...@athieme.com> wrote:
>>>> I apologize for asking what seems to be such a basic question, but I would use some help with submitting a job to a remote server.
>>>>
>>>> I have downloaded and installed hadoop locally in pseudo-distributed mode. I have written some Java code to submit a job.
>>>>
>>>> Here's the org.apache.hadoop.util.Tool and org.apache.hadoop.mapreduce.Mapper I've written.
>>>>
>>>> If I enable the conf.set("mapred.job.tracker", "localhost:9001") line, then I get the exception included below.
>>>>
>>>> If that line is disabled, then the job is completed. However, in reviewing the hadoop server administration page (http://localhost:50030/jobtracker.jsp) I don't see the job as processed by the server. Instead, I wonder if my Java code is simply running the necessary mapper Java code, bypassing the locally installed server.
>>>>
>>>> Thanks in advance.
>>>>
>>>> Alex
>>>>
>>>> public class OfflineDataTool extends Configured implements Tool {
>>>>
>>>> public int run(final String[] args) throws Exception {
>>>> final Configuration conf = getConf();
>>>> //conf.set("mapred.job.tracker", "localhost:9001");
>>>>
>>>> final Job job = new Job(conf);
>>>> job.setJarByClass(getClass());
>>>> job.setJobName(getClass().getName());
>>>>
>>>> job.setMapperClass(OfflineDataMapper.class);
>>>>
>>>> job.setInputFormatClass(TextInputFormat.class);
>>>>
>>>> job.setMapOutputKeyClass(Text.class);
>>>> job.setMapOutputValueClass(Text.class);
>>>>
>>>> job.setOutputKeyClass(Text.class);
>>>> job.setOutputValueClass(Text.class);
>>>>
>>>> FileInputFormat.addInputPath(job, new org.apache.hadoop.fs.Path(args[0]));
>>>>
>>>> final org.apache.hadoop.fs.Path output = new org.a
>
Re: Java submit job to remote server
Posted by Alex Thieme <at...@athieme.com>.
Any thoughts on why my connection to the hadoop server fails? An help provided would be greatly appreciated.
Alex Thieme
athieme@athieme.com
508-361-2788
On Feb 13, 2013, at 1:41 PM, Alex Thieme <at...@athieme.com> wrote:
> It appears this is the full extent of the stack trace. Anything prior to the org.apache.hadoop calls are from my container where hadoop is called from.
>
> Caused by: java.io.IOException: Call to /127.0.0.1:9001 failed on local exception: java.io.EOFException
> at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
> at org.apache.hadoop.ipc.Client.call(Client.java:743)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at org.apache.hadoop.mapred.$Proxy55.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
> at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:429)
> at org.apache.hadoop.mapred.JobClient.init(JobClient.java:423)
> at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:410)
> at org.apache.hadoop.mapreduce.Job.<init>(Job.java:50)
> at com.allenabi.sherlock.graph.OfflineDataTool.run(OfflineDataTool.java:25)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at com.allenabi.sherlock.graph.OfflineDataComponent.submitJob(OfflineDataComponent.java:67)
> ... 64 more
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
>
> Alex Thieme
> athieme@athieme.com
> 508-361-2788
> In
>
> On Feb 12, 2013, at 8:16 PM, Hemanth Yamijala <yh...@thoughtworks.com> wrote:
>
>> Can you please include the complete stack trace and not just the root. Also, have you set fs.default.name to a hdfs location like hdfs://localhost:9000 ?
>>
>> Thanks
>> Hemanth
>>
>> On Wednesday, February 13, 2013, Alex Thieme wrote:
>>> Thanks for the prompt reply and I'm sorry I forgot to include the exception. My bad. I've included it below. There certainly appears to be a server running on localhost:9001. At least, I was able to telnet to that address. While in development, I'm treating the server on localhost as the remote server. Moving to production, there'd obviously be a different remote server address configured.
>>>
>>> Root Exception stack trace:
>>> java.io.EOFException
>>> at java.io.DataInputStream.readInt(DataInputStream.java:375)
>>> at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
>>> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
>>> + 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
>>> ********************************************************************************
>>>
>>> On Feb 12, 2013, at 4:22 PM, Nitin Pawar <ni...@gmail.com> wrote:
>>>
>>>> conf.set("mapred.job.tracker", "localhost:9001");
>>>>
>>>> this means that your jobtracker is on port 9001 on localhost
>>>>
>>>> if you change it to the remote host and thats the port its running on then it should work as expected
>>>>
>>>> whats the exception you are getting?
>>>>
>>>>
>>>> On Wed, Feb 13, 2013 at 2:41 AM, Alex Thieme <at...@athieme.com> wrote:
>>>> I apologize for asking what seems to be such a basic question, but I would use some help with submitting a job to a remote server.
>>>>
>>>> I have downloaded and installed hadoop locally in pseudo-distributed mode. I have written some Java code to submit a job.
>>>>
>>>> Here's the org.apache.hadoop.util.Tool and org.apache.hadoop.mapreduce.Mapper I've written.
>>>>
>>>> If I enable the conf.set("mapred.job.tracker", "localhost:9001") line, then I get the exception included below.
>>>>
>>>> If that line is disabled, then the job is completed. However, in reviewing the hadoop server administration page (http://localhost:50030/jobtracker.jsp) I don't see the job as processed by the server. Instead, I wonder if my Java code is simply running the necessary mapper Java code, bypassing the locally installed server.
>>>>
>>>> Thanks in advance.
>>>>
>>>> Alex
>>>>
>>>> public class OfflineDataTool extends Configured implements Tool {
>>>>
>>>> public int run(final String[] args) throws Exception {
>>>> final Configuration conf = getConf();
>>>> //conf.set("mapred.job.tracker", "localhost:9001");
>>>>
>>>> final Job job = new Job(conf);
>>>> job.setJarByClass(getClass());
>>>> job.setJobName(getClass().getName());
>>>>
>>>> job.setMapperClass(OfflineDataMapper.class);
>>>>
>>>> job.setInputFormatClass(TextInputFormat.class);
>>>>
>>>> job.setMapOutputKeyClass(Text.class);
>>>> job.setMapOutputValueClass(Text.class);
>>>>
>>>> job.setOutputKeyClass(Text.class);
>>>> job.setOutputValueClass(Text.class);
>>>>
>>>> FileInputFormat.addInputPath(job, new org.apache.hadoop.fs.Path(args[0]));
>>>>
>>>> final org.apache.hadoop.fs.Path output = new org.a
>
Re: Java submit job to remote server
Posted by Alex Thieme <at...@athieme.com>.
It appears this is the full extent of the stack trace. Anything prior to the org.apache.hadoop calls are from my container where hadoop is called from.
Caused by: java.io.IOException: Call to /127.0.0.1:9001 failed on local exception: java.io.EOFException
at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
at org.apache.hadoop.ipc.Client.call(Client.java:743)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at org.apache.hadoop.mapred.$Proxy55.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:429)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:423)
at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:410)
at org.apache.hadoop.mapreduce.Job.<init>(Job.java:50)
at com.allenabi.sherlock.graph.OfflineDataTool.run(OfflineDataTool.java:25)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at com.allenabi.sherlock.graph.OfflineDataComponent.submitJob(OfflineDataComponent.java:67)
... 64 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
Alex Thieme
athieme@athieme.com
508-361-2788
In
On Feb 12, 2013, at 8:16 PM, Hemanth Yamijala <yh...@thoughtworks.com> wrote:
> Can you please include the complete stack trace and not just the root. Also, have you set fs.default.name to a hdfs location like hdfs://localhost:9000 ?
>
> Thanks
> Hemanth
>
> On Wednesday, February 13, 2013, Alex Thieme wrote:
> Thanks for the prompt reply and I'm sorry I forgot to include the exception. My bad. I've included it below. There certainly appears to be a server running on localhost:9001. At least, I was able to telnet to that address. While in development, I'm treating the server on localhost as the remote server. Moving to production, there'd obviously be a different remote server address configured.
>
> Root Exception stack trace:
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
> + 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
> ********************************************************************************
>
> On Feb 12, 2013, at 4:22 PM, Nitin Pawar <ni...@gmail.com> wrote:
>
>> conf.set("mapred.job.tracker", "localhost:9001");
>>
>> this means that your jobtracker is on port 9001 on localhost
>>
>> if you change it to the remote host and thats the port its running on then it should work as expected
>>
>> whats the exception you are getting?
>>
>>
>> On Wed, Feb 13, 2013 at 2:41 AM, Alex Thieme <at...@athieme.com> wrote:
>> I apologize for asking what seems to be such a basic question, but I would use some help with submitting a job to a remote server.
>>
>> I have downloaded and installed hadoop locally in pseudo-distributed mode. I have written some Java code to submit a job.
>>
>> Here's the org.apache.hadoop.util.Tool and org.apache.hadoop.mapreduce.Mapper I've written.
>>
>> If I enable the conf.set("mapred.job.tracker", "localhost:9001") line, then I get the exception included below.
>>
>> If that line is disabled, then the job is completed. However, in reviewing the hadoop server administration page (http://localhost:50030/jobtracker.jsp) I don't see the job as processed by the server. Instead, I wonder if my Java code is simply running the necessary mapper Java code, bypassing the locally installed server.
>>
>> Thanks in advance.
>>
>> Alex
>>
>> public class OfflineDataTool extends Configured implements Tool {
>>
>> public int run(final String[] args) throws Exception {
>> final Configuration conf = getConf();
>> //conf.set("mapred.job.tracker", "localhost:9001");
>>
>> final Job job = new Job(conf);
>> job.setJarByClass(getClass());
>> job.setJobName(getClass().getName());
>>
>> job.setMapperClass(OfflineDataMapper.class);
>>
>> job.setInputFormatClass(TextInputFormat.class);
>>
>> job.setMapOutputKeyClass(Text.class);
>> job.setMapOutputValueClass(Text.class);
>>
>> job.setOutputKeyClass(Text.class);
>> job.setOutputValueClass(Text.class);
>>
>> FileInputFormat.addInputPath(job, new org.apache.hadoop.fs.Path(args[0]));
>>
>> final org.apache.hadoop.fs.Path output = new org.a
Re: Java submit job to remote server
Posted by Alex Thieme <at...@athieme.com>.
It appears this is the full extent of the stack trace. Anything prior to the org.apache.hadoop calls are from my container where hadoop is called from.
Caused by: java.io.IOException: Call to /127.0.0.1:9001 failed on local exception: java.io.EOFException
at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
at org.apache.hadoop.ipc.Client.call(Client.java:743)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at org.apache.hadoop.mapred.$Proxy55.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:429)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:423)
at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:410)
at org.apache.hadoop.mapreduce.Job.<init>(Job.java:50)
at com.allenabi.sherlock.graph.OfflineDataTool.run(OfflineDataTool.java:25)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at com.allenabi.sherlock.graph.OfflineDataComponent.submitJob(OfflineDataComponent.java:67)
... 64 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
Alex Thieme
athieme@athieme.com
508-361-2788
In
On Feb 12, 2013, at 8:16 PM, Hemanth Yamijala <yh...@thoughtworks.com> wrote:
> Can you please include the complete stack trace and not just the root. Also, have you set fs.default.name to a hdfs location like hdfs://localhost:9000 ?
>
> Thanks
> Hemanth
>
> On Wednesday, February 13, 2013, Alex Thieme wrote:
> Thanks for the prompt reply and I'm sorry I forgot to include the exception. My bad. I've included it below. There certainly appears to be a server running on localhost:9001. At least, I was able to telnet to that address. While in development, I'm treating the server on localhost as the remote server. Moving to production, there'd obviously be a different remote server address configured.
>
> Root Exception stack trace:
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
> + 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
> ********************************************************************************
>
> On Feb 12, 2013, at 4:22 PM, Nitin Pawar <ni...@gmail.com> wrote:
>
>> conf.set("mapred.job.tracker", "localhost:9001");
>>
>> this means that your jobtracker is on port 9001 on localhost
>>
>> if you change it to the remote host and thats the port its running on then it should work as expected
>>
>> whats the exception you are getting?
>>
>>
>> On Wed, Feb 13, 2013 at 2:41 AM, Alex Thieme <at...@athieme.com> wrote:
>> I apologize for asking what seems to be such a basic question, but I would use some help with submitting a job to a remote server.
>>
>> I have downloaded and installed hadoop locally in pseudo-distributed mode. I have written some Java code to submit a job.
>>
>> Here's the org.apache.hadoop.util.Tool and org.apache.hadoop.mapreduce.Mapper I've written.
>>
>> If I enable the conf.set("mapred.job.tracker", "localhost:9001") line, then I get the exception included below.
>>
>> If that line is disabled, then the job is completed. However, in reviewing the hadoop server administration page (http://localhost:50030/jobtracker.jsp) I don't see the job as processed by the server. Instead, I wonder if my Java code is simply running the necessary mapper Java code, bypassing the locally installed server.
>>
>> Thanks in advance.
>>
>> Alex
>>
>> public class OfflineDataTool extends Configured implements Tool {
>>
>> public int run(final String[] args) throws Exception {
>> final Configuration conf = getConf();
>> //conf.set("mapred.job.tracker", "localhost:9001");
>>
>> final Job job = new Job(conf);
>> job.setJarByClass(getClass());
>> job.setJobName(getClass().getName());
>>
>> job.setMapperClass(OfflineDataMapper.class);
>>
>> job.setInputFormatClass(TextInputFormat.class);
>>
>> job.setMapOutputKeyClass(Text.class);
>> job.setMapOutputValueClass(Text.class);
>>
>> job.setOutputKeyClass(Text.class);
>> job.setOutputValueClass(Text.class);
>>
>> FileInputFormat.addInputPath(job, new org.apache.hadoop.fs.Path(args[0]));
>>
>> final org.apache.hadoop.fs.Path output = new org.a
Re: Java submit job to remote server
Posted by Alex Thieme <at...@athieme.com>.
It appears this is the full extent of the stack trace. Anything prior to the org.apache.hadoop calls are from my container where hadoop is called from.
Caused by: java.io.IOException: Call to /127.0.0.1:9001 failed on local exception: java.io.EOFException
at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
at org.apache.hadoop.ipc.Client.call(Client.java:743)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at org.apache.hadoop.mapred.$Proxy55.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:429)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:423)
at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:410)
at org.apache.hadoop.mapreduce.Job.<init>(Job.java:50)
at com.allenabi.sherlock.graph.OfflineDataTool.run(OfflineDataTool.java:25)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at com.allenabi.sherlock.graph.OfflineDataComponent.submitJob(OfflineDataComponent.java:67)
... 64 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
Alex Thieme
athieme@athieme.com
508-361-2788
In
On Feb 12, 2013, at 8:16 PM, Hemanth Yamijala <yh...@thoughtworks.com> wrote:
> Can you please include the complete stack trace and not just the root. Also, have you set fs.default.name to a hdfs location like hdfs://localhost:9000 ?
>
> Thanks
> Hemanth
>
> On Wednesday, February 13, 2013, Alex Thieme wrote:
> Thanks for the prompt reply and I'm sorry I forgot to include the exception. My bad. I've included it below. There certainly appears to be a server running on localhost:9001. At least, I was able to telnet to that address. While in development, I'm treating the server on localhost as the remote server. Moving to production, there'd obviously be a different remote server address configured.
>
> Root Exception stack trace:
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
> + 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
> ********************************************************************************
>
> On Feb 12, 2013, at 4:22 PM, Nitin Pawar <ni...@gmail.com> wrote:
>
>> conf.set("mapred.job.tracker", "localhost:9001");
>>
>> this means that your jobtracker is on port 9001 on localhost
>>
>> if you change it to the remote host and thats the port its running on then it should work as expected
>>
>> whats the exception you are getting?
>>
>>
>> On Wed, Feb 13, 2013 at 2:41 AM, Alex Thieme <at...@athieme.com> wrote:
>> I apologize for asking what seems to be such a basic question, but I would use some help with submitting a job to a remote server.
>>
>> I have downloaded and installed hadoop locally in pseudo-distributed mode. I have written some Java code to submit a job.
>>
>> Here's the org.apache.hadoop.util.Tool and org.apache.hadoop.mapreduce.Mapper I've written.
>>
>> If I enable the conf.set("mapred.job.tracker", "localhost:9001") line, then I get the exception included below.
>>
>> If that line is disabled, then the job is completed. However, in reviewing the hadoop server administration page (http://localhost:50030/jobtracker.jsp) I don't see the job as processed by the server. Instead, I wonder if my Java code is simply running the necessary mapper Java code, bypassing the locally installed server.
>>
>> Thanks in advance.
>>
>> Alex
>>
>> public class OfflineDataTool extends Configured implements Tool {
>>
>> public int run(final String[] args) throws Exception {
>> final Configuration conf = getConf();
>> //conf.set("mapred.job.tracker", "localhost:9001");
>>
>> final Job job = new Job(conf);
>> job.setJarByClass(getClass());
>> job.setJobName(getClass().getName());
>>
>> job.setMapperClass(OfflineDataMapper.class);
>>
>> job.setInputFormatClass(TextInputFormat.class);
>>
>> job.setMapOutputKeyClass(Text.class);
>> job.setMapOutputValueClass(Text.class);
>>
>> job.setOutputKeyClass(Text.class);
>> job.setOutputValueClass(Text.class);
>>
>> FileInputFormat.addInputPath(job, new org.apache.hadoop.fs.Path(args[0]));
>>
>> final org.apache.hadoop.fs.Path output = new org.a
Re: Java submit job to remote server
Posted by Alex Thieme <at...@athieme.com>.
It appears this is the full extent of the stack trace. Anything prior to the org.apache.hadoop calls are from my container where hadoop is called from.
Caused by: java.io.IOException: Call to /127.0.0.1:9001 failed on local exception: java.io.EOFException
at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
at org.apache.hadoop.ipc.Client.call(Client.java:743)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at org.apache.hadoop.mapred.$Proxy55.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:429)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:423)
at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:410)
at org.apache.hadoop.mapreduce.Job.<init>(Job.java:50)
at com.allenabi.sherlock.graph.OfflineDataTool.run(OfflineDataTool.java:25)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at com.allenabi.sherlock.graph.OfflineDataComponent.submitJob(OfflineDataComponent.java:67)
... 64 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
Alex Thieme
athieme@athieme.com
508-361-2788
In
On Feb 12, 2013, at 8:16 PM, Hemanth Yamijala <yh...@thoughtworks.com> wrote:
> Can you please include the complete stack trace and not just the root. Also, have you set fs.default.name to a hdfs location like hdfs://localhost:9000 ?
>
> Thanks
> Hemanth
>
> On Wednesday, February 13, 2013, Alex Thieme wrote:
> Thanks for the prompt reply and I'm sorry I forgot to include the exception. My bad. I've included it below. There certainly appears to be a server running on localhost:9001. At least, I was able to telnet to that address. While in development, I'm treating the server on localhost as the remote server. Moving to production, there'd obviously be a different remote server address configured.
>
> Root Exception stack trace:
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
> + 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
> ********************************************************************************
>
> On Feb 12, 2013, at 4:22 PM, Nitin Pawar <ni...@gmail.com> wrote:
>
>> conf.set("mapred.job.tracker", "localhost:9001");
>>
>> this means that your jobtracker is on port 9001 on localhost
>>
>> if you change it to the remote host and thats the port its running on then it should work as expected
>>
>> whats the exception you are getting?
>>
>>
>> On Wed, Feb 13, 2013 at 2:41 AM, Alex Thieme <at...@athieme.com> wrote:
>> I apologize for asking what seems to be such a basic question, but I would use some help with submitting a job to a remote server.
>>
>> I have downloaded and installed hadoop locally in pseudo-distributed mode. I have written some Java code to submit a job.
>>
>> Here's the org.apache.hadoop.util.Tool and org.apache.hadoop.mapreduce.Mapper I've written.
>>
>> If I enable the conf.set("mapred.job.tracker", "localhost:9001") line, then I get the exception included below.
>>
>> If that line is disabled, then the job is completed. However, in reviewing the hadoop server administration page (http://localhost:50030/jobtracker.jsp) I don't see the job as processed by the server. Instead, I wonder if my Java code is simply running the necessary mapper Java code, bypassing the locally installed server.
>>
>> Thanks in advance.
>>
>> Alex
>>
>> public class OfflineDataTool extends Configured implements Tool {
>>
>> public int run(final String[] args) throws Exception {
>> final Configuration conf = getConf();
>> //conf.set("mapred.job.tracker", "localhost:9001");
>>
>> final Job job = new Job(conf);
>> job.setJarByClass(getClass());
>> job.setJobName(getClass().getName());
>>
>> job.setMapperClass(OfflineDataMapper.class);
>>
>> job.setInputFormatClass(TextInputFormat.class);
>>
>> job.setMapOutputKeyClass(Text.class);
>> job.setMapOutputValueClass(Text.class);
>>
>> job.setOutputKeyClass(Text.class);
>> job.setOutputValueClass(Text.class);
>>
>> FileInputFormat.addInputPath(job, new org.apache.hadoop.fs.Path(args[0]));
>>
>> final org.apache.hadoop.fs.Path output = new org.a
Re: Java submit job to remote server
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
Can you please include the complete stack trace and not just the root.
Also, have you set fs.default.name to a hdfs location like
hdfs://localhost:9000 ?
Thanks
Hemanth
On Wednesday, February 13, 2013, Alex Thieme wrote:
> Thanks for the prompt reply and I'm sorry I forgot to include the
> exception. My bad. I've included it below. There certainly appears to be a
> server running on localhost:9001. At least, I was able to telnet to that
> address. While in development, I'm treating the server on localhost as the
> remote server. Moving to production, there'd obviously be a different
> remote server address configured.
>
> Root Exception stack trace:
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
> + 3 more (set debug level logging or '-Dmule.verbose.exceptions=true'
> for everything)
>
> ********************************************************************************
>
> On Feb 12, 2013, at 4:22 PM, Nitin Pawar <ni...@gmail.com> wrote:
>
> conf.set("mapred.job.tracker", "localhost:9001");
>
> this means that your jobtracker is on port 9001 on localhost
>
> if you change it to the remote host and thats the port its running on then
> it should work as expected
>
> whats the exception you are getting?
>
>
> On Wed, Feb 13, 2013 at 2:41 AM, Alex Thieme <at...@athieme.com> wrote:
>
> I apologize for asking what seems to be such a basic question, but I would
> use some help with submitting a job to a remote server.
>
> I have downloaded and installed hadoop locally in pseudo-distributed mode.
> I have written some Java code to submit a job.
>
> Here's the org.apache.hadoop.util.Tool
> and org.apache.hadoop.mapreduce.Mapper I've written.
>
> If I enable the conf.set("mapred.job.tracker", "localhost:9001") line,
> then I get the exception included below.
>
> If that line is disabled, then the job is completed. However, in reviewing
> the hadoop server administration page (
> http://localhost:50030/jobtracker.jsp) I don't see the job as processed
> by the server. Instead, I wonder if my Java code is simply running the
> necessary mapper Java code, bypassing the locally installed server.
>
> Thanks in advance.
>
> Alex
>
> public class OfflineDataTool extends Configured implements Tool {
>
> public int run(final String[] args) throws Exception {
> final Configuration conf = getConf();
> //conf.set("mapred.job.tracker", "localhost:9001");
>
> final Job job = new Job(conf);
> job.setJarByClass(getClass());
> job.setJobName(getClass().getName());
>
> job.setMapperClass(OfflineDataMapper.class);
>
> job.setInputFormatClass(TextInputFormat.class);
>
> job.setMapOutputKeyClass(Text.class);
> job.setMapOutputValueClass(Text.class);
>
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(Text.class);
>
> FileInputFormat.addInputPath(job, new
> org.apache.hadoop.fs.Path(args[0]));
>
> final org.apache.hadoop.fs.Path output = new org.a
>
>
Re: Java submit job to remote server
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
Can you please include the complete stack trace and not just the root.
Also, have you set fs.default.name to a hdfs location like
hdfs://localhost:9000 ?
Thanks
Hemanth
On Wednesday, February 13, 2013, Alex Thieme wrote:
> Thanks for the prompt reply and I'm sorry I forgot to include the
> exception. My bad. I've included it below. There certainly appears to be a
> server running on localhost:9001. At least, I was able to telnet to that
> address. While in development, I'm treating the server on localhost as the
> remote server. Moving to production, there'd obviously be a different
> remote server address configured.
>
> Root Exception stack trace:
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
> + 3 more (set debug level logging or '-Dmule.verbose.exceptions=true'
> for everything)
>
> ********************************************************************************
>
> On Feb 12, 2013, at 4:22 PM, Nitin Pawar <ni...@gmail.com> wrote:
>
> conf.set("mapred.job.tracker", "localhost:9001");
>
> this means that your jobtracker is on port 9001 on localhost
>
> if you change it to the remote host and thats the port its running on then
> it should work as expected
>
> whats the exception you are getting?
>
>
> On Wed, Feb 13, 2013 at 2:41 AM, Alex Thieme <at...@athieme.com> wrote:
>
> I apologize for asking what seems to be such a basic question, but I would
> use some help with submitting a job to a remote server.
>
> I have downloaded and installed hadoop locally in pseudo-distributed mode.
> I have written some Java code to submit a job.
>
> Here's the org.apache.hadoop.util.Tool
> and org.apache.hadoop.mapreduce.Mapper I've written.
>
> If I enable the conf.set("mapred.job.tracker", "localhost:9001") line,
> then I get the exception included below.
>
> If that line is disabled, then the job is completed. However, in reviewing
> the hadoop server administration page (
> http://localhost:50030/jobtracker.jsp) I don't see the job as processed
> by the server. Instead, I wonder if my Java code is simply running the
> necessary mapper Java code, bypassing the locally installed server.
>
> Thanks in advance.
>
> Alex
>
> public class OfflineDataTool extends Configured implements Tool {
>
> public int run(final String[] args) throws Exception {
> final Configuration conf = getConf();
> //conf.set("mapred.job.tracker", "localhost:9001");
>
> final Job job = new Job(conf);
> job.setJarByClass(getClass());
> job.setJobName(getClass().getName());
>
> job.setMapperClass(OfflineDataMapper.class);
>
> job.setInputFormatClass(TextInputFormat.class);
>
> job.setMapOutputKeyClass(Text.class);
> job.setMapOutputValueClass(Text.class);
>
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(Text.class);
>
> FileInputFormat.addInputPath(job, new
> org.apache.hadoop.fs.Path(args[0]));
>
> final org.apache.hadoop.fs.Path output = new org.a
>
>
Re: Java submit job to remote server
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
Can you please include the complete stack trace and not just the root.
Also, have you set fs.default.name to a hdfs location like
hdfs://localhost:9000 ?
Thanks
Hemanth
On Wednesday, February 13, 2013, Alex Thieme wrote:
> Thanks for the prompt reply and I'm sorry I forgot to include the
> exception. My bad. I've included it below. There certainly appears to be a
> server running on localhost:9001. At least, I was able to telnet to that
> address. While in development, I'm treating the server on localhost as the
> remote server. Moving to production, there'd obviously be a different
> remote server address configured.
>
> Root Exception stack trace:
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
> + 3 more (set debug level logging or '-Dmule.verbose.exceptions=true'
> for everything)
>
> ********************************************************************************
>
> On Feb 12, 2013, at 4:22 PM, Nitin Pawar <ni...@gmail.com> wrote:
>
> conf.set("mapred.job.tracker", "localhost:9001");
>
> this means that your jobtracker is on port 9001 on localhost
>
> if you change it to the remote host and thats the port its running on then
> it should work as expected
>
> whats the exception you are getting?
>
>
> On Wed, Feb 13, 2013 at 2:41 AM, Alex Thieme <at...@athieme.com> wrote:
>
> I apologize for asking what seems to be such a basic question, but I would
> use some help with submitting a job to a remote server.
>
> I have downloaded and installed hadoop locally in pseudo-distributed mode.
> I have written some Java code to submit a job.
>
> Here's the org.apache.hadoop.util.Tool
> and org.apache.hadoop.mapreduce.Mapper I've written.
>
> If I enable the conf.set("mapred.job.tracker", "localhost:9001") line,
> then I get the exception included below.
>
> If that line is disabled, then the job is completed. However, in reviewing
> the hadoop server administration page (
> http://localhost:50030/jobtracker.jsp) I don't see the job as processed
> by the server. Instead, I wonder if my Java code is simply running the
> necessary mapper Java code, bypassing the locally installed server.
>
> Thanks in advance.
>
> Alex
>
> public class OfflineDataTool extends Configured implements Tool {
>
> public int run(final String[] args) throws Exception {
> final Configuration conf = getConf();
> //conf.set("mapred.job.tracker", "localhost:9001");
>
> final Job job = new Job(conf);
> job.setJarByClass(getClass());
> job.setJobName(getClass().getName());
>
> job.setMapperClass(OfflineDataMapper.class);
>
> job.setInputFormatClass(TextInputFormat.class);
>
> job.setMapOutputKeyClass(Text.class);
> job.setMapOutputValueClass(Text.class);
>
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(Text.class);
>
> FileInputFormat.addInputPath(job, new
> org.apache.hadoop.fs.Path(args[0]));
>
> final org.apache.hadoop.fs.Path output = new org.a
>
>
Re: Java submit job to remote server
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
Can you please include the complete stack trace and not just the root.
Also, have you set fs.default.name to a hdfs location like
hdfs://localhost:9000 ?
Thanks
Hemanth
On Wednesday, February 13, 2013, Alex Thieme wrote:
> Thanks for the prompt reply and I'm sorry I forgot to include the
> exception. My bad. I've included it below. There certainly appears to be a
> server running on localhost:9001. At least, I was able to telnet to that
> address. While in development, I'm treating the server on localhost as the
> remote server. Moving to production, there'd obviously be a different
> remote server address configured.
>
> Root Exception stack trace:
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
> + 3 more (set debug level logging or '-Dmule.verbose.exceptions=true'
> for everything)
>
> ********************************************************************************
>
> On Feb 12, 2013, at 4:22 PM, Nitin Pawar <ni...@gmail.com> wrote:
>
> conf.set("mapred.job.tracker", "localhost:9001");
>
> this means that your jobtracker is on port 9001 on localhost
>
> if you change it to the remote host and thats the port its running on then
> it should work as expected
>
> whats the exception you are getting?
>
>
> On Wed, Feb 13, 2013 at 2:41 AM, Alex Thieme <at...@athieme.com> wrote:
>
> I apologize for asking what seems to be such a basic question, but I would
> use some help with submitting a job to a remote server.
>
> I have downloaded and installed hadoop locally in pseudo-distributed mode.
> I have written some Java code to submit a job.
>
> Here's the org.apache.hadoop.util.Tool
> and org.apache.hadoop.mapreduce.Mapper I've written.
>
> If I enable the conf.set("mapred.job.tracker", "localhost:9001") line,
> then I get the exception included below.
>
> If that line is disabled, then the job is completed. However, in reviewing
> the hadoop server administration page (
> http://localhost:50030/jobtracker.jsp) I don't see the job as processed
> by the server. Instead, I wonder if my Java code is simply running the
> necessary mapper Java code, bypassing the locally installed server.
>
> Thanks in advance.
>
> Alex
>
> public class OfflineDataTool extends Configured implements Tool {
>
> public int run(final String[] args) throws Exception {
> final Configuration conf = getConf();
> //conf.set("mapred.job.tracker", "localhost:9001");
>
> final Job job = new Job(conf);
> job.setJarByClass(getClass());
> job.setJobName(getClass().getName());
>
> job.setMapperClass(OfflineDataMapper.class);
>
> job.setInputFormatClass(TextInputFormat.class);
>
> job.setMapOutputKeyClass(Text.class);
> job.setMapOutputValueClass(Text.class);
>
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(Text.class);
>
> FileInputFormat.addInputPath(job, new
> org.apache.hadoop.fs.Path(args[0]));
>
> final org.apache.hadoop.fs.Path output = new org.a
>
>
Re: Java submit job to remote server
Posted by Alex Thieme <at...@athieme.com>.
Thanks for the prompt reply and I'm sorry I forgot to include the exception. My bad. I've included it below. There certainly appears to be a server running on localhost:9001. At least, I was able to telnet to that address. While in development, I'm treating the server on localhost as the remote server. Moving to production, there'd obviously be a different remote server address configured.
Root Exception stack trace:
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
+ 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
********************************************************************************
On Feb 12, 2013, at 4:22 PM, Nitin Pawar <ni...@gmail.com> wrote:
> conf.set("mapred.job.tracker", "localhost:9001");
>
> this means that your jobtracker is on port 9001 on localhost
>
> if you change it to the remote host and thats the port its running on then it should work as expected
>
> whats the exception you are getting?
>
>
> On Wed, Feb 13, 2013 at 2:41 AM, Alex Thieme <at...@athieme.com> wrote:
> I apologize for asking what seems to be such a basic question, but I would use some help with submitting a job to a remote server.
>
> I have downloaded and installed hadoop locally in pseudo-distributed mode. I have written some Java code to submit a job.
>
> Here's the org.apache.hadoop.util.Tool and org.apache.hadoop.mapreduce.Mapper I've written.
>
> If I enable the conf.set("mapred.job.tracker", "localhost:9001") line, then I get the exception included below.
>
> If that line is disabled, then the job is completed. However, in reviewing the hadoop server administration page (http://localhost:50030/jobtracker.jsp) I don't see the job as processed by the server. Instead, I wonder if my Java code is simply running the necessary mapper Java code, bypassing the locally installed server.
>
> Thanks in advance.
>
> Alex
>
> public class OfflineDataTool extends Configured implements Tool {
>
> public int run(final String[] args) throws Exception {
> final Configuration conf = getConf();
> //conf.set("mapred.job.tracker", "localhost:9001");
>
> final Job job = new Job(conf);
> job.setJarByClass(getClass());
> job.setJobName(getClass().getName());
>
> job.setMapperClass(OfflineDataMapper.class);
>
> job.setInputFormatClass(TextInputFormat.class);
>
> job.setMapOutputKeyClass(Text.class);
> job.setMapOutputValueClass(Text.class);
>
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(Text.class);
>
> FileInputFormat.addInputPath(job, new org.apache.hadoop.fs.Path(args[0]));
>
> final org.apache.hadoop.fs.Path output = new org.apache.hadoop.fs.Path(args[1]);
> FileSystem.get(conf).delete(output, true);
> FileOutputFormat.setOutputPath(job, output);
>
> return job.waitForCompletion(true) ? 0 : 1;
> }
>
> public static void main(final String[] args) {
> try {
> int result = ToolRunner.run(new Configuration(), new OfflineDataTool(), new String[]{"offline/input", "offline/output"});
> log.error("result = {}", result);
> } catch (final Exception e) {
> throw new RuntimeException(e);
> }
> }
> }
>
> public class OfflineDataMapper extends Mapper<LongWritable, Text, Text, Text> {
>
> public OfflineDataMapper() {
> super();
> }
>
> @Override
> protected void map(final LongWritable key, final Text value, final Context context) throws IOException, InterruptedException {
> final String inputString = value.toString();
> OfflineDataMapper.log.error("inputString = {}", inputString);
> }
> }
>
>
>
>
> --
> Nitin Pawar
Re: Java submit job to remote server
Posted by Alex Thieme <at...@athieme.com>.
Thanks for the prompt reply and I'm sorry I forgot to include the exception. My bad. I've included it below. There certainly appears to be a server running on localhost:9001. At least, I was able to telnet to that address. While in development, I'm treating the server on localhost as the remote server. Moving to production, there'd obviously be a different remote server address configured.
Root Exception stack trace:
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
+ 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
********************************************************************************
On Feb 12, 2013, at 4:22 PM, Nitin Pawar <ni...@gmail.com> wrote:
> conf.set("mapred.job.tracker", "localhost:9001");
>
> this means that your jobtracker is on port 9001 on localhost
>
> if you change it to the remote host and thats the port its running on then it should work as expected
>
> whats the exception you are getting?
>
>
> On Wed, Feb 13, 2013 at 2:41 AM, Alex Thieme <at...@athieme.com> wrote:
> I apologize for asking what seems to be such a basic question, but I would use some help with submitting a job to a remote server.
>
> I have downloaded and installed hadoop locally in pseudo-distributed mode. I have written some Java code to submit a job.
>
> Here's the org.apache.hadoop.util.Tool and org.apache.hadoop.mapreduce.Mapper I've written.
>
> If I enable the conf.set("mapred.job.tracker", "localhost:9001") line, then I get the exception included below.
>
> If that line is disabled, then the job is completed. However, in reviewing the hadoop server administration page (http://localhost:50030/jobtracker.jsp) I don't see the job as processed by the server. Instead, I wonder if my Java code is simply running the necessary mapper Java code, bypassing the locally installed server.
>
> Thanks in advance.
>
> Alex
>
> public class OfflineDataTool extends Configured implements Tool {
>
> public int run(final String[] args) throws Exception {
> final Configuration conf = getConf();
> //conf.set("mapred.job.tracker", "localhost:9001");
>
> final Job job = new Job(conf);
> job.setJarByClass(getClass());
> job.setJobName(getClass().getName());
>
> job.setMapperClass(OfflineDataMapper.class);
>
> job.setInputFormatClass(TextInputFormat.class);
>
> job.setMapOutputKeyClass(Text.class);
> job.setMapOutputValueClass(Text.class);
>
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(Text.class);
>
> FileInputFormat.addInputPath(job, new org.apache.hadoop.fs.Path(args[0]));
>
> final org.apache.hadoop.fs.Path output = new org.apache.hadoop.fs.Path(args[1]);
> FileSystem.get(conf).delete(output, true);
> FileOutputFormat.setOutputPath(job, output);
>
> return job.waitForCompletion(true) ? 0 : 1;
> }
>
> public static void main(final String[] args) {
> try {
> int result = ToolRunner.run(new Configuration(), new OfflineDataTool(), new String[]{"offline/input", "offline/output"});
> log.error("result = {}", result);
> } catch (final Exception e) {
> throw new RuntimeException(e);
> }
> }
> }
>
> public class OfflineDataMapper extends Mapper<LongWritable, Text, Text, Text> {
>
> public OfflineDataMapper() {
> super();
> }
>
> @Override
> protected void map(final LongWritable key, final Text value, final Context context) throws IOException, InterruptedException {
> final String inputString = value.toString();
> OfflineDataMapper.log.error("inputString = {}", inputString);
> }
> }
>
>
>
>
> --
> Nitin Pawar
Re: Java submit job to remote server
Posted by Alex Thieme <at...@athieme.com>.
Thanks for the prompt reply and I'm sorry I forgot to include the exception. My bad. I've included it below. There certainly appears to be a server running on localhost:9001. At least, I was able to telnet to that address. While in development, I'm treating the server on localhost as the remote server. Moving to production, there'd obviously be a different remote server address configured.
Root Exception stack trace:
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
+ 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
********************************************************************************
On Feb 12, 2013, at 4:22 PM, Nitin Pawar <ni...@gmail.com> wrote:
> conf.set("mapred.job.tracker", "localhost:9001");
>
> this means that your jobtracker is on port 9001 on localhost
>
> if you change it to the remote host and thats the port its running on then it should work as expected
>
> whats the exception you are getting?
>
>
> On Wed, Feb 13, 2013 at 2:41 AM, Alex Thieme <at...@athieme.com> wrote:
> I apologize for asking what seems to be such a basic question, but I would use some help with submitting a job to a remote server.
>
> I have downloaded and installed hadoop locally in pseudo-distributed mode. I have written some Java code to submit a job.
>
> Here's the org.apache.hadoop.util.Tool and org.apache.hadoop.mapreduce.Mapper I've written.
>
> If I enable the conf.set("mapred.job.tracker", "localhost:9001") line, then I get the exception included below.
>
> If that line is disabled, then the job is completed. However, in reviewing the hadoop server administration page (http://localhost:50030/jobtracker.jsp) I don't see the job as processed by the server. Instead, I wonder if my Java code is simply running the necessary mapper Java code, bypassing the locally installed server.
>
> Thanks in advance.
>
> Alex
>
> public class OfflineDataTool extends Configured implements Tool {
>
> public int run(final String[] args) throws Exception {
> final Configuration conf = getConf();
> //conf.set("mapred.job.tracker", "localhost:9001");
>
> final Job job = new Job(conf);
> job.setJarByClass(getClass());
> job.setJobName(getClass().getName());
>
> job.setMapperClass(OfflineDataMapper.class);
>
> job.setInputFormatClass(TextInputFormat.class);
>
> job.setMapOutputKeyClass(Text.class);
> job.setMapOutputValueClass(Text.class);
>
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(Text.class);
>
> FileInputFormat.addInputPath(job, new org.apache.hadoop.fs.Path(args[0]));
>
> final org.apache.hadoop.fs.Path output = new org.apache.hadoop.fs.Path(args[1]);
> FileSystem.get(conf).delete(output, true);
> FileOutputFormat.setOutputPath(job, output);
>
> return job.waitForCompletion(true) ? 0 : 1;
> }
>
> public static void main(final String[] args) {
> try {
> int result = ToolRunner.run(new Configuration(), new OfflineDataTool(), new String[]{"offline/input", "offline/output"});
> log.error("result = {}", result);
> } catch (final Exception e) {
> throw new RuntimeException(e);
> }
> }
> }
>
> public class OfflineDataMapper extends Mapper<LongWritable, Text, Text, Text> {
>
> public OfflineDataMapper() {
> super();
> }
>
> @Override
> protected void map(final LongWritable key, final Text value, final Context context) throws IOException, InterruptedException {
> final String inputString = value.toString();
> OfflineDataMapper.log.error("inputString = {}", inputString);
> }
> }
>
>
>
>
> --
> Nitin Pawar
Re: Java submit job to remote server
Posted by Alex Thieme <at...@athieme.com>.
Thanks for the prompt reply and I'm sorry I forgot to include the exception. My bad. I've included it below. There certainly appears to be a server running on localhost:9001. At least, I was able to telnet to that address. While in development, I'm treating the server on localhost as the remote server. Moving to production, there'd obviously be a different remote server address configured.
Root Exception stack trace:
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
+ 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
********************************************************************************
On Feb 12, 2013, at 4:22 PM, Nitin Pawar <ni...@gmail.com> wrote:
> conf.set("mapred.job.tracker", "localhost:9001");
>
> this means that your jobtracker is on port 9001 on localhost
>
> if you change it to the remote host and thats the port its running on then it should work as expected
>
> whats the exception you are getting?
>
>
> On Wed, Feb 13, 2013 at 2:41 AM, Alex Thieme <at...@athieme.com> wrote:
> I apologize for asking what seems to be such a basic question, but I would use some help with submitting a job to a remote server.
>
> I have downloaded and installed hadoop locally in pseudo-distributed mode. I have written some Java code to submit a job.
>
> Here's the org.apache.hadoop.util.Tool and org.apache.hadoop.mapreduce.Mapper I've written.
>
> If I enable the conf.set("mapred.job.tracker", "localhost:9001") line, then I get the exception included below.
>
> If that line is disabled, then the job is completed. However, in reviewing the hadoop server administration page (http://localhost:50030/jobtracker.jsp) I don't see the job as processed by the server. Instead, I wonder if my Java code is simply running the necessary mapper Java code, bypassing the locally installed server.
>
> Thanks in advance.
>
> Alex
>
> public class OfflineDataTool extends Configured implements Tool {
>
> public int run(final String[] args) throws Exception {
> final Configuration conf = getConf();
> //conf.set("mapred.job.tracker", "localhost:9001");
>
> final Job job = new Job(conf);
> job.setJarByClass(getClass());
> job.setJobName(getClass().getName());
>
> job.setMapperClass(OfflineDataMapper.class);
>
> job.setInputFormatClass(TextInputFormat.class);
>
> job.setMapOutputKeyClass(Text.class);
> job.setMapOutputValueClass(Text.class);
>
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(Text.class);
>
> FileInputFormat.addInputPath(job, new org.apache.hadoop.fs.Path(args[0]));
>
> final org.apache.hadoop.fs.Path output = new org.apache.hadoop.fs.Path(args[1]);
> FileSystem.get(conf).delete(output, true);
> FileOutputFormat.setOutputPath(job, output);
>
> return job.waitForCompletion(true) ? 0 : 1;
> }
>
> public static void main(final String[] args) {
> try {
> int result = ToolRunner.run(new Configuration(), new OfflineDataTool(), new String[]{"offline/input", "offline/output"});
> log.error("result = {}", result);
> } catch (final Exception e) {
> throw new RuntimeException(e);
> }
> }
> }
>
> public class OfflineDataMapper extends Mapper<LongWritable, Text, Text, Text> {
>
> public OfflineDataMapper() {
> super();
> }
>
> @Override
> protected void map(final LongWritable key, final Text value, final Context context) throws IOException, InterruptedException {
> final String inputString = value.toString();
> OfflineDataMapper.log.error("inputString = {}", inputString);
> }
> }
>
>
>
>
> --
> Nitin Pawar
Re: Java submit job to remote server
Posted by Nitin Pawar <ni...@gmail.com>.
conf.set("mapred.job.tracker", "localhost:9001");
this means that your jobtracker is on port 9001 on localhost
if you change it to the remote host and thats the port its running on then
it should work as expected
whats the exception you are getting?
On Wed, Feb 13, 2013 at 2:41 AM, Alex Thieme <at...@athieme.com> wrote:
> I apologize for asking what seems to be such a basic question, but I would
> use some help with submitting a job to a remote server.
>
> I have downloaded and installed hadoop locally in pseudo-distributed mode.
> I have written some Java code to submit a job.
>
> Here's the org.apache.hadoop.util.Tool
> and org.apache.hadoop.mapreduce.Mapper I've written.
>
> If I enable the conf.set("mapred.job.tracker", "localhost:9001") line,
> then I get the exception included below.
>
> If that line is disabled, then the job is completed. However, in reviewing
> the hadoop server administration page (
> http://localhost:50030/jobtracker.jsp) I don't see the job as processed
> by the server. Instead, I wonder if my Java code is simply running the
> necessary mapper Java code, bypassing the locally installed server.
>
> Thanks in advance.
>
> Alex
>
> public class OfflineDataTool extends Configured implements Tool {
>
> public int run(final String[] args) throws Exception {
> final Configuration conf = getConf();
> //conf.set("mapred.job.tracker", "localhost:9001");
>
> final Job job = new Job(conf);
> job.setJarByClass(getClass());
> job.setJobName(getClass().getName());
>
> job.setMapperClass(OfflineDataMapper.class);
>
> job.setInputFormatClass(TextInputFormat.class);
>
> job.setMapOutputKeyClass(Text.class);
> job.setMapOutputValueClass(Text.class);
>
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(Text.class);
>
> FileInputFormat.addInputPath(job, new
> org.apache.hadoop.fs.Path(args[0]));
>
> final org.apache.hadoop.fs.Path output = new
> org.apache.hadoop.fs.Path(args[1]);
> FileSystem.get(conf).delete(output, true);
> FileOutputFormat.setOutputPath(job, output);
>
> return job.waitForCompletion(true) ? 0 : 1;
> }
>
> public static void main(final String[] args) {
> try {
> int result = ToolRunner.run(new Configuration(), new
> OfflineDataTool(), new String[]{"offline/input", "offline/output"});
> log.error("result = {}", result);
> } catch (final Exception e) {
> throw new RuntimeException(e);
> }
> }
> }
>
> public class OfflineDataMapper extends Mapper<LongWritable, Text, Text,
> Text> {
>
> public OfflineDataMapper() {
> super();
> }
>
> @Override
> protected void map(final LongWritable key, final Text value, final
> Context context) throws IOException, InterruptedException {
> final String inputString = value.toString();
> OfflineDataMapper.log.error("inputString = {}", inputString);
> }
> }
>
>
--
Nitin Pawar
Re: Java submit job to remote server
Posted by Nitin Pawar <ni...@gmail.com>.
conf.set("mapred.job.tracker", "localhost:9001");
this means that your jobtracker is on port 9001 on localhost
if you change it to the remote host and thats the port its running on then
it should work as expected
whats the exception you are getting?
On Wed, Feb 13, 2013 at 2:41 AM, Alex Thieme <at...@athieme.com> wrote:
> I apologize for asking what seems to be such a basic question, but I would
> use some help with submitting a job to a remote server.
>
> I have downloaded and installed hadoop locally in pseudo-distributed mode.
> I have written some Java code to submit a job.
>
> Here's the org.apache.hadoop.util.Tool
> and org.apache.hadoop.mapreduce.Mapper I've written.
>
> If I enable the conf.set("mapred.job.tracker", "localhost:9001") line,
> then I get the exception included below.
>
> If that line is disabled, then the job is completed. However, in reviewing
> the hadoop server administration page (
> http://localhost:50030/jobtracker.jsp) I don't see the job as processed
> by the server. Instead, I wonder if my Java code is simply running the
> necessary mapper Java code, bypassing the locally installed server.
>
> Thanks in advance.
>
> Alex
>
> public class OfflineDataTool extends Configured implements Tool {
>
> public int run(final String[] args) throws Exception {
> final Configuration conf = getConf();
> //conf.set("mapred.job.tracker", "localhost:9001");
>
> final Job job = new Job(conf);
> job.setJarByClass(getClass());
> job.setJobName(getClass().getName());
>
> job.setMapperClass(OfflineDataMapper.class);
>
> job.setInputFormatClass(TextInputFormat.class);
>
> job.setMapOutputKeyClass(Text.class);
> job.setMapOutputValueClass(Text.class);
>
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(Text.class);
>
> FileInputFormat.addInputPath(job, new
> org.apache.hadoop.fs.Path(args[0]));
>
> final org.apache.hadoop.fs.Path output = new
> org.apache.hadoop.fs.Path(args[1]);
> FileSystem.get(conf).delete(output, true);
> FileOutputFormat.setOutputPath(job, output);
>
> return job.waitForCompletion(true) ? 0 : 1;
> }
>
> public static void main(final String[] args) {
> try {
> int result = ToolRunner.run(new Configuration(), new
> OfflineDataTool(), new String[]{"offline/input", "offline/output"});
> log.error("result = {}", result);
> } catch (final Exception e) {
> throw new RuntimeException(e);
> }
> }
> }
>
> public class OfflineDataMapper extends Mapper<LongWritable, Text, Text,
> Text> {
>
> public OfflineDataMapper() {
> super();
> }
>
> @Override
> protected void map(final LongWritable key, final Text value, final
> Context context) throws IOException, InterruptedException {
> final String inputString = value.toString();
> OfflineDataMapper.log.error("inputString = {}", inputString);
> }
> }
>
>
--
Nitin Pawar
Re: Java submit job to remote server
Posted by Nitin Pawar <ni...@gmail.com>.
conf.set("mapred.job.tracker", "localhost:9001");
this means that your jobtracker is on port 9001 on localhost
if you change it to the remote host and thats the port its running on then
it should work as expected
whats the exception you are getting?
On Wed, Feb 13, 2013 at 2:41 AM, Alex Thieme <at...@athieme.com> wrote:
> I apologize for asking what seems to be such a basic question, but I would
> use some help with submitting a job to a remote server.
>
> I have downloaded and installed hadoop locally in pseudo-distributed mode.
> I have written some Java code to submit a job.
>
> Here's the org.apache.hadoop.util.Tool
> and org.apache.hadoop.mapreduce.Mapper I've written.
>
> If I enable the conf.set("mapred.job.tracker", "localhost:9001") line,
> then I get the exception included below.
>
> If that line is disabled, then the job is completed. However, in reviewing
> the hadoop server administration page (
> http://localhost:50030/jobtracker.jsp) I don't see the job as processed
> by the server. Instead, I wonder if my Java code is simply running the
> necessary mapper Java code, bypassing the locally installed server.
>
> Thanks in advance.
>
> Alex
>
> public class OfflineDataTool extends Configured implements Tool {
>
> public int run(final String[] args) throws Exception {
> final Configuration conf = getConf();
> //conf.set("mapred.job.tracker", "localhost:9001");
>
> final Job job = new Job(conf);
> job.setJarByClass(getClass());
> job.setJobName(getClass().getName());
>
> job.setMapperClass(OfflineDataMapper.class);
>
> job.setInputFormatClass(TextInputFormat.class);
>
> job.setMapOutputKeyClass(Text.class);
> job.setMapOutputValueClass(Text.class);
>
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(Text.class);
>
> FileInputFormat.addInputPath(job, new
> org.apache.hadoop.fs.Path(args[0]));
>
> final org.apache.hadoop.fs.Path output = new
> org.apache.hadoop.fs.Path(args[1]);
> FileSystem.get(conf).delete(output, true);
> FileOutputFormat.setOutputPath(job, output);
>
> return job.waitForCompletion(true) ? 0 : 1;
> }
>
> public static void main(final String[] args) {
> try {
> int result = ToolRunner.run(new Configuration(), new
> OfflineDataTool(), new String[]{"offline/input", "offline/output"});
> log.error("result = {}", result);
> } catch (final Exception e) {
> throw new RuntimeException(e);
> }
> }
> }
>
> public class OfflineDataMapper extends Mapper<LongWritable, Text, Text,
> Text> {
>
> public OfflineDataMapper() {
> super();
> }
>
> @Override
> protected void map(final LongWritable key, final Text value, final
> Context context) throws IOException, InterruptedException {
> final String inputString = value.toString();
> OfflineDataMapper.log.error("inputString = {}", inputString);
> }
> }
>
>
--
Nitin Pawar
Re: Java submit job to remote server
Posted by Nitin Pawar <ni...@gmail.com>.
conf.set("mapred.job.tracker", "localhost:9001");
this means that your jobtracker is on port 9001 on localhost
if you change it to the remote host and thats the port its running on then
it should work as expected
whats the exception you are getting?
On Wed, Feb 13, 2013 at 2:41 AM, Alex Thieme <at...@athieme.com> wrote:
> I apologize for asking what seems to be such a basic question, but I would
> use some help with submitting a job to a remote server.
>
> I have downloaded and installed hadoop locally in pseudo-distributed mode.
> I have written some Java code to submit a job.
>
> Here's the org.apache.hadoop.util.Tool
> and org.apache.hadoop.mapreduce.Mapper I've written.
>
> If I enable the conf.set("mapred.job.tracker", "localhost:9001") line,
> then I get the exception included below.
>
> If that line is disabled, then the job is completed. However, in reviewing
> the hadoop server administration page (
> http://localhost:50030/jobtracker.jsp) I don't see the job as processed
> by the server. Instead, I wonder if my Java code is simply running the
> necessary mapper Java code, bypassing the locally installed server.
>
> Thanks in advance.
>
> Alex
>
> public class OfflineDataTool extends Configured implements Tool {
>
> public int run(final String[] args) throws Exception {
> final Configuration conf = getConf();
> //conf.set("mapred.job.tracker", "localhost:9001");
>
> final Job job = new Job(conf);
> job.setJarByClass(getClass());
> job.setJobName(getClass().getName());
>
> job.setMapperClass(OfflineDataMapper.class);
>
> job.setInputFormatClass(TextInputFormat.class);
>
> job.setMapOutputKeyClass(Text.class);
> job.setMapOutputValueClass(Text.class);
>
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(Text.class);
>
> FileInputFormat.addInputPath(job, new
> org.apache.hadoop.fs.Path(args[0]));
>
> final org.apache.hadoop.fs.Path output = new
> org.apache.hadoop.fs.Path(args[1]);
> FileSystem.get(conf).delete(output, true);
> FileOutputFormat.setOutputPath(job, output);
>
> return job.waitForCompletion(true) ? 0 : 1;
> }
>
> public static void main(final String[] args) {
> try {
> int result = ToolRunner.run(new Configuration(), new
> OfflineDataTool(), new String[]{"offline/input", "offline/output"});
> log.error("result = {}", result);
> } catch (final Exception e) {
> throw new RuntimeException(e);
> }
> }
> }
>
> public class OfflineDataMapper extends Mapper<LongWritable, Text, Text,
> Text> {
>
> public OfflineDataMapper() {
> super();
> }
>
> @Override
> protected void map(final LongWritable key, final Text value, final
> Context context) throws IOException, InterruptedException {
> final String inputString = value.toString();
> OfflineDataMapper.log.error("inputString = {}", inputString);
> }
> }
>
>
--
Nitin Pawar