You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Pedro Sá da Costa <ps...@gmail.com> on 2012/11/28 18:11:09 UTC
Get JobInProgress given jobId
I'm building a Java class and given a JobID, how can I get the
JobInProgress? Can anyone give me an example?
--
Best regards,
Re: Get JobInProgress given jobId
Posted by Harsh J <ha...@cloudera.com>.
The JobInProgress object is a private one and can't be fetched unless
you are running a service within the JobTracker environment.
What are you looking for, from the JobInProgress' information,
specifically, that is not found in the RunningJob interface provided
to clients?
On Wed, Nov 28, 2012 at 10:41 PM, Pedro Sá da Costa <ps...@gmail.com> wrote:
> I'm building a Java class and given a JobID, how can I get the
> JobInProgress? Can anyone give me an example?
>
> --
> Best regards,
--
Harsh J
Re: Get JobInProgress given jobId
Posted by Harsh J <ha...@cloudera.com>.
Such a mix won't work, and hence my (1) point of check. Please use a
jar that matches the server version, and you'll have it working.
On Thu, Nov 29, 2012 at 12:53 AM, Pedro Sá da Costa <ps...@gmail.com> wrote:
> And for this error, after all maybe I'm running hadoop.jar with
> different versions. I'm running hadoop-0.20 and trying to run
> JobClient in with hadoo-1.0
>
> On 28 November 2012 19:20, Pedro Sá da Costa <ps...@gmail.com> wrote:
>> I've this error in jobtracker log. Maybe this is the reason. What this
>> error means?
>>
>> 2012-11-28 19:19:17,697 WARN org.apache.hadoop.ipc.Server: Incorrect
>> header or version mismatch from 127.0.0.1:60089 got version 4 expected
>> version 3
>>
>>
>>
>> On 28 November 2012 18:28, Pedro Sá da Costa <ps...@gmail.com> wrote:
>>> On 28 November 2012 18:12, Harsh J <ha...@cloudera.com> wrote:
>>>> nt application's hadoop jar same version as the server?
>>>
>>> Yes it is.
>>>
>>>> 2. Is the port 54311 the proper JobTracker port?
>>>
>>> This jobtracker port is set to:
>>>
>>> <property>
>>> <name>mapred.job.tracker</name>
>>> <value>localhost:54311</value>
>>> <!--value>local</value-->
>>> <description>The host and port that the MapReduce job tracker runs
>>> at. If "local", then jobs are run in-process as a single map and
>>> reduce task.
>>> </description>
>>> </property>
>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>
>>
>>
>> --
>> Best regards,
>
>
>
> --
> Best regards,
--
Harsh J
Re: Get JobInProgress given jobId
Posted by Pedro Sá da Costa <ps...@gmail.com>.
And for this error, after all maybe I'm running hadoop.jar with
different versions. I'm running hadoop-0.20 and trying to run
JobClient in with hadoo-1.0
On 28 November 2012 19:20, Pedro Sá da Costa <ps...@gmail.com> wrote:
> I've this error in jobtracker log. Maybe this is the reason. What this
> error means?
>
> 2012-11-28 19:19:17,697 WARN org.apache.hadoop.ipc.Server: Incorrect
> header or version mismatch from 127.0.0.1:60089 got version 4 expected
> version 3
>
>
>
> On 28 November 2012 18:28, Pedro Sá da Costa <ps...@gmail.com> wrote:
>> On 28 November 2012 18:12, Harsh J <ha...@cloudera.com> wrote:
>>> nt application's hadoop jar same version as the server?
>>
>> Yes it is.
>>
>>> 2. Is the port 54311 the proper JobTracker port?
>>
>> This jobtracker port is set to:
>>
>> <property>
>> <name>mapred.job.tracker</name>
>> <value>localhost:54311</value>
>> <!--value>local</value-->
>> <description>The host and port that the MapReduce job tracker runs
>> at. If "local", then jobs are run in-process as a single map and
>> reduce task.
>> </description>
>> </property>
>>
>>
>>
>>
>> --
>> Best regards,
>
>
>
> --
> Best regards,
--
Best regards,
Re: Get JobInProgress given jobId
Posted by Pedro Sá da Costa <ps...@gmail.com>.
I've this error in jobtracker log. Maybe this is the reason. What this
error means?
2012-11-28 19:19:17,697 WARN org.apache.hadoop.ipc.Server: Incorrect
header or version mismatch from 127.0.0.1:60089 got version 4 expected
version 3
On 28 November 2012 18:28, Pedro Sá da Costa <ps...@gmail.com> wrote:
> On 28 November 2012 18:12, Harsh J <ha...@cloudera.com> wrote:
>> nt application's hadoop jar same version as the server?
>
> Yes it is.
>
>> 2. Is the port 54311 the proper JobTracker port?
>
> This jobtracker port is set to:
>
> <property>
> <name>mapred.job.tracker</name>
> <value>localhost:54311</value>
> <!--value>local</value-->
> <description>The host and port that the MapReduce job tracker runs
> at. If "local", then jobs are run in-process as a single map and
> reduce task.
> </description>
> </property>
>
>
>
>
> --
> Best regards,
--
Best regards,
Re: Get JobInProgress given jobId
Posted by Pedro Sá da Costa <ps...@gmail.com>.
On 28 November 2012 18:12, Harsh J <ha...@cloudera.com> wrote:
> nt application's hadoop jar same version as the server?
Yes it is.
> 2. Is the port 54311 the proper JobTracker port?
This jobtracker port is set to:
<property>
<name>mapred.job.tracker</name>
<value>localhost:54311</value>
<!--value>local</value-->
<description>The host and port that the MapReduce job tracker runs
at. If "local", then jobs are run in-process as a single map and
reduce task.
</description>
</property>
--
Best regards,
Re: Get JobInProgress given jobId
Posted by Harsh J <ha...@cloudera.com>.
Two things to check and address if not true:
1. Is your client application's hadoop jar same version as the server?
2. Is the port 54311 the proper JobTracker port?
On Wed, Nov 28, 2012 at 11:37 PM, Pedro Sá da Costa <ps...@gmail.com> wrote:
> Yes, I already did like you tell, but I get an error in the
> connection, and this doesn't make any sense.
>
> java.io.IOException: Call to localhost/127.0.0.1:54311 failed on local
> exception: java.io.EOFException
> at org.apache.hadoop.ipc.Client.wrapException(Client.java:1107)
> at org.apache.hadoop.ipc.Client.call(Client.java:1075)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
> at org.apache.hadoop.mapred.$Proxy1.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
> at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:498)
> at org.apache.hadoop.mapred.JobClient.init(JobClient.java:492)
> at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:475)
> at org.apache.hadoop.tools.JobProgression.status(JobProgression.java:38)
> at org.apache.hadoop.tools.JobProgression.run(JobProgression.java:98)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.JobProgression.main(JobProgression.java:69)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:804)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:749)
> With failures, global counters are inaccurate; consider running with -i
> Copy failed: java.lang.NullPointerException
> at org.apache.hadoop.tools.JobProgression.status(JobProgression.java:45)
> at org.apache.hadoop.tools.JobProgression.run(JobProgression.java:98)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.JobProgression.main(JobProgression.java:69)
>
>
>
> On 28 November 2012 17:44, Harsh J <ha...@cloudera.com> wrote:
>> Mahesh,
>>
>> The JobTracker is not a user-end API, it is a service. You can't query
>> it for its private objects as simply as that, and hence what you've
>> suggested would not work.
>>
>> Pedro,
>>
>> To get a RunningJob instance from a String representing the JobID you
>> need the JobClient class, and you can do like so:
>>
>> JobClient jc = new JobClient(jobConf);
>> org.apache.hadoop.mapred.JobID jId =
>> org.apache.hadoop.mapred.JobID.forName(jobIdString);
>> RunningJob rJob = jc.getJob(jId);
>> // Play with the rJob object as you want.
>>
>> On Wed, Nov 28, 2012 at 10:59 PM, Pedro Sá da Costa <ps...@gmail.com> wrote:
>>> I have the jobId as a String, and from that I want to access the
>>> RunningJob API for that jobId. I think that it is only possible to
>>> access this API through the JobInProgress class, but maybe I'm wrong.
>>> Is this true?
>>>
>>>
>>> On 28 November 2012 17:24, Mahesh Balija <ba...@gmail.com> wrote:
>>>> Hi Pedro,
>>>>
>>>> You can get the JobInProgress instance from JobTracker.
>>>> JobInProgress getJob(JobID jobid);
>>>>
>>>> Best,
>>>> Mahesh Balija,
>>>> Calsoft Labs.
>>>>
>>>> On Wed, Nov 28, 2012 at 10:41 PM, Pedro Sá da Costa <ps...@gmail.com>
>>>> wrote:
>>>>>
>>>>> I'm building a Java class and given a JobID, how can I get the
>>>>> JobInProgress? Can anyone give me an example?
>>>>>
>>>>> --
>>>>> Best regards,
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>
>>
>>
>> --
>> Harsh J
>
>
>
> --
> Best regards,
--
Harsh J
Re: Get JobInProgress given jobId
Posted by Pedro Sá da Costa <ps...@gmail.com>.
Yes, I already did like you tell, but I get an error in the
connection, and this doesn't make any sense.
java.io.IOException: Call to localhost/127.0.0.1:54311 failed on local
exception: java.io.EOFException
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1107)
at org.apache.hadoop.ipc.Client.call(Client.java:1075)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
at org.apache.hadoop.mapred.$Proxy1.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:498)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:492)
at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:475)
at org.apache.hadoop.tools.JobProgression.status(JobProgression.java:38)
at org.apache.hadoop.tools.JobProgression.run(JobProgression.java:98)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.tools.JobProgression.main(JobProgression.java:69)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:804)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:749)
With failures, global counters are inaccurate; consider running with -i
Copy failed: java.lang.NullPointerException
at org.apache.hadoop.tools.JobProgression.status(JobProgression.java:45)
at org.apache.hadoop.tools.JobProgression.run(JobProgression.java:98)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.tools.JobProgression.main(JobProgression.java:69)
On 28 November 2012 17:44, Harsh J <ha...@cloudera.com> wrote:
> Mahesh,
>
> The JobTracker is not a user-end API, it is a service. You can't query
> it for its private objects as simply as that, and hence what you've
> suggested would not work.
>
> Pedro,
>
> To get a RunningJob instance from a String representing the JobID you
> need the JobClient class, and you can do like so:
>
> JobClient jc = new JobClient(jobConf);
> org.apache.hadoop.mapred.JobID jId =
> org.apache.hadoop.mapred.JobID.forName(jobIdString);
> RunningJob rJob = jc.getJob(jId);
> // Play with the rJob object as you want.
>
> On Wed, Nov 28, 2012 at 10:59 PM, Pedro Sá da Costa <ps...@gmail.com> wrote:
>> I have the jobId as a String, and from that I want to access the
>> RunningJob API for that jobId. I think that it is only possible to
>> access this API through the JobInProgress class, but maybe I'm wrong.
>> Is this true?
>>
>>
>> On 28 November 2012 17:24, Mahesh Balija <ba...@gmail.com> wrote:
>>> Hi Pedro,
>>>
>>> You can get the JobInProgress instance from JobTracker.
>>> JobInProgress getJob(JobID jobid);
>>>
>>> Best,
>>> Mahesh Balija,
>>> Calsoft Labs.
>>>
>>> On Wed, Nov 28, 2012 at 10:41 PM, Pedro Sá da Costa <ps...@gmail.com>
>>> wrote:
>>>>
>>>> I'm building a Java class and given a JobID, how can I get the
>>>> JobInProgress? Can anyone give me an example?
>>>>
>>>> --
>>>> Best regards,
>>>
>>>
>>
>>
>>
>> --
>> Best regards,
>
>
>
> --
> Harsh J
--
Best regards,
Re: Get JobInProgress given jobId
Posted by Harsh J <ha...@cloudera.com>.
Mahesh,
The JobTracker is not a user-end API, it is a service. You can't query
it for its private objects as simply as that, and hence what you've
suggested would not work.
Pedro,
To get a RunningJob instance from a String representing the JobID you
need the JobClient class, and you can do like so:
JobClient jc = new JobClient(jobConf);
org.apache.hadoop.mapred.JobID jId =
org.apache.hadoop.mapred.JobID.forName(jobIdString);
RunningJob rJob = jc.getJob(jId);
// Play with the rJob object as you want.
On Wed, Nov 28, 2012 at 10:59 PM, Pedro Sá da Costa <ps...@gmail.com> wrote:
> I have the jobId as a String, and from that I want to access the
> RunningJob API for that jobId. I think that it is only possible to
> access this API through the JobInProgress class, but maybe I'm wrong.
> Is this true?
>
>
> On 28 November 2012 17:24, Mahesh Balija <ba...@gmail.com> wrote:
>> Hi Pedro,
>>
>> You can get the JobInProgress instance from JobTracker.
>> JobInProgress getJob(JobID jobid);
>>
>> Best,
>> Mahesh Balija,
>> Calsoft Labs.
>>
>> On Wed, Nov 28, 2012 at 10:41 PM, Pedro Sá da Costa <ps...@gmail.com>
>> wrote:
>>>
>>> I'm building a Java class and given a JobID, how can I get the
>>> JobInProgress? Can anyone give me an example?
>>>
>>> --
>>> Best regards,
>>
>>
>
>
>
> --
> Best regards,
--
Harsh J
Re: Get JobInProgress given jobId
Posted by Pedro Sá da Costa <ps...@gmail.com>.
I have the jobId as a String, and from that I want to access the
RunningJob API for that jobId. I think that it is only possible to
access this API through the JobInProgress class, but maybe I'm wrong.
Is this true?
On 28 November 2012 17:24, Mahesh Balija <ba...@gmail.com> wrote:
> Hi Pedro,
>
> You can get the JobInProgress instance from JobTracker.
> JobInProgress getJob(JobID jobid);
>
> Best,
> Mahesh Balija,
> Calsoft Labs.
>
> On Wed, Nov 28, 2012 at 10:41 PM, Pedro Sá da Costa <ps...@gmail.com>
> wrote:
>>
>> I'm building a Java class and given a JobID, how can I get the
>> JobInProgress? Can anyone give me an example?
>>
>> --
>> Best regards,
>
>
--
Best regards,
Re: Get JobInProgress given jobId
Posted by Mahesh Balija <ba...@gmail.com>.
Hi Pedro,
You can get the JobInProgress instance from JobTracker.
JobInProgress getJob(JobID jobid);
Best,
Mahesh Balija,
Calsoft Labs.
On Wed, Nov 28, 2012 at 10:41 PM, Pedro Sá da Costa <ps...@gmail.com>wrote:
> I'm building a Java class and given a JobID, how can I get the
> JobInProgress? Can anyone give me an example?
>
> --
> Best regards,
>