Re: purpose of JobTracker copying job.jar

[Raghu Angadi <ra...@apache.org>]

On Wed, Apr 27, 2011 at 11:20 AM, Arun C Murthy <ac...@yahoo-inc.com> wrote:

> So that it can instruct TTs to copy it from the system-dir for actually
> running jobs..
>

Thanks Arun.
Task trackers copy the job jar from HDFS (placed by client).
JobTracker copying the same file to its local filesystem seems to be
unnecessary.. This could have very noticeable effects on the central server.

I commented out JobInProgress.java:266 on 0.20 trunk, and my test jobs run
fine. I will try to double check what I am missing.

Raghu.


> hope that helps,
> Arun
>
>
> On Apr 26, 2011, at 11:57 AM, Raghu Angadi wrote:
>
>
>> Job tracker copies job jar from mapred system directory on HDFS to its
>> local file system ( ${mapred.local.dir}/jobTracker ).
>>
>> What is the purpose of copying this file? It does not run any user code.
>>
>> thanks,
>> Raghu.
>>
>>
>

Re: purpose of JobTracker copying job.jar

[Dmitriy Ryaboy <dv...@gmail.com>]

Looking around 0.22, looks like this behavior is gone there, it's not
copying the jar. The only copyToLocal call it makes is to get the job
file (not the jar file).  0.20 did both.

Does anyone remember why 0.20 needed the jar?

-D

On Mon, May 9, 2011 at 12:55 AM, Raghu Angadi <ra...@apache.org> wrote:
> On Wed, Apr 27, 2011 at 11:20 AM, Arun C Murthy <ac...@yahoo-inc.com> wrote:
>>
>> So that it can instruct TTs to copy it from the system-dir for actually
>> running jobs..
>
> Thanks Arun.
> Task trackers copy the job jar from HDFS (placed by client).
> JobTracker copying the same file to its local filesystem seems to be
> unnecessary.. This could have very noticeable effects on the central server.
> I commented out JobInProgress.java:266 on 0.20 trunk, and my test jobs run
> fine. I will try to double check what I am missing.
> Raghu.
>
>>
>> hope that helps,
>> Arun
>>
>> On Apr 26, 2011, at 11:57 AM, Raghu Angadi wrote:
>>
>>>
>>> Job tracker copies job jar from mapred system directory on HDFS to its
>>> local file system ( ${mapred.local.dir}/jobTracker ).
>>>
>>> What is the purpose of copying this file? It does not run any user code.
>>>
>>> thanks,
>>> Raghu.
>>>
>>
>
>