Who creates job.jar?

[Pedro Costa <ps...@gmail.com>]

Hi,

1 - Hadoop MR uses a job.jar used to execute MR. But who creates and when is
created this job.jar?

2 - This job.jar is only created at run time during the execution of an MR
task?

3 - It's created only 1 job.jar per MR execution?


Thanks
-- 
Pedro

Re: Who creates job.jar?

[Hemanth Yamijala <yh...@gmail.com>]

Pedro,


> I don't understand when you say "you" :). I've created the
> hadoop-0.20.2-dev-core.jar and hadoop-0.20.2-dev-examples.jar and none of
> them has the job.jar. I've also searched all hadoop directories, and none of
> them have the jar.

job.jar is a name given by the Map/Reduce framework to any jar that is
submitted with the map, reduce and other user classes. So, in your
case the examples.jar is submitted to the cluster as "job.jar". This
happens when the JobClient submits the job to the Map/Reduce cluster.

>
> - When the jar is created and it's exported?
>
> - What do you mean by aws?

Amazon web services - though for the purpose of this discussion, it is
immaterial on what platform Hadoop is running.

Thanks
hemanth

>
>
> On Thu, Jun 24, 2010 at 11:52 PM, Steve Lewis <lo...@gmail.com> wrote:
>>
>> You create the jar and export it to aws before running the job. The jar
>> should contain all of your code and in a directory called lib all libraries
>> you need - I have a tool for generating jars but it is a little rough -
>>
>> On Thu, Jun 24, 2010 at 2:38 PM, Pedro Costa <ps...@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> 1 - Hadoop MR uses a job.jar used to execute MR. But who creates and when
>>> is created this job.jar?
>>>
>>> 2 - This job.jar is only created at run time during the execution of an
>>> MR task?
>>>
>>> 3 - It's created only 1 job.jar per MR execution?
>>>
>>>
>>> Thanks
>>> --
>>> Pedro
>>
>>
>>
>> --
>> Steven M. Lewis PhD
>> Institute for Systems Biology
>> Seattle WA
>
>
>
> --
> Pedro
>

Re: How to get context in Close() method in hadoop Pipes

[Eric Sammer <es...@cloudera.com>]

I'm not intimately familiar with pipes, but in the Java APIs, the
easiest thing to do is to capture a reference to the context in an
instance variable in the map() method and then use it in close(). The
instance is guaranteed to still be valid in close(). Hope this helps.

On Sat, Jun 26, 2010 at 9:25 PM, Mohamed Riadh Trad
<Mo...@inria.fr> wrote:
> Dear All;
>
> I had to emit final Key/Values in the Mapper Close Method but I can't get the context.
>
> Any suggestion?
>
> Regard.



-- 
Eric Sammer
twitter: esammer
data: www.cloudera.com

How to get context in Close() method in hadoop Pipes

[Mohamed Riadh Trad <Mo...@inria.fr>]

Dear All;

I had to emit final Key/Values in the Mapper Close Method but I can't get the context.

Any suggestion?

Regard.

Re: Who creates job.jar?

[Steve Lewis <lo...@gmail.com>]

I have attached code for creating a Hadoop Jar - All you need to do is run
HadoopDeployer in the same environment
that your hadoop job runs as a local process (You did test your job in this
way - It jars everything in the class path except
jars you add to an excluded list because they are not needed (such as Junit)
or already in the path such as the hadoop jar.


On Fri, Jun 25, 2010 at 3:32 AM, Kiyoshi Mizumaru <
kiyoshi.mizumaru@gmail.com> wrote:

> > - What do you mean by aws?
>
> It must be Amazon Web Services, I think.
>
>
> On Fri, Jun 25, 2010 at 7:22 PM, Pedro Costa <ps...@gmail.com> wrote:
> > I don't understand when you say "you" :). I've created the
> > hadoop-0.20.2-dev-core.jar and hadoop-0.20.2-dev-examples.jar and none of
> > them has the job.jar. I've also searched all hadoop directories, and none
> of
> > them have the jar.
> >
> > - When the jar is created and it's exported?
> >
> > - What do you mean by aws?
> >
> >
> > On Thu, Jun 24, 2010 at 11:52 PM, Steve Lewis <lo...@gmail.com>
> wrote:
> >>
> >> You create the jar and export it to aws before running the job. The jar
> >> should contain all of your code and in a directory called lib all
> libraries
> >> you need - I have a tool for generating jars but it is a little rough -
> >>
> >> On Thu, Jun 24, 2010 at 2:38 PM, Pedro Costa <ps...@gmail.com>
> wrote:
> >>>
> >>> Hi,
> >>>
> >>> 1 - Hadoop MR uses a job.jar used to execute MR. But who creates and
> when
> >>> is created this job.jar?
> >>>
> >>> 2 - This job.jar is only created at run time during the execution of an
> >>> MR task?
> >>>
> >>> 3 - It's created only 1 job.jar per MR execution?
> >>>
> >>>
> >>> Thanks
> >>> --
> >>> Pedro
> >>
> >>
> >>
> >> --
> >> Steven M. Lewis PhD
> >> Institute for Systems Biology
> >> Seattle WA
> >
> >
> >
> > --
> > Pedro
> >
>



-- 
Steven M. Lewis PhD
Institute for Systems Biology
Seattle WA

Re: Who creates job.jar?

[Kiyoshi Mizumaru <ki...@gmail.com>]

> - What do you mean by aws?

It must be Amazon Web Services, I think.


On Fri, Jun 25, 2010 at 7:22 PM, Pedro Costa <ps...@gmail.com> wrote:
> I don't understand when you say "you" :). I've created the
> hadoop-0.20.2-dev-core.jar and hadoop-0.20.2-dev-examples.jar and none of
> them has the job.jar. I've also searched all hadoop directories, and none of
> them have the jar.
>
> - When the jar is created and it's exported?
>
> - What do you mean by aws?
>
>
> On Thu, Jun 24, 2010 at 11:52 PM, Steve Lewis <lo...@gmail.com> wrote:
>>
>> You create the jar and export it to aws before running the job. The jar
>> should contain all of your code and in a directory called lib all libraries
>> you need - I have a tool for generating jars but it is a little rough -
>>
>> On Thu, Jun 24, 2010 at 2:38 PM, Pedro Costa <ps...@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> 1 - Hadoop MR uses a job.jar used to execute MR. But who creates and when
>>> is created this job.jar?
>>>
>>> 2 - This job.jar is only created at run time during the execution of an
>>> MR task?
>>>
>>> 3 - It's created only 1 job.jar per MR execution?
>>>
>>>
>>> Thanks
>>> --
>>> Pedro
>>
>>
>>
>> --
>> Steven M. Lewis PhD
>> Institute for Systems Biology
>> Seattle WA
>
>
>
> --
> Pedro
>

Re: Who creates job.jar?

[welman Lu <we...@gmail.com>]

*job.jar* is used to encapsulate the program code you write by yourself.
It's not an automatic produced file, but you self should create it manually.

And of course you can use other names to call this jar file.

If you don't understand, you can learn to study the
hadoop-0.20.2-dev-examples.jar file.


On Fri, Jun 25, 2010 at 12:22 PM, Pedro Costa <ps...@gmail.com> wrote:

> I don't understand when you say "you" :). I've created the
> hadoop-0.20.2-dev-core.jar and hadoop-0.20.2-dev-examples.jar and none of
> them has the job.jar. I've also searched all hadoop directories, and none of
> them have the jar.
>
> - When the jar is created and it's exported?
>
> - What do you mean by aws?
>
>
>
> On Thu, Jun 24, 2010 at 11:52 PM, Steve Lewis <lo...@gmail.com>wrote:
>
>> You create the jar and export it to aws before running the job. The jar
>> should contain all of your code and in a directory called lib all libraries
>> you need - I have a tool for generating jars but it is a little rough -
>>
>>
>> On Thu, Jun 24, 2010 at 2:38 PM, Pedro Costa <ps...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> 1 - Hadoop MR uses a job.jar used to execute MR. But who creates and when
>>> is created this job.jar?
>>>
>>> 2 - This job.jar is only created at run time during the execution of an
>>> MR task?
>>>
>>> 3 - It's created only 1 job.jar per MR execution?
>>>
>>>
>>> Thanks
>>> --
>>> Pedro
>>>
>>
>>
>>
>> --
>> Steven M. Lewis PhD
>> Institute for Systems Biology
>> Seattle WA
>>
>
>
>
> --
> Pedro
>

Re: Who creates job.jar?

[Pedro Costa <ps...@gmail.com>]

I don't understand when you say "you" :). I've created the
hadoop-0.20.2-dev-core.jar and hadoop-0.20.2-dev-examples.jar and none of
them has the job.jar. I've also searched all hadoop directories, and none of
them have the jar.

- When the jar is created and it's exported?

- What do you mean by aws?


On Thu, Jun 24, 2010 at 11:52 PM, Steve Lewis <lo...@gmail.com> wrote:

> You create the jar and export it to aws before running the job. The jar
> should contain all of your code and in a directory called lib all libraries
> you need - I have a tool for generating jars but it is a little rough -
>
>
> On Thu, Jun 24, 2010 at 2:38 PM, Pedro Costa <ps...@gmail.com> wrote:
>
>> Hi,
>>
>> 1 - Hadoop MR uses a job.jar used to execute MR. But who creates and when
>> is created this job.jar?
>>
>> 2 - This job.jar is only created at run time during the execution of an MR
>> task?
>>
>> 3 - It's created only 1 job.jar per MR execution?
>>
>>
>> Thanks
>> --
>> Pedro
>>
>
>
>
> --
> Steven M. Lewis PhD
> Institute for Systems Biology
> Seattle WA
>



-- 
Pedro

Re: Who creates job.jar?

[Steve Lewis <lo...@gmail.com>]

You create the jar and export it to aws before running the job. The jar
should contain all of your code and in a directory called lib all libraries
you need - I have a tool for generating jars but it is a little rough -


On Thu, Jun 24, 2010 at 2:38 PM, Pedro Costa <ps...@gmail.com> wrote:

> Hi,
>
> 1 - Hadoop MR uses a job.jar used to execute MR. But who creates and when
> is created this job.jar?
>
> 2 - This job.jar is only created at run time during the execution of an MR
> task?
>
> 3 - It's created only 1 job.jar per MR execution?
>
>
> Thanks
> --
> Pedro
>



-- 
Steven M. Lewis PhD
Institute for Systems Biology
Seattle WA