You are viewing a plain text version of this content. The canonical link for it is here.

Posted to hdfs-user@hadoop.apache.org by jamal sasha <ja...@gmail.com> on 2013/01/16 02:24:49 UTC

newbie question

I have a mapper


    public class BuildGraph{
       public void config(JobConf job){ *<==this block doesnt seems to be
exexcuting at all :(*
        super.configure(job);
        this.currentId = job.getInt("currentId",0);
        if (this.currentId!=0){
        // I call a method from differnt class to build a distributed cache
        }
       }
      public void map(....){
    ....
    }
}

now the main code where this is called..

    public void run( String params,curId){
     JobConf conf = new JobConf(classname.class);
     conf.setInt("currentId",299); <*--note this i am setting the value here
*
     conf.setMapperClass(Mapper.class);
     //....
      JobClient.runJob(conf);
    }

But the problem is config method in the code is not executing as though
"currentId" returns 299 in main loop but it is not set at all in the mapper
class.
what am i doing wrong

RE: newbie question

Posted by Dave Shine <Da...@channelintelligence.com>.

I get an Invalid ID when I try to access pastebin, so I haven't seen the actual code.  But in the snippet below method that is not getting called should be named configure, not config.

public class BuildGraph extends MapReduceBase implements Mapper<> {

@Override
public void configure(JobConf conf) {
                                ...
}

@Override
public void map() {
                ...
}
}



Dave Shine
Sr. Software Engineer
321.939.5093 direct |  407.314.0122 mobile
CI Boost(tm) Clients  Outperform Online(tm)  www.ciboost.com<http://www.ciboost.com/>

From: jamal sasha [mailto:jamalshasha@gmail.com]
Sent: Tuesday, January 15, 2013 9:04 PM
To: user@hadoop.apache.org
Subject: Re: newbie question

Hi,
 The relevant code snippet posted here
http://pastebin.com/DRPXUm62

On Tue, Jan 15, 2013 at 5:31 PM, jamal sasha <ja...@gmail.com>> wrote:
My bad. Sorry I fixed it. It is BuildGraph.class

On Tue, Jan 15, 2013 at 5:30 PM, Serge Blazhiyevskyy <Se...@nice.com>> wrote:
Don't you need to set this

     conf.setMapperClass(Mapper.class);

to BuildGraph?


Serge


SERGE BLAZHIYEVSKY
Architect
(T) +1 (650) 226-0511<tel:%2B1%20%28650%29%20226-0511>
(M) +1 (408) 772-2615<tel:%2B1%20%28408%29%20772-2615>
serge@nice.com<ma...@nice.com>>
www.nice.com<http://www.nice.com><http://www.nice.com>


On Jan 15, 2013, at 5:24 PM, jamal sasha <ja...@gmail.com>>> wrote:

I have a mapper


    public class BuildGraph{
       public void config(JobConf job){ <==this block doesnt seems to be exexcuting at all :(
        super.configure(job);
        this.currentId = job.getInt("currentId",0);
        if (this.currentId!=0){
        // I call a method from differnt class to build a distributed cache
        }
       }
      public void map(....){
    ....
    }
}

now the main code where this is called..

    public void run( String params,curId){
     JobConf conf = new JobConf(classname.class);
     conf.setInt("currentId",299); <--note this i am setting the value here
     conf.setMapperClass(Mapper.class);
     //....
      JobClient.runJob(conf);
    }

But the problem is config method in the code is not executing as though "currentId" returns 299 in main loop but it is not set at all in the mapper class.
what am i doing wrong



________________________________
The information contained in this email message is considered confidential and proprietary to the sender and is intended solely for review and use by the named recipient. Any unauthorized review, use or distribution is strictly prohibited. If you have received this message in error, please advise the sender by reply email and delete the message.

Re: newbie question

Posted by feng lu <am...@gmail.com>.

Hi Jamal

i think you use the old MR API, you should implements the Mapper interface.

Maybe you can take a look an WordCount.java example in
example.org.apache.hadoop.examples.


On Wed, Jan 16, 2013 at 10:03 AM, jamal sasha <ja...@gmail.com> wrote:

> Hi,
>  The relevant code snippet posted here
> http://pastebin.com/DRPXUm62
>
>
> On Tue, Jan 15, 2013 at 5:31 PM, jamal sasha <ja...@gmail.com>wrote:
>
>> My bad. Sorry I fixed it. It is BuildGraph.class
>>
>>
>> On Tue, Jan 15, 2013 at 5:30 PM, Serge Blazhiyevskyy <
>> Serge.Blazhiyevskyy@nice.com> wrote:
>>
>>> Don't you need to set this
>>>
>>>      conf.setMapperClass(Mapper.class);
>>>
>>> to BuildGraph?
>>>
>>>
>>> Serge
>>>
>>>
>>> SERGE BLAZHIYEVSKY
>>> Architect
>>> (T) +1 (650) 226-0511
>>> (M) +1 (408) 772-2615
>>> serge@nice.com<ma...@nice.com>
>>> www.nice.com<http://www.nice.com>
>>>
>>>
>>> On Jan 15, 2013, at 5:24 PM, jamal sasha <jamalshasha@gmail.com<mailto:
>>> jamalshasha@gmail.com>> wrote:
>>>
>>> I have a mapper
>>>
>>>
>>>     public class BuildGraph{
>>>        public void config(JobConf job){ <==this block doesnt seems to be
>>> exexcuting at all :(
>>>         super.configure(job);
>>>         this.currentId = job.getInt("currentId",0);
>>>         if (this.currentId!=0){
>>>         // I call a method from differnt class to build a distributed
>>> cache
>>>         }
>>>        }
>>>       public void map(....){
>>>     ....
>>>     }
>>> }
>>>
>>> now the main code where this is called..
>>>
>>>     public void run( String params,curId){
>>>      JobConf conf = new JobConf(classname.class);
>>>      conf.setInt("currentId",299); <--note this i am setting the value
>>> here
>>>      conf.setMapperClass(Mapper.class);
>>>      //....
>>>       JobClient.runJob(conf);
>>>     }
>>>
>>> But the problem is config method in the code is not executing as though
>>> "currentId" returns 299 in main loop but it is not set at all in the mapper
>>> class.
>>> what am i doing wrong
>>>
>>>
>>
>


-- 
Don't Grow Old, Grow Up... :-)

RE: newbie question

Posted by Dave Shine <Da...@channelintelligence.com>.

I get an Invalid ID when I try to access pastebin, so I haven't seen the actual code.  But in the snippet below method that is not getting called should be named configure, not config.

public class BuildGraph extends MapReduceBase implements Mapper<> {

@Override
public void configure(JobConf conf) {
                                ...
}

@Override
public void map() {
                ...
}
}



Dave Shine
Sr. Software Engineer
321.939.5093 direct |  407.314.0122 mobile
CI Boost(tm) Clients  Outperform Online(tm)  www.ciboost.com<http://www.ciboost.com/>

From: jamal sasha [mailto:jamalshasha@gmail.com]
Sent: Tuesday, January 15, 2013 9:04 PM
To: user@hadoop.apache.org
Subject: Re: newbie question

Hi,
 The relevant code snippet posted here
http://pastebin.com/DRPXUm62

On Tue, Jan 15, 2013 at 5:31 PM, jamal sasha <ja...@gmail.com>> wrote:
My bad. Sorry I fixed it. It is BuildGraph.class

On Tue, Jan 15, 2013 at 5:30 PM, Serge Blazhiyevskyy <Se...@nice.com>> wrote:
Don't you need to set this

     conf.setMapperClass(Mapper.class);

to BuildGraph?


Serge


SERGE BLAZHIYEVSKY
Architect
(T) +1 (650) 226-0511<tel:%2B1%20%28650%29%20226-0511>
(M) +1 (408) 772-2615<tel:%2B1%20%28408%29%20772-2615>
serge@nice.com<ma...@nice.com>>
www.nice.com<http://www.nice.com><http://www.nice.com>


On Jan 15, 2013, at 5:24 PM, jamal sasha <ja...@gmail.com>>> wrote:

I have a mapper


    public class BuildGraph{
       public void config(JobConf job){ <==this block doesnt seems to be exexcuting at all :(
        super.configure(job);
        this.currentId = job.getInt("currentId",0);
        if (this.currentId!=0){
        // I call a method from differnt class to build a distributed cache
        }
       }
      public void map(....){
    ....
    }
}

now the main code where this is called..

    public void run( String params,curId){
     JobConf conf = new JobConf(classname.class);
     conf.setInt("currentId",299); <--note this i am setting the value here
     conf.setMapperClass(Mapper.class);
     //....
      JobClient.runJob(conf);
    }

But the problem is config method in the code is not executing as though "currentId" returns 299 in main loop but it is not set at all in the mapper class.
what am i doing wrong



________________________________
The information contained in this email message is considered confidential and proprietary to the sender and is intended solely for review and use by the named recipient. Any unauthorized review, use or distribution is strictly prohibited. If you have received this message in error, please advise the sender by reply email and delete the message.

Re: newbie question

Posted by feng lu <am...@gmail.com>.

Hi Jamal

i think you use the old MR API, you should implements the Mapper interface.

Maybe you can take a look an WordCount.java example in
example.org.apache.hadoop.examples.


On Wed, Jan 16, 2013 at 10:03 AM, jamal sasha <ja...@gmail.com> wrote:

> Hi,
>  The relevant code snippet posted here
> http://pastebin.com/DRPXUm62
>
>
> On Tue, Jan 15, 2013 at 5:31 PM, jamal sasha <ja...@gmail.com>wrote:
>
>> My bad. Sorry I fixed it. It is BuildGraph.class
>>
>>
>> On Tue, Jan 15, 2013 at 5:30 PM, Serge Blazhiyevskyy <
>> Serge.Blazhiyevskyy@nice.com> wrote:
>>
>>> Don't you need to set this
>>>
>>>      conf.setMapperClass(Mapper.class);
>>>
>>> to BuildGraph?
>>>
>>>
>>> Serge
>>>
>>>
>>> SERGE BLAZHIYEVSKY
>>> Architect
>>> (T) +1 (650) 226-0511
>>> (M) +1 (408) 772-2615
>>> serge@nice.com<ma...@nice.com>
>>> www.nice.com<http://www.nice.com>
>>>
>>>
>>> On Jan 15, 2013, at 5:24 PM, jamal sasha <jamalshasha@gmail.com<mailto:
>>> jamalshasha@gmail.com>> wrote:
>>>
>>> I have a mapper
>>>
>>>
>>>     public class BuildGraph{
>>>        public void config(JobConf job){ <==this block doesnt seems to be
>>> exexcuting at all :(
>>>         super.configure(job);
>>>         this.currentId = job.getInt("currentId",0);
>>>         if (this.currentId!=0){
>>>         // I call a method from differnt class to build a distributed
>>> cache
>>>         }
>>>        }
>>>       public void map(....){
>>>     ....
>>>     }
>>> }
>>>
>>> now the main code where this is called..
>>>
>>>     public void run( String params,curId){
>>>      JobConf conf = new JobConf(classname.class);
>>>      conf.setInt("currentId",299); <--note this i am setting the value
>>> here
>>>      conf.setMapperClass(Mapper.class);
>>>      //....
>>>       JobClient.runJob(conf);
>>>     }
>>>
>>> But the problem is config method in the code is not executing as though
>>> "currentId" returns 299 in main loop but it is not set at all in the mapper
>>> class.
>>> what am i doing wrong
>>>
>>>
>>
>


-- 
Don't Grow Old, Grow Up... :-)

Re: newbie question

Posted by feng lu <am...@gmail.com>.

Hi Jamal

i think you use the old MR API, you should implements the Mapper interface.

Maybe you can take a look an WordCount.java example in
example.org.apache.hadoop.examples.


On Wed, Jan 16, 2013 at 10:03 AM, jamal sasha <ja...@gmail.com> wrote:

> Hi,
>  The relevant code snippet posted here
> http://pastebin.com/DRPXUm62
>
>
> On Tue, Jan 15, 2013 at 5:31 PM, jamal sasha <ja...@gmail.com>wrote:
>
>> My bad. Sorry I fixed it. It is BuildGraph.class
>>
>>
>> On Tue, Jan 15, 2013 at 5:30 PM, Serge Blazhiyevskyy <
>> Serge.Blazhiyevskyy@nice.com> wrote:
>>
>>> Don't you need to set this
>>>
>>>      conf.setMapperClass(Mapper.class);
>>>
>>> to BuildGraph?
>>>
>>>
>>> Serge
>>>
>>>
>>> SERGE BLAZHIYEVSKY
>>> Architect
>>> (T) +1 (650) 226-0511
>>> (M) +1 (408) 772-2615
>>> serge@nice.com<ma...@nice.com>
>>> www.nice.com<http://www.nice.com>
>>>
>>>
>>> On Jan 15, 2013, at 5:24 PM, jamal sasha <jamalshasha@gmail.com<mailto:
>>> jamalshasha@gmail.com>> wrote:
>>>
>>> I have a mapper
>>>
>>>
>>>     public class BuildGraph{
>>>        public void config(JobConf job){ <==this block doesnt seems to be
>>> exexcuting at all :(
>>>         super.configure(job);
>>>         this.currentId = job.getInt("currentId",0);
>>>         if (this.currentId!=0){
>>>         // I call a method from differnt class to build a distributed
>>> cache
>>>         }
>>>        }
>>>       public void map(....){
>>>     ....
>>>     }
>>> }
>>>
>>> now the main code where this is called..
>>>
>>>     public void run( String params,curId){
>>>      JobConf conf = new JobConf(classname.class);
>>>      conf.setInt("currentId",299); <--note this i am setting the value
>>> here
>>>      conf.setMapperClass(Mapper.class);
>>>      //....
>>>       JobClient.runJob(conf);
>>>     }
>>>
>>> But the problem is config method in the code is not executing as though
>>> "currentId" returns 299 in main loop but it is not set at all in the mapper
>>> class.
>>> what am i doing wrong
>>>
>>>
>>
>


-- 
Don't Grow Old, Grow Up... :-)

RE: newbie question

Posted by Dave Shine <Da...@channelintelligence.com>.

I get an Invalid ID when I try to access pastebin, so I haven't seen the actual code.  But in the snippet below method that is not getting called should be named configure, not config.

public class BuildGraph extends MapReduceBase implements Mapper<> {

@Override
public void configure(JobConf conf) {
                                ...
}

@Override
public void map() {
                ...
}
}



Dave Shine
Sr. Software Engineer
321.939.5093 direct |  407.314.0122 mobile
CI Boost(tm) Clients  Outperform Online(tm)  www.ciboost.com<http://www.ciboost.com/>

From: jamal sasha [mailto:jamalshasha@gmail.com]
Sent: Tuesday, January 15, 2013 9:04 PM
To: user@hadoop.apache.org
Subject: Re: newbie question

Hi,
 The relevant code snippet posted here
http://pastebin.com/DRPXUm62

On Tue, Jan 15, 2013 at 5:31 PM, jamal sasha <ja...@gmail.com>> wrote:
My bad. Sorry I fixed it. It is BuildGraph.class

On Tue, Jan 15, 2013 at 5:30 PM, Serge Blazhiyevskyy <Se...@nice.com>> wrote:
Don't you need to set this

     conf.setMapperClass(Mapper.class);

to BuildGraph?


Serge


SERGE BLAZHIYEVSKY
Architect
(T) +1 (650) 226-0511<tel:%2B1%20%28650%29%20226-0511>
(M) +1 (408) 772-2615<tel:%2B1%20%28408%29%20772-2615>
serge@nice.com<ma...@nice.com>>
www.nice.com<http://www.nice.com><http://www.nice.com>


On Jan 15, 2013, at 5:24 PM, jamal sasha <ja...@gmail.com>>> wrote:

I have a mapper


    public class BuildGraph{
       public void config(JobConf job){ <==this block doesnt seems to be exexcuting at all :(
        super.configure(job);
        this.currentId = job.getInt("currentId",0);
        if (this.currentId!=0){
        // I call a method from differnt class to build a distributed cache
        }
       }
      public void map(....){
    ....
    }
}

now the main code where this is called..

    public void run( String params,curId){
     JobConf conf = new JobConf(classname.class);
     conf.setInt("currentId",299); <--note this i am setting the value here
     conf.setMapperClass(Mapper.class);
     //....
      JobClient.runJob(conf);
    }

But the problem is config method in the code is not executing as though "currentId" returns 299 in main loop but it is not set at all in the mapper class.
what am i doing wrong



________________________________
The information contained in this email message is considered confidential and proprietary to the sender and is intended solely for review and use by the named recipient. Any unauthorized review, use or distribution is strictly prohibited. If you have received this message in error, please advise the sender by reply email and delete the message.

Re: newbie question

Posted by feng lu <am...@gmail.com>.

Hi Jamal

i think you use the old MR API, you should implements the Mapper interface.

Maybe you can take a look an WordCount.java example in
example.org.apache.hadoop.examples.


On Wed, Jan 16, 2013 at 10:03 AM, jamal sasha <ja...@gmail.com> wrote:

> Hi,
>  The relevant code snippet posted here
> http://pastebin.com/DRPXUm62
>
>
> On Tue, Jan 15, 2013 at 5:31 PM, jamal sasha <ja...@gmail.com>wrote:
>
>> My bad. Sorry I fixed it. It is BuildGraph.class
>>
>>
>> On Tue, Jan 15, 2013 at 5:30 PM, Serge Blazhiyevskyy <
>> Serge.Blazhiyevskyy@nice.com> wrote:
>>
>>> Don't you need to set this
>>>
>>>      conf.setMapperClass(Mapper.class);
>>>
>>> to BuildGraph?
>>>
>>>
>>> Serge
>>>
>>>
>>> SERGE BLAZHIYEVSKY
>>> Architect
>>> (T) +1 (650) 226-0511
>>> (M) +1 (408) 772-2615
>>> serge@nice.com<ma...@nice.com>
>>> www.nice.com<http://www.nice.com>
>>>
>>>
>>> On Jan 15, 2013, at 5:24 PM, jamal sasha <jamalshasha@gmail.com<mailto:
>>> jamalshasha@gmail.com>> wrote:
>>>
>>> I have a mapper
>>>
>>>
>>>     public class BuildGraph{
>>>        public void config(JobConf job){ <==this block doesnt seems to be
>>> exexcuting at all :(
>>>         super.configure(job);
>>>         this.currentId = job.getInt("currentId",0);
>>>         if (this.currentId!=0){
>>>         // I call a method from differnt class to build a distributed
>>> cache
>>>         }
>>>        }
>>>       public void map(....){
>>>     ....
>>>     }
>>> }
>>>
>>> now the main code where this is called..
>>>
>>>     public void run( String params,curId){
>>>      JobConf conf = new JobConf(classname.class);
>>>      conf.setInt("currentId",299); <--note this i am setting the value
>>> here
>>>      conf.setMapperClass(Mapper.class);
>>>      //....
>>>       JobClient.runJob(conf);
>>>     }
>>>
>>> But the problem is config method in the code is not executing as though
>>> "currentId" returns 299 in main loop but it is not set at all in the mapper
>>> class.
>>> what am i doing wrong
>>>
>>>
>>
>


-- 
Don't Grow Old, Grow Up... :-)