You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by Praveen Yarlagadda <pr...@gmail.com> on 2010/07/20 01:39:40 UTC

Concurrent Mappers

Hello all,

Is it possible to run mappers concurrently using the new API? Using old API,
I used to do that by setting the following:

jobConf.setMapRunnerClass(ConcurrentMapper.class);

ConcurrentMapper implementation is similar to how Fetcher works in Nutch. I
couldn't find the similar call in the new API. If anybody knows how to do
that using new API, please let me know.

Thanks,
Praveen

Re: Concurrent Mappers

Posted by Praveen Yarlagadda <pr...@gmail.com>.

Ok.Thanks for the response!

-Praveen

On Mon, Jul 19, 2010 at 9:52 PM, Amareshwari Sri Ramadasu <
amarsri@yahoo-inc.com> wrote:

> There is no separate MapRunner in new api. You can override Mapper.run()
> method to achieve similar goal.
>
> On 7/20/10 5:09 AM, "Praveen Yarlagadda" <pr...@gmail.com>
> wrote:
>
> Hello all,
>
> Is it possible to run mappers concurrently using the new API? Using old
> API,
> I used to do that by setting the following:
>
> jobConf.setMapRunnerClass(ConcurrentMapper.class);
>
> ConcurrentMapper implementation is similar to how Fetcher works in Nutch. I
> couldn't find the similar call in the new API. If anybody knows how to do
> that using new API, please let me know.
>
> Thanks,
> Praveen
>
>

Re: Concurrent Mappers

Posted by Amareshwari Sri Ramadasu <am...@yahoo-inc.com>.

There is no separate MapRunner in new api. You can override Mapper.run() method to achieve similar goal.

On 7/20/10 5:09 AM, "Praveen Yarlagadda" <pr...@gmail.com> wrote:

Hello all,

Is it possible to run mappers concurrently using the new API? Using old API,
I used to do that by setting the following:

jobConf.setMapRunnerClass(ConcurrentMapper.class);

ConcurrentMapper implementation is similar to how Fetcher works in Nutch. I
couldn't find the similar call in the new API. If anybody knows how to do
that using new API, please let me know.

Thanks,
Praveen