You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by lei liu <li...@gmail.com> on 2011/01/20 07:52:06 UTC
can one map instance handle many data of input paths at the same time
There are two input paths, example: /user/test1/ and /user/test2/ path. Can
one map instance handle many data of input paths at the same time?
Thanks,
LiuLei
Re: can one map instance handle many data of input paths at the same time
Posted by Harsh J <qw...@gmail.com>.
If you meant a single Mapper instance reading files/splits from those
two sources, it should be doable too.
> On Thu, Jan 20, 2011 at 1:52 AM, lei liu <li...@gmail.com> wrote:
>
> There are two input paths, example: /user/test1/ and /user/test2/ path.
>
> Can
>
> one map instance handle many data of input paths at the same time?
>
>
> Thanks,
>
> LiuLei
>
>
>
>
> --
> Eric Sammer
> twitter: esammer
> data: www.cloudera.com
>
>
--
Harsh J
www.harshj.com
Re: can one map instance handle many data of input paths at the same time
Posted by Owen O'Malley <om...@apache.org>.
As defined in http://hadoop.apache.org/mailing_lists.html , please
send user questions to
mapreduce-user@hadoop.apache.org.
-- Owen
On Jan 20, 2011, at 10:21 AM, Eric Sammer wrote:
> LiuLei:
>
> Yes. What you're looking for is TextInputFormat.addPath() (assuming
> you're
> talking about text). You can call this multiple times and add
> multiple input
> paths if they are all of the same data format (i.e. text). If you have
> multiple paths that contain different format data, you'll need to use
> MultipleInputs. See the javadoc for details on usage.
>
> On Thu, Jan 20, 2011 at 1:52 AM, lei liu <li...@gmail.com> wrote:
>
>> There are two input paths, example: /user/test1/ and /user/test2/
>> path.
>> Can
>> one map instance handle many data of input paths at the same time?
>>
>>
>> Thanks,
>>
>> LiuLei
>>
>
>
>
> --
> Eric Sammer
> twitter: esammer
> data: www.cloudera.com
Re: can one map instance handle many data of input paths at the same time
Posted by lei liu <li...@gmail.com>.
Thanks everyone，
I detailed describe my question. There are two input
direcoties:/user/test1/ and /user/test2/ path, I want to join the two
direcoties content, in order to join the two directories, I need to identity
the content from which directory, so I use below code in mapper:
private int tag = -1;
@Override
public void configure(JobConf conf) {
try {
this.conf = conf;
String pathsToAliasStr = conf.get("paths.to.alias");//example:
conf.set("paths.to.alias", "0=/user/test1/,1=/user/test2/"
String[] pathsToAlias = pathsToAliasStr.split(",");
Path fpath = new Path((new
Path(conf.get("map.input.file"))).toUri().getPath());
String path = fpath.toUri().toString();
for (int i = 0; i < pathsToAlias.length; i++) {
String[] pathToAlias = pathsToAlias[i].split("=");
if (path.startsWith(pathToAlias[1])) {
tag = Integer.valueOf(pathToAlias[0].trim());//identity
current map instatnce are handling which directory content.
}
}
} catch (Throwable e) {
e.printStackTrace();
throw new RuntimeException(e);
}
}
So when map method run， the content are handled by the mapper are
identified for same direcoty.
I want to know whether one mapper instatnce only handle content of one
directory at same time.
Thanks
LiuLei
2011/1/21 Eric Sammer <es...@cloudera.com>
> LiuLei:
>
> Yes. What you're looking for is TextInputFormat.addPath() (assuming you're
> talking about text). You can call this multiple times and add multiple
> input
> paths if they are all of the same data format (i.e. text). If you have
> multiple paths that contain different format data, you'll need to use
> MultipleInputs. See the javadoc for details on usage.
>
> On Thu, Jan 20, 2011 at 1:52 AM, lei liu <li...@gmail.com> wrote:
>
> > There are two input paths, example: /user/test1/ and /user/test2/ path.
> > Can
> > one map instance handle many data of input paths at the same time?
> >
> >
> > Thanks,
> >
> > LiuLei
> >
>
>
>
> --
> Eric Sammer
> twitter: esammer
> data: www.cloudera.com
>
Re: can one map instance handle many data of input paths at the same time
Posted by Owen O'Malley <om...@apache.org>.
As defined in http://hadoop.apache.org/mailing_lists.html , please
send user questions to
mapreduce-user@hadoop.apache.org.
-- Owen
On Jan 20, 2011, at 10:21 AM, Eric Sammer wrote:
> LiuLei:
>
> Yes. What you're looking for is TextInputFormat.addPath() (assuming
> you're
> talking about text). You can call this multiple times and add
> multiple input
> paths if they are all of the same data format (i.e. text). If you have
> multiple paths that contain different format data, you'll need to use
> MultipleInputs. See the javadoc for details on usage.
>
> On Thu, Jan 20, 2011 at 1:52 AM, lei liu <li...@gmail.com> wrote:
>
>> There are two input paths, example: /user/test1/ and /user/test2/
>> path.
>> Can
>> one map instance handle many data of input paths at the same time?
>>
>>
>> Thanks,
>>
>> LiuLei
>>
>
>
>
> --
> Eric Sammer
> twitter: esammer
> data: www.cloudera.com
Re: can one map instance handle many data of input paths at the same time
Posted by Eric Sammer <es...@cloudera.com>.
LiuLei:
Yes. What you're looking for is TextInputFormat.addPath() (assuming you're
talking about text). You can call this multiple times and add multiple input
paths if they are all of the same data format (i.e. text). If you have
multiple paths that contain different format data, you'll need to use
MultipleInputs. See the javadoc for details on usage.
On Thu, Jan 20, 2011 at 1:52 AM, lei liu <li...@gmail.com> wrote:
> There are two input paths, example: /user/test1/ and /user/test2/ path.
> Can
> one map instance handle many data of input paths at the same time?
>
>
> Thanks,
>
> LiuLei
>
--
Eric Sammer
twitter: esammer
data: www.cloudera.com
RE: can one map instance handle many data of input paths at the
same time
Posted by "Segel, Mike" <ms...@navteq.com>.
First a caveat... I'm going from memory and I haven't had my first cup of coffee yet... :-)
The answer is it depends.
If your input is a directory, then I believe that you will pick up all of the files in the directory.
If you are trying to specify just two files... I don't believe so.
(Now watch. I'm wrong and I do need to get more sleep.)
HTH
-Mike
-----Original Message-----
From: lei liu [mailto:liulei412@gmail.com]
Sent: Thursday, January 20, 2011 12:52 AM
To: general@hadoop.apache.org
Subject: can one map instance handle many data of input paths at the same time
There are two input paths, example: /user/test1/ and /user/test2/ path. Can
one map instance handle many data of input paths at the same time?
Thanks,
LiuLei
The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.