You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Sh...@cognizant.com on 2011/09/15 13:10:54 UTC
Error calling config method in Mapper
Hi
I have written a mapper class that has the following structure
public class TestMapper extends MapReduceBase implements
Mapper<LongWritable, Text, Text, Text> {
public void configure(JobConf conf) {
Path [] cacheFiles = DistributedCache.getLocalCacheFiles(conf);
if (null != cacheFiles && cacheFiles.length > 0) {
for (Path cachePath : cacheFiles) {
if (cachePath.getName().equals(stopwordCacheName)) {
loadStopWords(cachePath);
break;
}
if (cachePath.getName().equals(positiveCacheName)) {
loadPositiveWords(cachePath);
break;
}
if (cachePath.getName().equals(negativeCacheName)) {
loadNegativeWords(cachePath);
break;
}
if (cachePath.getName().equals(searchCacheName)) {
loadSearchWords(cachePath);
break;
}
}
//Mapper logic goes here
}
Config method should get called before the mapper method by default
right?
But that's not happening..... and the loggers I have used inside mapper
are not getting printed on console and there are no loga in userlogs.
Userlogs is empty. Logger
statements(sLogger.debug("*****************uploaded positive words");)
in the driver class are getting printed on the console though.
Regards,
Shreya
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful.
RE: Error calling config method in Mapper
Posted by Sh...@cognizant.com.
Hi Harsh
I rechecked the code, it seems to be fine :(
public static final String HDFS_STOPWORD_LIST = "/data/stop_words.txt";
public static final String HDFS_POSITIVE_LIST =
"/data/Positive_Words.txt";
public static final String HDFS_NEGATIVE_LIST =
"/data/Negative_Words.txt";
public static final String HDFS_SEARCH_LIST =
"/data/Search_Criteria_Twitter.txt";
@Override
public void configure(JobConf conf) {
try {
super.configure(conf);
String stopwordCacheName = new
Path(HDFS_STOPWORD_LIST).getName();
String positiveCacheName = new
Path(HDFS_POSITIVE_LIST).getName();
String negativeCacheName = new
Path(HDFS_NEGATIVE_LIST).getName();
String searchCacheName = new
Path(HDFS_SEARCH_LIST).getName();
sLogger.debug("*********Inside Configure");
Path [] cacheFiles =
DistributedCache.getLocalCacheFiles(conf);
if (null != cacheFiles && cacheFiles.length > 0) {
for (Path cachePath : cacheFiles) {
if
(cachePath.getName().equals(stopwordCacheName)) {
loadStopWords(cachePath);
break;
}
if
(cachePath.getName().equals(positiveCacheName)) {
loadPositiveWords(cachePath);
break;
}
if
(cachePath.getName().equals(negativeCacheName)) {
loadNegativeWords(cachePath);
break;
}
if
(cachePath.getName().equals(searchCacheName)) {
loadSearchWords(cachePath);
break;
}
}
}else {
System.out.println("Got no cache files...");
}
} catch (IOException ioe) {
System.err.println("IOException reading from distributed
cache");
System.err.println(ioe.toString());
}
}
Regards,
Shreya
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Thursday, September 15, 2011 6:23 PM
To: mapreduce-user@hadoop.apache.org
Subject: Re: Error calling config method in Mapper
Shreya,
On Thu, Sep 15, 2011 at 5:37 PM, <Sh...@cognizant.com> wrote:
> I used @override does not seem to work.
Its just to guarantee that you're actually overriding things. Its a
good practice, not a trouble fixer at all times :)
> Configure is not getting called as I populate some collections there
that are used later in the mapper and I am getting those collections as
blank in the mapper.
FWIW, here's my results of your snippet, and it apparently looks like
configure() is called just fine:
https://gist.github.com/bae64045e55377089afd. I think the trouble is
with your code, on how you're trying to load the distributed cache
elements out and back in. My guess is that you haven't initialized
some of the variables you may be .equals()-ing against and thereby
they're being treated as nulls?
Its definitely not a Hadoop-side issue afaict.
--
Harsh J
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful.
Re: Error calling config method in Mapper
Posted by Harsh J <ha...@cloudera.com>.
Shreya,
On Thu, Sep 15, 2011 at 5:37 PM, <Sh...@cognizant.com> wrote:
> I used @override does not seem to work.
Its just to guarantee that you're actually overriding things. Its a
good practice, not a trouble fixer at all times :)
> Configure is not getting called as I populate some collections there that are used later in the mapper and I am getting those collections as blank in the mapper.
FWIW, here's my results of your snippet, and it apparently looks like
configure() is called just fine:
https://gist.github.com/bae64045e55377089afd. I think the trouble is
with your code, on how you're trying to load the distributed cache
elements out and back in. My guess is that you haven't initialized
some of the variables you may be .equals()-ing against and thereby
they're being treated as nulls?
Its definitely not a Hadoop-side issue afaict.
--
Harsh J
RE: Error calling config method in Mapper
Posted by Sh...@cognizant.com.
Hi Harsh,
I resolved the earlier issue I had with mapper but the configure is not
getting called.
The signature for Mapper is
public void map(LongWritable key, Text value, OutputCollector output,
Reporter reporter) throws IOException {
I used @override does not seem to work.
Configure is not getting called as I populate some collections there
that are used later in the mapper and I am getting those collections as
blank in the mapper.
Regards,
Shreya
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Thursday, September 15, 2011 5:14 PM
To: mapreduce-user@hadoop.apache.org
Subject: Re: Error calling config method in Mapper
Hello again Shreya,
That signature (for configure) looks right and I think it indeed gets
called. Unsure about your map call issue as I do not see that signature.
Perhaps what you need to resolve this issue by yourself is to make use
of the @Override method annotations in your code. That helps a lot when
dealing with frameworks, such as Hadoop.
On Thu, Sep 15, 2011 at 4:40 PM, <Sh...@cognizant.com> wrote:
Hi
I have written a mapper class that has the following structure
public class TestMapper extends MapReduceBase implements
Mapper<LongWritable, Text, Text, Text> {
public void configure(JobConf conf) {
Path [] cacheFiles = DistributedCache.getLocalCacheFiles(conf);
if (null != cacheFiles && cacheFiles.length > 0) {
for (Path cachePath : cacheFiles) {
if (cachePath.getName().equals(stopwordCacheName)) {
loadStopWords(cachePath);
break;
}
if (cachePath.getName().equals(positiveCacheName)) {
loadPositiveWords(cachePath);
break;
}
if (cachePath.getName().equals(negativeCacheName)) {
loadNegativeWords(cachePath);
break;
}
if (cachePath.getName().equals(searchCacheName)) {
loadSearchWords(cachePath);
break;
}
}
//Mapper logic goes here
}
Config method should get called before the mapper method by default
right?
But that's not happening..... and the loggers I have used inside mapper
are not getting printed on console and there are no loga in userlogs.
Userlogs is empty. Logger
statements(sLogger.debug("*****************uploaded positive words");)
in the driver class are getting printed on the console though.
Regards,
Shreya
This e-mail and any files transmitted with it are for the sole use of
the intended recipient(s) and may contain confidential and privileged
information. If you are not the intended recipient(s), please reply to
the sender and destroy all copies of the original message. Any
unauthorized review, use, disclosure, dissemination, forwarding,
printing or copying of this email, and/or any action taken in reliance
on the contents of this e-mail is strictly prohibited and may be
unlawful.
--
Harsh J
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful.
Re: Error calling config method in Mapper
Posted by Harsh J <ha...@cloudera.com>.
Hello again Shreya,
That signature (for configure) looks right and I think it indeed gets
called. Unsure about your map call issue as I do not see that signature.
Perhaps what you need to resolve this issue by yourself is to make use of
the @Override method annotations in your code. That helps a lot when dealing
with frameworks, such as Hadoop.
On Thu, Sep 15, 2011 at 4:40 PM, <Sh...@cognizant.com> wrote:
> Hi****
>
> ** **
>
> I have written a mapper class that has the following structure****
>
> ** **
>
> *public* *class* TestMapper *extends* MapReduceBase *implements*Mapper<LongWritable, Text, Text, Text> {
> ****
>
> ** **
>
> ** **
>
> ** **
>
> *public* *void* configure(JobConf conf) {****
>
> Path [] cacheFiles = DistributedCache.*getLocalCacheFiles*(conf);****
>
> *if* (*null* != cacheFiles && cacheFiles.length > 0) {****
>
> *for* (Path cachePath : cacheFiles) {****
>
> *if* (cachePath.getName().equals(stopwordCacheName)) {****
>
> loadStopWords(cachePath);****
>
> *break*;****
>
> }****
>
> *if* (cachePath.getName().equals(positiveCacheName)) {****
>
> loadPositiveWords(cachePath);****
>
> *break*;****
>
> }****
>
> *if* (cachePath.getName().equals(negativeCacheName)) {****
>
> loadNegativeWords(cachePath);****
>
> *break*;****
>
> }****
>
> *if* (cachePath.getName().equals(searchCacheName)) {****
>
> loadSearchWords(cachePath);****
>
> *break*;****
>
> }****
>
> }****
>
> ** **
>
> ** **
>
> //Mapper logic goes here****
>
> }****
>
> ** **
>
> Config method should get called before the mapper method by default right?
> ****
>
> But that’s not happening….. and the loggers I have used inside mapper are
> not getting printed on console and there are no loga in userlogs. Userlogs
> is empty. Logger statements(*sLogger*.debug("*****************uploaded
> positive words");) in the driver class are getting printed on the console
> though.****
>
> ** **
>
> ** **
>
> Regards,****
>
> Shreya****
>
> ** **
> This e-mail and any files transmitted with it are for the sole use of the
> intended recipient(s) and may contain confidential and privileged
> information. If you are not the intended recipient(s), please reply to the
> sender and destroy all copies of the original message. Any unauthorized
> review, use, disclosure, dissemination, forwarding, printing or copying of
> this email, and/or any action taken in reliance on the contents of this
> e-mail is strictly prohibited and may be unlawful.
>
--
Harsh J