Question About Passing Properties to a Mapper

[Geoffry Roberts <ge...@gmail.com>]

All,

I am attempting to pass a string value from my driver to each one of my
mappers and it is not working.  I can set the value, but when I read it back
it returns null.  the value is not null when I set() it and I am using the
correct key when I attempt to get() it.  This should be a simple, straight
forward thing but it isn't working.

I am, however, passing another string value successfully. The difference
seems to be that the value that passes successfully is short and simple
whereas the other, the failure is long and in fact is an xml document
containing new lines, et al.

Question: What must I do to pass an xml document? url encode it? strip the
new lines? other?

Here's my code.

In my driver:

*public void run() {
  ...
**  String scenarioName="56crmid";
**  String **scenarioText = "xml document here; new lines and all."**
**  Job job = new Job(new Configuration());
**  job.getConfiguration().set("**scenarioName**", **scenarioName**);  **
**  job.getConfiguration().set("scenario", scenarioText);
  ...
  job.setMapperClass(StageMapper.class);
  ...
  job.waitForCompletion(true);
}*

In my mapper:

*protected void setup(Context ctx) throws IOException, InterruptedException
{
  ...
**  String **scenarioName**= ctx.getConfiguration().get("**scenarioName**");
*
*  // ****scenarioName** **is NOT null*
*  String scenarioText = ctx.getConfiguration().get("scenario");
  // **scenarioText is null*
*  ...
}*

-- 
Geoffry Roberts

Re: Question About Passing Properties to a Mapper

[Geoffry Roberts <ge...@gmail.com>]

All,

Thanks Harsh for your response.  I have, however, solved the problem and I
shall now share.

The upshot is the property in question, when I did the get() was in fact not
null.  It contained the full XML document.  I thought it was either null or
a zero length string because I was logging the value like this:

LOG.info("scenarioText=" + scenarioText);

When I viewed the log file I would see:

... scenarioText=

This made me think the value was null.  But here's the kicker.  I was using
my browser to view the log, drilling down from
http://qq000:50070/dfshealth.jsp.  When you do this... well it can be
misleading.  If I cat the file from the command line I see what I would
expect to see,  i.e. the text of my document.

This now makes two things about viewing log files in the browser that I wish
were different. (1) the aforementioned, and (2) the wrapping of the logged
output as if there were no new line characters in the file.  This has to be
something that Hadoop is doing because if you simply open the same file with
a browser, the file displays normally--no wrap lines and displayed xml.

On 11 May 2011 09:15, Harsh J <ha...@cloudera.com> wrote:

> Hello Geoffry,
>
> On Wed, May 11, 2011 at 8:40 PM, Geoffry Roberts
> <ge...@gmail.com> wrote:
> > All,
> >
> > I am attempting to pass a string value from my driver to each one of my
> > mappers and it is not working.  I can set the value, but when I read it
> back
> > it returns null.  the value is not null when I set() it and I am using
> the
> > correct key when I attempt to get() it.  This should be a simple,
> straight
> > forward thing but it isn't working.
> >
> > I am, however, passing another string value successfully. The difference
> > seems to be that the value that passes successfully is short and simple
> > whereas the other, the failure is long and in fact is an xml document
> > containing new lines, et al.
> >
> > Question: What must I do to pass an xml document? url encode it? strip
> the
> > new lines? other?
>
> Encoding would be a good idea for arbitrary strings containing
> possibly bad XML chars., etc..
>
> (I prefer doing Base64 enc., that one works well with XML I think)
>
> --
> Harsh J
>



-- 
Geoffry Roberts

Re: Question About Passing Properties to a Mapper

[Harsh J <ha...@cloudera.com>]

Hello Geoffry,

On Wed, May 11, 2011 at 8:40 PM, Geoffry Roberts
<ge...@gmail.com> wrote:
> All,
>
> I am attempting to pass a string value from my driver to each one of my
> mappers and it is not working.  I can set the value, but when I read it back
> it returns null.  the value is not null when I set() it and I am using the
> correct key when I attempt to get() it.  This should be a simple, straight
> forward thing but it isn't working.
>
> I am, however, passing another string value successfully. The difference
> seems to be that the value that passes successfully is short and simple
> whereas the other, the failure is long and in fact is an xml document
> containing new lines, et al.
>
> Question: What must I do to pass an xml document? url encode it? strip the
> new lines? other?

Encoding would be a good idea for arbitrary strings containing
possibly bad XML chars., etc..

(I prefer doing Base64 enc., that one works well with XML I think)

-- 
Harsh J