Reducer - getMapOutput

[Grandl Robert <rg...@yahoo.com>]

Hi,

I am trying to modify the code for data transfer of intermediate output. 


In this respect, on the reduce side in getMapOuput I want to have the connection with the TaskTracker(setupSecureConnection), but then on doGet the TaskTracker to be able to delay the response back to the client. I tried with Thread.sleep(time), before setting up the response.setHeader(FROM_MAP_TASK, mapId) and the subsequent headers/content. But it seems the reducer will continue to read the headers immediately and get NULL.

Do you know how can I do the reducer to wait for TaskTracker to update this information before reading it. I tried with a while loop and reading the connection.getHeaderField(...) till not null but seems it always remains in the loop :(.

Probably is more a HttpServlet related problem but I am not very familiar with that. 


Do you have any idea how can I do it ?

Thanks,
Robert

Re: Reducer - getMapOutput

[Grandl Robert <rg...@yahoo.com>]

Hey Arun,

What I am trying to do is the following:

- the reducer will initiate a request for some content(specified by a URL to a tasktracker) 

- the taskTracker whenever receive the request for the content (in doGet) will launch the task execution to get the content the reducer request in the URL. 

(basically the reducer will request for the content before the content exist on the taskTracker and taskTracker will be triggered by this request in order to execute the task and produce the content). 

- after the taskTracker will finish the task, the content is returned on the open connection back to the reducer. 


Therefore I would like to produce the connection, but then "delay" the answer till the content is available and can be returned. 

Thanks,

Robert



________________________________
 From: Arun C Murthy <ac...@hortonworks.com>
To: mapreduce-user@hadoop.apache.org 
Sent: Monday, July 30, 2012 1:45 PM
Subject: Re: Reducer - getMapOutput
 

Hey Robert,

 Can you pls share what you are trying to do?

Arun


On Jul 30, 2012, at 9:10 AM, Grandl Robert wrote:

Hi,
>
>
>I am trying to modify the code for data transfer of intermediate output. 
>
>
>
>In this respect, on the reduce side in getMapOuput I want to have the connection with the TaskTracker(setupSecureConnection), but then on doGet the TaskTracker to be able to delay the response back to the client. I tried with Thread.sleep(time), before setting up the response.setHeader(FROM_MAP_TASK, mapId) and the subsequent headers/content. But it seems the reducer will continue to read the headers immediately and get NULL.
>
>
>Do you know how can I do the reducer to wait for TaskTracker to update this information before reading it. I tried with a while loop and reading the connection.getHeaderField(...) till not null but seems it always remains in the loop :(.
>
>
>Probably is more a HttpServlet related problem but I am not very familiar with that. 
>
>
>
>Do you have any idea how can I do it ?
>
>
>Thanks,
>Robert
>

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Reducer - getMapOutput

[Arun C Murthy <ac...@hortonworks.com>]

Hey Robert,

 Can you pls share what you are trying to do?

Arun

On Jul 30, 2012, at 9:10 AM, Grandl Robert wrote:

> Hi,
> 
> I am trying to modify the code for data transfer of intermediate output. 
> 
> In this respect, on the reduce side in getMapOuput I want to have the connection with the TaskTracker(setupSecureConnection), but then on doGet the TaskTracker to be able to delay the response back to the client. I tried with Thread.sleep(time), before setting up the response.setHeader(FROM_MAP_TASK, mapId) and the subsequent headers/content. But it seems the reducer will continue to read the headers immediately and get NULL.
> 
> Do you know how can I do the reducer to wait for TaskTracker to update this information before reading it. I tried with a while loop and reading the connection.getHeaderField(...) till not null but seems it always remains in the loop :(.
> 
> Probably is more a HttpServlet related problem but I am not very familiar with that. 
> 
> Do you have any idea how can I do it ?
> 
> Thanks,
> Robert

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/