You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "arkady borkovsky (JIRA)" <ji...@apache.org> on 2006/08/29 23:21:22 UTC
[jira] Created: (HADOOP-491) there should be a protocol for a map
or reduce task to tell the jobtracker its alive
there should be a protocol for a map or reduce task to tell the jobtracker its alive
------------------------------------------------------------------------------------
Key: HADOOP-491
URL: http://issues.apache.org/jira/browse/HADOOP-491
Project: Hadoop
Issue Type: New Feature
Components: mapred
Reporter: arkady borkovsky
Currently, the jobtracker relies on task to write something to the output to know the tasks are still alive.
There is a 600 seconds timeout preset.
Some tasks spend longer that 600 seconds without any output while being perfectly healthy.
It would be good to have an API for a task to explicitly tell the jobtracker it is alive.
There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-491) streaming jobs should allow programs
that don't do any IO for a long time
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12470477 ]
Owen O'Malley commented on HADOOP-491:
--------------------------------------
In Java jobs, if it either reads input, writes output, or explicitly calls the Reporter object. However, in streaming it is a little more complicated, because everything is asynchronous. In theory the same is true in streaming, but the details are more subtle.
> streaming jobs should allow programs that don't do any IO for a long time
> -------------------------------------------------------------------------
>
> Key: HADOOP-491
> URL: https://issues.apache.org/jira/browse/HADOOP-491
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/streaming
> Reporter: arkady borkovsky
> Assigned To: Arun C Murthy
> Fix For: 0.12.0
>
> Attachments: HADOOP-491_20070205_1.patch, HADOOP-491_20070206_2.patch
>
>
> The jobtracker relies on task to send heartbeats to know the tasks are still alive.
> There is a 600 seconds timeout preset.
> hadoop streaming also uses input to or output from the program it spawns to indicate progress, sending appropriate heartbeats.
> Some spawned programs spend longer that 600 seconds without any output while being perfectly healthy.
> It would be good to enhance the interface between hadoop streaming and the programs it spawns to track a healthy program in the absense of output.
> There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-491) streaming jobs should allow programs
that don't do any IO for a long time
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-491:
--------------------------------
Status: Open (was: Patch Available)
I think instead of adding a new configuration option, this should simply use mapred.task.timeout, perhaps changing that to be per-job rather than per-tracker, and implementing zero-means-never.
> streaming jobs should allow programs that don't do any IO for a long time
> -------------------------------------------------------------------------
>
> Key: HADOOP-491
> URL: https://issues.apache.org/jira/browse/HADOOP-491
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/streaming
> Reporter: arkady borkovsky
> Assigned To: Arun C Murthy
> Attachments: HADOOP-491_20070131_1.patch
>
>
> The jobtracker relies on task to send heartbeats to know the tasks are still alive.
> There is a 600 seconds timeout preset.
> hadoop streaming also uses input to or output from the program it spawns to indicate progress, sending appropriate heartbeats.
> Some spawned programs spend longer that 600 seconds without any output while being perfectly healthy.
> It would be good to enhance the interface between hadoop streaming and the programs it spawns to track a healthy program in the absense of output.
> There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-491) there should be a protocol for a map
or reduce task to tell the jobtracker its alive
Posted by "Yoram Arnon (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-491?page=all ]
Yoram Arnon updated HADOOP-491:
-------------------------------
Component/s: contrib/streaming
(was: mapred)
Description:
The jobtracker relies on task to send heartbeats to know the tasks are still alive.
There is a 600 seconds timeout preset.
hadoop streaming also uses output from the program it spawns to indicate progress, sending appropriate heartbeats.
Some spawned programs spend longer that 600 seconds without any output while being perfectly healthy.
It would be good to enhance the interface between hadoop streaming and the programs it spawns to track a healthy program in the absense of output.
There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.
was:
Currently, the jobtracker relies on task to write something to the output to know the tasks are still alive.
There is a 600 seconds timeout preset.
Some tasks spend longer that 600 seconds without any output while being perfectly healthy.
It would be good to have an API for a task to explicitly tell the jobtracker it is alive.
There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.
> there should be a protocol for a map or reduce task to tell the jobtracker its alive
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-491
> URL: http://issues.apache.org/jira/browse/HADOOP-491
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/streaming
> Reporter: arkady borkovsky
>
> The jobtracker relies on task to send heartbeats to know the tasks are still alive.
> There is a 600 seconds timeout preset.
> hadoop streaming also uses output from the program it spawns to indicate progress, sending appropriate heartbeats.
> Some spawned programs spend longer that 600 seconds without any output while being perfectly healthy.
> It would be good to enhance the interface between hadoop streaming and the programs it spawns to track a healthy program in the absense of output.
> There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-491) streaming jobs should allow programs
that don't do any IO for a long time
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-491:
---------------------------------
Attachment: (was: HADOOP-491_20070131_1.patch)
> streaming jobs should allow programs that don't do any IO for a long time
> -------------------------------------------------------------------------
>
> Key: HADOOP-491
> URL: https://issues.apache.org/jira/browse/HADOOP-491
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/streaming
> Reporter: arkady borkovsky
> Assigned To: Arun C Murthy
>
> The jobtracker relies on task to send heartbeats to know the tasks are still alive.
> There is a 600 seconds timeout preset.
> hadoop streaming also uses input to or output from the program it spawns to indicate progress, sending appropriate heartbeats.
> Some spawned programs spend longer that 600 seconds without any output while being perfectly healthy.
> It would be good to enhance the interface between hadoop streaming and the programs it spawns to track a healthy program in the absense of output.
> There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-491) streaming jobs should allow programs
that don't do any IO for a long time
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-491:
---------------------------------
Fix Version/s: 0.11.1
Affects Version/s: 0.11.0
Status: Patch Available (was: Open)
> streaming jobs should allow programs that don't do any IO for a long time
> -------------------------------------------------------------------------
>
> Key: HADOOP-491
> URL: https://issues.apache.org/jira/browse/HADOOP-491
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.11.0
> Reporter: arkady borkovsky
> Assigned To: Arun C Murthy
> Fix For: 0.11.1
>
> Attachments: HADOOP-491_20070205_1.patch
>
>
> The jobtracker relies on task to send heartbeats to know the tasks are still alive.
> There is a 600 seconds timeout preset.
> hadoop streaming also uses input to or output from the program it spawns to indicate progress, sending appropriate heartbeats.
> Some spawned programs spend longer that 600 seconds without any output while being perfectly healthy.
> It would be good to enhance the interface between hadoop streaming and the programs it spawns to track a healthy program in the absense of output.
> There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-491) streaming jobs should allow programs
that don't do any IO for a long time
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-491:
---------------------------------
Attachment: HADOOP-491_20070205_1.patch
Thanks for the feedback Doug (actually the old patch was horribly whacked); here is another patch which uses 'mapred.task.timeout', which is now per-job and not per-tracker (hence, I've removed TaskTracker.taskTimeout) and has the necessary 'zero-means-never' semantics.
> streaming jobs should allow programs that don't do any IO for a long time
> -------------------------------------------------------------------------
>
> Key: HADOOP-491
> URL: https://issues.apache.org/jira/browse/HADOOP-491
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.11.0
> Reporter: arkady borkovsky
> Assigned To: Arun C Murthy
> Fix For: 0.11.1
>
> Attachments: HADOOP-491_20070205_1.patch
>
>
> The jobtracker relies on task to send heartbeats to know the tasks are still alive.
> There is a 600 seconds timeout preset.
> hadoop streaming also uses input to or output from the program it spawns to indicate progress, sending appropriate heartbeats.
> Some spawned programs spend longer that 600 seconds without any output while being perfectly healthy.
> It would be good to enhance the interface between hadoop streaming and the programs it spawns to track a healthy program in the absense of output.
> There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-491) streaming jobs should allow programs
that don't do any IO for a long time
Posted by "Sanjay Dahiya (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Dahiya updated HADOOP-491:
---------------------------------
Status: Patch Available (was: Open)
> streaming jobs should allow programs that don't do any IO for a long time
> -------------------------------------------------------------------------
>
> Key: HADOOP-491
> URL: https://issues.apache.org/jira/browse/HADOOP-491
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/streaming
> Reporter: arkady borkovsky
> Assigned To: Arun C Murthy
> Attachments: HADOOP-491_20070131_1.patch
>
>
> The jobtracker relies on task to send heartbeats to know the tasks are still alive.
> There is a 600 seconds timeout preset.
> hadoop streaming also uses input to or output from the program it spawns to indicate progress, sending appropriate heartbeats.
> Some spawned programs spend longer that 600 seconds without any output while being perfectly healthy.
> It would be good to enhance the interface between hadoop streaming and the programs it spawns to track a healthy program in the absense of output.
> There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-491) streaming jobs should allow programs
that don't do any IO for a long time
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-491:
--------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks Arun.
> streaming jobs should allow programs that don't do any IO for a long time
> -------------------------------------------------------------------------
>
> Key: HADOOP-491
> URL: https://issues.apache.org/jira/browse/HADOOP-491
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/streaming
> Reporter: arkady borkovsky
> Assigned To: Arun C Murthy
> Fix For: 0.12.0
>
> Attachments: HADOOP-491_20070205_1.patch, HADOOP-491_20070206_2.patch, HADOOP-491_20070212_3.patch
>
>
> The jobtracker relies on task to send heartbeats to know the tasks are still alive.
> There is a 600 seconds timeout preset.
> hadoop streaming also uses input to or output from the program it spawns to indicate progress, sending appropriate heartbeats.
> Some spawned programs spend longer that 600 seconds without any output while being perfectly healthy.
> It would be good to enhance the interface between hadoop streaming and the programs it spawns to track a healthy program in the absense of output.
> There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-491) streaming jobs should allow programs
that don't do any IO for a long time
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12471788 ]
Owen O'Malley commented on HADOOP-491:
--------------------------------------
This looks good +1
> streaming jobs should allow programs that don't do any IO for a long time
> -------------------------------------------------------------------------
>
> Key: HADOOP-491
> URL: https://issues.apache.org/jira/browse/HADOOP-491
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/streaming
> Reporter: arkady borkovsky
> Assigned To: Arun C Murthy
> Fix For: 0.12.0
>
> Attachments: HADOOP-491_20070205_1.patch, HADOOP-491_20070206_2.patch
>
>
> The jobtracker relies on task to send heartbeats to know the tasks are still alive.
> There is a 600 seconds timeout preset.
> hadoop streaming also uses input to or output from the program it spawns to indicate progress, sending appropriate heartbeats.
> Some spawned programs spend longer that 600 seconds without any output while being perfectly healthy.
> It would be good to enhance the interface between hadoop streaming and the programs it spawns to track a healthy program in the absense of output.
> There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-491) streaming jobs should allow programs
that don't do any IO for a long time
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy reassigned HADOOP-491:
------------------------------------
Assignee: Arun C Murthy (was: Sanjay Dahiya)
> streaming jobs should allow programs that don't do any IO for a long time
> -------------------------------------------------------------------------
>
> Key: HADOOP-491
> URL: https://issues.apache.org/jira/browse/HADOOP-491
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/streaming
> Reporter: arkady borkovsky
> Assigned To: Arun C Murthy
>
> The jobtracker relies on task to send heartbeats to know the tasks are still alive.
> There is a 600 seconds timeout preset.
> hadoop streaming also uses input to or output from the program it spawns to indicate progress, sending appropriate heartbeats.
> Some spawned programs spend longer that 600 seconds without any output while being perfectly healthy.
> It would be good to enhance the interface between hadoop streaming and the programs it spawns to track a healthy program in the absense of output.
> There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-491) streaming jobs should allow programs
that don't do any IO for a long time
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-491:
---------------------------------
Attachment: HADOOP-491_20070206_2.patch
Another patch without the default for streaming...
> streaming jobs should allow programs that don't do any IO for a long time
> -------------------------------------------------------------------------
>
> Key: HADOOP-491
> URL: https://issues.apache.org/jira/browse/HADOOP-491
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.11.0
> Reporter: arkady borkovsky
> Assigned To: Arun C Murthy
> Fix For: 0.11.1
>
> Attachments: HADOOP-491_20070205_1.patch, HADOOP-491_20070206_2.patch
>
>
> The jobtracker relies on task to send heartbeats to know the tasks are still alive.
> There is a 600 seconds timeout preset.
> hadoop streaming also uses input to or output from the program it spawns to indicate progress, sending appropriate heartbeats.
> Some spawned programs spend longer that 600 seconds without any output while being perfectly healthy.
> It would be good to enhance the interface between hadoop streaming and the programs it spawns to track a healthy program in the absense of output.
> There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-491) streaming jobs should allow programs
that don't do any IO for a long time
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12470357 ]
Doug Cutting commented on HADOOP-491:
-------------------------------------
Didn't we talk at some point about using stderr for progress from streaming? Did that ever happen? Would that address this?
> streaming jobs should allow programs that don't do any IO for a long time
> -------------------------------------------------------------------------
>
> Key: HADOOP-491
> URL: https://issues.apache.org/jira/browse/HADOOP-491
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/streaming
> Reporter: arkady borkovsky
> Assigned To: Arun C Murthy
> Fix For: 0.12.0
>
> Attachments: HADOOP-491_20070205_1.patch, HADOOP-491_20070206_2.patch
>
>
> The jobtracker relies on task to send heartbeats to know the tasks are still alive.
> There is a 600 seconds timeout preset.
> hadoop streaming also uses input to or output from the program it spawns to indicate progress, sending appropriate heartbeats.
> Some spawned programs spend longer that 600 seconds without any output while being perfectly healthy.
> It would be good to enhance the interface between hadoop streaming and the programs it spawns to track a healthy program in the absense of output.
> There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-491) streaming jobs should allow programs
that don't do any IO for a long time
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-491:
---------------------------------
Attachment: HADOOP-491_20070131_1.patch
Here is a straight-forward patch which lets a per-job configuration knob for task-launch timeout, and is set to '0' for streaming jobs...
> streaming jobs should allow programs that don't do any IO for a long time
> -------------------------------------------------------------------------
>
> Key: HADOOP-491
> URL: https://issues.apache.org/jira/browse/HADOOP-491
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/streaming
> Reporter: arkady borkovsky
> Assigned To: Arun C Murthy
> Attachments: HADOOP-491_20070131_1.patch
>
>
> The jobtracker relies on task to send heartbeats to know the tasks are still alive.
> There is a 600 seconds timeout preset.
> hadoop streaming also uses input to or output from the program it spawns to indicate progress, sending appropriate heartbeats.
> Some spawned programs spend longer that 600 seconds without any output while being perfectly healthy.
> It would be good to enhance the interface between hadoop streaming and the programs it spawns to track a healthy program in the absense of output.
> There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-491) streaming jobs should allow programs
that don't do any IO for a long time
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12469716 ]
Hadoop QA commented on HADOOP-491:
----------------------------------
+1, because http://issues.apache.org/jira/secure/attachment/12350017/HADOOP-491_20070131_1.patch applied and successfully tested against trunk revision r502402.
> streaming jobs should allow programs that don't do any IO for a long time
> -------------------------------------------------------------------------
>
> Key: HADOOP-491
> URL: https://issues.apache.org/jira/browse/HADOOP-491
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/streaming
> Reporter: arkady borkovsky
> Assigned To: Arun C Murthy
> Attachments: HADOOP-491_20070131_1.patch
>
>
> The jobtracker relies on task to send heartbeats to know the tasks are still alive.
> There is a 600 seconds timeout preset.
> hadoop streaming also uses input to or output from the program it spawns to indicate progress, sending appropriate heartbeats.
> Some spawned programs spend longer that 600 seconds without any output while being perfectly healthy.
> It would be good to enhance the interface between hadoop streaming and the programs it spawns to track a healthy program in the absense of output.
> There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-491) streaming jobs should allow programs
that don't do any IO for a long time
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-491:
---------------------------------
Status: Patch Available (was: Open)
> streaming jobs should allow programs that don't do any IO for a long time
> -------------------------------------------------------------------------
>
> Key: HADOOP-491
> URL: https://issues.apache.org/jira/browse/HADOOP-491
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/streaming
> Reporter: arkady borkovsky
> Assigned To: Arun C Murthy
> Fix For: 0.12.0
>
> Attachments: HADOOP-491_20070205_1.patch, HADOOP-491_20070206_2.patch, HADOOP-491_20070212_3.patch
>
>
> The jobtracker relies on task to send heartbeats to know the tasks are still alive.
> There is a 600 seconds timeout preset.
> hadoop streaming also uses input to or output from the program it spawns to indicate progress, sending appropriate heartbeats.
> Some spawned programs spend longer that 600 seconds without any output while being perfectly healthy.
> It would be good to enhance the interface between hadoop streaming and the programs it spawns to track a healthy program in the absense of output.
> There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-491) streaming jobs should allow programs
that don't do any IO for a long time
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12470453 ]
Owen O'Malley commented on HADOOP-491:
--------------------------------------
That did happen and it helped. Because streaming can use binaries that the user doesn't control, they don't always have the option to add printing bytes to stderr. Furthermore, as we start sending more of the output back to the user's console, it is less clear that having the application send data to stderr is a good idea.
> streaming jobs should allow programs that don't do any IO for a long time
> -------------------------------------------------------------------------
>
> Key: HADOOP-491
> URL: https://issues.apache.org/jira/browse/HADOOP-491
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/streaming
> Reporter: arkady borkovsky
> Assigned To: Arun C Murthy
> Fix For: 0.12.0
>
> Attachments: HADOOP-491_20070205_1.patch, HADOOP-491_20070206_2.patch
>
>
> The jobtracker relies on task to send heartbeats to know the tasks are still alive.
> There is a 600 seconds timeout preset.
> hadoop streaming also uses input to or output from the program it spawns to indicate progress, sending appropriate heartbeats.
> Some spawned programs spend longer that 600 seconds without any output while being perfectly healthy.
> It would be good to enhance the interface between hadoop streaming and the programs it spawns to track a healthy program in the absense of output.
> There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-491) streaming jobs should allow programs
that don't do any IO for a long time
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-491:
---------------------------------
Attachment: HADOOP-491_20070212_3.patch
My bad Doug; fixed now...
> streaming jobs should allow programs that don't do any IO for a long time
> -------------------------------------------------------------------------
>
> Key: HADOOP-491
> URL: https://issues.apache.org/jira/browse/HADOOP-491
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/streaming
> Reporter: arkady borkovsky
> Assigned To: Arun C Murthy
> Fix For: 0.12.0
>
> Attachments: HADOOP-491_20070205_1.patch, HADOOP-491_20070206_2.patch, HADOOP-491_20070212_3.patch
>
>
> The jobtracker relies on task to send heartbeats to know the tasks are still alive.
> There is a 600 seconds timeout preset.
> hadoop streaming also uses input to or output from the program it spawns to indicate progress, sending appropriate heartbeats.
> Some spawned programs spend longer that 600 seconds without any output while being perfectly healthy.
> It would be good to enhance the interface between hadoop streaming and the programs it spawns to track a healthy program in the absense of output.
> There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-491) streaming jobs should allow programs
that don't do any IO for a long time
Posted by "Yoram Arnon (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-491?page=all ]
Yoram Arnon updated HADOOP-491:
-------------------------------
Summary: streaming jobs should allow programs that don't do any IO for a long time (was: there should be a protocol for a map or reduce task to tell the jobtracker its alive)
Description:
The jobtracker relies on task to send heartbeats to know the tasks are still alive.
There is a 600 seconds timeout preset.
hadoop streaming also uses input to or output from the program it spawns to indicate progress, sending appropriate heartbeats.
Some spawned programs spend longer that 600 seconds without any output while being perfectly healthy.
It would be good to enhance the interface between hadoop streaming and the programs it spawns to track a healthy program in the absense of output.
There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.
was:
The jobtracker relies on task to send heartbeats to know the tasks are still alive.
There is a 600 seconds timeout preset.
hadoop streaming also uses output from the program it spawns to indicate progress, sending appropriate heartbeats.
Some spawned programs spend longer that 600 seconds without any output while being perfectly healthy.
It would be good to enhance the interface between hadoop streaming and the programs it spawns to track a healthy program in the absense of output.
There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.
> streaming jobs should allow programs that don't do any IO for a long time
> -------------------------------------------------------------------------
>
> Key: HADOOP-491
> URL: http://issues.apache.org/jira/browse/HADOOP-491
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/streaming
> Reporter: arkady borkovsky
>
> The jobtracker relies on task to send heartbeats to know the tasks are still alive.
> There is a 600 seconds timeout preset.
> hadoop streaming also uses input to or output from the program it spawns to indicate progress, sending appropriate heartbeats.
> Some spawned programs spend longer that 600 seconds without any output while being perfectly healthy.
> It would be good to enhance the interface between hadoop streaming and the programs it spawns to track a healthy program in the absense of output.
> There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-491) streaming jobs should allow programs
that don't do any IO for a long time
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12470464 ]
Raghu Angadi commented on HADOOP-491:
-------------------------------------
If a streaming jobs reads from Input but does not write to output for a long time, is that considered progress?
> streaming jobs should allow programs that don't do any IO for a long time
> -------------------------------------------------------------------------
>
> Key: HADOOP-491
> URL: https://issues.apache.org/jira/browse/HADOOP-491
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/streaming
> Reporter: arkady borkovsky
> Assigned To: Arun C Murthy
> Fix For: 0.12.0
>
> Attachments: HADOOP-491_20070205_1.patch, HADOOP-491_20070206_2.patch
>
>
> The jobtracker relies on task to send heartbeats to know the tasks are still alive.
> There is a 600 seconds timeout preset.
> hadoop streaming also uses input to or output from the program it spawns to indicate progress, sending appropriate heartbeats.
> Some spawned programs spend longer that 600 seconds without any output while being perfectly healthy.
> It would be good to enhance the interface between hadoop streaming and the programs it spawns to track a healthy program in the absense of output.
> There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-491) streaming jobs should allow programs
that don't do any IO for a long time
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-491:
---------------------------------
Fix Version/s: (was: 0.11.1)
0.12.0
Affects Version/s: (was: 0.11.0)
> streaming jobs should allow programs that don't do any IO for a long time
> -------------------------------------------------------------------------
>
> Key: HADOOP-491
> URL: https://issues.apache.org/jira/browse/HADOOP-491
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/streaming
> Reporter: arkady borkovsky
> Assigned To: Arun C Murthy
> Fix For: 0.12.0
>
> Attachments: HADOOP-491_20070205_1.patch, HADOOP-491_20070206_2.patch
>
>
> The jobtracker relies on task to send heartbeats to know the tasks are still alive.
> There is a 600 seconds timeout preset.
> hadoop streaming also uses input to or output from the program it spawns to indicate progress, sending appropriate heartbeats.
> Some spawned programs spend longer that 600 seconds without any output while being perfectly healthy.
> It would be good to enhance the interface between hadoop streaming and the programs it spawns to track a healthy program in the absense of output.
> There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-491) streaming jobs should allow programs
that don't do any IO for a long time
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12470338 ]
Doug Cutting commented on HADOOP-491:
-------------------------------------
So streaming jobs should have no timeout by default? I can sort of see adding the feature of disabling task timeouts, and also of facillitating this from streaming, but do streaming applications really never hang? Should we change the default for all applications, not just streaming? I'm trying to understand the logic here.
Also, as a new feature, shouldn't this be targetted for 0.12.0?
> streaming jobs should allow programs that don't do any IO for a long time
> -------------------------------------------------------------------------
>
> Key: HADOOP-491
> URL: https://issues.apache.org/jira/browse/HADOOP-491
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.11.0
> Reporter: arkady borkovsky
> Assigned To: Arun C Murthy
> Fix For: 0.11.1
>
> Attachments: HADOOP-491_20070205_1.patch
>
>
> The jobtracker relies on task to send heartbeats to know the tasks are still alive.
> There is a 600 seconds timeout preset.
> hadoop streaming also uses input to or output from the program it spawns to indicate progress, sending appropriate heartbeats.
> Some spawned programs spend longer that 600 seconds without any output while being perfectly healthy.
> It would be good to enhance the interface between hadoop streaming and the programs it spawns to track a healthy program in the absense of output.
> There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-491) streaming jobs should allow programs
that don't do any IO for a long time
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12462099 ]
Owen O'Malley commented on HADOOP-491:
--------------------------------------
I think the right way to address this is to support timeouts of "0" that mean there should be no task timeouts. The default in streaming can be set to 0, since it is impossible for the streaming process to call reporter.progress().
> streaming jobs should allow programs that don't do any IO for a long time
> -------------------------------------------------------------------------
>
> Key: HADOOP-491
> URL: https://issues.apache.org/jira/browse/HADOOP-491
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/streaming
> Reporter: arkady borkovsky
> Assigned To: Sanjay Dahiya
>
> The jobtracker relies on task to send heartbeats to know the tasks are still alive.
> There is a 600 seconds timeout preset.
> hadoop streaming also uses input to or output from the program it spawns to indicate progress, sending appropriate heartbeats.
> Some spawned programs spend longer that 600 seconds without any output while being perfectly healthy.
> It would be good to enhance the interface between hadoop streaming and the programs it spawns to track a healthy program in the absense of output.
> There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-491) there should be a protocol for a map
or reduce task to tell the jobtracker its alive
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-491?page=comments#action_12431391 ]
Doug Cutting commented on HADOOP-491:
-------------------------------------
There already is an API for tasks to say they're alive: Reporter.setStatus() and progress().
This may not be well documented, so perhaps this is a documentation bug?
> there should be a protocol for a map or reduce task to tell the jobtracker its alive
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-491
> URL: http://issues.apache.org/jira/browse/HADOOP-491
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Reporter: arkady borkovsky
>
> Currently, the jobtracker relies on task to write something to the output to know the tasks are still alive.
> There is a 600 seconds timeout preset.
> Some tasks spend longer that 600 seconds without any output while being perfectly healthy.
> It would be good to have an API for a task to explicitly tell the jobtracker it is alive.
> There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-491) streaming jobs should allow programs
that don't do any IO for a long time
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-491:
--------------------------------
Status: Open (was: Patch Available)
The call to Configuration#getLong() should be outside the loop over running tasks. Optimally, we'd make this once per job, but, at a minimum, it should be outside the loop.
> streaming jobs should allow programs that don't do any IO for a long time
> -------------------------------------------------------------------------
>
> Key: HADOOP-491
> URL: https://issues.apache.org/jira/browse/HADOOP-491
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/streaming
> Reporter: arkady borkovsky
> Assigned To: Arun C Murthy
> Fix For: 0.12.0
>
> Attachments: HADOOP-491_20070205_1.patch, HADOOP-491_20070206_2.patch
>
>
> The jobtracker relies on task to send heartbeats to know the tasks are still alive.
> There is a 600 seconds timeout preset.
> hadoop streaming also uses input to or output from the program it spawns to indicate progress, sending appropriate heartbeats.
> Some spawned programs spend longer that 600 seconds without any output while being perfectly healthy.
> It would be good to enhance the interface between hadoop streaming and the programs it spawns to track a healthy program in the absense of output.
> There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-491) streaming jobs should allow programs
that don't do any IO for a long time
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12472384 ]
Hadoop QA commented on HADOOP-491:
----------------------------------
+1, because http://issues.apache.org/jira/secure/attachment/12350901/HADOOP-491_20070212_3.patch applied and successfully tested against trunk revision r505557.
> streaming jobs should allow programs that don't do any IO for a long time
> -------------------------------------------------------------------------
>
> Key: HADOOP-491
> URL: https://issues.apache.org/jira/browse/HADOOP-491
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/streaming
> Reporter: arkady borkovsky
> Assigned To: Arun C Murthy
> Fix For: 0.12.0
>
> Attachments: HADOOP-491_20070205_1.patch, HADOOP-491_20070206_2.patch, HADOOP-491_20070212_3.patch
>
>
> The jobtracker relies on task to send heartbeats to know the tasks are still alive.
> There is a 600 seconds timeout preset.
> hadoop streaming also uses input to or output from the program it spawns to indicate progress, sending appropriate heartbeats.
> Some spawned programs spend longer that 600 seconds without any output while being perfectly healthy.
> It would be good to enhance the interface between hadoop streaming and the programs it spawns to track a healthy program in the absense of output.
> There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-491) streaming jobs should allow programs
that don't do any IO for a long time
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12470351 ]
Owen O'Malley commented on HADOOP-491:
--------------------------------------
This looks good so far, it just needs the default for streaming set. The logic here is that streaming applications don't have an easy way of reporting to the framework that they are making progress. So it is far too easy for them to be killed as being dead when they are just working away. Clearly, disabling the timeout makes it easy to write applications that just get stuck, but it seems better to allow applications to get stuck rather than killing productive tasks.
> streaming jobs should allow programs that don't do any IO for a long time
> -------------------------------------------------------------------------
>
> Key: HADOOP-491
> URL: https://issues.apache.org/jira/browse/HADOOP-491
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/streaming
> Reporter: arkady borkovsky
> Assigned To: Arun C Murthy
> Fix For: 0.12.0
>
> Attachments: HADOOP-491_20070205_1.patch, HADOOP-491_20070206_2.patch
>
>
> The jobtracker relies on task to send heartbeats to know the tasks are still alive.
> There is a 600 seconds timeout preset.
> hadoop streaming also uses input to or output from the program it spawns to indicate progress, sending appropriate heartbeats.
> Some spawned programs spend longer that 600 seconds without any output while being perfectly healthy.
> It would be good to enhance the interface between hadoop streaming and the programs it spawns to track a healthy program in the absense of output.
> There are certain dangers with this protocol: e.g. a task can run a separate thread that does nothing but send "i'm alive" message. This would be a user bug to abuse the API in such way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.