You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Aroop Maliakkal (JIRA)" <ji...@apache.org> on 2009/03/12 16:18:50 UTC

[jira] Created: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Provide a node health check script and run it periodically to check the node health status
------------------------------------------------------------------------------------------

                 Key: HADOOP-5478
                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
             Project: Hadoop Core
          Issue Type: New Feature
          Components: mapred
    Affects Versions: 0.20.0
            Reporter: Aroop Maliakkal


Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.



-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718749#action_12718749 ] 

Hemanth Yamijala commented on HADOOP-5478:
------------------------------------------

To summarize some of the comments above, the issue we are discussing is whether the node health checker script should be launched as a separate process from the tasktracker (TT) itself, rather than as a thread in the TT, as done in the patch currently. There are some motivations for doing the same:

- A periodic process launch from a java service like the TT has caused problems in the past - for e.g. look at HADOOP-5059.
- Owen also mentioned instances where they'd seen the service itself lock up due to the process launch (and the underlying fork()/exec()) failing.

So, the proposal is to solve this problem by having the node health checker script as a separate process. This process can be configured with the following:
- Path to a script
- An interval
- TT's address for communication.

The process would periodically run the script (as done in the patch today) and report the status to the TT using RPC. To keep management simple, we can, in the first cut, launch this process from the TT itself and stop it when the TT is going down. In future, it should be possible to decouple this even more and have them run independently. The simplicity we buy in the first iteration is to not require administrators from worrying about managing this independently for the time being - until we gain some experience with how the health check script is running.

Does this sound fine ?


> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Issue Comment Edited: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720185#action_12720185 ] 

Hong Tang edited comment on HADOOP-5478 at 6/16/09 8:20 AM:
------------------------------------------------------------

bq. Hong, is this to check if the TT is alive ? In which case, did you mean another signal, like -0 or kill -3. -9 is SIGKILL and would kill the TT. Also, in that case are you suggesting that we could keep the health checker around and continue trying to report after a while ?

@hemanth sorry for not being clear. I gave a bit more thoughts on the problem, and I think the following logic may be simpler and more robust (1,2 are the current logic, 3 is my suggestion) : 

(1) periodically launch the health checking script; 
(2) reporting status that back to TT (both good and bad); 
(3) if it fails to receive response from TT, wait for X seconds, do an extra kill (to ensure TT is dead), and quit itself. 

I scanned through the code, it seems that NodeHealthChecker.stop() would be a good place to perform step (3).

      was (Author: hong.tang):
    bq Hong, is this to check if the TT is alive ? In which case, did you mean another signal, like -0 or kill -3. -9 is SIGKILL and would kill the TT. Also, in that case are you suggesting that we could keep the health checker around and continue trying to report after a while ?

@hemanth sorry for not being clear. I gave a bit more thoughts on the problem, and I think the following logic may be simpler and more robust (1,2 are the current logic, 3 is my suggestion) : 

(1) periodically launch the health checking script; 
(2) reporting status that back to TT (both good and bad); 
(3) if it fails to receive response from TT, wait for X seconds, do an extra kill (to ensure TT is dead), and quit itself. 

I scanned through the code, it seems that NodeHealthChecker.stop() would be a good place to perform step (3).
  
> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Sreekanth Ramakrishnan
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch, hadoop-5478-3.patch, hadoop-5478-4.patch, hadoop-5478-5.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708883#action_12708883 ] 

Hemanth Yamijala commented on HADOOP-5478:
------------------------------------------

bq. We really need the equivalent of dfsadmin -report for the JobTracker to provide this same level of output.

Allen, I had command line reporting in mind when writing the specs. Worst case, if the effort seems large to cover we will do this as a follow-up JIRA, restricting the information to the web UI for now.

bq. Additionally, torque ignores the exit status

If this is acceptable to all, we can do the same as well.

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708395#action_12708395 ] 

Hemanth Yamijala commented on HADOOP-5478:
------------------------------------------

We might start working on this soon. I thought it might be a good idea to share current thinking about the specs for this feature, and start a discussion.

In a brief discussion with the team, Eric and Owen, we came up with the following points:

- Provide an ability to the administrator to give a path to a script file that will be periodically run on the tasktracker. The interval of running can be configured.
- The tasktracker would run this in a separate thread and look for the exit code. If the script exits with a non-zero code, this will be reported to the JobTracker.
- Any output from the script (upon error) will be logged and if possible, also displayed on the web UI of the tasktracker.
- The jobtracker will blacklist this node when such a condition is reported.
- It will be a good idea to display the 'unhealthy' nodes on the UI.
- The tasktracker will need to continue to run the script so that if the condition is corrected, it will be reported again to the jobtracker for becoming available.
- A point came up about whether this mechanism can lead to the state of the node toggling. Maybe we can do some hysterisis, but as a follow-up for this jira.
- However, It may also be a good idea to show some stats like how oftern this node was blacklisted in the last 24 hours, and the current status by going to the tasktracker page on the web ui. This might help us decide if it's worthwhile introducing hysterisis.

Does this sound good ? Any other thoughts ?


> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720007#action_12720007 ] 

Hemanth Yamijala commented on HADOOP-5478:
------------------------------------------

bq. when the health check fails to report status to TT, try to do a "kill -9" for the TT ...

Hong, is this to check if the TT is alive ? In which case, did you mean another signal, like -0 or kill -3. -9 is SIGKILL and would kill the TT. Also, in that case are you suggesting that we could keep the health checker around and continue trying to report after a while ?

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch, hadoop-5478-3.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720098#action_12720098 ] 

Hemanth Yamijala commented on HADOOP-5478:
------------------------------------------

I've started reviewing the patch. I think the last patch missed adding the new files. Please correct it.

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch, hadoop-5478-3.patch, hadoop-5478-4.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718766#action_12718766 ] 

Hemanth Yamijala commented on HADOOP-5478:
------------------------------------------

bq. Steve's comments seem to imply that we need the capability of launch the health checking script in response to external commands. 

Hong, while this is true, we can make it work as long as we are using an RPC to communicate with the health checking process. One command in the RPC is certainly the heartbeat that will both report the status as well as let the TT know the health checker is running. The other could be a 'run script now' command. This can be added as an extension on top of the basic framework - for e.g. when the ping is added to the TT. Would that work ?

bq. Can't the status reporting be simply done via stdout/stderr with START and END markers?

I suppose so. We've seen in the past a couple of issues if we're not careful with interacting with i/o streams of sub processes. Maybe a lot of these are fixed in Hadoop, and so not really an issue. But RPC seems simpler and cleaner. Would be glad to find what others think as well.

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718752#action_12718752 ] 

Hong Tang commented on HADOOP-5478:
-----------------------------------

@owen: Steve's comments seem to imply that we need the capability of launch the health checking script in response to external commands.

@Streekanth: If I understand correctly, the new patch introduces another thread to check for command execution timeout. Is there a way to eliminate it (possibly by modifying shell execution API to accept timeouts)? 

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720731#action_12720731 ] 

Steve Loughran commented on HADOOP-5478:
----------------------------------------

> Though, at both times, the only one who knows about the trouble is the health checker and not the rest of the world.

why is why your management tools
# need to be HA toys themselves
# need to be able to ask the apps for their health
# may need to be able to do test jobs to probe system health
# may need the ability to react to failure according to the infrastructure in which HDFS is running, and your policy. 

If HDFS is running in anything that supports the EC2 APIs, if a TT is playing up I'd start by rebooting that node, if it still doesn't come up, decomission the namenode, terminate the VM and ask for a new one. That's a very different policy from a physical cluster, where you may want to blacklist the TT while its datanode services stays live. 

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Sreekanth Ramakrishnan
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch, hadoop-5478-3.patch, hadoop-5478-4.patch, hadoop-5478-5.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sreekanth Ramakrishnan updated HADOOP-5478:
-------------------------------------------

    Attachment: hadoop-5478-5.patch

Patch which includes the missed out file.

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch, hadoop-5478-3.patch, hadoop-5478-4.patch, hadoop-5478-5.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720519#action_12720519 ] 

Hong Tang commented on HADOOP-5478:
-----------------------------------

@owen Is there ever any chance the main RPC being overwhelmed and thus may not be able to respond promptly to health status information?

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Sreekanth Ramakrishnan
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch, hadoop-5478-3.patch, hadoop-5478-4.patch, hadoop-5478-5.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720061#action_12720061 ] 

Steve Loughran commented on HADOOP-5478:
----------------------------------------

1. The timeouts in Shell would seem useful on their own; every shell operation ought to have timeouts for extra robustness.

2. This would fit fairly well under the HADOOP-3628 stuff, where the monitor would start and stop with the TT lifecycle; we'd have to think about how to integrate it with the ping operation -I think returning the most recent status would be good.


3. At some point in the future, it would be good for the policy of acting on TT failure to be moved out of the JT. In infrastructure where the response to failure is to terminate that (virtual) host and ask for a new one, you react very differently to failure. It's not something the JT needs to handle, other than pass up bad news.

4. I'm not sure about all the kill -9 and shutdown hook stuff, it's getting into fragile waters. Hard to test, hard to debug, creates complex situations especially  in test runs or stuff hosted in different runtimes

* this helper script stuff must be optional; I would turn it off on my systems as I test health in different ways.
* kill handlers are best designed to do very little and be robust against odd system states -and not assume any other parts of the cluster are live.

For the curious, the way SmartFrog  manages is its health is that every component tracks the last time it was asked by its parent for its health, if that time ever exceeds a (programmed) limit then it terminates itself. Every process pings the root component; its up to that to ping its children and act on failures -and to  recognise and act on timeouts. This works OK for single host work, in a cluster you don't want any SPOFs and tend to take an aggregate view : there has to be one Namenode, one JT, "enough" workers. I have a component to check the health of a file in the filesystem; every time it's health is checked, it looks for the file it was bound to, checks that it is present and within a specified size range. This is handy for checking that files you value are there, and that the FS is visible across the network (very important on virtual servers with odd networking). I dont have anything similar for checking that TT's are good, the best check would be test work.

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch, hadoop-5478-3.patch, hadoop-5478-4.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sreekanth Ramakrishnan updated HADOOP-5478:
-------------------------------------------

    Attachment: cluster_setup.pdf
                blacklist1.png
                active.png

Attaching documentation pdf incorporating documentation change.

Attaching screenshot listing UI changes.

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Sreekanth Ramakrishnan
>         Attachments: active.png, blacklist1.png, cluster_setup.pdf, hadoop-5478-1.patch, hadoop-5478-2.patch, hadoop-5478-3.patch, hadoop-5478-4.patch, hadoop-5478-5.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717040#action_12717040 ] 

Steve Loughran commented on HADOOP-5478:
----------------------------------------

One thing to consider here is do you really want to run the health check in-VM, or do you want to bring up a second monitoring VM that can do the health checks, keep an eye on the VM running the node, and not be a threat to the stability of the main VM itself. 

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>         Attachments: hadoop-5478-1.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720514#action_12720514 ] 

Hemanth Yamijala commented on HADOOP-5478:
------------------------------------------

bq. The problem is that if there is something wrong that prevents the health checker from communicating to TT, health checker would quit voluntarily without TT's knowledge

That does sound like an issue. Maybe one simple solution is to send a timestamp with the TaskTrackerStatus report about when the health checker was last run. I am of course borrowing the idea from the information we have about when the last heartbeat was received from a TT. We could use that information to find out trackers that haven't updated their health for longer than a certain interval. Would that work ?

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Sreekanth Ramakrishnan
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch, hadoop-5478-3.patch, hadoop-5478-4.patch, hadoop-5478-5.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "eric baldeschwieler (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12709290#action_12709290 ] 

eric baldeschwieler commented on HADOOP-5478:
---------------------------------------------

looks good.

I think we should track some stats per node on the JT.  Just the total success & failures reported over the current and last hour and day long windows.  Showing this and current health and the error line (as allen suggests) on the JT console will let an operator quickly determine if any nodes are ill.

Do we currently track such success/failure ratios for tasks on a node?  That would also be great to display on the same console page.

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721317#action_12721317 ] 

Allen Wittenauer commented on HADOOP-5478:
------------------------------------------

bq. But the fact that we are reporting timestamps of the last health status gives the administrators an opportunity to know that something is amiss on this node, because it's health has not been updated for a while.

Hmm.  What interface do admins have that make this obvious?  If a cluster has 2500 TTs, it isn't going to be obvious in a web UI that any given TT is sick.  

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Sreekanth Ramakrishnan
>         Attachments: active.png, blacklist1.png, cluster_setup.pdf, hadoop-5478-1.patch, hadoop-5478-2.patch, hadoop-5478-3.patch, hadoop-5478-4.patch, hadoop-5478-5.patch, hadoop-5478-6.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720122#action_12720122 ] 

Hemanth Yamijala commented on HADOOP-5478:
------------------------------------------

This is a reasonably complex patch, and in the current state, it has already come a long way. I do have the following comments:

Comments on the spec:

- It may be useful to allow arguments to be provided to the health check script.

Shell:

- It seems like it will be easier to use a Timer to do the interrupting rather than implementing the logic ourselves.
- The timeout thread seems to be being interrupted and joined twice.
- Do we need the setTimeoutInterval ? What happens if it is set after the command has started. I think it is better to have it set only once.

Configuration:

- Default value for mapred.healthChecker.interval can be a little larger, like 60000.
- Let's call mapred.healthChecker.failure_interval mapred.healthChecker.script.timeout. Likewise the variable in NodeHealthChecker failureintervalTime as scriptTimeout or some such name.

NodeHealthCheckerService:

- Instead of setting the health status in the TaskTracker, I think the variable can simply be stored as a local variable by NodeHealthCheckerService, and TT can call it using an appropriately synchronized accessor method.
- If the thread launching the health checker VM is a daemon, I don't think there's a need to join it in close.
- Does NodeHealthCheckerService need to be a public class ?
- Please give a name to the thread launching the health checker VM

NodeHealthChecker:

- Typo: HEALTH_CHECK_INTERVAL_PREOPERTY should be HEALTH_CHECK_INTERVAL_PROPERTY
- If there is an exception in executing the script - should we treat this as a failure ? As a follow-up, should we ignore the output in such a case, as it may not be meaningful to rely upon ?
- A related question is what to do if the output is null. Currently we don't seem to be reporting anything at all. This clearly seems like a bug. I think if it did not timeout, this should be treated as a success.
- Also, should we be more careful in where we check for ERROR. I think the output should begin with the string ERROR, rather than ERROR appearing anywhere in the string.
- I think we should check whether the script timed out before we check the output. It reads more logically that way.
- I would also suggest we move the code in the finally block to a separate method for making it more obvious. Among other things it would help us unit test this very important logic easily.
- I think the check for the health check script should be done in NodeHealthCheckerService. This will avoid us the cost of launching the VM and then failing. However, basic checks for the script can still be there in NodeHealthChecker so it can be used as an independent unit.
- Where do the log messages of the NodeHealthChecker go ? I think it should go to a separate log file.
Give a name to the Timer object created for running the health check script.

JobTracker:

- unBlacklistTracker may not blacklist a tracker if the reason for blacklisting was different. Shouldn't this be checked whenever a call is made, and the subsequent actions be done only if unBlacklistTracker returns a success.
- When tasktracker status is updated because the node becomes healthy, shouldn't isBlackListed be set to true ?

TaskTracker:
- shutdown() seems to have got an extraneous log message by mistake.

Test cases:
- In the testcase, can we also verify the healthReport value for unhealthy trackers ?
- I think we also need a testcase that combines task failures with health script failing and make sure we don't unblacklist a tracker if it was blacklisted due to another reason.

Documentation:
- New classes (like NodeHealthCheckerService) need Javadoc. Indeed, please document all new methods.
- We also need Forrest documentation on this super cool feature. *smile* 


> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch, hadoop-5478-3.patch, hadoop-5478-4.patch, hadoop-5478-5.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708402#action_12708402 ] 

Steve Loughran commented on HADOOP-5478:
----------------------------------------

-this fits in well with the ping/liveness stuff I've been doing. I already do this with things that GET the various JSP pages of a node, checking the health of the jetty endpoints. when those pages return a non-200 code, I return an error that includes the entire HTTP page sent back, as that is often useful. 

# It may be handy to have this stuff independent of the TT itself, so you can run a node-health checker on anything, and even if the TT refuses to play, you could do some checking of the node.
# Also, could it be a bit of JavaScript instead of a shell script?
# A scenario to worry about is what if something bad happens (e.g. a bit of NFS goes away) that causes all health checks in a big cluster to fail simultaneously. Would this overload the JT?
# Incidentally, I could imagine some scripts being slow, so I wouldnt have my ping code run the .sh every call, but instead run it on a regular frequency; a ping() would return the latest results. 


> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720518#action_12720518 ] 

Hong Tang commented on HADOOP-5478:
-----------------------------------

bq. Maybe one simple solution is to send a timestamp with the TaskTrackerStatus report about when the health checker was last run. I am of course borrowing the idea from the information we have about when the last heartbeat was received from a TT. We could use that information to find out trackers that haven't updated their health for longer than a certain interval. Would that work ?

I like it.

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Sreekanth Ramakrishnan
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch, hadoop-5478-3.patch, hadoop-5478-4.patch, hadoop-5478-5.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sreekanth Ramakrishnan updated HADOOP-5478:
-------------------------------------------

    Attachment: hadoop-5478-1.patch

Attaching first cut patch to address the issue: 

The patch does following:

* Patch requires two configuration items to be present in TaskTracker nodes, {{mapred.tasktracker.health_check_script}} and {{mapred.tasktracker.health_check_interval}} the {{mapred.tasktracker.health_check_script}} needs to be absolute path to script file. If the file does not exist when the TT starts up then the monitor is turned off.
* The monitor periodically runs the shell script. It ignores the exit code of the shell script, gets the output from the script, searches for a pattern "ERROR" in the output. 
* If ERROR is present in output, the monitor, sets health of the node as unhealthy and puts entire output as status to be set to JT.
* JT then depending on the value of the health of the node, decides to blacklist or white list the node.
* Attached test case which tests black listing and white listing as per output of the script.

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>         Attachments: hadoop-5478-1.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720001#action_12720001 ] 

Hong Tang commented on HADOOP-5478:
-----------------------------------

Both points make sense to me. One suggestion: when the health check fails to report status to TT, try to do a "kill -9" for the TT in case there might be a bug in the reporting sub-system that prevents health checker reporting states, but the other part of TT is still working fine. 

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch, hadoop-5478-3.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720655#action_12720655 ] 

Steve Loughran commented on HADOOP-5478:
----------------------------------------

# timestamps would help, would push more analysis up to whoever is asking the TT.
# If the main RPC is so owverwhelmed it can't answer a health query, that's a sign of a problem anyway. Your TT is no longer _live_

The health checking would fit in with the notion of a {{Ping}} operation, as raised in HADOOP-5622. Every service should have a way of saying "are you up". The failure to answer the query: trouble. If the call returns with an error: trouble. If the call returns saying it is well, then all you know is that the service thinks it is well, but still may not be capable of useful work.

What this operation does do is set more requirements on what gets returned -you probably want to return something machine readable, that can be extended by different services, depending on their view of the world. A hashtable containing writable stuff, perhaps. 

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Sreekanth Ramakrishnan
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch, hadoop-5478-3.patch, hadoop-5478-4.patch, hadoop-5478-5.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12700332#action_12700332 ] 

Allen Wittenauer commented on HADOOP-5478:
------------------------------------------

We were chatting a bit about this at staff today.  One of the topics that came up was what should happen if the TaskTracker is found to be in a bad state.  The general agreement was that it should be blacklisted with the option of coming back to life.   This helps prevents transient problems from becoming permanent problems.

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718761#action_12718761 ] 

Vinod K V commented on HADOOP-5478:
-----------------------------------

bq. The process would periodically run the script (as done in the patch today) and report the status to the TT using RPC.
Can't the status reporting be simply done via stdout/stderr with START and END markers?

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Richard Theige (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682056#action_12682056 ] 

Richard Theige commented on HADOOP-5478:
----------------------------------------

 For consideration:

   * check_network_speed
    * check_filesystems
          o check for read-only filesystems 
    * check_ylock
          o required, if not installed some jobs which are using ycore libs will fail 
    * check_symlinks
          o checks for hadoop versions 
    * check_diskspace
          o checks if there is sufficient mapred tmp space
          o TT should handle this 


> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708535#action_12708535 ] 

Allen Wittenauer commented on HADOOP-5478:
------------------------------------------



Torque specifically looks for a line that begins with ERROR on stdout, but reports the whole line in the node status.  So running pbsnodes will show the full status message and provides an easy way to audit all nodes on a giving torque server.  We really need the equivalent of dfsadmin -report for the JobTracker to provide this same level of output.

Additionally, torque ignores the exit status. In the vast majority of cases, the node is going to be good.  So the approach they take is that if a script has a syntax error (and would therefore have a 'fail' as an exit code), the node should be considered good anyway.

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sreekanth Ramakrishnan updated HADOOP-5478:
-------------------------------------------

    Attachment: hadoop-5478-4.patch

Attaching patch merged with latest trunk.

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch, hadoop-5478-3.patch, hadoop-5478-4.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715800#action_12715800 ] 

Owen O'Malley commented on HADOOP-5478:
---------------------------------------

One thing that bothers me is that it requires a fork and we've already seen that forks are expensive for   our servers. We'll also need a timer if the fork locks up to count as a failure.

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>         Attachments: hadoop-5478-1.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720491#action_12720491 ] 

Hemanth Yamijala commented on HADOOP-5478:
------------------------------------------

bq. if it fails to receive response from TT, wait for X seconds, do an extra kill (to ensure TT is dead), and quit itself.

Hong, I am not certain about this. I am essentially viewing the TT as the master still, and the health monitor is just a helper service that monitors the health of the node, not the health of the TT itself. It seems wrong that this service could kill the master. I can conceive in future that we extend this to monitor the health of the TT also. And take corrective actions in case something is wrong with the TT. But I think that should be the topic of a different JIRA, or at a minimum an extension to this one. I would still like the scope of this to be restricted to providing a plug-in for checking the health of a node.





> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Sreekanth Ramakrishnan
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch, hadoop-5478-3.patch, hadoop-5478-4.patch, hadoop-5478-5.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720513#action_12720513 ] 

Hong Tang commented on HADOOP-5478:
-----------------------------------

@hemanth The problem is that if there is something wrong that prevents the health checker from communicating to TT, health checker would quit voluntarily without TT's knowledge. So the value of this health checker service could be discounted. But on a second thought, I think maybe we should stop worrying about that for now and defer more complex logic to a later time.


> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Sreekanth Ramakrishnan
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch, hadoop-5478-3.patch, hadoop-5478-4.patch, hadoop-5478-5.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720263#action_12720263 ] 

Allen Wittenauer commented on HADOOP-5478:
------------------------------------------

bq.We are using a new port number for the TT to bind to for the health checker script to send updates. The other option was to use the same port as that used for the TaskUmbilicalProtocol. We thought the health service should not mix with child tasks reporting status and hence kept it different.

This is disappointing.  Hadoop has enough ports open that I think it qualifies as a cheese.

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Sreekanth Ramakrishnan
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch, hadoop-5478-3.patch, hadoop-5478-4.patch, hadoop-5478-5.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721327#action_12721327 ] 

Hemanth Yamijala commented on HADOOP-5478:
------------------------------------------

bq. What interface do admins have that make this obvious? If a cluster has 2500 TTs, it isn't going to be obvious in a web UI that any given TT is sick.

Allen, just to be clear, nodes detected as unhealthy by the health check script are blacklisted as was discussed and as you commented in one of your earliest comments on this JIRA. Therefore the 'mapred job -list-blacklisted-trackers' will easily point out these.

We are only discussing a specific case where the health checker VM exits on a node, say because of some transient problem on the TT, *but* the TT is still up, and possibly unhealthy. It was only for detecting these TTs that I was talking about the timestamps. I agree with you that it will be difficult to look this information up on a web UI. But the fact that we are having this information centrally will help us build / enhance existing tools. 

I believe even with the current scope this feature is worth a try on live clusters. Given we have no way of detecting unhealthy nodes now, this is an improvement. If indeed we find many instances where the health checker is going down while TTs are up, we can easily provide additional tools based on the information currently being reported. Hope that makes sense.

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Sreekanth Ramakrishnan
>         Attachments: active.png, blacklist1.png, cluster_setup.pdf, hadoop-5478-1.patch, hadoop-5478-2.patch, hadoop-5478-3.patch, hadoop-5478-4.patch, hadoop-5478-5.patch, hadoop-5478-6.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717209#action_12717209 ] 

Steve Loughran commented on HADOOP-5478:
----------------------------------------

Cron? No, because you want to be able to force a health check with a ping() over the network, have a URL you can hit for the load balancer. 

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>         Attachments: hadoop-5478-1.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708459#action_12708459 ] 

Steve Loughran commented on HADOOP-5478:
----------------------------------------

some related issues
* HADOOP-3323
* HADOOP-3767 
* HADOOP-3893
* HADOOP-5622

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716682#action_12716682 ] 

Hong Tang commented on HADOOP-5478:
-----------------------------------

- The health checking script may end up running more frequently than desired if the thread gets interrupted.
- The health checking script may also run at a lower frequency as desired because the code did not count the actual time spent on running the script. We should have a variable remember the last launch time, and in the beginning of the loop, get the current time, and either launch the script or sleep for the difference of the check-interval and the elapsed time since the last launch time.
- Like Owen said, we probably need to have a timeout when executing the script (and if it indeed happens, count it as failure).

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>         Attachments: hadoop-5478-1.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720490#action_12720490 ] 

Hemanth Yamijala commented on HADOOP-5478:
------------------------------------------

bq. This is disappointing. Hadoop has enough ports open that I think it qualifies as a cheese.

I was counting on you to object, Allen *smile*.

What do others feel ? I guess we could piggyback on the port used for the TaskUmbilicalProtocol - need Sreekanth to confirm this though. The main concern is if it will interfere with the processing of the tasks reporting their status to TT. To be clear though, the amount of work done in the RPC for the health reporter call is minimal - it just sets two variables.

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Sreekanth Ramakrishnan
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch, hadoop-5478-3.patch, hadoop-5478-4.patch, hadoop-5478-5.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721175#action_12721175 ] 

Hong Tang commented on HADOOP-5478:
-----------------------------------

+1 to hemanth's comments. 

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Sreekanth Ramakrishnan
>         Attachments: active.png, blacklist1.png, cluster_setup.pdf, hadoop-5478-1.patch, hadoop-5478-2.patch, hadoop-5478-3.patch, hadoop-5478-4.patch, hadoop-5478-5.patch, hadoop-5478-6.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12709807#action_12709807 ] 

Devaraj Das commented on HADOOP-5478:
-------------------------------------

TaskTrackers do report the number of failures, and it should be straightforward to maintain and report the number of successes as well.

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717146#action_12717146 ] 

Hong Tang commented on HADOOP-5478:
-----------------------------------

+1. Might it be even easier by just hooking the script with cron?

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>         Attachments: hadoop-5478-1.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sreekanth Ramakrishnan updated HADOOP-5478:
-------------------------------------------

    Attachment: hadoop-5478-3.patch

Attaching a patch which does following:

* Added a new protocol which is used for node health reporting.
* Added a new Class which implements this Interface.
* The new class spawns off a new JVM, which does the health monitoring.
* Moved the timeout logic into {{Shell}}
* Retained the old test case which does
 1. Black listing due to node script reporting error 
2. White-listing of same tracker when script returns no error 
3. Blacklisting when the script times out.

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch, hadoop-5478-3.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719991#action_12719991 ] 

Hemanth Yamijala commented on HADOOP-5478:
------------------------------------------

To add to Sreekanth's comments:

- We are using a new port number for the TT to bind to for the health checker script to send updates. The other option was to use the same port as that used for the TaskUmbilicalProtocol. We thought the health service should not mix with child tasks reporting status and hence kept it different.

- The other important point is about how the health checker stops. Currently, the model is similar to how a child stops, in that if it can't report status to the TT, it kills itself. This is anyway required because it has to handle the case of the TT dying unexpectedly. However this is the extreme case. When the TT is stopped normally there are better options to stop the health check script. For e.g. we could add a shutdown hook to TT and send a signal to the health checker. We could make the health checker a separate daemon as well so that stop-mapred could stop it. Any of these options can be easily implemented as a follow-up once the basic structure is in place.

Please let us know if these points make sense.



> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch, hadoop-5478-3.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682098#action_12682098 ] 

Allen Wittenauer commented on HADOOP-5478:
------------------------------------------

*sigh*

ylock is internal yahoo junk.

The real solution here is that there needs to be a script call out to provide user defined functionality.  Specific checks should not come out of the box by default.

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708879#action_12708879 ] 

Hemanth Yamijala commented on HADOOP-5478:
------------------------------------------

bq. this fits in well with the ping/liveness stuff I've been doing

This coupled with your comment on ping() returning the latest results seems to indicate that we have a thread that periodically executes and stores the results. In that sense, maybe we could build this solution now, and when HADOOP-3628 is committed to trunk, we could integrate the solution and results to be returned as part of ping(). Does that make sense ?

bq. It may be handy to have this stuff independent of the TT itself, so you can run a node-health checker on anything, and even if the TT refuses to play, you could do some checking of the node.

The health check script itself is definitely external and could be anything. All the TT would provide is the ability to run it periodically. So, I can imagine this being run standalone, or integrated with another daemon that provides a similar interface.

bq. Also, could it be a bit of JavaScript instead of a shell script?

Umm. Can we execute this from the TT directly ? AFAIK, this is not possible, right ? As of now, there is no plan to support anything other than a shell script.

bq. A scenario to worry about is what if something bad happens (e.g. a bit of NFS goes away) that causes all health checks in a big cluster to fail simultaneously. Would this overload the JT?

Since the plan is to send the information using the heartbeats itself, handling the load of requests should not be a problem. I am not sure how costly blacklist processing itself is on the JT, but hopefully not bad. We'll keep this in mind though.


> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720495#action_12720495 ] 

Owen O'Malley commented on HADOOP-5478:
---------------------------------------

I'd vote for using the same RPC port. I don't think it buys us enough to be worth the hassle of dealing with the extra port.

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Sreekanth Ramakrishnan
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch, hadoop-5478-3.patch, hadoop-5478-4.patch, hadoop-5478-5.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sreekanth Ramakrishnan updated HADOOP-5478:
-------------------------------------------

    Attachment: hadoop-5478-6.patch

Attaching latest patch incorporating Hemanths comments:

Major Changes:

* Now {{NodeHealthChecker}} reports health status on same port which tasks use to report status.
* {{NodeHealthChecker}} can now be run in seperate VM or as thread, thread based start up can be used in {{MiniMRCluster}}
* Changed testcase to also test conditions with blacklisting across jobs and also verifying cluster capacity after we blacklist tracker.
* Also added new configuration entry which takes node health scripts arguments.


> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Sreekanth Ramakrishnan
>         Attachments: active.png, blacklist1.png, cluster_setup.pdf, hadoop-5478-1.patch, hadoop-5478-2.patch, hadoop-5478-3.patch, hadoop-5478-4.patch, hadoop-5478-5.patch, hadoop-5478-6.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Issue Comment Edited: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718752#action_12718752 ] 

Hong Tang edited comment on HADOOP-5478 at 6/12/09 12:49 AM:
-------------------------------------------------------------

@owen: Steve's comments seem to imply that we need the capability of launch the health checking script in response to external commands.

@Sreekanth: If I understand correctly, the new patch introduces another thread to check for command execution timeout. Is there a way to eliminate it (possibly by modifying shell execution API to accept timeouts)? 

      was (Author: hong.tang):
    @owen: Steve's comments seem to imply that we need the capability of launch the health checking script in response to external commands.

@Streekanth: If I understand correctly, the new patch introduces another thread to check for command execution timeout. Is there a way to eliminate it (possibly by modifying shell execution API to accept timeouts)? 
  
> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720185#action_12720185 ] 

Hong Tang commented on HADOOP-5478:
-----------------------------------

bq Hong, is this to check if the TT is alive ? In which case, did you mean another signal, like -0 or kill -3. -9 is SIGKILL and would kill the TT. Also, in that case are you suggesting that we could keep the health checker around and continue trying to report after a while ?

@hemanth sorry for not being clear. I gave a bit more thoughts on the problem, and I think the following logic may be simpler and more robust (1,2 are the current logic, 3 is my suggestion) : 

(1) periodically launch the health checking script; 
(2) reporting status that back to TT (both good and bad); 
(3) if it fails to receive response from TT, wait for X seconds, do an extra kill (to ensure TT is dead), and quit itself. 

I scanned through the code, it seems that NodeHealthChecker.stop() would be a good place to perform step (3).

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Sreekanth Ramakrishnan
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch, hadoop-5478-3.patch, hadoop-5478-4.patch, hadoop-5478-5.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708943#action_12708943 ] 

Steve Loughran commented on HADOOP-5478:
----------------------------------------


Hemanth,

I could certainly merge this in with HADOOP-3628; I'm busy dealing with svn merge issues right now, and don't want this held up. I think it would be handy for me if we could run this on startup and then have ping query the latest state when checked.

>>     Also, could it be a bit of JavaScript instead of a shell script?

>Umm. Can we execute this from the TT directly ? AFAIK, this is not possible, right ? As of now, there is no plan to support anything other than a shell script.

you can run JS direct from a java6 jvm; the script engine is in the box. I've been wondering what it would take for JS support in MR jobs, but it could be handy for system health checks too, though native scripts give you the edge for low level system health. 




> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720689#action_12720689 ] 

Hong Tang commented on HADOOP-5478:
-----------------------------------

bq. The health checking would fit in with the notion of a Ping operation, as raised in HADOOP-5622. Every service should have a way of saying "are you up". The failure to answer the query: trouble. If the call returns with an error: trouble. If the call returns saying it is well, then all you know is that the service thinks it is well, but still may not be capable of useful work.

Though, at both times, the only one who knows about the trouble is the health checker and not the rest of the world. 

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Sreekanth Ramakrishnan
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch, hadoop-5478-3.patch, hadoop-5478-4.patch, hadoop-5478-5.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719484#action_12719484 ] 

Sreekanth Ramakrishnan commented on HADOOP-5478:
------------------------------------------------

Adding a little more to discussion, following is approach which I am taking to generate a new patch:

* Introduce a new health monitor service which is spawned off by task tracker when it starts.
* The service periodically reports the status of the node to the task tracker.
* The protocol is modeled out of {{TaskUmbricalProtocol}}
* The service would receive the host address and port as the command line arguments while starting up.
* The service then periodically sends the status update to task tracker based on the host and port specified to the service. 
* When TaskTracker is shutdown, the {{NodeHealthChecker}} would not be able to contact {{TaskTracker}} and would shut itself down.  The reason why this is done, is because task tracker's {{shutdown()}} or {{close()}} is not called when we do a {{stop-mapred.sh}} or task tracker can be killed with direct {{kill -9 ttpid}} in this case the TT might not inform all the clients which contact it to report services.


> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718729#action_12718729 ] 

Owen O'Malley commented on HADOOP-5478:
---------------------------------------

I'd propose that we fork a small process off that runs the script for us every N minutes and reports status to the TT. That way we minimize the danger of causing the TT to fail. Furthermore, since it can have a tiny heap and few threads, the fork will be much cheaper. Thoughts?

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718819#action_12718819 ] 

Steve Loughran commented on HADOOP-5478:
----------------------------------------

I do a fair amount of health monitoring , and there is a lot to be said for something that runs is a separate process from any of the Hadoop services to do the checking.
# it could be its own service, fairly lightweight
# gives you the option of monitoring (and if need be killing) the TT process itself.
# stops you accidentally stamping on bits of the JVM. As an example, I'd deployed something that checked the health of bits of HDFS by checking that files where there, but that code closed the handle after use, killing any TT in the same process. 

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sreekanth Ramakrishnan updated HADOOP-5478:
-------------------------------------------

    Attachment: hadoop-5478-2.patch

Attaching a patch which address all other concern except forking a shell every delay interval.

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hemanth Yamijala updated HADOOP-5478:
-------------------------------------

    Assignee: Sreekanth Ramakrishnan  (was: Vinod K V)

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Sreekanth Ramakrishnan
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch, hadoop-5478-3.patch, hadoop-5478-4.patch, hadoop-5478-5.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Assigned: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hemanth Yamijala reassigned HADOOP-5478:
----------------------------------------

    Assignee: Vinod K V

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Vinod K V
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5478) Provide a node health check script and run it periodically to check the node health status

Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721042#action_12721042 ] 

Hemanth Yamijala commented on HADOOP-5478:
------------------------------------------

Folks, I still maintain that the focus of this jira is just checking health of the node as determined by an administrator supplied script. The last few comments are focusing more on health of a TT. For the purpose of making incremental progress, let us stick to the original scope and defer discussions of checking the health on the TT, and corrective actions there-of, to a separate jira.

So, to summarize, the health checker kills itself if it cannot communicate with the TT (similar to the child JVM). If this happens because the TT is down, well and good. The 'lost tasktracker' logic of the jobtracker would ensure this status is captured. If this happens because the TT was overwhelmed, well, maybe the TT is not 'healthy' any more. But the fact that we are reporting timestamps of the last health status gives the administrators an opportunity to know that something is amiss on this node, because it's health has not been updated for a while. Either way we can alert ourselves to problems. So, the purpose is still solved. Of course, there are better, more automated ways to do it. That would qualify for a next increment.

Hope this makes sense.

> Provide a node health check script and run it periodically to check the node health status
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5478
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Aroop Maliakkal
>            Assignee: Sreekanth Ramakrishnan
>         Attachments: hadoop-5478-1.patch, hadoop-5478-2.patch, hadoop-5478-3.patch, hadoop-5478-4.patch, hadoop-5478-5.patch
>
>
> Hadoop must have some mechanism to find the health status of a node . It should run the health check script periodically and if there is any errors, it should black list the node. This will be really helpful when we run static mapred clusters. Else we may have to run some scripts/daemons periodically to find the node status and take it offline manually.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.