[jira] [Created] (TRAFODION-1593) use 'shell -c node info' to up a node may sometimes hang if kill monitor processes for some times

["Joshua Liu (JIRA)" <ji...@apache.org>]

Joshua  Liu created TRAFODION-1593:
--------------------------------------

             Summary: use 'shell -c node info' to up a node may sometimes hang if kill monitor processes for some times
                 Key: TRAFODION-1593
                 URL: https://issues.apache.org/jira/browse/TRAFODION-1593
             Project: Apache Trafodion
          Issue Type: Bug
          Components: foundation
    Affects Versions: 2.0-incubating
            Reporter: Joshua  Liu
            Priority: Minor


 Today and yesterday we faced one issue on nap007~012 that:
         If we killed monitor on some node, then used ‘shell –c up node’ to up this node. Sometimes it failed to up and just hung there.

         The log file is like below:
         Cat mon.snmp.20151028.08.58.14.nap007.esgyn.local.1701.log
2015-10-28 08:58:15,037, INFO, MON, Node Number: 0,, PIN: 1701 , Process Name: $MONITOR,,, TID: 1701, Message ID: 101010401, [CCluster::CCluster] Validation of node down is enabled
2015-10-28 08:58:15,038, ERROR, MON, Node Number: 0,, PIN: 1701 , Process Name: $MONITOR,,, TID: 1701, Message ID: 101010402, [CCluster::CCluster], Error= No monitor.ini found, setting to defaults.

Cat node_up_nap007.esgyn.local.log
…
In perform_work: nap007.esgyn.local
remove shared memory ids on host nap007.esgyn.local
Unable to get the monitor port number
ssh -q nap007.esgyn.local ipcrm -m 65538
ssh -q nap007.esgyn.local ipcrm -m 327690
…

By the way, yesterday we faced the issue after killing monitor or tm for 20 times. Today 40 times.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)