You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@manifoldcf.apache.org by lalit jangra <la...@gmail.com> on 2014/09/14 18:18:43 UTC
MCF cluster agents do not shut down.
Hi,
I am using MCF 1.7 in clustered mode with 6 zookeeper nodes and two agents
running on separate machines. Once i start the cluster (running agent1 on
node1 and agent 2 on node 2), everything is fine but after a long time say
a day or two, if i try to kill agents or restart them, agents do not get
killed and using ./stop-agents.sh shows right logs but do not kill agents.
Please suggest.
Regards,
Lalit.
Re: MCF cluster agents do not shut down.
Posted by Karl Wright <da...@gmail.com>.
Hi Lalit,
I asked for thread dumps, not logs. To get a thread dump, use your jdk's
"jstack" command on the agents process that won't shut down.
Thanks,
Karl
On Tue, Sep 16, 2014 at 4:51 AM, lalit jangra <la...@gmail.com>
wrote:
> Hi Karl,
>
> Below are the logs i get when i try to stop agents. Please suggest.
>
>
> [root@iwdc1preecma03 ecmadmin]# cd
> /app/IW/MCF1.7/dist/multiprocess-zk-example
>
> [root@iwdc1preecma03 multiprocess-zk-example]# ./stop-agents.sh
>
> Configuration file successfully read
>
> [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
>
> [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=
> iwdc1preecma03.iwater.ie
>
> [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.version=1.7.0_25
>
> [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.vendor=Oracle Corporation
>
> [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.home=/app/IW/java/jre
>
> [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.class.path=.:../lib/mcf-pull-agent.jar:../lib/mcf-agents.jar:../lib/mcf-core.jar:../lib/hsqldb.jar:../lib/derbyLocale_zh_TW.jar:../lib/derbyLocale_zh_CN.jar:../lib/derbyLocale_ru.jar:../lib/derbyLocale_pt_BR.jar:../lib/derbyLocale_pl.jar:../lib/derbyLocale_ko_KR.jar:../lib/derbyLocale_ja_JP.jar:../lib/derbyLocale_it.jar:../lib/derbyLocale_hu.jar:../lib/derbyLocale_fr.jar:../lib/derbyLocale_es.jar:../lib/derbyLocale_de_DE.jar:../lib/derbyLocale_cs.jar:../lib/derbytools.jar:../lib/derbynet.jar:../lib/derby.jar:../lib/postgresql.jar:../lib/mail.jar:../lib/slf4j-simple.jar:../lib/slf4j-api.jar:../lib/velocity.jar:../lib/xml-apis.jar:../lib/xercesImpl.jar:../lib/xalan.jar:../lib/servlet-api.jar:../lib/serializer.jar:../lib/log4j.jar:../lib/commons-logging.jar:../lib/commons-lang.jar:../lib/commons-io.jar:../lib/httpclient.jar:../lib/httpcore.jar:../lib/commons-fileupload.jar:../lib/commons-el.jar:../lib/commons-collections.jar:../lib/commons-codec.jar:../lib/json-simple.jar:../lib/json.jar:../lib/zookeeper.jar:../lib/activation.jar:../lib/saaj-impl.jar:../lib/saaj-api.jar:../lib/wsdl4j.jar:../lib/wss4j.jar:../lib/axis.jar:../lib/axis-jaxrpc.jar:../lib/commons-discovery.jar:../lib/geronimo-javamail_1.4_spec.jar:../lib/castor.jar:
>
> [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>
> [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.io.tmpdir=/tmp
>
> [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.compiler=<NA>
>
> [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name
> =Linux
>
> [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:os.arch=amd64
>
> [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:os.version=2.6.32-358.el6.x86_64
>
> [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name
> =root
>
> [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:user.home=/root
>
> [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:user.dir=/app/IW/MCF1.7/dist/multiprocess-zk-example
>
> [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection,
> connectString=iwdc1preecma03:2181,iwdc1preecma03:2182,iwdc1preecma03:2183,iwdc2preecma04:2181,iwdc2preecma04:2182,iwdc2preecma04:2183
> sessionTimeout=4000
> watcher=org.apache.manifoldcf.core.lockmanager.ZooKeeperConnection$ZooKeeperWatcher@78618ac5
>
> [main-SendThread(iwdc1preecma03.iwater.ie:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> iwdc1preecma03.iwater.ie/10.231.72.24:2181. Will not attempt to
> authenticate using SASL (unknown error)
>
> [main-SendThread(iwdc1preecma03.iwater.ie:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Socket connection established to
> iwdc1preecma03.iwater.ie/10.231.72.24:2181, initiating session
>
> [main-SendThread(iwdc1preecma03.iwater.ie:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Unable to read additional data from
> server sessionid 0x0, likely server has closed socket, closing socket
> connection and attempting reconnect
>
> [main-SendThread(iwdc2preecma04.iwater.ie:2182)] INFO
> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> iwdc2preecma04.iwater.ie/10.231.72.25:2182. Will not attempt to
> authenticate using SASL (unknown error)
>
> [main-SendThread(iwdc2preecma04.iwater.ie:2182)] INFO
> org.apache.zookeeper.ClientCnxn - Socket connection established to
> iwdc2preecma04.iwater.ie/10.231.72.25:2182, initiating session
>
> [main-SendThread(iwdc2preecma04.iwater.ie:2182)] INFO
> org.apache.zookeeper.ClientCnxn - Unable to read additional data from
> server sessionid 0x0, likely server has closed socket, closing socket
> connection and attempting reconnect
>
> [main-SendThread(iwdc1preecma03.iwater.ie:2183)] INFO
> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> iwdc1preecma03.iwater.ie/10.231.72.24:2183. Will not attempt to
> authenticate using SASL (unknown error)
>
> [main-SendThread(iwdc1preecma03.iwater.ie:2183)] INFO
> org.apache.zookeeper.ClientCnxn - Socket connection established to
> iwdc1preecma03.iwater.ie/10.231.72.24:2183, initiating session
>
> [main-SendThread(iwdc1preecma03.iwater.ie:2183)] INFO
> org.apache.zookeeper.ClientCnxn - Unable to read additional data from
> server sessionid 0x0, likely server has closed socket, closing socket
> connection and attempting reconnect
>
> [main-SendThread(iwdc2preecma04.iwater.ie:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> iwdc2preecma04.iwater.ie/10.231.72.25:2181. Will not attempt to
> authenticate using SASL (unknown error)
>
> [main-SendThread(iwdc2preecma04.iwater.ie:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Socket connection established to
> iwdc2preecma04.iwater.ie/10.231.72.25:2181, initiating session
>
> [main-SendThread(iwdc2preecma04.iwater.ie:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Unable to read additional data from
> server sessionid 0x0, likely server has closed socket, closing socket
> connection and attempting reconnect
>
> [main-SendThread(iwdc1preecma03.iwater.ie:2182)] INFO
> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> iwdc1preecma03.iwater.ie/10.231.72.24:2182. Will not attempt to
> authenticate using SASL (unknown error)
>
> [main-SendThread(iwdc1preecma03.iwater.ie:2182)] INFO
> org.apache.zookeeper.ClientCnxn - Socket connection established to
> iwdc1preecma03.iwater.ie/10.231.72.24:2182, initiating session
>
> [main-SendThread(iwdc1preecma03.iwater.ie:2182)] INFO
> org.apache.zookeeper.ClientCnxn - Session establishment complete on server
> iwdc1preecma03.iwater.ie/10.231.72.24:2182, sessionid =
> 0x2487d06d7540078, negotiated timeout = 4000
>
> Shutdown signal sent
>
> [Shutdown thread] INFO org.apache.zookeeper.ZooKeeper - Session:
> 0x2487d06d7540078 closed
>
> [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut
> down
>
>
> On Mon, Sep 15, 2014 at 1:46 AM, Karl Wright <da...@gmail.com> wrote:
>
>> Hi Lalit,
>>
>> Please obtain thread dumps from the agents processese that will not stop.
>>
>> Thanks,
>> Karl
>>
>>
>> On Sun, Sep 14, 2014 at 12:18 PM, lalit jangra <la...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I am using MCF 1.7 in clustered mode with 6 zookeeper nodes and two
>>> agents running on separate machines. Once i start the cluster (running
>>> agent1 on node1 and agent 2 on node 2), everything is fine but after a long
>>> time say a day or two, if i try to kill agents or restart them, agents do
>>> not get killed and using ./stop-agents.sh shows right logs but do not kill
>>> agents.
>>>
>>> Please suggest.
>>>
>>> Regards,
>>> Lalit.
>>>
>>
>>
>
>
> --
> Regards,
> Lalit.
>
Re: MCF cluster agents do not shut down.
Posted by lalit jangra <la...@gmail.com>.
Hi Karl,
Below are the logs i get when i try to stop agents. Please suggest.
[root@iwdc1preecma03 ecmadmin]# cd
/app/IW/MCF1.7/dist/multiprocess-zk-example
[root@iwdc1preecma03 multiprocess-zk-example]# ./stop-agents.sh
Configuration file successfully read
[main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=
iwdc1preecma03.iwater.ie
[main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.version=1.7.0_25
[main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.vendor=Oracle Corporation
[main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.home=/app/IW/java/jre
[main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.class.path=.:../lib/mcf-pull-agent.jar:../lib/mcf-agents.jar:../lib/mcf-core.jar:../lib/hsqldb.jar:../lib/derbyLocale_zh_TW.jar:../lib/derbyLocale_zh_CN.jar:../lib/derbyLocale_ru.jar:../lib/derbyLocale_pt_BR.jar:../lib/derbyLocale_pl.jar:../lib/derbyLocale_ko_KR.jar:../lib/derbyLocale_ja_JP.jar:../lib/derbyLocale_it.jar:../lib/derbyLocale_hu.jar:../lib/derbyLocale_fr.jar:../lib/derbyLocale_es.jar:../lib/derbyLocale_de_DE.jar:../lib/derbyLocale_cs.jar:../lib/derbytools.jar:../lib/derbynet.jar:../lib/derby.jar:../lib/postgresql.jar:../lib/mail.jar:../lib/slf4j-simple.jar:../lib/slf4j-api.jar:../lib/velocity.jar:../lib/xml-apis.jar:../lib/xercesImpl.jar:../lib/xalan.jar:../lib/servlet-api.jar:../lib/serializer.jar:../lib/log4j.jar:../lib/commons-logging.jar:../lib/commons-lang.jar:../lib/commons-io.jar:../lib/httpclient.jar:../lib/httpcore.jar:../lib/commons-fileupload.jar:../lib/commons-el.jar:../lib/commons-collections.jar:../lib/commons-codec.jar:../lib/json-simple.jar:../lib/json.jar:../lib/zookeeper.jar:../lib/activation.jar:../lib/saaj-impl.jar:../lib/saaj-api.jar:../lib/wsdl4j.jar:../lib/wss4j.jar:../lib/axis.jar:../lib/axis-jaxrpc.jar:../lib/commons-discovery.jar:../lib/geronimo-javamail_1.4_spec.jar:../lib/castor.jar:
[main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
[main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.io.tmpdir=/tmp
[main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.compiler=<NA>
[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name
=Linux
[main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:os.arch=amd64
[main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:os.version=2.6.32-358.el6.x86_64
[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name
=root
[main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:user.home=/root
[main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:user.dir=/app/IW/MCF1.7/dist/multiprocess-zk-example
[main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection,
connectString=iwdc1preecma03:2181,iwdc1preecma03:2182,iwdc1preecma03:2183,iwdc2preecma04:2181,iwdc2preecma04:2182,iwdc2preecma04:2183
sessionTimeout=4000
watcher=org.apache.manifoldcf.core.lockmanager.ZooKeeperConnection$ZooKeeperWatcher@78618ac5
[main-SendThread(iwdc1preecma03.iwater.ie:2181)] INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
iwdc1preecma03.iwater.ie/10.231.72.24:2181. Will not attempt to
authenticate using SASL (unknown error)
[main-SendThread(iwdc1preecma03.iwater.ie:2181)] INFO
org.apache.zookeeper.ClientCnxn - Socket connection established to
iwdc1preecma03.iwater.ie/10.231.72.24:2181, initiating session
[main-SendThread(iwdc1preecma03.iwater.ie:2181)] INFO
org.apache.zookeeper.ClientCnxn - Unable to read additional data from
server sessionid 0x0, likely server has closed socket, closing socket
connection and attempting reconnect
[main-SendThread(iwdc2preecma04.iwater.ie:2182)] INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
iwdc2preecma04.iwater.ie/10.231.72.25:2182. Will not attempt to
authenticate using SASL (unknown error)
[main-SendThread(iwdc2preecma04.iwater.ie:2182)] INFO
org.apache.zookeeper.ClientCnxn - Socket connection established to
iwdc2preecma04.iwater.ie/10.231.72.25:2182, initiating session
[main-SendThread(iwdc2preecma04.iwater.ie:2182)] INFO
org.apache.zookeeper.ClientCnxn - Unable to read additional data from
server sessionid 0x0, likely server has closed socket, closing socket
connection and attempting reconnect
[main-SendThread(iwdc1preecma03.iwater.ie:2183)] INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
iwdc1preecma03.iwater.ie/10.231.72.24:2183. Will not attempt to
authenticate using SASL (unknown error)
[main-SendThread(iwdc1preecma03.iwater.ie:2183)] INFO
org.apache.zookeeper.ClientCnxn - Socket connection established to
iwdc1preecma03.iwater.ie/10.231.72.24:2183, initiating session
[main-SendThread(iwdc1preecma03.iwater.ie:2183)] INFO
org.apache.zookeeper.ClientCnxn - Unable to read additional data from
server sessionid 0x0, likely server has closed socket, closing socket
connection and attempting reconnect
[main-SendThread(iwdc2preecma04.iwater.ie:2181)] INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
iwdc2preecma04.iwater.ie/10.231.72.25:2181. Will not attempt to
authenticate using SASL (unknown error)
[main-SendThread(iwdc2preecma04.iwater.ie:2181)] INFO
org.apache.zookeeper.ClientCnxn - Socket connection established to
iwdc2preecma04.iwater.ie/10.231.72.25:2181, initiating session
[main-SendThread(iwdc2preecma04.iwater.ie:2181)] INFO
org.apache.zookeeper.ClientCnxn - Unable to read additional data from
server sessionid 0x0, likely server has closed socket, closing socket
connection and attempting reconnect
[main-SendThread(iwdc1preecma03.iwater.ie:2182)] INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
iwdc1preecma03.iwater.ie/10.231.72.24:2182. Will not attempt to
authenticate using SASL (unknown error)
[main-SendThread(iwdc1preecma03.iwater.ie:2182)] INFO
org.apache.zookeeper.ClientCnxn - Socket connection established to
iwdc1preecma03.iwater.ie/10.231.72.24:2182, initiating session
[main-SendThread(iwdc1preecma03.iwater.ie:2182)] INFO
org.apache.zookeeper.ClientCnxn - Session establishment complete on server
iwdc1preecma03.iwater.ie/10.231.72.24:2182, sessionid = 0x2487d06d7540078,
negotiated timeout = 4000
Shutdown signal sent
[Shutdown thread] INFO org.apache.zookeeper.ZooKeeper - Session:
0x2487d06d7540078 closed
[main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut
down
On Mon, Sep 15, 2014 at 1:46 AM, Karl Wright <da...@gmail.com> wrote:
> Hi Lalit,
>
> Please obtain thread dumps from the agents processese that will not stop.
>
> Thanks,
> Karl
>
>
> On Sun, Sep 14, 2014 at 12:18 PM, lalit jangra <la...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I am using MCF 1.7 in clustered mode with 6 zookeeper nodes and two
>> agents running on separate machines. Once i start the cluster (running
>> agent1 on node1 and agent 2 on node 2), everything is fine but after a long
>> time say a day or two, if i try to kill agents or restart them, agents do
>> not get killed and using ./stop-agents.sh shows right logs but do not kill
>> agents.
>>
>> Please suggest.
>>
>> Regards,
>> Lalit.
>>
>
>
--
Regards,
Lalit.
Re: MCF cluster agents do not shut down.
Posted by Karl Wright <da...@gmail.com>.
Hi Lalit,
Please obtain thread dumps from the agents processese that will not stop.
Thanks,
Karl
On Sun, Sep 14, 2014 at 12:18 PM, lalit jangra <la...@gmail.com>
wrote:
> Hi,
>
> I am using MCF 1.7 in clustered mode with 6 zookeeper nodes and two agents
> running on separate machines. Once i start the cluster (running agent1 on
> node1 and agent 2 on node 2), everything is fine but after a long time say
> a day or two, if i try to kill agents or restart them, agents do not get
> killed and using ./stop-agents.sh shows right logs but do not kill agents.
>
> Please suggest.
>
> Regards,
> Lalit.
>