You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by "Lukasz Osipiuk (JIRA)" <ji...@apache.org> on 2010/03/18 12:40:32 UTC
[jira] Created: (ZOOKEEPER-710) permanent ZSESSIONMOVED error after
client app reconnects to zookeeper cluster
permanent ZSESSIONMOVED error after client app reconnects to zookeeper cluster
------------------------------------------------------------------------------
Key: ZOOKEEPER-710
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-710
Project: Zookeeper
Issue Type: Bug
Environment: debian lenny; ia64; xen virtualization
Reporter: Lukasz Osipiuk
Originally problem was described on Users mailing list starting with this [post|http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201003.mbox/<3b...@mail.gmail.com>].
Below I restate it in more organized form.
We occasionally (few times a day) observe that our client application disconnects from Zookeeper cluster.
Application is written in C++ and we are using libzookeeper_mt library. In version 3.2.2.
The disconnects we are observing are probably related to some problems with our network infrastructure - we are observing periods with great packet loss between machines in our DC.
Sometimes after client application (i.e. zookeeper library) reconnects to zookeeper cluster we are observing that all subsequent requests return ZSESSIONMOVED error. Restarting client app helps - we always pass 0 as clientid to zookeeper_init function so old session is not reused.
On 16-03-2010 we observed few occurences of problem. Example ones:
- 22:08; client IP 10.1.112.60 (app1)
- 14:21; client IP 10.1.112.61 (app2)
I attach logs of cluster and application nodes (only stuff concerining zookeeper):
- zookeeper-node1.log.2010-03-16.gz
- zookeeper-node2.log.2010-03-16.gz
- zookeeper-node3.log.2010-03-16.gz
- app1.log.2010-03-16.gz
- app2.log.2010-03-16.gz
I also made some analysis of case at 22:08:
- Network glitch which resulted in problem occurred at about 22:08.
- From what I see since 17:48 node2 was the leader and it did not
change later yesterday.
- Client was connected to node2 since 17:50
- At around 22:09 client tried to connect to every node (1,2,3).
Connections to node1 and node3 were closed
with exception "Exception causing close of session 0x22767e1c9630000
due to java.io.IOException: Read error".
Connection to node2 stood alive.
- All subsequent operations were refused with ZSESSIONMOVED error.
Error visible both on client and on server side.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-710) permanent ZSESSIONMOVED error after
client app reconnects to zookeeper cluster
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt updated ZOOKEEPER-710:
-----------------------------------
Attachment: ZOOKEEPER-710_3.2.patch
this 3.2 patch is against the current 3.2 svn branch
fixes the issue in my tests
> permanent ZSESSIONMOVED error after client app reconnects to zookeeper cluster
> ------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-710
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-710
> Project: Zookeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.2.2
> Environment: debian lenny; ia64; xen virtualization
> Reporter: Lukasz Osipiuk
> Assignee: Patrick Hunt
> Priority: Blocker
> Fix For: 3.2.3, 3.3.0
>
> Attachments: app1.log.2010-03-16.gz, app2.log.2010-03-16.gz, ZOOKEEPER-710_3.2.patch, ZOOKEEPER-710_3.3.patch, zookeeper-node1.log.2010-03-16.gz, zookeeper-node2.log.2010-03-16.gz, zookeeper-node3.log.2010-03-16.gz
>
>
> Originally problem was described on Users mailing list starting with this [post|http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201003.mbox/<3b...@mail.gmail.com>].
> Below I restate it in more organized form.
> We occasionally (few times a day) observe that our client application disconnects from Zookeeper cluster.
> Application is written in C++ and we are using libzookeeper_mt library. In version 3.2.2.
> The disconnects we are observing are probably related to some problems with our network infrastructure - we are observing periods with great packet loss between machines in our DC.
> Sometimes after client application (i.e. zookeeper library) reconnects to zookeeper cluster we are observing that all subsequent requests return ZSESSIONMOVED error. Restarting client app helps - we always pass 0 as clientid to zookeeper_init function so old session is not reused.
> On 16-03-2010 we observed few occurences of problem. Example ones:
> - 22:08; client IP 10.1.112.60 (app1); sessionID 0x22767e1c9630000
> - 14:21; client IP 10.1.112.61 (app2); sessionID 0x324dcc1ba580085
> I attach logs of cluster and application nodes (only stuff concerining zookeeper):
> - [^zookeeper-node1.log.2010-03-16.gz] - logs of zookeepr cluster node 1 10.1.112.62
> - [^zookeeper-node2.log.2010-03-16.gz] - logs of zookeepr cluster node 2 10.1.112.63
> - [^zookeeper-node3.log.2010-03-16.gz] - logs of zookeepr cluster node 3 10.1.112.64
> - [^app1.log.2010-03-16.gz] - application logs of app1 10.1.112.60
> - [^app2.log.2010-03-16.gz] - application logs of app2 10.1.112.61
> I also made some analysis of case at 22:08:
> - Network glitch which resulted in problem occurred at about 22:08.
> - From what I see since 17:48 node2 was the leader and it did not
> change later yesterday.
> - Client was connected to node2 since 17:50
> - At around 22:09 client tried to connect to every node (1,2,3).
> Connections to node1 and node3 were closed
> with exception "Exception causing close of session 0x22767e1c9630000
> due to java.io.IOException: Read error".
> Connection to node2 stood alive.
> - All subsequent operations were refused with ZSESSIONMOVED error.
> Error visible both on client and on server side.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-710) permanent ZSESSIONMOVED error after
client app reconnects to zookeeper cluster
Posted by "Lukasz Osipiuk (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lukasz Osipiuk updated ZOOKEEPER-710:
-------------------------------------
Description:
Originally problem was described on Users mailing list starting with this [post|http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201003.mbox/<3b...@mail.gmail.com>].
Below I restate it in more organized form.
We occasionally (few times a day) observe that our client application disconnects from Zookeeper cluster.
Application is written in C++ and we are using libzookeeper_mt library. In version 3.2.2.
The disconnects we are observing are probably related to some problems with our network infrastructure - we are observing periods with great packet loss between machines in our DC.
Sometimes after client application (i.e. zookeeper library) reconnects to zookeeper cluster we are observing that all subsequent requests return ZSESSIONMOVED error. Restarting client app helps - we always pass 0 as clientid to zookeeper_init function so old session is not reused.
On 16-03-2010 we observed few occurences of problem. Example ones:
- 22:08; client IP 10.1.112.60 (app1); sessionID 0x22767e1c9630000
- 14:21; client IP 10.1.112.61 (app2); sessionID 0x324dcc1ba580085
I attach logs of cluster and application nodes (only stuff concerining zookeeper):
- [^zookeeper-node1.log.2010-03-16.gz] - logs of zookeepr cluster node 1 10.1.112.62
- [^zookeeper-node2.log.2010-03-16.gz] - logs of zookeepr cluster node 2 10.1.112.63
- [^zookeeper-node3.log.2010-03-16.gz] - logs of zookeepr cluster node 3 10.1.112.64
- [^app1.log.2010-03-16.gz] - application logs of app1 10.1.112.60
- [^app2.log.2010-03-16.gz] - application logs of app2 10.1.112.61
I also made some analysis of case at 22:08:
- Network glitch which resulted in problem occurred at about 22:08.
- From what I see since 17:48 node2 was the leader and it did not
change later yesterday.
- Client was connected to node2 since 17:50
- At around 22:09 client tried to connect to every node (1,2,3).
Connections to node1 and node3 were closed
with exception "Exception causing close of session 0x22767e1c9630000
due to java.io.IOException: Read error".
Connection to node2 stood alive.
- All subsequent operations were refused with ZSESSIONMOVED error.
Error visible both on client and on server side.
was:
Originally problem was described on Users mailing list starting with this [post|http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201003.mbox/<3b...@mail.gmail.com>].
Below I restate it in more organized form.
We occasionally (few times a day) observe that our client application disconnects from Zookeeper cluster.
Application is written in C++ and we are using libzookeeper_mt library. In version 3.2.2.
The disconnects we are observing are probably related to some problems with our network infrastructure - we are observing periods with great packet loss between machines in our DC.
Sometimes after client application (i.e. zookeeper library) reconnects to zookeeper cluster we are observing that all subsequent requests return ZSESSIONMOVED error. Restarting client app helps - we always pass 0 as clientid to zookeeper_init function so old session is not reused.
On 16-03-2010 we observed few occurences of problem. Example ones:
- 22:08; client IP 10.1.112.60 (app1); sessionID 0x22767e1c9630000
- 14:21; client IP 10.1.112.61 (app2); sessionID 0x324dcc1ba580085
I attach logs of cluster and application nodes (only stuff concerining zookeeper):
- zookeeper-node1.log.2010-03-16.gz
- zookeeper-node2.log.2010-03-16.gz
- zookeeper-node3.log.2010-03-16.gz
- app1.log.2010-03-16.gz
- app2.log.2010-03-16.gz
I also made some analysis of case at 22:08:
- Network glitch which resulted in problem occurred at about 22:08.
- From what I see since 17:48 node2 was the leader and it did not
change later yesterday.
- Client was connected to node2 since 17:50
- At around 22:09 client tried to connect to every node (1,2,3).
Connections to node1 and node3 were closed
with exception "Exception causing close of session 0x22767e1c9630000
due to java.io.IOException: Read error".
Connection to node2 stood alive.
- All subsequent operations were refused with ZSESSIONMOVED error.
Error visible both on client and on server side.
> permanent ZSESSIONMOVED error after client app reconnects to zookeeper cluster
> ------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-710
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-710
> Project: Zookeeper
> Issue Type: Bug
> Affects Versions: 3.2.2
> Environment: debian lenny; ia64; xen virtualization
> Reporter: Lukasz Osipiuk
> Attachments: app1.log.2010-03-16.gz, app2.log.2010-03-16.gz, zookeeper-node1.log.2010-03-16.gz, zookeeper-node2.log.2010-03-16.gz, zookeeper-node3.log.2010-03-16.gz
>
>
> Originally problem was described on Users mailing list starting with this [post|http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201003.mbox/<3b...@mail.gmail.com>].
> Below I restate it in more organized form.
> We occasionally (few times a day) observe that our client application disconnects from Zookeeper cluster.
> Application is written in C++ and we are using libzookeeper_mt library. In version 3.2.2.
> The disconnects we are observing are probably related to some problems with our network infrastructure - we are observing periods with great packet loss between machines in our DC.
> Sometimes after client application (i.e. zookeeper library) reconnects to zookeeper cluster we are observing that all subsequent requests return ZSESSIONMOVED error. Restarting client app helps - we always pass 0 as clientid to zookeeper_init function so old session is not reused.
> On 16-03-2010 we observed few occurences of problem. Example ones:
> - 22:08; client IP 10.1.112.60 (app1); sessionID 0x22767e1c9630000
> - 14:21; client IP 10.1.112.61 (app2); sessionID 0x324dcc1ba580085
> I attach logs of cluster and application nodes (only stuff concerining zookeeper):
> - [^zookeeper-node1.log.2010-03-16.gz] - logs of zookeepr cluster node 1 10.1.112.62
> - [^zookeeper-node2.log.2010-03-16.gz] - logs of zookeepr cluster node 2 10.1.112.63
> - [^zookeeper-node3.log.2010-03-16.gz] - logs of zookeepr cluster node 3 10.1.112.64
> - [^app1.log.2010-03-16.gz] - application logs of app1 10.1.112.60
> - [^app2.log.2010-03-16.gz] - application logs of app2 10.1.112.61
> I also made some analysis of case at 22:08:
> - Network glitch which resulted in problem occurred at about 22:08.
> - From what I see since 17:48 node2 was the leader and it did not
> change later yesterday.
> - Client was connected to node2 since 17:50
> - At around 22:09 client tried to connect to every node (1,2,3).
> Connections to node1 and node3 were closed
> with exception "Exception causing close of session 0x22767e1c9630000
> due to java.io.IOException: Read error".
> Connection to node2 stood alive.
> - All subsequent operations were refused with ZSESSIONMOVED error.
> Error visible both on client and on server side.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-710) permanent ZSESSIONMOVED error after
client app reconnects to zookeeper cluster
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt updated ZOOKEEPER-710:
-----------------------------------
Affects Version/s: (was: 3.3.0)
> permanent ZSESSIONMOVED error after client app reconnects to zookeeper cluster
> ------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-710
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-710
> Project: Zookeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.2.2
> Environment: debian lenny; ia64; xen virtualization
> Reporter: Lukasz Osipiuk
> Assignee: Patrick Hunt
> Priority: Blocker
> Fix For: 3.2.3, 3.3.0
>
> Attachments: app1.log.2010-03-16.gz, app2.log.2010-03-16.gz, zookeeper-node1.log.2010-03-16.gz, zookeeper-node2.log.2010-03-16.gz, zookeeper-node3.log.2010-03-16.gz
>
>
> Originally problem was described on Users mailing list starting with this [post|http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201003.mbox/<3b...@mail.gmail.com>].
> Below I restate it in more organized form.
> We occasionally (few times a day) observe that our client application disconnects from Zookeeper cluster.
> Application is written in C++ and we are using libzookeeper_mt library. In version 3.2.2.
> The disconnects we are observing are probably related to some problems with our network infrastructure - we are observing periods with great packet loss between machines in our DC.
> Sometimes after client application (i.e. zookeeper library) reconnects to zookeeper cluster we are observing that all subsequent requests return ZSESSIONMOVED error. Restarting client app helps - we always pass 0 as clientid to zookeeper_init function so old session is not reused.
> On 16-03-2010 we observed few occurences of problem. Example ones:
> - 22:08; client IP 10.1.112.60 (app1); sessionID 0x22767e1c9630000
> - 14:21; client IP 10.1.112.61 (app2); sessionID 0x324dcc1ba580085
> I attach logs of cluster and application nodes (only stuff concerining zookeeper):
> - [^zookeeper-node1.log.2010-03-16.gz] - logs of zookeepr cluster node 1 10.1.112.62
> - [^zookeeper-node2.log.2010-03-16.gz] - logs of zookeepr cluster node 2 10.1.112.63
> - [^zookeeper-node3.log.2010-03-16.gz] - logs of zookeepr cluster node 3 10.1.112.64
> - [^app1.log.2010-03-16.gz] - application logs of app1 10.1.112.60
> - [^app2.log.2010-03-16.gz] - application logs of app2 10.1.112.61
> I also made some analysis of case at 22:08:
> - Network glitch which resulted in problem occurred at about 22:08.
> - From what I see since 17:48 node2 was the leader and it did not
> change later yesterday.
> - Client was connected to node2 since 17:50
> - At around 22:09 client tried to connect to every node (1,2,3).
> Connections to node1 and node3 were closed
> with exception "Exception causing close of session 0x22767e1c9630000
> due to java.io.IOException: Read error".
> Connection to node2 stood alive.
> - All subsequent operations were refused with ZSESSIONMOVED error.
> Error visible both on client and on server side.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-710) permanent ZSESSIONMOVED error
after client app reconnects to zookeeper cluster
Posted by "Lukasz Osipiuk (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12847275#action_12847275 ]
Lukasz Osipiuk commented on ZOOKEEPER-710:
------------------------------------------
I will build version with patch and deploy it on Monday. But as I will also tweak client timeout parameter and GC; and bug itself is not manually reproducable, it wont be best check if fix is working :)
> permanent ZSESSIONMOVED error after client app reconnects to zookeeper cluster
> ------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-710
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-710
> Project: Zookeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.2.2
> Environment: debian lenny; ia64; xen virtualization
> Reporter: Lukasz Osipiuk
> Assignee: Patrick Hunt
> Priority: Blocker
> Fix For: 3.2.3, 3.3.0
>
> Attachments: app1.log.2010-03-16.gz, app2.log.2010-03-16.gz, ZOOKEEPER-710_3.2.patch, ZOOKEEPER-710_3.3.patch, zookeeper-node1.log.2010-03-16.gz, zookeeper-node2.log.2010-03-16.gz, zookeeper-node3.log.2010-03-16.gz
>
>
> Originally problem was described on Users mailing list starting with this [post|http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201003.mbox/<3b...@mail.gmail.com>].
> Below I restate it in more organized form.
> We occasionally (few times a day) observe that our client application disconnects from Zookeeper cluster.
> Application is written in C++ and we are using libzookeeper_mt library. In version 3.2.2.
> The disconnects we are observing are probably related to some problems with our network infrastructure - we are observing periods with great packet loss between machines in our DC.
> Sometimes after client application (i.e. zookeeper library) reconnects to zookeeper cluster we are observing that all subsequent requests return ZSESSIONMOVED error. Restarting client app helps - we always pass 0 as clientid to zookeeper_init function so old session is not reused.
> On 16-03-2010 we observed few occurences of problem. Example ones:
> - 22:08; client IP 10.1.112.60 (app1); sessionID 0x22767e1c9630000
> - 14:21; client IP 10.1.112.61 (app2); sessionID 0x324dcc1ba580085
> I attach logs of cluster and application nodes (only stuff concerining zookeeper):
> - [^zookeeper-node1.log.2010-03-16.gz] - logs of zookeepr cluster node 1 10.1.112.62
> - [^zookeeper-node2.log.2010-03-16.gz] - logs of zookeepr cluster node 2 10.1.112.63
> - [^zookeeper-node3.log.2010-03-16.gz] - logs of zookeepr cluster node 3 10.1.112.64
> - [^app1.log.2010-03-16.gz] - application logs of app1 10.1.112.60
> - [^app2.log.2010-03-16.gz] - application logs of app2 10.1.112.61
> I also made some analysis of case at 22:08:
> - Network glitch which resulted in problem occurred at about 22:08.
> - From what I see since 17:48 node2 was the leader and it did not
> change later yesterday.
> - Client was connected to node2 since 17:50
> - At around 22:09 client tried to connect to every node (1,2,3).
> Connections to node1 and node3 were closed
> with exception "Exception causing close of session 0x22767e1c9630000
> due to java.io.IOException: Read error".
> Connection to node2 stood alive.
> - All subsequent operations were refused with ZSESSIONMOVED error.
> Error visible both on client and on server side.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-710) permanent ZSESSIONMOVED error
after client app reconnects to zookeeper cluster
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12847017#action_12847017 ]
Mahadev konar commented on ZOOKEEPER-710:
-----------------------------------------
lukasz, can you elaborate on what this code is doing?
{code}
Mar 16 14:21:49 XXXXXXXXXX-c1-mbox-2 XXXXXXXXXX_mbox[25112]: DEBUG: ZooKeeper: ZooKeeper::execWatch() executing watch
Mar 16 14:21:49 XXXXXXXXXX-c1-mbox-2 XXXXXXXXXX_mbox[25112]: INFO: ZooKeeper: ZooKeeper::watcherFunHandleSessionEvent() state change to 3
Mar 16 14:21:49 XXXXXXXXXX-c1-mbox-2 XXXXXXXXXX_mbox[25112]: INFO: ZooKeeper: ZooKeeper::fireAllWatchFunctions() calling all watches
{code}
who all watches are your firing?
> permanent ZSESSIONMOVED error after client app reconnects to zookeeper cluster
> ------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-710
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-710
> Project: Zookeeper
> Issue Type: Bug
> Affects Versions: 3.2.2
> Environment: debian lenny; ia64; xen virtualization
> Reporter: Lukasz Osipiuk
> Attachments: app1.log.2010-03-16.gz, app2.log.2010-03-16.gz, zookeeper-node1.log.2010-03-16.gz, zookeeper-node2.log.2010-03-16.gz, zookeeper-node3.log.2010-03-16.gz
>
>
> Originally problem was described on Users mailing list starting with this [post|http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201003.mbox/<3b...@mail.gmail.com>].
> Below I restate it in more organized form.
> We occasionally (few times a day) observe that our client application disconnects from Zookeeper cluster.
> Application is written in C++ and we are using libzookeeper_mt library. In version 3.2.2.
> The disconnects we are observing are probably related to some problems with our network infrastructure - we are observing periods with great packet loss between machines in our DC.
> Sometimes after client application (i.e. zookeeper library) reconnects to zookeeper cluster we are observing that all subsequent requests return ZSESSIONMOVED error. Restarting client app helps - we always pass 0 as clientid to zookeeper_init function so old session is not reused.
> On 16-03-2010 we observed few occurences of problem. Example ones:
> - 22:08; client IP 10.1.112.60 (app1); sessionID 0x22767e1c9630000
> - 14:21; client IP 10.1.112.61 (app2); sessionID 0x324dcc1ba580085
> I attach logs of cluster and application nodes (only stuff concerining zookeeper):
> - [^zookeeper-node1.log.2010-03-16.gz] - logs of zookeepr cluster node 1 10.1.112.62
> - [^zookeeper-node2.log.2010-03-16.gz] - logs of zookeepr cluster node 2 10.1.112.63
> - [^zookeeper-node3.log.2010-03-16.gz] - logs of zookeepr cluster node 3 10.1.112.64
> - [^app1.log.2010-03-16.gz] - application logs of app1 10.1.112.60
> - [^app2.log.2010-03-16.gz] - application logs of app2 10.1.112.61
> I also made some analysis of case at 22:08:
> - Network glitch which resulted in problem occurred at about 22:08.
> - From what I see since 17:48 node2 was the leader and it did not
> change later yesterday.
> - Client was connected to node2 since 17:50
> - At around 22:09 client tried to connect to every node (1,2,3).
> Connections to node1 and node3 were closed
> with exception "Exception causing close of session 0x22767e1c9630000
> due to java.io.IOException: Read error".
> Connection to node2 stood alive.
> - All subsequent operations were refused with ZSESSIONMOVED error.
> Error visible both on client and on server side.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-710) permanent ZSESSIONMOVED error after
client app reconnects to zookeeper cluster
Posted by "Lukasz Osipiuk (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lukasz Osipiuk updated ZOOKEEPER-710:
-------------------------------------
Description:
Originally problem was described on Users mailing list starting with this [post|http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201003.mbox/<3b...@mail.gmail.com>].
Below I restate it in more organized form.
We occasionally (few times a day) observe that our client application disconnects from Zookeeper cluster.
Application is written in C++ and we are using libzookeeper_mt library. In version 3.2.2.
The disconnects we are observing are probably related to some problems with our network infrastructure - we are observing periods with great packet loss between machines in our DC.
Sometimes after client application (i.e. zookeeper library) reconnects to zookeeper cluster we are observing that all subsequent requests return ZSESSIONMOVED error. Restarting client app helps - we always pass 0 as clientid to zookeeper_init function so old session is not reused.
On 16-03-2010 we observed few occurences of problem. Example ones:
- 22:08; client IP 10.1.112.60 (app1); sessionID 0x22767e1c9630000
- 14:21; client IP 10.1.112.61 (app2); sessionID 0x324dcc1ba580085
I attach logs of cluster and application nodes (only stuff concerining zookeeper):
- zookeeper-node1.log.2010-03-16.gz
- zookeeper-node2.log.2010-03-16.gz
- zookeeper-node3.log.2010-03-16.gz
- app1.log.2010-03-16.gz
- app2.log.2010-03-16.gz
I also made some analysis of case at 22:08:
- Network glitch which resulted in problem occurred at about 22:08.
- From what I see since 17:48 node2 was the leader and it did not
change later yesterday.
- Client was connected to node2 since 17:50
- At around 22:09 client tried to connect to every node (1,2,3).
Connections to node1 and node3 were closed
with exception "Exception causing close of session 0x22767e1c9630000
due to java.io.IOException: Read error".
Connection to node2 stood alive.
- All subsequent operations were refused with ZSESSIONMOVED error.
Error visible both on client and on server side.
was:
Originally problem was described on Users mailing list starting with this [post|http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201003.mbox/<3b...@mail.gmail.com>].
Below I restate it in more organized form.
We occasionally (few times a day) observe that our client application disconnects from Zookeeper cluster.
Application is written in C++ and we are using libzookeeper_mt library. In version 3.2.2.
The disconnects we are observing are probably related to some problems with our network infrastructure - we are observing periods with great packet loss between machines in our DC.
Sometimes after client application (i.e. zookeeper library) reconnects to zookeeper cluster we are observing that all subsequent requests return ZSESSIONMOVED error. Restarting client app helps - we always pass 0 as clientid to zookeeper_init function so old session is not reused.
On 16-03-2010 we observed few occurences of problem. Example ones:
- 22:08; client IP 10.1.112.60 (app1)
- 14:21; client IP 10.1.112.61 (app2)
I attach logs of cluster and application nodes (only stuff concerining zookeeper):
- zookeeper-node1.log.2010-03-16.gz
- zookeeper-node2.log.2010-03-16.gz
- zookeeper-node3.log.2010-03-16.gz
- app1.log.2010-03-16.gz
- app2.log.2010-03-16.gz
I also made some analysis of case at 22:08:
- Network glitch which resulted in problem occurred at about 22:08.
- From what I see since 17:48 node2 was the leader and it did not
change later yesterday.
- Client was connected to node2 since 17:50
- At around 22:09 client tried to connect to every node (1,2,3).
Connections to node1 and node3 were closed
with exception "Exception causing close of session 0x22767e1c9630000
due to java.io.IOException: Read error".
Connection to node2 stood alive.
- All subsequent operations were refused with ZSESSIONMOVED error.
Error visible both on client and on server side.
Affects Version/s: 3.2.2
> permanent ZSESSIONMOVED error after client app reconnects to zookeeper cluster
> ------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-710
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-710
> Project: Zookeeper
> Issue Type: Bug
> Affects Versions: 3.2.2
> Environment: debian lenny; ia64; xen virtualization
> Reporter: Lukasz Osipiuk
> Attachments: app1.log.2010-03-16.gz, app2.log.2010-03-16.gz, zookeeper-node1.log.2010-03-16.gz, zookeeper-node2.log.2010-03-16.gz, zookeeper-node3.log.2010-03-16.gz
>
>
> Originally problem was described on Users mailing list starting with this [post|http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201003.mbox/<3b...@mail.gmail.com>].
> Below I restate it in more organized form.
> We occasionally (few times a day) observe that our client application disconnects from Zookeeper cluster.
> Application is written in C++ and we are using libzookeeper_mt library. In version 3.2.2.
> The disconnects we are observing are probably related to some problems with our network infrastructure - we are observing periods with great packet loss between machines in our DC.
> Sometimes after client application (i.e. zookeeper library) reconnects to zookeeper cluster we are observing that all subsequent requests return ZSESSIONMOVED error. Restarting client app helps - we always pass 0 as clientid to zookeeper_init function so old session is not reused.
> On 16-03-2010 we observed few occurences of problem. Example ones:
> - 22:08; client IP 10.1.112.60 (app1); sessionID 0x22767e1c9630000
> - 14:21; client IP 10.1.112.61 (app2); sessionID 0x324dcc1ba580085
> I attach logs of cluster and application nodes (only stuff concerining zookeeper):
> - zookeeper-node1.log.2010-03-16.gz
> - zookeeper-node2.log.2010-03-16.gz
> - zookeeper-node3.log.2010-03-16.gz
> - app1.log.2010-03-16.gz
> - app2.log.2010-03-16.gz
> I also made some analysis of case at 22:08:
> - Network glitch which resulted in problem occurred at about 22:08.
> - From what I see since 17:48 node2 was the leader and it did not
> change later yesterday.
> - Client was connected to node2 since 17:50
> - At around 22:09 client tried to connect to every node (1,2,3).
> Connections to node1 and node3 were closed
> with exception "Exception causing close of session 0x22767e1c9630000
> due to java.io.IOException: Read error".
> Connection to node2 stood alive.
> - All subsequent operations were refused with ZSESSIONMOVED error.
> Error visible both on client and on server side.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (ZOOKEEPER-710) permanent ZSESSIONMOVED error
after client app reconnects to zookeeper cluster
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar resolved ZOOKEEPER-710.
-------------------------------------
Resolution: Fixed
I just committed this to 3.2 branch. thanks pat!
> permanent ZSESSIONMOVED error after client app reconnects to zookeeper cluster
> ------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-710
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-710
> Project: Zookeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.2.2
> Environment: debian lenny; ia64; xen virtualization
> Reporter: Lukasz Osipiuk
> Assignee: Patrick Hunt
> Priority: Blocker
> Fix For: 3.2.3, 3.3.0
>
> Attachments: app1.log.2010-03-16.gz, app2.log.2010-03-16.gz, ZOOKEEPER-710_3.2.patch, ZOOKEEPER-710_3.3.patch, zookeeper-node1.log.2010-03-16.gz, zookeeper-node2.log.2010-03-16.gz, zookeeper-node3.log.2010-03-16.gz
>
>
> Originally problem was described on Users mailing list starting with this [post|http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201003.mbox/<3b...@mail.gmail.com>].
> Below I restate it in more organized form.
> We occasionally (few times a day) observe that our client application disconnects from Zookeeper cluster.
> Application is written in C++ and we are using libzookeeper_mt library. In version 3.2.2.
> The disconnects we are observing are probably related to some problems with our network infrastructure - we are observing periods with great packet loss between machines in our DC.
> Sometimes after client application (i.e. zookeeper library) reconnects to zookeeper cluster we are observing that all subsequent requests return ZSESSIONMOVED error. Restarting client app helps - we always pass 0 as clientid to zookeeper_init function so old session is not reused.
> On 16-03-2010 we observed few occurences of problem. Example ones:
> - 22:08; client IP 10.1.112.60 (app1); sessionID 0x22767e1c9630000
> - 14:21; client IP 10.1.112.61 (app2); sessionID 0x324dcc1ba580085
> I attach logs of cluster and application nodes (only stuff concerining zookeeper):
> - [^zookeeper-node1.log.2010-03-16.gz] - logs of zookeepr cluster node 1 10.1.112.62
> - [^zookeeper-node2.log.2010-03-16.gz] - logs of zookeepr cluster node 2 10.1.112.63
> - [^zookeeper-node3.log.2010-03-16.gz] - logs of zookeepr cluster node 3 10.1.112.64
> - [^app1.log.2010-03-16.gz] - application logs of app1 10.1.112.60
> - [^app2.log.2010-03-16.gz] - application logs of app2 10.1.112.61
> I also made some analysis of case at 22:08:
> - Network glitch which resulted in problem occurred at about 22:08.
> - From what I see since 17:48 node2 was the leader and it did not
> change later yesterday.
> - Client was connected to node2 since 17:50
> - At around 22:09 client tried to connect to every node (1,2,3).
> Connections to node1 and node3 were closed
> with exception "Exception causing close of session 0x22767e1c9630000
> due to java.io.IOException: Read error".
> Connection to node2 stood alive.
> - All subsequent operations were refused with ZSESSIONMOVED error.
> Error visible both on client and on server side.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-710) permanent ZSESSIONMOVED error after
client app reconnects to zookeeper cluster
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt updated ZOOKEEPER-710:
-----------------------------------
Component/s: server
Priority: Blocker (was: Major)
Affects Version/s: 3.3.0
Fix Version/s: 3.3.0
3.2.3
Assignee: Patrick Hunt
Marking this to fix for 3.3.0 and 3.2.3
> permanent ZSESSIONMOVED error after client app reconnects to zookeeper cluster
> ------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-710
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-710
> Project: Zookeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.2.2
> Environment: debian lenny; ia64; xen virtualization
> Reporter: Lukasz Osipiuk
> Assignee: Patrick Hunt
> Priority: Blocker
> Fix For: 3.2.3, 3.3.0
>
> Attachments: app1.log.2010-03-16.gz, app2.log.2010-03-16.gz, zookeeper-node1.log.2010-03-16.gz, zookeeper-node2.log.2010-03-16.gz, zookeeper-node3.log.2010-03-16.gz
>
>
> Originally problem was described on Users mailing list starting with this [post|http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201003.mbox/<3b...@mail.gmail.com>].
> Below I restate it in more organized form.
> We occasionally (few times a day) observe that our client application disconnects from Zookeeper cluster.
> Application is written in C++ and we are using libzookeeper_mt library. In version 3.2.2.
> The disconnects we are observing are probably related to some problems with our network infrastructure - we are observing periods with great packet loss between machines in our DC.
> Sometimes after client application (i.e. zookeeper library) reconnects to zookeeper cluster we are observing that all subsequent requests return ZSESSIONMOVED error. Restarting client app helps - we always pass 0 as clientid to zookeeper_init function so old session is not reused.
> On 16-03-2010 we observed few occurences of problem. Example ones:
> - 22:08; client IP 10.1.112.60 (app1); sessionID 0x22767e1c9630000
> - 14:21; client IP 10.1.112.61 (app2); sessionID 0x324dcc1ba580085
> I attach logs of cluster and application nodes (only stuff concerining zookeeper):
> - [^zookeeper-node1.log.2010-03-16.gz] - logs of zookeepr cluster node 1 10.1.112.62
> - [^zookeeper-node2.log.2010-03-16.gz] - logs of zookeepr cluster node 2 10.1.112.63
> - [^zookeeper-node3.log.2010-03-16.gz] - logs of zookeepr cluster node 3 10.1.112.64
> - [^app1.log.2010-03-16.gz] - application logs of app1 10.1.112.60
> - [^app2.log.2010-03-16.gz] - application logs of app2 10.1.112.61
> I also made some analysis of case at 22:08:
> - Network glitch which resulted in problem occurred at about 22:08.
> - From what I see since 17:48 node2 was the leader and it did not
> change later yesterday.
> - Client was connected to node2 since 17:50
> - At around 22:09 client tried to connect to every node (1,2,3).
> Connections to node1 and node3 were closed
> with exception "Exception causing close of session 0x22767e1c9630000
> due to java.io.IOException: Read error".
> Connection to node2 stood alive.
> - All subsequent operations were refused with ZSESSIONMOVED error.
> Error visible both on client and on server side.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-710) permanent ZSESSIONMOVED error after
client app reconnects to zookeeper cluster
Posted by "Lukasz Osipiuk (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lukasz Osipiuk updated ZOOKEEPER-710:
-------------------------------------
Attachment: zookeeper-node1.log.2010-03-16.gz
> permanent ZSESSIONMOVED error after client app reconnects to zookeeper cluster
> ------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-710
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-710
> Project: Zookeeper
> Issue Type: Bug
> Environment: debian lenny; ia64; xen virtualization
> Reporter: Lukasz Osipiuk
> Attachments: app1.log.2010-03-16.gz, zookeeper-node1.log.2010-03-16.gz, zookeeper-node2.log.2010-03-16.gz, zookeeper-node3.log.2010-03-16.gz
>
>
> Originally problem was described on Users mailing list starting with this [post|http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201003.mbox/<3b...@mail.gmail.com>].
> Below I restate it in more organized form.
> We occasionally (few times a day) observe that our client application disconnects from Zookeeper cluster.
> Application is written in C++ and we are using libzookeeper_mt library. In version 3.2.2.
> The disconnects we are observing are probably related to some problems with our network infrastructure - we are observing periods with great packet loss between machines in our DC.
> Sometimes after client application (i.e. zookeeper library) reconnects to zookeeper cluster we are observing that all subsequent requests return ZSESSIONMOVED error. Restarting client app helps - we always pass 0 as clientid to zookeeper_init function so old session is not reused.
> On 16-03-2010 we observed few occurences of problem. Example ones:
> - 22:08; client IP 10.1.112.60 (app1)
> - 14:21; client IP 10.1.112.61 (app2)
> I attach logs of cluster and application nodes (only stuff concerining zookeeper):
> - zookeeper-node1.log.2010-03-16.gz
> - zookeeper-node2.log.2010-03-16.gz
> - zookeeper-node3.log.2010-03-16.gz
> - app1.log.2010-03-16.gz
> - app2.log.2010-03-16.gz
> I also made some analysis of case at 22:08:
> - Network glitch which resulted in problem occurred at about 22:08.
> - From what I see since 17:48 node2 was the leader and it did not
> change later yesterday.
> - Client was connected to node2 since 17:50
> - At around 22:09 client tried to connect to every node (1,2,3).
> Connections to node1 and node3 were closed
> with exception "Exception causing close of session 0x22767e1c9630000
> due to java.io.IOException: Read error".
> Connection to node2 stood alive.
> - All subsequent operations were refused with ZSESSIONMOVED error.
> Error visible both on client and on server side.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-710) permanent ZSESSIONMOVED error after
client app reconnects to zookeeper cluster
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt updated ZOOKEEPER-710:
-----------------------------------
Attachment: ZOOKEEPER-710_3.3.patch
fixes this session moved issue by closing the invalid connection
> permanent ZSESSIONMOVED error after client app reconnects to zookeeper cluster
> ------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-710
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-710
> Project: Zookeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.2.2
> Environment: debian lenny; ia64; xen virtualization
> Reporter: Lukasz Osipiuk
> Assignee: Patrick Hunt
> Priority: Blocker
> Fix For: 3.2.3, 3.3.0
>
> Attachments: app1.log.2010-03-16.gz, app2.log.2010-03-16.gz, ZOOKEEPER-710_3.3.patch, zookeeper-node1.log.2010-03-16.gz, zookeeper-node2.log.2010-03-16.gz, zookeeper-node3.log.2010-03-16.gz
>
>
> Originally problem was described on Users mailing list starting with this [post|http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201003.mbox/<3b...@mail.gmail.com>].
> Below I restate it in more organized form.
> We occasionally (few times a day) observe that our client application disconnects from Zookeeper cluster.
> Application is written in C++ and we are using libzookeeper_mt library. In version 3.2.2.
> The disconnects we are observing are probably related to some problems with our network infrastructure - we are observing periods with great packet loss between machines in our DC.
> Sometimes after client application (i.e. zookeeper library) reconnects to zookeeper cluster we are observing that all subsequent requests return ZSESSIONMOVED error. Restarting client app helps - we always pass 0 as clientid to zookeeper_init function so old session is not reused.
> On 16-03-2010 we observed few occurences of problem. Example ones:
> - 22:08; client IP 10.1.112.60 (app1); sessionID 0x22767e1c9630000
> - 14:21; client IP 10.1.112.61 (app2); sessionID 0x324dcc1ba580085
> I attach logs of cluster and application nodes (only stuff concerining zookeeper):
> - [^zookeeper-node1.log.2010-03-16.gz] - logs of zookeepr cluster node 1 10.1.112.62
> - [^zookeeper-node2.log.2010-03-16.gz] - logs of zookeepr cluster node 2 10.1.112.63
> - [^zookeeper-node3.log.2010-03-16.gz] - logs of zookeepr cluster node 3 10.1.112.64
> - [^app1.log.2010-03-16.gz] - application logs of app1 10.1.112.60
> - [^app2.log.2010-03-16.gz] - application logs of app2 10.1.112.61
> I also made some analysis of case at 22:08:
> - Network glitch which resulted in problem occurred at about 22:08.
> - From what I see since 17:48 node2 was the leader and it did not
> change later yesterday.
> - Client was connected to node2 since 17:50
> - At around 22:09 client tried to connect to every node (1,2,3).
> Connections to node1 and node3 were closed
> with exception "Exception causing close of session 0x22767e1c9630000
> due to java.io.IOException: Read error".
> Connection to node2 stood alive.
> - All subsequent operations were refused with ZSESSIONMOVED error.
> Error visible both on client and on server side.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-710) permanent ZSESSIONMOVED error after
client app reconnects to zookeeper cluster
Posted by "Lukasz Osipiuk (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lukasz Osipiuk updated ZOOKEEPER-710:
-------------------------------------
Attachment: zookeeper-node3.log.2010-03-16.gz
zookeeper-node2.log.2010-03-16.gz
> permanent ZSESSIONMOVED error after client app reconnects to zookeeper cluster
> ------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-710
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-710
> Project: Zookeeper
> Issue Type: Bug
> Environment: debian lenny; ia64; xen virtualization
> Reporter: Lukasz Osipiuk
> Attachments: app1.log.2010-03-16.gz, zookeeper-node1.log.2010-03-16.gz, zookeeper-node2.log.2010-03-16.gz, zookeeper-node3.log.2010-03-16.gz
>
>
> Originally problem was described on Users mailing list starting with this [post|http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201003.mbox/<3b...@mail.gmail.com>].
> Below I restate it in more organized form.
> We occasionally (few times a day) observe that our client application disconnects from Zookeeper cluster.
> Application is written in C++ and we are using libzookeeper_mt library. In version 3.2.2.
> The disconnects we are observing are probably related to some problems with our network infrastructure - we are observing periods with great packet loss between machines in our DC.
> Sometimes after client application (i.e. zookeeper library) reconnects to zookeeper cluster we are observing that all subsequent requests return ZSESSIONMOVED error. Restarting client app helps - we always pass 0 as clientid to zookeeper_init function so old session is not reused.
> On 16-03-2010 we observed few occurences of problem. Example ones:
> - 22:08; client IP 10.1.112.60 (app1)
> - 14:21; client IP 10.1.112.61 (app2)
> I attach logs of cluster and application nodes (only stuff concerining zookeeper):
> - zookeeper-node1.log.2010-03-16.gz
> - zookeeper-node2.log.2010-03-16.gz
> - zookeeper-node3.log.2010-03-16.gz
> - app1.log.2010-03-16.gz
> - app2.log.2010-03-16.gz
> I also made some analysis of case at 22:08:
> - Network glitch which resulted in problem occurred at about 22:08.
> - From what I see since 17:48 node2 was the leader and it did not
> change later yesterday.
> - Client was connected to node2 since 17:50
> - At around 22:09 client tried to connect to every node (1,2,3).
> Connections to node1 and node3 were closed
> with exception "Exception causing close of session 0x22767e1c9630000
> due to java.io.IOException: Read error".
> Connection to node2 stood alive.
> - All subsequent operations were refused with ZSESSIONMOVED error.
> Error visible both on client and on server side.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-710) permanent ZSESSIONMOVED error after
client app reconnects to zookeeper cluster
Posted by "Lukasz Osipiuk (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lukasz Osipiuk updated ZOOKEEPER-710:
-------------------------------------
Attachment: app2.log.2010-03-16.gz
> permanent ZSESSIONMOVED error after client app reconnects to zookeeper cluster
> ------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-710
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-710
> Project: Zookeeper
> Issue Type: Bug
> Environment: debian lenny; ia64; xen virtualization
> Reporter: Lukasz Osipiuk
> Attachments: app1.log.2010-03-16.gz, app2.log.2010-03-16.gz, zookeeper-node1.log.2010-03-16.gz, zookeeper-node2.log.2010-03-16.gz, zookeeper-node3.log.2010-03-16.gz
>
>
> Originally problem was described on Users mailing list starting with this [post|http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201003.mbox/<3b...@mail.gmail.com>].
> Below I restate it in more organized form.
> We occasionally (few times a day) observe that our client application disconnects from Zookeeper cluster.
> Application is written in C++ and we are using libzookeeper_mt library. In version 3.2.2.
> The disconnects we are observing are probably related to some problems with our network infrastructure - we are observing periods with great packet loss between machines in our DC.
> Sometimes after client application (i.e. zookeeper library) reconnects to zookeeper cluster we are observing that all subsequent requests return ZSESSIONMOVED error. Restarting client app helps - we always pass 0 as clientid to zookeeper_init function so old session is not reused.
> On 16-03-2010 we observed few occurences of problem. Example ones:
> - 22:08; client IP 10.1.112.60 (app1)
> - 14:21; client IP 10.1.112.61 (app2)
> I attach logs of cluster and application nodes (only stuff concerining zookeeper):
> - zookeeper-node1.log.2010-03-16.gz
> - zookeeper-node2.log.2010-03-16.gz
> - zookeeper-node3.log.2010-03-16.gz
> - app1.log.2010-03-16.gz
> - app2.log.2010-03-16.gz
> I also made some analysis of case at 22:08:
> - Network glitch which resulted in problem occurred at about 22:08.
> - From what I see since 17:48 node2 was the leader and it did not
> change later yesterday.
> - Client was connected to node2 since 17:50
> - At around 22:09 client tried to connect to every node (1,2,3).
> Connections to node1 and node3 were closed
> with exception "Exception causing close of session 0x22767e1c9630000
> due to java.io.IOException: Read error".
> Connection to node2 stood alive.
> - All subsequent operations were refused with ZSESSIONMOVED error.
> Error visible both on client and on server side.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-710) permanent ZSESSIONMOVED error after
client app reconnects to zookeeper cluster
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt updated ZOOKEEPER-710:
-----------------------------------
Status: Patch Available (was: Open)
> permanent ZSESSIONMOVED error after client app reconnects to zookeeper cluster
> ------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-710
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-710
> Project: Zookeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.2.2
> Environment: debian lenny; ia64; xen virtualization
> Reporter: Lukasz Osipiuk
> Assignee: Patrick Hunt
> Priority: Blocker
> Fix For: 3.2.3, 3.3.0
>
> Attachments: app1.log.2010-03-16.gz, app2.log.2010-03-16.gz, ZOOKEEPER-710_3.3.patch, zookeeper-node1.log.2010-03-16.gz, zookeeper-node2.log.2010-03-16.gz, zookeeper-node3.log.2010-03-16.gz
>
>
> Originally problem was described on Users mailing list starting with this [post|http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201003.mbox/<3b...@mail.gmail.com>].
> Below I restate it in more organized form.
> We occasionally (few times a day) observe that our client application disconnects from Zookeeper cluster.
> Application is written in C++ and we are using libzookeeper_mt library. In version 3.2.2.
> The disconnects we are observing are probably related to some problems with our network infrastructure - we are observing periods with great packet loss between machines in our DC.
> Sometimes after client application (i.e. zookeeper library) reconnects to zookeeper cluster we are observing that all subsequent requests return ZSESSIONMOVED error. Restarting client app helps - we always pass 0 as clientid to zookeeper_init function so old session is not reused.
> On 16-03-2010 we observed few occurences of problem. Example ones:
> - 22:08; client IP 10.1.112.60 (app1); sessionID 0x22767e1c9630000
> - 14:21; client IP 10.1.112.61 (app2); sessionID 0x324dcc1ba580085
> I attach logs of cluster and application nodes (only stuff concerining zookeeper):
> - [^zookeeper-node1.log.2010-03-16.gz] - logs of zookeepr cluster node 1 10.1.112.62
> - [^zookeeper-node2.log.2010-03-16.gz] - logs of zookeepr cluster node 2 10.1.112.63
> - [^zookeeper-node3.log.2010-03-16.gz] - logs of zookeepr cluster node 3 10.1.112.64
> - [^app1.log.2010-03-16.gz] - application logs of app1 10.1.112.60
> - [^app2.log.2010-03-16.gz] - application logs of app2 10.1.112.61
> I also made some analysis of case at 22:08:
> - Network glitch which resulted in problem occurred at about 22:08.
> - From what I see since 17:48 node2 was the leader and it did not
> change later yesterday.
> - Client was connected to node2 since 17:50
> - At around 22:09 client tried to connect to every node (1,2,3).
> Connections to node1 and node3 were closed
> with exception "Exception causing close of session 0x22767e1c9630000
> due to java.io.IOException: Read error".
> Connection to node2 stood alive.
> - All subsequent operations were refused with ZSESSIONMOVED error.
> Error visible both on client and on server side.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-710) permanent ZSESSIONMOVED error
after client app reconnects to zookeeper cluster
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12847250#action_12847250 ]
Patrick Hunt commented on ZOOKEEPER-710:
----------------------------------------
Lukasz feel free to try the attached 3.2 patch. It passes all tests on my system. You should be able to patch/compile against the 3.2 release (patch -p0 < patchfile) or svn 3.2 branch.
It would be great if you could try this out and verify that it fixes the issue you've been seeing with moved.
> permanent ZSESSIONMOVED error after client app reconnects to zookeeper cluster
> ------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-710
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-710
> Project: Zookeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.2.2
> Environment: debian lenny; ia64; xen virtualization
> Reporter: Lukasz Osipiuk
> Assignee: Patrick Hunt
> Priority: Blocker
> Fix For: 3.2.3, 3.3.0
>
> Attachments: app1.log.2010-03-16.gz, app2.log.2010-03-16.gz, ZOOKEEPER-710_3.2.patch, ZOOKEEPER-710_3.3.patch, zookeeper-node1.log.2010-03-16.gz, zookeeper-node2.log.2010-03-16.gz, zookeeper-node3.log.2010-03-16.gz
>
>
> Originally problem was described on Users mailing list starting with this [post|http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201003.mbox/<3b...@mail.gmail.com>].
> Below I restate it in more organized form.
> We occasionally (few times a day) observe that our client application disconnects from Zookeeper cluster.
> Application is written in C++ and we are using libzookeeper_mt library. In version 3.2.2.
> The disconnects we are observing are probably related to some problems with our network infrastructure - we are observing periods with great packet loss between machines in our DC.
> Sometimes after client application (i.e. zookeeper library) reconnects to zookeeper cluster we are observing that all subsequent requests return ZSESSIONMOVED error. Restarting client app helps - we always pass 0 as clientid to zookeeper_init function so old session is not reused.
> On 16-03-2010 we observed few occurences of problem. Example ones:
> - 22:08; client IP 10.1.112.60 (app1); sessionID 0x22767e1c9630000
> - 14:21; client IP 10.1.112.61 (app2); sessionID 0x324dcc1ba580085
> I attach logs of cluster and application nodes (only stuff concerining zookeeper):
> - [^zookeeper-node1.log.2010-03-16.gz] - logs of zookeepr cluster node 1 10.1.112.62
> - [^zookeeper-node2.log.2010-03-16.gz] - logs of zookeepr cluster node 2 10.1.112.63
> - [^zookeeper-node3.log.2010-03-16.gz] - logs of zookeepr cluster node 3 10.1.112.64
> - [^app1.log.2010-03-16.gz] - application logs of app1 10.1.112.60
> - [^app2.log.2010-03-16.gz] - application logs of app2 10.1.112.61
> I also made some analysis of case at 22:08:
> - Network glitch which resulted in problem occurred at about 22:08.
> - From what I see since 17:48 node2 was the leader and it did not
> change later yesterday.
> - Client was connected to node2 since 17:50
> - At around 22:09 client tried to connect to every node (1,2,3).
> Connections to node1 and node3 were closed
> with exception "Exception causing close of session 0x22767e1c9630000
> due to java.io.IOException: Read error".
> Connection to node2 stood alive.
> - All subsequent operations were refused with ZSESSIONMOVED error.
> Error visible both on client and on server side.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-710) permanent ZSESSIONMOVED error after
client app reconnects to zookeeper cluster
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Benjamin Reed updated ZOOKEEPER-710:
------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed revision 925104.
> permanent ZSESSIONMOVED error after client app reconnects to zookeeper cluster
> ------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-710
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-710
> Project: Zookeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.2.2
> Environment: debian lenny; ia64; xen virtualization
> Reporter: Lukasz Osipiuk
> Assignee: Patrick Hunt
> Priority: Blocker
> Fix For: 3.2.3, 3.3.0
>
> Attachments: app1.log.2010-03-16.gz, app2.log.2010-03-16.gz, ZOOKEEPER-710_3.2.patch, ZOOKEEPER-710_3.3.patch, zookeeper-node1.log.2010-03-16.gz, zookeeper-node2.log.2010-03-16.gz, zookeeper-node3.log.2010-03-16.gz
>
>
> Originally problem was described on Users mailing list starting with this [post|http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201003.mbox/<3b...@mail.gmail.com>].
> Below I restate it in more organized form.
> We occasionally (few times a day) observe that our client application disconnects from Zookeeper cluster.
> Application is written in C++ and we are using libzookeeper_mt library. In version 3.2.2.
> The disconnects we are observing are probably related to some problems with our network infrastructure - we are observing periods with great packet loss between machines in our DC.
> Sometimes after client application (i.e. zookeeper library) reconnects to zookeeper cluster we are observing that all subsequent requests return ZSESSIONMOVED error. Restarting client app helps - we always pass 0 as clientid to zookeeper_init function so old session is not reused.
> On 16-03-2010 we observed few occurences of problem. Example ones:
> - 22:08; client IP 10.1.112.60 (app1); sessionID 0x22767e1c9630000
> - 14:21; client IP 10.1.112.61 (app2); sessionID 0x324dcc1ba580085
> I attach logs of cluster and application nodes (only stuff concerining zookeeper):
> - [^zookeeper-node1.log.2010-03-16.gz] - logs of zookeepr cluster node 1 10.1.112.62
> - [^zookeeper-node2.log.2010-03-16.gz] - logs of zookeepr cluster node 2 10.1.112.63
> - [^zookeeper-node3.log.2010-03-16.gz] - logs of zookeepr cluster node 3 10.1.112.64
> - [^app1.log.2010-03-16.gz] - application logs of app1 10.1.112.60
> - [^app2.log.2010-03-16.gz] - application logs of app2 10.1.112.61
> I also made some analysis of case at 22:08:
> - Network glitch which resulted in problem occurred at about 22:08.
> - From what I see since 17:48 node2 was the leader and it did not
> change later yesterday.
> - Client was connected to node2 since 17:50
> - At around 22:09 client tried to connect to every node (1,2,3).
> Connections to node1 and node3 were closed
> with exception "Exception causing close of session 0x22767e1c9630000
> due to java.io.IOException: Read error".
> Connection to node2 stood alive.
> - All subsequent operations were refused with ZSESSIONMOVED error.
> Error visible both on client and on server side.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (ZOOKEEPER-710) permanent ZSESSIONMOVED error
after client app reconnects to zookeeper cluster
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt reopened ZOOKEEPER-710:
------------------------------------
> permanent ZSESSIONMOVED error after client app reconnects to zookeeper cluster
> ------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-710
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-710
> Project: Zookeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.2.2
> Environment: debian lenny; ia64; xen virtualization
> Reporter: Lukasz Osipiuk
> Assignee: Patrick Hunt
> Priority: Blocker
> Fix For: 3.2.3, 3.3.0
>
> Attachments: app1.log.2010-03-16.gz, app2.log.2010-03-16.gz, ZOOKEEPER-710_3.2.patch, ZOOKEEPER-710_3.3.patch, zookeeper-node1.log.2010-03-16.gz, zookeeper-node2.log.2010-03-16.gz, zookeeper-node3.log.2010-03-16.gz
>
>
> Originally problem was described on Users mailing list starting with this [post|http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201003.mbox/<3b...@mail.gmail.com>].
> Below I restate it in more organized form.
> We occasionally (few times a day) observe that our client application disconnects from Zookeeper cluster.
> Application is written in C++ and we are using libzookeeper_mt library. In version 3.2.2.
> The disconnects we are observing are probably related to some problems with our network infrastructure - we are observing periods with great packet loss between machines in our DC.
> Sometimes after client application (i.e. zookeeper library) reconnects to zookeeper cluster we are observing that all subsequent requests return ZSESSIONMOVED error. Restarting client app helps - we always pass 0 as clientid to zookeeper_init function so old session is not reused.
> On 16-03-2010 we observed few occurences of problem. Example ones:
> - 22:08; client IP 10.1.112.60 (app1); sessionID 0x22767e1c9630000
> - 14:21; client IP 10.1.112.61 (app2); sessionID 0x324dcc1ba580085
> I attach logs of cluster and application nodes (only stuff concerining zookeeper):
> - [^zookeeper-node1.log.2010-03-16.gz] - logs of zookeepr cluster node 1 10.1.112.62
> - [^zookeeper-node2.log.2010-03-16.gz] - logs of zookeepr cluster node 2 10.1.112.63
> - [^zookeeper-node3.log.2010-03-16.gz] - logs of zookeepr cluster node 3 10.1.112.64
> - [^app1.log.2010-03-16.gz] - application logs of app1 10.1.112.60
> - [^app2.log.2010-03-16.gz] - application logs of app2 10.1.112.61
> I also made some analysis of case at 22:08:
> - Network glitch which resulted in problem occurred at about 22:08.
> - From what I see since 17:48 node2 was the leader and it did not
> change later yesterday.
> - Client was connected to node2 since 17:50
> - At around 22:09 client tried to connect to every node (1,2,3).
> Connections to node1 and node3 were closed
> with exception "Exception causing close of session 0x22767e1c9630000
> due to java.io.IOException: Read error".
> Connection to node2 stood alive.
> - All subsequent operations were refused with ZSESSIONMOVED error.
> Error visible both on client and on server side.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-710) permanent ZSESSIONMOVED error
after client app reconnects to zookeeper cluster
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12847815#action_12847815 ]
Hudson commented on ZOOKEEPER-710:
----------------------------------
Integrated in ZooKeeper-trunk #763 (See [http://hudson.zones.apache.org/hudson/job/ZooKeeper-trunk/763/])
. Forgot to run ant docs
> permanent ZSESSIONMOVED error after client app reconnects to zookeeper cluster
> ------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-710
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-710
> Project: Zookeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.2.2
> Environment: debian lenny; ia64; xen virtualization
> Reporter: Lukasz Osipiuk
> Assignee: Patrick Hunt
> Priority: Blocker
> Fix For: 3.2.3, 3.3.0
>
> Attachments: app1.log.2010-03-16.gz, app2.log.2010-03-16.gz, ZOOKEEPER-710_3.2.patch, ZOOKEEPER-710_3.3.patch, zookeeper-node1.log.2010-03-16.gz, zookeeper-node2.log.2010-03-16.gz, zookeeper-node3.log.2010-03-16.gz
>
>
> Originally problem was described on Users mailing list starting with this [post|http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201003.mbox/<3b...@mail.gmail.com>].
> Below I restate it in more organized form.
> We occasionally (few times a day) observe that our client application disconnects from Zookeeper cluster.
> Application is written in C++ and we are using libzookeeper_mt library. In version 3.2.2.
> The disconnects we are observing are probably related to some problems with our network infrastructure - we are observing periods with great packet loss between machines in our DC.
> Sometimes after client application (i.e. zookeeper library) reconnects to zookeeper cluster we are observing that all subsequent requests return ZSESSIONMOVED error. Restarting client app helps - we always pass 0 as clientid to zookeeper_init function so old session is not reused.
> On 16-03-2010 we observed few occurences of problem. Example ones:
> - 22:08; client IP 10.1.112.60 (app1); sessionID 0x22767e1c9630000
> - 14:21; client IP 10.1.112.61 (app2); sessionID 0x324dcc1ba580085
> I attach logs of cluster and application nodes (only stuff concerining zookeeper):
> - [^zookeeper-node1.log.2010-03-16.gz] - logs of zookeepr cluster node 1 10.1.112.62
> - [^zookeeper-node2.log.2010-03-16.gz] - logs of zookeepr cluster node 2 10.1.112.63
> - [^zookeeper-node3.log.2010-03-16.gz] - logs of zookeepr cluster node 3 10.1.112.64
> - [^app1.log.2010-03-16.gz] - application logs of app1 10.1.112.60
> - [^app2.log.2010-03-16.gz] - application logs of app2 10.1.112.61
> I also made some analysis of case at 22:08:
> - Network glitch which resulted in problem occurred at about 22:08.
> - From what I see since 17:48 node2 was the leader and it did not
> change later yesterday.
> - Client was connected to node2 since 17:50
> - At around 22:09 client tried to connect to every node (1,2,3).
> Connections to node1 and node3 were closed
> with exception "Exception causing close of session 0x22767e1c9630000
> due to java.io.IOException: Read error".
> Connection to node2 stood alive.
> - All subsequent operations were refused with ZSESSIONMOVED error.
> Error visible both on client and on server side.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-710) permanent ZSESSIONMOVED error after
client app reconnects to zookeeper cluster
Posted by "Lukasz Osipiuk (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lukasz Osipiuk updated ZOOKEEPER-710:
-------------------------------------
Attachment: app1.log.2010-03-16.gz
> permanent ZSESSIONMOVED error after client app reconnects to zookeeper cluster
> ------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-710
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-710
> Project: Zookeeper
> Issue Type: Bug
> Environment: debian lenny; ia64; xen virtualization
> Reporter: Lukasz Osipiuk
> Attachments: app1.log.2010-03-16.gz, zookeeper-node1.log.2010-03-16.gz, zookeeper-node2.log.2010-03-16.gz, zookeeper-node3.log.2010-03-16.gz
>
>
> Originally problem was described on Users mailing list starting with this [post|http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201003.mbox/<3b...@mail.gmail.com>].
> Below I restate it in more organized form.
> We occasionally (few times a day) observe that our client application disconnects from Zookeeper cluster.
> Application is written in C++ and we are using libzookeeper_mt library. In version 3.2.2.
> The disconnects we are observing are probably related to some problems with our network infrastructure - we are observing periods with great packet loss between machines in our DC.
> Sometimes after client application (i.e. zookeeper library) reconnects to zookeeper cluster we are observing that all subsequent requests return ZSESSIONMOVED error. Restarting client app helps - we always pass 0 as clientid to zookeeper_init function so old session is not reused.
> On 16-03-2010 we observed few occurences of problem. Example ones:
> - 22:08; client IP 10.1.112.60 (app1)
> - 14:21; client IP 10.1.112.61 (app2)
> I attach logs of cluster and application nodes (only stuff concerining zookeeper):
> - zookeeper-node1.log.2010-03-16.gz
> - zookeeper-node2.log.2010-03-16.gz
> - zookeeper-node3.log.2010-03-16.gz
> - app1.log.2010-03-16.gz
> - app2.log.2010-03-16.gz
> I also made some analysis of case at 22:08:
> - Network glitch which resulted in problem occurred at about 22:08.
> - From what I see since 17:48 node2 was the leader and it did not
> change later yesterday.
> - Client was connected to node2 since 17:50
> - At around 22:09 client tried to connect to every node (1,2,3).
> Connections to node1 and node3 were closed
> with exception "Exception causing close of session 0x22767e1c9630000
> due to java.io.IOException: Read error".
> Connection to node2 stood alive.
> - All subsequent operations were refused with ZSESSIONMOVED error.
> Error visible both on client and on server side.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-710) permanent ZSESSIONMOVED error after
client app reconnects to zookeeper cluster
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Benjamin Reed updated ZOOKEEPER-710:
------------------------------------
Hadoop Flags: [Reviewed]
+1 great work pat! and thanx Lukasz for identifying this failure condition.
> permanent ZSESSIONMOVED error after client app reconnects to zookeeper cluster
> ------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-710
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-710
> Project: Zookeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.2.2
> Environment: debian lenny; ia64; xen virtualization
> Reporter: Lukasz Osipiuk
> Assignee: Patrick Hunt
> Priority: Blocker
> Fix For: 3.2.3, 3.3.0
>
> Attachments: app1.log.2010-03-16.gz, app2.log.2010-03-16.gz, ZOOKEEPER-710_3.3.patch, zookeeper-node1.log.2010-03-16.gz, zookeeper-node2.log.2010-03-16.gz, zookeeper-node3.log.2010-03-16.gz
>
>
> Originally problem was described on Users mailing list starting with this [post|http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201003.mbox/<3b...@mail.gmail.com>].
> Below I restate it in more organized form.
> We occasionally (few times a day) observe that our client application disconnects from Zookeeper cluster.
> Application is written in C++ and we are using libzookeeper_mt library. In version 3.2.2.
> The disconnects we are observing are probably related to some problems with our network infrastructure - we are observing periods with great packet loss between machines in our DC.
> Sometimes after client application (i.e. zookeeper library) reconnects to zookeeper cluster we are observing that all subsequent requests return ZSESSIONMOVED error. Restarting client app helps - we always pass 0 as clientid to zookeeper_init function so old session is not reused.
> On 16-03-2010 we observed few occurences of problem. Example ones:
> - 22:08; client IP 10.1.112.60 (app1); sessionID 0x22767e1c9630000
> - 14:21; client IP 10.1.112.61 (app2); sessionID 0x324dcc1ba580085
> I attach logs of cluster and application nodes (only stuff concerining zookeeper):
> - [^zookeeper-node1.log.2010-03-16.gz] - logs of zookeepr cluster node 1 10.1.112.62
> - [^zookeeper-node2.log.2010-03-16.gz] - logs of zookeepr cluster node 2 10.1.112.63
> - [^zookeeper-node3.log.2010-03-16.gz] - logs of zookeepr cluster node 3 10.1.112.64
> - [^app1.log.2010-03-16.gz] - application logs of app1 10.1.112.60
> - [^app2.log.2010-03-16.gz] - application logs of app2 10.1.112.61
> I also made some analysis of case at 22:08:
> - Network glitch which resulted in problem occurred at about 22:08.
> - From what I see since 17:48 node2 was the leader and it did not
> change later yesterday.
> - Client was connected to node2 since 17:50
> - At around 22:09 client tried to connect to every node (1,2,3).
> Connections to node1 and node3 were closed
> with exception "Exception causing close of session 0x22767e1c9630000
> due to java.io.IOException: Read error".
> Connection to node2 stood alive.
> - All subsequent operations were refused with ZSESSIONMOVED error.
> Error visible both on client and on server side.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-710) permanent ZSESSIONMOVED error
after client app reconnects to zookeeper cluster
Posted by "Lukasz Osipiuk (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12847080#action_12847080 ]
Lukasz Osipiuk commented on ZOOKEEPER-710:
------------------------------------------
This is application stuff.
Lets take exist function from our code which wraps zookeeper library as an example.
Its declaration is:
{noformat}
ZooKeeper::ErrorCode::Type ZooKeeper::exists(const std::string& path,
Stat &stat,
const WatchFunction& watchFunction,
WatchId& watchId,
apr_time_t timeout)
{noformat}
As you see caller passes high level watchFunction to be called.
Our wrapper maintains a map (node,operationtype)->watcherFunction. Internal watch function which is used in zoo_* calls
is responsible for calling user provided watchFunctions stored in map in case of change concerning one of nodes.
When client disconnects from zookeeper all watches are invalidated (at least AFAIK) so we are calling all user provided functions in such case.
We are doing this as soon client is reconnected to zookeeper. We simulate situation when change occured to all nodes application was interested in. That is what fireAllWatchFunctions function do.
Is it somewhat clear?
> permanent ZSESSIONMOVED error after client app reconnects to zookeeper cluster
> ------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-710
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-710
> Project: Zookeeper
> Issue Type: Bug
> Affects Versions: 3.2.2
> Environment: debian lenny; ia64; xen virtualization
> Reporter: Lukasz Osipiuk
> Attachments: app1.log.2010-03-16.gz, app2.log.2010-03-16.gz, zookeeper-node1.log.2010-03-16.gz, zookeeper-node2.log.2010-03-16.gz, zookeeper-node3.log.2010-03-16.gz
>
>
> Originally problem was described on Users mailing list starting with this [post|http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201003.mbox/<3b...@mail.gmail.com>].
> Below I restate it in more organized form.
> We occasionally (few times a day) observe that our client application disconnects from Zookeeper cluster.
> Application is written in C++ and we are using libzookeeper_mt library. In version 3.2.2.
> The disconnects we are observing are probably related to some problems with our network infrastructure - we are observing periods with great packet loss between machines in our DC.
> Sometimes after client application (i.e. zookeeper library) reconnects to zookeeper cluster we are observing that all subsequent requests return ZSESSIONMOVED error. Restarting client app helps - we always pass 0 as clientid to zookeeper_init function so old session is not reused.
> On 16-03-2010 we observed few occurences of problem. Example ones:
> - 22:08; client IP 10.1.112.60 (app1); sessionID 0x22767e1c9630000
> - 14:21; client IP 10.1.112.61 (app2); sessionID 0x324dcc1ba580085
> I attach logs of cluster and application nodes (only stuff concerining zookeeper):
> - [^zookeeper-node1.log.2010-03-16.gz] - logs of zookeepr cluster node 1 10.1.112.62
> - [^zookeeper-node2.log.2010-03-16.gz] - logs of zookeepr cluster node 2 10.1.112.63
> - [^zookeeper-node3.log.2010-03-16.gz] - logs of zookeepr cluster node 3 10.1.112.64
> - [^app1.log.2010-03-16.gz] - application logs of app1 10.1.112.60
> - [^app2.log.2010-03-16.gz] - application logs of app2 10.1.112.61
> I also made some analysis of case at 22:08:
> - Network glitch which resulted in problem occurred at about 22:08.
> - From what I see since 17:48 node2 was the leader and it did not
> change later yesterday.
> - Client was connected to node2 since 17:50
> - At around 22:09 client tried to connect to every node (1,2,3).
> Connections to node1 and node3 were closed
> with exception "Exception causing close of session 0x22767e1c9630000
> due to java.io.IOException: Read error".
> Connection to node2 stood alive.
> - All subsequent operations were refused with ZSESSIONMOVED error.
> Error visible both on client and on server side.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (ZOOKEEPER-710) permanent
ZSESSIONMOVED error after client app reconnects to zookeeper cluster
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12847017#action_12847017 ]
Mahadev konar edited comment on ZOOKEEPER-710 at 3/18/10 6:05 PM:
------------------------------------------------------------------
lukasz, can you elaborate on what this code is doing?
{code}
Mar 16 14:21:49 XXXXXXXXXX-c1-mbox-2 XXXXXXXXXX_mbox[25112]: DEBUG: ZooKeeper: ZooKeeper::execWatch() executing watch
Mar 16 14:21:49 XXXXXXXXXX-c1-mbox-2 XXXXXXXXXX_mbox[25112]: INFO: ZooKeeper: ZooKeeper::watcherFunHandleSessionEvent() state change to 3
Mar 16 14:21:49 XXXXXXXXXX-c1-mbox-2 XXXXXXXXXX_mbox[25112]: INFO: ZooKeeper: ZooKeeper::fireAllWatchFunctions() calling all watches
{code}
which all watches are your firing?
was (Author: mahadev):
lukasz, can you elaborate on what this code is doing?
{code}
Mar 16 14:21:49 XXXXXXXXXX-c1-mbox-2 XXXXXXXXXX_mbox[25112]: DEBUG: ZooKeeper: ZooKeeper::execWatch() executing watch
Mar 16 14:21:49 XXXXXXXXXX-c1-mbox-2 XXXXXXXXXX_mbox[25112]: INFO: ZooKeeper: ZooKeeper::watcherFunHandleSessionEvent() state change to 3
Mar 16 14:21:49 XXXXXXXXXX-c1-mbox-2 XXXXXXXXXX_mbox[25112]: INFO: ZooKeeper: ZooKeeper::fireAllWatchFunctions() calling all watches
{code}
who all watches are your firing?
> permanent ZSESSIONMOVED error after client app reconnects to zookeeper cluster
> ------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-710
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-710
> Project: Zookeeper
> Issue Type: Bug
> Affects Versions: 3.2.2
> Environment: debian lenny; ia64; xen virtualization
> Reporter: Lukasz Osipiuk
> Attachments: app1.log.2010-03-16.gz, app2.log.2010-03-16.gz, zookeeper-node1.log.2010-03-16.gz, zookeeper-node2.log.2010-03-16.gz, zookeeper-node3.log.2010-03-16.gz
>
>
> Originally problem was described on Users mailing list starting with this [post|http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201003.mbox/<3b...@mail.gmail.com>].
> Below I restate it in more organized form.
> We occasionally (few times a day) observe that our client application disconnects from Zookeeper cluster.
> Application is written in C++ and we are using libzookeeper_mt library. In version 3.2.2.
> The disconnects we are observing are probably related to some problems with our network infrastructure - we are observing periods with great packet loss between machines in our DC.
> Sometimes after client application (i.e. zookeeper library) reconnects to zookeeper cluster we are observing that all subsequent requests return ZSESSIONMOVED error. Restarting client app helps - we always pass 0 as clientid to zookeeper_init function so old session is not reused.
> On 16-03-2010 we observed few occurences of problem. Example ones:
> - 22:08; client IP 10.1.112.60 (app1); sessionID 0x22767e1c9630000
> - 14:21; client IP 10.1.112.61 (app2); sessionID 0x324dcc1ba580085
> I attach logs of cluster and application nodes (only stuff concerining zookeeper):
> - [^zookeeper-node1.log.2010-03-16.gz] - logs of zookeepr cluster node 1 10.1.112.62
> - [^zookeeper-node2.log.2010-03-16.gz] - logs of zookeepr cluster node 2 10.1.112.63
> - [^zookeeper-node3.log.2010-03-16.gz] - logs of zookeepr cluster node 3 10.1.112.64
> - [^app1.log.2010-03-16.gz] - application logs of app1 10.1.112.60
> - [^app2.log.2010-03-16.gz] - application logs of app2 10.1.112.61
> I also made some analysis of case at 22:08:
> - Network glitch which resulted in problem occurred at about 22:08.
> - From what I see since 17:48 node2 was the leader and it did not
> change later yesterday.
> - Client was connected to node2 since 17:50
> - At around 22:09 client tried to connect to every node (1,2,3).
> Connections to node1 and node3 were closed
> with exception "Exception causing close of session 0x22767e1c9630000
> due to java.io.IOException: Read error".
> Connection to node2 stood alive.
> - All subsequent operations were refused with ZSESSIONMOVED error.
> Error visible both on client and on server side.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-710) permanent ZSESSIONMOVED error
after client app reconnects to zookeeper cluster
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12847231#action_12847231 ]
Hadoop QA commented on ZOOKEEPER-710:
-------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12439239/ZOOKEEPER-710_3.3.patch
against trunk revision 924942.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/143/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/143/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/143/console
This message is automatically generated.
> permanent ZSESSIONMOVED error after client app reconnects to zookeeper cluster
> ------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-710
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-710
> Project: Zookeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.2.2
> Environment: debian lenny; ia64; xen virtualization
> Reporter: Lukasz Osipiuk
> Assignee: Patrick Hunt
> Priority: Blocker
> Fix For: 3.2.3, 3.3.0
>
> Attachments: app1.log.2010-03-16.gz, app2.log.2010-03-16.gz, ZOOKEEPER-710_3.3.patch, zookeeper-node1.log.2010-03-16.gz, zookeeper-node2.log.2010-03-16.gz, zookeeper-node3.log.2010-03-16.gz
>
>
> Originally problem was described on Users mailing list starting with this [post|http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201003.mbox/<3b...@mail.gmail.com>].
> Below I restate it in more organized form.
> We occasionally (few times a day) observe that our client application disconnects from Zookeeper cluster.
> Application is written in C++ and we are using libzookeeper_mt library. In version 3.2.2.
> The disconnects we are observing are probably related to some problems with our network infrastructure - we are observing periods with great packet loss between machines in our DC.
> Sometimes after client application (i.e. zookeeper library) reconnects to zookeeper cluster we are observing that all subsequent requests return ZSESSIONMOVED error. Restarting client app helps - we always pass 0 as clientid to zookeeper_init function so old session is not reused.
> On 16-03-2010 we observed few occurences of problem. Example ones:
> - 22:08; client IP 10.1.112.60 (app1); sessionID 0x22767e1c9630000
> - 14:21; client IP 10.1.112.61 (app2); sessionID 0x324dcc1ba580085
> I attach logs of cluster and application nodes (only stuff concerining zookeeper):
> - [^zookeeper-node1.log.2010-03-16.gz] - logs of zookeepr cluster node 1 10.1.112.62
> - [^zookeeper-node2.log.2010-03-16.gz] - logs of zookeepr cluster node 2 10.1.112.63
> - [^zookeeper-node3.log.2010-03-16.gz] - logs of zookeepr cluster node 3 10.1.112.64
> - [^app1.log.2010-03-16.gz] - application logs of app1 10.1.112.60
> - [^app2.log.2010-03-16.gz] - application logs of app2 10.1.112.61
> I also made some analysis of case at 22:08:
> - Network glitch which resulted in problem occurred at about 22:08.
> - From what I see since 17:48 node2 was the leader and it did not
> change later yesterday.
> - Client was connected to node2 since 17:50
> - At around 22:09 client tried to connect to every node (1,2,3).
> Connections to node1 and node3 were closed
> with exception "Exception causing close of session 0x22767e1c9630000
> due to java.io.IOException: Read error".
> Connection to node2 stood alive.
> - All subsequent operations were refused with ZSESSIONMOVED error.
> Error visible both on client and on server side.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-710) permanent ZSESSIONMOVED error
after client app reconnects to zookeeper cluster
Posted by "Lukasz Osipiuk (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12846864#action_12846864 ]
Lukasz Osipiuk commented on ZOOKEEPER-710:
------------------------------------------
ZooKeeper::getChildren() path='/ggmboxlocks/16314'; callback error='session moved to another server, so operation is ignored' logs are printed using
log_error(logger, "ZooKeeper::getChildren() path='%s'; callback error='%s'", path.c_str(), zerror(retval)) where retval is rc code from callback.
> permanent ZSESSIONMOVED error after client app reconnects to zookeeper cluster
> ------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-710
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-710
> Project: Zookeeper
> Issue Type: Bug
> Affects Versions: 3.2.2
> Environment: debian lenny; ia64; xen virtualization
> Reporter: Lukasz Osipiuk
> Attachments: app1.log.2010-03-16.gz, app2.log.2010-03-16.gz, zookeeper-node1.log.2010-03-16.gz, zookeeper-node2.log.2010-03-16.gz, zookeeper-node3.log.2010-03-16.gz
>
>
> Originally problem was described on Users mailing list starting with this [post|http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201003.mbox/<3b...@mail.gmail.com>].
> Below I restate it in more organized form.
> We occasionally (few times a day) observe that our client application disconnects from Zookeeper cluster.
> Application is written in C++ and we are using libzookeeper_mt library. In version 3.2.2.
> The disconnects we are observing are probably related to some problems with our network infrastructure - we are observing periods with great packet loss between machines in our DC.
> Sometimes after client application (i.e. zookeeper library) reconnects to zookeeper cluster we are observing that all subsequent requests return ZSESSIONMOVED error. Restarting client app helps - we always pass 0 as clientid to zookeeper_init function so old session is not reused.
> On 16-03-2010 we observed few occurences of problem. Example ones:
> - 22:08; client IP 10.1.112.60 (app1); sessionID 0x22767e1c9630000
> - 14:21; client IP 10.1.112.61 (app2); sessionID 0x324dcc1ba580085
> I attach logs of cluster and application nodes (only stuff concerining zookeeper):
> - [^zookeeper-node1.log.2010-03-16.gz] - logs of zookeepr cluster node 1 10.1.112.62
> - [^zookeeper-node2.log.2010-03-16.gz] - logs of zookeepr cluster node 2 10.1.112.63
> - [^zookeeper-node3.log.2010-03-16.gz] - logs of zookeepr cluster node 3 10.1.112.64
> - [^app1.log.2010-03-16.gz] - application logs of app1 10.1.112.60
> - [^app2.log.2010-03-16.gz] - application logs of app2 10.1.112.61
> I also made some analysis of case at 22:08:
> - Network glitch which resulted in problem occurred at about 22:08.
> - From what I see since 17:48 node2 was the leader and it did not
> change later yesterday.
> - Client was connected to node2 since 17:50
> - At around 22:09 client tried to connect to every node (1,2,3).
> Connections to node1 and node3 were closed
> with exception "Exception causing close of session 0x22767e1c9630000
> due to java.io.IOException: Read error".
> Connection to node2 stood alive.
> - All subsequent operations were refused with ZSESSIONMOVED error.
> Error visible both on client and on server side.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-710) permanent ZSESSIONMOVED error
after client app reconnects to zookeeper cluster
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12847085#action_12847085 ]
Benjamin Reed commented on ZOOKEEPER-710:
-----------------------------------------
lukasz thank you for the logs! they have really helped pinpoint the problem. it appears that message delays are the cause of the problem. it appears that what is happening is that the client is connecting to a follower that is running very slow. the follower is slow processing the reconnect request and forwarding it to the leader. for example, the following shows a case where it takes 7 seconds for the follower to process a reestablish connection request.
{quote}
zookeeper-node3.log.2010-03-16:2010-03-16 14:21:41,267 - INFO [NIOServerCxn.Factory:2181:NIOServerCnxn@964] - Finished init of 0x224ea5675b4007b valid:true
zookeeper-node3.log.2010-03-16:2010-03-16 14:21:48,937 - INFO [NIOServerCxn.Factory:2181:NIOServerCnxn@612] - Renewing session 0x224ea5675b4007b
{quote}
the problem is that within that 7 seconds a client can drop the connection to the follower and establish a connection to the leader. if the renew session request is delayed until after the client reestablishes the connection with the leader, the session will get marked as managed by the previous follower and all requests set to the leader will get MOVED error, which is what you are seeing.
the problem is that we consider the MOVED error to be a request error rather than a connection error. the leader needs to drop the connection when it sees the MOVED error so that the client can reestablish the session properly with another machine.
> permanent ZSESSIONMOVED error after client app reconnects to zookeeper cluster
> ------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-710
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-710
> Project: Zookeeper
> Issue Type: Bug
> Affects Versions: 3.2.2
> Environment: debian lenny; ia64; xen virtualization
> Reporter: Lukasz Osipiuk
> Attachments: app1.log.2010-03-16.gz, app2.log.2010-03-16.gz, zookeeper-node1.log.2010-03-16.gz, zookeeper-node2.log.2010-03-16.gz, zookeeper-node3.log.2010-03-16.gz
>
>
> Originally problem was described on Users mailing list starting with this [post|http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201003.mbox/<3b...@mail.gmail.com>].
> Below I restate it in more organized form.
> We occasionally (few times a day) observe that our client application disconnects from Zookeeper cluster.
> Application is written in C++ and we are using libzookeeper_mt library. In version 3.2.2.
> The disconnects we are observing are probably related to some problems with our network infrastructure - we are observing periods with great packet loss between machines in our DC.
> Sometimes after client application (i.e. zookeeper library) reconnects to zookeeper cluster we are observing that all subsequent requests return ZSESSIONMOVED error. Restarting client app helps - we always pass 0 as clientid to zookeeper_init function so old session is not reused.
> On 16-03-2010 we observed few occurences of problem. Example ones:
> - 22:08; client IP 10.1.112.60 (app1); sessionID 0x22767e1c9630000
> - 14:21; client IP 10.1.112.61 (app2); sessionID 0x324dcc1ba580085
> I attach logs of cluster and application nodes (only stuff concerining zookeeper):
> - [^zookeeper-node1.log.2010-03-16.gz] - logs of zookeepr cluster node 1 10.1.112.62
> - [^zookeeper-node2.log.2010-03-16.gz] - logs of zookeepr cluster node 2 10.1.112.63
> - [^zookeeper-node3.log.2010-03-16.gz] - logs of zookeepr cluster node 3 10.1.112.64
> - [^app1.log.2010-03-16.gz] - application logs of app1 10.1.112.60
> - [^app2.log.2010-03-16.gz] - application logs of app2 10.1.112.61
> I also made some analysis of case at 22:08:
> - Network glitch which resulted in problem occurred at about 22:08.
> - From what I see since 17:48 node2 was the leader and it did not
> change later yesterday.
> - Client was connected to node2 since 17:50
> - At around 22:09 client tried to connect to every node (1,2,3).
> Connections to node1 and node3 were closed
> with exception "Exception causing close of session 0x22767e1c9630000
> due to java.io.IOException: Read error".
> Connection to node2 stood alive.
> - All subsequent operations were refused with ZSESSIONMOVED error.
> Error visible both on client and on server side.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.