You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bookkeeper.apache.org by "Sijie Guo (Created) (JIRA)" <ji...@apache.org> on 2011/10/14 09:38:12 UTC
[jira] [Created] (BOOKKEEPER-79) randomly
startDelivery/stopDelivery will core dump in c++ hedwig client
randomly startDelivery/stopDelivery will core dump in c++ hedwig client
-----------------------------------------------------------------------
Key: BOOKKEEPER-79
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-79
Project: Bookkeeper
Issue Type: Bug
Components: hedwig-client
Affects Versions: 3.4.0
Reporter: Sijie Guo
in our test program, we tried to startDelivery/stopDelivery different subscriptions randomly. And it core dump.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-79) randomly
startDelivery/stopDelivery will core dump in c++ hedwig client
Posted by "Sijie Guo (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152982#comment-13152982 ]
Sijie Guo commented on BOOKKEEPER-79:
-------------------------------------
I have tried running 'make check' several times. it is OK for me, only one time there is an Exception thrown in socket.close, the exception indicates closing a Not Connect socket.
I think the exception is due to shut down a channel:
{code}
void DuplexChannel::kill() {
...
socket.cancel();
socket.shutdown(boost::asio::ip::tcp::socket::shutdown_both);
socket.close();
...
}
{code}
I think we can change using cancel/shutdown/close to cancel(&error)/shutdown(&error)/close(&error), which returning an error code instead of throwing exception. http://www.boost.org/doc/libs/1_48_0/doc/html/boost_asio/reference/basic_stream_socket/close/overload2.html
I am not sure the problems described in your comments, could you give more information about them?
> randomly startDelivery/stopDelivery will core dump in c++ hedwig client
> -----------------------------------------------------------------------
>
> Key: BOOKKEEPER-79
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-79
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-client
> Affects Versions: 4.0.0
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.0.0
>
> Attachments: BOOKKEEPER-79.patch_v2, BOOKKEEPER-79.patch_v3, bookkeeper-79.patch, pubsubtest.cpp
>
>
> in our test program, we tried to startDelivery/stopDelivery different subscriptions randomly. And it core dump.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (BOOKKEEPER-79) randomly
startDelivery/stopDelivery will core dump in c++ hedwig client
Posted by "Ivan Kelly (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152019#comment-13152019 ]
Ivan Kelly edited comment on BOOKKEEPER-79 at 11/17/11 12:21 PM:
-----------------------------------------------------------------
I'm running the testcase (without the fix) and I'm getting a number of different errors. What error do you get which shows this problem?
Also, I'm not sure the fix is correct. I don't think it's possible for receiving to be false, while stopReceivingBefore is false, because there's a lock,
{code}
boost::lock_guard<boost::mutex> lock(receiving_lock);
{code}
which explicitly stops this situation.
What this change seems to do is that, once you stopReceiving, the channel can never start receiving again?
was (Author: ikelly):
I'm running the testcase (without the fix) and I'm getting a number of different errors. What error do you get which shows this problem?
Also, I'm not sure the fix is correct. I don't think it's possible for receiving to be false, while stopReceivingBefore is false, because there's a lock,
{code}
boost::lock_guard<boost::mutex> lock(receiving_lock);
{code}
which explicitly stops this situation.
> randomly startDelivery/stopDelivery will core dump in c++ hedwig client
> -----------------------------------------------------------------------
>
> Key: BOOKKEEPER-79
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-79
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-client
> Affects Versions: 4.0.0
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.0.0
>
> Attachments: BOOKKEEPER-79.patch_v2, bookkeeper-79.patch
>
>
> in our test program, we tried to startDelivery/stopDelivery different subscriptions randomly. And it core dump.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-79) randomly
startDelivery/stopDelivery will core dump in c++ hedwig client
Posted by "Ivan Kelly (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-79?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Kelly updated BOOKKEEPER-79:
---------------------------------
Attachment: pubsubtest.cpp
> randomly startDelivery/stopDelivery will core dump in c++ hedwig client
> -----------------------------------------------------------------------
>
> Key: BOOKKEEPER-79
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-79
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-client
> Affects Versions: 4.0.0
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.0.0
>
> Attachments: BOOKKEEPER-79.patch_v2, bookkeeper-79.patch, pubsubtest.cpp
>
>
> in our test program, we tried to startDelivery/stopDelivery different subscriptions randomly. And it core dump.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-79) randomly
startDelivery/stopDelivery will core dump in c++ hedwig client
Posted by "Sijie Guo (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-79?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sijie Guo updated BOOKKEEPER-79:
--------------------------------
Attachment: BOOKKEEPER-79.patch_v4
new patch, rename last_response to outstanding_response
> randomly startDelivery/stopDelivery will core dump in c++ hedwig client
> -----------------------------------------------------------------------
>
> Key: BOOKKEEPER-79
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-79
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-client
> Affects Versions: 4.0.0
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.0.0
>
> Attachments: BOOKKEEPER-79.patch_v2, BOOKKEEPER-79.patch_v3, BOOKKEEPER-79.patch_v4, bookkeeper-79.patch, pubsubtest.cpp
>
>
> in our test program, we tried to startDelivery/stopDelivery different subscriptions randomly. And it core dump.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-79) randomly
startDelivery/stopDelivery will core dump in c++ hedwig client
Posted by "Sijie Guo (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-79?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sijie Guo updated BOOKKEEPER-79:
--------------------------------
Fix Version/s: 4.0.0
> randomly startDelivery/stopDelivery will core dump in c++ hedwig client
> -----------------------------------------------------------------------
>
> Key: BOOKKEEPER-79
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-79
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-client
> Affects Versions: 3.4.0
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.0.0
>
> Attachments: bookkeeper-79.patch
>
>
> in our test program, we tried to startDelivery/stopDelivery different subscriptions randomly. And it core dump.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-79) randomly
startDelivery/stopDelivery will core dump in c++ hedwig client
Posted by "Ivan Kelly (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-79?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Kelly updated BOOKKEEPER-79:
---------------------------------
Attachment: make-check-errors.txt
> randomly startDelivery/stopDelivery will core dump in c++ hedwig client
> -----------------------------------------------------------------------
>
> Key: BOOKKEEPER-79
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-79
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-client
> Affects Versions: 4.0.0
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.0.0
>
> Attachments: BOOKKEEPER-79.patch_v2, BOOKKEEPER-79.patch_v3, BOOKKEEPER-79.patch_v4, bookkeeper-79.patch, make-check-errors.txt, pubsubtest.cpp
>
>
> in our test program, we tried to startDelivery/stopDelivery different subscriptions randomly. And it core dump.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-79) randomly
startDelivery/stopDelivery will core dump in c++ hedwig client
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13154162#comment-13154162 ]
Hudson commented on BOOKKEEPER-79:
----------------------------------
Integrated in bookkeeper-trunk #234 (See [https://builds.apache.org/job/bookkeeper-trunk/234/])
BOOKKEEPER-79: randomly startDelivery/stopDelivery will core dump in c++ hedwig client (Sijie Guo via ivank)
ivank :
Files :
* /zookeeper/bookkeeper/trunk/CHANGES.txt
* /zookeeper/bookkeeper/trunk/hedwig-client/src/main/cpp/lib/channel.cpp
* /zookeeper/bookkeeper/trunk/hedwig-client/src/main/cpp/lib/channel.h
* /zookeeper/bookkeeper/trunk/hedwig-client/src/main/cpp/lib/subscriberimpl.cpp
* /zookeeper/bookkeeper/trunk/hedwig-client/src/main/cpp/scripts/server-control.sh
* /zookeeper/bookkeeper/trunk/hedwig-client/src/main/cpp/test/pubsubtest.cpp
> randomly startDelivery/stopDelivery will core dump in c++ hedwig client
> -----------------------------------------------------------------------
>
> Key: BOOKKEEPER-79
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-79
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-client
> Affects Versions: 4.0.0
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.0.0
>
> Attachments: BOOKKEEPER-79.patch_v2, BOOKKEEPER-79.patch_v3, BOOKKEEPER-79.patch_v4, BOOKKEEPER-79.patch_v5, bookkeeper-79.patch, make-check-errors.txt, pubsubtest.cpp
>
>
> in our test program, we tried to startDelivery/stopDelivery different subscriptions randomly. And it core dump.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-79) randomly
startDelivery/stopDelivery will core dump in c++ hedwig client
Posted by "Sijie Guo (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-79?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sijie Guo updated BOOKKEEPER-79:
--------------------------------
Attachment: bookkeeper-79.patch
avoid putting asyc io in boost io queue when startDelivery again.
> randomly startDelivery/stopDelivery will core dump in c++ hedwig client
> -----------------------------------------------------------------------
>
> Key: BOOKKEEPER-79
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-79
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-client
> Affects Versions: 3.4.0
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Attachments: bookkeeper-79.patch
>
>
> in our test program, we tried to startDelivery/stopDelivery different subscriptions randomly. And it core dump.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-79) randomly
startDelivery/stopDelivery will core dump in c++ hedwig client
Posted by "Ivan Kelly (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13153013#comment-13153013 ]
Ivan Kelly commented on BOOKKEEPER-79:
--------------------------------------
Have to you patched against the latest trunk? Did you clean before the check? I've attached a file with the errors I get. The cancel/shutdown/error idea is good.
Also, pubsubtest.cpp gives warnings.
> randomly startDelivery/stopDelivery will core dump in c++ hedwig client
> -----------------------------------------------------------------------
>
> Key: BOOKKEEPER-79
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-79
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-client
> Affects Versions: 4.0.0
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.0.0
>
> Attachments: BOOKKEEPER-79.patch_v2, BOOKKEEPER-79.patch_v3, BOOKKEEPER-79.patch_v4, bookkeeper-79.patch, make-check-errors.txt, pubsubtest.cpp
>
>
> in our test program, we tried to startDelivery/stopDelivery different subscriptions randomly. And it core dump.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-79) randomly
startDelivery/stopDelivery will core dump in c++ hedwig client
Posted by "Sijie Guo (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-79?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sijie Guo updated BOOKKEEPER-79:
--------------------------------
Attachment: BOOKKEEPER-79.patch_v2
attach new patch with test case.
> randomly startDelivery/stopDelivery will core dump in c++ hedwig client
> -----------------------------------------------------------------------
>
> Key: BOOKKEEPER-79
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-79
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-client
> Affects Versions: 4.0.0
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.0.0
>
> Attachments: BOOKKEEPER-79.patch_v2, bookkeeper-79.patch
>
>
> in our test program, we tried to startDelivery/stopDelivery different subscriptions randomly. And it core dump.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-79) randomly
startDelivery/stopDelivery will core dump in c++ hedwig client
Posted by "Ivan Kelly (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130660#comment-13130660 ]
Ivan Kelly commented on BOOKKEEPER-79:
--------------------------------------
Could you add your test program to the tests in hedwig-client/src/main/cpp/test/. I've added another JIRA, BOOKKEEPER-71 which fixes the tests, which hadn't really been run since we opensourced hedwig. Perhaps you could review that change?
> randomly startDelivery/stopDelivery will core dump in c++ hedwig client
> -----------------------------------------------------------------------
>
> Key: BOOKKEEPER-79
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-79
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-client
> Affects Versions: 3.4.0
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Attachments: bookkeeper-79.patch
>
>
> in our test program, we tried to startDelivery/stopDelivery different subscriptions randomly. And it core dump.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-79) randomly
startDelivery/stopDelivery will core dump in c++ hedwig client
Posted by "Ivan Kelly (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152082#comment-13152082 ]
Ivan Kelly commented on BOOKKEEPER-79:
--------------------------------------
I think we agree on where the problem is. I think the fix will need a bit more though. Currently we seem to allow the messageReceived to be called even if stopSending has been called. stopReceiving should stop receiving, so no messages should come from the channel after it call.
I've attached pubsubtest.cpp which seems to trigger the problem more reliably.
> randomly startDelivery/stopDelivery will core dump in c++ hedwig client
> -----------------------------------------------------------------------
>
> Key: BOOKKEEPER-79
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-79
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-client
> Affects Versions: 4.0.0
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.0.0
>
> Attachments: BOOKKEEPER-79.patch_v2, bookkeeper-79.patch, pubsubtest.cpp
>
>
> in our test program, we tried to startDelivery/stopDelivery different subscriptions randomly. And it core dump.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-79) randomly
startDelivery/stopDelivery will core dump in c++ hedwig client
Posted by "Ivan Kelly (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152804#comment-13152804 ]
Ivan Kelly commented on BOOKKEEPER-79:
--------------------------------------
I think your patch is good. Could you rename last_response to outstanding_response? Also, the full test suite doesn't succeed when I run this on trunk. One problem is BOOKKEEPER-115, but once that is fixed, there are still problems. (running with make check)
The thread issue is something we should look at in general in the future. Im of the opinion that we should only have one one channel per hub. But that requires server side buffering of messages.
{quote}
Beside the number of threads issue, #stopReceiving will be blocked if no data sent from server in the channel, since it only be notified in messageCallbackHandler. async_read will not trigger the bound callback if no data read from the channel.
{quote}
This was where I got stuck yesterday before giving up.
> randomly startDelivery/stopDelivery will core dump in c++ hedwig client
> -----------------------------------------------------------------------
>
> Key: BOOKKEEPER-79
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-79
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-client
> Affects Versions: 4.0.0
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.0.0
>
> Attachments: BOOKKEEPER-79.patch_v2, BOOKKEEPER-79.patch_v3, bookkeeper-79.patch, pubsubtest.cpp
>
>
> in our test program, we tried to startDelivery/stopDelivery different subscriptions randomly. And it core dump.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (BOOKKEEPER-79) randomly
startDelivery/stopDelivery will core dump in c++ hedwig client
Posted by "Sijie Guo (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-79?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sijie Guo reassigned BOOKKEEPER-79:
-----------------------------------
Assignee: Sijie Guo
> randomly startDelivery/stopDelivery will core dump in c++ hedwig client
> -----------------------------------------------------------------------
>
> Key: BOOKKEEPER-79
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-79
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-client
> Affects Versions: 3.4.0
> Reporter: Sijie Guo
> Assignee: Sijie Guo
>
> in our test program, we tried to startDelivery/stopDelivery different subscriptions randomly. And it core dump.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-79) randomly
startDelivery/stopDelivery will core dump in c++ hedwig client
Posted by "Sijie Guo (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-79?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sijie Guo updated BOOKKEEPER-79:
--------------------------------
Attachment: BOOKKEEPER-79.patch_v5
new patch remove warnings in pubsubtest.cpp due to unused variables. also catch exception in IntegerPublisher for testRandomDelivery.
> randomly startDelivery/stopDelivery will core dump in c++ hedwig client
> -----------------------------------------------------------------------
>
> Key: BOOKKEEPER-79
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-79
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-client
> Affects Versions: 4.0.0
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.0.0
>
> Attachments: BOOKKEEPER-79.patch_v2, BOOKKEEPER-79.patch_v3, BOOKKEEPER-79.patch_v4, BOOKKEEPER-79.patch_v5, bookkeeper-79.patch, make-check-errors.txt, pubsubtest.cpp
>
>
> in our test program, we tried to startDelivery/stopDelivery different subscriptions randomly. And it core dump.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-79) randomly
startDelivery/stopDelivery will core dump in c++ hedwig client
Posted by "Sijie Guo (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-79?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sijie Guo updated BOOKKEEPER-79:
--------------------------------
Attachment: BOOKKEEPER-79.patch_v3
> randomly startDelivery/stopDelivery will core dump in c++ hedwig client
> -----------------------------------------------------------------------
>
> Key: BOOKKEEPER-79
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-79
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-client
> Affects Versions: 4.0.0
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.0.0
>
> Attachments: BOOKKEEPER-79.patch_v2, BOOKKEEPER-79.patch_v3, bookkeeper-79.patch, pubsubtest.cpp
>
>
> in our test program, we tried to startDelivery/stopDelivery different subscriptions randomly. And it core dump.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-79) randomly
startDelivery/stopDelivery will core dump in c++ hedwig client
Posted by "Sijie Guo (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13153427#comment-13153427 ]
Sijie Guo commented on BOOKKEEPER-79:
-------------------------------------
I have run 'make clean check' against latest trunk (including BOOKKEEPER-114 and BOOKKEEPER-115) for several times. most is OK, sometimes I got following 2 exceptions.
first one is Exception during socket.close, as commented in previous comments.
second one is SERVER_DOWN exception thrown in publish_thread (running more than ~250 seconds) in #testRandomDelivery, which due to hub server disconnect to zk. I added code in publish thread to catch such exception to avoid it.
{code}
2011-11-19 11:59:46,856 - INFO [Thread-2-SendThread(feelpartfirm-lm.local:2181): ClientCnxn$SendThread@1057] - Client session timed out, have not heard from server in 2908ms for sessionid 0x133b9f6474f0006, closing socket connection and attempting reconnect
2011-11-19 11:59:46,988 - ERROR [Thread-2-EventThread:ZkUtils@75] - Topic: randomDeliveryTopic subscriberId: mysub-randomDelivery could not set subscription state: consumeSeqId: local:21950,zkPath: /hedwig/CppUnitTest/topics/randomDeliveryTopic/subscribers/ mysub-randomDelivery
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hedwig/CppUnitTest/topics/randomDeliveryTopic/subscribers/mysub- randomDelivery
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.hedwig.zookeeper.ZkUtils.logErrorAndCreateZKException(ZkUtils.java:74) at org.apache.hedwig.server.subscriptions.ZkSubscriptionManager$3. safeProcessResult(ZkSubscriptionManager.java:186) at org.apache.hedwig.zookeeper.SafeAsyncZKCallback$StatCallback. processResult(SafeAsyncZKCallback.java:30) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:554)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
2011-11-19 11:59:46,991 - WARN [Thread-2-EventThread:ZkTopicManager$2@94] - ZK client has been disconnected to the ZK server! 2011-11-19 11:59:47,054 - INFO [Thread-2-SendThread(feelpartfirm-lm.local:2181): ClientCnxn$SendThread@933] - Opening socket connection to server /127.0.0.1:21812011-11-19 11:59:47,058 - INFO [Thread-2-SendThread(localhost:2181):ClientCnxn$SendThread@ 846] - Socket connection established to localhost/127.0.0.1:2181, initiating session2011-11-19 11:59:47,069 - INFO [Thread-2-SendThread(localhost:2181):ClientCnxn$SendThread@ 1181] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x133b9f6474f0006, negotiated timeout = 4362
2011-11-19 11:59:47,069 - INFO [Thread-2-EventThread:ZkTopicManager$2@98] - ZK client has been reconnected to the ZK server!
{code}
> randomly startDelivery/stopDelivery will core dump in c++ hedwig client
> -----------------------------------------------------------------------
>
> Key: BOOKKEEPER-79
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-79
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-client
> Affects Versions: 4.0.0
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.0.0
>
> Attachments: BOOKKEEPER-79.patch_v2, BOOKKEEPER-79.patch_v3, BOOKKEEPER-79.patch_v4, bookkeeper-79.patch, make-check-errors.txt, pubsubtest.cpp
>
>
> in our test program, we tried to startDelivery/stopDelivery different subscriptions randomly. And it core dump.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-79) randomly
startDelivery/stopDelivery will core dump in c++ hedwig client
Posted by "Ivan Kelly (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152227#comment-13152227 ]
Ivan Kelly commented on BOOKKEEPER-79:
--------------------------------------
The above code will deadlock on mutexes. There's a number of issues, one being that we try to handle the messages in the async io thread.
I've made an attempt to sort this out, and it just gets ugly. Also, I think we should block stopDelivery from returning until we have stopped all reads, otherwise callbacks may trigger when we don't expect them. Trying this with some locks leads to deadlocks.
https://github.com/ivankelly/bookkeeper/tree/BOOKKEEPER-79
We can discuss this tomorrow morning.
> randomly startDelivery/stopDelivery will core dump in c++ hedwig client
> -----------------------------------------------------------------------
>
> Key: BOOKKEEPER-79
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-79
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-client
> Affects Versions: 4.0.0
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.0.0
>
> Attachments: BOOKKEEPER-79.patch_v2, bookkeeper-79.patch, pubsubtest.cpp
>
>
> in our test program, we tried to startDelivery/stopDelivery different subscriptions randomly. And it core dump.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-79) randomly
startDelivery/stopDelivery will core dump in c++ hedwig client
Posted by "Ivan Kelly (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152019#comment-13152019 ]
Ivan Kelly commented on BOOKKEEPER-79:
--------------------------------------
I'm running the testcase (without the fix) and I'm getting a number of different errors. What error do you get which shows this problem?
Also, I'm not sure the fix is correct. I don't think it's possible for receiving to be false, while stopReceivingBefore is false, because there's a lock,
{code}
boost::lock_guard<boost::mutex> lock(receiving_lock);
{code}
which explicitly stops this situation.
> randomly startDelivery/stopDelivery will core dump in c++ hedwig client
> -----------------------------------------------------------------------
>
> Key: BOOKKEEPER-79
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-79
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-client
> Affects Versions: 4.0.0
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.0.0
>
> Attachments: BOOKKEEPER-79.patch_v2, bookkeeper-79.patch
>
>
> in our test program, we tried to startDelivery/stopDelivery different subscriptions randomly. And it core dump.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-79) randomly
startDelivery/stopDelivery will core dump in c++ hedwig client
Posted by "Ivan Kelly (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152020#comment-13152020 ]
Ivan Kelly commented on BOOKKEEPER-79:
--------------------------------------
Actually, I think the problem is that we always startReceiving with readSize() even though the channel may have read the size, and be in the message reading part. I think this needs to be fixed so that the state of the channel is recorded when you stopReceiving, and when you startReceiving again, you decide what you need to read by looking at the previous state.
> randomly startDelivery/stopDelivery will core dump in c++ hedwig client
> -----------------------------------------------------------------------
>
> Key: BOOKKEEPER-79
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-79
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-client
> Affects Versions: 4.0.0
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.0.0
>
> Attachments: BOOKKEEPER-79.patch_v2, bookkeeper-79.patch
>
>
> in our test program, we tried to startDelivery/stopDelivery different subscriptions randomly. And it core dump.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-79) randomly
startDelivery/stopDelivery will core dump in c++ hedwig client
Posted by "Sijie Guo (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152763#comment-13152763 ]
Sijie Guo commented on BOOKKEEPER-79:
-------------------------------------
Ivan, thanks for your comments. It is great that you tried to avoid callbacks be triggered after stopReceiving in new patch. But a thread is introduced for each channel, if we have lots of subscriptions we will have same number of threads. it is not good.
Beside the number of threads issue, #stopReceiving will be blocked if no data sent from server in the channel, since it only be notified in messageCallbackHandler. async_read will not trigger the bound callback if no data read from the channel.
I attached a new patch based on my previous comments, which can
1) ensure no callbacks after #stopReceiving, also ensure no message lost between stopDelivery and startDelivery (add order checking in MessageHandlerCallback)
2) avoid the deadlock on mutexes.
> randomly startDelivery/stopDelivery will core dump in c++ hedwig client
> -----------------------------------------------------------------------
>
> Key: BOOKKEEPER-79
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-79
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-client
> Affects Versions: 4.0.0
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.0.0
>
> Attachments: BOOKKEEPER-79.patch_v2, BOOKKEEPER-79.patch_v3, bookkeeper-79.patch, pubsubtest.cpp
>
>
> in our test program, we tried to startDelivery/stopDelivery different subscriptions randomly. And it core dump.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-79) randomly
startDelivery/stopDelivery will core dump in c++ hedwig client
Posted by "Sijie Guo (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127336#comment-13127336 ]
Sijie Guo commented on BOOKKEEPER-79:
-------------------------------------
Did investigation on startDelivery/stopDelivery.
I found that stopDelivery just set flag *receiving* as false without doing other things.
Let me explain the startDelivery/stopDelivery flow.
startDelivery:
1) check if *receiving* is true
2) if *receiving* is true, it just return without doing anything. since it has been started.
3) if *receiving* is false, it set *receiving* as true. and put an async io read in boost io queue to read 4bytes message size.
4) if async sizeRead succeed, it will trigger sizeReadCallback and put an async io read in boost io queue to read actual message data.
if we just startDelivery without stopDelivery, the boost io queue will be:
> boost io queue <
| read size |
| read msg |
| read size |
| read msg |
| ... |
but if we first startDelivery (1) -> stopDelivery (1) -> startDelivery (2),
the second delivery will add another read size io in the queue, then it will be:
> boost io queue <
| read size |
| read msg |
| ... |
| *read size* |
| *read size* |
| read msg |
so the data stream is corrupted which will cause memory corrupted. it core dump.
> randomly startDelivery/stopDelivery will core dump in c++ hedwig client
> -----------------------------------------------------------------------
>
> Key: BOOKKEEPER-79
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-79
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-client
> Affects Versions: 3.4.0
> Reporter: Sijie Guo
>
> in our test program, we tried to startDelivery/stopDelivery different subscriptions randomly. And it core dump.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-79) randomly
startDelivery/stopDelivery will core dump in c++ hedwig client
Posted by "Sijie Guo (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13131271#comment-13131271 ]
Sijie Guo commented on BOOKKEEPER-79:
-------------------------------------
Ivan, Let me add the test program.
> randomly startDelivery/stopDelivery will core dump in c++ hedwig client
> -----------------------------------------------------------------------
>
> Key: BOOKKEEPER-79
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-79
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-client
> Affects Versions: 3.4.0
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Attachments: bookkeeper-79.patch
>
>
> in our test program, we tried to startDelivery/stopDelivery different subscriptions randomly. And it core dump.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-79) randomly
startDelivery/stopDelivery will core dump in c++ hedwig client
Posted by "Sijie Guo (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152068#comment-13152068 ]
Sijie Guo commented on BOOKKEEPER-79:
-------------------------------------
> What error do you get which shows this problem?
It core dump running the test case.
> there's a lock, which explicitly stops this situation.
The lock is for changing received value. stopReceiving can set receiving to false since it can obtain the lock.
> I think this needs to be fixed so that the state of the channel is recorded when you stopReceiving
We don't need to record the reading state of the channel, since the receiving check is in readSize which ensure it stop exactly in a frame of message.
{code}
/*static*/ void DuplexChannel::readSize(DuplexChannelPtr channel) {
if (!channel->isReceiving()) {
return;
}
int toread = sizeof(uint32_t) - channel->in_buf.size();
LOG4CXX_DEBUG(logger, " size of incoming message " << sizeof(uint32_t)
<< ", currently in buffer " << channel->in_buf.size()
<< " channel(" << channel.get() << ")");
if (toread < 0) {
DuplexChannel::sizeReadCallbackHandler(channel, boost::system::error_code(), 0);
} else {
// in_buf_size.prepare(sizeof(uint32_t));
boost::asio::async_read(channel->socket, channel->in_buf,
boost::asio::transfer_at_least(sizeof(uint32_t)),
boost::bind(&DuplexChannel::sizeReadCallbackHandler,
channel,
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred));
}
}
{code}
The race here is between #startReceiving, #stopReceiving, and #readSize, as below
1. #readSize passed isReceiving() checking and before async_read is called.
2. main thread do #startReceiving #stopReceiving. it insert a new async_read op to read size.
3. #readSize calls async_read to insert its async_read op to read size.
But you comment remind me that the patch only fix the above case. In following case, it failed:
1. main thread #stopReceiving first
2. #readSize check receiving, return. no more async_read op are inserted.
3. main thread #startReceiving, it doesn't insert any async_read op, since stopReceivingBefore is true.
I think the root cause of race here is between checking receiving flag and changing receiving flag. we have lock in #startReceiving and #stopReceiving, but we have no lock in #readSize which checks receiving flag.
The right fix may be:
need a flag which indicates whether there is a async_read size op in io service. we only do readSize if there is no async_read size op in io service when #startReceiving. said the flag is reading, its lock is reading_lock. the code is showed as below.
{code}
void DuplexChannel::startReceiving() {
LOG4CXX_DEBUG(logger, "DuplexChannel::startReceiving channel(" << this << ") currently receiving = " << receiving);
boost::lock_guard<boost::mutex> lock(receiving_lock);
if (receiving) {
return;
}
receiving = true;
{
boost::lock_guard<boost::mutex> lock(reading_lock);
if (!reading) {
reading = true;
DuplexChannel::doReadSize(shared_from_this());
}
}
}
// no change in #stopReceiving
void DuplexChannel::stopReceiving() {
LOG4CXX_DEBUG(logger, "DuplexChannel::stopReceiving channel(" << this << ")");
boost::lock_guard<boost::mutex> lock(receiving_lock);
receiving = false;
stopReceivingBefore = true;
}
/*static*/ void DuplexChannel::readSize(DuplexChannelPtr channel) {
boost::lock_guard<boost::mutex> lock(receiving_lock);
if (!channel->isReceiving()) {
boost::lock_guard<boost::mutex> lock(reading_lock);
reading = false;
return;
}
doReadSize(channel);
}
void doReadSize(DuplexChannelPtr channel) {
int toread = sizeof(uint32_t) - channel->in_buf.size();
LOG4CXX_DEBUG(logger, " size of incoming message " << sizeof(uint32_t)
<< ", currently in buffer " << channel->in_buf.size()
<< " channel(" << channel.get() << ")");
if (toread < 0) {
DuplexChannel::sizeReadCallbackHandler(channel, boost::system::error_code(), 0);
} else {
// in_buf_size.prepare(sizeof(uint32_t));
boost::asio::async_read(channel->socket, channel->in_buf,
boost::asio::transfer_at_least(sizeof(uint32_t)),
boost::bind(&DuplexChannel::sizeReadCallbackHandler,
channel,
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred));
}
}
{code}
> randomly startDelivery/stopDelivery will core dump in c++ hedwig client
> -----------------------------------------------------------------------
>
> Key: BOOKKEEPER-79
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-79
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-client
> Affects Versions: 4.0.0
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.0.0
>
> Attachments: BOOKKEEPER-79.patch_v2, bookkeeper-79.patch
>
>
> in our test program, we tried to startDelivery/stopDelivery different subscriptions randomly. And it core dump.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira