You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Vinod Kone <vi...@gmail.com> on 2012/11/02 23:26:35 UTC
Review Request: More ZooKeeper Master Detector tests
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7842/
-----------------------------------------------------------
Review request for mesos, Benjamin Hindman and Ben Mahler.
Description
-------
Its quite hard to test true network partition behavior, so these tests expire zk sessions as a close approximation.
Diffs
-----
src/detector/detector.cpp 57febbb22e2abf193f7881839b31db69d49b14df
src/tests/zookeeper_tests.cpp fff53b84356243663cfc75d66ddb3a1a7faf8043
Diff: https://reviews.apache.org/r/7842/diff/
Testing
-------
make check.
When running in a repeat loop, I found the following JVM error. It looks un-related to current tests
[vinod@smfd-aki-27-sr1:~/mesos/build] $ GLOG_v=1 ./bin/mesos-tests.sh --gtest_filter="*Expire*" --verbose --gtest_repeat=-1 --gtest_break_on_failure
....
....
I1102 22:21:46.203616 28200 detector.cpp:331] Created ephemeral/sequence znode at '/mesos/0000000003'
I1102 22:21:46.203902 28198 detector.cpp:486] Master detector ((290)@10.35.12.124:57104) found 2 registered masters
I1102 22:21:46.204463 28200 detector.cpp:486] Master detector ((287)@10.35.12.124:57104) found 2 registered masters
I1102 22:21:46.205245 28200 detector.cpp:532] Master detector ((287)@10.35.12.124:57104) got new master pid: (290)@10.35.12.124:57104
../../src/tests/zookeeper_tests.cpp:441: Failure
Mock function called more times than expected - returning directly.
Function call: newMasterDetected(@0x451ffe90 (290)@10.35.12.124:57104)
Expected: to be called once
Actual: called twice - over-saturated and active
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x000000000075a304, pid=28178, tid=1159727424
#
# JRE version: 7.0_10-b10
# Java VM: Java HotSpot(TM) 64-Bit Server VM (23.6-b04 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [lt-mesos-tests+0x35a304] cgroups::removeCgroup(std::string const&, std::string const&)+0x35a304
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/vinod/mesos/build/hs_err_pid28178.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.sun.com/bugreport/crash.jsp
#
Aborted
Thanks,
Vinod Kone
Re: Review Request: More ZooKeeper Master Detector tests
Posted by Benjamin Hindman <be...@berkeley.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7842/#review14105
-----------------------------------------------------------
Ship it!
Ship It!
- Benjamin Hindman
On Nov. 2, 2012, 10:29 p.m., Vinod Kone wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7842/
> -----------------------------------------------------------
>
> (Updated Nov. 2, 2012, 10:29 p.m.)
>
>
> Review request for mesos, Benjamin Hindman and Ben Mahler.
>
>
> Description
> -------
>
> Its quite hard to test true network partition behavior, so these tests expire zk sessions as a close approximation.
>
>
> Diffs
> -----
>
> src/detector/detector.cpp 57febbb22e2abf193f7881839b31db69d49b14df
> src/tests/zookeeper_tests.cpp fff53b84356243663cfc75d66ddb3a1a7faf8043
>
> Diff: https://reviews.apache.org/r/7842/diff/
>
>
> Testing
> -------
>
> make check.
>
> When running in a repeat loop, I found the following JVM error (in iteration 12). It looks un-related to current tests
>
> [vinod@smfd-aki-27-sr1:~/mesos/build] $ GLOG_v=1 ./bin/mesos-tests.sh --gtest_filter="*Expire*" --verbose --gtest_repeat=-1 --gtest_break_on_failure
> ....
> ....
> I1102 22:21:46.203616 28200 detector.cpp:331] Created ephemeral/sequence znode at '/mesos/0000000003'
> I1102 22:21:46.203902 28198 detector.cpp:486] Master detector ((290)@10.35.12.124:57104) found 2 registered masters
> I1102 22:21:46.204463 28200 detector.cpp:486] Master detector ((287)@10.35.12.124:57104) found 2 registered masters
> I1102 22:21:46.205245 28200 detector.cpp:532] Master detector ((287)@10.35.12.124:57104) got new master pid: (290)@10.35.12.124:57104
> ../../src/tests/zookeeper_tests.cpp:441: Failure
> Mock function called more times than expected - returning directly.
> Function call: newMasterDetected(@0x451ffe90 (290)@10.35.12.124:57104)
> Expected: to be called once
> Actual: called twice - over-saturated and active
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x000000000075a304, pid=28178, tid=1159727424
> #
> # JRE version: 7.0_10-b10
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (23.6-b04 mixed mode linux-amd64 compressed oops)
> # Problematic frame:
> # C [lt-mesos-tests+0x35a304] cgroups::removeCgroup(std::string const&, std::string const&)+0x35a304
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /home/vinod/mesos/build/hs_err_pid28178.log
> #
> # If you would like to submit a bug report, please visit:
> # http://bugreport.sun.com/bugreport/crash.jsp
> #
> Aborted
>
>
> Thanks,
>
> Vinod Kone
>
>
Re: Review Request: More ZooKeeper Master Detector tests
Posted by Vinod Kone <vi...@gmail.com>.
> On Nov. 15, 2012, 4:21 p.m., Benjamin Hindman wrote:
> > Have the segfault issues been resolved?
>
> Vinod Kone wrote:
> not yet.
Wasn't able to repro the above again :/
Ran all master detector tests in a loop (~200 times) and the new expiration tests (~50 times) with NO segfaults!
I'm not sure what was happening before to cause the SEGFAULTs and what changed now.
Feel free to test this patch (in a loop) on your own before committing to the trunk.
- Vinod
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7842/#review13469
-----------------------------------------------------------
On Nov. 2, 2012, 10:29 p.m., Vinod Kone wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7842/
> -----------------------------------------------------------
>
> (Updated Nov. 2, 2012, 10:29 p.m.)
>
>
> Review request for mesos, Benjamin Hindman and Ben Mahler.
>
>
> Description
> -------
>
> Its quite hard to test true network partition behavior, so these tests expire zk sessions as a close approximation.
>
>
> Diffs
> -----
>
> src/detector/detector.cpp 57febbb22e2abf193f7881839b31db69d49b14df
> src/tests/zookeeper_tests.cpp fff53b84356243663cfc75d66ddb3a1a7faf8043
>
> Diff: https://reviews.apache.org/r/7842/diff/
>
>
> Testing
> -------
>
> make check.
>
> When running in a repeat loop, I found the following JVM error (in iteration 12). It looks un-related to current tests
>
> [vinod@smfd-aki-27-sr1:~/mesos/build] $ GLOG_v=1 ./bin/mesos-tests.sh --gtest_filter="*Expire*" --verbose --gtest_repeat=-1 --gtest_break_on_failure
> ....
> ....
> I1102 22:21:46.203616 28200 detector.cpp:331] Created ephemeral/sequence znode at '/mesos/0000000003'
> I1102 22:21:46.203902 28198 detector.cpp:486] Master detector ((290)@10.35.12.124:57104) found 2 registered masters
> I1102 22:21:46.204463 28200 detector.cpp:486] Master detector ((287)@10.35.12.124:57104) found 2 registered masters
> I1102 22:21:46.205245 28200 detector.cpp:532] Master detector ((287)@10.35.12.124:57104) got new master pid: (290)@10.35.12.124:57104
> ../../src/tests/zookeeper_tests.cpp:441: Failure
> Mock function called more times than expected - returning directly.
> Function call: newMasterDetected(@0x451ffe90 (290)@10.35.12.124:57104)
> Expected: to be called once
> Actual: called twice - over-saturated and active
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x000000000075a304, pid=28178, tid=1159727424
> #
> # JRE version: 7.0_10-b10
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (23.6-b04 mixed mode linux-amd64 compressed oops)
> # Problematic frame:
> # C [lt-mesos-tests+0x35a304] cgroups::removeCgroup(std::string const&, std::string const&)+0x35a304
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /home/vinod/mesos/build/hs_err_pid28178.log
> #
> # If you would like to submit a bug report, please visit:
> # http://bugreport.sun.com/bugreport/crash.jsp
> #
> Aborted
>
>
> Thanks,
>
> Vinod Kone
>
>
Re: Review Request: More ZooKeeper Master Detector tests
Posted by Vinod Kone <vi...@gmail.com>.
> On Nov. 15, 2012, 4:21 p.m., Benjamin Hindman wrote:
> > Have the segfault issues been resolved?
not yet.
- Vinod
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7842/#review13469
-----------------------------------------------------------
On Nov. 2, 2012, 10:29 p.m., Vinod Kone wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7842/
> -----------------------------------------------------------
>
> (Updated Nov. 2, 2012, 10:29 p.m.)
>
>
> Review request for mesos, Benjamin Hindman and Ben Mahler.
>
>
> Description
> -------
>
> Its quite hard to test true network partition behavior, so these tests expire zk sessions as a close approximation.
>
>
> Diffs
> -----
>
> src/detector/detector.cpp 57febbb22e2abf193f7881839b31db69d49b14df
> src/tests/zookeeper_tests.cpp fff53b84356243663cfc75d66ddb3a1a7faf8043
>
> Diff: https://reviews.apache.org/r/7842/diff/
>
>
> Testing
> -------
>
> make check.
>
> When running in a repeat loop, I found the following JVM error (in iteration 12). It looks un-related to current tests
>
> [vinod@smfd-aki-27-sr1:~/mesos/build] $ GLOG_v=1 ./bin/mesos-tests.sh --gtest_filter="*Expire*" --verbose --gtest_repeat=-1 --gtest_break_on_failure
> ....
> ....
> I1102 22:21:46.203616 28200 detector.cpp:331] Created ephemeral/sequence znode at '/mesos/0000000003'
> I1102 22:21:46.203902 28198 detector.cpp:486] Master detector ((290)@10.35.12.124:57104) found 2 registered masters
> I1102 22:21:46.204463 28200 detector.cpp:486] Master detector ((287)@10.35.12.124:57104) found 2 registered masters
> I1102 22:21:46.205245 28200 detector.cpp:532] Master detector ((287)@10.35.12.124:57104) got new master pid: (290)@10.35.12.124:57104
> ../../src/tests/zookeeper_tests.cpp:441: Failure
> Mock function called more times than expected - returning directly.
> Function call: newMasterDetected(@0x451ffe90 (290)@10.35.12.124:57104)
> Expected: to be called once
> Actual: called twice - over-saturated and active
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x000000000075a304, pid=28178, tid=1159727424
> #
> # JRE version: 7.0_10-b10
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (23.6-b04 mixed mode linux-amd64 compressed oops)
> # Problematic frame:
> # C [lt-mesos-tests+0x35a304] cgroups::removeCgroup(std::string const&, std::string const&)+0x35a304
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /home/vinod/mesos/build/hs_err_pid28178.log
> #
> # If you would like to submit a bug report, please visit:
> # http://bugreport.sun.com/bugreport/crash.jsp
> #
> Aborted
>
>
> Thanks,
>
> Vinod Kone
>
>
Re: Review Request: More ZooKeeper Master Detector tests
Posted by Benjamin Hindman <be...@berkeley.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7842/#review13469
-----------------------------------------------------------
Have the segfault issues been resolved?
- Benjamin Hindman
On Nov. 2, 2012, 10:29 p.m., Vinod Kone wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7842/
> -----------------------------------------------------------
>
> (Updated Nov. 2, 2012, 10:29 p.m.)
>
>
> Review request for mesos, Benjamin Hindman and Ben Mahler.
>
>
> Description
> -------
>
> Its quite hard to test true network partition behavior, so these tests expire zk sessions as a close approximation.
>
>
> Diffs
> -----
>
> src/detector/detector.cpp 57febbb22e2abf193f7881839b31db69d49b14df
> src/tests/zookeeper_tests.cpp fff53b84356243663cfc75d66ddb3a1a7faf8043
>
> Diff: https://reviews.apache.org/r/7842/diff/
>
>
> Testing
> -------
>
> make check.
>
> When running in a repeat loop, I found the following JVM error (in iteration 12). It looks un-related to current tests
>
> [vinod@smfd-aki-27-sr1:~/mesos/build] $ GLOG_v=1 ./bin/mesos-tests.sh --gtest_filter="*Expire*" --verbose --gtest_repeat=-1 --gtest_break_on_failure
> ....
> ....
> I1102 22:21:46.203616 28200 detector.cpp:331] Created ephemeral/sequence znode at '/mesos/0000000003'
> I1102 22:21:46.203902 28198 detector.cpp:486] Master detector ((290)@10.35.12.124:57104) found 2 registered masters
> I1102 22:21:46.204463 28200 detector.cpp:486] Master detector ((287)@10.35.12.124:57104) found 2 registered masters
> I1102 22:21:46.205245 28200 detector.cpp:532] Master detector ((287)@10.35.12.124:57104) got new master pid: (290)@10.35.12.124:57104
> ../../src/tests/zookeeper_tests.cpp:441: Failure
> Mock function called more times than expected - returning directly.
> Function call: newMasterDetected(@0x451ffe90 (290)@10.35.12.124:57104)
> Expected: to be called once
> Actual: called twice - over-saturated and active
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x000000000075a304, pid=28178, tid=1159727424
> #
> # JRE version: 7.0_10-b10
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (23.6-b04 mixed mode linux-amd64 compressed oops)
> # Problematic frame:
> # C [lt-mesos-tests+0x35a304] cgroups::removeCgroup(std::string const&, std::string const&)+0x35a304
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /home/vinod/mesos/build/hs_err_pid28178.log
> #
> # If you would like to submit a bug report, please visit:
> # http://bugreport.sun.com/bugreport/crash.jsp
> #
> Aborted
>
>
> Thanks,
>
> Vinod Kone
>
>
Re: Review Request: More ZooKeeper Master Detector tests
Posted by Vinod Kone <vi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7842/
-----------------------------------------------------------
(Updated Nov. 2, 2012, 10:29 p.m.)
Review request for mesos, Benjamin Hindman and Ben Mahler.
Changes
-------
the SEGFAULT happens after a few iterations (and is repeatable).
Description
-------
Its quite hard to test true network partition behavior, so these tests expire zk sessions as a close approximation.
Diffs
-----
src/detector/detector.cpp 57febbb22e2abf193f7881839b31db69d49b14df
src/tests/zookeeper_tests.cpp fff53b84356243663cfc75d66ddb3a1a7faf8043
Diff: https://reviews.apache.org/r/7842/diff/
Testing (updated)
-------
make check.
When running in a repeat loop, I found the following JVM error (in iteration 12). It looks un-related to current tests
[vinod@smfd-aki-27-sr1:~/mesos/build] $ GLOG_v=1 ./bin/mesos-tests.sh --gtest_filter="*Expire*" --verbose --gtest_repeat=-1 --gtest_break_on_failure
....
....
I1102 22:21:46.203616 28200 detector.cpp:331] Created ephemeral/sequence znode at '/mesos/0000000003'
I1102 22:21:46.203902 28198 detector.cpp:486] Master detector ((290)@10.35.12.124:57104) found 2 registered masters
I1102 22:21:46.204463 28200 detector.cpp:486] Master detector ((287)@10.35.12.124:57104) found 2 registered masters
I1102 22:21:46.205245 28200 detector.cpp:532] Master detector ((287)@10.35.12.124:57104) got new master pid: (290)@10.35.12.124:57104
../../src/tests/zookeeper_tests.cpp:441: Failure
Mock function called more times than expected - returning directly.
Function call: newMasterDetected(@0x451ffe90 (290)@10.35.12.124:57104)
Expected: to be called once
Actual: called twice - over-saturated and active
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x000000000075a304, pid=28178, tid=1159727424
#
# JRE version: 7.0_10-b10
# Java VM: Java HotSpot(TM) 64-Bit Server VM (23.6-b04 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [lt-mesos-tests+0x35a304] cgroups::removeCgroup(std::string const&, std::string const&)+0x35a304
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/vinod/mesos/build/hs_err_pid28178.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.sun.com/bugreport/crash.jsp
#
Aborted
Thanks,
Vinod Kone