You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Vinod Kone (JIRA)" <ji...@apache.org> on 2019/02/06 14:15:00 UTC
[jira] [Commented] (MESOS-8796) Some GroupTest.* are flaky on Mac.
[ https://issues.apache.org/jira/browse/MESOS-8796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16761769#comment-16761769 ]
Vinod Kone commented on MESOS-8796:
-----------------------------------
Saw this again on internal CI (on Mac).
{code}
[ RUN ] GroupTest.GroupPathWithRestrictivePerms
I0205 21:14:33.530055 296834496 zookeeper_test_server.cpp:156] Started ZooKeeperTestServer on port 50946
2019-02-05 21:14:33,530:8369(0x7000036ae000):ZOO_INFO@log_env@753: Client environment:zookeeper.version=zookeeper C client 3.4.8
2019-02-05 21:14:33,530:8369(0x7000036ae000):ZOO_INFO@log_env@757: Client environment:host.name=Jenkinss-Mac-mini.local
2019-02-05 21:14:33,530:8369(0x7000036ae000):ZOO_INFO@log_env@764: Client environment:os.name=Darwin
2019-02-05 21:14:33,530:8369(0x7000036ae000):ZOO_INFO@log_env@765: Client environment:os.arch=18.2.0
2019-02-05 21:14:33,530:8369(0x7000036ae000):ZOO_INFO@log_env@766: Client environment:os.version=Darwin Kernel Version 18.2.0: Mon Nov 12 20:24:46 PST 2018; root:xnu-4903.231.4~2/
RELEASE_X86_64
2019-02-05 21:14:33,530:8369(0x7000036ae000):ZOO_INFO@log_env@774: Client environment:user.name=jenkins
2019-02-05 21:14:33,530:8369(0x7000036ae000):ZOO_INFO@log_env@782: Client environment:user.home=/Users/jenkins
2019-02-05 21:14:33,530:8369(0x7000036ae000):ZOO_INFO@log_env@794: Client environment:user.dir=/Users/jenkins/workspace/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mac/mesos/bui
ld
2019-02-05 21:14:33,530:8369(0x7000036ae000):ZOO_INFO@zookeeper_init@827: Initiating client connection, host=127.0.0.1:50946 sessionTimeout=10000 watcher=0x1145565d0 sessionId=0 s
essionPasswd=<null> context=0x7fb3e0c9bc90 flags=0
2019-02-05 21:14:33,530:8369(0x700003fcf000):ZOO_INFO@check_events@1764: initiated connection to server [127.0.0.1:50946]
2019-02-05 21:14:33,532:8369(0x700003fcf000):ZOO_INFO@check_events@1811: session establishment complete on server [127.0.0.1:50946], sessionId=0x168c13aa8b90000, negotiated timeou
t=10000
2019-02-05 21:14:36,875:8369(0x700003fcf000):ZOO_INFO@auth_completion_func@1327: Authentication scheme digest succeeded
2019-02-05 21:14:36,878:8369(0x70000341f000):ZOO_INFO@log_env@753: Client environment:zookeeper.version=zookeeper C client 3.4.8
2019-02-05 21:14:36,878:8369(0x70000341f000):ZOO_INFO@log_env@757: Client environment:host.name=Jenkinss-Mac-mini.local
2019-02-05 21:14:36,878:8369(0x70000341f000):ZOO_INFO@log_env@764: Client environment:os.name=Darwin
2019-02-05 21:14:36,878:8369(0x70000341f000):ZOO_INFO@log_env@765: Client environment:os.arch=18.2.0
2019-02-05 21:14:36,878:8369(0x70000341f000):ZOO_INFO@log_env@766: Client environment:os.version=Darwin Kernel Version 18.2.0: Mon Nov 12 20:24:46 PST 2018; root:xnu-4903.231.4~2/
RELEASE_X86_64
2019-02-05 21:14:36,878:8369(0x70000341f000):ZOO_INFO@log_env@774: Client environment:user.name=jenkins
2019-02-05 21:14:36,878:8369(0x70000341f000):ZOO_INFO@log_env@782: Client environment:user.home=/Users/jenkins
2019-02-05 21:14:36,878:8369(0x70000341f000):ZOO_INFO@log_env@794: Client environment:user.dir=/Users/jenkins/workspace/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mac/mesos/bui
ld
2019-02-05 21:14:36,878:8369(0x70000341f000):ZOO_INFO@zookeeper_init@827: Initiating client connection, host=127.0.0.1:50946 sessionTimeout=10000 watcher=0x1145565d0 sessionId=0 s
essionPasswd=<null> context=0x7fb3e0a4db10 flags=0
2019-02-05 21:14:36,879:8369(0x700004767000):ZOO_INFO@check_events@1764: initiated connection to server [127.0.0.1:50946]
2019-02-05 21:14:36,880:8369(0x700004767000):ZOO_INFO@check_events@1811: session establishment complete on server [127.0.0.1:50946], sessionId=0x168c13aa8b90001, negotiated timeou
t=10000
I0205 21:14:36.880167 55189504 group.cpp:341] Group process (zookeeper-group(48)@10.0.49.4:65013) connected to ZooKeeper
I0205 21:14:36.880213 55189504 group.cpp:831] Syncing group operations: queue size (joins, cancels, datas) = (1, 0, 0)
I0205 21:14:36.880225 55189504 group.cpp:395] Authenticating with ZooKeeper using digest
2019-02-05 21:14:40,222:8369(0x700004767000):ZOO_INFO@auth_completion_func@1327: Authentication scheme digest succeeded
I0205 21:14:40.222224 55189504 group.cpp:419] Trying to create path '/read-only' in ZooKeeper
2019-02-05 21:14:40,223:8369(0x7000036ae000):ZOO_INFO@log_env@753: Client environment:zookeeper.version=zookeeper C client 3.4.8
2019-02-05 21:14:40,224:8369(0x7000036ae000):ZOO_INFO@log_env@757: Client environment:host.name=Jenkinss-Mac-mini.local
2019-02-05 21:14:40,224:8369(0x7000036ae000):ZOO_INFO@log_env@764: Client environment:os.name=Darwin
2019-02-05 21:14:40,224:8369(0x7000036ae000):ZOO_INFO@log_env@765: Client environment:os.arch=18.2.0
2019-02-05 21:14:40,224:8369(0x7000036ae000):ZOO_INFO@log_env@766: Client environment:os.version=Darwin Kernel Version 18.2.0: Mon Nov 12 20:24:46 PST 2018; root:xnu-4903.231.4~2/
RELEASE_X86_64
2019-02-05 21:14:40,224:8369(0x7000036ae000):ZOO_INFO@log_env@774: Client environment:user.name=jenkins
2019-02-05 21:14:40,224:8369(0x7000036ae000):ZOO_INFO@log_env@782: Client environment:user.home=/Users/jenkins
2019-02-05 21:14:40,224:8369(0x7000036ae000):ZOO_INFO@log_env@794: Client environment:user.dir=/Users/jenkins/workspace/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mac/mesos/bui
ld
2019-02-05 21:14:40,224:8369(0x7000036ae000):ZOO_INFO@zookeeper_init@827: Initiating client connection, host=127.0.0.1:50946 sessionTimeout=10000 watcher=0x1145565d0 sessionId=0 s
essionPasswd=<null> context=0x7fb3e03c00c0 flags=0
I0205 21:14:40.224323 55189504 group.cpp:758] Found non-sequence node 'writable' at '/read-only' in ZooKeeper
2019-02-05 21:14:40,224:8369(0x70000486d000):ZOO_INFO@check_events@1764: initiated connection to server [127.0.0.1:50946]
2019-02-05 21:14:40,225:8369(0x70000486d000):ZOO_INFO@check_events@1811: session establishment complete on server [127.0.0.1:50946], sessionId=0x168c13aa8b90002, negotiated timeou
t=10000
I0205 21:14:40.225809 57335808 group.cpp:341] Group process (zookeeper-group(49)@10.0.49.4:65013) connected to ZooKeeper
I0205 21:14:40.225847 57335808 group.cpp:831] Syncing group operations: queue size (joins, cancels, datas) = (1, 0, 0)
I0205 21:14:40.225859 57335808 group.cpp:395] Authenticating with ZooKeeper using digest
2019-02-05 21:14:49,320:8369(0x700004767000):ZOO_WARN@zookeeper_interest@1597: Exceeded deadline by 5764ms
2019-02-05 21:14:49,320:8369(0x700003fcf000):ZOO_WARN@zookeeper_interest@1597: Exceeded deadline by 5766ms
2019-02-05 21:14:49,320:8369(0x700004767000):ZOO_ERROR@handle_socket_error_msg@1702: Socket [127.0.0.1:50946] zk retcode=-7, errno=60(Operation timed out): connection to 127.0.0.1:50946 timed out (exceeded timeout by 2430ms)
2019-02-05 21:14:49,321:8369(0x700003fcf000):ZOO_ERROR@handle_socket_error_msg@1702: Socket [127.0.0.1:50946] zk retcode=-7, errno=60(Operation timed out): connection to 127.0.0.1:50946 timed out (exceeded timeout by 2432ms)
2019-02-05 21:14:49,321:8369(0x700004767000):ZOO_WARN@zookeeper_interest@1597: Exceeded deadline by 5764ms
2019-02-05 21:14:49,321:8369(0x700003fcf000):ZOO_WARN@zookeeper_interest@1597: Exceeded deadline by 5766ms
I0205 21:14:49.321324 54652928 group.cpp:452] Lost connection to ZooKeeper, attempting to reconnect ...
2019-02-05 21:14:49,330:8369(0x70000486d000):ZOO_WARN@zookeeper_interest@1597: Exceeded deadline by 5773ms
2019-02-05 21:14:49,330:8369(0x70000486d000):ZOO_ERROR@handle_socket_error_msg@1702: Socket [127.0.0.1:50946] zk retcode=-7, errno=60(Operation timed out): connection to 127.0.0.1:50946 timed out (exceeded timeout by 2438ms)
2019-02-05 21:14:49,330:8369(0x70000486d000):ZOO_WARN@zookeeper_interest@1597: Exceeded deadline by 5773ms
I0205 21:14:49.330642 57335808 group.cpp:452] Lost connection to ZooKeeper, attempting to reconnect ...
2019-02-05 21:14:52,664:8369(0x700003fcf000):ZOO_WARN@zookeeper_interest@1597: Exceeded deadline by 9109ms
2019-02-05 21:14:52,664:8369(0x700004767000):ZOO_WARN@zookeeper_interest@1597: Exceeded deadline by 9107ms
2019-02-05 21:14:52,664:8369(0x70000486d000):ZOO_WARN@zookeeper_interest@1597: Exceeded deadline by 9106ms
2019-02-05 21:14:52,664:8369(0x700004767000):ZOO_INFO@check_events@1764: initiated connection to server [127.0.0.1:50946]
2019-02-05 21:14:52,664:8369(0x700003fcf000):ZOO_INFO@check_events@1764: initiated connection to server [127.0.0.1:50946]
2019-02-05 21:14:52,664:8369(0x70000486d000):ZOO_INFO@check_events@1764: initiated connection to server [127.0.0.1:50946]
2019-02-05 21:14:52,665:8369(0x700003fcf000):ZOO_ERROR@handle_socket_error_msg@1800: Socket [127.0.0.1:50946] zk retcode=-112, errno=70(Stale NFS file handle): sessionId=0x168c13aa8b90000 has expired.
2019-02-05 21:14:52,665:8369(0x700004767000):ZOO_ERROR@handle_socket_error_msg@1800: Socket [127.0.0.1:50946] zk retcode=-112, errno=70(Stale NFS file handle): sessionId=0x168c13aa8b90001 has expired.
I0205 21:14:52.665789 56799232 group.cpp:511] ZooKeeper session expired
2019-02-05 21:14:52,665:8369(0x70000362b000):ZOO_INFO@zookeeper_close@2579: Freeing zookeeper resources for sessionId=0x168c13aa8b90001
2019-02-05 21:14:52,665:8369(0x70000486d000):ZOO_ERROR@handle_socket_error_msg@1800: Socket [127.0.0.1:50946] zk retcode=-112, errno=70(Stale NFS file handle): sessionId=0x168c13aa8b90002 has expired.
2019-02-05 21:14:52,666:8369(0x700003525000):ZOO_INFO@log_env@753: Client environment:zookeeper.version=zookeeper C client 3.4.8
I0205 21:14:52.666157 55189504 group.cpp:511] ZooKeeper session expired
2019-02-05 21:14:52,666:8369(0x700003525000):ZOO_INFO@log_env@757: Client environment:host.name=Jenkinss-Mac-mini.local
2019-02-05 21:14:52,666:8369(0x700003525000):ZOO_INFO@log_env@764: Client environment:os.name=Darwin
2019-02-05 21:14:52,666:8369(0x700003525000):ZOO_INFO@log_env@765: Client environment:os.arch=18.2.0
2019-02-05 21:14:52,666:8369(0x700003525000):ZOO_INFO@log_env@766: Client environment:os.version=Darwin Kernel Version 18.2.0: Mon Nov 12 20:24:46 PST 2018; root:xnu-4903.231.4~2/RELEASE_X86_64
2019-02-05 21:14:52,666:8369(0x700003525000):ZOO_INFO@log_env@774: Client environment:user.name=jenkins
2019-02-05 21:14:52,666:8369(0x700003525000):ZOO_INFO@log_env@782: Client environment:user.home=/Users/jenkins
2019-02-05 21:14:52,666:8369(0x7000034a2000):ZOO_INFO@zookeeper_close@2579: Freeing zookeeper resources for sessionId=0x168c13aa8b90002
2019-02-05 21:14:52,666:8369(0x700003525000):ZOO_INFO@log_env@794: Client environment:user.dir=/Users/jenkins/workspace/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mac/mesos/build
2019-02-05 21:14:52,666:8369(0x700003525000):ZOO_INFO@zookeeper_init@827: Initiating client connection, host=127.0.0.1:50946 sessionTimeout=10000 watcher=0x1145565d0 sessionId=0 sessionPasswd=<null> context=0x7fb3e0cad090 flags=0
2019-02-05 21:14:52,666:8369(0x7000035a8000):ZOO_INFO@log_env@753: Client environment:zookeeper.version=zookeeper C client 3.4.8
2019-02-05 21:14:52,666:8369(0x7000035a8000):ZOO_INFO@log_env@757: Client environment:host.name=Jenkinss-Mac-mini.local
2019-02-05 21:14:52,666:8369(0x7000035a8000):ZOO_INFO@log_env@764: Client environment:os.name=Darwin
2019-02-05 21:14:52,666:8369(0x7000035a8000):ZOO_INFO@log_env@765: Client environment:os.arch=18.2.0
2019-02-05 21:14:52,666:8369(0x7000035a8000):ZOO_INFO@log_env@766: Client environment:os.version=Darwin Kernel Version 18.2.0: Mon Nov 12 20:24:46 PST 2018; root:xnu-4903.231.4~2/:
RELEASE_X86_64
2019-02-05 21:14:52,666:8369(0x7000035a8000):ZOO_INFO@log_env@774: Client environment:user.name=jenkins
2019-02-05 21:14:52,666:8369(0x7000035a8000):ZOO_INFO@log_env@782: Client environment:user.home=/Users/jenkins
2019-02-05 21:14:52,666:8369(0x7000035a8000):ZOO_INFO@log_env@794: Client environment:user.dir=/Users/jenkins/workspace/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mac/mesos/build
2019-02-05 21:14:52,666:8369(0x7000035a8000):ZOO_INFO@zookeeper_init@827: Initiating client connection, host=127.0.0.1:50946 sessionTimeout=10000 watcher=0x1145565d0 sessionId=0 sessionPasswd=<null> context=0x7fb3e03c00c0 flags=0
2019-02-05 21:14:52,666:8369(0x700004767000):ZOO_INFO@check_events@1764: initiated connection to server [127.0.0.1:50946]
2019-02-05 21:14:52,666:8369(0x70000486d000):ZOO_INFO@check_events@1764: initiated connection to server [127.0.0.1:50946]
2019-02-05 21:14:52,667:8369(0x700004767000):ZOO_INFO@check_events@1811: session establishment complete on server [127.0.0.1:50946], sessionId=0x168c13aa8b90003, negotiated timeout=10000
I0205 21:14:52.668120 54116352 group.cpp:341] Group process (zookeeper-group(48)@10.0.49.4:65013) connected to ZooKeeper
I0205 21:14:52.668161 54116352 group.cpp:831] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I0205 21:14:52.668203 54116352 group.cpp:395] Authenticating with ZooKeeper using digest
2019-02-05 21:14:52,668:8369(0x70000486d000):ZOO_INFO@check_events@1811: session establishment complete on server [127.0.0.1:50946], sessionId=0x168c13aa8b90004, negotiated timeout=10000
I0205 21:14:52.668565 54652928 group.cpp:341] Group process (zookeeper-group(49)@10.0.49.4:65013) connected to ZooKeeper
I0205 21:14:52.668601 54652928 group.cpp:831] Syncing group operations: queue size (joins, cancels, datas) = (1, 0, 0)
I0205 21:14:52.668615 54652928 group.cpp:395] Authenticating with ZooKeeper using digest
../../src/tests/group_tests.cpp:333: Failure
Failed to wait 15secs for failedGroup2.join("fail")
2019-02-05 21:14:56,010:8369(0x700004767000):ZOO_WARN@zookeeper_interest@1597: Exceeded deadline by 11ms
2019-02-05 21:14:56,010:8369(0x70000486d000):ZOO_INFO@auth_completion_func@1327: Authentication scheme digest succeeded
2019-02-05 21:14:56,010:8369(0x700004767000):ZOO_INFO@auth_completion_func@1327: Authentication scheme digest succeeded
I0205 21:14:56.011027 54652928 group.cpp:419] Trying to create path '/read-only/new' in ZooKeeper
I0205 21:14:56.011297 54116352 group.cpp:419] Trying to create path '/read-only' in ZooKeeper
I0205 21:14:56.012171 54116352 group.cpp:758] Found non-sequence node 'writable' at '/read-only' in ZooKeeper
E0205 21:14:56.012761 54652928 group.cpp:961] Group aborting: Failed to create '/read-only/new' in ZooKeeper: not authenticated
2019-02-05 21:14:56,012:8369(0x70000341f000):ZOO_INFO@zookeeper_close@2562: Closing zookeeper sessionId=0x168c13aa8b90004 to [127.0.0.1:50946]
2019-02-05 21:14:56,013:8369(0x111b155c0):ZOO_INFO@zookeeper_close@2562: Closing zookeeper sessionId=0x168c13aa8b90003 to [127.0.0.1:50946]
2019-02-05 21:14:56,013:8369(0x111b155c0):ZOO_INFO@zookeeper_close@2579: Freeing zookeeper resources for sessionId=0x168c13aa8b90000
I0205 21:14:56.013558 296834496 zookeeper_test_server.cpp:116] Shutting down ZooKeeperTestServer on port 50946
[ FAILED ] GroupTest.GroupPathWithRestrictivePerms (22486 ms)
{code}
> Some GroupTest.* are flaky on Mac.
> ----------------------------------
>
> Key: MESOS-8796
> URL: https://issues.apache.org/jira/browse/MESOS-8796
> Project: Mesos
> Issue Type: Bug
> Components: test
> Affects Versions: 1.6.0, 1.8.0
> Environment: Mac OS with SSL enabled
> Reporter: Alexander Rukletsov
> Priority: Minor
> Labels: flaky, flaky-test, zookeeper
> Attachments: GroupJoinWithDisconnect-badrun.txt, GroupPathWithRestrictivePerms-badrun.txt, GroupPathWithRestrictivePerms-goodrun.txt, GroupTest.GroupPathWithRest-another-bad-run.txt, RetryableErrors-badrun.txt
>
>
> I see some failures related to zookeeper on our Mac machine. Current list of failing tests:
> {noformat}
> GroupTest.GroupPathWithRestrictivePerms
> GroupTest.RetryableErrors
> {noformat}
> Full logs attached.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)