You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "James Peach (JIRA)" <ji...@apache.org> on 2017/10/06 18:28:00 UTC
[jira] [Commented] (MESOS-5309)
PortMappingIsolatorTest.ROOT_NC_ContainerToContainerTCP failed.
[ https://issues.apache.org/jira/browse/MESOS-5309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16194995#comment-16194995 ]
James Peach commented on MESOS-5309:
------------------------------------
On my Fedora 26 system, this reliably fails with:
{noformat}
+ mount --make-rslave /run/netns
+ test -f /proc/sys/net/ipv6/conf/all/disable_ipv6
+ echo 1
+ ip link set lo address 00:3e:e1:c8:84:d1 mtu 1500 up
+ ethtool -K enp12s0 rx off
+ ip link set enp12s0 address 00:3e:e1:c8:84:d1 up
+ ip addr add 17.228.224.108/24 dev enp12s0
+ ip route add default via 17.228.224.1
+ echo 30016 30031
+ echo 1
+ echo 1
+ echo 1
+ '[' -f /proc/sys/net/ipv4/tcp_keepalive_time ']'
+ echo 7200
+ '[' -f /proc/sys/net/core/rmem_max ']'
+ '[' -f /proc/sys/net/ipv4/tcp_keepalive_intvl ']'
+ echo 75
+ '[' -f /proc/sys/net/core/somaxconn ']'
+ echo 128
+ '[' -f /proc/sys/net/core/wmem_max ']'
+ '[' -f /proc/sys/net/core/netdev_max_backlog ']'
+ '[' -f /proc/sys/net/ipv4/tcp_keepalive_probes ']'
+ echo 9
+ '[' -f /proc/sys/net/ipv4/tcp_max_syn_backlog ']'
+ echo 2048
+ '[' -f /proc/sys/net/ipv4/neigh/default/gc_thresh2 ']'
+ '[' -f /proc/sys/net/ipv4/neigh/default/gc_thresh3 ']'
+ '[' -f /proc/sys/net/ipv4/tcp_wmem ']'
+ '[' -f /proc/sys/net/ipv4/neigh/default/gc_thresh1 ']'
+ '[' -f /proc/sys/net/ipv4/tcp_synack_retries ']'
+ echo 5
+ '[' -f /proc/sys/net/ipv4/tcp_rmem ']'
+ '[' -f /proc/sys/net/ipv4/tcp_retries2 ']'
+ echo 15
+ tc qdisc add dev lo ingress
+ tc qdisc add dev enp12s0 ingress
+ tc filter add dev lo parent ffff:0 protocol ip prio 770 u32 flowid ffff:0 match ip dst 17.228.224.108 action mirred egress redirect dev enp12s0
+ tc filter add dev lo parent ffff:0 protocol ip prio 770 u32 flowid ffff:0 match ip dst 127.0.0.1 action mirred egress redirect dev enp12s0
+ tc filter add dev lo parent ffff:0 protocol ip prio 769 u32 flowid ffff:0 match ip dport 30016 fff0
+ tc filter add dev enp12s0 parent ffff:0 protocol ip prio 770 u32 flowid ffff:0 match ip dst 127.0.0.1 match ip dport 30016 fff0 action mirred egress redirect dev lo
+ tc filter add dev lo parent ffff:0 protocol ip prio 769 u32 flowid ffff:0 match ip dport 31000 fff8
+ tc filter add dev enp12s0 parent ffff:0 protocol ip prio 770 u32 flowid ffff:0 match ip dst 127.0.0.1 match ip dport 31000 fff8 action mirred egress redirect dev lo
+ tc filter add dev lo parent ffff:0 protocol ip prio 769 u32 flowid ffff:0 match ip dport 31008 ffe0
+ tc filter add dev enp12s0 parent ffff:0 protocol ip prio 770 u32 flowid ffff:0 match ip dst 127.0.0.1 match ip dport 31008 ffe0 action mirred egress redirect dev lo
+ tc filter add dev lo parent ffff:0 protocol ip prio 769 u32 flowid ffff:0 match ip dport 31040 ffc0
+ tc filter add dev enp12s0 parent ffff:0 protocol ip prio 770 u32 flowid ffff:0 match ip dst 127.0.0.1 match ip dport 31040 ffc0 action mirred egress redirect dev lo
+ tc filter add dev lo parent ffff:0 protocol ip prio 769 u32 flowid ffff:0 match ip dport 31104 ff80
+ tc filter add dev enp12s0 parent ffff:0 protocol ip prio 770 u32 flowid ffff:0 match ip dst 127.0.0.1 match ip dport 31104 ff80 action mirred egress redirect dev lo
+ tc filter add dev lo parent ffff:0 protocol ip prio 769 u32 flowid ffff:0 match ip dport 31232 ff00
+ tc filter add dev enp12s0 parent ffff:0 protocol ip prio 770 u32 flowid ffff:0 match ip dst 127.0.0.1 match ip dport 31232 ff00 action mirred egress redirect dev lo
+ tc filter add dev lo parent ffff:0 protocol ip prio 769 u32 flowid ffff:0 match ip dport 31488 fff8
+ tc filter add dev enp12s0 parent ffff:0 protocol ip prio 770 u32 flowid ffff:0 match ip dst 127.0.0.1 match ip dport 31488 fff8 action mirred egress redirect dev lo
+ tc filter add dev lo parent ffff:0 protocol ip prio 769 u32 flowid ffff:0 match ip dport 31496 fffc
+ tc filter add dev enp12s0 parent ffff:0 protocol ip prio 770 u32 flowid ffff:0 match ip dst 127.0.0.1 match ip dport 31496 fffc action mirred egress redirect dev lo
+ tc filter add dev lo parent ffff:0 protocol ip prio 514 u32 flowid ffff:0 match ip protocol 1 0xff match ip dst 17.228.224.108
+ tc filter add dev lo parent ffff:0 protocol ip prio 514 u32 flowid ffff:0 match ip protocol 1 0xff match ip dst 127.0.0.1
+ tc filter show dev enp12s0 parent ffff:0
filter protocol ip pref 770 u32
filter protocol ip pref 770 u32 fh 800: ht divisor 1
filter protocol ip pref 770 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid ffff:
match 7f000001/ffffffff at 16
match 00007540/0000fff0 at 20
action order 1: mirred (Egress Redirect to device lo) stolen
index 3 ref 1 bind 1
filter protocol ip pref 770 u32 fh 800::801 order 2049 key ht 800 bkt 0 flowid ffff:
match 7f000001/ffffffff at 16
match 00007918/0000fff8 at 20
action order 1: mirred (Egress Redirect to device lo) stolen
index 4 ref 1 bind 1
filter protocol ip pref 770 u32 fh 800::802 order 2050 key ht 800 bkt 0 flowid ffff:
match 7f000001/ffffffff at 16
match 00007920/0000ffe0 at 20
action order 1: mirred (Egress Redirect to device lo) stolen
index 5 ref 1 bind 1
filter protocol ip pref 770 u32 fh 800::803 order 2051 key ht 800 bkt 0 flowid ffff:
match 7f000001/ffffffff at 16
match 00007940/0000ffc0 at 20
action order 1: mirred (Egress Redirect to device lo) stolen
index 6 ref 1 bind 1
filter protocol ip pref 770 u32 fh 800::804 order 2052 key ht 800 bkt 0 flowid ffff:
match 7f000001/ffffffff at 16
match 00007980/0000ff80 at 20
action order 1: mirred (Egress Redirect to device lo) stolen
index 7 ref 1 bind 1
filter protocol ip pref 770 u32 fh 800::805 order 2053 key ht 800 bkt 0 flowid ffff:
match 7f000001/ffffffff at 16
match 00007a00/0000ff00 at 20
action order 1: mirred (Egress Redirect to device lo) stolen
index 8 ref 1 bind 1
filter protocol ip pref 770 u32 fh 800::806 order 2054 key ht 800 bkt 0 flowid ffff:
match 7f000001/ffffffff at 16
match 00007b00/0000fff8 at 20
action order 1: mirred (Egress Redirect to device lo) stolen
index 9 ref 1 bind 1
filter protocol ip pref 770 u32 fh 800::807 order 2055 key ht 800 bkt 0 flowid ffff:
match 7f000001/ffffffff at 16
match 00007b08/0000fffc at 20
action order 1: mirred (Egress Redirect to device lo) stolen
index 10 ref 1 bind 1
+ tc filter show dev lo parent ffff:0
filter protocol ip pref 514 u32
filter protocol ip pref 514 u32 fh 802: ht divisor 1
filter protocol ip pref 514 u32 fh 802::800 order 2048 key ht 802 bkt 0 flowid ffff:
match 00010000/00ff0000 at 8
match 11e4e06c/ffffffff at 16
filter protocol ip pref 514 u32 fh 802::801 order 2049 key ht 802 bkt 0 flowid ffff:
match 00010000/00ff0000 at 8
match 7f000001/ffffffff at 16
filter protocol ip pref 769 u32
filter protocol ip pref 769 u32 fh 801: ht divisor 1
filter protocol ip pref 769 u32 fh 801::800 order 2048 key ht 801 bkt 0 flowid ffff:
match 00007540/0000fff0 at 20
filter protocol ip pref 769 u32 fh 801::801 order 2049 key ht 801 bkt 0 flowid ffff:
match 00007918/0000fff8 at 20
filter protocol ip pref 769 u32 fh 801::802 order 2050 key ht 801 bkt 0 flowid ffff:
match 00007920/0000ffe0 at 20
filter protocol ip pref 769 u32 fh 801::803 order 2051 key ht 801 bkt 0 flowid ffff:
match 00007940/0000ffc0 at 20
filter protocol ip pref 769 u32 fh 801::804 order 2052 key ht 801 bkt 0 flowid ffff:
match 00007980/0000ff80 at 20
filter protocol ip pref 769 u32 fh 801::805 order 2053 key ht 801 bkt 0 flowid ffff:
match 00007a00/0000ff00 at 20
filter protocol ip pref 769 u32 fh 801::806 order 2054 key ht 801 bkt 0 flowid ffff:
match 00007b00/0000fff8 at 20
filter protocol ip pref 769 u32 fh 801::807 order 2055 key ht 801 bkt 0 flowid ffff:
match 00007b08/0000fffc at 20
filter protocol ip pref 770 u32
filter protocol ip pref 770 u32 fh 800: ht divisor 1
filter protocol ip pref 770 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid ffff:
match 11e4e06c/ffffffff at 16
action order 1: mirred (Egress Redirect to device enp12s0) stolen
index 1 ref 1 bind 1
filter protocol ip pref 770 u32 fh 800::801 order 2049 key ht 800 bkt 0 flowid ffff:
match 7f000001/ffffffff at 16
action order 1: mirred (Egress Redirect to device enp12s0) stolen
index 2 ref 1 bind 1
../../src/tests/containerizer/port_mapping_tests.cpp:588: Failure
(isolator.get()->isolate(containerId2, pid.get())).failure(): Failed to append a ICMP mirror action from host enp12s0 to mesos24638: Invalid input data or parameter
{noformat}
> PortMappingIsolatorTest.ROOT_NC_ContainerToContainerTCP failed.
> ---------------------------------------------------------------
>
> Key: MESOS-5309
> URL: https://issues.apache.org/jira/browse/MESOS-5309
> Project: Mesos
> Issue Type: Bug
> Components: containerization
> Environment: Fedora 23 with network isolator enabled
> Reporter: Gilbert Song
> Labels: isolation
>
> Here is the log:
> {code}
> [01:22:18] : [ RUN ] PortMappingIsolatorTest.ROOT_NC_ContainerToContainerTCP
> [01:22:18]W: I0429 01:22:18.416817 24850 port_mapping_tests.cpp:229] Using eth0 as the public interface
> [01:22:18]W: I0429 01:22:18.417135 24850 port_mapping_tests.cpp:237] Using lo as the loopback interface
> [01:22:18]W: I0429 01:22:18.429095 24850 resources.cpp:572] Parsing resources as JSON failed: cpus:2;mem:1024;disk:1024;ephemeral_ports:[30001-30999];ports:[31000-32000]
> [01:22:18]W: Trying semicolon-delimited string format instead
> [01:22:18]W: I0429 01:22:18.430194 24850 port_mapping.cpp:1557] Using eth0 as the public interface
> [01:22:18]W: I0429 01:22:18.430490 24850 port_mapping.cpp:1582] Using lo as the loopback interface
> [01:22:18]W: I0429 01:22:18.431619 24850 port_mapping.cpp:1869] /proc/sys/net/ipv4/neigh/default/gc_thresh3 = '1024'
> [01:22:18]W: I0429 01:22:18.431668 24850 port_mapping.cpp:1869] /proc/sys/net/ipv4/neigh/default/gc_thresh1 = '128'
> [01:22:18]W: I0429 01:22:18.431723 24850 port_mapping.cpp:1869] /proc/sys/net/ipv4/tcp_wmem = '4096 16384 4194304'
> [01:22:18]W: I0429 01:22:18.431761 24850 port_mapping.cpp:1869] /proc/sys/net/ipv4/tcp_synack_retries = '5'
> [01:22:18]W: I0429 01:22:18.431797 24850 port_mapping.cpp:1869] /proc/sys/net/core/rmem_max = '212992'
> [01:22:18]W: I0429 01:22:18.431830 24850 port_mapping.cpp:1869] /proc/sys/net/core/somaxconn = '128'
> [01:22:18]W: I0429 01:22:18.431864 24850 port_mapping.cpp:1869] /proc/sys/net/core/wmem_max = '212992'
> [01:22:18]W: I0429 01:22:18.431900 24850 port_mapping.cpp:1869] /proc/sys/net/ipv4/tcp_rmem = '4096 87380 6291456'
> [01:22:18]W: I0429 01:22:18.431933 24850 port_mapping.cpp:1869] /proc/sys/net/ipv4/tcp_keepalive_time = '7200'
> [01:22:18]W: I0429 01:22:18.431967 24850 port_mapping.cpp:1869] /proc/sys/net/ipv4/neigh/default/gc_thresh2 = '512'
> [01:22:18]W: I0429 01:22:18.432001 24850 port_mapping.cpp:1869] /proc/sys/net/core/netdev_max_backlog = '1000'
> [01:22:18]W: I0429 01:22:18.432036 24850 port_mapping.cpp:1869] /proc/sys/net/ipv4/tcp_keepalive_intvl = '75'
> [01:22:18]W: I0429 01:22:18.432070 24850 port_mapping.cpp:1869] /proc/sys/net/ipv4/tcp_keepalive_probes = '9'
> [01:22:18]W: I0429 01:22:18.432101 24850 port_mapping.cpp:1869] /proc/sys/net/ipv4/tcp_max_syn_backlog = '512'
> [01:22:18]W: I0429 01:22:18.432134 24850 port_mapping.cpp:1869] /proc/sys/net/ipv4/tcp_retries2 = '15'
> [01:22:18]W: F0429 01:22:18.432205 24850 port_mapping_tests.cpp:448] CHECK_SOME(isolator): Failed to get realpath for bind mount root '/var/run/netns': Not found
> [01:22:18]W: *** Check failure stack trace: ***
> [01:22:18]W: @ 0x7fc8dccfc986 google::LogMessage::Fail()
> [01:22:18]W: @ 0x7fc8dccfc8df google::LogMessage::SendToLog()
> [01:22:18]W: @ 0x7fc8dccfc2d5 google::LogMessage::Flush()
> [01:22:18]W: @ 0x7fc8dccff146 google::LogMessageFatal::~LogMessageFatal()
> [01:22:18]W: @ 0xa6031d _CheckFatal::~_CheckFatal()
> [01:22:18]W: @ 0x1890d61 mesos::internal::tests::PortMappingIsolatorTest_ROOT_NC_ContainerToContainerTCP_Test::TestBody()
> [01:22:18]W: @ 0x19437a8 testing::internal::HandleSehExceptionsInMethodIfSupported<>()
> [01:22:18]W: @ 0x193e81a testing::internal::HandleExceptionsInMethodIfSupported<>()
> [01:22:18]W: @ 0x191f264 testing::Test::Run()
> [01:22:18]W: @ 0x191fa1c testing::TestInfo::Run()
> [01:22:18]W: @ 0x192006d testing::TestCase::Run()
> [01:22:18]W: @ 0x1926bab testing::internal::UnitTestImpl::RunAllTests()
> [01:22:18]W: @ 0x194446f testing::internal::HandleSehExceptionsInMethodIfSupported<>()
> [01:22:18]W: @ 0x193f35a testing::internal::HandleExceptionsInMethodIfSupported<>()
> [01:22:18]W: @ 0x1925887 testing::UnitTest::Run()
> [01:22:18]W: @ 0xf9131d RUN_ALL_TESTS()
> [01:22:18]W: @ 0xf90f15 main
> [01:22:18]W: @ 0x7fc8d68d8580 __libc_start_main
> [01:22:18]W: @ 0xa5e919 _start
> [01:22:19]W: /mnt/teamcity/temp/agentTmp/custom_script1282998915150293546: line 3: 24850 Aborted (core dumped) GLOG_v=1 ./bin/mesos-tests.sh --verbose --gtest_filter="$GTEST_FILTER"
> [01:22:19]W: Process exited with code 134
> [01:22:19]i: ##teamcity[buildStatisticValue key='buildStageDuration:buildStepRUNNER_299' value='815124.0']
> [01:22:19]E: Step Run tests (Command Line) failed
> [01:22:19]i: ##teamcity[buildStatisticValue key='buildStageDuration:buildFinishing' value='8.0']
> [01:22:19] : Publishing internal artifacts
> [01:22:19] : Publishing 1 file using [WebPublisher]
> [01:22:19] : Publishing 1 file using [ArtifactsCachePublisher]
> [01:22:19]i: ##teamcity[buildStatisticValue key='buildStageDuration:artifactsPublishing' value='49.0']
> [01:22:20] : Build finished
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)