You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mesos.apache.org by ch...@apache.org on 2018/11/28 23:27:02 UTC
[mesos] branch master updated (84a6eef -> f24736d)
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git.
from 84a6eef Fixed some incorrect CentOS RPM build settings.
new 276774c Fixed master crash when executors send messages to recovered frameworks.
new 9f268d2 Made the `createTask` helper work for both v0 and v1 API.
new aabd67f Added a test for executors sending messages to recovered frameworks.
new dee1010 Added MESOS-9418 to the 1.7.1 CHANGELOG.
new 15abc35 Added MESOS-9419 to the 1.7.1 CHANGELOG.
new d3069e3 Added MESOS-9418 to the 1.6.2 CHANGELOG.
new ce17a3f Added MESOS-9419 to the 1.6.2 CHANGELOG.
new fb983cf Added MESOS-9419 to the 1.5.3 CHANGELOG.
new f24736d Added MESOS-9419 to the 1.4.3 CHANGELOG.
The 9 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
CHANGELOG | 6 ++
src/master/master.cpp | 10 +++
src/master/master.hpp | 26 +++++--
src/tests/master_tests.cpp | 189 +++++++++++++++++++++++++++++++++++++++++++++
src/tests/mesos.hpp | 14 +++-
5 files changed, 238 insertions(+), 7 deletions(-)
[mesos] 07/09: Added MESOS-9419 to the 1.6.2 CHANGELOG.
Posted by ch...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit ce17a3fb691d757cae7e7aa77494fe4b678606cf
Author: Chun-Hung Hsiao <ch...@mesosphere.io>
AuthorDate: Wed Nov 28 10:18:52 2018 -0800
Added MESOS-9419 to the 1.6.2 CHANGELOG.
---
CHANGELOG | 1 +
1 file changed, 1 insertion(+)
diff --git a/CHANGELOG b/CHANGELOG
index ea62cae..cf00990 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -426,6 +426,7 @@ Release Notes - Mesos - Version 1.6.2 (WIP)
* [MESOS-9332] - Nested container should run as the same user of its parent container by default.
* [MESOS-9334] - Container stuck at ISOLATING state due to libevent poll never returns.
* [MESOS-9418] - Add support for the `Discard` blkio operation type.
+ * [MESOS-9419] - Executor to framework message crashes master if framework has not re-registered.
** Improvement
* [MESOS-9305] - Create cgoup recursively to workaround systemd deleting cgroups_root.
[mesos] 01/09: Fixed master crash when executors send messages to
recovered frameworks.
Posted by ch...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 276774c4d88dbd2f5147c90c53b795e1ec889494
Author: Chun-Hung Hsiao <ch...@mesosphere.io>
AuthorDate: Mon Nov 26 20:12:36 2018 -0800
Fixed master crash when executors send messages to recovered frameworks.
The `Framework::send` function assumes that either `http` or `pid` is
set, which is not true for a framework that hasn't yet reregistered yet
but recovered from a reregistered agent. As a result, the master would
crash when a recovered executor tries to send a message to such a
framework (see MESOS-9419). This patch fixes this crash bug.
Review: https://reviews.apache.org/r/69451
---
src/master/master.cpp | 10 ++++++++++
src/master/master.hpp | 26 ++++++++++++++++++++------
2 files changed, 30 insertions(+), 6 deletions(-)
diff --git a/src/master/master.cpp b/src/master/master.cpp
index b4b02d8..3b3824a 100644
--- a/src/master/master.cpp
+++ b/src/master/master.cpp
@@ -6482,6 +6482,16 @@ void Master::executorMessage(
return;
}
+ if (!framework->connected()) {
+ LOG(WARNING) << "Not forwarding executor message for executor '"
+ << executorId << "' of framework " << frameworkId
+ << " on agent " << *slave
+ << " because the framework is disconnected";
+
+ metrics->invalid_executor_to_framework_messages++;
+ return;
+ }
+
ExecutorToFrameworkMessage message;
*message.mutable_slave_id() =
std::move(*executorToFrameworkMessage.mutable_slave_id());
diff --git a/src/master/master.hpp b/src/master/master.hpp
index 3b3c1a4..c7becfa 100644
--- a/src/master/master.hpp
+++ b/src/master/master.hpp
@@ -2591,21 +2591,35 @@ private:
template <typename Message>
void Framework::send(const Message& message)
{
+ metrics.incrementEvent(message);
+
if (!connected()) {
- LOG(WARNING) << "Master attempted to send message to disconnected"
+ LOG(WARNING) << "Master attempting to send message to disconnected"
<< " framework " << *this;
- }
- metrics.incrementEvent(message);
+ // NOTE: We proceed here without returning to support the case where a
+ // "disconnected" framework is still talking to the master and the master
+ // wants to shut it down by sending a `FrameworkErrorMessage`. This can
+ // occur in a one-way network partition where the master -> framework link
+ // is broken but the framework -> master link remains intact. Note that we
+ // have no periodic heartbeats between the master and pid-based schedulers.
+ //
+ // TODO(chhsiao): Update the `FrameworkErrorMessage` call-sites that rely on
+ // the lack of a `return` here to directly call `process::send` so that this
+ // function doesn't need to deal with the special case. Then we can check
+ // that one of `http` or `pid` is set if the framework is connected.
+ }
if (http.isSome()) {
if (!http->send(message)) {
- LOG(WARNING) << "Unable to send event to framework " << *this << ":"
+ LOG(WARNING) << "Unable to send message to framework " << *this << ":"
<< " connection closed";
}
- } else {
- CHECK_SOME(pid);
+ } else if (pid.isSome()) {
master->send(pid.get(), message);
+ } else {
+ LOG(WARNING) << "Unable to send message to framework " << *this << ":"
+ << " framework is recovered but has not reregistered";
}
}
[mesos] 04/09: Added MESOS-9418 to the 1.7.1 CHANGELOG.
Posted by ch...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit dee101099bbf34515653e3761e77e005227b651b
Author: James Peach <jp...@apache.org>
AuthorDate: Tue Nov 27 09:59:33 2018 -0800
Added MESOS-9418 to the 1.7.1 CHANGELOG.
---
CHANGELOG | 1 +
1 file changed, 1 insertion(+)
diff --git a/CHANGELOG b/CHANGELOG
index 8a08f0f..345d88b 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -27,6 +27,7 @@ Release Notes - Mesos - Version 1.7.1 (WIP)
* [MESOS-9325] - Optimize `Resources::filter` operation.
* [MESOS-9332] - Nested container should run as the same user of its parent container by default.
* [MESOS-9334] - Container stuck at ISOLATING state due to libevent poll never returns.
+ * [MESOS-9418] - Add support for the `Discard` blkio operation type.
** Improvement:
* [MESOS-6765] - Make the Resources wrapper "copy-on-write" to improve performance.
[mesos] 08/09: Added MESOS-9419 to the 1.5.3 CHANGELOG.
Posted by ch...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit fb983cfce38b9f4103a01d85c2b54354337b9f95
Author: Chun-Hung Hsiao <ch...@mesosphere.io>
AuthorDate: Wed Nov 28 10:20:29 2018 -0800
Added MESOS-9419 to the 1.5.3 CHANGELOG.
---
CHANGELOG | 1 +
1 file changed, 1 insertion(+)
diff --git a/CHANGELOG b/CHANGELOG
index cf00990..5ba77a1 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -824,6 +824,7 @@ Release Notes - Mesos - Version 1.5.3 (WIP)
* [MESOS-7474] - Mesos fetcher cache doesn't retry when missed.
* [MESOS-9317] - Some master endpoints do not handle failed authorization properly.
* [MESOS-9332] - Nested container should run as the same user of its parent container by default.
+ * [MESOS-9419] - Executor to framework message crashes master if framework has not re-registered.
Release Notes - Mesos - Version 1.5.2
[mesos] 02/09: Made the `createTask` helper work for both v0 and v1
API.
Posted by ch...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 9f268d255ccd785d9c69904df4827cfa8a44225e
Author: Chun-Hung Hsiao <ch...@mesosphere.io>
AuthorDate: Tue Nov 27 13:50:37 2018 -0800
Made the `createTask` helper work for both v0 and v1 API.
The patched `createTask` helper originally uses the `slave_id` accessor,
which does not apply to the v1 API. This patch fixes this problem.
Review: https://reviews.apache.org/r/69464
---
src/tests/mesos.hpp | 14 +++++++++++++-
1 file changed, 13 insertions(+), 1 deletion(-)
diff --git a/src/tests/mesos.hpp b/src/tests/mesos.hpp
index 576f4bd..b181f73 100644
--- a/src/tests/mesos.hpp
+++ b/src/tests/mesos.hpp
@@ -798,6 +798,18 @@ inline TContainerInfo createContainerInfo(
}
+inline SlaveID getAgentID(const Offer& offer)
+{
+ return offer.slave_id();
+}
+
+
+inline mesos::v1::AgentID getAgentID(const mesos::v1::Offer& offer)
+{
+ return offer.agent_id();
+}
+
+
inline void setAgentID(TaskInfo* task, const SlaveID& slaveId)
{
task->mutable_slave_id()->CopyFrom(slaveId);
@@ -904,7 +916,7 @@ inline TTaskInfo createTask(
TExecutorInfo,
TCommandInfo,
TOffer>(
- offer.slave_id(),
+ getAgentID(offer),
offer.resources(),
command,
executorId,
[mesos] 03/09: Added a test for executors sending messages to
recovered frameworks.
Posted by ch...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit aabd67f7500863a9874878081e09e37fdc0c14e9
Author: Chun-Hung Hsiao <ch...@mesosphere.io>
AuthorDate: Tue Nov 27 13:53:03 2018 -0800
Added a test for executors sending messages to recovered frameworks.
This patch adds the `ExecutorMessageToRecoveredHttpFramework` test to
ensure that a master will not crash when forwarding an executor message
to a recovered framework to prevent any future regression of MESOS-9419.
Review: https://reviews.apache.org/r/69452
---
src/tests/master_tests.cpp | 189 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 189 insertions(+)
diff --git a/src/tests/master_tests.cpp b/src/tests/master_tests.cpp
index 651bb9b..369bd33 100644
--- a/src/tests/master_tests.cpp
+++ b/src/tests/master_tests.cpp
@@ -3841,6 +3841,195 @@ TEST_F(MasterTest, RecoveredFramework)
}
+// This test ensures that when a master fails over, the recovered master will
+// not crash when executors that belong to recovered (but not yet reregistered)
+// frameworks try to send executor -> framework messages. See MESOS-8623.
+//
+// NOTE: This test uses an HTTP framework which has no PID and thus any
+// executor -> framework message is forwarded by the master. Otherwise, the
+// agent would bypass the master and directly sends it to the scheduler driver.
+TEST_F(MasterTest, ExecutorMessageToRecoveredHttpFramework)
+{
+ Clock::pause();
+
+ master::Flags masterFlags = CreateMasterFlags();
+ Try<Owned<cluster::Master>> master = StartMaster(masterFlags);
+
+ ASSERT_SOME(master);
+
+ StandaloneMasterDetector detector(master.get()->pid);
+
+ // Set up a mock executor to send messages.
+ auto executor = std::make_shared<v1::MockHTTPExecutor>();
+ TestContainerizer containerizer(DEFAULT_EXECUTOR_ID, executor);
+
+ Future<v1::executor::Mesos*> executorLibrary;
+ EXPECT_CALL(*executor, connected(_))
+ .WillOnce(FutureArg<0>(&executorLibrary));
+
+ Future<Nothing> executorSubscribed;
+ EXPECT_CALL(*executor, subscribed(_, _))
+ .WillOnce(FutureSatisfy(&executorSubscribed));
+
+ EXPECT_CALL(*executor, launch(_, _));
+
+ EXPECT_CALL(*executor, shutdown(_))
+ .Times(AtMost(1));
+
+ slave::Flags slaveFlags = CreateSlaveFlags();
+ Try<Owned<cluster::Slave>> slave =
+ StartSlave(&detector, &containerizer, slaveFlags);
+
+ ASSERT_SOME(slave);
+
+ // Register an HTTP framework to launch the mock executor.
+ auto scheduler = std::make_shared<v1::MockHTTPScheduler>();
+ Owned<v1::scheduler::TestMesos> schedulerLibrary(new v1::scheduler::TestMesos(
+ master.get()->pid, ContentType::PROTOBUF, scheduler));
+
+ Future<Nothing> schedulerConnected;
+ EXPECT_CALL(*scheduler, connected(_))
+ .WillOnce(FutureSatisfy(&schedulerConnected))
+ .WillRepeatedly(Return()); // Ignore teardown reconnections, see MESOS-6033.
+
+ Future<v1::scheduler::Event::Subscribed> schedulerSubscribed;
+ EXPECT_CALL(*scheduler, subscribed(_, _))
+ .WillOnce(FutureArg<1>(&schedulerSubscribed));
+
+ EXPECT_CALL(*scheduler, heartbeat(_))
+ .WillRepeatedly(Return()); // Ignore heartbeats.
+
+ Future<v1::scheduler::Event::Offers> offers;
+ EXPECT_CALL(*scheduler, offers(_, _))
+ .WillOnce(FutureArg<1>(&offers))
+ .WillRepeatedly(Return()); // Ignore subsequent offers.
+
+ EXPECT_CALL(*scheduler, disconnected(_))
+ .Times(AtMost(1));
+
+ AWAIT_READY(schedulerConnected);
+
+ {
+ v1::scheduler::Call call;
+ call.set_type(v1::scheduler::Call::SUBSCRIBE);
+ call.mutable_subscribe()->mutable_framework_info()
+ ->CopyFrom(v1::DEFAULT_FRAMEWORK_INFO);
+
+ schedulerLibrary->send(call);
+ }
+
+ AWAIT_READY(schedulerSubscribed);
+
+ const v1::FrameworkID& frameworkId = schedulerSubscribed->framework_id();
+
+ // Advance the clock and trigger a batch allocation.
+ Clock::advance(masterFlags.allocation_interval);
+
+ AWAIT_READY(offers);
+ ASSERT_FALSE(offers->offers().empty());
+
+ // Launch the mock executor through a `LAUNCH` operation.
+ {
+ v1::scheduler::Call call;
+ call.mutable_framework_id()->CopyFrom(frameworkId);
+ call.set_type(v1::scheduler::Call::ACCEPT);
+ call.mutable_accept()->add_offer_ids()->CopyFrom(offers->offers(0).id());
+
+ v1::Offer::Operation* operation = call.mutable_accept()->add_operations();
+ operation->set_type(v1::Offer::Operation::LAUNCH);
+ operation->mutable_launch()->add_task_infos()->CopyFrom(
+ v1::createTask(offers->offers(0), "hello", v1::DEFAULT_EXECUTOR_ID));
+
+ schedulerLibrary->send(call);
+ }
+
+ AWAIT_READY(executorLibrary);
+
+ {
+ v1::executor::Call call;
+ call.mutable_framework_id()->CopyFrom(frameworkId);
+ call.mutable_executor_id()->CopyFrom(v1::DEFAULT_EXECUTOR_ID);
+ call.set_type(v1::executor::Call::SUBSCRIBE);
+ call.mutable_subscribe();
+
+ executorLibrary.get()->send(call);
+ }
+
+ AWAIT_READY(executorSubscribed);
+
+ // Shutdown the master.
+ master->reset();
+
+ // Fail over the framework and ensure that no more callback will be made to
+ // the scheduler.
+ schedulerLibrary.reset();
+ Clock::settle();
+
+ // Restart the master.
+ master = StartMaster(masterFlags);
+
+ ASSERT_SOME(master);
+
+ Future<SlaveReregisteredMessage> slaveReregisteredMessage =
+ FUTURE_PROTOBUF(SlaveReregisteredMessage(), master.get()->pid, _);
+
+ detector.appoint(master.get()->pid);
+
+ // Advance the clock to trigger agent reregistration.
+ Clock::advance(slaveFlags.registration_backoff_factor);
+
+ AWAIT_READY(slaveReregisteredMessage);
+
+ // Validate that the framework is recovered through agent reregistration.
+ {
+ Future<Response> response = process::http::get(
+ master.get()->pid,
+ "state",
+ None(),
+ createBasicAuthHeaders(DEFAULT_CREDENTIAL));
+
+ AWAIT_ASSERT_RESPONSE_STATUS_EQ(OK().status, response);
+ AWAIT_ASSERT_RESPONSE_HEADER_EQ(APPLICATION_JSON, "Content-Type", response);
+
+ Try<JSON::Object> state = JSON::parse<JSON::Object>(response.get().body);
+ ASSERT_SOME(state);
+ ASSERT_EQ(1u, state->values["frameworks"].as<JSON::Array>().values.size());
+
+ const JSON::Object& framework =
+ state->values.at("frameworks").as<JSON::Array>()
+ .values.at(0).as<JSON::Object>();
+
+ EXPECT_EQ(
+ frameworkId.value(),
+ framework.values.at("id").as<JSON::String>().value);
+
+ EXPECT_FALSE(framework.values.at("active").as<JSON::Boolean>().value);
+ EXPECT_FALSE(framework.values.at("connected").as<JSON::Boolean>().value);
+ EXPECT_TRUE(framework.values.at("recovered").as<JSON::Boolean>().value);
+ }
+
+ Future<ExecutorToFrameworkMessage> executorMessage =
+ FUTURE_PROTOBUF(ExecutorToFrameworkMessage(), _, master.get()->pid);
+
+ // Send an executor -> framework message to trigger the
+ // `Master::executorMessage` handler.
+ {
+ v1::executor::Call call;
+ call.mutable_framework_id()->CopyFrom(frameworkId);
+ call.mutable_executor_id()->CopyFrom(v1::DEFAULT_EXECUTOR_ID);
+ call.set_type(v1::executor::Call::MESSAGE);
+ call.mutable_message()->set_data("world");
+
+ executorLibrary.get()->send(call);
+ }
+
+ AWAIT_READY(executorMessage);
+
+ // Wait for the `Master::executorMessage` handler to finish.
+ Clock::settle();
+}
+
+
// This test verifies that a framework that has not yet reregistered
// after a master failover doesn't show up multiple times in
// "frameworks" when querying "/state" or "/frameworks" endpoints. This
[mesos] 05/09: Added MESOS-9419 to the 1.7.1 CHANGELOG.
Posted by ch...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 15abc35613387e35e18de27431349d1871e341a3
Author: Chun-Hung Hsiao <ch...@mesosphere.io>
AuthorDate: Wed Nov 28 10:18:20 2018 -0800
Added MESOS-9419 to the 1.7.1 CHANGELOG.
---
CHANGELOG | 1 +
1 file changed, 1 insertion(+)
diff --git a/CHANGELOG b/CHANGELOG
index 345d88b..c8a3521 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -28,6 +28,7 @@ Release Notes - Mesos - Version 1.7.1 (WIP)
* [MESOS-9332] - Nested container should run as the same user of its parent container by default.
* [MESOS-9334] - Container stuck at ISOLATING state due to libevent poll never returns.
* [MESOS-9418] - Add support for the `Discard` blkio operation type.
+ * [MESOS-9419] - Executor to framework message crashes master if framework has not re-registered.
** Improvement:
* [MESOS-6765] - Make the Resources wrapper "copy-on-write" to improve performance.
[mesos] 06/09: Added MESOS-9418 to the 1.6.2 CHANGELOG.
Posted by ch...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit d3069e358ed704e3b64d0ed9a9bb520aef9da8a5
Author: James Peach <jp...@apache.org>
AuthorDate: Tue Nov 27 11:11:47 2018 -0800
Added MESOS-9418 to the 1.6.2 CHANGELOG.
---
CHANGELOG | 1 +
1 file changed, 1 insertion(+)
diff --git a/CHANGELOG b/CHANGELOG
index c8a3521..ea62cae 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -425,6 +425,7 @@ Release Notes - Mesos - Version 1.6.2 (WIP)
* [MESOS-9317] - Some master endpoints do not handle failed authorization properly.
* [MESOS-9332] - Nested container should run as the same user of its parent container by default.
* [MESOS-9334] - Container stuck at ISOLATING state due to libevent poll never returns.
+ * [MESOS-9418] - Add support for the `Discard` blkio operation type.
** Improvement
* [MESOS-9305] - Create cgoup recursively to workaround systemd deleting cgroups_root.
[mesos] 09/09: Added MESOS-9419 to the 1.4.3 CHANGELOG.
Posted by ch...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
chhsiao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit f24736d41b109c474f93d7e16fd1e2448074d91e
Author: Chun-Hung Hsiao <ch...@mesosphere.io>
AuthorDate: Wed Nov 28 10:21:01 2018 -0800
Added MESOS-9419 to the 1.4.3 CHANGELOG.
---
CHANGELOG | 1 +
1 file changed, 1 insertion(+)
diff --git a/CHANGELOG b/CHANGELOG
index 5ba77a1..6e82394 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -1422,6 +1422,7 @@ Release Notes - Mesos - Version 1.4.3 (WIP)
* [MESOS-9279] - Docker Containerizer 'usage' call might be expensive if mount table is big.
* [MESOS-9283] - Docker containerizer actor can get backlogged with large number of containers.
* [MESOS-9334] - Container stuck at ISOLATING state due to libevent poll never returns.
+ * [MESOS-9419] - Executor to framework message crashes master if framework has not re-registered.
Release Notes - Mesos - Version 1.4.2