You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modules@mesos.apache.org by liang tang <li...@gmail.com> on 2017/05/31 03:38:48 UTC
two questions about mesos module
1. Cannot find mesos module entry where module will start to run? didn't
find in slave/main.cc or module/mananger.cc
2. Why process in mesos module exits, service cannot be run in Marathon?
In my option, module shouldn't have an influence on service running.
Which code will invoke code about container logger module in
https://github.com/apache/mesos/blob/master/src/slave/container_loggers/lib_logrotate.cpp#L309-L339
mesos::modules::Module<ContainerLogger>
org_apache_mesos_LogrotateContainerLogger(
MESOS_MODULE_API_VERSION,
MESOS_VERSION,
"Apache Mesos",
"modules@mesos.apache.org",
"Logrotate Container Logger module.",
nullptr,
[](const Parameters& parameters) -> ContainerLogger* {
// Convert `parameters` into a map.
std::map<std::string, std::string> values;
foreach (const Parameter& parameter, parameters.parameter()) {
values[parameter.key()] = parameter.value();
}
// Load and validate flags from the map.
mesos::internal::logger::Flags flags;
Try<flags::Warnings> load = flags.load(values);
if (load.isError()) {
LOG(ERROR) << "Failed to parse parameters: " << load.error();
return nullptr;
}
// Log any flag warnings.
foreach (const flags::Warning& warning, load->warnings) {
LOG(WARNING) << warning.message;
}
return new mesos::internal::logger::LogrotateContainerLogger(flags);
});
Re: two questions about mesos module
Posted by Joseph Wu <jo...@mesosphere.io>.
That depends on what you consider more important.
Since your logging service (Graylog) is stuck, that means your module can
do the following:
* Stall and queue up logs until the logging service is unstuck.
* Drop logs.
* Log to some other location/service.
Basically, it is up to you whether you want to keep logs or keep your
services healthy, albeit without logs. For a centralized logging service
like Graylog, it is usually a good idea to set up your logging pipeline in
three components:
Your tasks + your logging module
=> log forwarding service located on the same machine/node
=> log aggregation server(s)
Since the log forwarding service is co-located, it is much less likely for
that service to stall. And when there is a network partition, or if the
upstream aggregation server stalls, it is up to the forwarding service to
buffer the logs and remain responsive.
On Wed, Jun 7, 2017 at 8:49 PM, liang tang <li...@gmail.com>
wrote:
> I think
> https://github.com/apache/mesos/blob/master/src/slave/
> containerizer/mesos/io/switchboard.cpp#L278-L289
> is the real entry where prepare function in container logger will be
> invoked.
>
> I wrote a module which sends stderr and stdout to graylog. If graylog got
> stuck, service in Marathon even cannot run successfully, at the same time
> we cannot kill the stuck task in Marathon.
>
> How can I deal with the stuck process in mesos module not having any bad
> effect on mesos services?
>
>
>
> 2017-06-02 9:35 GMT+08:00 Joseph Wu <jo...@mesosphere.io>:
>
> > 1) The entry point for the container logger module is here:
> > https://github.com/apache/mesos/blob/1.3.x/src/slave/
> > container_logger.cpp#L42-L52
> >
> > All modules have a "create" method which attempts to load the dynamic
> > library containing the module. The --modules flag you specify at
> > agent/master startup time tells the agent/master where to find the
> library
> > and what the symbol name to load is.
> >
> > For other invocations of the module(s), you'll need to look for places in
> > the code where the module interface is called. The precise module class
> > obviously won't be referenced directly within the codebase (because
> they're
> > loaded at runtime, rather than at compile time).
> >
> > 2) Modules are loaded precisely so that they have an influence
> (otherwise,
> > why load a module at all?). If you don't like the side effects of a
> > module, your choices are to either disable the module, or to tweak the
> > module yourself and reload it.
> >
> >
> > On Tue, May 30, 2017 at 8:38 PM, liang tang <liangtang20080522@gmail.com
> >
> > wrote:
> >
> > > 1. Cannot find mesos module entry where module will start to run?
> didn't
> > > find in slave/main.cc or module/mananger.cc
> > >
> > > 2. Why process in mesos module exits, service cannot be run in
> Marathon?
> > > In my option, module shouldn't have an influence on service running.
> > >
> > > Which code will invoke code about container logger module in
> > > https://github.com/apache/mesos/blob/master/src/slave/
> > > container_loggers/lib_logrotate.cpp#L309-L339
> > >
> > >
> > >
> > > mesos::modules::Module<ContainerLogger>
> > > org_apache_mesos_LogrotateContainerLogger(
> > > MESOS_MODULE_API_VERSION,
> > > MESOS_VERSION,
> > > "Apache Mesos",
> > > "modules@mesos.apache.org",
> > > "Logrotate Container Logger module.",
> > > nullptr,
> > > [](const Parameters& parameters) -> ContainerLogger* {
> > > // Convert `parameters` into a map.
> > > std::map<std::string, std::string> values;
> > > foreach (const Parameter& parameter, parameters.parameter()) {
> > > values[parameter.key()] = parameter.value();
> > > }
> > >
> > > // Load and validate flags from the map.
> > > mesos::internal::logger::Flags flags;
> > > Try<flags::Warnings> load = flags.load(values);
> > >
> > > if (load.isError()) {
> > > LOG(ERROR) << "Failed to parse parameters: " << load.error();
> > > return nullptr;
> > > }
> > >
> > > // Log any flag warnings.
> > > foreach (const flags::Warning& warning, load->warnings) {
> > > LOG(WARNING) << warning.message;
> > > }
> > >
> > > return new mesos::internal::logger::LogrotateContainerLogger(
> > flags);
> > > });
> > >
> >
>
Re: two questions about mesos module
Posted by liang tang <li...@gmail.com>.
I think
https://github.com/apache/mesos/blob/master/src/slave/containerizer/mesos/io/switchboard.cpp#L278-L289
is the real entry where prepare function in container logger will be
invoked.
I wrote a module which sends stderr and stdout to graylog. If graylog got
stuck, service in Marathon even cannot run successfully, at the same time
we cannot kill the stuck task in Marathon.
How can I deal with the stuck process in mesos module not having any bad
effect on mesos services?
2017-06-02 9:35 GMT+08:00 Joseph Wu <jo...@mesosphere.io>:
> 1) The entry point for the container logger module is here:
> https://github.com/apache/mesos/blob/1.3.x/src/slave/
> container_logger.cpp#L42-L52
>
> All modules have a "create" method which attempts to load the dynamic
> library containing the module. The --modules flag you specify at
> agent/master startup time tells the agent/master where to find the library
> and what the symbol name to load is.
>
> For other invocations of the module(s), you'll need to look for places in
> the code where the module interface is called. The precise module class
> obviously won't be referenced directly within the codebase (because they're
> loaded at runtime, rather than at compile time).
>
> 2) Modules are loaded precisely so that they have an influence (otherwise,
> why load a module at all?). If you don't like the side effects of a
> module, your choices are to either disable the module, or to tweak the
> module yourself and reload it.
>
>
> On Tue, May 30, 2017 at 8:38 PM, liang tang <li...@gmail.com>
> wrote:
>
> > 1. Cannot find mesos module entry where module will start to run? didn't
> > find in slave/main.cc or module/mananger.cc
> >
> > 2. Why process in mesos module exits, service cannot be run in Marathon?
> > In my option, module shouldn't have an influence on service running.
> >
> > Which code will invoke code about container logger module in
> > https://github.com/apache/mesos/blob/master/src/slave/
> > container_loggers/lib_logrotate.cpp#L309-L339
> >
> >
> >
> > mesos::modules::Module<ContainerLogger>
> > org_apache_mesos_LogrotateContainerLogger(
> > MESOS_MODULE_API_VERSION,
> > MESOS_VERSION,
> > "Apache Mesos",
> > "modules@mesos.apache.org",
> > "Logrotate Container Logger module.",
> > nullptr,
> > [](const Parameters& parameters) -> ContainerLogger* {
> > // Convert `parameters` into a map.
> > std::map<std::string, std::string> values;
> > foreach (const Parameter& parameter, parameters.parameter()) {
> > values[parameter.key()] = parameter.value();
> > }
> >
> > // Load and validate flags from the map.
> > mesos::internal::logger::Flags flags;
> > Try<flags::Warnings> load = flags.load(values);
> >
> > if (load.isError()) {
> > LOG(ERROR) << "Failed to parse parameters: " << load.error();
> > return nullptr;
> > }
> >
> > // Log any flag warnings.
> > foreach (const flags::Warning& warning, load->warnings) {
> > LOG(WARNING) << warning.message;
> > }
> >
> > return new mesos::internal::logger::LogrotateContainerLogger(
> flags);
> > });
> >
>
Re: two questions about mesos module
Posted by Joseph Wu <jo...@mesosphere.io>.
1) The entry point for the container logger module is here:
https://github.com/apache/mesos/blob/1.3.x/src/slave/container_logger.cpp#L42-L52
All modules have a "create" method which attempts to load the dynamic
library containing the module. The --modules flag you specify at
agent/master startup time tells the agent/master where to find the library
and what the symbol name to load is.
For other invocations of the module(s), you'll need to look for places in
the code where the module interface is called. The precise module class
obviously won't be referenced directly within the codebase (because they're
loaded at runtime, rather than at compile time).
2) Modules are loaded precisely so that they have an influence (otherwise,
why load a module at all?). If you don't like the side effects of a
module, your choices are to either disable the module, or to tweak the
module yourself and reload it.
On Tue, May 30, 2017 at 8:38 PM, liang tang <li...@gmail.com>
wrote:
> 1. Cannot find mesos module entry where module will start to run? didn't
> find in slave/main.cc or module/mananger.cc
>
> 2. Why process in mesos module exits, service cannot be run in Marathon?
> In my option, module shouldn't have an influence on service running.
>
> Which code will invoke code about container logger module in
> https://github.com/apache/mesos/blob/master/src/slave/
> container_loggers/lib_logrotate.cpp#L309-L339
>
>
>
> mesos::modules::Module<ContainerLogger>
> org_apache_mesos_LogrotateContainerLogger(
> MESOS_MODULE_API_VERSION,
> MESOS_VERSION,
> "Apache Mesos",
> "modules@mesos.apache.org",
> "Logrotate Container Logger module.",
> nullptr,
> [](const Parameters& parameters) -> ContainerLogger* {
> // Convert `parameters` into a map.
> std::map<std::string, std::string> values;
> foreach (const Parameter& parameter, parameters.parameter()) {
> values[parameter.key()] = parameter.value();
> }
>
> // Load and validate flags from the map.
> mesos::internal::logger::Flags flags;
> Try<flags::Warnings> load = flags.load(values);
>
> if (load.isError()) {
> LOG(ERROR) << "Failed to parse parameters: " << load.error();
> return nullptr;
> }
>
> // Log any flag warnings.
> foreach (const flags::Warning& warning, load->warnings) {
> LOG(WARNING) << warning.message;
> }
>
> return new mesos::internal::logger::LogrotateContainerLogger(flags);
> });
>