You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Alexander Rukletsov (JIRA)" <ji...@apache.org> on 2018/06/14 13:38:00 UTC
[jira] [Comment Edited] (MESOS-8985) Posting to the operator api
with 'accept recordio' header can crash the agent
[ https://issues.apache.org/jira/browse/MESOS-8985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16512475#comment-16512475 ]
Alexander Rukletsov edited comment on MESOS-8985 at 6/14/18 1:37 PM:
---------------------------------------------------------------------
{noformat}
commit e25385a2f9337194164786d60a5e107140a78ae6
Author: Benno Evers <be...@mesosphere.com>
AuthorDate: Thu Jun 14 15:34:42 2018 +0200
Commit: Alexander Rukletsov <al...@apache.org>
CommitDate: Thu Jun 14 15:34:42 2018 +0200
Added check for API endpoints not supporting streaming responses.
A client can request any endpoint to stream its response using
the RecordIO format. However, we never checked whether the selected
endpoint actually supports streaming, leading to an agent crash
when it was used for endpoints that did not expect such a request.
Review: https://reviews.apache.org/r/67547/
{noformat}
was (Author: alexr):
{noformat}
commit e25385a2f9337194164786d60a5e107140a78ae6
Author: Benno Evers <be...@mesosphere.com>
AuthorDate: Thu Jun 14 15:34:42 2018 +0200
Commit: Alexander Rukletsov <al...@apache.org>
CommitDate: Thu Jun 14 15:34:42 2018 +0200
Added check for API endpoints not supporting streaming responses.
A client can request any endpoint to stream its response using
the RecordIO format. However, we never checked whether the selected
endpoint actually supports streaming, leading to an agent crash
when it was used for endpoints that did not expect such a request.
Review: https://reviews.apache.org/r/67547/
[noformat}
> Posting to the operator api with 'accept recordio' header can crash the agent
> -----------------------------------------------------------------------------
>
> Key: MESOS-8985
> URL: https://issues.apache.org/jira/browse/MESOS-8985
> Project: Mesos
> Issue Type: Bug
> Components: HTTP API
> Affects Versions: 1.4.1, 1.5.1
> Reporter: Philip Norman
> Assignee: Benno Evers
> Priority: Major
> Labels: mesosphere
> Fix For: 1.7.0
>
> Attachments: mesos-slave-crash.log
>
>
> It's possible to crash the mesos agent by posting a reasonable request to the operator API.
> h3. Background:
> Sending a request to the v1 api endpoint with an unsupported 'accept' header:
> {code:java}
> curl -X POST http://10.0.3.27:5051/api/v1 \
> -H 'accept: application/atom+xml' \
> -H 'content-type: application/json' \
> -d '{"type":"GET_CONTAINERS","get_containers":{"show_nested": true,"show_standalone": true}}'{code}
> Results in the following friendly error message:
> {code:java}
> Expecting 'Accept' to allow application/json or application/x-protobuf or application/recordio{code}
> h3. Reproducible crash:
> However, sending the same request with 'application/recordio' 'accept' header:
> {code:java}
> curl -X POST \
> http://10.0.3.27:5051/api/v1 \
> -H 'accept: application/recordio' \
> -H 'content-type: application/json' \
> -d '{"type":"GET_CONTAINERS","get_containers":{"show_nested": true,"show_standalone": true}}'{code}
> causes the agent to crash (no response is received).
> Crash log is shown below, full log from the agent is attached here:
> {code:java}
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: I0607 22:30:32.397320 3743 logfmt.cpp:178] type=audit timestamp=2018-06-07 22:30:32.397243904+00:00 reason="Error in token 'Missing 'Authorization' header from HTTP request'. Allowing anonymous connection" object="/slave(1)/api/v1" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36" authorizer="mesos-agent" action="POST" result=allow srcip=10.0.6.99 dstport=5051 srcport=42084 dstip=10.0.3.27
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: W0607 22:30:32.397434 3743 authenticator.cpp:289] Error in token on request from '10.0.6.99:42084': Missing 'Authorization' header from HTTP request
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: W0607 22:30:32.397466 3743 authenticator.cpp:291] Falling back to anonymous connection using user 'dcos_anonymous'
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: I0607 22:30:32.397629 3748 http.cpp:1099] HTTP POST for /slave(1)/api/v1 from 10.0.6.99:42084 with User-Agent='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36'
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: I0607 22:30:32.397784 3748 http.cpp:2030] Processing GET_CONTAINERS call
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: F0607 22:30:32.398736 3747 http.cpp:121] Serializing a RecordIO stream is not supported
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: *** Check failure stack trace: ***
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f619478636d google::LogMessage::Fail()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f619478819d google::LogMessage::SendToLog()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6194785f5c google::LogMessage::Flush()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6194788a99 google::LogMessageFatal::~LogMessageFatal()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f61935e2b9d mesos::internal::serialize()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a4c0ef _ZNO6lambda12CallableOnceIFN7process6FutureINS1_4http8ResponseEEERKN4JSON5ArrayEEE10CallableFnIZNK5mesos8internal5slave4Http13getContainersERKNSD_5agent4CallENSD_11ContentTypeERK6OptionINS3_14authentication9PrincipalEEEUlRKNS2_IS7_EEE0_EclES9_
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a81d61 process::internal::thenf<>()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a59b15 _ZNO6lambda12CallableOnceIFvRKN7process6FutureIN4JSON5ArrayEEEEE10CallableFnINS_8internal7PartialIPFvONS0_IFNS2_INS1_4http8ResponseEEERKS4_EEESt10unique_ptrINS1_7PromiseISE_EESt14default_deleteISN_EES7_EJSJ_SQ_St12_PlaceholderILi1EEEEEEclES7_
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a6e4e9 process::internal::run<>()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a7fa28 process::Future<>::_set<>()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a7f9fe process::Future<>::_set<>()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a7f9fe process::Future<>::_set<>()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a7f9fe process::Future<>::_set<>()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a7f9fe process::Future<>::_set<>()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a84e00 process::Future<>::onReady()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a8509e process::Promise<>::associate()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a856ac process::internal::thenf<>()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a59935 _ZNO6lambda12CallableOnceIFvRKN7process6FutureISt5tupleIINS2_ISt4listINS2_IN5mesos15ContainerStatusEEESaIS7_EEEENS2_IS4_INS2_INS5_18ResourceStatisticsEEESaISC_EEEEEEEEEE10CallableFnINS_8internal7PartialIPFvONS0_IFNS2_IN4JSON5ArrayEEERKSG_EEESt10unique_ptrINS1_7PromiseISQ_EESt14default_deleteISZ_EESJ_EISV_S12_St12_PlaceholderILi1EEEEEEclESJ_
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a81359 process::internal::run<>()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a83f12 _ZN7process6FutureISt5tupleIJNS0_ISt4listINS0_IN5mesos15ContainerStatusEEESaIS5_EEEENS0_IS2_INS0_INS3_18ResourceStatisticsEEESaISA_EEEEEEE4_setIRKSE_EEbOT_
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a85f10 _ZNK7process6FutureISt5tupleIJNS0_ISt4listINS0_IN5mesos15ContainerStatusEEESaIS5_EEEENS0_IS2_INS0_INS3_18ResourceStatisticsEEESaISA_EEEEEEE7onReadyEON6lambda12CallableOnceIFvRKSE_EEE
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a861ae process::Promise<>::associate()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a866ac process::internal::thenf<>()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a59875 _ZNO6lambda12CallableOnceIFvRKN7process6FutureISt4listINS2_I7NothingEESaIS5_EEEEEE10CallableFnINS_8internal7PartialIPFvONS0_IFNS2_ISt5tupleIINS2_IS3_INS2_IN5mesos15ContainerStatusEEESaISJ_EEEENS2_IS3_INS2_INSH_18ResourceStatisticsEEESaISO_EEEEEEEERKS7_EEESt10unique_ptrINS1_7PromiseISS_EESt14default_deleteIS11_EESA_EISX_S14_St12_PlaceholderILi1EEEEEEclESA_
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f61935c1a19 process::internal::run<>()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f61935cf25f process::Future<>::_set<>()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f61935cf44b process::internal::AwaitProcess<>::waited()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f61946d79d1 process::ProcessBase::consume()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f61946e8dcc process::ProcessManager::resume()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f61946ee7a6 _ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEEE6_M_runEv
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f61918d8d73 (unknown)
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f61913d952c (unknown)
> Jun 07 22:30:34 ip-10-0-3-27.us-west-2.compute.internal systemd[1]: dcos-mesos-slave.service: Main process exited, code=killed, status=6/ABRT
> Jun 07 22:30:34 ip-10-0-3-27.us-west-2.compute.internal systemd[1]: dcos-mesos-slave.service: Unit entered failed state.
> Jun 07 22:30:34 ip-10-0-3-27.us-west-2.compute.internal systemd[1]: dcos-mesos-slave.service: Failed with result 'signal'.
> Jun 07 22:30:39 ip-10-0-3-27.us-west-2.compute.internal systemd[1]: dcos-mesos-slave.service: Service hold-off time over, scheduling restart.
> Jun 07 22:30:39 ip-10-0-3-27.us-west-2.compute.internal systemd[1]: Stopped Mesos Agent: distributed systems kernel agent.{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)