Problems with mgmt API unix socket ownership

[David Carlin <dc...@yahoo-inc.com>]

Since switching to 3.1.3 I've been seeing the following errors in my logs
when starting trafficserver:

[Apr 26 23:57:01.823] Manager {0x7f49fd072700} NOTE: [newUNIXsocket]
Unable to chmod unix-domain socket: Operation not permitted
[Apr 26 23:57:01.823] Manager {0x7f49fd072700} NOTE: [WebIntrMain] Unable
to set up socket for handling managment API calls. API socket path =
/var/run/trafficserver/mgmtapisocket

This also impacts /var/run/trafficserver/eventapisocket

I have specified proxy.config.admin.user_id = nobody in records.config.  I
am using trafficserver script to start ATS, but I saw the same error
message above intermittently running "traffic_manager" directly as a test.

Basically, 2/3rds of the time one of these sockets ends up being owned by
root upon startup ­ only about 1 in 3 times that I start ATS is the
ownership correct.  This results in problems such as:

$ traffic_line -s proxy.config.http.keep_alive_no_activity_timeout_in -v 60
[connect] ERROR (main_socket_fd 3): Connection refused
traffic_line: Please correct your variable name and|or value

I'll then restart ATS, the ownership of the socket changes and
traffic_line works again.

I switched back to 3.0.4 and restarted ATS tens of times - the ownership
of mgmtapisocket and eventapisocket is always nobody - I never saw it
switch to root.

The v3 docs say that proxy.config.admin.user_id is "also used to specify
ownership of config and log files."  I don't know if this is part of the
problem or related in any way, but I noticed all of the logs are owned by
nobody except for traffic.out, traffic_server.stderr,
traffic_server.stdout - this is true in 3.1.3 or 3.0.4.  Again, not really
a problem.. But possible related to my issue?  Something is setting the
ownership incorrectly?

Anyone else seeing this in 3.1.3?  Thanks!

David

Re: Problems with mgmt API unix socket ownership

[Leif Hedstrom <zw...@apache.org>]

On 5/14/12 4:43 PM, David Carlin wrote:
> I noticed something else interesting while trying to troubleshoot this
> problem today:
>

Bryan Call noticed a similar problem. Out of curiosity, do you have libcap
and libcap-devel (or _dev) installed? The problem would most likely go away
with this library available. (Note that the devel package has to be
installed, so that configure can find the include file).

Fwiw, we always recommend compiling with libcap when possible.

-- Leif

Re: Problems with mgmt API unix socket ownership

[David Carlin <dc...@yahoo-inc.com>]

I noticed something else interesting while trying to troubleshoot this
problem today:

[May 14 22:24:24.023] Manager {0x7f99ebf397e0} NOTE:
[ClusterCom::ClusterCom] Node running on OS: 'Linux' Release:
'2.6.32-220.4.2.el6.YAHOO.201202'
[May 14 22:24:24.024] Manager {0x7f99ebf397e0} NOTE:
[LocalManager::listenForProxy] Listening on port: 80
[May 14 22:24:24.024] Manager {0x7f99ebf397e0} NOTE:
[LocalManager::listenForProxy] Listening on port: 81
[May 14 22:24:24.024] Manager {0x7f99ea8bf700} NOTE: [newUNIXsocket]
Unable to chmod unix-domain socket: Operation not permitted
[May 14 22:24:24.024] Manager {0x7f99ea8bf700} NOTE: [WebIntrMain] Unable
to set up so for handling managment API event calls. Event Socket path:
/var/run/trafficserver/eventapisocket
[May 14 22:24:24.024] Manager {0x7f99ebf397e0} NOTE:
[LocalManager::listenForProxy] Listening on port: 82
[May 14 22:24:24.024] Manager {0x7f99ebf397e0} NOTE:
[LocalManager::listenForProxy] Listening on port: 83
[May 14 22:24:24.025] Manager {0x7f99ebf397e0} NOTE:
[LocalManager::listenForProxy] Listening on port: 84
[May 14 22:24:24.025] Manager {0x7f99ebf397e0} NOTE:
[LocalManager::listenForProxy] Listening on port: 85
[May 14 22:24:24.026] Manager {0x7f99ebf397e0} NOTE:
[LocalManager::listenForProxy] Listening on port: 443
[May 14 22:24:24.026] Manager {0x7f99ebf397e0} NOTE:
[LocalManager::listenForProxy] Listening on port: 444
[May 14 22:24:24.026] Manager {0x7f99ebf397e0} NOTE:
[LocalManager::listenForProxy] Listening on port: 445
[May 14 22:24:24.026] Manager {0x7f99ebf397e0} NOTE:
[LocalManager::listenForProxy] Listening on port: 446
[May 14 22:24:24.026] Manager {0x7f99ebf397e0} NOTE:
[LocalManager::listenForProxy] Listening on port: 447
[May 14 22:24:24.026] Manager {0x7f99ebf397e0} NOTE:
[LocalManager::listenForProxy] Listening on port: 448


The error messages for the unix socket create appear in the middle of the
log msgs for opening the ports ATS listens on.  I changed
proxy.config.http.server_ports from my usual '80 81 82 83 84 85 443:ssl
444:ssl 445:ssl 446:ssl 447:ssl 448:ssl' to simply "80" then the problem
creating the unix socket goes away!

David

On 5/11/12 6:06 PM, "David Carlin" <dc...@yahoo-inc.com> wrote:

>Since switching to 3.1.3 I've been seeing the following errors in my logs
>when starting trafficserver:
>
>[Apr 26 23:57:01.823] Manager {0x7f49fd072700} NOTE: [newUNIXsocket]
>Unable to chmod unix-domain socket: Operation not permitted
>[Apr 26 23:57:01.823] Manager {0x7f49fd072700} NOTE: [WebIntrMain] Unable
>to set up socket for handling managment API calls. API socket path =
>/var/run/trafficserver/mgmtapisocket
>
>This also impacts /var/run/trafficserver/eventapisocket
>
>I have specified proxy.config.admin.user_id = nobody in records.config.  I
>am using trafficserver script to start ATS, but I saw the same error
>message above intermittently running "traffic_manager" directly as a test.
>
>Basically, 2/3rds of the time one of these sockets ends up being owned by
>root upon startup ­ only about 1 in 3 times that I start ATS is the
>ownership correct.  This results in problems such as:
>
>$ traffic_line -s proxy.config.http.keep_alive_no_activity_timeout_in -v
>60
>[connect] ERROR (main_socket_fd 3): Connection refused
>traffic_line: Please correct your variable name and|or value
>
>I'll then restart ATS, the ownership of the socket changes and
>traffic_line works again.
>
>I switched back to 3.0.4 and restarted ATS tens of times - the ownership
>of mgmtapisocket and eventapisocket is always nobody - I never saw it
>switch to root.
>
>The v3 docs say that proxy.config.admin.user_id is "also used to specify
>ownership of config and log files."  I don't know if this is part of the
>problem or related in any way, but I noticed all of the logs are owned by
>nobody except for traffic.out, traffic_server.stderr,
>traffic_server.stdout - this is true in 3.1.3 or 3.0.4.  Again, not really
>a problem.. But possible related to my issue?  Something is setting the
>ownership incorrectly?
>
>Anyone else seeing this in 3.1.3?  Thanks!
>
>David
>