[jira] [Created] (MESOS-5884) Internal virtual files path map does not update to reflect GLOG rotation

["Harpreet (JIRA)" <ji...@apache.org>]

Harpreet created MESOS-5884:
-------------------------------

             Summary: Internal virtual files path map does not update to reflect GLOG rotation 
                 Key: MESOS-5884
                 URL: https://issues.apache.org/jira/browse/MESOS-5884
             Project: Mesos
          Issue Type: Bug
    Affects Versions: 0.28.1
         Environment: CoreOS 766.5.0, Mesos 0.28.1 (DC/OS 1.7.2)
            Reporter: Harpreet
            Priority: Minor


I was trying to access Mesos Master logs on a log running cluster from the Mesos UI and was greeted with {{(FAILED TO INITIALIZE...RETRYING)}} in the log pailer. 

To debug this I hit {{/file/read?path=/master/log}} endpoint and was greeted with {{Failed to open file at 'Failed to open file at '/var/log/mesos/mesos-master.<hostname>.invalid-user.log.INFO.20160623-082236.807': No such file or directory.}}

I ssh-ed into the node and checked, sure enough there was no file named {{/var/log/mesos/mesos-master.<hostname>.invalid-user.log.INFO.20160623-082236.807}}, but there was {{/var/log/mesos/mesos-master.<hostname>.invalid-user.log.INFO.20160719-060435.807}} where the newest log lines were going. This made me suspect log rotation.

I confirmed using {{/files/debug}} to make sure that the virtual path mapping for {{/master/log}} was stale.

This was a production cluster so I did not mess with it anymore since I could access the logs by ssh-ing into the box anyways. I created another cluster with same version of DC/OS to experiment with.  This is what I did to confirm my suspicion : 

# Spun up a DC/OS 1.7.2 cluster
# Stopped `dcos-logrotate.timer` on the leading master to  eliminate this as a factor/variable
# Confirmed everything worked as expected by :
## accessing master logs from Mesos UI to verify logs streaming in
## hitting {{/files/read?path=/master/log}}
## matching {{/files/debug}} output with actual file name for logs on the box
# Flooded the master logs (using curl requests) to initiate GLOG rotation
# After GLOG rotation verified that file path mapping was not updated by
## hitting {{/files/debug}} to confirm mapping was stale
## hitting {{/files/read?path=/master/log}} lead to {{No such file or directory. error}}
## trying to access master logs from Mesos UI showed last lines from older log file rather than streaming logs. This file would have eventually gotten deleted by {{dcos-logrotate}} and lead to {{(FAILED TO INITIALIZE...RETRYING)}}
# Restarted mesos master process to notice that the mapping is updated and the logs are accessible again from the Mesos UI  

So the impact of internal virtual files path map not updating is that you would either see stale master logs or no master logs from the Mesos UI. Master process restart fixes the mapping until its broken again by GLOG rotation



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)