RFC: how to do file monitor and config file reload

[永豪 <yo...@taobao.com>]

Phil submit the TS-2014 Add an API to register a plugin config file to monitor, get me pickup this file, the file monitor and config + plugin management need a better solution, please comment.

thanks



## RFC: how to do file monitor and config file reload

    In the Apache Traffic Server, there is a file monitor system in the manager process, which will monitor the change of the config files, here we will discuss how to we should do with the file monitoring and config reloading.

### The Current Implement

* ATS config files:

    We do config file monitor, sync, backup, snap, roll-back in the manager, and the globule config file is register in 'initializeRegistry()'.


### What We Need To Manage?

    In Apache Traffic Server, the most important part is for plugins, which will expand the functions and features of the proxy, playing the killer feature compare to the competitors.

* Global config files, those are files we now covered:

        log_hosts.config
        logs_xml.config
        storage.config
        socks.config
        proxy.pac
        records.config
        vaddrs.config
        cache.config
        icp.config
        ip_allow.config
        parent.config
        remap.config
        update.config
        volume.config
        hosting.config
        congestion.config
        plugin.config
        splitdns.config
        ssl_multicert.config
        stats.config.xml
        prefetch.config

* plugin config files, plugin configs are often changed in daily usage, users should not be forced to restart the cache if they just have some minor change in one of the plugin config file:
    * plugin config file for global plugin defined in 'plugin.config'
    * plugin config file for remap defined in 'remap.config'
* plugin binary files, if there is many plugins, then you may have to deal with the plugin upgrading in production, we should handle it in the smooth way.
    * global plugin so files
    * remap plugin so files

### How To Deal With The Files?

1. monitor the change of the files, and call back for the corresponding functions
2. be able to add and remove file from the monitoring file list
3. be able to manage in the cluster env, sync and update

### What is the root problem?

* most of the plugin related config and so files is defined in the traffic_server scope while the file management is done in the traffic_manager
* unify the directory where the plugin and plugin config is stored, to make it easy to manage for cluster and other tools
* if you'd like to track all the config & so files in the cluster, how to work with operation side management tools such as 'cfengine', 'puppet' and 'chef' etc.
    * make own plugin for records.config management?
    * disable cluster management of the config files and plugin so?

### Things We Need To Prepare:
* set rules for plugin management, so files and config files
* set plugin config specs, set standards and APIs
* make it able to register files to the file management system, even in plugin
* get the common config options into records style config options, for most usage
* get it clear how to update the plugin so, config file and get it actived

Re: RFC: how to do file monitor and config file reload

[Nick Kew <ni...@webthing.com>]

On 14 Sep 2013, at 04:12, Yongming Zhao wrote:

> here is my known of the 'traffic_line -x', it is the central control of taking config update in to running, when you changed the records.config with VIM for example.
> but we have some problem here too:
> 1, there is 'traffic_line -s xx -v yy', which will take into action without any hesitate.
> 2, if you changed your plugin so, or plugin.config, it won't get in update at all, for example TS-2104.

Are you sure your problem isn't just TS-2036?

There may be merit in making it self-aware, but would that have to
involve anything more than traffic_line -x on a timer?

-- 
Nick Kew

Re: RFC: how to do file monitor and config file reload

[James Peach <jp...@apache.org>]

On Sep 15, 2013, at 11:27 AM, Leif Hedstrom <zw...@apache.org> wrote:

> On Sep 14, 2013, at 4:30 PM, James Peach <jp...@apache.org> wrote:
> 
>> On Sep 13, 2013, at 8:12 PM, Yongming Zhao <mi...@gmail.com> wrote:
>> 
>>> I'd like we can get more known of all the confusing problems we have, I don't have any conclusion or suggestion at all.
>>> 
>>> 
>>> here is my known of the 'traffic_line -x', it is the central control of taking config update in to running, when you changed the records.config with VIM for example.
>>> but we have some problem here too:
>>> 1, there is 'traffic_line -s xx -v yy', which will take into action without any hesitate.
>>> 2, if you changed your plugin so, or plugin.config, it won't get in update at all, for example TS-2104.
>>> 
>>> here, we use rpm to manage the ATS binary and plugins, we find it not so easy for plugin management too. the plugin is a lightweight change in our daily management, we'd not like to restart ATS.
>> 
>> I have some ideas about how to do a graceful restart of ATS so that plugins could be reloaded without dropping any connections. Not sure whether there is a ticket on it. I hope to work on it this year.
> 
> 
> Would that retain e.g. the RAM cache across restarts?

Not in the first version, but it's probably possible.

> I like this idea a lot, for things such as server or plugin code upgrades. But is it really the best approach for something you might do very frequently, such as reloading plugin configs?

No, it's primarily for reloading code (plugin and ATS upgrades).

> I think that better APIs and infrastructure to let all configs, including plugins, reload gracefully and at close to zero cost is a good idea too.

Yes, +10

J

Re: RFC: how to do file monitor and config file reload

[Leif Hedstrom <zw...@apache.org>]

On Sep 14, 2013, at 4:30 PM, James Peach <jp...@apache.org> wrote:

> On Sep 13, 2013, at 8:12 PM, Yongming Zhao <mi...@gmail.com> wrote:
> 
>> I'd like we can get more known of all the confusing problems we have, I don't have any conclusion or suggestion at all.
>> 
>> 
>> here is my known of the 'traffic_line -x', it is the central control of taking config update in to running, when you changed the records.config with VIM for example.
>> but we have some problem here too:
>> 1, there is 'traffic_line -s xx -v yy', which will take into action without any hesitate.
>> 2, if you changed your plugin so, or plugin.config, it won't get in update at all, for example TS-2104.
>> 
>> here, we use rpm to manage the ATS binary and plugins, we find it not so easy for plugin management too. the plugin is a lightweight change in our daily management, we'd not like to restart ATS.
> 
> I have some ideas about how to do a graceful restart of ATS so that plugins could be reloaded without dropping any connections. Not sure whether there is a ticket on it. I hope to work on it this year.


Would that retain e.g. the RAM cache across restarts?I like this idea a lot, for things such as server or plugin code upgrades. But is it really the best approach for something you might do very frequently, such as reloading plugin configs?

I think that better APIs and infrastructure to let all configs, including plugins, reload gracefully and at close to zero cost is a good idea too.

Cheers,

-- Leif 


> 
>> we found out that if we update the plugin rpm into another new version, with so file replaced, it won't take into action, because it is not changed in the inode of the filesystem. we have to do a symbol link to for it to be included in the remap.config, and name the plugin so in the standard library version naming way.
>> but the change make it impossible to first install the plugin, then active it later, it will take into action as soon as it is installed. so 'traffic_line -x' is useless here.
> 
> That sounds like a fixable problem with the reloading process.
> 
>> 
>> that is what I am wondering, is there any better solution? how should we make it for the future?
>> 
>> 
>> 在 2013-9-14，上午6:41，Nick Kew <ni...@webthing.com> 写道：
>> 
>>> On Fri, 13 Sep 2013 22:43:22 +0800
>>> 永豪 <yo...@taobao.com> wrote:
>>> 
>>> 
>>>> ## RFC: how to do file monitor and config file reload
>>> 
>>> How would you expect this to relate to "traffic_line -x"?
>>> 
>>> -- 
>>> Nick Kew
> 

Re: RFC: how to do file monitor and config file reload

[James Peach <jp...@apache.org>]

On Sep 13, 2013, at 8:12 PM, Yongming Zhao <mi...@gmail.com> wrote:

> I'd like we can get more known of all the confusing problems we have, I don't have any conclusion or suggestion at all.
> 
> 
> here is my known of the 'traffic_line -x', it is the central control of taking config update in to running, when you changed the records.config with VIM for example.
> but we have some problem here too:
> 1, there is 'traffic_line -s xx -v yy', which will take into action without any hesitate.
> 2, if you changed your plugin so, or plugin.config, it won't get in update at all, for example TS-2104.
> 
> here, we use rpm to manage the ATS binary and plugins, we find it not so easy for plugin management too. the plugin is a lightweight change in our daily management, we'd not like to restart ATS.

I have some ideas about how to do a graceful restart of ATS so that plugins could be reloaded without dropping any connections. Not sure whether there is a ticket on it. I hope to work on it this year.

> we found out that if we update the plugin rpm into another new version, with so file replaced, it won't take into action, because it is not changed in the inode of the filesystem. we have to do a symbol link to for it to be included in the remap.config, and name the plugin so in the standard library version naming way.
> but the change make it impossible to first install the plugin, then active it later, it will take into action as soon as it is installed. so 'traffic_line -x' is useless here.

That sounds like a fixable problem with the reloading process.

> 
> that is what I am wondering, is there any better solution? how should we make it for the future?
> 
> 
> 在 2013-9-14，上午6:41，Nick Kew <ni...@webthing.com> 写道：
> 
>> On Fri, 13 Sep 2013 22:43:22 +0800
>> 永豪 <yo...@taobao.com> wrote:
>> 
>> 
>>> ## RFC: how to do file monitor and config file reload
>> 
>> How would you expect this to relate to "traffic_line -x"?
>> 
>> -- 
>> Nick Kew
> 

Re: RFC: how to do file monitor and config file reload

[Yongming Zhao <mi...@gmail.com>]

I'd like we can get more known of all the confusing problems we have, I don't have any conclusion or suggestion at all.


here is my known of the 'traffic_line -x', it is the central control of taking config update in to running, when you changed the records.config with VIM for example.
but we have some problem here too:
1, there is 'traffic_line -s xx -v yy', which will take into action without any hesitate.
2, if you changed your plugin so, or plugin.config, it won't get in update at all, for example TS-2104.

here, we use rpm to manage the ATS binary and plugins, we find it not so easy for plugin management too. the plugin is a lightweight change in our daily management, we'd not like to restart ATS.
we found out that if we update the plugin rpm into another new version, with so file replaced, it won't take into action, because it is not changed in the inode of the filesystem. we have to do a symbol link to for it to be included in the remap.config, and name the plugin so in the standard library version naming way.
but the change make it impossible to first install the plugin, then active it later, it will take into action as soon as it is installed. so 'traffic_line -x' is useless here.

that is what I am wondering, is there any better solution? how should we make it for the future?


在 2013-9-14，上午6:41，Nick Kew <ni...@webthing.com> 写道：

> On Fri, 13 Sep 2013 22:43:22 +0800
> 永豪 <yo...@taobao.com> wrote:
> 
> 
>> ## RFC: how to do file monitor and config file reload
> 
> How would you expect this to relate to "traffic_line -x"?
> 
> -- 
> Nick Kew

Re: RFC: how to do file monitor and config file reload

[Nick Kew <ni...@webthing.com>]

On Fri, 13 Sep 2013 22:43:22 +0800
永豪 <yo...@taobao.com> wrote:


> ## RFC: how to do file monitor and config file reload

How would you expect this to relate to "traffic_line -x"?

-- 
Nick Kew