You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by Alan Carroll <so...@verizonmedia.com> on 2020/05/20 18:08:56 UTC
Converting "records.config" to YAML
If "records.config" is changed to be YAML for ATS 10, there are two
reasonable approaches to changing it.
Option 1) Use a flat namespace. The file would look something like
config:
proxy.http.cache: true
proxy.http.insert_request_via_str: 2
proxy.http.chunking_enabled: true
proxy.dns.resolv_conf: "/etc/resolv.conf"
# .... etc.
Option 2) Use a tree. The file would look something like
config:
proxy:
http:
cache: true
insert_request_via: 2
chunking_enabled: true
dns:
resolv_conf: "/etc/resolv.conf"
From an automation point of view these are not really different - there is
an obvious isomorphism between them such that converting between them is
trivial.
Any comments?
Re: Converting "records.config" to YAML
Posted by Alan Carroll <so...@verizonmedia.com>.
Heh, it occurs to me we could use "/" as the separator to be like "/proc".
Re: Converting "records.config" to YAML
Posted by Robert O Butts <ro...@apache.org>.
I'm still flat namespace. But I'm like, +0.1 on it.
On Wed, May 20, 2020 at 3:36 PM Alan Carroll <
solidwallofcode@verizonmedia.com> wrote:
> Dang, you guys were all "flat name space!" at the hackathon.
>
> As you can see from my example, I did clean up the name space a bit and
> removed a "config" from those (e.g. "proxy.config.http.cache" is the
> current name). We could remove "proxy" as well.
>
> Sudheer's question is a good one that I"ve been pondering.
>
> 1) This would only be "records.config". It's too much to convert
> everything to YAML, but every file we change is one step closer to full
> YAML. ("Never go full YAML" - Leif)
>
> 2) Overrides would need to use the full name. For the flat style this is
> trivial because that's the same as the key name. For the tree, we'd need a
> notation to indicate descent along the objects. A simple rule would be to
> forbid a name to contain "." and then use "." as a separator that indicates
> to descend. Then the names would again be exactly as they are now and no
> changes would be needed. This isn't really any different from file system
> paths or FQDNs, so I don't think it will be a large hurdle for ATS ops
> people.
>
> On Wed, May 20, 2020 at 3:22 PM Leif Hedstrom <zw...@apache.org> wrote:
>
>> IF we are going down this path, we should restructure the configuration
>> name space now too. A lot of things makes no sense any more, including the
>> proxy. prefix.
>>
>> I do agree with Randal though, we should use proper YAML structure for
>> the name spaces, for both configurations and metrics.
>>
>> — leif
>>
>>
>> On May 20, 2020, at 1:18 PM, Randall Meyer <ra...@yahoo.com>
>> wrote:
>>
>> I vote for #2, though I could live with either.
>>
>> -r
>>
>> On Wednesday, May 20, 2020, 11:09:25 AM PDT, Alan Carroll <
>> solidwallofcode@verizonmedia.com> wrote:
>>
>>
>> If "records.config" is changed to be YAML for ATS 10, there are two
>> reasonable approaches to changing it.
>>
>> Option 1) Use a flat namespace. The file would look something like
>>
>> config:
>> proxy.http.cache: true
>> proxy.http.insert_request_via_str: 2
>> proxy.http.chunking_enabled: true
>> proxy.dns.resolv_conf: "/etc/resolv.conf"
>> # .... etc.
>>
>> Option 2) Use a tree. The file would look something like
>>
>> config:
>> proxy:
>> http:
>> cache: true
>> insert_request_via: 2
>> chunking_enabled: true
>> dns:
>> resolv_conf: "/etc/resolv.conf"
>>
>> From an automation point of view these are not really different - there
>> is an obvious isomorphism between them such that converting between them is
>> trivial.
>>
>> Any comments?
>>
>>
>>
Re: Converting "records.config" to YAML
Posted by Alan Carroll <so...@verizonmedia.com>.
That sort of translation is one of the functions for which I designed the
class Lexicon. The stuff there now is a hacky precursor. Hooking an
instance up to a configuration variable is the point of the "MgmtConverter"
logic.
Documentation - http://docs.solidwallofcode.com/libswoc/code/Lexicon.en.html
On Wed, May 20, 2020 at 6:19 PM Leif Hedstrom <zw...@apache.org> wrote:
>
>
> On May 20, 2020, at 4:07 PM, Robert O Butts <ro...@apache.org> wrote:
>
> While we're changing it, thoughts on descriptive text instead of numbers
> for enums? `insert_request_via: basic_transaction_codes` instead of
> `insert_request_via: 2` ?
>
>
>
> Yeh, that’s something someone has been asking for for a long time. The
> complication is that this means some more universal parsing mechanism, such
> that the overridable configurations can call the right “parser” for each
> configuration. Very doable, just more complex C++17 mad code. Some of it
> will be easy (“on” / “off”), but others are not (as your example). Lets not
> write box in the configuration values though :).
>
> This would be the time to do that as well, break records.config once, and
> be done with it.
>
> — Leif
>
>
>
> On Wed, May 20, 2020 at 3:52 PM Leif Hedstrom <zw...@apache.org> wrote:
>
>>
>>
>> On May 20, 2020, at 3:35 PM, Alan Carroll <
>> solidwallofcode@verizonmedia.com> wrote:
>>
>> Dang, you guys were all "flat name space!" at the hackathon.
>>
>> As you can see from my example, I did clean up the name space a bit and
>> removed a "config" from those (e.g. "proxy.config.http.cache" is the
>> current name). We could remove "proxy" as well.
>>
>>
>>
>> Pretty sure I would not have agreed to keep the existing name space. I
>> don’t consider the one we have “flat” though, it’s definitely a hierarchy.
>> However, that hierarchy is not well designed, definitely not well
>> maintained and absolutely not consistent.
>>
>> My suggestion is, when we do this, lets also clean this up. As a bonus,
>> it’s likely that the same hierarchy could be incorporated into how we use
>> Debug(), to make it a lot more easy to trace on some particular portion of
>> the feature/configuration hierarchy, and have it do so reliably. The bonus
>> here then is that we have one name space to maintain, and document.
>>
>> — Leif
>>
>>
>> Sudheer's question is a good one that I"ve been pondering.
>>
>> 1) This would only be "records.config". It's too much to convert
>> everything to YAML, but every file we change is one step closer to full
>> YAML. ("Never go full YAML" - Leif)
>>
>> 2) Overrides would need to use the full name. For the flat style this is
>> trivial because that's the same as the key name. For the tree, we'd need a
>> notation to indicate descent along the objects. A simple rule would be to
>> forbid a name to contain "." and then use "." as a separator that indicates
>> to descend. Then the names would again be exactly as they are now and no
>> changes would be needed. This isn't really any different from file system
>> paths or FQDNs, so I don't think it will be a large hurdle for ATS ops
>> people.
>>
>> On Wed, May 20, 2020 at 3:22 PM Leif Hedstrom <zw...@apache.org> wrote:
>>
>>> IF we are going down this path, we should restructure the configuration
>>> name space now too. A lot of things makes no sense any more, including the
>>> proxy. prefix.
>>>
>>> I do agree with Randal though, we should use proper YAML structure for
>>> the name spaces, for both configurations and metrics.
>>>
>>> — leif
>>>
>>>
>>> On May 20, 2020, at 1:18 PM, Randall Meyer <ra...@yahoo.com>
>>> wrote:
>>>
>>> I vote for #2, though I could live with either.
>>>
>>> -r
>>>
>>> On Wednesday, May 20, 2020, 11:09:25 AM PDT, Alan Carroll <
>>> solidwallofcode@verizonmedia.com> wrote:
>>>
>>>
>>> If "records.config" is changed to be YAML for ATS 10, there are two
>>> reasonable approaches to changing it.
>>>
>>> Option 1) Use a flat namespace. The file would look something like
>>>
>>> config:
>>> proxy.http.cache: true
>>> proxy.http.insert_request_via_str: 2
>>> proxy.http.chunking_enabled: true
>>> proxy.dns.resolv_conf: "/etc/resolv.conf"
>>> # .... etc.
>>>
>>> Option 2) Use a tree. The file would look something like
>>>
>>> config:
>>> proxy:
>>> http:
>>> cache: true
>>> insert_request_via: 2
>>> chunking_enabled: true
>>> dns:
>>> resolv_conf: "/etc/resolv.conf"
>>>
>>> From an automation point of view these are not really different - there
>>> is an obvious isomorphism between them such that converting between them is
>>> trivial.
>>>
>>> Any comments?
>>>
>>>
>>>
>>
>
Re: Converting "records.config" to YAML
Posted by Leif Hedstrom <zw...@apache.org>.
> On May 20, 2020, at 4:07 PM, Robert O Butts <ro...@apache.org> wrote:
>
> While we're changing it, thoughts on descriptive text instead of numbers for enums? `insert_request_via: basic_transaction_codes` instead of `insert_request_via: 2` ?
Yeh, that’s something someone has been asking for for a long time. The complication is that this means some more universal parsing mechanism, such that the overridable configurations can call the right “parser” for each configuration. Very doable, just more complex C++17 mad code. Some of it will be easy (“on” / “off”), but others are not (as your example). Lets not write box in the configuration values though :).
This would be the time to do that as well, break records.config once, and be done with it.
— Leif
>
>
> On Wed, May 20, 2020 at 3:52 PM Leif Hedstrom <zwoop@apache.org <ma...@apache.org>> wrote:
>
>
>> On May 20, 2020, at 3:35 PM, Alan Carroll <solidwallofcode@verizonmedia.com <ma...@verizonmedia.com>> wrote:
>>
>> Dang, you guys were all "flat name space!" at the hackathon.
>>
>> As you can see from my example, I did clean up the name space a bit and removed a "config" from those (e.g. "proxy.config.http.cache" is the current name). We could remove "proxy" as well.
>
>
> Pretty sure I would not have agreed to keep the existing name space. I don’t consider the one we have “flat” though, it’s definitely a hierarchy. However, that hierarchy is not well designed, definitely not well maintained and absolutely not consistent.
>
> My suggestion is, when we do this, lets also clean this up. As a bonus, it’s likely that the same hierarchy could be incorporated into how we use Debug(), to make it a lot more easy to trace on some particular portion of the feature/configuration hierarchy, and have it do so reliably. The bonus here then is that we have one name space to maintain, and document.
>
> — Leif
>>
>> Sudheer's question is a good one that I"ve been pondering.
>>
>> 1) This would only be "records.config". It's too much to convert everything to YAML, but every file we change is one step closer to full YAML. ("Never go full YAML" - Leif)
>>
>> 2) Overrides would need to use the full name. For the flat style this is trivial because that's the same as the key name. For the tree, we'd need a notation to indicate descent along the objects. A simple rule would be to forbid a name to contain "." and then use "." as a separator that indicates to descend. Then the names would again be exactly as they are now and no changes would be needed. This isn't really any different from file system paths or FQDNs, so I don't think it will be a large hurdle for ATS ops people.
>>
>> On Wed, May 20, 2020 at 3:22 PM Leif Hedstrom <zwoop@apache.org <ma...@apache.org>> wrote:
>> IF we are going down this path, we should restructure the configuration name space now too. A lot of things makes no sense any more, including the proxy. prefix.
>>
>> I do agree with Randal though, we should use proper YAML structure for the name spaces, for both configurations and metrics.
>>
>> — leif
>>
>>
>>> On May 20, 2020, at 1:18 PM, Randall Meyer <randallmeyer@yahoo.com <ma...@yahoo.com>> wrote:
>>>
>>> I vote for #2, though I could live with either.
>>>
>>> -r
>>>
>>> On Wednesday, May 20, 2020, 11:09:25 AM PDT, Alan Carroll <solidwallofcode@verizonmedia.com <ma...@verizonmedia.com>> wrote:
>>>
>>>
>>> If "records.config" is changed to be YAML for ATS 10, there are two reasonable approaches to changing it.
>>>
>>> Option 1) Use a flat namespace. The file would look something like
>>>
>>> config:
>>> proxy.http.cache: true
>>> proxy.http.insert_request_via_str: 2
>>> proxy.http.chunking_enabled: true
>>> proxy.dns.resolv_conf: "/etc/resolv.conf"
>>> # .... etc.
>>>
>>> Option 2) Use a tree. The file would look something like
>>>
>>> config:
>>> proxy:
>>> http:
>>> cache: true
>>> insert_request_via: 2
>>> chunking_enabled: true
>>> dns:
>>> resolv_conf: "/etc/resolv.conf"
>>>
>>> From an automation point of view these are not really different - there is an obvious isomorphism between them such that converting between them is trivial.
>>>
>>> Any comments?
>>>
>>
>
Re: Converting "records.config" to YAML
Posted by Robert O Butts <ro...@apache.org>.
While we're changing it, thoughts on descriptive text instead of numbers
for enums? `insert_request_via: basic_transaction_codes` instead of
`insert_request_via: 2` ?
On Wed, May 20, 2020 at 3:52 PM Leif Hedstrom <zw...@apache.org> wrote:
>
>
> On May 20, 2020, at 3:35 PM, Alan Carroll <
> solidwallofcode@verizonmedia.com> wrote:
>
> Dang, you guys were all "flat name space!" at the hackathon.
>
> As you can see from my example, I did clean up the name space a bit and
> removed a "config" from those (e.g. "proxy.config.http.cache" is the
> current name). We could remove "proxy" as well.
>
>
>
> Pretty sure I would not have agreed to keep the existing name space. I
> don’t consider the one we have “flat” though, it’s definitely a hierarchy.
> However, that hierarchy is not well designed, definitely not well
> maintained and absolutely not consistent.
>
> My suggestion is, when we do this, lets also clean this up. As a bonus,
> it’s likely that the same hierarchy could be incorporated into how we use
> Debug(), to make it a lot more easy to trace on some particular portion of
> the feature/configuration hierarchy, and have it do so reliably. The bonus
> here then is that we have one name space to maintain, and document.
>
> — Leif
>
>
> Sudheer's question is a good one that I"ve been pondering.
>
> 1) This would only be "records.config". It's too much to convert
> everything to YAML, but every file we change is one step closer to full
> YAML. ("Never go full YAML" - Leif)
>
> 2) Overrides would need to use the full name. For the flat style this is
> trivial because that's the same as the key name. For the tree, we'd need a
> notation to indicate descent along the objects. A simple rule would be to
> forbid a name to contain "." and then use "." as a separator that indicates
> to descend. Then the names would again be exactly as they are now and no
> changes would be needed. This isn't really any different from file system
> paths or FQDNs, so I don't think it will be a large hurdle for ATS ops
> people.
>
> On Wed, May 20, 2020 at 3:22 PM Leif Hedstrom <zw...@apache.org> wrote:
>
>> IF we are going down this path, we should restructure the configuration
>> name space now too. A lot of things makes no sense any more, including the
>> proxy. prefix.
>>
>> I do agree with Randal though, we should use proper YAML structure for
>> the name spaces, for both configurations and metrics.
>>
>> — leif
>>
>>
>> On May 20, 2020, at 1:18 PM, Randall Meyer <ra...@yahoo.com>
>> wrote:
>>
>> I vote for #2, though I could live with either.
>>
>> -r
>>
>> On Wednesday, May 20, 2020, 11:09:25 AM PDT, Alan Carroll <
>> solidwallofcode@verizonmedia.com> wrote:
>>
>>
>> If "records.config" is changed to be YAML for ATS 10, there are two
>> reasonable approaches to changing it.
>>
>> Option 1) Use a flat namespace. The file would look something like
>>
>> config:
>> proxy.http.cache: true
>> proxy.http.insert_request_via_str: 2
>> proxy.http.chunking_enabled: true
>> proxy.dns.resolv_conf: "/etc/resolv.conf"
>> # .... etc.
>>
>> Option 2) Use a tree. The file would look something like
>>
>> config:
>> proxy:
>> http:
>> cache: true
>> insert_request_via: 2
>> chunking_enabled: true
>> dns:
>> resolv_conf: "/etc/resolv.conf"
>>
>> From an automation point of view these are not really different - there
>> is an obvious isomorphism between them such that converting between them is
>> trivial.
>>
>> Any comments?
>>
>>
>>
>
Re: Converting "records.config" to YAML
Posted by Leif Hedstrom <zw...@apache.org>.
> On May 20, 2020, at 3:35 PM, Alan Carroll <so...@verizonmedia.com> wrote:
>
> Dang, you guys were all "flat name space!" at the hackathon.
>
> As you can see from my example, I did clean up the name space a bit and removed a "config" from those (e.g. "proxy.config.http.cache" is the current name). We could remove "proxy" as well.
Pretty sure I would not have agreed to keep the existing name space. I don’t consider the one we have “flat” though, it’s definitely a hierarchy. However, that hierarchy is not well designed, definitely not well maintained and absolutely not consistent.
My suggestion is, when we do this, lets also clean this up. As a bonus, it’s likely that the same hierarchy could be incorporated into how we use Debug(), to make it a lot more easy to trace on some particular portion of the feature/configuration hierarchy, and have it do so reliably. The bonus here then is that we have one name space to maintain, and document.
— Leif
>
> Sudheer's question is a good one that I"ve been pondering.
>
> 1) This would only be "records.config". It's too much to convert everything to YAML, but every file we change is one step closer to full YAML. ("Never go full YAML" - Leif)
>
> 2) Overrides would need to use the full name. For the flat style this is trivial because that's the same as the key name. For the tree, we'd need a notation to indicate descent along the objects. A simple rule would be to forbid a name to contain "." and then use "." as a separator that indicates to descend. Then the names would again be exactly as they are now and no changes would be needed. This isn't really any different from file system paths or FQDNs, so I don't think it will be a large hurdle for ATS ops people.
>
> On Wed, May 20, 2020 at 3:22 PM Leif Hedstrom <zwoop@apache.org <ma...@apache.org>> wrote:
> IF we are going down this path, we should restructure the configuration name space now too. A lot of things makes no sense any more, including the proxy. prefix.
>
> I do agree with Randal though, we should use proper YAML structure for the name spaces, for both configurations and metrics.
>
> — leif
>
>
>> On May 20, 2020, at 1:18 PM, Randall Meyer <randallmeyer@yahoo.com <ma...@yahoo.com>> wrote:
>>
>> I vote for #2, though I could live with either.
>>
>> -r
>>
>> On Wednesday, May 20, 2020, 11:09:25 AM PDT, Alan Carroll <solidwallofcode@verizonmedia.com <ma...@verizonmedia.com>> wrote:
>>
>>
>> If "records.config" is changed to be YAML for ATS 10, there are two reasonable approaches to changing it.
>>
>> Option 1) Use a flat namespace. The file would look something like
>>
>> config:
>> proxy.http.cache: true
>> proxy.http.insert_request_via_str: 2
>> proxy.http.chunking_enabled: true
>> proxy.dns.resolv_conf: "/etc/resolv.conf"
>> # .... etc.
>>
>> Option 2) Use a tree. The file would look something like
>>
>> config:
>> proxy:
>> http:
>> cache: true
>> insert_request_via: 2
>> chunking_enabled: true
>> dns:
>> resolv_conf: "/etc/resolv.conf"
>>
>> From an automation point of view these are not really different - there is an obvious isomorphism between them such that converting between them is trivial.
>>
>> Any comments?
>>
>
Re: Converting "records.config" to YAML
Posted by Alan Carroll <so...@verizonmedia.com>.
Dang, you guys were all "flat name space!" at the hackathon.
As you can see from my example, I did clean up the name space a bit and
removed a "config" from those (e.g. "proxy.config.http.cache" is the
current name). We could remove "proxy" as well.
Sudheer's question is a good one that I"ve been pondering.
1) This would only be "records.config". It's too much to convert everything
to YAML, but every file we change is one step closer to full YAML. ("Never
go full YAML" - Leif)
2) Overrides would need to use the full name. For the flat style this is
trivial because that's the same as the key name. For the tree, we'd need a
notation to indicate descent along the objects. A simple rule would be to
forbid a name to contain "." and then use "." as a separator that indicates
to descend. Then the names would again be exactly as they are now and no
changes would be needed. This isn't really any different from file system
paths or FQDNs, so I don't think it will be a large hurdle for ATS ops
people.
On Wed, May 20, 2020 at 3:22 PM Leif Hedstrom <zw...@apache.org> wrote:
> IF we are going down this path, we should restructure the configuration
> name space now too. A lot of things makes no sense any more, including the
> proxy. prefix.
>
> I do agree with Randal though, we should use proper YAML structure for the
> name spaces, for both configurations and metrics.
>
> — leif
>
>
> On May 20, 2020, at 1:18 PM, Randall Meyer <ra...@yahoo.com> wrote:
>
> I vote for #2, though I could live with either.
>
> -r
>
> On Wednesday, May 20, 2020, 11:09:25 AM PDT, Alan Carroll <
> solidwallofcode@verizonmedia.com> wrote:
>
>
> If "records.config" is changed to be YAML for ATS 10, there are two
> reasonable approaches to changing it.
>
> Option 1) Use a flat namespace. The file would look something like
>
> config:
> proxy.http.cache: true
> proxy.http.insert_request_via_str: 2
> proxy.http.chunking_enabled: true
> proxy.dns.resolv_conf: "/etc/resolv.conf"
> # .... etc.
>
> Option 2) Use a tree. The file would look something like
>
> config:
> proxy:
> http:
> cache: true
> insert_request_via: 2
> chunking_enabled: true
> dns:
> resolv_conf: "/etc/resolv.conf"
>
> From an automation point of view these are not really different - there is
> an obvious isomorphism between them such that converting between them is
> trivial.
>
> Any comments?
>
>
>
Re: Converting "records.config" to YAML
Posted by Leif Hedstrom <zw...@apache.org>.
IF we are going down this path, we should restructure the configuration name space now too. A lot of things makes no sense any more, including the proxy. prefix.
I do agree with Randal though, we should use proper YAML structure for the name spaces, for both configurations and metrics.
— leif
> On May 20, 2020, at 1:18 PM, Randall Meyer <ra...@yahoo.com> wrote:
>
> I vote for #2, though I could live with either.
>
> -r
>
> On Wednesday, May 20, 2020, 11:09:25 AM PDT, Alan Carroll <so...@verizonmedia.com> wrote:
>
>
> If "records.config" is changed to be YAML for ATS 10, there are two reasonable approaches to changing it.
>
> Option 1) Use a flat namespace. The file would look something like
>
> config:
> proxy.http.cache: true
> proxy.http.insert_request_via_str: 2
> proxy.http.chunking_enabled: true
> proxy.dns.resolv_conf: "/etc/resolv.conf"
> # .... etc.
>
> Option 2) Use a tree. The file would look something like
>
> config:
> proxy:
> http:
> cache: true
> insert_request_via: 2
> chunking_enabled: true
> dns:
> resolv_conf: "/etc/resolv.conf"
>
> From an automation point of view these are not really different - there is an obvious isomorphism between them such that converting between them is trivial.
>
> Any comments?
>
Re: Converting "records.config" to YAML
Posted by Randall Meyer <ra...@yahoo.com>.
I vote for #2, though I could live with either.
-r
On Wednesday, May 20, 2020, 11:09:25 AM PDT, Alan Carroll <so...@verizonmedia.com> wrote:
If "records.config" is changed to be YAML for ATS 10, there are two reasonable approaches to changing it.
Option 1) Use a flat namespace. The file would look something like
config:
proxy.http.cache: true
proxy.http.insert_request_via_str: 2
proxy.http.chunking_enabled: true proxy.dns.resolv_conf: "/etc/resolv.conf" # .... etc.
Option 2) Use a tree. The file would look something like
config: proxy: http: cache: true insert_request_via: 2 chunking_enabled: true dns: resolv_conf: "/etc/resolv.conf"
From an automation point of view these are not really different - there is an obvious isomorphism between them such that converting between them is trivial.
Any comments?
Re: Converting "records.config" to YAML
Posted by Sudheer Vinukonda <su...@yahoo.com>.
Hmm..is the change limited to records.config alone? How would the overrides work? Will they use the full config name like today?
Would be nice to find a consistent mechanism between the two?
On Wednesday, May 20, 2020, 11:09:24 AM PDT, Alan Carroll <so...@verizonmedia.com> wrote:
If "records.config" is changed to be YAML for ATS 10, there are two reasonable approaches to changing it.
Option 1) Use a flat namespace. The file would look something like
config:
proxy.http.cache: true
proxy.http.insert_request_via_str: 2
proxy.http.chunking_enabled: true proxy.dns.resolv_conf: "/etc/resolv.conf" # .... etc.
Option 2) Use a tree. The file would look something like
config: proxy: http: cache: true insert_request_via: 2 chunking_enabled: true dns: resolv_conf: "/etc/resolv.conf"
From an automation point of view these are not really different - there is an obvious isomorphism between them such that converting between them is trivial.
Any comments?