You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by Hobin Yoon <ho...@gmail.com> on 2019/01/10 06:52:40 UTC
Downtime when traffic_ctl config reload
Hi,
We are noticing there is quite a bit of delay when we reload the config
with traffic_ctl config reload. The delay is up to about 30 seconds, during
which period we don't get any caching. We are using consistency hashing
plugin. The number of nodes changes dynamically between 5 to 30.
Here is an example balancer (consistent hash) configuration in remap.config.
map / http://127.0.0.1 @plugin=cachekey.so @pparam=--include-params=p0,p1
@pparam=--sort-params=true @plugin=balancer.so @pparam=--policy=hash,key
@pparam=123.123.123.1 @pparam=123.123.123.2 @pparam=123.123.123.3
@pparam=123.123.123.4
...
Is this downtime normal? How do you guys avoid the service downtime while
reconfiguring the cache nodes in the cluster?
Hobin
Re: Downtime when traffic_ctl config reload
Posted by Hobin Yoon <ho...@gmail.com>.
With the new setup, I'm happy to report that the downtime, the 30 second
downtime we had with balancer plugin, is gone.
Hobin
On Sat, Jan 12, 2019 at 2:34 PM Hobin Yoon <ho...@gmail.com> wrote:
> dest_ip did the magic. Thanks Miles! Here is my setup for the record.
>
> On child node: (It's ip address is 100.100.100.1)
> records.config
> CONFIG proxy.config.http.parent_proxy_routing_enable INT 1
> parent.config:
> dest_domain=100.100.100.1 scheme=http parent="100.100.200.1:80,
> 100.100.200.1:80" round_robin=consistent_hash go_direct=false
>
> On parent nodes (100.100.200.1 and 100.100.200.2):
> records.config
> CONFIG proxy.config.http.parent_proxy_routing_enable INT 0
> remap.config
> map / http://(origin_ip)
>
> Hobin
>
> On Sat, Jan 12, 2019 at 12:15 PM Miles Libbey <ml...@apache.org> wrote:
>
>> A loop would indicate its not matching parent.config then. I've never
>> used an IP in a map. In
>>
>> https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/parent.config.en.html
>> it has a
>> dest_ip
>> field that would seem to make more sense than dest_domain|host for
>> that map rule.
>>
>> You'll also need a map http://52.91.93.206/ http://someorigiin on your
>> parent...
>> miles
>>
>> On Fri, Jan 11, 2019 at 10:49 PM Hobin Yoon <ho...@gmail.com> wrote:
>> >
>> > Now, I'm getting
>> >
>> > Cycle Prohibited
>> > ________________________________
>> > Description: Could not process your request for the document because it
>> would cause an HTTP proxy cycle. Please check the URL and your browser's
>> proxy settings.
>> >
>> > I think it's because remap.config has the same ip addresses ... I tried
>> to comment out the map rule, but got the same error...
>> >
>> > On child node: (It's ip address is 52.91.93.206)
>> > records.config
>> > CONFIG proxy.config.http.parent_proxy_routing_enable INT 1
>> > CONFIG proxy.config.http.cache.http INT 0
>> > remap.config:
>> > map http://52.91.93.206 http://52.91.93.206
>> > parent.config:
>> > dest_domain=52.91.93.206 scheme=http parent="18.212.133.3:80,
>> 34.207.101.159:80" round_robin=consistent_hash go_direct=false
>> >
>> > On parent nodes:
>> > records.config
>> > CONFIG proxy.config.http.parent_proxy_routing_enable INT 0
>> > CONFIG proxy.config.http.cache.http INT 1
>> >
>> > Hobin
>> >
>> > On Fri, Jan 11, 2019 at 5:31 PM Miles Libbey <ml...@apache.org>
>> wrote:
>> >>
>> >> First thing I'd check is proxy.config.http.parent_proxy_routing_enable
>> >> on the children :)
>> >>
>> >> On Fri, Jan 11, 2019 at 2:15 PM Hobin Yoon <ho...@gmail.com>
>> wrote:
>> >> >
>> >> > It is disabled. I just tried with both on and off, but none of them
>> worked. What bothers me is ATS doesn't seem to be reading parent.config at
>> all since I don't get any error when I intentionally made a typo in
>> parent.config.
>> >> >
>> >> > Hobin
>> >> >
>> >> > On Fri, Jan 11, 2019 at 4:38 PM Leif Hedstrom <zw...@apache.org>
>> wrote:
>> >> >>
>> >> >>
>> >> >>
>> >> >> On Jan 11, 2019, at 12:54 PM, Hobin Yoon <ho...@gmail.com>
>> wrote:
>> >> >>
>> >> >> Ok, I added the rule in remap.config. Now, the requests are going
>> to "origin". But, they bypasses parent nodes; I don't see any requests on
>> either of the parent nodes.
>> >> >>
>> >> >> Here is my system setup for test. "child" is a pure load balancer
>> and doesn't cache any data items. parent1 and parent2 do cache.
>> >> >> Client (web browser) - child (ATS)--- parent1 (ATS) --- origin
>> (Apache2)
>> >> >> └- parent2 (ATS) -┘
>> >> >>
>> >> >> Here is the configuration in "child" node.
>> >> >>
>> >> >> remap.config
>> >> >> map / http://origin
>> >> >>
>> >> >> parent.config
>> >> >> dest_domain=origin scheme=http parent="parent1:80;parent2:80"
>> round_robin=consistent_hash
>> >> >>
>> >> >>
>> >> >>
>> >> >> Do you have “pristine” host headers enabled? The mapping in parent
>> selection happens on the “remapped” URL, but I’m pretty sure it’d be
>> different if you have pristine host headers disabled.
>> >> >>
>> >> >> That much said, I’d run with a diagnostics (-T) on “parent”.
>> >> >>
>> >> >> — Leif
>> >> >>
>> >> >>
>> >> >>
>> >> >> records.config
>> >> >> CONFIG proxy.config.http.parent_proxy_routing_enable INT 1
>> >> >>
>> >> >> Hobin
>> >> >>
>> >> >> On Fri, Jan 11, 2019 at 1:08 PM Leif Hedstrom <zw...@apache.org>
>> wrote:
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> On Jan 11, 2019, at 10:00 AM, Hobin Yoon <ho...@gmail.com>
>> wrote:
>> >> >>>
>> >> >>> Should you still specify the mapping in remap.config when you have
>> the mapping in parent.config?
>> >> >>>
>> >> >>>
>> >> >>> Yes. Unless you run an open proxy, and then you have to turn off
>> “require remap”. Not recommended for almost all cases.
>> >> >>>
>> >> >>> — leif
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> On Fri, Jan 11, 2019 at 11:33 AM Randall Meyer <
>> randallmeyer@yahoo.com> wrote:
>> >> >>>>
>> >> >>>>
>> >> >>>> Looking at your remap.config in that repo, I see that all the
>> mappings are all commented out. Was that intentional?
>> >> >>>> On Thursday, January 10, 2019, 8:28:53 PM PST, Hobin Yoon <
>> hobinyoon@gmail.com> wrote:
>> >> >>>>
>> >> >>>>
>> >> >>>> Yes, they are also ATS nodes and they worked fine when requests
>> were made to them directly.
>> >> >>>>
>> >> >>>> I uploaded the config files here, if you could take a look:
>> https://gitlab.com/hobinyoon/trafficserver-config
>> >> >>>>
>> >> >>>> Hobin
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>> On Thu, Jan 10, 2019 at 7:31 PM Miles Libbey <ml...@apache.org>
>> wrote:
>> >> >>>>
>> >> >>>> Typically that error means the request doesn't match a rule in
>> >> >>>> remap.config. Is the error coming from the first hop or one of the
>> >> >>>> 123.123.123.[1-4] nodes? That is, if 123.123.123.[1-4] are ATS
>> nodes,
>> >> >>>> are they configured to accept the requests they are getting?
>> >> >>>>
>> >> >>>> On Thu, Jan 10, 2019 at 2:51 PM Hobin Yoon <ho...@gmail.com>
>> wrote:
>> >> >>>> >
>> >> >>>> > With parent.config
>> >> >>>> > dest_domain=. scheme=http parent="123.123.123.1:80,
>> 123.123.123.2:80" round_robin=consistent_hash go_direct=false
>> >> >>>> >
>> >> >>>> > I'm getting
>> >> >>>> >
>> >> >>>> > Not Found on Accelerator
>> >> >>>> > ________________________________
>> >> >>>> > Description: Your request on the specified host was not found.
>> Check the location and try again.
>> >> >>>> >
>> >> >>>> > I must be missing something ...
>> >> >>>> >
>> >> >>>> > Hobin
>> >> >>>> >
>> >> >>>> > On Thu, Jan 10, 2019 at 1:18 PM Hobin Yoon <ho...@gmail.com>
>> wrote:
>> >> >>>> >>
>> >> >>>> >> Alan, that is the only map rule we have with the varying
>> number of cache nodes. During the down time, ATS doesn't return "HTTP/1.1
>> 200 OK" for the requests. I didn't check what it returned.
>> >> >>>> >>
>> >> >>>> >> Miles, I'll check out parent plugin!
>> >> >>>> >>
>> >> >>>> >> Hobin
>> >> >>>> >>
>> >> >>>> >> On Thu, Jan 10, 2019 at 1:13 PM Alan Carroll <
>> solidwallofcode@oath.com> wrote:
>> >> >>>> >>>
>> >> >>>> >>> It could be an artifact of reloading plugin configurations if
>> you have a lot of remap rules with plugins, although internally ATS should
>> do the load and then swap the configuration. During the down time, does ATS
>> process any traffic, or there is traffic but no caching?
>> >> >>>> >>>
>> >> >>>> >>> On Thu, Jan 10, 2019 at 10:53 AM Miles Libbey <
>> mlibbey@apache.org> wrote:
>> >> >>>> >>>>
>> >> >>>> >>>> We don't experience downtime when using traffic_ctl config
>> reload (we
>> >> >>>> >>>> use that ~daily).
>> >> >>>> >>>>
>> >> >>>> >>>> We don't use the balancer plugin. Instead, we use
>> parent.config
>> >> >>>> >>>> (
>> https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/parent.config.en.html
>> )
>> >> >>>> >>>> to achieve the same consistent hash. Your config would
>> translate to
>> >> >>>> >>>>
>> >> >>>> >>>> remap.config
>> >> >>>> >>>> map / http://127.0.0.1 @plugin=cachekey.so
>> >> >>>> >>>> @pparam=--include-params=p0,p1 @pparam=--sort-params=true
>> >> >>>> >>>>
>> >> >>>> >>>> parent.config
>> >> >>>> >>>> dest_domain=. scheme=http
>> >> >>>> >>>> parent="123.123.123.1:80,23.123.123.2:80,123.123.123.3:80,
>> 123.123.123.4:80"
>> >> >>>> >>>> round_robin=consistent_hash go_direct=false
>> >> >>>> >>>> dest_domain=. scheme=https
>> >> >>>> >>>> parent="123.123.123.1:443,23.123.123.2:443,123.123.123.3:443
>> ,123.123.123.4:443"
>> >> >>>> >>>> round_robin=consistent_hash go_direct=false
>> >> >>>> >>>>
>> >> >>>> >>>> miles
>> >> >>>> >>>>
>> >> >>>> >>>> On Wed, Jan 9, 2019 at 10:53 PM Hobin Yoon <
>> hobinyoon@gmail.com> wrote:
>> >> >>>> >>>> >
>> >> >>>> >>>> > Hi,
>> >> >>>> >>>> >
>> >> >>>> >>>> > We are noticing there is quite a bit of delay when we
>> reload the config with traffic_ctl config reload. The delay is up to about
>> 30 seconds, during which period we don't get any caching. We are using
>> consistency hashing plugin. The number of nodes changes dynamically between
>> 5 to 30.
>> >> >>>> >>>> >
>> >> >>>> >>>> > Here is an example balancer (consistent hash)
>> configuration in remap.config.
>> >> >>>> >>>> >
>> >> >>>> >>>> > map / http://127.0.0.1 @plugin=cachekey.so
>> @pparam=--include-params=p0,p1 @pparam=--sort-params=true
>> @plugin=balancer.so @pparam=--policy=hash,key @pparam=123.123.123.1
>> @pparam=123.123.123.2 @pparam=123.123.123.3 @pparam=123.123.123.4 ...
>> >> >>>> >>>> >
>> >> >>>> >>>> > Is this downtime normal? How do you guys avoid the service
>> downtime while reconfiguring the cache nodes in the cluster?
>> >> >>>> >>>> >
>> >> >>>> >>>> > Hobin
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>> >>>> >
>> >> >>>> >>>
>> >> >>>> >>>
>> >> >>>> >>>
>> >> >>>> >>> --
>> >> >>>> >>> Beware the fisherman who's casting out his line in to a dried
>> up riverbed.
>> >> >>>> >>> Oh don't try to tell him 'cause he won't believe. Throw some
>> bread to the ducks instead.
>> >> >>>> >>> It's easier that way. - Genesis : Duke : VI 25-28
>> >> >>>
>> >> >>>
>> >> >>
>>
>
Re: Downtime when traffic_ctl config reload
Posted by Hobin Yoon <ho...@gmail.com>.
dest_ip did the magic. Thanks Miles! Here is my setup for the record.
On child node: (It's ip address is 100.100.100.1)
records.config
CONFIG proxy.config.http.parent_proxy_routing_enable INT 1
parent.config:
dest_domain=100.100.100.1 scheme=http parent="100.100.200.1:80,
100.100.200.1:80" round_robin=consistent_hash go_direct=false
On parent nodes (100.100.200.1 and 100.100.200.2):
records.config
CONFIG proxy.config.http.parent_proxy_routing_enable INT 0
remap.config
map / http://(origin_ip)
Hobin
On Sat, Jan 12, 2019 at 12:15 PM Miles Libbey <ml...@apache.org> wrote:
> A loop would indicate its not matching parent.config then. I've never
> used an IP in a map. In
>
> https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/parent.config.en.html
> it has a
> dest_ip
> field that would seem to make more sense than dest_domain|host for
> that map rule.
>
> You'll also need a map http://52.91.93.206/ http://someorigiin on your
> parent...
> miles
>
> On Fri, Jan 11, 2019 at 10:49 PM Hobin Yoon <ho...@gmail.com> wrote:
> >
> > Now, I'm getting
> >
> > Cycle Prohibited
> > ________________________________
> > Description: Could not process your request for the document because it
> would cause an HTTP proxy cycle. Please check the URL and your browser's
> proxy settings.
> >
> > I think it's because remap.config has the same ip addresses ... I tried
> to comment out the map rule, but got the same error...
> >
> > On child node: (It's ip address is 52.91.93.206)
> > records.config
> > CONFIG proxy.config.http.parent_proxy_routing_enable INT 1
> > CONFIG proxy.config.http.cache.http INT 0
> > remap.config:
> > map http://52.91.93.206 http://52.91.93.206
> > parent.config:
> > dest_domain=52.91.93.206 scheme=http parent="18.212.133.3:80,
> 34.207.101.159:80" round_robin=consistent_hash go_direct=false
> >
> > On parent nodes:
> > records.config
> > CONFIG proxy.config.http.parent_proxy_routing_enable INT 0
> > CONFIG proxy.config.http.cache.http INT 1
> >
> > Hobin
> >
> > On Fri, Jan 11, 2019 at 5:31 PM Miles Libbey <ml...@apache.org> wrote:
> >>
> >> First thing I'd check is proxy.config.http.parent_proxy_routing_enable
> >> on the children :)
> >>
> >> On Fri, Jan 11, 2019 at 2:15 PM Hobin Yoon <ho...@gmail.com> wrote:
> >> >
> >> > It is disabled. I just tried with both on and off, but none of them
> worked. What bothers me is ATS doesn't seem to be reading parent.config at
> all since I don't get any error when I intentionally made a typo in
> parent.config.
> >> >
> >> > Hobin
> >> >
> >> > On Fri, Jan 11, 2019 at 4:38 PM Leif Hedstrom <zw...@apache.org>
> wrote:
> >> >>
> >> >>
> >> >>
> >> >> On Jan 11, 2019, at 12:54 PM, Hobin Yoon <ho...@gmail.com>
> wrote:
> >> >>
> >> >> Ok, I added the rule in remap.config. Now, the requests are going to
> "origin". But, they bypasses parent nodes; I don't see any requests on
> either of the parent nodes.
> >> >>
> >> >> Here is my system setup for test. "child" is a pure load balancer
> and doesn't cache any data items. parent1 and parent2 do cache.
> >> >> Client (web browser) - child (ATS)--- parent1 (ATS) --- origin
> (Apache2)
> >> >> └- parent2 (ATS) -┘
> >> >>
> >> >> Here is the configuration in "child" node.
> >> >>
> >> >> remap.config
> >> >> map / http://origin
> >> >>
> >> >> parent.config
> >> >> dest_domain=origin scheme=http parent="parent1:80;parent2:80"
> round_robin=consistent_hash
> >> >>
> >> >>
> >> >>
> >> >> Do you have “pristine” host headers enabled? The mapping in parent
> selection happens on the “remapped” URL, but I’m pretty sure it’d be
> different if you have pristine host headers disabled.
> >> >>
> >> >> That much said, I’d run with a diagnostics (-T) on “parent”.
> >> >>
> >> >> — Leif
> >> >>
> >> >>
> >> >>
> >> >> records.config
> >> >> CONFIG proxy.config.http.parent_proxy_routing_enable INT 1
> >> >>
> >> >> Hobin
> >> >>
> >> >> On Fri, Jan 11, 2019 at 1:08 PM Leif Hedstrom <zw...@apache.org>
> wrote:
> >> >>>
> >> >>>
> >> >>>
> >> >>> On Jan 11, 2019, at 10:00 AM, Hobin Yoon <ho...@gmail.com>
> wrote:
> >> >>>
> >> >>> Should you still specify the mapping in remap.config when you have
> the mapping in parent.config?
> >> >>>
> >> >>>
> >> >>> Yes. Unless you run an open proxy, and then you have to turn off
> “require remap”. Not recommended for almost all cases.
> >> >>>
> >> >>> — leif
> >> >>>
> >> >>>
> >> >>>
> >> >>> On Fri, Jan 11, 2019 at 11:33 AM Randall Meyer <
> randallmeyer@yahoo.com> wrote:
> >> >>>>
> >> >>>>
> >> >>>> Looking at your remap.config in that repo, I see that all the
> mappings are all commented out. Was that intentional?
> >> >>>> On Thursday, January 10, 2019, 8:28:53 PM PST, Hobin Yoon <
> hobinyoon@gmail.com> wrote:
> >> >>>>
> >> >>>>
> >> >>>> Yes, they are also ATS nodes and they worked fine when requests
> were made to them directly.
> >> >>>>
> >> >>>> I uploaded the config files here, if you could take a look:
> https://gitlab.com/hobinyoon/trafficserver-config
> >> >>>>
> >> >>>> Hobin
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> On Thu, Jan 10, 2019 at 7:31 PM Miles Libbey <ml...@apache.org>
> wrote:
> >> >>>>
> >> >>>> Typically that error means the request doesn't match a rule in
> >> >>>> remap.config. Is the error coming from the first hop or one of the
> >> >>>> 123.123.123.[1-4] nodes? That is, if 123.123.123.[1-4] are ATS
> nodes,
> >> >>>> are they configured to accept the requests they are getting?
> >> >>>>
> >> >>>> On Thu, Jan 10, 2019 at 2:51 PM Hobin Yoon <ho...@gmail.com>
> wrote:
> >> >>>> >
> >> >>>> > With parent.config
> >> >>>> > dest_domain=. scheme=http parent="123.123.123.1:80,
> 123.123.123.2:80" round_robin=consistent_hash go_direct=false
> >> >>>> >
> >> >>>> > I'm getting
> >> >>>> >
> >> >>>> > Not Found on Accelerator
> >> >>>> > ________________________________
> >> >>>> > Description: Your request on the specified host was not found.
> Check the location and try again.
> >> >>>> >
> >> >>>> > I must be missing something ...
> >> >>>> >
> >> >>>> > Hobin
> >> >>>> >
> >> >>>> > On Thu, Jan 10, 2019 at 1:18 PM Hobin Yoon <ho...@gmail.com>
> wrote:
> >> >>>> >>
> >> >>>> >> Alan, that is the only map rule we have with the varying number
> of cache nodes. During the down time, ATS doesn't return "HTTP/1.1 200 OK"
> for the requests. I didn't check what it returned.
> >> >>>> >>
> >> >>>> >> Miles, I'll check out parent plugin!
> >> >>>> >>
> >> >>>> >> Hobin
> >> >>>> >>
> >> >>>> >> On Thu, Jan 10, 2019 at 1:13 PM Alan Carroll <
> solidwallofcode@oath.com> wrote:
> >> >>>> >>>
> >> >>>> >>> It could be an artifact of reloading plugin configurations if
> you have a lot of remap rules with plugins, although internally ATS should
> do the load and then swap the configuration. During the down time, does ATS
> process any traffic, or there is traffic but no caching?
> >> >>>> >>>
> >> >>>> >>> On Thu, Jan 10, 2019 at 10:53 AM Miles Libbey <
> mlibbey@apache.org> wrote:
> >> >>>> >>>>
> >> >>>> >>>> We don't experience downtime when using traffic_ctl config
> reload (we
> >> >>>> >>>> use that ~daily).
> >> >>>> >>>>
> >> >>>> >>>> We don't use the balancer plugin. Instead, we use
> parent.config
> >> >>>> >>>> (
> https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/parent.config.en.html
> )
> >> >>>> >>>> to achieve the same consistent hash. Your config would
> translate to
> >> >>>> >>>>
> >> >>>> >>>> remap.config
> >> >>>> >>>> map / http://127.0.0.1 @plugin=cachekey.so
> >> >>>> >>>> @pparam=--include-params=p0,p1 @pparam=--sort-params=true
> >> >>>> >>>>
> >> >>>> >>>> parent.config
> >> >>>> >>>> dest_domain=. scheme=http
> >> >>>> >>>> parent="123.123.123.1:80,23.123.123.2:80,123.123.123.3:80,
> 123.123.123.4:80"
> >> >>>> >>>> round_robin=consistent_hash go_direct=false
> >> >>>> >>>> dest_domain=. scheme=https
> >> >>>> >>>> parent="123.123.123.1:443,23.123.123.2:443,123.123.123.3:443,
> 123.123.123.4:443"
> >> >>>> >>>> round_robin=consistent_hash go_direct=false
> >> >>>> >>>>
> >> >>>> >>>> miles
> >> >>>> >>>>
> >> >>>> >>>> On Wed, Jan 9, 2019 at 10:53 PM Hobin Yoon <
> hobinyoon@gmail.com> wrote:
> >> >>>> >>>> >
> >> >>>> >>>> > Hi,
> >> >>>> >>>> >
> >> >>>> >>>> > We are noticing there is quite a bit of delay when we
> reload the config with traffic_ctl config reload. The delay is up to about
> 30 seconds, during which period we don't get any caching. We are using
> consistency hashing plugin. The number of nodes changes dynamically between
> 5 to 30.
> >> >>>> >>>> >
> >> >>>> >>>> > Here is an example balancer (consistent hash) configuration
> in remap.config.
> >> >>>> >>>> >
> >> >>>> >>>> > map / http://127.0.0.1 @plugin=cachekey.so
> @pparam=--include-params=p0,p1 @pparam=--sort-params=true
> @plugin=balancer.so @pparam=--policy=hash,key @pparam=123.123.123.1
> @pparam=123.123.123.2 @pparam=123.123.123.3 @pparam=123.123.123.4 ...
> >> >>>> >>>> >
> >> >>>> >>>> > Is this downtime normal? How do you guys avoid the service
> downtime while reconfiguring the cache nodes in the cluster?
> >> >>>> >>>> >
> >> >>>> >>>> > Hobin
> >> >>>> >>>> >
> >> >>>> >>>> >
> >> >>>> >>>> >
> >> >>>> >>>> >
> >> >>>> >>>> >
> >> >>>> >>>
> >> >>>> >>>
> >> >>>> >>>
> >> >>>> >>> --
> >> >>>> >>> Beware the fisherman who's casting out his line in to a dried
> up riverbed.
> >> >>>> >>> Oh don't try to tell him 'cause he won't believe. Throw some
> bread to the ducks instead.
> >> >>>> >>> It's easier that way. - Genesis : Duke : VI 25-28
> >> >>>
> >> >>>
> >> >>
>
Re: Downtime when traffic_ctl config reload
Posted by Miles Libbey <ml...@apache.org>.
A loop would indicate its not matching parent.config then. I've never
used an IP in a map. In
https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/parent.config.en.html
it has a
dest_ip
field that would seem to make more sense than dest_domain|host for
that map rule.
You'll also need a map http://52.91.93.206/ http://someorigiin on your parent...
miles
On Fri, Jan 11, 2019 at 10:49 PM Hobin Yoon <ho...@gmail.com> wrote:
>
> Now, I'm getting
>
> Cycle Prohibited
> ________________________________
> Description: Could not process your request for the document because it would cause an HTTP proxy cycle. Please check the URL and your browser's proxy settings.
>
> I think it's because remap.config has the same ip addresses ... I tried to comment out the map rule, but got the same error...
>
> On child node: (It's ip address is 52.91.93.206)
> records.config
> CONFIG proxy.config.http.parent_proxy_routing_enable INT 1
> CONFIG proxy.config.http.cache.http INT 0
> remap.config:
> map http://52.91.93.206 http://52.91.93.206
> parent.config:
> dest_domain=52.91.93.206 scheme=http parent="18.212.133.3:80,34.207.101.159:80" round_robin=consistent_hash go_direct=false
>
> On parent nodes:
> records.config
> CONFIG proxy.config.http.parent_proxy_routing_enable INT 0
> CONFIG proxy.config.http.cache.http INT 1
>
> Hobin
>
> On Fri, Jan 11, 2019 at 5:31 PM Miles Libbey <ml...@apache.org> wrote:
>>
>> First thing I'd check is proxy.config.http.parent_proxy_routing_enable
>> on the children :)
>>
>> On Fri, Jan 11, 2019 at 2:15 PM Hobin Yoon <ho...@gmail.com> wrote:
>> >
>> > It is disabled. I just tried with both on and off, but none of them worked. What bothers me is ATS doesn't seem to be reading parent.config at all since I don't get any error when I intentionally made a typo in parent.config.
>> >
>> > Hobin
>> >
>> > On Fri, Jan 11, 2019 at 4:38 PM Leif Hedstrom <zw...@apache.org> wrote:
>> >>
>> >>
>> >>
>> >> On Jan 11, 2019, at 12:54 PM, Hobin Yoon <ho...@gmail.com> wrote:
>> >>
>> >> Ok, I added the rule in remap.config. Now, the requests are going to "origin". But, they bypasses parent nodes; I don't see any requests on either of the parent nodes.
>> >>
>> >> Here is my system setup for test. "child" is a pure load balancer and doesn't cache any data items. parent1 and parent2 do cache.
>> >> Client (web browser) - child (ATS)--- parent1 (ATS) --- origin (Apache2)
>> >> └- parent2 (ATS) -┘
>> >>
>> >> Here is the configuration in "child" node.
>> >>
>> >> remap.config
>> >> map / http://origin
>> >>
>> >> parent.config
>> >> dest_domain=origin scheme=http parent="parent1:80;parent2:80" round_robin=consistent_hash
>> >>
>> >>
>> >>
>> >> Do you have “pristine” host headers enabled? The mapping in parent selection happens on the “remapped” URL, but I’m pretty sure it’d be different if you have pristine host headers disabled.
>> >>
>> >> That much said, I’d run with a diagnostics (-T) on “parent”.
>> >>
>> >> — Leif
>> >>
>> >>
>> >>
>> >> records.config
>> >> CONFIG proxy.config.http.parent_proxy_routing_enable INT 1
>> >>
>> >> Hobin
>> >>
>> >> On Fri, Jan 11, 2019 at 1:08 PM Leif Hedstrom <zw...@apache.org> wrote:
>> >>>
>> >>>
>> >>>
>> >>> On Jan 11, 2019, at 10:00 AM, Hobin Yoon <ho...@gmail.com> wrote:
>> >>>
>> >>> Should you still specify the mapping in remap.config when you have the mapping in parent.config?
>> >>>
>> >>>
>> >>> Yes. Unless you run an open proxy, and then you have to turn off “require remap”. Not recommended for almost all cases.
>> >>>
>> >>> — leif
>> >>>
>> >>>
>> >>>
>> >>> On Fri, Jan 11, 2019 at 11:33 AM Randall Meyer <ra...@yahoo.com> wrote:
>> >>>>
>> >>>>
>> >>>> Looking at your remap.config in that repo, I see that all the mappings are all commented out. Was that intentional?
>> >>>> On Thursday, January 10, 2019, 8:28:53 PM PST, Hobin Yoon <ho...@gmail.com> wrote:
>> >>>>
>> >>>>
>> >>>> Yes, they are also ATS nodes and they worked fine when requests were made to them directly.
>> >>>>
>> >>>> I uploaded the config files here, if you could take a look: https://gitlab.com/hobinyoon/trafficserver-config
>> >>>>
>> >>>> Hobin
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Thu, Jan 10, 2019 at 7:31 PM Miles Libbey <ml...@apache.org> wrote:
>> >>>>
>> >>>> Typically that error means the request doesn't match a rule in
>> >>>> remap.config. Is the error coming from the first hop or one of the
>> >>>> 123.123.123.[1-4] nodes? That is, if 123.123.123.[1-4] are ATS nodes,
>> >>>> are they configured to accept the requests they are getting?
>> >>>>
>> >>>> On Thu, Jan 10, 2019 at 2:51 PM Hobin Yoon <ho...@gmail.com> wrote:
>> >>>> >
>> >>>> > With parent.config
>> >>>> > dest_domain=. scheme=http parent="123.123.123.1:80,123.123.123.2:80" round_robin=consistent_hash go_direct=false
>> >>>> >
>> >>>> > I'm getting
>> >>>> >
>> >>>> > Not Found on Accelerator
>> >>>> > ________________________________
>> >>>> > Description: Your request on the specified host was not found. Check the location and try again.
>> >>>> >
>> >>>> > I must be missing something ...
>> >>>> >
>> >>>> > Hobin
>> >>>> >
>> >>>> > On Thu, Jan 10, 2019 at 1:18 PM Hobin Yoon <ho...@gmail.com> wrote:
>> >>>> >>
>> >>>> >> Alan, that is the only map rule we have with the varying number of cache nodes. During the down time, ATS doesn't return "HTTP/1.1 200 OK" for the requests. I didn't check what it returned.
>> >>>> >>
>> >>>> >> Miles, I'll check out parent plugin!
>> >>>> >>
>> >>>> >> Hobin
>> >>>> >>
>> >>>> >> On Thu, Jan 10, 2019 at 1:13 PM Alan Carroll <so...@oath.com> wrote:
>> >>>> >>>
>> >>>> >>> It could be an artifact of reloading plugin configurations if you have a lot of remap rules with plugins, although internally ATS should do the load and then swap the configuration. During the down time, does ATS process any traffic, or there is traffic but no caching?
>> >>>> >>>
>> >>>> >>> On Thu, Jan 10, 2019 at 10:53 AM Miles Libbey <ml...@apache.org> wrote:
>> >>>> >>>>
>> >>>> >>>> We don't experience downtime when using traffic_ctl config reload (we
>> >>>> >>>> use that ~daily).
>> >>>> >>>>
>> >>>> >>>> We don't use the balancer plugin. Instead, we use parent.config
>> >>>> >>>> (https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/parent.config.en.html)
>> >>>> >>>> to achieve the same consistent hash. Your config would translate to
>> >>>> >>>>
>> >>>> >>>> remap.config
>> >>>> >>>> map / http://127.0.0.1 @plugin=cachekey.so
>> >>>> >>>> @pparam=--include-params=p0,p1 @pparam=--sort-params=true
>> >>>> >>>>
>> >>>> >>>> parent.config
>> >>>> >>>> dest_domain=. scheme=http
>> >>>> >>>> parent="123.123.123.1:80,23.123.123.2:80,123.123.123.3:80,123.123.123.4:80"
>> >>>> >>>> round_robin=consistent_hash go_direct=false
>> >>>> >>>> dest_domain=. scheme=https
>> >>>> >>>> parent="123.123.123.1:443,23.123.123.2:443,123.123.123.3:443,123.123.123.4:443"
>> >>>> >>>> round_robin=consistent_hash go_direct=false
>> >>>> >>>>
>> >>>> >>>> miles
>> >>>> >>>>
>> >>>> >>>> On Wed, Jan 9, 2019 at 10:53 PM Hobin Yoon <ho...@gmail.com> wrote:
>> >>>> >>>> >
>> >>>> >>>> > Hi,
>> >>>> >>>> >
>> >>>> >>>> > We are noticing there is quite a bit of delay when we reload the config with traffic_ctl config reload. The delay is up to about 30 seconds, during which period we don't get any caching. We are using consistency hashing plugin. The number of nodes changes dynamically between 5 to 30.
>> >>>> >>>> >
>> >>>> >>>> > Here is an example balancer (consistent hash) configuration in remap.config.
>> >>>> >>>> >
>> >>>> >>>> > map / http://127.0.0.1 @plugin=cachekey.so @pparam=--include-params=p0,p1 @pparam=--sort-params=true @plugin=balancer.so @pparam=--policy=hash,key @pparam=123.123.123.1 @pparam=123.123.123.2 @pparam=123.123.123.3 @pparam=123.123.123.4 ...
>> >>>> >>>> >
>> >>>> >>>> > Is this downtime normal? How do you guys avoid the service downtime while reconfiguring the cache nodes in the cluster?
>> >>>> >>>> >
>> >>>> >>>> > Hobin
>> >>>> >>>> >
>> >>>> >>>> >
>> >>>> >>>> >
>> >>>> >>>> >
>> >>>> >>>> >
>> >>>> >>>
>> >>>> >>>
>> >>>> >>>
>> >>>> >>> --
>> >>>> >>> Beware the fisherman who's casting out his line in to a dried up riverbed.
>> >>>> >>> Oh don't try to tell him 'cause he won't believe. Throw some bread to the ducks instead.
>> >>>> >>> It's easier that way. - Genesis : Duke : VI 25-28
>> >>>
>> >>>
>> >>
Re: Downtime when traffic_ctl config reload
Posted by Hobin Yoon <ho...@gmail.com>.
Now, I'm getting
Cycle Prohibited
________________________________
Description: Could not process your request for the document because it
would cause an HTTP proxy cycle. Please check the URL and your browser's
proxy settings.
I think it's because remap.config has the same ip addresses ... I tried to
comment out the map rule, but got the same error...
On child node: (It's ip address is 52.91.93.206)
records.config
CONFIG proxy.config.http.parent_proxy_routing_enable INT 1
CONFIG proxy.config.http.cache.http INT 0
remap.config:
map http://52.91.93.206 http://52.91.93.206
parent.config:
dest_domain=52.91.93.206 scheme=http parent="18.212.133.3:80,
34.207.101.159:80" round_robin=consistent_hash go_direct=false
On parent nodes:
records.config
CONFIG proxy.config.http.parent_proxy_routing_enable INT 0
CONFIG proxy.config.http.cache.http INT 1
Hobin
On Fri, Jan 11, 2019 at 5:31 PM Miles Libbey <ml...@apache.org> wrote:
> First thing I'd check is proxy.config.http.parent_proxy_routing_enable
> on the children :)
>
> On Fri, Jan 11, 2019 at 2:15 PM Hobin Yoon <ho...@gmail.com> wrote:
> >
> > It is disabled. I just tried with both on and off, but none of them
> worked. What bothers me is ATS doesn't seem to be reading parent.config at
> all since I don't get any error when I intentionally made a typo in
> parent.config.
> >
> > Hobin
> >
> > On Fri, Jan 11, 2019 at 4:38 PM Leif Hedstrom <zw...@apache.org> wrote:
> >>
> >>
> >>
> >> On Jan 11, 2019, at 12:54 PM, Hobin Yoon <ho...@gmail.com> wrote:
> >>
> >> Ok, I added the rule in remap.config. Now, the requests are going to
> "origin". But, they bypasses parent nodes; I don't see any requests on
> either of the parent nodes.
> >>
> >> Here is my system setup for test. "child" is a pure load balancer and
> doesn't cache any data items. parent1 and parent2 do cache.
> >> Client (web browser) - child (ATS)--- parent1 (ATS) --- origin (Apache2)
> >> └- parent2 (ATS) -┘
> >>
> >> Here is the configuration in "child" node.
> >>
> >> remap.config
> >> map / http://origin
> >>
> >> parent.config
> >> dest_domain=origin scheme=http parent="parent1:80;parent2:80"
> round_robin=consistent_hash
> >>
> >>
> >>
> >> Do you have “pristine” host headers enabled? The mapping in parent
> selection happens on the “remapped” URL, but I’m pretty sure it’d be
> different if you have pristine host headers disabled.
> >>
> >> That much said, I’d run with a diagnostics (-T) on “parent”.
> >>
> >> — Leif
> >>
> >>
> >>
> >> records.config
> >> CONFIG proxy.config.http.parent_proxy_routing_enable INT 1
> >>
> >> Hobin
> >>
> >> On Fri, Jan 11, 2019 at 1:08 PM Leif Hedstrom <zw...@apache.org> wrote:
> >>>
> >>>
> >>>
> >>> On Jan 11, 2019, at 10:00 AM, Hobin Yoon <ho...@gmail.com> wrote:
> >>>
> >>> Should you still specify the mapping in remap.config when you have the
> mapping in parent.config?
> >>>
> >>>
> >>> Yes. Unless you run an open proxy, and then you have to turn off
> “require remap”. Not recommended for almost all cases.
> >>>
> >>> — leif
> >>>
> >>>
> >>>
> >>> On Fri, Jan 11, 2019 at 11:33 AM Randall Meyer <ra...@yahoo.com>
> wrote:
> >>>>
> >>>>
> >>>> Looking at your remap.config in that repo, I see that all the
> mappings are all commented out. Was that intentional?
> >>>> On Thursday, January 10, 2019, 8:28:53 PM PST, Hobin Yoon <
> hobinyoon@gmail.com> wrote:
> >>>>
> >>>>
> >>>> Yes, they are also ATS nodes and they worked fine when requests were
> made to them directly.
> >>>>
> >>>> I uploaded the config files here, if you could take a look:
> https://gitlab.com/hobinyoon/trafficserver-config
> >>>>
> >>>> Hobin
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Jan 10, 2019 at 7:31 PM Miles Libbey <ml...@apache.org>
> wrote:
> >>>>
> >>>> Typically that error means the request doesn't match a rule in
> >>>> remap.config. Is the error coming from the first hop or one of the
> >>>> 123.123.123.[1-4] nodes? That is, if 123.123.123.[1-4] are ATS nodes,
> >>>> are they configured to accept the requests they are getting?
> >>>>
> >>>> On Thu, Jan 10, 2019 at 2:51 PM Hobin Yoon <ho...@gmail.com>
> wrote:
> >>>> >
> >>>> > With parent.config
> >>>> > dest_domain=. scheme=http parent="123.123.123.1:80,
> 123.123.123.2:80" round_robin=consistent_hash go_direct=false
> >>>> >
> >>>> > I'm getting
> >>>> >
> >>>> > Not Found on Accelerator
> >>>> > ________________________________
> >>>> > Description: Your request on the specified host was not found.
> Check the location and try again.
> >>>> >
> >>>> > I must be missing something ...
> >>>> >
> >>>> > Hobin
> >>>> >
> >>>> > On Thu, Jan 10, 2019 at 1:18 PM Hobin Yoon <ho...@gmail.com>
> wrote:
> >>>> >>
> >>>> >> Alan, that is the only map rule we have with the varying number of
> cache nodes. During the down time, ATS doesn't return "HTTP/1.1 200 OK" for
> the requests. I didn't check what it returned.
> >>>> >>
> >>>> >> Miles, I'll check out parent plugin!
> >>>> >>
> >>>> >> Hobin
> >>>> >>
> >>>> >> On Thu, Jan 10, 2019 at 1:13 PM Alan Carroll <
> solidwallofcode@oath.com> wrote:
> >>>> >>>
> >>>> >>> It could be an artifact of reloading plugin configurations if you
> have a lot of remap rules with plugins, although internally ATS should do
> the load and then swap the configuration. During the down time, does ATS
> process any traffic, or there is traffic but no caching?
> >>>> >>>
> >>>> >>> On Thu, Jan 10, 2019 at 10:53 AM Miles Libbey <ml...@apache.org>
> wrote:
> >>>> >>>>
> >>>> >>>> We don't experience downtime when using traffic_ctl config
> reload (we
> >>>> >>>> use that ~daily).
> >>>> >>>>
> >>>> >>>> We don't use the balancer plugin. Instead, we use parent.config
> >>>> >>>> (
> https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/parent.config.en.html
> )
> >>>> >>>> to achieve the same consistent hash. Your config would translate
> to
> >>>> >>>>
> >>>> >>>> remap.config
> >>>> >>>> map / http://127.0.0.1 @plugin=cachekey.so
> >>>> >>>> @pparam=--include-params=p0,p1 @pparam=--sort-params=true
> >>>> >>>>
> >>>> >>>> parent.config
> >>>> >>>> dest_domain=. scheme=http
> >>>> >>>> parent="123.123.123.1:80,23.123.123.2:80,123.123.123.3:80,
> 123.123.123.4:80"
> >>>> >>>> round_robin=consistent_hash go_direct=false
> >>>> >>>> dest_domain=. scheme=https
> >>>> >>>> parent="123.123.123.1:443,23.123.123.2:443,123.123.123.3:443,
> 123.123.123.4:443"
> >>>> >>>> round_robin=consistent_hash go_direct=false
> >>>> >>>>
> >>>> >>>> miles
> >>>> >>>>
> >>>> >>>> On Wed, Jan 9, 2019 at 10:53 PM Hobin Yoon <ho...@gmail.com>
> wrote:
> >>>> >>>> >
> >>>> >>>> > Hi,
> >>>> >>>> >
> >>>> >>>> > We are noticing there is quite a bit of delay when we reload
> the config with traffic_ctl config reload. The delay is up to about 30
> seconds, during which period we don't get any caching. We are using
> consistency hashing plugin. The number of nodes changes dynamically between
> 5 to 30.
> >>>> >>>> >
> >>>> >>>> > Here is an example balancer (consistent hash) configuration in
> remap.config.
> >>>> >>>> >
> >>>> >>>> > map / http://127.0.0.1 @plugin=cachekey.so
> @pparam=--include-params=p0,p1 @pparam=--sort-params=true
> @plugin=balancer.so @pparam=--policy=hash,key @pparam=123.123.123.1
> @pparam=123.123.123.2 @pparam=123.123.123.3 @pparam=123.123.123.4 ...
> >>>> >>>> >
> >>>> >>>> > Is this downtime normal? How do you guys avoid the service
> downtime while reconfiguring the cache nodes in the cluster?
> >>>> >>>> >
> >>>> >>>> > Hobin
> >>>> >>>> >
> >>>> >>>> >
> >>>> >>>> >
> >>>> >>>> >
> >>>> >>>> >
> >>>> >>>
> >>>> >>>
> >>>> >>>
> >>>> >>> --
> >>>> >>> Beware the fisherman who's casting out his line in to a dried up
> riverbed.
> >>>> >>> Oh don't try to tell him 'cause he won't believe. Throw some
> bread to the ducks instead.
> >>>> >>> It's easier that way. - Genesis : Duke : VI 25-28
> >>>
> >>>
> >>
>
Re: Downtime when traffic_ctl config reload
Posted by Miles Libbey <ml...@apache.org>.
First thing I'd check is proxy.config.http.parent_proxy_routing_enable
on the children :)
On Fri, Jan 11, 2019 at 2:15 PM Hobin Yoon <ho...@gmail.com> wrote:
>
> It is disabled. I just tried with both on and off, but none of them worked. What bothers me is ATS doesn't seem to be reading parent.config at all since I don't get any error when I intentionally made a typo in parent.config.
>
> Hobin
>
> On Fri, Jan 11, 2019 at 4:38 PM Leif Hedstrom <zw...@apache.org> wrote:
>>
>>
>>
>> On Jan 11, 2019, at 12:54 PM, Hobin Yoon <ho...@gmail.com> wrote:
>>
>> Ok, I added the rule in remap.config. Now, the requests are going to "origin". But, they bypasses parent nodes; I don't see any requests on either of the parent nodes.
>>
>> Here is my system setup for test. "child" is a pure load balancer and doesn't cache any data items. parent1 and parent2 do cache.
>> Client (web browser) - child (ATS)--- parent1 (ATS) --- origin (Apache2)
>> └- parent2 (ATS) -┘
>>
>> Here is the configuration in "child" node.
>>
>> remap.config
>> map / http://origin
>>
>> parent.config
>> dest_domain=origin scheme=http parent="parent1:80;parent2:80" round_robin=consistent_hash
>>
>>
>>
>> Do you have “pristine” host headers enabled? The mapping in parent selection happens on the “remapped” URL, but I’m pretty sure it’d be different if you have pristine host headers disabled.
>>
>> That much said, I’d run with a diagnostics (-T) on “parent”.
>>
>> — Leif
>>
>>
>>
>> records.config
>> CONFIG proxy.config.http.parent_proxy_routing_enable INT 1
>>
>> Hobin
>>
>> On Fri, Jan 11, 2019 at 1:08 PM Leif Hedstrom <zw...@apache.org> wrote:
>>>
>>>
>>>
>>> On Jan 11, 2019, at 10:00 AM, Hobin Yoon <ho...@gmail.com> wrote:
>>>
>>> Should you still specify the mapping in remap.config when you have the mapping in parent.config?
>>>
>>>
>>> Yes. Unless you run an open proxy, and then you have to turn off “require remap”. Not recommended for almost all cases.
>>>
>>> — leif
>>>
>>>
>>>
>>> On Fri, Jan 11, 2019 at 11:33 AM Randall Meyer <ra...@yahoo.com> wrote:
>>>>
>>>>
>>>> Looking at your remap.config in that repo, I see that all the mappings are all commented out. Was that intentional?
>>>> On Thursday, January 10, 2019, 8:28:53 PM PST, Hobin Yoon <ho...@gmail.com> wrote:
>>>>
>>>>
>>>> Yes, they are also ATS nodes and they worked fine when requests were made to them directly.
>>>>
>>>> I uploaded the config files here, if you could take a look: https://gitlab.com/hobinyoon/trafficserver-config
>>>>
>>>> Hobin
>>>>
>>>>
>>>>
>>>> On Thu, Jan 10, 2019 at 7:31 PM Miles Libbey <ml...@apache.org> wrote:
>>>>
>>>> Typically that error means the request doesn't match a rule in
>>>> remap.config. Is the error coming from the first hop or one of the
>>>> 123.123.123.[1-4] nodes? That is, if 123.123.123.[1-4] are ATS nodes,
>>>> are they configured to accept the requests they are getting?
>>>>
>>>> On Thu, Jan 10, 2019 at 2:51 PM Hobin Yoon <ho...@gmail.com> wrote:
>>>> >
>>>> > With parent.config
>>>> > dest_domain=. scheme=http parent="123.123.123.1:80,123.123.123.2:80" round_robin=consistent_hash go_direct=false
>>>> >
>>>> > I'm getting
>>>> >
>>>> > Not Found on Accelerator
>>>> > ________________________________
>>>> > Description: Your request on the specified host was not found. Check the location and try again.
>>>> >
>>>> > I must be missing something ...
>>>> >
>>>> > Hobin
>>>> >
>>>> > On Thu, Jan 10, 2019 at 1:18 PM Hobin Yoon <ho...@gmail.com> wrote:
>>>> >>
>>>> >> Alan, that is the only map rule we have with the varying number of cache nodes. During the down time, ATS doesn't return "HTTP/1.1 200 OK" for the requests. I didn't check what it returned.
>>>> >>
>>>> >> Miles, I'll check out parent plugin!
>>>> >>
>>>> >> Hobin
>>>> >>
>>>> >> On Thu, Jan 10, 2019 at 1:13 PM Alan Carroll <so...@oath.com> wrote:
>>>> >>>
>>>> >>> It could be an artifact of reloading plugin configurations if you have a lot of remap rules with plugins, although internally ATS should do the load and then swap the configuration. During the down time, does ATS process any traffic, or there is traffic but no caching?
>>>> >>>
>>>> >>> On Thu, Jan 10, 2019 at 10:53 AM Miles Libbey <ml...@apache.org> wrote:
>>>> >>>>
>>>> >>>> We don't experience downtime when using traffic_ctl config reload (we
>>>> >>>> use that ~daily).
>>>> >>>>
>>>> >>>> We don't use the balancer plugin. Instead, we use parent.config
>>>> >>>> (https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/parent.config.en.html)
>>>> >>>> to achieve the same consistent hash. Your config would translate to
>>>> >>>>
>>>> >>>> remap.config
>>>> >>>> map / http://127.0.0.1 @plugin=cachekey.so
>>>> >>>> @pparam=--include-params=p0,p1 @pparam=--sort-params=true
>>>> >>>>
>>>> >>>> parent.config
>>>> >>>> dest_domain=. scheme=http
>>>> >>>> parent="123.123.123.1:80,23.123.123.2:80,123.123.123.3:80,123.123.123.4:80"
>>>> >>>> round_robin=consistent_hash go_direct=false
>>>> >>>> dest_domain=. scheme=https
>>>> >>>> parent="123.123.123.1:443,23.123.123.2:443,123.123.123.3:443,123.123.123.4:443"
>>>> >>>> round_robin=consistent_hash go_direct=false
>>>> >>>>
>>>> >>>> miles
>>>> >>>>
>>>> >>>> On Wed, Jan 9, 2019 at 10:53 PM Hobin Yoon <ho...@gmail.com> wrote:
>>>> >>>> >
>>>> >>>> > Hi,
>>>> >>>> >
>>>> >>>> > We are noticing there is quite a bit of delay when we reload the config with traffic_ctl config reload. The delay is up to about 30 seconds, during which period we don't get any caching. We are using consistency hashing plugin. The number of nodes changes dynamically between 5 to 30.
>>>> >>>> >
>>>> >>>> > Here is an example balancer (consistent hash) configuration in remap.config.
>>>> >>>> >
>>>> >>>> > map / http://127.0.0.1 @plugin=cachekey.so @pparam=--include-params=p0,p1 @pparam=--sort-params=true @plugin=balancer.so @pparam=--policy=hash,key @pparam=123.123.123.1 @pparam=123.123.123.2 @pparam=123.123.123.3 @pparam=123.123.123.4 ...
>>>> >>>> >
>>>> >>>> > Is this downtime normal? How do you guys avoid the service downtime while reconfiguring the cache nodes in the cluster?
>>>> >>>> >
>>>> >>>> > Hobin
>>>> >>>> >
>>>> >>>> >
>>>> >>>> >
>>>> >>>> >
>>>> >>>> >
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> Beware the fisherman who's casting out his line in to a dried up riverbed.
>>>> >>> Oh don't try to tell him 'cause he won't believe. Throw some bread to the ducks instead.
>>>> >>> It's easier that way. - Genesis : Duke : VI 25-28
>>>
>>>
>>
Re: Downtime when traffic_ctl config reload
Posted by Hobin Yoon <ho...@gmail.com>.
It is disabled. I just tried with both on and off, but none of them worked.
What bothers me is ATS doesn't seem to be reading parent.config at all
since I don't get any error when I intentionally made a typo in
parent.config.
Hobin
On Fri, Jan 11, 2019 at 4:38 PM Leif Hedstrom <zw...@apache.org> wrote:
>
>
> On Jan 11, 2019, at 12:54 PM, Hobin Yoon <ho...@gmail.com> wrote:
>
> Ok, I added the rule in remap.config. Now, the requests are going to
> "origin". But, they bypasses parent nodes; I don't see any requests on
> either of the parent nodes.
>
> Here is my system setup for test. "child" is a pure load balancer and
> doesn't cache any data items. parent1 and parent2 do cache.
> Client (web browser) - child (ATS)--- parent1 (ATS) --- origin (Apache2)
> └- parent2 (ATS) -┘
>
> Here is the configuration in "child" node.
>
> remap.config
> map / http://origin
>
> parent.config
> dest_domain=origin scheme=http parent="parent1:80;parent2:80"
> round_robin=consistent_hash
>
>
>
> Do you have “pristine” host headers enabled? The mapping in parent
> selection happens on the “remapped” URL, but I’m pretty sure it’d be
> different if you have pristine host headers disabled.
>
> That much said, I’d run with a diagnostics (-T) on “parent”.
>
> — Leif
>
>
>
> records.config
> CONFIG proxy.config.http.parent_proxy_routing_enable INT 1
>
> Hobin
>
> On Fri, Jan 11, 2019 at 1:08 PM Leif Hedstrom <zw...@apache.org> wrote:
>
>>
>>
>> On Jan 11, 2019, at 10:00 AM, Hobin Yoon <ho...@gmail.com> wrote:
>>
>> Should you still specify the mapping in remap.config when you have the
>> mapping in parent.config?
>>
>>
>> Yes. Unless you run an open proxy, and then you have to turn off “require
>> remap”. Not recommended for almost all cases.
>>
>> — leif
>>
>>
>>
>> On Fri, Jan 11, 2019 at 11:33 AM Randall Meyer <ra...@yahoo.com>
>> wrote:
>>
>>>
>>> Looking at your remap.config in that repo, I see that all the mappings
>>> are all commented out. Was that intentional?
>>> On Thursday, January 10, 2019, 8:28:53 PM PST, Hobin Yoon <
>>> hobinyoon@gmail.com> wrote:
>>>
>>>
>>> Yes, they are also ATS nodes and they worked fine when requests were
>>> made to them directly.
>>>
>>> I uploaded the config files here, if you could take a look:
>>> https://gitlab.com/hobinyoon/trafficserver-config
>>>
>>> Hobin
>>>
>>>
>>>
>>> On Thu, Jan 10, 2019 at 7:31 PM Miles Libbey <ml...@apache.org> wrote:
>>>
>>> Typically that error means the request doesn't match a rule in
>>> remap.config. Is the error coming from the first hop or one of the
>>> 123.123.123.[1-4] nodes? That is, if 123.123.123.[1-4] are ATS nodes,
>>> are they configured to accept the requests they are getting?
>>>
>>> On Thu, Jan 10, 2019 at 2:51 PM Hobin Yoon <ho...@gmail.com> wrote:
>>> >
>>> > With parent.config
>>> > dest_domain=. scheme=http parent="123.123.123.1:80
>>> <http://123.123.123.1/>,123.123.123.2:80 <http://123.123.123.2/>"
>>> round_robin=consistent_hash go_direct=false
>>> >
>>> > I'm getting
>>> >
>>> > Not Found on Accelerator
>>> > ________________________________
>>> > Description: Your request on the specified host was not found. Check
>>> the location and try again.
>>> >
>>> > I must be missing something ...
>>> >
>>> > Hobin
>>> >
>>> > On Thu, Jan 10, 2019 at 1:18 PM Hobin Yoon <ho...@gmail.com>
>>> wrote:
>>> >>
>>> >> Alan, that is the only map rule we have with the varying number of
>>> cache nodes. During the down time, ATS doesn't return "HTTP/1.1 200 OK" for
>>> the requests. I didn't check what it returned.
>>> >>
>>> >> Miles, I'll check out parent plugin!
>>> >>
>>> >> Hobin
>>> >>
>>> >> On Thu, Jan 10, 2019 at 1:13 PM Alan Carroll <
>>> solidwallofcode@oath.com> wrote:
>>> >>>
>>> >>> It could be an artifact of reloading plugin configurations if you
>>> have a lot of remap rules with plugins, although internally ATS should do
>>> the load and then swap the configuration. During the down time, does ATS
>>> process any traffic, or there is traffic but no caching?
>>> >>>
>>> >>> On Thu, Jan 10, 2019 at 10:53 AM Miles Libbey <ml...@apache.org>
>>> wrote:
>>> >>>>
>>> >>>> We don't experience downtime when using traffic_ctl config reload
>>> (we
>>> >>>> use that ~daily).
>>> >>>>
>>> >>>> We don't use the balancer plugin. Instead, we use parent.config
>>> >>>> (
>>> https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/parent.config.en.html
>>> )
>>> >>>> to achieve the same consistent hash. Your config would translate to
>>> >>>>
>>> >>>> remap.config
>>> >>>> map / http://127.0.0.1 @plugin=cachekey.so
>>> >>>> @pparam=--include-params=p0,p1 @pparam=--sort-params=true
>>> >>>>
>>> >>>> parent.config
>>> >>>> dest_domain=. scheme=http
>>> >>>> parent="123.123.123.1:80 <http://123.123.123.1/>,23.123.123.2:80
>>> <http://23.123.123.2/>,123.123.123.3:80 <http://123.123.123.3/>,
>>> 123.123.123.4:80 <http://123.123.123.4/>"
>>> >>>> round_robin=consistent_hash go_direct=false
>>> >>>> dest_domain=. scheme=https
>>> >>>> parent="123.123.123.1:443,23.123.123.2:443,123.123.123.3:443,
>>> 123.123.123.4:443"
>>> >>>> round_robin=consistent_hash go_direct=false
>>> >>>>
>>> >>>> miles
>>> >>>>
>>> >>>> On Wed, Jan 9, 2019 at 10:53 PM Hobin Yoon <ho...@gmail.com>
>>> wrote:
>>> >>>> >
>>> >>>> > Hi,
>>> >>>> >
>>> >>>> > We are noticing there is quite a bit of delay when we reload the
>>> config with traffic_ctl config reload. The delay is up to about 30 seconds,
>>> during which period we don't get any caching. We are using consistency
>>> hashing plugin. The number of nodes changes dynamically between 5 to 30.
>>> >>>> >
>>> >>>> > Here is an example balancer (consistent hash) configuration in
>>> remap.config.
>>> >>>> >
>>> >>>> > map / http://127.0.0.1 @plugin=cachekey.so
>>> @pparam=--include-params=p0,p1 @pparam=--sort-params=true
>>> @plugin=balancer.so @pparam=--policy=hash,key @pparam=123.123.123.1
>>> @pparam=123.123.123.2 @pparam=123.123.123.3 @pparam=123.123.123.4 ...
>>> >>>> >
>>> >>>> > Is this downtime normal? How do you guys avoid the service
>>> downtime while reconfiguring the cache nodes in the cluster?
>>> >>>> >
>>> >>>> > Hobin
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Beware the fisherman who's casting out his line in to a dried up
>>> riverbed.
>>> >>> Oh don't try to tell him 'cause he won't believe. Throw some bread
>>> to the ducks instead.
>>> >>> It's easier that way. - Genesis : Duke : VI 25-28
>>>
>>>
>>
>
Re: Downtime when traffic_ctl config reload
Posted by Leif Hedstrom <zw...@apache.org>.
> On Jan 11, 2019, at 12:54 PM, Hobin Yoon <ho...@gmail.com> wrote:
>
> Ok, I added the rule in remap.config. Now, the requests are going to "origin". But, they bypasses parent nodes; I don't see any requests on either of the parent nodes.
>
> Here is my system setup for test. "child" is a pure load balancer and doesn't cache any data items. parent1 and parent2 do cache.
> Client (web browser) - child (ATS)--- parent1 (ATS) --- origin (Apache2)
> └- parent2 (ATS) -┘
>
> Here is the configuration in "child" node.
>
> remap.config
> map / http://origin <http://origin/>
>
> parent.config
> dest_domain=origin scheme=http parent="parent1:80;parent2:80" round_robin=consistent_hash
Do you have “pristine” host headers enabled? The mapping in parent selection happens on the “remapped” URL, but I’m pretty sure it’d be different if you have pristine host headers disabled.
That much said, I’d run with a diagnostics (-T) on “parent”.
— Leif
>
> records.config
> CONFIG proxy.config.http.parent_proxy_routing_enable INT 1
>
> Hobin
>
> On Fri, Jan 11, 2019 at 1:08 PM Leif Hedstrom <zwoop@apache.org <ma...@apache.org>> wrote:
>
>
>> On Jan 11, 2019, at 10:00 AM, Hobin Yoon <hobinyoon@gmail.com <ma...@gmail.com>> wrote:
>>
>> Should you still specify the mapping in remap.config when you have the mapping in parent.config?
>
> Yes. Unless you run an open proxy, and then you have to turn off “require remap”. Not recommended for almost all cases.
>
> — leif
>
>>
>>
>> On Fri, Jan 11, 2019 at 11:33 AM Randall Meyer <randallmeyer@yahoo.com <ma...@yahoo.com>> wrote:
>>
>> Looking at your remap.config in that repo, I see that all the mappings are all commented out. Was that intentional?
>> On Thursday, January 10, 2019, 8:28:53 PM PST, Hobin Yoon <hobinyoon@gmail.com <ma...@gmail.com>> wrote:
>>
>>
>> Yes, they are also ATS nodes and they worked fine when requests were made to them directly.
>>
>> I uploaded the config files here, if you could take a look: https://gitlab.com/hobinyoon/trafficserver-config <https://gitlab.com/hobinyoon/trafficserver-config>
>>
>> Hobin
>>
>>
>>
>> On Thu, Jan 10, 2019 at 7:31 PM Miles Libbey <mlibbey@apache.org <ma...@apache.org>> wrote:
>> Typically that error means the request doesn't match a rule in
>> remap.config. Is the error coming from the first hop or one of the
>> 123.123.123.[1-4] nodes? That is, if 123.123.123.[1-4] are ATS nodes,
>> are they configured to accept the requests they are getting?
>>
>> On Thu, Jan 10, 2019 at 2:51 PM Hobin Yoon <hobinyoon@gmail.com <ma...@gmail.com>> wrote:
>> >
>> > With parent.config
>> > dest_domain=. scheme=http parent="123.123.123.1:80 <http://123.123.123.1/>,123.123.123.2:80 <http://123.123.123.2/>" round_robin=consistent_hash go_direct=false
>> >
>> > I'm getting
>> >
>> > Not Found on Accelerator
>> > ________________________________
>> > Description: Your request on the specified host was not found. Check the location and try again.
>> >
>> > I must be missing something ...
>> >
>> > Hobin
>> >
>> > On Thu, Jan 10, 2019 at 1:18 PM Hobin Yoon <hobinyoon@gmail.com <ma...@gmail.com>> wrote:
>> >>
>> >> Alan, that is the only map rule we have with the varying number of cache nodes. During the down time, ATS doesn't return "HTTP/1.1 200 OK" for the requests. I didn't check what it returned.
>> >>
>> >> Miles, I'll check out parent plugin!
>> >>
>> >> Hobin
>> >>
>> >> On Thu, Jan 10, 2019 at 1:13 PM Alan Carroll <solidwallofcode@oath.com <ma...@oath.com>> wrote:
>> >>>
>> >>> It could be an artifact of reloading plugin configurations if you have a lot of remap rules with plugins, although internally ATS should do the load and then swap the configuration. During the down time, does ATS process any traffic, or there is traffic but no caching?
>> >>>
>> >>> On Thu, Jan 10, 2019 at 10:53 AM Miles Libbey <mlibbey@apache.org <ma...@apache.org>> wrote:
>> >>>>
>> >>>> We don't experience downtime when using traffic_ctl config reload (we
>> >>>> use that ~daily).
>> >>>>
>> >>>> We don't use the balancer plugin. Instead, we use parent.config
>> >>>> (https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/parent.config.en.html <https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/parent.config.en.html>)
>> >>>> to achieve the same consistent hash. Your config would translate to
>> >>>>
>> >>>> remap.config
>> >>>> map / http://127.0.0.1 <http://127.0.0.1/> @plugin=cachekey.so
>> >>>> @pparam=--include-params=p0,p1 @pparam=--sort-params=true
>> >>>>
>> >>>> parent.config
>> >>>> dest_domain=. scheme=http
>> >>>> parent="123.123.123.1:80 <http://123.123.123.1/>,23.123.123.2:80 <http://23.123.123.2/>,123.123.123.3:80 <http://123.123.123.3/>,123.123.123.4:80 <http://123.123.123.4/>"
>> >>>> round_robin=consistent_hash go_direct=false
>> >>>> dest_domain=. scheme=https
>> >>>> parent="123.123.123.1:443 <http://123.123.123.1:443/>,23.123.123.2:443 <http://23.123.123.2:443/>,123.123.123.3:443 <http://123.123.123.3:443/>,123.123.123.4:443 <http://123.123.123.4:443/>"
>> >>>> round_robin=consistent_hash go_direct=false
>> >>>>
>> >>>> miles
>> >>>>
>> >>>> On Wed, Jan 9, 2019 at 10:53 PM Hobin Yoon <hobinyoon@gmail.com <ma...@gmail.com>> wrote:
>> >>>> >
>> >>>> > Hi,
>> >>>> >
>> >>>> > We are noticing there is quite a bit of delay when we reload the config with traffic_ctl config reload. The delay is up to about 30 seconds, during which period we don't get any caching. We are using consistency hashing plugin. The number of nodes changes dynamically between 5 to 30.
>> >>>> >
>> >>>> > Here is an example balancer (consistent hash) configuration in remap.config.
>> >>>> >
>> >>>> > map / http://127.0.0.1 <http://127.0.0.1/> @plugin=cachekey.so @pparam=--include-params=p0,p1 @pparam=--sort-params=true @plugin=balancer.so @pparam=--policy=hash,key @pparam=123.123.123.1 @pparam=123.123.123.2 @pparam=123.123.123.3 @pparam=123.123.123.4 ...
>> >>>> >
>> >>>> > Is this downtime normal? How do you guys avoid the service downtime while reconfiguring the cache nodes in the cluster?
>> >>>> >
>> >>>> > Hobin
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Beware the fisherman who's casting out his line in to a dried up riverbed.
>> >>> Oh don't try to tell him 'cause he won't believe. Throw some bread to the ducks instead.
>> >>> It's easier that way. - Genesis : Duke : VI 25-28
>
Re: Downtime when traffic_ctl config reload
Posted by Hobin Yoon <ho...@gmail.com>.
Ok, I added the rule in remap.config. Now, the requests are going to
"origin". But, they bypasses parent nodes; I don't see any requests on
either of the parent nodes.
Here is my system setup for test. "child" is a pure load balancer and
doesn't cache any data items. parent1 and parent2 do cache.
Client (web browser) - child (ATS)--- parent1 (ATS) --- origin (Apache2)
└- parent2 (ATS) -┘
Here is the configuration in "child" node.
remap.config
map / http://origin
parent.config
dest_domain=origin scheme=http parent="parent1:80;parent2:80"
round_robin=consistent_hash
records.config
CONFIG proxy.config.http.parent_proxy_routing_enable INT 1
Hobin
On Fri, Jan 11, 2019 at 1:08 PM Leif Hedstrom <zw...@apache.org> wrote:
>
>
> On Jan 11, 2019, at 10:00 AM, Hobin Yoon <ho...@gmail.com> wrote:
>
> Should you still specify the mapping in remap.config when you have the
> mapping in parent.config?
>
>
> Yes. Unless you run an open proxy, and then you have to turn off “require
> remap”. Not recommended for almost all cases.
>
> — leif
>
>
>
> On Fri, Jan 11, 2019 at 11:33 AM Randall Meyer <ra...@yahoo.com>
> wrote:
>
>>
>> Looking at your remap.config in that repo, I see that all the mappings
>> are all commented out. Was that intentional?
>> On Thursday, January 10, 2019, 8:28:53 PM PST, Hobin Yoon <
>> hobinyoon@gmail.com> wrote:
>>
>>
>> Yes, they are also ATS nodes and they worked fine when requests were made
>> to them directly.
>>
>> I uploaded the config files here, if you could take a look:
>> https://gitlab.com/hobinyoon/trafficserver-config
>>
>> Hobin
>>
>>
>>
>> On Thu, Jan 10, 2019 at 7:31 PM Miles Libbey <ml...@apache.org> wrote:
>>
>> Typically that error means the request doesn't match a rule in
>> remap.config. Is the error coming from the first hop or one of the
>> 123.123.123.[1-4] nodes? That is, if 123.123.123.[1-4] are ATS nodes,
>> are they configured to accept the requests they are getting?
>>
>> On Thu, Jan 10, 2019 at 2:51 PM Hobin Yoon <ho...@gmail.com> wrote:
>> >
>> > With parent.config
>> > dest_domain=. scheme=http parent="123.123.123.1:80
>> <http://123.123.123.1/>,123.123.123.2:80 <http://123.123.123.2/>"
>> round_robin=consistent_hash go_direct=false
>> >
>> > I'm getting
>> >
>> > Not Found on Accelerator
>> > ________________________________
>> > Description: Your request on the specified host was not found. Check
>> the location and try again.
>> >
>> > I must be missing something ...
>> >
>> > Hobin
>> >
>> > On Thu, Jan 10, 2019 at 1:18 PM Hobin Yoon <ho...@gmail.com> wrote:
>> >>
>> >> Alan, that is the only map rule we have with the varying number of
>> cache nodes. During the down time, ATS doesn't return "HTTP/1.1 200 OK" for
>> the requests. I didn't check what it returned.
>> >>
>> >> Miles, I'll check out parent plugin!
>> >>
>> >> Hobin
>> >>
>> >> On Thu, Jan 10, 2019 at 1:13 PM Alan Carroll <so...@oath.com>
>> wrote:
>> >>>
>> >>> It could be an artifact of reloading plugin configurations if you
>> have a lot of remap rules with plugins, although internally ATS should do
>> the load and then swap the configuration. During the down time, does ATS
>> process any traffic, or there is traffic but no caching?
>> >>>
>> >>> On Thu, Jan 10, 2019 at 10:53 AM Miles Libbey <ml...@apache.org>
>> wrote:
>> >>>>
>> >>>> We don't experience downtime when using traffic_ctl config reload (we
>> >>>> use that ~daily).
>> >>>>
>> >>>> We don't use the balancer plugin. Instead, we use parent.config
>> >>>> (
>> https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/parent.config.en.html
>> )
>> >>>> to achieve the same consistent hash. Your config would translate to
>> >>>>
>> >>>> remap.config
>> >>>> map / http://127.0.0.1 @plugin=cachekey.so
>> >>>> @pparam=--include-params=p0,p1 @pparam=--sort-params=true
>> >>>>
>> >>>> parent.config
>> >>>> dest_domain=. scheme=http
>> >>>> parent="123.123.123.1:80 <http://123.123.123.1/>,23.123.123.2:80
>> <http://23.123.123.2/>,123.123.123.3:80 <http://123.123.123.3/>,
>> 123.123.123.4:80 <http://123.123.123.4/>"
>> >>>> round_robin=consistent_hash go_direct=false
>> >>>> dest_domain=. scheme=https
>> >>>> parent="123.123.123.1:443,23.123.123.2:443,123.123.123.3:443,
>> 123.123.123.4:443"
>> >>>> round_robin=consistent_hash go_direct=false
>> >>>>
>> >>>> miles
>> >>>>
>> >>>> On Wed, Jan 9, 2019 at 10:53 PM Hobin Yoon <ho...@gmail.com>
>> wrote:
>> >>>> >
>> >>>> > Hi,
>> >>>> >
>> >>>> > We are noticing there is quite a bit of delay when we reload the
>> config with traffic_ctl config reload. The delay is up to about 30 seconds,
>> during which period we don't get any caching. We are using consistency
>> hashing plugin. The number of nodes changes dynamically between 5 to 30.
>> >>>> >
>> >>>> > Here is an example balancer (consistent hash) configuration in
>> remap.config.
>> >>>> >
>> >>>> > map / http://127.0.0.1 @plugin=cachekey.so
>> @pparam=--include-params=p0,p1 @pparam=--sort-params=true
>> @plugin=balancer.so @pparam=--policy=hash,key @pparam=123.123.123.1
>> @pparam=123.123.123.2 @pparam=123.123.123.3 @pparam=123.123.123.4 ...
>> >>>> >
>> >>>> > Is this downtime normal? How do you guys avoid the service
>> downtime while reconfiguring the cache nodes in the cluster?
>> >>>> >
>> >>>> > Hobin
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Beware the fisherman who's casting out his line in to a dried up
>> riverbed.
>> >>> Oh don't try to tell him 'cause he won't believe. Throw some bread to
>> the ducks instead.
>> >>> It's easier that way. - Genesis : Duke : VI 25-28
>>
>>
>
Re: Downtime when traffic_ctl config reload
Posted by Leif Hedstrom <zw...@apache.org>.
> On Jan 11, 2019, at 10:00 AM, Hobin Yoon <ho...@gmail.com> wrote:
>
> Should you still specify the mapping in remap.config when you have the mapping in parent.config?
Yes. Unless you run an open proxy, and then you have to turn off “require remap”. Not recommended for almost all cases.
— leif
>
>
> On Fri, Jan 11, 2019 at 11:33 AM Randall Meyer <randallmeyer@yahoo.com <ma...@yahoo.com>> wrote:
>
> Looking at your remap.config in that repo, I see that all the mappings are all commented out. Was that intentional?
> On Thursday, January 10, 2019, 8:28:53 PM PST, Hobin Yoon <hobinyoon@gmail.com <ma...@gmail.com>> wrote:
>
>
> Yes, they are also ATS nodes and they worked fine when requests were made to them directly.
>
> I uploaded the config files here, if you could take a look: https://gitlab.com/hobinyoon/trafficserver-config <https://gitlab.com/hobinyoon/trafficserver-config>
>
> Hobin
>
>
>
> On Thu, Jan 10, 2019 at 7:31 PM Miles Libbey <mlibbey@apache.org <ma...@apache.org>> wrote:
> Typically that error means the request doesn't match a rule in
> remap.config. Is the error coming from the first hop or one of the
> 123.123.123.[1-4] nodes? That is, if 123.123.123.[1-4] are ATS nodes,
> are they configured to accept the requests they are getting?
>
> On Thu, Jan 10, 2019 at 2:51 PM Hobin Yoon <hobinyoon@gmail.com <ma...@gmail.com>> wrote:
> >
> > With parent.config
> > dest_domain=. scheme=http parent="123.123.123.1:80 <http://123.123.123.1/>,123.123.123.2:80 <http://123.123.123.2/>" round_robin=consistent_hash go_direct=false
> >
> > I'm getting
> >
> > Not Found on Accelerator
> > ________________________________
> > Description: Your request on the specified host was not found. Check the location and try again.
> >
> > I must be missing something ...
> >
> > Hobin
> >
> > On Thu, Jan 10, 2019 at 1:18 PM Hobin Yoon <hobinyoon@gmail.com <ma...@gmail.com>> wrote:
> >>
> >> Alan, that is the only map rule we have with the varying number of cache nodes. During the down time, ATS doesn't return "HTTP/1.1 200 OK" for the requests. I didn't check what it returned.
> >>
> >> Miles, I'll check out parent plugin!
> >>
> >> Hobin
> >>
> >> On Thu, Jan 10, 2019 at 1:13 PM Alan Carroll <solidwallofcode@oath.com <ma...@oath.com>> wrote:
> >>>
> >>> It could be an artifact of reloading plugin configurations if you have a lot of remap rules with plugins, although internally ATS should do the load and then swap the configuration. During the down time, does ATS process any traffic, or there is traffic but no caching?
> >>>
> >>> On Thu, Jan 10, 2019 at 10:53 AM Miles Libbey <mlibbey@apache.org <ma...@apache.org>> wrote:
> >>>>
> >>>> We don't experience downtime when using traffic_ctl config reload (we
> >>>> use that ~daily).
> >>>>
> >>>> We don't use the balancer plugin. Instead, we use parent.config
> >>>> (https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/parent.config.en.html <https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/parent.config.en.html>)
> >>>> to achieve the same consistent hash. Your config would translate to
> >>>>
> >>>> remap.config
> >>>> map / http://127.0.0.1 <http://127.0.0.1/> @plugin=cachekey.so
> >>>> @pparam=--include-params=p0,p1 @pparam=--sort-params=true
> >>>>
> >>>> parent.config
> >>>> dest_domain=. scheme=http
> >>>> parent="123.123.123.1:80 <http://123.123.123.1/>,23.123.123.2:80 <http://23.123.123.2/>,123.123.123.3:80 <http://123.123.123.3/>,123.123.123.4:80 <http://123.123.123.4/>"
> >>>> round_robin=consistent_hash go_direct=false
> >>>> dest_domain=. scheme=https
> >>>> parent="123.123.123.1:443 <http://123.123.123.1:443/>,23.123.123.2:443 <http://23.123.123.2:443/>,123.123.123.3:443 <http://123.123.123.3:443/>,123.123.123.4:443 <http://123.123.123.4:443/>"
> >>>> round_robin=consistent_hash go_direct=false
> >>>>
> >>>> miles
> >>>>
> >>>> On Wed, Jan 9, 2019 at 10:53 PM Hobin Yoon <hobinyoon@gmail.com <ma...@gmail.com>> wrote:
> >>>> >
> >>>> > Hi,
> >>>> >
> >>>> > We are noticing there is quite a bit of delay when we reload the config with traffic_ctl config reload. The delay is up to about 30 seconds, during which period we don't get any caching. We are using consistency hashing plugin. The number of nodes changes dynamically between 5 to 30.
> >>>> >
> >>>> > Here is an example balancer (consistent hash) configuration in remap.config.
> >>>> >
> >>>> > map / http://127.0.0.1 <http://127.0.0.1/> @plugin=cachekey.so @pparam=--include-params=p0,p1 @pparam=--sort-params=true @plugin=balancer.so @pparam=--policy=hash,key @pparam=123.123.123.1 @pparam=123.123.123.2 @pparam=123.123.123.3 @pparam=123.123.123.4 ...
> >>>> >
> >>>> > Is this downtime normal? How do you guys avoid the service downtime while reconfiguring the cache nodes in the cluster?
> >>>> >
> >>>> > Hobin
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Beware the fisherman who's casting out his line in to a dried up riverbed.
> >>> Oh don't try to tell him 'cause he won't believe. Throw some bread to the ducks instead.
> >>> It's easier that way. - Genesis : Duke : VI 25-28
Re: Downtime when traffic_ctl config reload
Posted by Miles Libbey <ml...@apache.org>.
On Fri, Jan 11, 2019 at 10:13 AM Hobin Yoon <ho...@gmail.com> wrote:
>
> Miles, can I get a minimal working set of config files including parent.config so I can hopefully figure out what's missing?
We do this:
child:
remap.config:
map http://inbound.domain.com http://inbound.domain.com
map https://inbound.domain.com https://inbound.domain.com
parent.config:
dest_domain=inbound.domain.com scheme=http
parent="123.123.123.1:80,23.123.123.2:80,123.123.123.3:80,123.123.123.4:80"
round_robin=consistent_hash go_direct=false
dest_domain=inbound.domain.com scheme=https
parent="123.123.123.1:443,23.123.123.2:443,123.123.123.3:443,123.123.123.4:443"
round_robin=consistent_hash go_direct=false
records.config:
CONFIG proxy.config.http.parent_proxy_routing_enable INT 1
<all the other configs>
parent:
remap.config
map http://inbound.domain.com http://origin.domain.com
map https://inbound.domain.com https://origin.domain.com
records.config has proxy.config.http.parent_proxy_routing_enable=0
> On Fri, Jan 11, 2019 at 12:33 PM Hobin Yoon <ho...@gmail.com> wrote:
>>
>> Another thing I noticed was ATS didn't generate any error when I intentionally introduced an error in parent.config such as typing dest_domain1 instead of dest_domain, which made me wondering if parent.config was being read at all.
>>
>> On Fri, Jan 11, 2019 at 12:00 PM Hobin Yoon <ho...@gmail.com> wrote:
>>>
>>> Should you still specify the mapping in remap.config when you have the mapping in parent.config?
>>>
>>>
>>> On Fri, Jan 11, 2019 at 11:33 AM Randall Meyer <ra...@yahoo.com> wrote:
>>>>
>>>>
>>>> Looking at your remap.config in that repo, I see that all the mappings are all commented out. Was that intentional?
>>>> On Thursday, January 10, 2019, 8:28:53 PM PST, Hobin Yoon <ho...@gmail.com> wrote:
>>>>
>>>>
>>>> Yes, they are also ATS nodes and they worked fine when requests were made to them directly.
>>>>
>>>> I uploaded the config files here, if you could take a look: https://gitlab.com/hobinyoon/trafficserver-config
>>>>
>>>> Hobin
>>>>
>>>>
>>>>
>>>> On Thu, Jan 10, 2019 at 7:31 PM Miles Libbey <ml...@apache.org> wrote:
>>>>
>>>> Typically that error means the request doesn't match a rule in
>>>> remap.config. Is the error coming from the first hop or one of the
>>>> 123.123.123.[1-4] nodes? That is, if 123.123.123.[1-4] are ATS nodes,
>>>> are they configured to accept the requests they are getting?
>>>>
>>>> On Thu, Jan 10, 2019 at 2:51 PM Hobin Yoon <ho...@gmail.com> wrote:
>>>> >
>>>> > With parent.config
>>>> > dest_domain=. scheme=http parent="123.123.123.1:80,123.123.123.2:80" round_robin=consistent_hash go_direct=false
>>>> >
>>>> > I'm getting
>>>> >
>>>> > Not Found on Accelerator
>>>> > ________________________________
>>>> > Description: Your request on the specified host was not found. Check the location and try again.
>>>> >
>>>> > I must be missing something ...
>>>> >
>>>> > Hobin
>>>> >
>>>> > On Thu, Jan 10, 2019 at 1:18 PM Hobin Yoon <ho...@gmail.com> wrote:
>>>> >>
>>>> >> Alan, that is the only map rule we have with the varying number of cache nodes. During the down time, ATS doesn't return "HTTP/1.1 200 OK" for the requests. I didn't check what it returned.
>>>> >>
>>>> >> Miles, I'll check out parent plugin!
>>>> >>
>>>> >> Hobin
>>>> >>
>>>> >> On Thu, Jan 10, 2019 at 1:13 PM Alan Carroll <so...@oath.com> wrote:
>>>> >>>
>>>> >>> It could be an artifact of reloading plugin configurations if you have a lot of remap rules with plugins, although internally ATS should do the load and then swap the configuration. During the down time, does ATS process any traffic, or there is traffic but no caching?
>>>> >>>
>>>> >>> On Thu, Jan 10, 2019 at 10:53 AM Miles Libbey <ml...@apache.org> wrote:
>>>> >>>>
>>>> >>>> We don't experience downtime when using traffic_ctl config reload (we
>>>> >>>> use that ~daily).
>>>> >>>>
>>>> >>>> We don't use the balancer plugin. Instead, we use parent.config
>>>> >>>> (https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/parent.config.en.html)
>>>> >>>> to achieve the same consistent hash. Your config would translate to
>>>> >>>>
>>>> >>>> remap.config
>>>> >>>> map / http://127.0.0.1 @plugin=cachekey.so
>>>> >>>> @pparam=--include-params=p0,p1 @pparam=--sort-params=true
>>>> >>>>
>>>> >>>> parent.config
>>>> >>>> dest_domain=. scheme=http
>>>> >>>> parent="123.123.123.1:80,23.123.123.2:80,123.123.123.3:80,123.123.123.4:80"
>>>> >>>> round_robin=consistent_hash go_direct=false
>>>> >>>> dest_domain=. scheme=https
>>>> >>>> parent="123.123.123.1:443,23.123.123.2:443,123.123.123.3:443,123.123.123.4:443"
>>>> >>>> round_robin=consistent_hash go_direct=false
>>>> >>>>
>>>> >>>> miles
>>>> >>>>
>>>> >>>> On Wed, Jan 9, 2019 at 10:53 PM Hobin Yoon <ho...@gmail.com> wrote:
>>>> >>>> >
>>>> >>>> > Hi,
>>>> >>>> >
>>>> >>>> > We are noticing there is quite a bit of delay when we reload the config with traffic_ctl config reload. The delay is up to about 30 seconds, during which period we don't get any caching. We are using consistency hashing plugin. The number of nodes changes dynamically between 5 to 30.
>>>> >>>> >
>>>> >>>> > Here is an example balancer (consistent hash) configuration in remap.config.
>>>> >>>> >
>>>> >>>> > map / http://127.0.0.1 @plugin=cachekey.so @pparam=--include-params=p0,p1 @pparam=--sort-params=true @plugin=balancer.so @pparam=--policy=hash,key @pparam=123.123.123.1 @pparam=123.123.123.2 @pparam=123.123.123.3 @pparam=123.123.123.4 ...
>>>> >>>> >
>>>> >>>> > Is this downtime normal? How do you guys avoid the service downtime while reconfiguring the cache nodes in the cluster?
>>>> >>>> >
>>>> >>>> > Hobin
>>>> >>>> >
>>>> >>>> >
>>>> >>>> >
>>>> >>>> >
>>>> >>>> >
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> Beware the fisherman who's casting out his line in to a dried up riverbed.
>>>> >>> Oh don't try to tell him 'cause he won't believe. Throw some bread to the ducks instead.
>>>> >>> It's easier that way. - Genesis : Duke : VI 25-28
Re: Downtime when traffic_ctl config reload
Posted by Hobin Yoon <ho...@gmail.com>.
Miles, can I get a minimal working set of config files including
parent.config so I can hopefully figure out what's missing?
On Fri, Jan 11, 2019 at 12:33 PM Hobin Yoon <ho...@gmail.com> wrote:
> Another thing I noticed was ATS didn't generate any error when I
> intentionally introduced an error in parent.config such as typing
> dest_domain1 instead of dest_domain, which made me wondering if
> parent.config was being read at all.
>
> On Fri, Jan 11, 2019 at 12:00 PM Hobin Yoon <ho...@gmail.com> wrote:
>
>> Should you still specify the mapping in remap.config when you have the
>> mapping in parent.config?
>>
>>
>> On Fri, Jan 11, 2019 at 11:33 AM Randall Meyer <ra...@yahoo.com>
>> wrote:
>>
>>>
>>> Looking at your remap.config in that repo, I see that all the mappings
>>> are all commented out. Was that intentional?
>>> On Thursday, January 10, 2019, 8:28:53 PM PST, Hobin Yoon <
>>> hobinyoon@gmail.com> wrote:
>>>
>>>
>>> Yes, they are also ATS nodes and they worked fine when requests were
>>> made to them directly.
>>>
>>> I uploaded the config files here, if you could take a look:
>>> https://gitlab.com/hobinyoon/trafficserver-config
>>>
>>> Hobin
>>>
>>>
>>>
>>> On Thu, Jan 10, 2019 at 7:31 PM Miles Libbey <ml...@apache.org> wrote:
>>>
>>> Typically that error means the request doesn't match a rule in
>>> remap.config. Is the error coming from the first hop or one of the
>>> 123.123.123.[1-4] nodes? That is, if 123.123.123.[1-4] are ATS nodes,
>>> are they configured to accept the requests they are getting?
>>>
>>> On Thu, Jan 10, 2019 at 2:51 PM Hobin Yoon <ho...@gmail.com> wrote:
>>> >
>>> > With parent.config
>>> > dest_domain=. scheme=http parent="123.123.123.1:80,123.123.123.2:80"
>>> round_robin=consistent_hash go_direct=false
>>> >
>>> > I'm getting
>>> >
>>> > Not Found on Accelerator
>>> > ________________________________
>>> > Description: Your request on the specified host was not found. Check
>>> the location and try again.
>>> >
>>> > I must be missing something ...
>>> >
>>> > Hobin
>>> >
>>> > On Thu, Jan 10, 2019 at 1:18 PM Hobin Yoon <ho...@gmail.com>
>>> wrote:
>>> >>
>>> >> Alan, that is the only map rule we have with the varying number of
>>> cache nodes. During the down time, ATS doesn't return "HTTP/1.1 200 OK" for
>>> the requests. I didn't check what it returned.
>>> >>
>>> >> Miles, I'll check out parent plugin!
>>> >>
>>> >> Hobin
>>> >>
>>> >> On Thu, Jan 10, 2019 at 1:13 PM Alan Carroll <
>>> solidwallofcode@oath.com> wrote:
>>> >>>
>>> >>> It could be an artifact of reloading plugin configurations if you
>>> have a lot of remap rules with plugins, although internally ATS should do
>>> the load and then swap the configuration. During the down time, does ATS
>>> process any traffic, or there is traffic but no caching?
>>> >>>
>>> >>> On Thu, Jan 10, 2019 at 10:53 AM Miles Libbey <ml...@apache.org>
>>> wrote:
>>> >>>>
>>> >>>> We don't experience downtime when using traffic_ctl config reload
>>> (we
>>> >>>> use that ~daily).
>>> >>>>
>>> >>>> We don't use the balancer plugin. Instead, we use parent.config
>>> >>>> (
>>> https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/parent.config.en.html
>>> )
>>> >>>> to achieve the same consistent hash. Your config would translate to
>>> >>>>
>>> >>>> remap.config
>>> >>>> map / http://127.0.0.1 @plugin=cachekey.so
>>> >>>> @pparam=--include-params=p0,p1 @pparam=--sort-params=true
>>> >>>>
>>> >>>> parent.config
>>> >>>> dest_domain=. scheme=http
>>> >>>> parent="123.123.123.1:80,23.123.123.2:80,123.123.123.3:80,
>>> 123.123.123.4:80"
>>> >>>> round_robin=consistent_hash go_direct=false
>>> >>>> dest_domain=. scheme=https
>>> >>>> parent="123.123.123.1:443,23.123.123.2:443,123.123.123.3:443,
>>> 123.123.123.4:443"
>>> >>>> round_robin=consistent_hash go_direct=false
>>> >>>>
>>> >>>> miles
>>> >>>>
>>> >>>> On Wed, Jan 9, 2019 at 10:53 PM Hobin Yoon <ho...@gmail.com>
>>> wrote:
>>> >>>> >
>>> >>>> > Hi,
>>> >>>> >
>>> >>>> > We are noticing there is quite a bit of delay when we reload the
>>> config with traffic_ctl config reload. The delay is up to about 30 seconds,
>>> during which period we don't get any caching. We are using consistency
>>> hashing plugin. The number of nodes changes dynamically between 5 to 30.
>>> >>>> >
>>> >>>> > Here is an example balancer (consistent hash) configuration in
>>> remap.config.
>>> >>>> >
>>> >>>> > map / http://127.0.0.1 @plugin=cachekey.so
>>> @pparam=--include-params=p0,p1 @pparam=--sort-params=true
>>> @plugin=balancer.so @pparam=--policy=hash,key @pparam=123.123.123.1
>>> @pparam=123.123.123.2 @pparam=123.123.123.3 @pparam=123.123.123.4 ...
>>> >>>> >
>>> >>>> > Is this downtime normal? How do you guys avoid the service
>>> downtime while reconfiguring the cache nodes in the cluster?
>>> >>>> >
>>> >>>> > Hobin
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Beware the fisherman who's casting out his line in to a dried up
>>> riverbed.
>>> >>> Oh don't try to tell him 'cause he won't believe. Throw some bread
>>> to the ducks instead.
>>> >>> It's easier that way. - Genesis : Duke : VI 25-28
>>>
>>>
Re: Downtime when traffic_ctl config reload
Posted by Hobin Yoon <ho...@gmail.com>.
Another thing I noticed was ATS didn't generate any error when I
intentionally introduced an error in parent.config such as typing
dest_domain1 instead of dest_domain, which made me wondering if
parent.config was being read at all.
On Fri, Jan 11, 2019 at 12:00 PM Hobin Yoon <ho...@gmail.com> wrote:
> Should you still specify the mapping in remap.config when you have the
> mapping in parent.config?
>
>
> On Fri, Jan 11, 2019 at 11:33 AM Randall Meyer <ra...@yahoo.com>
> wrote:
>
>>
>> Looking at your remap.config in that repo, I see that all the mappings
>> are all commented out. Was that intentional?
>> On Thursday, January 10, 2019, 8:28:53 PM PST, Hobin Yoon <
>> hobinyoon@gmail.com> wrote:
>>
>>
>> Yes, they are also ATS nodes and they worked fine when requests were made
>> to them directly.
>>
>> I uploaded the config files here, if you could take a look:
>> https://gitlab.com/hobinyoon/trafficserver-config
>>
>> Hobin
>>
>>
>>
>> On Thu, Jan 10, 2019 at 7:31 PM Miles Libbey <ml...@apache.org> wrote:
>>
>> Typically that error means the request doesn't match a rule in
>> remap.config. Is the error coming from the first hop or one of the
>> 123.123.123.[1-4] nodes? That is, if 123.123.123.[1-4] are ATS nodes,
>> are they configured to accept the requests they are getting?
>>
>> On Thu, Jan 10, 2019 at 2:51 PM Hobin Yoon <ho...@gmail.com> wrote:
>> >
>> > With parent.config
>> > dest_domain=. scheme=http parent="123.123.123.1:80,123.123.123.2:80"
>> round_robin=consistent_hash go_direct=false
>> >
>> > I'm getting
>> >
>> > Not Found on Accelerator
>> > ________________________________
>> > Description: Your request on the specified host was not found. Check
>> the location and try again.
>> >
>> > I must be missing something ...
>> >
>> > Hobin
>> >
>> > On Thu, Jan 10, 2019 at 1:18 PM Hobin Yoon <ho...@gmail.com> wrote:
>> >>
>> >> Alan, that is the only map rule we have with the varying number of
>> cache nodes. During the down time, ATS doesn't return "HTTP/1.1 200 OK" for
>> the requests. I didn't check what it returned.
>> >>
>> >> Miles, I'll check out parent plugin!
>> >>
>> >> Hobin
>> >>
>> >> On Thu, Jan 10, 2019 at 1:13 PM Alan Carroll <so...@oath.com>
>> wrote:
>> >>>
>> >>> It could be an artifact of reloading plugin configurations if you
>> have a lot of remap rules with plugins, although internally ATS should do
>> the load and then swap the configuration. During the down time, does ATS
>> process any traffic, or there is traffic but no caching?
>> >>>
>> >>> On Thu, Jan 10, 2019 at 10:53 AM Miles Libbey <ml...@apache.org>
>> wrote:
>> >>>>
>> >>>> We don't experience downtime when using traffic_ctl config reload (we
>> >>>> use that ~daily).
>> >>>>
>> >>>> We don't use the balancer plugin. Instead, we use parent.config
>> >>>> (
>> https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/parent.config.en.html
>> )
>> >>>> to achieve the same consistent hash. Your config would translate to
>> >>>>
>> >>>> remap.config
>> >>>> map / http://127.0.0.1 @plugin=cachekey.so
>> >>>> @pparam=--include-params=p0,p1 @pparam=--sort-params=true
>> >>>>
>> >>>> parent.config
>> >>>> dest_domain=. scheme=http
>> >>>> parent="123.123.123.1:80,23.123.123.2:80,123.123.123.3:80,
>> 123.123.123.4:80"
>> >>>> round_robin=consistent_hash go_direct=false
>> >>>> dest_domain=. scheme=https
>> >>>> parent="123.123.123.1:443,23.123.123.2:443,123.123.123.3:443,
>> 123.123.123.4:443"
>> >>>> round_robin=consistent_hash go_direct=false
>> >>>>
>> >>>> miles
>> >>>>
>> >>>> On Wed, Jan 9, 2019 at 10:53 PM Hobin Yoon <ho...@gmail.com>
>> wrote:
>> >>>> >
>> >>>> > Hi,
>> >>>> >
>> >>>> > We are noticing there is quite a bit of delay when we reload the
>> config with traffic_ctl config reload. The delay is up to about 30 seconds,
>> during which period we don't get any caching. We are using consistency
>> hashing plugin. The number of nodes changes dynamically between 5 to 30.
>> >>>> >
>> >>>> > Here is an example balancer (consistent hash) configuration in
>> remap.config.
>> >>>> >
>> >>>> > map / http://127.0.0.1 @plugin=cachekey.so
>> @pparam=--include-params=p0,p1 @pparam=--sort-params=true
>> @plugin=balancer.so @pparam=--policy=hash,key @pparam=123.123.123.1
>> @pparam=123.123.123.2 @pparam=123.123.123.3 @pparam=123.123.123.4 ...
>> >>>> >
>> >>>> > Is this downtime normal? How do you guys avoid the service
>> downtime while reconfiguring the cache nodes in the cluster?
>> >>>> >
>> >>>> > Hobin
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Beware the fisherman who's casting out his line in to a dried up
>> riverbed.
>> >>> Oh don't try to tell him 'cause he won't believe. Throw some bread to
>> the ducks instead.
>> >>> It's easier that way. - Genesis : Duke : VI 25-28
>>
>>
Re: Downtime when traffic_ctl config reload
Posted by Hobin Yoon <ho...@gmail.com>.
Should you still specify the mapping in remap.config when you have the
mapping in parent.config?
On Fri, Jan 11, 2019 at 11:33 AM Randall Meyer <ra...@yahoo.com>
wrote:
>
> Looking at your remap.config in that repo, I see that all the mappings are
> all commented out. Was that intentional?
> On Thursday, January 10, 2019, 8:28:53 PM PST, Hobin Yoon <
> hobinyoon@gmail.com> wrote:
>
>
> Yes, they are also ATS nodes and they worked fine when requests were made
> to them directly.
>
> I uploaded the config files here, if you could take a look:
> https://gitlab.com/hobinyoon/trafficserver-config
>
> Hobin
>
>
>
> On Thu, Jan 10, 2019 at 7:31 PM Miles Libbey <ml...@apache.org> wrote:
>
> Typically that error means the request doesn't match a rule in
> remap.config. Is the error coming from the first hop or one of the
> 123.123.123.[1-4] nodes? That is, if 123.123.123.[1-4] are ATS nodes,
> are they configured to accept the requests they are getting?
>
> On Thu, Jan 10, 2019 at 2:51 PM Hobin Yoon <ho...@gmail.com> wrote:
> >
> > With parent.config
> > dest_domain=. scheme=http parent="123.123.123.1:80,123.123.123.2:80"
> round_robin=consistent_hash go_direct=false
> >
> > I'm getting
> >
> > Not Found on Accelerator
> > ________________________________
> > Description: Your request on the specified host was not found. Check the
> location and try again.
> >
> > I must be missing something ...
> >
> > Hobin
> >
> > On Thu, Jan 10, 2019 at 1:18 PM Hobin Yoon <ho...@gmail.com> wrote:
> >>
> >> Alan, that is the only map rule we have with the varying number of
> cache nodes. During the down time, ATS doesn't return "HTTP/1.1 200 OK" for
> the requests. I didn't check what it returned.
> >>
> >> Miles, I'll check out parent plugin!
> >>
> >> Hobin
> >>
> >> On Thu, Jan 10, 2019 at 1:13 PM Alan Carroll <so...@oath.com>
> wrote:
> >>>
> >>> It could be an artifact of reloading plugin configurations if you have
> a lot of remap rules with plugins, although internally ATS should do the
> load and then swap the configuration. During the down time, does ATS
> process any traffic, or there is traffic but no caching?
> >>>
> >>> On Thu, Jan 10, 2019 at 10:53 AM Miles Libbey <ml...@apache.org>
> wrote:
> >>>>
> >>>> We don't experience downtime when using traffic_ctl config reload (we
> >>>> use that ~daily).
> >>>>
> >>>> We don't use the balancer plugin. Instead, we use parent.config
> >>>> (
> https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/parent.config.en.html
> )
> >>>> to achieve the same consistent hash. Your config would translate to
> >>>>
> >>>> remap.config
> >>>> map / http://127.0.0.1 @plugin=cachekey.so
> >>>> @pparam=--include-params=p0,p1 @pparam=--sort-params=true
> >>>>
> >>>> parent.config
> >>>> dest_domain=. scheme=http
> >>>> parent="123.123.123.1:80,23.123.123.2:80,123.123.123.3:80,
> 123.123.123.4:80"
> >>>> round_robin=consistent_hash go_direct=false
> >>>> dest_domain=. scheme=https
> >>>> parent="123.123.123.1:443,23.123.123.2:443,123.123.123.3:443,
> 123.123.123.4:443"
> >>>> round_robin=consistent_hash go_direct=false
> >>>>
> >>>> miles
> >>>>
> >>>> On Wed, Jan 9, 2019 at 10:53 PM Hobin Yoon <ho...@gmail.com>
> wrote:
> >>>> >
> >>>> > Hi,
> >>>> >
> >>>> > We are noticing there is quite a bit of delay when we reload the
> config with traffic_ctl config reload. The delay is up to about 30 seconds,
> during which period we don't get any caching. We are using consistency
> hashing plugin. The number of nodes changes dynamically between 5 to 30.
> >>>> >
> >>>> > Here is an example balancer (consistent hash) configuration in
> remap.config.
> >>>> >
> >>>> > map / http://127.0.0.1 @plugin=cachekey.so
> @pparam=--include-params=p0,p1 @pparam=--sort-params=true
> @plugin=balancer.so @pparam=--policy=hash,key @pparam=123.123.123.1
> @pparam=123.123.123.2 @pparam=123.123.123.3 @pparam=123.123.123.4 ...
> >>>> >
> >>>> > Is this downtime normal? How do you guys avoid the service downtime
> while reconfiguring the cache nodes in the cluster?
> >>>> >
> >>>> > Hobin
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Beware the fisherman who's casting out his line in to a dried up
> riverbed.
> >>> Oh don't try to tell him 'cause he won't believe. Throw some bread to
> the ducks instead.
> >>> It's easier that way. - Genesis : Duke : VI 25-28
>
>
Re: Downtime when traffic_ctl config reload
Posted by Randall Meyer <ra...@yahoo.com>.
Looking at your remap.config in that repo, I see that all the mappings are all commented out. Was that intentional? On Thursday, January 10, 2019, 8:28:53 PM PST, Hobin Yoon <ho...@gmail.com> wrote:
Yes, they are also ATS nodes and they worked fine when requests were made to them directly.
I uploaded the config files here, if you could take a look: https://gitlab.com/hobinyoon/trafficserver-config
Hobin
On Thu, Jan 10, 2019 at 7:31 PM Miles Libbey <ml...@apache.org> wrote:
Typically that error means the request doesn't match a rule in
remap.config. Is the error coming from the first hop or one of the
123.123.123.[1-4] nodes? That is, if 123.123.123.[1-4] are ATS nodes,
are they configured to accept the requests they are getting?
On Thu, Jan 10, 2019 at 2:51 PM Hobin Yoon <ho...@gmail.com> wrote:
>
> With parent.config
> dest_domain=. scheme=http parent="123.123.123.1:80,123.123.123.2:80" round_robin=consistent_hash go_direct=false
>
> I'm getting
>
> Not Found on Accelerator
> ________________________________
> Description: Your request on the specified host was not found. Check the location and try again.
>
> I must be missing something ...
>
> Hobin
>
> On Thu, Jan 10, 2019 at 1:18 PM Hobin Yoon <ho...@gmail.com> wrote:
>>
>> Alan, that is the only map rule we have with the varying number of cache nodes. During the down time, ATS doesn't return "HTTP/1.1 200 OK" for the requests. I didn't check what it returned.
>>
>> Miles, I'll check out parent plugin!
>>
>> Hobin
>>
>> On Thu, Jan 10, 2019 at 1:13 PM Alan Carroll <so...@oath.com> wrote:
>>>
>>> It could be an artifact of reloading plugin configurations if you have a lot of remap rules with plugins, although internally ATS should do the load and then swap the configuration. During the down time, does ATS process any traffic, or there is traffic but no caching?
>>>
>>> On Thu, Jan 10, 2019 at 10:53 AM Miles Libbey <ml...@apache.org> wrote:
>>>>
>>>> We don't experience downtime when using traffic_ctl config reload (we
>>>> use that ~daily).
>>>>
>>>> We don't use the balancer plugin. Instead, we use parent.config
>>>> (https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/parent.config.en.html)
>>>> to achieve the same consistent hash. Your config would translate to
>>>>
>>>> remap.config
>>>> map / http://127.0.0.1 @plugin=cachekey.so
>>>> @pparam=--include-params=p0,p1 @pparam=--sort-params=true
>>>>
>>>> parent.config
>>>> dest_domain=. scheme=http
>>>> parent="123.123.123.1:80,23.123.123.2:80,123.123.123.3:80,123.123.123.4:80"
>>>> round_robin=consistent_hash go_direct=false
>>>> dest_domain=. scheme=https
>>>> parent="123.123.123.1:443,23.123.123.2:443,123.123.123.3:443,123.123.123.4:443"
>>>> round_robin=consistent_hash go_direct=false
>>>>
>>>> miles
>>>>
>>>> On Wed, Jan 9, 2019 at 10:53 PM Hobin Yoon <ho...@gmail.com> wrote:
>>>> >
>>>> > Hi,
>>>> >
>>>> > We are noticing there is quite a bit of delay when we reload the config with traffic_ctl config reload. The delay is up to about 30 seconds, during which period we don't get any caching. We are using consistency hashing plugin. The number of nodes changes dynamically between 5 to 30.
>>>> >
>>>> > Here is an example balancer (consistent hash) configuration in remap.config.
>>>> >
>>>> > map / http://127.0.0.1 @plugin=cachekey.so @pparam=--include-params=p0,p1 @pparam=--sort-params=true @plugin=balancer.so @pparam=--policy=hash,key @pparam=123.123.123.1 @pparam=123.123.123.2 @pparam=123.123.123.3 @pparam=123.123.123.4 ...
>>>> >
>>>> > Is this downtime normal? How do you guys avoid the service downtime while reconfiguring the cache nodes in the cluster?
>>>> >
>>>> > Hobin
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>
>>>
>>>
>>> --
>>> Beware the fisherman who's casting out his line in to a dried up riverbed.
>>> Oh don't try to tell him 'cause he won't believe. Throw some bread to the ducks instead.
>>> It's easier that way. - Genesis : Duke : VI 25-28
Re: Downtime when traffic_ctl config reload
Posted by Hobin Yoon <ho...@gmail.com>.
Yes, they are also ATS nodes and they worked fine when requests were made
to them directly.
I uploaded the config files here, if you could take a look:
https://gitlab.com/hobinyoon/trafficserver-config
Hobin
On Thu, Jan 10, 2019 at 7:31 PM Miles Libbey <ml...@apache.org> wrote:
> Typically that error means the request doesn't match a rule in
> remap.config. Is the error coming from the first hop or one of the
> 123.123.123.[1-4] nodes? That is, if 123.123.123.[1-4] are ATS nodes,
> are they configured to accept the requests they are getting?
>
> On Thu, Jan 10, 2019 at 2:51 PM Hobin Yoon <ho...@gmail.com> wrote:
> >
> > With parent.config
> > dest_domain=. scheme=http parent="123.123.123.1:80,123.123.123.2:80"
> round_robin=consistent_hash go_direct=false
> >
> > I'm getting
> >
> > Not Found on Accelerator
> > ________________________________
> > Description: Your request on the specified host was not found. Check the
> location and try again.
> >
> > I must be missing something ...
> >
> > Hobin
> >
> > On Thu, Jan 10, 2019 at 1:18 PM Hobin Yoon <ho...@gmail.com> wrote:
> >>
> >> Alan, that is the only map rule we have with the varying number of
> cache nodes. During the down time, ATS doesn't return "HTTP/1.1 200 OK" for
> the requests. I didn't check what it returned.
> >>
> >> Miles, I'll check out parent plugin!
> >>
> >> Hobin
> >>
> >> On Thu, Jan 10, 2019 at 1:13 PM Alan Carroll <so...@oath.com>
> wrote:
> >>>
> >>> It could be an artifact of reloading plugin configurations if you have
> a lot of remap rules with plugins, although internally ATS should do the
> load and then swap the configuration. During the down time, does ATS
> process any traffic, or there is traffic but no caching?
> >>>
> >>> On Thu, Jan 10, 2019 at 10:53 AM Miles Libbey <ml...@apache.org>
> wrote:
> >>>>
> >>>> We don't experience downtime when using traffic_ctl config reload (we
> >>>> use that ~daily).
> >>>>
> >>>> We don't use the balancer plugin. Instead, we use parent.config
> >>>> (
> https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/parent.config.en.html
> )
> >>>> to achieve the same consistent hash. Your config would translate to
> >>>>
> >>>> remap.config
> >>>> map / http://127.0.0.1 @plugin=cachekey.so
> >>>> @pparam=--include-params=p0,p1 @pparam=--sort-params=true
> >>>>
> >>>> parent.config
> >>>> dest_domain=. scheme=http
> >>>> parent="123.123.123.1:80,23.123.123.2:80,123.123.123.3:80,
> 123.123.123.4:80"
> >>>> round_robin=consistent_hash go_direct=false
> >>>> dest_domain=. scheme=https
> >>>> parent="123.123.123.1:443,23.123.123.2:443,123.123.123.3:443,
> 123.123.123.4:443"
> >>>> round_robin=consistent_hash go_direct=false
> >>>>
> >>>> miles
> >>>>
> >>>> On Wed, Jan 9, 2019 at 10:53 PM Hobin Yoon <ho...@gmail.com>
> wrote:
> >>>> >
> >>>> > Hi,
> >>>> >
> >>>> > We are noticing there is quite a bit of delay when we reload the
> config with traffic_ctl config reload. The delay is up to about 30 seconds,
> during which period we don't get any caching. We are using consistency
> hashing plugin. The number of nodes changes dynamically between 5 to 30.
> >>>> >
> >>>> > Here is an example balancer (consistent hash) configuration in
> remap.config.
> >>>> >
> >>>> > map / http://127.0.0.1 @plugin=cachekey.so
> @pparam=--include-params=p0,p1 @pparam=--sort-params=true
> @plugin=balancer.so @pparam=--policy=hash,key @pparam=123.123.123.1
> @pparam=123.123.123.2 @pparam=123.123.123.3 @pparam=123.123.123.4 ...
> >>>> >
> >>>> > Is this downtime normal? How do you guys avoid the service downtime
> while reconfiguring the cache nodes in the cluster?
> >>>> >
> >>>> > Hobin
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Beware the fisherman who's casting out his line in to a dried up
> riverbed.
> >>> Oh don't try to tell him 'cause he won't believe. Throw some bread to
> the ducks instead.
> >>> It's easier that way. - Genesis : Duke : VI 25-28
>
Re: Downtime when traffic_ctl config reload
Posted by Miles Libbey <ml...@apache.org>.
Typically that error means the request doesn't match a rule in
remap.config. Is the error coming from the first hop or one of the
123.123.123.[1-4] nodes? That is, if 123.123.123.[1-4] are ATS nodes,
are they configured to accept the requests they are getting?
On Thu, Jan 10, 2019 at 2:51 PM Hobin Yoon <ho...@gmail.com> wrote:
>
> With parent.config
> dest_domain=. scheme=http parent="123.123.123.1:80,123.123.123.2:80" round_robin=consistent_hash go_direct=false
>
> I'm getting
>
> Not Found on Accelerator
> ________________________________
> Description: Your request on the specified host was not found. Check the location and try again.
>
> I must be missing something ...
>
> Hobin
>
> On Thu, Jan 10, 2019 at 1:18 PM Hobin Yoon <ho...@gmail.com> wrote:
>>
>> Alan, that is the only map rule we have with the varying number of cache nodes. During the down time, ATS doesn't return "HTTP/1.1 200 OK" for the requests. I didn't check what it returned.
>>
>> Miles, I'll check out parent plugin!
>>
>> Hobin
>>
>> On Thu, Jan 10, 2019 at 1:13 PM Alan Carroll <so...@oath.com> wrote:
>>>
>>> It could be an artifact of reloading plugin configurations if you have a lot of remap rules with plugins, although internally ATS should do the load and then swap the configuration. During the down time, does ATS process any traffic, or there is traffic but no caching?
>>>
>>> On Thu, Jan 10, 2019 at 10:53 AM Miles Libbey <ml...@apache.org> wrote:
>>>>
>>>> We don't experience downtime when using traffic_ctl config reload (we
>>>> use that ~daily).
>>>>
>>>> We don't use the balancer plugin. Instead, we use parent.config
>>>> (https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/parent.config.en.html)
>>>> to achieve the same consistent hash. Your config would translate to
>>>>
>>>> remap.config
>>>> map / http://127.0.0.1 @plugin=cachekey.so
>>>> @pparam=--include-params=p0,p1 @pparam=--sort-params=true
>>>>
>>>> parent.config
>>>> dest_domain=. scheme=http
>>>> parent="123.123.123.1:80,23.123.123.2:80,123.123.123.3:80,123.123.123.4:80"
>>>> round_robin=consistent_hash go_direct=false
>>>> dest_domain=. scheme=https
>>>> parent="123.123.123.1:443,23.123.123.2:443,123.123.123.3:443,123.123.123.4:443"
>>>> round_robin=consistent_hash go_direct=false
>>>>
>>>> miles
>>>>
>>>> On Wed, Jan 9, 2019 at 10:53 PM Hobin Yoon <ho...@gmail.com> wrote:
>>>> >
>>>> > Hi,
>>>> >
>>>> > We are noticing there is quite a bit of delay when we reload the config with traffic_ctl config reload. The delay is up to about 30 seconds, during which period we don't get any caching. We are using consistency hashing plugin. The number of nodes changes dynamically between 5 to 30.
>>>> >
>>>> > Here is an example balancer (consistent hash) configuration in remap.config.
>>>> >
>>>> > map / http://127.0.0.1 @plugin=cachekey.so @pparam=--include-params=p0,p1 @pparam=--sort-params=true @plugin=balancer.so @pparam=--policy=hash,key @pparam=123.123.123.1 @pparam=123.123.123.2 @pparam=123.123.123.3 @pparam=123.123.123.4 ...
>>>> >
>>>> > Is this downtime normal? How do you guys avoid the service downtime while reconfiguring the cache nodes in the cluster?
>>>> >
>>>> > Hobin
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>
>>>
>>>
>>> --
>>> Beware the fisherman who's casting out his line in to a dried up riverbed.
>>> Oh don't try to tell him 'cause he won't believe. Throw some bread to the ducks instead.
>>> It's easier that way. - Genesis : Duke : VI 25-28
Re: Downtime when traffic_ctl config reload
Posted by Hobin Yoon <ho...@gmail.com>.
With parent.config
dest_domain=. scheme=http parent="123.123.123.1:80,123.123.123.2:80"
round_robin=consistent_hash go_direct=false
I'm getting
Not Found on Accelerator
________________________________
Description: Your request on the specified host was not found. Check the
location and try again.
I must be missing something ...
Hobin
On Thu, Jan 10, 2019 at 1:18 PM Hobin Yoon <ho...@gmail.com> wrote:
> Alan, that is the only map rule we have with the varying number of cache
> nodes. During the down time, ATS doesn't return "HTTP/1.1 200 OK" for the
> requests. I didn't check what it returned.
>
> Miles, I'll check out parent plugin!
>
> Hobin
>
> On Thu, Jan 10, 2019 at 1:13 PM Alan Carroll <so...@oath.com>
> wrote:
>
>> It could be an artifact of reloading plugin configurations if you have a
>> lot of remap rules with plugins, although internally ATS should do the load
>> and then swap the configuration. During the down time, does ATS process any
>> traffic, or there is traffic but no caching?
>>
>> On Thu, Jan 10, 2019 at 10:53 AM Miles Libbey <ml...@apache.org> wrote:
>>
>>> We don't experience downtime when using traffic_ctl config reload (we
>>> use that ~daily).
>>>
>>> We don't use the balancer plugin. Instead, we use parent.config
>>> (
>>> https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/parent.config.en.html
>>> )
>>> to achieve the same consistent hash. Your config would translate to
>>>
>>> remap.config
>>> map / http://127.0.0.1 @plugin=cachekey.so
>>> @pparam=--include-params=p0,p1 @pparam=--sort-params=true
>>>
>>> parent.config
>>> dest_domain=. scheme=http
>>> parent="123.123.123.1:80,23.123.123.2:80,123.123.123.3:80,
>>> 123.123.123.4:80"
>>> round_robin=consistent_hash go_direct=false
>>> dest_domain=. scheme=https
>>> parent="123.123.123.1:443,23.123.123.2:443,123.123.123.3:443,
>>> 123.123.123.4:443"
>>> round_robin=consistent_hash go_direct=false
>>>
>>> miles
>>>
>>> On Wed, Jan 9, 2019 at 10:53 PM Hobin Yoon <ho...@gmail.com> wrote:
>>> >
>>> > Hi,
>>> >
>>> > We are noticing there is quite a bit of delay when we reload the
>>> config with traffic_ctl config reload. The delay is up to about 30 seconds,
>>> during which period we don't get any caching. We are using consistency
>>> hashing plugin. The number of nodes changes dynamically between 5 to 30.
>>> >
>>> > Here is an example balancer (consistent hash) configuration in
>>> remap.config.
>>> >
>>> > map / http://127.0.0.1 @plugin=cachekey.so
>>> @pparam=--include-params=p0,p1 @pparam=--sort-params=true
>>> @plugin=balancer.so @pparam=--policy=hash,key @pparam=123.123.123.1
>>> @pparam=123.123.123.2 @pparam=123.123.123.3 @pparam=123.123.123.4 ...
>>> >
>>> > Is this downtime normal? How do you guys avoid the service downtime
>>> while reconfiguring the cache nodes in the cluster?
>>> >
>>> > Hobin
>>> >
>>> >
>>> >
>>> >
>>> >
>>>
>>
>>
>> --
>> *Beware the fisherman who's casting out his line in to a dried up
>> riverbed.*
>> *Oh don't try to tell him 'cause he won't believe. Throw some bread to
>> the ducks instead.*
>> *It's easier that way. *- Genesis : Duke : VI 25-28
>>
>
Re: Downtime when traffic_ctl config reload
Posted by Hobin Yoon <ho...@gmail.com>.
Alan, that is the only map rule we have with the varying number of cache
nodes. During the down time, ATS doesn't return "HTTP/1.1 200 OK" for the
requests. I didn't check what it returned.
Miles, I'll check out parent plugin!
Hobin
On Thu, Jan 10, 2019 at 1:13 PM Alan Carroll <so...@oath.com>
wrote:
> It could be an artifact of reloading plugin configurations if you have a
> lot of remap rules with plugins, although internally ATS should do the load
> and then swap the configuration. During the down time, does ATS process any
> traffic, or there is traffic but no caching?
>
> On Thu, Jan 10, 2019 at 10:53 AM Miles Libbey <ml...@apache.org> wrote:
>
>> We don't experience downtime when using traffic_ctl config reload (we
>> use that ~daily).
>>
>> We don't use the balancer plugin. Instead, we use parent.config
>> (
>> https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/parent.config.en.html
>> )
>> to achieve the same consistent hash. Your config would translate to
>>
>> remap.config
>> map / http://127.0.0.1 @plugin=cachekey.so
>> @pparam=--include-params=p0,p1 @pparam=--sort-params=true
>>
>> parent.config
>> dest_domain=. scheme=http
>> parent="123.123.123.1:80,23.123.123.2:80,123.123.123.3:80,
>> 123.123.123.4:80"
>> round_robin=consistent_hash go_direct=false
>> dest_domain=. scheme=https
>> parent="123.123.123.1:443,23.123.123.2:443,123.123.123.3:443,
>> 123.123.123.4:443"
>> round_robin=consistent_hash go_direct=false
>>
>> miles
>>
>> On Wed, Jan 9, 2019 at 10:53 PM Hobin Yoon <ho...@gmail.com> wrote:
>> >
>> > Hi,
>> >
>> > We are noticing there is quite a bit of delay when we reload the config
>> with traffic_ctl config reload. The delay is up to about 30 seconds, during
>> which period we don't get any caching. We are using consistency hashing
>> plugin. The number of nodes changes dynamically between 5 to 30.
>> >
>> > Here is an example balancer (consistent hash) configuration in
>> remap.config.
>> >
>> > map / http://127.0.0.1 @plugin=cachekey.so
>> @pparam=--include-params=p0,p1 @pparam=--sort-params=true
>> @plugin=balancer.so @pparam=--policy=hash,key @pparam=123.123.123.1
>> @pparam=123.123.123.2 @pparam=123.123.123.3 @pparam=123.123.123.4 ...
>> >
>> > Is this downtime normal? How do you guys avoid the service downtime
>> while reconfiguring the cache nodes in the cluster?
>> >
>> > Hobin
>> >
>> >
>> >
>> >
>> >
>>
>
>
> --
> *Beware the fisherman who's casting out his line in to a dried up
> riverbed.*
> *Oh don't try to tell him 'cause he won't believe. Throw some bread to the
> ducks instead.*
> *It's easier that way. *- Genesis : Duke : VI 25-28
>
Re: Downtime when traffic_ctl config reload
Posted by Alan Carroll <so...@oath.com>.
It could be an artifact of reloading plugin configurations if you have a
lot of remap rules with plugins, although internally ATS should do the load
and then swap the configuration. During the down time, does ATS process any
traffic, or there is traffic but no caching?
On Thu, Jan 10, 2019 at 10:53 AM Miles Libbey <ml...@apache.org> wrote:
> We don't experience downtime when using traffic_ctl config reload (we
> use that ~daily).
>
> We don't use the balancer plugin. Instead, we use parent.config
> (
> https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/parent.config.en.html
> )
> to achieve the same consistent hash. Your config would translate to
>
> remap.config
> map / http://127.0.0.1 @plugin=cachekey.so
> @pparam=--include-params=p0,p1 @pparam=--sort-params=true
>
> parent.config
> dest_domain=. scheme=http
> parent="123.123.123.1:80,23.123.123.2:80,123.123.123.3:80,123.123.123.4:80
> "
> round_robin=consistent_hash go_direct=false
> dest_domain=. scheme=https
> parent="123.123.123.1:443,23.123.123.2:443,123.123.123.3:443,
> 123.123.123.4:443"
> round_robin=consistent_hash go_direct=false
>
> miles
>
> On Wed, Jan 9, 2019 at 10:53 PM Hobin Yoon <ho...@gmail.com> wrote:
> >
> > Hi,
> >
> > We are noticing there is quite a bit of delay when we reload the config
> with traffic_ctl config reload. The delay is up to about 30 seconds, during
> which period we don't get any caching. We are using consistency hashing
> plugin. The number of nodes changes dynamically between 5 to 30.
> >
> > Here is an example balancer (consistent hash) configuration in
> remap.config.
> >
> > map / http://127.0.0.1 @plugin=cachekey.so
> @pparam=--include-params=p0,p1 @pparam=--sort-params=true
> @plugin=balancer.so @pparam=--policy=hash,key @pparam=123.123.123.1
> @pparam=123.123.123.2 @pparam=123.123.123.3 @pparam=123.123.123.4 ...
> >
> > Is this downtime normal? How do you guys avoid the service downtime
> while reconfiguring the cache nodes in the cluster?
> >
> > Hobin
> >
> >
> >
> >
> >
>
--
*Beware the fisherman who's casting out his line in to a dried up riverbed.*
*Oh don't try to tell him 'cause he won't believe. Throw some bread to the
ducks instead.*
*It's easier that way. *- Genesis : Duke : VI 25-28
Re: Downtime when traffic_ctl config reload
Posted by Miles Libbey <ml...@apache.org>.
We don't experience downtime when using traffic_ctl config reload (we
use that ~daily).
We don't use the balancer plugin. Instead, we use parent.config
(https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/parent.config.en.html)
to achieve the same consistent hash. Your config would translate to
remap.config
map / http://127.0.0.1 @plugin=cachekey.so
@pparam=--include-params=p0,p1 @pparam=--sort-params=true
parent.config
dest_domain=. scheme=http
parent="123.123.123.1:80,23.123.123.2:80,123.123.123.3:80,123.123.123.4:80"
round_robin=consistent_hash go_direct=false
dest_domain=. scheme=https
parent="123.123.123.1:443,23.123.123.2:443,123.123.123.3:443,123.123.123.4:443"
round_robin=consistent_hash go_direct=false
miles
On Wed, Jan 9, 2019 at 10:53 PM Hobin Yoon <ho...@gmail.com> wrote:
>
> Hi,
>
> We are noticing there is quite a bit of delay when we reload the config with traffic_ctl config reload. The delay is up to about 30 seconds, during which period we don't get any caching. We are using consistency hashing plugin. The number of nodes changes dynamically between 5 to 30.
>
> Here is an example balancer (consistent hash) configuration in remap.config.
>
> map / http://127.0.0.1 @plugin=cachekey.so @pparam=--include-params=p0,p1 @pparam=--sort-params=true @plugin=balancer.so @pparam=--policy=hash,key @pparam=123.123.123.1 @pparam=123.123.123.2 @pparam=123.123.123.3 @pparam=123.123.123.4 ...
>
> Is this downtime normal? How do you guys avoid the service downtime while reconfiguring the cache nodes in the cluster?
>
> Hobin
>
>
>
>
>