You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by Erez Koler <Er...@harmonicinc.com> on 2019/02/19 20:44:37 UTC
RE: [EXTERNAL] Re: corrupt data with multiple concurrent client
No , diags looks normal
-----Original Message-----
From: Jeremy Payne [mailto:jp557198@gmail.com]
Sent: Tuesday, February 19, 2019 10:36 PM
To: users@trafficserver.apache.org
Subject: [EXTERNAL] Re: corrupt data with multiple concurrent client
question... are you seeing 'document truncated' in the diag log?
On Tue, Feb 19, 2019 at 1:45 PM ezko <er...@harmonicinc.com> wrote:
>
> hi,
> with ATS 7.1.6 reverse proxy for live HLS , we are seeing corrupted data on
> the playlist when 300 concurrent clients are playing the same content (same
> profile). we tried several client simulators like locust and jmeter HLS
> plugins with same results.
> we configured collapsed forward plugin and RWW as described in the docs.
> collapsed_forwarding
> <https://docs.trafficserver.apache.org/en/7.1.x/admin-guide/plugins/collapsed_forwarding.en.html?highlight=collapsed%20forwarding%20plugin#collapsed-forwarding-plugin>
> reducing-origin-server-requests-avoiding-the-thundering-herd
> <https://docs.trafficserver.apache.org/en/7.1.x/admin-guide/configuration/cache-basics.en.html?highlight=thundering#reducing-origin-server-requests-avoiding-the-thundering-herd>
> any ideas how to debug ?
> Thanks.
> erez
>
>
>
>
>
> --
> Sent from: http://apache-traffic-server.24303.n7.nabble.com/
Re: [EXTERNAL] Re: corrupt data with multiple concurrent client
Posted by Erez Koler <Er...@harmonicinc.com>.
Wrong content length. Corruption in data itself , not headers (response code is 200)
Get Outlook for Android<https://aka.ms/ghei36>
From: Leif Hedstrom
Sent: Tuesday, 19 February, 22:51
Subject: Re: [EXTERNAL] Re: corrupt data with multiple concurrent client
To: Mailing-List trafficserver
> On Feb 19, 2019, at 1:44 PM, Erez Koler wrote: > > No , diags looks normal > How are the corruptions manifesting themselves? Truncated data/ Wrong Content-Length ? Actual corruption in the data itself ? — leif
Re: [EXTERNAL] Re: corrupt data with multiple concurrent client
Posted by Leif Hedstrom <zw...@apache.org>.
> On Feb 19, 2019, at 1:44 PM, Erez Koler <Er...@harmonicinc.com> wrote:
>
> No , diags looks normal
>
How are the corruptions manifesting themselves? Truncated data/ Wrong Content-Length ? Actual corruption in the data itself ?
— leif
Re: [EXTERNAL] Re: corrupt data with multiple concurrent client
Posted by ezko <er...@harmonicinc.com>.
so looking even closer we now believe that the problem is with the client
simulator. it seems locust for example accepts gzip encoding , and the
origin does sometimes returns gzip (so ATS has 2 alternates). so probably
locust doesn't handle correctly the missing Content-Length with gzip
encoding , thus showing false negatives.
thanks.
--
Sent from: http://apache-traffic-server.24303.n7.nabble.com/
Re: [EXTERNAL] Re: corrupt data with multiple concurrent client
Posted by ezko <er...@harmonicinc.com>.
OK , so i took a closer look and it seems ATS sometimes returns
Content-Encoding: gzip (so it's not garbage) , although origin always
returns Content-Type: application/vnd.apple.mpegurl.
we are not using any compress plugins , so why does ATS return compress data
?
thanks for the support,
erez
origin response:
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Access-Control-Allow-Origin: *
< Access-Control-Expose-Headers: Date
< Cache-Control: max-age=1
< Cache-Control: public
< Cache-control: no-cache="set-cookie"
< Content-Type: application/vnd.apple.mpegurl
< Date: Wed, 20 Feb 2019 07:04:51 GMT
< ETag: "5c6cfbe0-341"
< Expires: Wed, 20 Feb 2019 07:04:52 GMT
< Last-Modified: Wed, 20 Feb 2019 07:04:00 GMT
< Set-Cookie: AWSELB=XXXXXX;PATH=/;MAX-AGE=7200
< Vary: Accept-Encoding
< Vary: Accept-Encoding
< Content-Length: 833
< Connection: keep-alive
<
ATS response:
< HTTP/1.1 200 OK
< Access-Control-Allow-Origin: *
< Access-Control-Expose-Headers: Date
< Content-Encoding: gzip
< Content-Type: application/vnd.apple.mpegurl
< Content-Length: 180
< Cache-Control: max-age=1
< Cache-Control: public
< Cache-Control: no-cache="set-cookie"
< Date: Wed, 20 Feb 2019 07:07:14 GMT
< Etag: "5c6cfc9e-1b5"
< Expires: Wed, 20 Feb 2019 07:07:15 GMT
< Last-Modified: Wed, 20 Feb 2019 07:07:10 GMT
< Vary: Accept-Encoding
< Age: 0
< Connection: keep-alive
< Via: https/1.1 vds-cache (ApacheTrafficServer/7.1.6 [uScSsNfUpSeN:t cCSi p
sS])
< Server: ATS/7.1.6
<
* Connection #0 to host .... left intact
��A�0�= �`��i���h��Ĵh
�M�������/hf_MyV,��6���-�C0�Ǔ�-�t�( Ȉ�AU�J'p�8��*%zO| �>
--
Sent from: http://apache-traffic-server.24303.n7.nabble.com/
Re: [EXTERNAL] Re: corrupt data with multiple concurrent client
Posted by Erez Koler <Er...@harmonicinc.com>.
No. I don't think so but i will double check.
Get Outlook for Android<https://aka.ms/ghei36>
From: Miles Libbey
Sent: Tuesday, 19 February, 23:03
Subject: Re: [EXTERNAL] Re: corrupt data with multiple concurrent client
To: users@trafficserver.apache.org
Is the origin sending the data as Transfer-Encoding: Chunked? On Tue, Feb 19, 2019 at 12:53 PM Jeremy Payne wrote: > > ok.. i'm currently looking at some production 'document truncated' > errors.. which seem to only occur during these thundering herd > events.. > we're not using collapsed forwarding, but we do have rww set to 1. > > On Tue, Feb 19, 2019 at 2:44 PM Erez Koler wrote: > > > > No , diags looks normal > > > > -----Original Message----- > > From: Jeremy Payne [mailto:jp557198@gmail.com] > > Sent: Tuesday, February 19, 2019 10:36 PM > > To: users@trafficserver.apache.org > > Subject: [EXTERNAL] Re: corrupt data with multiple concurrent client > > > > question... are you seeing 'document truncated' in the diag log? > > > > On Tue, Feb 19, 2019 at 1:45 PM ezko wrote: > > > > > > hi, > > > with ATS 7.1.6 reverse proxy for live HLS , we are seeing corrupted data on > > > the playlist when 300 concurrent clients are playing the same content (same > > > profile). we tried several client simulators like locust and jmeter HLS > > > plugins with same results. > > > we configured collapsed forward plugin and RWW as described in the docs. > > > collapsed_forwarding > > > > > > reducing-origin-server-requests-avoiding-the-thundering-herd > > > > > > any ideas how to debug ? > > > Thanks. > > > erez > > > > > > > > > > > > > > > > > > -- > > > Sent from: http://apache-traffic-server.24303.n7.nabble.com/
Re: [EXTERNAL] Re: corrupt data with multiple concurrent client
Posted by Miles Libbey <ml...@apache.org>.
Is the origin sending the data as Transfer-Encoding: Chunked?
On Tue, Feb 19, 2019 at 12:53 PM Jeremy Payne <jp...@gmail.com> wrote:
>
> ok.. i'm currently looking at some production 'document truncated'
> errors.. which seem to only occur during these thundering herd
> events..
> we're not using collapsed forwarding, but we do have rww set to 1.
>
> On Tue, Feb 19, 2019 at 2:44 PM Erez Koler <Er...@harmonicinc.com> wrote:
> >
> > No , diags looks normal
> >
> > -----Original Message-----
> > From: Jeremy Payne [mailto:jp557198@gmail.com]
> > Sent: Tuesday, February 19, 2019 10:36 PM
> > To: users@trafficserver.apache.org
> > Subject: [EXTERNAL] Re: corrupt data with multiple concurrent client
> >
> > question... are you seeing 'document truncated' in the diag log?
> >
> > On Tue, Feb 19, 2019 at 1:45 PM ezko <er...@harmonicinc.com> wrote:
> > >
> > > hi,
> > > with ATS 7.1.6 reverse proxy for live HLS , we are seeing corrupted data on
> > > the playlist when 300 concurrent clients are playing the same content (same
> > > profile). we tried several client simulators like locust and jmeter HLS
> > > plugins with same results.
> > > we configured collapsed forward plugin and RWW as described in the docs.
> > > collapsed_forwarding
> > > <https://docs.trafficserver.apache.org/en/7.1.x/admin-guide/plugins/collapsed_forwarding.en.html?highlight=collapsed%20forwarding%20plugin#collapsed-forwarding-plugin>
> > > reducing-origin-server-requests-avoiding-the-thundering-herd
> > > <https://docs.trafficserver.apache.org/en/7.1.x/admin-guide/configuration/cache-basics.en.html?highlight=thundering#reducing-origin-server-requests-avoiding-the-thundering-herd>
> > > any ideas how to debug ?
> > > Thanks.
> > > erez
> > >
> > >
> > >
> > >
> > >
> > > --
> > > Sent from: http://apache-traffic-server.24303.n7.nabble.com/
Re: [EXTERNAL] Re: corrupt data with multiple concurrent client
Posted by Jeremy Payne <jp...@gmail.com>.
ok.. i'm currently looking at some production 'document truncated'
errors.. which seem to only occur during these thundering herd
events..
we're not using collapsed forwarding, but we do have rww set to 1.
On Tue, Feb 19, 2019 at 2:44 PM Erez Koler <Er...@harmonicinc.com> wrote:
>
> No , diags looks normal
>
> -----Original Message-----
> From: Jeremy Payne [mailto:jp557198@gmail.com]
> Sent: Tuesday, February 19, 2019 10:36 PM
> To: users@trafficserver.apache.org
> Subject: [EXTERNAL] Re: corrupt data with multiple concurrent client
>
> question... are you seeing 'document truncated' in the diag log?
>
> On Tue, Feb 19, 2019 at 1:45 PM ezko <er...@harmonicinc.com> wrote:
> >
> > hi,
> > with ATS 7.1.6 reverse proxy for live HLS , we are seeing corrupted data on
> > the playlist when 300 concurrent clients are playing the same content (same
> > profile). we tried several client simulators like locust and jmeter HLS
> > plugins with same results.
> > we configured collapsed forward plugin and RWW as described in the docs.
> > collapsed_forwarding
> > <https://docs.trafficserver.apache.org/en/7.1.x/admin-guide/plugins/collapsed_forwarding.en.html?highlight=collapsed%20forwarding%20plugin#collapsed-forwarding-plugin>
> > reducing-origin-server-requests-avoiding-the-thundering-herd
> > <https://docs.trafficserver.apache.org/en/7.1.x/admin-guide/configuration/cache-basics.en.html?highlight=thundering#reducing-origin-server-requests-avoiding-the-thundering-herd>
> > any ideas how to debug ?
> > Thanks.
> > erez
> >
> >
> >
> >
> >
> > --
> > Sent from: http://apache-traffic-server.24303.n7.nabble.com/