You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by Mateusz Zajakala <za...@gmail.com> on 2018/02/22 11:59:17 UTC
very long cache_open_read_end - timeout
Hi,
I have a problem - from time to time my ATS 7.2.1 has problems with serving
static files from origin, client transactions last over 10s and I see the
following diagnostic (I have enabled logging "slow requests")
cache_open_read_begin: 0.000 cache_open_read_end: 10.640
Any requests to the same URL are blocked for over 10s. Restarting ATS
solves the problem.
What does it mean? Does it have to do with open_read feature? but the
timeouts don't add...
Any suggestions appreciated!
Here's my records.config with configs related to avoiding thundering herd...
CONFIG proxy.config.cache.enable_read_while_writer INT 1
CONFIG proxy.config.http.background_fill_active_timeout INT 0
CONFIG proxy.config.http.background_fill_completed_threshold FLOAT 0.000000
CONFIG proxy.config.cache.read_while_writer.max_retries INT 5
CONFIG proxy.config.cache.read_while_writer_retry.delay INT 200
CONFIG proxy.config.http.cache.max_open_read_retries INT 5
CONFIG proxy.config.http.cache.open_read_retry_time INT 200
CONFIG proxy.config.http.cache.open_write_fail_action INT 2
Re: very long cache_open_read_end - timeout
Posted by Bryan Call <bc...@apache.org>.
That is what I set ATS to do in production for us. Funny that we both came up with 4. It was mainly do to lock contention issues and people benchmarking or running ATS at high request rates.
$ tail -4 volume.config
volume=1 scheme=http size=25%
volume=2 scheme=http size=25%
volume=3 scheme=http size=25%
volume=4 scheme=http size=25%
-Bryan
> On Feb 26, 2018, at 12:15 AM, Veiko Kukk <ve...@gmail.com> wrote:
>
> Hi Bryan,
>
> We decided to create 4 volumes. No strong reasoning behind it, just 2 seemed too little to avoid those blocking timeouts and 4 was next comfortable to set using 25%.
>
> Veiko
>
>
> 2018-02-23 21:21 GMT+02:00 Bryan Call <bcall@apache.org <ma...@apache.org>>:
> Veiko,
>
> How many volumes did you create?
>
> -Bryan
>
>
>> On Feb 23, 2018, at 3:14 AM, Veiko Kukk <veiko.kukk@gmail.com <ma...@gmail.com>> wrote:
>>
>> Hi, Mateusz
>>
>> I wonder if you might experience same issue as me.
>> https://github.com/apache/trafficserver/issues/3057 <https://github.com/apache/trafficserver/issues/3057>
>>
>> For me, solution, or I would rather call it workaround was to create volumes.
>>
>> Unfortunately, documentation regarding to storage performance issues/tuning is missing.
>>
>> Best regards,
>> Veiko
>>
>>
>> 2018-02-22 13:59 GMT+02:00 Mateusz Zajakala <zajakala@gmail.com <ma...@gmail.com>>:
>> Hi,
>>
>> I have a problem - from time to time my ATS 7.2.1 has problems with serving static files from origin, client transactions last over 10s and I see the following diagnostic (I have enabled logging "slow requests")
>>
>> cache_open_read_begin: 0.000 cache_open_read_end: 10.640
>>
>> Any requests to the same URL are blocked for over 10s. Restarting ATS solves the problem.
>>
>> What does it mean? Does it have to do with open_read feature? but the timeouts don't add...
>>
>> Any suggestions appreciated!
>>
>> Here's my records.config with configs related to avoiding thundering herd...
>>
>> CONFIG proxy.config.cache.enable_read_while_writer INT 1
>> CONFIG proxy.config.http.background_fill_active_timeout INT 0
>> CONFIG proxy.config.http.background_fill_completed_threshold FLOAT 0.000000
>> CONFIG proxy.config.cache.read_while_writer.max_retries INT 5
>> CONFIG proxy.config.cache.read_while_writer_retry.delay INT 200
>> CONFIG proxy.config.http.cache.max_open_read_retries INT 5
>> CONFIG proxy.config.http.cache.open_read_retry_time INT 200
>>
>> CONFIG proxy.config.http.cache.open_write_fail_action INT 2
>>
>>
>
>
Re: very long cache_open_read_end - timeout
Posted by Veiko Kukk <ve...@gmail.com>.
Hi Bryan,
We decided to create 4 volumes. No strong reasoning behind it, just 2
seemed too little to avoid those blocking timeouts and 4 was next
comfortable to set using 25%.
Veiko
2018-02-23 21:21 GMT+02:00 Bryan Call <bc...@apache.org>:
> Veiko,
>
> How many volumes did you create?
>
> -Bryan
>
>
> On Feb 23, 2018, at 3:14 AM, Veiko Kukk <ve...@gmail.com> wrote:
>
> Hi, Mateusz
>
> I wonder if you might experience same issue as me.
> https://github.com/apache/trafficserver/issues/3057
>
> For me, solution, or I would rather call it workaround was to create
> volumes.
>
> Unfortunately, documentation regarding to storage performance
> issues/tuning is missing.
>
> Best regards,
> Veiko
>
>
> 2018-02-22 13:59 GMT+02:00 Mateusz Zajakala <za...@gmail.com>:
>
>> Hi,
>>
>> I have a problem - from time to time my ATS 7.2.1 has problems with
>> serving static files from origin, client transactions last over 10s and I
>> see the following diagnostic (I have enabled logging "slow requests")
>>
>> cache_open_read_begin: 0.000 cache_open_read_end: 10.640
>>
>> Any requests to the same URL are blocked for over 10s. Restarting ATS
>> solves the problem.
>>
>> What does it mean? Does it have to do with open_read feature? but the
>> timeouts don't add...
>>
>> Any suggestions appreciated!
>>
>> Here's my records.config with configs related to avoiding thundering
>> herd...
>>
>> CONFIG proxy.config.cache.enable_read_while_writer INT 1
>> CONFIG proxy.config.http.background_fill_active_timeout INT 0
>> CONFIG proxy.config.http.background_fill_completed_threshold FLOAT
>> 0.000000
>> CONFIG proxy.config.cache.read_while_writer.max_retries INT 5
>> CONFIG proxy.config.cache.read_while_writer_retry.delay INT 200
>> CONFIG proxy.config.http.cache.max_open_read_retries INT 5
>> CONFIG proxy.config.http.cache.open_read_retry_time INT 200
>>
>> CONFIG proxy.config.http.cache.open_write_fail_action INT 2
>>
>>
>
>
Re: very long cache_open_read_end - timeout
Posted by Bryan Call <bc...@apache.org>.
Veiko,
How many volumes did you create?
-Bryan
> On Feb 23, 2018, at 3:14 AM, Veiko Kukk <ve...@gmail.com> wrote:
>
> Hi, Mateusz
>
> I wonder if you might experience same issue as me.
> https://github.com/apache/trafficserver/issues/3057 <https://github.com/apache/trafficserver/issues/3057>
>
> For me, solution, or I would rather call it workaround was to create volumes.
>
> Unfortunately, documentation regarding to storage performance issues/tuning is missing.
>
> Best regards,
> Veiko
>
>
> 2018-02-22 13:59 GMT+02:00 Mateusz Zajakala <zajakala@gmail.com <ma...@gmail.com>>:
> Hi,
>
> I have a problem - from time to time my ATS 7.2.1 has problems with serving static files from origin, client transactions last over 10s and I see the following diagnostic (I have enabled logging "slow requests")
>
> cache_open_read_begin: 0.000 cache_open_read_end: 10.640
>
> Any requests to the same URL are blocked for over 10s. Restarting ATS solves the problem.
>
> What does it mean? Does it have to do with open_read feature? but the timeouts don't add...
>
> Any suggestions appreciated!
>
> Here's my records.config with configs related to avoiding thundering herd...
>
> CONFIG proxy.config.cache.enable_read_while_writer INT 1
> CONFIG proxy.config.http.background_fill_active_timeout INT 0
> CONFIG proxy.config.http.background_fill_completed_threshold FLOAT 0.000000
> CONFIG proxy.config.cache.read_while_writer.max_retries INT 5
> CONFIG proxy.config.cache.read_while_writer_retry.delay INT 200
> CONFIG proxy.config.http.cache.max_open_read_retries INT 5
> CONFIG proxy.config.http.cache.open_read_retry_time INT 200
>
> CONFIG proxy.config.http.cache.open_write_fail_action INT 2
>
>
Re: very long cache_open_read_end - timeout
Posted by Veiko Kukk <ve...@gmail.com>.
Hi, Mateusz
I wonder if you might experience same issue as me.
https://github.com/apache/trafficserver/issues/3057
For me, solution, or I would rather call it workaround was to create
volumes.
Unfortunately, documentation regarding to storage performance issues/tuning
is missing.
Best regards,
Veiko
2018-02-22 13:59 GMT+02:00 Mateusz Zajakala <za...@gmail.com>:
> Hi,
>
> I have a problem - from time to time my ATS 7.2.1 has problems with
> serving static files from origin, client transactions last over 10s and I
> see the following diagnostic (I have enabled logging "slow requests")
>
> cache_open_read_begin: 0.000 cache_open_read_end: 10.640
>
> Any requests to the same URL are blocked for over 10s. Restarting ATS
> solves the problem.
>
> What does it mean? Does it have to do with open_read feature? but the
> timeouts don't add...
>
> Any suggestions appreciated!
>
> Here's my records.config with configs related to avoiding thundering
> herd...
>
> CONFIG proxy.config.cache.enable_read_while_writer INT 1
> CONFIG proxy.config.http.background_fill_active_timeout INT 0
> CONFIG proxy.config.http.background_fill_completed_threshold FLOAT
> 0.000000
> CONFIG proxy.config.cache.read_while_writer.max_retries INT 5
> CONFIG proxy.config.cache.read_while_writer_retry.delay INT 200
> CONFIG proxy.config.http.cache.max_open_read_retries INT 5
> CONFIG proxy.config.http.cache.open_read_retry_time INT 200
>
> CONFIG proxy.config.http.cache.open_write_fail_action INT 2
>
>