You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@trafficserver.apache.org by GitBox <gi...@apache.org> on 2022/02/08 20:06:46 UTC
[GitHub] [trafficserver] cmcfarlen opened a new pull request #8657: Support transforming range requests when origin returns full resource.
cmcfarlen opened a new pull request #8657:
URL: https://github.com/apache/trafficserver/pull/8657
Resolves #8161:
This PR adds support for handling range requests after a response is received and the response source is the origin server.
- Allow caching responses when a range is requested (requires proxy.config.http.cache.range.write is set, but slightly changes behavior)
- Handle range setup even if the response source is the origin server.
- Call do_range_setup_if_necessary on forward server response.
- Enable API flag to cache original response when a partial response is tranformed (api_info.cache_untransformed = true)
This doesn't currently support transforming chunked responses from the origin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] masaori335 commented on pull request #8657: Support transforming range requests when origin returns full resource.
Posted by GitBox <gi...@apache.org>.
masaori335 commented on pull request #8657:
URL: https://github.com/apache/trafficserver/pull/8657#issuecomment-1034536516
[approve ci centos]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] vmamidi commented on pull request #8657: Support transforming range requests when origin returns full resource.
Posted by GitBox <gi...@apache.org>.
vmamidi commented on pull request #8657:
URL: https://github.com/apache/trafficserver/pull/8657#issuecomment-1034354217
[approve ci]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] bryancall commented on pull request #8657: Support transforming range requests when origin returns full resource.
Posted by GitBox <gi...@apache.org>.
bryancall commented on pull request #8657:
URL: https://github.com/apache/trafficserver/pull/8657#issuecomment-1039731415
@traeak Can you please review this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] bryancall commented on pull request #8657: Support transforming range requests when origin returns full resource.
Posted by GitBox <gi...@apache.org>.
bryancall commented on pull request #8657:
URL: https://github.com/apache/trafficserver/pull/8657#issuecomment-1039731415
@traeak Can you please review this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] bneradt removed a comment on pull request #8657: Support transforming range requests when origin returns full resource.
Posted by GitBox <gi...@apache.org>.
bneradt removed a comment on pull request #8657:
URL: https://github.com/apache/trafficserver/pull/8657#issuecomment-1050387987
[approve ci fedora]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] vmamidi merged pull request #8657: Support transforming range requests when origin returns full resource.
Posted by GitBox <gi...@apache.org>.
vmamidi merged pull request #8657:
URL: https://github.com/apache/trafficserver/pull/8657
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] cmcfarlen commented on pull request #8657: Support transforming range requests when origin returns full resource.
Posted by GitBox <gi...@apache.org>.
cmcfarlen commented on pull request #8657:
URL: https://github.com/apache/trafficserver/pull/8657#issuecomment-1040747292
I added a check for a Content-Encoding and skip out on setting up a server transform to a 206.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] bneradt commented on a change in pull request #8657: Support transforming range requests when origin returns full resource.
Posted by GitBox <gi...@apache.org>.
bneradt commented on a change in pull request #8657:
URL: https://github.com/apache/trafficserver/pull/8657#discussion_r806357934
##########
File path: tests/gold_tests/cache/cache-range-response.test.py
##########
@@ -0,0 +1,38 @@
+'''
+Verify that caching a range request when origin returns full response works.
+'''
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+Test.Summary = '''
+Verify correct caching behavior for range requests.
+'''
+
+ts = Test.MakeATSProcess("ts")
+replay_file = "replay/cache-range-response.replay.yaml"
+server = Test.MakeVerifierServerProcess("server0", replay_file)
+ts.Disk.records_config.update({
+ 'proxy.config.diags.debug.enabled': 1,
+ 'proxy.config.diags.debug.tags': 'http.*|cache.*',
+ 'proxy.config.http.cache.range.write': 1,
+})
+ts.Disk.remap_config.AddLine(
+ 'map / http://127.0.0.1:{0}'.format(server.Variables.http_port)
Review comment:
Small thing: we're generally using Python f stings now. Thus:
```
f'map / http://127.0.0.1:{server.Variables.http_port}'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] masaori335 commented on pull request #8657: Support transforming range requests when origin returns full resource.
Posted by GitBox <gi...@apache.org>.
masaori335 commented on pull request #8657:
URL: https://github.com/apache/trafficserver/pull/8657#issuecomment-1033199504
@ezelkow1 Would you add Chris (@cmcfarlen) to the ci list?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] bneradt commented on a change in pull request #8657: Support transforming range requests when origin returns full resource.
Posted by GitBox <gi...@apache.org>.
bneradt commented on a change in pull request #8657:
URL: https://github.com/apache/trafficserver/pull/8657#discussion_r806357934
##########
File path: tests/gold_tests/cache/cache-range-response.test.py
##########
@@ -0,0 +1,38 @@
+'''
+Verify that caching a range request when origin returns full response works.
+'''
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+Test.Summary = '''
+Verify correct caching behavior for range requests.
+'''
+
+ts = Test.MakeATSProcess("ts")
+replay_file = "replay/cache-range-response.replay.yaml"
+server = Test.MakeVerifierServerProcess("server0", replay_file)
+ts.Disk.records_config.update({
+ 'proxy.config.diags.debug.enabled': 1,
+ 'proxy.config.diags.debug.tags': 'http.*|cache.*',
+ 'proxy.config.http.cache.range.write': 1,
+})
+ts.Disk.remap_config.AddLine(
+ 'map / http://127.0.0.1:{0}'.format(server.Variables.http_port)
Review comment:
Small thing: we're generally using Python f stings now. Thus:
```
f'map / http://127.0.0.1:{server.Variables.http_port}'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] ezelkow1 commented on pull request #8657: Support transforming range requests when origin returns full resource.
Posted by GitBox <gi...@apache.org>.
ezelkow1 commented on pull request #8657:
URL: https://github.com/apache/trafficserver/pull/8657#issuecomment-1034189824
[approve ci]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] cmcfarlen commented on pull request #8657: Support transforming range requests when origin returns full resource.
Posted by GitBox <gi...@apache.org>.
cmcfarlen commented on pull request #8657:
URL: https://github.com/apache/trafficserver/pull/8657#issuecomment-1047077190
I will check the case when a large asset is requested with just the first part specified in Range and see what happens. Will report back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] cmcfarlen commented on pull request #8657: Support transforming range requests when origin returns full resource.
Posted by GitBox <gi...@apache.org>.
cmcfarlen commented on pull request #8657:
URL: https://github.com/apache/trafficserver/pull/8657#issuecomment-1047244507
I setup a scenario to try to answer the question about short ranges and caching the full asset. TL;DR: It caches the full response
### Scenario
- https as the origin to serve a ~50MB file configured to ignore ranges to always return a 200 and the full request
- ATS mapping to origin with this new feature turned on to transform 200 to 206 and cache the full response (staring with an empty cache)
- curl client requesting to close the connection and requesting a range.
Here is the simple request script:
```
#!/bin/sh
curl -v -o first.bin --header "Connection: close" --header 'Range: bytes=0-10' http://ats:8080/docker.iso &
sleep 0.1 # added or both requests would be cache misses
curl -v -o second.bin --header "Connection: close" --header 'Range: bytes=10000000-' http://ats:8080/docker.iso &
```
### Expected behavior
The first request will cause ats start the transfer from origin, transforming the response and sending 11 bytes to the client and caching the full response.
The second request will do a RWW and serve the request from the cache as its written
### Results
(I removed the output related to the transfer progress, leaving just the verbose output)
```
* Connected to ats (127.0.0.1) port 8080 (#0)
> GET /docker.iso HTTP/1.1
> Host: ats:8080
> User-Agent: curl/7.77.0
> Accept: */*
> Connection: close
> Range: bytes=0-10
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 206 Partial Content
< Date: Mon, 21 Feb 2022 20:31:36 GMT
< Server: ATS/10.0.0
< Last-Modified: Mon, 21 Feb 2022 17:59:12 GMT
< ETag: "3000000-5d88afd724296"
< Accept-Ranges: none
< Cache-Control: public, max-age=60, must-revalidate
< Content-Type: application/x-iso9660-image
< Content-Range: bytes 0-10/50331648
< Content-Length: 11
< Age: 0
< Connection: close
< Via: http/1.1 traffic_server (ApacheTrafficServer/10.0.0 [uScMsSfWpSeN:t cCMp sS])
<
* Closing connection 0
* Connected to ats (127.0.0.1) port 8080 (#0)
> GET /docker.iso HTTP/1.1
> Host: ats:8080
> User-Agent: curl/7.77.0
> Accept: */*
> Connection: close
> Range: bytes=10000000-
>
$ * Mark bundle as not supporting multiuse
< HTTP/1.1 206 Partial Content
< Date: Mon, 21 Feb 2022 20:31:36 GMT
< Server: ATS/10.0.0
< Last-Modified: Mon, 21 Feb 2022 17:59:12 GMT
< ETag: "3000000-5d88afd724296"
< Accept-Ranges: none
< Cache-Control: public, max-age=60, must-revalidate
< Content-Type: application/x-iso9660-image
< Content-Range: bytes 10000000-50331647/50331648
< Content-Length: 40331648
< Age: 0
< Connection: close
< Via: http/1.1 traffic_server (ApacheTrafficServer/10.0.0 [uScWs f p eN:t cCHp s ])
<
* Closing connection 0
```
A subsequent request for the full content will serve the full resource from cache:
```
< Content-Length: 50331648
< Cache-Control: public, max-age=60, must-revalidate
< Content-Type: application/x-iso9660-image
< Age: 9
< Connection: close
< Via: http/1.1 traffic_server (ApacheTrafficServer/10.0.0 [uScHs f p eN:t cCHp s ])
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] bneradt commented on pull request #8657: Support transforming range requests when origin returns full resource.
Posted by GitBox <gi...@apache.org>.
bneradt commented on pull request #8657:
URL: https://github.com/apache/trafficserver/pull/8657#issuecomment-1050387987
[approve ci fedora]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] traeak commented on pull request #8657: Support transforming range requests when origin returns full resource.
Posted by GitBox <gi...@apache.org>.
traeak commented on pull request #8657:
URL: https://github.com/apache/trafficserver/pull/8657#issuecomment-1060715662
Broke this while testing with an IMS header.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] ezelkow1 commented on pull request #8657: Support transforming range requests when origin returns full resource.
Posted by GitBox <gi...@apache.org>.
ezelkow1 commented on pull request #8657:
URL: https://github.com/apache/trafficserver/pull/8657#issuecomment-1045221812
[approve ci rocky]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] ezelkow1 edited a comment on pull request #8657: Support transforming range requests when origin returns full resource.
Posted by GitBox <gi...@apache.org>.
ezelkow1 edited a comment on pull request #8657:
URL: https://github.com/apache/trafficserver/pull/8657#issuecomment-1034189824
[approve ci]
Done, he should be in the list now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] traeak commented on pull request #8657: Support transforming range requests when origin returns full resource.
Posted by GitBox <gi...@apache.org>.
traeak commented on pull request #8657:
URL: https://github.com/apache/trafficserver/pull/8657#issuecomment-1072623207
[approve ci fedora]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] zwoop commented on pull request #8657: Support transforming range requests when origin returns full resource.
Posted by GitBox <gi...@apache.org>.
zwoop commented on pull request #8657:
URL: https://github.com/apache/trafficserver/pull/8657#issuecomment-1082007196
Cherry-picked to v9.2.x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] cmcfarlen edited a comment on pull request #8657: Support transforming range requests when origin returns full resource.
Posted by GitBox <gi...@apache.org>.
cmcfarlen edited a comment on pull request #8657:
URL: https://github.com/apache/trafficserver/pull/8657#issuecomment-1040747292
I added a check for a Content-Encoding in the server response and skip out on setting up a server transform to a 206.
I also investigated the compress plugin and it will skip compression for requests that include a Range header unless a setting overrides this. It may be that that configuration is not compatible with configuring the feature added by this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org