You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by Pavel Vazharov <pa...@x3me.net> on 2024/04/17 13:40:02 UTC
Using Apache Traffic Server as HTTP client to store some content in the storage
Hi there,
Is there a way to use the ATS as an HTTP client to download and store given
content?
I'm aware that I can achieve this with a local HTTP client (wget, curl)
which uses the ATS as a forward proxy and then the ATS will store the
content (which is the actual goal).
I was wondering if there is a way without using an additional HTTP client
so that I can skip the additional content moving between sockets.
Thanks,
Pavel.
Re: Using Apache Traffic Server as HTTP client to store some content in the storage
Posted by Leif Hedstrom <ap...@gmail.com>.
> On Apr 19, 2024, at 21:42, dave seddon <da...@gmail.com> wrote:
>
>
>
>
> Thanks Brian! Very cool
>
>
>
>
> I don't know if you're supposed to retroactively update release notes, but
> that's a pretty big feature not in the notes.
>
>
>
>
>
> <https://docs.trafficserver.apache.org/en/10.0.x/release-
> notes/upgrading.en.html>
>
>
>
>
Yeh the release notes for 10.x are not done yet.
> Stale-while-revaldate feature was a major challenge for some of our use
> cases, so it's awesome to hear that is resolved. Congratulations and thank
> you!
>
>
>
>
> A potentially larger challenge is memory footprint. Generally, for Yahoo,
> Apple, Comcast CDNs, and all you guys all have beefy machines with plenty of
> RAM, so this isn't a drama. In those scenarios ATS is an amazing solution!
> Woot woot. Love it!
>
>
>
>
> There are more unique situations, home caching for example, where you want
> large amounts of cache storage, but much lower in memory indexes to that
> storage.
>
>
>
>
> I'm not suggesting ATS needs to solve is, but, for the sake of the mailing
> list history, people need to keep in mind.
>
>
>
>
> Specifically, and unless I'm mistaken, RAM consumption is directly linear
> with the available cache size, regardless of the "working set".
>
>
>
>
> E.g. If you have a large storage you need a (relatively) large amount of RAM
> to index into that storage, even if most of the storage is not accessed.
Well, yes. However it’s 10x less memory than e.g. squid for the same disk
cache size :). The cost is 10 bytes of memory per directory entry (index).
Now, you can, and should, tune the number of directory entries to your data
set. The default is pretty conservative, one dir entry for every 8KB of disk
cache. If your average object size is noticeably larger than this (very
likely), you can reduce memory usage accordingly. So setting this
configuration to 16KB would reduce memory use in half, 32KB is 1/4 memory etc.
Cheers,
— Leif
>
>
>
> Again, this is a totally understandable engineering decision, and makes
> sense in most situations.
>
>
>
>
> On Fri, Apr 19, 2024, 14:02 Brian Neradt
> <[brian.neradt@gmail.com](mailto:brian.neradt@gmail.com)> wrote:
>
>
>> > If I understand correctly, ATS does NOT support stale while revalidate,
which is surprising given the RFC was created by a Yahoo-ligan. ;)
>
>>
>>
>
>>
>> I'm glad you mentioned this. Actually Yahoo open-sourced the functionality
for stale while revalidate via the stale_response.so plugin for ATS 10:
>>
>>
>
>>
>> <https://docs.trafficserver.apache.org/en/latest/admin-
guide/plugins/stale_response.en.html#stale-response-plugin>
>
>>
>>
>
>>
>> On Fri, Apr 19, 2024 at 3:38 PM dave seddon
<[dave.seddon.ca@gmail.com](mailto:dave.seddon.ca@gmail.com)> wrote:
>
>>
>>> Depending on your use case, you also want to carefully consider the cache
control headers for the downloaded object.
>>>
>>>
>
>>>
>>> If I understand correctly, ATS does NOT support stale while revalidate,
which is surprising given the RFC was created by a Yahoo-ligan. ;)
>
>>>
>>>
>
>>>
>>> On Fri, Apr 19, 2024 at 9:51 AM Leif Hedstrom
<[zwoop@apache.org](mailto:zwoop@apache.org)> wrote:
>
>>>
>>>> We discussed this in the slack channel. Probably the best option for you
(other than writing a new plugin) is to use the background_fetch plugin. You
will still need to trigger a download, but you can avoid having to wait for
the full response in the client. So, what you’d do is
>>>>
>>>>
>
>>>>
>>>> curl -H “Range: bytes=0-1”
<https://www.example.com/some/url/foo.img>
>>>>
>>>>
>
>>>>
>>>>
>
>>>>
>>>> And make sure background_fetch is enabled on this remap rule. No matter
what you do, you will have to have something that tells ATS to kick off a
download, and the above is probably as good as any alternative.
>>>>
>>>>
>
>>>>
>>>> Cheers,
>>>>
>>>>
>
>>>>
>>>> — Leif
>>>>
>>>>
>
>>>>
>>>>
>
>>>>
>>>>> On Apr 17, 2024, at 9:33 AM, Pavel Vazharov
<[pavel@x3me.net](mailto:pavel@x3me.net)> wrote:
>>>>>
>>>>>
>
>>>>>
>>>>> Hi Brian,
>
>
>>>>>
>>>>> Thank you for your response.
> It'll do the job but, as far as I understand it, an external functionality
> will need to download the content in order to push it into the ATS.
>
>>>>>
>>>>> The content that I want to write in the ATS storage is on the Internet.
>
>
>>>>>
>>>>> Regards,
>
>>>>>
>>>>> Pavel.
>
>>>>>
>>>>>
>
>>>>>
>>>>> On Wed, Apr 17, 2024 at 6:26 PM Brian Neradt
<[brian.neradt@gmail.com](mailto:brian.neradt@gmail.com)> wrote:
>
>>>>>
>>>>>> Hi Pavel,
>>>>>>
>>>>>>
>
>>>>>>
>>>>>> This isn't a direct answer to your question, but are you aware of the
ATS HTTP PUSH feature? That allows you to push objects into the ATS cache
without the typical caching of proxied response. Can that help you in this
situation?
>>>>>>
>>>>>>
>
>>>>>>
>>>>>> <https://docs.trafficserver.apache.org/en/latest/admin-
guide/configuration/cache-basics.en.html#pushing-content-into-the-cache>
>
>>>>>>
>>>>>>
>
>>>>>>
>>>>>> On Wed, Apr 17, 2024 at 8:41 AM Pavel Vazharov
<[pavel@x3me.net](mailto:pavel@x3me.net)> wrote:
>
>>>>>>
>>>>>>> Hi there,
>
>
>>>>>>>
>>>>>>> Is there a way to use the ATS as an HTTP client to download and store
given content?
>
>>>>>>>
>>>>>>> I'm aware that I can achieve this with a local HTTP client (wget,
curl) which uses the ATS as a forward proxy and then the ATS will store the
content (which is the actual goal).
>
>>>>>>>
>>>>>>> I was wondering if there is a way without using an additional HTTP
client so that I can skip the additional content moving between sockets.
>
>
>>>>>>>
>>>>>>> Thanks,
>
>>>>>>>
>>>>>>> Pavel.
>
>>>>>>
>>>>>>
>
>>>>>>
>>>>>>
>
>>>>>>
>>>>>> \--
>
>>>>>>
>>>>>> "Come to Me, all who are weary and heavy-laden, and I will
>
>>>>>>
>>>>>> give you rest. Take My yoke upon you and learn from Me, for
> I am gentle and humble in heart, and you will find rest for
> your souls. For My yoke is easy and My burden is light."
>
> ~ Matthew 11:28-30
>
>>>>
>>>>
>
>>>
>>>
>
> \--
>
>>>
>>> Regards,
>
>>>
>>> Dave Seddon
> +1 415 857 5102
>
>>
>>
>
>>
>>
>
>>
>> \--
>
>>
>> "Come to Me, all who are weary and heavy-laden, and I will
>
>>
>> give you rest. Take My yoke upon you and learn from Me, for
> I am gentle and humble in heart, and you will find rest for
> your souls. For My yoke is easy and My burden is light."
>
> ~ Matthew 11:28-30
>
Re: Using Apache Traffic Server as HTTP client to store some content in the storage
Posted by dave seddon <da...@gmail.com>.
Thanks Brian! Very cool
I don't know if you're supposed to retroactively update release notes, but
that's a pretty big feature not in the notes.
https://docs.trafficserver.apache.org/en/10.0.x/release-notes/upgrading.en.html
Stale-while-revaldate feature was a major challenge for some of our use
cases, so it's awesome to hear that is resolved. Congratulations and thank
you!
A potentially larger challenge is memory footprint. Generally, for Yahoo,
Apple, Comcast CDNs, and all you guys all have beefy machines with plenty
of RAM, so this isn't a drama. In those scenarios ATS is an amazing
solution! Woot woot. Love it!
There are more unique situations, home caching for example, where you want
large amounts of cache storage, but much lower in memory indexes to that
storage.
I'm not suggesting ATS needs to solve is, but, for the sake of the mailing
list history, people need to keep in mind.
Specifically, and unless I'm mistaken, RAM consumption is directly linear
with the available cache size, regardless of the "working set".
E.g. If you have a large storage you need a (relatively) large amount of
RAM to index into that storage, even if most of the storage is not accessed.
Again, this is a totally understandable engineering decision, and makes
sense in most situations.
On Fri, Apr 19, 2024, 14:02 Brian Neradt <br...@gmail.com> wrote:
> If I understand correctly, ATS does NOT support stale while revalidate,
>> which is surprising given the RFC was created by a Yahoo-ligan. ;)
>>
>
> I'm glad you mentioned this. Actually Yahoo open-sourced the functionality
> for stale while revalidate via the stale_response.so plugin for ATS 10:
>
>
> https://docs.trafficserver.apache.org/en/latest/admin-guide/plugins/stale_response.en.html#stale-response-plugin
>
> On Fri, Apr 19, 2024 at 3:38 PM dave seddon <da...@gmail.com>
> wrote:
>
>> Depending on your use case, you also want to carefully consider the cache
>> control headers for the downloaded object.
>>
>> If I understand correctly, ATS does NOT support stale while revalidate,
>> which is surprising given the RFC was created by a Yahoo-ligan. ;)
>>
>> On Fri, Apr 19, 2024 at 9:51 AM Leif Hedstrom <zw...@apache.org> wrote:
>>
>>> We discussed this in the slack channel. Probably the best option for you
>>> (other than writing a new plugin) is to use the background_fetch plugin.
>>> You will still need to trigger a download, but you can avoid having to wait
>>> for the full response in the client. So, what you’d do is
>>>
>>> curl -H “Range: bytes=0-1” https://www.example.com/some/url/foo.img
>>>
>>>
>>> And make sure background_fetch is enabled on this remap rule. No matter
>>> what you do, you will have to have something that tells ATS to kick off a
>>> download, and the above is probably as good as any alternative.
>>>
>>> Cheers,
>>>
>>> — Leif
>>>
>>>
>>> On Apr 17, 2024, at 9:33 AM, Pavel Vazharov <pa...@x3me.net> wrote:
>>>
>>> Hi Brian,
>>>
>>> Thank you for your response.
>>> It'll do the job but, as far as I understand it, an external
>>> functionality will need to download the content in order to push it into
>>> the ATS.
>>> The content that I want to write in the ATS storage is on the Internet.
>>>
>>> Regards,
>>> Pavel.
>>>
>>> On Wed, Apr 17, 2024 at 6:26 PM Brian Neradt <br...@gmail.com>
>>> wrote:
>>>
>>>> Hi Pavel,
>>>>
>>>> This isn't a direct answer to your question, but are you aware of the
>>>> ATS HTTP PUSH feature? That allows you to push objects into the ATS cache
>>>> without the typical caching of proxied response. Can that help you in this
>>>> situation?
>>>>
>>>>
>>>> https://docs.trafficserver.apache.org/en/latest/admin-guide/configuration/cache-basics.en.html#pushing-content-into-the-cache
>>>>
>>>>
>>>> On Wed, Apr 17, 2024 at 8:41 AM Pavel Vazharov <pa...@x3me.net> wrote:
>>>>
>>>>> Hi there,
>>>>>
>>>>> Is there a way to use the ATS as an HTTP client to download and store
>>>>> given content?
>>>>> I'm aware that I can achieve this with a local HTTP client (wget,
>>>>> curl) which uses the ATS as a forward proxy and then the ATS will store the
>>>>> content (which is the actual goal).
>>>>> I was wondering if there is a way without using an additional HTTP
>>>>> client so that I can skip the additional content moving between sockets.
>>>>>
>>>>> Thanks,
>>>>> Pavel.
>>>>>
>>>>
>>>>
>>>> --
>>>> "Come to Me, all who are weary and heavy-laden, and I will
>>>> give you rest. Take My yoke upon you and learn from Me, for
>>>> I am gentle and humble in heart, and you will find rest for
>>>> your souls. For My yoke is easy and My burden is light."
>>>>
>>>> ~ Matthew 11:28-30
>>>>
>>>
>>>
>>
>> --
>> Regards,
>> Dave Seddon
>> +1 415 857 5102
>>
>
>
> --
> "Come to Me, all who are weary and heavy-laden, and I will
> give you rest. Take My yoke upon you and learn from Me, for
> I am gentle and humble in heart, and you will find rest for
> your souls. For My yoke is easy and My burden is light."
>
> ~ Matthew 11:28-30
>
Re: Using Apache Traffic Server as HTTP client to store some content in the storage
Posted by Pavel Vazharov <pa...@x3me.net>.
Thank you all for the responses.
On Sat, Apr 20, 2024 at 12:02 AM Brian Neradt <br...@gmail.com>
wrote:
> If I understand correctly, ATS does NOT support stale while revalidate,
>> which is surprising given the RFC was created by a Yahoo-ligan. ;)
>>
>
> I'm glad you mentioned this. Actually Yahoo open-sourced the functionality
> for stale while revalidate via the stale_response.so plugin for ATS 10:
>
>
> https://docs.trafficserver.apache.org/en/latest/admin-guide/plugins/stale_response.en.html#stale-response-plugin
>
> On Fri, Apr 19, 2024 at 3:38 PM dave seddon <da...@gmail.com>
> wrote:
>
>> Depending on your use case, you also want to carefully consider the cache
>> control headers for the downloaded object.
>>
>> If I understand correctly, ATS does NOT support stale while revalidate,
>> which is surprising given the RFC was created by a Yahoo-ligan. ;)
>>
>> On Fri, Apr 19, 2024 at 9:51 AM Leif Hedstrom <zw...@apache.org> wrote:
>>
>>> We discussed this in the slack channel. Probably the best option for you
>>> (other than writing a new plugin) is to use the background_fetch plugin.
>>> You will still need to trigger a download, but you can avoid having to wait
>>> for the full response in the client. So, what you’d do is
>>>
>>> curl -H “Range: bytes=0-1” https://www.example.com/some/url/foo.img
>>>
>>>
>>> And make sure background_fetch is enabled on this remap rule. No matter
>>> what you do, you will have to have something that tells ATS to kick off a
>>> download, and the above is probably as good as any alternative.
>>>
>>> Cheers,
>>>
>>> — Leif
>>>
>>>
>>> On Apr 17, 2024, at 9:33 AM, Pavel Vazharov <pa...@x3me.net> wrote:
>>>
>>> Hi Brian,
>>>
>>> Thank you for your response.
>>> It'll do the job but, as far as I understand it, an external
>>> functionality will need to download the content in order to push it into
>>> the ATS.
>>> The content that I want to write in the ATS storage is on the Internet.
>>>
>>> Regards,
>>> Pavel.
>>>
>>> On Wed, Apr 17, 2024 at 6:26 PM Brian Neradt <br...@gmail.com>
>>> wrote:
>>>
>>>> Hi Pavel,
>>>>
>>>> This isn't a direct answer to your question, but are you aware of the
>>>> ATS HTTP PUSH feature? That allows you to push objects into the ATS cache
>>>> without the typical caching of proxied response. Can that help you in this
>>>> situation?
>>>>
>>>>
>>>> https://docs.trafficserver.apache.org/en/latest/admin-guide/configuration/cache-basics.en.html#pushing-content-into-the-cache
>>>>
>>>>
>>>> On Wed, Apr 17, 2024 at 8:41 AM Pavel Vazharov <pa...@x3me.net> wrote:
>>>>
>>>>> Hi there,
>>>>>
>>>>> Is there a way to use the ATS as an HTTP client to download and store
>>>>> given content?
>>>>> I'm aware that I can achieve this with a local HTTP client (wget,
>>>>> curl) which uses the ATS as a forward proxy and then the ATS will store the
>>>>> content (which is the actual goal).
>>>>> I was wondering if there is a way without using an additional HTTP
>>>>> client so that I can skip the additional content moving between sockets.
>>>>>
>>>>> Thanks,
>>>>> Pavel.
>>>>>
>>>>
>>>>
>>>> --
>>>> "Come to Me, all who are weary and heavy-laden, and I will
>>>> give you rest. Take My yoke upon you and learn from Me, for
>>>> I am gentle and humble in heart, and you will find rest for
>>>> your souls. For My yoke is easy and My burden is light."
>>>>
>>>> ~ Matthew 11:28-30
>>>>
>>>
>>>
>>
>> --
>> Regards,
>> Dave Seddon
>> +1 415 857 5102
>>
>
>
> --
> "Come to Me, all who are weary and heavy-laden, and I will
> give you rest. Take My yoke upon you and learn from Me, for
> I am gentle and humble in heart, and you will find rest for
> your souls. For My yoke is easy and My burden is light."
>
> ~ Matthew 11:28-30
>
Re: Using Apache Traffic Server as HTTP client to store some content in the storage
Posted by Brian Neradt <br...@gmail.com>.
>
> If I understand correctly, ATS does NOT support stale while revalidate,
> which is surprising given the RFC was created by a Yahoo-ligan. ;)
>
I'm glad you mentioned this. Actually Yahoo open-sourced the functionality
for stale while revalidate via the stale_response.so plugin for ATS 10:
https://docs.trafficserver.apache.org/en/latest/admin-guide/plugins/stale_response.en.html#stale-response-plugin
On Fri, Apr 19, 2024 at 3:38 PM dave seddon <da...@gmail.com>
wrote:
> Depending on your use case, you also want to carefully consider the cache
> control headers for the downloaded object.
>
> If I understand correctly, ATS does NOT support stale while revalidate,
> which is surprising given the RFC was created by a Yahoo-ligan. ;)
>
> On Fri, Apr 19, 2024 at 9:51 AM Leif Hedstrom <zw...@apache.org> wrote:
>
>> We discussed this in the slack channel. Probably the best option for you
>> (other than writing a new plugin) is to use the background_fetch plugin.
>> You will still need to trigger a download, but you can avoid having to wait
>> for the full response in the client. So, what you’d do is
>>
>> curl -H “Range: bytes=0-1” https://www.example.com/some/url/foo.img
>>
>>
>> And make sure background_fetch is enabled on this remap rule. No matter
>> what you do, you will have to have something that tells ATS to kick off a
>> download, and the above is probably as good as any alternative.
>>
>> Cheers,
>>
>> — Leif
>>
>>
>> On Apr 17, 2024, at 9:33 AM, Pavel Vazharov <pa...@x3me.net> wrote:
>>
>> Hi Brian,
>>
>> Thank you for your response.
>> It'll do the job but, as far as I understand it, an external
>> functionality will need to download the content in order to push it into
>> the ATS.
>> The content that I want to write in the ATS storage is on the Internet.
>>
>> Regards,
>> Pavel.
>>
>> On Wed, Apr 17, 2024 at 6:26 PM Brian Neradt <br...@gmail.com>
>> wrote:
>>
>>> Hi Pavel,
>>>
>>> This isn't a direct answer to your question, but are you aware of the
>>> ATS HTTP PUSH feature? That allows you to push objects into the ATS cache
>>> without the typical caching of proxied response. Can that help you in this
>>> situation?
>>>
>>>
>>> https://docs.trafficserver.apache.org/en/latest/admin-guide/configuration/cache-basics.en.html#pushing-content-into-the-cache
>>>
>>>
>>> On Wed, Apr 17, 2024 at 8:41 AM Pavel Vazharov <pa...@x3me.net> wrote:
>>>
>>>> Hi there,
>>>>
>>>> Is there a way to use the ATS as an HTTP client to download and store
>>>> given content?
>>>> I'm aware that I can achieve this with a local HTTP client (wget, curl)
>>>> which uses the ATS as a forward proxy and then the ATS will store the
>>>> content (which is the actual goal).
>>>> I was wondering if there is a way without using an additional HTTP
>>>> client so that I can skip the additional content moving between sockets.
>>>>
>>>> Thanks,
>>>> Pavel.
>>>>
>>>
>>>
>>> --
>>> "Come to Me, all who are weary and heavy-laden, and I will
>>> give you rest. Take My yoke upon you and learn from Me, for
>>> I am gentle and humble in heart, and you will find rest for
>>> your souls. For My yoke is easy and My burden is light."
>>>
>>> ~ Matthew 11:28-30
>>>
>>
>>
>
> --
> Regards,
> Dave Seddon
> +1 415 857 5102
>
--
"Come to Me, all who are weary and heavy-laden, and I will
give you rest. Take My yoke upon you and learn from Me, for
I am gentle and humble in heart, and you will find rest for
your souls. For My yoke is easy and My burden is light."
~ Matthew 11:28-30
Re: Using Apache Traffic Server as HTTP client to store some content in the storage
Posted by dave seddon <da...@gmail.com>.
Depending on your use case, you also want to carefully consider the cache
control headers for the downloaded object.
If I understand correctly, ATS does NOT support stale while revalidate,
which is surprising given the RFC was created by a Yahoo-ligan. ;)
On Fri, Apr 19, 2024 at 9:51 AM Leif Hedstrom <zw...@apache.org> wrote:
> We discussed this in the slack channel. Probably the best option for you
> (other than writing a new plugin) is to use the background_fetch plugin.
> You will still need to trigger a download, but you can avoid having to wait
> for the full response in the client. So, what you’d do is
>
> curl -H “Range: bytes=0-1” https://www.example.com/some/url/foo.img
>
>
> And make sure background_fetch is enabled on this remap rule. No matter
> what you do, you will have to have something that tells ATS to kick off a
> download, and the above is probably as good as any alternative.
>
> Cheers,
>
> — Leif
>
>
> On Apr 17, 2024, at 9:33 AM, Pavel Vazharov <pa...@x3me.net> wrote:
>
> Hi Brian,
>
> Thank you for your response.
> It'll do the job but, as far as I understand it, an external functionality
> will need to download the content in order to push it into the ATS.
> The content that I want to write in the ATS storage is on the Internet.
>
> Regards,
> Pavel.
>
> On Wed, Apr 17, 2024 at 6:26 PM Brian Neradt <br...@gmail.com>
> wrote:
>
>> Hi Pavel,
>>
>> This isn't a direct answer to your question, but are you aware of the ATS
>> HTTP PUSH feature? That allows you to push objects into the ATS cache
>> without the typical caching of proxied response. Can that help you in this
>> situation?
>>
>>
>> https://docs.trafficserver.apache.org/en/latest/admin-guide/configuration/cache-basics.en.html#pushing-content-into-the-cache
>>
>>
>> On Wed, Apr 17, 2024 at 8:41 AM Pavel Vazharov <pa...@x3me.net> wrote:
>>
>>> Hi there,
>>>
>>> Is there a way to use the ATS as an HTTP client to download and store
>>> given content?
>>> I'm aware that I can achieve this with a local HTTP client (wget, curl)
>>> which uses the ATS as a forward proxy and then the ATS will store the
>>> content (which is the actual goal).
>>> I was wondering if there is a way without using an additional HTTP
>>> client so that I can skip the additional content moving between sockets.
>>>
>>> Thanks,
>>> Pavel.
>>>
>>
>>
>> --
>> "Come to Me, all who are weary and heavy-laden, and I will
>> give you rest. Take My yoke upon you and learn from Me, for
>> I am gentle and humble in heart, and you will find rest for
>> your souls. For My yoke is easy and My burden is light."
>>
>> ~ Matthew 11:28-30
>>
>
>
--
Regards,
Dave Seddon
+1 415 857 5102
Re: Using Apache Traffic Server as HTTP client to store some content in the storage
Posted by Leif Hedstrom <zw...@apache.org>.
We discussed this in the slack channel. Probably the best option for you (other than writing a new plugin) is to use the background_fetch plugin. You will still need to trigger a download, but you can avoid having to wait for the full response in the client. So, what you’d do is
curl -H “Range: bytes=0-1” https://www.example.com/some/url/foo.img
And make sure background_fetch is enabled on this remap rule. No matter what you do, you will have to have something that tells ATS to kick off a download, and the above is probably as good as any alternative.
Cheers,
— Leif
> On Apr 17, 2024, at 9:33 AM, Pavel Vazharov <pa...@x3me.net> wrote:
>
> Hi Brian,
>
> Thank you for your response.
> It'll do the job but, as far as I understand it, an external functionality will need to download the content in order to push it into the ATS.
> The content that I want to write in the ATS storage is on the Internet.
>
> Regards,
> Pavel.
>
> On Wed, Apr 17, 2024 at 6:26 PM Brian Neradt <brian.neradt@gmail.com <ma...@gmail.com>> wrote:
>> Hi Pavel,
>>
>> This isn't a direct answer to your question, but are you aware of the ATS HTTP PUSH feature? That allows you to push objects into the ATS cache without the typical caching of proxied response. Can that help you in this situation?
>>
>> https://docs.trafficserver.apache.org/en/latest/admin-guide/configuration/cache-basics.en.html#pushing-content-into-the-cache
>>
>> On Wed, Apr 17, 2024 at 8:41 AM Pavel Vazharov <pavel@x3me.net <ma...@x3me.net>> wrote:
>>> Hi there,
>>>
>>> Is there a way to use the ATS as an HTTP client to download and store given content?
>>> I'm aware that I can achieve this with a local HTTP client (wget, curl) which uses the ATS as a forward proxy and then the ATS will store the content (which is the actual goal).
>>> I was wondering if there is a way without using an additional HTTP client so that I can skip the additional content moving between sockets.
>>>
>>> Thanks,
>>> Pavel.
>>
>>
>> --
>> "Come to Me, all who are weary and heavy-laden, and I will
>> give you rest. Take My yoke upon you and learn from Me, for
>> I am gentle and humble in heart, and you will find rest for
>> your souls. For My yoke is easy and My burden is light."
>>
>> ~ Matthew 11:28-30
Re: [E] Re: Using Apache Traffic Server as HTTP client to store some content in the storage
Posted by Walt Karas via users <us...@trafficserver.apache.org>.
Seems like it would be dangerous to enable PUSH on a proxy that is
accessible to untrusted clients. I guess you could set
proxy.local.incoming_ip_to_bind to a loopback address on a second instance
of ATS, and use remap rules to access content pushed to it.
Could it be that the cache control headers of the third-party content
prevent it from being cached?
On Wed, Apr 17, 2024 at 11:36 AM Pavel Vazharov <pa...@x3me.net> wrote:
> Hi Brian,
>
> Thank you for your response.
> It'll do the job but, as far as I understand it, an external functionality
> will need to download the content in order to push it into the ATS.
> The content that I want to write in the ATS storage is on the Internet.
>
> Regards,
> Pavel.
>
> On Wed, Apr 17, 2024 at 6:26 PM Brian Neradt <br...@gmail.com>
> wrote:
>
>> Hi Pavel,
>>
>> This isn't a direct answer to your question, but are you aware of the ATS
>> HTTP PUSH feature? That allows you to push objects into the ATS cache
>> without the typical caching of proxied response. Can that help you in this
>> situation?
>>
>>
>> https://docs.trafficserver.apache.org/en/latest/admin-guide/configuration/cache-basics.en.html#pushing-content-into-the-cache
>> <https://urldefense.com/v3/__https://docs.trafficserver.apache.org/en/latest/admin-guide/configuration/cache-basics.en.html*pushing-content-into-the-cache__;Iw!!Op6eflyXZCqGR5I!Fd2IyeiAblRE5dSqGt5VMcM_96LFFNko6_C1uSy6puDEI9WRcEtBxEsWVLKtIu1zaVKNbspY4bAU$>
>>
>>
>> On Wed, Apr 17, 2024 at 8:41 AM Pavel Vazharov <pa...@x3me.net> wrote:
>>
>>> Hi there,
>>>
>>> Is there a way to use the ATS as an HTTP client to download and store
>>> given content?
>>> I'm aware that I can achieve this with a local HTTP client (wget, curl)
>>> which uses the ATS as a forward proxy and then the ATS will store the
>>> content (which is the actual goal).
>>> I was wondering if there is a way without using an additional HTTP
>>> client so that I can skip the additional content moving between sockets.
>>>
>>> Thanks,
>>> Pavel.
>>>
>>
>>
>> --
>> "Come to Me, all who are weary and heavy-laden, and I will
>> give you rest. Take My yoke upon you and learn from Me, for
>> I am gentle and humble in heart, and you will find rest for
>> your souls. For My yoke is easy and My burden is light."
>>
>> ~ Matthew 11:28-30
>>
>
Re: Using Apache Traffic Server as HTTP client to store some content in the storage
Posted by Pavel Vazharov <pa...@x3me.net>.
Hi Brian,
Thank you for your response.
It'll do the job but, as far as I understand it, an external functionality
will need to download the content in order to push it into the ATS.
The content that I want to write in the ATS storage is on the Internet.
Regards,
Pavel.
On Wed, Apr 17, 2024 at 6:26 PM Brian Neradt <br...@gmail.com> wrote:
> Hi Pavel,
>
> This isn't a direct answer to your question, but are you aware of the ATS
> HTTP PUSH feature? That allows you to push objects into the ATS cache
> without the typical caching of proxied response. Can that help you in this
> situation?
>
>
> https://docs.trafficserver.apache.org/en/latest/admin-guide/configuration/cache-basics.en.html#pushing-content-into-the-cache
>
>
> On Wed, Apr 17, 2024 at 8:41 AM Pavel Vazharov <pa...@x3me.net> wrote:
>
>> Hi there,
>>
>> Is there a way to use the ATS as an HTTP client to download and store
>> given content?
>> I'm aware that I can achieve this with a local HTTP client (wget, curl)
>> which uses the ATS as a forward proxy and then the ATS will store the
>> content (which is the actual goal).
>> I was wondering if there is a way without using an additional HTTP client
>> so that I can skip the additional content moving between sockets.
>>
>> Thanks,
>> Pavel.
>>
>
>
> --
> "Come to Me, all who are weary and heavy-laden, and I will
> give you rest. Take My yoke upon you and learn from Me, for
> I am gentle and humble in heart, and you will find rest for
> your souls. For My yoke is easy and My burden is light."
>
> ~ Matthew 11:28-30
>
Re: Using Apache Traffic Server as HTTP client to store some content in the storage
Posted by Brian Neradt <br...@gmail.com>.
Hi Pavel,
This isn't a direct answer to your question, but are you aware of the ATS
HTTP PUSH feature? That allows you to push objects into the ATS cache
without the typical caching of proxied response. Can that help you in this
situation?
https://docs.trafficserver.apache.org/en/latest/admin-guide/configuration/cache-basics.en.html#pushing-content-into-the-cache
On Wed, Apr 17, 2024 at 8:41 AM Pavel Vazharov <pa...@x3me.net> wrote:
> Hi there,
>
> Is there a way to use the ATS as an HTTP client to download and store
> given content?
> I'm aware that I can achieve this with a local HTTP client (wget, curl)
> which uses the ATS as a forward proxy and then the ATS will store the
> content (which is the actual goal).
> I was wondering if there is a way without using an additional HTTP client
> so that I can skip the additional content moving between sockets.
>
> Thanks,
> Pavel.
>
--
"Come to Me, all who are weary and heavy-laden, and I will
give you rest. Take My yoke upon you and learn from Me, for
I am gentle and humble in heart, and you will find rest for
your souls. For My yoke is easy and My burden is light."
~ Matthew 11:28-30