You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficcontrol.apache.org by Abdulrahman Elawady <ae...@apache.org> on 2023/10/05 10:55:54 UTC

Next Step for Varnish Support

Hi Everyone,

Now that Varnish is somewhat in a good state with a lot of the basics done,
I wanted to discuss with you the next steps for making Varnish a viable
option as a cache server for Traffic Control.

There are multiple options for what to do next and what to focus on and I
am interested in your opinions on that.

Some of these options are:

Varnish configuration:
- Cache server storage configuration. It might not be identical to ATS due
to different cache storage models between Varnish and ATS.
- ATS plugins like: header rewrite, regex remap. Some of the plugins are
straightforward and supported directly by Varnish while others will need
some work.
- DS improvements (dscp, qstring, go_direct...) and backends health checks.
- Error pages to match ATS.

Functional Tests:
- Some integration tests are needed for Varnish to ensure the functionality
of VCL files generated are correct and make it easier for future
development to be integrated. It is somewhat different than ORT tests
available as it tests the functionality of the cache server itself rather
than t3c or some baseline configuration files.

Varnish development:
- It would be great to make Varnish development easier for newcomers and
people interested. So, some of the previous points could be simplified into
good first issues.
- Varnish configuration package "varnishcfg" could be improved based on how
easy it is to add features to it. If it turned out to be somewhat difficult
we can see other options like go template language or maybe document it
more. Of course I can't be the judge of that since I wrote a lot of it so
hopefully the good first issues might give more insight to that.

Traffic Monitor:
- After #7805 DS stats will be reported except bytes in and out because
Varnish does not report this data. So, it might need a solution outside
Varnish. It's probably not a priority right now but just worth mentioning.

These are just some ideas of what to do next, feel free to suggest other
ideas. I am interested in your opinions on what to prioritise and focus on.

Regards,
Abdulrahman

Re: Next Step for Varnish Support

Posted by Eric Friedrich <fr...@alum.rpi.edu>.
Definitely a tough question and I see arguments both ways.

One might consider it as
1) Will users mix and match between ATS and Varnish in a single Delivery
service?
2) If yes, is it okay to for them to have to maintain two sets of header
rewrite rules?

In a perfect world, we'd only want one copy of any CDN policies, less to
maintain and eliminates problems with them getting out of sync.
However, in this prototype implementation, I think its as good a starting
point as any to stick with the native formats for both.

Would love to hear others thoughts as well

--Eric


On Wed, Oct 11, 2023 at 10:42 AM Abdulrahman Elawady <
abdoelawady125@gmail.com> wrote:

> I had some questions about how we should approach supporting header
> rewrites in Varnish.
>
> As you know, using VCL we can directly rewrite headers by accessing the
> request and response objects. The concern is how should the user of Traffic
> Control specify which headers to rewrite or based on what conditions.
> Normally with Traffic Server we can use something like the plugin.config
> parameter to specify the header_rewrite plugin with the path to files
> containing rewrite directives based on ATS supported syntax.
>
> Should we expect the user to also provide ATS syntax?
>
> While it's doable, it is not that easy to parse this configuration and
> maintaining it will be non-trivial.
>
> We can also expect from the user to input the directives in a Varnish way
> using VCL. Since it is just like any VCL we will be basically allowing the
> user to add their own blocks of code whether it's a header rewrite or just
> a custom logic. I think that would be easy to maintain and will offer more
> customization but will require more parameters.
>
> What do you think?
>
> Abdulrahman
>
> On Sat, Oct 7, 2023 at 3:44 PM Eric Friedrich <fr...@alum.rpi.edu> wrote:
>
> > Great suggestions Abdulrahman!
> >
> > I think in terms of base functionality, that header rewrites are very
> > frequently used, so this might be a good next step. Lots of times extra
> > response headers are needed, especially if CORS is required.
> >
> > Regex remap is also useful because CDNs frequently are asked to modify
> the
> > URL, and Varnish has great support for this with regsub() already.
> >
> > After that, paying off some tech debt on the testing might make sense?
> > Would be very cool if we could find a way to integrate varnishtest
> somehow
> > as its purpose built for testing VCL functionality
> >
> > Best,
> > Eric
> >
> > On Thu, Oct 5, 2023 at 6:56 AM Abdulrahman Elawady <ae...@apache.org>
> > wrote:
> >
> > > Hi Everyone,
> > >
> > > Now that Varnish is somewhat in a good state with a lot of the basics
> > done,
> > > I wanted to discuss with you the next steps for making Varnish a viable
> > > option as a cache server for Traffic Control.
> > >
> > > There are multiple options for what to do next and what to focus on
> and I
> > > am interested in your opinions on that.
> > >
> > > Some of these options are:
> > >
> > > Varnish configuration:
> > > - Cache server storage configuration. It might not be identical to ATS
> > due
> > > to different cache storage models between Varnish and ATS.
> > > - ATS plugins like: header rewrite, regex remap. Some of the plugins
> are
> > > straightforward and supported directly by Varnish while others will
> need
> > > some work.
> > > - DS improvements (dscp, qstring, go_direct...) and backends health
> > checks.
> > > - Error pages to match ATS.
> > >
> > > Functional Tests:
> > > - Some integration tests are needed for Varnish to ensure the
> > functionality
> > > of VCL files generated are correct and make it easier for future
> > > development to be integrated. It is somewhat different than ORT tests
> > > available as it tests the functionality of the cache server itself
> rather
> > > than t3c or some baseline configuration files.
> > >
> > > Varnish development:
> > > - It would be great to make Varnish development easier for newcomers
> and
> > > people interested. So, some of the previous points could be simplified
> > into
> > > good first issues.
> > > - Varnish configuration package "varnishcfg" could be improved based on
> > how
> > > easy it is to add features to it. If it turned out to be somewhat
> > difficult
> > > we can see other options like go template language or maybe document it
> > > more. Of course I can't be the judge of that since I wrote a lot of it
> so
> > > hopefully the good first issues might give more insight to that.
> > >
> > > Traffic Monitor:
> > > - After #7805 DS stats will be reported except bytes in and out because
> > > Varnish does not report this data. So, it might need a solution outside
> > > Varnish. It's probably not a priority right now but just worth
> > mentioning.
> > >
> > > These are just some ideas of what to do next, feel free to suggest
> other
> > > ideas. I am interested in your opinions on what to prioritise and focus
> > on.
> > >
> > > Regards,
> > > Abdulrahman
> > >
> >
>

Re: Next Step for Varnish Support

Posted by Abdulrahman Elawady <ab...@gmail.com>.
I had some questions about how we should approach supporting header
rewrites in Varnish.

As you know, using VCL we can directly rewrite headers by accessing the
request and response objects. The concern is how should the user of Traffic
Control specify which headers to rewrite or based on what conditions.
Normally with Traffic Server we can use something like the plugin.config
parameter to specify the header_rewrite plugin with the path to files
containing rewrite directives based on ATS supported syntax.

Should we expect the user to also provide ATS syntax?

While it's doable, it is not that easy to parse this configuration and
maintaining it will be non-trivial.

We can also expect from the user to input the directives in a Varnish way
using VCL. Since it is just like any VCL we will be basically allowing the
user to add their own blocks of code whether it's a header rewrite or just
a custom logic. I think that would be easy to maintain and will offer more
customization but will require more parameters.

What do you think?

Abdulrahman

On Sat, Oct 7, 2023 at 3:44 PM Eric Friedrich <fr...@alum.rpi.edu> wrote:

> Great suggestions Abdulrahman!
>
> I think in terms of base functionality, that header rewrites are very
> frequently used, so this might be a good next step. Lots of times extra
> response headers are needed, especially if CORS is required.
>
> Regex remap is also useful because CDNs frequently are asked to modify the
> URL, and Varnish has great support for this with regsub() already.
>
> After that, paying off some tech debt on the testing might make sense?
> Would be very cool if we could find a way to integrate varnishtest somehow
> as its purpose built for testing VCL functionality
>
> Best,
> Eric
>
> On Thu, Oct 5, 2023 at 6:56 AM Abdulrahman Elawady <ae...@apache.org>
> wrote:
>
> > Hi Everyone,
> >
> > Now that Varnish is somewhat in a good state with a lot of the basics
> done,
> > I wanted to discuss with you the next steps for making Varnish a viable
> > option as a cache server for Traffic Control.
> >
> > There are multiple options for what to do next and what to focus on and I
> > am interested in your opinions on that.
> >
> > Some of these options are:
> >
> > Varnish configuration:
> > - Cache server storage configuration. It might not be identical to ATS
> due
> > to different cache storage models between Varnish and ATS.
> > - ATS plugins like: header rewrite, regex remap. Some of the plugins are
> > straightforward and supported directly by Varnish while others will need
> > some work.
> > - DS improvements (dscp, qstring, go_direct...) and backends health
> checks.
> > - Error pages to match ATS.
> >
> > Functional Tests:
> > - Some integration tests are needed for Varnish to ensure the
> functionality
> > of VCL files generated are correct and make it easier for future
> > development to be integrated. It is somewhat different than ORT tests
> > available as it tests the functionality of the cache server itself rather
> > than t3c or some baseline configuration files.
> >
> > Varnish development:
> > - It would be great to make Varnish development easier for newcomers and
> > people interested. So, some of the previous points could be simplified
> into
> > good first issues.
> > - Varnish configuration package "varnishcfg" could be improved based on
> how
> > easy it is to add features to it. If it turned out to be somewhat
> difficult
> > we can see other options like go template language or maybe document it
> > more. Of course I can't be the judge of that since I wrote a lot of it so
> > hopefully the good first issues might give more insight to that.
> >
> > Traffic Monitor:
> > - After #7805 DS stats will be reported except bytes in and out because
> > Varnish does not report this data. So, it might need a solution outside
> > Varnish. It's probably not a priority right now but just worth
> mentioning.
> >
> > These are just some ideas of what to do next, feel free to suggest other
> > ideas. I am interested in your opinions on what to prioritise and focus
> on.
> >
> > Regards,
> > Abdulrahman
> >
>

Re: Next Step for Varnish Support

Posted by Eric Friedrich <fr...@alum.rpi.edu>.
Great suggestions Abdulrahman!

I think in terms of base functionality, that header rewrites are very
frequently used, so this might be a good next step. Lots of times extra
response headers are needed, especially if CORS is required.

Regex remap is also useful because CDNs frequently are asked to modify the
URL, and Varnish has great support for this with regsub() already.

After that, paying off some tech debt on the testing might make sense?
Would be very cool if we could find a way to integrate varnishtest somehow
as its purpose built for testing VCL functionality

Best,
Eric

On Thu, Oct 5, 2023 at 6:56 AM Abdulrahman Elawady <ae...@apache.org>
wrote:

> Hi Everyone,
>
> Now that Varnish is somewhat in a good state with a lot of the basics done,
> I wanted to discuss with you the next steps for making Varnish a viable
> option as a cache server for Traffic Control.
>
> There are multiple options for what to do next and what to focus on and I
> am interested in your opinions on that.
>
> Some of these options are:
>
> Varnish configuration:
> - Cache server storage configuration. It might not be identical to ATS due
> to different cache storage models between Varnish and ATS.
> - ATS plugins like: header rewrite, regex remap. Some of the plugins are
> straightforward and supported directly by Varnish while others will need
> some work.
> - DS improvements (dscp, qstring, go_direct...) and backends health checks.
> - Error pages to match ATS.
>
> Functional Tests:
> - Some integration tests are needed for Varnish to ensure the functionality
> of VCL files generated are correct and make it easier for future
> development to be integrated. It is somewhat different than ORT tests
> available as it tests the functionality of the cache server itself rather
> than t3c or some baseline configuration files.
>
> Varnish development:
> - It would be great to make Varnish development easier for newcomers and
> people interested. So, some of the previous points could be simplified into
> good first issues.
> - Varnish configuration package "varnishcfg" could be improved based on how
> easy it is to add features to it. If it turned out to be somewhat difficult
> we can see other options like go template language or maybe document it
> more. Of course I can't be the judge of that since I wrote a lot of it so
> hopefully the good first issues might give more insight to that.
>
> Traffic Monitor:
> - After #7805 DS stats will be reported except bytes in and out because
> Varnish does not report this data. So, it might need a solution outside
> Varnish. It's probably not a priority right now but just worth mentioning.
>
> These are just some ideas of what to do next, feel free to suggest other
> ideas. I am interested in your opinions on what to prioritise and focus on.
>
> Regards,
> Abdulrahman
>