Suggested changes to default configuration

[John Scharber <jo...@ban.io>]

It think if would reduce the number of posts / amount of time to get TS
working if was configured as a standard forward proxy in the base
configuration, to that end I would suggest the following changes in the
records.config file

CONFIG proxy.config.reverse_proxy.enabled INT 0 // Disable reverse proxy
CONFIG proxy.config.url_remap.remap_required INT 0 //Disable remap required

Also my problem starting traffic server was in the trafficserver script in
the /urs/local/bin directory.

As documented in the Makefile it the pidfile has an erroneous "internal"
directory in the path
# pidfile: /usr/local/var/trafficserver/internal/server.lock

Proper without the "internal" directory in the path
PIDFILE=${PIDFILE:-/usr/local/var/trafficserver/server.lock}

Re: Suggested changes to default configuration

[Jim Tosh <ji...@authentic8.com>]

I would second this. The default config files can require proxy auth for 
the source address (i.e. subnet). This would make it easier for newbs 
(like me) to get started as well as helping illustrate how to make it 
secure.

Having a recipe book (with samples) does not make it any less secure 
than it is today, but sure does make it more approachable to get started.

My 2 cents,
- Jim

On 12/20/09 9:40 PM, John Scharber wrote:
> Fully understand, however I still think the comments are valid.  A user
> should be able to get TS installed an running in a useful configuration with
> minimal fuss or knowledge, just Apache HTTPD.  In my case, being new to TS
> but not proxies or web servers administration or development I took the time
> to read the docs build the code and expect that after following some minimal
> number of steps have a typical configuration running.  So I think from my
> experience there could be a couple of tasks added to the project:
>
> 1) Define what would be a typical installation or possible provide multiple
> sample configuration for different uses
>
> a) Forward
> b) Reverse
> c) Parent/Sibling
>
> 2) Clean up the documentation to clearly explain minimal changes to create a
> functional system.  So for example a forward proxy may be protected by an
> allowed IP range on a private network and a configuration step would bet o
> add my own network or select from an alternatives (proxy auth, forward
> mapping, etc).  This make it clear to a new user what steps that choices
> they needed to make.
>
> Totally unrelated, I'm starting to delve into the code a bit deeper and it
> would be helpful if I would pick someones brain with regards to the cache
> store and event system.  Also is I didn't see Connection Collapsing in the
> documentation does this feature still work?
>
> Thanks everyone has been very helpful
>
> /jms
>
>
> ##############################################################################
> #
> # Connection Collapsing
> #
> ##############################################################################
>    # Adds the ability to collaspe connections going to the orgin server
>    # see Admin Guide Addendum for more details
> CONFIG proxy.config.connection_collapsing.hashtable_enabled INT 0
> CONFIG proxy.config.connection_collapsing.rww_wait_time INT 0
> CONFIG proxy.config.connection_collapsing.revalidate_window_period INT 0
>
>
> On Sun, Dec 20, 2009 at 4:15 PM, Eric Balsa<er...@ericbalsa.com>  wrote:
>
>    
>> Hi John,
>> See the faq on this subject on the wiki. Having open forward proxies on the
>> internet is a bad thing.
>>
>> --Eric
>>
>> On Dec 20, 2009 3:41 PM, "John Scharber"<jo...@ban.io>  wrote:
>>
>> It think if would reduce the number of posts / amount of time to get TS
>> working if was configured as a standard forward proxy in the base
>> configuration, to that end I would suggest the following changes in the
>> records.config file
>>
>> CONFIG proxy.config.reverse_proxy.enabled INT 0 // Disable reverse proxy
>> CONFIG proxy.config.url_remap.remap_required INT 0 //Disable remap required
>>
>> Also my problem starting traffic server was in the trafficserver script in
>> the /urs/local/bin directory.
>>
>> As documented in the Makefile it the pidfile has an erroneous "internal"
>> directory in the path
>> # pidfile: /usr/local/var/trafficserver/internal/server.lock
>>
>> Proper without the "internal" directory in the path
>> PIDFILE=${PIDFILE:-/usr/local/var/trafficserver/server.lock}
>>
>>      
>    

Re: Suggested changes to default configuration

[John Scharber <jo...@ban.io>]

That makes sense but I was hoping it was referring to connection pooling.
 I've been trying to shift through the code and figure out the best location
to insert code to do dynamic site acceleration / wan optimization between
ingress / egress caches which would also needs to include protocol gateways
and path selection.  Ideally it would be done in a generic fashion such that
other developers could add new methods.

Any thoughts on the above would be appreciated.

Given the asynchronous model I assumed you had already done read-while-write
which is certainly a needed feature given the number of streaming solutions
that use http and byte range requests.

Thanks

/jms


On Wed, Dec 23, 2009 at 10:45 AM, Dima Ruban <di...@yahoo-inc.com> wrote:

> On Mon, Dec 21, 2009 at 09:23:36PM -0700, Leif Hedstrom wrote:
> >
> > I believe this is being worked on right now, not sure quite what the
> > status is. Dima Ruban, want to comment on this (since you "own" this
> > part)? It is supposed to work, but I think those "connection_collapsing"
> > configs are either not functioning properly, or not at all ;). There is
> > a feature called "read-while-write" (rww) which should work, and I
> > believe Dima is working on cleaning up this mess a bit.
>
> Read-while-write feature (proxy.config.cache.enable_read_while_writer)
> should work w/ builtin cache. Keep in mind that this feature is nearly
> identical to what other products/projects call a forward connection
> collapsing.
> The way it works is - as soon as writer starts writing object into a cache,
> all the subsequent requests to that object will be able to serve it out
> from
> cache even before writer completes writing the entire object.
> Writer starts writing into cache once it receives headers from the origin
> server, so if second request came before first request received headers,
> read while write isn't gonna come into play. Current code tries to
> deal w/ it with 'connection_collapsing' options but as far as I know it
> doesn't work properly.
> It's gonna be addressed in the newer version of read-while-write that I'm
> working
> on right now.
>
> Cheers!
>
> >
> > Cheers!
> >
> > -- leif
> >
> >
>
> --dima
>

Re: Suggested changes to default configuration

[John Plevyak <jp...@acm.org>]

This feature also has some problems with large documents in the cache.
As far as I can tell it forwards what the writer has on read open
then the rest when the writer closes the connection.  This is fine
for small documents on short lived connections, but is unacceptable for
large documents on long lived connections.  It also buffers the entire
document in memory which could be a problem as well.

I have to do some work on this code to support random reads for efficient
range operations as well, so if you have any cache API needs/ideasn related
to this feature, I'd like to hear them.

john


Dima Ruban wrote:
> On Mon, Dec 21, 2009 at 09:23:36PM -0700, Leif Hedstrom wrote:
>> I believe this is being worked on right now, not sure quite what the 
>> status is. Dima Ruban, want to comment on this (since you "own" this 
>> part)? It is supposed to work, but I think those "connection_collapsing" 
>> configs are either not functioning properly, or not at all ;). There is 
>> a feature called "read-while-write" (rww) which should work, and I 
>> believe Dima is working on cleaning up this mess a bit.
> 
> Read-while-write feature (proxy.config.cache.enable_read_while_writer)
> should work w/ builtin cache. Keep in mind that this feature is nearly
> identical to what other products/projects call a forward connection collapsing.
> The way it works is - as soon as writer starts writing object into a cache,
> all the subsequent requests to that object will be able to serve it out from
> cache even before writer completes writing the entire object.
> Writer starts writing into cache once it receives headers from the origin
> server, so if second request came before first request received headers,
> read while write isn't gonna come into play. Current code tries to
> deal w/ it with 'connection_collapsing' options but as far as I know it
> doesn't work properly.
> It's gonna be addressed in the newer version of read-while-write that I'm working
> on right now.
> 
> Cheers!
> 
>> Cheers!
>>
>> -- leif
>>
>>
> 
> --dima

Re: Suggested changes to default configuration

[Dima Ruban <di...@yahoo-inc.com>]

On Mon, Dec 21, 2009 at 09:23:36PM -0700, Leif Hedstrom wrote:
> 
> I believe this is being worked on right now, not sure quite what the 
> status is. Dima Ruban, want to comment on this (since you "own" this 
> part)? It is supposed to work, but I think those "connection_collapsing" 
> configs are either not functioning properly, or not at all ;). There is 
> a feature called "read-while-write" (rww) which should work, and I 
> believe Dima is working on cleaning up this mess a bit.

Read-while-write feature (proxy.config.cache.enable_read_while_writer)
should work w/ builtin cache. Keep in mind that this feature is nearly
identical to what other products/projects call a forward connection collapsing.
The way it works is - as soon as writer starts writing object into a cache,
all the subsequent requests to that object will be able to serve it out from
cache even before writer completes writing the entire object.
Writer starts writing into cache once it receives headers from the origin
server, so if second request came before first request received headers,
read while write isn't gonna come into play. Current code tries to
deal w/ it with 'connection_collapsing' options but as far as I know it
doesn't work properly.
It's gonna be addressed in the newer version of read-while-write that I'm working
on right now.

Cheers!

> 
> Cheers!
> 
> -- leif
> 
> 

--dima

Re: Suggested changes to default configuration

[Leif Hedstrom <le...@ogre.com>]

On 12/20/2009 10:40 PM, John Scharber wrote:
> Fully understand, however I still think the comments are valid.  A user
> should be able to get TS installed an running in a useful configuration with
> minimal fuss or knowledge, just Apache HTTPD.  In my case, being new to TS
> but not proxies or web servers administration or development I took the time
>    

I'm kinda with Eric on keeping the defaults as conservative as possible. 
But, I'm thinking, we really need to provide "basic" configs for the 
common setups that people would use (to make it a lot easier to get 
started). E.g. forward and reverse proxy configs at least.


Ideally, if you could file bugs on your ideas and suggestions (including 
docs changes etc.), that'd be great!

     http://issues.apache.org/jira/browse/TS


> would be helpful if I would pick someones brain with regards to the cache
> store and event system.  Also is I didn't see Connection Collapsing in the
> documentation does this feature still work?
>    

I believe this is being worked on right now, not sure quite what the 
status is. Dima Ruban, want to comment on this (since you "own" this 
part)? It is supposed to work, but I think those "connection_collapsing" 
configs are either not functioning properly, or not at all ;). There is 
a feature called "read-while-write" (rww) which should work, and I 
believe Dima is working on cleaning up this mess a bit.

Cheers!

-- leif

Re: Suggested changes to default configuration

[John Scharber <jo...@ban.io>]

Fully understand, however I still think the comments are valid.  A user
should be able to get TS installed an running in a useful configuration with
minimal fuss or knowledge, just Apache HTTPD.  In my case, being new to TS
but not proxies or web servers administration or development I took the time
to read the docs build the code and expect that after following some minimal
number of steps have a typical configuration running.  So I think from my
experience there could be a couple of tasks added to the project:

1) Define what would be a typical installation or possible provide multiple
sample configuration for different uses

a) Forward
b) Reverse
c) Parent/Sibling

2) Clean up the documentation to clearly explain minimal changes to create a
functional system.  So for example a forward proxy may be protected by an
allowed IP range on a private network and a configuration step would bet o
add my own network or select from an alternatives (proxy auth, forward
mapping, etc).  This make it clear to a new user what steps that choices
they needed to make.

Totally unrelated, I'm starting to delve into the code a bit deeper and it
would be helpful if I would pick someones brain with regards to the cache
store and event system.  Also is I didn't see Connection Collapsing in the
documentation does this feature still work?

Thanks everyone has been very helpful

/jms


##############################################################################
#
# Connection Collapsing
#
##############################################################################
  # Adds the ability to collaspe connections going to the orgin server
  # see Admin Guide Addendum for more details
CONFIG proxy.config.connection_collapsing.hashtable_enabled INT 0
CONFIG proxy.config.connection_collapsing.rww_wait_time INT 0
CONFIG proxy.config.connection_collapsing.revalidate_window_period INT 0


On Sun, Dec 20, 2009 at 4:15 PM, Eric Balsa <er...@ericbalsa.com> wrote:

> Hi John,
> See the faq on this subject on the wiki. Having open forward proxies on the
> internet is a bad thing.
>
> --Eric
>
> On Dec 20, 2009 3:41 PM, "John Scharber" <jo...@ban.io> wrote:
>
> It think if would reduce the number of posts / amount of time to get TS
> working if was configured as a standard forward proxy in the base
> configuration, to that end I would suggest the following changes in the
> records.config file
>
> CONFIG proxy.config.reverse_proxy.enabled INT 0 // Disable reverse proxy
> CONFIG proxy.config.url_remap.remap_required INT 0 //Disable remap required
>
> Also my problem starting traffic server was in the trafficserver script in
> the /urs/local/bin directory.
>
> As documented in the Makefile it the pidfile has an erroneous "internal"
> directory in the path
> # pidfile: /usr/local/var/trafficserver/internal/server.lock
>
> Proper without the "internal" directory in the path
> PIDFILE=${PIDFILE:-/usr/local/var/trafficserver/server.lock}
>

Re: Suggested changes to default configuration

[Eric Balsa <er...@ericbalsa.com>]

Hi John,
See the faq on this subject on the wiki. Having open forward proxies on the
internet is a bad thing.

--Eric

On Dec 20, 2009 3:41 PM, "John Scharber" <jo...@ban.io> wrote:

It think if would reduce the number of posts / amount of time to get TS
working if was configured as a standard forward proxy in the base
configuration, to that end I would suggest the following changes in the
records.config file

CONFIG proxy.config.reverse_proxy.enabled INT 0 // Disable reverse proxy
CONFIG proxy.config.url_remap.remap_required INT 0 //Disable remap required

Also my problem starting traffic server was in the trafficserver script in
the /urs/local/bin directory.

As documented in the Makefile it the pidfile has an erroneous "internal"
directory in the path
# pidfile: /usr/local/var/trafficserver/internal/server.lock

Proper without the "internal" directory in the path
PIDFILE=${PIDFILE:-/usr/local/var/trafficserver/server.lock}