You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@trafficserver.apache.org by ml...@apache.org on 2013/12/19 02:05:12 UTC
[01/12] git commit: [TS-2328] Document failover configuration for log
collation
Updated Branches:
refs/heads/master 0c953e129 -> 90af89c47
[TS-2328] Document failover configuration for log collation
Project: http://git-wip-us.apache.org/repos/asf/trafficserver/repo
Commit: http://git-wip-us.apache.org/repos/asf/trafficserver/commit/5871b0af
Tree: http://git-wip-us.apache.org/repos/asf/trafficserver/tree/5871b0af
Diff: http://git-wip-us.apache.org/repos/asf/trafficserver/diff/5871b0af
Branch: refs/heads/master
Commit: 5871b0af68d9a6e17e00efb09600c550188247c5
Parents: 3388fd8
Author: Miles Libbey <ml...@apache.org>
Authored: Fri Dec 13 10:55:21 2013 -0800
Committer: Miles Libbey <ml...@apache.org>
Committed: Fri Dec 13 10:55:21 2013 -0800
----------------------------------------------------------------------
doc/reference/configuration/logs_xml.config.en.rst | 16 +++++++++++-----
1 file changed, 11 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/trafficserver/blob/5871b0af/doc/reference/configuration/logs_xml.config.en.rst
----------------------------------------------------------------------
diff --git a/doc/reference/configuration/logs_xml.config.en.rst b/doc/reference/configuration/logs_xml.config.en.rst
index 1d2a057..92d90f1 100644
--- a/doc/reference/configuration/logs_xml.config.en.rst
+++ b/doc/reference/configuration/logs_xml.config.en.rst
@@ -217,12 +217,18 @@ The following list shows the ``LogObject`` specifications.
.. _logs-xml-logobject-collationhost:
-``<CollationHosts = "list_of_valid_hostnames"/>``
+``<CollationHosts = "list_of_valid_hostnames:port|failover hosts"/>``
Optional
- A comma-separated list of collation servers to which all log entries
- (for this object) are forwarded. Collation servers can be specified
- by name or IP address. Specify the collation port with a colon after
- the name; for example, ``host:port`` .
+ A comma-separated list of collation servers (with pipe delimited
+ failover servers) to which all log entries (for this object) are
+ forwarded. Collation servers can be specified by name or IP address.
+ Specify the collation port with a colon after the name. For example,
+ in ``host1:5000|failhostA:5000|failhostB:6000, host2:6000`` logs
+ would be sent to host1 and host2, with failhostA and failhostB
+ acting as failover hosts for host1. When host1 disconnects,
+ logs would be sent to failhostA. If failhostA disconnects, log
+ entries would be sent to failhostB until host1 or failhostA comes
+ back. Logs would also be sent to host2.
``<Header = "header"/>``
Optional
[09/12] git commit: [TS-625] Copy remap.conf description of
map_with_referer over to site documentation
Posted by ml...@apache.org.
[TS-625] Copy remap.conf description of map_with_referer over to site documentation
Project: http://git-wip-us.apache.org/repos/asf/trafficserver/repo
Commit: http://git-wip-us.apache.org/repos/asf/trafficserver/commit/f2c84527
Tree: http://git-wip-us.apache.org/repos/asf/trafficserver/tree/f2c84527
Diff: http://git-wip-us.apache.org/repos/asf/trafficserver/diff/f2c84527
Branch: refs/heads/master
Commit: f2c845270412962f540680de76c9d2e07ba54668
Parents: 44a8614
Author: Miles Libbey <ml...@apache.org>
Authored: Mon Dec 16 14:58:58 2013 -0800
Committer: Miles Libbey <ml...@apache.org>
Committed: Mon Dec 16 14:58:58 2013 -0800
----------------------------------------------------------------------
doc/reference/configuration/remap.config.en.rst | 73 ++++++++++++++++++++
1 file changed, 73 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/trafficserver/blob/f2c84527/doc/reference/configuration/remap.config.en.rst
----------------------------------------------------------------------
diff --git a/doc/reference/configuration/remap.config.en.rst b/doc/reference/configuration/remap.config.en.rst
index b6223da..dc5a47f 100644
--- a/doc/reference/configuration/remap.config.en.rst
+++ b/doc/reference/configuration/remap.config.en.rst
@@ -63,6 +63,10 @@ Traffic Server recognizes three space-delimited fields: ``type``,
- ``map`` --translates an incoming request URL to the appropriate
origin server URL.
+ - ``map_with_referer`` -- extended version of 'map', which can be used to activate
+ "deep linking protection", where target URLs are only accessible when the Referer
+ header is set to a URL that is allowed to link to the target.
+
- ``reverse_map`` --translates the URL in origin server redirect
responses to point to the Traffic Server.
@@ -270,6 +274,75 @@ Examples
regex_map http://x([0-9]+).z.com/ http://real-x$1.z.com/
regex_redirect http://old.(.*).z.com http://new.$1.z.com
+.. _map_with_referer:
+
+map_with_referer
+================
+
+the format of is the following::
+
+ map_with_referer client-URL origin-server-URL redirect-URL regex1 [regex2 ...]
+
+'redirect-URL' is a redirection URL specified according to RFC 2616 and can
+contain special formatting instructions for run-time modifications of the
+resulting redirection URL. All regexes Perl compatible regular expressions,
+which describes the content of the "Referer" header which must be
+verified. In case an actual request does not have "Referer" header or it
+does not match with referer regular expression, the HTTP request will be
+redirected to 'redirect-URL'.
+
+At least one regular expressions must be specified in order to activate
+'deep linking protection'. There are limitations for the number of referer
+regular expression strings - 2048. In order to enable the 'deep linking
+protection' feature in Traffic Server, configure records.config with::
+
+ CONFIG proxy.config.http.referer_filter INT 1
+
+In order to enable run-time formatting for redirect URL, configure::
+
+ CONFIG proxy.config.http.referer_format_redirect INT 1
+
+When run-time formatting for redirect-URL was enabled the following format
+symbols can be used::
+
+ %r - to substitute original "Referer" header string
+ %f - to substitute client-URL from 'map_with_referer' record
+ %t - to substitute origin-server-URL from 'map_with_referer' record
+ %o - to substitute request URL to origin server, which was created a
+ the result of a mapping operation
+
+Note: There is a special referer type "~*" that can be used in order to
+specify that the Referer header is optional in the request. If "~*" referer
+was used in map_with_referer mapping, only requests with Referer header will
+be verified for validity. If the "~" symbol was specified before referer
+regular expression, it means that the request with a matching referer header
+will be redirected to redirectURL. It can be used to create a so-called
+negative referer list. If "*" was used as a referer regular expression -
+all referers are allowed. Various combinations of "*" and "~" in a referer
+list can be used to create different filtering rules.
+
+map_with_referer Examples
+-------------------------
+
+::
+
+ map_with_referer http://y.foo.bar.com/x/yy/ http://foo.bar.com/x/yy/ http://games.bar.com/new_games .*\.bar\.com www.bar-friends.com
+
+Explanation: Referer header must be in the request, only ".*\.bar\.com" and "www.bar-friends.com" are allowed.
+
+::
+
+ map_with_referer http://y.foo.bar.com/x/yy/ http://foo.bar.com/x/yy/ http://games.bar.com/new_games * ~.*\.evil\.com
+
+Explanation: Referer header must be in the request but all referers are allowed except ".*\.evil\.com".
+
+::
+
+ map_with_referer http://y.foo.bar.com/x/yy/ http://foo.bar.com/x/yy/ http://games.bar.com/error ~* * ~.*\.evil\.com
+
+Explanation: Referer header is optional. However, if Referer header exists, only request from ".*\.evil\.com" will be redirected to redirect-URL.
+
+
.. _remap-config-plugin-chaining:
Plugin Chaining
[10/12] git commit: [TS-2310] Copy over remap.conf description of
map_with_recv_port remap rule type
Posted by ml...@apache.org.
[TS-2310] Copy over remap.conf description of map_with_recv_port remap rule type
Project: http://git-wip-us.apache.org/repos/asf/trafficserver/repo
Commit: http://git-wip-us.apache.org/repos/asf/trafficserver/commit/29f3a12b
Tree: http://git-wip-us.apache.org/repos/asf/trafficserver/tree/29f3a12b
Diff: http://git-wip-us.apache.org/repos/asf/trafficserver/diff/29f3a12b
Branch: refs/heads/master
Commit: 29f3a12b1cffe15c1f89a292e1e9c5f53147847b
Parents: f2c8452
Author: Miles Libbey <ml...@apache.org>
Authored: Mon Dec 16 15:11:10 2013 -0800
Committer: Miles Libbey <ml...@apache.org>
Committed: Mon Dec 16 15:11:10 2013 -0800
----------------------------------------------------------------------
doc/reference/configuration/remap.config.en.rst | 15 +++++++++++----
1 file changed, 11 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/trafficserver/blob/29f3a12b/doc/reference/configuration/remap.config.en.rst
----------------------------------------------------------------------
diff --git a/doc/reference/configuration/remap.config.en.rst b/doc/reference/configuration/remap.config.en.rst
index dc5a47f..18f725c 100644
--- a/doc/reference/configuration/remap.config.en.rst
+++ b/doc/reference/configuration/remap.config.en.rst
@@ -63,6 +63,12 @@ Traffic Server recognizes three space-delimited fields: ``type``,
- ``map`` --translates an incoming request URL to the appropriate
origin server URL.
+ - ``map_with_recv_port`` --exactly like 'map' except that it uses the port at
+ which the request was received to perform the mapping instead of the port present
+ in the request. The regex qualifier can also be used for this type. When present,
+ 'map_with_recv_port' mappings are checked first. If there is a match, then it is
+ chosen without evaluating the "regular" forward mapping rules.
+
- ``map_with_referer`` -- extended version of 'map', which can be used to activate
"deep linking protection", where target URLs are only accessible when the Referer
header is set to a URL that is allowed to link to the target.
@@ -109,10 +115,11 @@ Precedence
Remap rules are not processed top-down, but based on an internal
priority
-1. ``map`` and ``reverse_map``
-2. ``regex_map``
-3. ``redirect`` and ``redirect_temporary``
-4. ``regex_redirect`` and ``regex_redirect_temporary``
+1. ``map_with_recv_port``
+#. ``map`` and ``reverse_map``
+#. ``regex_map``
+#. ``redirect`` and ``redirect_temporary``
+#. ``regex_redirect`` and ``regex_redirect_temporary``
Match-All
Re: [12/12] git commit: Merge branch 'master' of
https://git-wip-us.apache.org/repos/asf/trafficserver
Posted by Igor Galić <i....@brainsware.org>.
Please whenever possible, rebase before pushing.
Otherwise it's quite easy for reviewers to miss a commit..
----- Original Message -----
> Merge branch 'master' of
> https://git-wip-us.apache.org/repos/asf/trafficserver
>
>
> Project: http://git-wip-us.apache.org/repos/asf/trafficserver/repo
> Commit: http://git-wip-us.apache.org/repos/asf/trafficserver/commit/90af89c4
> Tree: http://git-wip-us.apache.org/repos/asf/trafficserver/tree/90af89c4
> Diff: http://git-wip-us.apache.org/repos/asf/trafficserver/diff/90af89c4
>
> Branch: refs/heads/master
> Commit: 90af89c47425cf55ca1a472ec68847afb2efc57a
> Parents: 9e06528 0c953e1
> Author: Miles Libbey <ml...@apache.org>
> Authored: Wed Dec 18 17:03:02 2013 -0800
> Committer: Miles Libbey <ml...@apache.org>
> Committed: Wed Dec 18 17:03:02 2013 -0800
>
> ----------------------------------------------------------------------
> CHANGES | 2 +-
> doc/arch/cache/cache-arch.en.rst | 148 ++++++++++++-------
> doc/glossary.en.rst | 42 +++++-
> ...viewing-statistics-using-traffic-line.en.rst | 2 +-
> lib/atscppapi/src/include/atscppapi/Headers.h | 2 +-
> 5 files changed, 142 insertions(+), 54 deletions(-)
> ----------------------------------------------------------------------
>
>
>
--
Igor Galić
Tel: +43 (0) 664 886 22 883
Mail: i.galic@brainsware.org
URL: http://brainsware.org/
GPG: 8716 7A9F 989B ABD5 100F 4008 F266 55D6 2998 1641
[12/12] git commit: Merge branch 'master' of
https://git-wip-us.apache.org/repos/asf/trafficserver
Posted by ml...@apache.org.
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/trafficserver
Project: http://git-wip-us.apache.org/repos/asf/trafficserver/repo
Commit: http://git-wip-us.apache.org/repos/asf/trafficserver/commit/90af89c4
Tree: http://git-wip-us.apache.org/repos/asf/trafficserver/tree/90af89c4
Diff: http://git-wip-us.apache.org/repos/asf/trafficserver/diff/90af89c4
Branch: refs/heads/master
Commit: 90af89c47425cf55ca1a472ec68847afb2efc57a
Parents: 9e06528 0c953e1
Author: Miles Libbey <ml...@apache.org>
Authored: Wed Dec 18 17:03:02 2013 -0800
Committer: Miles Libbey <ml...@apache.org>
Committed: Wed Dec 18 17:03:02 2013 -0800
----------------------------------------------------------------------
CHANGES | 2 +-
doc/arch/cache/cache-arch.en.rst | 148 ++++++++++++-------
doc/glossary.en.rst | 42 +++++-
...viewing-statistics-using-traffic-line.en.rst | 2 +-
lib/atscppapi/src/include/atscppapi/Headers.h | 2 +-
5 files changed, 142 insertions(+), 54 deletions(-)
----------------------------------------------------------------------
[11/12] git commit: Merge branch 'master' of
https://git-wip-us.apache.org/repos/asf/trafficserver
Posted by ml...@apache.org.
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/trafficserver
Project: http://git-wip-us.apache.org/repos/asf/trafficserver/repo
Commit: http://git-wip-us.apache.org/repos/asf/trafficserver/commit/9e06528c
Tree: http://git-wip-us.apache.org/repos/asf/trafficserver/tree/9e06528c
Diff: http://git-wip-us.apache.org/repos/asf/trafficserver/diff/9e06528c
Branch: refs/heads/master
Commit: 9e06528ce14803821b921c1134ecb4a54c9e5176
Parents: 29f3a12 fb46319
Author: Miles Libbey <ml...@apache.org>
Authored: Tue Dec 17 10:50:06 2013 -0800
Committer: Miles Libbey <ml...@apache.org>
Committed: Tue Dec 17 10:50:06 2013 -0800
----------------------------------------------------------------------
CHANGES | 9 +
ci/tsqa/functions | 244 +++++++++++++++++++
ci/tsqa/test-bootstrap | 41 ++++
ci/tsqa/test-log-configuration | 63 +++++
doc/admin/reverse-proxy-http-redirects.en.rst | 2 +-
doc/admin/security-options.en.rst | 4 +-
.../configuration/congestion.config.en.rst | 2 +-
.../configuration/plugin.config.en.rst | 52 ++--
.../configuration/records.config.en.rst | 10 +-
.../configuration/ssl_multicert.config.en.rst | 2 +-
.../configuration/storage.config.en.rst | 2 +-
doc/reference/plugins/cacheurl.en.rst | 3 +-
doc/reference/plugins/combo_handler.en.rst | 4 +-
doc/reference/plugins/gzip.en.rst | 3 +-
doc/reference/plugins/header_filter.en.rst | 3 +-
iocore/aio/AIO.cc | 1 +
iocore/net/P_SSLUtils.h | 9 +
iocore/net/SSLConfig.cc | 6 +
mgmt/RecordsConfig.cc | 4 +
plugins/experimental/ts_lua/Makefile.am | 2 +-
.../experimental/ts_lua/ts_lua_client_request.c | 45 +++-
plugins/experimental/ts_lua/ts_lua_remap.c | 67 +++++
plugins/experimental/ts_lua/ts_lua_remap.h | 28 +++
plugins/experimental/ts_lua/ts_lua_util.c | 3 +
proxy/Main.cc | 6 +-
proxy/Main.h | 1 -
proxy/Plugin.cc | 14 +-
proxy/http/HttpSM.cc | 2 +-
proxy/http/HttpTransact.cc | 26 +-
proxy/logging/LogStandalone.cc | 14 +-
tools/tsxs.in | 5 +
31 files changed, 608 insertions(+), 69 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/trafficserver/blob/9e06528c/doc/reference/configuration/records.config.en.rst
----------------------------------------------------------------------
Re: [05/12] git commit: [TS-2183] Make a restructured text version
of the ram-cache wiki page
Posted by Igor Galić <i....@brainsware.org>.
----- Original Message -----
> [TS-2183] Make a restructured text version of the ram-cache wiki page
>
>
> Project: http://git-wip-us.apache.org/repos/asf/trafficserver/repo
> Commit: http://git-wip-us.apache.org/repos/asf/trafficserver/commit/219ba692
> Tree: http://git-wip-us.apache.org/repos/asf/trafficserver/tree/219ba692
> Diff: http://git-wip-us.apache.org/repos/asf/trafficserver/diff/219ba692
>
> Branch: refs/heads/master
> Commit: 219ba692eaf6e85fe99c41efdd176a9f2ed18c4b
> Parents: 0f45c06
> Author: Miles Libbey <ml...@apache.org>
> Authored: Fri Dec 13 13:52:15 2013 -0800
> Committer: Miles Libbey <ml...@apache.org>
> Committed: Fri Dec 13 13:52:15 2013 -0800
>
> ----------------------------------------------------------------------
> doc/arch/cache/ram-cache.en.rst | 88 ++++++++++++++++++++++++++++++++++++
> 1 file changed, 88 insertions(+)
> ----------------------------------------------------------------------
>
>
> http://git-wip-us.apache.org/repos/asf/trafficserver/blob/219ba692/doc/arch/cache/ram-cache.en.rst
> ----------------------------------------------------------------------
> diff --git a/doc/arch/cache/ram-cache.en.rst
> b/doc/arch/cache/ram-cache.en.rst
> new file mode 100644
> index 0000000..b0b15e1
> --- /dev/null
> +++ b/doc/arch/cache/ram-cache.en.rst
> @@ -0,0 +1,88 @@
> +.. Licensed to the Apache Software Foundation (ASF) under one
> + or more contributor license agreements. See the NOTICE file
> + distributed with this work for additional information
> + regarding copyright ownership. The ASF licenses this file
> + to you under the Apache License, Version 2.0 (the
> + "License"); you may not use this file except in compliance
> + with the License. You may obtain a copy of the License at
> +
> + http://www.apache.org/licenses/LICENSE-2.0
> +
> + Unless required by applicable law or agreed to in writing,
> + software distributed under the License is distributed on an
> + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
> + KIND, either express or implied. See the License for the
> + specific language governing permissions and limitations
> + under the License.
> +
> +.. include:: common.defs
> +
> +*********
> +Ram Cache
> +*********
> +
> +New Ram Cache Algorithm (CLFUS)
> +===============================
> +
> +The new Ram Cache uses ideas from a number of cache replacement policies and
> algorithms, including LRU, LFU, CLOCK, GDFS and 2Q, called CLFUS (Clocked
> Least Frequently Used by Size). It avoids any patented algorithms and
> includes the following features:
> +
for easier reading, please break at max 120 char.
> +* Balances Recentness, Frequency and Size to maximize hit rate (not byte hit
> rate).
> +* Is Scan Resistant and extracts robust hit rates even when the working set
> does not fit in the Ram Cache.
> +* Supports compression at 3 levels fastlz, gzip(libz), and xz(liblzma).
> Compression can be moved to another thread.
> +* Has very low CPU overhead, only little more than a basic LRU. Rather than
> using an O(lg n) heap, it uses a probabilistic replacement policy for O(1)
> cost with low C.
What's a very low C?
> +* Has relatively low memory overhead of approximately 200 bytes per object
> in memory.
> +
> +The rational for emphasizing hit rate over byte hit rate is that the
> overhead of pulling more bytes from secondary storage is low compared to the
> cost of a request.
> +
> +The Ram Cache consists of an object hash fronting 2 LRU/CLOCK lists and a
> "Seen" hash table. The first "Cached" list contains objects in memory while
> the second contains a "History" of objects which have either recently been
> in memory or are being considered for keeping in memory. The "Seen" hash
> table is used to make the algorithm scan resistant.
> +
> +The list entries record the following information:
> +
> +* key - 16 byte unique object identifier
> +* auxkeys - 8 bytes worth of version number (in our system the block in the
> partition). When the version of an object changes old entries are purged
> from the cache.
> +* hits - number of hits within this clock period
> +* size - the size of the object in the cache
> +* len - the actual length of the object (differs from size because of
> compression and padding)
> +* compressed_len - the compressed length of the object
> +* compressed (none, fastlz, libz, liblzma)
> +* uncompressible (flag)
> +* copy - whether or not this object should be copied in and copied out (e.g.
> HTTP HDR)
> +* LRU link
> +* HASH link
> +* IOBufferData (smart point to the data buffer)
> +
> +
> +The interface to the cache is Get and Put operations. Get operations check
are those different from GET and PUT?
> if an object is in the cache and are called on a read attempt. The Put
> operation decides whether or not to cache the provided object in memory. It
> is called after a read from secondary storage.
> +
> +Seen Hash
> +=========
> +
> +The Seen List becomes active after the Cached and History lists become full
> after a cold start. The purpose is to make the cache scan resistant which
> means that the cache state must not be effected at all by a long sequence
> Get and Put operations on objects which are seen only once. This is
> essential, without it not only would the cache be polluted, but it could
> lose critical information about the objects that it cares about. It is
> therefore essential that the Cache and History lists are not effected by Get
> or Put operations on objects seen the first time. The Seen Hash maintains a
> set of 16 bit hash tags, and requests which do not hit in the object cache
> (are in the Cache List or History List) and do not match the hash tag result
> in the hash tag begin updated but are otherwise ignored. The Seen Hash is
> sized to approximately the number of objects in the cache in order to match
> the number that are passed through it with the CLOCK rate of the Cached and
> History Lists.
> +
> +Cached List
> +===========
> +
> +The Cached list contains objects actually in memory. The basic operation is
> LRU with new entries inserted into a FIFO (queue) and hits causing objects
> to be reinserted. The interesting bit comes when an object is being
> considered for insertion. First we check if the Object Hash to see if the
> object is in the Cached List or History. Hits result in updating the "hit"
> field and reinsertion. History hits result in the "hit" field being updated
> and a comparison to see if this object should be kept in memory. The
> comparison is against the least recently used members of the Cache List, and
> is based on a weighted frequency::
> +
> + CACHE_VALUE = hits / (size + overhead)
> +
> +A new object must beat enough bytes worth of currently cached objects to
> cover itself. Each time an object is considered for replacement the CLOCK
> moves forward. If the History object has a greater value then it is
> inserted into the Cached List and the replaced objects are removed from
> memory and their list entries are inserted into the History List. If the
> History object has a lesser value it is reinserted into the History List.
> Objects considered for replacement (at least one) but not replaced have
> their "hits" field set to zero and are reinserted into the Cached List.
> This is the CLOCK operation on the Cached List.
> +
> +History List
> +============
> +
> +Each CLOCK the least recently used entry in the History List is dequeued and
> if the "hits" field is not greater than 1 (it was hit at least once in the
> History or Cached List) it is deleted, otherwise the "hits" is set to zero
> and it is requeued on the History List.
> +
> +Compression/Decompression
> +=========================
> +
> +Compression is performed by a background operation (currently called as part
> of Put) which maintains a pointer into the Cached List and runs toward the
> head compressing entries. Decompression occurs on demand during a Get. In
> the case of objects tagged "copy" the compressed version is reinserted in
> the LRU since we need to make a copy anyway. Those not tagged "copy" are
> inserted uncompressed in the hope that they can be reused in uncompressed
> form. This is a compile time option and may be something we want to change.
> +
> +There are 3 algorithms and levels of compression (speed on 1 thread i7 920)
> :
> +
> +* fastlz: 173 MB/sec compression, 442 MB/sec decompression : basically free
> since disk or network will limit first, ~53% final size
> +* libz: 55 MB/sec compression, 234 MB/sec decompression : almost free,
> particularly decompression, ~37% final size
> +* liblzma: 3 MB/sec compression, 50 MB/sec decompression : expensive, ~27%
> final size
> +
> +These are ballpark numbers, and your millage will vary enormously. JPEG for
mileage
> example will not compress with any of these. The RamCache does detect
> compression level and will declare something "incompressible" if it doesn't
> get below 90% of the original size. This value is cached so that the
> RamCache will not attempt to compress it again (at least as long as it is in
> the history).
> +
General: all of the CAPSLOCK words should be put in the glossary
--
Igor Galić
Tel: +43 (0) 664 886 22 883
Mail: i.galic@brainsware.org
URL: http://brainsware.org/
GPG: 8716 7A9F 989B ABD5 100F 4008 F266 55D6 2998 1641
[05/12] git commit: [TS-2183] Make a restructured text version of the
ram-cache wiki page
Posted by ml...@apache.org.
[TS-2183] Make a restructured text version of the ram-cache wiki page
Project: http://git-wip-us.apache.org/repos/asf/trafficserver/repo
Commit: http://git-wip-us.apache.org/repos/asf/trafficserver/commit/219ba692
Tree: http://git-wip-us.apache.org/repos/asf/trafficserver/tree/219ba692
Diff: http://git-wip-us.apache.org/repos/asf/trafficserver/diff/219ba692
Branch: refs/heads/master
Commit: 219ba692eaf6e85fe99c41efdd176a9f2ed18c4b
Parents: 0f45c06
Author: Miles Libbey <ml...@apache.org>
Authored: Fri Dec 13 13:52:15 2013 -0800
Committer: Miles Libbey <ml...@apache.org>
Committed: Fri Dec 13 13:52:15 2013 -0800
----------------------------------------------------------------------
doc/arch/cache/ram-cache.en.rst | 88 ++++++++++++++++++++++++++++++++++++
1 file changed, 88 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/trafficserver/blob/219ba692/doc/arch/cache/ram-cache.en.rst
----------------------------------------------------------------------
diff --git a/doc/arch/cache/ram-cache.en.rst b/doc/arch/cache/ram-cache.en.rst
new file mode 100644
index 0000000..b0b15e1
--- /dev/null
+++ b/doc/arch/cache/ram-cache.en.rst
@@ -0,0 +1,88 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. include:: common.defs
+
+*********
+Ram Cache
+*********
+
+New Ram Cache Algorithm (CLFUS)
+===============================
+
+The new Ram Cache uses ideas from a number of cache replacement policies and algorithms, including LRU, LFU, CLOCK, GDFS and 2Q, called CLFUS (Clocked Least Frequently Used by Size). It avoids any patented algorithms and includes the following features:
+
+* Balances Recentness, Frequency and Size to maximize hit rate (not byte hit rate).
+* Is Scan Resistant and extracts robust hit rates even when the working set does not fit in the Ram Cache.
+* Supports compression at 3 levels fastlz, gzip(libz), and xz(liblzma). Compression can be moved to another thread.
+* Has very low CPU overhead, only little more than a basic LRU. Rather than using an O(lg n) heap, it uses a probabilistic replacement policy for O(1) cost with low C.
+* Has relatively low memory overhead of approximately 200 bytes per object in memory.
+
+The rational for emphasizing hit rate over byte hit rate is that the overhead of pulling more bytes from secondary storage is low compared to the cost of a request.
+
+The Ram Cache consists of an object hash fronting 2 LRU/CLOCK lists and a "Seen" hash table. The first "Cached" list contains objects in memory while the second contains a "History" of objects which have either recently been in memory or are being considered for keeping in memory. The "Seen" hash table is used to make the algorithm scan resistant.
+
+The list entries record the following information:
+
+* key - 16 byte unique object identifier
+* auxkeys - 8 bytes worth of version number (in our system the block in the partition). When the version of an object changes old entries are purged from the cache.
+* hits - number of hits within this clock period
+* size - the size of the object in the cache
+* len - the actual length of the object (differs from size because of compression and padding)
+* compressed_len - the compressed length of the object
+* compressed (none, fastlz, libz, liblzma)
+* uncompressible (flag)
+* copy - whether or not this object should be copied in and copied out (e.g. HTTP HDR)
+* LRU link
+* HASH link
+* IOBufferData (smart point to the data buffer)
+
+
+The interface to the cache is Get and Put operations. Get operations check if an object is in the cache and are called on a read attempt. The Put operation decides whether or not to cache the provided object in memory. It is called after a read from secondary storage.
+
+Seen Hash
+=========
+
+The Seen List becomes active after the Cached and History lists become full after a cold start. The purpose is to make the cache scan resistant which means that the cache state must not be effected at all by a long sequence Get and Put operations on objects which are seen only once. This is essential, without it not only would the cache be polluted, but it could lose critical information about the objects that it cares about. It is therefore essential that the Cache and History lists are not effected by Get or Put operations on objects seen the first time. The Seen Hash maintains a set of 16 bit hash tags, and requests which do not hit in the object cache (are in the Cache List or History List) and do not match the hash tag result in the hash tag begin updated but are otherwise ignored. The Seen Hash is sized to approximately the number of objects in the cache in order to match the number that are passed through it with the CLOCK rate of the Cached and History Lists.
+
+Cached List
+===========
+
+The Cached list contains objects actually in memory. The basic operation is LRU with new entries inserted into a FIFO (queue) and hits causing objects to be reinserted. The interesting bit comes when an object is being considered for insertion. First we check if the Object Hash to see if the object is in the Cached List or History. Hits result in updating the "hit" field and reinsertion. History hits result in the "hit" field being updated and a comparison to see if this object should be kept in memory. The comparison is against the least recently used members of the Cache List, and is based on a weighted frequency::
+
+ CACHE_VALUE = hits / (size + overhead)
+
+A new object must beat enough bytes worth of currently cached objects to cover itself. Each time an object is considered for replacement the CLOCK moves forward. If the History object has a greater value then it is inserted into the Cached List and the replaced objects are removed from memory and their list entries are inserted into the History List. If the History object has a lesser value it is reinserted into the History List. Objects considered for replacement (at least one) but not replaced have their "hits" field set to zero and are reinserted into the Cached List. This is the CLOCK operation on the Cached List.
+
+History List
+============
+
+Each CLOCK the least recently used entry in the History List is dequeued and if the "hits" field is not greater than 1 (it was hit at least once in the History or Cached List) it is deleted, otherwise the "hits" is set to zero and it is requeued on the History List.
+
+Compression/Decompression
+=========================
+
+Compression is performed by a background operation (currently called as part of Put) which maintains a pointer into the Cached List and runs toward the head compressing entries. Decompression occurs on demand during a Get. In the case of objects tagged "copy" the compressed version is reinserted in the LRU since we need to make a copy anyway. Those not tagged "copy" are inserted uncompressed in the hope that they can be reused in uncompressed form. This is a compile time option and may be something we want to change.
+
+There are 3 algorithms and levels of compression (speed on 1 thread i7 920) :
+
+* fastlz: 173 MB/sec compression, 442 MB/sec decompression : basically free since disk or network will limit first, ~53% final size
+* libz: 55 MB/sec compression, 234 MB/sec decompression : almost free, particularly decompression, ~37% final size
+* liblzma: 3 MB/sec compression, 50 MB/sec decompression : expensive, ~27% final size
+
+These are ballpark numbers, and your millage will vary enormously. JPEG for example will not compress with any of these. The RamCache does detect compression level and will declare something "incompressible" if it doesn't get below 90% of the original size. This value is cached so that the RamCache will not attempt to compress it again (at least as long as it is in the history).
+
[02/12] git commit: [TS-145] Clarify cache.config items in docs.
Posted by ml...@apache.org.
[TS-145] Clarify cache.config items in docs.
Project: http://git-wip-us.apache.org/repos/asf/trafficserver/repo
Commit: http://git-wip-us.apache.org/repos/asf/trafficserver/commit/8e02b12c
Tree: http://git-wip-us.apache.org/repos/asf/trafficserver/tree/8e02b12c
Diff: http://git-wip-us.apache.org/repos/asf/trafficserver/diff/8e02b12c
Branch: refs/heads/master
Commit: 8e02b12c204c1ab9647b3e4505d3aa9da9159def
Parents: 5871b0a
Author: Miles Libbey <ml...@apache.org>
Authored: Fri Dec 13 10:57:03 2013 -0800
Committer: Miles Libbey <ml...@apache.org>
Committed: Fri Dec 13 10:57:03 2013 -0800
----------------------------------------------------------------------
doc/reference/configuration/cache.config.en.rst | 37 ++++++++++++++++----
1 file changed, 31 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/trafficserver/blob/8e02b12c/doc/reference/configuration/cache.config.en.rst
----------------------------------------------------------------------
diff --git a/doc/reference/configuration/cache.config.en.rst b/doc/reference/configuration/cache.config.en.rst
index 0a3de13..e9a691e 100644
--- a/doc/reference/configuration/cache.config.en.rst
+++ b/doc/reference/configuration/cache.config.en.rst
@@ -127,11 +127,27 @@ The following list shows possible actions and their allowed values.
- ``cluster-cache-local`` configures the cluster cache to allow for
this content to be stored locally on every cluster node.
+.. _cache-responses-to-cookies:
+``cache-responses-to-cookies``
+ Change the style of caching with regard to cookies. This effectively
+ overrides the configuration parameter
+ :ref:`proxy.config.http.cache.cache_responses_to_cookies`
+ and uses the same values with the same semantics. The override happens
+ only for requests that match.
+
.. _cache-config-format-pin-in-cache:
``pin-in-cache``
- The amount of time you want to keep the object(s) in the cache. The
+ Preserves objects in cache, preventing them from being overwritten.
+ Does not affect objects that are determined not to be cacheable. This
+ setting can have performance issues, and severely affect the cache.
+ For instance, if the primary destination matches all objects, once the
+ cache is full, no new objects could get written as nothing would be
+ evicted. Similarly, for each cache-miss, each object would incur extra
+ checks to determine if the object it would replace could be overwritten.
+
+ The value is the amount of time you want to keep the object(s) in the cache. The
following time formats are allowed:
- ``d`` for days; for example: 2d
@@ -143,15 +159,16 @@ The following list shows possible actions and their allowed values.
.. _cache-config-format-revalidate:
``revalidate``
- The amount of time object(s) are to be considered fresh. Use the
- same time formats as ``pin-in-cache``.
+ For objects that are in cache, overrides the the amount of time the object(s)
+ are to be considered fresh. Use the same time formats as ``pin-in-cache``.
.. _cache-config-format-ttl-in-cache:
``ttl-in-cache``
- The amount of time object(s) are to be kept in the cache, regardless
- of Cache-Control response headers. Use the same time formats as
- pin-in-cache and revalidate .
+ Forces object(s) to become cached, as if they had a Cache-Control: max-age:<time>
+ header. Can be overruled by requests with cookies. The value is the amount of
+ time object(s) are to be kept in the cache, regardless of Cache-Control response
+ headers. Use the same time formats as pin-in-cache and revalidate.
Examples
========
@@ -165,3 +182,11 @@ in the order listed. ::
dest_domain=mydomain.com suffix=jpeg revalidate=6h
dest_domain=mydomain.com revalidate=1h
+Force a specific regex to be in cache between 7-11pm of the server's time for 26hours. ::
+
+ url_regex=example.com/articles/popular.* time=19:00-23:00 ttl-in-cache=1d2h
+
+Prevent objects from being evicted from cache:
+
+ url_regex=example.com/game/.* pin-in-cache=1h
+
[07/12] git commit: [TS-446] Add
proxy.config.cache.target_fragment_size based on jira notes.
Posted by ml...@apache.org.
[TS-446] Add proxy.config.cache.target_fragment_size based on jira notes.
Project: http://git-wip-us.apache.org/repos/asf/trafficserver/repo
Commit: http://git-wip-us.apache.org/repos/asf/trafficserver/commit/1b814a78
Tree: http://git-wip-us.apache.org/repos/asf/trafficserver/tree/1b814a78
Diff: http://git-wip-us.apache.org/repos/asf/trafficserver/diff/1b814a78
Branch: refs/heads/master
Commit: 1b814a785c4c60914361595fb0ac21b9fc7ab4af
Parents: 5c6c53b
Author: Miles Libbey <ml...@apache.org>
Authored: Mon Dec 16 13:46:19 2013 -0800
Committer: Miles Libbey <ml...@apache.org>
Committed: Mon Dec 16 13:48:26 2013 -0800
----------------------------------------------------------------------
doc/reference/configuration/records.config.en.rst | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/trafficserver/blob/1b814a78/doc/reference/configuration/records.config.en.rst
----------------------------------------------------------------------
diff --git a/doc/reference/configuration/records.config.en.rst b/doc/reference/configuration/records.config.en.rst
index c3c7b80..fd662c5 100644
--- a/doc/reference/configuration/records.config.en.rst
+++ b/doc/reference/configuration/records.config.en.rst
@@ -1200,7 +1200,13 @@ Cache Control
Disable by setting to 0. Note that this setting will not strictly enforce
this if the variable ``proxy.config.cache.vary_on_user_agent`` is set
to 1 (by default it is 0).
-
+
+.. ts:cv:: CONFIG proxy.config.cache.target_fragment_size INT 1048576
+
+ Sets the target size of a contiguous fragment of a file in the disk cache. Accepts values that are powers of 2, e.g. 65536, 131072,
+ 262144, 524288, 1048576, 2097152, etc. When setting this, consider that larger numbers could waste memory on slow connections,
+ but smaller numbers could increase (waste) seeks.
+
RAM Cache
=========
[03/12] git commit: [TS-2430] Indicate that extra cache options in
cache.config; clarify ram cache algorithm a bit
Posted by ml...@apache.org.
[TS-2430] Indicate that extra cache options in cache.config; clarify ram cache algorithm a bit
Project: http://git-wip-us.apache.org/repos/asf/trafficserver/repo
Commit: http://git-wip-us.apache.org/repos/asf/trafficserver/commit/b5494fd7
Tree: http://git-wip-us.apache.org/repos/asf/trafficserver/tree/b5494fd7
Diff: http://git-wip-us.apache.org/repos/asf/trafficserver/diff/b5494fd7
Branch: refs/heads/master
Commit: b5494fd7b099c9bb12304b4321f09098492828d4
Parents: 8e02b12
Author: Miles Libbey <ml...@apache.org>
Authored: Fri Dec 13 10:59:54 2013 -0800
Committer: Miles Libbey <ml...@apache.org>
Committed: Fri Dec 13 10:59:54 2013 -0800
----------------------------------------------------------------------
doc/admin/configuring-cache.en.rst | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/trafficserver/blob/b5494fd7/doc/admin/configuring-cache.en.rst
----------------------------------------------------------------------
diff --git a/doc/admin/configuring-cache.en.rst b/doc/admin/configuring-cache.en.rst
index 3c6d1c4..7847fad 100644
--- a/doc/admin/configuring-cache.en.rst
+++ b/doc/admin/configuring-cache.en.rst
@@ -55,6 +55,10 @@ You can perform the following cache configuration tasks:
- Partition the cache by reserving cache disk space for specific
protocols and origin servers/domains: refer to `Partitioning the Cache`_.
- Delete all data in the cache: refer to `Clearing the Cache`_.
+- Override cache directives for a requested domain name, regex on a url,
+ hostname or ip, with extra filters for time, port, method of the request
+ (and more). ATS can be configured to never cache; always cache;
+ ignore no-cache directives, etc. These are configured in :file:`cache.config`.
The RAM Cache
=============
@@ -67,8 +71,10 @@ You can configure the RAM cache size to suit your needs, as described in
The RAM cache supports two cache eviction algorithms, a regular **LRU**
(*Least Recently Used*) and the more advanced **CLFUS** (*Clocked Least
-Frequently Used by Size*). The default is to use **CLFUS**, and this is
-controlled via :ts:cv:`proxy.config.cache.ram_cache.algorithm`.
+Frequently Used by Size*, which balances recentness, frequency and size
+to maximize hit rate -- similar to a most frequently used algorithm).
+The default is to use **CLFUS**, and this is controlled via
+:ts:cv:`proxy.config.cache.ram_cache.algorithm`.
Both the **LRU** and **CLFUS** RAM caches support a configuration to increase
scan resistance. In a typical **LRU**, if you request all possible objects in
Re: [08/12] git commit: [TS-428] Add
proxy.config.dns.validate_query_name to drecords.config doc from Jira
notes.
Posted by Igor Galić <i....@brainsware.org>.
----- Original Message -----
> [TS-428] Add proxy.config.dns.validate_query_name to drecords.config doc from
> Jira notes.
>
>
> Project: http://git-wip-us.apache.org/repos/asf/trafficserver/repo
> Commit: http://git-wip-us.apache.org/repos/asf/trafficserver/commit/44a86148
> Tree: http://git-wip-us.apache.org/repos/asf/trafficserver/tree/44a86148
> Diff: http://git-wip-us.apache.org/repos/asf/trafficserver/diff/44a86148
>
> Branch: refs/heads/master
> Commit: 44a8614853f2d622861aa3ee434b1eb9fe255bb7
> Parents: 1b814a7
> Author: Miles Libbey <ml...@apache.org>
> Authored: Mon Dec 16 14:12:48 2013 -0800
> Committer: Miles Libbey <ml...@apache.org>
> Committed: Mon Dec 16 14:12:48 2013 -0800
>
> ----------------------------------------------------------------------
> doc/reference/configuration/records.config.en.rst | 6 ++++++
> 1 file changed, 6 insertions(+)
> ----------------------------------------------------------------------
>
>
> http://git-wip-us.apache.org/repos/asf/trafficserver/blob/44a86148/doc/reference/configuration/records.config.en.rst
> ----------------------------------------------------------------------
> diff --git a/doc/reference/configuration/records.config.en.rst
> b/doc/reference/configuration/records.config.en.rst
> index fd662c5..353b8e6 100644
> --- a/doc/reference/configuration/records.config.en.rst
> +++ b/doc/reference/configuration/records.config.en.rst
> @@ -1403,6 +1403,12 @@ hostname to ``host_x.y.com``.
> contention on the first worker thread (which otherwise takes on the
> burden of
> all DNS lookups).
>
> +.. ts:cv:: CONFIG proxy.config.dns.validate_query_name INT 0
> +
> + When enabled (1) provides additional resilience against DNS forgery (for
> instance
> + in DNS Injection attacks), particularly in forward or transparent
> proxies, but
> + requires that the resolver populates the queries section of the response
> properly.
What does that mean?
* who/what is the resolver? (we? HostDB? The system? something else)
* what is the queries section
* what qualifies as "properly"
> +
> HostDB
> ======
>
>
>
--
Igor Galić
Tel: +43 (0) 664 886 22 883
Mail: i.galic@brainsware.org
URL: http://brainsware.org/
GPG: 8716 7A9F 989B ABD5 100F 4008 F266 55D6 2998 1641
[08/12] git commit: [TS-428] Add proxy.config.dns.validate_query_name
to drecords.config doc from Jira notes.
Posted by ml...@apache.org.
[TS-428] Add proxy.config.dns.validate_query_name to drecords.config doc from Jira notes.
Project: http://git-wip-us.apache.org/repos/asf/trafficserver/repo
Commit: http://git-wip-us.apache.org/repos/asf/trafficserver/commit/44a86148
Tree: http://git-wip-us.apache.org/repos/asf/trafficserver/tree/44a86148
Diff: http://git-wip-us.apache.org/repos/asf/trafficserver/diff/44a86148
Branch: refs/heads/master
Commit: 44a8614853f2d622861aa3ee434b1eb9fe255bb7
Parents: 1b814a7
Author: Miles Libbey <ml...@apache.org>
Authored: Mon Dec 16 14:12:48 2013 -0800
Committer: Miles Libbey <ml...@apache.org>
Committed: Mon Dec 16 14:12:48 2013 -0800
----------------------------------------------------------------------
doc/reference/configuration/records.config.en.rst | 6 ++++++
1 file changed, 6 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/trafficserver/blob/44a86148/doc/reference/configuration/records.config.en.rst
----------------------------------------------------------------------
diff --git a/doc/reference/configuration/records.config.en.rst b/doc/reference/configuration/records.config.en.rst
index fd662c5..353b8e6 100644
--- a/doc/reference/configuration/records.config.en.rst
+++ b/doc/reference/configuration/records.config.en.rst
@@ -1403,6 +1403,12 @@ hostname to ``host_x.y.com``.
contention on the first worker thread (which otherwise takes on the burden of
all DNS lookups).
+.. ts:cv:: CONFIG proxy.config.dns.validate_query_name INT 0
+
+ When enabled (1) provides additional resilience against DNS forgery (for instance
+ in DNS Injection attacks), particularly in forward or transparent proxies, but
+ requires that the resolver populates the queries section of the response properly.
+
HostDB
======
[06/12] git commit: [TS-2389] Add back
proxy.config.http.uncacheable_requests_bypass_parent and a few others from
the undocumented section.
Posted by ml...@apache.org.
[TS-2389] Add back proxy.config.http.uncacheable_requests_bypass_parent and a few others from the undocumented section.
Project: http://git-wip-us.apache.org/repos/asf/trafficserver/repo
Commit: http://git-wip-us.apache.org/repos/asf/trafficserver/commit/5c6c53bd
Tree: http://git-wip-us.apache.org/repos/asf/trafficserver/tree/5c6c53bd
Diff: http://git-wip-us.apache.org/repos/asf/trafficserver/diff/5c6c53bd
Branch: refs/heads/master
Commit: 5c6c53bda1aa86908e9d873b63c50705595923cf
Parents: 219ba69
Author: Miles Libbey <ml...@apache.org>
Authored: Mon Dec 16 13:27:02 2013 -0800
Committer: Miles Libbey <ml...@apache.org>
Committed: Mon Dec 16 13:27:02 2013 -0800
----------------------------------------------------------------------
.../configuration/records.config.en.rst | 22 +++++++++++++++++---
1 file changed, 19 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/trafficserver/blob/5c6c53bd/doc/reference/configuration/records.config.en.rst
----------------------------------------------------------------------
diff --git a/doc/reference/configuration/records.config.en.rst b/doc/reference/configuration/records.config.en.rst
index 1f45a7a..c3c7b80 100644
--- a/doc/reference/configuration/records.config.en.rst
+++ b/doc/reference/configuration/records.config.en.rst
@@ -720,6 +720,9 @@ Parent Proxy Configuration
Don't try to resolve DNS, forward all DNS requests to the parent. This is off (``0``) by default.
+
+
+
HTTP Connection Timeouts
========================
@@ -839,6 +842,10 @@ Origin Server Connect Attempts
The number of seconds before Traffic Server marks an origin server as unavailable after a client abandons a request
because the origin server was too slow in sending the response header.
+.. ts:cv:: CONFIG proxy.config.http.uncacheable_requests_bypass_parent INT 1
+
+ When enabled (1), Traffic Server bypasses the parent proxy for a request that is not cacheable.
+
Congestion Control
==================
@@ -896,6 +903,11 @@ Negative Response Caching
The cache lifetime for objects cached from this setting is controlled via
:ts:cv:`proxy.config.http.negative_caching_lifetime`.
+.. ts:cv:: CONFIG proxy.config.http.negative_caching_lifetime INT 1800
+
+ How long (in seconds) Traffic Server keeps the negative responses valid in cache. This value only affects negative
+ responses that do have explicit ``Expires:`` or ``Cache-Control:`` lifetimes set by the server.
+
Proxy User Variables
====================
@@ -1182,6 +1194,13 @@ Cache Control
Objects larger than the limit are not hit evacuated. A value of 0 disables the limit.
+.. ts:cv:: CONFIG proxy.config.cache.limits.http.max_alts INT 5
+
+ The maximum number of alternates that are allowed for any given URL.
+ Disable by setting to 0. Note that this setting will not strictly enforce
+ this if the variable ``proxy.config.cache.vary_on_user_agent`` is set
+ to 1 (by default it is 0).
+
RAM Cache
=========
@@ -2193,10 +2212,7 @@ Undocumented
These are referenced but not documented. Please contribute a definition.
-.. ts:cv:: CONFIG proxy.config.http.negative_caching_lifetime INT 0
.. ts:cv:: CONFIG proxy.config.task_threads INT 0
-.. ts:cv:: CONFIG proxy.config.cache.limits.http.max_alts INT 5
-
.. ts:cv:: CONFIG proxy.config.http.enabled INT 1
Re: [04/12] git commit: [TS-2329] Add a restructured text version
of the header rewrite's README
Posted by Igor Galić <i....@brainsware.org>.
----- Original Message -----
> [TS-2329] Add a restructured text version of the header rewrite's README
>
>
> Project: http://git-wip-us.apache.org/repos/asf/trafficserver/repo
> Commit: http://git-wip-us.apache.org/repos/asf/trafficserver/commit/0f45c068
> Tree: http://git-wip-us.apache.org/repos/asf/trafficserver/tree/0f45c068
> Diff: http://git-wip-us.apache.org/repos/asf/trafficserver/diff/0f45c068
>
> Branch: refs/heads/master
> Commit: 0f45c0684164c74ffe73e430e080ec62193dc001
> Parents: b5494fd
> Author: Miles Libbey <ml...@apache.org>
> Authored: Fri Dec 13 13:31:41 2013 -0800
> Committer: Miles Libbey <ml...@apache.org>
> Committed: Fri Dec 13 13:31:41 2013 -0800
>
> ----------------------------------------------------------------------
> doc/reference/plugins/header_rewrite.en.rst | 174 +++++++++++++++++++++++
> 1 file changed, 174 insertions(+)
> ----------------------------------------------------------------------
>
>
> http://git-wip-us.apache.org/repos/asf/trafficserver/blob/0f45c068/doc/reference/plugins/header_rewrite.en.rst
> ----------------------------------------------------------------------
> diff --git a/doc/reference/plugins/header_rewrite.en.rst
> b/doc/reference/plugins/header_rewrite.en.rst
> new file mode 100644
> index 0000000..2069d26
> --- /dev/null
> +++ b/doc/reference/plugins/header_rewrite.en.rst
> @@ -0,0 +1,174 @@
> +.. _header_rewrite-plugin
> +
> +Header Rewrite Plugin
> +*********************
> +
> +.. Licensed to the Apache Software Foundation (ASF) under one
> + or more contributor license agreements. See the NOTICE file
> + distributed with this work for additional information
> + regarding copyright ownership. The ASF licenses this file
> + to you under the Apache License, Version 2.0 (the
> + "License"); you may not use this file except in compliance
> + with the License. You may obtain a copy of the License at
> +
> + http://www.apache.org/licenses/LICENSE-2.0
> +
> + Unless required by applicable law or agreed to in writing,
> + software distributed under the License is distributed on an
> + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
> + KIND, either express or implied. See the License for the
> + specific language governing permissions and limitations
> + under the License.
> +
> +
> +This is a plugin for Apache Traffic Server that allows you to
> +modify various headers based on defined rules (operations) on a request or
> +response. Currently, only one operation is supported.
a couple of lines down this file when we describe operators the
statement that only one operation is supported seems a bit shaky ;)
> +Using the plugin
> +----------------
> +
> +This plugin can be used as both a global plugin, enabled in plugin.config::
> +
> + header_rewrite.so config_file_1.conf config_file_2.conf ...
> +
> +These are global rules, which would apply to all requests. Another option is
> +to use per remap rules in remap.config::
> +
> + map http://a http://b @plugin=header_rewrite.so @pparam=rules1.conf ...
> +
> +In the second example, hooks which are not to be executed during the remap
> +phase (the default) causes a transaction hook to be instantiated and used
> +at a later time. This allows you to setup e.g. a rule that gets executed
> +during the origin response header parsing, using READ_RESPONSE_HDR_HOOK.
> +
> +Operators
> +---------
> +
> +The following operators are available::
> +
> + rm-header header-name [flags]
> + add-header header <value> [flags]
> + set-status <status-code> [flags]
> + set-status-reason <value> [flags]
> + set-config config <value> [flags]
> + no-op [flags]
> + counter counter-name [flags]
> +
> +The following operator(s) currently only works when instantiating the
> +plugin as a remap plugin::
> +
> + set-destination [qual] value
> +
> +Where qual is one of the support URL qualifiers::
> +
> + HOST
> + PORT
> + PATH
> + QUERY
> +
> +For example (as a remap rule)::
> +
> + cond %{HEADER:X-Mobile} = "foo"
> + set-destination HOST foo.mobile.bar.com [L]
> +
> +Operator flags
> +--------------
> +::
> +
> + [L] Last rule, do not continue
> +
> +Variable expansion
> +------------------
> +Currently only limited variable expansion is supported in add-header.
> Supported
> +substitutions include::
> +
> + %<proto> Protocol
> + %<port> Port
> + %<chi> Client IP
> + %<cqhl> Client request length
> + %<cqhm> Client HTTP method
> + %<cquup> Client unmapped URI
> +
> +Conditions
> +----------
> +The conditions are used as qualifiers: The operators specified will
> +only be evaluated if the condition(s) are met::
> +
> + cond %{STATUS} operand [flags]
> + cond %{RANDOM:nn} operand [flags]
> + cond %{ACCESS:file} [flags]
> + cond %{TRUE} [flags]
> + cond %{FALSE} [flags]
> + cond %{HEADER:header-name} operand [flags]
> + cond %{COOKIE:cookie-name} operand [flags]
> + cond %{CLIENT-HEADER:header-name} operand [flags]
> + cond %{PROTOCOL} operand [flags]
> + cond %{PORT} operand [flags]
> + cond %{HOST} operand [flags]
> + cond %{TOHOST} operand [false]
> + cond %{FROMHOST} operand [false]
> + cond %{PATH} operand [false]
> + cond %{PARAMS} operand [false]
> + cond %{QUERY} operand [false]
> +
> +The difference between HEADER and CLIENT-HEADER is that HEADER adapts to the
> +hook it's running in, whereas CLIENT-HEADER always applies to the client
> +request header. The %{TRUE} condition is also the default condition if no
> +other conditions are specified.
> +
> +These conditions have to be first in a ruleset, and you can only have one in
> +each rule. This implies that a new hook condition starts a new rule as
> well.::
> +
> + cond %{READ_RESPONSE_HDR_HOOK} (this is the default hook)
> + cond %{READ_REQUEST_HDR_HOOK}
> + cond %{SEND_REQUEST_HDR_HOOK}
> + cond %{SEND_RESPONSE_HDR_HOOK}
> +
> +For remap.config plugin instanations, the default hook is named
> +REMAP_PSEUDO_HOOK. This can be useful if you are mixing other hooks in a
> +configuration, but being the default it is also optional.
> +
> +---------------
> +Condition flags
> +---------------
> +::
> +
> + [NC] Not case sensitive condition (when applicable)
> + [AND] AND with next condition (default)
> + [OR] OR with next condition
> + [NOT] Invert this condition
> +
> +Operands to conditions
> +----------------------
> +::
> +
> + /string/ # regular expression
> + <string # lexically lower
> + >string # lexically greater
> + =string # lexically equal
> +
> +The absence of a "matcher" means value exists).
> +
> +Values
> +------
> +Setting e.g. a header with a value can take the following formats:
> +
> +- Any of the cond definitions, that extracts a value from the request
> +- $N 0 <= N <= 9, as grouped in a regular expression
> +- string (which can contain the above)
> +- null
> +
> +Examples
> +--------
> +::
> +
> + cond %{HEADER:X-Y-Foobar}
> + cond %{COOKIE:X-DC} =DC1
> + add-header X-Y-Fiefum %{HEADER:X-Y-Foobar}
> + add-header X-Forwarded-For %<chi>
> + rm-header X-Y-Foobar
> + rm-header Set-Cookie
> + counter plugin.header_rewrite.x-y-foobar-dc1
> + cond %{HEADER:X-Y-Foobar} "Some string" [AND,NC]
> +
>
>
--
Igor Galić
Tel: +43 (0) 664 886 22 883
Mail: i.galic@brainsware.org
URL: http://brainsware.org/
GPG: 8716 7A9F 989B ABD5 100F 4008 F266 55D6 2998 1641
Re: [04/12] git commit: [TS-2329] Add a restructured text version
of the header rewrite's README
Posted by Igor Galić <i....@brainsware.org>.
----- Original Message -----
> [TS-2329] Add a restructured text version of the header rewrite's README
>
>
> Project: http://git-wip-us.apache.org/repos/asf/trafficserver/repo
> Commit: http://git-wip-us.apache.org/repos/asf/trafficserver/commit/0f45c068
> Tree: http://git-wip-us.apache.org/repos/asf/trafficserver/tree/0f45c068
> Diff: http://git-wip-us.apache.org/repos/asf/trafficserver/diff/0f45c068
>
> Branch: refs/heads/master
> Commit: 0f45c0684164c74ffe73e430e080ec62193dc001
> Parents: b5494fd
> Author: Miles Libbey <ml...@apache.org>
> Authored: Fri Dec 13 13:31:41 2013 -0800
> Committer: Miles Libbey <ml...@apache.org>
> Committed: Fri Dec 13 13:31:41 2013 -0800
>
> ----------------------------------------------------------------------
> doc/reference/plugins/header_rewrite.en.rst | 174 +++++++++++++++++++++++
> 1 file changed, 174 insertions(+)
> ----------------------------------------------------------------------
>
>
> http://git-wip-us.apache.org/repos/asf/trafficserver/blob/0f45c068/doc/reference/plugins/header_rewrite.en.rst
> ----------------------------------------------------------------------
> diff --git a/doc/reference/plugins/header_rewrite.en.rst
> b/doc/reference/plugins/header_rewrite.en.rst
> new file mode 100644
> index 0000000..2069d26
> --- /dev/null
> +++ b/doc/reference/plugins/header_rewrite.en.rst
> @@ -0,0 +1,174 @@
> +.. _header_rewrite-plugin
> +
> +Header Rewrite Plugin
> +*********************
> +
> +.. Licensed to the Apache Software Foundation (ASF) under one
> + or more contributor license agreements. See the NOTICE file
> + distributed with this work for additional information
> + regarding copyright ownership. The ASF licenses this file
> + to you under the Apache License, Version 2.0 (the
> + "License"); you may not use this file except in compliance
> + with the License. You may obtain a copy of the License at
> +
> + http://www.apache.org/licenses/LICENSE-2.0
> +
> + Unless required by applicable law or agreed to in writing,
> + software distributed under the License is distributed on an
> + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
> + KIND, either express or implied. See the License for the
> + specific language governing permissions and limitations
> + under the License.
> +
> +
> +This is a plugin for Apache Traffic Server that allows you to
> +modify various headers based on defined rules (operations) on a request or
> +response. Currently, only one operation is supported.
> +
> +Using the plugin
> +----------------
> +
> +This plugin can be used as both a global plugin, enabled in plugin.config::
> +
> + header_rewrite.so config_file_1.conf config_file_2.conf ...
> +
> +These are global rules, which would apply to all requests. Another option is
> +to use per remap rules in remap.config::
> +
> + map http://a http://b @plugin=header_rewrite.so @pparam=rules1.conf ...
> +
> +In the second example, hooks which are not to be executed during the remap
> +phase (the default) causes a transaction hook to be instantiated and used
> +at a later time. This allows you to setup e.g. a rule that gets executed
> +during the origin response header parsing, using READ_RESPONSE_HDR_HOOK.
> +
> +Operators
> +---------
> +
> +The following operators are available::
> +
> + rm-header header-name [flags]
> + add-header header <value> [flags]
> + set-status <status-code> [flags]
> + set-status-reason <value> [flags]
> + set-config config <value> [flags]
> + no-op [flags]
> + counter counter-name [flags]
> +
> +The following operator(s) currently only works when instantiating the
> +plugin as a remap plugin::
> +
> + set-destination [qual] value
> +
> +Where qual is one of the support URL qualifiers::
> +
> + HOST
> + PORT
> + PATH
> + QUERY
> +
> +For example (as a remap rule)::
> +
> + cond %{HEADER:X-Mobile} = "foo"
> + set-destination HOST foo.mobile.bar.com [L]
> +
> +Operator flags
> +--------------
> +::
> +
> + [L] Last rule, do not continue
> +
> +Variable expansion
> +------------------
> +Currently only limited variable expansion is supported in add-header.
> Supported
> +substitutions include::
> +
> + %<proto> Protocol
> + %<port> Port
> + %<chi> Client IP
> + %<cqhl> Client request length
> + %<cqhm> Client HTTP method
> + %<cquup> Client unmapped URI
> +
> +Conditions
> +----------
> +The conditions are used as qualifiers: The operators specified will
> +only be evaluated if the condition(s) are met::
> +
> + cond %{STATUS} operand [flags]
> + cond %{RANDOM:nn} operand [flags]
> + cond %{ACCESS:file} [flags]
> + cond %{TRUE} [flags]
> + cond %{FALSE} [flags]
> + cond %{HEADER:header-name} operand [flags]
> + cond %{COOKIE:cookie-name} operand [flags]
> + cond %{CLIENT-HEADER:header-name} operand [flags]
> + cond %{PROTOCOL} operand [flags]
> + cond %{PORT} operand [flags]
> + cond %{HOST} operand [flags]
> + cond %{TOHOST} operand [false]
> + cond %{FROMHOST} operand [false]
> + cond %{PATH} operand [false]
> + cond %{PARAMS} operand [false]
> + cond %{QUERY} operand [false]
> +
> +The difference between HEADER and CLIENT-HEADER is that HEADER adapts to the
> +hook it's running in, whereas CLIENT-HEADER always applies to the client
> +request header. The %{TRUE} condition is also the default condition if no
> +other conditions are specified.
> +
> +These conditions have to be first in a ruleset, and you can only have one in
> +each rule. This implies that a new hook condition starts a new rule as
> well.::
> +
> + cond %{READ_RESPONSE_HDR_HOOK} (this is the default hook)
> + cond %{READ_REQUEST_HDR_HOOK}
> + cond %{SEND_REQUEST_HDR_HOOK}
> + cond %{SEND_RESPONSE_HDR_HOOK}
> +
> +For remap.config plugin instanations, the default hook is named
> +REMAP_PSEUDO_HOOK. This can be useful if you are mixing other hooks in a
we should link to remap.config and the HOOK here.
> +configuration, but being the default it is also optional.
> +
> +---------------
> +Condition flags
> +---------------
> +::
> +
> + [NC] Not case sensitive condition (when applicable)
> + [AND] AND with next condition (default)
> + [OR] OR with next condition
> + [NOT] Invert this condition
> +
> +Operands to conditions
> +----------------------
> +::
> +
> + /string/ # regular expression
> + <string # lexically lower
> + >string # lexically greater
> + =string # lexically equal
> +
> +The absence of a "matcher" means value exists).
> +
> +Values
> +------
> +Setting e.g. a header with a value can take the following formats:
> +
> +- Any of the cond definitions, that extracts a value from the request
> +- $N 0 <= N <= 9, as grouped in a regular expression
> +- string (which can contain the above)
> +- null
> +
> +Examples
> +--------
> +::
> +
> + cond %{HEADER:X-Y-Foobar}
> + cond %{COOKIE:X-DC} =DC1
> + add-header X-Y-Fiefum %{HEADER:X-Y-Foobar}
> + add-header X-Forwarded-For %<chi>
> + rm-header X-Y-Foobar
> + rm-header Set-Cookie
> + counter plugin.header_rewrite.x-y-foobar-dc1
> + cond %{HEADER:X-Y-Foobar} "Some string" [AND,NC]
> +
>
>
--
Igor Galić
Tel: +43 (0) 664 886 22 883
Mail: i.galic@brainsware.org
URL: http://brainsware.org/
GPG: 8716 7A9F 989B ABD5 100F 4008 F266 55D6 2998 1641
[04/12] git commit: [TS-2329] Add a restructured text version of the
header rewrite's README
Posted by ml...@apache.org.
[TS-2329] Add a restructured text version of the header rewrite's README
Project: http://git-wip-us.apache.org/repos/asf/trafficserver/repo
Commit: http://git-wip-us.apache.org/repos/asf/trafficserver/commit/0f45c068
Tree: http://git-wip-us.apache.org/repos/asf/trafficserver/tree/0f45c068
Diff: http://git-wip-us.apache.org/repos/asf/trafficserver/diff/0f45c068
Branch: refs/heads/master
Commit: 0f45c0684164c74ffe73e430e080ec62193dc001
Parents: b5494fd
Author: Miles Libbey <ml...@apache.org>
Authored: Fri Dec 13 13:31:41 2013 -0800
Committer: Miles Libbey <ml...@apache.org>
Committed: Fri Dec 13 13:31:41 2013 -0800
----------------------------------------------------------------------
doc/reference/plugins/header_rewrite.en.rst | 174 +++++++++++++++++++++++
1 file changed, 174 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/trafficserver/blob/0f45c068/doc/reference/plugins/header_rewrite.en.rst
----------------------------------------------------------------------
diff --git a/doc/reference/plugins/header_rewrite.en.rst b/doc/reference/plugins/header_rewrite.en.rst
new file mode 100644
index 0000000..2069d26
--- /dev/null
+++ b/doc/reference/plugins/header_rewrite.en.rst
@@ -0,0 +1,174 @@
+.. _header_rewrite-plugin
+
+Header Rewrite Plugin
+*********************
+
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+This is a plugin for Apache Traffic Server that allows you to
+modify various headers based on defined rules (operations) on a request or
+response. Currently, only one operation is supported.
+
+Using the plugin
+----------------
+
+This plugin can be used as both a global plugin, enabled in plugin.config::
+
+ header_rewrite.so config_file_1.conf config_file_2.conf ...
+
+These are global rules, which would apply to all requests. Another option is
+to use per remap rules in remap.config::
+
+ map http://a http://b @plugin=header_rewrite.so @pparam=rules1.conf ...
+
+In the second example, hooks which are not to be executed during the remap
+phase (the default) causes a transaction hook to be instantiated and used
+at a later time. This allows you to setup e.g. a rule that gets executed
+during the origin response header parsing, using READ_RESPONSE_HDR_HOOK.
+
+Operators
+---------
+
+The following operators are available::
+
+ rm-header header-name [flags]
+ add-header header <value> [flags]
+ set-status <status-code> [flags]
+ set-status-reason <value> [flags]
+ set-config config <value> [flags]
+ no-op [flags]
+ counter counter-name [flags]
+
+The following operator(s) currently only works when instantiating the
+plugin as a remap plugin::
+
+ set-destination [qual] value
+
+Where qual is one of the support URL qualifiers::
+
+ HOST
+ PORT
+ PATH
+ QUERY
+
+For example (as a remap rule)::
+
+ cond %{HEADER:X-Mobile} = "foo"
+ set-destination HOST foo.mobile.bar.com [L]
+
+Operator flags
+--------------
+::
+
+ [L] Last rule, do not continue
+
+Variable expansion
+------------------
+Currently only limited variable expansion is supported in add-header. Supported
+substitutions include::
+
+ %<proto> Protocol
+ %<port> Port
+ %<chi> Client IP
+ %<cqhl> Client request length
+ %<cqhm> Client HTTP method
+ %<cquup> Client unmapped URI
+
+Conditions
+----------
+The conditions are used as qualifiers: The operators specified will
+only be evaluated if the condition(s) are met::
+
+ cond %{STATUS} operand [flags]
+ cond %{RANDOM:nn} operand [flags]
+ cond %{ACCESS:file} [flags]
+ cond %{TRUE} [flags]
+ cond %{FALSE} [flags]
+ cond %{HEADER:header-name} operand [flags]
+ cond %{COOKIE:cookie-name} operand [flags]
+ cond %{CLIENT-HEADER:header-name} operand [flags]
+ cond %{PROTOCOL} operand [flags]
+ cond %{PORT} operand [flags]
+ cond %{HOST} operand [flags]
+ cond %{TOHOST} operand [false]
+ cond %{FROMHOST} operand [false]
+ cond %{PATH} operand [false]
+ cond %{PARAMS} operand [false]
+ cond %{QUERY} operand [false]
+
+The difference between HEADER and CLIENT-HEADER is that HEADER adapts to the
+hook it's running in, whereas CLIENT-HEADER always applies to the client
+request header. The %{TRUE} condition is also the default condition if no
+other conditions are specified.
+
+These conditions have to be first in a ruleset, and you can only have one in
+each rule. This implies that a new hook condition starts a new rule as well.::
+
+ cond %{READ_RESPONSE_HDR_HOOK} (this is the default hook)
+ cond %{READ_REQUEST_HDR_HOOK}
+ cond %{SEND_REQUEST_HDR_HOOK}
+ cond %{SEND_RESPONSE_HDR_HOOK}
+
+For remap.config plugin instanations, the default hook is named
+REMAP_PSEUDO_HOOK. This can be useful if you are mixing other hooks in a
+configuration, but being the default it is also optional.
+
+---------------
+Condition flags
+---------------
+::
+
+ [NC] Not case sensitive condition (when applicable)
+ [AND] AND with next condition (default)
+ [OR] OR with next condition
+ [NOT] Invert this condition
+
+Operands to conditions
+----------------------
+::
+
+ /string/ # regular expression
+ <string # lexically lower
+ >string # lexically greater
+ =string # lexically equal
+
+The absence of a "matcher" means value exists).
+
+Values
+------
+Setting e.g. a header with a value can take the following formats:
+
+- Any of the cond definitions, that extracts a value from the request
+- $N 0 <= N <= 9, as grouped in a regular expression
+- string (which can contain the above)
+- null
+
+Examples
+--------
+::
+
+ cond %{HEADER:X-Y-Foobar}
+ cond %{COOKIE:X-DC} =DC1
+ add-header X-Y-Fiefum %{HEADER:X-Y-Foobar}
+ add-header X-Forwarded-For %<chi>
+ rm-header X-Y-Foobar
+ rm-header Set-Cookie
+ counter plugin.header_rewrite.x-y-foobar-dc1
+ cond %{HEADER:X-Y-Foobar} "Some string" [AND,NC]
+