document update for hipes and buffer_upload plugins
Project: http://git-wip-us.apache.org/repos/asf/trafficserver/repo
Commit: http://git-wip-us.apache.org/repos/asf/trafficserver/commit/b36a7e12
Tree: http://git-wip-us.apache.org/repos/asf/trafficserver/tree/b36a7e12
Diff: http://git-wip-us.apache.org/repos/asf/trafficserver/diff/b36a7e12
Branch: refs/heads/5.0.x
Commit: b36a7e120e4b9c216fe2f13a97015a508911e0a4
Parents: 8d8507f
Author: Kit Chan <ki...@apache.org>
Authored: Sat Dec 28 09:06:04 2013 -0800
Committer: Kit Chan <ki...@apache.org>
Committed: Sat Dec 28 09:06:04 2013 -0800
----------------------------------------------------------------------
doc/reference/plugins/buffer_upload.en.rst | 93 +++++++++++--------------
doc/reference/plugins/hipes.en.rst | 22 ++++--
2 files changed, 58 insertions(+), 57 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/trafficserver/blob/b36a7e12/doc/reference/plugins/buffer_upload.en.rst
----------------------------------------------------------------------
diff --git a/doc/reference/plugins/buffer_upload.en.rst b/doc/reference/plugins/buffer_upload.en.rst
index ab5f209..022d61e 100644
--- a/doc/reference/plugins/buffer_upload.en.rst
+++ b/doc/reference/plugins/buffer_upload.en.rst
@@ -18,72 +18,63 @@ Buffer Upload Plugin
specific language governing permissions and limitations
under the License.
+The Buffer Upload plugin offers the following features
-.. XXX Discribe what the heck this plugin actually does.
+Installation
+============
-Upload proxy specs for phase I:
+Configuration can be explicitly specified as a parameter in ``plugin.config`` ::
-1. Memory buffering (buffer the entire POST data in IOBuffer before
- connecting to OS) 1.1. Memory buffer size is configured with
- "mem_buffer_size" in config file. Default and minimum value is 32K
- You can increase it in the config file. If the size of a request is
- larger than the "mem_buffer_size" value specifiied in the config
- file, then the upload proxy feature will be disabled for this
- particular request
+ buffer_upload.so /FOOBAR/upload.conf
-2. Disk buffering (buffer the entire POST data on disk before connecting
- to OS) 2.1. Use disk async IO. This involved some changes in ATS core
- . new APIs wrapping around ink_aio_read() and ink_aio_write() .
- change to distinguish between api call's AIO and cache's AIO .
- guarantee api call's AIO only involves certain amount of threads .
- the number of threads is configurable in plugin's config file
- (default is 4)
+Memory buffering (buffer the entire POST data in IOBuffer before connecting to OS)
+==================================================================================
-3.
+Memory buffer size is configured with "mem_buffer_size" in config file. Default and minimum value is 32K. You can
+increase it in the config file. If the size of a request is larger than the "mem_buffer_size" value specifiied in the
+config file, then the upload proxy feature will be disabled for this particular request
- 2. Directories and files generated on disk . base directory:
- FOOBAR/var/buffer_upload_tmp/ (configurable in config file) .
- number of subdirectories: 64 (configurable in config file) .
- filename are randomly generated . files will be removed when the
- entire data have been sent out to OS . remove dangling files (left
- on disk due to transaction interruption or traffic server crash)
- at startup time
+Disk buffering (buffer the entire POST data on disk before connecting to OS)
+============================================================================
-4.
+1. Disk async IO is used. AIO api call only involves certain amount of threads. The number of threads is configurable in
+plugin's config file (default is 4)
- 3. Default chunk size when reading from disk: 16K, configurable in
- config file
+2. Directories and files are generated on disk . Base directory is /FOOBAR/var/buffer_upload_tmp/ (configurable in
+config file). Number of subdirectories is 64 (configurable in config file). Filename are randomly generated. Files will
+be removed when the entire data have been sent out to OS . At startup time, dangling files are removed (left on disk due
+to transaction interruption or traffic server crash)
-5. Default buffering mode: disk aio buffering mode 3.1. to turn off disk
- buffering, add a "use_disk_buffer 0" line in config file
+3. Default chunk size when reading from disk is 16K, configurable in config file
-6. Trigger POST buffering on certain URLs 4.1. certain URLs will be
- provided in a plain text file (one URL each line) 4.2. specify
- filename in config file by "url_list_file" 4.3. max length of each
- URL: 4096 (configurable in config file) 4.4. use exact match, don't
- support regex for now
+Trigger POST buffering on certain URLs
+======================================
-7. URL conversion for Mail's specific URL format 5.1. for now check if
- the "host" part in the URL is same as the proxy server name, then
- will do this conversion 5.2. To turn on URL conversion feature, set
- "convert_url 1" in config file
+1. Certain URLs will be provided in a plain text file (one URL each line)
+2. Specify filename in config file by "url_list_file"
+3. max length of each URL is 4096 (configurable in config file)
+4. use exact match, don't support regex for now
-8. All request headers inlcuding cookies plus the entire POST data will
- be buffered (either in memory or on disk)
+Other Features
+==============
-9. Config file can be explicitly sepcified as a parameter in command
- line (in plugin.config file)
+1. Default buffering mode is disk aio buffering mode. To turn off disk buffering, add a "use_disk_buffer 0" line in
+config file
-a sample config file:
+2. All request headers inlcuding cookies plus the entire POST data will be buffered (either in memory or on disk)
-use_disk_buffer 1 convert_url 1 chunk_size 1024 url_list_file
-/tmp/url_list.conf max_url_length 10000 base_dir /tmp/test1
-subdir_num 100 thread_num 10 mem_buffer_size 40000
+Configuration File
+==================
-default config file: FOOBAR/etc/upload.conf
+sample config file ::
-default config values: use_disk_buffer 1 convert_url 0 chunk_size
-16384 url_list_file none max_url_length 4096 base_dir
-FOOBAR/var/buffer_upload_tmp subdir_num 64 thread_num 4
-mem_buffer_size 32768
+ use_disk_buffer 1
+ convert_url 1
+ chunk_size 1024
+ url_list_file /tmp/url_list.conf
+ max_url_length 10000
+ base_dir /tmp/test1
+ subdir_num 100
+ thread_num 10
+ mem_buffer_size 40000
http://git-wip-us.apache.org/repos/asf/trafficserver/blob/b36a7e12/doc/reference/plugins/hipes.en.rst
----------------------------------------------------------------------
diff --git a/doc/reference/plugins/hipes.en.rst b/doc/reference/plugins/hipes.en.rst
index e74a58a..20c847b 100644
--- a/doc/reference/plugins/hipes.en.rst
+++ b/doc/reference/plugins/hipes.en.rst
@@ -62,7 +62,8 @@ only apply to the connection to the specific "service" host.
Notes on HIPES
==============
-HTTP Pipes (aka HIPES and pronounced "Hippies") allows data services to be pipelined together, as illustrated by the example below.
+HTTP Pipes (aka HIPES and pronounced "Hippies") allows data services to be pipelined together, as illustrated by the
+example below.
Example
=======
@@ -90,25 +91,34 @@ to this URL ::
http://127.0.0.1:8080/svc_case.php?case=u&url=http%3A%2F%2F127.0.0.1%2Ftest.txt
-5. In this example, the service ``svc_case.php`` fetches and transforms the response of ``http://127.0.0.1/test.txt`` (which ATS proxies the request to a local file) to upper case. And subsequently the service ``svc_reverse.php`` receives the response and reverse the order before the response is sent back to the client by ATS.
+5. In this example, the service ``svc_case.php`` fetches and transforms the response of ``http://127.0.0.1/test.txt``
+(which ATS proxies the request to a local file) to upper case. And subsequently the service ``svc_reverse.php`` receives
+the response and reverse the order before the response is sent back to the client by ATS.
Notes on reducing traffic
=========================
-There can be significant overhead using HIPES because the data can traverse through ATS many times. Caching can be important to reduce traffic to services/through ATS and can be achieved via a proper ``Cache-Control`` header returned by the services. Another way to reduce traffic through ATS is to have ATS to return 302 redirects to url for the requests made by service, instead of proxying the requests to that url. However, the service must then be able to follow the redirect. The down side is that we cannot use ATS to cache intermediate results. Below is an example of using redirect.
+There can be significant overhead using HIPES because the data can traverse through ATS many times. Caching can be
+important to reduce traffic to services/through ATS and can be achieved via a proper ``Cache-Control`` header returned
+by the services. Another way to reduce traffic through ATS is to have ATS to return 302 redirects to url for the
+requests made by service, instead of proxying the requests to that url. However, the service must then be able to follow
+the redirect. The down side is that we cannot use ATS to cache intermediate results. Below is an example of using
+redirect.
Modification to above example to reduce traffic using redirect
==============================================================
-1. The service ``svc_reverse.php`` is modified to add a header of ``X-HIPES-Redirect: 2`` to the request made against ``url``.
+1. The service ``svc_reverse.php`` is modified to add a header of ``X-HIPES-Redirect: 2`` to the request made against
+``url``.
2. HIPES plugin will instruct ATS to return a redirect response to this url ::
http://127.0.0.1:8080/svc_case.php?case=u&url=http%3A%2F%2F127.0.0.1%2Ftest.txt
-for the following request
+for the following request ::
http://127.0.0.1/svc_case;case=u/test.txt
-3. The service ``svc_reverse.php`` is also modified to follow the redirect. Thus the response of the service of ``svc_case.php`` will not pass through ATS and will pass to ``svc_reverse.php`` service instead.
+3. The service ``svc_reverse.php`` is also modified to follow the redirect. Thus the response of the service of
+``svc_case.php`` will not pass through ATS and will pass to ``svc_reverse.php`` service instead.