You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by bu...@apache.org on 2014/12/17 08:19:49 UTC
svn commit: r933087 [1/2] - in /websites/production/camel/content:
book-component-appendix.html book-in-one-page.html cache/main.pageCache
download.html hdfs.html hdfs2.html
Author: buildbot
Date: Wed Dec 17 07:19:49 2014
New Revision: 933087
Log:
Production update by buildbot for camel
Modified:
websites/production/camel/content/book-component-appendix.html
websites/production/camel/content/book-in-one-page.html
websites/production/camel/content/cache/main.pageCache
websites/production/camel/content/download.html
websites/production/camel/content/hdfs.html
websites/production/camel/content/hdfs2.html
Modified: websites/production/camel/content/book-component-appendix.html
==============================================================================
--- websites/production/camel/content/book-component-appendix.html (original)
+++ websites/production/camel/content/book-component-appendix.html Wed Dec 17 07:19:49 2014
@@ -1415,11 +1415,11 @@ template.send("direct:alias-verify&
</div>
</div>
<p>The <strong>cxf:</strong> component provides integration with <a shape="rect" href="http://cxf.apache.org">Apache CXF</a> for connecting to JAX-WS services hosted in CXF.</p><p><style type="text/css">/*<![CDATA[*/
-div.rbtoc1418714315607 {padding: 0px;}
-div.rbtoc1418714315607 ul {list-style: disc;margin-left: 0px;}
-div.rbtoc1418714315607 li {margin-left: 0px;padding-left: 0px;}
+div.rbtoc1418800713901 {padding: 0px;}
+div.rbtoc1418800713901 ul {list-style: disc;margin-left: 0px;}
+div.rbtoc1418800713901 li {margin-left: 0px;padding-left: 0px;}
-/*]]>*/</style></p><div class="toc-macro rbtoc1418714315607">
+/*]]>*/</style></p><div class="toc-macro rbtoc1418800713901">
<ul class="toc-indentation"><li><a shape="rect" href="#CXF-CXFComponent">CXF Component</a>
<ul class="toc-indentation"><li><a shape="rect" href="#CXF-URIformat">URI format</a></li><li><a shape="rect" href="#CXF-Options">Options</a>
<ul class="toc-indentation"><li><a shape="rect" href="#CXF-Thedescriptionsofthedataformats">The descriptions of the dataformats</a>
@@ -4827,9 +4827,13 @@ template.sendBodyAndHeader("direct:
</div></div><h3 id="BookComponentAppendix-URIformat.24">URI format</h3><div class="code panel pdl" style="border-width: 1px;"><div class="codeContent panelContent pdl">
<script class="theme: Default; brush: java; gutter: false" type="syntaxhighlighter"><![CDATA[hdfs://hostname[:port][/path][?options]
]]></script>
-</div></div><p>You can append query options to the URI in the following format, <code>?option=value&option=value&...</code><br clear="none"> The path is treated in the following way:</p><ol><li>as a consumer, if it's a file, it just reads the file, otherwise if it represents a directory it scans all the file under the path satisfying the configured pattern. All the files under that directory must be of the same type.</li><li>as a producer, if at least one split strategy is defined, the path is considered a directory and under that directory the producer creates a different file per split named using the configured <a shape="rect" href="uuidgenerator.html">UuidGenerator</a>.</li></ol><h3 id="BookComponentAppendix-Options.19">Options</h3><div class="confluenceTableSmall">
-<div class="table-wrap"><table class="confluenceTable"><tbody><tr><th colspan="1" rowspan="1" class="confluenceTh"><p> Name </p></th><th colspan="1" rowspan="1" class="confluenceTh"><p> Default Value </p></th><th colspan="1" rowspan="1" class="confluenceTh"><p> Description </p></th></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>overwrite</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>true</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> The file can be overwritten </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>append</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>false</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> Append to existing file. Notice that not all HDFS file systems support the append option. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>bufferSize</code> </p></td><td colspan="1" rowspan="1" cla
ss="confluenceTd"><p> <code>4096</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> The buffer size used by HDFS </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>replication</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>3</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> The HDFS replication factor </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>blockSize</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>67108864</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> The size of the HDFS blocks </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>fileType</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>NORMAL_FILE</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> It can be SEQUENCE_FILE, MAP_FILE, ARRAY_FILE, or BLOOMMAP_FILE, see Hadoop </p></td></tr>
<tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>fileSystemType</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>HDFS</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> It can be LOCAL for local filesystem </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>keyType</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>NULL</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> The type for the key in case of sequence or map files. See below. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>valueType</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>TEXT</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> The type for the key in case of sequence or map files. See below. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>splitStrategy</code> </p></td><td colspan="1" rowsp
an="1" class="confluenceTd"><p> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> A string describing the strategy on how to split the file based on different criteria. See below. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>openedSuffix</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>opened</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> When a file is opened for reading/writing the file is renamed with this suffix to avoid to read it during the writing phase. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>readSuffix</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>read</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> Once the file has been read is renamed with this suffix to avoid to read it again. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>initialDelay</code> </p></td><td co
lspan="1" rowspan="1" class="confluenceTd"><p> <code>0</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> For the consumer, how much to wait (milliseconds) before to start scanning the directory. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>delay</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>0</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> The interval (milliseconds) between the directory scans. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>pattern</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>*</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> The pattern used for scanning the directory </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>chunkSize</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>4096</code> </p></td><td colspan="1" rowspan="1" class="co
nfluenceTd"><p> When reading a normal file, this is split into chunks producing a message per chunk. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>connectOnStartup</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>true</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <strong>Camel 2.9.3/2.10.1:</strong> Whether to connect to the HDFS file system on starting the producer/consumer. If <code>false</code> then the connection is created on-demand. Notice that HDFS may take up till 15 minutes to establish a connection, as it has hardcoded 45 x 20 sec redelivery. By setting this option to <code>false</code> allows your application to startup, and not block for up till 15 minutes. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>owner</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <strong>Camel 2.13/2
.12.4:</strong> The file owner must match this owner for the consumer to pickup the file. Otherwise the file is skipped. </p></td></tr></tbody></table></div>
-</div><h4 id="BookComponentAppendix-KeyTypeandValueType">KeyType and ValueType</h4><ul><li>NULL it means that the key or the value is absent</li><li>BYTE for writing a byte, the java Byte class is mapped into a BYTE</li><li>BYTES for writing a sequence of bytes. It maps the java ByteBuffer class</li><li>INT for writing java integer</li><li>FLOAT for writing java float</li><li>LONG for writing java long</li><li>DOUBLE for writing java double</li><li>TEXT for writing java strings</li></ul><p>BYTES is also used with everything else, for example, in Camel a file is sent around as an InputStream, int this case is written in a sequence file or a map file as a sequence of bytes.</p><h3 id="BookComponentAppendix-SplittingStrategy">Splitting Strategy</h3><p>In the current version of Hadoop opening a file in append mode is disabled since it's not very reliable. So, for the moment, it's only possible to create new files. The Camel HDFS endpoint tries to solve this problem in this way:</p><ul><
li>If the split strategy option has been defined, the hdfs path will be used as a directory and files will be created using the configured <a shape="rect" href="uuidgenerator.html">UuidGenerator</a></li><li>Every time a splitting condition is met, a new file is created.<br clear="none"> The splitStrategy option is defined as a string with the following syntax:<br clear="none"> splitStrategy=<ST>:<value>,<ST>:<value>,*</li></ul><p>where <ST> can be:</p><ul><li>BYTES a new file is created, and the old is closed when the number of written bytes is more than <value></li><li>MESSAGES a new file is created, and the old is closed when the number of written messages is more than <value></li><li>IDLE a new file is created, and the old is closed when no writing happened in the last <value> milliseconds</li></ul> <div class="aui-message warning shadowed information-macro">
+</div></div><p>You can append query options to the URI in the following format, <code>?option=value&option=value&...</code><br clear="none"> The path is treated in the following way:</p><ol><li>as a consumer, if it's a file, it just reads the file, otherwise if it represents a directory it scans all the file under the path satisfying the configured pattern. All the files under that directory must be of the same type.</li><li>as a producer, if at least one split strategy is defined, the path is considered a directory and under that directory the producer creates a different file per split named using the configured <a shape="rect" href="uuidgenerator.html">UuidGenerator</a>.</li></ol> <div class="aui-message warning shadowed information-macro">
+ <span class="aui-icon icon-warning">Icon</span>
+ <div class="message-content">
+ <p><span>When consuming from hdfs then in normal mode, a file is split into chunks, producing a message per chunk. You can configure the size of the chunk using the chunkSize option. If you want to read from hdfs and write to a regular file using the file component, then you can use the fileMode=Append to append each of the chunks together.</span></p>
+ </div>
+ </div>
+<p> </p><h3 id="BookComponentAppendix-Options.19">Options</h3><div class="confluenceTableSmall"><div class="table-wrap"><table class="confluenceTable"><tbody><tr><th colspan="1" rowspan="1" class="confluenceTh"><p>Name</p></th><th colspan="1" rowspan="1" class="confluenceTh"><p>Default Value</p></th><th colspan="1" rowspan="1" class="confluenceTh"><p>Description</p></th></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>overwrite</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>true</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>The file can be overwritten</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>append</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>false</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>Append to existing file. Notice that not all HDFS file systems support the append option.</p></td></tr><tr><td colspan="1" rowspan="1" class=
"confluenceTd"><p><code>bufferSize</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>4096</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>The buffer size used by HDFS</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>replication</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>3</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>The HDFS replication factor</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>blockSize</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>67108864</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>The size of the HDFS blocks</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>fileType</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>NORMAL_FILE</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>It can be SEQUENCE_FILE, MAP_FILE
, ARRAY_FILE, or BLOOMMAP_FILE, see Hadoop</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>fileSystemType</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>HDFS</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>It can be LOCAL for local filesystem</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>keyType</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>NULL</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>The type for the key in case of sequence or map files. See below.</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>valueType</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>TEXT</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>The type for the key in case of sequence or map files. See below.</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>splitStrategy</co
de></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>A string describing the strategy on how to split the file based on different criteria. See below.</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>openedSuffix</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>opened</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>When a file is opened for reading/writing the file is renamed with this suffix to avoid to read it during the writing phase.</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>readSuffix</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>read</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>Once the file has been read is renamed with this suffix to avoid to read it again.</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>initialDelay</code>
</p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>0</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>For the consumer, how much to wait (milliseconds) before to start scanning the directory.</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>delay</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>0</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>The interval (milliseconds) between the directory scans.</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>pattern</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>*</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>The pattern used for scanning the directory</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>chunkSize</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>4096</code></p></td><td colspan="1" rowspan="1" class="confluenc
eTd"><p>When reading a normal file, this is split into chunks producing a message per chunk.</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>connectOnStartup</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>true</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><strong>Camel 2.9.3/2.10.1:</strong> Whether to connect to the HDFS file system on starting the producer/consumer. If <code>false</code> then the connection is created on-demand. Notice that HDFS may take up till 15 minutes to establish a connection, as it has hardcoded 45 x 20 sec redelivery. By setting this option to <code>false</code> allows your application to startup, and not block for up till 15 minutes.</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>owner</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><strong>Camel 2.13/2.12.4:</strong> Th
e file owner must match this owner for the consumer to pickup the file. Otherwise the file is skipped.</p></td></tr></tbody></table></div></div><h4 id="BookComponentAppendix-KeyTypeandValueType">KeyType and ValueType</h4><ul><li>NULL it means that the key or the value is absent</li><li>BYTE for writing a byte, the java Byte class is mapped into a BYTE</li><li>BYTES for writing a sequence of bytes. It maps the java ByteBuffer class</li><li>INT for writing java integer</li><li>FLOAT for writing java float</li><li>LONG for writing java long</li><li>DOUBLE for writing java double</li><li>TEXT for writing java strings</li></ul><p>BYTES is also used with everything else, for example, in Camel a file is sent around as an InputStream, int this case is written in a sequence file or a map file as a sequence of bytes.</p><h3 id="BookComponentAppendix-SplittingStrategy">Splitting Strategy</h3><p>In the current version of Hadoop opening a file in append mode is disabled since it's not very relia
ble. So, for the moment, it's only possible to create new files. The Camel HDFS endpoint tries to solve this problem in this way:</p><ul><li>If the split strategy option has been defined, the hdfs path will be used as a directory and files will be created using the configured <a shape="rect" href="uuidgenerator.html">UuidGenerator</a></li><li>Every time a splitting condition is met, a new file is created.<br clear="none"> The splitStrategy option is defined as a string with the following syntax:<br clear="none"> splitStrategy=<ST>:<value>,<ST>:<value>,*</li></ul><p>where <ST> can be:</p><ul><li>BYTES a new file is created, and the old is closed when the number of written bytes is more than <value></li><li>MESSAGES a new file is created, and the old is closed when the number of written messages is more than <value></li><li>IDLE a new file is created, and the old is closed when no writing happened in the last <value> milliseconds</li></u
l> <div class="aui-message warning shadowed information-macro">
<span class="aui-icon icon-warning">Icon</span>
<div class="message-content">
<p>note that this strategy currently requires either setting an IDLE value or setting the HdfsConstants.HDFS_CLOSE header to false to use the BYTES/MESSAGES configuration...otherwise, the file will be closed with each message</p>
@@ -4838,9 +4842,7 @@ template.sendBodyAndHeader("direct:
<p>for example:</p><div class="code panel pdl" style="border-width: 1px;"><div class="codeContent panelContent pdl">
<script class="theme: Default; brush: java; gutter: false" type="syntaxhighlighter"><![CDATA[hdfs://localhost/tmp/simple-file?splitStrategy=IDLE:1000,BYTES:5
]]></script>
-</div></div><p>it means: a new file is created either when it has been idle for more than 1 second or if more than 5 bytes have been written. So, running <code>hadoop fs -ls /tmp/simple-file</code> you'll see that multiple files have been created.</p><h3 id="BookComponentAppendix-MessageHeaders.4">Message Headers</h3><p>The following headers are supported by this component:</p><h4 id="BookComponentAppendix-Produceronly">Producer only</h4><div class="confluenceTableSmall">
-<div class="table-wrap"><table class="confluenceTable"><tbody><tr><th colspan="1" rowspan="1" class="confluenceTh"><p> Header </p></th><th colspan="1" rowspan="1" class="confluenceTh"><p> Description </p></th></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>CamelFileName</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <strong>Camel 2.13:</strong> Specifies the name of the file to write (relative to the endpoint path). The name can be a <code>String</code> or an <a shape="rect" href="expression.html">Expression</a> object. Only relevant when not using a split strategy. </p></td></tr></tbody></table></div>
-</div><h3 id="BookComponentAppendix-Controllingtoclosefilestream">Controlling to close file stream</h3><p><strong>Available as of Camel 2.10.4</strong></p><p>When using the <a shape="rect" href="hdfs.html">HDFS</a> producer <strong>without</strong> a split strategy, then the file output stream is by default closed after the write. However you may want to keep the stream open, and only explicitly close the stream later. For that you can use the header <code>HdfsConstants.HDFS_CLOSE</code> (value = <code>"CamelHdfsClose"</code>) to control this. Setting this value to a boolean allows you to explicit control whether the stream should be closed or not.</p><p>Notice this does not apply if you use a split strategy, as there are various strategies that can control when the stream is closed.</p><h3 id="BookComponentAppendix-UsingthiscomponentinOSGi">Using this component in OSGi</h3><p>This component is fully functional in an OSGi environment, however, it requires some actions from the user.
Hadoop uses the thread context class loader in order to load resources. Usually, the thread context classloader will be the bundle class loader of the bundle that contains the routes. So, the default configuration files need to be visible from the bundle class loader. A typical way to deal with it is to keep a copy of core-default.xml in your bundle root. That file can be found in the hadoop-common.jar.</p>
+</div></div><p>it means: a new file is created either when it has been idle for more than 1 second or if more than 5 bytes have been written. So, running <code>hadoop fs -ls /tmp/simple-file</code> you'll see that multiple files have been created.</p><h3 id="BookComponentAppendix-MessageHeaders.4">Message Headers</h3><p>The following headers are supported by this component:</p><h4 id="BookComponentAppendix-Produceronly">Producer only</h4><div class="confluenceTableSmall"><div class="table-wrap"><table class="confluenceTable"><tbody><tr><th colspan="1" rowspan="1" class="confluenceTh"><p>Header</p></th><th colspan="1" rowspan="1" class="confluenceTh"><p>Description</p></th></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>CamelFileName</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><strong>Camel 2.13:</strong> Specifies the name of the file to write (relative to the endpoint path). The name can be a <code>String</code> or an <a shape="rect" href="
expression.html">Expression</a> object. Only relevant when not using a split strategy.</p></td></tr></tbody></table></div></div><h3 id="BookComponentAppendix-Controllingtoclosefilestream">Controlling to close file stream</h3><p><strong>Available as of Camel 2.10.4</strong></p><p>When using the <a shape="rect" href="hdfs.html">HDFS</a> producer <strong>without</strong> a split strategy, then the file output stream is by default closed after the write. However you may want to keep the stream open, and only explicitly close the stream later. For that you can use the header <code>HdfsConstants.HDFS_CLOSE</code> (value = <code>"CamelHdfsClose"</code>) to control this. Setting this value to a boolean allows you to explicit control whether the stream should be closed or not.</p><p>Notice this does not apply if you use a split strategy, as there are various strategies that can control when the stream is closed.</p><h3 id="BookComponentAppendix-UsingthiscomponentinOSGi">Using this component
in OSGi</h3><p>This component is fully functional in an OSGi environment, however, it requires some actions from the user. Hadoop uses the thread context class loader in order to load resources. Usually, the thread context classloader will be the bundle class loader of the bundle that contains the routes. So, the default configuration files need to be visible from the bundle class loader. A typical way to deal with it is to keep a copy of core-default.xml in your bundle root. That file can be found in the hadoop-common.jar.</p>
<h2 id="BookComponentAppendix-HibernateComponent">Hibernate Component</h2>
<p>The <strong>hibernate:</strong> component allows you to work with databases using Hibernate as the object relational mapping technology to map POJOs to database tables. The <strong>camel-hibernate</strong> library is provided by the <a shape="rect" class="external-link" href="http://code.google.com/p/camel-extra/" rel="nofollow">Camel Extra</a> project which hosts all *GPL related components for Camel.</p>
Modified: websites/production/camel/content/book-in-one-page.html
==============================================================================
--- websites/production/camel/content/book-in-one-page.html (original)
+++ websites/production/camel/content/book-in-one-page.html Wed Dec 17 07:19:49 2014
@@ -4126,11 +4126,11 @@ While not actual tutorials you might fin
</div>
</div>
<h2 id="BookInOnePage-Preface">Preface</h2><p>This tutorial aims to guide the reader through the stages of creating a project which uses Camel to facilitate the routing of messages from a JMS queue to a <a shape="rect" class="external-link" href="http://www.springramework.org" rel="nofollow">Spring</a> service. The route works in a synchronous fashion returning a response to the client.</p><p><style type="text/css">/*<![CDATA[*/
-div.rbtoc1418732329449 {padding: 0px;}
-div.rbtoc1418732329449 ul {list-style: disc;margin-left: 0px;}
-div.rbtoc1418732329449 li {margin-left: 0px;padding-left: 0px;}
+div.rbtoc1418800724259 {padding: 0px;}
+div.rbtoc1418800724259 ul {list-style: disc;margin-left: 0px;}
+div.rbtoc1418800724259 li {margin-left: 0px;padding-left: 0px;}
-/*]]>*/</style></p><div class="toc-macro rbtoc1418732329449">
+/*]]>*/</style></p><div class="toc-macro rbtoc1418800724259">
<ul class="toc-indentation"><li><a shape="rect" href="#Tutorial-JmsRemoting-TutorialonSpringRemotingwithJMS">Tutorial on Spring Remoting with JMS</a></li><li><a shape="rect" href="#Tutorial-JmsRemoting-Preface">Preface</a></li><li><a shape="rect" href="#Tutorial-JmsRemoting-Prerequisites">Prerequisites</a></li><li><a shape="rect" href="#Tutorial-JmsRemoting-Distribution">Distribution</a></li><li><a shape="rect" href="#Tutorial-JmsRemoting-About">About</a></li><li><a shape="rect" href="#Tutorial-JmsRemoting-CreatetheCamelProject">Create the Camel Project</a>
<ul class="toc-indentation"><li><a shape="rect" href="#Tutorial-JmsRemoting-UpdatethePOMwithDependencies">Update the POM with Dependencies</a></li></ul>
</li><li><a shape="rect" href="#Tutorial-JmsRemoting-WritingtheServer">Writing the Server</a>
@@ -6316,11 +6316,11 @@ So we completed the last piece in the pi
<style type="text/css">/*<![CDATA[*/
-div.rbtoc1418732330791 {padding: 0px;}
-div.rbtoc1418732330791 ul {list-style: disc;margin-left: 0px;}
-div.rbtoc1418732330791 li {margin-left: 0px;padding-left: 0px;}
+div.rbtoc1418800724844 {padding: 0px;}
+div.rbtoc1418800724844 ul {list-style: disc;margin-left: 0px;}
+div.rbtoc1418800724844 li {margin-left: 0px;padding-left: 0px;}
-/*]]>*/</style><div class="toc-macro rbtoc1418732330791">
+/*]]>*/</style><div class="toc-macro rbtoc1418800724844">
<ul class="toc-indentation"><li><a shape="rect" href="#Tutorial-AXIS-Camel-TutorialusingAxis1.4withApacheCamel">Tutorial using Axis 1.4 with Apache Camel</a>
<ul class="toc-indentation"><li><a shape="rect" href="#Tutorial-AXIS-Camel-Prerequisites">Prerequisites</a></li><li><a shape="rect" href="#Tutorial-AXIS-Camel-Distribution">Distribution</a></li><li><a shape="rect" href="#Tutorial-AXIS-Camel-Introduction">Introduction</a></li><li><a shape="rect" href="#Tutorial-AXIS-Camel-SettinguptheprojecttorunAxis">Setting up the project to run Axis</a>
<ul class="toc-indentation"><li><a shape="rect" href="#Tutorial-AXIS-Camel-Maven2">Maven 2</a></li><li><a shape="rect" href="#Tutorial-AXIS-Camel-wsdl">wsdl</a></li><li><a shape="rect" href="#Tutorial-AXIS-Camel-ConfiguringAxis">Configuring Axis</a></li><li><a shape="rect" href="#Tutorial-AXIS-Camel-RunningtheExample">Running the Example</a></li></ul>
@@ -19229,11 +19229,11 @@ template.send("direct:alias-verify&
</div>
</div>
<p>The <strong>cxf:</strong> component provides integration with <a shape="rect" href="http://cxf.apache.org">Apache CXF</a> for connecting to JAX-WS services hosted in CXF.</p><p><style type="text/css">/*<![CDATA[*/
-div.rbtoc1418732349169 {padding: 0px;}
-div.rbtoc1418732349169 ul {list-style: disc;margin-left: 0px;}
-div.rbtoc1418732349169 li {margin-left: 0px;padding-left: 0px;}
+div.rbtoc1418800730865 {padding: 0px;}
+div.rbtoc1418800730865 ul {list-style: disc;margin-left: 0px;}
+div.rbtoc1418800730865 li {margin-left: 0px;padding-left: 0px;}
-/*]]>*/</style></p><div class="toc-macro rbtoc1418732349169">
+/*]]>*/</style></p><div class="toc-macro rbtoc1418800730865">
<ul class="toc-indentation"><li><a shape="rect" href="#CXF-CXFComponent">CXF Component</a>
<ul class="toc-indentation"><li><a shape="rect" href="#CXF-URIformat">URI format</a></li><li><a shape="rect" href="#CXF-Options">Options</a>
<ul class="toc-indentation"><li><a shape="rect" href="#CXF-Thedescriptionsofthedataformats">The descriptions of the dataformats</a>
@@ -22641,9 +22641,13 @@ template.sendBodyAndHeader("direct:
</div></div><h3 id="BookInOnePage-URIformat.25">URI format</h3><div class="code panel pdl" style="border-width: 1px;"><div class="codeContent panelContent pdl">
<script class="theme: Default; brush: java; gutter: false" type="syntaxhighlighter"><![CDATA[hdfs://hostname[:port][/path][?options]
]]></script>
-</div></div><p>You can append query options to the URI in the following format, <code>?option=value&option=value&...</code><br clear="none"> The path is treated in the following way:</p><ol><li>as a consumer, if it's a file, it just reads the file, otherwise if it represents a directory it scans all the file under the path satisfying the configured pattern. All the files under that directory must be of the same type.</li><li>as a producer, if at least one split strategy is defined, the path is considered a directory and under that directory the producer creates a different file per split named using the configured <a shape="rect" href="uuidgenerator.html">UuidGenerator</a>.</li></ol><h3 id="BookInOnePage-Options.40">Options</h3><div class="confluenceTableSmall">
-<div class="table-wrap"><table class="confluenceTable"><tbody><tr><th colspan="1" rowspan="1" class="confluenceTh"><p> Name </p></th><th colspan="1" rowspan="1" class="confluenceTh"><p> Default Value </p></th><th colspan="1" rowspan="1" class="confluenceTh"><p> Description </p></th></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>overwrite</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>true</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> The file can be overwritten </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>append</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>false</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> Append to existing file. Notice that not all HDFS file systems support the append option. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>bufferSize</code> </p></td><td colspan="1" rowspan="1" cla
ss="confluenceTd"><p> <code>4096</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> The buffer size used by HDFS </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>replication</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>3</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> The HDFS replication factor </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>blockSize</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>67108864</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> The size of the HDFS blocks </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>fileType</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>NORMAL_FILE</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> It can be SEQUENCE_FILE, MAP_FILE, ARRAY_FILE, or BLOOMMAP_FILE, see Hadoop </p></td></tr>
<tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>fileSystemType</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>HDFS</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> It can be LOCAL for local filesystem </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>keyType</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>NULL</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> The type for the key in case of sequence or map files. See below. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>valueType</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>TEXT</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> The type for the key in case of sequence or map files. See below. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>splitStrategy</code> </p></td><td colspan="1" rowsp
an="1" class="confluenceTd"><p> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> A string describing the strategy on how to split the file based on different criteria. See below. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>openedSuffix</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>opened</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> When a file is opened for reading/writing the file is renamed with this suffix to avoid to read it during the writing phase. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>readSuffix</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>read</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> Once the file has been read is renamed with this suffix to avoid to read it again. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>initialDelay</code> </p></td><td co
lspan="1" rowspan="1" class="confluenceTd"><p> <code>0</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> For the consumer, how much to wait (milliseconds) before to start scanning the directory. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>delay</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>0</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> The interval (milliseconds) between the directory scans. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>pattern</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>*</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> The pattern used for scanning the directory </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>chunkSize</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>4096</code> </p></td><td colspan="1" rowspan="1" class="co
nfluenceTd"><p> When reading a normal file, this is split into chunks producing a message per chunk. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>connectOnStartup</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>true</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <strong>Camel 2.9.3/2.10.1:</strong> Whether to connect to the HDFS file system on starting the producer/consumer. If <code>false</code> then the connection is created on-demand. Notice that HDFS may take up till 15 minutes to establish a connection, as it has hardcoded 45 x 20 sec redelivery. By setting this option to <code>false</code> allows your application to startup, and not block for up till 15 minutes. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>owner</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <strong>Camel 2.13/2
.12.4:</strong> The file owner must match this owner for the consumer to pickup the file. Otherwise the file is skipped. </p></td></tr></tbody></table></div>
-</div><h4 id="BookInOnePage-KeyTypeandValueType">KeyType and ValueType</h4><ul><li>NULL it means that the key or the value is absent</li><li>BYTE for writing a byte, the java Byte class is mapped into a BYTE</li><li>BYTES for writing a sequence of bytes. It maps the java ByteBuffer class</li><li>INT for writing java integer</li><li>FLOAT for writing java float</li><li>LONG for writing java long</li><li>DOUBLE for writing java double</li><li>TEXT for writing java strings</li></ul><p>BYTES is also used with everything else, for example, in Camel a file is sent around as an InputStream, int this case is written in a sequence file or a map file as a sequence of bytes.</p><h3 id="BookInOnePage-SplittingStrategy">Splitting Strategy</h3><p>In the current version of Hadoop opening a file in append mode is disabled since it's not very reliable. So, for the moment, it's only possible to create new files. The Camel HDFS endpoint tries to solve this problem in this way:</p><ul><li>If the split
strategy option has been defined, the hdfs path will be used as a directory and files will be created using the configured <a shape="rect" href="uuidgenerator.html">UuidGenerator</a></li><li>Every time a splitting condition is met, a new file is created.<br clear="none"> The splitStrategy option is defined as a string with the following syntax:<br clear="none"> splitStrategy=<ST>:<value>,<ST>:<value>,*</li></ul><p>where <ST> can be:</p><ul><li>BYTES a new file is created, and the old is closed when the number of written bytes is more than <value></li><li>MESSAGES a new file is created, and the old is closed when the number of written messages is more than <value></li><li>IDLE a new file is created, and the old is closed when no writing happened in the last <value> milliseconds</li></ul> <div class="aui-message warning shadowed information-macro">
+</div></div><p>You can append query options to the URI in the following format, <code>?option=value&option=value&...</code><br clear="none"> The path is treated in the following way:</p><ol><li>as a consumer, if it's a file, it just reads the file, otherwise if it represents a directory it scans all the file under the path satisfying the configured pattern. All the files under that directory must be of the same type.</li><li>as a producer, if at least one split strategy is defined, the path is considered a directory and under that directory the producer creates a different file per split named using the configured <a shape="rect" href="uuidgenerator.html">UuidGenerator</a>.</li></ol> <div class="aui-message warning shadowed information-macro">
+ <span class="aui-icon icon-warning">Icon</span>
+ <div class="message-content">
+ <p><span>When consuming from hdfs then in normal mode, a file is split into chunks, producing a message per chunk. You can configure the size of the chunk using the chunkSize option. If you want to read from hdfs and write to a regular file using the file component, then you can use the fileMode=Append to append each of the chunks together.</span></p>
+ </div>
+ </div>
+<p> </p><h3 id="BookInOnePage-Options.40">Options</h3><div class="confluenceTableSmall"><div class="table-wrap"><table class="confluenceTable"><tbody><tr><th colspan="1" rowspan="1" class="confluenceTh"><p>Name</p></th><th colspan="1" rowspan="1" class="confluenceTh"><p>Default Value</p></th><th colspan="1" rowspan="1" class="confluenceTh"><p>Description</p></th></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>overwrite</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>true</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>The file can be overwritten</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>append</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>false</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>Append to existing file. Notice that not all HDFS file systems support the append option.</p></td></tr><tr><td colspan="1" rowspan="1" class="conflue
nceTd"><p><code>bufferSize</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>4096</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>The buffer size used by HDFS</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>replication</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>3</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>The HDFS replication factor</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>blockSize</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>67108864</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>The size of the HDFS blocks</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>fileType</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>NORMAL_FILE</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>It can be SEQUENCE_FILE, MAP_FILE, ARRAY_
FILE, or BLOOMMAP_FILE, see Hadoop</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>fileSystemType</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>HDFS</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>It can be LOCAL for local filesystem</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>keyType</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>NULL</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>The type for the key in case of sequence or map files. See below.</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>valueType</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>TEXT</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>The type for the key in case of sequence or map files. See below.</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>splitStrategy</code></p><
/td><td colspan="1" rowspan="1" class="confluenceTd"><p> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>A string describing the strategy on how to split the file based on different criteria. See below.</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>openedSuffix</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>opened</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>When a file is opened for reading/writing the file is renamed with this suffix to avoid to read it during the writing phase.</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>readSuffix</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>read</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>Once the file has been read is renamed with this suffix to avoid to read it again.</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>initialDelay</code></p></td
><td colspan="1" rowspan="1" class="confluenceTd"><p><code>0</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>For the consumer, how much to wait (milliseconds) before to start scanning the directory.</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>delay</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>0</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>The interval (milliseconds) between the directory scans.</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>pattern</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>*</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>The pattern used for scanning the directory</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>chunkSize</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>4096</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>
When reading a normal file, this is split into chunks producing a message per chunk.</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>connectOnStartup</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>true</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><strong>Camel 2.9.3/2.10.1:</strong> Whether to connect to the HDFS file system on starting the producer/consumer. If <code>false</code> then the connection is created on-demand. Notice that HDFS may take up till 15 minutes to establish a connection, as it has hardcoded 45 x 20 sec redelivery. By setting this option to <code>false</code> allows your application to startup, and not block for up till 15 minutes.</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>owner</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><strong>Camel 2.13/2.12.4:</strong> The file o
wner must match this owner for the consumer to pickup the file. Otherwise the file is skipped.</p></td></tr></tbody></table></div></div><h4 id="BookInOnePage-KeyTypeandValueType">KeyType and ValueType</h4><ul><li>NULL it means that the key or the value is absent</li><li>BYTE for writing a byte, the java Byte class is mapped into a BYTE</li><li>BYTES for writing a sequence of bytes. It maps the java ByteBuffer class</li><li>INT for writing java integer</li><li>FLOAT for writing java float</li><li>LONG for writing java long</li><li>DOUBLE for writing java double</li><li>TEXT for writing java strings</li></ul><p>BYTES is also used with everything else, for example, in Camel a file is sent around as an InputStream, int this case is written in a sequence file or a map file as a sequence of bytes.</p><h3 id="BookInOnePage-SplittingStrategy">Splitting Strategy</h3><p>In the current version of Hadoop opening a file in append mode is disabled since it's not very reliable. So, for the moment,
it's only possible to create new files. The Camel HDFS endpoint tries to solve this problem in this way:</p><ul><li>If the split strategy option has been defined, the hdfs path will be used as a directory and files will be created using the configured <a shape="rect" href="uuidgenerator.html">UuidGenerator</a></li><li>Every time a splitting condition is met, a new file is created.<br clear="none"> The splitStrategy option is defined as a string with the following syntax:<br clear="none"> splitStrategy=<ST>:<value>,<ST>:<value>,*</li></ul><p>where <ST> can be:</p><ul><li>BYTES a new file is created, and the old is closed when the number of written bytes is more than <value></li><li>MESSAGES a new file is created, and the old is closed when the number of written messages is more than <value></li><li>IDLE a new file is created, and the old is closed when no writing happened in the last <value> milliseconds</li></ul> <div class="aui-me
ssage warning shadowed information-macro">
<span class="aui-icon icon-warning">Icon</span>
<div class="message-content">
<p>note that this strategy currently requires either setting an IDLE value or setting the HdfsConstants.HDFS_CLOSE header to false to use the BYTES/MESSAGES configuration...otherwise, the file will be closed with each message</p>
@@ -22652,9 +22656,7 @@ template.sendBodyAndHeader("direct:
<p>for example:</p><div class="code panel pdl" style="border-width: 1px;"><div class="codeContent panelContent pdl">
<script class="theme: Default; brush: java; gutter: false" type="syntaxhighlighter"><![CDATA[hdfs://localhost/tmp/simple-file?splitStrategy=IDLE:1000,BYTES:5
]]></script>
-</div></div><p>it means: a new file is created either when it has been idle for more than 1 second or if more than 5 bytes have been written. So, running <code>hadoop fs -ls /tmp/simple-file</code> you'll see that multiple files have been created.</p><h3 id="BookInOnePage-MessageHeaders.4">Message Headers</h3><p>The following headers are supported by this component:</p><h4 id="BookInOnePage-Produceronly">Producer only</h4><div class="confluenceTableSmall">
-<div class="table-wrap"><table class="confluenceTable"><tbody><tr><th colspan="1" rowspan="1" class="confluenceTh"><p> Header </p></th><th colspan="1" rowspan="1" class="confluenceTh"><p> Description </p></th></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>CamelFileName</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <strong>Camel 2.13:</strong> Specifies the name of the file to write (relative to the endpoint path). The name can be a <code>String</code> or an <a shape="rect" href="expression.html">Expression</a> object. Only relevant when not using a split strategy. </p></td></tr></tbody></table></div>
-</div><h3 id="BookInOnePage-Controllingtoclosefilestream">Controlling to close file stream</h3><p><strong>Available as of Camel 2.10.4</strong></p><p>When using the <a shape="rect" href="hdfs.html">HDFS</a> producer <strong>without</strong> a split strategy, then the file output stream is by default closed after the write. However you may want to keep the stream open, and only explicitly close the stream later. For that you can use the header <code>HdfsConstants.HDFS_CLOSE</code> (value = <code>"CamelHdfsClose"</code>) to control this. Setting this value to a boolean allows you to explicit control whether the stream should be closed or not.</p><p>Notice this does not apply if you use a split strategy, as there are various strategies that can control when the stream is closed.</p><h3 id="BookInOnePage-UsingthiscomponentinOSGi">Using this component in OSGi</h3><p>This component is fully functional in an OSGi environment, however, it requires some actions from the user. Hadoop uses the
thread context class loader in order to load resources. Usually, the thread context classloader will be the bundle class loader of the bundle that contains the routes. So, the default configuration files need to be visible from the bundle class loader. A typical way to deal with it is to keep a copy of core-default.xml in your bundle root. That file can be found in the hadoop-common.jar.</p>
+</div></div><p>it means: a new file is created either when it has been idle for more than 1 second or if more than 5 bytes have been written. So, running <code>hadoop fs -ls /tmp/simple-file</code> you'll see that multiple files have been created.</p><h3 id="BookInOnePage-MessageHeaders.4">Message Headers</h3><p>The following headers are supported by this component:</p><h4 id="BookInOnePage-Produceronly">Producer only</h4><div class="confluenceTableSmall"><div class="table-wrap"><table class="confluenceTable"><tbody><tr><th colspan="1" rowspan="1" class="confluenceTh"><p>Header</p></th><th colspan="1" rowspan="1" class="confluenceTh"><p>Description</p></th></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>CamelFileName</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><strong>Camel 2.13:</strong> Specifies the name of the file to write (relative to the endpoint path). The name can be a <code>String</code> or an <a shape="rect" href="expression.html"
>Expression</a> object. Only relevant when not using a split strategy.</p></td></tr></tbody></table></div></div><h3 id="BookInOnePage-Controllingtoclosefilestream">Controlling to close file stream</h3><p><strong>Available as of Camel 2.10.4</strong></p><p>When using the <a shape="rect" href="hdfs.html">HDFS</a> producer <strong>without</strong> a split strategy, then the file output stream is by default closed after the write. However you may want to keep the stream open, and only explicitly close the stream later. For that you can use the header <code>HdfsConstants.HDFS_CLOSE</code> (value = <code>"CamelHdfsClose"</code>) to control this. Setting this value to a boolean allows you to explicit control whether the stream should be closed or not.</p><p>Notice this does not apply if you use a split strategy, as there are various strategies that can control when the stream is closed.</p><h3 id="BookInOnePage-UsingthiscomponentinOSGi">Using this component in OSGi</h3><p>This component is
fully functional in an OSGi environment, however, it requires some actions from the user. Hadoop uses the thread context class loader in order to load resources. Usually, the thread context classloader will be the bundle class loader of the bundle that contains the routes. So, the default configuration files need to be visible from the bundle class loader. A typical way to deal with it is to keep a copy of core-default.xml in your bundle root. That file can be found in the hadoop-common.jar.</p>
<h2 id="BookInOnePage-HibernateComponent">Hibernate Component</h2>
<p>The <strong>hibernate:</strong> component allows you to work with databases using Hibernate as the object relational mapping technology to map POJOs to database tables. The <strong>camel-hibernate</strong> library is provided by the <a shape="rect" class="external-link" href="http://code.google.com/p/camel-extra/" rel="nofollow">Camel Extra</a> project which hosts all *GPL related components for Camel.</p>
Modified: websites/production/camel/content/cache/main.pageCache
==============================================================================
Binary files - no diff available.
Modified: websites/production/camel/content/download.html
==============================================================================
--- websites/production/camel/content/download.html (original)
+++ websites/production/camel/content/download.html Wed Dec 17 07:19:49 2014
@@ -84,7 +84,7 @@
<tbody>
<tr>
<td valign="top" width="100%">
-<div class="wiki-content maincontent"><h2 id="Download-LatestReleases">Latest Releases</h2><div class="sectionColumnWrapper"><div class="sectionMacro"><div class="sectionMacroRow"><div class="columnMacro" style="width:20%;min-width:20%;max-width:20%;"><div style="padding-right:20px;float:left;"><p><img class="confluence-embedded-image confluence-external-resource" src="http://camel.apache.org/images/camel-box-small.png" data-image-src="http://camel.apache.org/images/camel-box-small.png"></p></div></div><div class="columnMacro" style="width:80%;min-width:80%;max-width:80%;"><div style="min-height:200px"><p>Grab these releases while they are hot!</p><ul><li><span>The latest release for Camel 2.14.x is <a shape="rect" href="camel-2140-release.html">Camel 2.14.1 Release</a>.</span></li><li>The latest release for Camel 2.13.x is <a shape="rect" href="camel-2133-release.html">Camel 2.13.3 Release</a>.</li><li>The latest release for Camel 2.12.x is <a shape="rect" href="camel-2125-release.
html">Camel 2.12.5 Release</a>. (This is last planned release for 2.12.x which now is no longer maintained)</li><li>Camel versions 2.11.x and older are no longer actively developed.</li></ul></div></div></div></div></div><h2 id="Download-OlderReleases">Older Releases</h2><p>See <a shape="rect" href="download-archives.html">Download Archives</a></p><h2 id="Download-Gettingthelatestdistributions">Getting the latest distributions</h2> <div class="aui-message hint shadowed information-macro">
+<div class="wiki-content maincontent"><h2 id="Download-LatestReleases">Latest Releases</h2><div class="sectionColumnWrapper"><div class="sectionMacro"><div class="sectionMacroRow"><div class="columnMacro" style="width:20%;min-width:20%;max-width:20%;"><div style="padding-right:20px;float:left;"><p><img class="confluence-embedded-image confluence-external-resource" src="http://camel.apache.org/images/camel-box-small.png" data-image-src="http://camel.apache.org/images/camel-box-small.png"></p></div></div><div class="columnMacro" style="width:80%;min-width:80%;max-width:80%;"><div style="min-height:200px"><p>Grab these releases while they are hot!</p><ul><li><span>The latest release for Camel 2.14.x is <a shape="rect" href="camel-2141-release.html">Camel 2.14.1 Release</a>.</span></li><li>The latest release for Camel 2.13.x is <a shape="rect" href="camel-2133-release.html">Camel 2.13.3 Release</a>.</li><li>The latest release for Camel 2.12.x is <a shape="rect" href="camel-2125-release.
html">Camel 2.12.5 Release</a>. (This is last planned release for 2.12.x which now is no longer maintained)</li><li>Camel versions 2.11.x and older are no longer actively developed.</li></ul></div></div></div></div></div><h2 id="Download-OlderReleases">Older Releases</h2><p>See <a shape="rect" href="download-archives.html">Download Archives</a></p><h2 id="Download-Gettingthelatestdistributions">Getting the latest distributions</h2> <div class="aui-message hint shadowed information-macro">
<p class="title">The URLs below use redirection</p>
<span class="aui-icon icon-hint">Icon</span>
<div class="message-content">
Modified: websites/production/camel/content/hdfs.html
==============================================================================
--- websites/production/camel/content/hdfs.html (original)
+++ websites/production/camel/content/hdfs.html Wed Dec 17 07:19:49 2014
@@ -96,9 +96,13 @@
</div></div><h3 id="HDFS-URIformat">URI format</h3><div class="code panel pdl" style="border-width: 1px;"><div class="codeContent panelContent pdl">
<script class="theme: Default; brush: java; gutter: false" type="syntaxhighlighter"><![CDATA[hdfs://hostname[:port][/path][?options]
]]></script>
-</div></div><p>You can append query options to the URI in the following format, <code>?option=value&option=value&...</code><br clear="none"> The path is treated in the following way:</p><ol><li>as a consumer, if it's a file, it just reads the file, otherwise if it represents a directory it scans all the file under the path satisfying the configured pattern. All the files under that directory must be of the same type.</li><li>as a producer, if at least one split strategy is defined, the path is considered a directory and under that directory the producer creates a different file per split named using the configured <a shape="rect" href="uuidgenerator.html">UuidGenerator</a>.</li></ol><h3 id="HDFS-Options">Options</h3><div class="confluenceTableSmall">
-<div class="table-wrap"><table class="confluenceTable"><tbody><tr><th colspan="1" rowspan="1" class="confluenceTh"><p> Name </p></th><th colspan="1" rowspan="1" class="confluenceTh"><p> Default Value </p></th><th colspan="1" rowspan="1" class="confluenceTh"><p> Description </p></th></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>overwrite</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>true</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> The file can be overwritten </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>append</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>false</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> Append to existing file. Notice that not all HDFS file systems support the append option. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>bufferSize</code> </p></td><td colspan="1" rowspan="1" cla
ss="confluenceTd"><p> <code>4096</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> The buffer size used by HDFS </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>replication</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>3</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> The HDFS replication factor </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>blockSize</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>67108864</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> The size of the HDFS blocks </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>fileType</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>NORMAL_FILE</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> It can be SEQUENCE_FILE, MAP_FILE, ARRAY_FILE, or BLOOMMAP_FILE, see Hadoop </p></td></tr>
<tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>fileSystemType</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>HDFS</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> It can be LOCAL for local filesystem </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>keyType</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>NULL</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> The type for the key in case of sequence or map files. See below. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>valueType</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>TEXT</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> The type for the key in case of sequence or map files. See below. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>splitStrategy</code> </p></td><td colspan="1" rowsp
an="1" class="confluenceTd"><p> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> A string describing the strategy on how to split the file based on different criteria. See below. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>openedSuffix</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>opened</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> When a file is opened for reading/writing the file is renamed with this suffix to avoid to read it during the writing phase. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>readSuffix</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>read</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> Once the file has been read is renamed with this suffix to avoid to read it again. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>initialDelay</code> </p></td><td co
lspan="1" rowspan="1" class="confluenceTd"><p> <code>0</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> For the consumer, how much to wait (milliseconds) before to start scanning the directory. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>delay</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>0</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> The interval (milliseconds) between the directory scans. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>pattern</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>*</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> The pattern used for scanning the directory </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>chunkSize</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>4096</code> </p></td><td colspan="1" rowspan="1" class="co
nfluenceTd"><p> When reading a normal file, this is split into chunks producing a message per chunk. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>connectOnStartup</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>true</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <strong>Camel 2.9.3/2.10.1:</strong> Whether to connect to the HDFS file system on starting the producer/consumer. If <code>false</code> then the connection is created on-demand. Notice that HDFS may take up till 15 minutes to establish a connection, as it has hardcoded 45 x 20 sec redelivery. By setting this option to <code>false</code> allows your application to startup, and not block for up till 15 minutes. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>owner</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <strong>Camel 2.13/2
.12.4:</strong> The file owner must match this owner for the consumer to pickup the file. Otherwise the file is skipped. </p></td></tr></tbody></table></div>
-</div><h4 id="HDFS-KeyTypeandValueType">KeyType and ValueType</h4><ul><li>NULL it means that the key or the value is absent</li><li>BYTE for writing a byte, the java Byte class is mapped into a BYTE</li><li>BYTES for writing a sequence of bytes. It maps the java ByteBuffer class</li><li>INT for writing java integer</li><li>FLOAT for writing java float</li><li>LONG for writing java long</li><li>DOUBLE for writing java double</li><li>TEXT for writing java strings</li></ul><p>BYTES is also used with everything else, for example, in Camel a file is sent around as an InputStream, int this case is written in a sequence file or a map file as a sequence of bytes.</p><h3 id="HDFS-SplittingStrategy">Splitting Strategy</h3><p>In the current version of Hadoop opening a file in append mode is disabled since it's not very reliable. So, for the moment, it's only possible to create new files. The Camel HDFS endpoint tries to solve this problem in this way:</p><ul><li>If the split strategy option ha
s been defined, the hdfs path will be used as a directory and files will be created using the configured <a shape="rect" href="uuidgenerator.html">UuidGenerator</a></li><li>Every time a splitting condition is met, a new file is created.<br clear="none"> The splitStrategy option is defined as a string with the following syntax:<br clear="none"> splitStrategy=<ST>:<value>,<ST>:<value>,*</li></ul><p>where <ST> can be:</p><ul><li>BYTES a new file is created, and the old is closed when the number of written bytes is more than <value></li><li>MESSAGES a new file is created, and the old is closed when the number of written messages is more than <value></li><li>IDLE a new file is created, and the old is closed when no writing happened in the last <value> milliseconds</li></ul> <div class="aui-message warning shadowed information-macro">
+</div></div><p>You can append query options to the URI in the following format, <code>?option=value&option=value&...</code><br clear="none"> The path is treated in the following way:</p><ol><li>as a consumer, if it's a file, it just reads the file, otherwise if it represents a directory it scans all the file under the path satisfying the configured pattern. All the files under that directory must be of the same type.</li><li>as a producer, if at least one split strategy is defined, the path is considered a directory and under that directory the producer creates a different file per split named using the configured <a shape="rect" href="uuidgenerator.html">UuidGenerator</a>.</li></ol> <div class="aui-message warning shadowed information-macro">
+ <span class="aui-icon icon-warning">Icon</span>
+ <div class="message-content">
+ <p><span>When consuming from hdfs then in normal mode, a file is split into chunks, producing a message per chunk. You can configure the size of the chunk using the chunkSize option. If you want to read from hdfs and write to a regular file using the file component, then you can use the fileMode=Append to append each of the chunks together.</span></p>
+ </div>
+ </div>
+<p> </p><h3 id="HDFS-Options">Options</h3><div class="confluenceTableSmall"><div class="table-wrap"><table class="confluenceTable"><tbody><tr><th colspan="1" rowspan="1" class="confluenceTh"><p>Name</p></th><th colspan="1" rowspan="1" class="confluenceTh"><p>Default Value</p></th><th colspan="1" rowspan="1" class="confluenceTh"><p>Description</p></th></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>overwrite</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>true</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>The file can be overwritten</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>append</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>false</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>Append to existing file. Notice that not all HDFS file systems support the append option.</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><c
ode>bufferSize</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>4096</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>The buffer size used by HDFS</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>replication</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>3</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>The HDFS replication factor</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>blockSize</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>67108864</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>The size of the HDFS blocks</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>fileType</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>NORMAL_FILE</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>It can be SEQUENCE_FILE, MAP_FILE, ARRAY_FILE, or BLO
OMMAP_FILE, see Hadoop</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>fileSystemType</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>HDFS</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>It can be LOCAL for local filesystem</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>keyType</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>NULL</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>The type for the key in case of sequence or map files. See below.</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>valueType</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>TEXT</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>The type for the key in case of sequence or map files. See below.</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>splitStrategy</code></p></td><td cols
pan="1" rowspan="1" class="confluenceTd"><p> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>A string describing the strategy on how to split the file based on different criteria. See below.</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>openedSuffix</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>opened</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>When a file is opened for reading/writing the file is renamed with this suffix to avoid to read it during the writing phase.</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>readSuffix</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>read</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>Once the file has been read is renamed with this suffix to avoid to read it again.</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>initialDelay</code></p></td><td colspan
="1" rowspan="1" class="confluenceTd"><p><code>0</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>For the consumer, how much to wait (milliseconds) before to start scanning the directory.</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>delay</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>0</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>The interval (milliseconds) between the directory scans.</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>pattern</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>*</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>The pattern used for scanning the directory</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>chunkSize</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>4096</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p>When reading
a normal file, this is split into chunks producing a message per chunk.</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>connectOnStartup</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><code>true</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><strong>Camel 2.9.3/2.10.1:</strong> Whether to connect to the HDFS file system on starting the producer/consumer. If <code>false</code> then the connection is created on-demand. Notice that HDFS may take up till 15 minutes to establish a connection, as it has hardcoded 45 x 20 sec redelivery. By setting this option to <code>false</code> allows your application to startup, and not block for up till 15 minutes.</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>owner</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><strong>Camel 2.13/2.12.4:</strong> The file owner must ma
tch this owner for the consumer to pickup the file. Otherwise the file is skipped.</p></td></tr></tbody></table></div></div><h4 id="HDFS-KeyTypeandValueType">KeyType and ValueType</h4><ul><li>NULL it means that the key or the value is absent</li><li>BYTE for writing a byte, the java Byte class is mapped into a BYTE</li><li>BYTES for writing a sequence of bytes. It maps the java ByteBuffer class</li><li>INT for writing java integer</li><li>FLOAT for writing java float</li><li>LONG for writing java long</li><li>DOUBLE for writing java double</li><li>TEXT for writing java strings</li></ul><p>BYTES is also used with everything else, for example, in Camel a file is sent around as an InputStream, int this case is written in a sequence file or a map file as a sequence of bytes.</p><h3 id="HDFS-SplittingStrategy">Splitting Strategy</h3><p>In the current version of Hadoop opening a file in append mode is disabled since it's not very reliable. So, for the moment, it's only possible to create
new files. The Camel HDFS endpoint tries to solve this problem in this way:</p><ul><li>If the split strategy option has been defined, the hdfs path will be used as a directory and files will be created using the configured <a shape="rect" href="uuidgenerator.html">UuidGenerator</a></li><li>Every time a splitting condition is met, a new file is created.<br clear="none"> The splitStrategy option is defined as a string with the following syntax:<br clear="none"> splitStrategy=<ST>:<value>,<ST>:<value>,*</li></ul><p>where <ST> can be:</p><ul><li>BYTES a new file is created, and the old is closed when the number of written bytes is more than <value></li><li>MESSAGES a new file is created, and the old is closed when the number of written messages is more than <value></li><li>IDLE a new file is created, and the old is closed when no writing happened in the last <value> milliseconds</li></ul> <div class="aui-message warning shadowed informa
tion-macro">
<span class="aui-icon icon-warning">Icon</span>
<div class="message-content">
<p>note that this strategy currently requires either setting an IDLE value or setting the HdfsConstants.HDFS_CLOSE header to false to use the BYTES/MESSAGES configuration...otherwise, the file will be closed with each message</p>
@@ -107,9 +111,7 @@
<p>for example:</p><div class="code panel pdl" style="border-width: 1px;"><div class="codeContent panelContent pdl">
<script class="theme: Default; brush: java; gutter: false" type="syntaxhighlighter"><![CDATA[hdfs://localhost/tmp/simple-file?splitStrategy=IDLE:1000,BYTES:5
]]></script>
-</div></div><p>it means: a new file is created either when it has been idle for more than 1 second or if more than 5 bytes have been written. So, running <code>hadoop fs -ls /tmp/simple-file</code> you'll see that multiple files have been created.</p><h3 id="HDFS-MessageHeaders">Message Headers</h3><p>The following headers are supported by this component:</p><h4 id="HDFS-Produceronly">Producer only</h4><div class="confluenceTableSmall">
-<div class="table-wrap"><table class="confluenceTable"><tbody><tr><th colspan="1" rowspan="1" class="confluenceTh"><p> Header </p></th><th colspan="1" rowspan="1" class="confluenceTh"><p> Description </p></th></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>CamelFileName</code> </p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <strong>Camel 2.13:</strong> Specifies the name of the file to write (relative to the endpoint path). The name can be a <code>String</code> or an <a shape="rect" href="expression.html">Expression</a> object. Only relevant when not using a split strategy. </p></td></tr></tbody></table></div>
-</div><h3 id="HDFS-Controllingtoclosefilestream">Controlling to close file stream</h3><p><strong>Available as of Camel 2.10.4</strong></p><p>When using the <a shape="rect" href="hdfs.html">HDFS</a> producer <strong>without</strong> a split strategy, then the file output stream is by default closed after the write. However you may want to keep the stream open, and only explicitly close the stream later. For that you can use the header <code>HdfsConstants.HDFS_CLOSE</code> (value = <code>"CamelHdfsClose"</code>) to control this. Setting this value to a boolean allows you to explicit control whether the stream should be closed or not.</p><p>Notice this does not apply if you use a split strategy, as there are various strategies that can control when the stream is closed.</p><h3 id="HDFS-UsingthiscomponentinOSGi">Using this component in OSGi</h3><p>This component is fully functional in an OSGi environment, however, it requires some actions from the user. Hadoop uses the thread context cl
ass loader in order to load resources. Usually, the thread context classloader will be the bundle class loader of the bundle that contains the routes. So, the default configuration files need to be visible from the bundle class loader. A typical way to deal with it is to keep a copy of core-default.xml in your bundle root. That file can be found in the hadoop-common.jar.</p></div>
+</div></div><p>it means: a new file is created either when it has been idle for more than 1 second or if more than 5 bytes have been written. So, running <code>hadoop fs -ls /tmp/simple-file</code> you'll see that multiple files have been created.</p><h3 id="HDFS-MessageHeaders">Message Headers</h3><p>The following headers are supported by this component:</p><h4 id="HDFS-Produceronly">Producer only</h4><div class="confluenceTableSmall"><div class="table-wrap"><table class="confluenceTable"><tbody><tr><th colspan="1" rowspan="1" class="confluenceTh"><p>Header</p></th><th colspan="1" rowspan="1" class="confluenceTh"><p>Description</p></th></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p><code>CamelFileName</code></p></td><td colspan="1" rowspan="1" class="confluenceTd"><p><strong>Camel 2.13:</strong> Specifies the name of the file to write (relative to the endpoint path). The name can be a <code>String</code> or an <a shape="rect" href="expression.html">Expression</a> obje
ct. Only relevant when not using a split strategy.</p></td></tr></tbody></table></div></div><h3 id="HDFS-Controllingtoclosefilestream">Controlling to close file stream</h3><p><strong>Available as of Camel 2.10.4</strong></p><p>When using the <a shape="rect" href="hdfs.html">HDFS</a> producer <strong>without</strong> a split strategy, then the file output stream is by default closed after the write. However you may want to keep the stream open, and only explicitly close the stream later. For that you can use the header <code>HdfsConstants.HDFS_CLOSE</code> (value = <code>"CamelHdfsClose"</code>) to control this. Setting this value to a boolean allows you to explicit control whether the stream should be closed or not.</p><p>Notice this does not apply if you use a split strategy, as there are various strategies that can control when the stream is closed.</p><h3 id="HDFS-UsingthiscomponentinOSGi">Using this component in OSGi</h3><p>This component is fully functional in an OSGi environme
nt, however, it requires some actions from the user. Hadoop uses the thread context class loader in order to load resources. Usually, the thread context classloader will be the bundle class loader of the bundle that contains the routes. So, the default configuration files need to be visible from the bundle class loader. A typical way to deal with it is to keep a copy of core-default.xml in your bundle root. That file can be found in the hadoop-common.jar.</p></div>
</td>
<td valign="top">
<div class="navigation">