You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@nifi.apache.org by jo...@apache.org on 2018/01/12 21:00:25 UTC
svn commit: r1821033 [26/46] - in /nifi/site/trunk/docs: nifi-docs/
nifi-docs/components/ nifi-docs/components/org.apache.nifi/
nifi-docs/components/org.apache.nifi/nifi-ambari-nar/
nifi-docs/components/org.apache.nifi/nifi-ambari-nar/1.5.0/ nifi-docs/...
Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.ListenUDP/index.html
URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.ListenUDP/index.html?rev=1821033&view=auto
==============================================================================
--- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.ListenUDP/index.html (added)
+++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.ListenUDP/index.html Fri Jan 12 21:00:14 2018
@@ -0,0 +1 @@
+<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>ListenUDP</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">ListenUDP</h1><h2>Description: </h2><p>Listens for Datagram Packets on a given port. The default behavior produces a FlowFile per datagram, however for higher throughput the Max Batch Size property may be increased to specify the number of datagrams to batch together in a single FlowFile. This processor can be restricted to listening for datagrams from a specific remote host and port by specifying the Sending Host and Sending Host Port properties, otherwise it will listen for datagrams from all hosts and ports.</p><h3>Tags: </h3><p>ingest, udp, listen, source</p><h3>Properties: </h3><p>In the list be
low, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a>.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name">Local Network Interface</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The name of a local network interface to be used to restrict listening to a specific LAN.<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name"><strong>Port</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The port to listen on for communication.</td></tr><tr><td id="name"><strong>Receive Buffer Size</strong></td><td id="default-value">65507 B</td><td id="allowable-values"></td
><td id="description">The size of each buffer used to receive messages. Adjust this value appropriately based on the expected size of the incoming messages.</td></tr><tr><td id="name"><strong>Max Size of Message Queue</strong></td><td id="default-value">10000</td><td id="allowable-values"></td><td id="description">The maximum size of the internal queue used to buffer messages being transferred from the underlying channel to the processor. Setting this value higher allows more messages to be buffered in memory during surges of incoming messages, but increases the total memory used by the processor.</td></tr><tr><td id="name"><strong>Max Size of Socket Buffer</strong></td><td id="default-value">1 MB</td><td id="allowable-values"></td><td id="description">The maximum size of the socket buffer that should be used. This is a suggestion to the Operating System to indicate how big the socket buffer should be. If this value is set too low, the buffer may fill up before the data can be read,
and incoming data will be dropped.</td></tr><tr><td id="name"><strong>Character Set</strong></td><td id="default-value">UTF-8</td><td id="allowable-values"></td><td id="description">Specifies the character set of the received data.</td></tr><tr><td id="name"><strong>Max Batch Size</strong></td><td id="default-value">1</td><td id="allowable-values"></td><td id="description">The maximum number of messages to add to a single FlowFile. If multiple messages are available, they will be concatenated along with the <Message Delimiter> up to this configured maximum number of messages</td></tr><tr><td id="name"><strong>Batching Message Delimiter</strong></td><td id="default-value">\n</td><td id="allowable-values"></td><td id="description">Specifies the delimiter to place between messages when multiple messages are bundled together (see <Max Batch Size> property).</td></tr><tr><td id="name">Sending Host</td><td id="default-value"></td><td id="allowable-values"></td><td id="descrip
tion">IP, or name, of a remote host. Only Datagrams from the specified Sending Host Port and this host will be accepted. Improves Performance. May be a system property or an environment variable.<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name">Sending Host Port</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Port being used by remote host to send Datagrams. Only Datagrams from the specified Sending Host and this port will be accepted. Improves Performance. May be a system property or an environment variable.<br/><strong>Supports Expression Language: true</strong></td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td>Messages received successfully will be sent out this relationship.</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3><table id="writes-attributes"><tr><th>Name</th><th>Description</th></
tr><tr><td>udp.sender</td><td>The sending host of the messages.</td></tr><tr><td>udp.port</td><td>The sending port the messages were received.</td></tr></table><h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>Input requirement: </h3>This component does not allow an incoming relationship.</body></html>
\ No newline at end of file
Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.ListenUDPRecord/index.html
URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.ListenUDPRecord/index.html?rev=1821033&view=auto
==============================================================================
--- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.ListenUDPRecord/index.html (added)
+++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.ListenUDPRecord/index.html Fri Jan 12 21:00:14 2018
@@ -0,0 +1 @@
+<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>ListenUDPRecord</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">ListenUDPRecord</h1><h2>Description: </h2><p>Listens for Datagram Packets on a given port and reads the content of each datagram using the configured Record Reader. Each record will then be written to a flow file using the configured Record Writer. This processor can be restricted to listening for datagrams from a specific remote host and port by specifying the Sending Host and Sending Host Port properties, otherwise it will listen for datagrams from all hosts and ports.</p><h3>Tags: </h3><p>ingest, udp, listen, source, record</p><h3>Properties: </h3><p>In the list below, the names of required
properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a>.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name">Local Network Interface</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The name of a local network interface to be used to restrict listening to a specific LAN.<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name"><strong>Port</strong></td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The port to listen on for communication.</td></tr><tr><td id="name"><strong>Receive Buffer Size</strong></td><td id="default-value">65507 B</td><td id="allowable-values"></td><td id="description">The s
ize of each buffer used to receive messages. Adjust this value appropriately based on the expected size of the incoming messages.</td></tr><tr><td id="name"><strong>Max Size of Message Queue</strong></td><td id="default-value">10000</td><td id="allowable-values"></td><td id="description">The maximum size of the internal queue used to buffer messages being transferred from the underlying channel to the processor. Setting this value higher allows more messages to be buffered in memory during surges of incoming messages, but increases the total memory used by the processor.</td></tr><tr><td id="name"><strong>Max Size of Socket Buffer</strong></td><td id="default-value">1 MB</td><td id="allowable-values"></td><td id="description">The maximum size of the socket buffer that should be used. This is a suggestion to the Operating System to indicate how big the socket buffer should be. If this value is set too low, the buffer may fill up before the data can be read, and incoming data will be
dropped.</td></tr><tr><td id="name"><strong>Character Set</strong></td><td id="default-value">UTF-8</td><td id="allowable-values"></td><td id="description">Specifies the character set of the received data.</td></tr><tr><td id="name"><strong>Poll Timeout</strong></td><td id="default-value">50 ms</td><td id="allowable-values"></td><td id="description">The amount of time to wait when polling the internal queue for more datagrams. If no datagrams are found after waiting for the configured timeout, then the processor will emit whatever records have been obtained up to that point.</td></tr><tr><td id="name"><strong>Batch Size</strong></td><td id="default-value">1000</td><td id="allowable-values"></td><td id="description">The maximum number of datagrams to write as records to a single FlowFile. The Batch Size will only be reached when data is coming in more frequently than the Poll Timeout.</td></tr><tr><td id="name"><strong>Record Reader</strong></td><td id="default-value"></td><td id="al
lowable-values"><strong>Controller Service API: </strong><br/>RecordReaderFactory<br/><strong>Implementations: </strong><a href="../../../nifi-record-serialization-services-nar/1.5.0/org.apache.nifi.csv.CSVReader/index.html">CSVReader</a><br/><a href="../../../nifi-scripting-nar/1.5.0/org.apache.nifi.record.script.ScriptedReader/index.html">ScriptedReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.5.0/org.apache.nifi.grok.GrokReader/index.html">GrokReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.5.0/org.apache.nifi.json.JsonTreeReader/index.html">JsonTreeReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.5.0/org.apache.nifi.avro.AvroReader/index.html">AvroReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.5.0/org.apache.nifi.json.JsonPathReader/index.html">JsonPathReader</a></td><td id="description">The Record Reader to use for reading the content of incoming datagrams.</td></tr><tr><td i
d="name"><strong>Record Writer</strong></td><td id="default-value"></td><td id="allowable-values"><strong>Controller Service API: </strong><br/>RecordSetWriterFactory<br/><strong>Implementations: </strong><a href="../../../nifi-record-serialization-services-nar/1.5.0/org.apache.nifi.text.FreeFormTextRecordSetWriter/index.html">FreeFormTextRecordSetWriter</a><br/><a href="../../../nifi-record-serialization-services-nar/1.5.0/org.apache.nifi.csv.CSVRecordSetWriter/index.html">CSVRecordSetWriter</a><br/><a href="../../../nifi-record-serialization-services-nar/1.5.0/org.apache.nifi.avro.AvroRecordSetWriter/index.html">AvroRecordSetWriter</a><br/><a href="../../../nifi-record-serialization-services-nar/1.5.0/org.apache.nifi.json.JsonRecordSetWriter/index.html">JsonRecordSetWriter</a><br/><a href="../../../nifi-scripting-nar/1.5.0/org.apache.nifi.record.script.ScriptedRecordSetWriter/index.html">ScriptedRecordSetWriter</a></td><td id="description">The Record Writer to use in order to seri
alize the data before writing to a flow file.</td></tr><tr><td id="name">Sending Host</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">IP, or name, of a remote host. Only Datagrams from the specified Sending Host Port and this host will be accepted. Improves Performance. May be a system property or an environment variable.<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name">Sending Host Port</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Port being used by remote host to send Datagrams. Only Datagrams from the specified Sending Host and this port will be accepted. Improves Performance. May be a system property or an environment variable.<br/><strong>Supports Expression Language: true</strong></td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td>Messages received successfully will be sent out this relati
onship.</td></tr><tr><td>parse.failure</td><td>If a datagram cannot be parsed using the configured Record Reader, the contents of the message will be routed to this Relationship as its own individual FlowFile.</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3><table id="writes-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>udp.sender</td><td>The sending host of the messages.</td></tr><tr><td>udp.port</td><td>The sending port the messages were received.</td></tr><tr><td>record.count</td><td>The number of records written to the flow file.</td></tr><tr><td>mime.type</td><td>The mime-type of the writer used to write the records to the flow file.</td></tr></table><h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>Input requirement: </h3>This component does not allow an incoming relationship.</body></html>
\ No newline at end of file
Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.LogAttribute/index.html
URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.LogAttribute/index.html?rev=1821033&view=auto
==============================================================================
--- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.LogAttribute/index.html (added)
+++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.LogAttribute/index.html Fri Jan 12 21:00:14 2018
@@ -0,0 +1 @@
+<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>LogAttribute</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">LogAttribute</h1><h2>Description: </h2><p>No description provided.</p><h3>Tags: </h3><p>attributes, logging</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a>.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><strong>Log Level</s
trong></td><td id="default-value">info</td><td id="allowable-values"><ul><li>trace</li><li>debug</li><li>info</li><li>warn</li><li>error</li></ul></td><td id="description">The Log Level to use when logging the Attributes</td></tr><tr><td id="name"><strong>Log Payload</strong></td><td id="default-value">false</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">If true, the FlowFile's payload will be logged, in addition to its attributes; otherwise, just the Attributes will be logged.</td></tr><tr><td id="name">Attributes to Log</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">A comma-separated list of Attributes to Log. If not specified, all attributes will be logged unless `Attributes to Log by Regular Expression` is modified. There's an AND relationship between the two properties.</td></tr><tr><td id="name">Attributes to Log by Regular Expression</td><td id="default-value">.*</td><td id="allowable-values"></
td><td id="description">A regular expression indicating the Attributes to Log. If not specified, all attributes will be logged unless `Attributes to Log` is modified. There's an AND relationship between the two properties.</td></tr><tr><td id="name">Attributes to Ignore</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">A comma-separated list of Attributes to ignore. If not specified, no attributes will be ignored unless `Attributes to Ignore by Regular Expression` is modified. There's an OR relationship between the two properties.</td></tr><tr><td id="name">Attributes to Ignore by Regular Expression</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">A regular expression indicating the Attributes to Ignore. If not specified, no attributes will be ignored unless `Attributes to Ignore` is modified. There's an OR relationship between the two properties.</td></tr><tr><td id="name">Log prefix</td><td id="default-value"></t
d><td id="allowable-values"></td><td id="description">Log prefix appended to the log lines. It helps to distinguish the output of multiple LogAttribute processors.<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name"><strong>Character Set</strong></td><td id="default-value">UTF-8</td><td id="allowable-values"></td><td id="description">The name of the CharacterSet to use<br/><strong>Supports Expression Language: true</strong></td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td>All FlowFiles are routed to this relationship</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3>None specified.<h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>Input requirement: </h3>This component requires an incoming relationship.</body></html>
\ No newline at end of file
Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.LogMessage/index.html
URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.LogMessage/index.html?rev=1821033&view=auto
==============================================================================
--- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.LogMessage/index.html (added)
+++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.LogMessage/index.html Fri Jan 12 21:00:14 2018
@@ -0,0 +1 @@
+<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>LogMessage</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">LogMessage</h1><h2>Description: </h2><p>Emits a log message at the specified log level</p><h3>Tags: </h3><p>attributes, logging</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a>.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><s
trong>Log Level</strong></td><td id="default-value">info</td><td id="allowable-values"><ul><li>trace</li><li>debug</li><li>info</li><li>warn</li><li>error</li></ul></td><td id="description">The Log Level to use when logging the message<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name">Log prefix</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Log prefix appended to the log lines. It helps to distinguish the output of multiple LogMessage processors.<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name">Log message</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The log message to emit<br/><strong>Supports Expression Language: true</strong></td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td>All FlowFiles are routed to this relationship</td></tr></table><h3>Reads Attributes
: </h3>None specified.<h3>Writes Attributes: </h3>None specified.<h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>Input requirement: </h3>This component requires an incoming relationship.</body></html>
\ No newline at end of file
Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.LookupAttribute/index.html
URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.LookupAttribute/index.html?rev=1821033&view=auto
==============================================================================
--- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.LookupAttribute/index.html (added)
+++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.LookupAttribute/index.html Fri Jan 12 21:00:14 2018
@@ -0,0 +1 @@
+<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>LookupAttribute</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">LookupAttribute</h1><h2>Description: </h2><p>Lookup attributes from a lookup service</p><h3>Tags: </h3><p>lookup, cache, enrich, join, attributes, Attribute Expression Language</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><strong>Lookup Service</strong></td><td id="default-value"></td><td id="all
owable-values"><strong>Controller Service API: </strong><br/>StringLookupService<br/><strong>Implementations: </strong><a href="../../../nifi-lookup-services-nar/1.5.0/org.apache.nifi.lookup.SimpleCsvFileLookupService/index.html">SimpleCsvFileLookupService</a><br/><a href="../../../nifi-lookup-services-nar/1.5.0/org.apache.nifi.lookup.PropertiesFileLookupService/index.html">PropertiesFileLookupService</a><br/><a href="../../../nifi-lookup-services-nar/1.5.0/org.apache.nifi.lookup.XMLFileLookupService/index.html">XMLFileLookupService</a><br/><a href="../../../nifi-lookup-services-nar/1.5.0/org.apache.nifi.lookup.SimpleKeyValueLookupService/index.html">SimpleKeyValueLookupService</a></td><td id="description">The lookup service to use for attribute lookups</td></tr><tr><td id="name"><strong>Include Empty Values</strong></td><td id="default-value">true</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">Include null or blank values for keys that a
re null or blank</td></tr></table><h3>Dynamic Properties: </h3><p>Dynamic Properties allow the user to specify both the name and value of a property.<table id="dynamic-properties"><tr><th>Name</th><th>Value</th><th>Description</th></tr><tr><td id="name">The name of the attribute to add to the FlowFile</td><td id="value">The name of the key or property to retrieve from the lookup service</td><td>Adds a FlowFile attribute specified by the dynamic property's key with the value found in the lookup service using the the dynamic property's value<br/><strong>Supports Expression Language: true</strong></td></tr></table></p><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>failure</td><td>FlowFiles with failing lookups are routed to this relationship</td></tr><tr><td>unmatched</td><td>FlowFiles with missing lookups are routed to this relationship</td></tr><tr><td>matched</td><td>FlowFiles with matching lookups are routed to this relationship<
/td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3>None specified.<h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>Input requirement: </h3>This component requires an incoming relationship.</body></html>
\ No newline at end of file
Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.LookupRecord/index.html
URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.LookupRecord/index.html?rev=1821033&view=auto
==============================================================================
--- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.LookupRecord/index.html (added)
+++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.LookupRecord/index.html Fri Jan 12 21:00:14 2018
@@ -0,0 +1 @@
+<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>LookupRecord</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">LookupRecord</h1><h2>Description: </h2><p>Extracts one or more fields from a Record and looks up a value for those fields in a LookupService. If a result is returned by the LookupService, that result is optionally added to the Record. In this case, the processor functions as an Enrichment processor. Regardless, the Record is then routed to either the 'matched' relationship or 'unmatched' relationship (if the 'Routing Strategy' property is configured to do so), indicating whether or not a result was returned by the LookupService, allowing the processor to also function as a Routing processor. The "c
oordinates" to use for looking up a value in the Lookup Service are defined by adding a user-defined property. Each property that is added will have an entry added to a Map, where the name of the property becomes the Map Key and the value returned by the RecordPath becomes the value for that key. If multiple values are returned by the RecordPath, then the Record will be routed to the 'unmatched' relationship (or 'success', depending on the 'Routing Strategy' property's configuration). If one or more fields match the Result RecordPath, all fields that match will be updated. If there is no match in the configured LookupService, then no fields will be updated. I.e., it will not overwrite an existing value in the Record with a null value. Please note, however, that if the results returned by the LookupService are not accounted for in your schema (specifically, the schema that is configured for your Record Writer) then the fields will not be written out to the FlowFile.</p><h3>Tags: </h3
><p>lookup, enrichment, route, record, csv, json, avro, logs, convert, filter</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a>.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><strong>Record Reader</strong></td><td id="default-value"></td><td id="allowable-values"><strong>Controller Service API: </strong><br/>RecordReaderFactory<br/><strong>Implementations: </strong><a href="../../../nifi-record-serialization-services-nar/1.5.0/org.apache.nifi.csv.CSVReader/index.html">CSVReader</a><br/><a href="../../../nifi-scripting-nar/1.5.0/org.apache.nifi.record.script.ScriptedReader/index.html">ScriptedReader</a><br/><a
href="../../../nifi-record-serialization-services-nar/1.5.0/org.apache.nifi.grok.GrokReader/index.html">GrokReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.5.0/org.apache.nifi.json.JsonTreeReader/index.html">JsonTreeReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.5.0/org.apache.nifi.avro.AvroReader/index.html">AvroReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.5.0/org.apache.nifi.json.JsonPathReader/index.html">JsonPathReader</a></td><td id="description">Specifies the Controller Service to use for reading incoming data</td></tr><tr><td id="name"><strong>Record Writer</strong></td><td id="default-value"></td><td id="allowable-values"><strong>Controller Service API: </strong><br/>RecordSetWriterFactory<br/><strong>Implementations: </strong><a href="../../../nifi-record-serialization-services-nar/1.5.0/org.apache.nifi.text.FreeFormTextRecordSetWriter/index.html">FreeFormTextRecordSetWriter</a><br/><a href=".
./../../nifi-record-serialization-services-nar/1.5.0/org.apache.nifi.csv.CSVRecordSetWriter/index.html">CSVRecordSetWriter</a><br/><a href="../../../nifi-record-serialization-services-nar/1.5.0/org.apache.nifi.avro.AvroRecordSetWriter/index.html">AvroRecordSetWriter</a><br/><a href="../../../nifi-record-serialization-services-nar/1.5.0/org.apache.nifi.json.JsonRecordSetWriter/index.html">JsonRecordSetWriter</a><br/><a href="../../../nifi-scripting-nar/1.5.0/org.apache.nifi.record.script.ScriptedRecordSetWriter/index.html">ScriptedRecordSetWriter</a></td><td id="description">Specifies the Controller Service to use for writing out the records</td></tr><tr><td id="name"><strong>Lookup Service</strong></td><td id="default-value"></td><td id="allowable-values"><strong>Controller Service API: </strong><br/>LookupService<br/><strong>Implementations: </strong><a href="../../../nifi-lookup-services-nar/1.5.0/org.apache.nifi.lookup.SimpleCsvFileLookupService/index.html">SimpleCsvFileLookupSer
vice</a><br/><a href="../../../nifi-lookup-services-nar/1.5.0/org.apache.nifi.lookup.PropertiesFileLookupService/index.html">PropertiesFileLookupService</a><br/><a href="../../../nifi-lookup-services-nar/1.5.0/org.apache.nifi.lookup.maxmind.IPLookupService/index.html">IPLookupService</a><br/><a href="../../../nifi-scripting-nar/1.5.0/org.apache.nifi.lookup.script.ScriptedLookupService/index.html">ScriptedLookupService</a><br/><a href="../../../nifi-mongodb-services-nar/1.5.0/org.apache.nifi.mongodb.MongoDBLookupService/index.html">MongoDBLookupService</a><br/><a href="../../../nifi-lookup-services-nar/1.5.0/org.apache.nifi.lookup.XMLFileLookupService/index.html">XMLFileLookupService</a><br/><a href="../../../nifi-lookup-services-nar/1.5.0/org.apache.nifi.lookup.SimpleKeyValueLookupService/index.html">SimpleKeyValueLookupService</a><br/><a href="../../../nifi-lookup-services-nar/1.5.0/org.apache.nifi.lookup.CSVRecordLookupService/index.html">CSVRecordLookupService</a><br/><a href="..
/../../nifi-hbase_1_1_2-client-service-nar/1.5.0/org.apache.nifi.hbase.HBase_1_1_2_RecordLookupService/index.html">HBase_1_1_2_RecordLookupService</a></td><td id="description">The Lookup Service to use in order to lookup a value in each Record</td></tr><tr><td id="name">Result RecordPath</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">A RecordPath that points to the field whose value should be updated with whatever value is returned from the Lookup Service. If not specified, the value that is returned from the Lookup Service will be ignored, except for determining whether the FlowFile should be routed to the 'matched' or 'unmatched' Relationship.<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name"><strong>Routing Strategy</strong></td><td id="default-value">route-to-success</td><td id="allowable-values"><ul><li>Route to 'success' <img src="../../../../../html/images/iconInfo.png" alt="Records will be routed to a 'suc
cess' Relationship regardless of whether or not there is a match in the configured Lookup Service" title="Records will be routed to a 'success' Relationship regardless of whether or not there is a match in the configured Lookup Service"></img></li><li>Route to 'matched' or 'unmatched' <img src="../../../../../html/images/iconInfo.png" alt="Records will be routed to either a 'matched' or an 'unmatched' Relationship depending on whether or not there was a match in the configured Lookup Service. A single input FlowFile may result in two different output FlowFiles." title="Records will be routed to either a 'matched' or an 'unmatched' Relationship depending on whether or not there was a match in the configured Lookup Service. A single input FlowFile may result in two different output FlowFiles."></img></li></ul></td><td id="description">Specifies how to route records after a Lookup has completed</td></tr><tr><td id="name"><strong>Record Result Contents</strong></td><td id="default-value
">insert-entire-record</td><td id="allowable-values"><ul><li>Insert Entire Record <img src="../../../../../html/images/iconInfo.png" alt="The entire Record that is retrieved from the Lookup Service will be inserted into the destination path." title="The entire Record that is retrieved from the Lookup Service will be inserted into the destination path."></img></li><li>Insert Record Fields <img src="../../../../../html/images/iconInfo.png" alt="All of the fields in the Record that is retrieved from the Lookup Service will be inserted into the destination path." title="All of the fields in the Record that is retrieved from the Lookup Service will be inserted into the destination path."></img></li></ul></td><td id="description">When a result is obtained that contains a Record, this property determines whether the Record itself is inserted at the configured path or if the contents of the Record (i.e., the sub-fields) will be inserted at the configured path.</td></tr></table><h3>Dynamic P
roperties: </h3><p>Dynamic Properties allow the user to specify both the name and value of a property.<table id="dynamic-properties"><tr><th>Name</th><th>Value</th><th>Description</th></tr><tr><td id="name">Value To Lookup</td><td id="value">Valid Record Path</td><td>A RecordPath that points to the field whose value will be looked up in the configured Lookup Service<br/><strong>Supports Expression Language: true</strong></td></tr></table></p><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>success</td><td>All records will be sent to this Relationship if configured to do so, unless a failure occurs</td></tr><tr><td>failure</td><td>If a FlowFile cannot be transformed from the configured input format to the configured output format, the unchanged FlowFile will be routed to this relationship</td></tr></table><h3>Reads Attributes: </h3>None specified.<h3>Writes Attributes: </h3><table id="writes-attributes"><tr><th>Name</th><th>Descripti
on</th></tr><tr><td>mime.type</td><td>Sets the mime.type attribute to the MIME Type specified by the Record Writer</td></tr><tr><td>record.count</td><td>The number of records in the FlowFile</td></tr></table><h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>Input requirement: </h3>This component requires an incoming relationship.<h3>See Also:</h3><p><a href="../org.apache.nifi.processors.standard.ConvertRecord/index.html">ConvertRecord</a>, <a href="../org.apache.nifi.processors.standard.SplitRecord/index.html">SplitRecord</a>, <a href="../../../nifi-lookup-services-nar/1.5.0/org.apache.nifi.lookup.SimpleKeyValueLookupService/index.html">SimpleKeyValueLookupService</a>, <a href="../../../nifi-lookup-services-nar/1.5.0/org.apache.nifi.lookup.maxmind.IPLookupService/index.html">IPLookupService</a></p></body></html>
\ No newline at end of file
Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.MergeContent/index.html
URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.MergeContent/index.html?rev=1821033&view=auto
==============================================================================
--- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.MergeContent/index.html (added)
+++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.MergeContent/index.html Fri Jan 12 21:00:14 2018
@@ -0,0 +1 @@
+<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>MergeContent</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">MergeContent</h1><h2>Description: </h2><p>Merges a Group of FlowFiles together based on a user-defined strategy and packages them into a single FlowFile. It is recommended that the Processor be configured with only a single incoming connection, as Group of FlowFiles will not be created from FlowFiles in different connections. This processor updates the mime.type attribute as appropriate.</p><h3>Tags: </h3><p>merge, content, correlation, tar, zip, stream, concatenation, archive, flowfile-stream, flowfile-stream-v3</p><h3>Properties: </h3><p>In the list below, the names of required properties appear
in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the <a href="../../../../../html/expression-language-guide.html">NiFi Expression Language</a>.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><strong>Merge Strategy</strong></td><td id="default-value">Bin-Packing Algorithm</td><td id="allowable-values"><ul><li>Bin-Packing Algorithm <img src="../../../../../html/images/iconInfo.png" alt="Generates 'bins' of FlowFiles and fills each bin as full as possible. FlowFiles are placed into a bin based on their size and optionally their attributes (if the <Correlation Attribute> property is set)" title="Generates 'bins' of FlowFiles and fills each bin as full as possible. FlowFiles are placed into a bin based on their size and optionally their attributes (if the <Correlation Attribute>
property is set)"></img></li><li>Defragment <img src="../../../../../html/images/iconInfo.png" alt="Combines fragments that are associated by attributes back into a single cohesive FlowFile. If using this strategy, all FlowFiles must have the attributes <fragment.identifier>, <fragment.count>, and <fragment.index> or alternatively (for backward compatibility purposes) <segment.identifier>, <segment.count>, and <segment.index>. All FlowFiles with the same value for "fragment.identifier" will be grouped together. All FlowFiles in this group must have the same value for the "fragment.count" attribute. All FlowFiles in this group must have a unique value for the "fragment.index" attribute between 0 and the value of the "fragment.count" attribute." title="Combines fragments that are associated by attributes back into a single cohesive FlowFile. If using this strategy, all FlowFiles must have the attributes <fr
agment.identifier>, <fragment.count>, and <fragment.index> or alternatively (for backward compatibility purposes) <segment.identifier>, <segment.count>, and <segment.index>. All FlowFiles with the same value for "fragment.identifier" will be grouped together. All FlowFiles in this group must have the same value for the "fragment.count" attribute. All FlowFiles in this group must have a unique value for the "fragment.index" attribute between 0 and the value of the "fragment.count" attribute."></img></li></ul></td><td id="description">Specifies the algorithm used to merge content. The 'Defragment' algorithm combines fragments that are associated by attributes back into a single cohesive FlowFile. The 'Bin-Packing Algorithm' generates a FlowFile populated by arbitrarily chosen FlowFiles</td></tr><tr><td id="name"><strong>Merge Format</strong></td><td id="default-value">Binary Concatenation</td><td id="allowable-va
lues"><ul><li>TAR <img src="../../../../../html/images/iconInfo.png" alt="A bin of FlowFiles will be combined into a single TAR file. The FlowFiles' <path> attribute will be used to create a directory in the TAR file if the <Keep Paths> property is set to true; otherwise, all FlowFiles will be added at the root of the TAR file. If a FlowFile has an attribute named <tar.permissions> that is 3 characters, each between 0-7, that attribute will be used as the TAR entry's 'mode'." title="A bin of FlowFiles will be combined into a single TAR file. The FlowFiles' <path> attribute will be used to create a directory in the TAR file if the <Keep Paths> property is set to true; otherwise, all FlowFiles will be added at the root of the TAR file. If a FlowFile has an attribute named <tar.permissions> that is 3 characters, each between 0-7, that attribute will be used as the TAR entry's 'mode'."></img></li><li>ZIP <img src="../../../../../html/images/iconInfo.p
ng" alt="A bin of FlowFiles will be combined into a single ZIP file. The FlowFiles' <path> attribute will be used to create a directory in the ZIP file if the <Keep Paths> property is set to true; otherwise, all FlowFiles will be added at the root of the ZIP file. The <Compression Level> property indicates the ZIP compression to use." title="A bin of FlowFiles will be combined into a single ZIP file. The FlowFiles' <path> attribute will be used to create a directory in the ZIP file if the <Keep Paths> property is set to true; otherwise, all FlowFiles will be added at the root of the ZIP file. The <Compression Level> property indicates the ZIP compression to use."></img></li><li>FlowFile Stream, v3 <img src="../../../../../html/images/iconInfo.png" alt="A bin of FlowFiles will be combined into a single Version 3 FlowFile Stream" title="A bin of FlowFiles will be combined into a single Version 3 FlowFile Stream"></img></li><li>FlowFile Stream, v2 <i
mg src="../../../../../html/images/iconInfo.png" alt="A bin of FlowFiles will be combined into a single Version 2 FlowFile Stream" title="A bin of FlowFiles will be combined into a single Version 2 FlowFile Stream"></img></li><li>FlowFile Tar, v1 <img src="../../../../../html/images/iconInfo.png" alt="A bin of FlowFiles will be combined into a single Version 1 FlowFile Package" title="A bin of FlowFiles will be combined into a single Version 1 FlowFile Package"></img></li><li>Binary Concatenation <img src="../../../../../html/images/iconInfo.png" alt="The contents of all FlowFiles will be concatenated together into a single FlowFile" title="The contents of all FlowFiles will be concatenated together into a single FlowFile"></img></li><li>Avro <img src="../../../../../html/images/iconInfo.png" alt="The Avro contents of all FlowFiles will be concatenated together into a single FlowFile" title="The Avro contents of all FlowFiles will be concatenated together into a single FlowFile"></i
mg></li></ul></td><td id="description">Determines the format that will be used to merge the content.</td></tr><tr><td id="name"><strong>Attribute Strategy</strong></td><td id="default-value">Keep Only Common Attributes</td><td id="allowable-values"><ul><li>Keep Only Common Attributes <img src="../../../../../html/images/iconInfo.png" alt="Any attribute that is not the same on all FlowFiles in a bin will be dropped. Those that are the same across all FlowFiles will be retained." title="Any attribute that is not the same on all FlowFiles in a bin will be dropped. Those that are the same across all FlowFiles will be retained."></img></li><li>Keep All Unique Attributes <img src="../../../../../html/images/iconInfo.png" alt="Any attribute that has the same value for all FlowFiles in a bin, or has no value for a FlowFile, will be kept. For example, if a bin consists of 3 FlowFiles and 2 of them have a value of 'hello' for the 'greeting' attribute and the third FlowFile has no 'greeting' a
ttribute then the outbound FlowFile will get a 'greeting' attribute with the value 'hello'." title="Any attribute that has the same value for all FlowFiles in a bin, or has no value for a FlowFile, will be kept. For example, if a bin consists of 3 FlowFiles and 2 of them have a value of 'hello' for the 'greeting' attribute and the third FlowFile has no 'greeting' attribute then the outbound FlowFile will get a 'greeting' attribute with the value 'hello'."></img></li></ul></td><td id="description">Determines which FlowFile attributes should be added to the bundle. If 'Keep All Unique Attributes' is selected, any attribute on any FlowFile that gets bundled will be kept unless its value conflicts with the value from another FlowFile. If 'Keep Only Common Attributes' is selected, only the attributes that exist on all FlowFiles in the bundle, with the same value, will be preserved.</td></tr><tr><td id="name">Correlation Attribute Name</td><td id="default-value"></td><td id="allowable-val
ues"></td><td id="description">If specified, like FlowFiles will be binned together, where 'like FlowFiles' means FlowFiles that have the same value for this Attribute. If not specified, FlowFiles are bundled by the order in which they are pulled from the queue.<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name"><strong>Metadata Strategy</strong></td><td id="default-value">Do Not Merge Uncommon Metadata</td><td id="allowable-values"><ul><li>Use First Metadata <img src="../../../../../html/images/iconInfo.png" alt="For any input format that supports metadata (Avro, e.g.), the metadata for the first FlowFile in the bin will be set on the output FlowFile." title="For any input format that supports metadata (Avro, e.g.), the metadata for the first FlowFile in the bin will be set on the output FlowFile."></img></li><li>Keep Only Common Metadata <img src="../../../../../html/images/iconInfo.png" alt="For any input format that supports metadata (Avro, e.g.),
any FlowFile whose metadata values match those of the first FlowFile, any additional metadata will be dropped but the FlowFile will be merged. Any FlowFile whose metadata values do not match those of the first FlowFile in the bin will not be merged." title="For any input format that supports metadata (Avro, e.g.), any FlowFile whose metadata values match those of the first FlowFile, any additional metadata will be dropped but the FlowFile will be merged. Any FlowFile whose metadata values do not match those of the first FlowFile in the bin will not be merged."></img></li><li>Do Not Merge Uncommon Metadata <img src="../../../../../html/images/iconInfo.png" alt="For any input format that supports metadata (Avro, e.g.), any FlowFile whose metadata values do not match those of the first FlowFile in the bin will not be merged." title="For any input format that supports metadata (Avro, e.g.), any FlowFile whose metadata values do not match those of the first FlowFile in the bin will not
be merged."></img></li><li>Ignore Metadata <img src="../../../../../html/images/iconInfo.png" alt="Ignores (does not transfer, compare, etc.) any metadata from a FlowFile whose content supports embedded metadata." title="Ignores (does not transfer, compare, etc.) any metadata from a FlowFile whose content supports embedded metadata."></img></li></ul></td><td id="description">For FlowFiles whose input format supports metadata (Avro, e.g.), this property determines which metadata should be added to the bundle. If 'Use First Metadata' is selected, the metadata keys/values from the first FlowFile to be bundled will be used. If 'Keep Only Common Metadata' is selected, only the metadata that exists on all FlowFiles in the bundle, with the same value, will be preserved. If 'Ignore Metadata' is selected, no metadata is transferred to the outgoing bundled FlowFile. If 'Do Not Merge Uncommon Metadata' is selected, any FlowFile whose metadata values do not match those of the first bundled Flow
File will not be merged.</td></tr><tr><td id="name"><strong>Minimum Number of Entries</strong></td><td id="default-value">1</td><td id="allowable-values"></td><td id="description">The minimum number of files to include in a bundle</td></tr><tr><td id="name">Maximum Number of Entries</td><td id="default-value">1000</td><td id="allowable-values"></td><td id="description">The maximum number of files to include in a bundle. If not specified, there is no maximum.</td></tr><tr><td id="name"><strong>Minimum Group Size</strong></td><td id="default-value">0 B</td><td id="allowable-values"></td><td id="description">The minimum size of for the bundle</td></tr><tr><td id="name">Maximum Group Size</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The maximum size for the bundle. If not specified, there is no maximum.</td></tr><tr><td id="name">Max Bin Age</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The maximum age of a Bin
that will trigger a Bin to be complete. Expected format is <duration> <time unit> where <duration> is a positive integer and time unit is one of seconds, minutes, hours</td></tr><tr><td id="name"><strong>Maximum number of Bins</strong></td><td id="default-value">5</td><td id="allowable-values"></td><td id="description">Specifies the maximum number of bins that can be held in memory at any one time</td></tr><tr><td id="name"><strong>Delimiter Strategy</strong></td><td id="default-value">Filename</td><td id="allowable-values"><ul><li>Filename <img src="../../../../../html/images/iconInfo.png" alt="The values of Header, Footer, and Demarcator will be retrieved from the contents of a file" title="The values of Header, Footer, and Demarcator will be retrieved from the contents of a file"></img></li><li>Text <img src="../../../../../html/images/iconInfo.png" alt="The values of Header, Footer, and Demarcator will be specified as property values" title="The values of Head
er, Footer, and Demarcator will be specified as property values"></img></li></ul></td><td id="description">Determines if Header, Footer, and Demarcator should point to files containing the respective content, or if the values of the properties should be used as the content.</td></tr><tr><td id="name">Header</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Filename specifying the header to use. If not specified, no header is supplied. This property is valid only when using the binary-concatenation merge strategy; otherwise, it is ignored.<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name">Footer</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Filename specifying the footer to use. If not specified, no footer is supplied. This property is valid only when using the binary-concatenation merge strategy; otherwise, it is ignored.<br/><strong>Supports Expression Language: true</strong></t
d></tr><tr><td id="name">Demarcator</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">Filename specifying the demarcator to use. If not specified, no demarcator is supplied. This property is valid only when using the binary-concatenation merge strategy; otherwise, it is ignored.<br/><strong>Supports Expression Language: true</strong></td></tr><tr><td id="name"><strong>Compression Level</strong></td><td id="default-value">1</td><td id="allowable-values"><ul><li>0</li><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li></ul></td><td id="description">Specifies the compression level to use when using the Zip Merge Format; if not using the Zip Merge Format, this value is ignored</td></tr><tr><td id="name"><strong>Keep Path</strong></td><td id="default-value">false</td><td id="allowable-values"><ul><li>true</li><li>false</li></ul></td><td id="description">If using the Zip or Tar Merge Format, specifies whether or not
the FlowFiles' paths should be included in their entry names; if using other merge strategy, this value is ignored</td></tr><tr><td id="name">Tar Modified Time</td><td id="default-value">${file.lastModifiedTime}</td><td id="allowable-values"></td><td id="description">If using the Tar Merge Format, specifies if the Tar entry should store the modified timestamp either by expression (e.g. ${file.lastModifiedTime} or static value, both of which must match the ISO8601 format 'yyyy-MM-dd'T'HH:mm:ssZ'; if using other merge strategy or left blank, this value is ignored<br/><strong>Supports Expression Language: true</strong></td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>failure</td><td>If the bundle cannot be created, all FlowFiles that would have been used to created the bundle will be transferred to failure</td></tr><tr><td>original</td><td>The FlowFiles that were used to create the bundle</td></tr><tr><td>merged</td><t
d>The FlowFile containing the merged content</td></tr></table><h3>Reads Attributes: </h3><table id="reads-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>fragment.identifier</td><td>Applicable only if the <Merge Strategy> property is set to Defragment. All FlowFiles with the same value for this attribute will be bundled together.</td></tr><tr><td>fragment.index</td><td>Applicable only if the <Merge Strategy> property is set to Defragment. This attribute indicates the order in which the fragments should be assembled. This attribute must be present on all FlowFiles when using the Defragment Merge Strategy and must be a unique (i.e., unique across all FlowFiles that have the same value for the "fragment.identifier" attribute) integer between 0 and the value of the fragment.count attribute. If two or more FlowFiles have the same value for the "fragment.identifier" attribute and the same value for the "fragment.index" attribute, the behavior of this Processor is
undefined.</td></tr><tr><td>fragment.count</td><td>Applicable only if the <Merge Strategy> property is set to Defragment. This attribute must be present on all FlowFiles with the same value for the fragment.identifier attribute. All FlowFiles in the same bundle must have the same value for this attribute. The value of this attribute indicates how many FlowFiles should be expected in the given bundle.</td></tr><tr><td>segment.original.filename</td><td>Applicable only if the <Merge Strategy> property is set to Defragment. This attribute must be present on all FlowFiles with the same value for the fragment.identifier attribute. All FlowFiles in the same bundle must have the same value for this attribute. The value of this attribute will be used for the filename of the completed merged FlowFile.</td></tr><tr><td>tar.permissions</td><td>Applicable only if the <Merge Format> property is set to TAR. The value of this attribute must be 3 characters; each character must be
in the range 0 to 7 (inclusive) and indicates the file permissions that should be used for the FlowFile's TAR entry. If this attribute is missing or has an invalid value, the default value of 644 will be used</td></tr></table><h3>Writes Attributes: </h3><table id="writes-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>filename</td><td>When more than 1 file is merged, the filename comes from the segment.original.filename attribute. If that attribute does not exist in the source FlowFiles, then the filename is set to the number of nanoseconds matching system time. Then a filename extension may be applied:if Merge Format is TAR, then the filename will be appended with .tar, if Merge Format is ZIP, then the filename will be appended with .zip, if Merge Format is FlowFileStream, then the filename will be appended with .pkg</td></tr><tr><td>merge.count</td><td>The number of FlowFiles that were merged into this bundle</td></tr><tr><td>merge.bin.age</td><td>The age of the bin
, in milliseconds, when it was merged and output. Effectively this is the greatest amount of time that any FlowFile in this bundle remained waiting in this processor before it was output</td></tr></table><h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>Input requirement: </h3>This component requires an incoming relationship.<h3>See Also:</h3><p><a href="../org.apache.nifi.processors.standard.SegmentContent/index.html">SegmentContent</a>, <a href="../org.apache.nifi.processors.standard.MergeRecord/index.html">MergeRecord</a></p></body></html>
\ No newline at end of file
Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.MergeRecord/additionalDetails.html
URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.MergeRecord/additionalDetails.html?rev=1821033&view=auto
==============================================================================
--- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.MergeRecord/additionalDetails.html (added)
+++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.MergeRecord/additionalDetails.html Fri Jan 12 21:00:14 2018
@@ -0,0 +1,229 @@
+<!DOCTYPE html>
+<html lang="en">
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+ <head>
+ <meta charset="utf-8" />
+ <title>MergeRecord</title>
+
+ <link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css" />
+ </head>
+
+ <body>
+
+ <h3>Introduction</h3>
+ <p>
+ The MergeRecord Processor allows the user to take many FlowFiles that consist of record-oriented data (any data format for which there is
+ a Record Reader available) and combine the FlowFiles into one larger FlowFile. This may be preferable before pushing the data to a downstream
+ system that prefers larger batches of data, such as HDFS, or in order to improve performance of a NiFi flow by reducing the number of FlowFiles
+ that flow through the system (thereby reducing the contention placed on the FlowFile Repository, Provenance Repository, Content Repository, and
+ FlowFile Queues).
+ </p>
+
+ <p>
+ The Processor creates several 'bins' to put the FlowFiles in. The maximum number of bins to use is set to 5 by default, but this can be changed
+ by updating the value of the <Maximum number of Bins> property. The number of bins is bound in order to avoid running out of Java heap space.
+ Note: while the contents of a FlowFile are stored in the Content Repository and not in the Java heap space, the Processor must hold the FlowFile
+ objects themselves in memory. As a result, these FlowFiles with their attributes can potentially take up a great deal of heap space and cause
+ OutOfMemoryError's to be thrown. In order to avoid this, if you expect to merge many small FlowFiles together, it is advisable to instead use a
+ MergeContent that merges no more than say 1,000 FlowFiles into a bundle and then use a second MergeContent to merges these small bundles into
+ larger bundles. For example, to merge 1,000,000 FlowFiles together, use MergeRecord that uses a <Maximum Number of Records> of 1,000 and route the
+ "merged" Relationship to a second MergeRecord that also sets the <Maximum Number of Records> to 1,000. The second MergeRecord will then merge 1,000 bundles
+ of 1,000, which in effect produces bundles of 1,000,000.
+ </p>
+
+
+
+ <h3>How FlowFiles are Binned</h3>
+ <p>
+ How the Processor determines which bin to place a FlowFile in depends on a few different configuration options. Firstly, the Merge Strategy
+ is considered. The Merge Strategy can be set to one of two options: Bin Packing Algorithm, or Defragment. When the goal is to simply combine
+ smaller FlowFiles into one larger FlowFiles, the Bin Packing Algorithm should be used. This algorithm picks a bin based on whether or not the FlowFile
+ can fit in the bin according to its size and the <Maximum Bin Size> property and whether or not the FlowFile is 'like' the other FlowFiles in
+ the bin. What it means for two FlowFiles to be 'like FlowFiles' is discussed at the end of this section.
+ </p>
+
+ <p>
+ The "Defragment" Merge Strategy can be used when records need to be explicitly assigned to the same bin. For example, if data is split apart using
+ the SplitRecord Processor, each 'split' can be processed independently and later merged back together using this Processor with the
+ Merge Strategy set to Defragment. In order for FlowFiles to be added to the same bin when using this configuration, the FlowFiles must have the same
+ value for the "fragment.identifier" attribute. Each FlowFile with the same identifier must also have the same value for the "fragment.count" attribute
+ (which indicates how many FlowFiles belong in the bin) and a unique value for the "fragment.index" attribute so that the FlowFiles can be ordered
+ correctly.
+ </p>
+
+ <p>
+ In order to be added to the same bin, two FlowFiles must be 'like FlowFiles.' In order for two FlowFiles to be like FlowFiles, they must have the same
+ schema, and if the <Correlation Attribute Name> property is set, they must have the same value for the specified attribute. For example, if the
+ <Correlation Attribute Name> is set to "filename" then two FlowFiles must have the same value for the "filename" attribute in order to be binned
+ together. If more than one attribute is needed in order to correlate two FlowFiles, it is recommended to use an UpdateAttribute processor before the
+ MergeRecord processor and combine the attributes. For example, if the goal is to bin together two FlowFiles only if they have the same value for the
+ "abc" attribute and the "xyz" attribute, then we could accomplish this by using UpdateAttribute and adding a property with name "correlation.attribute"
+ and a value of "abc=${abc},xyz=${xyz}" and then setting MergeRecord's <Correlation Attribute Name> property to "correlation.attribute".
+ </p>
+
+ <p>
+ It is often useful to bin together only Records that have the same value for some field. For example, if we have point-of-sale data, perhaps the desire
+ is to bin together records that belong to the same store, as identified by the 'storeId' field. This can be accomplished by making use of the PartitionRecord
+ Processor ahead of MergeRecord. This Processor will allow one or more fields to be configured as the partitioning criteria and will create attributes for those
+ corresponding values. An UpdateAttribute processor could then be used, if necessary, to combine multiple attributes into a single correlation attribute,
+ as described above. See documentation for those processors for more details.
+ </p>
+
+
+
+ <h3>When a Bin is Merged</h3>
+ <p>
+ Above, we discussed how a bin is chosen for a given FlowFile. Once a bin has been created and FlowFiles added to it, we must have some way to determine
+ when a bin is "full" so that we can bin those FlowFiles together into a "merged" FlowFile. There are a few criteria that are used to make a determination as
+ to whether or not a bin should be merged.
+ </p>
+
+ <p>
+ If the <Merge Strategy> property is set to "Bin Packing Algorithm" then then the following rules will be evaluated.
+ Firstly, in order for a bin to be full, both of the thresholds specified by the <Minimum Bin Size> and the <Minimum Number of Records> properties
+ must be satisfied. If one of these properties is not set, then it is ignored. Secondly, if either the <Maximum Bin Size> or the <Maximum Number of
+ Records> property is reached, then the bin is merged. That is, both of the minimum values must be reached but only one of the maximum values need be reached.
+ Note that the <Maximum Number of Records> property is a "soft limit," meaning that all records in a given input FlowFile will be added to the same bin, and
+ as a result the number of records may exceed the maximum configured number of records. Once this happens, though, no more Records will be added to that same bin
+ from another FlowFile.
+ If the <Max Bin Age> is reached for a bin, then the FlowFiles in that bin will be merged, <b>even if</b> the minimum bin size and minimum number of records
+ have not yet been met. Finally, if the maximum number of bins have been created (as specified by the <Maximum number of Bins> property), and some input FlowFiles
+ cannot fit into any of the existing bins, then the oldest bin will be merged to make room. This is done because otherwise we would not be able to add any
+ additional FlowFiles to the existing bins and would have to wait until the Max Bin Age is reached (if ever) in order to merge any FlowFiles.
+ </p>
+
+ <p>
+ If the <Merge Strategy> property is set to "Defragment" then a bin is full only when the number of FlowFiles in the bin is equal to the number specified
+ by the "fragment.count" attribute of one of the FlowFiles in the bin. All FlowFiles that have this attribute must have the same value for this attribute,
+ or else they will be routed to the "failure" relationship. It is not necessary that all FlowFiles have this value, but at least one FlowFile in the bin must have
+ this value or the bin will never be complete. If all of the necessary FlowFiles are not binned together by the point at which the bin times amount
+ (as specified by the <Max Bin Age> property), then the FlowFiles will all be routed to the 'failure' relationship instead of being merged together.
+ </p>
+
+ <p>
+ Once a bin is merged into a single FlowFile, it can sometimes be useful to understand why exactly the bin was merged when it was. For example, if the maximum number
+ of allowable bins is reached, a merged FlowFile may consist of far fewer records than expected. In order to help understand the behavior, the Processor will emit
+ a JOIN Provenance Events when creating the merged FlowFile, and the JOIN event will include in it a "Details" field that explains why the bin was merged when it was.
+ For example, the event will indicate "Records Merged due to: Bin is full" if the bin reached its minimum thresholds and no more subsequent FlowFiles were able to be
+ added to it. Or it may indicate "Records Merged due to: Maximum number of bins has been exceeded" if the bin was merged due to the configured maximum number of bins
+ being filled and needing to free up space for a new bin.
+ </p>
+
+
+ <h3>When a Failure Occurs</h3>
+ <p>
+ When a bin is filled, the Processor is responsible for merging together all of the records in those FlowFiles into a single FlowFile. If the Processor fails
+ to do so for any reason (for example, a Record cannot be read from an input FlowFile), then all of the FlowFiles in that bin are routed to the 'failure'
+ Relationship. The Processor does not skip the single problematic FlowFile and merge the others. This behavior was chosen because of two different considerations.
+ Firstly, without those problematic records, the bin may not truly be full, as the minimum bin size may not be reached without those records.
+ Secondly, and more importantly, if the problematic FlowFile contains 100 "good" records before the problematic ones, those 100 records would already have been
+ written to the "merged" FlowFile. We cannot un-write those records. If we were to then send those 100 records on and route the problematic FlowFile to 'failure'
+ then in a situation where the "failure" relationship is eventually routed back to MergeRecord, we could end up continually duplicating those 100 successfully
+ processed records.
+ </p>
+
+
+
+ <h2>Examples</h2>
+
+ <p>
+ To better understand how this Processor works, we will lay out a few examples. For the sake of simplicity of these examples, we will use CSV-formatted data and
+ write the merged data as CSV-formatted data, but
+ the format of the data is not really relevant, as long as there is a Record Reader that is capable of reading the data and a Record Writer capable of writing
+ the data in the desired format.
+ </p>
+
+
+
+ <h3>Example 1 - Batching Together Many Small FlowFiles</h3>
+
+ <p>
+ When we want to batch together many small FlowFiles in order to create one larger FlowFile, we will accomplish this by using the "Bin Packing Algorithm"
+ Merge Strategy. The idea here is to bundle together as many FlowFiles as we can within our minimum and maximum number of records and bin size.
+ Consider that we have the following properties set:
+ </p>
+
+<table>
+ <tr>
+ <th>Property Name</th>
+ <th>Property Value</th>
+ </tr>
+ <tr>
+ <td>Merge Strategy</td>
+ <td>Bin Packing Algorithm</td>
+ </tr>
+ <tr>
+ <td>Minimum Number of Records</td>
+ <td>3</td>
+ </tr>
+ <tr>
+ <td>Maximum Number of Records</td>
+ <td>5</td>
+ </tr>
+</table>
+
+ <p>
+ Also consider that we have the following data on the queue, with the schema indicating a Name and an Age field:
+ </p>
+
+<table>
+ <tr>
+ <th>FlowFile ID</th>
+ <th>FlowFile Contents</th>
+ </tr>
+ <tr>
+ <td>1</td>
+ <td>Mark, 33</td>
+ </tr>
+ <tr>
+ <td>2</td>
+ <td>John, 45<br />Jane, 43</td>
+ </tr>
+ <tr>
+ <td>3</td>
+ <td>Jake, 3</td>
+ </tr>
+ <tr>
+ <td>4</td>
+ <td>Jan, 2</td>
+ </tr>
+</table>
+
+ <p>
+ In this, because we have not configured a Correlation Attribute, and because all FlowFiles have the same schema, the Processor
+ will attempt to add all of these FlowFiles to the same bin. Because the Minimum Number of Records is 3 and the Maximum Number of Records is 5,
+ all of the FlowFiles will be added to the same bin. The output, then, is a single FlowFile with the following content:
+ </p>
+
+<code>
+<pre>
+Mark, 33
+John, 45
+Jane, 43
+Jake, 3
+Jan, 2
+</pre>
+</code>
+
+ <p>
+ When the Processor runs, it will bin all of the FlowFiles that it can get from the queue. After that, it will merge any bin that is "full enough."
+ So if we had only 3 FlowFiles on the queue, those 3 would have been added, and a new bin would have been created in the next iteration, once the
+ 4th FlowFile showed up. However, if we had 8 FlowFiles queued up, only 5 would have been added to the first bin. The other 3 would have been added
+ to a second bin, and that bin would then be merged since it reached the minimum threshold of 3 also.
+ </p>
+
+ </body>
+</html>
\ No newline at end of file
Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.MergeRecord/index.html
URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.MergeRecord/index.html?rev=1821033&view=auto
==============================================================================
--- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.MergeRecord/index.html (added)
+++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.MergeRecord/index.html Fri Jan 12 21:00:14 2018
@@ -0,0 +1 @@
+<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"></meta><title>MergeRecord</title><link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"></link></head><script type="text/javascript">window.onload = function(){if(self==top) { document.getElementById('nameHeader').style.display = "inherit"; } }</script><body><h1 id="nameHeader" style="display: none;">MergeRecord</h1><h2>Description: </h2><p>This Processor merges together multiple record-oriented FlowFiles into a single FlowFile that contains all of the Records of the input FlowFiles. This Processor works by creating 'bins' and then adding FlowFiles to these bins until they are full. Once a bin is full, all of the FlowFiles will be combined into a single output FlowFile, and that FlowFile will be routed to the 'merged' Relationship. A bin will consist of potentially many 'like FlowFiles'. In order for two FlowFiles to be considered 'like FlowFiles', they must have the same Schema (as identified b
y the Record Reader) and, if the <Correlation Attribute Name> property is set, the same value for the specified attribute. See Processor Usage and Additional Details for more information.</p><p><a href="additionalDetails.html">Additional Details...</a></p><h3>Tags: </h3><p>merge, record, content, correlation, stream, event</p><h3>Properties: </h3><p>In the list below, the names of required properties appear in <strong>bold</strong>. Any other properties (not in bold) are considered optional. The table also indicates any default values.</p><table id="properties"><tr><th>Name</th><th>Default Value</th><th>Allowable Values</th><th>Description</th></tr><tr><td id="name"><strong>Record Reader</strong></td><td id="default-value"></td><td id="allowable-values"><strong>Controller Service API: </strong><br/>RecordReaderFactory<br/><strong>Implementations: </strong><a href="../../../nifi-record-serialization-services-nar/1.5.0/org.apache.nifi.csv.CSVReader/index.html">CSVReader</a><br/>
<a href="../../../nifi-scripting-nar/1.5.0/org.apache.nifi.record.script.ScriptedReader/index.html">ScriptedReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.5.0/org.apache.nifi.grok.GrokReader/index.html">GrokReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.5.0/org.apache.nifi.json.JsonTreeReader/index.html">JsonTreeReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.5.0/org.apache.nifi.avro.AvroReader/index.html">AvroReader</a><br/><a href="../../../nifi-record-serialization-services-nar/1.5.0/org.apache.nifi.json.JsonPathReader/index.html">JsonPathReader</a></td><td id="description">Specifies the Controller Service to use for reading incoming data</td></tr><tr><td id="name"><strong>Record Writer</strong></td><td id="default-value"></td><td id="allowable-values"><strong>Controller Service API: </strong><br/>RecordSetWriterFactory<br/><strong>Implementations: </strong><a href="../../../nifi-record-serialization-
services-nar/1.5.0/org.apache.nifi.text.FreeFormTextRecordSetWriter/index.html">FreeFormTextRecordSetWriter</a><br/><a href="../../../nifi-record-serialization-services-nar/1.5.0/org.apache.nifi.csv.CSVRecordSetWriter/index.html">CSVRecordSetWriter</a><br/><a href="../../../nifi-record-serialization-services-nar/1.5.0/org.apache.nifi.avro.AvroRecordSetWriter/index.html">AvroRecordSetWriter</a><br/><a href="../../../nifi-record-serialization-services-nar/1.5.0/org.apache.nifi.json.JsonRecordSetWriter/index.html">JsonRecordSetWriter</a><br/><a href="../../../nifi-scripting-nar/1.5.0/org.apache.nifi.record.script.ScriptedRecordSetWriter/index.html">ScriptedRecordSetWriter</a></td><td id="description">Specifies the Controller Service to use for writing out the records</td></tr><tr><td id="name"><strong>Merge Strategy</strong></td><td id="default-value">Bin-Packing Algorithm</td><td id="allowable-values"><ul><li>Bin-Packing Algorithm <img src="../../../../../html/images/iconInfo.png" alt
="Generates 'bins' of FlowFiles and fills each bin as full as possible. FlowFiles are placed into a bin based on their size and optionally their attributes (if the <Correlation Attribute> property is set)" title="Generates 'bins' of FlowFiles and fills each bin as full as possible. FlowFiles are placed into a bin based on their size and optionally their attributes (if the <Correlation Attribute> property is set)"></img></li><li>Defragment <img src="../../../../../html/images/iconInfo.png" alt="Combines fragments that are associated by attributes back into a single cohesive FlowFile. If using this strategy, all FlowFiles must have the attributes <fragment.identifier> and <fragment.count>. All FlowFiles with the same value for "fragment.identifier" will be grouped together. All FlowFiles in this group must have the same value for the "fragment.count" attribute. The ordering of the Records that are output is not guaranteed." title="Combines f
ragments that are associated by attributes back into a single cohesive FlowFile. If using this strategy, all FlowFiles must have the attributes <fragment.identifier> and <fragment.count>. All FlowFiles with the same value for "fragment.identifier" will be grouped together. All FlowFiles in this group must have the same value for the "fragment.count" attribute. The ordering of the Records that are output is not guaranteed."></img></li></ul></td><td id="description">Specifies the algorithm used to merge records. The 'Defragment' algorithm combines fragments that are associated by attributes back into a single cohesive FlowFile. The 'Bin-Packing Algorithm' generates a FlowFile populated by arbitrarily chosen FlowFiles</td></tr><tr><td id="name">Correlation Attribute Name</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">If specified, two FlowFiles will be binned together only if they have the same value for this Attribut
e. If not specified, FlowFiles are bundled by the order in which they are pulled from the queue.</td></tr><tr><td id="name"><strong>Attribute Strategy</strong></td><td id="default-value">Keep Only Common Attributes</td><td id="allowable-values"><ul><li>Keep Only Common Attributes <img src="../../../../../html/images/iconInfo.png" alt="Any attribute that is not the same on all FlowFiles in a bin will be dropped. Those that are the same across all FlowFiles will be retained." title="Any attribute that is not the same on all FlowFiles in a bin will be dropped. Those that are the same across all FlowFiles will be retained."></img></li><li>Keep All Unique Attributes <img src="../../../../../html/images/iconInfo.png" alt="Any attribute that has the same value for all FlowFiles in a bin, or has no value for a FlowFile, will be kept. For example, if a bin consists of 3 FlowFiles and 2 of them have a value of 'hello' for the 'greeting' attribute and the third FlowFile has no 'greeting' attri
bute then the outbound FlowFile will get a 'greeting' attribute with the value 'hello'." title="Any attribute that has the same value for all FlowFiles in a bin, or has no value for a FlowFile, will be kept. For example, if a bin consists of 3 FlowFiles and 2 of them have a value of 'hello' for the 'greeting' attribute and the third FlowFile has no 'greeting' attribute then the outbound FlowFile will get a 'greeting' attribute with the value 'hello'."></img></li></ul></td><td id="description">Determines which FlowFile attributes should be added to the bundle. If 'Keep All Unique Attributes' is selected, any attribute on any FlowFile that gets bundled will be kept unless its value conflicts with the value from another FlowFile. If 'Keep Only Common Attributes' is selected, only the attributes that exist on all FlowFiles in the bundle, with the same value, will be preserved.</td></tr><tr><td id="name"><strong>Minimum Number of Records</strong></td><td id="default-value">1</td><td id="
allowable-values"></td><td id="description">The minimum number of records to include in a bin</td></tr><tr><td id="name">Maximum Number of Records</td><td id="default-value">1000</td><td id="allowable-values"></td><td id="description">The maximum number of Records to include in a bin. This is a 'soft limit' in that if a FlowFIle is added to a bin, all records in that FlowFile will be added, so this limit may be exceeded by up to the number of records in the last input FlowFile.</td></tr><tr><td id="name"><strong>Minimum Bin Size</strong></td><td id="default-value">0 B</td><td id="allowable-values"></td><td id="description">The minimum size of for the bin</td></tr><tr><td id="name">Maximum Bin Size</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The maximum size for the bundle. If not specified, there is no maximum. This is a 'soft limit' in that if a FlowFile is added to a bin, all records in that FlowFile will be added, so this limit may be excee
ded by up to the number of bytes in last input FlowFile.</td></tr><tr><td id="name">Max Bin Age</td><td id="default-value"></td><td id="allowable-values"></td><td id="description">The maximum age of a Bin that will trigger a Bin to be complete. Expected format is <duration> <time unit> where <duration> is a positive integer and time unit is one of seconds, minutes, hours</td></tr><tr><td id="name"><strong>Maximum Number of Bins</strong></td><td id="default-value">10</td><td id="allowable-values"></td><td id="description">Specifies the maximum number of bins that can be held in memory at any one time. This number should not be smaller than the maximum number of conurrent threads for this Processor, or the bins that are created will often consist only of a single incoming FlowFile.</td></tr></table><h3>Relationships: </h3><table id="relationships"><tr><th>Name</th><th>Description</th></tr><tr><td>failure</td><td>If the bundle cannot be created, all FlowFiles that wou
ld have been used to created the bundle will be transferred to failure</td></tr><tr><td>original</td><td>The FlowFiles that were used to create the bundle</td></tr><tr><td>merged</td><td>The FlowFile containing the merged records</td></tr></table><h3>Reads Attributes: </h3><table id="reads-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>fragment.identifier</td><td>Applicable only if the <Merge Strategy> property is set to Defragment. All FlowFiles with the same value for this attribute will be bundled together.</td></tr><tr><td>fragment.count</td><td>Applicable only if the <Merge Strategy> property is set to Defragment. This attribute must be present on all FlowFiles with the same value for the fragment.identifier attribute. All FlowFiles in the same bundle must have the same value for this attribute. The value of this attribute indicates how many FlowFiles should be expected in the given bundle.</td></tr></table><h3>Writes Attributes: </h3><table id="write
s-attributes"><tr><th>Name</th><th>Description</th></tr><tr><td>record.count</td><td>The merged FlowFile will have a 'record.count' attribute indicating the number of records that were written to the FlowFile.</td></tr><tr><td>mime.type</td><td>The MIME Type indicated by the Record Writer</td></tr><tr><td>merge.count</td><td>The number of FlowFiles that were merged into this bundle</td></tr><tr><td>merge.bin.age</td><td>The age of the bin, in milliseconds, when it was merged and output. Effectively this is the greatest amount of time that any FlowFile in this bundle remained waiting in this processor before it was output</td></tr><tr><td><Attributes from Record Writer></td><td>Any Attribute that the configured Record Writer returns will be added to the FlowFile.</td></tr></table><h3>State management: </h3>This component does not store state.<h3>Restricted: </h3>This component is not restricted.<h3>Input requirement: </h3>This component requires an incoming relationship.<h3>See
Also:</h3><p><a href="../org.apache.nifi.processors.standard.MergeContent/index.html">MergeContent</a>, <a href="../org.apache.nifi.processors.standard.SplitRecord/index.html">SplitRecord</a>, <a href="../org.apache.nifi.processors.standard.PartitionRecord/index.html">PartitionRecord</a></p></body></html>
\ No newline at end of file