[jira] [Commented] (NIFI-7320) HDFS Content Repository

["Joe Witt (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17075870#comment-17075870 ] 

Joe Witt commented on NIFI-7320:
--------------------------------

removed fix version.

this would likely need to occur as a profile activated build.  different hdfs libs create compatibility issues are mindblowingly absurd on dependencies.

the L&N for this and its impact on the build and config and all that...needs to be deeply considered and dealt with.

will be very interesting to see the performance implications of this but with the slab allocation style of the content repo there is some reason to believe it could do well.

> HDFS Content Repository
> -----------------------
>
>                 Key: NIFI-7320
>                 URL: https://issues.apache.org/jira/browse/NIFI-7320
>             Project: Apache NiFi
>          Issue Type: New Feature
>          Components: Extensions
>    Affects Versions: 1.12.0
>            Reporter: Andrew Glowacki
>            Priority: Major
>              Labels: ready-to-commit
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> This issue is for a new content repository that stores FlowFile content in HDFS instead of the local disk. While storage capacity for a NiFi cluster can be expanded by adding more nodes, this doesn't make sense if the only reason is to increase content repository storage or I/O capacity.
> In addition to just storing all content in HDFS, the repository can also:
>  * Store content in the local disk as normal, and fallback to HDFS when the local disk is full
>  * Store content in HDFS but fallback to a local disk if a failure occurs.
>  * Store content in the local disk as normal, and only store archived content in HDFS for longer retention.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)