You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Rajib Mandal <m_...@hotmail.com> on 2016/06/28 12:13:13 UTC

Big Data Ingest

Hi all,


We have files (size of each files  600GB- 700 GB ).  We want to load (bulk upload)these files into HDFS.

We are planning to choose Apache NiFi to ingest high volume of data.

Could you please advise if Apache NiFi is the right technology for  the high volume  of data  ingest?

If we use hadoop fs -put commands to load (bulk upload)these files into HDFS, do you foresee  any performance differnce between hadoop fs -put  and Apache NiFi?

Also we have couple of questions here?

  *   What  is the difference between NIFI and PUT - in terms of architecture, performance, scalability, high availability, fault tolerance ?
  *   Do we have any comparison metrics for the following items -supported features, unsupported features, limitations, free GA version and availability for commercial purpose etc?
  *   What Hadoop distributions are supported by NIFI?

Thanks & regards,
Rajib