Nifi Output port Use case

["Aditya Gaurav (agaurav)" <ag...@cisco.com>]

Hi,

We are building a system to consume an AVRO file for ingest into HDFS. We are planning to use MAPR-STREAM for the same and found Nifi interesting for handling input requests.

I wanted to check if we can put a file to a Nifi port which can be read via a Spark job?

Pls let me know.


Thanks
Aditya

Re: Nifi Output port Use case

[Andre <an...@fucs.org>]

Aditya,

MapR Stream is Kafka compatible so you could in theory just push from NiFi
into the MapR cluster and get Spark to consume from there. However, if you
want to bypass MapR streams then you can use Output ports as documented
here:

https://blogs.apache.org/nifi/entry/stream_processing_nifi_and_spark

and

https://community.hortonworks.com/articles/12708/nifi-feeding-data-to-spark-streaming.html

If I recall correctly, when using direct Spark to NiFi communications, the
Spark job will behave like a NiFi site 2 site peer (be mindful of access
controls) and consume data from there.

If what you want is to upload data from the Spark job into NiFi then you
once again have a few options one of them (easiest one?) is configuring
NiFi to read straight out of MapR-FS and send data into HDFS.

Cheers



On Tue, Mar 7, 2017 at 9:32 PM, Aditya Gaurav (agaurav) <ag...@cisco.com>
wrote:

> Hi,
>
> We are building a system to consume an AVRO file for ingest into HDFS. We
> are planning to use MAPR-STREAM for the same and found Nifi interesting for
> handling input requests.
>
> I wanted to check if we can put a file to a Nifi port which can be read
> via a Spark job?
>
> Pls let me know.
>
>
> Thanks
> Aditya
>