NiFi and Spark Streaming

["Savage, Edward (ISE)" <Ed...@euro.ngc.com>]

Hi,

I have been reading your blog: https://blogs.apache.org/nifi/entry/stream_processing_nifi_and_spark.

This has been a useful resource for learning a new technology. So thanks for putting it together.
Firstly I believe there is a typo. The code when creating the receiver passes in an object called "ClientConfig". However from the code above I believe it is meant to be simply "config".

Secondly when trying to run your example on my own machine I am getting issues when submitting the job to Spark. It moans about not finding the "NiFiReceiver" class. How do I add this external dependency to spark?

Thanks in advance for any help you can offer,
Edd


EDWARD SAVAGE | Software Engineer, Cyber | NORTHROP GRUMMAN

Northrop Grumman
Cyber Division
95, Promenade
Cheltenham,
GL50 1HH

Email: Edward.savage@euro.ngc.com<ma...@euro.ngc.com>

RE: NiFi and Spark Streaming

[Mark Payne <ma...@hotmail.com>]

Ed,

Thanks for your interest!

You're definitely right - ClientConfig should be "config". I have updated the code on the blog post. Thanks for pointing that out.

To include the Receiver in your application, you'll need the following dependency in your application:

  <dependency>
    <groupId>org.apache.nifi</groupId>
    <artifactId>nifi-spark-receiver</artifactId>
    <version>0.1.0-incubating</version>
  </dependency>

Then, you should create an "uber jar" that you can deploy (in Maven it's usually done with something like the following snippet):

	<build>
		<plugins>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-shade-plugin</artifactId>
				<executions>
					<execution>
						<phase>package</phase>
						<goals>
							<goal>shade</goal>
						</goals>
					</execution>
				</executions>
			</plugin>
		</plugins>
	</build>
You should then be able to use spark-deploy or whatever mechanism you usually use to deploy your applications to Spark.

Thanks
-Mark



----------------------------------------
> From: Edward.Savage@euro.ngc.com
> To: dev@nifi.incubator.apache.org
> Subject: NiFi and Spark Streaming
> Date: Tue, 26 May 2015 15:04:36 +0000
>
> Hi,
>
> I have been reading your blog: https://blogs.apache.org/nifi/entry/stream_processing_nifi_and_spark.
>
> This has been a useful resource for learning a new technology. So thanks for putting it together.
> Firstly I believe there is a typo. The code when creating the receiver passes in an object called "ClientConfig". However from the code above I believe it is meant to be simply "config".
>
> Secondly when trying to run your example on my own machine I am getting issues when submitting the job to Spark. It moans about not finding the "NiFiReceiver" class. How do I add this external dependency to spark?
>
> Thanks in advance for any help you can offer,
> Edd
>
>
> EDWARD SAVAGE | Software Engineer, Cyber | NORTHROP GRUMMAN
>
> Northrop Grumman
> Cyber Division
> 95, Promenade
> Cheltenham,
> GL50 1HH
>
> Email: Edward.savage@euro.ngc.com<ma...@euro.ngc.com>
>