You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by shankhamajumdar <sh...@lexmark.com> on 2016/11/16 08:16:20 UTC
NiFi Spark Processor
Hi,
I am trying to use custom Spark Nifi processor. I have downloaded the
processor from https://github.com/qiansl127/nifi-spark-bundle and then
executed the maven build. After the maven build I have copied
nifi-spark-nar-0.3.0.nar inside NiFi lib folder and restarted the NiFi
server which is not working.
Please advice.
Regards,
Shankha
--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-Spark-Processor-tp13899.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: NiFi Spark Processor
Posted by Oleg Zhurakousky <oz...@hortonworks.com>.
Shankha
I know it may not always be all that obvious, but have you started the OutputPort? Also, have you defined ‘nifi.remote.input.socket.port’ property in wifi.properties?
Also, unless you’ve already discovered it there is an excellent blog (https://blogs.apache.org/nifi/entry/stream_processing_nifi_and_spark) from Mark on this which covers every little detail, so hope that will help.
Cheers
Oleg
On Nov 18, 2016, at 12:29 AM, shankhamajumdar <sh...@lexmark.com>> wrote:
Hi joey,
I am trying the POC in different way now.
1. GetFile processor which is getting the files from the local drive.
2. Connecting the processor to the NiFi Output Port.
3. Written a Spark streaming Job which is connecting to the NiFi Output Port
to stream the data.
The Spark Job is able to connect the NiFi Output Port successfully. But not
able to get any data in spark though in the NiFi the data has been queued
from GetFile processor. I have checked the NiFi Data Provenance of the NiFi
Output Port but no data is showing there as well where as GetFile to Output
Port connector is showing the data has been queued.
Regards,
Shankha
--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-Spark-Processor-tp13899p13913.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com<http://nabble.com>.
Re: NiFi Spark Processor
Posted by shankhamajumdar <sh...@lexmark.com>.
Hi joey,
I am trying the POC in different way now.
1. GetFile processor which is getting the files from the local drive.
2. Connecting the processor to the NiFi Output Port.
3. Written a Spark streaming Job which is connecting to the NiFi Output Port
to stream the data.
The Spark Job is able to connect the NiFi Output Port successfully. But not
able to get any data in spark though in the NiFi the data has been queued
from GetFile processor. I have checked the NiFi Data Provenance of the NiFi
Output Port but no data is showing there as well where as GetFile to Output
Port connector is showing the data has been queued.
Regards,
Shankha
--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-Spark-Processor-tp13899p13913.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: NiFi Spark Processor
Posted by Joey Frazee <jo...@icloud.com>.
Shanka,
It's hard to tell without more details, which you can probably find in the logs, but the pom.xml in that project has not been updated for any of the recent (< 1y) releases of NiFi (0.6, 0.7, 1.0), so you're probably going to have to change the version properties and, if using NiFi 1.0, change ProcessorLog to ComponentLog in the actual processor implementation.
If you want to check the logs for relevant messages you can find these in ./logs/nifi-app.log and ./logs/nifi-bootstrap.log.
I'll add that, from a stylistic point of view, you might be better off looking into one the Spark job servers (spark-jobserver, livy) and using NiFi's HTTP processors to submit jobs that way. It should be more manageable and then you won't need this custom processor.
-joey
> On Nov 16, 2016, at 2:16 AM, shankhamajumdar <sh...@lexmark.com> wrote:
>
> Hi,
>
> I am trying to use custom Spark Nifi processor. I have downloaded the
> processor from https://github.com/qiansl127/nifi-spark-bundle and then
> executed the maven build. After the maven build I have copied
> nifi-spark-nar-0.3.0.nar inside NiFi lib folder and restarted the NiFi
> server which is not working.
>
> Please advice.
>
> Regards,
> Shankha
>
>
>
> --
> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-Spark-Processor-tp13899.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.