You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Bryan Bende <bb...@gmail.com> on 2016/05/19 16:08:55 UTC
Re: Spark & NiFi question
Hi Conrad,
I think there are a couple of things at play here...
One is that the SSL properties need to be set on the
SiteToSiteClientBuilder, rather than through system properties. There
should be methods to set the keystore and other values.
In a secured NiFi instance, the certificate you are authenticating with
(the keystore used by the s2s client) would need to have an account in
NiFi, and would need to have access to the output port.
If you attempt to make a request with that cert, and then you go into the
NiFi UI as another user, you should be able to go into the accounts section
(top right) and approve the account for that certificate.
Then if you stop your output port, right-click and Configure... and from
the Access Controls tab started typing the DN from your cert and add that
user to the Allowed Users list. Hit Apply and started the port again.
We probably need to document this better, or write up an article about it
somewhere.
Let us know if its still not working.
Thanks,
Bryan
On Thu, May 19, 2016 at 11:54 AM, Conrad Crampton <
conrad.crampton@secdata.com> wrote:
> Hi,
> Tried following a couple of blog posts about this [1], [2], but neither of
> these refer to using NiFi in clustered environment with SSL and I suspect
> this is where I am hitting problems (but don’t know where).
>
> The blogs state that using an output port (in the root process group I.e.
> on main canvas) which I have done and tried to connect thus..
>
> System.setProperty("javax.net.ssl.keyStore", "/spark-processor.jks");
> System.setProperty("javax.net.ssl.keyStorePassword", *“******");
> System.setProperty("javax.net.ssl.trustStore", *“*/cacerts.jks");
>
> SiteToSiteClientConfig config = new SiteToSiteClient.Builder()
> .url("https://yarn-cm1.mis-cds.local:9090/nifi")
> .portName("Spark test out")
> .buildConfig();
>
> SparkConf sparkConf = new SparkConf().setMaster("local[2]").setAppName("NiFi Spark Log Processor");
> JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, new Duration(5000));
> JavaReceiverInputDStream<NiFiDataPacket> packetStream = jssc.receiverStream(new NiFiReceiver(config, StorageLevel.MEMORY_ONLY()));
>
> JavaDStream text = packetStream.map(dataPacket -> new String(dataPacket.getContent(), StandardCharsets.UTF_8));
> text.print();
> jssc.start();
> jssc.awaitTermination();
>
> The error I am getting is
>
> 16/05/19 16:39:03 WARN ReceiverSupervisorImpl: Restarting receiver with
> delay 2000 ms: Failed to receive data from NiFi
> java.io.IOException: Server returned HTTP response code: 401 for URL:
> https://yarn-cm1.mis-cds.local:9090/nifi-api/controller
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1889)
> at
> sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1884)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1883)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1456)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
> at
> sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
> at
> org.apache.nifi.remote.util.NiFiRestApiUtil.getController(NiFiRestApiUtil.java:69)
> at
> org.apache.nifi.remote.client.socket.EndpointConnectionPool.refreshRemoteInfo(EndpointConnectionPool.java:891)
> at
> org.apache.nifi.remote.client.socket.EndpointConnectionPool.getPortIdentifier(EndpointConnectionPool.java:878)
> at
> org.apache.nifi.remote.client.socket.EndpointConnectionPool.getOutputPortIdentifier(EndpointConnectionPool.java:862)
> at
> org.apache.nifi.remote.client.socket.SocketClient.getPortIdentifier(SocketClient.java:81)
> at
> org.apache.nifi.remote.client.socket.SocketClient.createTransaction(SocketClient.java:123)
> at
> org.apache.nifi.spark.NiFiReceiver$ReceiveRunnable.run(NiFiReceiver.java:149)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Server returned HTTP response code: 401
> for URL: https://yarn-cm1.mis-cds.local:9090/nifi-api/controller
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1839)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
> at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
> at
> sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
> at
> org.apache.nifi.remote.util.NiFiRestApiUtil.getController(NiFiRestApiUtil.java:66)
> ... 7 more
>
> Any pointers would be helpful in getting this working. I don’t know if I
> have to set up a remote process group with the output port (not sure how
> this works), or what. When I go to
> https://yarn-cm1.mis-cds.local:9090/nifi-api/controller in the browser, I
> get an access denied error.
> I have created keystore and signed by the RootCA used to sign all the self
> signed certs for the cluster.
>
> Running 0.6.1, 6 node cluster.
>
> Thanks
> Conrad
>
> [1[ -
> https://community.hortonworks.com/articles/12708/nifi-feeding-data-to-spark-streaming.html
> [2] - https://blogs.apache.org/nifi/entry/stream_processing_nifi_and_spark
>
>
> SecureData, combating cyber threats
>
> ------------------------------
>
> The information contained in this message or any of its attachments may be
> privileged and confidential and intended for the exclusive use of the
> intended recipient. If you are not the intended recipient any disclosure,
> reproduction, distribution or other dissemination or use of this
> communications is strictly prohibited. The views expressed in this email
> are those of the individual and not necessarily of SecureData Europe Ltd.
> Any prices quoted are only valid if followed up by a formal written quote.
>
> SecureData Europe Limited. Registered in England & Wales 04365896.
> Registered Address: SecureData House, Hermitage Court, Hermitage Lane,
> Maidstone, Kent, ME16 9NT
>
Re: Spark & NiFi question
Posted by Joe Witt <jo...@gmail.com>.
I am very proud to be part of a community with threads like this!
On Fri, May 20, 2016 at 9:34 AM, Bryan Bende <bb...@gmail.com> wrote:
> Hi Conrad,
>
> Sorry this has been so challenging to setup. After trying it out myself, I
> believe the problem you ran into when you didn't set the System properties
> is actually a legit bug in the SiteToSiteClient...
> I wrote it up in this JIRA [1], but the short answer is that it never uses
> those properties to create an SSLContext and ends up trying to make a normal
> connection to the https end-point, and thus ends up failing.
>
> I made some quick code changes to work around the above issue, and
> eventually got it working using Storm, since I don't have spark streaming
> setup. Here is what I did...
>
> In conf/nifi.properties I set the following:
>
> # Site to Site properties
> nifi.remote.input.socket.host=
> nifi.remote.input.socket.port=8088
> nifi.remote.input.secure=true
>
> # web properties #
> nifi.web.war.directory=./lib
> nifi.web.http.host=
> nifi.web.http.port=8080
> nifi.web.https.host=
> nifi.web.https.port=8443
> nifi.web.jetty.working.directory=./work/jetty
> nifi.web.jetty.threads=200
>
> # security properties #
> nifi.sensitive.props.key=
> nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
> nifi.sensitive.props.provider=BC
>
> nifi.security.keystore=/path/to/nifi/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/localhost-ks.jks
> nifi.security.keystoreType=JKS
> nifi.security.keystorePasswd=localtest
> nifi.security.keyPasswd=localtest
> nifi.security.truststore=/path/to/nifi//nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/localhost-ts.jks
> nifi.security.truststoreType=JKS
> nifi.security.truststorePasswd=localtest
>
>
> I started NiFi and used the unsecure url (http://localhost:8080/nifi) to
> create a flow with GenerateFlowFile -> Output Port named "Data for Storm".
>
> There is an example Storm topology that is part of the code base [2], so I
> started with that, and modified the SiteToSiteClientConfig:
>
> final SiteToSiteClientConfig inputConfig = new SiteToSiteClient.Builder()
> .url("https://localhost:8443/nifi")
> .portName("Data for Storm")
>
> .keystoreFilename("/path/to/nifi//nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/localhost-ks.jks")
> .keystoreType(KeystoreType.JKS)
> .keystorePass("localtest")
>
> .truststoreFilename("/path/to/nifi//nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/localhost-ts.jks")
> .truststoreType(KeystoreType.JKS)
> .truststorePass("localtest")
> .buildConfig();
>
> Now of course setting those properties only worked because of local changes
> I made, but after that I got a 401 Unauthorized when I ran the topology,
> which I think was where you were originally at.
>
> I went back into the unsecure url and checked the users section and didn't
> see anything, so I think I was incorrect that it automatically creates a
> pending account.
> I then put that localhost cert into my browser (I already had it as p12 from
> something else) and I went to https://localhost:8443/nifi and it prompted
> for the account request and I submitted it.
> Went back to the unsecure UI and approved the account with role NiFi, then
> went to the Output Port and gave access to the localhost user.
>
> After that it was working... I think since you were already at the point of
> getting the 401, if you can just get the account created for that
> certificate and the access controls on the ports, then it should probably
> work using the System properties as a work around for now, but not totally
> sure.
>
> Again, sorry for all the confusion, definitely planning to address the JIRA
> soon.
>
> -Bryan
>
> [1] https://issues.apache.org/jira/browse/NIFI-1907
> [2]
> https://github.com/apache/nifi/blob/e12a79ea929a222a93fd64bfc63382441e31060f/nifi-external/nifi-storm-spout/src/test/java/org/apache/nifi/storm/NiFiStormTopology.java
>
>
> On Fri, May 20, 2016 at 4:16 AM, Conrad Crampton
> <co...@secdata.com> wrote:
>>
>> Thanks for the pointers Bryan, however wrt your first suggestion. I tried
>> without setting SSL properties on System properties and get an unable to
>> find ssl path error – this gets resolved by doing as I have done (but of
>> course this may be a red herring). I initially tried setting on site builder
>> but got the same error as below – it appears to make no difference as to
>> what is logged in the nifi-users.log if I include SSL props on site builder
>> or not, I get the same error viz:
>>
>> 2016-05-20 08:59:47,082 INFO [NiFi Web Server-29590180]
>> o.a.n.w.s.NiFiAuthenticationFilter Attempting request for
>> (<CN=spark-processor.m.xxx, OU=Development, O=Secure Data Europe Ltd,
>> L=Maidstone, ST=Kent, C=GB>) GET
>> https://yarn-cm1.m.xxxx:9090/nifi-api/controller (source ip: xx.xx.xx.1)
>> 2016-05-20 08:59:47,082 INFO [NiFi Web Server-29494759]
>> o.a.n.w.s.NiFiAuthenticationFilter Attempting request for
>> (<CN=spark-processor.m.xxx, OU=Development, O=Secure Data Europe Ltd,
>> L=Maidstone, ST=Kent, C=GB>) GET
>> https://yarn-cm1.m.xxx:9090/nifi-api/controller (source ip: xx.xx.xx.1)
>> 2016-05-20 08:59:47,083 INFO [NiFi Web Server-29590180]
>> o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Unable to
>> verify access for CN=spark-processor.m.xxx, OU=Development, O=Secure Data
>> Europe Ltd, L=Maidstone, ST=Kent, C=GB
>>
>> I am using self signed certs if that makes a difference (but these work
>> fine on across the cluster). I am not seeing my spark user appear in the
>> list of users to grant access.
>>
>> I have turned on debug for ssl to see if that is throwing up anything but
>> nothing appears obvious – here is the snipet that I would expect errors to
>> be shown from that log.
>>
>> ... no IV derived for this protocol
>> %% Server resumed [Session-4, TLS_RSA_WITH_AES_128_CBC_SHA256]
>> NiFi Receiver, READ: TLSv1.2 Change Cipher Spec, length = 1
>> NiFi Receiver, READ: TLSv1.2 Handshake, length = 80
>> *** Finished
>> verify_data: { 109, 126, 134, 14, 33, 110, 224, 83, 198, 116, 54, 228 }
>> ***
>> NiFi Receiver, WRITE: TLSv1.2 Change Cipher Spec, length = 1
>> *** Finished
>> verify_data: { 83, 120, 49, 158, 181, 136, 127, 219, 30, 194, 58, 167 }
>> ***
>> NiFi Receiver, WRITE: TLSv1.2 Handshake, length = 80
>> NiFi Receiver, WRITE: TLSv1.2 Application Data, length = 240
>>
>> I don’t really know enough about certificates and how client java apps
>> would use them wrt to the host name/ ip address etc. of details is included
>> in them. The nifi-user.log is showing access from a specific IP address
>> which clearly doesn’t match the CN details in the cert. Just clutching at
>> straws here!
>>
>> Any other suggestions?
>>
>> Thanks
>> Conrad
>>
>> From: Bryan Bende <bb...@gmail.com>
>> Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> Date: Thursday, 19 May 2016 at 17:08
>> To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> Subject: Re: Spark & NiFi question
>>
>> Hi Conrad,
>>
>> I think there are a couple of things at play here...
>>
>> One is that the SSL properties need to be set on the
>> SiteToSiteClientBuilder, rather than through system properties. There should
>> be methods to set the keystore and other values.
>>
>> In a secured NiFi instance, the certificate you are authenticating with
>> (the keystore used by the s2s client) would need to have an account in NiFi,
>> and would need to have access to the output port.
>> If you attempt to make a request with that cert, and then you go into the
>> NiFi UI as another user, you should be able to go into the accounts section
>> (top right) and approve the account for that certificate.
>>
>> Then if you stop your output port, right-click and Configure... and from
>> the Access Controls tab started typing the DN from your cert and add that
>> user to the Allowed Users list. Hit Apply and started the port again.
>>
>> We probably need to document this better, or write up an article about it
>> somewhere.
>>
>> Let us know if its still not working.
>>
>> Thanks,
>>
>> Bryan
>>
>>
>> On Thu, May 19, 2016 at 11:54 AM, Conrad Crampton
>> <co...@secdata.com> wrote:
>>>
>>> Hi,
>>> Tried following a couple of blog posts about this [1], [2], but neither
>>> of these refer to using NiFi in clustered environment with SSL and I suspect
>>> this is where I am hitting problems (but don’t know where).
>>>
>>> The blogs state that using an output port (in the root process group I.e.
>>> on main canvas) which I have done and tried to connect thus..
>>>
>>> System.setProperty("javax.net.ssl.keyStore", "/spark-processor.jks");
>>> System.setProperty("javax.net.ssl.keyStorePassword", “*****");
>>> System.setProperty("javax.net.ssl.trustStore", “/cacerts.jks");
>>>
>>> SiteToSiteClientConfig config = new SiteToSiteClient.Builder()
>>> .url("https://yarn-cm1.mis-cds.local:9090/nifi")
>>> .portName("Spark test out")
>>> .buildConfig();
>>>
>>> SparkConf sparkConf = new
>>> SparkConf().setMaster("local[2]").setAppName("NiFi Spark Log Processor");
>>> JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, new
>>> Duration(5000));
>>> JavaReceiverInputDStream<NiFiDataPacket> packetStream =
>>> jssc.receiverStream(new NiFiReceiver(config, StorageLevel.MEMORY_ONLY()));
>>>
>>> JavaDStream text = packetStream.map(dataPacket -> new
>>> String(dataPacket.getContent(), StandardCharsets.UTF_8));
>>> text.print();
>>> jssc.start();
>>> jssc.awaitTermination();
>>>
>>> The error I am getting is
>>>
>>> 16/05/19 16:39:03 WARN ReceiverSupervisorImpl: Restarting receiver with
>>> delay 2000 ms: Failed to receive data from NiFi
>>> java.io.IOException: Server returned HTTP response code: 401 for URL:
>>> https://yarn-cm1.mis-cds.local:9090/nifi-api/controller
>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>>> at
>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>>> at
>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
>>> at
>>> sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1889)
>>> at
>>> sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1884)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at
>>> sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1883)
>>> at
>>> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1456)
>>> at
>>> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
>>> at
>>> sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
>>> at
>>> org.apache.nifi.remote.util.NiFiRestApiUtil.getController(NiFiRestApiUtil.java:69)
>>> at
>>> org.apache.nifi.remote.client.socket.EndpointConnectionPool.refreshRemoteInfo(EndpointConnectionPool.java:891)
>>> at
>>> org.apache.nifi.remote.client.socket.EndpointConnectionPool.getPortIdentifier(EndpointConnectionPool.java:878)
>>> at
>>> org.apache.nifi.remote.client.socket.EndpointConnectionPool.getOutputPortIdentifier(EndpointConnectionPool.java:862)
>>> at
>>> org.apache.nifi.remote.client.socket.SocketClient.getPortIdentifier(SocketClient.java:81)
>>> at
>>> org.apache.nifi.remote.client.socket.SocketClient.createTransaction(SocketClient.java:123)
>>> at
>>> org.apache.nifi.spark.NiFiReceiver$ReceiveRunnable.run(NiFiReceiver.java:149)
>>> at java.lang.Thread.run(Thread.java:745)
>>> Caused by: java.io.IOException: Server returned HTTP response code: 401
>>> for URL: https://yarn-cm1.mis-cds.local:9090/nifi-api/controller
>>> at
>>> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1839)
>>> at
>>> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
>>> at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
>>> at
>>> sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
>>> at
>>> org.apache.nifi.remote.util.NiFiRestApiUtil.getController(NiFiRestApiUtil.java:66)
>>> ... 7 more
>>>
>>> Any pointers would be helpful in getting this working. I don’t know if I
>>> have to set up a remote process group with the output port (not sure how
>>> this works), or what. When I go to
>>> https://yarn-cm1.mis-cds.local:9090/nifi-api/controller in the browser, I
>>> get an access denied error.
>>> I have created keystore and signed by the RootCA used to sign all the
>>> self signed certs for the cluster.
>>>
>>> Running 0.6.1, 6 node cluster.
>>>
>>> Thanks
>>> Conrad
>>>
>>> [1[ -
>>> https://community.hortonworks.com/articles/12708/nifi-feeding-data-to-spark-streaming.html
>>> [2] -
>>> https://blogs.apache.org/nifi/entry/stream_processing_nifi_and_spark
>>>
>>>
>>> SecureData, combating cyber threats
>>>
>>> ________________________________
>>>
>>> The information contained in this message or any of its attachments may
>>> be privileged and confidential and intended for the exclusive use of the
>>> intended recipient. If you are not the intended recipient any disclosure,
>>> reproduction, distribution or other dissemination or use of this
>>> communications is strictly prohibited. The views expressed in this email are
>>> those of the individual and not necessarily of SecureData Europe Ltd. Any
>>> prices quoted are only valid if followed up by a formal written quote.
>>>
>>> SecureData Europe Limited. Registered in England & Wales 04365896.
>>> Registered Address: SecureData House, Hermitage Court, Hermitage Lane,
>>> Maidstone, Kent, ME16 9NT
>>
>>
>>
>>
>> ***This email originated outside SecureData***
>>
>> Click here to report this email as spam.
>
>
Re: Spark & NiFi question
Posted by Bryan Bende <bb...@gmail.com>.
Hi Conrad,
Sorry this has been so challenging to setup. After trying it out myself, I
believe the problem you ran into when you didn't set the System properties
is actually a legit bug in the SiteToSiteClient...
I wrote it up in this JIRA [1], but the short answer is that it never uses
those properties to create an SSLContext and ends up trying to make a
normal connection to the https end-point, and thus ends up failing.
I made some quick code changes to work around the above issue, and
eventually got it working using Storm, since I don't have spark streaming
setup. Here is what I did...
In conf/nifi.properties I set the following:
# Site to Site properties
nifi.remote.input.socket.host=
nifi.remote.input.socket.port=8088
nifi.remote.input.secure=true
# web properties #
nifi.web.war.directory=./lib
nifi.web.http.host=
nifi.web.http.port=8080
nifi.web.https.host=
nifi.web.https.port=8443
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
# security properties #
nifi.sensitive.props.key=
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
nifi.sensitive.props.provider=BC
nifi.security.keystore=/path/to/nifi/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/localhost-ks.jks
nifi.security.keystoreType=JKS
nifi.security.keystorePasswd=localtest
nifi.security.keyPasswd=localtest
nifi.security.truststore=/path/to/nifi//nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/localhost-ts.jks
nifi.security.truststoreType=JKS
nifi.security.truststorePasswd=localtest
I started NiFi and used the unsecure url (http://localhost:8080/nifi) to
create a flow with GenerateFlowFile -> Output Port named "Data for Storm".
There is an example Storm topology that is part of the code base [2], so I
started with that, and modified the SiteToSiteClientConfig:
final SiteToSiteClientConfig inputConfig = new SiteToSiteClient.Builder()
.url("https://localhost:8443/nifi")
.portName("Data for Storm")
.keystoreFilename("/path/to/nifi//nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/localhost-ks.jks")
.keystoreType(KeystoreType.JKS)
.keystorePass("localtest")
.truststoreFilename("/path/to/nifi//nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/localhost-ts.jks")
.truststoreType(KeystoreType.JKS)
.truststorePass("localtest")
.buildConfig();
Now of course setting those properties only worked because of local changes
I made, but after that I got a 401 Unauthorized when I ran the topology,
which I think was where you were originally at.
I went back into the unsecure url and checked the users section and didn't
see anything, so I think I was incorrect that it automatically creates a
pending account.
I then put that localhost cert into my browser (I already had it as p12
from something else) and I went to https://localhost:8443/nifi and it
prompted for the account request and I submitted it.
Went back to the unsecure UI and approved the account with role NiFi, then
went to the Output Port and gave access to the localhost user.
After that it was working... I think since you were already at the point of
getting the 401, if you can just get the account created for that
certificate and the access controls on the ports, then it should probably
work using the System properties as a work around for now, but not totally
sure.
Again, sorry for all the confusion, definitely planning to address the JIRA
soon.
-Bryan
[1] https://issues.apache.org/jira/browse/NIFI-1907
[2]
https://github.com/apache/nifi/blob/e12a79ea929a222a93fd64bfc63382441e31060f/nifi-external/nifi-storm-spout/src/test/java/org/apache/nifi/storm/NiFiStormTopology.java
On Fri, May 20, 2016 at 4:16 AM, Conrad Crampton <
conrad.crampton@secdata.com> wrote:
> Thanks for the pointers Bryan, however wrt your first suggestion. I tried
> without setting SSL properties on System properties and get an unable to
> find ssl path error – this gets resolved by doing as I have done (but of
> course this may be a red herring). I initially tried setting on site
> builder but got the same error as below – it appears to make no difference
> as to what is logged in the nifi-users.log if I include SSL props on site
> builder or not, I get the same error viz:
>
> 2016-05-20 08:59:47,082 INFO [NiFi Web Server-29590180]
> o.a.n.w.s.NiFiAuthenticationFilter Attempting request for
> (<CN=spark-processor.m.xxx, OU=Development, O=Secure Data Europe Ltd,
> L=Maidstone, ST=Kent, C=GB>) GET
> https://yarn-cm1.m.xxxx:9090/nifi-api/controller (source ip: xx.xx.xx.1)
> 2016-05-20 08:59:47,082 INFO [NiFi Web Server-29494759]
> o.a.n.w.s.NiFiAuthenticationFilter Attempting request for
> (<CN=spark-processor.m.xxx, OU=Development, O=Secure Data Europe Ltd,
> L=Maidstone, ST=Kent, C=GB>) GET
> https://yarn-cm1.m.xxx:9090/nifi-api/controller (source ip: xx.xx.xx.1)
> 2016-05-20 08:59:47,083 INFO [NiFi Web Server-29590180]
> o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Unable to
> verify access for CN=spark-processor.m.xxx, OU=Development, O=Secure Data
> Europe Ltd, L=Maidstone, ST=Kent, C=GB
>
> I am using self signed certs if that makes a difference (but these work
> fine on across the cluster). I am not seeing my spark user appear in the
> list of users to grant access.
>
> I have turned on debug for ssl to see if that is throwing up anything but
> nothing appears obvious – here is the snipet that I would expect errors to
> be shown from that log.
>
> ... no IV derived for this protocol
> %% Server resumed [Session-4, TLS_RSA_WITH_AES_128_CBC_SHA256]
> NiFi Receiver, READ: TLSv1.2 Change Cipher Spec, length = 1
> NiFi Receiver, READ: TLSv1.2 Handshake, length = 80
> *** Finished
> verify_data: { 109, 126, 134, 14, 33, 110, 224, 83, 198, 116, 54, 228 }
> ***
> NiFi Receiver, WRITE: TLSv1.2 Change Cipher Spec, length = 1
> *** Finished
> verify_data: { 83, 120, 49, 158, 181, 136, 127, 219, 30, 194, 58, 167 }
> ***
> NiFi Receiver, WRITE: TLSv1.2 Handshake, length = 80
> NiFi Receiver, WRITE: TLSv1.2 Application Data, length = 240
>
> I don’t really know enough about certificates and how client java apps
> would use them wrt to the host name/ ip address etc. of details is included
> in them. The nifi-user.log is showing access from a specific IP address
> which clearly doesn’t match the CN details in the cert. Just clutching at
> straws here!
>
> Any other suggestions?
>
> Thanks
> Conrad
>
> From: Bryan Bende <bb...@gmail.com>
> Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Date: Thursday, 19 May 2016 at 17:08
> To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Subject: Re: Spark & NiFi question
>
> Hi Conrad,
>
> I think there are a couple of things at play here...
>
> One is that the SSL properties need to be set on the
> SiteToSiteClientBuilder, rather than through system properties. There
> should be methods to set the keystore and other values.
>
> In a secured NiFi instance, the certificate you are authenticating with
> (the keystore used by the s2s client) would need to have an account in
> NiFi, and would need to have access to the output port.
> If you attempt to make a request with that cert, and then you go into the
> NiFi UI as another user, you should be able to go into the accounts section
> (top right) and approve the account for that certificate.
>
> Then if you stop your output port, right-click and Configure... and from
> the Access Controls tab started typing the DN from your cert and add that
> user to the Allowed Users list. Hit Apply and started the port again.
>
> We probably need to document this better, or write up an article about it
> somewhere.
>
> Let us know if its still not working.
>
> Thanks,
>
> Bryan
>
>
> On Thu, May 19, 2016 at 11:54 AM, Conrad Crampton <
> conrad.crampton@secdata.com> wrote:
>
>> Hi,
>> Tried following a couple of blog posts about this [1], [2], but neither
>> of these refer to using NiFi in clustered environment with SSL and I
>> suspect this is where I am hitting problems (but don’t know where).
>>
>> The blogs state that using an output port (in the root process group I.e.
>> on main canvas) which I have done and tried to connect thus..
>>
>> System.setProperty("javax.net.ssl.keyStore", "/spark-processor.jks");
>> System.setProperty("javax.net.ssl.keyStorePassword", *“******");
>> System.setProperty("javax.net.ssl.trustStore", *“*/cacerts.jks");
>>
>> SiteToSiteClientConfig config = new SiteToSiteClient.Builder()
>> .url("https://yarn-cm1.mis-cds.local:9090/nifi")
>> .portName("Spark test out")
>> .buildConfig();
>>
>> SparkConf sparkConf = new SparkConf().setMaster("local[2]").setAppName("NiFi Spark Log Processor");
>> JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, new Duration(5000));
>> JavaReceiverInputDStream<NiFiDataPacket> packetStream = jssc.receiverStream(new NiFiReceiver(config, StorageLevel.MEMORY_ONLY()));
>>
>> JavaDStream text = packetStream.map(dataPacket -> new String(dataPacket.getContent(), StandardCharsets.UTF_8));
>> text.print();
>> jssc.start();
>> jssc.awaitTermination();
>>
>> The error I am getting is
>>
>> 16/05/19 16:39:03 WARN ReceiverSupervisorImpl: Restarting receiver with
>> delay 2000 ms: Failed to receive data from NiFi
>> java.io.IOException: Server returned HTTP response code: 401 for URL:
>> https://yarn-cm1.mis-cds.local:9090/nifi-api/controller
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>> at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>> at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
>> at
>> sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1889)
>> at
>> sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1884)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at
>> sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1883)
>> at
>> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1456)
>> at
>> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
>> at
>> sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
>> at
>> org.apache.nifi.remote.util.NiFiRestApiUtil.getController(NiFiRestApiUtil.java:69)
>> at
>> org.apache.nifi.remote.client.socket.EndpointConnectionPool.refreshRemoteInfo(EndpointConnectionPool.java:891)
>> at
>> org.apache.nifi.remote.client.socket.EndpointConnectionPool.getPortIdentifier(EndpointConnectionPool.java:878)
>> at
>> org.apache.nifi.remote.client.socket.EndpointConnectionPool.getOutputPortIdentifier(EndpointConnectionPool.java:862)
>> at
>> org.apache.nifi.remote.client.socket.SocketClient.getPortIdentifier(SocketClient.java:81)
>> at
>> org.apache.nifi.remote.client.socket.SocketClient.createTransaction(SocketClient.java:123)
>> at
>> org.apache.nifi.spark.NiFiReceiver$ReceiveRunnable.run(NiFiReceiver.java:149)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.io.IOException: Server returned HTTP response code: 401
>> for URL: https://yarn-cm1.mis-cds.local:9090/nifi-api/controller
>> at
>> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1839)
>> at
>> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
>> at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
>> at
>> sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
>> at
>> org.apache.nifi.remote.util.NiFiRestApiUtil.getController(NiFiRestApiUtil.java:66)
>> ... 7 more
>>
>> Any pointers would be helpful in getting this working. I don’t know if I
>> have to set up a remote process group with the output port (not sure how
>> this works), or what. When I go to
>> https://yarn-cm1.mis-cds.local:9090/nifi-api/controller in the browser,
>> I get an access denied error.
>> I have created keystore and signed by the RootCA used to sign all the
>> self signed certs for the cluster.
>>
>> Running 0.6.1, 6 node cluster.
>>
>> Thanks
>> Conrad
>>
>> [1[ -
>> https://community.hortonworks.com/articles/12708/nifi-feeding-data-to-spark-streaming.html
>> [2] -
>> https://blogs.apache.org/nifi/entry/stream_processing_nifi_and_spark
>>
>>
>> SecureData, combating cyber threats
>>
>> ------------------------------
>>
>> The information contained in this message or any of its attachments may
>> be privileged and confidential and intended for the exclusive use of the
>> intended recipient. If you are not the intended recipient any disclosure,
>> reproduction, distribution or other dissemination or use of this
>> communications is strictly prohibited. The views expressed in this email
>> are those of the individual and not necessarily of SecureData Europe Ltd.
>> Any prices quoted are only valid if followed up by a formal written quote.
>>
>> SecureData Europe Limited. Registered in England & Wales 04365896.
>> Registered Address: SecureData House, Hermitage Court, Hermitage Lane,
>> Maidstone, Kent, ME16 9NT
>>
>
>
>
> ***This email originated outside SecureData***
>
> Click here
> <https://www.mailcontrol.com/sr/JOj4ovws70LGX2PQPOmvUqa7UuQeNDoM5CPuVUMi!aLghcUmWuJbL8QAhL3vPgRnasXOF8Vdo14NCU1!U1Tbvw==>
> to report this email as spam.
>
Re: Spark & NiFi question
Posted by Bryan Bende <bb...@gmail.com>.
Conrad,
I think the error message is mis-leading a little bit, it says...
"Unable to communicate with yarn-cm1.mis-cds.local:9870 because it requires
Secure Site-to-Site communications, but this instance is not configured for
secure communications"
That statement is saying that your NiFi cluster is configured for secure
site-to-site (which you proved from the debug logs), but that "this
instance" which is actually your Spark streaming job, is not configured for
secure communication.
The reason it thinks your Spark streaming job is not configured for secure
communication is because of the bug I mentioned in the previous email,
where it will never create the SSLContext.
The error message was originally written in the context of two NiFi
instances talking to each other, so it makes more sense in that context.
Perhaps it should be changed to... "this site-to-site client is not
configured for secure communication".
-Bryan
On Mon, May 23, 2016 at 11:04 AM, Conrad Crampton <
conrad.crampton@secdata.com> wrote:
> Hi,
> I don’t know if I’m hitting some bug here but something doesn’t make sense.
> With ssl debug on I get the following
> NiFi Receiver, READ: TLSv1.2 Application Data, length = 1648
> Padded plaintext after DECRYPTION: len = 1648
> 0000: 65 A2 B8 34 DF 20 6B 95 56 88 97 16 7A EC 8F E3 e..4. k.V...z...
> 0010: 48 54 54 50 2F 31 2E 31 20 32 30 30 20 4F 4B 0D HTTP/1.1 200 OK.
> 0020: 0A 44 61 74 65 3A 20 4D 6F 6E 2C 20 32 33 20 4D .Date: Mon, 23 M
> 0030: 61 79 20 32 30 31 36 20 31 34 3A 34 39 3A 33 39 ay 2016 14:49:39
> 0040: 20 47 4D 54 0D 0A 53 65 72 76 65 72 3A 20 4A 65 GMT..Server: Je
> 0050: 74 74 79 28 39 2E 32 2E 31 31 2E 76 32 30 31 35 tty(9.2.11.v2015
> 0060: 30 35 32 39 29 0D 0A 43 61 63 68 65 2D 43 6F 6E 0529)..Cache-Con
> 0070: 74 72 6F 6C 3A 20 70 72 69 76 61 74 65 2C 20 6E trol: private, n
> 0080: 6F 2D 63 61 63 68 65 2C 20 6E 6F 2D 73 74 6F 72 o-cache, no-stor
> 0090: 65 2C 20 6E 6F 2D 74 72 61 6E 73 66 6F 72 6D 0D e, no-transform.
> 00A0: 0A 56 61 72 79 3A 20 41 63 63 65 70 74 2D 45 6E .Vary: Accept-En
> 00B0: 63 6F 64 69 6E 67 2C 20 55 73 65 72 2D 41 67 65 coding, User-Age
> 00C0: 6E 74 0D 0A 44 61 74 65 3A 20 4D 6F 6E 2C 20 32 nt..Date: Mon, 2
> 00D0: 33 20 4D 61 79 20 32 30 31 36 20 31 34 3A 34 39 3 May 2016 14:49
> 00E0: 3A 33 39 20 47 4D 54 0D 0A 43 6F 6E 74 65 6E 74 :39 GMT..Content
> 00F0: 2D 54 79 70 65 3A 20 61 70 70 6C 69 63 61 74 69 -Type: applicati
> 0100: 6F 6E 2F 6A 73 6F 6E 0D 0A 56 61 72 79 3A 20 41 on/json..Vary: A
> 0110: 63 63 65 70 74 2D 45 6E 63 6F 64 69 6E 67 2C 20 ccept-Encoding,
> 0120: 55 73 65 72 2D 41 67 65 6E 74 0D 0A 43 6F 6E 74 User-Agent..Cont
> 0130: 65 6E 74 2D 4C 65 6E 67 74 68 3A 20 31 32 38 35 ent-Length: 1285
> 0140: 0D 0A 0D 0A 7B 22 72 65 76 69 73 69 6F 6E 22 3A ....."revision":
> 0150: 7B 22 63 6C 69 65 6E 74 49 64 22 3A 22 39 34 38 ."clientId":"948
> 0160: 66 62 34 31 33 2D 65 39 37 64 2D 34 32 37 65 2D fb413-e97d-427e-
> 0170: 61 34 38 36 2D 31 31 63 39 65 37 31 63 63 62 62 a486-11c9e71ccbb
> 0180: 32 22 7D 2C 22 63 6F 6E 74 72 6F 6C 6C 65 72 22 2".,"controller"
> 0190: 3A 7B 22 69 64 22 3A 22 31 38 63 38 39 64 32 33 :."id":"18c89d23
> 01A0: 2D 61 35 31 65 2D 34 35 35 38 2D 62 30 31 61 2D -a51e-4558-b01a-
> 01B0: 33 66 36 30 64 66 31 31 63 39 61 64 22 2C 22 6E 3f60df11c9ad","n
> 01C0: 61 6D 65 22 3A 22 4E 69 46 69 20 46 6C 6F 77 22 ame":"NiFi Flow"
> 01D0: 2C 22 63 6F 6D 6D 65 6E 74 73 22 3A 22 22 2C 22 ,"comments":"","
> 01E0: 72 75 6E 6E 69 6E 67 43 6F 75 6E 74 22 3A 31 36 runningCount":16
> 01F0: 34 2C 22 73 74 6F 70 70 65 64 43 6F 75 6E 74 22 4,"stoppedCount"
> 0200: 3A 34 33 2C 22 69 6E 76 61 6C 69 64 43 6F 75 6E :43,"invalidCoun
> 0210: 74 22 3A 31 2C 22 64 69 73 61 62 6C 65 64 43 6F t":1,"disabledCo
> 0220: 75 6E 74 22 3A 30 2C 22 69 6E 70 75 74 50 6F 72 unt":0,"inputPor
> 0230: 74 43 6F 75 6E 74 22 3A 37 2C 22 6F 75 74 70 75 tCount":7,"outpu
> 0240: 74 50 6F 72 74 43 6F 75 6E 74 22 3A 31 2C 22 72 tPortCount":1,"r
> 0250: 65 6D 6F 74 65 53 69 74 65 4C 69 73 74 65 6E 69 emoteSiteListeni
> 0260: 6E 67 50 6F 72 74 22 3A 39 38 37 30 2C 22 73 69 ngPort":9870,"si
> 0270: 74 65 54 6F 53 69 74 65 53 65 63 75 72 65 22 3A teToSiteSecure":
> 0280: 74 72 75 65 2C 22 69 6E 73 74 61 6E 63 65 49 64 true,"instanceId
> 0290: 22 3A 22 30 35 38 30 63 35 31 38 2D 39 62 63 37 ":"0580c518-9bc7
> 02A0: 2D 34 37 38 33 2D 39 32 34 38 2D 35 38 30 61 36 -4783-9248-580a6
> 02B0: 37 34 65 34 33 35 62 22 2C 22 69 6E 70 75 74 50 74e435b","inputP
> 02C0: 6F 72 74 73 22 3A 5B 7B 22 69 64 22 3A 22 33 32 orts":[."id":"32
> 02D0: 37 30 39 33 31 66 2D 64 61 38 35 2D 34 63 34 65 70931f-da85-4c4e
> 02E0: 2D 62 61 65 36 2D 38 63 36 32 37 62 30 39 62 37 -bae6-8c627b09b7
> 02F0: 32 66 22 2C 22 6E 61 6D 65 22 3A 22 48 44 46 53 2f","name":"HDFS
> 0300: 49 6E 63 6F 6D 69 6E 67 22 2C 22 63 6F 6D 6D 65 Incoming","comme
> 0310: 6E 74 73 22 3A 22 22 2C 22 73 74 61 74 65 22 3A nts":"","state":
> 0320: 22 53 54 4F 50 50 45 44 22 7D 2C 7B 22 69 64 22 "STOPPED".,."id"
> 0330: 3A 22 30 39 33 30 63 62 32 63 2D 37 61 38 33 2D :"0930cb2c-7a83-
> 0340: 34 38 36 64 2D 62 62 61 65 2D 38 62 33 30 31 32 486d-bbae-8b3012
> 0350: 64 36 31 39 66 37 22 2C 22 6E 61 6D 65 22 3A 22 d619f7","name":"
> 0360: 50 6F 72 74 20 39 30 39 38 20 49 6E 63 6F 6D 69 Port 9098 Incomi
> 0370: 6E 67 20 53 79 73 6C 6F 67 73 22 2C 22 63 6F 6D ng Syslogs","com
> 0380: 6D 65 6E 74 73 22 3A 22 22 2C 22 73 74 61 74 65 ments":"","state
> 0390: 22 3A 22 52 55 4E 4E 49 4E 47 22 7D 2C 7B 22 69 ":"RUNNING".,."i
> 03A0: 64 22 3A 22 31 34 62 64 32 66 66 35 2D 38 38 36 d":"14bd2ff5-886
> 03B0: 61 2D 34 61 32 39 2D 62 39 39 61 2D 38 64 34 34 a-4a29-b99a-8d44
> 03C0: 65 66 37 38 66 30 31 30 22 2C 22 6E 61 6D 65 22 ef78f010","name"
> 03D0: 3A 22 48 44 46 53 57 65 62 73 65 6E 73 65 53 65 :"HDFSWebsenseSe
> 03E0: 63 75 72 69 74 79 22 2C 22 63 6F 6D 6D 65 6E 74 curity","comment
> 03F0: 73 22 3A 22 22 2C 22 73 74 61 74 65 22 3A 22 53 s":"","state":"S
> 0400: 54 4F 50 50 45 44 22 7D 2C 7B 22 69 64 22 3A 22 TOPPED".,."id":"
> 0410: 33 61 66 30 33 66 66 36 2D 39 62 65 37 2D 33 32 3af03ff6-9be7-32
> 0420: 35 61 2D 61 63 66 33 2D 63 36 62 39 61 37 64 32 5a-acf3-c6b9a7d2
> 0430: 31 36 65 33 22 2C 22 6E 61 6D 65 22 3A 22 50 6F 16e3","name":"Po
> 0440: 72 74 20 39 30 39 39 20 49 6E 63 6F 6D 69 6E 67 rt 9099 Incoming
> 0450: 20 53 79 73 6C 6F 67 73 22 2C 22 63 6F 6D 6D 65 Syslogs","comme
> 0460: 6E 74 73 22 3A 22 22 2C 22 73 74 61 74 65 22 3A nts":"","state":
> 0470: 22 52 55 4E 4E 49 4E 47 22 7D 2C 7B 22 69 64 22 "RUNNING".,."id"
> 0480: 3A 22 65 65 34 31 37 64 35 61 2D 62 64 39 38 2D :"ee417d5a-bd98-
> 0490: 33 32 65 61 2D 61 63 35 38 2D 63 36 32 33 64 66 32ea-ac58-c623df
> 04A0: 35 65 64 64 66 35 22 2C 22 6E 61 6D 65 22 3A 22 5eddf5","name":"
> 04B0: 50 6F 72 74 20 39 31 30 31 20 49 6E 63 6F 6D 69 Port 9101 Incomi
> 04C0: 6E 67 20 53 79 73 6C 6F 67 73 22 2C 22 63 6F 6D ng Syslogs","com
> 04D0: 6D 65 6E 74 73 22 3A 22 22 2C 22 73 74 61 74 65 ments":"","state
> 04E0: 22 3A 22 52 55 4E 4E 49 4E 47 22 7D 2C 7B 22 69 ":"RUNNING".,."i
> 04F0: 64 22 3A 22 39 34 37 30 38 30 61 36 2D 34 65 61 d":"947080a6-4ea
> 0500: 66 2D 33 37 64 37 2D 62 36 32 62 2D 39 37 62 61 f-37d7-b62b-97ba
> 0510: 62 35 37 66 34 64 39 38 22 2C 22 6E 61 6D 65 22 b57f4d98","name"
> 0520: 3A 22 50 6F 72 74 20 39 31 30 30 20 49 6E 63 6F :"Port 9100 Inco
> 0530: 6D 69 6E 67 20 53 79 73 6C 6F 67 73 22 2C 22 63 ming Syslogs","c
> 0540: 6F 6D 6D 65 6E 74 73 22 3A 22 22 2C 22 73 74 61 omments":"","sta
> 0550: 74 65 22 3A 22 52 55 4E 4E 49 4E 47 22 7D 2C 7B te":"RUNNING".,.
> 0560: 22 69 64 22 3A 22 63 33 37 34 35 64 37 65 2D 39 "id":"c3745d7e-9
> 0570: 62 66 66 2D 33 31 31 32 2D 38 65 33 63 2D 39 36 bff-3112-8e3c-96
> 0580: 34 61 66 62 39 63 36 36 37 33 22 2C 22 6E 61 6D 4afb9c6673","nam
> 0590: 65 22 3A 22 50 6F 72 74 20 39 31 30 32 20 49 6E e":"Port 9102 In
> 05A0: 63 6F 6D 69 6E 67 20 53 79 73 6C 6F 67 73 22 2C coming Syslogs",
> 05B0: 22 63 6F 6D 6D 65 6E 74 73 22 3A 22 22 2C 22 73 "comments":"","s
> 05C0: 74 61 74 65 22 3A 22 52 55 4E 4E 49 4E 47 22 7D tate":"RUNNING".
> 05D0: 5D 2C 22 6F 75 74 70 75 74 50 6F 72 74 73 22 3A ],"outputPorts":
> 05E0: 5B 7B 22 69 64 22 3A 22 61 62 38 36 62 37 34 36 [."id":"ab86b746
> 05F0: 2D 37 39 63 33 2D 34 30 31 65 2D 62 35 30 35 2D -79c3-401e-b505-
> 0600: 39 64 39 34 30 35 62 32 32 62 33 31 22 2C 22 6E 9d9405b22b31","n
> 0610: 61 6D 65 22 3A 22 53 70 61 72 6B 20 74 65 73 74 ame":"Spark test
> 0620: 20 6F 75 74 22 2C 22 63 6F 6D 6D 65 6E 74 73 22 out","comments"
> 0630: 3A 22 22 2C 22 73 74 61 74 65 22 3A 22 52 55 4E :"","state":"RUN
> 0640: 4E 49 4E 47 22 7D 5D 7D 7D 15 C4 DA 96 85 23 76 NING".].......#v
> 0650: 2B DB 4B 46 5A 9A DD 4F 9B EF D8 46 70 FF CD EC +.KFZ..O...Fp...
> 0660: 99 19 31 F3 7F CC C1 14 07 06 06 06 06 06 06 06 ..1.............
> 16/05/23 15:49:39 WARN EndpointConnectionPool:
> EndpointConnectionPool[Cluster URL=
> https://yarn-cm1.mis-cds.local:9090/nifi/] Unable to refresh Remote
> Group's peers due to java.io.IOException: Unable to communicate with
> yarn-cm1.mis-cds.local:9870 because it requires Secure Site-to-Site
> communications, but this instance is not configured for secure
> communications
> 16/05/23 15:49:39 WARN EndpointConnectionPool:
> EndpointConnectionPool[Cluster URL=
> https://yarn-cm1.mis-cds.local:9090/nifi/] Unable to refresh Remote
> Group's peers due to java.io.IOException: Unable to communicate with
> yarn-cm1.mis-cds.local:9870 because it requires Secure Site-to-Site
> communications, but this instance is not configured for secure
> communications
> Exception in thread "NiFi Receiver" java.lang.NullPointerException
> at
> org.apache.nifi.spark.NiFiReceiver$ReceiveRunnable.run(NiFiReceiver.java:150)
> at java.lang.Thread.run(Thread.java:745)
>
> Which clearly shows that secure site to site communication is true
> "r
> 0250: 65 6D 6F 74 65 53 69 74 65 4C 69 73 74 65 6E 69 emoteSiteListeni
> 0260: 6E 67 50 6F 72 74 22 3A 39 38 37 30 2C 22 73 69 ngPort":9870,"si
> 0270: 74 65 54 6F 53 69 74 65 53 65 63 75 72 65 22 3A teToSiteSecure":
> 0280: 74 72 75 65 2C 22 69 6E 73 74 61 6E 63 65 49 64 true,”
>
> But the exception thrown looks like it is being coming from line 150 in
> NifiReceiver
>
> Transaction ioe1 = ioe.createTransaction(TransferDirection.RECEIVE);
> DataPacket dataPacket = ioe1.receive(); <—— here,
>
> As a result of attempting to create the transaction on the
> SiteToSiteClient. The docs state that client may have to query the server’s
> RESTful interface which could throw an IOException. Without the full stack
> trace I’m only guessing that the isSecure method is returning false when it
> should be returning true.
>
> Anyone?
> Thanks
> Conrad
>
> From: Conrad Crampton <co...@SecData.com>
> Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Date: Monday, 23 May 2016 at 10:39
>
> To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Subject: SPOOFED: Re: Spark & NiFi question
>
> Hi,
> An update to this but still not working
> I have now set keystore and truststore as system properties, and included
> these as part of the SiteToSiteClientConfig building. I have used a cert
> that I have for one of the servers in my cluster as I know they can
> communicate over ssl with NCM as my 6 node cluster works over ssl and has
> remote ports working (as I read from syslog on a primary server then
> distribute to other via remote ports as suggested somewhere else) .
> When I try now to connect to output port via Spark, I get a
> "EndpointConnectionPool[Cluster URL=
> https://yarn-cm1.mis-cds.local:9090/nifi/] Unable to refresh Remote
> Group's peers due to java.io.IOException: Unable to communicate with
> yarn-cm1.mis-cds.local:9870 because it requires Secure Site-to-Site
> communications, but this instance is not configured for secure
> communications"
> Exception even though I know Secure Site-to-Site communication is working
> (9870 being the port set up for remote s2s comms in nifi.properties), so I
> am now really confused!!
>
> Does the port that I wish to read from need to be set up with remote
> process group (conceptually I’m struggling with how to do this for an
> output port), or is it is sufficient to be ‘just an output port’?
>
> I have this working when connecting to an unsecured (http) instance of
> NiFi running on my laptop with Spark and a standard output port. Does it
> make a difference that my production cluster is a cluster and therefore
> needs setting up differently?
>
> So many questions but I’m stuck now so any suggestions welcome.
> Thanks
> Conrad
>
> From: Conrad Crampton <co...@SecData.com>
> Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Date: Friday, 20 May 2016 at 09:16
> To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Subject: SPOOFED: Re: Spark & NiFi question
>
> Thanks for the pointers Bryan, however wrt your first suggestion. I tried
> without setting SSL properties on System properties and get an unable to
> find ssl path error – this gets resolved by doing as I have done (but of
> course this may be a red herring). I initially tried setting on site
> builder but got the same error as below – it appears to make no difference
> as to what is logged in the nifi-users.log if I include SSL props on site
> builder or not, I get the same error viz:
>
> 2016-05-20 08:59:47,082 INFO [NiFi Web Server-29590180]
> o.a.n.w.s.NiFiAuthenticationFilter Attempting request for
> (<CN=spark-processor.m.xxx, OU=Development, O=Secure Data Europe Ltd,
> L=Maidstone, ST=Kent, C=GB>) GET
> https://yarn-cm1.m.xxxx:9090/nifi-api/controller (source ip: xx.xx.xx.1)
> 2016-05-20 08:59:47,082 INFO [NiFi Web Server-29494759]
> o.a.n.w.s.NiFiAuthenticationFilter Attempting request for
> (<CN=spark-processor.m.xxx, OU=Development, O=Secure Data Europe Ltd,
> L=Maidstone, ST=Kent, C=GB>) GET
> https://yarn-cm1.m.xxx:9090/nifi-api/controller (source ip: xx.xx.xx.1)
> 2016-05-20 08:59:47,083 INFO [NiFi Web Server-29590180]
> o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Unable to
> verify access for CN=spark-processor.m.xxx, OU=Development, O=Secure Data
> Europe Ltd, L=Maidstone, ST=Kent, C=GB
>
> I am using self signed certs if that makes a difference (but these work
> fine on across the cluster). I am not seeing my spark user appear in the
> list of users to grant access.
>
> I have turned on debug for ssl to see if that is throwing up anything but
> nothing appears obvious – here is the snipet that I would expect errors to
> be shown from that log.
>
> ... no IV derived for this protocol
> %% Server resumed [Session-4, TLS_RSA_WITH_AES_128_CBC_SHA256]
> NiFi Receiver, READ: TLSv1.2 Change Cipher Spec, length = 1
> NiFi Receiver, READ: TLSv1.2 Handshake, length = 80
> *** Finished
> verify_data: { 109, 126, 134, 14, 33, 110, 224, 83, 198, 116, 54, 228 }
> ***
> NiFi Receiver, WRITE: TLSv1.2 Change Cipher Spec, length = 1
> *** Finished
> verify_data: { 83, 120, 49, 158, 181, 136, 127, 219, 30, 194, 58, 167 }
> ***
> NiFi Receiver, WRITE: TLSv1.2 Handshake, length = 80
> NiFi Receiver, WRITE: TLSv1.2 Application Data, length = 240
>
> I don’t really know enough about certificates and how client java apps
> would use them wrt to the host name/ ip address etc. of details is included
> in them. The nifi-user.log is showing access from a specific IP address
> which clearly doesn’t match the CN details in the cert. Just clutching at
> straws here!
>
> Any other suggestions?
>
> Thanks
> Conrad
>
> From: Bryan Bende <bb...@gmail.com>
> Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Date: Thursday, 19 May 2016 at 17:08
> To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Subject: Re: Spark & NiFi question
>
> Hi Conrad,
>
> I think there are a couple of things at play here...
>
> One is that the SSL properties need to be set on the
> SiteToSiteClientBuilder, rather than through system properties. There
> should be methods to set the keystore and other values.
>
> In a secured NiFi instance, the certificate you are authenticating with
> (the keystore used by the s2s client) would need to have an account in
> NiFi, and would need to have access to the output port.
> If you attempt to make a request with that cert, and then you go into the
> NiFi UI as another user, you should be able to go into the accounts section
> (top right) and approve the account for that certificate.
>
> Then if you stop your output port, right-click and Configure... and from
> the Access Controls tab started typing the DN from your cert and add that
> user to the Allowed Users list. Hit Apply and started the port again.
>
> We probably need to document this better, or write up an article about it
> somewhere.
>
> Let us know if its still not working.
>
> Thanks,
>
> Bryan
>
>
> On Thu, May 19, 2016 at 11:54 AM, Conrad Crampton <
> conrad.crampton@secdata.com> wrote:
>
>> Hi,
>> Tried following a couple of blog posts about this [1], [2], but neither
>> of these refer to using NiFi in clustered environment with SSL and I
>> suspect this is where I am hitting problems (but don’t know where).
>>
>> The blogs state that using an output port (in the root process group I.e.
>> on main canvas) which I have done and tried to connect thus..
>>
>> System.setProperty("javax.net.ssl.keyStore", "/spark-processor.jks");
>> System.setProperty("javax.net.ssl.keyStorePassword", *“******");
>> System.setProperty("javax.net.ssl.trustStore", *“*/cacerts.jks");
>>
>> SiteToSiteClientConfig config = new SiteToSiteClient.Builder()
>> .url("https://yarn-cm1.mis-cds.local:9090/nifi")
>> .portName("Spark test out")
>> .buildConfig();
>>
>> SparkConf sparkConf = new SparkConf().setMaster("local[2]").setAppName("NiFi Spark Log Processor");
>> JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, new Duration(5000));
>> JavaReceiverInputDStream<NiFiDataPacket> packetStream = jssc.receiverStream(new NiFiReceiver(config, StorageLevel.MEMORY_ONLY()));
>>
>> JavaDStream text = packetStream.map(dataPacket -> new String(dataPacket.getContent(), StandardCharsets.UTF_8));
>> text.print();
>> jssc.start();
>> jssc.awaitTermination();
>>
>> The error I am getting is
>>
>> 16/05/19 16:39:03 WARN ReceiverSupervisorImpl: Restarting receiver with
>> delay 2000 ms: Failed to receive data from NiFi
>> java.io.IOException: Server returned HTTP response code: 401 for URL:
>> https://yarn-cm1.mis-cds.local:9090/nifi-api/controller
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>> at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>> at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
>> at
>> sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1889)
>> at
>> sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1884)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at
>> sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1883)
>> at
>> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1456)
>> at
>> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
>> at
>> sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
>> at
>> org.apache.nifi.remote.util.NiFiRestApiUtil.getController(NiFiRestApiUtil.java:69)
>> at
>> org.apache.nifi.remote.client.socket.EndpointConnectionPool.refreshRemoteInfo(EndpointConnectionPool.java:891)
>> at
>> org.apache.nifi.remote.client.socket.EndpointConnectionPool.getPortIdentifier(EndpointConnectionPool.java:878)
>> at
>> org.apache.nifi.remote.client.socket.EndpointConnectionPool.getOutputPortIdentifier(EndpointConnectionPool.java:862)
>> at
>> org.apache.nifi.remote.client.socket.SocketClient.getPortIdentifier(SocketClient.java:81)
>> at
>> org.apache.nifi.remote.client.socket.SocketClient.createTransaction(SocketClient.java:123)
>> at
>> org.apache.nifi.spark.NiFiReceiver$ReceiveRunnable.run(NiFiReceiver.java:149)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.io.IOException: Server returned HTTP response code: 401
>> for URL: https://yarn-cm1.mis-cds.local:9090/nifi-api/controller
>> at
>> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1839)
>> at
>> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
>> at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
>> at
>> sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
>> at
>> org.apache.nifi.remote.util.NiFiRestApiUtil.getController(NiFiRestApiUtil.java:66)
>> ... 7 more
>>
>> Any pointers would be helpful in getting this working. I don’t know if I
>> have to set up a remote process group with the output port (not sure how
>> this works), or what. When I go to
>> https://yarn-cm1.mis-cds.local:9090/nifi-api/controller in the browser,
>> I get an access denied error.
>> I have created keystore and signed by the RootCA used to sign all the
>> self signed certs for the cluster.
>>
>> Running 0.6.1, 6 node cluster.
>>
>> Thanks
>> Conrad
>>
>> [1[ -
>> https://community.hortonworks.com/articles/12708/nifi-feeding-data-to-spark-streaming.html
>> [2] -
>> https://blogs.apache.org/nifi/entry/stream_processing_nifi_and_spark
>>
>>
>> SecureData, combating cyber threats
>>
>> ------------------------------
>>
>> The information contained in this message or any of its attachments may
>> be privileged and confidential and intended for the exclusive use of the
>> intended recipient. If you are not the intended recipient any disclosure,
>> reproduction, distribution or other dissemination or use of this
>> communications is strictly prohibited. The views expressed in this email
>> are those of the individual and not necessarily of SecureData Europe Ltd.
>> Any prices quoted are only valid if followed up by a formal written quote.
>>
>> SecureData Europe Limited. Registered in England & Wales 04365896.
>> Registered Address: SecureData House, Hermitage Court, Hermitage Lane,
>> Maidstone, Kent, ME16 9NT
>>
>
>
>
> ***This email originated outside SecureData***
>
> Click here <https://www.mailcontrol.com/sr/MZbqvYs5QwJvpeaetUwhCQ==> to
> report this email as spam.
>
Re: Spark & NiFi question
Posted by Conrad Crampton <co...@SecData.com>.
Hi Bryan
Firstly, let me apologise for my constant stream of emails on this that appear not to be taking any of your replies into consideration. I thought no one was looking at it! My email client/ server appears to have stopped letting any emails relating to this thread though even though I get all others in the list! I must appear to be a complete numbnuts! I checked the archive list on the mail-archives website and found all of your posts!
Having been on this mailing list for a while now, I couldn’t quite believe no-one was assisting given the usually brilliant responses (so I definitely concur with Joe’s previous comment) :-)
Anyway, I can’t thank you enough Bryan for confirming that I, in fact, am not going mad and there is a bug here. I will park the work here and wait for 0.7 to be released as I know what I want to do actually works on the Spark end (having proved on a local (insecure) NiFi.
Thanks again,
Conrad
From: Conrad Crampton <co...@SecData.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Monday, 23 May 2016 at 16:04
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Spark & NiFi question
Hi,
I don’t know if I’m hitting some bug here but something doesn’t make sense.
With ssl debug on I get the following
NiFi Receiver, READ: TLSv1.2 Application Data, length = 1648
Padded plaintext after DECRYPTION: len = 1648
0000: 65 A2 B8 34 DF 20 6B 95 56 88 97 16 7A EC 8F E3 e..4. k.V...z...
0010: 48 54 54 50 2F 31 2E 31 20 32 30 30 20 4F 4B 0D HTTP/1.1 200 OK.
0020: 0A 44 61 74 65 3A 20 4D 6F 6E 2C 20 32 33 20 4D .Date: Mon, 23 M
0030: 61 79 20 32 30 31 36 20 31 34 3A 34 39 3A 33 39 ay 2016 14:49:39
0040: 20 47 4D 54 0D 0A 53 65 72 76 65 72 3A 20 4A 65 GMT..Server: Je
0050: 74 74 79 28 39 2E 32 2E 31 31 2E 76 32 30 31 35 tty(9.2.11.v2015
0060: 30 35 32 39 29 0D 0A 43 61 63 68 65 2D 43 6F 6E 0529)..Cache-Con
0070: 74 72 6F 6C 3A 20 70 72 69 76 61 74 65 2C 20 6E trol: private, n
0080: 6F 2D 63 61 63 68 65 2C 20 6E 6F 2D 73 74 6F 72 o-cache, no-stor
0090: 65 2C 20 6E 6F 2D 74 72 61 6E 73 66 6F 72 6D 0D e, no-transform.
00A0: 0A 56 61 72 79 3A 20 41 63 63 65 70 74 2D 45 6E .Vary: Accept-En
00B0: 63 6F 64 69 6E 67 2C 20 55 73 65 72 2D 41 67 65 coding, User-Age
00C0: 6E 74 0D 0A 44 61 74 65 3A 20 4D 6F 6E 2C 20 32 nt..Date: Mon, 2
00D0: 33 20 4D 61 79 20 32 30 31 36 20 31 34 3A 34 39 3 May 2016 14:49
00E0: 3A 33 39 20 47 4D 54 0D 0A 43 6F 6E 74 65 6E 74 :39 GMT..Content
00F0: 2D 54 79 70 65 3A 20 61 70 70 6C 69 63 61 74 69 -Type: applicati
0100: 6F 6E 2F 6A 73 6F 6E 0D 0A 56 61 72 79 3A 20 41 on/json..Vary: A
0110: 63 63 65 70 74 2D 45 6E 63 6F 64 69 6E 67 2C 20 ccept-Encoding,
0120: 55 73 65 72 2D 41 67 65 6E 74 0D 0A 43 6F 6E 74 User-Agent..Cont
0130: 65 6E 74 2D 4C 65 6E 67 74 68 3A 20 31 32 38 35 ent-Length: 1285
0140: 0D 0A 0D 0A 7B 22 72 65 76 69 73 69 6F 6E 22 3A ....."revision":
0150: 7B 22 63 6C 69 65 6E 74 49 64 22 3A 22 39 34 38 ."clientId":"948
0160: 66 62 34 31 33 2D 65 39 37 64 2D 34 32 37 65 2D fb413-e97d-427e-
0170: 61 34 38 36 2D 31 31 63 39 65 37 31 63 63 62 62 a486-11c9e71ccbb
0180: 32 22 7D 2C 22 63 6F 6E 74 72 6F 6C 6C 65 72 22 2".,"controller"
0190: 3A 7B 22 69 64 22 3A 22 31 38 63 38 39 64 32 33 :."id":"18c89d23
01A0: 2D 61 35 31 65 2D 34 35 35 38 2D 62 30 31 61 2D -a51e-4558-b01a-
01B0: 33 66 36 30 64 66 31 31 63 39 61 64 22 2C 22 6E 3f60df11c9ad","n
01C0: 61 6D 65 22 3A 22 4E 69 46 69 20 46 6C 6F 77 22 ame":"NiFi Flow"
01D0: 2C 22 63 6F 6D 6D 65 6E 74 73 22 3A 22 22 2C 22 ,"comments":"","
01E0: 72 75 6E 6E 69 6E 67 43 6F 75 6E 74 22 3A 31 36 runningCount":16
01F0: 34 2C 22 73 74 6F 70 70 65 64 43 6F 75 6E 74 22 4,"stoppedCount"
0200: 3A 34 33 2C 22 69 6E 76 61 6C 69 64 43 6F 75 6E :43,"invalidCoun
0210: 74 22 3A 31 2C 22 64 69 73 61 62 6C 65 64 43 6F t":1,"disabledCo
0220: 75 6E 74 22 3A 30 2C 22 69 6E 70 75 74 50 6F 72 unt":0,"inputPor
0230: 74 43 6F 75 6E 74 22 3A 37 2C 22 6F 75 74 70 75 tCount":7,"outpu
0240: 74 50 6F 72 74 43 6F 75 6E 74 22 3A 31 2C 22 72 tPortCount":1,"r
0250: 65 6D 6F 74 65 53 69 74 65 4C 69 73 74 65 6E 69 emoteSiteListeni
0260: 6E 67 50 6F 72 74 22 3A 39 38 37 30 2C 22 73 69 ngPort":9870,"si
0270: 74 65 54 6F 53 69 74 65 53 65 63 75 72 65 22 3A teToSiteSecure":
0280: 74 72 75 65 2C 22 69 6E 73 74 61 6E 63 65 49 64 true,"instanceId
0290: 22 3A 22 30 35 38 30 63 35 31 38 2D 39 62 63 37 ":"0580c518-9bc7
02A0: 2D 34 37 38 33 2D 39 32 34 38 2D 35 38 30 61 36 -4783-9248-580a6
02B0: 37 34 65 34 33 35 62 22 2C 22 69 6E 70 75 74 50 74e435b","inputP
02C0: 6F 72 74 73 22 3A 5B 7B 22 69 64 22 3A 22 33 32 orts":[."id":"32
02D0: 37 30 39 33 31 66 2D 64 61 38 35 2D 34 63 34 65 70931f-da85-4c4e
02E0: 2D 62 61 65 36 2D 38 63 36 32 37 62 30 39 62 37 -bae6-8c627b09b7
02F0: 32 66 22 2C 22 6E 61 6D 65 22 3A 22 48 44 46 53 2f","name":"HDFS
0300: 49 6E 63 6F 6D 69 6E 67 22 2C 22 63 6F 6D 6D 65 Incoming","comme
0310: 6E 74 73 22 3A 22 22 2C 22 73 74 61 74 65 22 3A nts":"","state":
0320: 22 53 54 4F 50 50 45 44 22 7D 2C 7B 22 69 64 22 "STOPPED".,."id"
0330: 3A 22 30 39 33 30 63 62 32 63 2D 37 61 38 33 2D :"0930cb2c-7a83-
0340: 34 38 36 64 2D 62 62 61 65 2D 38 62 33 30 31 32 486d-bbae-8b3012
0350: 64 36 31 39 66 37 22 2C 22 6E 61 6D 65 22 3A 22 d619f7","name":"
0360: 50 6F 72 74 20 39 30 39 38 20 49 6E 63 6F 6D 69 Port 9098 Incomi
0370: 6E 67 20 53 79 73 6C 6F 67 73 22 2C 22 63 6F 6D ng Syslogs","com
0380: 6D 65 6E 74 73 22 3A 22 22 2C 22 73 74 61 74 65 ments":"","state
0390: 22 3A 22 52 55 4E 4E 49 4E 47 22 7D 2C 7B 22 69 ":"RUNNING".,."i
03A0: 64 22 3A 22 31 34 62 64 32 66 66 35 2D 38 38 36 d":"14bd2ff5-886
03B0: 61 2D 34 61 32 39 2D 62 39 39 61 2D 38 64 34 34 a-4a29-b99a-8d44
03C0: 65 66 37 38 66 30 31 30 22 2C 22 6E 61 6D 65 22 ef78f010","name"
03D0: 3A 22 48 44 46 53 57 65 62 73 65 6E 73 65 53 65 :"HDFSWebsenseSe
03E0: 63 75 72 69 74 79 22 2C 22 63 6F 6D 6D 65 6E 74 curity","comment
03F0: 73 22 3A 22 22 2C 22 73 74 61 74 65 22 3A 22 53 s":"","state":"S
0400: 54 4F 50 50 45 44 22 7D 2C 7B 22 69 64 22 3A 22 TOPPED".,."id":"
0410: 33 61 66 30 33 66 66 36 2D 39 62 65 37 2D 33 32 3af03ff6-9be7-32
0420: 35 61 2D 61 63 66 33 2D 63 36 62 39 61 37 64 32 5a-acf3-c6b9a7d2
0430: 31 36 65 33 22 2C 22 6E 61 6D 65 22 3A 22 50 6F 16e3","name":"Po
0440: 72 74 20 39 30 39 39 20 49 6E 63 6F 6D 69 6E 67 rt 9099 Incoming
0450: 20 53 79 73 6C 6F 67 73 22 2C 22 63 6F 6D 6D 65 Syslogs","comme
0460: 6E 74 73 22 3A 22 22 2C 22 73 74 61 74 65 22 3A nts":"","state":
0470: 22 52 55 4E 4E 49 4E 47 22 7D 2C 7B 22 69 64 22 "RUNNING".,."id"
0480: 3A 22 65 65 34 31 37 64 35 61 2D 62 64 39 38 2D :"ee417d5a-bd98-
0490: 33 32 65 61 2D 61 63 35 38 2D 63 36 32 33 64 66 32ea-ac58-c623df
04A0: 35 65 64 64 66 35 22 2C 22 6E 61 6D 65 22 3A 22 5eddf5","name":"
04B0: 50 6F 72 74 20 39 31 30 31 20 49 6E 63 6F 6D 69 Port 9101 Incomi
04C0: 6E 67 20 53 79 73 6C 6F 67 73 22 2C 22 63 6F 6D ng Syslogs","com
04D0: 6D 65 6E 74 73 22 3A 22 22 2C 22 73 74 61 74 65 ments":"","state
04E0: 22 3A 22 52 55 4E 4E 49 4E 47 22 7D 2C 7B 22 69 ":"RUNNING".,."i
04F0: 64 22 3A 22 39 34 37 30 38 30 61 36 2D 34 65 61 d":"947080a6-4ea
0500: 66 2D 33 37 64 37 2D 62 36 32 62 2D 39 37 62 61 f-37d7-b62b-97ba
0510: 62 35 37 66 34 64 39 38 22 2C 22 6E 61 6D 65 22 b57f4d98","name"
0520: 3A 22 50 6F 72 74 20 39 31 30 30 20 49 6E 63 6F :"Port 9100 Inco
0530: 6D 69 6E 67 20 53 79 73 6C 6F 67 73 22 2C 22 63 ming Syslogs","c
0540: 6F 6D 6D 65 6E 74 73 22 3A 22 22 2C 22 73 74 61 omments":"","sta
0550: 74 65 22 3A 22 52 55 4E 4E 49 4E 47 22 7D 2C 7B te":"RUNNING".,.
0560: 22 69 64 22 3A 22 63 33 37 34 35 64 37 65 2D 39 "id":"c3745d7e-9
0570: 62 66 66 2D 33 31 31 32 2D 38 65 33 63 2D 39 36 bff-3112-8e3c-96
0580: 34 61 66 62 39 63 36 36 37 33 22 2C 22 6E 61 6D 4afb9c6673","nam
0590: 65 22 3A 22 50 6F 72 74 20 39 31 30 32 20 49 6E e":"Port 9102 In
05A0: 63 6F 6D 69 6E 67 20 53 79 73 6C 6F 67 73 22 2C coming Syslogs",
05B0: 22 63 6F 6D 6D 65 6E 74 73 22 3A 22 22 2C 22 73 "comments":"","s
05C0: 74 61 74 65 22 3A 22 52 55 4E 4E 49 4E 47 22 7D tate":"RUNNING".
05D0: 5D 2C 22 6F 75 74 70 75 74 50 6F 72 74 73 22 3A ],"outputPorts":
05E0: 5B 7B 22 69 64 22 3A 22 61 62 38 36 62 37 34 36 [."id":"ab86b746
05F0: 2D 37 39 63 33 2D 34 30 31 65 2D 62 35 30 35 2D -79c3-401e-b505-
0600: 39 64 39 34 30 35 62 32 32 62 33 31 22 2C 22 6E 9d9405b22b31","n
0610: 61 6D 65 22 3A 22 53 70 61 72 6B 20 74 65 73 74 ame":"Spark test
0620: 20 6F 75 74 22 2C 22 63 6F 6D 6D 65 6E 74 73 22 out","comments"
0630: 3A 22 22 2C 22 73 74 61 74 65 22 3A 22 52 55 4E :"","state":"RUN
0640: 4E 49 4E 47 22 7D 5D 7D 7D 15 C4 DA 96 85 23 76 NING".].......#v
0650: 2B DB 4B 46 5A 9A DD 4F 9B EF D8 46 70 FF CD EC +.KFZ..O...Fp...
0660: 99 19 31 F3 7F CC C1 14 07 06 06 06 06 06 06 06 ..1.............
16/05/23 15:49:39 WARN EndpointConnectionPool: EndpointConnectionPool[Cluster URL=https://yarn-cm1.mis-cds.local:9090/nifi/] Unable to refresh Remote Group's peers due to java.io.IOException: Unable to communicate with yarn-cm1.mis-cds.local:9870 because it requires Secure Site-to-Site communications, but this instance is not configured for secure communications
16/05/23 15:49:39 WARN EndpointConnectionPool: EndpointConnectionPool[Cluster URL=https://yarn-cm1.mis-cds.local:9090/nifi/] Unable to refresh Remote Group's peers due to java.io.IOException: Unable to communicate with yarn-cm1.mis-cds.local:9870 because it requires Secure Site-to-Site communications, but this instance is not configured for secure communications
Exception in thread "NiFi Receiver" java.lang.NullPointerException
at org.apache.nifi.spark.NiFiReceiver$ReceiveRunnable.run(NiFiReceiver.java:150)
at java.lang.Thread.run(Thread.java:745)
Which clearly shows that secure site to site communication is true
"r
0250: 65 6D 6F 74 65 53 69 74 65 4C 69 73 74 65 6E 69 emoteSiteListeni
0260: 6E 67 50 6F 72 74 22 3A 39 38 37 30 2C 22 73 69 ngPort":9870,"si
0270: 74 65 54 6F 53 69 74 65 53 65 63 75 72 65 22 3A teToSiteSecure":
0280: 74 72 75 65 2C 22 69 6E 73 74 61 6E 63 65 49 64 true,”
But the exception thrown looks like it is being coming from line 150 in NifiReceiver
Transaction ioe1 = ioe.createTransaction(TransferDirection.RECEIVE);
DataPacket dataPacket = ioe1.receive(); <—— here,
As a result of attempting to create the transaction on the SiteToSiteClient. The docs state that client may have to query the server’s RESTful interface which could throw an IOException. Without the full stack trace I’m only guessing that the isSecure method is returning false when it should be returning true.
Anyone?
Thanks
Conrad
From: Conrad Crampton <co...@SecData.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Monday, 23 May 2016 at 10:39
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Spark & NiFi question
Hi,
An update to this but still not working
I have now set keystore and truststore as system properties, and included these as part of the SiteToSiteClientConfig building. I have used a cert that I have for one of the servers in my cluster as I know they can communicate over ssl with NCM as my 6 node cluster works over ssl and has remote ports working (as I read from syslog on a primary server then distribute to other via remote ports as suggested somewhere else) .
When I try now to connect to output port via Spark, I get a
"EndpointConnectionPool[Cluster URL=https://yarn-cm1.mis-cds.local:9090/nifi/] Unable to refresh Remote Group's peers due to java.io.IOException: Unable to communicate with yarn-cm1.mis-cds.local:9870 because it requires Secure Site-to-Site communications, but this instance is not configured for secure communications"
Exception even though I know Secure Site-to-Site communication is working (9870 being the port set up for remote s2s comms in nifi.properties), so I am now really confused!!
Does the port that I wish to read from need to be set up with remote process group (conceptually I’m struggling with how to do this for an output port), or is it is sufficient to be ‘just an output port’?
I have this working when connecting to an unsecured (http) instance of NiFi running on my laptop with Spark and a standard output port. Does it make a difference that my production cluster is a cluster and therefore needs setting up differently?
So many questions but I’m stuck now so any suggestions welcome.
Thanks
Conrad
From: Conrad Crampton <co...@SecData.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Friday, 20 May 2016 at 09:16
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject:Re: Spark & NiFi question
Thanks for the pointers Bryan, however wrt your first suggestion. I tried without setting SSL properties on System properties and get an unable to find ssl path error – this gets resolved by doing as I have done (but of course this may be a red herring). I initially tried setting on site builder but got the same error as below – it appears to make no difference as to what is logged in the nifi-users.log if I include SSL props on site builder or not, I get the same error viz:
2016-05-20 08:59:47,082 INFO [NiFi Web Server-29590180] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<CN=spark-processor.m.xxx, OU=Development, O=Secure Data Europe Ltd, L=Maidstone, ST=Kent, C=GB>) GET https://yarn-cm1.m.xxxx:9090/nifi-api/controller (source ip: xx.xx.xx.1)
2016-05-20 08:59:47,082 INFO [NiFi Web Server-29494759] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<CN=spark-processor.m.xxx, OU=Development, O=Secure Data Europe Ltd, L=Maidstone, ST=Kent, C=GB>) GET https://yarn-cm1.m.xxx:9090/nifi-api/controller (source ip: xx.xx.xx.1)
2016-05-20 08:59:47,083 INFO [NiFi Web Server-29590180] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Unable to verify access for CN=spark-processor.m.xxx, OU=Development, O=Secure Data Europe Ltd, L=Maidstone, ST=Kent, C=GB
I am using self signed certs if that makes a difference (but these work fine on across the cluster). I am not seeing my spark user appear in the list of users to grant access.
I have turned on debug for ssl to see if that is throwing up anything but nothing appears obvious – here is the snipet that I would expect errors to be shown from that log.
... no IV derived for this protocol
%% Server resumed [Session-4, TLS_RSA_WITH_AES_128_CBC_SHA256]
NiFi Receiver, READ: TLSv1.2 Change Cipher Spec, length = 1
NiFi Receiver, READ: TLSv1.2 Handshake, length = 80
*** Finished
verify_data: { 109, 126, 134, 14, 33, 110, 224, 83, 198, 116, 54, 228 }
***
NiFi Receiver, WRITE: TLSv1.2 Change Cipher Spec, length = 1
*** Finished
verify_data: { 83, 120, 49, 158, 181, 136, 127, 219, 30, 194, 58, 167 }
***
NiFi Receiver, WRITE: TLSv1.2 Handshake, length = 80
NiFi Receiver, WRITE: TLSv1.2 Application Data, length = 240
I don’t really know enough about certificates and how client java apps would use them wrt to the host name/ ip address etc. of details is included in them. The nifi-user.log is showing access from a specific IP address which clearly doesn’t match the CN details in the cert. Just clutching at straws here!
Any other suggestions?
Thanks
Conrad
From: Bryan Bende <bb...@gmail.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Thursday, 19 May 2016 at 17:08
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Spark & NiFi question
Hi Conrad,
I think there are a couple of things at play here...
One is that the SSL properties need to be set on the SiteToSiteClientBuilder, rather than through system properties. There should be methods to set the keystore and other values.
In a secured NiFi instance, the certificate you are authenticating with (the keystore used by the s2s client) would need to have an account in NiFi, and would need to have access to the output port.
If you attempt to make a request with that cert, and then you go into the NiFi UI as another user, you should be able to go into the accounts section (top right) and approve the account for that certificate.
Then if you stop your output port, right-click and Configure... and from the Access Controls tab started typing the DN from your cert and add that user to the Allowed Users list. Hit Apply and started the port again.
We probably need to document this better, or write up an article about it somewhere.
Let us know if its still not working.
Thanks,
Bryan
On Thu, May 19, 2016 at 11:54 AM, Conrad Crampton <co...@secdata.com>> wrote:
Hi,
Tried following a couple of blog posts about this [1], [2], but neither of these refer to using NiFi in clustered environment with SSL and I suspect this is where I am hitting problems (but don’t know where).
The blogs state that using an output port (in the root process group I.e. on main canvas) which I have done and tried to connect thus..
System.setProperty("javax.net.ssl.keyStore", "/spark-processor.jks");
System.setProperty("javax.net.ssl.keyStorePassword", “*****");
System.setProperty("javax.net.ssl.trustStore", “/cacerts.jks");
SiteToSiteClientConfig config = new SiteToSiteClient.Builder()
.url("https://yarn-cm1.mis-cds.local:9090/nifi")
.portName("Spark test out")
.buildConfig();
SparkConf sparkConf = new SparkConf().setMaster("local[2]").setAppName("NiFi Spark Log Processor");
JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, new Duration(5000));
JavaReceiverInputDStream<NiFiDataPacket> packetStream = jssc.receiverStream(new NiFiReceiver(config, StorageLevel.MEMORY_ONLY()));
JavaDStream text = packetStream.map(dataPacket -> new String(dataPacket.getContent(), StandardCharsets.UTF_8));
text.print();
jssc.start();
jssc.awaitTermination();
The error I am getting is
16/05/19 16:39:03 WARN ReceiverSupervisorImpl: Restarting receiver with delay 2000 ms: Failed to receive data from NiFi
java.io.IOException: Server returned HTTP response code: 401 for URL: https://yarn-cm1.mis-cds.local:9090/nifi-api/controller
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1889)
at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1884)
at java.security.AccessController.doPrivileged(Native Method)
at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1883)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1456)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
at org.apache.nifi.remote.util.NiFiRestApiUtil.getController(NiFiRestApiUtil.java:69)
at org.apache.nifi.remote.client.socket.EndpointConnectionPool.refreshRemoteInfo(EndpointConnectionPool.java:891)
at org.apache.nifi.remote.client.socket.EndpointConnectionPool.getPortIdentifier(EndpointConnectionPool.java:878)
at org.apache.nifi.remote.client.socket.EndpointConnectionPool.getOutputPortIdentifier(EndpointConnectionPool.java:862)
at org.apache.nifi.remote.client.socket.SocketClient.getPortIdentifier(SocketClient.java:81)
at org.apache.nifi.remote.client.socket.SocketClient.createTransaction(SocketClient.java:123)
at org.apache.nifi.spark.NiFiReceiver$ReceiveRunnable.run(NiFiReceiver.java:149)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Server returned HTTP response code: 401 for URL: https://yarn-cm1.mis-cds.local:9090/nifi-api/controller
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1839)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
at org.apache.nifi.remote.util.NiFiRestApiUtil.getController(NiFiRestApiUtil.java:66)
... 7 more
Any pointers would be helpful in getting this working. I don’t know if I have to set up a remote process group with the output port (not sure how this works), or what. When I go to https://yarn-cm1.mis-cds.local:9090/nifi-api/controller in the browser, I get an access denied error.
I have created keystore and signed by the RootCA used to sign all the self signed certs for the cluster.
Running 0.6.1, 6 node cluster.
Thanks
Conrad
[1[ - https://community.hortonworks.com/articles/12708/nifi-feeding-data-to-spark-streaming.html
[2] - https://blogs.apache.org/nifi/entry/stream_processing_nifi_and_spark
SecureData, combating cyber threats
________________________________
The information contained in this message or any of its attachments may be privileged and confidential and intended for the exclusive use of the intended recipient. If you are not the intended recipient any disclosure, reproduction, distribution or other dissemination or use of this communications is strictly prohibited. The views expressed in this email are those of the individual and not necessarily of SecureData Europe Ltd. Any prices quoted are only valid if followed up by a formal written quote.
SecureData Europe Limited. Registered in England & Wales 04365896. Registered Address: SecureData House, Hermitage Court, Hermitage Lane, Maidstone, Kent, ME16 9NT
***This email originated outside SecureData***
Click here<https://www.mailcontrol.com/sr/MZbqvYs5QwJvpeaetUwhCQ==> to report this email as spam.
Re: Spark & NiFi question
Posted by Conrad Crampton <co...@SecData.com>.
Hi,
I don’t know if I’m hitting some bug here but something doesn’t make sense.
With ssl debug on I get the following
NiFi Receiver, READ: TLSv1.2 Application Data, length = 1648
Padded plaintext after DECRYPTION: len = 1648
0000: 65 A2 B8 34 DF 20 6B 95 56 88 97 16 7A EC 8F E3 e..4. k.V...z...
0010: 48 54 54 50 2F 31 2E 31 20 32 30 30 20 4F 4B 0D HTTP/1.1 200 OK.
0020: 0A 44 61 74 65 3A 20 4D 6F 6E 2C 20 32 33 20 4D .Date: Mon, 23 M
0030: 61 79 20 32 30 31 36 20 31 34 3A 34 39 3A 33 39 ay 2016 14:49:39
0040: 20 47 4D 54 0D 0A 53 65 72 76 65 72 3A 20 4A 65 GMT..Server: Je
0050: 74 74 79 28 39 2E 32 2E 31 31 2E 76 32 30 31 35 tty(9.2.11.v2015
0060: 30 35 32 39 29 0D 0A 43 61 63 68 65 2D 43 6F 6E 0529)..Cache-Con
0070: 74 72 6F 6C 3A 20 70 72 69 76 61 74 65 2C 20 6E trol: private, n
0080: 6F 2D 63 61 63 68 65 2C 20 6E 6F 2D 73 74 6F 72 o-cache, no-stor
0090: 65 2C 20 6E 6F 2D 74 72 61 6E 73 66 6F 72 6D 0D e, no-transform.
00A0: 0A 56 61 72 79 3A 20 41 63 63 65 70 74 2D 45 6E .Vary: Accept-En
00B0: 63 6F 64 69 6E 67 2C 20 55 73 65 72 2D 41 67 65 coding, User-Age
00C0: 6E 74 0D 0A 44 61 74 65 3A 20 4D 6F 6E 2C 20 32 nt..Date: Mon, 2
00D0: 33 20 4D 61 79 20 32 30 31 36 20 31 34 3A 34 39 3 May 2016 14:49
00E0: 3A 33 39 20 47 4D 54 0D 0A 43 6F 6E 74 65 6E 74 :39 GMT..Content
00F0: 2D 54 79 70 65 3A 20 61 70 70 6C 69 63 61 74 69 -Type: applicati
0100: 6F 6E 2F 6A 73 6F 6E 0D 0A 56 61 72 79 3A 20 41 on/json..Vary: A
0110: 63 63 65 70 74 2D 45 6E 63 6F 64 69 6E 67 2C 20 ccept-Encoding,
0120: 55 73 65 72 2D 41 67 65 6E 74 0D 0A 43 6F 6E 74 User-Agent..Cont
0130: 65 6E 74 2D 4C 65 6E 67 74 68 3A 20 31 32 38 35 ent-Length: 1285
0140: 0D 0A 0D 0A 7B 22 72 65 76 69 73 69 6F 6E 22 3A ....."revision":
0150: 7B 22 63 6C 69 65 6E 74 49 64 22 3A 22 39 34 38 ."clientId":"948
0160: 66 62 34 31 33 2D 65 39 37 64 2D 34 32 37 65 2D fb413-e97d-427e-
0170: 61 34 38 36 2D 31 31 63 39 65 37 31 63 63 62 62 a486-11c9e71ccbb
0180: 32 22 7D 2C 22 63 6F 6E 74 72 6F 6C 6C 65 72 22 2".,"controller"
0190: 3A 7B 22 69 64 22 3A 22 31 38 63 38 39 64 32 33 :."id":"18c89d23
01A0: 2D 61 35 31 65 2D 34 35 35 38 2D 62 30 31 61 2D -a51e-4558-b01a-
01B0: 33 66 36 30 64 66 31 31 63 39 61 64 22 2C 22 6E 3f60df11c9ad","n
01C0: 61 6D 65 22 3A 22 4E 69 46 69 20 46 6C 6F 77 22 ame":"NiFi Flow"
01D0: 2C 22 63 6F 6D 6D 65 6E 74 73 22 3A 22 22 2C 22 ,"comments":"","
01E0: 72 75 6E 6E 69 6E 67 43 6F 75 6E 74 22 3A 31 36 runningCount":16
01F0: 34 2C 22 73 74 6F 70 70 65 64 43 6F 75 6E 74 22 4,"stoppedCount"
0200: 3A 34 33 2C 22 69 6E 76 61 6C 69 64 43 6F 75 6E :43,"invalidCoun
0210: 74 22 3A 31 2C 22 64 69 73 61 62 6C 65 64 43 6F t":1,"disabledCo
0220: 75 6E 74 22 3A 30 2C 22 69 6E 70 75 74 50 6F 72 unt":0,"inputPor
0230: 74 43 6F 75 6E 74 22 3A 37 2C 22 6F 75 74 70 75 tCount":7,"outpu
0240: 74 50 6F 72 74 43 6F 75 6E 74 22 3A 31 2C 22 72 tPortCount":1,"r
0250: 65 6D 6F 74 65 53 69 74 65 4C 69 73 74 65 6E 69 emoteSiteListeni
0260: 6E 67 50 6F 72 74 22 3A 39 38 37 30 2C 22 73 69 ngPort":9870,"si
0270: 74 65 54 6F 53 69 74 65 53 65 63 75 72 65 22 3A teToSiteSecure":
0280: 74 72 75 65 2C 22 69 6E 73 74 61 6E 63 65 49 64 true,"instanceId
0290: 22 3A 22 30 35 38 30 63 35 31 38 2D 39 62 63 37 ":"0580c518-9bc7
02A0: 2D 34 37 38 33 2D 39 32 34 38 2D 35 38 30 61 36 -4783-9248-580a6
02B0: 37 34 65 34 33 35 62 22 2C 22 69 6E 70 75 74 50 74e435b","inputP
02C0: 6F 72 74 73 22 3A 5B 7B 22 69 64 22 3A 22 33 32 orts":[."id":"32
02D0: 37 30 39 33 31 66 2D 64 61 38 35 2D 34 63 34 65 70931f-da85-4c4e
02E0: 2D 62 61 65 36 2D 38 63 36 32 37 62 30 39 62 37 -bae6-8c627b09b7
02F0: 32 66 22 2C 22 6E 61 6D 65 22 3A 22 48 44 46 53 2f","name":"HDFS
0300: 49 6E 63 6F 6D 69 6E 67 22 2C 22 63 6F 6D 6D 65 Incoming","comme
0310: 6E 74 73 22 3A 22 22 2C 22 73 74 61 74 65 22 3A nts":"","state":
0320: 22 53 54 4F 50 50 45 44 22 7D 2C 7B 22 69 64 22 "STOPPED".,."id"
0330: 3A 22 30 39 33 30 63 62 32 63 2D 37 61 38 33 2D :"0930cb2c-7a83-
0340: 34 38 36 64 2D 62 62 61 65 2D 38 62 33 30 31 32 486d-bbae-8b3012
0350: 64 36 31 39 66 37 22 2C 22 6E 61 6D 65 22 3A 22 d619f7","name":"
0360: 50 6F 72 74 20 39 30 39 38 20 49 6E 63 6F 6D 69 Port 9098 Incomi
0370: 6E 67 20 53 79 73 6C 6F 67 73 22 2C 22 63 6F 6D ng Syslogs","com
0380: 6D 65 6E 74 73 22 3A 22 22 2C 22 73 74 61 74 65 ments":"","state
0390: 22 3A 22 52 55 4E 4E 49 4E 47 22 7D 2C 7B 22 69 ":"RUNNING".,."i
03A0: 64 22 3A 22 31 34 62 64 32 66 66 35 2D 38 38 36 d":"14bd2ff5-886
03B0: 61 2D 34 61 32 39 2D 62 39 39 61 2D 38 64 34 34 a-4a29-b99a-8d44
03C0: 65 66 37 38 66 30 31 30 22 2C 22 6E 61 6D 65 22 ef78f010","name"
03D0: 3A 22 48 44 46 53 57 65 62 73 65 6E 73 65 53 65 :"HDFSWebsenseSe
03E0: 63 75 72 69 74 79 22 2C 22 63 6F 6D 6D 65 6E 74 curity","comment
03F0: 73 22 3A 22 22 2C 22 73 74 61 74 65 22 3A 22 53 s":"","state":"S
0400: 54 4F 50 50 45 44 22 7D 2C 7B 22 69 64 22 3A 22 TOPPED".,."id":"
0410: 33 61 66 30 33 66 66 36 2D 39 62 65 37 2D 33 32 3af03ff6-9be7-32
0420: 35 61 2D 61 63 66 33 2D 63 36 62 39 61 37 64 32 5a-acf3-c6b9a7d2
0430: 31 36 65 33 22 2C 22 6E 61 6D 65 22 3A 22 50 6F 16e3","name":"Po
0440: 72 74 20 39 30 39 39 20 49 6E 63 6F 6D 69 6E 67 rt 9099 Incoming
0450: 20 53 79 73 6C 6F 67 73 22 2C 22 63 6F 6D 6D 65 Syslogs","comme
0460: 6E 74 73 22 3A 22 22 2C 22 73 74 61 74 65 22 3A nts":"","state":
0470: 22 52 55 4E 4E 49 4E 47 22 7D 2C 7B 22 69 64 22 "RUNNING".,."id"
0480: 3A 22 65 65 34 31 37 64 35 61 2D 62 64 39 38 2D :"ee417d5a-bd98-
0490: 33 32 65 61 2D 61 63 35 38 2D 63 36 32 33 64 66 32ea-ac58-c623df
04A0: 35 65 64 64 66 35 22 2C 22 6E 61 6D 65 22 3A 22 5eddf5","name":"
04B0: 50 6F 72 74 20 39 31 30 31 20 49 6E 63 6F 6D 69 Port 9101 Incomi
04C0: 6E 67 20 53 79 73 6C 6F 67 73 22 2C 22 63 6F 6D ng Syslogs","com
04D0: 6D 65 6E 74 73 22 3A 22 22 2C 22 73 74 61 74 65 ments":"","state
04E0: 22 3A 22 52 55 4E 4E 49 4E 47 22 7D 2C 7B 22 69 ":"RUNNING".,."i
04F0: 64 22 3A 22 39 34 37 30 38 30 61 36 2D 34 65 61 d":"947080a6-4ea
0500: 66 2D 33 37 64 37 2D 62 36 32 62 2D 39 37 62 61 f-37d7-b62b-97ba
0510: 62 35 37 66 34 64 39 38 22 2C 22 6E 61 6D 65 22 b57f4d98","name"
0520: 3A 22 50 6F 72 74 20 39 31 30 30 20 49 6E 63 6F :"Port 9100 Inco
0530: 6D 69 6E 67 20 53 79 73 6C 6F 67 73 22 2C 22 63 ming Syslogs","c
0540: 6F 6D 6D 65 6E 74 73 22 3A 22 22 2C 22 73 74 61 omments":"","sta
0550: 74 65 22 3A 22 52 55 4E 4E 49 4E 47 22 7D 2C 7B te":"RUNNING".,.
0560: 22 69 64 22 3A 22 63 33 37 34 35 64 37 65 2D 39 "id":"c3745d7e-9
0570: 62 66 66 2D 33 31 31 32 2D 38 65 33 63 2D 39 36 bff-3112-8e3c-96
0580: 34 61 66 62 39 63 36 36 37 33 22 2C 22 6E 61 6D 4afb9c6673","nam
0590: 65 22 3A 22 50 6F 72 74 20 39 31 30 32 20 49 6E e":"Port 9102 In
05A0: 63 6F 6D 69 6E 67 20 53 79 73 6C 6F 67 73 22 2C coming Syslogs",
05B0: 22 63 6F 6D 6D 65 6E 74 73 22 3A 22 22 2C 22 73 "comments":"","s
05C0: 74 61 74 65 22 3A 22 52 55 4E 4E 49 4E 47 22 7D tate":"RUNNING".
05D0: 5D 2C 22 6F 75 74 70 75 74 50 6F 72 74 73 22 3A ],"outputPorts":
05E0: 5B 7B 22 69 64 22 3A 22 61 62 38 36 62 37 34 36 [."id":"ab86b746
05F0: 2D 37 39 63 33 2D 34 30 31 65 2D 62 35 30 35 2D -79c3-401e-b505-
0600: 39 64 39 34 30 35 62 32 32 62 33 31 22 2C 22 6E 9d9405b22b31","n
0610: 61 6D 65 22 3A 22 53 70 61 72 6B 20 74 65 73 74 ame":"Spark test
0620: 20 6F 75 74 22 2C 22 63 6F 6D 6D 65 6E 74 73 22 out","comments"
0630: 3A 22 22 2C 22 73 74 61 74 65 22 3A 22 52 55 4E :"","state":"RUN
0640: 4E 49 4E 47 22 7D 5D 7D 7D 15 C4 DA 96 85 23 76 NING".].......#v
0650: 2B DB 4B 46 5A 9A DD 4F 9B EF D8 46 70 FF CD EC +.KFZ..O...Fp...
0660: 99 19 31 F3 7F CC C1 14 07 06 06 06 06 06 06 06 ..1.............
16/05/23 15:49:39 WARN EndpointConnectionPool: EndpointConnectionPool[Cluster URL=https://yarn-cm1.mis-cds.local:9090/nifi/] Unable to refresh Remote Group's peers due to java.io.IOException: Unable to communicate with yarn-cm1.mis-cds.local:9870 because it requires Secure Site-to-Site communications, but this instance is not configured for secure communications
16/05/23 15:49:39 WARN EndpointConnectionPool: EndpointConnectionPool[Cluster URL=https://yarn-cm1.mis-cds.local:9090/nifi/] Unable to refresh Remote Group's peers due to java.io.IOException: Unable to communicate with yarn-cm1.mis-cds.local:9870 because it requires Secure Site-to-Site communications, but this instance is not configured for secure communications
Exception in thread "NiFi Receiver" java.lang.NullPointerException
at org.apache.nifi.spark.NiFiReceiver$ReceiveRunnable.run(NiFiReceiver.java:150)
at java.lang.Thread.run(Thread.java:745)
Which clearly shows that secure site to site communication is true
"r
0250: 65 6D 6F 74 65 53 69 74 65 4C 69 73 74 65 6E 69 emoteSiteListeni
0260: 6E 67 50 6F 72 74 22 3A 39 38 37 30 2C 22 73 69 ngPort":9870,"si
0270: 74 65 54 6F 53 69 74 65 53 65 63 75 72 65 22 3A teToSiteSecure":
0280: 74 72 75 65 2C 22 69 6E 73 74 61 6E 63 65 49 64 true,”
But the exception thrown looks like it is being coming from line 150 in NifiReceiver
Transaction ioe1 = ioe.createTransaction(TransferDirection.RECEIVE);
DataPacket dataPacket = ioe1.receive(); <—— here,
As a result of attempting to create the transaction on the SiteToSiteClient. The docs state that client may have to query the server’s RESTful interface which could throw an IOException. Without the full stack trace I’m only guessing that the isSecure method is returning false when it should be returning true.
Anyone?
Thanks
Conrad
From: Conrad Crampton <co...@SecData.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Monday, 23 May 2016 at 10:39
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: SPOOFED: Re: Spark & NiFi question
Hi,
An update to this but still not working
I have now set keystore and truststore as system properties, and included these as part of the SiteToSiteClientConfig building. I have used a cert that I have for one of the servers in my cluster as I know they can communicate over ssl with NCM as my 6 node cluster works over ssl and has remote ports working (as I read from syslog on a primary server then distribute to other via remote ports as suggested somewhere else) .
When I try now to connect to output port via Spark, I get a
"EndpointConnectionPool[Cluster URL=https://yarn-cm1.mis-cds.local:9090/nifi/] Unable to refresh Remote Group's peers due to java.io.IOException: Unable to communicate with yarn-cm1.mis-cds.local:9870 because it requires Secure Site-to-Site communications, but this instance is not configured for secure communications"
Exception even though I know Secure Site-to-Site communication is working (9870 being the port set up for remote s2s comms in nifi.properties), so I am now really confused!!
Does the port that I wish to read from need to be set up with remote process group (conceptually I’m struggling with how to do this for an output port), or is it is sufficient to be ‘just an output port’?
I have this working when connecting to an unsecured (http) instance of NiFi running on my laptop with Spark and a standard output port. Does it make a difference that my production cluster is a cluster and therefore needs setting up differently?
So many questions but I’m stuck now so any suggestions welcome.
Thanks
Conrad
From: Conrad Crampton <co...@SecData.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Friday, 20 May 2016 at 09:16
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: SPOOFED: Re: Spark & NiFi question
Thanks for the pointers Bryan, however wrt your first suggestion. I tried without setting SSL properties on System properties and get an unable to find ssl path error – this gets resolved by doing as I have done (but of course this may be a red herring). I initially tried setting on site builder but got the same error as below – it appears to make no difference as to what is logged in the nifi-users.log if I include SSL props on site builder or not, I get the same error viz:
2016-05-20 08:59:47,082 INFO [NiFi Web Server-29590180] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<CN=spark-processor.m.xxx, OU=Development, O=Secure Data Europe Ltd, L=Maidstone, ST=Kent, C=GB>) GET https://yarn-cm1.m.xxxx:9090/nifi-api/controller (source ip: xx.xx.xx.1)
2016-05-20 08:59:47,082 INFO [NiFi Web Server-29494759] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<CN=spark-processor.m.xxx, OU=Development, O=Secure Data Europe Ltd, L=Maidstone, ST=Kent, C=GB>) GET https://yarn-cm1.m.xxx:9090/nifi-api/controller (source ip: xx.xx.xx.1)
2016-05-20 08:59:47,083 INFO [NiFi Web Server-29590180] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Unable to verify access for CN=spark-processor.m.xxx, OU=Development, O=Secure Data Europe Ltd, L=Maidstone, ST=Kent, C=GB
I am using self signed certs if that makes a difference (but these work fine on across the cluster). I am not seeing my spark user appear in the list of users to grant access.
I have turned on debug for ssl to see if that is throwing up anything but nothing appears obvious – here is the snipet that I would expect errors to be shown from that log.
... no IV derived for this protocol
%% Server resumed [Session-4, TLS_RSA_WITH_AES_128_CBC_SHA256]
NiFi Receiver, READ: TLSv1.2 Change Cipher Spec, length = 1
NiFi Receiver, READ: TLSv1.2 Handshake, length = 80
*** Finished
verify_data: { 109, 126, 134, 14, 33, 110, 224, 83, 198, 116, 54, 228 }
***
NiFi Receiver, WRITE: TLSv1.2 Change Cipher Spec, length = 1
*** Finished
verify_data: { 83, 120, 49, 158, 181, 136, 127, 219, 30, 194, 58, 167 }
***
NiFi Receiver, WRITE: TLSv1.2 Handshake, length = 80
NiFi Receiver, WRITE: TLSv1.2 Application Data, length = 240
I don’t really know enough about certificates and how client java apps would use them wrt to the host name/ ip address etc. of details is included in them. The nifi-user.log is showing access from a specific IP address which clearly doesn’t match the CN details in the cert. Just clutching at straws here!
Any other suggestions?
Thanks
Conrad
From: Bryan Bende <bb...@gmail.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Thursday, 19 May 2016 at 17:08
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Spark & NiFi question
Hi Conrad,
I think there are a couple of things at play here...
One is that the SSL properties need to be set on the SiteToSiteClientBuilder, rather than through system properties. There should be methods to set the keystore and other values.
In a secured NiFi instance, the certificate you are authenticating with (the keystore used by the s2s client) would need to have an account in NiFi, and would need to have access to the output port.
If you attempt to make a request with that cert, and then you go into the NiFi UI as another user, you should be able to go into the accounts section (top right) and approve the account for that certificate.
Then if you stop your output port, right-click and Configure... and from the Access Controls tab started typing the DN from your cert and add that user to the Allowed Users list. Hit Apply and started the port again.
We probably need to document this better, or write up an article about it somewhere.
Let us know if its still not working.
Thanks,
Bryan
On Thu, May 19, 2016 at 11:54 AM, Conrad Crampton <co...@secdata.com>> wrote:
Hi,
Tried following a couple of blog posts about this [1], [2], but neither of these refer to using NiFi in clustered environment with SSL and I suspect this is where I am hitting problems (but don’t know where).
The blogs state that using an output port (in the root process group I.e. on main canvas) which I have done and tried to connect thus..
System.setProperty("javax.net.ssl.keyStore", "/spark-processor.jks");
System.setProperty("javax.net.ssl.keyStorePassword", “*****");
System.setProperty("javax.net.ssl.trustStore", “/cacerts.jks");
SiteToSiteClientConfig config = new SiteToSiteClient.Builder()
.url("https://yarn-cm1.mis-cds.local:9090/nifi")
.portName("Spark test out")
.buildConfig();
SparkConf sparkConf = new SparkConf().setMaster("local[2]").setAppName("NiFi Spark Log Processor");
JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, new Duration(5000));
JavaReceiverInputDStream<NiFiDataPacket> packetStream = jssc.receiverStream(new NiFiReceiver(config, StorageLevel.MEMORY_ONLY()));
JavaDStream text = packetStream.map(dataPacket -> new String(dataPacket.getContent(), StandardCharsets.UTF_8));
text.print();
jssc.start();
jssc.awaitTermination();
The error I am getting is
16/05/19 16:39:03 WARN ReceiverSupervisorImpl: Restarting receiver with delay 2000 ms: Failed to receive data from NiFi
java.io.IOException: Server returned HTTP response code: 401 for URL: https://yarn-cm1.mis-cds.local:9090/nifi-api/controller
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1889)
at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1884)
at java.security.AccessController.doPrivileged(Native Method)
at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1883)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1456)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
at org.apache.nifi.remote.util.NiFiRestApiUtil.getController(NiFiRestApiUtil.java:69)
at org.apache.nifi.remote.client.socket.EndpointConnectionPool.refreshRemoteInfo(EndpointConnectionPool.java:891)
at org.apache.nifi.remote.client.socket.EndpointConnectionPool.getPortIdentifier(EndpointConnectionPool.java:878)
at org.apache.nifi.remote.client.socket.EndpointConnectionPool.getOutputPortIdentifier(EndpointConnectionPool.java:862)
at org.apache.nifi.remote.client.socket.SocketClient.getPortIdentifier(SocketClient.java:81)
at org.apache.nifi.remote.client.socket.SocketClient.createTransaction(SocketClient.java:123)
at org.apache.nifi.spark.NiFiReceiver$ReceiveRunnable.run(NiFiReceiver.java:149)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Server returned HTTP response code: 401 for URL: https://yarn-cm1.mis-cds.local:9090/nifi-api/controller
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1839)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
at org.apache.nifi.remote.util.NiFiRestApiUtil.getController(NiFiRestApiUtil.java:66)
... 7 more
Any pointers would be helpful in getting this working. I don’t know if I have to set up a remote process group with the output port (not sure how this works), or what. When I go to https://yarn-cm1.mis-cds.local:9090/nifi-api/controller in the browser, I get an access denied error.
I have created keystore and signed by the RootCA used to sign all the self signed certs for the cluster.
Running 0.6.1, 6 node cluster.
Thanks
Conrad
[1[ - https://community.hortonworks.com/articles/12708/nifi-feeding-data-to-spark-streaming.html
[2] - https://blogs.apache.org/nifi/entry/stream_processing_nifi_and_spark
SecureData, combating cyber threats
________________________________
The information contained in this message or any of its attachments may be privileged and confidential and intended for the exclusive use of the intended recipient. If you are not the intended recipient any disclosure, reproduction, distribution or other dissemination or use of this communications is strictly prohibited. The views expressed in this email are those of the individual and not necessarily of SecureData Europe Ltd. Any prices quoted are only valid if followed up by a formal written quote.
SecureData Europe Limited. Registered in England & Wales 04365896. Registered Address: SecureData House, Hermitage Court, Hermitage Lane, Maidstone, Kent, ME16 9NT
***This email originated outside SecureData***
Click here<https://www.mailcontrol.com/sr/MZbqvYs5QwJvpeaetUwhCQ==> to report this email as spam.
Re: Spark & NiFi question
Posted by Bryan Bende <bb...@gmail.com>.
Conrad,
Unfortunately I think this is a result of the issue you discovered with the
SSLContext not getting created from the properties on the
SiteToSiteClientBuilder...
Whats happening is the spark side is hitting this:
if (siteToSiteSecure) {
if (sslContext == null) {
throw new IOException("Unable to communicate with " + hostname +
":" + port
+ " because it requires Secure Site-to-Site communications,
but this instance is not configured for secure communications");
}
And siteToSiteSecure is true, but the sslContext is null so it can never
get past this point. I submitted a pull request on Friday that should
address the issue [1].
Once we get this merged in you could possibly build the source to get the
fixed SiteToSiteClient code, otherwise you could wait for the 0.7.0 release
to happen.
-Bryan
[1] https://github.com/apache/nifi/pull/457
On Mon, May 23, 2016 at 5:39 AM, Conrad Crampton <
conrad.crampton@secdata.com> wrote:
> Hi,
> An update to this but still not working
> I have now set keystore and truststore as system properties, and included
> these as part of the SiteToSiteClientConfig building. I have used a cert
> that I have for one of the servers in my cluster as I know they can
> communicate over ssl with NCM as my 6 node cluster works over ssl and has
> remote ports working (as I read from syslog on a primary server then
> distribute to other via remote ports as suggested somewhere else) .
> When I try now to connect to output port via Spark, I get a
> "EndpointConnectionPool[Cluster URL=
> https://yarn-cm1.mis-cds.local:9090/nifi/] Unable to refresh Remote
> Group's peers due to java.io.IOException: Unable to communicate with
> yarn-cm1.mis-cds.local:9870 because it requires Secure Site-to-Site
> communications, but this instance is not configured for secure
> communications"
> Exception even though I know Secure Site-to-Site communication is working
> (9870 being the port set up for remote s2s comms in nifi.properties), so I
> am now really confused!!
>
> Does the port that I wish to read from need to be set up with remote
> process group (conceptually I’m struggling with how to do this for an
> output port), or is it is sufficient to be ‘just an output port’?
>
> I have this working when connecting to an unsecured (http) instance of
> NiFi running on my laptop with Spark and a standard output port. Does it
> make a difference that my production cluster is a cluster and therefore
> needs setting up differently?
>
> So many questions but I’m stuck now so any suggestions welcome.
> Thanks
> Conrad
>
> From: Conrad Crampton <co...@SecData.com>
> Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Date: Friday, 20 May 2016 at 09:16
> To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Subject: SPOOFED: Re: Spark & NiFi question
>
> Thanks for the pointers Bryan, however wrt your first suggestion. I tried
> without setting SSL properties on System properties and get an unable to
> find ssl path error – this gets resolved by doing as I have done (but of
> course this may be a red herring). I initially tried setting on site
> builder but got the same error as below – it appears to make no difference
> as to what is logged in the nifi-users.log if I include SSL props on site
> builder or not, I get the same error viz:
>
> 2016-05-20 08:59:47,082 INFO [NiFi Web Server-29590180]
> o.a.n.w.s.NiFiAuthenticationFilter Attempting request for
> (<CN=spark-processor.m.xxx, OU=Development, O=Secure Data Europe Ltd,
> L=Maidstone, ST=Kent, C=GB>) GET
> https://yarn-cm1.m.xxxx:9090/nifi-api/controller (source ip: xx.xx.xx.1)
> 2016-05-20 08:59:47,082 INFO [NiFi Web Server-29494759]
> o.a.n.w.s.NiFiAuthenticationFilter Attempting request for
> (<CN=spark-processor.m.xxx, OU=Development, O=Secure Data Europe Ltd,
> L=Maidstone, ST=Kent, C=GB>) GET
> https://yarn-cm1.m.xxx:9090/nifi-api/controller (source ip: xx.xx.xx.1)
> 2016-05-20 08:59:47,083 INFO [NiFi Web Server-29590180]
> o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Unable to
> verify access for CN=spark-processor.m.xxx, OU=Development, O=Secure Data
> Europe Ltd, L=Maidstone, ST=Kent, C=GB
>
> I am using self signed certs if that makes a difference (but these work
> fine on across the cluster). I am not seeing my spark user appear in the
> list of users to grant access.
>
> I have turned on debug for ssl to see if that is throwing up anything but
> nothing appears obvious – here is the snipet that I would expect errors to
> be shown from that log.
>
> ... no IV derived for this protocol
> %% Server resumed [Session-4, TLS_RSA_WITH_AES_128_CBC_SHA256]
> NiFi Receiver, READ: TLSv1.2 Change Cipher Spec, length = 1
> NiFi Receiver, READ: TLSv1.2 Handshake, length = 80
> *** Finished
> verify_data: { 109, 126, 134, 14, 33, 110, 224, 83, 198, 116, 54, 228 }
> ***
> NiFi Receiver, WRITE: TLSv1.2 Change Cipher Spec, length = 1
> *** Finished
> verify_data: { 83, 120, 49, 158, 181, 136, 127, 219, 30, 194, 58, 167 }
> ***
> NiFi Receiver, WRITE: TLSv1.2 Handshake, length = 80
> NiFi Receiver, WRITE: TLSv1.2 Application Data, length = 240
>
> I don’t really know enough about certificates and how client java apps
> would use them wrt to the host name/ ip address etc. of details is included
> in them. The nifi-user.log is showing access from a specific IP address
> which clearly doesn’t match the CN details in the cert. Just clutching at
> straws here!
>
> Any other suggestions?
>
> Thanks
> Conrad
>
> From: Bryan Bende <bb...@gmail.com>
> Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Date: Thursday, 19 May 2016 at 17:08
> To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Subject: Re: Spark & NiFi question
>
> Hi Conrad,
>
> I think there are a couple of things at play here...
>
> One is that the SSL properties need to be set on the
> SiteToSiteClientBuilder, rather than through system properties. There
> should be methods to set the keystore and other values.
>
> In a secured NiFi instance, the certificate you are authenticating with
> (the keystore used by the s2s client) would need to have an account in
> NiFi, and would need to have access to the output port.
> If you attempt to make a request with that cert, and then you go into the
> NiFi UI as another user, you should be able to go into the accounts section
> (top right) and approve the account for that certificate.
>
> Then if you stop your output port, right-click and Configure... and from
> the Access Controls tab started typing the DN from your cert and add that
> user to the Allowed Users list. Hit Apply and started the port again.
>
> We probably need to document this better, or write up an article about it
> somewhere.
>
> Let us know if its still not working.
>
> Thanks,
>
> Bryan
>
>
> On Thu, May 19, 2016 at 11:54 AM, Conrad Crampton <
> conrad.crampton@secdata.com> wrote:
>
>> Hi,
>> Tried following a couple of blog posts about this [1], [2], but neither
>> of these refer to using NiFi in clustered environment with SSL and I
>> suspect this is where I am hitting problems (but don’t know where).
>>
>> The blogs state that using an output port (in the root process group I.e.
>> on main canvas) which I have done and tried to connect thus..
>>
>> System.setProperty("javax.net.ssl.keyStore", "/spark-processor.jks");
>> System.setProperty("javax.net.ssl.keyStorePassword", *“******");
>> System.setProperty("javax.net.ssl.trustStore", *“*/cacerts.jks");
>>
>> SiteToSiteClientConfig config = new SiteToSiteClient.Builder()
>> .url("https://yarn-cm1.mis-cds.local:9090/nifi")
>> .portName("Spark test out")
>> .buildConfig();
>>
>> SparkConf sparkConf = new SparkConf().setMaster("local[2]").setAppName("NiFi Spark Log Processor");
>> JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, new Duration(5000));
>> JavaReceiverInputDStream<NiFiDataPacket> packetStream = jssc.receiverStream(new NiFiReceiver(config, StorageLevel.MEMORY_ONLY()));
>>
>> JavaDStream text = packetStream.map(dataPacket -> new String(dataPacket.getContent(), StandardCharsets.UTF_8));
>> text.print();
>> jssc.start();
>> jssc.awaitTermination();
>>
>> The error I am getting is
>>
>> 16/05/19 16:39:03 WARN ReceiverSupervisorImpl: Restarting receiver with
>> delay 2000 ms: Failed to receive data from NiFi
>> java.io.IOException: Server returned HTTP response code: 401 for URL:
>> https://yarn-cm1.mis-cds.local:9090/nifi-api/controller
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>> at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>> at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
>> at
>> sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1889)
>> at
>> sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1884)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at
>> sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1883)
>> at
>> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1456)
>> at
>> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
>> at
>> sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
>> at
>> org.apache.nifi.remote.util.NiFiRestApiUtil.getController(NiFiRestApiUtil.java:69)
>> at
>> org.apache.nifi.remote.client.socket.EndpointConnectionPool.refreshRemoteInfo(EndpointConnectionPool.java:891)
>> at
>> org.apache.nifi.remote.client.socket.EndpointConnectionPool.getPortIdentifier(EndpointConnectionPool.java:878)
>> at
>> org.apache.nifi.remote.client.socket.EndpointConnectionPool.getOutputPortIdentifier(EndpointConnectionPool.java:862)
>> at
>> org.apache.nifi.remote.client.socket.SocketClient.getPortIdentifier(SocketClient.java:81)
>> at
>> org.apache.nifi.remote.client.socket.SocketClient.createTransaction(SocketClient.java:123)
>> at
>> org.apache.nifi.spark.NiFiReceiver$ReceiveRunnable.run(NiFiReceiver.java:149)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.io.IOException: Server returned HTTP response code: 401
>> for URL: https://yarn-cm1.mis-cds.local:9090/nifi-api/controller
>> at
>> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1839)
>> at
>> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
>> at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
>> at
>> sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
>> at
>> org.apache.nifi.remote.util.NiFiRestApiUtil.getController(NiFiRestApiUtil.java:66)
>> ... 7 more
>>
>> Any pointers would be helpful in getting this working. I don’t know if I
>> have to set up a remote process group with the output port (not sure how
>> this works), or what. When I go to
>> https://yarn-cm1.mis-cds.local:9090/nifi-api/controller in the browser,
>> I get an access denied error.
>> I have created keystore and signed by the RootCA used to sign all the
>> self signed certs for the cluster.
>>
>> Running 0.6.1, 6 node cluster.
>>
>> Thanks
>> Conrad
>>
>> [1[ -
>> https://community.hortonworks.com/articles/12708/nifi-feeding-data-to-spark-streaming.html
>> [2] -
>> https://blogs.apache.org/nifi/entry/stream_processing_nifi_and_spark
>>
>>
>> SecureData, combating cyber threats
>>
>> ------------------------------
>>
>> The information contained in this message or any of its attachments may
>> be privileged and confidential and intended for the exclusive use of the
>> intended recipient. If you are not the intended recipient any disclosure,
>> reproduction, distribution or other dissemination or use of this
>> communications is strictly prohibited. The views expressed in this email
>> are those of the individual and not necessarily of SecureData Europe Ltd.
>> Any prices quoted are only valid if followed up by a formal written quote.
>>
>> SecureData Europe Limited. Registered in England & Wales 04365896.
>> Registered Address: SecureData House, Hermitage Court, Hermitage Lane,
>> Maidstone, Kent, ME16 9NT
>>
>
>
>
> ***This email originated outside SecureData***
>
> Click here
> <https://www.mailcontrol.com/sr/JOj4ovws70LGX2PQPOmvUqa7UuQeNDoM5CPuVUMi!aLghcUmWuJbL8QAhL3vPgRnasXOF8Vdo14NCU1!U1Tbvw==>
> to report this email as spam.
>
Re: Spark & NiFi question
Posted by Conrad Crampton <co...@SecData.com>.
Hi,
An update to this but still not working
I have now set keystore and truststore as system properties, and included these as part of the SiteToSiteClientConfig building. I have used a cert that I have for one of the servers in my cluster as I know they can communicate over ssl with NCM as my 6 node cluster works over ssl and has remote ports working (as I read from syslog on a primary server then distribute to other via remote ports as suggested somewhere else) .
When I try now to connect to output port via Spark, I get a
"EndpointConnectionPool[Cluster URL=https://yarn-cm1.mis-cds.local:9090/nifi/] Unable to refresh Remote Group's peers due to java.io.IOException: Unable to communicate with yarn-cm1.mis-cds.local:9870 because it requires Secure Site-to-Site communications, but this instance is not configured for secure communications"
Exception even though I know Secure Site-to-Site communication is working (9870 being the port set up for remote s2s comms in nifi.properties), so I am now really confused!!
Does the port that I wish to read from need to be set up with remote process group (conceptually I’m struggling with how to do this for an output port), or is it is sufficient to be ‘just an output port’?
I have this working when connecting to an unsecured (http) instance of NiFi running on my laptop with Spark and a standard output port. Does it make a difference that my production cluster is a cluster and therefore needs setting up differently?
So many questions but I’m stuck now so any suggestions welcome.
Thanks
Conrad
From: Conrad Crampton <co...@SecData.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Friday, 20 May 2016 at 09:16
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: SPOOFED: Re: Spark & NiFi question
Thanks for the pointers Bryan, however wrt your first suggestion. I tried without setting SSL properties on System properties and get an unable to find ssl path error – this gets resolved by doing as I have done (but of course this may be a red herring). I initially tried setting on site builder but got the same error as below – it appears to make no difference as to what is logged in the nifi-users.log if I include SSL props on site builder or not, I get the same error viz:
2016-05-20 08:59:47,082 INFO [NiFi Web Server-29590180] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<CN=spark-processor.m.xxx, OU=Development, O=Secure Data Europe Ltd, L=Maidstone, ST=Kent, C=GB>) GET https://yarn-cm1.m.xxxx:9090/nifi-api/controller (source ip: xx.xx.xx.1)
2016-05-20 08:59:47,082 INFO [NiFi Web Server-29494759] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<CN=spark-processor.m.xxx, OU=Development, O=Secure Data Europe Ltd, L=Maidstone, ST=Kent, C=GB>) GET https://yarn-cm1.m.xxx:9090/nifi-api/controller (source ip: xx.xx.xx.1)
2016-05-20 08:59:47,083 INFO [NiFi Web Server-29590180] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Unable to verify access for CN=spark-processor.m.xxx, OU=Development, O=Secure Data Europe Ltd, L=Maidstone, ST=Kent, C=GB
I am using self signed certs if that makes a difference (but these work fine on across the cluster). I am not seeing my spark user appear in the list of users to grant access.
I have turned on debug for ssl to see if that is throwing up anything but nothing appears obvious – here is the snipet that I would expect errors to be shown from that log.
... no IV derived for this protocol
%% Server resumed [Session-4, TLS_RSA_WITH_AES_128_CBC_SHA256]
NiFi Receiver, READ: TLSv1.2 Change Cipher Spec, length = 1
NiFi Receiver, READ: TLSv1.2 Handshake, length = 80
*** Finished
verify_data: { 109, 126, 134, 14, 33, 110, 224, 83, 198, 116, 54, 228 }
***
NiFi Receiver, WRITE: TLSv1.2 Change Cipher Spec, length = 1
*** Finished
verify_data: { 83, 120, 49, 158, 181, 136, 127, 219, 30, 194, 58, 167 }
***
NiFi Receiver, WRITE: TLSv1.2 Handshake, length = 80
NiFi Receiver, WRITE: TLSv1.2 Application Data, length = 240
I don’t really know enough about certificates and how client java apps would use them wrt to the host name/ ip address etc. of details is included in them. The nifi-user.log is showing access from a specific IP address which clearly doesn’t match the CN details in the cert. Just clutching at straws here!
Any other suggestions?
Thanks
Conrad
From: Bryan Bende <bb...@gmail.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Thursday, 19 May 2016 at 17:08
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Spark & NiFi question
Hi Conrad,
I think there are a couple of things at play here...
One is that the SSL properties need to be set on the SiteToSiteClientBuilder, rather than through system properties. There should be methods to set the keystore and other values.
In a secured NiFi instance, the certificate you are authenticating with (the keystore used by the s2s client) would need to have an account in NiFi, and would need to have access to the output port.
If you attempt to make a request with that cert, and then you go into the NiFi UI as another user, you should be able to go into the accounts section (top right) and approve the account for that certificate.
Then if you stop your output port, right-click and Configure... and from the Access Controls tab started typing the DN from your cert and add that user to the Allowed Users list. Hit Apply and started the port again.
We probably need to document this better, or write up an article about it somewhere.
Let us know if its still not working.
Thanks,
Bryan
On Thu, May 19, 2016 at 11:54 AM, Conrad Crampton <co...@secdata.com>> wrote:
Hi,
Tried following a couple of blog posts about this [1], [2], but neither of these refer to using NiFi in clustered environment with SSL and I suspect this is where I am hitting problems (but don’t know where).
The blogs state that using an output port (in the root process group I.e. on main canvas) which I have done and tried to connect thus..
System.setProperty("javax.net.ssl.keyStore", "/spark-processor.jks");
System.setProperty("javax.net.ssl.keyStorePassword", “*****");
System.setProperty("javax.net.ssl.trustStore", “/cacerts.jks");
SiteToSiteClientConfig config = new SiteToSiteClient.Builder()
.url("https://yarn-cm1.mis-cds.local:9090/nifi")
.portName("Spark test out")
.buildConfig();
SparkConf sparkConf = new SparkConf().setMaster("local[2]").setAppName("NiFi Spark Log Processor");
JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, new Duration(5000));
JavaReceiverInputDStream<NiFiDataPacket> packetStream = jssc.receiverStream(new NiFiReceiver(config, StorageLevel.MEMORY_ONLY()));
JavaDStream text = packetStream.map(dataPacket -> new String(dataPacket.getContent(), StandardCharsets.UTF_8));
text.print();
jssc.start();
jssc.awaitTermination();
The error I am getting is
16/05/19 16:39:03 WARN ReceiverSupervisorImpl: Restarting receiver with delay 2000 ms: Failed to receive data from NiFi
java.io.IOException: Server returned HTTP response code: 401 for URL: https://yarn-cm1.mis-cds.local:9090/nifi-api/controller
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1889)
at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1884)
at java.security.AccessController.doPrivileged(Native Method)
at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1883)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1456)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
at org.apache.nifi.remote.util.NiFiRestApiUtil.getController(NiFiRestApiUtil.java:69)
at org.apache.nifi.remote.client.socket.EndpointConnectionPool.refreshRemoteInfo(EndpointConnectionPool.java:891)
at org.apache.nifi.remote.client.socket.EndpointConnectionPool.getPortIdentifier(EndpointConnectionPool.java:878)
at org.apache.nifi.remote.client.socket.EndpointConnectionPool.getOutputPortIdentifier(EndpointConnectionPool.java:862)
at org.apache.nifi.remote.client.socket.SocketClient.getPortIdentifier(SocketClient.java:81)
at org.apache.nifi.remote.client.socket.SocketClient.createTransaction(SocketClient.java:123)
at org.apache.nifi.spark.NiFiReceiver$ReceiveRunnable.run(NiFiReceiver.java:149)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Server returned HTTP response code: 401 for URL: https://yarn-cm1.mis-cds.local:9090/nifi-api/controller
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1839)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
at org.apache.nifi.remote.util.NiFiRestApiUtil.getController(NiFiRestApiUtil.java:66)
... 7 more
Any pointers would be helpful in getting this working. I don’t know if I have to set up a remote process group with the output port (not sure how this works), or what. When I go to https://yarn-cm1.mis-cds.local:9090/nifi-api/controller in the browser, I get an access denied error.
I have created keystore and signed by the RootCA used to sign all the self signed certs for the cluster.
Running 0.6.1, 6 node cluster.
Thanks
Conrad
[1[ - https://community.hortonworks.com/articles/12708/nifi-feeding-data-to-spark-streaming.html
[2] - https://blogs.apache.org/nifi/entry/stream_processing_nifi_and_spark
SecureData, combating cyber threats
________________________________
The information contained in this message or any of its attachments may be privileged and confidential and intended for the exclusive use of the intended recipient. If you are not the intended recipient any disclosure, reproduction, distribution or other dissemination or use of this communications is strictly prohibited. The views expressed in this email are those of the individual and not necessarily of SecureData Europe Ltd. Any prices quoted are only valid if followed up by a formal written quote.
SecureData Europe Limited. Registered in England & Wales 04365896. Registered Address: SecureData House, Hermitage Court, Hermitage Lane, Maidstone, Kent, ME16 9NT
***This email originated outside SecureData***
Click here<https://www.mailcontrol.com/sr/JOj4ovws70LGX2PQPOmvUqa7UuQeNDoM5CPuVUMi!aLghcUmWuJbL8QAhL3vPgRnasXOF8Vdo14NCU1!U1Tbvw==> to report this email as spam.
Re: Spark & NiFi question
Posted by Conrad Crampton <co...@SecData.com>.
Thanks for the pointers Bryan, however wrt your first suggestion. I tried without setting SSL properties on System properties and get an unable to find ssl path error – this gets resolved by doing as I have done (but of course this may be a red herring). I initially tried setting on site builder but got the same error as below – it appears to make no difference as to what is logged in the nifi-users.log if I include SSL props on site builder or not, I get the same error viz:
2016-05-20 08:59:47,082 INFO [NiFi Web Server-29590180] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<CN=spark-processor.m.xxx, OU=Development, O=Secure Data Europe Ltd, L=Maidstone, ST=Kent, C=GB>) GET https://yarn-cm1.m.xxxx:9090/nifi-api/controller (source ip: xx.xx.xx.1)
2016-05-20 08:59:47,082 INFO [NiFi Web Server-29494759] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<CN=spark-processor.m.xxx, OU=Development, O=Secure Data Europe Ltd, L=Maidstone, ST=Kent, C=GB>) GET https://yarn-cm1.m.xxx:9090/nifi-api/controller (source ip: xx.xx.xx.1)
2016-05-20 08:59:47,083 INFO [NiFi Web Server-29590180] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Unable to verify access for CN=spark-processor.m.xxx, OU=Development, O=Secure Data Europe Ltd, L=Maidstone, ST=Kent, C=GB
I am using self signed certs if that makes a difference (but these work fine on across the cluster). I am not seeing my spark user appear in the list of users to grant access.
I have turned on debug for ssl to see if that is throwing up anything but nothing appears obvious – here is the snipet that I would expect errors to be shown from that log.
... no IV derived for this protocol
%% Server resumed [Session-4, TLS_RSA_WITH_AES_128_CBC_SHA256]
NiFi Receiver, READ: TLSv1.2 Change Cipher Spec, length = 1
NiFi Receiver, READ: TLSv1.2 Handshake, length = 80
*** Finished
verify_data: { 109, 126, 134, 14, 33, 110, 224, 83, 198, 116, 54, 228 }
***
NiFi Receiver, WRITE: TLSv1.2 Change Cipher Spec, length = 1
*** Finished
verify_data: { 83, 120, 49, 158, 181, 136, 127, 219, 30, 194, 58, 167 }
***
NiFi Receiver, WRITE: TLSv1.2 Handshake, length = 80
NiFi Receiver, WRITE: TLSv1.2 Application Data, length = 240
I don’t really know enough about certificates and how client java apps would use them wrt to the host name/ ip address etc. of details is included in them. The nifi-user.log is showing access from a specific IP address which clearly doesn’t match the CN details in the cert. Just clutching at straws here!
Any other suggestions?
Thanks
Conrad
From: Bryan Bende <bb...@gmail.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Thursday, 19 May 2016 at 17:08
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Spark & NiFi question
Hi Conrad,
I think there are a couple of things at play here...
One is that the SSL properties need to be set on the SiteToSiteClientBuilder, rather than through system properties. There should be methods to set the keystore and other values.
In a secured NiFi instance, the certificate you are authenticating with (the keystore used by the s2s client) would need to have an account in NiFi, and would need to have access to the output port.
If you attempt to make a request with that cert, and then you go into the NiFi UI as another user, you should be able to go into the accounts section (top right) and approve the account for that certificate.
Then if you stop your output port, right-click and Configure... and from the Access Controls tab started typing the DN from your cert and add that user to the Allowed Users list. Hit Apply and started the port again.
We probably need to document this better, or write up an article about it somewhere.
Let us know if its still not working.
Thanks,
Bryan
On Thu, May 19, 2016 at 11:54 AM, Conrad Crampton <co...@secdata.com>> wrote:
Hi,
Tried following a couple of blog posts about this [1], [2], but neither of these refer to using NiFi in clustered environment with SSL and I suspect this is where I am hitting problems (but don’t know where).
The blogs state that using an output port (in the root process group I.e. on main canvas) which I have done and tried to connect thus..
System.setProperty("javax.net.ssl.keyStore", "/spark-processor.jks");
System.setProperty("javax.net.ssl.keyStorePassword", “*****");
System.setProperty("javax.net.ssl.trustStore", “/cacerts.jks");
SiteToSiteClientConfig config = new SiteToSiteClient.Builder()
.url("https://yarn-cm1.mis-cds.local:9090/nifi")
.portName("Spark test out")
.buildConfig();
SparkConf sparkConf = new SparkConf().setMaster("local[2]").setAppName("NiFi Spark Log Processor");
JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, new Duration(5000));
JavaReceiverInputDStream<NiFiDataPacket> packetStream = jssc.receiverStream(new NiFiReceiver(config, StorageLevel.MEMORY_ONLY()));
JavaDStream text = packetStream.map(dataPacket -> new String(dataPacket.getContent(), StandardCharsets.UTF_8));
text.print();
jssc.start();
jssc.awaitTermination();
The error I am getting is
16/05/19 16:39:03 WARN ReceiverSupervisorImpl: Restarting receiver with delay 2000 ms: Failed to receive data from NiFi
java.io.IOException: Server returned HTTP response code: 401 for URL: https://yarn-cm1.mis-cds.local:9090/nifi-api/controller
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1889)
at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1884)
at java.security.AccessController.doPrivileged(Native Method)
at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1883)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1456)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
at org.apache.nifi.remote.util.NiFiRestApiUtil.getController(NiFiRestApiUtil.java:69)
at org.apache.nifi.remote.client.socket.EndpointConnectionPool.refreshRemoteInfo(EndpointConnectionPool.java:891)
at org.apache.nifi.remote.client.socket.EndpointConnectionPool.getPortIdentifier(EndpointConnectionPool.java:878)
at org.apache.nifi.remote.client.socket.EndpointConnectionPool.getOutputPortIdentifier(EndpointConnectionPool.java:862)
at org.apache.nifi.remote.client.socket.SocketClient.getPortIdentifier(SocketClient.java:81)
at org.apache.nifi.remote.client.socket.SocketClient.createTransaction(SocketClient.java:123)
at org.apache.nifi.spark.NiFiReceiver$ReceiveRunnable.run(NiFiReceiver.java:149)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Server returned HTTP response code: 401 for URL: https://yarn-cm1.mis-cds.local:9090/nifi-api/controller
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1839)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
at org.apache.nifi.remote.util.NiFiRestApiUtil.getController(NiFiRestApiUtil.java:66)
... 7 more
Any pointers would be helpful in getting this working. I don’t know if I have to set up a remote process group with the output port (not sure how this works), or what. When I go to https://yarn-cm1.mis-cds.local:9090/nifi-api/controller in the browser, I get an access denied error.
I have created keystore and signed by the RootCA used to sign all the self signed certs for the cluster.
Running 0.6.1, 6 node cluster.
Thanks
Conrad
[1[ - https://community.hortonworks.com/articles/12708/nifi-feeding-data-to-spark-streaming.html
[2] - https://blogs.apache.org/nifi/entry/stream_processing_nifi_and_spark
SecureData, combating cyber threats
________________________________
The information contained in this message or any of its attachments may be privileged and confidential and intended for the exclusive use of the intended recipient. If you are not the intended recipient any disclosure, reproduction, distribution or other dissemination or use of this communications is strictly prohibited. The views expressed in this email are those of the individual and not necessarily of SecureData Europe Ltd. Any prices quoted are only valid if followed up by a formal written quote.
SecureData Europe Limited. Registered in England & Wales 04365896. Registered Address: SecureData House, Hermitage Court, Hermitage Lane, Maidstone, Kent, ME16 9NT
***This email originated outside SecureData***
Click here<https://www.mailcontrol.com/sr/JOj4ovws70LGX2PQPOmvUqa7UuQeNDoM5CPuVUMi!aLghcUmWuJbL8QAhL3vPgRnasXOF8Vdo14NCU1!U1Tbvw==> to report this email as spam.