You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by dkommineni <dk...@gmail.com> on 2019/10/09 19:34:47 UTC
Apache Nifi with IBM Event Streams
I am trying to use Apache Nifi docker image "apache/nifi" to connect to Kafka
in IBM Event Streams platform. I am trying to use "SASL_SSL" as the security
protocol.
My questions are:
What is the "Kerberos Service Name" to be used? If some configuration
required how and where to setup?
What is the "SSL Context Service" and how do I create one?
I tried adding additional properties like "sasl.mechanism=PLAIN"and
"sasl.jaas.config" as
org.apache.kafka.common.security.plain.PlainLoginModule required
username="XxxxXXxxx password="XxxXxxxXXxxxxx";
--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
Re: Apache Nifi with IBM Event Streams
Posted by Bryan Bende <bb...@gmail.com>.
https://issues.apache.org/jira/browse/NIFI-4820
On Thu, Oct 10, 2019 at 9:56 AM Joe Witt <jo...@gmail.com> wrote:
>
> You can use SASL/Plain today with the kafka 2 procs in NiFi but as noted it is unpleasant to configure. A much easier/clear configuration is being worked on right now. Not positive what the JIRA for it is though. Should be available quite soon.
>
> Thanks
>
> On Thu, Oct 10, 2019 at 9:31 AM dkommineni <dk...@gmail.com> wrote:
>>
>> Thanks Chris.
>>
>> I am trying to setup Kafka for "PublishKafka_2_0".
>>
>> I did followed the same steps, when I try to setup SSLContextService, I am
>> not sure what details to provide for keystore/truststore file name/path(s).
>>
>> When I run Java application to connect to IBM Event Streams, the kafka log
>> does show *null* for keystore/truststore path.
>>
>> I am attaching Nifi setup and Java logs for reference.
>>
>> <http://apache-nifi-users-list.2361937.n4.nabble.com/file/t785/Screen_Shot_2019-10-10_at_9.png>
>> <http://apache-nifi-users-list.2361937.n4.nabble.com/file/t785/Screen_Shot_2019-10-10_at_9.png>
>> <http://apache-nifi-users-list.2361937.n4.nabble.com/file/t785/Screen_Shot_2019-10-10_at_9.png>
>> <http://apache-nifi-users-list.2361937.n4.nabble.com/file/t785/Screen_Shot_2019-10-10_at_9.png>
>> <http://apache-nifi-users-list.2361937.n4.nabble.com/file/t785/Screen_Shot_2019-10-10_at_9.png>
>>
>>
>> Regards
>> Devaraj
>>
>>
>>
>> --
>> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
Re: Apache Nifi with IBM Event Streams
Posted by Joe Witt <jo...@gmail.com>.
You can use SASL/Plain today with the kafka 2 procs in NiFi but as noted it
is unpleasant to configure. A much easier/clear configuration is being
worked on right now. Not positive what the JIRA for it is though. Should
be available quite soon.
Thanks
On Thu, Oct 10, 2019 at 9:31 AM dkommineni <dk...@gmail.com> wrote:
> Thanks Chris.
>
> I am trying to setup Kafka for "PublishKafka_2_0".
>
> I did followed the same steps, when I try to setup SSLContextService, I am
> not sure what details to provide for keystore/truststore file
> name/path(s).
>
> When I run Java application to connect to IBM Event Streams, the kafka log
> does show *null* for keystore/truststore path.
>
> I am attaching Nifi setup and Java logs for reference.
>
> <
> http://apache-nifi-users-list.2361937.n4.nabble.com/file/t785/Screen_Shot_2019-10-10_at_9.png>
>
> <
> http://apache-nifi-users-list.2361937.n4.nabble.com/file/t785/Screen_Shot_2019-10-10_at_9.png>
>
> <
> http://apache-nifi-users-list.2361937.n4.nabble.com/file/t785/Screen_Shot_2019-10-10_at_9.png>
>
> <
> http://apache-nifi-users-list.2361937.n4.nabble.com/file/t785/Screen_Shot_2019-10-10_at_9.png>
>
> <
> http://apache-nifi-users-list.2361937.n4.nabble.com/file/t785/Screen_Shot_2019-10-10_at_9.png>
>
>
>
> Regards
> Devaraj
>
>
>
> --
> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
>
Re: Apache Nifi with IBM Event Streams
Posted by dkommineni <dk...@gmail.com>.
Thanks Bryan.
I am using the docker container for running Apache Nifi. I got the JABA_HOME
PATH and navigated to the cacerts file and configured SSLSecurityContext and
it is working now.
Regards
Devaraj
--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
Re: Apache Nifi with IBM Event Streams
Posted by Bryan Bende <bb...@gmail.com>.
You can't create a TLS connection to something if you don't have
access to the public key of the CA. This is how all TLS/https works,
your browser has the public keys of all the major CAs in order to
trust the websites you visit over https.
If IBM Cloud is generating certificates and signing them with one of
the major CAs, then you might be able to use a truststore that points
to the system CA certs...
http://apache-nifi.1125220.n5.nabble.com/Nifi-GetHttp-https-endpoint-keystore-password-error-td1216.html
https://mail-archives.apache.org/mod_mbox/nifi-users/201509.mbox/%3CEB8BF379-164B-49D0-82B5-7043F2891FC1@gmail.com%3E
On Thu, Oct 10, 2019 at 11:45 AM dkommineni <dk...@gmail.com> wrote:
>
> Hi Bryan,
>
> The problem is I am using Kafka hosted in IBM Cloud and I don't have access
> to any certificates/keys.
>
> Regards
> Devaraj
>
>
>
> --
> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
Re: Apache Nifi with IBM Event Streams
Posted by dkommineni <dk...@gmail.com>.
Hi Bryan,
The problem is I am using Kafka hosted in IBM Cloud and I don't have access
to any certificates/keys.
Regards
Devaraj
--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
Re: Apache Nifi with IBM Event Streams
Posted by Bryan Bende <bb...@gmail.com>.
You need to set the truststore filename and password.
The truststore needs to have public key of the CA that signed the
certs used by the Kafka broker (i.e. it needs to trust the certs used
by the Kafka broker).
On Thu, Oct 10, 2019 at 10:00 AM dkommineni <dk...@gmail.com> wrote:
>
> I think I have messed up with attachments.
> <http://apache-nifi-users-list.2361937.n4.nabble.com/file/t785/nifi_processor.png>
> <http://apache-nifi-users-list.2361937.n4.nabble.com/file/t785/nifi_producer_kafka_1.png>
> <http://apache-nifi-users-list.2361937.n4.nabble.com/file/t785/nifi_producer_kafka_2.png>
> <http://apache-nifi-users-list.2361937.n4.nabble.com/file/t785/nifi_security_context.png>
> <http://apache-nifi-users-list.2361937.n4.nabble.com/file/t785/java_kafka_log.png>
>
>
>
>
> --
> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
RE: Apache Nifi with IBM Event Streams
Posted by dkommineni <dk...@gmail.com>.
I think I have messed up with attachments.
<http://apache-nifi-users-list.2361937.n4.nabble.com/file/t785/nifi_processor.png>
<http://apache-nifi-users-list.2361937.n4.nabble.com/file/t785/nifi_producer_kafka_1.png>
<http://apache-nifi-users-list.2361937.n4.nabble.com/file/t785/nifi_producer_kafka_2.png>
<http://apache-nifi-users-list.2361937.n4.nabble.com/file/t785/nifi_security_context.png>
<http://apache-nifi-users-list.2361937.n4.nabble.com/file/t785/java_kafka_log.png>
--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
RE: Apache Nifi with IBM Event Streams
Posted by dkommineni <dk...@gmail.com>.
Thanks Chris.
I am trying to setup Kafka for "PublishKafka_2_0".
I did followed the same steps, when I try to setup SSLContextService, I am
not sure what details to provide for keystore/truststore file name/path(s).
When I run Java application to connect to IBM Event Streams, the kafka log
does show *null* for keystore/truststore path.
I am attaching Nifi setup and Java logs for reference.
<http://apache-nifi-users-list.2361937.n4.nabble.com/file/t785/Screen_Shot_2019-10-10_at_9.png>
<http://apache-nifi-users-list.2361937.n4.nabble.com/file/t785/Screen_Shot_2019-10-10_at_9.png>
<http://apache-nifi-users-list.2361937.n4.nabble.com/file/t785/Screen_Shot_2019-10-10_at_9.png>
<http://apache-nifi-users-list.2361937.n4.nabble.com/file/t785/Screen_Shot_2019-10-10_at_9.png>
<http://apache-nifi-users-list.2361937.n4.nabble.com/file/t785/Screen_Shot_2019-10-10_at_9.png>
Regards
Devaraj
--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
RE: Apache Nifi with IBM Event Streams
Posted by ch...@post.ch.
Hi
It seems you want to configure ConsumeKafka / ConsumeKafkaRecord for SASL_SSL PLAIN. It is possible but not straightforward.
The "Kerberos Service Name" cannot be empty but its content does not matter since it will not be used in this case.
The trick is to specify the 2 additional properties sasl.jaas.config and sasl.mechanism, exactly like you did.
You must create a Controller Service of type StandardSSLContextService, see https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-ssl-context-service-nar/1.5.0/org.apache.nifi.ssl.StandardSSLContextService/index.html
And reference the Truststore so that Nifi can securely connect to your Kafka platform.
Gruss
Chris
-----Original Message-----
From: dkommineni <dk...@gmail.com>
Sent: Wednesday, October 9, 2019 21:35
To: users@nifi.apache.org
Subject: Apache Nifi with IBM Event Streams
I am trying to use Apache Nifi docker image "apache/nifi" to connect to Kafka in IBM Event Streams platform. I am trying to use "SASL_SSL" as the security protocol.
My questions are:
What is the "Kerberos Service Name" to be used? If some configuration required how and where to setup?
What is the "SSL Context Service" and how do I create one?
I tried adding additional properties like "sasl.mechanism=PLAIN"and "sasl.jaas.config" as org.apache.kafka.common.security.plain.PlainLoginModule required username="XxxxXXxxx password="XxxXxxxXXxxxxx";
--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/