You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Milan Das <md...@interset.com> on 2018/06/21 15:23:33 UTC
NIFI Multiple Kerberos configuration
Hello Team,
I have very unique problem. We are integration two kerberized haddop system and they have their own Kerbros setup.
Is it possible to two Kerberos kdc configurations in NIFI ? Integration is Kafka from one Hadoop to Kafka on 2nd Hadoop.
Really appreciate any thoughts.
Regards,
Milan Das
Milan Das
Sr. System Architect
email: mdas@interset.com
mobile: +1 678 216 5660
www.interset.com
Re: NIFI Multiple Kerberos configuration
Posted by Jeff <jt...@gmail.com>.
I'll have to set up a test this week and see if I can reproduce this. If
you'd like, you can file a JIRA [1] with sanitized details of your
krb5.conf and an example flow.
[1] https://issues.apache.org/jira/projects/NIFI/issues
On Sat, Jun 23, 2018 at 3:48 AM Hiroaki Miyanaga <hi...@gmail.com>
wrote:
> I tried a similar case last week and it could not access to both cluster at
> the same time.
>
> Try to connect kafka and hadoop managed by their own KDCs.
> I set both KDCs in realms section of krb5.conf.
> But NiFi looks using default realms in krb5.conf.
>
> I find a similar ticket.
>
> https://community.hortonworks.com/questions/149808/unable-to-connect-to-two-kdcs-from-nifi.html
>
>
> On Sat, Jun 23, 2018 at 4:01 AM, Jeff <jt...@gmail.com> wrote:
>
> > You can do this by configuring a realm for each KDC to krb5.conf.
> >
> > On Fri, Jun 22, 2018 at 10:37 AM Bryan Bende <bb...@gmail.com> wrote:
> >
> > > Java assumes there is one krb5.conf file loaded by the JVM. It looks
> > > for the system property java.security.krb5.conf or falls back to
> > > looking in well-known locations, but still only expects one [1].
> > >
> > > NiFi requires you to set the location in nifi.properties and uses that
> > > value to set the system property above.
> > >
> > > There may be a way to create a single krb5.conf with multiple KDCs,
> > > but I'm not sure exactly how to do it.
> > >
> > > [1]
> > > https://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/
> > tutorials/KerberosReq.html
> > >
> > > On Fri, Jun 22, 2018 at 10:10 AM, Milan Das <md...@interset.com> wrote:
> > > > The problem is krb5.conf. There are two different krb5.conf with two
> > > different kdc server.
> > > > Regards,
> > > > Milan Das
> > > >
> > > > On 6/22/18, 2:04 AM, "Koji Kawamura" <ij...@gmail.com>
> wrote:
> > > >
> > > > Hi Milan,
> > > >
> > > > I haven't tried myself, but since NiFi has Kerberos configuration
> > per
> > > > Processor instance, e.g. ListHDFS or PutHDFS, NiFi should be able
> > to
> > > > connect multiple Hadoop clusters accessed by different Kerberos
> > > principals
> > > > and keytabs. Principals must resolve domain (realm) correctly, if
> > > both
> > > > Hadoop cluster use the same domain such as 'EXAMPLE.COM', then
> it
> > > will be
> > > > problematic for NiFi to find the right KDC server.
> > > >
> > > > Thanks,
> > > > Koji
> > > >
> > > > On Fri, Jun 22, 2018 at 12:23 AM, Milan Das <md...@interset.com>
> > > wrote:
> > > >
> > > > > Hello Team,
> > > > >
> > > > > I have very unique problem. We are integration two kerberized
> > > haddop
> > > > > system and they have their own Kerbros setup.
> > > > >
> > > > > Is it possible to two Kerberos kdc configurations in NIFI ?
> > > Integration is
> > > > > Kafka from one Hadoop to Kafka on 2nd Hadoop.
> > > > >
> > > > > Really appreciate any thoughts.
> > > > >
> > > > >
> > > > >
> > > > > Regards,
> > > > >
> > > > > Milan Das
> > > > >
> > > > >
> > > > >
> > > > > [image: ograph]
> > > > >
> > > > > *Milan Das*
> > > > > Sr. System Architect
> > > > >
> > > > > email: mdas@interset.com
> > > > > mobile: +1 678 216 5660 <(678)%20216-5660> <(678)%20216-5660>
> > > > >
> > > > > [image: edIn icon] <https://www.linkedin.com/in/milandas/>
> > > > >
> > > > > www.interset.com
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> >
>
Re: NIFI Multiple Kerberos configuration
Posted by Hiroaki Miyanaga <hi...@gmail.com>.
I tried a similar case last week and it could not access to both cluster at
the same time.
Try to connect kafka and hadoop managed by their own KDCs.
I set both KDCs in realms section of krb5.conf.
But NiFi looks using default realms in krb5.conf.
I find a similar ticket.
https://community.hortonworks.com/questions/149808/unable-to-connect-to-two-kdcs-from-nifi.html
On Sat, Jun 23, 2018 at 4:01 AM, Jeff <jt...@gmail.com> wrote:
> You can do this by configuring a realm for each KDC to krb5.conf.
>
> On Fri, Jun 22, 2018 at 10:37 AM Bryan Bende <bb...@gmail.com> wrote:
>
> > Java assumes there is one krb5.conf file loaded by the JVM. It looks
> > for the system property java.security.krb5.conf or falls back to
> > looking in well-known locations, but still only expects one [1].
> >
> > NiFi requires you to set the location in nifi.properties and uses that
> > value to set the system property above.
> >
> > There may be a way to create a single krb5.conf with multiple KDCs,
> > but I'm not sure exactly how to do it.
> >
> > [1]
> > https://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/
> tutorials/KerberosReq.html
> >
> > On Fri, Jun 22, 2018 at 10:10 AM, Milan Das <md...@interset.com> wrote:
> > > The problem is krb5.conf. There are two different krb5.conf with two
> > different kdc server.
> > > Regards,
> > > Milan Das
> > >
> > > On 6/22/18, 2:04 AM, "Koji Kawamura" <ij...@gmail.com> wrote:
> > >
> > > Hi Milan,
> > >
> > > I haven't tried myself, but since NiFi has Kerberos configuration
> per
> > > Processor instance, e.g. ListHDFS or PutHDFS, NiFi should be able
> to
> > > connect multiple Hadoop clusters accessed by different Kerberos
> > principals
> > > and keytabs. Principals must resolve domain (realm) correctly, if
> > both
> > > Hadoop cluster use the same domain such as 'EXAMPLE.COM', then it
> > will be
> > > problematic for NiFi to find the right KDC server.
> > >
> > > Thanks,
> > > Koji
> > >
> > > On Fri, Jun 22, 2018 at 12:23 AM, Milan Das <md...@interset.com>
> > wrote:
> > >
> > > > Hello Team,
> > > >
> > > > I have very unique problem. We are integration two kerberized
> > haddop
> > > > system and they have their own Kerbros setup.
> > > >
> > > > Is it possible to two Kerberos kdc configurations in NIFI ?
> > Integration is
> > > > Kafka from one Hadoop to Kafka on 2nd Hadoop.
> > > >
> > > > Really appreciate any thoughts.
> > > >
> > > >
> > > >
> > > > Regards,
> > > >
> > > > Milan Das
> > > >
> > > >
> > > >
> > > > [image: ograph]
> > > >
> > > > *Milan Das*
> > > > Sr. System Architect
> > > >
> > > > email: mdas@interset.com
> > > > mobile: +1 678 216 5660 <(678)%20216-5660>
> > > >
> > > > [image: edIn icon] <https://www.linkedin.com/in/milandas/>
> > > >
> > > > www.interset.com
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> >
>
Re: NIFI Multiple Kerberos configuration
Posted by Jeff <jt...@gmail.com>.
You can do this by configuring a realm for each KDC to krb5.conf.
On Fri, Jun 22, 2018 at 10:37 AM Bryan Bende <bb...@gmail.com> wrote:
> Java assumes there is one krb5.conf file loaded by the JVM. It looks
> for the system property java.security.krb5.conf or falls back to
> looking in well-known locations, but still only expects one [1].
>
> NiFi requires you to set the location in nifi.properties and uses that
> value to set the system property above.
>
> There may be a way to create a single krb5.conf with multiple KDCs,
> but I'm not sure exactly how to do it.
>
> [1]
> https://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/tutorials/KerberosReq.html
>
> On Fri, Jun 22, 2018 at 10:10 AM, Milan Das <md...@interset.com> wrote:
> > The problem is krb5.conf. There are two different krb5.conf with two
> different kdc server.
> > Regards,
> > Milan Das
> >
> > On 6/22/18, 2:04 AM, "Koji Kawamura" <ij...@gmail.com> wrote:
> >
> > Hi Milan,
> >
> > I haven't tried myself, but since NiFi has Kerberos configuration per
> > Processor instance, e.g. ListHDFS or PutHDFS, NiFi should be able to
> > connect multiple Hadoop clusters accessed by different Kerberos
> principals
> > and keytabs. Principals must resolve domain (realm) correctly, if
> both
> > Hadoop cluster use the same domain such as 'EXAMPLE.COM', then it
> will be
> > problematic for NiFi to find the right KDC server.
> >
> > Thanks,
> > Koji
> >
> > On Fri, Jun 22, 2018 at 12:23 AM, Milan Das <md...@interset.com>
> wrote:
> >
> > > Hello Team,
> > >
> > > I have very unique problem. We are integration two kerberized
> haddop
> > > system and they have their own Kerbros setup.
> > >
> > > Is it possible to two Kerberos kdc configurations in NIFI ?
> Integration is
> > > Kafka from one Hadoop to Kafka on 2nd Hadoop.
> > >
> > > Really appreciate any thoughts.
> > >
> > >
> > >
> > > Regards,
> > >
> > > Milan Das
> > >
> > >
> > >
> > > [image: ograph]
> > >
> > > *Milan Das*
> > > Sr. System Architect
> > >
> > > email: mdas@interset.com
> > > mobile: +1 678 216 5660 <(678)%20216-5660>
> > >
> > > [image: edIn icon] <https://www.linkedin.com/in/milandas/>
> > >
> > > www.interset.com
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
>
Re: NIFI Multiple Kerberos configuration
Posted by Bryan Bende <bb...@gmail.com>.
Java assumes there is one krb5.conf file loaded by the JVM. It looks
for the system property java.security.krb5.conf or falls back to
looking in well-known locations, but still only expects one [1].
NiFi requires you to set the location in nifi.properties and uses that
value to set the system property above.
There may be a way to create a single krb5.conf with multiple KDCs,
but I'm not sure exactly how to do it.
[1] https://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/tutorials/KerberosReq.html
On Fri, Jun 22, 2018 at 10:10 AM, Milan Das <md...@interset.com> wrote:
> The problem is krb5.conf. There are two different krb5.conf with two different kdc server.
> Regards,
> Milan Das
>
> On 6/22/18, 2:04 AM, "Koji Kawamura" <ij...@gmail.com> wrote:
>
> Hi Milan,
>
> I haven't tried myself, but since NiFi has Kerberos configuration per
> Processor instance, e.g. ListHDFS or PutHDFS, NiFi should be able to
> connect multiple Hadoop clusters accessed by different Kerberos principals
> and keytabs. Principals must resolve domain (realm) correctly, if both
> Hadoop cluster use the same domain such as 'EXAMPLE.COM', then it will be
> problematic for NiFi to find the right KDC server.
>
> Thanks,
> Koji
>
> On Fri, Jun 22, 2018 at 12:23 AM, Milan Das <md...@interset.com> wrote:
>
> > Hello Team,
> >
> > I have very unique problem. We are integration two kerberized haddop
> > system and they have their own Kerbros setup.
> >
> > Is it possible to two Kerberos kdc configurations in NIFI ? Integration is
> > Kafka from one Hadoop to Kafka on 2nd Hadoop.
> >
> > Really appreciate any thoughts.
> >
> >
> >
> > Regards,
> >
> > Milan Das
> >
> >
> >
> > [image: ograph]
> >
> > *Milan Das*
> > Sr. System Architect
> >
> > email: mdas@interset.com
> > mobile: +1 678 216 5660
> >
> > [image: edIn icon] <https://www.linkedin.com/in/milandas/>
> >
> > www.interset.com
> >
> >
> >
> >
> >
>
>
>
Re: NIFI Multiple Kerberos configuration
Posted by Milan Das <md...@interset.com>.
The problem is krb5.conf. There are two different krb5.conf with two different kdc server.
Regards,
Milan Das
On 6/22/18, 2:04 AM, "Koji Kawamura" <ij...@gmail.com> wrote:
Hi Milan,
I haven't tried myself, but since NiFi has Kerberos configuration per
Processor instance, e.g. ListHDFS or PutHDFS, NiFi should be able to
connect multiple Hadoop clusters accessed by different Kerberos principals
and keytabs. Principals must resolve domain (realm) correctly, if both
Hadoop cluster use the same domain such as 'EXAMPLE.COM', then it will be
problematic for NiFi to find the right KDC server.
Thanks,
Koji
On Fri, Jun 22, 2018 at 12:23 AM, Milan Das <md...@interset.com> wrote:
> Hello Team,
>
> I have very unique problem. We are integration two kerberized haddop
> system and they have their own Kerbros setup.
>
> Is it possible to two Kerberos kdc configurations in NIFI ? Integration is
> Kafka from one Hadoop to Kafka on 2nd Hadoop.
>
> Really appreciate any thoughts.
>
>
>
> Regards,
>
> Milan Das
>
>
>
> [image: ograph]
>
> *Milan Das*
> Sr. System Architect
>
> email: mdas@interset.com
> mobile: +1 678 216 5660
>
> [image: edIn icon] <https://www.linkedin.com/in/milandas/>
>
> www.interset.com
>
>
>
>
>
Re: NIFI Multiple Kerberos configuration
Posted by Koji Kawamura <ij...@gmail.com>.
Hi Milan,
I haven't tried myself, but since NiFi has Kerberos configuration per
Processor instance, e.g. ListHDFS or PutHDFS, NiFi should be able to
connect multiple Hadoop clusters accessed by different Kerberos principals
and keytabs. Principals must resolve domain (realm) correctly, if both
Hadoop cluster use the same domain such as 'EXAMPLE.COM', then it will be
problematic for NiFi to find the right KDC server.
Thanks,
Koji
On Fri, Jun 22, 2018 at 12:23 AM, Milan Das <md...@interset.com> wrote:
> Hello Team,
>
> I have very unique problem. We are integration two kerberized haddop
> system and they have their own Kerbros setup.
>
> Is it possible to two Kerberos kdc configurations in NIFI ? Integration is
> Kafka from one Hadoop to Kafka on 2nd Hadoop.
>
> Really appreciate any thoughts.
>
>
>
> Regards,
>
> Milan Das
>
>
>
> [image: ograph]
>
> *Milan Das*
> Sr. System Architect
>
> email: mdas@interset.com
> mobile: +1 678 216 5660
>
> [image: edIn icon] <https://www.linkedin.com/in/milandas/>
>
> www.interset.com
>
>
>
>
>