You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Maulin Vasavada <ma...@gmail.com> on 2019/08/08 06:31:22 UTC
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Bump! Can somebody please review this?
On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <ma...@gmail.com>
wrote:
> Bump! Can somebody please review this?
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Colin McCabe <cm...@apache.org>.
Harsha made a good point that you can achieve your goals through KIP-492. Security configuration is starting to get pretty complex-- is there a reason not to use the existing configurations?
Also, it seems like most people who want a custom truststore / keystore will also want a custom SSL provider, so that they can integrate their custom SSL infra with more applications than just Kafka. So I'm not sure how big the audience for the proposed new configurations would be.
regards,
Colin
On Thu, Aug 8, 2019, at 15:29, Maulin Vasavada wrote:
> Hi Harsha
>
> We don't have spire (or similar) agents and we do not have keys/certs
> locally on any brokers. To elaborate more on my previous email,
>
> I agree that Java security Providers are used in much broader sense - to
> have a particular implementation of an algorithm, use specific cipher
> suites for SSL , OR in our current team's case have a particular way to
> leverage pre-generated SSL sessions. However, the scope of our KIP (486) is
> much restricted than that. We merely intend to provide a custom
> keystore/truststore for our SSL connections and not really worry about
> underlying specific SSL/TLS implementation. This simplifies it a lot for
> us to keep the concerns separate and clear.
>
> I feel our approach is more complimentary such that it allows for using
> keystores of choice while retaining the flexibility to use any
> underlying/available Provider for actually making the SSL call.
>
> We agree with KIP-492's approach based on Providers (and Java's
> recommendation), but also strongly believe that our approach can compliment
> it very effectively for reasons explained above.
>
> Thanks
> Maulin
>
> On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <ka...@harsha.io> wrote:
>
> > Hi Maulin,
> >
> > On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
> > maulin.vasavada@gmail.com>
> > wrote:
> >
> > > Hi Harsha
> > >
> > > The reason we rejected the SslProvider route is that - we only needed a
> > > custom way to load keys/certs. Not touch any policy that existing
> > Providers
> > > govern like SunJSSE Provider.
> > >
> >
> > We have exactly the same requirements to load certs and keys through spire
> > agent. We used security.provider to do that exactly. I am not sure why you
> > would be modifying any policies provided by default SunJSSE provider. Can
> > you give me an example of having custom provider that will override an
> > existing policy in SunJSSE provider.
> >
> > As pointed out earlier, this kip
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> > allows
> > you to load security.provider through config.
> > Take a look at the examples I gave before
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
> > It registers KeyManagerFactory and TrustManagerFactory and Keystore
> > algorithm.
> > Implement your custom way of loading Keystore in here
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java
> >
> > and Trust manager like here
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java
> >
> > In your Kafka client you can set the security.provider to your custom
> > implementation and with this fix
> > https://issues.apache.org/jira/browse/KAFKA-8191 you can set
> > keyManagerAlgorigthm and trustManagerAlgorithm configs.
> >
> > All of this is in your clients and broker side and do not need to touch any
> > policy changes at JVM level. You'll register the providers in the priority
> > order and can still have SunJSSE provider and have your custom provider to
> > implement the key and trust managers.
> >
> >
> >
> > The ask here is different than KIP-492. We don't have any need to
> > > modify/specify the algorithm parameter. Does that make sense?
> > >
> >
> > The ask in KIP is introducing new interfaces where the KIP's
> > goal/motivation can be achieved through the security.provider and we worked
> > on similar goal without touching any Keystore or Truststore interfaces.
> > My advise is against changing or introducing new interfaces when it can
> > work through security.provider.
> >
> > Thanks,
> > Harsha
> >
> >
> > Thanks
> > > Maulin
> > >
> > > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <ka...@harsha.io>
> > > wrote:
> > >
> > > In your KIP you added security. provider as rejected alternative and
> > > specified "its not the correct way". Do you mind explaining why its not?
> > I
> > > didn't find any evidence in Java docs to say so. Contrary to your
> > statement
> > > it does say in the java docs
> > > " However, please note that a provider can be used to implement any
> > > security service in Java that uses a pluggable architecture with a choice
> > > of implementations that fit underneath."
> > >
> > > Java Security Providers have been used by other projects to provide such
> > > integration . I am not sure if you looked into Spiffe project to
> > > efficiently distribute certificates but here is an example of Java
> > provider
> > >
> > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >
> > spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > > java which
> > > obtains certificates from local daemons.
> > > These integrations are being used in Tomcat, Jetty etc.. We are also
> > using
> > > Security provider to do the same in our Kafka clusters. So unless I see
> > > more evidence why security.provider doesn't work for you adding new
> > > interfaces while there exists more cleaner way of achieving the goals of
> > > this KIP is unnecessary and breaks the well known security interfaces
> > > provided by Java itself.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <ka...@harsha.io>
> > > wrote:
> > >
> > > Hi Maulin,
> > > Not sure if you looked at my previous replies. This
> > >
> > > changes
> > >
> > > are not required as there is already security Provider to do what you are
> > > proposing. This KIP https://cwiki.apache.org/confluence/display/KAFKA/
> > > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config also
> > > addresses easy registration of such providers.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada <maulin.vasavada@gmail.
> > > com> wrote:
> > >
> > > Bump! Can somebody please review this?
> > >
> > > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
> > >
> > > maulin.vasavada@gmail.com>
> > >
> > > wrote:
> > >
> > > Bump! Can somebody please review this?
> > >
> > >
> >
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Maulin Vasavada <ma...@gmail.com>.
Hi Colin,
To your point - "Also, it seems like most people who want a custom
truststore / keystore will also want a custom SSL provider," I don't think
so. Take our own example. We have a fairly large Kafka eco-system (500B+
messages a day flowing through with poly-glot client-base) with strict
InfoSec requirements but we still do not need Custom SSL Provider for Kafka.
Thanks
Maulin
On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <ma...@gmail.com>
wrote:
> Imagine a scenario like - We know we have a custom KMS and as a Kafka
> owner we want to comply to using that KMS source to load keys/certs. As a
> Kafka owner we know how to integrate with KMS but doesn't necessarily have
> to know anything about cipher suites, algorithms, and SSL/TLS
> implementation. Going the Provider way requires to know lot more than we
> should, isn't it? Not that we would have concern/shy-away knowing those
> details - but if we don't have to - why should we?
>
> On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <ma...@gmail.com>
> wrote:
>
>> Hi Harsha
>>
>> We don't have spire (or similar) agents and we do not have keys/certs
>> locally on any brokers. To elaborate more on my previous email,
>>
>> I agree that Java security Providers are used in much broader sense - to
>> have a particular implementation of an algorithm, use specific cipher
>> suites for SSL , OR in our current team's case have a particular way to
>> leverage pre-generated SSL sessions. However, the scope of our KIP (486) is
>> much restricted than that. We merely intend to provide a custom
>> keystore/truststore for our SSL connections and not really worry about
>> underlying specific SSL/TLS implementation. This simplifies it a lot for
>> us to keep the concerns separate and clear.
>>
>> I feel our approach is more complimentary such that it allows for using
>> keystores of choice while retaining the flexibility to use any
>> underlying/available Provider for actually making the SSL call.
>>
>> We agree with KIP-492's approach based on Providers (and Java's
>> recommendation), but also strongly believe that our approach can compliment
>> it very effectively for reasons explained above.
>>
>> Thanks
>> Maulin
>>
>> On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <ka...@harsha.io>
>> wrote:
>>
>>> Hi Maulin,
>>>
>>> On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
>>> maulin.vasavada@gmail.com>
>>> wrote:
>>>
>>> > Hi Harsha
>>> >
>>> > The reason we rejected the SslProvider route is that - we only needed a
>>> > custom way to load keys/certs. Not touch any policy that existing
>>> Providers
>>> > govern like SunJSSE Provider.
>>> >
>>>
>>> We have exactly the same requirements to load certs and keys through
>>> spire
>>> agent. We used security.provider to do that exactly. I am not sure why
>>> you
>>> would be modifying any policies provided by default SunJSSE provider.
>>> Can
>>> you give me an example of having custom provider that will override an
>>> existing policy in SunJSSE provider.
>>>
>>> As pointed out earlier, this kip
>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>>> allows
>>> you to load security.provider through config.
>>> Take a look at the examples I gave before
>>>
>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
>>> It registers KeyManagerFactory and TrustManagerFactory and Keystore
>>> algorithm.
>>> Implement your custom way of loading Keystore in here
>>>
>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java
>>>
>>> and Trust manager like here
>>>
>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java
>>>
>>> In your Kafka client you can set the security.provider to your custom
>>> implementation and with this fix
>>> https://issues.apache.org/jira/browse/KAFKA-8191 you can set
>>> keyManagerAlgorigthm and trustManagerAlgorithm configs.
>>>
>>> All of this is in your clients and broker side and do not need to touch
>>> any
>>> policy changes at JVM level. You'll register the providers in the
>>> priority
>>> order and can still have SunJSSE provider and have your custom provider
>>> to
>>> implement the key and trust managers.
>>>
>>>
>>>
>>> The ask here is different than KIP-492. We don't have any need to
>>> > modify/specify the algorithm parameter. Does that make sense?
>>> >
>>>
>>> The ask in KIP is introducing new interfaces where the KIP's
>>> goal/motivation can be achieved through the security.provider and we
>>> worked
>>> on similar goal without touching any Keystore or Truststore interfaces.
>>> My advise is against changing or introducing new interfaces when it can
>>> work through security.provider.
>>>
>>> Thanks,
>>> Harsha
>>>
>>>
>>> Thanks
>>> > Maulin
>>> >
>>> > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <ka...@harsha.io>
>>> > wrote:
>>> >
>>> > In your KIP you added security. provider as rejected alternative and
>>> > specified "its not the correct way". Do you mind explaining why its
>>> not? I
>>> > didn't find any evidence in Java docs to say so. Contrary to your
>>> statement
>>> > it does say in the java docs
>>> > " However, please note that a provider can be used to implement any
>>> > security service in Java that uses a pluggable architecture with a
>>> choice
>>> > of implementations that fit underneath."
>>> >
>>> > Java Security Providers have been used by other projects to provide
>>> such
>>> > integration . I am not sure if you looked into Spiffe project to
>>> > efficiently distribute certificates but here is an example of Java
>>> provider
>>> >
>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>>> >
>>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>>> > java which
>>> > obtains certificates from local daemons.
>>> > These integrations are being used in Tomcat, Jetty etc.. We are also
>>> using
>>> > Security provider to do the same in our Kafka clusters. So unless I see
>>> > more evidence why security.provider doesn't work for you adding new
>>> > interfaces while there exists more cleaner way of achieving the goals
>>> of
>>> > this KIP is unnecessary and breaks the well known security interfaces
>>> > provided by Java itself.
>>> >
>>> > Thanks,
>>> > Harsha
>>> >
>>> > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <ka...@harsha.io>
>>> > wrote:
>>> >
>>> > Hi Maulin,
>>> > Not sure if you looked at my previous replies. This
>>> >
>>> > changes
>>> >
>>> > are not required as there is already security Provider to do what you
>>> are
>>> > proposing. This KIP https://cwiki.apache.org/confluence/display/KAFKA/
>>> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config also
>>> > addresses easy registration of such providers.
>>> >
>>> > Thanks,
>>> > Harsha
>>> >
>>> > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
>>> <maulin.vasavada@gmail.
>>> > com> wrote:
>>> >
>>> > Bump! Can somebody please review this?
>>> >
>>> > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
>>> >
>>> > maulin.vasavada@gmail.com>
>>> >
>>> > wrote:
>>> >
>>> > Bump! Can somebody please review this?
>>> >
>>> >
>>>
>>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Maulin Vasavada <ma...@gmail.com>.
Instead of keep diverging in different directions, it would be helpful if
you guys take my detailed posts with 1st to 4th points I mentioned and
start referring/commenting on each of those if you agree with them or not.
On Mon, Aug 19, 2019 at 10:45 PM Maulin Vasavada <ma...@gmail.com>
wrote:
> Hi Colin
>
> When I refer to "standard" or "custom" algorithms I am following Java
> security Provider Terminology. You can refer to
> https://docs.oracle.com/javase/7/docs/technotes/guides/security/StandardNames.html#TrustManagerFactory
> link I provided earlier in the emails. It says PKIX is the default
> Algorithm for TrustManagerFactory.
>
> 1. For SPIFFE, I am not sure why you are saying 'it does not implement
> custom algorithms' because the following file clearly indicates that it
> does use custom algorithm-
>
>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/SpiffeProvider.java#L17
>
>
> Algorithm value:
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/SpiffeProviderConstants.java#L6
>
> @Harsha do you want to chime in since you use that provider?
>
> 2. I already mentioned in my 3rd point, in my previous post, why using
> ssl.provider does NOT work. I updated KIP-486 in "rejected alternatives"
> also why ssl.provider does not work.
>
> 3. Security.insertProviderAt() comments were based on assumption if
> KIP-492 changes are done and we use that mechanism to configure providers
> instead of ssl.provider configuration.
>
> Can you read my all the points, I mentioned in my previous post, very
> carefully? I am covering all the aspects in explaining. I am open to still
> discuss more to clarify any doubts.
>
> Thanks
> Maulin
>
>
>
> On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <cm...@apache.org> wrote:
>
>> Hi Maulin,
>>
>> A lot of JSSE providers don't implement custom algorithms. Spire is a
>> good example of a JSSE provider that doesn't, and yet is still useful to
>> many people. Your JSSE provider can work fine even if it doesn't implement
>> a custom algorithm.
>>
>> Maybe I'm missing something, but I don't understand the discussion of
>> Security.insertProviderAt() that you included. SslEngineBuilder doesn't
>> use that API to get the security provider. Instead, it calls
>> "SSLContext.getInstance(protocol, provider)", where provider is the name of
>> the provider.
>>
>> best,
>> Colin
>>
>>
>> On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
>> > On top of everything above I feel strongly to add the 4th point which is
>> > based on Java APIs for TrustManagerFactory.init(KeyStore) (
>> >
>> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/TrustManagerFactory.html#init(java.security.KeyStore)
>> )
>> > and KeyManagerFactory.init(KeyStore, char[]) (
>> >
>> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory.html#init(java.security.KeyStore,%20char[])
>> > ).
>> >
>> > 4. The above APIs are intended to support providing "trust/key material"
>> > from the user without having to write their own
>> TrustManager/KeyManagers.
>> > To quote from the TrustManagerFactory.init()'s documentation
>> "Initializes
>> > this factory with a source of certificate authorities and related trust
>> > material."
>> > To quote from the KeyManagerFactory.init()'s documentation "Initializes
>> > this factory with a source of key material."
>> >
>> > Based on this it is clear that there is a flexibility provided by Java
>> to
>> > to enable developers to provide the required trust/key material loaded
>> from
>> > "anywhere" without requiring them to write custom provider OR trust/key
>> > managers. This same flexibility is reflected in Kafka code also where it
>> > loads the trust/keys from a local file and doesn't require writing a
>> > Provider necessarily. If we do NOT have a custom algorithm, it makes
>> less
>> > sense to write a Provider.
>> >
>> > Thanks
>> > Maulin
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
>> maulin.vasavada@gmail.com>
>> > wrote:
>> >
>> > > Hi Harsha/Colin
>> > >
>> > > I did the sample with a custom Provider for TrustStoreManager and
>> tried
>> > > using ssl.provider Kafka config AND the way KIP-492 is suggesting (by
>> > > adding Provider programmatically instead of relying on
>> > > ssl.provider+java.security. The below sample is followed by my
>> detailed
>> > > findings. I'll appreciate if you can go through it carefully and see
>> if you
>> > > see my point.
>> > >
>> > > package providertest;
>> > >
>> > > import java.security.Provider;
>> > >
>> > > public class MyProvider extends Provider {
>> > >
>> > > private static final String name = "MyProvider";
>> > > private static double version = 1.0d;
>> > > private static String info = "Maulin's SSL Provider v"+version;
>> > >
>> > > public MyProvider() {
>> > > super(name, version, info);
>> > > this.put("TrustManagerFactory.PKIX",
>> "providertest.MyTrustManagerFactory");
>> > > }
>> > > }
>> > >
>> > >
>> > >
>> > > *Details:*
>> > >
>> > > KIP-492 documents that it will use Security.addProvider() assuming it
>> will
>> > > add it as position '0' which is not a correct assumption. The
>> > > addProvider()'s documentation says it will add it to the last
>> available
>> > > position. You may want to correct that to say
>> > > Security.insertProviderAt(provider, 1).
>> > >
>> > > Now coming back to our specific discussion,
>> > >
>> > > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add
>> that
>> > > provider in the provider list via Security.addProvider() the position
>> where
>> > > it gets added doesn't matter (even if you don't end up adding it as
>> first
>> > > entry) since that is the ONLY provider for SPIFFE specific algorithm
>> you
>> > > might have.
>> > >
>> > > We do *not* have custom algorithm for Key/Trust StoreMangers. Which
>> means
>> > > we have to use X509, PKIX etc "Standard Algorithms" ((
>> > >
>> https://docs.oracle.com/javase/7/docs/technotes/guides/security/StandardNames.html
>> ))
>> > > in our provider to override the TrustStoreManager (see my sample
>> code) and
>> > > KeyStoreManger and KeyManager. This creates another challenge
>> mentioned in
>> > > the below point.
>> > >
>> > > 2. In order to make our Provider for loading custom TrustStore work,
>> we
>> > > have to add the provider as 'first' in the list since there are
>> others with
>> > > the same algorithm.
>> > >
>> > > However, the programatic way of adding provider
>> > > (Security.insertProviderAt()) is *not* deterministic for ordering
>> since
>> > > different code can call the method for a different provider and
>> depending
>> > > upon the order of the call our provider can be first or pushed down
>> the
>> > > list. This can happen very well in any client application using
>> Kafka. This
>> > > is specially problematic for a case when you want to guarantee order
>> for a
>> > > Provider having "Standard Algorithms".
>> > >
>> > > If we add our provider in java.security file that definitely
>> guarantees
>> > > the order(unless somebody calls removeProvider() which is less
>> likely). But
>> > > if we add our provider in java.security file it will defeat the
>> purpose of
>> > > the KIP-492.
>> > >
>> > > In the gist - Apache Kafka must not rely on "un-deterministic" method
>> to
>> > > rely on Provider ordering.
>> > >
>> > > 3. If we just use existing ssl.provider kafka configuration then our
>> > > provider will be used in SSLContext.getInstance(protocol, provider)
>> call in
>> > > SslFactory.java and if our provider does not have implementation for
>> > > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we tested it). Example:
>> In
>> > > MyProvider sample above you see that I didn't add SSLContext.TLSv1 as
>> > > "Service+Algorithm" and that didn't work for me. In SPIFFE provider
>> you
>> > > don't have this challenge since you are planning to bypass
>> ssl.provider as
>> > > you mention in the KIP-492.
>> > >
>> > >
>> > > *Overall summary,*
>> > >
>> > > 1. Any provider based mechanisms- a) existing ssl.provider and
>> b)KIP-492,
>> > > for loading key/trust store using "Standard Algorithms" do not work
>> > >
>> > > 2. Approach suggested in our KIP-486 works without any issue and it is
>> > > *not* our context specific solve
>> > >
>> > > 3. Based on above we feel KIP-492 and KIP-486 are complimentary
>> changes
>> > > and not contradicting or redundent.
>> > >
>> > > If you want we can do a joint session somehow to walk through the
>> sample I
>> > > have and various experiments I did. I would encourage you to do
>> similar
>> > > exercise by writing a Provider for "Standard Algorithm" for
>> > > TrustStoreManager (like our needs) and see what you find since only
>> writing
>> > > samples can bring out the complexity/challenges we face.
>> > >
>> > > Thanks
>> > > Maulin
>> > >
>> > > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <
>> > > maulin.vasavada@gmail.com> wrote:
>> > >
>> > >> Just to update - still working on it. Get to work only on and off on
>> it :(
>> > >>
>> > >> On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
>> maulin.vasavada@gmail.com>
>> > >> wrote:
>> > >>
>> > >>> Hi Harsha
>> > >>>
>> > >>> Let me try to write samples and will let you know.
>> > >>>
>> > >>> Thanks
>> > >>> Maulin
>> > >>>
>> > >>> On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <ha...@gmail.com>
>> wrote:
>> > >>>
>> > >>>> Hi Maulin,
>> > >>>> With java security providers can be as custom you would
>> like
>> > >>>> it to
>> > >>>> be. If you only want to to implement a custom way of loading the
>> > >>>> keystore and truststore and not implement any protocol/encryption
>> > >>>> handling
>> > >>>> you can leave them empty and no need to implement.
>> > >>>> Have you looked into the links I pasted before?
>> > >>>>
>> > >>>>
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java
>> > >>>>
>> > >>>>
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java
>> > >>>>
>> > >>>>
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
>> > >>>>
>> > >>>> Can you please tell me which methods are too complex in above to
>> > >>>> implement
>> > >>>> or unnecessary? You are changing anything in SSL/TLS
>> implementations
>> > >>>> provided by
>> > >>>>
>> > >>>> All of the implementations delegating the checks to the default
>> > >>>> implementation anyway.
>> > >>>> Spire agent is an example, its nothing but a GRPC server listening
>> on a
>> > >>>> unix domain socket . Above code is making a RPC call to the local
>> > >>>> daemon to
>> > >>>> get the certificate and keys. The mechanics are pretty much same
>> as what
>> > >>>> you are asking for.
>> > >>>>
>> > >>>> Thanks,
>> > >>>> Harsha
>> > >>>>
>> > >>>> On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
>> > >>>> maulin.vasavada@gmail.com>
>> > >>>> wrote:
>> > >>>>
>> > >>>> > Imagine a scenario like - We know we have a custom KMS and as a
>> Kafka
>> > >>>> owner
>> > >>>> > we want to comply to using that KMS source to load keys/certs.
>> As a
>> > >>>> Kafka
>> > >>>> > owner we know how to integrate with KMS but doesn't necessarily
>> have
>> > >>>> to
>> > >>>> > know anything about cipher suites, algorithms, and SSL/TLS
>> > >>>> implementation.
>> > >>>> > Going the Provider way requires to know lot more than we should,
>> > >>>> isn't it?
>> > >>>> > Not that we would have concern/shy-away knowing those details -
>> but
>> > >>>> if we
>> > >>>> > don't have to - why should we?
>> > >>>> >
>> > >>>> > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
>> > >>>> maulin.vasavada@gmail.com>
>> > >>>> > wrote:
>> > >>>> >
>> > >>>> > > Hi Harsha
>> > >>>> > >
>> > >>>> > > We don't have spire (or similar) agents and we do not have
>> > >>>> keys/certs
>> > >>>> > > locally on any brokers. To elaborate more on my previous email,
>> > >>>> > >
>> > >>>> > > I agree that Java security Providers are used in much broader
>> sense
>> > >>>> - to
>> > >>>> > > have a particular implementation of an algorithm, use specific
>> > >>>> cipher
>> > >>>> > > suites for SSL , OR in our current team's case have a
>> particular
>> > >>>> way to
>> > >>>> > > leverage pre-generated SSL sessions. However, the scope of our
>> KIP
>> > >>>> (486)
>> > >>>> > is
>> > >>>> > > much restricted than that. We merely intend to provide a custom
>> > >>>> > > keystore/truststore for our SSL connections and not really
>> worry
>> > >>>> about
>> > >>>> > > underlying specific SSL/TLS implementation. This simplifies
>> it a
>> > >>>> lot for
>> > >>>> > > us to keep the concerns separate and clear.
>> > >>>> > >
>> > >>>> > > I feel our approach is more complimentary such that it allows
>> for
>> > >>>> using
>> > >>>> > > keystores of choice while retaining the flexibility to use any
>> > >>>> > > underlying/available Provider for actually making the SSL call.
>> > >>>> > >
>> > >>>> > > We agree with KIP-492's approach based on Providers (and Java's
>> > >>>> > > recommendation), but also strongly believe that our approach
>> can
>> > >>>> > compliment
>> > >>>> > > it very effectively for reasons explained above.
>> > >>>> > >
>> > >>>> > > Thanks
>> > >>>> > > Maulin
>> > >>>> > >
>> > >>>> > > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <
>> kafka@harsha.io
>> > >>>> >
>> > >>>> > > wrote:
>> > >>>> > >
>> > >>>> > >> Hi Maulin,
>> > >>>> > >>
>> > >>>> > >> On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
>> > >>>> > >> maulin.vasavada@gmail.com>
>> > >>>> > >> wrote:
>> > >>>> > >>
>> > >>>> > >> > Hi Harsha
>> > >>>> > >> >
>> > >>>> > >> > The reason we rejected the SslProvider route is that - we
>> only
>> > >>>> needed
>> > >>>> > a
>> > >>>> > >> > custom way to load keys/certs. Not touch any policy that
>> existing
>> > >>>> > >> Providers
>> > >>>> > >> > govern like SunJSSE Provider.
>> > >>>> > >> >
>> > >>>> > >>
>> > >>>> > >> We have exactly the same requirements to load certs and keys
>> > >>>> through
>> > >>>> > spire
>> > >>>> > >> agent. We used security.provider to do that exactly. I am not
>> sure
>> > >>>> why
>> > >>>> > you
>> > >>>> > >> would be modifying any policies provided by default SunJSSE
>> > >>>> provider.
>> > >>>> > Can
>> > >>>> > >> you give me an example of having custom provider that will
>> > >>>> override an
>> > >>>> > >> existing policy in SunJSSE provider.
>> > >>>> > >>
>> > >>>> > >> As pointed out earlier, this kip
>> > >>>> > >>
>> > >>>> > >>
>> > >>>> >
>> > >>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>> > >>>> > >> allows
>> > >>>> > >> you to load security.provider through config.
>> > >>>> > >> Take a look at the examples I gave before
>> > >>>> > >>
>> > >>>> > >>
>> > >>>> >
>> > >>>>
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
>> > >>>> > >> It registers KeyManagerFactory and TrustManagerFactory and
>> Keystore
>> > >>>> > >> algorithm.
>> > >>>> > >> Implement your custom way of loading Keystore in here
>> > >>>> > >>
>> > >>>> > >>
>> > >>>> >
>> > >>>>
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java
>> > >>>> > >>
>> > >>>> > >> and Trust manager like here
>> > >>>> > >>
>> > >>>> > >>
>> > >>>> >
>> > >>>>
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java
>> > >>>> > >>
>> > >>>> > >> In your Kafka client you can set the security.provider to
>> your
>> > >>>> custom
>> > >>>> > >> implementation and with this fix
>> > >>>> > >> https://issues.apache.org/jira/browse/KAFKA-8191 you can set
>> > >>>> > >> keyManagerAlgorigthm and trustManagerAlgorithm configs.
>> > >>>> > >>
>> > >>>> > >> All of this is in your clients and broker side and do not
>> need to
>> > >>>> touch
>> > >>>> > >> any
>> > >>>> > >> policy changes at JVM level. You'll register the providers in
>> the
>> > >>>> > priority
>> > >>>> > >> order and can still have SunJSSE provider and have your custom
>> > >>>> provider
>> > >>>> > to
>> > >>>> > >> implement the key and trust managers.
>> > >>>> > >>
>> > >>>> > >>
>> > >>>> > >>
>> > >>>> > >> The ask here is different than KIP-492. We don't have any
>> need to
>> > >>>> > >> > modify/specify the algorithm parameter. Does that make
>> sense?
>> > >>>> > >> >
>> > >>>> > >>
>> > >>>> > >> The ask in KIP is introducing new interfaces where the KIP's
>> > >>>> > >> goal/motivation can be achieved through the security.provider
>> and
>> > >>>> we
>> > >>>> > >> worked
>> > >>>> > >> on similar goal without touching any Keystore or Truststore
>> > >>>> interfaces.
>> > >>>> > >> My advise is against changing or introducing new interfaces
>> when
>> > >>>> it can
>> > >>>> > >> work through security.provider.
>> > >>>> > >>
>> > >>>> > >> Thanks,
>> > >>>> > >> Harsha
>> > >>>> > >>
>> > >>>> > >>
>> > >>>> > >> Thanks
>> > >>>> > >> > Maulin
>> > >>>> > >> >
>> > >>>> > >> > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
>> > >>>> kafka@harsha.io>
>> > >>>> > >> > wrote:
>> > >>>> > >> >
>> > >>>> > >> > In your KIP you added security. provider as rejected
>> alternative
>> > >>>> and
>> > >>>> > >> > specified "its not the correct way". Do you mind explaining
>> why
>> > >>>> its
>> > >>>> > >> not? I
>> > >>>> > >> > didn't find any evidence in Java docs to say so. Contrary
>> to your
>> > >>>> > >> statement
>> > >>>> > >> > it does say in the java docs
>> > >>>> > >> > " However, please note that a provider can be used to
>> implement
>> > >>>> any
>> > >>>> > >> > security service in Java that uses a pluggable architecture
>> with
>> > >>>> a
>> > >>>> > >> choice
>> > >>>> > >> > of implementations that fit underneath."
>> > >>>> > >> >
>> > >>>> > >> > Java Security Providers have been used by other projects to
>> > >>>> provide
>> > >>>> > such
>> > >>>> > >> > integration . I am not sure if you looked into Spiffe
>> project to
>> > >>>> > >> > efficiently distribute certificates but here is an example
>> of
>> > >>>> Java
>> > >>>> > >> provider
>> > >>>> > >> >
>> > >>>> > >> >
>> > >>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>> > >>>> > >> >
>> > >>>> > >>
>> > >>>> >
>> > >>>>
>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>> > >>>> > >> > java which
>> > >>>> > >> > obtains certificates from local daemons.
>> > >>>> > >> > These integrations are being used in Tomcat, Jetty etc.. We
>> are
>> > >>>> also
>> > >>>> > >> using
>> > >>>> > >> > Security provider to do the same in our Kafka clusters. So
>> > >>>> unless I
>> > >>>> > see
>> > >>>> > >> > more evidence why security.provider doesn't work for you
>> adding
>> > >>>> new
>> > >>>> > >> > interfaces while there exists more cleaner way of achieving
>> the
>> > >>>> goals
>> > >>>> > of
>> > >>>> > >> > this KIP is unnecessary and breaks the well known security
>> > >>>> interfaces
>> > >>>> > >> > provided by Java itself.
>> > >>>> > >> >
>> > >>>> > >> > Thanks,
>> > >>>> > >> > Harsha
>> > >>>> > >> >
>> > >>>> > >> > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
>> > >>>> kafka@harsha.io
>> > >>>> > >
>> > >>>> > >> > wrote:
>> > >>>> > >> >
>> > >>>> > >> > Hi Maulin,
>> > >>>> > >> > Not sure if you looked at my previous replies. This
>> > >>>> > >> >
>> > >>>> > >> > changes
>> > >>>> > >> >
>> > >>>> > >> > are not required as there is already security Provider to do
>> > >>>> what you
>> > >>>> > >> are
>> > >>>> > >> > proposing. This KIP
>> > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/
>> > >>>> > >> >
>> KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>> > >>>> also
>> > >>>> > >> > addresses easy registration of such providers.
>> > >>>> > >> >
>> > >>>> > >> > Thanks,
>> > >>>> > >> > Harsha
>> > >>>> > >> >
>> > >>>> > >> > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
>> > >>>> > >> <maulin.vasavada@gmail.
>> > >>>> > >> > com> wrote:
>> > >>>> > >> >
>> > >>>> > >> > Bump! Can somebody please review this?
>> > >>>> > >> >
>> > >>>> > >> > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
>> > >>>> > >> >
>> > >>>> > >> > maulin.vasavada@gmail.com>
>> > >>>> > >> >
>> > >>>> > >> > wrote:
>> > >>>> > >> >
>> > >>>> > >> > Bump! Can somebody please review this?
>> > >>>> > >> >
>> > >>>> > >> >
>> > >>>> > >>
>> > >>>> > >
>> > >>>> >
>> > >>>>
>> > >>>
>> >
>>
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Maulin Vasavada <ma...@gmail.com>.
Thanks Clement. I"ll start accommodating your suggestions and addressing
comments on the KIP-519. However, can we please start discussing on the
KIP-519 discussion thread?
On Mon, Sep 9, 2019 at 9:23 PM Pellerin, Clement <Cl...@ibi.com>
wrote:
> Please specify in the KIP if the new config is reconfigurable or not.
>
> -----Original Message-----
> From: Pellerin, Clement
> Sent: Monday, September 9, 2019 8:28 PM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-486 Support for pluggable KeyStore and
> TrustStore
>
> This is a good start.
>
> Please document the constants for the config you are creating.
> You will notice you need to make the name of your default implementation
> public.
>
> How do the configs get into your factory instance? Is it through the
> constructor or you will add method arguments?
> Please define your constructor in the KIP even if it is the default
> constructor.
> You will also need to document the constructor in the class comment of
> your interface.
>
> In your implementation, I suggest to default the config value and always
> call the constructor dynamically even for the default implementation.
>
> I don't understand why this is a factory for both SSLContext and SslEngine.
> The name of the factory depends on this choice.
>
> I am not a fan of the method name shouldBeRebuilt()
> This was OK in a private implementation before but this will become a
> public API.
> What would be a better name that fits with the naming conventions in
> SslFactory?
>
> Please mention how your interface could handle custom configs.
> I would assume you plan to merge the SslFactory reconfigurableConfigs with
> those of the SslEngineFactory.
> How can you be sure SslFactory always creates its SslEngineFactory before
> any calls to SslFactory.reconfigurableConfigs()?
>
> Your Rejected Alternatives need to be beefed up.
> In particular, why is SslFactory not the pluggable interface directly?
> Hint: because we want to reuse all the complex validation code and make it
> mandatory.
> This is also the place to argue against KIP-383. Hint: because it does not
> handle reconfiguration in the presence of custom configs.
>
> When I wrote KIP-383, I felt I needed a prototype before I could solidify
> the proposal.
> That's part of the reason why there was never a third iteration.
>
> -----Original Message-----
> From: Maulin Vasavada [mailto:maulin.vasavada@gmail.com]
> Sent: Monday, September 9, 2019 6:41 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and
> TrustStore
>
> Hi all
>
> I created a KIP-519 for pluggability of SSLContext/Engine object. Looking
> forward for a great discussion and feedback on the same. I'll keep KIP-486
> in discussion state until we reach to some good conclusion on how to allow
> custom key/trust stores after KIP-519 work is done. Based on that, we will
> update the KIP-486 appropriately.
>
> Thanks
> Maulin
>
>
> This email did not originate from inside Information Builders.
>
RE: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by "Pellerin, Clement" <Cl...@ibi.com>.
Please specify in the KIP if the new config is reconfigurable or not.
-----Original Message-----
From: Pellerin, Clement
Sent: Monday, September 9, 2019 8:28 PM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
This is a good start.
Please document the constants for the config you are creating.
You will notice you need to make the name of your default implementation public.
How do the configs get into your factory instance? Is it through the constructor or you will add method arguments?
Please define your constructor in the KIP even if it is the default constructor.
You will also need to document the constructor in the class comment of your interface.
In your implementation, I suggest to default the config value and always call the constructor dynamically even for the default implementation.
I don't understand why this is a factory for both SSLContext and SslEngine.
The name of the factory depends on this choice.
I am not a fan of the method name shouldBeRebuilt()
This was OK in a private implementation before but this will become a public API.
What would be a better name that fits with the naming conventions in SslFactory?
Please mention how your interface could handle custom configs.
I would assume you plan to merge the SslFactory reconfigurableConfigs with those of the SslEngineFactory.
How can you be sure SslFactory always creates its SslEngineFactory before any calls to SslFactory.reconfigurableConfigs()?
Your Rejected Alternatives need to be beefed up.
In particular, why is SslFactory not the pluggable interface directly? Hint: because we want to reuse all the complex validation code and make it mandatory.
This is also the place to argue against KIP-383. Hint: because it does not handle reconfiguration in the presence of custom configs.
When I wrote KIP-383, I felt I needed a prototype before I could solidify the proposal.
That's part of the reason why there was never a third iteration.
-----Original Message-----
From: Maulin Vasavada [mailto:maulin.vasavada@gmail.com]
Sent: Monday, September 9, 2019 6:41 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Hi all
I created a KIP-519 for pluggability of SSLContext/Engine object. Looking
forward for a great discussion and feedback on the same. I'll keep KIP-486
in discussion state until we reach to some good conclusion on how to allow
custom key/trust stores after KIP-519 work is done. Based on that, we will
update the KIP-486 appropriately.
Thanks
Maulin
This email did not originate from inside Information Builders.
RE: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by "Pellerin, Clement" <Cl...@ibi.com>.
This is a good start.
Please document the constants for the config you are creating.
You will notice you need to make the name of your default implementation public.
How do the configs get into your factory instance? Is it through the constructor or you will add method arguments?
Please define your constructor in the KIP even if it is the default constructor.
You will also need to document the constructor in the class comment of your interface.
In your implementation, I suggest to default the config value and always call the constructor dynamically even for the default implementation.
I don't understand why this is a factory for both SSLContext and SslEngine.
The name of the factory depends on this choice.
I am not a fan of the method name shouldBeRebuilt()
This was OK in a private implementation before but this will become a public API.
What would be a better name that fits with the naming conventions in SslFactory?
Please mention how your interface could handle custom configs.
I would assume you plan to merge the SslFactory reconfigurableConfigs with those of the SslEngineFactory.
How can you be sure SslFactory always creates its SslEngineFactory before any calls to SslFactory.reconfigurableConfigs()?
Your Rejected Alternatives need to be beefed up.
In particular, why is SslFactory not the pluggable interface directly? Hint: because we want to reuse all the complex validation code and make it mandatory.
This is also the place to argue against KIP-383. Hint: because it does not handle reconfiguration in the presence of custom configs.
When I wrote KIP-383, I felt I needed a prototype before I could solidify the proposal.
That's part of the reason why there was never a third iteration.
-----Original Message-----
From: Maulin Vasavada [mailto:maulin.vasavada@gmail.com]
Sent: Monday, September 9, 2019 6:41 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Hi all
I created a KIP-519 for pluggability of SSLContext/Engine object. Looking
forward for a great discussion and feedback on the same. I'll keep KIP-486
in discussion state until we reach to some good conclusion on how to allow
custom key/trust stores after KIP-519 work is done. Based on that, we will
update the KIP-486 appropriately.
Thanks
Maulin
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Maulin Vasavada <ma...@gmail.com>.
Hi all
I created a KIP-519 for pluggability of SSLContext/Engine object. Looking
forward for a great discussion and feedback on the same. I'll keep KIP-486
in discussion state until we reach to some good conclusion on how to allow
custom key/trust stores after KIP-519 work is done. Based on that, we will
update the KIP-486 appropriately.
Thanks
Maulin
On Sun, Sep 8, 2019 at 11:08 PM Maulin Vasavada <ma...@gmail.com>
wrote:
> Thank you Clement and Colin for shading light on the process for me. I
> will probably start working on a new kip for pluggable SslEngine.
>
> Side note: The pull request is from my fork to my fork to make it easier
> to see what changes I am making. I have no intention to getting it reviewed
> as a pull request.
>
> Okay, let me start with a new KIP and we will go from there. I'll keep
> this KIP-486 since I feel ultimately we need that kip.
>
> Thanks
> Maulin
>
> On Fri, Sep 6, 2019 at 2:52 PM Colin McCabe <cm...@apache.org> wrote:
>
>> Sorry, I meant to write "removing an API typically requires a new major
>> release of Kafka". Deprecating an API can be done in a minor release.
>>
>> regards,
>> Colin
>>
>>
>> On Fri, Sep 6, 2019, at 14:49, Colin McCabe wrote:
>> > Hi Maulin,
>> >
>> > Clement brought up a good point, which is that we should have agreement
>> > on the overall design before we start looking at PRs. In Kafka, that
>> > means starting a discussion on a KIP, and then getting that KIP voted
>> > in.
>> >
>> > There is more information about the KIP process here:
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>> > Feel free to ask more detailed questions about the process on the
>> > mailing list as well :)
>> >
>> > It sounds like Clement is proposing that you create a new KIP rather
>> > than reusing KIP-383. So you should probably either repurpose KIP-486
>> > for this, or choose a new number. Both options seem reasonable here.
>> > As far as I can see, once we implement a pluggable SslEngineBuilder,
>> > KIP-383 would no longer be needed, and so we would put that one into
>> > "rejected KIPs."
>> >
>> > In general, coming up with good APIs can be harder than it might seem
>> > at first. Once we add an API, we have to assume that we're going to
>> > support it forever. Deprecating an API can take a very long time, and
>> > typically requires a new major release of Kafka. Therefore, sometimes
>> > it's worth making users go through a little bit of copying and pasting
>> > in order to avoid creating an API that would constrain the project in
>> > the future. Hopefully, we can come up with something good here that
>> > will be useful to everyone.
>> >
>> > best,
>> > Colin
>> >
>> >
>> > On Fri, Sep 6, 2019, at 07:48, Pellerin, Clement wrote:
>> > > This is the way I see it, which is in no way authoritative.
>> > >
>> > > First you have to find a complete solution, then you need to document
>> > > it in a KIP and that KIP needs to pass a vote. Any votes before the
>> KIP
>> > > vote starts is meaningless.
>> > >
>> > > As for the ownership and authorship of the KIPs, I don't plan to work
>> > > on this, so KIP-383 is better left the way it is. I would prefer if
>> you
>> > > would update your KIP or maybe create a new one. We can mark KIP-383
>> as
>> > > obsolete after your KIP passes its vote.
>> > >
>> > > -----Original Message-----
>> > > From: Maulin Vasavada [mailto:maulin.vasavada@gmail.com]
>> > > Sent: Friday, September 6, 2019 2:36 AM
>> > > To: dev@kafka.apache.org
>> > > Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and
>> TrustStore
>> > >
>> > > Hi all
>> > >
>> > > Any input on my last question about the process?
>> > >
>> > > Also, I have updated the code based on the feedback so far at:
>> > > https://github.com/maulin-vasavada/kafka/pull/2/files. Still have to
>> figure
>> > > out how to plug keys/certs loading while using
>> DefaultSslEngineFactory's
>> > > implementation (still need to make it final).
>> > >
>> > > Thanks
>> > > Maulin
>> > >
>> > >
>> > > On Thu, Sep 5, 2019 at 4:34 PM Maulin Vasavada <
>> maulin.vasavada@gmail.com>
>> > > wrote:
>> > >
>> > > > Hi all
>> > > >
>> > > > It seems we are in some sort of agreement so far apart from code
>> > > > review/comments. However, I have a fundamental question - going
>> forward how
>> > > > this work from the process standpoint? What do we do with this
>> KIP-486 vs
>> > > > KIP-383? I feel that both needs to come together since in order to
>> make
>> > > > Pluggable key/trust store on top of making SslEngineBuilder
>> pluggable we
>> > > > will need changes suggested by KIP-486 with some differences to the
>> > > > original proposal. It would great if someone can help us clarify
>> the next
>> > > > steps.
>> > > >
>> > > > Thanks
>> > > > Maulin
>> > > >
>> > > > On Wed, Sep 4, 2019 at 1:54 PM Maulin Vasavada <
>> maulin.vasavada@gmail.com>
>> > > > wrote:
>> > > >
>> > > >> Do you guys think it would be easier if you can provide comments on
>> > > >> GitHub and we can continue there and summarize the conclusion here?
>> > > >>
>> > > >> We should not lose addressing any comments.
>> > > >>
>> > > >> On Wed, Sep 4, 2019 at 12:34 PM Pellerin, Clement <
>> > > >> Clement_Pellerin@ibi.com> wrote:
>> > > >>
>> > > >>> The proposed interface does not look like the Builder pattern I
>> am used
>> > > >>> to.
>> > > >>> Should SslEngineBuilder be called SslEngineFactory instead?
>> > > >>>
>> > > >>> On Mon, Sep 2, 2019, at 03:33, Rajini Sivaram wrote:
>> > > >>> > I would expect SslEngineBuilder interface to look something
>> like this,
>> > > >>> > perhaps with some tweaking:
>> > > >>> >
>> > > >>> > public interface SslEngineBuilder extends Configurable,
>> Closeable {
>> > > >>> >
>> > > >>> > Set<String> reconfigurableConfigs();
>> > > >>> >
>> > > >>> > boolean shouldBeRebuilt(Map<String, Object> nextConfigs);
>> > > >>> >
>> > > >>> > SSLEngine createSslEngine(Mode mode, String peerHost, int
>> > > >>> > peerPort, String endpointIdentification);
>> > > >>> >
>> > > >>> > }
>> > > >>>
>> > > >>
>> > >
>> >
>>
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Maulin Vasavada <ma...@gmail.com>.
Thank you Clement and Colin for shading light on the process for me. I will
probably start working on a new kip for pluggable SslEngine.
Side note: The pull request is from my fork to my fork to make it easier to
see what changes I am making. I have no intention to getting it reviewed as
a pull request.
Okay, let me start with a new KIP and we will go from there. I'll keep this
KIP-486 since I feel ultimately we need that kip.
Thanks
Maulin
On Fri, Sep 6, 2019 at 2:52 PM Colin McCabe <cm...@apache.org> wrote:
> Sorry, I meant to write "removing an API typically requires a new major
> release of Kafka". Deprecating an API can be done in a minor release.
>
> regards,
> Colin
>
>
> On Fri, Sep 6, 2019, at 14:49, Colin McCabe wrote:
> > Hi Maulin,
> >
> > Clement brought up a good point, which is that we should have agreement
> > on the overall design before we start looking at PRs. In Kafka, that
> > means starting a discussion on a KIP, and then getting that KIP voted
> > in.
> >
> > There is more information about the KIP process here:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > Feel free to ask more detailed questions about the process on the
> > mailing list as well :)
> >
> > It sounds like Clement is proposing that you create a new KIP rather
> > than reusing KIP-383. So you should probably either repurpose KIP-486
> > for this, or choose a new number. Both options seem reasonable here.
> > As far as I can see, once we implement a pluggable SslEngineBuilder,
> > KIP-383 would no longer be needed, and so we would put that one into
> > "rejected KIPs."
> >
> > In general, coming up with good APIs can be harder than it might seem
> > at first. Once we add an API, we have to assume that we're going to
> > support it forever. Deprecating an API can take a very long time, and
> > typically requires a new major release of Kafka. Therefore, sometimes
> > it's worth making users go through a little bit of copying and pasting
> > in order to avoid creating an API that would constrain the project in
> > the future. Hopefully, we can come up with something good here that
> > will be useful to everyone.
> >
> > best,
> > Colin
> >
> >
> > On Fri, Sep 6, 2019, at 07:48, Pellerin, Clement wrote:
> > > This is the way I see it, which is in no way authoritative.
> > >
> > > First you have to find a complete solution, then you need to document
> > > it in a KIP and that KIP needs to pass a vote. Any votes before the
> KIP
> > > vote starts is meaningless.
> > >
> > > As for the ownership and authorship of the KIPs, I don't plan to work
> > > on this, so KIP-383 is better left the way it is. I would prefer if
> you
> > > would update your KIP or maybe create a new one. We can mark KIP-383
> as
> > > obsolete after your KIP passes its vote.
> > >
> > > -----Original Message-----
> > > From: Maulin Vasavada [mailto:maulin.vasavada@gmail.com]
> > > Sent: Friday, September 6, 2019 2:36 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and
> TrustStore
> > >
> > > Hi all
> > >
> > > Any input on my last question about the process?
> > >
> > > Also, I have updated the code based on the feedback so far at:
> > > https://github.com/maulin-vasavada/kafka/pull/2/files. Still have to
> figure
> > > out how to plug keys/certs loading while using
> DefaultSslEngineFactory's
> > > implementation (still need to make it final).
> > >
> > > Thanks
> > > Maulin
> > >
> > >
> > > On Thu, Sep 5, 2019 at 4:34 PM Maulin Vasavada <
> maulin.vasavada@gmail.com>
> > > wrote:
> > >
> > > > Hi all
> > > >
> > > > It seems we are in some sort of agreement so far apart from code
> > > > review/comments. However, I have a fundamental question - going
> forward how
> > > > this work from the process standpoint? What do we do with this
> KIP-486 vs
> > > > KIP-383? I feel that both needs to come together since in order to
> make
> > > > Pluggable key/trust store on top of making SslEngineBuilder
> pluggable we
> > > > will need changes suggested by KIP-486 with some differences to the
> > > > original proposal. It would great if someone can help us clarify the
> next
> > > > steps.
> > > >
> > > > Thanks
> > > > Maulin
> > > >
> > > > On Wed, Sep 4, 2019 at 1:54 PM Maulin Vasavada <
> maulin.vasavada@gmail.com>
> > > > wrote:
> > > >
> > > >> Do you guys think it would be easier if you can provide comments on
> > > >> GitHub and we can continue there and summarize the conclusion here?
> > > >>
> > > >> We should not lose addressing any comments.
> > > >>
> > > >> On Wed, Sep 4, 2019 at 12:34 PM Pellerin, Clement <
> > > >> Clement_Pellerin@ibi.com> wrote:
> > > >>
> > > >>> The proposed interface does not look like the Builder pattern I am
> used
> > > >>> to.
> > > >>> Should SslEngineBuilder be called SslEngineFactory instead?
> > > >>>
> > > >>> On Mon, Sep 2, 2019, at 03:33, Rajini Sivaram wrote:
> > > >>> > I would expect SslEngineBuilder interface to look something like
> this,
> > > >>> > perhaps with some tweaking:
> > > >>> >
> > > >>> > public interface SslEngineBuilder extends Configurable,
> Closeable {
> > > >>> >
> > > >>> > Set<String> reconfigurableConfigs();
> > > >>> >
> > > >>> > boolean shouldBeRebuilt(Map<String, Object> nextConfigs);
> > > >>> >
> > > >>> > SSLEngine createSslEngine(Mode mode, String peerHost, int
> > > >>> > peerPort, String endpointIdentification);
> > > >>> >
> > > >>> > }
> > > >>>
> > > >>
> > >
> >
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Colin McCabe <cm...@apache.org>.
Sorry, I meant to write "removing an API typically requires a new major release of Kafka". Deprecating an API can be done in a minor release.
regards,
Colin
On Fri, Sep 6, 2019, at 14:49, Colin McCabe wrote:
> Hi Maulin,
>
> Clement brought up a good point, which is that we should have agreement
> on the overall design before we start looking at PRs. In Kafka, that
> means starting a discussion on a KIP, and then getting that KIP voted
> in.
>
> There is more information about the KIP process here:
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> Feel free to ask more detailed questions about the process on the
> mailing list as well :)
>
> It sounds like Clement is proposing that you create a new KIP rather
> than reusing KIP-383. So you should probably either repurpose KIP-486
> for this, or choose a new number. Both options seem reasonable here.
> As far as I can see, once we implement a pluggable SslEngineBuilder,
> KIP-383 would no longer be needed, and so we would put that one into
> "rejected KIPs."
>
> In general, coming up with good APIs can be harder than it might seem
> at first. Once we add an API, we have to assume that we're going to
> support it forever. Deprecating an API can take a very long time, and
> typically requires a new major release of Kafka. Therefore, sometimes
> it's worth making users go through a little bit of copying and pasting
> in order to avoid creating an API that would constrain the project in
> the future. Hopefully, we can come up with something good here that
> will be useful to everyone.
>
> best,
> Colin
>
>
> On Fri, Sep 6, 2019, at 07:48, Pellerin, Clement wrote:
> > This is the way I see it, which is in no way authoritative.
> >
> > First you have to find a complete solution, then you need to document
> > it in a KIP and that KIP needs to pass a vote. Any votes before the KIP
> > vote starts is meaningless.
> >
> > As for the ownership and authorship of the KIPs, I don't plan to work
> > on this, so KIP-383 is better left the way it is. I would prefer if you
> > would update your KIP or maybe create a new one. We can mark KIP-383 as
> > obsolete after your KIP passes its vote.
> >
> > -----Original Message-----
> > From: Maulin Vasavada [mailto:maulin.vasavada@gmail.com]
> > Sent: Friday, September 6, 2019 2:36 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
> >
> > Hi all
> >
> > Any input on my last question about the process?
> >
> > Also, I have updated the code based on the feedback so far at:
> > https://github.com/maulin-vasavada/kafka/pull/2/files. Still have to figure
> > out how to plug keys/certs loading while using DefaultSslEngineFactory's
> > implementation (still need to make it final).
> >
> > Thanks
> > Maulin
> >
> >
> > On Thu, Sep 5, 2019 at 4:34 PM Maulin Vasavada <ma...@gmail.com>
> > wrote:
> >
> > > Hi all
> > >
> > > It seems we are in some sort of agreement so far apart from code
> > > review/comments. However, I have a fundamental question - going forward how
> > > this work from the process standpoint? What do we do with this KIP-486 vs
> > > KIP-383? I feel that both needs to come together since in order to make
> > > Pluggable key/trust store on top of making SslEngineBuilder pluggable we
> > > will need changes suggested by KIP-486 with some differences to the
> > > original proposal. It would great if someone can help us clarify the next
> > > steps.
> > >
> > > Thanks
> > > Maulin
> > >
> > > On Wed, Sep 4, 2019 at 1:54 PM Maulin Vasavada <ma...@gmail.com>
> > > wrote:
> > >
> > >> Do you guys think it would be easier if you can provide comments on
> > >> GitHub and we can continue there and summarize the conclusion here?
> > >>
> > >> We should not lose addressing any comments.
> > >>
> > >> On Wed, Sep 4, 2019 at 12:34 PM Pellerin, Clement <
> > >> Clement_Pellerin@ibi.com> wrote:
> > >>
> > >>> The proposed interface does not look like the Builder pattern I am used
> > >>> to.
> > >>> Should SslEngineBuilder be called SslEngineFactory instead?
> > >>>
> > >>> On Mon, Sep 2, 2019, at 03:33, Rajini Sivaram wrote:
> > >>> > I would expect SslEngineBuilder interface to look something like this,
> > >>> > perhaps with some tweaking:
> > >>> >
> > >>> > public interface SslEngineBuilder extends Configurable, Closeable {
> > >>> >
> > >>> > Set<String> reconfigurableConfigs();
> > >>> >
> > >>> > boolean shouldBeRebuilt(Map<String, Object> nextConfigs);
> > >>> >
> > >>> > SSLEngine createSslEngine(Mode mode, String peerHost, int
> > >>> > peerPort, String endpointIdentification);
> > >>> >
> > >>> > }
> > >>>
> > >>
> >
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Colin McCabe <cm...@apache.org>.
Hi Maulin,
Clement brought up a good point, which is that we should have agreement on the overall design before we start looking at PRs. In Kafka, that means starting a discussion on a KIP, and then getting that KIP voted in.
There is more information about the KIP process here:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
Feel free to ask more detailed questions about the process on the mailing list as well :)
It sounds like Clement is proposing that you create a new KIP rather than reusing KIP-383. So you should probably either repurpose KIP-486 for this, or choose a new number. Both options seem reasonable here. As far as I can see, once we implement a pluggable SslEngineBuilder, KIP-383 would no longer be needed, and so we would put that one into "rejected KIPs."
In general, coming up with good APIs can be harder than it might seem at first. Once we add an API, we have to assume that we're going to support it forever. Deprecating an API can take a very long time, and typically requires a new major release of Kafka. Therefore, sometimes it's worth making users go through a little bit of copying and pasting in order to avoid creating an API that would constrain the project in the future. Hopefully, we can come up with something good here that will be useful to everyone.
best,
Colin
On Fri, Sep 6, 2019, at 07:48, Pellerin, Clement wrote:
> This is the way I see it, which is in no way authoritative.
>
> First you have to find a complete solution, then you need to document
> it in a KIP and that KIP needs to pass a vote. Any votes before the KIP
> vote starts is meaningless.
>
> As for the ownership and authorship of the KIPs, I don't plan to work
> on this, so KIP-383 is better left the way it is. I would prefer if you
> would update your KIP or maybe create a new one. We can mark KIP-383 as
> obsolete after your KIP passes its vote.
>
> -----Original Message-----
> From: Maulin Vasavada [mailto:maulin.vasavada@gmail.com]
> Sent: Friday, September 6, 2019 2:36 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
>
> Hi all
>
> Any input on my last question about the process?
>
> Also, I have updated the code based on the feedback so far at:
> https://github.com/maulin-vasavada/kafka/pull/2/files. Still have to figure
> out how to plug keys/certs loading while using DefaultSslEngineFactory's
> implementation (still need to make it final).
>
> Thanks
> Maulin
>
>
> On Thu, Sep 5, 2019 at 4:34 PM Maulin Vasavada <ma...@gmail.com>
> wrote:
>
> > Hi all
> >
> > It seems we are in some sort of agreement so far apart from code
> > review/comments. However, I have a fundamental question - going forward how
> > this work from the process standpoint? What do we do with this KIP-486 vs
> > KIP-383? I feel that both needs to come together since in order to make
> > Pluggable key/trust store on top of making SslEngineBuilder pluggable we
> > will need changes suggested by KIP-486 with some differences to the
> > original proposal. It would great if someone can help us clarify the next
> > steps.
> >
> > Thanks
> > Maulin
> >
> > On Wed, Sep 4, 2019 at 1:54 PM Maulin Vasavada <ma...@gmail.com>
> > wrote:
> >
> >> Do you guys think it would be easier if you can provide comments on
> >> GitHub and we can continue there and summarize the conclusion here?
> >>
> >> We should not lose addressing any comments.
> >>
> >> On Wed, Sep 4, 2019 at 12:34 PM Pellerin, Clement <
> >> Clement_Pellerin@ibi.com> wrote:
> >>
> >>> The proposed interface does not look like the Builder pattern I am used
> >>> to.
> >>> Should SslEngineBuilder be called SslEngineFactory instead?
> >>>
> >>> On Mon, Sep 2, 2019, at 03:33, Rajini Sivaram wrote:
> >>> > I would expect SslEngineBuilder interface to look something like this,
> >>> > perhaps with some tweaking:
> >>> >
> >>> > public interface SslEngineBuilder extends Configurable, Closeable {
> >>> >
> >>> > Set<String> reconfigurableConfigs();
> >>> >
> >>> > boolean shouldBeRebuilt(Map<String, Object> nextConfigs);
> >>> >
> >>> > SSLEngine createSslEngine(Mode mode, String peerHost, int
> >>> > peerPort, String endpointIdentification);
> >>> >
> >>> > }
> >>>
> >>
>
RE: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by "Pellerin, Clement" <Cl...@ibi.com>.
This is the way I see it, which is in no way authoritative.
First you have to find a complete solution, then you need to document it in a KIP and that KIP needs to pass a vote. Any votes before the KIP vote starts is meaningless.
As for the ownership and authorship of the KIPs, I don't plan to work on this, so KIP-383 is better left the way it is. I would prefer if you would update your KIP or maybe create a new one. We can mark KIP-383 as obsolete after your KIP passes its vote.
-----Original Message-----
From: Maulin Vasavada [mailto:maulin.vasavada@gmail.com]
Sent: Friday, September 6, 2019 2:36 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Hi all
Any input on my last question about the process?
Also, I have updated the code based on the feedback so far at:
https://github.com/maulin-vasavada/kafka/pull/2/files. Still have to figure
out how to plug keys/certs loading while using DefaultSslEngineFactory's
implementation (still need to make it final).
Thanks
Maulin
On Thu, Sep 5, 2019 at 4:34 PM Maulin Vasavada <ma...@gmail.com>
wrote:
> Hi all
>
> It seems we are in some sort of agreement so far apart from code
> review/comments. However, I have a fundamental question - going forward how
> this work from the process standpoint? What do we do with this KIP-486 vs
> KIP-383? I feel that both needs to come together since in order to make
> Pluggable key/trust store on top of making SslEngineBuilder pluggable we
> will need changes suggested by KIP-486 with some differences to the
> original proposal. It would great if someone can help us clarify the next
> steps.
>
> Thanks
> Maulin
>
> On Wed, Sep 4, 2019 at 1:54 PM Maulin Vasavada <ma...@gmail.com>
> wrote:
>
>> Do you guys think it would be easier if you can provide comments on
>> GitHub and we can continue there and summarize the conclusion here?
>>
>> We should not lose addressing any comments.
>>
>> On Wed, Sep 4, 2019 at 12:34 PM Pellerin, Clement <
>> Clement_Pellerin@ibi.com> wrote:
>>
>>> The proposed interface does not look like the Builder pattern I am used
>>> to.
>>> Should SslEngineBuilder be called SslEngineFactory instead?
>>>
>>> On Mon, Sep 2, 2019, at 03:33, Rajini Sivaram wrote:
>>> > I would expect SslEngineBuilder interface to look something like this,
>>> > perhaps with some tweaking:
>>> >
>>> > public interface SslEngineBuilder extends Configurable, Closeable {
>>> >
>>> > Set<String> reconfigurableConfigs();
>>> >
>>> > boolean shouldBeRebuilt(Map<String, Object> nextConfigs);
>>> >
>>> > SSLEngine createSslEngine(Mode mode, String peerHost, int
>>> > peerPort, String endpointIdentification);
>>> >
>>> > }
>>>
>>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Maulin Vasavada <ma...@gmail.com>.
Hi all
Any input on my last question about the process?
Also, I have updated the code based on the feedback so far at:
https://github.com/maulin-vasavada/kafka/pull/2/files. Still have to figure
out how to plug keys/certs loading while using DefaultSslEngineFactory's
implementation (still need to make it final).
Thanks
Maulin
On Thu, Sep 5, 2019 at 4:34 PM Maulin Vasavada <ma...@gmail.com>
wrote:
> Hi all
>
> It seems we are in some sort of agreement so far apart from code
> review/comments. However, I have a fundamental question - going forward how
> this work from the process standpoint? What do we do with this KIP-486 vs
> KIP-383? I feel that both needs to come together since in order to make
> Pluggable key/trust store on top of making SslEngineBuilder pluggable we
> will need changes suggested by KIP-486 with some differences to the
> original proposal. It would great if someone can help us clarify the next
> steps.
>
> Thanks
> Maulin
>
> On Wed, Sep 4, 2019 at 1:54 PM Maulin Vasavada <ma...@gmail.com>
> wrote:
>
>> Do you guys think it would be easier if you can provide comments on
>> GitHub and we can continue there and summarize the conclusion here?
>>
>> We should not lose addressing any comments.
>>
>> On Wed, Sep 4, 2019 at 12:34 PM Pellerin, Clement <
>> Clement_Pellerin@ibi.com> wrote:
>>
>>> The proposed interface does not look like the Builder pattern I am used
>>> to.
>>> Should SslEngineBuilder be called SslEngineFactory instead?
>>>
>>> On Mon, Sep 2, 2019, at 03:33, Rajini Sivaram wrote:
>>> > I would expect SslEngineBuilder interface to look something like this,
>>> > perhaps with some tweaking:
>>> >
>>> > public interface SslEngineBuilder extends Configurable, Closeable {
>>> >
>>> > Set<String> reconfigurableConfigs();
>>> >
>>> > boolean shouldBeRebuilt(Map<String, Object> nextConfigs);
>>> >
>>> > SSLEngine createSslEngine(Mode mode, String peerHost, int
>>> > peerPort, String endpointIdentification);
>>> >
>>> > }
>>>
>>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Maulin Vasavada <ma...@gmail.com>.
Hi all
It seems we are in some sort of agreement so far apart from code
review/comments. However, I have a fundamental question - going forward how
this work from the process standpoint? What do we do with this KIP-486 vs
KIP-383? I feel that both needs to come together since in order to make
Pluggable key/trust store on top of making SslEngineBuilder pluggable we
will need changes suggested by KIP-486 with some differences to the
original proposal. It would great if someone can help us clarify the next
steps.
Thanks
Maulin
On Wed, Sep 4, 2019 at 1:54 PM Maulin Vasavada <ma...@gmail.com>
wrote:
> Do you guys think it would be easier if you can provide comments on GitHub
> and we can continue there and summarize the conclusion here?
>
> We should not lose addressing any comments.
>
> On Wed, Sep 4, 2019 at 12:34 PM Pellerin, Clement <
> Clement_Pellerin@ibi.com> wrote:
>
>> The proposed interface does not look like the Builder pattern I am used
>> to.
>> Should SslEngineBuilder be called SslEngineFactory instead?
>>
>> On Mon, Sep 2, 2019, at 03:33, Rajini Sivaram wrote:
>> > I would expect SslEngineBuilder interface to look something like this,
>> > perhaps with some tweaking:
>> >
>> > public interface SslEngineBuilder extends Configurable, Closeable {
>> >
>> > Set<String> reconfigurableConfigs();
>> >
>> > boolean shouldBeRebuilt(Map<String, Object> nextConfigs);
>> >
>> > SSLEngine createSslEngine(Mode mode, String peerHost, int
>> > peerPort, String endpointIdentification);
>> >
>> > }
>>
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Maulin Vasavada <ma...@gmail.com>.
Do you guys think it would be easier if you can provide comments on GitHub
and we can continue there and summarize the conclusion here?
We should not lose addressing any comments.
On Wed, Sep 4, 2019 at 12:34 PM Pellerin, Clement <Cl...@ibi.com>
wrote:
> The proposed interface does not look like the Builder pattern I am used to.
> Should SslEngineBuilder be called SslEngineFactory instead?
>
> On Mon, Sep 2, 2019, at 03:33, Rajini Sivaram wrote:
> > I would expect SslEngineBuilder interface to look something like this,
> > perhaps with some tweaking:
> >
> > public interface SslEngineBuilder extends Configurable, Closeable {
> >
> > Set<String> reconfigurableConfigs();
> >
> > boolean shouldBeRebuilt(Map<String, Object> nextConfigs);
> >
> > SSLEngine createSslEngine(Mode mode, String peerHost, int
> > peerPort, String endpointIdentification);
> >
> > }
>
RE: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by "Pellerin, Clement" <Cl...@ibi.com>.
The proposed interface does not look like the Builder pattern I am used to.
Should SslEngineBuilder be called SslEngineFactory instead?
On Mon, Sep 2, 2019, at 03:33, Rajini Sivaram wrote:
> I would expect SslEngineBuilder interface to look something like this,
> perhaps with some tweaking:
>
> public interface SslEngineBuilder extends Configurable, Closeable {
>
> Set<String> reconfigurableConfigs();
>
> boolean shouldBeRebuilt(Map<String, Object> nextConfigs);
>
> SSLEngine createSslEngine(Mode mode, String peerHost, int
> peerPort, String endpointIdentification);
>
> }
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Maulin Vasavada <ma...@gmail.com>.
Hi Colin
Thanks for your comments. I agree with most of them. This is not a
pull-request ready code yet :)
If we want to make DefaultSslEngineBuilder final then what do you propose
to address our requirements to be able to plugin custom way for loading
keys/certs? (The original challenge we have documented with this KIP). What
I gather from your comment on that is - we will have to standardize those
requirements as public APIs like interfaces documented in the KIP-486 for
KeyStoreLoader/TrustStoreLoader, correct? We have those pluggable APIs to
provide custom source for keys/certs and have SslEngineBuilder Interface
somehow take those?
On Wed, Sep 4, 2019 at 10:58 AM Colin McCabe <cm...@apache.org> wrote:
> On Tue, Sep 3, 2019, at 22:56, Maulin Vasavada wrote:
> > Hi all
> >
> > Please check
> >
> https://github.com/maulin-vasavada/kafka/commit/44f86395b1ba3fe4bd87de89029d72da77995ff8
> >
> >
> > This is just the first cut obviously. There are few call outs I would
> like
> > to make,
> >
> > 1. So far I kept the old SslEngineBuilder hence I had to name the
> interface
> > with "I" (that can change later)
>
> Hi Maulin,
>
> Thanks for working on this.
>
> We don't use Hungarian notation in Kafka. The interface should probably
> just be SslEngineBuilder. The default implementation can be
> DefaultSslEngineBuilder.
>
> >
> > 2. I did not yet add the creation of SslEngineBuilder via loading the
> > configuration like 'ssl.engine.builder.class'. Hence you see direct
> > creation of DefaultSslEngineBuilder class
> >
> > 3. Due to validation logic in the current SslFactory I had to add more
> > methods in ISslEngineBuilder interface (like keystore(), truststore()
> etc).
> > Due to other classes like EchoServer depending upon needing SSLContext, I
> > had to add getSSLContext() also in the interface.
>
> Hmm. I don't think we want to expose this stuff. EchoServer is just used
> for testing, so it can cast the SslEngineBuilder to DefaultEngineBuilder
> (the only one that it will use during JUnit tests) and get what it needs
> that way.
>
> >
> > 4. With these changes and with existing old SslEngineBuilder, the
> > clients/core projects builds with tests successfully but I didn't add any
> > additional tests yet
> >
> > 5. I wanted to have DefaultSslEngineBuilder in such a way that if
> somebody
> > wants to implement custom SslEngineBuilder they can extend and override
> > only key required methods without repeating any logic.
>
> No, DefaultSslEngineBuilder should be final. We should not allow people
> to extend the default engine builder, since then it becomes a public API.
> If there are utility functions that we feel would be useful to everyone, we
> can spell those out explicitly and standardize them as public APIs that
> can't be changed.
>
> >
> > 6. For reconfigurable interface I kept the way suggested by Rajini -
> > meaning SslFactory really is reconfigurable BUT it relies on the
> > ISslEngineBuilder to define the reconfigurable options. This means that
> > ISslEngineBuilder dictates based on which reconfigurable params the
> > SslFactory should try to reconfigure the SSLEngine.
>
> Each SslEngineBuilder should define its own set of reconfigurable
> configurations. We don't know ahead of time what they will need. We want
> to be flexible. People might want to fetch certificates from a central
> location via HTTPS, for example. Or maybe they want to use a native
> library of some kind. And so on.
>
> best,
> Colin
>
> >
> > With this - open to all the suggestions and further improvements.
> >
> > Thanks
> > Maulin
> >
> >
> > On Tue, Sep 3, 2019 at 10:07 AM Colin McCabe <cm...@apache.org> wrote:
> >
> > > On Mon, Sep 2, 2019, at 03:33, Rajini Sivaram wrote:
> > > > I would expect SslEngineBuilder interface to look something like
> this,
> > > > perhaps with some tweaking:
> > > >
> > > > public interface SslEngineBuilder extends Configurable, Closeable {
> > > >
> > > > Set<String> reconfigurableConfigs();
> > > >
> > > > boolean shouldBeRebuilt(Map<String, Object> nextConfigs);
> > > >
> > > > SSLEngine createSslEngine(Mode mode, String peerHost, int
> > > > peerPort, String endpointIdentification);
> > > >
> > > > }
> > > >
> > > > The existing SslEngineBuilder class would be renamed and will
> implement
> > > > this interface. Loading of keystore/truststore will be in
> > > SslEngineBuilder
> > > > as it is now. The method `shouldBeRebuilt()` will validate configs
> > > during
> > > > reconfiguration and decide if reconfiguration is required because
> > > keystore
> > > > or truststore changed. SslFactory.reconfigurableConfigs() will return
> > > > SslEngineBuilder.reconfigurableConfigs() as well including any custom
> > > > configs of SslEngineBuilder, so no other changes will be required
> when we
> > > > eventually support custom SSL configs.
> > > >
> > > > We don't want to make SslFactory the pluggable class since that
> contains
> > > > validation logic for SSL engines. Everything that we want to
> customise is
> > > > contained in SslEngineBuilder. Basically custom SslEngineBuilder will
> > > > validate custom configs during reconfiguration and create SSLEngine.
> > > > SslFactory will continue to perform validation of SSLEngines and this
> > > will
> > > > not be customizable. SslEngineBuilder will not be reconfigurable,
> instead
> > > > we create a new builder as we do now to avoid having to deal with
> > > > thread-safety and atomicity of updates. We could consider using a
> public
> > > > Reconfigurable interface as the pluggable interface for consistency,
> but
> > > I
> > > > think we would still want to create a new Builder on reconfiguration
> and
> > > > retain non-pluggable SSL engine validation in SslFactory.
> > >
> > > +1
> > >
> > > C.
> > >
> > > >
> > > >
> > > > On Fri, Aug 30, 2019 at 10:21 PM Maulin Vasavada <
> > > maulin.vasavada@gmail.com>
> > > > wrote:
> > > >
> > > > > Looking at SslFactory and SslEngineBuilder I feel the
> responsibilities
> > > are
> > > > > not clear. Both has public method for createSSLEngine for example.
> I
> > > feel
> > > > > the SslEngineBuilder was created to move out lot of code but it is
> not
> > > > > necessarily a public class (e.g. I don't think anybody calling
> > > > > SslEngineBuilder separately without SslFactory in between). I am
> > > currently
> > > > > inclined toward what Celement is suggesting - having pluggable
> > > SslFactory.
> > > > >
> > > > > Let me do this - let me refactor SslFactory and SslEngineBuilder
> and
> > > review
> > > > > what I can come up with you guys. Let us see if we can address all
> the
> > > > > objections raised previously for KIP-383's iterations. I'll need
> > > sometime
> > > > > though. Let me try to do it by next of next week.
> > > > >
> > > > > Thanks
> > > > > Maulin
> > > > >
> > > > > On Fri, Aug 30, 2019 at 12:25 PM Pellerin, Clement <
> > > > > Clement_Pellerin@ibi.com>
> > > > > wrote:
> > > > >
> > > > > > What is your solution to the objection that killed the second
> > > iteration
> > > > > of
> > > > > > KIP-383?
> > > > > > Mainly, how do you support validation of reconfiguration requests
> > > that
> > > > > > involve new custom properties implemented by the pluggable
> factory?
> > > > > > Custom properties do not exist yet, but that is very legitimate
> > > thing to
> > > > > > design for the future.
> > > > > >
> > > > > > That's why I favor a pluggable SslFactory instead of an
> > > SslEngineBuilder
> > > > > > factory.
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Maulin Vasavada [mailto:maulin.vasavada@gmail.com]
> > > > > > Sent: Friday, August 30, 2019 3:07 PM
> > > > > > To: dev@kafka.apache.org
> > > > > > Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and
> > > > > > TrustStore
> > > > > >
> > > > > > +1 for making SslEngineBuilder configurable upon more thoughts.
> > > > > >
> > > > > > However, in the abstraction and default implementation we should
> make
> > > > > sure
> > > > > > when we do have a requirement to plugin custom key/trust store
> people
> > > > > don't
> > > > > > have to write lot more code which may not be related to it.
> > > > > >
> > > > > > Having said that, does this mean, we resurrect KIP-383 and
> update it
> > > with
> > > > > > latest context and go from there?
> > > > > >
> > > > > > We are willing to take up that work for making it configurable.
> > > > > >
> > > > > > Thanks
> > > > > > Maulin
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Aug 30, 2019 at 10:34 AM Maulin Vasavada <
> > > > > > maulin.vasavada@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Why don't we make SSLEngineBuilder code delegate the whole
> > > Key/Trust
> > > > > > store
> > > > > > > initialization to the interfaces we are proposing? Default
> > > > > implementation
> > > > > > > for those key/trust store loader interfaces will be "file
> based"
> > > > > loading
> > > > > > vs
> > > > > > > if somebody wants to customize any of it they can.
> > > > > > >
> > > > > > > Would that make sense?
> > > > > > >
> > > > > > > On Fri, Aug 30, 2019 at 10:03 AM Colin McCabe <
> cmccabe@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > >> +1 for making SslEngineBuilder configurable. This would give
> > > > > > >> implementers a lot more flexibility-- to use key distribution
> > > methods
> > > > > > that
> > > > > > >> were not files, for example.
> > > > > > >>
> > > > > > >> best,
> > > > > > >> Colin
> > > > > > >>
> > > > > > >>
> > > > > > >> On Fri, Aug 30, 2019, at 02:03, Rajini Sivaram wrote:
> > > > > > >> > Just to make sure we are on the same page - KIP-383 was
> written
> > > > > before
> > > > > > >> > the
> > > > > > >> > code was refactored. The refactoring addressed some of the
> > > concerns
> > > > > of
> > > > > > >> > KIP-383. My suggestion was to make SslEngineBuilder
> > > configurable.
> > > > > The
> > > > > > >> > default implementation of this pluggable class would be
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > >
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/ssl/SslEngineBuilder.java
> > > > > > >> .
> > > > > > >> > That should give an idea of the size of the configurable
> part
> > > that a
> > > > > > >> > custom
> > > > > > >> > class needs to implement. A large part of that is about
> loading
> > > > > > >> > keystore/truststore. I agree it has slightly more code than
> > > KIP-486
> > > > > > >> > proposes, but since it lets you customize creation of
> > > SSLEngine, it
> > > > > > >> > would
> > > > > > >> > address every possible scenario.
> > > > > > >> >
> > > > > > >> > Thoughts?
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Fri, Aug 30, 2019 at 2:02 AM Maulin Vasavada <
> > > > > > >> maulin.vasavada@gmail.com>
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> > > I thought about it more. I feel that no matter how we
> > > refactor the
> > > > > > >> code
> > > > > > >> > > (with or without KIP-383 integrated), ultimately the need
> of
> > > > > > >> customizing
> > > > > > >> > > loading for keys and certs will still remain. Whenever
> that
> > > need
> > > > > > >> arises we
> > > > > > >> > > might end up thinking about the solution suggested by our
> > > KIP-486.
> > > > > > >> Hence
> > > > > > >> > > regardless of the other KIPs and configurations "if we do
> > > need to
> > > > > > >> customize
> > > > > > >> > > loading of keys/certs, we will need the code changes
> > > suggested by
> > > > > > this
> > > > > > >> > > KIP".
> > > > > > >> > >
> > > > > > >> > > Let me know what you guys think.
> > > > > > >> > >
> > > > > > >> > > Harsha, we are working on changing the interfaces for
> > > key/trust
> > > > > > store
> > > > > > >> > > loaders with Certificate and PrivateKey objects. Will
> > > probably be
> > > > > > >> able to
> > > > > > >> > > update it later today or tomorrow.
> > > > > > >> > >
> > > > > > >> > > Thanks
> > > > > > >> > > Maulin
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > On Thu, Aug 29, 2019 at 2:30 PM Maulin Vasavada <
> > > > > > >> maulin.vasavada@gmail.com
> > > > > > >> > > >
> > > > > > >> > > wrote:
> > > > > > >> > >
> > > > > > >> > > > On that, I actually looked at KIP-383 before briefly.
> > > However,
> > > > > > that
> > > > > > >> > > > sounded like lot of changes suggested.
> > > > > > >> > > >
> > > > > > >> > > > One "key" thing we have to keep in mind is - IF we need
> lot
> > > of
> > > > > > >> > > > customization Kafka already allows you to use your
> > > SslProvider
> > > > > via
> > > > > > >> > > > ssl.providers or the changes done by KIP-492 and
> > > > > > >> > > > SSLContext.getInstance(protocol, provider) call allows
> us to
> > > > > > return
> > > > > > >> the
> > > > > > >> > > > SSLContext with "ALL" the details we would like to
> > > customize.
> > > > > > Hence
> > > > > > >> I am
> > > > > > >> > > > not sure that customization suggested by KIP-383 would
> be
> > > worth
> > > > > > the
> > > > > > >> > > effort.
> > > > > > >> > > > We also have similar SSLContext customization outside of
> > > Kafka.
> > > > > > >> > > >
> > > > > > >> > > > Thanks
> > > > > > >> > > > Maulin
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > > On Thu, Aug 29, 2019 at 12:47 PM Pellerin, Clement <
> > > > > > >> > > > Clement_Pellerin@ibi.com> wrote:
> > > > > > >> > > >
> > > > > > >> > > >> KIP-383 in its present form was vetoed because it was
> not
> > > > > > possible
> > > > > > >> to
> > > > > > >> > > add
> > > > > > >> > > >> validation of custom properties in a future KIP. The
> > > solution
> > > > > to
> > > > > > >> that is
> > > > > > >> > > >> the first proposal I wrote for KIP-383 which made the
> whole
> > > > > > >> SslFactory
> > > > > > >> > > >> pluggable. That first solution was also vetoed hence
> the
> > > > > > deadlock.
> > > > > > >> > > >>
> > > > > > >> > > >> Replacing the whole factory was a much nicer solution.
> It
> > > was
> > > > > > >> vetoed
> > > > > > >> > > >> because doing this almost invariably meant the
> replacement
> > > lost
> > > > > > >> all the
> > > > > > >> > > >> complex validation code in the default SslFactory.
> > > > > > >> > > >>
> > > > > > >> > > >> My current idea is to extract the validation code into
> > > another
> > > > > > >> public
> > > > > > >> > > API
> > > > > > >> > > >> that SslFactory would call. I did not look at the newly
> > > > > > refactored
> > > > > > >> code
> > > > > > >> > > and
> > > > > > >> > > >> I did not study how to do this yet. KIP-383 was not
> > > popular at
> > > > > > the
> > > > > > >> time
> > > > > > >> > > and
> > > > > > >> > > >> designing a new solution is a lot of work.
> > > > > > >> > > >>
> > > > > > >> > > >> Is there interest from 3 binding voters for something
> like
> > > > > this?
> > > > > > >> > > >>
> > > > > > >> > > >> -----Original Message-----
> > > > > > >> > > >> From: Rajini Sivaram [mailto:rajinisivaram@gmail.com]
> > > > > > >> > > >> Sent: Thursday, August 29, 2019 2:57 PM
> > > > > > >> > > >> To: dev
> > > > > > >> > > >> Subject: Re: [DISCUSS] KIP-486 Support for pluggable
> > > KeyStore
> > > > > and
> > > > > > >> > > >> TrustStore
> > > > > > >> > > >>
> > > > > > >> > > >> Hi Maulin,
> > > > > > >> > > >>
> > > > > > >> > > >> In SSL scenarios, I imagine security providers
> introduced
> > > by
> > > > > > >> KIP-492 are
> > > > > > >> > > >> likely to be most useful when you want to use third
> party
> > > > > > >> providers. The
> > > > > > >> > > >> biggest advantage of the config from that KIP is that
> you
> > > don't
> > > > > > >> need to
> > > > > > >> > > >> write much code to integrate existing security
> providers
> > > into
> > > > > > Kafka
> > > > > > >> > > >> brokers
> > > > > > >> > > >> or clients. As I understand it, KIP-486 is a more
> > > convenient
> > > > > > >> option for
> > > > > > >> > > >> the
> > > > > > >> > > >> specific problem of loading keystores/truststores
> > > differently.
> > > > > It
> > > > > > >> can be
> > > > > > >> > > >> achieved in theory with KIP-492, but KIP-486 is a much
> > > simpler
> > > > > > >> option
> > > > > > >> > > for
> > > > > > >> > > >> this case.
> > > > > > >> > > >>
> > > > > > >> > > >> My concern about KIP-486 is that it introduces yet
> another
> > > > > > >> interface
> > > > > > >> > > into
> > > > > > >> > > >> our already complex security code, while only solving
> one
> > > > > > >> particular use
> > > > > > >> > > >> case. Have you looked at
> > > > > > >> > > >>
> > > > > > >> > > >>
> > > > > > >> > >
> > > > > > >>
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-383%3A++Pluggable+interface+for+SSL+Factory
> > > > > > >> > > >> ?
> > > > > > >> > > >> The goal was to make
> > > > > > >> > > >> org.apache.kafka.common.security.ssl.SslEngineBuilder
> > > > > pluggable.
> > > > > > >> > > >> The code has already been refactored by Colin after
> that
> > > KIP
> > > > > was
> > > > > > >> > > written,
> > > > > > >> > > >> making it easier to implement KIP-383. This should
> enable
> > > you
> > > > > to
> > > > > > >> load
> > > > > > >> > > your
> > > > > > >> > > >> keystores and truststores differently. Using a
> pluggable
> > > > > > >> > > SslEngineBuilder
> > > > > > >> > > >> will also solve several other use cases at the same
> time.
> > > > > KIP-383
> > > > > > >> hasn't
> > > > > > >> > > >> been voted through yet, but perhaps you could take a
> look
> > > and
> > > > > we
> > > > > > >> could
> > > > > > >> > > >> revive that instead if it solves your use case as well?
> > > > > > >> > > >>
> > > > > > >> > > >> Regards,
> > > > > > >> > > >>
> > > > > > >> > > >> Rajini
> > > > > > >> > > >>
> > > > > > >> > > >>
> > > > > > >> > > >> On Thu, Aug 29, 2019 at 6:42 PM Maulin Vasavada <
> > > > > > >> > > >> maulin.vasavada@gmail.com>
> > > > > > >> > > >> wrote:
> > > > > > >> > > >>
> > > > > > >> > > >> > Hi Harsha
> > > > > > >> > > >> >
> > > > > > >> > > >> > Thank you. Appreciate your time and support on this.
> Let
> > > me
> > > > > go
> > > > > > >> back do
> > > > > > >> > > >> some
> > > > > > >> > > >> > more research and get back to you on the KeyStore
> > > interface
> > > > > > part.
> > > > > > >> > > >> > Basically, if we return certs and keys in the
> interface
> > > then
> > > > > > >> Kafka
> > > > > > >> > > code
> > > > > > >> > > >> > will have to build KeyStore object - which is also
> > > > > reasonable.
> > > > > > >> > > >> >
> > > > > > >> > > >> > Thanks
> > > > > > >> > > >> > Maulin
> > > > > > >> > > >> >
> > > > > > >> > > >> > On Thu, Aug 29, 2019 at 10:01 AM Harsha Chintalapani
> <
> > > > > > >> kafka@harsha.io
> > > > > > >> > > >
> > > > > > >> > > >> > wrote:
> > > > > > >> > > >> >
> > > > > > >> > > >> > > Hi Maulin,
> > > > > > >> > > >> > > Use cases are clear now. I am
> +1
> > > for
> > > > > > moving
> > > > > > >> > > >> forward
> > > > > > >> > > >> > > with the discussions on having such configurable
> > > option for
> > > > > > >> users.
> > > > > > >> > > But
> > > > > > >> > > >> > the
> > > > > > >> > > >> > > interfaces is proposed doesn't look right to me.
> We are
> > > > > still
> > > > > > >> > > talking
> > > > > > >> > > >> > about
> > > > > > >> > > >> > > keystore interfaces. Given keystore's are used as
> > > > > filebased
> > > > > > >> way of
> > > > > > >> > > >> > > transporting certificates I am not sure it will
> help
> > > the
> > > > > rest
> > > > > > >> of the
> > > > > > >> > > >> > > user-base.
> > > > > > >> > > >> > > In short, I am +1 on the KIP's
> > > motivation
> > > > > > >> and only
> > > > > > >> > > >> have
> > > > > > >> > > >> > > questions around returning keystores instead of
> > > returning
> > > > > > >> certs,
> > > > > > >> > > >> private
> > > > > > >> > > >> > > keys etc. . If others in the community are ok with
> such
> > > > > > >> interface we
> > > > > > >> > > >> can
> > > > > > >> > > >> > > move forward.
> > > > > > >> > > >> > >
> > > > > > >> > > >> > > Thanks,
> > > > > > >> > > >> > > Harsha
> > > > > > >> > > >> > >
> > > > > > >> > > >> > >
> > > > > > >> > > >> > > On Wed, Aug 28, 2019 at 1:51 PM, Maulin Vasavada <
> > > > > > >> > > >> > > maulin.vasavada@gmail.com>
> > > > > > >> > > >> > > wrote:
> > > > > > >> > > >> > >
> > > > > > >> > > >> > > > Hi Harsha
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > As we synced-up offline on this topic, we hope
> you
> > > don't
> > > > > > >> have any
> > > > > > >> > > >> more
> > > > > > >> > > >> > > > clarifications that you are seeking. If that is
> the
> > > case,
> > > > > > >> can you
> > > > > > >> > > >> > please
> > > > > > >> > > >> > > > help us move this forward and discuss what
> changes
> > > you
> > > > > > would
> > > > > > >> > > expect
> > > > > > >> > > >> on
> > > > > > >> > > >> > > the
> > > > > > >> > > >> > > > KIP design in order to make it valuable
> contribution?
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Just FYI - we verified our primary design change
> > > with the
> > > > > > >> author
> > > > > > >> > > of
> > > > > > >> > > >> > Sun's
> > > > > > >> > > >> > > > X509 Trustmanager implementation and the outcome
> is
> > > that
> > > > > > >> what we
> > > > > > >> > > are
> > > > > > >> > > >> > > > proposing makes sense at the heart of it -
> "Instead
> > > of
> > > > > > >> writing
> > > > > > >> > > >> > > TrustManager
> > > > > > >> > > >> > > > just plugin the Trust store". We are open to
> discuss
> > > > > > >> additional
> > > > > > >> > > >> changes
> > > > > > >> > > >> > > > that you/anybody else would like to see on the
> > > > > > functionality
> > > > > > >> > > >> however.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Thanks
> > > > > > >> > > >> > > > Maulin
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada <
> > > > > > >> > > >> > > maulin.vasavada@gmail.com>
> > > > > > >> > > >> > > > wrote:
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Hi Harsha
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Any response on my question? I feel this KIP is
> worth
> > > > > > >> > > accommodating.
> > > > > > >> > > >> > Your
> > > > > > >> > > >> > > > help is much appreciated.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Thanks
> > > > > > >> > > >> > > > Maulin
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada
> <
> > > > > > >> > > >> > maulin.vasavada@gmail.
> > > > > > >> > > >> > > > com> wrote:
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Hi Harsha
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > I've examined the SPIFFE provider more and have
> one
> > > > > > question
> > > > > > >> -
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > If SPIFFE didn't have a need to do
> checkSpiffeId()
> > > call
> > > > > at
> > > > > > >> the
> > > > > > >> > > below
> > > > > > >> > > >> > > > location, would you really still write the
> Provider?
> > > *OR*
> > > > > > >> Would
> > > > > > >> > > you
> > > > > > >> > > >> > just
> > > > > > >> > > >> > > > use TrustManagerFactory.init(KeyStore) signature
> to
> > > pass
> > > > > > the
> > > > > > >> > > >> KeyStore
> > > > > > >> > > >> > > from
> > > > > > >> > > >> > > > set of certs returned by spiffeIdManager.
> > > > > > getTrustedCerts()?
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > >
> > > > > > >> > > >> >
> > > > > > >> > >
> > > > > > >>
> > > > >
> > >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > > > > >> > > >> > > > provider/CertificateUtils.java#L100
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > /**
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > * Validates that the SPIFFE ID is present and
> > > matches the
> > > > > > >> SPIFFE
> > > > > > >> > > ID
> > > > > > >> > > >> > > > configured in
> > > > > > >> > > >> > > > * the java.security property ssl.spiffe.accept
> > > > > > >> > > >> > > > *
> > > > > > >> > > >> > > > * If the authorized spiffe ids list is empty any
> > > spiffe
> > > > > id
> > > > > > is
> > > > > > >> > > >> > authorized
> > > > > > >> > > >> > > > *
> > > > > > >> > > >> > > > * @param chain an array of X509Certificate that
> > > contains
> > > > > > the
> > > > > > >> > > Peer's
> > > > > > >> > > >> > SVID
> > > > > > >> > > >> > > > to be validated
> > > > > > >> > > >> > > > * @throws CertificateException when either the
> > > > > certificates
> > > > > > >> > > doesn't
> > > > > > >> > > >> > have
> > > > > > >> > > >> > > a
> > > > > > >> > > >> > > > SPIFFE ID or the SPIFFE ID is not authorized
> > > > > > >> > > >> > > > */
> > > > > > >> > > >> > > > static void checkSpiffeId(X509Certificate[]
> chain)
> > > throws
> > > > > > >> > > >> > > > CertificateException {
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Thanks
> > > > > > >> > > >> > > > Maulin
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > On Tue, Aug 20, 2019 at 4:49 PM Harsha
> Chintalapani <
> > > > > > >> > > >> kafka@harsha.io>
> > > > > > >> > > >> > > > wrote:
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Maulin,
> > > > > > >> > > >> > > > The code parts you are pointing are specific for
> > > Spiffe
> > > > > and
> > > > > > >> if
> > > > > > >> > > >> > > > you are talking about validate method which uses
> PKIX
> > > > > check
> > > > > > >> like
> > > > > > >> > > any
> > > > > > >> > > >> > > other
> > > > > > >> > > >> > > > provider does.
> > > > > > >> > > >> > > > If you want to default to SunJSSE everywhere you
> can
> > > do
> > > > > so
> > > > > > by
> > > > > > >> > > >> > delegating
> > > > > > >> > > >> > > > the calls in these methods to SunJSSE provider.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > TrustManagerFactory tmf = TrustManagerFactory
> > > > > > >> > > >> > > >
> > > > > .getInstance(TrustManagerFactory.getDefaultAlgorithm());and
> > > > > > >> use
> > > > > > >> > > >> > > > tmf.chekServerTrusted()
> > > > > > >> > > >> > > > or use
> > > > > > >> > > >> > > >
> > > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> > > > > > >> > > >> > > >
> > > TrustManagerFactory.html#getInstance(java.lang.String)if
> > > > > > you
> > > > > > >> want
> > > > > > >> > > a
> > > > > > >> > > >> > > > specific provider.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > -Harsha
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada
> <
> > > > > > >> > > >> > maulin.vasavada@gmail.
> > > > > > >> > > >> > > > com>
> > > > > > >> > > >> > > > wrote:
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Okay, so I take that you guys agree that I have
> to
> > > write
> > > > > a
> > > > > > >> > > 'custom'
> > > > > > >> > > >> > > > algorithm and a provider to make it work ,
> correct?
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Now, for Harsha's comment "Here the 'Custom'
> > > Algorithm is
> > > > > > >> not an
> > > > > > >> > > >> > > > implementation per say , ..." , I diagree. You
> can
> > > refer
> > > > > to
> > > > > > >> > > https://
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > >
> > > > > > >> > > >> >
> > > > > > >> > > >>
> > > > > > >> > >
> > > > > > >>
> > > > > >
> > >
> github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > SpiffeTrustManager.java#L91 <
> > > > > > >> http://spiffetrustmanager.java/#L91>
> > > > > > >> > > >> and
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > >
> > > > > > >> > > >> >
> > > > > > >> > >
> > > > > > >>
> > > > >
> > >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > provider/CertificateUtils.java#L100
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > "that code" is the customization you have for the
> > > custom
> > > > > > way
> > > > > > >> to
> > > > > > >> > > >> check
> > > > > > >> > > >> > > > something on top of regular checks. That method
> is
> > > NOT
> > > > > > doing
> > > > > > >> > > custom
> > > > > > >> > > >> > > > truststore loading. It is validating/verifying
> > > something
> > > > > in
> > > > > > >> the
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > "custom"
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > way with spiffeId.
> > > > > > >> > > >> > > > I bet that without that you won't have a need of
> the
> > > > > custom
> > > > > > >> > > >> algorithm
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > in
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > the first place.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Let me know if you agree to this.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Thanks
> > > > > > >> > > >> > > > Maulin
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <
> > > > > > >> mprsai@gmail.com>
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > wrote:
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Hi Maulin, thanks for the discussion. As Harsha
> > > pointed
> > > > > > out,
> > > > > > >> to
> > > > > > >> > > use
> > > > > > >> > > >> the
> > > > > > >> > > >> > > > KIP492, you need to create a new provider,
> register a
> > > > > *new*
> > > > > > >> custom
> > > > > > >> > > >> > > > algorithm for your keymanager and trustmanager
> > > factory
> > > > > > >> > > >> implementations.
> > > > > > >> > > >> > > > After this, the kafka server configuration can be
> > > done as
> > > > > > >> given
> > > > > > >> > > >> below
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > # Register the provider class with custom
> algorithm,
> > > say
> > > > > > >> CUSTOM
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > security.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > >
> provider.classes=com.company.security.CustomProvider
> > > > > > >> > > >> > > > <http://provider.classes
> > > > > > >> =com.company.security.customprovider/>
> > > > > > >> > > >> > > > <http://security.provider.classes
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > =com.company.security.customprovider/>
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > # Register the keymanager and trustmanager
> algorithms
> > > > > > >> > > >> > > > # These algorithms indicate that the Keymanager
> and
> > > > > > >> Trustmanagers
> > > > > > >> > > >> > > > registered under the algorithm "CUSTOM" needs to
> be
> > > used
> > > > > > >> > > >> > > > ssl.trustmanager.algorithm=CUSTOM
> > > > > > >> > > >> > > > ssl.keymanager.algorithm=CUSTOM
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > And the customprovider looks like this...
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > public class CustomProvider extends Provider {
> > > > > > >> > > >> > > > public CustomProvider() {
> > > > > > >> > > >> > > > super("NEW_CUSTOM_PROVIDER", 0.1, "Custom
> KeyStore
> > > and
> > > > > > >> > > TrustStore");
> > > > > > >> > > >> > > > super.put("KeyManagerFactory.CUSTOM",
> > > > > > >> "customKeyManagerFactory");
> > > > > > >> > > >> > > > super.put("TrustManagerFactory.CUSTOM",
> > > > > > >> > > >> > > > "customTrustManagerFactory");
> > > > > > >> > > >> > > > }
> > > > > > >> > > >> > > > }
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > The PR for this is in review and can be found
> here:
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > https://github.com/
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > apache/kafka/pull/7090
> > > > > > >> > > >> > > > This PR includes the fixed insertProviderAt
> function
> > > > > call.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > On Tue, Aug 20, 2019 at 9:56 AM Harsha
> Chintalapani <
> > > > > > >> > > >> kafka@harsha.io>
> > > > > > >> > > >> > > > wrote:
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Answers are in-line
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > On Mon, Aug 19, 2019 at 10:45 PM, Maulin
> Vasavada <
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > maulin.vasavada@gmail.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > com
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > wrote:
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Hi Colin
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > When I refer to "standard" or "custom"
> algorithms I
> > > am
> > > > > > >> following
> > > > > > >> > > >> Java
> > > > > > >> > > >> > > > security Provider Terminology. You can refer to
> > > > > > >> > > >> > > >
> > > > > > >>
> https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> > > > > > >> > > >> > > > StandardNames.html#TrustManagerFactory link I
> > > provided
> > > > > > >> earlier in
> > > > > > >> > > >> the
> > > > > > >> > > >> > > > emails. It says PKIX is the default Algorithm for
> > > > > > >> > > >> TrustManagerFactory.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > 1. For SPIFFE, I am not sure why you are saying
> 'it
> > > does
> > > > > > not
> > > > > > >> > > >> implement
> > > > > > >> > > >> > > > custom algorithms' because the following file
> clearly
> > > > > > >> indicates
> > > > > > >> > > >> that it
> > > > > > >> > > >> > > > does use custom algorithm-
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > >
> > > > > > >> > > >> >
> > > > > > >> > >
> > > > > > >>
> > > > >
> > >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > provider/SpiffeProvider.java#L17
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Algorithm value:
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > >
> > > > > > >> > > >> >
> > > > > > >> > >
> > > > > > >>
> > > > >
> > >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > provider/SpiffeProviderConstants.java#L6
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > @Harsha do you want to chime in since you use
> that
> > > > > > provider?
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Here the "Custom" Algorithm is not an
> implementation
> > > per
> > > > > > say
> > > > > > >> ,
> > > > > > >> > > >> rather
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > used
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > to invoke the custom trust store factory and key
> > > manager
> > > > > > >> factory.
> > > > > > >> > > >> You
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > are
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > not moving away from "standard" alogrithms that
> are
> > > > > > >> available.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > >
> > > > > > >> > > >> >
> > > > > > >> > >
> > > > > > >>
> > > > >
> > >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > provider/SpiffeTrustManager.java
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > As you can see it delegates all the calls of
> > > verification
> > > > > > of
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > certificate
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > to
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > the default implementation available.
> > > > > > >> > > >> > > > So in our implementation we still use PKIX to
> verify
> > > the
> > > > > > >> > > certificate
> > > > > > >> > > >> > > > chain. So you are not losing anything here and
> > > Spiffe is
> > > > > > not
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > reimplementing
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > the verification process.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > 2. I already mentioned in my 3rd point, in my
> > > previous
> > > > > > post,
> > > > > > >> why
> > > > > > >> > > >> using
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > ssl.provider does NOT work. I updated KIP-486 in
> > > > > "rejected
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > alternatives"
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > also why ssl.provider does not work.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > As mentioned before , provider is the right way
> to
> > > go. I
> > > > > am
> > > > > > >> still
> > > > > > >> > > >> not
> > > > > > >> > > >> > > > understanding the gap is.
> > > > > > >> > > >> > > > If I understand correctly your argument is ,
> > > provider is
> > > > > > >> going to
> > > > > > >> > > >> ask
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > to
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > implement a custom algorithm.
> > > > > > >> > > >> > > > My answer to that is , no you are not
> > > re-implementing the
> > > > > > >> > > algorithm.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Please
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > check the above link , TrustManager
> implementation it
> > > > > > >> delegates
> > > > > > >> > > the
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > calls
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > back. There is no need to implement your own
> here.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > 3. Security.insertProviderAt() comments were
> based on
> > > > > > >> assumption
> > > > > > >> > > if
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > KIP-492
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > changes are done and we use that mechanism to
> > > configure
> > > > > > >> providers
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > instead
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > of ssl.provider configuration.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > KIP-492 has patch available and going through
> review.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Can you read my all the points, I mentioned in my
> > > > > previous
> > > > > > >> post,
> > > > > > >> > > >> very
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > carefully? I am covering all the aspects in
> > > explaining. I
> > > > > > am
> > > > > > >> open
> > > > > > >> > > to
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > still
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > discuss more to clarify any doubts.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > "3. If we just use existing ssl.provider kafka
> > > > > > configuration
> > > > > > >> then
> > > > > > >> > > >> our
> > > > > > >> > > >> > > > provider will be used in
> > > SSLContext.getInstance(protocol,
> > > > > > >> > > provider)
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > call
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > in
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > SslFactory.java <http://sslfactory.java/> <
> > > > > > >> > > http://sslfactory.java/>
> > > > > > >> > > >> <
> > > > > > >> > > >> > > > http://sslfactory.java/>
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > and
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > if our provider does not have
> > > > > > >> > > >> > > > implementation for SSLContext.TLS/TLSv1.1/TLSv1.2
> > > etc it
> > > > > > >> breaks
> > > > > > >> > > (we
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > tested
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > it). Example: In MyProvider sample above you see
> > > that I
> > > > > > >> didn't add
> > > > > > >> > > >> > > > SSLContext.TLSv1 as
> > > > > > >> > > >> > > > "Service+Algorithm" and that didn't work for me.
> In
> > > > > SPIFFE
> > > > > > >> > > provider
> > > > > > >> > > >> you
> > > > > > >> > > >> > > > don't have this challenge since you are planning
> to
> > > > > bypass
> > > > > > >> > > >> ssl.provider
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > as
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > you mention in the KIP-492."
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Yes here you need to pass the protocol that your
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > KeyManager/TrustManager
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > registered with and in no way its deviating from
> TLS
> > > RFC
> > > > > > >> spec.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > >
> > > > > > >> > > >> >
> > > > > > >> > >
> > > > > > >>
> > > > >
> > >
> https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > javax/net/ssl/SSLContext.java#L134
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > My suggestion here is for you to implement a
> simple
> > > > > > Security
> > > > > > >> > > >> Provider
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > as
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > you did before and register a Algorithm. You can
> use
> > > the
> > > > > > >> existing
> > > > > > >> > > >> > > > implementation in SpiffeProvider and plug in
> changes
> > > > > where
> > > > > > >> you
> > > > > > >> > > need
> > > > > > >> > > >> to
> > > > > > >> > > >> > > > retrieve the certificates from by making RPC
> call.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Run an end-to-end test with Kafka broker coming
> up
> > > with
> > > > > > your
> > > > > > >> > > >> provider
> > > > > > >> > > >> > > > making calls to RPC call. You do need to pass the
> > > "custom
> > > > > > >> > > algorithm"
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > that
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > you registered in your key manager to invoke the
> > > > > provider.
> > > > > > I
> > > > > > >> think
> > > > > > >> > > >> your
> > > > > > >> > > >> > > > concern is this approach is replacing the
> existing
> > > known
> > > > > > >> > > >> ciphersuites
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > and
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > certificate validation provided by java. Which
> its
> > > not.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Now test the TLS connection you can do so via
> openssl
> > > > > > >> -s_client
> > > > > > >> > > >> options
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > to
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > test if everything else is working.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > I am happy to share configs that we used if you
> are
> > > > > > >> interested.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Thanks,
> > > > > > >> > > >> > > > Harsha
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Thanks
> > > > > > >> > > >> > > > Maulin
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <
> > > > > > >> cmccabe@apache.org>
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > wrote:
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Hi Maulin,
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > A lot of JSSE providers don't implement custom
> > > > > algorithms.
> > > > > > >> Spire
> > > > > > >> > > is
> > > > > > >> > > >> a
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > good
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > example of a JSSE provider that doesn't, and yet
> is
> > > still
> > > > > > >> useful
> > > > > > >> > > to
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > many
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > people. Your JSSE provider can work fine even if
> it
> > > > > doesn't
> > > > > > >> > > >> implement a
> > > > > > >> > > >> > > > custom algorithm.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Maybe I'm missing something, but I don't
> understand
> > > the
> > > > > > >> discussion
> > > > > > >> > > >> of
> > > > > > >> > > >> > > > Security.insertProviderAt() that you included.
> > > > > > >> SslEngineBuilder
> > > > > > >> > > >> doesn't
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > use
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > that API to get the security provider. Instead,
> it
> > > calls
> > > > > > >> > > >> > > > "SSLContext.getInstance(protocol, provider)",
> where
> > > > > > provider
> > > > > > >> is
> > > > > > >> > > the
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > name
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > of the provider.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > best,
> > > > > > >> > > >> > > > Colin
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada
> > > wrote:
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > On top of everything above I feel strongly to add
> > > the 4th
> > > > > > >> point
> > > > > > >> > > >> which
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > is
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > based on Java APIs for
> > > > > TrustManagerFactory.init(KeyStore) (
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > >
> > > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> > > > > > >> > > >> > > >
> TrustManagerFactory.html#init(java.security.KeyStore
> > > > > > >> > > >> > > > <http://java.security.keystore/>
> > > > > > >> > > >> > > > <http://java.security.keystore/>)
> > > > > > >> > > >> > > > )
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > and KeyManagerFactory.init(KeyStore, char[]) (
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > >
> > > > > > >> > > >> >
> > > > > > >> > > >>
> > > > > > >> > >
> > > > > > >>
> > > > > >
> > > > >
> > >
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory
> > > > > > >> > > >> > > > .
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > html#init(java.security.KeyStore <
> > > > > > >> http://java.security.keystore/>
> > > > > > >> > > >> > > <http://
> > > > > > >> > > >> > > > java.security.keystore/
> > > > > > >> > > >> > > > ,%20char[])
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > ).
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > 4. The above APIs are intended to support
> providing
> > > > > > >> "trust/key
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > material"
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > from the user without having to write their own
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > TrustManager/KeyManagers.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > To quote from the TrustManagerFactory.init()'s
> > > > > > documentation
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > "Initializes
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > this factory with a source of certificate
> > > authorities and
> > > > > > >> related
> > > > > > >> > > >> trust
> > > > > > >> > > >> > > > material."
> > > > > > >> > > >> > > > To quote from the KeyManagerFactory.init()'s
> > > > > documentation
> > > > > > >> > > >> "Initializes
> > > > > > >> > > >> > > > this factory with a source of key material."
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Based on this it is clear that there is a
> flexibility
> > > > > > >> provided by
> > > > > > >> > > >> Java
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > to
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > to enable developers to provide the required
> > > trust/key
> > > > > > >> material
> > > > > > >> > > >> loaded
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > from
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > "anywhere" without requiring them to write custom
> > > > > provider
> > > > > > OR
> > > > > > >> > > >> trust/key
> > > > > > >> > > >> > > > managers. This same flexibility is reflected in
> Kafka
> > > > > code
> > > > > > >> also
> > > > > > >> > > >> where
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > it
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > loads the trust/keys from a local file and
> doesn't
> > > > > require
> > > > > > >> > > writing a
> > > > > > >> > > >> > > > Provider necessarily. If we do NOT have a custom
> > > > > algorithm,
> > > > > > >> it
> > > > > > >> > > makes
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > less
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > sense to write a Provider.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Thanks
> > > > > > >> > > >> > > > Maulin
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada
> <
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > maulin.vasavada@gmail.com>
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > wrote:
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Hi Harsha/Colin
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > I did the sample with a custom Provider for
> > > > > > >> TrustStoreManager and
> > > > > > >> > > >> tried
> > > > > > >> > > >> > > > using ssl.provider Kafka config AND the way
> KIP-492
> > > is
> > > > > > >> suggesting
> > > > > > >> > > >> (by
> > > > > > >> > > >> > > > adding Provider programmatically instead of
> relying
> > > on
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > ssl.provider+java.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > security. The below sample is followed by my
> detailed
> > > > > > >> findings.
> > > > > > >> > > I'll
> > > > > > >> > > >> > > > appreciate if you can go through it carefully
> and see
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > if you
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > see my point.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > package providertest;
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > import java.security.Provider <
> > > > > > >> http://java.security.provider/>
> > > > > > >> > > >> > <http://
> > > > > > >> > > >> > > > java.security.provider/>
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > <http://
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > java.security.provider/>;
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > public class MyProvider extends Provider {
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > private static final String name = "MyProvider";
> > > private
> > > > > > >> static
> > > > > > >> > > >> double
> > > > > > >> > > >> > > > version = 1.0d;
> > > > > > >> > > >> > > > private static String info = "Maulin's SSL
> Provider
> > > > > > >> v"+version;
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > public MyProvider() {
> > > > > > >> > > >> > > > super(name, version, info);
> > > > > > >> > > >> > > > this.put("TrustManagerFactory.PKIX",
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > "providertest.MyTrustManagerFactory");
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > }
> > > > > > >> > > >> > > > }
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > *Details:*
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > KIP-492 documents that it will use
> > > Security.addProvider()
> > > > > > >> assuming
> > > > > > >> > > >> it
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > will
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > add it as position '0' which is not a correct
> > > assumption.
> > > > > > The
> > > > > > >> > > >> > > > addProvider()'s documentation says it will add
> it to
> > > the
> > > > > > last
> > > > > > >> > > >> available
> > > > > > >> > > >> > > > position. You may want to correct that to say
> > > > > > >> > > >> > > > Security.insertProviderAt(provider, 1).
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Now coming back to our specific discussion,
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > 1. SPIFFE example uses Custom Algorithm - spiffe.
> > > Hence
> > > > > > when
> > > > > > >> you
> > > > > > >> > > add
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > that
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > provider in the provider list via
> > > Security.addProvider()
> > > > > > the
> > > > > > >> > > >> position
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > where
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > it gets added doesn't matter (even if you don't
> end
> > > up
> > > > > > >> adding it
> > > > > > >> > > as
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > first
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > entry) since that is the ONLY provider for SPIFFE
> > > > > specific
> > > > > > >> > > algorithm
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > you
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > might have.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > We do *not* have custom algorithm for Key/Trust
> > > > > > StoreMangers.
> > > > > > >> > > Which
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > means
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > we have to use X509, PKIX etc "Standard
> Algorithms"
> > > ((
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > >
> > > > > > >>
> https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> > > > > > >> > > >> > > > StandardNames.html
> > > > > > >> > > >> > > > ))
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > in our provider to override the TrustStoreManager
> > > (see my
> > > > > > >> sample
> > > > > > >> > > >> code)
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > and
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > KeyStoreManger and KeyManager. This creates
> another
> > > > > > challenge
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > mentioned in
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > the below point.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > 2. In order to make our Provider for loading
> custom
> > > > > > >> TrustStore
> > > > > > >> > > >> work, we
> > > > > > >> > > >> > > > have to add the provider as 'first' in the list
> since
> > > > > there
> > > > > > >> are
> > > > > > >> > > >> others
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > with
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > the same algorithm.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > However, the programatic way of adding provider
> > > > > > >> > > >> > > > (Security.insertProviderAt()) is *not*
> deterministic
> > > for
> > > > > > >> ordering
> > > > > > >> > > >> since
> > > > > > >> > > >> > > > different code can call the method for a
> different
> > > > > provider
> > > > > > >> and
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > depending
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > upon the order of the call our provider can be
> first
> > > or
> > > > > > >> pushed
> > > > > > >> > > down
> > > > > > >> > > >> the
> > > > > > >> > > >> > > > list. This can happen very well in any client
> > > application
> > > > > > >> using
> > > > > > >> > > >> Kafka.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > This
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > is specially problematic for a case when you
> want to
> > > > > > >> guarantee
> > > > > > >> > > order
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > for a
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Provider having "Standard Algorithms".
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > If we add our provider in java.security file that
> > > > > > definitely
> > > > > > >> > > >> guarantees
> > > > > > >> > > >> > > > the order(unless somebody calls removeProvider()
> > > which is
> > > > > > >> less
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > likely). But
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > if we add our provider in java.security file it
> will
> > > > > defeat
> > > > > > >> the
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > purpose of
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > the KIP-492.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > In the gist - Apache Kafka must not rely on
> > > > > > >> "un-deterministic"
> > > > > > >> > > >> method
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > to
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > rely on Provider ordering.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > 3. If we just use existing ssl.provider kafka
> > > > > configuration
> > > > > > >> then
> > > > > > >> > > our
> > > > > > >> > > >> > > > provider will be used in
> > > SSLContext.getInstance(protocol,
> > > > > > >> > > provider)
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > call in
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > SslFactory.java <http://sslfactory.java/> <
> > > > > > >> > > http://sslfactory.java/>
> > > > > > >> > > >> <
> > > > > > >> > > >> > > > http://sslfactory.java/>
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > and
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > if our provider does not have implementation for
> > > > > > >> > > >> > > > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > (we
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > tested it). Example:
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > In
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > MyProvider sample above you see that I didn't add
> > > > > > >> SSLContext.TLSv1
> > > > > > >> > > >> as
> > > > > > >> > > >> > > > "Service+Algorithm" and that didn't work for me.
> In
> > > > > SPIFFE
> > > > > > >> > > provider
> > > > > > >> > > >> you
> > > > > > >> > > >> > > > don't have this challenge since you are planning
> to
> > > > > bypass
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > ssl.provider as
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > you mention in the KIP-492.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > *Overall summary,*
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > 1. Any provider based mechanisms- a) existing
> > > > > ssl.provider
> > > > > > >> and
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > b)KIP-492,
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > for loading key/trust store using "Standard
> > > Algorithms"
> > > > > do
> > > > > > >> not
> > > > > > >> > > work
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > 2. Approach suggested in our KIP-486 works
> without
> > > any
> > > > > > issue
> > > > > > >> and
> > > > > > >> > > it
> > > > > > >> > > >> is
> > > > > > >> > > >> > > > *not* our context specific solve
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > 3. Based on above we feel KIP-492 and KIP-486 are
> > > > > > >> complimentary
> > > > > > >> > > >> changes
> > > > > > >> > > >> > > > and not contradicting or redundent.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > If you want we can do a joint session somehow to
> walk
> > > > > > >> through the
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > sample I
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > have and various experiments I did. I would
> > > encourage you
> > > > > > to
> > > > > > >> do
> > > > > > >> > > >> similar
> > > > > > >> > > >> > > > exercise by writing a Provider for "Standard
> > > Algorithm"
> > > > > for
> > > > > > >> > > >> > > > TrustStoreManager (like our needs) and see what
> you
> > > find
> > > > > > >> since
> > > > > > >> > > only
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > writing
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > samples can bring out the complexity/challenges
> we
> > > face.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Thanks
> > > > > > >> > > >> > > > Maulin
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada
> <
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > maulin.vasavada@gmail.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > com> wrote:
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Just to update - still working on it. Get to work
> > > only on
> > > > > > >> and off
> > > > > > >> > > on
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > it :(
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > maulin.vasavada@gmail.com>
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > wrote:
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Hi Harsha
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Let me try to write samples and will let you
> know.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Thanks
> > > > > > >> > > >> > > > Maulin
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <
> > > > > > >> harsha.ch@gmail.com>
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > wrote:
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Hi Maulin,
> > > > > > >> > > >> > > > With java security providers can be as custom you
> > > would
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > like
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > it to
> > > > > > >> > > >> > > > be. If you only want to to implement a custom
> way of
> > > > > > loading
> > > > > > >> the
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > keystore
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > and truststore and not implement any
> > > protocol/encryption
> > > > > > >> handling
> > > > > > >> > > >> you
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > can
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > leave them empty and no need to implement. Have
> you
> > > > > looked
> > > > > > >> into
> > > > > > >> > > the
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > links I
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > pasted before?
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > >
> > > > > > >>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > >
> > > > > > >> > > >> >
> > > > > > >> > > >>
> > > > > > >> > >
> > > > > > >>
> > > > > >
> > > > >
> > >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > java
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > >
> > > > > > >>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > > > > >> > > >> > > >
> > > > > spiffe-security-provider/src/main/java/spiffe/api/provider/
> > > > > > >> > > >> > > > SpiffeTrustManager.java <
> > > http://spiffetrustmanager.java/
> > > > > >
> > > > > > >> > > <http://
> > > > > > >> > > >> > > > spiffetrustmanager.java/>
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > >
> > > > > > >>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > >
> > > > > > >> > > >> >
> > > > > > >> > > >>
> > > > > > >> > >
> > > > > > >>
> > > > > >
> > > > >
> > >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > java
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Can you please tell me which methods are too
> complex
> > > in
> > > > > > >> above to
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > implement
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > or unnecessary? You are changing anything in
> SSL/TLS
> > > > > > >> > > implementations
> > > > > > >> > > >> > > > provided by
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > All of the implementations delegating the checks
> to
> > > the
> > > > > > >> default
> > > > > > >> > > >> > > > implementation anyway.
> > > > > > >> > > >> > > > Spire agent is an example, its nothing but a GRPC
> > > server
> > > > > > >> listening
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > on a
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > unix domain socket . Above code is making a RPC
> call
> > > to
> > > > > the
> > > > > > >> local
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > daemon
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > to
> > > > > > >> > > >> > > > get the certificate and keys. The mechanics are
> > > pretty
> > > > > much
> > > > > > >> same
> > > > > > >> > > as
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > what
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > you are asking for.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Thanks,
> > > > > > >> > > >> > > > Harsha
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > maulin.vasavada@gmail.com>
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > wrote:
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Imagine a scenario like - We know we have a
> custom
> > > KMS
> > > > > and
> > > > > > >> as a
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Kafka
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > owner
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > we want to comply to using that KMS source to
> load
> > > > > > >> keys/certs. As
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > a
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Kafka
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > owner we know how to integrate with KMS but
> doesn't
> > > > > > >> necessarily
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > have
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > to
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > know anything about cipher suites, algorithms,
> and
> > > > > SSL/TLS
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > implementation.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Going the Provider way requires to know lot more
> > > than we
> > > > > > >> should,
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > isn't it?
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Not that we would have concern/shy-away knowing
> those
> > > > > > >> details -
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > but
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > if we
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > don't have to - why should we?
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > maulin.vasavada@gmail.com>
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > wrote:
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Hi Harsha
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > We don't have spire (or similar) agents and we
> do not
> > > > > have
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > keys/certs
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > locally on any brokers. To elaborate more on my
> > > previous
> > > > > > >> email,
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > I agree that Java security Providers are used in
> much
> > > > > > broader
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > sense
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > - to
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > have a particular implementation of an
> algorithm, use
> > > > > > >> specific
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > cipher
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > suites for SSL , OR in our current team's case
> have a
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > particular
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > way to
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > leverage pre-generated SSL sessions. However, the
> > > scope
> > > > > of
> > > > > > >> our
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > KIP
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > (486)
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > is
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > much restricted than that. We merely intend to
> > > provide a
> > > > > > >> custom
> > > > > > >> > > >> > > > keystore/truststore for our SSL connections and
> not
> > > > > really
> > > > > > >> worry
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > about
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > underlying specific SSL/TLS implementation. This
> > > > > simplifies
> > > > > > >> it
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > a
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > lot for
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > us to keep the concerns separate and clear.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > I feel our approach is more complimentary such
> that
> > > it
> > > > > > allows
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > for
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > using
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > keystores of choice while retaining the
> flexibility
> > > to
> > > > > use
> > > > > > >> any
> > > > > > >> > > >> > > > underlying/available Provider for actually making
> > > the SSL
> > > > > > >> call.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > We agree with KIP-492's approach based on
> Providers
> > > (and
> > > > > > >> Java's
> > > > > > >> > > >> > > > recommendation), but also strongly believe that
> our
> > > > > > approach
> > > > > > >> can
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > compliment
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > it very effectively for reasons explained above.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Thanks
> > > > > > >> > > >> > > > Maulin
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > On Thu, Aug 8, 2019 at 3:05 PM Harsha
> Chintalapani <
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > kafka@harsha.io
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > wrote:
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Hi Maulin,
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada
> <
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > maulin.vasavada@gmail.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > com>
> > > > > > >> > > >> > > > wrote:
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Hi Harsha
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > The reason we rejected the SslProvider route is
> that
> > > - we
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > only
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > needed
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > a
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > custom way to load keys/certs. Not touch any
> policy
> > > that
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > existing
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Providers
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > govern like SunJSSE Provider.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > We have exactly the same requirements to load
> certs
> > > and
> > > > > > keys
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > through
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > spire
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > agent. We used security.provider to do that
> exactly.
> > > I am
> > > > > > not
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > sure
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > why
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > you
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > would be modifying any policies provided by
> default
> > > > > SunJSSE
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > provider.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Can
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > you give me an example of having custom provider
> that
> > > > > will
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > override an
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > existing policy in SunJSSE provider.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > As pointed out earlier, this kip
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > > >> > > >> > > >
> > > > > > >>
> KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > allows
> > > > > > >> > > >> > > > you to load security.provider through config.
> > > > > > >> > > >> > > > Take a look at the examples I gave before
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > >
> > > > > > >>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > >
> > > > > > >> > > >> >
> > > > > > >> > > >>
> > > > > > >> > >
> > > > > > >>
> > > > > >
> > > > >
> > >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > java
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > It registers KeyManagerFactory and
> > > TrustManagerFactory
> > > > > and
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Keystore
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > algorithm.
> > > > > > >> > > >> > > > Implement your custom way of loading Keystore in
> here
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > >
> > > > > > >>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > >
> > > > > > >> > > >> >
> > > > > > >> > > >>
> > > > > > >> > >
> > > > > > >>
> > > > > >
> > > > >
> > >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > java
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > and Trust manager like here
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > >
> > > > > > >>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > > > > >> > > >> > > >
> > > > > spiffe-security-provider/src/main/java/spiffe/api/provider/
> > > > > > >> > > >> > > > SpiffeTrustManager.java <
> > > http://spiffetrustmanager.java/
> > > > > >
> > > > > > >> > > <http://
> > > > > > >> > > >> > > > spiffetrustmanager.java/>
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > In your Kafka client you can set the
> > > security.provider to
> > > > > > >> your
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > custom
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > implementation and with this fix
> > > > > > >> > > >> > > > https://issues.apache.org/jira/browse/KAFKA-8191
> > > you can
> > > > > > set
> > > > > > >> > > >> > > > keyManagerAlgorigthm and trustManagerAlgorithm
> > > configs.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > All of this is in your clients and broker side
> and
> > > do not
> > > > > > >> need
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > to
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > touch
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > any
> > > > > > >> > > >> > > > policy changes at JVM level. You'll register the
> > > > > providers
> > > > > > in
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > the
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > priority
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > order and can still have SunJSSE provider and
> have
> > > your
> > > > > > >> custom
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > provider
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > to
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > implement the key and trust managers.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > The ask here is different than KIP-492. We don't
> > > have any
> > > > > > >> need
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > to
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > modify/specify the algorithm parameter. Does that
> > > make
> > > > > > sense?
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > The ask in KIP is introducing new interfaces
> where
> > > the
> > > > > > KIP's
> > > > > > >> > > >> > > > goal/motivation can be achieved through the
> > > > > > security.provider
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > and
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > we
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > worked
> > > > > > >> > > >> > > > on similar goal without touching any Keystore or
> > > > > Truststore
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > interfaces.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > My advise is against changing or introducing new
> > > > > interfaces
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > when
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > it can
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > work through security.provider.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Thanks,
> > > > > > >> > > >> > > > Harsha
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Thanks
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Maulin
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > On Thu, Aug 8, 2019 at 7:48 AM Harsha
> Chintalapani <
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > kafka@harsha.io>
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > wrote:
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > In your KIP you added security. provider as
> rejected
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > alternative
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > and
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > specified "its not the correct way". Do you mind
> > > > > explaining
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > why
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > its
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > not? I
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > didn't find any evidence in Java docs to say so.
> > > Contrary
> > > > > > to
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > your
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > statement
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > it does say in the java docs
> > > > > > >> > > >> > > > " However, please note that a provider can be
> used to
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > implement
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > any
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > security service in Java that uses a pluggable
> > > > > architecture
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > with
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > a
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > choice
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > of implementations that fit underneath."
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Java Security Providers have been used by other
> > > projects
> > > > > to
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > provide
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > such
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > integration . I am not sure if you looked into
> Spiffe
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > project to
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > efficiently distribute certificates but here is
> an
> > > > > example
> > > > > > of
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Java
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > provider
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > >
> > > > > > >>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > >
> > > > > > >> > > >> >
> > > > > > >> > > >>
> > > > > > >> > >
> > > > > > >>
> > > > > >
> > > > >
> > >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > java which
> > > > > > >> > > >> > > > obtains certificates from local daemons.
> > > > > > >> > > >> > > > These integrations are being used in Tomcat,
> Jetty
> > > etc..
> > > > > We
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > are
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > also
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > using
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Security provider to do the same in our Kafka
> > > clusters.
> > > > > So
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > unless I
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > see
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > more evidence why security.provider doesn't work
> for
> > > you
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > adding
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > new
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > interfaces while there exists more cleaner way of
> > > > > achieving
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > the
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > goals
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > of
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > this KIP is unnecessary and breaks the well known
> > > > > security
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > interfaces
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > provided by Java itself.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Thanks,
> > > > > > >> > > >> > > > Harsha
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > On Thu, Aug 08, 2019 at 6:54 AM, Harsha
> Chintalapani
> > > <
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > kafka@harsha.io
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > wrote:
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Hi Maulin,
> > > > > > >> > > >> > > > Not sure if you looked at my previous replies.
> This
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > changes
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > are not required as there is already security
> > > Provider to
> > > > > > do
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > what you
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > are
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > proposing. This KIP
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > >
> > > > > > >>
> KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > also
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > addresses easy registration of such providers.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Thanks,
> > > > > > >> > > >> > > > Harsha
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > <maulin.vasavada@gmail.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > com> wrote:
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Bump! Can somebody please review this?
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > maulin.vasavada@gmail.com>
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > wrote:
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Bump! Can somebody please review this?
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > --
> > > > > > >> > > >> > > > Thanks,
> > > > > > >> > > >> > > > M.Sai Sandeep
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > >
> > > > > > >> > > >> >
> > > > > > >> > > >>
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Colin McCabe <cm...@apache.org>.
On Tue, Sep 3, 2019, at 22:56, Maulin Vasavada wrote:
> Hi all
>
> Please check
> https://github.com/maulin-vasavada/kafka/commit/44f86395b1ba3fe4bd87de89029d72da77995ff8
>
>
> This is just the first cut obviously. There are few call outs I would like
> to make,
>
> 1. So far I kept the old SslEngineBuilder hence I had to name the interface
> with "I" (that can change later)
Hi Maulin,
Thanks for working on this.
We don't use Hungarian notation in Kafka. The interface should probably just be SslEngineBuilder. The default implementation can be DefaultSslEngineBuilder.
>
> 2. I did not yet add the creation of SslEngineBuilder via loading the
> configuration like 'ssl.engine.builder.class'. Hence you see direct
> creation of DefaultSslEngineBuilder class
>
> 3. Due to validation logic in the current SslFactory I had to add more
> methods in ISslEngineBuilder interface (like keystore(), truststore() etc).
> Due to other classes like EchoServer depending upon needing SSLContext, I
> had to add getSSLContext() also in the interface.
Hmm. I don't think we want to expose this stuff. EchoServer is just used for testing, so it can cast the SslEngineBuilder to DefaultEngineBuilder (the only one that it will use during JUnit tests) and get what it needs that way.
>
> 4. With these changes and with existing old SslEngineBuilder, the
> clients/core projects builds with tests successfully but I didn't add any
> additional tests yet
>
> 5. I wanted to have DefaultSslEngineBuilder in such a way that if somebody
> wants to implement custom SslEngineBuilder they can extend and override
> only key required methods without repeating any logic.
No, DefaultSslEngineBuilder should be final. We should not allow people to extend the default engine builder, since then it becomes a public API. If there are utility functions that we feel would be useful to everyone, we can spell those out explicitly and standardize them as public APIs that can't be changed.
>
> 6. For reconfigurable interface I kept the way suggested by Rajini -
> meaning SslFactory really is reconfigurable BUT it relies on the
> ISslEngineBuilder to define the reconfigurable options. This means that
> ISslEngineBuilder dictates based on which reconfigurable params the
> SslFactory should try to reconfigure the SSLEngine.
Each SslEngineBuilder should define its own set of reconfigurable configurations. We don't know ahead of time what they will need. We want to be flexible. People might want to fetch certificates from a central location via HTTPS, for example. Or maybe they want to use a native library of some kind. And so on.
best,
Colin
>
> With this - open to all the suggestions and further improvements.
>
> Thanks
> Maulin
>
>
> On Tue, Sep 3, 2019 at 10:07 AM Colin McCabe <cm...@apache.org> wrote:
>
> > On Mon, Sep 2, 2019, at 03:33, Rajini Sivaram wrote:
> > > I would expect SslEngineBuilder interface to look something like this,
> > > perhaps with some tweaking:
> > >
> > > public interface SslEngineBuilder extends Configurable, Closeable {
> > >
> > > Set<String> reconfigurableConfigs();
> > >
> > > boolean shouldBeRebuilt(Map<String, Object> nextConfigs);
> > >
> > > SSLEngine createSslEngine(Mode mode, String peerHost, int
> > > peerPort, String endpointIdentification);
> > >
> > > }
> > >
> > > The existing SslEngineBuilder class would be renamed and will implement
> > > this interface. Loading of keystore/truststore will be in
> > SslEngineBuilder
> > > as it is now. The method `shouldBeRebuilt()` will validate configs
> > during
> > > reconfiguration and decide if reconfiguration is required because
> > keystore
> > > or truststore changed. SslFactory.reconfigurableConfigs() will return
> > > SslEngineBuilder.reconfigurableConfigs() as well including any custom
> > > configs of SslEngineBuilder, so no other changes will be required when we
> > > eventually support custom SSL configs.
> > >
> > > We don't want to make SslFactory the pluggable class since that contains
> > > validation logic for SSL engines. Everything that we want to customise is
> > > contained in SslEngineBuilder. Basically custom SslEngineBuilder will
> > > validate custom configs during reconfiguration and create SSLEngine.
> > > SslFactory will continue to perform validation of SSLEngines and this
> > will
> > > not be customizable. SslEngineBuilder will not be reconfigurable, instead
> > > we create a new builder as we do now to avoid having to deal with
> > > thread-safety and atomicity of updates. We could consider using a public
> > > Reconfigurable interface as the pluggable interface for consistency, but
> > I
> > > think we would still want to create a new Builder on reconfiguration and
> > > retain non-pluggable SSL engine validation in SslFactory.
> >
> > +1
> >
> > C.
> >
> > >
> > >
> > > On Fri, Aug 30, 2019 at 10:21 PM Maulin Vasavada <
> > maulin.vasavada@gmail.com>
> > > wrote:
> > >
> > > > Looking at SslFactory and SslEngineBuilder I feel the responsibilities
> > are
> > > > not clear. Both has public method for createSSLEngine for example. I
> > feel
> > > > the SslEngineBuilder was created to move out lot of code but it is not
> > > > necessarily a public class (e.g. I don't think anybody calling
> > > > SslEngineBuilder separately without SslFactory in between). I am
> > currently
> > > > inclined toward what Celement is suggesting - having pluggable
> > SslFactory.
> > > >
> > > > Let me do this - let me refactor SslFactory and SslEngineBuilder and
> > review
> > > > what I can come up with you guys. Let us see if we can address all the
> > > > objections raised previously for KIP-383's iterations. I'll need
> > sometime
> > > > though. Let me try to do it by next of next week.
> > > >
> > > > Thanks
> > > > Maulin
> > > >
> > > > On Fri, Aug 30, 2019 at 12:25 PM Pellerin, Clement <
> > > > Clement_Pellerin@ibi.com>
> > > > wrote:
> > > >
> > > > > What is your solution to the objection that killed the second
> > iteration
> > > > of
> > > > > KIP-383?
> > > > > Mainly, how do you support validation of reconfiguration requests
> > that
> > > > > involve new custom properties implemented by the pluggable factory?
> > > > > Custom properties do not exist yet, but that is very legitimate
> > thing to
> > > > > design for the future.
> > > > >
> > > > > That's why I favor a pluggable SslFactory instead of an
> > SslEngineBuilder
> > > > > factory.
> > > > >
> > > > > -----Original Message-----
> > > > > From: Maulin Vasavada [mailto:maulin.vasavada@gmail.com]
> > > > > Sent: Friday, August 30, 2019 3:07 PM
> > > > > To: dev@kafka.apache.org
> > > > > Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and
> > > > > TrustStore
> > > > >
> > > > > +1 for making SslEngineBuilder configurable upon more thoughts.
> > > > >
> > > > > However, in the abstraction and default implementation we should make
> > > > sure
> > > > > when we do have a requirement to plugin custom key/trust store people
> > > > don't
> > > > > have to write lot more code which may not be related to it.
> > > > >
> > > > > Having said that, does this mean, we resurrect KIP-383 and update it
> > with
> > > > > latest context and go from there?
> > > > >
> > > > > We are willing to take up that work for making it configurable.
> > > > >
> > > > > Thanks
> > > > > Maulin
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Aug 30, 2019 at 10:34 AM Maulin Vasavada <
> > > > > maulin.vasavada@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Why don't we make SSLEngineBuilder code delegate the whole
> > Key/Trust
> > > > > store
> > > > > > initialization to the interfaces we are proposing? Default
> > > > implementation
> > > > > > for those key/trust store loader interfaces will be "file based"
> > > > loading
> > > > > vs
> > > > > > if somebody wants to customize any of it they can.
> > > > > >
> > > > > > Would that make sense?
> > > > > >
> > > > > > On Fri, Aug 30, 2019 at 10:03 AM Colin McCabe <cm...@apache.org>
> > > > > wrote:
> > > > > >
> > > > > >> +1 for making SslEngineBuilder configurable. This would give
> > > > > >> implementers a lot more flexibility-- to use key distribution
> > methods
> > > > > that
> > > > > >> were not files, for example.
> > > > > >>
> > > > > >> best,
> > > > > >> Colin
> > > > > >>
> > > > > >>
> > > > > >> On Fri, Aug 30, 2019, at 02:03, Rajini Sivaram wrote:
> > > > > >> > Just to make sure we are on the same page - KIP-383 was written
> > > > before
> > > > > >> > the
> > > > > >> > code was refactored. The refactoring addressed some of the
> > concerns
> > > > of
> > > > > >> > KIP-383. My suggestion was to make SslEngineBuilder
> > configurable.
> > > > The
> > > > > >> > default implementation of this pluggable class would be
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/ssl/SslEngineBuilder.java
> > > > > >> .
> > > > > >> > That should give an idea of the size of the configurable part
> > that a
> > > > > >> > custom
> > > > > >> > class needs to implement. A large part of that is about loading
> > > > > >> > keystore/truststore. I agree it has slightly more code than
> > KIP-486
> > > > > >> > proposes, but since it lets you customize creation of
> > SSLEngine, it
> > > > > >> > would
> > > > > >> > address every possible scenario.
> > > > > >> >
> > > > > >> > Thoughts?
> > > > > >> >
> > > > > >> >
> > > > > >> > On Fri, Aug 30, 2019 at 2:02 AM Maulin Vasavada <
> > > > > >> maulin.vasavada@gmail.com>
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > I thought about it more. I feel that no matter how we
> > refactor the
> > > > > >> code
> > > > > >> > > (with or without KIP-383 integrated), ultimately the need of
> > > > > >> customizing
> > > > > >> > > loading for keys and certs will still remain. Whenever that
> > need
> > > > > >> arises we
> > > > > >> > > might end up thinking about the solution suggested by our
> > KIP-486.
> > > > > >> Hence
> > > > > >> > > regardless of the other KIPs and configurations "if we do
> > need to
> > > > > >> customize
> > > > > >> > > loading of keys/certs, we will need the code changes
> > suggested by
> > > > > this
> > > > > >> > > KIP".
> > > > > >> > >
> > > > > >> > > Let me know what you guys think.
> > > > > >> > >
> > > > > >> > > Harsha, we are working on changing the interfaces for
> > key/trust
> > > > > store
> > > > > >> > > loaders with Certificate and PrivateKey objects. Will
> > probably be
> > > > > >> able to
> > > > > >> > > update it later today or tomorrow.
> > > > > >> > >
> > > > > >> > > Thanks
> > > > > >> > > Maulin
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Thu, Aug 29, 2019 at 2:30 PM Maulin Vasavada <
> > > > > >> maulin.vasavada@gmail.com
> > > > > >> > > >
> > > > > >> > > wrote:
> > > > > >> > >
> > > > > >> > > > On that, I actually looked at KIP-383 before briefly.
> > However,
> > > > > that
> > > > > >> > > > sounded like lot of changes suggested.
> > > > > >> > > >
> > > > > >> > > > One "key" thing we have to keep in mind is - IF we need lot
> > of
> > > > > >> > > > customization Kafka already allows you to use your
> > SslProvider
> > > > via
> > > > > >> > > > ssl.providers or the changes done by KIP-492 and
> > > > > >> > > > SSLContext.getInstance(protocol, provider) call allows us to
> > > > > return
> > > > > >> the
> > > > > >> > > > SSLContext with "ALL" the details we would like to
> > customize.
> > > > > Hence
> > > > > >> I am
> > > > > >> > > > not sure that customization suggested by KIP-383 would be
> > worth
> > > > > the
> > > > > >> > > effort.
> > > > > >> > > > We also have similar SSLContext customization outside of
> > Kafka.
> > > > > >> > > >
> > > > > >> > > > Thanks
> > > > > >> > > > Maulin
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > On Thu, Aug 29, 2019 at 12:47 PM Pellerin, Clement <
> > > > > >> > > > Clement_Pellerin@ibi.com> wrote:
> > > > > >> > > >
> > > > > >> > > >> KIP-383 in its present form was vetoed because it was not
> > > > > possible
> > > > > >> to
> > > > > >> > > add
> > > > > >> > > >> validation of custom properties in a future KIP. The
> > solution
> > > > to
> > > > > >> that is
> > > > > >> > > >> the first proposal I wrote for KIP-383 which made the whole
> > > > > >> SslFactory
> > > > > >> > > >> pluggable. That first solution was also vetoed hence the
> > > > > deadlock.
> > > > > >> > > >>
> > > > > >> > > >> Replacing the whole factory was a much nicer solution. It
> > was
> > > > > >> vetoed
> > > > > >> > > >> because doing this almost invariably meant the replacement
> > lost
> > > > > >> all the
> > > > > >> > > >> complex validation code in the default SslFactory.
> > > > > >> > > >>
> > > > > >> > > >> My current idea is to extract the validation code into
> > another
> > > > > >> public
> > > > > >> > > API
> > > > > >> > > >> that SslFactory would call. I did not look at the newly
> > > > > refactored
> > > > > >> code
> > > > > >> > > and
> > > > > >> > > >> I did not study how to do this yet. KIP-383 was not
> > popular at
> > > > > the
> > > > > >> time
> > > > > >> > > and
> > > > > >> > > >> designing a new solution is a lot of work.
> > > > > >> > > >>
> > > > > >> > > >> Is there interest from 3 binding voters for something like
> > > > this?
> > > > > >> > > >>
> > > > > >> > > >> -----Original Message-----
> > > > > >> > > >> From: Rajini Sivaram [mailto:rajinisivaram@gmail.com]
> > > > > >> > > >> Sent: Thursday, August 29, 2019 2:57 PM
> > > > > >> > > >> To: dev
> > > > > >> > > >> Subject: Re: [DISCUSS] KIP-486 Support for pluggable
> > KeyStore
> > > > and
> > > > > >> > > >> TrustStore
> > > > > >> > > >>
> > > > > >> > > >> Hi Maulin,
> > > > > >> > > >>
> > > > > >> > > >> In SSL scenarios, I imagine security providers introduced
> > by
> > > > > >> KIP-492 are
> > > > > >> > > >> likely to be most useful when you want to use third party
> > > > > >> providers. The
> > > > > >> > > >> biggest advantage of the config from that KIP is that you
> > don't
> > > > > >> need to
> > > > > >> > > >> write much code to integrate existing security providers
> > into
> > > > > Kafka
> > > > > >> > > >> brokers
> > > > > >> > > >> or clients. As I understand it, KIP-486 is a more
> > convenient
> > > > > >> option for
> > > > > >> > > >> the
> > > > > >> > > >> specific problem of loading keystores/truststores
> > differently.
> > > > It
> > > > > >> can be
> > > > > >> > > >> achieved in theory with KIP-492, but KIP-486 is a much
> > simpler
> > > > > >> option
> > > > > >> > > for
> > > > > >> > > >> this case.
> > > > > >> > > >>
> > > > > >> > > >> My concern about KIP-486 is that it introduces yet another
> > > > > >> interface
> > > > > >> > > into
> > > > > >> > > >> our already complex security code, while only solving one
> > > > > >> particular use
> > > > > >> > > >> case. Have you looked at
> > > > > >> > > >>
> > > > > >> > > >>
> > > > > >> > >
> > > > > >>
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-383%3A++Pluggable+interface+for+SSL+Factory
> > > > > >> > > >> ?
> > > > > >> > > >> The goal was to make
> > > > > >> > > >> org.apache.kafka.common.security.ssl.SslEngineBuilder
> > > > pluggable.
> > > > > >> > > >> The code has already been refactored by Colin after that
> > KIP
> > > > was
> > > > > >> > > written,
> > > > > >> > > >> making it easier to implement KIP-383. This should enable
> > you
> > > > to
> > > > > >> load
> > > > > >> > > your
> > > > > >> > > >> keystores and truststores differently. Using a pluggable
> > > > > >> > > SslEngineBuilder
> > > > > >> > > >> will also solve several other use cases at the same time.
> > > > KIP-383
> > > > > >> hasn't
> > > > > >> > > >> been voted through yet, but perhaps you could take a look
> > and
> > > > we
> > > > > >> could
> > > > > >> > > >> revive that instead if it solves your use case as well?
> > > > > >> > > >>
> > > > > >> > > >> Regards,
> > > > > >> > > >>
> > > > > >> > > >> Rajini
> > > > > >> > > >>
> > > > > >> > > >>
> > > > > >> > > >> On Thu, Aug 29, 2019 at 6:42 PM Maulin Vasavada <
> > > > > >> > > >> maulin.vasavada@gmail.com>
> > > > > >> > > >> wrote:
> > > > > >> > > >>
> > > > > >> > > >> > Hi Harsha
> > > > > >> > > >> >
> > > > > >> > > >> > Thank you. Appreciate your time and support on this. Let
> > me
> > > > go
> > > > > >> back do
> > > > > >> > > >> some
> > > > > >> > > >> > more research and get back to you on the KeyStore
> > interface
> > > > > part.
> > > > > >> > > >> > Basically, if we return certs and keys in the interface
> > then
> > > > > >> Kafka
> > > > > >> > > code
> > > > > >> > > >> > will have to build KeyStore object - which is also
> > > > reasonable.
> > > > > >> > > >> >
> > > > > >> > > >> > Thanks
> > > > > >> > > >> > Maulin
> > > > > >> > > >> >
> > > > > >> > > >> > On Thu, Aug 29, 2019 at 10:01 AM Harsha Chintalapani <
> > > > > >> kafka@harsha.io
> > > > > >> > > >
> > > > > >> > > >> > wrote:
> > > > > >> > > >> >
> > > > > >> > > >> > > Hi Maulin,
> > > > > >> > > >> > > Use cases are clear now. I am +1
> > for
> > > > > moving
> > > > > >> > > >> forward
> > > > > >> > > >> > > with the discussions on having such configurable
> > option for
> > > > > >> users.
> > > > > >> > > But
> > > > > >> > > >> > the
> > > > > >> > > >> > > interfaces is proposed doesn't look right to me. We are
> > > > still
> > > > > >> > > talking
> > > > > >> > > >> > about
> > > > > >> > > >> > > keystore interfaces. Given keystore's are used as
> > > > filebased
> > > > > >> way of
> > > > > >> > > >> > > transporting certificates I am not sure it will help
> > the
> > > > rest
> > > > > >> of the
> > > > > >> > > >> > > user-base.
> > > > > >> > > >> > > In short, I am +1 on the KIP's
> > motivation
> > > > > >> and only
> > > > > >> > > >> have
> > > > > >> > > >> > > questions around returning keystores instead of
> > returning
> > > > > >> certs,
> > > > > >> > > >> private
> > > > > >> > > >> > > keys etc. . If others in the community are ok with such
> > > > > >> interface we
> > > > > >> > > >> can
> > > > > >> > > >> > > move forward.
> > > > > >> > > >> > >
> > > > > >> > > >> > > Thanks,
> > > > > >> > > >> > > Harsha
> > > > > >> > > >> > >
> > > > > >> > > >> > >
> > > > > >> > > >> > > On Wed, Aug 28, 2019 at 1:51 PM, Maulin Vasavada <
> > > > > >> > > >> > > maulin.vasavada@gmail.com>
> > > > > >> > > >> > > wrote:
> > > > > >> > > >> > >
> > > > > >> > > >> > > > Hi Harsha
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > As we synced-up offline on this topic, we hope you
> > don't
> > > > > >> have any
> > > > > >> > > >> more
> > > > > >> > > >> > > > clarifications that you are seeking. If that is the
> > case,
> > > > > >> can you
> > > > > >> > > >> > please
> > > > > >> > > >> > > > help us move this forward and discuss what changes
> > you
> > > > > would
> > > > > >> > > expect
> > > > > >> > > >> on
> > > > > >> > > >> > > the
> > > > > >> > > >> > > > KIP design in order to make it valuable contribution?
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Just FYI - we verified our primary design change
> > with the
> > > > > >> author
> > > > > >> > > of
> > > > > >> > > >> > Sun's
> > > > > >> > > >> > > > X509 Trustmanager implementation and the outcome is
> > that
> > > > > >> what we
> > > > > >> > > are
> > > > > >> > > >> > > > proposing makes sense at the heart of it - "Instead
> > of
> > > > > >> writing
> > > > > >> > > >> > > TrustManager
> > > > > >> > > >> > > > just plugin the Trust store". We are open to discuss
> > > > > >> additional
> > > > > >> > > >> changes
> > > > > >> > > >> > > > that you/anybody else would like to see on the
> > > > > functionality
> > > > > >> > > >> however.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Thanks
> > > > > >> > > >> > > > Maulin
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada <
> > > > > >> > > >> > > maulin.vasavada@gmail.com>
> > > > > >> > > >> > > > wrote:
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Hi Harsha
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Any response on my question? I feel this KIP is worth
> > > > > >> > > accommodating.
> > > > > >> > > >> > Your
> > > > > >> > > >> > > > help is much appreciated.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Thanks
> > > > > >> > > >> > > > Maulin
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada <
> > > > > >> > > >> > maulin.vasavada@gmail.
> > > > > >> > > >> > > > com> wrote:
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Hi Harsha
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > I've examined the SPIFFE provider more and have one
> > > > > question
> > > > > >> -
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > If SPIFFE didn't have a need to do checkSpiffeId()
> > call
> > > > at
> > > > > >> the
> > > > > >> > > below
> > > > > >> > > >> > > > location, would you really still write the Provider?
> > *OR*
> > > > > >> Would
> > > > > >> > > you
> > > > > >> > > >> > just
> > > > > >> > > >> > > > use TrustManagerFactory.init(KeyStore) signature to
> > pass
> > > > > the
> > > > > >> > > >> KeyStore
> > > > > >> > > >> > > from
> > > > > >> > > >> > > > set of certs returned by spiffeIdManager.
> > > > > getTrustedCerts()?
> > > > > >> > > >> > > >
> > > > > >> > > >> > > >
> > > > > >> > > >> >
> > > > > >> > >
> > > > > >>
> > > >
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > > > >> > > >> > > > provider/CertificateUtils.java#L100
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > /**
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > * Validates that the SPIFFE ID is present and
> > matches the
> > > > > >> SPIFFE
> > > > > >> > > ID
> > > > > >> > > >> > > > configured in
> > > > > >> > > >> > > > * the java.security property ssl.spiffe.accept
> > > > > >> > > >> > > > *
> > > > > >> > > >> > > > * If the authorized spiffe ids list is empty any
> > spiffe
> > > > id
> > > > > is
> > > > > >> > > >> > authorized
> > > > > >> > > >> > > > *
> > > > > >> > > >> > > > * @param chain an array of X509Certificate that
> > contains
> > > > > the
> > > > > >> > > Peer's
> > > > > >> > > >> > SVID
> > > > > >> > > >> > > > to be validated
> > > > > >> > > >> > > > * @throws CertificateException when either the
> > > > certificates
> > > > > >> > > doesn't
> > > > > >> > > >> > have
> > > > > >> > > >> > > a
> > > > > >> > > >> > > > SPIFFE ID or the SPIFFE ID is not authorized
> > > > > >> > > >> > > > */
> > > > > >> > > >> > > > static void checkSpiffeId(X509Certificate[] chain)
> > throws
> > > > > >> > > >> > > > CertificateException {
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Thanks
> > > > > >> > > >> > > > Maulin
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani <
> > > > > >> > > >> kafka@harsha.io>
> > > > > >> > > >> > > > wrote:
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Maulin,
> > > > > >> > > >> > > > The code parts you are pointing are specific for
> > Spiffe
> > > > and
> > > > > >> if
> > > > > >> > > >> > > > you are talking about validate method which uses PKIX
> > > > check
> > > > > >> like
> > > > > >> > > any
> > > > > >> > > >> > > other
> > > > > >> > > >> > > > provider does.
> > > > > >> > > >> > > > If you want to default to SunJSSE everywhere you can
> > do
> > > > so
> > > > > by
> > > > > >> > > >> > delegating
> > > > > >> > > >> > > > the calls in these methods to SunJSSE provider.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > TrustManagerFactory tmf = TrustManagerFactory
> > > > > >> > > >> > > >
> > > > .getInstance(TrustManagerFactory.getDefaultAlgorithm());and
> > > > > >> use
> > > > > >> > > >> > > > tmf.chekServerTrusted()
> > > > > >> > > >> > > > or use
> > > > > >> > > >> > > >
> > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> > > > > >> > > >> > > >
> > TrustManagerFactory.html#getInstance(java.lang.String)if
> > > > > you
> > > > > >> want
> > > > > >> > > a
> > > > > >> > > >> > > > specific provider.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > -Harsha
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada <
> > > > > >> > > >> > maulin.vasavada@gmail.
> > > > > >> > > >> > > > com>
> > > > > >> > > >> > > > wrote:
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Okay, so I take that you guys agree that I have to
> > write
> > > > a
> > > > > >> > > 'custom'
> > > > > >> > > >> > > > algorithm and a provider to make it work , correct?
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Now, for Harsha's comment "Here the 'Custom'
> > Algorithm is
> > > > > >> not an
> > > > > >> > > >> > > > implementation per say , ..." , I diagree. You can
> > refer
> > > > to
> > > > > >> > > https://
> > > > > >> > > >> > > >
> > > > > >> > > >> > > >
> > > > > >> > > >> >
> > > > > >> > > >>
> > > > > >> > >
> > > > > >>
> > > > >
> > github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > SpiffeTrustManager.java#L91 <
> > > > > >> http://spiffetrustmanager.java/#L91>
> > > > > >> > > >> and
> > > > > >> > > >> > > >
> > > > > >> > > >> > > >
> > > > > >> > > >> >
> > > > > >> > >
> > > > > >>
> > > >
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > provider/CertificateUtils.java#L100
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > "that code" is the customization you have for the
> > custom
> > > > > way
> > > > > >> to
> > > > > >> > > >> check
> > > > > >> > > >> > > > something on top of regular checks. That method is
> > NOT
> > > > > doing
> > > > > >> > > custom
> > > > > >> > > >> > > > truststore loading. It is validating/verifying
> > something
> > > > in
> > > > > >> the
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > "custom"
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > way with spiffeId.
> > > > > >> > > >> > > > I bet that without that you won't have a need of the
> > > > custom
> > > > > >> > > >> algorithm
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > in
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > the first place.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Let me know if you agree to this.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Thanks
> > > > > >> > > >> > > > Maulin
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <
> > > > > >> mprsai@gmail.com>
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > wrote:
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Hi Maulin, thanks for the discussion. As Harsha
> > pointed
> > > > > out,
> > > > > >> to
> > > > > >> > > use
> > > > > >> > > >> the
> > > > > >> > > >> > > > KIP492, you need to create a new provider, register a
> > > > *new*
> > > > > >> custom
> > > > > >> > > >> > > > algorithm for your keymanager and trustmanager
> > factory
> > > > > >> > > >> implementations.
> > > > > >> > > >> > > > After this, the kafka server configuration can be
> > done as
> > > > > >> given
> > > > > >> > > >> below
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > # Register the provider class with custom algorithm,
> > say
> > > > > >> CUSTOM
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > security.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > provider.classes=com.company.security.CustomProvider
> > > > > >> > > >> > > > <http://provider.classes
> > > > > >> =com.company.security.customprovider/>
> > > > > >> > > >> > > > <http://security.provider.classes
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > =com.company.security.customprovider/>
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > # Register the keymanager and trustmanager algorithms
> > > > > >> > > >> > > > # These algorithms indicate that the Keymanager and
> > > > > >> Trustmanagers
> > > > > >> > > >> > > > registered under the algorithm "CUSTOM" needs to be
> > used
> > > > > >> > > >> > > > ssl.trustmanager.algorithm=CUSTOM
> > > > > >> > > >> > > > ssl.keymanager.algorithm=CUSTOM
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > And the customprovider looks like this...
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > public class CustomProvider extends Provider {
> > > > > >> > > >> > > > public CustomProvider() {
> > > > > >> > > >> > > > super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore
> > and
> > > > > >> > > TrustStore");
> > > > > >> > > >> > > > super.put("KeyManagerFactory.CUSTOM",
> > > > > >> "customKeyManagerFactory");
> > > > > >> > > >> > > > super.put("TrustManagerFactory.CUSTOM",
> > > > > >> > > >> > > > "customTrustManagerFactory");
> > > > > >> > > >> > > > }
> > > > > >> > > >> > > > }
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > The PR for this is in review and can be found here:
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > https://github.com/
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > apache/kafka/pull/7090
> > > > > >> > > >> > > > This PR includes the fixed insertProviderAt function
> > > > call.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani <
> > > > > >> > > >> kafka@harsha.io>
> > > > > >> > > >> > > > wrote:
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Answers are in-line
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada <
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > maulin.vasavada@gmail.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > com
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > wrote:
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Hi Colin
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > When I refer to "standard" or "custom" algorithms I
> > am
> > > > > >> following
> > > > > >> > > >> Java
> > > > > >> > > >> > > > security Provider Terminology. You can refer to
> > > > > >> > > >> > > >
> > > > > >> https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> > > > > >> > > >> > > > StandardNames.html#TrustManagerFactory link I
> > provided
> > > > > >> earlier in
> > > > > >> > > >> the
> > > > > >> > > >> > > > emails. It says PKIX is the default Algorithm for
> > > > > >> > > >> TrustManagerFactory.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > 1. For SPIFFE, I am not sure why you are saying 'it
> > does
> > > > > not
> > > > > >> > > >> implement
> > > > > >> > > >> > > > custom algorithms' because the following file clearly
> > > > > >> indicates
> > > > > >> > > >> that it
> > > > > >> > > >> > > > does use custom algorithm-
> > > > > >> > > >> > > >
> > > > > >> > > >> > > >
> > > > > >> > > >> >
> > > > > >> > >
> > > > > >>
> > > >
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > provider/SpiffeProvider.java#L17
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Algorithm value:
> > > > > >> > > >> > > >
> > > > > >> > > >> > > >
> > > > > >> > > >> >
> > > > > >> > >
> > > > > >>
> > > >
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > provider/SpiffeProviderConstants.java#L6
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > @Harsha do you want to chime in since you use that
> > > > > provider?
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Here the "Custom" Algorithm is not an implementation
> > per
> > > > > say
> > > > > >> ,
> > > > > >> > > >> rather
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > used
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > to invoke the custom trust store factory and key
> > manager
> > > > > >> factory.
> > > > > >> > > >> You
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > are
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > not moving away from "standard" alogrithms that are
> > > > > >> available.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > >
> > > > > >> > > >> >
> > > > > >> > >
> > > > > >>
> > > >
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > provider/SpiffeTrustManager.java
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > As you can see it delegates all the calls of
> > verification
> > > > > of
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > certificate
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > to
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > the default implementation available.
> > > > > >> > > >> > > > So in our implementation we still use PKIX to verify
> > the
> > > > > >> > > certificate
> > > > > >> > > >> > > > chain. So you are not losing anything here and
> > Spiffe is
> > > > > not
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > reimplementing
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > the verification process.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > 2. I already mentioned in my 3rd point, in my
> > previous
> > > > > post,
> > > > > >> why
> > > > > >> > > >> using
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > ssl.provider does NOT work. I updated KIP-486 in
> > > > "rejected
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > alternatives"
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > also why ssl.provider does not work.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > As mentioned before , provider is the right way to
> > go. I
> > > > am
> > > > > >> still
> > > > > >> > > >> not
> > > > > >> > > >> > > > understanding the gap is.
> > > > > >> > > >> > > > If I understand correctly your argument is ,
> > provider is
> > > > > >> going to
> > > > > >> > > >> ask
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > to
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > implement a custom algorithm.
> > > > > >> > > >> > > > My answer to that is , no you are not
> > re-implementing the
> > > > > >> > > algorithm.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Please
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > check the above link , TrustManager implementation it
> > > > > >> delegates
> > > > > >> > > the
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > calls
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > back. There is no need to implement your own here.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > 3. Security.insertProviderAt() comments were based on
> > > > > >> assumption
> > > > > >> > > if
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > KIP-492
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > changes are done and we use that mechanism to
> > configure
> > > > > >> providers
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > instead
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > of ssl.provider configuration.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > KIP-492 has patch available and going through review.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Can you read my all the points, I mentioned in my
> > > > previous
> > > > > >> post,
> > > > > >> > > >> very
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > carefully? I am covering all the aspects in
> > explaining. I
> > > > > am
> > > > > >> open
> > > > > >> > > to
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > still
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > discuss more to clarify any doubts.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > "3. If we just use existing ssl.provider kafka
> > > > > configuration
> > > > > >> then
> > > > > >> > > >> our
> > > > > >> > > >> > > > provider will be used in
> > SSLContext.getInstance(protocol,
> > > > > >> > > provider)
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > call
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > in
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > SslFactory.java <http://sslfactory.java/> <
> > > > > >> > > http://sslfactory.java/>
> > > > > >> > > >> <
> > > > > >> > > >> > > > http://sslfactory.java/>
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > and
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > if our provider does not have
> > > > > >> > > >> > > > implementation for SSLContext.TLS/TLSv1.1/TLSv1.2
> > etc it
> > > > > >> breaks
> > > > > >> > > (we
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > tested
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > it). Example: In MyProvider sample above you see
> > that I
> > > > > >> didn't add
> > > > > >> > > >> > > > SSLContext.TLSv1 as
> > > > > >> > > >> > > > "Service+Algorithm" and that didn't work for me. In
> > > > SPIFFE
> > > > > >> > > provider
> > > > > >> > > >> you
> > > > > >> > > >> > > > don't have this challenge since you are planning to
> > > > bypass
> > > > > >> > > >> ssl.provider
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > as
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > you mention in the KIP-492."
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Yes here you need to pass the protocol that your
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > KeyManager/TrustManager
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > registered with and in no way its deviating from TLS
> > RFC
> > > > > >> spec.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > >
> > > > > >> > > >> >
> > > > > >> > >
> > > > > >>
> > > >
> > https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > javax/net/ssl/SSLContext.java#L134
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > My suggestion here is for you to implement a simple
> > > > > Security
> > > > > >> > > >> Provider
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > as
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > you did before and register a Algorithm. You can use
> > the
> > > > > >> existing
> > > > > >> > > >> > > > implementation in SpiffeProvider and plug in changes
> > > > where
> > > > > >> you
> > > > > >> > > need
> > > > > >> > > >> to
> > > > > >> > > >> > > > retrieve the certificates from by making RPC call.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Run an end-to-end test with Kafka broker coming up
> > with
> > > > > your
> > > > > >> > > >> provider
> > > > > >> > > >> > > > making calls to RPC call. You do need to pass the
> > "custom
> > > > > >> > > algorithm"
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > that
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > you registered in your key manager to invoke the
> > > > provider.
> > > > > I
> > > > > >> think
> > > > > >> > > >> your
> > > > > >> > > >> > > > concern is this approach is replacing the existing
> > known
> > > > > >> > > >> ciphersuites
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > and
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > certificate validation provided by java. Which its
> > not.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Now test the TLS connection you can do so via openssl
> > > > > >> -s_client
> > > > > >> > > >> options
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > to
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > test if everything else is working.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > I am happy to share configs that we used if you are
> > > > > >> interested.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Thanks,
> > > > > >> > > >> > > > Harsha
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Thanks
> > > > > >> > > >> > > > Maulin
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <
> > > > > >> cmccabe@apache.org>
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > wrote:
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Hi Maulin,
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > A lot of JSSE providers don't implement custom
> > > > algorithms.
> > > > > >> Spire
> > > > > >> > > is
> > > > > >> > > >> a
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > good
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > example of a JSSE provider that doesn't, and yet is
> > still
> > > > > >> useful
> > > > > >> > > to
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > many
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > people. Your JSSE provider can work fine even if it
> > > > doesn't
> > > > > >> > > >> implement a
> > > > > >> > > >> > > > custom algorithm.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Maybe I'm missing something, but I don't understand
> > the
> > > > > >> discussion
> > > > > >> > > >> of
> > > > > >> > > >> > > > Security.insertProviderAt() that you included.
> > > > > >> SslEngineBuilder
> > > > > >> > > >> doesn't
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > use
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > that API to get the security provider. Instead, it
> > calls
> > > > > >> > > >> > > > "SSLContext.getInstance(protocol, provider)", where
> > > > > provider
> > > > > >> is
> > > > > >> > > the
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > name
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > of the provider.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > best,
> > > > > >> > > >> > > > Colin
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada
> > wrote:
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > On top of everything above I feel strongly to add
> > the 4th
> > > > > >> point
> > > > > >> > > >> which
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > is
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > based on Java APIs for
> > > > TrustManagerFactory.init(KeyStore) (
> > > > > >> > > >> > > >
> > > > > >> > > >> > > >
> > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> > > > > >> > > >> > > > TrustManagerFactory.html#init(java.security.KeyStore
> > > > > >> > > >> > > > <http://java.security.keystore/>
> > > > > >> > > >> > > > <http://java.security.keystore/>)
> > > > > >> > > >> > > > )
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > and KeyManagerFactory.init(KeyStore, char[]) (
> > > > > >> > > >> > > >
> > > > > >> > > >> > > >
> > > > > >> > > >> > >
> > > > > >> > > >> >
> > > > > >> > > >>
> > > > > >> > >
> > > > > >>
> > > > >
> > > >
> > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory
> > > > > >> > > >> > > > .
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > html#init(java.security.KeyStore <
> > > > > >> http://java.security.keystore/>
> > > > > >> > > >> > > <http://
> > > > > >> > > >> > > > java.security.keystore/
> > > > > >> > > >> > > > ,%20char[])
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > ).
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > 4. The above APIs are intended to support providing
> > > > > >> "trust/key
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > material"
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > from the user without having to write their own
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > TrustManager/KeyManagers.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > To quote from the TrustManagerFactory.init()'s
> > > > > documentation
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > "Initializes
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > this factory with a source of certificate
> > authorities and
> > > > > >> related
> > > > > >> > > >> trust
> > > > > >> > > >> > > > material."
> > > > > >> > > >> > > > To quote from the KeyManagerFactory.init()'s
> > > > documentation
> > > > > >> > > >> "Initializes
> > > > > >> > > >> > > > this factory with a source of key material."
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Based on this it is clear that there is a flexibility
> > > > > >> provided by
> > > > > >> > > >> Java
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > to
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > to enable developers to provide the required
> > trust/key
> > > > > >> material
> > > > > >> > > >> loaded
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > from
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > "anywhere" without requiring them to write custom
> > > > provider
> > > > > OR
> > > > > >> > > >> trust/key
> > > > > >> > > >> > > > managers. This same flexibility is reflected in Kafka
> > > > code
> > > > > >> also
> > > > > >> > > >> where
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > it
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > loads the trust/keys from a local file and doesn't
> > > > require
> > > > > >> > > writing a
> > > > > >> > > >> > > > Provider necessarily. If we do NOT have a custom
> > > > algorithm,
> > > > > >> it
> > > > > >> > > makes
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > less
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > sense to write a Provider.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Thanks
> > > > > >> > > >> > > > Maulin
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > maulin.vasavada@gmail.com>
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > wrote:
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Hi Harsha/Colin
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > I did the sample with a custom Provider for
> > > > > >> TrustStoreManager and
> > > > > >> > > >> tried
> > > > > >> > > >> > > > using ssl.provider Kafka config AND the way KIP-492
> > is
> > > > > >> suggesting
> > > > > >> > > >> (by
> > > > > >> > > >> > > > adding Provider programmatically instead of relying
> > on
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > ssl.provider+java.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > security. The below sample is followed by my detailed
> > > > > >> findings.
> > > > > >> > > I'll
> > > > > >> > > >> > > > appreciate if you can go through it carefully and see
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > if you
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > see my point.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > package providertest;
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > import java.security.Provider <
> > > > > >> http://java.security.provider/>
> > > > > >> > > >> > <http://
> > > > > >> > > >> > > > java.security.provider/>
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > <http://
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > java.security.provider/>;
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > public class MyProvider extends Provider {
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > private static final String name = "MyProvider";
> > private
> > > > > >> static
> > > > > >> > > >> double
> > > > > >> > > >> > > > version = 1.0d;
> > > > > >> > > >> > > > private static String info = "Maulin's SSL Provider
> > > > > >> v"+version;
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > public MyProvider() {
> > > > > >> > > >> > > > super(name, version, info);
> > > > > >> > > >> > > > this.put("TrustManagerFactory.PKIX",
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > "providertest.MyTrustManagerFactory");
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > }
> > > > > >> > > >> > > > }
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > *Details:*
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > KIP-492 documents that it will use
> > Security.addProvider()
> > > > > >> assuming
> > > > > >> > > >> it
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > will
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > add it as position '0' which is not a correct
> > assumption.
> > > > > The
> > > > > >> > > >> > > > addProvider()'s documentation says it will add it to
> > the
> > > > > last
> > > > > >> > > >> available
> > > > > >> > > >> > > > position. You may want to correct that to say
> > > > > >> > > >> > > > Security.insertProviderAt(provider, 1).
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Now coming back to our specific discussion,
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > 1. SPIFFE example uses Custom Algorithm - spiffe.
> > Hence
> > > > > when
> > > > > >> you
> > > > > >> > > add
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > that
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > provider in the provider list via
> > Security.addProvider()
> > > > > the
> > > > > >> > > >> position
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > where
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > it gets added doesn't matter (even if you don't end
> > up
> > > > > >> adding it
> > > > > >> > > as
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > first
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > entry) since that is the ONLY provider for SPIFFE
> > > > specific
> > > > > >> > > algorithm
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > you
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > might have.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > We do *not* have custom algorithm for Key/Trust
> > > > > StoreMangers.
> > > > > >> > > Which
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > means
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > we have to use X509, PKIX etc "Standard Algorithms"
> > ((
> > > > > >> > > >> > > >
> > > > > >> > > >> > > >
> > > > > >> https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> > > > > >> > > >> > > > StandardNames.html
> > > > > >> > > >> > > > ))
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > in our provider to override the TrustStoreManager
> > (see my
> > > > > >> sample
> > > > > >> > > >> code)
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > and
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > KeyStoreManger and KeyManager. This creates another
> > > > > challenge
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > mentioned in
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > the below point.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > 2. In order to make our Provider for loading custom
> > > > > >> TrustStore
> > > > > >> > > >> work, we
> > > > > >> > > >> > > > have to add the provider as 'first' in the list since
> > > > there
> > > > > >> are
> > > > > >> > > >> others
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > with
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > the same algorithm.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > However, the programatic way of adding provider
> > > > > >> > > >> > > > (Security.insertProviderAt()) is *not* deterministic
> > for
> > > > > >> ordering
> > > > > >> > > >> since
> > > > > >> > > >> > > > different code can call the method for a different
> > > > provider
> > > > > >> and
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > depending
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > upon the order of the call our provider can be first
> > or
> > > > > >> pushed
> > > > > >> > > down
> > > > > >> > > >> the
> > > > > >> > > >> > > > list. This can happen very well in any client
> > application
> > > > > >> using
> > > > > >> > > >> Kafka.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > This
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > is specially problematic for a case when you want to
> > > > > >> guarantee
> > > > > >> > > order
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > for a
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Provider having "Standard Algorithms".
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > If we add our provider in java.security file that
> > > > > definitely
> > > > > >> > > >> guarantees
> > > > > >> > > >> > > > the order(unless somebody calls removeProvider()
> > which is
> > > > > >> less
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > likely). But
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > if we add our provider in java.security file it will
> > > > defeat
> > > > > >> the
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > purpose of
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > the KIP-492.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > In the gist - Apache Kafka must not rely on
> > > > > >> "un-deterministic"
> > > > > >> > > >> method
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > to
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > rely on Provider ordering.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > 3. If we just use existing ssl.provider kafka
> > > > configuration
> > > > > >> then
> > > > > >> > > our
> > > > > >> > > >> > > > provider will be used in
> > SSLContext.getInstance(protocol,
> > > > > >> > > provider)
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > call in
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > SslFactory.java <http://sslfactory.java/> <
> > > > > >> > > http://sslfactory.java/>
> > > > > >> > > >> <
> > > > > >> > > >> > > > http://sslfactory.java/>
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > and
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > if our provider does not have implementation for
> > > > > >> > > >> > > > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > (we
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > tested it). Example:
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > In
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > MyProvider sample above you see that I didn't add
> > > > > >> SSLContext.TLSv1
> > > > > >> > > >> as
> > > > > >> > > >> > > > "Service+Algorithm" and that didn't work for me. In
> > > > SPIFFE
> > > > > >> > > provider
> > > > > >> > > >> you
> > > > > >> > > >> > > > don't have this challenge since you are planning to
> > > > bypass
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > ssl.provider as
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > you mention in the KIP-492.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > *Overall summary,*
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > 1. Any provider based mechanisms- a) existing
> > > > ssl.provider
> > > > > >> and
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > b)KIP-492,
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > for loading key/trust store using "Standard
> > Algorithms"
> > > > do
> > > > > >> not
> > > > > >> > > work
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > 2. Approach suggested in our KIP-486 works without
> > any
> > > > > issue
> > > > > >> and
> > > > > >> > > it
> > > > > >> > > >> is
> > > > > >> > > >> > > > *not* our context specific solve
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > 3. Based on above we feel KIP-492 and KIP-486 are
> > > > > >> complimentary
> > > > > >> > > >> changes
> > > > > >> > > >> > > > and not contradicting or redundent.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > If you want we can do a joint session somehow to walk
> > > > > >> through the
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > sample I
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > have and various experiments I did. I would
> > encourage you
> > > > > to
> > > > > >> do
> > > > > >> > > >> similar
> > > > > >> > > >> > > > exercise by writing a Provider for "Standard
> > Algorithm"
> > > > for
> > > > > >> > > >> > > > TrustStoreManager (like our needs) and see what you
> > find
> > > > > >> since
> > > > > >> > > only
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > writing
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > samples can bring out the complexity/challenges we
> > face.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Thanks
> > > > > >> > > >> > > > Maulin
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > maulin.vasavada@gmail.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > com> wrote:
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Just to update - still working on it. Get to work
> > only on
> > > > > >> and off
> > > > > >> > > on
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > it :(
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > maulin.vasavada@gmail.com>
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > wrote:
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Hi Harsha
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Let me try to write samples and will let you know.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Thanks
> > > > > >> > > >> > > > Maulin
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <
> > > > > >> harsha.ch@gmail.com>
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > wrote:
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Hi Maulin,
> > > > > >> > > >> > > > With java security providers can be as custom you
> > would
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > like
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > it to
> > > > > >> > > >> > > > be. If you only want to to implement a custom way of
> > > > > loading
> > > > > >> the
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > keystore
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > and truststore and not implement any
> > protocol/encryption
> > > > > >> handling
> > > > > >> > > >> you
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > can
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > leave them empty and no need to implement. Have you
> > > > looked
> > > > > >> into
> > > > > >> > > the
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > links I
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > pasted before?
> > > > > >> > > >> > > >
> > > > > >> > > >> > > >
> > > > > >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > > > >> > > >> > > >
> > > > > >> > > >> > > >
> > > > > >> > > >> > >
> > > > > >> > > >> >
> > > > > >> > > >>
> > > > > >> > >
> > > > > >>
> > > > >
> > > >
> > spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > java
> > > > > >> > > >> > > >
> > > > > >> > > >> > > >
> > > > > >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > > > >> > > >> > > >
> > > > spiffe-security-provider/src/main/java/spiffe/api/provider/
> > > > > >> > > >> > > > SpiffeTrustManager.java <
> > http://spiffetrustmanager.java/
> > > > >
> > > > > >> > > <http://
> > > > > >> > > >> > > > spiffetrustmanager.java/>
> > > > > >> > > >> > > >
> > > > > >> > > >> > > >
> > > > > >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > > > >> > > >> > > >
> > > > > >> > > >> > > >
> > > > > >> > > >> > >
> > > > > >> > > >> >
> > > > > >> > > >>
> > > > > >> > >
> > > > > >>
> > > > >
> > > >
> > spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > java
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Can you please tell me which methods are too complex
> > in
> > > > > >> above to
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > implement
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > or unnecessary? You are changing anything in SSL/TLS
> > > > > >> > > implementations
> > > > > >> > > >> > > > provided by
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > All of the implementations delegating the checks to
> > the
> > > > > >> default
> > > > > >> > > >> > > > implementation anyway.
> > > > > >> > > >> > > > Spire agent is an example, its nothing but a GRPC
> > server
> > > > > >> listening
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > on a
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > unix domain socket . Above code is making a RPC call
> > to
> > > > the
> > > > > >> local
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > daemon
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > to
> > > > > >> > > >> > > > get the certificate and keys. The mechanics are
> > pretty
> > > > much
> > > > > >> same
> > > > > >> > > as
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > what
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > you are asking for.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Thanks,
> > > > > >> > > >> > > > Harsha
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > maulin.vasavada@gmail.com>
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > wrote:
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Imagine a scenario like - We know we have a custom
> > KMS
> > > > and
> > > > > >> as a
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Kafka
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > owner
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > we want to comply to using that KMS source to load
> > > > > >> keys/certs. As
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > a
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Kafka
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > owner we know how to integrate with KMS but doesn't
> > > > > >> necessarily
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > have
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > to
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > know anything about cipher suites, algorithms, and
> > > > SSL/TLS
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > implementation.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Going the Provider way requires to know lot more
> > than we
> > > > > >> should,
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > isn't it?
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Not that we would have concern/shy-away knowing those
> > > > > >> details -
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > but
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > if we
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > don't have to - why should we?
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > maulin.vasavada@gmail.com>
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > wrote:
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Hi Harsha
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > We don't have spire (or similar) agents and we do not
> > > > have
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > keys/certs
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > locally on any brokers. To elaborate more on my
> > previous
> > > > > >> email,
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > I agree that Java security Providers are used in much
> > > > > broader
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > sense
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > - to
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > have a particular implementation of an algorithm, use
> > > > > >> specific
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > cipher
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > suites for SSL , OR in our current team's case have a
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > particular
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > way to
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > leverage pre-generated SSL sessions. However, the
> > scope
> > > > of
> > > > > >> our
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > KIP
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > (486)
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > is
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > much restricted than that. We merely intend to
> > provide a
> > > > > >> custom
> > > > > >> > > >> > > > keystore/truststore for our SSL connections and not
> > > > really
> > > > > >> worry
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > about
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > underlying specific SSL/TLS implementation. This
> > > > simplifies
> > > > > >> it
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > a
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > lot for
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > us to keep the concerns separate and clear.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > I feel our approach is more complimentary such that
> > it
> > > > > allows
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > for
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > using
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > keystores of choice while retaining the flexibility
> > to
> > > > use
> > > > > >> any
> > > > > >> > > >> > > > underlying/available Provider for actually making
> > the SSL
> > > > > >> call.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > We agree with KIP-492's approach based on Providers
> > (and
> > > > > >> Java's
> > > > > >> > > >> > > > recommendation), but also strongly believe that our
> > > > > approach
> > > > > >> can
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > compliment
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > it very effectively for reasons explained above.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Thanks
> > > > > >> > > >> > > > Maulin
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > kafka@harsha.io
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > wrote:
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Hi Maulin,
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > maulin.vasavada@gmail.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > com>
> > > > > >> > > >> > > > wrote:
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Hi Harsha
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > The reason we rejected the SslProvider route is that
> > - we
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > only
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > needed
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > a
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > custom way to load keys/certs. Not touch any policy
> > that
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > existing
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Providers
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > govern like SunJSSE Provider.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > We have exactly the same requirements to load certs
> > and
> > > > > keys
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > through
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > spire
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > agent. We used security.provider to do that exactly.
> > I am
> > > > > not
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > sure
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > why
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > you
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > would be modifying any policies provided by default
> > > > SunJSSE
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > provider.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Can
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > you give me an example of having custom provider that
> > > > will
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > override an
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > existing policy in SunJSSE provider.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > As pointed out earlier, this kip
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > >> > > >> > > >
> > > > > >> KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > allows
> > > > > >> > > >> > > > you to load security.provider through config.
> > > > > >> > > >> > > > Take a look at the examples I gave before
> > > > > >> > > >> > > >
> > > > > >> > > >> > > >
> > > > > >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > > > >> > > >> > > >
> > > > > >> > > >> > > >
> > > > > >> > > >> > >
> > > > > >> > > >> >
> > > > > >> > > >>
> > > > > >> > >
> > > > > >>
> > > > >
> > > >
> > spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > java
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > It registers KeyManagerFactory and
> > TrustManagerFactory
> > > > and
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Keystore
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > algorithm.
> > > > > >> > > >> > > > Implement your custom way of loading Keystore in here
> > > > > >> > > >> > > >
> > > > > >> > > >> > > >
> > > > > >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > > > >> > > >> > > >
> > > > > >> > > >> > > >
> > > > > >> > > >> > >
> > > > > >> > > >> >
> > > > > >> > > >>
> > > > > >> > >
> > > > > >>
> > > > >
> > > >
> > spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > java
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > and Trust manager like here
> > > > > >> > > >> > > >
> > > > > >> > > >> > > >
> > > > > >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > > > >> > > >> > > >
> > > > spiffe-security-provider/src/main/java/spiffe/api/provider/
> > > > > >> > > >> > > > SpiffeTrustManager.java <
> > http://spiffetrustmanager.java/
> > > > >
> > > > > >> > > <http://
> > > > > >> > > >> > > > spiffetrustmanager.java/>
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > In your Kafka client you can set the
> > security.provider to
> > > > > >> your
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > custom
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > implementation and with this fix
> > > > > >> > > >> > > > https://issues.apache.org/jira/browse/KAFKA-8191
> > you can
> > > > > set
> > > > > >> > > >> > > > keyManagerAlgorigthm and trustManagerAlgorithm
> > configs.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > All of this is in your clients and broker side and
> > do not
> > > > > >> need
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > to
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > touch
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > any
> > > > > >> > > >> > > > policy changes at JVM level. You'll register the
> > > > providers
> > > > > in
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > the
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > priority
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > order and can still have SunJSSE provider and have
> > your
> > > > > >> custom
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > provider
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > to
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > implement the key and trust managers.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > The ask here is different than KIP-492. We don't
> > have any
> > > > > >> need
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > to
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > modify/specify the algorithm parameter. Does that
> > make
> > > > > sense?
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > The ask in KIP is introducing new interfaces where
> > the
> > > > > KIP's
> > > > > >> > > >> > > > goal/motivation can be achieved through the
> > > > > security.provider
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > and
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > we
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > worked
> > > > > >> > > >> > > > on similar goal without touching any Keystore or
> > > > Truststore
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > interfaces.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > My advise is against changing or introducing new
> > > > interfaces
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > when
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > it can
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > work through security.provider.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Thanks,
> > > > > >> > > >> > > > Harsha
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Thanks
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Maulin
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > kafka@harsha.io>
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > wrote:
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > In your KIP you added security. provider as rejected
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > alternative
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > and
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > specified "its not the correct way". Do you mind
> > > > explaining
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > why
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > its
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > not? I
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > didn't find any evidence in Java docs to say so.
> > Contrary
> > > > > to
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > your
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > statement
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > it does say in the java docs
> > > > > >> > > >> > > > " However, please note that a provider can be used to
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > implement
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > any
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > security service in Java that uses a pluggable
> > > > architecture
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > with
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > a
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > choice
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > of implementations that fit underneath."
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Java Security Providers have been used by other
> > projects
> > > > to
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > provide
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > such
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > integration . I am not sure if you looked into Spiffe
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > project to
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > efficiently distribute certificates but here is an
> > > > example
> > > > > of
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Java
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > provider
> > > > > >> > > >> > > >
> > > > > >> > > >> > > >
> > > > > >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > > > >> > > >> > > >
> > > > > >> > > >> > > >
> > > > > >> > > >> > >
> > > > > >> > > >> >
> > > > > >> > > >>
> > > > > >> > >
> > > > > >>
> > > > >
> > > >
> > spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > java which
> > > > > >> > > >> > > > obtains certificates from local daemons.
> > > > > >> > > >> > > > These integrations are being used in Tomcat, Jetty
> > etc..
> > > > We
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > are
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > also
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > using
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Security provider to do the same in our Kafka
> > clusters.
> > > > So
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > unless I
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > see
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > more evidence why security.provider doesn't work for
> > you
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > adding
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > new
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > interfaces while there exists more cleaner way of
> > > > achieving
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > the
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > goals
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > of
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > this KIP is unnecessary and breaks the well known
> > > > security
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > interfaces
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > provided by Java itself.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Thanks,
> > > > > >> > > >> > > > Harsha
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani
> > <
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > kafka@harsha.io
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > wrote:
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Hi Maulin,
> > > > > >> > > >> > > > Not sure if you looked at my previous replies. This
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > changes
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > are not required as there is already security
> > Provider to
> > > > > do
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > what you
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > are
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > proposing. This KIP
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > >> > > >> > > >
> > > > > >> > > >> > > >
> > > > > >> KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > also
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > addresses easy registration of such providers.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Thanks,
> > > > > >> > > >> > > > Harsha
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > <maulin.vasavada@gmail.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > com> wrote:
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Bump! Can somebody please review this?
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > maulin.vasavada@gmail.com>
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > wrote:
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Bump! Can somebody please review this?
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > --
> > > > > >> > > >> > > > Thanks,
> > > > > >> > > >> > > > M.Sai Sandeep
> > > > > >> > > >> > > >
> > > > > >> > > >> > > >
> > > > > >> > > >> > >
> > > > > >> > > >> >
> > > > > >> > > >>
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Maulin Vasavada <ma...@gmail.com>.
Hi all
Please check
https://github.com/maulin-vasavada/kafka/commit/44f86395b1ba3fe4bd87de89029d72da77995ff8
This is just the first cut obviously. There are few call outs I would like
to make,
1. So far I kept the old SslEngineBuilder hence I had to name the interface
with "I" (that can change later)
2. I did not yet add the creation of SslEngineBuilder via loading the
configuration like 'ssl.engine.builder.class'. Hence you see direct
creation of DefaultSslEngineBuilder class
3. Due to validation logic in the current SslFactory I had to add more
methods in ISslEngineBuilder interface (like keystore(), truststore() etc).
Due to other classes like EchoServer depending upon needing SSLContext, I
had to add getSSLContext() also in the interface.
4. With these changes and with existing old SslEngineBuilder, the
clients/core projects builds with tests successfully but I didn't add any
additional tests yet
5. I wanted to have DefaultSslEngineBuilder in such a way that if somebody
wants to implement custom SslEngineBuilder they can extend and override
only key required methods without repeating any logic.
6. For reconfigurable interface I kept the way suggested by Rajini -
meaning SslFactory really is reconfigurable BUT it relies on the
ISslEngineBuilder to define the reconfigurable options. This means that
ISslEngineBuilder dictates based on which reconfigurable params the
SslFactory should try to reconfigure the SSLEngine.
With this - open to all the suggestions and further improvements.
Thanks
Maulin
On Tue, Sep 3, 2019 at 10:07 AM Colin McCabe <cm...@apache.org> wrote:
> On Mon, Sep 2, 2019, at 03:33, Rajini Sivaram wrote:
> > I would expect SslEngineBuilder interface to look something like this,
> > perhaps with some tweaking:
> >
> > public interface SslEngineBuilder extends Configurable, Closeable {
> >
> > Set<String> reconfigurableConfigs();
> >
> > boolean shouldBeRebuilt(Map<String, Object> nextConfigs);
> >
> > SSLEngine createSslEngine(Mode mode, String peerHost, int
> > peerPort, String endpointIdentification);
> >
> > }
> >
> > The existing SslEngineBuilder class would be renamed and will implement
> > this interface. Loading of keystore/truststore will be in
> SslEngineBuilder
> > as it is now. The method `shouldBeRebuilt()` will validate configs
> during
> > reconfiguration and decide if reconfiguration is required because
> keystore
> > or truststore changed. SslFactory.reconfigurableConfigs() will return
> > SslEngineBuilder.reconfigurableConfigs() as well including any custom
> > configs of SslEngineBuilder, so no other changes will be required when we
> > eventually support custom SSL configs.
> >
> > We don't want to make SslFactory the pluggable class since that contains
> > validation logic for SSL engines. Everything that we want to customise is
> > contained in SslEngineBuilder. Basically custom SslEngineBuilder will
> > validate custom configs during reconfiguration and create SSLEngine.
> > SslFactory will continue to perform validation of SSLEngines and this
> will
> > not be customizable. SslEngineBuilder will not be reconfigurable, instead
> > we create a new builder as we do now to avoid having to deal with
> > thread-safety and atomicity of updates. We could consider using a public
> > Reconfigurable interface as the pluggable interface for consistency, but
> I
> > think we would still want to create a new Builder on reconfiguration and
> > retain non-pluggable SSL engine validation in SslFactory.
>
> +1
>
> C.
>
> >
> >
> > On Fri, Aug 30, 2019 at 10:21 PM Maulin Vasavada <
> maulin.vasavada@gmail.com>
> > wrote:
> >
> > > Looking at SslFactory and SslEngineBuilder I feel the responsibilities
> are
> > > not clear. Both has public method for createSSLEngine for example. I
> feel
> > > the SslEngineBuilder was created to move out lot of code but it is not
> > > necessarily a public class (e.g. I don't think anybody calling
> > > SslEngineBuilder separately without SslFactory in between). I am
> currently
> > > inclined toward what Celement is suggesting - having pluggable
> SslFactory.
> > >
> > > Let me do this - let me refactor SslFactory and SslEngineBuilder and
> review
> > > what I can come up with you guys. Let us see if we can address all the
> > > objections raised previously for KIP-383's iterations. I'll need
> sometime
> > > though. Let me try to do it by next of next week.
> > >
> > > Thanks
> > > Maulin
> > >
> > > On Fri, Aug 30, 2019 at 12:25 PM Pellerin, Clement <
> > > Clement_Pellerin@ibi.com>
> > > wrote:
> > >
> > > > What is your solution to the objection that killed the second
> iteration
> > > of
> > > > KIP-383?
> > > > Mainly, how do you support validation of reconfiguration requests
> that
> > > > involve new custom properties implemented by the pluggable factory?
> > > > Custom properties do not exist yet, but that is very legitimate
> thing to
> > > > design for the future.
> > > >
> > > > That's why I favor a pluggable SslFactory instead of an
> SslEngineBuilder
> > > > factory.
> > > >
> > > > -----Original Message-----
> > > > From: Maulin Vasavada [mailto:maulin.vasavada@gmail.com]
> > > > Sent: Friday, August 30, 2019 3:07 PM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and
> > > > TrustStore
> > > >
> > > > +1 for making SslEngineBuilder configurable upon more thoughts.
> > > >
> > > > However, in the abstraction and default implementation we should make
> > > sure
> > > > when we do have a requirement to plugin custom key/trust store people
> > > don't
> > > > have to write lot more code which may not be related to it.
> > > >
> > > > Having said that, does this mean, we resurrect KIP-383 and update it
> with
> > > > latest context and go from there?
> > > >
> > > > We are willing to take up that work for making it configurable.
> > > >
> > > > Thanks
> > > > Maulin
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, Aug 30, 2019 at 10:34 AM Maulin Vasavada <
> > > > maulin.vasavada@gmail.com>
> > > > wrote:
> > > >
> > > > > Why don't we make SSLEngineBuilder code delegate the whole
> Key/Trust
> > > > store
> > > > > initialization to the interfaces we are proposing? Default
> > > implementation
> > > > > for those key/trust store loader interfaces will be "file based"
> > > loading
> > > > vs
> > > > > if somebody wants to customize any of it they can.
> > > > >
> > > > > Would that make sense?
> > > > >
> > > > > On Fri, Aug 30, 2019 at 10:03 AM Colin McCabe <cm...@apache.org>
> > > > wrote:
> > > > >
> > > > >> +1 for making SslEngineBuilder configurable. This would give
> > > > >> implementers a lot more flexibility-- to use key distribution
> methods
> > > > that
> > > > >> were not files, for example.
> > > > >>
> > > > >> best,
> > > > >> Colin
> > > > >>
> > > > >>
> > > > >> On Fri, Aug 30, 2019, at 02:03, Rajini Sivaram wrote:
> > > > >> > Just to make sure we are on the same page - KIP-383 was written
> > > before
> > > > >> > the
> > > > >> > code was refactored. The refactoring addressed some of the
> concerns
> > > of
> > > > >> > KIP-383. My suggestion was to make SslEngineBuilder
> configurable.
> > > The
> > > > >> > default implementation of this pluggable class would be
> > > > >> >
> > > > >>
> > > >
> > >
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/ssl/SslEngineBuilder.java
> > > > >> .
> > > > >> > That should give an idea of the size of the configurable part
> that a
> > > > >> > custom
> > > > >> > class needs to implement. A large part of that is about loading
> > > > >> > keystore/truststore. I agree it has slightly more code than
> KIP-486
> > > > >> > proposes, but since it lets you customize creation of
> SSLEngine, it
> > > > >> > would
> > > > >> > address every possible scenario.
> > > > >> >
> > > > >> > Thoughts?
> > > > >> >
> > > > >> >
> > > > >> > On Fri, Aug 30, 2019 at 2:02 AM Maulin Vasavada <
> > > > >> maulin.vasavada@gmail.com>
> > > > >> > wrote:
> > > > >> >
> > > > >> > > I thought about it more. I feel that no matter how we
> refactor the
> > > > >> code
> > > > >> > > (with or without KIP-383 integrated), ultimately the need of
> > > > >> customizing
> > > > >> > > loading for keys and certs will still remain. Whenever that
> need
> > > > >> arises we
> > > > >> > > might end up thinking about the solution suggested by our
> KIP-486.
> > > > >> Hence
> > > > >> > > regardless of the other KIPs and configurations "if we do
> need to
> > > > >> customize
> > > > >> > > loading of keys/certs, we will need the code changes
> suggested by
> > > > this
> > > > >> > > KIP".
> > > > >> > >
> > > > >> > > Let me know what you guys think.
> > > > >> > >
> > > > >> > > Harsha, we are working on changing the interfaces for
> key/trust
> > > > store
> > > > >> > > loaders with Certificate and PrivateKey objects. Will
> probably be
> > > > >> able to
> > > > >> > > update it later today or tomorrow.
> > > > >> > >
> > > > >> > > Thanks
> > > > >> > > Maulin
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > On Thu, Aug 29, 2019 at 2:30 PM Maulin Vasavada <
> > > > >> maulin.vasavada@gmail.com
> > > > >> > > >
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > On that, I actually looked at KIP-383 before briefly.
> However,
> > > > that
> > > > >> > > > sounded like lot of changes suggested.
> > > > >> > > >
> > > > >> > > > One "key" thing we have to keep in mind is - IF we need lot
> of
> > > > >> > > > customization Kafka already allows you to use your
> SslProvider
> > > via
> > > > >> > > > ssl.providers or the changes done by KIP-492 and
> > > > >> > > > SSLContext.getInstance(protocol, provider) call allows us to
> > > > return
> > > > >> the
> > > > >> > > > SSLContext with "ALL" the details we would like to
> customize.
> > > > Hence
> > > > >> I am
> > > > >> > > > not sure that customization suggested by KIP-383 would be
> worth
> > > > the
> > > > >> > > effort.
> > > > >> > > > We also have similar SSLContext customization outside of
> Kafka.
> > > > >> > > >
> > > > >> > > > Thanks
> > > > >> > > > Maulin
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > On Thu, Aug 29, 2019 at 12:47 PM Pellerin, Clement <
> > > > >> > > > Clement_Pellerin@ibi.com> wrote:
> > > > >> > > >
> > > > >> > > >> KIP-383 in its present form was vetoed because it was not
> > > > possible
> > > > >> to
> > > > >> > > add
> > > > >> > > >> validation of custom properties in a future KIP. The
> solution
> > > to
> > > > >> that is
> > > > >> > > >> the first proposal I wrote for KIP-383 which made the whole
> > > > >> SslFactory
> > > > >> > > >> pluggable. That first solution was also vetoed hence the
> > > > deadlock.
> > > > >> > > >>
> > > > >> > > >> Replacing the whole factory was a much nicer solution. It
> was
> > > > >> vetoed
> > > > >> > > >> because doing this almost invariably meant the replacement
> lost
> > > > >> all the
> > > > >> > > >> complex validation code in the default SslFactory.
> > > > >> > > >>
> > > > >> > > >> My current idea is to extract the validation code into
> another
> > > > >> public
> > > > >> > > API
> > > > >> > > >> that SslFactory would call. I did not look at the newly
> > > > refactored
> > > > >> code
> > > > >> > > and
> > > > >> > > >> I did not study how to do this yet. KIP-383 was not
> popular at
> > > > the
> > > > >> time
> > > > >> > > and
> > > > >> > > >> designing a new solution is a lot of work.
> > > > >> > > >>
> > > > >> > > >> Is there interest from 3 binding voters for something like
> > > this?
> > > > >> > > >>
> > > > >> > > >> -----Original Message-----
> > > > >> > > >> From: Rajini Sivaram [mailto:rajinisivaram@gmail.com]
> > > > >> > > >> Sent: Thursday, August 29, 2019 2:57 PM
> > > > >> > > >> To: dev
> > > > >> > > >> Subject: Re: [DISCUSS] KIP-486 Support for pluggable
> KeyStore
> > > and
> > > > >> > > >> TrustStore
> > > > >> > > >>
> > > > >> > > >> Hi Maulin,
> > > > >> > > >>
> > > > >> > > >> In SSL scenarios, I imagine security providers introduced
> by
> > > > >> KIP-492 are
> > > > >> > > >> likely to be most useful when you want to use third party
> > > > >> providers. The
> > > > >> > > >> biggest advantage of the config from that KIP is that you
> don't
> > > > >> need to
> > > > >> > > >> write much code to integrate existing security providers
> into
> > > > Kafka
> > > > >> > > >> brokers
> > > > >> > > >> or clients. As I understand it, KIP-486 is a more
> convenient
> > > > >> option for
> > > > >> > > >> the
> > > > >> > > >> specific problem of loading keystores/truststores
> differently.
> > > It
> > > > >> can be
> > > > >> > > >> achieved in theory with KIP-492, but KIP-486 is a much
> simpler
> > > > >> option
> > > > >> > > for
> > > > >> > > >> this case.
> > > > >> > > >>
> > > > >> > > >> My concern about KIP-486 is that it introduces yet another
> > > > >> interface
> > > > >> > > into
> > > > >> > > >> our already complex security code, while only solving one
> > > > >> particular use
> > > > >> > > >> case. Have you looked at
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > >
> > > > >>
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-383%3A++Pluggable+interface+for+SSL+Factory
> > > > >> > > >> ?
> > > > >> > > >> The goal was to make
> > > > >> > > >> org.apache.kafka.common.security.ssl.SslEngineBuilder
> > > pluggable.
> > > > >> > > >> The code has already been refactored by Colin after that
> KIP
> > > was
> > > > >> > > written,
> > > > >> > > >> making it easier to implement KIP-383. This should enable
> you
> > > to
> > > > >> load
> > > > >> > > your
> > > > >> > > >> keystores and truststores differently. Using a pluggable
> > > > >> > > SslEngineBuilder
> > > > >> > > >> will also solve several other use cases at the same time.
> > > KIP-383
> > > > >> hasn't
> > > > >> > > >> been voted through yet, but perhaps you could take a look
> and
> > > we
> > > > >> could
> > > > >> > > >> revive that instead if it solves your use case as well?
> > > > >> > > >>
> > > > >> > > >> Regards,
> > > > >> > > >>
> > > > >> > > >> Rajini
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >> On Thu, Aug 29, 2019 at 6:42 PM Maulin Vasavada <
> > > > >> > > >> maulin.vasavada@gmail.com>
> > > > >> > > >> wrote:
> > > > >> > > >>
> > > > >> > > >> > Hi Harsha
> > > > >> > > >> >
> > > > >> > > >> > Thank you. Appreciate your time and support on this. Let
> me
> > > go
> > > > >> back do
> > > > >> > > >> some
> > > > >> > > >> > more research and get back to you on the KeyStore
> interface
> > > > part.
> > > > >> > > >> > Basically, if we return certs and keys in the interface
> then
> > > > >> Kafka
> > > > >> > > code
> > > > >> > > >> > will have to build KeyStore object - which is also
> > > reasonable.
> > > > >> > > >> >
> > > > >> > > >> > Thanks
> > > > >> > > >> > Maulin
> > > > >> > > >> >
> > > > >> > > >> > On Thu, Aug 29, 2019 at 10:01 AM Harsha Chintalapani <
> > > > >> kafka@harsha.io
> > > > >> > > >
> > > > >> > > >> > wrote:
> > > > >> > > >> >
> > > > >> > > >> > > Hi Maulin,
> > > > >> > > >> > > Use cases are clear now. I am +1
> for
> > > > moving
> > > > >> > > >> forward
> > > > >> > > >> > > with the discussions on having such configurable
> option for
> > > > >> users.
> > > > >> > > But
> > > > >> > > >> > the
> > > > >> > > >> > > interfaces is proposed doesn't look right to me. We are
> > > still
> > > > >> > > talking
> > > > >> > > >> > about
> > > > >> > > >> > > keystore interfaces. Given keystore's are used as
> > > filebased
> > > > >> way of
> > > > >> > > >> > > transporting certificates I am not sure it will help
> the
> > > rest
> > > > >> of the
> > > > >> > > >> > > user-base.
> > > > >> > > >> > > In short, I am +1 on the KIP's
> motivation
> > > > >> and only
> > > > >> > > >> have
> > > > >> > > >> > > questions around returning keystores instead of
> returning
> > > > >> certs,
> > > > >> > > >> private
> > > > >> > > >> > > keys etc. . If others in the community are ok with such
> > > > >> interface we
> > > > >> > > >> can
> > > > >> > > >> > > move forward.
> > > > >> > > >> > >
> > > > >> > > >> > > Thanks,
> > > > >> > > >> > > Harsha
> > > > >> > > >> > >
> > > > >> > > >> > >
> > > > >> > > >> > > On Wed, Aug 28, 2019 at 1:51 PM, Maulin Vasavada <
> > > > >> > > >> > > maulin.vasavada@gmail.com>
> > > > >> > > >> > > wrote:
> > > > >> > > >> > >
> > > > >> > > >> > > > Hi Harsha
> > > > >> > > >> > > >
> > > > >> > > >> > > > As we synced-up offline on this topic, we hope you
> don't
> > > > >> have any
> > > > >> > > >> more
> > > > >> > > >> > > > clarifications that you are seeking. If that is the
> case,
> > > > >> can you
> > > > >> > > >> > please
> > > > >> > > >> > > > help us move this forward and discuss what changes
> you
> > > > would
> > > > >> > > expect
> > > > >> > > >> on
> > > > >> > > >> > > the
> > > > >> > > >> > > > KIP design in order to make it valuable contribution?
> > > > >> > > >> > > >
> > > > >> > > >> > > > Just FYI - we verified our primary design change
> with the
> > > > >> author
> > > > >> > > of
> > > > >> > > >> > Sun's
> > > > >> > > >> > > > X509 Trustmanager implementation and the outcome is
> that
> > > > >> what we
> > > > >> > > are
> > > > >> > > >> > > > proposing makes sense at the heart of it - "Instead
> of
> > > > >> writing
> > > > >> > > >> > > TrustManager
> > > > >> > > >> > > > just plugin the Trust store". We are open to discuss
> > > > >> additional
> > > > >> > > >> changes
> > > > >> > > >> > > > that you/anybody else would like to see on the
> > > > functionality
> > > > >> > > >> however.
> > > > >> > > >> > > >
> > > > >> > > >> > > > Thanks
> > > > >> > > >> > > > Maulin
> > > > >> > > >> > > >
> > > > >> > > >> > > > On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada <
> > > > >> > > >> > > maulin.vasavada@gmail.com>
> > > > >> > > >> > > > wrote:
> > > > >> > > >> > > >
> > > > >> > > >> > > > Hi Harsha
> > > > >> > > >> > > >
> > > > >> > > >> > > > Any response on my question? I feel this KIP is worth
> > > > >> > > accommodating.
> > > > >> > > >> > Your
> > > > >> > > >> > > > help is much appreciated.
> > > > >> > > >> > > >
> > > > >> > > >> > > > Thanks
> > > > >> > > >> > > > Maulin
> > > > >> > > >> > > >
> > > > >> > > >> > > > On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada <
> > > > >> > > >> > maulin.vasavada@gmail.
> > > > >> > > >> > > > com> wrote:
> > > > >> > > >> > > >
> > > > >> > > >> > > > Hi Harsha
> > > > >> > > >> > > >
> > > > >> > > >> > > > I've examined the SPIFFE provider more and have one
> > > > question
> > > > >> -
> > > > >> > > >> > > >
> > > > >> > > >> > > > If SPIFFE didn't have a need to do checkSpiffeId()
> call
> > > at
> > > > >> the
> > > > >> > > below
> > > > >> > > >> > > > location, would you really still write the Provider?
> *OR*
> > > > >> Would
> > > > >> > > you
> > > > >> > > >> > just
> > > > >> > > >> > > > use TrustManagerFactory.init(KeyStore) signature to
> pass
> > > > the
> > > > >> > > >> KeyStore
> > > > >> > > >> > > from
> > > > >> > > >> > > > set of certs returned by spiffeIdManager.
> > > > getTrustedCerts()?
> > > > >> > > >> > > >
> > > > >> > > >> > > >
> > > > >> > > >> >
> > > > >> > >
> > > > >>
> > >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > > >> > > >> > > > provider/CertificateUtils.java#L100
> > > > >> > > >> > > >
> > > > >> > > >> > > > /**
> > > > >> > > >> > > >
> > > > >> > > >> > > > * Validates that the SPIFFE ID is present and
> matches the
> > > > >> SPIFFE
> > > > >> > > ID
> > > > >> > > >> > > > configured in
> > > > >> > > >> > > > * the java.security property ssl.spiffe.accept
> > > > >> > > >> > > > *
> > > > >> > > >> > > > * If the authorized spiffe ids list is empty any
> spiffe
> > > id
> > > > is
> > > > >> > > >> > authorized
> > > > >> > > >> > > > *
> > > > >> > > >> > > > * @param chain an array of X509Certificate that
> contains
> > > > the
> > > > >> > > Peer's
> > > > >> > > >> > SVID
> > > > >> > > >> > > > to be validated
> > > > >> > > >> > > > * @throws CertificateException when either the
> > > certificates
> > > > >> > > doesn't
> > > > >> > > >> > have
> > > > >> > > >> > > a
> > > > >> > > >> > > > SPIFFE ID or the SPIFFE ID is not authorized
> > > > >> > > >> > > > */
> > > > >> > > >> > > > static void checkSpiffeId(X509Certificate[] chain)
> throws
> > > > >> > > >> > > > CertificateException {
> > > > >> > > >> > > >
> > > > >> > > >> > > > Thanks
> > > > >> > > >> > > > Maulin
> > > > >> > > >> > > >
> > > > >> > > >> > > > On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani <
> > > > >> > > >> kafka@harsha.io>
> > > > >> > > >> > > > wrote:
> > > > >> > > >> > > >
> > > > >> > > >> > > > Maulin,
> > > > >> > > >> > > > The code parts you are pointing are specific for
> Spiffe
> > > and
> > > > >> if
> > > > >> > > >> > > > you are talking about validate method which uses PKIX
> > > check
> > > > >> like
> > > > >> > > any
> > > > >> > > >> > > other
> > > > >> > > >> > > > provider does.
> > > > >> > > >> > > > If you want to default to SunJSSE everywhere you can
> do
> > > so
> > > > by
> > > > >> > > >> > delegating
> > > > >> > > >> > > > the calls in these methods to SunJSSE provider.
> > > > >> > > >> > > >
> > > > >> > > >> > > > TrustManagerFactory tmf = TrustManagerFactory
> > > > >> > > >> > > >
> > > .getInstance(TrustManagerFactory.getDefaultAlgorithm());and
> > > > >> use
> > > > >> > > >> > > > tmf.chekServerTrusted()
> > > > >> > > >> > > > or use
> > > > >> > > >> > > >
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> > > > >> > > >> > > >
> TrustManagerFactory.html#getInstance(java.lang.String)if
> > > > you
> > > > >> want
> > > > >> > > a
> > > > >> > > >> > > > specific provider.
> > > > >> > > >> > > >
> > > > >> > > >> > > > -Harsha
> > > > >> > > >> > > >
> > > > >> > > >> > > > On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada <
> > > > >> > > >> > maulin.vasavada@gmail.
> > > > >> > > >> > > > com>
> > > > >> > > >> > > > wrote:
> > > > >> > > >> > > >
> > > > >> > > >> > > > Okay, so I take that you guys agree that I have to
> write
> > > a
> > > > >> > > 'custom'
> > > > >> > > >> > > > algorithm and a provider to make it work , correct?
> > > > >> > > >> > > >
> > > > >> > > >> > > > Now, for Harsha's comment "Here the 'Custom'
> Algorithm is
> > > > >> not an
> > > > >> > > >> > > > implementation per say , ..." , I diagree. You can
> refer
> > > to
> > > > >> > > https://
> > > > >> > > >> > > >
> > > > >> > > >> > > >
> > > > >> > > >> >
> > > > >> > > >>
> > > > >> > >
> > > > >>
> > > >
> github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/
> > > > >> > > >> > > >
> > > > >> > > >> > > > SpiffeTrustManager.java#L91 <
> > > > >> http://spiffetrustmanager.java/#L91>
> > > > >> > > >> and
> > > > >> > > >> > > >
> > > > >> > > >> > > >
> > > > >> > > >> >
> > > > >> > >
> > > > >>
> > >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > > >> > > >> > > >
> > > > >> > > >> > > > provider/CertificateUtils.java#L100
> > > > >> > > >> > > >
> > > > >> > > >> > > > "that code" is the customization you have for the
> custom
> > > > way
> > > > >> to
> > > > >> > > >> check
> > > > >> > > >> > > > something on top of regular checks. That method is
> NOT
> > > > doing
> > > > >> > > custom
> > > > >> > > >> > > > truststore loading. It is validating/verifying
> something
> > > in
> > > > >> the
> > > > >> > > >> > > >
> > > > >> > > >> > > > "custom"
> > > > >> > > >> > > >
> > > > >> > > >> > > > way with spiffeId.
> > > > >> > > >> > > > I bet that without that you won't have a need of the
> > > custom
> > > > >> > > >> algorithm
> > > > >> > > >> > > >
> > > > >> > > >> > > > in
> > > > >> > > >> > > >
> > > > >> > > >> > > > the first place.
> > > > >> > > >> > > >
> > > > >> > > >> > > > Let me know if you agree to this.
> > > > >> > > >> > > >
> > > > >> > > >> > > > Thanks
> > > > >> > > >> > > > Maulin
> > > > >> > > >> > > >
> > > > >> > > >> > > > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <
> > > > >> mprsai@gmail.com>
> > > > >> > > >> > > >
> > > > >> > > >> > > > wrote:
> > > > >> > > >> > > >
> > > > >> > > >> > > > Hi Maulin, thanks for the discussion. As Harsha
> pointed
> > > > out,
> > > > >> to
> > > > >> > > use
> > > > >> > > >> the
> > > > >> > > >> > > > KIP492, you need to create a new provider, register a
> > > *new*
> > > > >> custom
> > > > >> > > >> > > > algorithm for your keymanager and trustmanager
> factory
> > > > >> > > >> implementations.
> > > > >> > > >> > > > After this, the kafka server configuration can be
> done as
> > > > >> given
> > > > >> > > >> below
> > > > >> > > >> > > >
> > > > >> > > >> > > > # Register the provider class with custom algorithm,
> say
> > > > >> CUSTOM
> > > > >> > > >> > > >
> > > > >> > > >> > > > security.
> > > > >> > > >> > > >
> > > > >> > > >> > > > provider.classes=com.company.security.CustomProvider
> > > > >> > > >> > > > <http://provider.classes
> > > > >> =com.company.security.customprovider/>
> > > > >> > > >> > > > <http://security.provider.classes
> > > > >> > > >> > > >
> > > > >> > > >> > > > =com.company.security.customprovider/>
> > > > >> > > >> > > >
> > > > >> > > >> > > > # Register the keymanager and trustmanager algorithms
> > > > >> > > >> > > > # These algorithms indicate that the Keymanager and
> > > > >> Trustmanagers
> > > > >> > > >> > > > registered under the algorithm "CUSTOM" needs to be
> used
> > > > >> > > >> > > > ssl.trustmanager.algorithm=CUSTOM
> > > > >> > > >> > > > ssl.keymanager.algorithm=CUSTOM
> > > > >> > > >> > > >
> > > > >> > > >> > > > And the customprovider looks like this...
> > > > >> > > >> > > >
> > > > >> > > >> > > > public class CustomProvider extends Provider {
> > > > >> > > >> > > > public CustomProvider() {
> > > > >> > > >> > > > super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore
> and
> > > > >> > > TrustStore");
> > > > >> > > >> > > > super.put("KeyManagerFactory.CUSTOM",
> > > > >> "customKeyManagerFactory");
> > > > >> > > >> > > > super.put("TrustManagerFactory.CUSTOM",
> > > > >> > > >> > > > "customTrustManagerFactory");
> > > > >> > > >> > > > }
> > > > >> > > >> > > > }
> > > > >> > > >> > > >
> > > > >> > > >> > > > The PR for this is in review and can be found here:
> > > > >> > > >> > > >
> > > > >> > > >> > > > https://github.com/
> > > > >> > > >> > > >
> > > > >> > > >> > > > apache/kafka/pull/7090
> > > > >> > > >> > > > This PR includes the fixed insertProviderAt function
> > > call.
> > > > >> > > >> > > >
> > > > >> > > >> > > > On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani <
> > > > >> > > >> kafka@harsha.io>
> > > > >> > > >> > > > wrote:
> > > > >> > > >> > > >
> > > > >> > > >> > > > Answers are in-line
> > > > >> > > >> > > >
> > > > >> > > >> > > > On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada <
> > > > >> > > >> > > >
> > > > >> > > >> > > > maulin.vasavada@gmail.
> > > > >> > > >> > > >
> > > > >> > > >> > > > com
> > > > >> > > >> > > >
> > > > >> > > >> > > > wrote:
> > > > >> > > >> > > >
> > > > >> > > >> > > > Hi Colin
> > > > >> > > >> > > >
> > > > >> > > >> > > > When I refer to "standard" or "custom" algorithms I
> am
> > > > >> following
> > > > >> > > >> Java
> > > > >> > > >> > > > security Provider Terminology. You can refer to
> > > > >> > > >> > > >
> > > > >> https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> > > > >> > > >> > > > StandardNames.html#TrustManagerFactory link I
> provided
> > > > >> earlier in
> > > > >> > > >> the
> > > > >> > > >> > > > emails. It says PKIX is the default Algorithm for
> > > > >> > > >> TrustManagerFactory.
> > > > >> > > >> > > >
> > > > >> > > >> > > > 1. For SPIFFE, I am not sure why you are saying 'it
> does
> > > > not
> > > > >> > > >> implement
> > > > >> > > >> > > > custom algorithms' because the following file clearly
> > > > >> indicates
> > > > >> > > >> that it
> > > > >> > > >> > > > does use custom algorithm-
> > > > >> > > >> > > >
> > > > >> > > >> > > >
> > > > >> > > >> >
> > > > >> > >
> > > > >>
> > >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > > >> > > >> > > >
> > > > >> > > >> > > > provider/SpiffeProvider.java#L17
> > > > >> > > >> > > >
> > > > >> > > >> > > > Algorithm value:
> > > > >> > > >> > > >
> > > > >> > > >> > > >
> > > > >> > > >> >
> > > > >> > >
> > > > >>
> > >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > > >> > > >> > > >
> > > > >> > > >> > > > provider/SpiffeProviderConstants.java#L6
> > > > >> > > >> > > >
> > > > >> > > >> > > > @Harsha do you want to chime in since you use that
> > > > provider?
> > > > >> > > >> > > >
> > > > >> > > >> > > > Here the "Custom" Algorithm is not an implementation
> per
> > > > say
> > > > >> ,
> > > > >> > > >> rather
> > > > >> > > >> > > >
> > > > >> > > >> > > > used
> > > > >> > > >> > > >
> > > > >> > > >> > > > to invoke the custom trust store factory and key
> manager
> > > > >> factory.
> > > > >> > > >> You
> > > > >> > > >> > > >
> > > > >> > > >> > > > are
> > > > >> > > >> > > >
> > > > >> > > >> > > > not moving away from "standard" alogrithms that are
> > > > >> available.
> > > > >> > > >> > > >
> > > > >> > > >> > > >
> > > > >> > > >> >
> > > > >> > >
> > > > >>
> > >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > > >> > > >> > > >
> > > > >> > > >> > > > provider/SpiffeTrustManager.java
> > > > >> > > >> > > >
> > > > >> > > >> > > > As you can see it delegates all the calls of
> verification
> > > > of
> > > > >> > > >> > > >
> > > > >> > > >> > > > certificate
> > > > >> > > >> > > >
> > > > >> > > >> > > > to
> > > > >> > > >> > > >
> > > > >> > > >> > > > the default implementation available.
> > > > >> > > >> > > > So in our implementation we still use PKIX to verify
> the
> > > > >> > > certificate
> > > > >> > > >> > > > chain. So you are not losing anything here and
> Spiffe is
> > > > not
> > > > >> > > >> > > >
> > > > >> > > >> > > > reimplementing
> > > > >> > > >> > > >
> > > > >> > > >> > > > the verification process.
> > > > >> > > >> > > >
> > > > >> > > >> > > > 2. I already mentioned in my 3rd point, in my
> previous
> > > > post,
> > > > >> why
> > > > >> > > >> using
> > > > >> > > >> > > >
> > > > >> > > >> > > > ssl.provider does NOT work. I updated KIP-486 in
> > > "rejected
> > > > >> > > >> > > >
> > > > >> > > >> > > > alternatives"
> > > > >> > > >> > > >
> > > > >> > > >> > > > also why ssl.provider does not work.
> > > > >> > > >> > > >
> > > > >> > > >> > > > As mentioned before , provider is the right way to
> go. I
> > > am
> > > > >> still
> > > > >> > > >> not
> > > > >> > > >> > > > understanding the gap is.
> > > > >> > > >> > > > If I understand correctly your argument is ,
> provider is
> > > > >> going to
> > > > >> > > >> ask
> > > > >> > > >> > > >
> > > > >> > > >> > > > to
> > > > >> > > >> > > >
> > > > >> > > >> > > > implement a custom algorithm.
> > > > >> > > >> > > > My answer to that is , no you are not
> re-implementing the
> > > > >> > > algorithm.
> > > > >> > > >> > > >
> > > > >> > > >> > > > Please
> > > > >> > > >> > > >
> > > > >> > > >> > > > check the above link , TrustManager implementation it
> > > > >> delegates
> > > > >> > > the
> > > > >> > > >> > > >
> > > > >> > > >> > > > calls
> > > > >> > > >> > > >
> > > > >> > > >> > > > back. There is no need to implement your own here.
> > > > >> > > >> > > >
> > > > >> > > >> > > > 3. Security.insertProviderAt() comments were based on
> > > > >> assumption
> > > > >> > > if
> > > > >> > > >> > > >
> > > > >> > > >> > > > KIP-492
> > > > >> > > >> > > >
> > > > >> > > >> > > > changes are done and we use that mechanism to
> configure
> > > > >> providers
> > > > >> > > >> > > >
> > > > >> > > >> > > > instead
> > > > >> > > >> > > >
> > > > >> > > >> > > > of ssl.provider configuration.
> > > > >> > > >> > > >
> > > > >> > > >> > > > KIP-492 has patch available and going through review.
> > > > >> > > >> > > >
> > > > >> > > >> > > > Can you read my all the points, I mentioned in my
> > > previous
> > > > >> post,
> > > > >> > > >> very
> > > > >> > > >> > > >
> > > > >> > > >> > > > carefully? I am covering all the aspects in
> explaining. I
> > > > am
> > > > >> open
> > > > >> > > to
> > > > >> > > >> > > >
> > > > >> > > >> > > > still
> > > > >> > > >> > > >
> > > > >> > > >> > > > discuss more to clarify any doubts.
> > > > >> > > >> > > >
> > > > >> > > >> > > > "3. If we just use existing ssl.provider kafka
> > > > configuration
> > > > >> then
> > > > >> > > >> our
> > > > >> > > >> > > > provider will be used in
> SSLContext.getInstance(protocol,
> > > > >> > > provider)
> > > > >> > > >> > > >
> > > > >> > > >> > > > call
> > > > >> > > >> > > >
> > > > >> > > >> > > > in
> > > > >> > > >> > > >
> > > > >> > > >> > > > SslFactory.java <http://sslfactory.java/> <
> > > > >> > > http://sslfactory.java/>
> > > > >> > > >> <
> > > > >> > > >> > > > http://sslfactory.java/>
> > > > >> > > >> > > >
> > > > >> > > >> > > > and
> > > > >> > > >> > > >
> > > > >> > > >> > > > if our provider does not have
> > > > >> > > >> > > > implementation for SSLContext.TLS/TLSv1.1/TLSv1.2
> etc it
> > > > >> breaks
> > > > >> > > (we
> > > > >> > > >> > > >
> > > > >> > > >> > > > tested
> > > > >> > > >> > > >
> > > > >> > > >> > > > it). Example: In MyProvider sample above you see
> that I
> > > > >> didn't add
> > > > >> > > >> > > > SSLContext.TLSv1 as
> > > > >> > > >> > > > "Service+Algorithm" and that didn't work for me. In
> > > SPIFFE
> > > > >> > > provider
> > > > >> > > >> you
> > > > >> > > >> > > > don't have this challenge since you are planning to
> > > bypass
> > > > >> > > >> ssl.provider
> > > > >> > > >> > > >
> > > > >> > > >> > > > as
> > > > >> > > >> > > >
> > > > >> > > >> > > > you mention in the KIP-492."
> > > > >> > > >> > > >
> > > > >> > > >> > > > Yes here you need to pass the protocol that your
> > > > >> > > >> > > >
> > > > >> > > >> > > > KeyManager/TrustManager
> > > > >> > > >> > > >
> > > > >> > > >> > > > registered with and in no way its deviating from TLS
> RFC
> > > > >> spec.
> > > > >> > > >> > > >
> > > > >> > > >> > > >
> > > > >> > > >> >
> > > > >> > >
> > > > >>
> > >
> https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/
> > > > >> > > >> > > >
> > > > >> > > >> > > > javax/net/ssl/SSLContext.java#L134
> > > > >> > > >> > > >
> > > > >> > > >> > > > My suggestion here is for you to implement a simple
> > > > Security
> > > > >> > > >> Provider
> > > > >> > > >> > > >
> > > > >> > > >> > > > as
> > > > >> > > >> > > >
> > > > >> > > >> > > > you did before and register a Algorithm. You can use
> the
> > > > >> existing
> > > > >> > > >> > > > implementation in SpiffeProvider and plug in changes
> > > where
> > > > >> you
> > > > >> > > need
> > > > >> > > >> to
> > > > >> > > >> > > > retrieve the certificates from by making RPC call.
> > > > >> > > >> > > >
> > > > >> > > >> > > > Run an end-to-end test with Kafka broker coming up
> with
> > > > your
> > > > >> > > >> provider
> > > > >> > > >> > > > making calls to RPC call. You do need to pass the
> "custom
> > > > >> > > algorithm"
> > > > >> > > >> > > >
> > > > >> > > >> > > > that
> > > > >> > > >> > > >
> > > > >> > > >> > > > you registered in your key manager to invoke the
> > > provider.
> > > > I
> > > > >> think
> > > > >> > > >> your
> > > > >> > > >> > > > concern is this approach is replacing the existing
> known
> > > > >> > > >> ciphersuites
> > > > >> > > >> > > >
> > > > >> > > >> > > > and
> > > > >> > > >> > > >
> > > > >> > > >> > > > certificate validation provided by java. Which its
> not.
> > > > >> > > >> > > >
> > > > >> > > >> > > > Now test the TLS connection you can do so via openssl
> > > > >> -s_client
> > > > >> > > >> options
> > > > >> > > >> > > >
> > > > >> > > >> > > > to
> > > > >> > > >> > > >
> > > > >> > > >> > > > test if everything else is working.
> > > > >> > > >> > > >
> > > > >> > > >> > > > I am happy to share configs that we used if you are
> > > > >> interested.
> > > > >> > > >> > > >
> > > > >> > > >> > > > Thanks,
> > > > >> > > >> > > > Harsha
> > > > >> > > >> > > >
> > > > >> > > >> > > > Thanks
> > > > >> > > >> > > > Maulin
> > > > >> > > >> > > >
> > > > >> > > >> > > > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <
> > > > >> cmccabe@apache.org>
> > > > >> > > >> > > >
> > > > >> > > >> > > > wrote:
> > > > >> > > >> > > >
> > > > >> > > >> > > > Hi Maulin,
> > > > >> > > >> > > >
> > > > >> > > >> > > > A lot of JSSE providers don't implement custom
> > > algorithms.
> > > > >> Spire
> > > > >> > > is
> > > > >> > > >> a
> > > > >> > > >> > > >
> > > > >> > > >> > > > good
> > > > >> > > >> > > >
> > > > >> > > >> > > > example of a JSSE provider that doesn't, and yet is
> still
> > > > >> useful
> > > > >> > > to
> > > > >> > > >> > > >
> > > > >> > > >> > > > many
> > > > >> > > >> > > >
> > > > >> > > >> > > > people. Your JSSE provider can work fine even if it
> > > doesn't
> > > > >> > > >> implement a
> > > > >> > > >> > > > custom algorithm.
> > > > >> > > >> > > >
> > > > >> > > >> > > > Maybe I'm missing something, but I don't understand
> the
> > > > >> discussion
> > > > >> > > >> of
> > > > >> > > >> > > > Security.insertProviderAt() that you included.
> > > > >> SslEngineBuilder
> > > > >> > > >> doesn't
> > > > >> > > >> > > >
> > > > >> > > >> > > > use
> > > > >> > > >> > > >
> > > > >> > > >> > > > that API to get the security provider. Instead, it
> calls
> > > > >> > > >> > > > "SSLContext.getInstance(protocol, provider)", where
> > > > provider
> > > > >> is
> > > > >> > > the
> > > > >> > > >> > > >
> > > > >> > > >> > > > name
> > > > >> > > >> > > >
> > > > >> > > >> > > > of the provider.
> > > > >> > > >> > > >
> > > > >> > > >> > > > best,
> > > > >> > > >> > > > Colin
> > > > >> > > >> > > >
> > > > >> > > >> > > > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada
> wrote:
> > > > >> > > >> > > >
> > > > >> > > >> > > > On top of everything above I feel strongly to add
> the 4th
> > > > >> point
> > > > >> > > >> which
> > > > >> > > >> > > >
> > > > >> > > >> > > > is
> > > > >> > > >> > > >
> > > > >> > > >> > > > based on Java APIs for
> > > TrustManagerFactory.init(KeyStore) (
> > > > >> > > >> > > >
> > > > >> > > >> > > >
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> > > > >> > > >> > > > TrustManagerFactory.html#init(java.security.KeyStore
> > > > >> > > >> > > > <http://java.security.keystore/>
> > > > >> > > >> > > > <http://java.security.keystore/>)
> > > > >> > > >> > > > )
> > > > >> > > >> > > >
> > > > >> > > >> > > > and KeyManagerFactory.init(KeyStore, char[]) (
> > > > >> > > >> > > >
> > > > >> > > >> > > >
> > > > >> > > >> > >
> > > > >> > > >> >
> > > > >> > > >>
> > > > >> > >
> > > > >>
> > > >
> > >
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory
> > > > >> > > >> > > > .
> > > > >> > > >> > > >
> > > > >> > > >> > > > html#init(java.security.KeyStore <
> > > > >> http://java.security.keystore/>
> > > > >> > > >> > > <http://
> > > > >> > > >> > > > java.security.keystore/
> > > > >> > > >> > > > ,%20char[])
> > > > >> > > >> > > >
> > > > >> > > >> > > > ).
> > > > >> > > >> > > >
> > > > >> > > >> > > > 4. The above APIs are intended to support providing
> > > > >> "trust/key
> > > > >> > > >> > > >
> > > > >> > > >> > > > material"
> > > > >> > > >> > > >
> > > > >> > > >> > > > from the user without having to write their own
> > > > >> > > >> > > >
> > > > >> > > >> > > > TrustManager/KeyManagers.
> > > > >> > > >> > > >
> > > > >> > > >> > > > To quote from the TrustManagerFactory.init()'s
> > > > documentation
> > > > >> > > >> > > >
> > > > >> > > >> > > > "Initializes
> > > > >> > > >> > > >
> > > > >> > > >> > > > this factory with a source of certificate
> authorities and
> > > > >> related
> > > > >> > > >> trust
> > > > >> > > >> > > > material."
> > > > >> > > >> > > > To quote from the KeyManagerFactory.init()'s
> > > documentation
> > > > >> > > >> "Initializes
> > > > >> > > >> > > > this factory with a source of key material."
> > > > >> > > >> > > >
> > > > >> > > >> > > > Based on this it is clear that there is a flexibility
> > > > >> provided by
> > > > >> > > >> Java
> > > > >> > > >> > > >
> > > > >> > > >> > > > to
> > > > >> > > >> > > >
> > > > >> > > >> > > > to enable developers to provide the required
> trust/key
> > > > >> material
> > > > >> > > >> loaded
> > > > >> > > >> > > >
> > > > >> > > >> > > > from
> > > > >> > > >> > > >
> > > > >> > > >> > > > "anywhere" without requiring them to write custom
> > > provider
> > > > OR
> > > > >> > > >> trust/key
> > > > >> > > >> > > > managers. This same flexibility is reflected in Kafka
> > > code
> > > > >> also
> > > > >> > > >> where
> > > > >> > > >> > > >
> > > > >> > > >> > > > it
> > > > >> > > >> > > >
> > > > >> > > >> > > > loads the trust/keys from a local file and doesn't
> > > require
> > > > >> > > writing a
> > > > >> > > >> > > > Provider necessarily. If we do NOT have a custom
> > > algorithm,
> > > > >> it
> > > > >> > > makes
> > > > >> > > >> > > >
> > > > >> > > >> > > > less
> > > > >> > > >> > > >
> > > > >> > > >> > > > sense to write a Provider.
> > > > >> > > >> > > >
> > > > >> > > >> > > > Thanks
> > > > >> > > >> > > > Maulin
> > > > >> > > >> > > >
> > > > >> > > >> > > > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
> > > > >> > > >> > > >
> > > > >> > > >> > > > maulin.vasavada@gmail.com>
> > > > >> > > >> > > >
> > > > >> > > >> > > > wrote:
> > > > >> > > >> > > >
> > > > >> > > >> > > > Hi Harsha/Colin
> > > > >> > > >> > > >
> > > > >> > > >> > > > I did the sample with a custom Provider for
> > > > >> TrustStoreManager and
> > > > >> > > >> tried
> > > > >> > > >> > > > using ssl.provider Kafka config AND the way KIP-492
> is
> > > > >> suggesting
> > > > >> > > >> (by
> > > > >> > > >> > > > adding Provider programmatically instead of relying
> on
> > > > >> > > >> > > >
> > > > >> > > >> > > > ssl.provider+java.
> > > > >> > > >> > > >
> > > > >> > > >> > > > security. The below sample is followed by my detailed
> > > > >> findings.
> > > > >> > > I'll
> > > > >> > > >> > > > appreciate if you can go through it carefully and see
> > > > >> > > >> > > >
> > > > >> > > >> > > > if you
> > > > >> > > >> > > >
> > > > >> > > >> > > > see my point.
> > > > >> > > >> > > >
> > > > >> > > >> > > > package providertest;
> > > > >> > > >> > > >
> > > > >> > > >> > > > import java.security.Provider <
> > > > >> http://java.security.provider/>
> > > > >> > > >> > <http://
> > > > >> > > >> > > > java.security.provider/>
> > > > >> > > >> > > >
> > > > >> > > >> > > > <http://
> > > > >> > > >> > > >
> > > > >> > > >> > > > java.security.provider/>;
> > > > >> > > >> > > >
> > > > >> > > >> > > > public class MyProvider extends Provider {
> > > > >> > > >> > > >
> > > > >> > > >> > > > private static final String name = "MyProvider";
> private
> > > > >> static
> > > > >> > > >> double
> > > > >> > > >> > > > version = 1.0d;
> > > > >> > > >> > > > private static String info = "Maulin's SSL Provider
> > > > >> v"+version;
> > > > >> > > >> > > >
> > > > >> > > >> > > > public MyProvider() {
> > > > >> > > >> > > > super(name, version, info);
> > > > >> > > >> > > > this.put("TrustManagerFactory.PKIX",
> > > > >> > > >> > > >
> > > > >> > > >> > > > "providertest.MyTrustManagerFactory");
> > > > >> > > >> > > >
> > > > >> > > >> > > > }
> > > > >> > > >> > > > }
> > > > >> > > >> > > >
> > > > >> > > >> > > > *Details:*
> > > > >> > > >> > > >
> > > > >> > > >> > > > KIP-492 documents that it will use
> Security.addProvider()
> > > > >> assuming
> > > > >> > > >> it
> > > > >> > > >> > > >
> > > > >> > > >> > > > will
> > > > >> > > >> > > >
> > > > >> > > >> > > > add it as position '0' which is not a correct
> assumption.
> > > > The
> > > > >> > > >> > > > addProvider()'s documentation says it will add it to
> the
> > > > last
> > > > >> > > >> available
> > > > >> > > >> > > > position. You may want to correct that to say
> > > > >> > > >> > > > Security.insertProviderAt(provider, 1).
> > > > >> > > >> > > >
> > > > >> > > >> > > > Now coming back to our specific discussion,
> > > > >> > > >> > > >
> > > > >> > > >> > > > 1. SPIFFE example uses Custom Algorithm - spiffe.
> Hence
> > > > when
> > > > >> you
> > > > >> > > add
> > > > >> > > >> > > >
> > > > >> > > >> > > > that
> > > > >> > > >> > > >
> > > > >> > > >> > > > provider in the provider list via
> Security.addProvider()
> > > > the
> > > > >> > > >> position
> > > > >> > > >> > > >
> > > > >> > > >> > > > where
> > > > >> > > >> > > >
> > > > >> > > >> > > > it gets added doesn't matter (even if you don't end
> up
> > > > >> adding it
> > > > >> > > as
> > > > >> > > >> > > >
> > > > >> > > >> > > > first
> > > > >> > > >> > > >
> > > > >> > > >> > > > entry) since that is the ONLY provider for SPIFFE
> > > specific
> > > > >> > > algorithm
> > > > >> > > >> > > >
> > > > >> > > >> > > > you
> > > > >> > > >> > > >
> > > > >> > > >> > > > might have.
> > > > >> > > >> > > >
> > > > >> > > >> > > > We do *not* have custom algorithm for Key/Trust
> > > > StoreMangers.
> > > > >> > > Which
> > > > >> > > >> > > >
> > > > >> > > >> > > > means
> > > > >> > > >> > > >
> > > > >> > > >> > > > we have to use X509, PKIX etc "Standard Algorithms"
> ((
> > > > >> > > >> > > >
> > > > >> > > >> > > >
> > > > >> https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> > > > >> > > >> > > > StandardNames.html
> > > > >> > > >> > > > ))
> > > > >> > > >> > > >
> > > > >> > > >> > > > in our provider to override the TrustStoreManager
> (see my
> > > > >> sample
> > > > >> > > >> code)
> > > > >> > > >> > > >
> > > > >> > > >> > > > and
> > > > >> > > >> > > >
> > > > >> > > >> > > > KeyStoreManger and KeyManager. This creates another
> > > > challenge
> > > > >> > > >> > > >
> > > > >> > > >> > > > mentioned in
> > > > >> > > >> > > >
> > > > >> > > >> > > > the below point.
> > > > >> > > >> > > >
> > > > >> > > >> > > > 2. In order to make our Provider for loading custom
> > > > >> TrustStore
> > > > >> > > >> work, we
> > > > >> > > >> > > > have to add the provider as 'first' in the list since
> > > there
> > > > >> are
> > > > >> > > >> others
> > > > >> > > >> > > >
> > > > >> > > >> > > > with
> > > > >> > > >> > > >
> > > > >> > > >> > > > the same algorithm.
> > > > >> > > >> > > >
> > > > >> > > >> > > > However, the programatic way of adding provider
> > > > >> > > >> > > > (Security.insertProviderAt()) is *not* deterministic
> for
> > > > >> ordering
> > > > >> > > >> since
> > > > >> > > >> > > > different code can call the method for a different
> > > provider
> > > > >> and
> > > > >> > > >> > > >
> > > > >> > > >> > > > depending
> > > > >> > > >> > > >
> > > > >> > > >> > > > upon the order of the call our provider can be first
> or
> > > > >> pushed
> > > > >> > > down
> > > > >> > > >> the
> > > > >> > > >> > > > list. This can happen very well in any client
> application
> > > > >> using
> > > > >> > > >> Kafka.
> > > > >> > > >> > > >
> > > > >> > > >> > > > This
> > > > >> > > >> > > >
> > > > >> > > >> > > > is specially problematic for a case when you want to
> > > > >> guarantee
> > > > >> > > order
> > > > >> > > >> > > >
> > > > >> > > >> > > > for a
> > > > >> > > >> > > >
> > > > >> > > >> > > > Provider having "Standard Algorithms".
> > > > >> > > >> > > >
> > > > >> > > >> > > > If we add our provider in java.security file that
> > > > definitely
> > > > >> > > >> guarantees
> > > > >> > > >> > > > the order(unless somebody calls removeProvider()
> which is
> > > > >> less
> > > > >> > > >> > > >
> > > > >> > > >> > > > likely). But
> > > > >> > > >> > > >
> > > > >> > > >> > > > if we add our provider in java.security file it will
> > > defeat
> > > > >> the
> > > > >> > > >> > > >
> > > > >> > > >> > > > purpose of
> > > > >> > > >> > > >
> > > > >> > > >> > > > the KIP-492.
> > > > >> > > >> > > >
> > > > >> > > >> > > > In the gist - Apache Kafka must not rely on
> > > > >> "un-deterministic"
> > > > >> > > >> method
> > > > >> > > >> > > >
> > > > >> > > >> > > > to
> > > > >> > > >> > > >
> > > > >> > > >> > > > rely on Provider ordering.
> > > > >> > > >> > > >
> > > > >> > > >> > > > 3. If we just use existing ssl.provider kafka
> > > configuration
> > > > >> then
> > > > >> > > our
> > > > >> > > >> > > > provider will be used in
> SSLContext.getInstance(protocol,
> > > > >> > > provider)
> > > > >> > > >> > > >
> > > > >> > > >> > > > call in
> > > > >> > > >> > > >
> > > > >> > > >> > > > SslFactory.java <http://sslfactory.java/> <
> > > > >> > > http://sslfactory.java/>
> > > > >> > > >> <
> > > > >> > > >> > > > http://sslfactory.java/>
> > > > >> > > >> > > >
> > > > >> > > >> > > > and
> > > > >> > > >> > > >
> > > > >> > > >> > > > if our provider does not have implementation for
> > > > >> > > >> > > > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks
> > > > >> > > >> > > >
> > > > >> > > >> > > > (we
> > > > >> > > >> > > >
> > > > >> > > >> > > > tested it). Example:
> > > > >> > > >> > > >
> > > > >> > > >> > > > In
> > > > >> > > >> > > >
> > > > >> > > >> > > > MyProvider sample above you see that I didn't add
> > > > >> SSLContext.TLSv1
> > > > >> > > >> as
> > > > >> > > >> > > > "Service+Algorithm" and that didn't work for me. In
> > > SPIFFE
> > > > >> > > provider
> > > > >> > > >> you
> > > > >> > > >> > > > don't have this challenge since you are planning to
> > > bypass
> > > > >> > > >> > > >
> > > > >> > > >> > > > ssl.provider as
> > > > >> > > >> > > >
> > > > >> > > >> > > > you mention in the KIP-492.
> > > > >> > > >> > > >
> > > > >> > > >> > > > *Overall summary,*
> > > > >> > > >> > > >
> > > > >> > > >> > > > 1. Any provider based mechanisms- a) existing
> > > ssl.provider
> > > > >> and
> > > > >> > > >> > > >
> > > > >> > > >> > > > b)KIP-492,
> > > > >> > > >> > > >
> > > > >> > > >> > > > for loading key/trust store using "Standard
> Algorithms"
> > > do
> > > > >> not
> > > > >> > > work
> > > > >> > > >> > > >
> > > > >> > > >> > > > 2. Approach suggested in our KIP-486 works without
> any
> > > > issue
> > > > >> and
> > > > >> > > it
> > > > >> > > >> is
> > > > >> > > >> > > > *not* our context specific solve
> > > > >> > > >> > > >
> > > > >> > > >> > > > 3. Based on above we feel KIP-492 and KIP-486 are
> > > > >> complimentary
> > > > >> > > >> changes
> > > > >> > > >> > > > and not contradicting or redundent.
> > > > >> > > >> > > >
> > > > >> > > >> > > > If you want we can do a joint session somehow to walk
> > > > >> through the
> > > > >> > > >> > > >
> > > > >> > > >> > > > sample I
> > > > >> > > >> > > >
> > > > >> > > >> > > > have and various experiments I did. I would
> encourage you
> > > > to
> > > > >> do
> > > > >> > > >> similar
> > > > >> > > >> > > > exercise by writing a Provider for "Standard
> Algorithm"
> > > for
> > > > >> > > >> > > > TrustStoreManager (like our needs) and see what you
> find
> > > > >> since
> > > > >> > > only
> > > > >> > > >> > > >
> > > > >> > > >> > > > writing
> > > > >> > > >> > > >
> > > > >> > > >> > > > samples can bring out the complexity/challenges we
> face.
> > > > >> > > >> > > >
> > > > >> > > >> > > > Thanks
> > > > >> > > >> > > > Maulin
> > > > >> > > >> > > >
> > > > >> > > >> > > > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <
> > > > >> > > >> > > >
> > > > >> > > >> > > > maulin.vasavada@gmail.
> > > > >> > > >> > > >
> > > > >> > > >> > > > com> wrote:
> > > > >> > > >> > > >
> > > > >> > > >> > > > Just to update - still working on it. Get to work
> only on
> > > > >> and off
> > > > >> > > on
> > > > >> > > >> > > >
> > > > >> > > >> > > > it :(
> > > > >> > > >> > > >
> > > > >> > > >> > > > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
> > > > >> > > >> > > >
> > > > >> > > >> > > > maulin.vasavada@gmail.com>
> > > > >> > > >> > > >
> > > > >> > > >> > > > wrote:
> > > > >> > > >> > > >
> > > > >> > > >> > > > Hi Harsha
> > > > >> > > >> > > >
> > > > >> > > >> > > > Let me try to write samples and will let you know.
> > > > >> > > >> > > >
> > > > >> > > >> > > > Thanks
> > > > >> > > >> > > > Maulin
> > > > >> > > >> > > >
> > > > >> > > >> > > > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <
> > > > >> harsha.ch@gmail.com>
> > > > >> > > >> > > >
> > > > >> > > >> > > > wrote:
> > > > >> > > >> > > >
> > > > >> > > >> > > > Hi Maulin,
> > > > >> > > >> > > > With java security providers can be as custom you
> would
> > > > >> > > >> > > >
> > > > >> > > >> > > > like
> > > > >> > > >> > > >
> > > > >> > > >> > > > it to
> > > > >> > > >> > > > be. If you only want to to implement a custom way of
> > > > loading
> > > > >> the
> > > > >> > > >> > > >
> > > > >> > > >> > > > keystore
> > > > >> > > >> > > >
> > > > >> > > >> > > > and truststore and not implement any
> protocol/encryption
> > > > >> handling
> > > > >> > > >> you
> > > > >> > > >> > > >
> > > > >> > > >> > > > can
> > > > >> > > >> > > >
> > > > >> > > >> > > > leave them empty and no need to implement. Have you
> > > looked
> > > > >> into
> > > > >> > > the
> > > > >> > > >> > > >
> > > > >> > > >> > > > links I
> > > > >> > > >> > > >
> > > > >> > > >> > > > pasted before?
> > > > >> > > >> > > >
> > > > >> > > >> > > >
> > > > >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > > >> > > >> > > >
> > > > >> > > >> > > >
> > > > >> > > >> > >
> > > > >> > > >> >
> > > > >> > > >>
> > > > >> > >
> > > > >>
> > > >
> > >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> > > > >> > > >> > > >
> > > > >> > > >> > > > java
> > > > >> > > >> > > >
> > > > >> > > >> > > >
> > > > >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > > >> > > >> > > >
> > > spiffe-security-provider/src/main/java/spiffe/api/provider/
> > > > >> > > >> > > > SpiffeTrustManager.java <
> http://spiffetrustmanager.java/
> > > >
> > > > >> > > <http://
> > > > >> > > >> > > > spiffetrustmanager.java/>
> > > > >> > > >> > > >
> > > > >> > > >> > > >
> > > > >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > > >> > > >> > > >
> > > > >> > > >> > > >
> > > > >> > > >> > >
> > > > >> > > >> >
> > > > >> > > >>
> > > > >> > >
> > > > >>
> > > >
> > >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > > > >> > > >> > > >
> > > > >> > > >> > > > java
> > > > >> > > >> > > >
> > > > >> > > >> > > > Can you please tell me which methods are too complex
> in
> > > > >> above to
> > > > >> > > >> > > >
> > > > >> > > >> > > > implement
> > > > >> > > >> > > >
> > > > >> > > >> > > > or unnecessary? You are changing anything in SSL/TLS
> > > > >> > > implementations
> > > > >> > > >> > > > provided by
> > > > >> > > >> > > >
> > > > >> > > >> > > > All of the implementations delegating the checks to
> the
> > > > >> default
> > > > >> > > >> > > > implementation anyway.
> > > > >> > > >> > > > Spire agent is an example, its nothing but a GRPC
> server
> > > > >> listening
> > > > >> > > >> > > >
> > > > >> > > >> > > > on a
> > > > >> > > >> > > >
> > > > >> > > >> > > > unix domain socket . Above code is making a RPC call
> to
> > > the
> > > > >> local
> > > > >> > > >> > > >
> > > > >> > > >> > > > daemon
> > > > >> > > >> > > >
> > > > >> > > >> > > > to
> > > > >> > > >> > > > get the certificate and keys. The mechanics are
> pretty
> > > much
> > > > >> same
> > > > >> > > as
> > > > >> > > >> > > >
> > > > >> > > >> > > > what
> > > > >> > > >> > > >
> > > > >> > > >> > > > you are asking for.
> > > > >> > > >> > > >
> > > > >> > > >> > > > Thanks,
> > > > >> > > >> > > > Harsha
> > > > >> > > >> > > >
> > > > >> > > >> > > > On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
> > > > >> > > >> > > >
> > > > >> > > >> > > > maulin.vasavada@gmail.com>
> > > > >> > > >> > > >
> > > > >> > > >> > > > wrote:
> > > > >> > > >> > > >
> > > > >> > > >> > > > Imagine a scenario like - We know we have a custom
> KMS
> > > and
> > > > >> as a
> > > > >> > > >> > > >
> > > > >> > > >> > > > Kafka
> > > > >> > > >> > > >
> > > > >> > > >> > > > owner
> > > > >> > > >> > > >
> > > > >> > > >> > > > we want to comply to using that KMS source to load
> > > > >> keys/certs. As
> > > > >> > > >> > > >
> > > > >> > > >> > > > a
> > > > >> > > >> > > >
> > > > >> > > >> > > > Kafka
> > > > >> > > >> > > >
> > > > >> > > >> > > > owner we know how to integrate with KMS but doesn't
> > > > >> necessarily
> > > > >> > > >> > > >
> > > > >> > > >> > > > have
> > > > >> > > >> > > >
> > > > >> > > >> > > > to
> > > > >> > > >> > > >
> > > > >> > > >> > > > know anything about cipher suites, algorithms, and
> > > SSL/TLS
> > > > >> > > >> > > >
> > > > >> > > >> > > > implementation.
> > > > >> > > >> > > >
> > > > >> > > >> > > > Going the Provider way requires to know lot more
> than we
> > > > >> should,
> > > > >> > > >> > > >
> > > > >> > > >> > > > isn't it?
> > > > >> > > >> > > >
> > > > >> > > >> > > > Not that we would have concern/shy-away knowing those
> > > > >> details -
> > > > >> > > >> > > >
> > > > >> > > >> > > > but
> > > > >> > > >> > > >
> > > > >> > > >> > > > if we
> > > > >> > > >> > > >
> > > > >> > > >> > > > don't have to - why should we?
> > > > >> > > >> > > >
> > > > >> > > >> > > > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
> > > > >> > > >> > > >
> > > > >> > > >> > > > maulin.vasavada@gmail.com>
> > > > >> > > >> > > >
> > > > >> > > >> > > > wrote:
> > > > >> > > >> > > >
> > > > >> > > >> > > > Hi Harsha
> > > > >> > > >> > > >
> > > > >> > > >> > > > We don't have spire (or similar) agents and we do not
> > > have
> > > > >> > > >> > > >
> > > > >> > > >> > > > keys/certs
> > > > >> > > >> > > >
> > > > >> > > >> > > > locally on any brokers. To elaborate more on my
> previous
> > > > >> email,
> > > > >> > > >> > > >
> > > > >> > > >> > > > I agree that Java security Providers are used in much
> > > > broader
> > > > >> > > >> > > >
> > > > >> > > >> > > > sense
> > > > >> > > >> > > >
> > > > >> > > >> > > > - to
> > > > >> > > >> > > >
> > > > >> > > >> > > > have a particular implementation of an algorithm, use
> > > > >> specific
> > > > >> > > >> > > >
> > > > >> > > >> > > > cipher
> > > > >> > > >> > > >
> > > > >> > > >> > > > suites for SSL , OR in our current team's case have a
> > > > >> > > >> > > >
> > > > >> > > >> > > > particular
> > > > >> > > >> > > >
> > > > >> > > >> > > > way to
> > > > >> > > >> > > >
> > > > >> > > >> > > > leverage pre-generated SSL sessions. However, the
> scope
> > > of
> > > > >> our
> > > > >> > > >> > > >
> > > > >> > > >> > > > KIP
> > > > >> > > >> > > >
> > > > >> > > >> > > > (486)
> > > > >> > > >> > > >
> > > > >> > > >> > > > is
> > > > >> > > >> > > >
> > > > >> > > >> > > > much restricted than that. We merely intend to
> provide a
> > > > >> custom
> > > > >> > > >> > > > keystore/truststore for our SSL connections and not
> > > really
> > > > >> worry
> > > > >> > > >> > > >
> > > > >> > > >> > > > about
> > > > >> > > >> > > >
> > > > >> > > >> > > > underlying specific SSL/TLS implementation. This
> > > simplifies
> > > > >> it
> > > > >> > > >> > > >
> > > > >> > > >> > > > a
> > > > >> > > >> > > >
> > > > >> > > >> > > > lot for
> > > > >> > > >> > > >
> > > > >> > > >> > > > us to keep the concerns separate and clear.
> > > > >> > > >> > > >
> > > > >> > > >> > > > I feel our approach is more complimentary such that
> it
> > > > allows
> > > > >> > > >> > > >
> > > > >> > > >> > > > for
> > > > >> > > >> > > >
> > > > >> > > >> > > > using
> > > > >> > > >> > > >
> > > > >> > > >> > > > keystores of choice while retaining the flexibility
> to
> > > use
> > > > >> any
> > > > >> > > >> > > > underlying/available Provider for actually making
> the SSL
> > > > >> call.
> > > > >> > > >> > > >
> > > > >> > > >> > > > We agree with KIP-492's approach based on Providers
> (and
> > > > >> Java's
> > > > >> > > >> > > > recommendation), but also strongly believe that our
> > > > approach
> > > > >> can
> > > > >> > > >> > > >
> > > > >> > > >> > > > compliment
> > > > >> > > >> > > >
> > > > >> > > >> > > > it very effectively for reasons explained above.
> > > > >> > > >> > > >
> > > > >> > > >> > > > Thanks
> > > > >> > > >> > > > Maulin
> > > > >> > > >> > > >
> > > > >> > > >> > > > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <
> > > > >> > > >> > > >
> > > > >> > > >> > > > kafka@harsha.io
> > > > >> > > >> > > >
> > > > >> > > >> > > > wrote:
> > > > >> > > >> > > >
> > > > >> > > >> > > > Hi Maulin,
> > > > >> > > >> > > >
> > > > >> > > >> > > > On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
> > > > >> > > >> > > >
> > > > >> > > >> > > > maulin.vasavada@gmail.
> > > > >> > > >> > > >
> > > > >> > > >> > > > com>
> > > > >> > > >> > > > wrote:
> > > > >> > > >> > > >
> > > > >> > > >> > > > Hi Harsha
> > > > >> > > >> > > >
> > > > >> > > >> > > > The reason we rejected the SslProvider route is that
> - we
> > > > >> > > >> > > >
> > > > >> > > >> > > > only
> > > > >> > > >> > > >
> > > > >> > > >> > > > needed
> > > > >> > > >> > > >
> > > > >> > > >> > > > a
> > > > >> > > >> > > >
> > > > >> > > >> > > > custom way to load keys/certs. Not touch any policy
> that
> > > > >> > > >> > > >
> > > > >> > > >> > > > existing
> > > > >> > > >> > > >
> > > > >> > > >> > > > Providers
> > > > >> > > >> > > >
> > > > >> > > >> > > > govern like SunJSSE Provider.
> > > > >> > > >> > > >
> > > > >> > > >> > > > We have exactly the same requirements to load certs
> and
> > > > keys
> > > > >> > > >> > > >
> > > > >> > > >> > > > through
> > > > >> > > >> > > >
> > > > >> > > >> > > > spire
> > > > >> > > >> > > >
> > > > >> > > >> > > > agent. We used security.provider to do that exactly.
> I am
> > > > not
> > > > >> > > >> > > >
> > > > >> > > >> > > > sure
> > > > >> > > >> > > >
> > > > >> > > >> > > > why
> > > > >> > > >> > > >
> > > > >> > > >> > > > you
> > > > >> > > >> > > >
> > > > >> > > >> > > > would be modifying any policies provided by default
> > > SunJSSE
> > > > >> > > >> > > >
> > > > >> > > >> > > > provider.
> > > > >> > > >> > > >
> > > > >> > > >> > > > Can
> > > > >> > > >> > > >
> > > > >> > > >> > > > you give me an example of having custom provider that
> > > will
> > > > >> > > >> > > >
> > > > >> > > >> > > > override an
> > > > >> > > >> > > >
> > > > >> > > >> > > > existing policy in SunJSSE provider.
> > > > >> > > >> > > >
> > > > >> > > >> > > > As pointed out earlier, this kip
> > > > >> > > >> > > >
> > > > >> > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > >> > > >> > > >
> > > > >> KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> > > > >> > > >> > > >
> > > > >> > > >> > > > allows
> > > > >> > > >> > > > you to load security.provider through config.
> > > > >> > > >> > > > Take a look at the examples I gave before
> > > > >> > > >> > > >
> > > > >> > > >> > > >
> > > > >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > > >> > > >> > > >
> > > > >> > > >> > > >
> > > > >> > > >> > >
> > > > >> > > >> >
> > > > >> > > >>
> > > > >> > >
> > > > >>
> > > >
> > >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > > > >> > > >> > > >
> > > > >> > > >> > > > java
> > > > >> > > >> > > >
> > > > >> > > >> > > > It registers KeyManagerFactory and
> TrustManagerFactory
> > > and
> > > > >> > > >> > > >
> > > > >> > > >> > > > Keystore
> > > > >> > > >> > > >
> > > > >> > > >> > > > algorithm.
> > > > >> > > >> > > > Implement your custom way of loading Keystore in here
> > > > >> > > >> > > >
> > > > >> > > >> > > >
> > > > >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > > >> > > >> > > >
> > > > >> > > >> > > >
> > > > >> > > >> > >
> > > > >> > > >> >
> > > > >> > > >>
> > > > >> > >
> > > > >>
> > > >
> > >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> > > > >> > > >> > > >
> > > > >> > > >> > > > java
> > > > >> > > >> > > >
> > > > >> > > >> > > > and Trust manager like here
> > > > >> > > >> > > >
> > > > >> > > >> > > >
> > > > >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > > >> > > >> > > >
> > > spiffe-security-provider/src/main/java/spiffe/api/provider/
> > > > >> > > >> > > > SpiffeTrustManager.java <
> http://spiffetrustmanager.java/
> > > >
> > > > >> > > <http://
> > > > >> > > >> > > > spiffetrustmanager.java/>
> > > > >> > > >> > > >
> > > > >> > > >> > > > In your Kafka client you can set the
> security.provider to
> > > > >> your
> > > > >> > > >> > > >
> > > > >> > > >> > > > custom
> > > > >> > > >> > > >
> > > > >> > > >> > > > implementation and with this fix
> > > > >> > > >> > > > https://issues.apache.org/jira/browse/KAFKA-8191
> you can
> > > > set
> > > > >> > > >> > > > keyManagerAlgorigthm and trustManagerAlgorithm
> configs.
> > > > >> > > >> > > >
> > > > >> > > >> > > > All of this is in your clients and broker side and
> do not
> > > > >> need
> > > > >> > > >> > > >
> > > > >> > > >> > > > to
> > > > >> > > >> > > >
> > > > >> > > >> > > > touch
> > > > >> > > >> > > >
> > > > >> > > >> > > > any
> > > > >> > > >> > > > policy changes at JVM level. You'll register the
> > > providers
> > > > in
> > > > >> > > >> > > >
> > > > >> > > >> > > > the
> > > > >> > > >> > > >
> > > > >> > > >> > > > priority
> > > > >> > > >> > > >
> > > > >> > > >> > > > order and can still have SunJSSE provider and have
> your
> > > > >> custom
> > > > >> > > >> > > >
> > > > >> > > >> > > > provider
> > > > >> > > >> > > >
> > > > >> > > >> > > > to
> > > > >> > > >> > > >
> > > > >> > > >> > > > implement the key and trust managers.
> > > > >> > > >> > > >
> > > > >> > > >> > > > The ask here is different than KIP-492. We don't
> have any
> > > > >> need
> > > > >> > > >> > > >
> > > > >> > > >> > > > to
> > > > >> > > >> > > >
> > > > >> > > >> > > > modify/specify the algorithm parameter. Does that
> make
> > > > sense?
> > > > >> > > >> > > >
> > > > >> > > >> > > > The ask in KIP is introducing new interfaces where
> the
> > > > KIP's
> > > > >> > > >> > > > goal/motivation can be achieved through the
> > > > security.provider
> > > > >> > > >> > > >
> > > > >> > > >> > > > and
> > > > >> > > >> > > >
> > > > >> > > >> > > > we
> > > > >> > > >> > > >
> > > > >> > > >> > > > worked
> > > > >> > > >> > > > on similar goal without touching any Keystore or
> > > Truststore
> > > > >> > > >> > > >
> > > > >> > > >> > > > interfaces.
> > > > >> > > >> > > >
> > > > >> > > >> > > > My advise is against changing or introducing new
> > > interfaces
> > > > >> > > >> > > >
> > > > >> > > >> > > > when
> > > > >> > > >> > > >
> > > > >> > > >> > > > it can
> > > > >> > > >> > > >
> > > > >> > > >> > > > work through security.provider.
> > > > >> > > >> > > >
> > > > >> > > >> > > > Thanks,
> > > > >> > > >> > > > Harsha
> > > > >> > > >> > > >
> > > > >> > > >> > > > Thanks
> > > > >> > > >> > > >
> > > > >> > > >> > > > Maulin
> > > > >> > > >> > > >
> > > > >> > > >> > > > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
> > > > >> > > >> > > >
> > > > >> > > >> > > > kafka@harsha.io>
> > > > >> > > >> > > >
> > > > >> > > >> > > > wrote:
> > > > >> > > >> > > >
> > > > >> > > >> > > > In your KIP you added security. provider as rejected
> > > > >> > > >> > > >
> > > > >> > > >> > > > alternative
> > > > >> > > >> > > >
> > > > >> > > >> > > > and
> > > > >> > > >> > > >
> > > > >> > > >> > > > specified "its not the correct way". Do you mind
> > > explaining
> > > > >> > > >> > > >
> > > > >> > > >> > > > why
> > > > >> > > >> > > >
> > > > >> > > >> > > > its
> > > > >> > > >> > > >
> > > > >> > > >> > > > not? I
> > > > >> > > >> > > >
> > > > >> > > >> > > > didn't find any evidence in Java docs to say so.
> Contrary
> > > > to
> > > > >> > > >> > > >
> > > > >> > > >> > > > your
> > > > >> > > >> > > >
> > > > >> > > >> > > > statement
> > > > >> > > >> > > >
> > > > >> > > >> > > > it does say in the java docs
> > > > >> > > >> > > > " However, please note that a provider can be used to
> > > > >> > > >> > > >
> > > > >> > > >> > > > implement
> > > > >> > > >> > > >
> > > > >> > > >> > > > any
> > > > >> > > >> > > >
> > > > >> > > >> > > > security service in Java that uses a pluggable
> > > architecture
> > > > >> > > >> > > >
> > > > >> > > >> > > > with
> > > > >> > > >> > > >
> > > > >> > > >> > > > a
> > > > >> > > >> > > >
> > > > >> > > >> > > > choice
> > > > >> > > >> > > >
> > > > >> > > >> > > > of implementations that fit underneath."
> > > > >> > > >> > > >
> > > > >> > > >> > > > Java Security Providers have been used by other
> projects
> > > to
> > > > >> > > >> > > >
> > > > >> > > >> > > > provide
> > > > >> > > >> > > >
> > > > >> > > >> > > > such
> > > > >> > > >> > > >
> > > > >> > > >> > > > integration . I am not sure if you looked into Spiffe
> > > > >> > > >> > > >
> > > > >> > > >> > > > project to
> > > > >> > > >> > > >
> > > > >> > > >> > > > efficiently distribute certificates but here is an
> > > example
> > > > of
> > > > >> > > >> > > >
> > > > >> > > >> > > > Java
> > > > >> > > >> > > >
> > > > >> > > >> > > > provider
> > > > >> > > >> > > >
> > > > >> > > >> > > >
> > > > >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > > >> > > >> > > >
> > > > >> > > >> > > >
> > > > >> > > >> > >
> > > > >> > > >> >
> > > > >> > > >>
> > > > >> > >
> > > > >>
> > > >
> > >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > > > >> > > >> > > >
> > > > >> > > >> > > > java which
> > > > >> > > >> > > > obtains certificates from local daemons.
> > > > >> > > >> > > > These integrations are being used in Tomcat, Jetty
> etc..
> > > We
> > > > >> > > >> > > >
> > > > >> > > >> > > > are
> > > > >> > > >> > > >
> > > > >> > > >> > > > also
> > > > >> > > >> > > >
> > > > >> > > >> > > > using
> > > > >> > > >> > > >
> > > > >> > > >> > > > Security provider to do the same in our Kafka
> clusters.
> > > So
> > > > >> > > >> > > >
> > > > >> > > >> > > > unless I
> > > > >> > > >> > > >
> > > > >> > > >> > > > see
> > > > >> > > >> > > >
> > > > >> > > >> > > > more evidence why security.provider doesn't work for
> you
> > > > >> > > >> > > >
> > > > >> > > >> > > > adding
> > > > >> > > >> > > >
> > > > >> > > >> > > > new
> > > > >> > > >> > > >
> > > > >> > > >> > > > interfaces while there exists more cleaner way of
> > > achieving
> > > > >> > > >> > > >
> > > > >> > > >> > > > the
> > > > >> > > >> > > >
> > > > >> > > >> > > > goals
> > > > >> > > >> > > >
> > > > >> > > >> > > > of
> > > > >> > > >> > > >
> > > > >> > > >> > > > this KIP is unnecessary and breaks the well known
> > > security
> > > > >> > > >> > > >
> > > > >> > > >> > > > interfaces
> > > > >> > > >> > > >
> > > > >> > > >> > > > provided by Java itself.
> > > > >> > > >> > > >
> > > > >> > > >> > > > Thanks,
> > > > >> > > >> > > > Harsha
> > > > >> > > >> > > >
> > > > >> > > >> > > > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani
> <
> > > > >> > > >> > > >
> > > > >> > > >> > > > kafka@harsha.io
> > > > >> > > >> > > >
> > > > >> > > >> > > > wrote:
> > > > >> > > >> > > >
> > > > >> > > >> > > > Hi Maulin,
> > > > >> > > >> > > > Not sure if you looked at my previous replies. This
> > > > >> > > >> > > >
> > > > >> > > >> > > > changes
> > > > >> > > >> > > >
> > > > >> > > >> > > > are not required as there is already security
> Provider to
> > > > do
> > > > >> > > >> > > >
> > > > >> > > >> > > > what you
> > > > >> > > >> > > >
> > > > >> > > >> > > > are
> > > > >> > > >> > > >
> > > > >> > > >> > > > proposing. This KIP
> > > > >> > > >> > > >
> > > > >> > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > >> > > >> > > >
> > > > >> > > >> > > >
> > > > >> KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> > > > >> > > >> > > >
> > > > >> > > >> > > > also
> > > > >> > > >> > > >
> > > > >> > > >> > > > addresses easy registration of such providers.
> > > > >> > > >> > > >
> > > > >> > > >> > > > Thanks,
> > > > >> > > >> > > > Harsha
> > > > >> > > >> > > >
> > > > >> > > >> > > > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
> > > > >> > > >> > > >
> > > > >> > > >> > > > <maulin.vasavada@gmail.
> > > > >> > > >> > > >
> > > > >> > > >> > > > com> wrote:
> > > > >> > > >> > > >
> > > > >> > > >> > > > Bump! Can somebody please review this?
> > > > >> > > >> > > >
> > > > >> > > >> > > > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
> > > > >> > > >> > > >
> > > > >> > > >> > > > maulin.vasavada@gmail.com>
> > > > >> > > >> > > >
> > > > >> > > >> > > > wrote:
> > > > >> > > >> > > >
> > > > >> > > >> > > > Bump! Can somebody please review this?
> > > > >> > > >> > > >
> > > > >> > > >> > > > --
> > > > >> > > >> > > > Thanks,
> > > > >> > > >> > > > M.Sai Sandeep
> > > > >> > > >> > > >
> > > > >> > > >> > > >
> > > > >> > > >> > >
> > > > >> > > >> >
> > > > >> > > >>
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Colin McCabe <cm...@apache.org>.
On Mon, Sep 2, 2019, at 03:33, Rajini Sivaram wrote:
> I would expect SslEngineBuilder interface to look something like this,
> perhaps with some tweaking:
>
> public interface SslEngineBuilder extends Configurable, Closeable {
>
> Set<String> reconfigurableConfigs();
>
> boolean shouldBeRebuilt(Map<String, Object> nextConfigs);
>
> SSLEngine createSslEngine(Mode mode, String peerHost, int
> peerPort, String endpointIdentification);
>
> }
>
> The existing SslEngineBuilder class would be renamed and will implement
> this interface. Loading of keystore/truststore will be in SslEngineBuilder
> as it is now. The method `shouldBeRebuilt()` will validate configs during
> reconfiguration and decide if reconfiguration is required because keystore
> or truststore changed. SslFactory.reconfigurableConfigs() will return
> SslEngineBuilder.reconfigurableConfigs() as well including any custom
> configs of SslEngineBuilder, so no other changes will be required when we
> eventually support custom SSL configs.
>
> We don't want to make SslFactory the pluggable class since that contains
> validation logic for SSL engines. Everything that we want to customise is
> contained in SslEngineBuilder. Basically custom SslEngineBuilder will
> validate custom configs during reconfiguration and create SSLEngine.
> SslFactory will continue to perform validation of SSLEngines and this will
> not be customizable. SslEngineBuilder will not be reconfigurable, instead
> we create a new builder as we do now to avoid having to deal with
> thread-safety and atomicity of updates. We could consider using a public
> Reconfigurable interface as the pluggable interface for consistency, but I
> think we would still want to create a new Builder on reconfiguration and
> retain non-pluggable SSL engine validation in SslFactory.
+1
C.
>
>
> On Fri, Aug 30, 2019 at 10:21 PM Maulin Vasavada <ma...@gmail.com>
> wrote:
>
> > Looking at SslFactory and SslEngineBuilder I feel the responsibilities are
> > not clear. Both has public method for createSSLEngine for example. I feel
> > the SslEngineBuilder was created to move out lot of code but it is not
> > necessarily a public class (e.g. I don't think anybody calling
> > SslEngineBuilder separately without SslFactory in between). I am currently
> > inclined toward what Celement is suggesting - having pluggable SslFactory.
> >
> > Let me do this - let me refactor SslFactory and SslEngineBuilder and review
> > what I can come up with you guys. Let us see if we can address all the
> > objections raised previously for KIP-383's iterations. I'll need sometime
> > though. Let me try to do it by next of next week.
> >
> > Thanks
> > Maulin
> >
> > On Fri, Aug 30, 2019 at 12:25 PM Pellerin, Clement <
> > Clement_Pellerin@ibi.com>
> > wrote:
> >
> > > What is your solution to the objection that killed the second iteration
> > of
> > > KIP-383?
> > > Mainly, how do you support validation of reconfiguration requests that
> > > involve new custom properties implemented by the pluggable factory?
> > > Custom properties do not exist yet, but that is very legitimate thing to
> > > design for the future.
> > >
> > > That's why I favor a pluggable SslFactory instead of an SslEngineBuilder
> > > factory.
> > >
> > > -----Original Message-----
> > > From: Maulin Vasavada [mailto:maulin.vasavada@gmail.com]
> > > Sent: Friday, August 30, 2019 3:07 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and
> > > TrustStore
> > >
> > > +1 for making SslEngineBuilder configurable upon more thoughts.
> > >
> > > However, in the abstraction and default implementation we should make
> > sure
> > > when we do have a requirement to plugin custom key/trust store people
> > don't
> > > have to write lot more code which may not be related to it.
> > >
> > > Having said that, does this mean, we resurrect KIP-383 and update it with
> > > latest context and go from there?
> > >
> > > We are willing to take up that work for making it configurable.
> > >
> > > Thanks
> > > Maulin
> > >
> > >
> > >
> > >
> > >
> > > On Fri, Aug 30, 2019 at 10:34 AM Maulin Vasavada <
> > > maulin.vasavada@gmail.com>
> > > wrote:
> > >
> > > > Why don't we make SSLEngineBuilder code delegate the whole Key/Trust
> > > store
> > > > initialization to the interfaces we are proposing? Default
> > implementation
> > > > for those key/trust store loader interfaces will be "file based"
> > loading
> > > vs
> > > > if somebody wants to customize any of it they can.
> > > >
> > > > Would that make sense?
> > > >
> > > > On Fri, Aug 30, 2019 at 10:03 AM Colin McCabe <cm...@apache.org>
> > > wrote:
> > > >
> > > >> +1 for making SslEngineBuilder configurable. This would give
> > > >> implementers a lot more flexibility-- to use key distribution methods
> > > that
> > > >> were not files, for example.
> > > >>
> > > >> best,
> > > >> Colin
> > > >>
> > > >>
> > > >> On Fri, Aug 30, 2019, at 02:03, Rajini Sivaram wrote:
> > > >> > Just to make sure we are on the same page - KIP-383 was written
> > before
> > > >> > the
> > > >> > code was refactored. The refactoring addressed some of the concerns
> > of
> > > >> > KIP-383. My suggestion was to make SslEngineBuilder configurable.
> > The
> > > >> > default implementation of this pluggable class would be
> > > >> >
> > > >>
> > >
> > https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/ssl/SslEngineBuilder.java
> > > >> .
> > > >> > That should give an idea of the size of the configurable part that a
> > > >> > custom
> > > >> > class needs to implement. A large part of that is about loading
> > > >> > keystore/truststore. I agree it has slightly more code than KIP-486
> > > >> > proposes, but since it lets you customize creation of SSLEngine, it
> > > >> > would
> > > >> > address every possible scenario.
> > > >> >
> > > >> > Thoughts?
> > > >> >
> > > >> >
> > > >> > On Fri, Aug 30, 2019 at 2:02 AM Maulin Vasavada <
> > > >> maulin.vasavada@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> > > I thought about it more. I feel that no matter how we refactor the
> > > >> code
> > > >> > > (with or without KIP-383 integrated), ultimately the need of
> > > >> customizing
> > > >> > > loading for keys and certs will still remain. Whenever that need
> > > >> arises we
> > > >> > > might end up thinking about the solution suggested by our KIP-486.
> > > >> Hence
> > > >> > > regardless of the other KIPs and configurations "if we do need to
> > > >> customize
> > > >> > > loading of keys/certs, we will need the code changes suggested by
> > > this
> > > >> > > KIP".
> > > >> > >
> > > >> > > Let me know what you guys think.
> > > >> > >
> > > >> > > Harsha, we are working on changing the interfaces for key/trust
> > > store
> > > >> > > loaders with Certificate and PrivateKey objects. Will probably be
> > > >> able to
> > > >> > > update it later today or tomorrow.
> > > >> > >
> > > >> > > Thanks
> > > >> > > Maulin
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > On Thu, Aug 29, 2019 at 2:30 PM Maulin Vasavada <
> > > >> maulin.vasavada@gmail.com
> > > >> > > >
> > > >> > > wrote:
> > > >> > >
> > > >> > > > On that, I actually looked at KIP-383 before briefly. However,
> > > that
> > > >> > > > sounded like lot of changes suggested.
> > > >> > > >
> > > >> > > > One "key" thing we have to keep in mind is - IF we need lot of
> > > >> > > > customization Kafka already allows you to use your SslProvider
> > via
> > > >> > > > ssl.providers or the changes done by KIP-492 and
> > > >> > > > SSLContext.getInstance(protocol, provider) call allows us to
> > > return
> > > >> the
> > > >> > > > SSLContext with "ALL" the details we would like to customize.
> > > Hence
> > > >> I am
> > > >> > > > not sure that customization suggested by KIP-383 would be worth
> > > the
> > > >> > > effort.
> > > >> > > > We also have similar SSLContext customization outside of Kafka.
> > > >> > > >
> > > >> > > > Thanks
> > > >> > > > Maulin
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > On Thu, Aug 29, 2019 at 12:47 PM Pellerin, Clement <
> > > >> > > > Clement_Pellerin@ibi.com> wrote:
> > > >> > > >
> > > >> > > >> KIP-383 in its present form was vetoed because it was not
> > > possible
> > > >> to
> > > >> > > add
> > > >> > > >> validation of custom properties in a future KIP. The solution
> > to
> > > >> that is
> > > >> > > >> the first proposal I wrote for KIP-383 which made the whole
> > > >> SslFactory
> > > >> > > >> pluggable. That first solution was also vetoed hence the
> > > deadlock.
> > > >> > > >>
> > > >> > > >> Replacing the whole factory was a much nicer solution. It was
> > > >> vetoed
> > > >> > > >> because doing this almost invariably meant the replacement lost
> > > >> all the
> > > >> > > >> complex validation code in the default SslFactory.
> > > >> > > >>
> > > >> > > >> My current idea is to extract the validation code into another
> > > >> public
> > > >> > > API
> > > >> > > >> that SslFactory would call. I did not look at the newly
> > > refactored
> > > >> code
> > > >> > > and
> > > >> > > >> I did not study how to do this yet. KIP-383 was not popular at
> > > the
> > > >> time
> > > >> > > and
> > > >> > > >> designing a new solution is a lot of work.
> > > >> > > >>
> > > >> > > >> Is there interest from 3 binding voters for something like
> > this?
> > > >> > > >>
> > > >> > > >> -----Original Message-----
> > > >> > > >> From: Rajini Sivaram [mailto:rajinisivaram@gmail.com]
> > > >> > > >> Sent: Thursday, August 29, 2019 2:57 PM
> > > >> > > >> To: dev
> > > >> > > >> Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore
> > and
> > > >> > > >> TrustStore
> > > >> > > >>
> > > >> > > >> Hi Maulin,
> > > >> > > >>
> > > >> > > >> In SSL scenarios, I imagine security providers introduced by
> > > >> KIP-492 are
> > > >> > > >> likely to be most useful when you want to use third party
> > > >> providers. The
> > > >> > > >> biggest advantage of the config from that KIP is that you don't
> > > >> need to
> > > >> > > >> write much code to integrate existing security providers into
> > > Kafka
> > > >> > > >> brokers
> > > >> > > >> or clients. As I understand it, KIP-486 is a more convenient
> > > >> option for
> > > >> > > >> the
> > > >> > > >> specific problem of loading keystores/truststores differently.
> > It
> > > >> can be
> > > >> > > >> achieved in theory with KIP-492, but KIP-486 is a much simpler
> > > >> option
> > > >> > > for
> > > >> > > >> this case.
> > > >> > > >>
> > > >> > > >> My concern about KIP-486 is that it introduces yet another
> > > >> interface
> > > >> > > into
> > > >> > > >> our already complex security code, while only solving one
> > > >> particular use
> > > >> > > >> case. Have you looked at
> > > >> > > >>
> > > >> > > >>
> > > >> > >
> > > >>
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-383%3A++Pluggable+interface+for+SSL+Factory
> > > >> > > >> ?
> > > >> > > >> The goal was to make
> > > >> > > >> org.apache.kafka.common.security.ssl.SslEngineBuilder
> > pluggable.
> > > >> > > >> The code has already been refactored by Colin after that KIP
> > was
> > > >> > > written,
> > > >> > > >> making it easier to implement KIP-383. This should enable you
> > to
> > > >> load
> > > >> > > your
> > > >> > > >> keystores and truststores differently. Using a pluggable
> > > >> > > SslEngineBuilder
> > > >> > > >> will also solve several other use cases at the same time.
> > KIP-383
> > > >> hasn't
> > > >> > > >> been voted through yet, but perhaps you could take a look and
> > we
> > > >> could
> > > >> > > >> revive that instead if it solves your use case as well?
> > > >> > > >>
> > > >> > > >> Regards,
> > > >> > > >>
> > > >> > > >> Rajini
> > > >> > > >>
> > > >> > > >>
> > > >> > > >> On Thu, Aug 29, 2019 at 6:42 PM Maulin Vasavada <
> > > >> > > >> maulin.vasavada@gmail.com>
> > > >> > > >> wrote:
> > > >> > > >>
> > > >> > > >> > Hi Harsha
> > > >> > > >> >
> > > >> > > >> > Thank you. Appreciate your time and support on this. Let me
> > go
> > > >> back do
> > > >> > > >> some
> > > >> > > >> > more research and get back to you on the KeyStore interface
> > > part.
> > > >> > > >> > Basically, if we return certs and keys in the interface then
> > > >> Kafka
> > > >> > > code
> > > >> > > >> > will have to build KeyStore object - which is also
> > reasonable.
> > > >> > > >> >
> > > >> > > >> > Thanks
> > > >> > > >> > Maulin
> > > >> > > >> >
> > > >> > > >> > On Thu, Aug 29, 2019 at 10:01 AM Harsha Chintalapani <
> > > >> kafka@harsha.io
> > > >> > > >
> > > >> > > >> > wrote:
> > > >> > > >> >
> > > >> > > >> > > Hi Maulin,
> > > >> > > >> > > Use cases are clear now. I am +1 for
> > > moving
> > > >> > > >> forward
> > > >> > > >> > > with the discussions on having such configurable option for
> > > >> users.
> > > >> > > But
> > > >> > > >> > the
> > > >> > > >> > > interfaces is proposed doesn't look right to me. We are
> > still
> > > >> > > talking
> > > >> > > >> > about
> > > >> > > >> > > keystore interfaces. Given keystore's are used as
> > filebased
> > > >> way of
> > > >> > > >> > > transporting certificates I am not sure it will help the
> > rest
> > > >> of the
> > > >> > > >> > > user-base.
> > > >> > > >> > > In short, I am +1 on the KIP's motivation
> > > >> and only
> > > >> > > >> have
> > > >> > > >> > > questions around returning keystores instead of returning
> > > >> certs,
> > > >> > > >> private
> > > >> > > >> > > keys etc. . If others in the community are ok with such
> > > >> interface we
> > > >> > > >> can
> > > >> > > >> > > move forward.
> > > >> > > >> > >
> > > >> > > >> > > Thanks,
> > > >> > > >> > > Harsha
> > > >> > > >> > >
> > > >> > > >> > >
> > > >> > > >> > > On Wed, Aug 28, 2019 at 1:51 PM, Maulin Vasavada <
> > > >> > > >> > > maulin.vasavada@gmail.com>
> > > >> > > >> > > wrote:
> > > >> > > >> > >
> > > >> > > >> > > > Hi Harsha
> > > >> > > >> > > >
> > > >> > > >> > > > As we synced-up offline on this topic, we hope you don't
> > > >> have any
> > > >> > > >> more
> > > >> > > >> > > > clarifications that you are seeking. If that is the case,
> > > >> can you
> > > >> > > >> > please
> > > >> > > >> > > > help us move this forward and discuss what changes you
> > > would
> > > >> > > expect
> > > >> > > >> on
> > > >> > > >> > > the
> > > >> > > >> > > > KIP design in order to make it valuable contribution?
> > > >> > > >> > > >
> > > >> > > >> > > > Just FYI - we verified our primary design change with the
> > > >> author
> > > >> > > of
> > > >> > > >> > Sun's
> > > >> > > >> > > > X509 Trustmanager implementation and the outcome is that
> > > >> what we
> > > >> > > are
> > > >> > > >> > > > proposing makes sense at the heart of it - "Instead of
> > > >> writing
> > > >> > > >> > > TrustManager
> > > >> > > >> > > > just plugin the Trust store". We are open to discuss
> > > >> additional
> > > >> > > >> changes
> > > >> > > >> > > > that you/anybody else would like to see on the
> > > functionality
> > > >> > > >> however.
> > > >> > > >> > > >
> > > >> > > >> > > > Thanks
> > > >> > > >> > > > Maulin
> > > >> > > >> > > >
> > > >> > > >> > > > On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada <
> > > >> > > >> > > maulin.vasavada@gmail.com>
> > > >> > > >> > > > wrote:
> > > >> > > >> > > >
> > > >> > > >> > > > Hi Harsha
> > > >> > > >> > > >
> > > >> > > >> > > > Any response on my question? I feel this KIP is worth
> > > >> > > accommodating.
> > > >> > > >> > Your
> > > >> > > >> > > > help is much appreciated.
> > > >> > > >> > > >
> > > >> > > >> > > > Thanks
> > > >> > > >> > > > Maulin
> > > >> > > >> > > >
> > > >> > > >> > > > On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada <
> > > >> > > >> > maulin.vasavada@gmail.
> > > >> > > >> > > > com> wrote:
> > > >> > > >> > > >
> > > >> > > >> > > > Hi Harsha
> > > >> > > >> > > >
> > > >> > > >> > > > I've examined the SPIFFE provider more and have one
> > > question
> > > >> -
> > > >> > > >> > > >
> > > >> > > >> > > > If SPIFFE didn't have a need to do checkSpiffeId() call
> > at
> > > >> the
> > > >> > > below
> > > >> > > >> > > > location, would you really still write the Provider? *OR*
> > > >> Would
> > > >> > > you
> > > >> > > >> > just
> > > >> > > >> > > > use TrustManagerFactory.init(KeyStore) signature to pass
> > > the
> > > >> > > >> KeyStore
> > > >> > > >> > > from
> > > >> > > >> > > > set of certs returned by spiffeIdManager.
> > > getTrustedCerts()?
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> > > >> >
> > > >> > >
> > > >>
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > >> > > >> > > > provider/CertificateUtils.java#L100
> > > >> > > >> > > >
> > > >> > > >> > > > /**
> > > >> > > >> > > >
> > > >> > > >> > > > * Validates that the SPIFFE ID is present and matches the
> > > >> SPIFFE
> > > >> > > ID
> > > >> > > >> > > > configured in
> > > >> > > >> > > > * the java.security property ssl.spiffe.accept
> > > >> > > >> > > > *
> > > >> > > >> > > > * If the authorized spiffe ids list is empty any spiffe
> > id
> > > is
> > > >> > > >> > authorized
> > > >> > > >> > > > *
> > > >> > > >> > > > * @param chain an array of X509Certificate that contains
> > > the
> > > >> > > Peer's
> > > >> > > >> > SVID
> > > >> > > >> > > > to be validated
> > > >> > > >> > > > * @throws CertificateException when either the
> > certificates
> > > >> > > doesn't
> > > >> > > >> > have
> > > >> > > >> > > a
> > > >> > > >> > > > SPIFFE ID or the SPIFFE ID is not authorized
> > > >> > > >> > > > */
> > > >> > > >> > > > static void checkSpiffeId(X509Certificate[] chain) throws
> > > >> > > >> > > > CertificateException {
> > > >> > > >> > > >
> > > >> > > >> > > > Thanks
> > > >> > > >> > > > Maulin
> > > >> > > >> > > >
> > > >> > > >> > > > On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani <
> > > >> > > >> kafka@harsha.io>
> > > >> > > >> > > > wrote:
> > > >> > > >> > > >
> > > >> > > >> > > > Maulin,
> > > >> > > >> > > > The code parts you are pointing are specific for Spiffe
> > and
> > > >> if
> > > >> > > >> > > > you are talking about validate method which uses PKIX
> > check
> > > >> like
> > > >> > > any
> > > >> > > >> > > other
> > > >> > > >> > > > provider does.
> > > >> > > >> > > > If you want to default to SunJSSE everywhere you can do
> > so
> > > by
> > > >> > > >> > delegating
> > > >> > > >> > > > the calls in these methods to SunJSSE provider.
> > > >> > > >> > > >
> > > >> > > >> > > > TrustManagerFactory tmf = TrustManagerFactory
> > > >> > > >> > > >
> > .getInstance(TrustManagerFactory.getDefaultAlgorithm());and
> > > >> use
> > > >> > > >> > > > tmf.chekServerTrusted()
> > > >> > > >> > > > or use
> > > >> > > >> > > > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> > > >> > > >> > > > TrustManagerFactory.html#getInstance(java.lang.String)if
> > > you
> > > >> want
> > > >> > > a
> > > >> > > >> > > > specific provider.
> > > >> > > >> > > >
> > > >> > > >> > > > -Harsha
> > > >> > > >> > > >
> > > >> > > >> > > > On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada <
> > > >> > > >> > maulin.vasavada@gmail.
> > > >> > > >> > > > com>
> > > >> > > >> > > > wrote:
> > > >> > > >> > > >
> > > >> > > >> > > > Okay, so I take that you guys agree that I have to write
> > a
> > > >> > > 'custom'
> > > >> > > >> > > > algorithm and a provider to make it work , correct?
> > > >> > > >> > > >
> > > >> > > >> > > > Now, for Harsha's comment "Here the 'Custom' Algorithm is
> > > >> not an
> > > >> > > >> > > > implementation per say , ..." , I diagree. You can refer
> > to
> > > >> > > https://
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> > > >> >
> > > >> > > >>
> > > >> > >
> > > >>
> > > github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/
> > > >> > > >> > > >
> > > >> > > >> > > > SpiffeTrustManager.java#L91 <
> > > >> http://spiffetrustmanager.java/#L91>
> > > >> > > >> and
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> > > >> >
> > > >> > >
> > > >>
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > >> > > >> > > >
> > > >> > > >> > > > provider/CertificateUtils.java#L100
> > > >> > > >> > > >
> > > >> > > >> > > > "that code" is the customization you have for the custom
> > > way
> > > >> to
> > > >> > > >> check
> > > >> > > >> > > > something on top of regular checks. That method is NOT
> > > doing
> > > >> > > custom
> > > >> > > >> > > > truststore loading. It is validating/verifying something
> > in
> > > >> the
> > > >> > > >> > > >
> > > >> > > >> > > > "custom"
> > > >> > > >> > > >
> > > >> > > >> > > > way with spiffeId.
> > > >> > > >> > > > I bet that without that you won't have a need of the
> > custom
> > > >> > > >> algorithm
> > > >> > > >> > > >
> > > >> > > >> > > > in
> > > >> > > >> > > >
> > > >> > > >> > > > the first place.
> > > >> > > >> > > >
> > > >> > > >> > > > Let me know if you agree to this.
> > > >> > > >> > > >
> > > >> > > >> > > > Thanks
> > > >> > > >> > > > Maulin
> > > >> > > >> > > >
> > > >> > > >> > > > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <
> > > >> mprsai@gmail.com>
> > > >> > > >> > > >
> > > >> > > >> > > > wrote:
> > > >> > > >> > > >
> > > >> > > >> > > > Hi Maulin, thanks for the discussion. As Harsha pointed
> > > out,
> > > >> to
> > > >> > > use
> > > >> > > >> the
> > > >> > > >> > > > KIP492, you need to create a new provider, register a
> > *new*
> > > >> custom
> > > >> > > >> > > > algorithm for your keymanager and trustmanager factory
> > > >> > > >> implementations.
> > > >> > > >> > > > After this, the kafka server configuration can be done as
> > > >> given
> > > >> > > >> below
> > > >> > > >> > > >
> > > >> > > >> > > > # Register the provider class with custom algorithm, say
> > > >> CUSTOM
> > > >> > > >> > > >
> > > >> > > >> > > > security.
> > > >> > > >> > > >
> > > >> > > >> > > > provider.classes=com.company.security.CustomProvider
> > > >> > > >> > > > <http://provider.classes
> > > >> =com.company.security.customprovider/>
> > > >> > > >> > > > <http://security.provider.classes
> > > >> > > >> > > >
> > > >> > > >> > > > =com.company.security.customprovider/>
> > > >> > > >> > > >
> > > >> > > >> > > > # Register the keymanager and trustmanager algorithms
> > > >> > > >> > > > # These algorithms indicate that the Keymanager and
> > > >> Trustmanagers
> > > >> > > >> > > > registered under the algorithm "CUSTOM" needs to be used
> > > >> > > >> > > > ssl.trustmanager.algorithm=CUSTOM
> > > >> > > >> > > > ssl.keymanager.algorithm=CUSTOM
> > > >> > > >> > > >
> > > >> > > >> > > > And the customprovider looks like this...
> > > >> > > >> > > >
> > > >> > > >> > > > public class CustomProvider extends Provider {
> > > >> > > >> > > > public CustomProvider() {
> > > >> > > >> > > > super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and
> > > >> > > TrustStore");
> > > >> > > >> > > > super.put("KeyManagerFactory.CUSTOM",
> > > >> "customKeyManagerFactory");
> > > >> > > >> > > > super.put("TrustManagerFactory.CUSTOM",
> > > >> > > >> > > > "customTrustManagerFactory");
> > > >> > > >> > > > }
> > > >> > > >> > > > }
> > > >> > > >> > > >
> > > >> > > >> > > > The PR for this is in review and can be found here:
> > > >> > > >> > > >
> > > >> > > >> > > > https://github.com/
> > > >> > > >> > > >
> > > >> > > >> > > > apache/kafka/pull/7090
> > > >> > > >> > > > This PR includes the fixed insertProviderAt function
> > call.
> > > >> > > >> > > >
> > > >> > > >> > > > On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani <
> > > >> > > >> kafka@harsha.io>
> > > >> > > >> > > > wrote:
> > > >> > > >> > > >
> > > >> > > >> > > > Answers are in-line
> > > >> > > >> > > >
> > > >> > > >> > > > On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada <
> > > >> > > >> > > >
> > > >> > > >> > > > maulin.vasavada@gmail.
> > > >> > > >> > > >
> > > >> > > >> > > > com
> > > >> > > >> > > >
> > > >> > > >> > > > wrote:
> > > >> > > >> > > >
> > > >> > > >> > > > Hi Colin
> > > >> > > >> > > >
> > > >> > > >> > > > When I refer to "standard" or "custom" algorithms I am
> > > >> following
> > > >> > > >> Java
> > > >> > > >> > > > security Provider Terminology. You can refer to
> > > >> > > >> > > >
> > > >> https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> > > >> > > >> > > > StandardNames.html#TrustManagerFactory link I provided
> > > >> earlier in
> > > >> > > >> the
> > > >> > > >> > > > emails. It says PKIX is the default Algorithm for
> > > >> > > >> TrustManagerFactory.
> > > >> > > >> > > >
> > > >> > > >> > > > 1. For SPIFFE, I am not sure why you are saying 'it does
> > > not
> > > >> > > >> implement
> > > >> > > >> > > > custom algorithms' because the following file clearly
> > > >> indicates
> > > >> > > >> that it
> > > >> > > >> > > > does use custom algorithm-
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> > > >> >
> > > >> > >
> > > >>
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > >> > > >> > > >
> > > >> > > >> > > > provider/SpiffeProvider.java#L17
> > > >> > > >> > > >
> > > >> > > >> > > > Algorithm value:
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> > > >> >
> > > >> > >
> > > >>
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > >> > > >> > > >
> > > >> > > >> > > > provider/SpiffeProviderConstants.java#L6
> > > >> > > >> > > >
> > > >> > > >> > > > @Harsha do you want to chime in since you use that
> > > provider?
> > > >> > > >> > > >
> > > >> > > >> > > > Here the "Custom" Algorithm is not an implementation per
> > > say
> > > >> ,
> > > >> > > >> rather
> > > >> > > >> > > >
> > > >> > > >> > > > used
> > > >> > > >> > > >
> > > >> > > >> > > > to invoke the custom trust store factory and key manager
> > > >> factory.
> > > >> > > >> You
> > > >> > > >> > > >
> > > >> > > >> > > > are
> > > >> > > >> > > >
> > > >> > > >> > > > not moving away from "standard" alogrithms that are
> > > >> available.
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> > > >> >
> > > >> > >
> > > >>
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > >> > > >> > > >
> > > >> > > >> > > > provider/SpiffeTrustManager.java
> > > >> > > >> > > >
> > > >> > > >> > > > As you can see it delegates all the calls of verification
> > > of
> > > >> > > >> > > >
> > > >> > > >> > > > certificate
> > > >> > > >> > > >
> > > >> > > >> > > > to
> > > >> > > >> > > >
> > > >> > > >> > > > the default implementation available.
> > > >> > > >> > > > So in our implementation we still use PKIX to verify the
> > > >> > > certificate
> > > >> > > >> > > > chain. So you are not losing anything here and Spiffe is
> > > not
> > > >> > > >> > > >
> > > >> > > >> > > > reimplementing
> > > >> > > >> > > >
> > > >> > > >> > > > the verification process.
> > > >> > > >> > > >
> > > >> > > >> > > > 2. I already mentioned in my 3rd point, in my previous
> > > post,
> > > >> why
> > > >> > > >> using
> > > >> > > >> > > >
> > > >> > > >> > > > ssl.provider does NOT work. I updated KIP-486 in
> > "rejected
> > > >> > > >> > > >
> > > >> > > >> > > > alternatives"
> > > >> > > >> > > >
> > > >> > > >> > > > also why ssl.provider does not work.
> > > >> > > >> > > >
> > > >> > > >> > > > As mentioned before , provider is the right way to go. I
> > am
> > > >> still
> > > >> > > >> not
> > > >> > > >> > > > understanding the gap is.
> > > >> > > >> > > > If I understand correctly your argument is , provider is
> > > >> going to
> > > >> > > >> ask
> > > >> > > >> > > >
> > > >> > > >> > > > to
> > > >> > > >> > > >
> > > >> > > >> > > > implement a custom algorithm.
> > > >> > > >> > > > My answer to that is , no you are not re-implementing the
> > > >> > > algorithm.
> > > >> > > >> > > >
> > > >> > > >> > > > Please
> > > >> > > >> > > >
> > > >> > > >> > > > check the above link , TrustManager implementation it
> > > >> delegates
> > > >> > > the
> > > >> > > >> > > >
> > > >> > > >> > > > calls
> > > >> > > >> > > >
> > > >> > > >> > > > back. There is no need to implement your own here.
> > > >> > > >> > > >
> > > >> > > >> > > > 3. Security.insertProviderAt() comments were based on
> > > >> assumption
> > > >> > > if
> > > >> > > >> > > >
> > > >> > > >> > > > KIP-492
> > > >> > > >> > > >
> > > >> > > >> > > > changes are done and we use that mechanism to configure
> > > >> providers
> > > >> > > >> > > >
> > > >> > > >> > > > instead
> > > >> > > >> > > >
> > > >> > > >> > > > of ssl.provider configuration.
> > > >> > > >> > > >
> > > >> > > >> > > > KIP-492 has patch available and going through review.
> > > >> > > >> > > >
> > > >> > > >> > > > Can you read my all the points, I mentioned in my
> > previous
> > > >> post,
> > > >> > > >> very
> > > >> > > >> > > >
> > > >> > > >> > > > carefully? I am covering all the aspects in explaining. I
> > > am
> > > >> open
> > > >> > > to
> > > >> > > >> > > >
> > > >> > > >> > > > still
> > > >> > > >> > > >
> > > >> > > >> > > > discuss more to clarify any doubts.
> > > >> > > >> > > >
> > > >> > > >> > > > "3. If we just use existing ssl.provider kafka
> > > configuration
> > > >> then
> > > >> > > >> our
> > > >> > > >> > > > provider will be used in SSLContext.getInstance(protocol,
> > > >> > > provider)
> > > >> > > >> > > >
> > > >> > > >> > > > call
> > > >> > > >> > > >
> > > >> > > >> > > > in
> > > >> > > >> > > >
> > > >> > > >> > > > SslFactory.java <http://sslfactory.java/> <
> > > >> > > http://sslfactory.java/>
> > > >> > > >> <
> > > >> > > >> > > > http://sslfactory.java/>
> > > >> > > >> > > >
> > > >> > > >> > > > and
> > > >> > > >> > > >
> > > >> > > >> > > > if our provider does not have
> > > >> > > >> > > > implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it
> > > >> breaks
> > > >> > > (we
> > > >> > > >> > > >
> > > >> > > >> > > > tested
> > > >> > > >> > > >
> > > >> > > >> > > > it). Example: In MyProvider sample above you see that I
> > > >> didn't add
> > > >> > > >> > > > SSLContext.TLSv1 as
> > > >> > > >> > > > "Service+Algorithm" and that didn't work for me. In
> > SPIFFE
> > > >> > > provider
> > > >> > > >> you
> > > >> > > >> > > > don't have this challenge since you are planning to
> > bypass
> > > >> > > >> ssl.provider
> > > >> > > >> > > >
> > > >> > > >> > > > as
> > > >> > > >> > > >
> > > >> > > >> > > > you mention in the KIP-492."
> > > >> > > >> > > >
> > > >> > > >> > > > Yes here you need to pass the protocol that your
> > > >> > > >> > > >
> > > >> > > >> > > > KeyManager/TrustManager
> > > >> > > >> > > >
> > > >> > > >> > > > registered with and in no way its deviating from TLS RFC
> > > >> spec.
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> > > >> >
> > > >> > >
> > > >>
> > https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/
> > > >> > > >> > > >
> > > >> > > >> > > > javax/net/ssl/SSLContext.java#L134
> > > >> > > >> > > >
> > > >> > > >> > > > My suggestion here is for you to implement a simple
> > > Security
> > > >> > > >> Provider
> > > >> > > >> > > >
> > > >> > > >> > > > as
> > > >> > > >> > > >
> > > >> > > >> > > > you did before and register a Algorithm. You can use the
> > > >> existing
> > > >> > > >> > > > implementation in SpiffeProvider and plug in changes
> > where
> > > >> you
> > > >> > > need
> > > >> > > >> to
> > > >> > > >> > > > retrieve the certificates from by making RPC call.
> > > >> > > >> > > >
> > > >> > > >> > > > Run an end-to-end test with Kafka broker coming up with
> > > your
> > > >> > > >> provider
> > > >> > > >> > > > making calls to RPC call. You do need to pass the "custom
> > > >> > > algorithm"
> > > >> > > >> > > >
> > > >> > > >> > > > that
> > > >> > > >> > > >
> > > >> > > >> > > > you registered in your key manager to invoke the
> > provider.
> > > I
> > > >> think
> > > >> > > >> your
> > > >> > > >> > > > concern is this approach is replacing the existing known
> > > >> > > >> ciphersuites
> > > >> > > >> > > >
> > > >> > > >> > > > and
> > > >> > > >> > > >
> > > >> > > >> > > > certificate validation provided by java. Which its not.
> > > >> > > >> > > >
> > > >> > > >> > > > Now test the TLS connection you can do so via openssl
> > > >> -s_client
> > > >> > > >> options
> > > >> > > >> > > >
> > > >> > > >> > > > to
> > > >> > > >> > > >
> > > >> > > >> > > > test if everything else is working.
> > > >> > > >> > > >
> > > >> > > >> > > > I am happy to share configs that we used if you are
> > > >> interested.
> > > >> > > >> > > >
> > > >> > > >> > > > Thanks,
> > > >> > > >> > > > Harsha
> > > >> > > >> > > >
> > > >> > > >> > > > Thanks
> > > >> > > >> > > > Maulin
> > > >> > > >> > > >
> > > >> > > >> > > > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <
> > > >> cmccabe@apache.org>
> > > >> > > >> > > >
> > > >> > > >> > > > wrote:
> > > >> > > >> > > >
> > > >> > > >> > > > Hi Maulin,
> > > >> > > >> > > >
> > > >> > > >> > > > A lot of JSSE providers don't implement custom
> > algorithms.
> > > >> Spire
> > > >> > > is
> > > >> > > >> a
> > > >> > > >> > > >
> > > >> > > >> > > > good
> > > >> > > >> > > >
> > > >> > > >> > > > example of a JSSE provider that doesn't, and yet is still
> > > >> useful
> > > >> > > to
> > > >> > > >> > > >
> > > >> > > >> > > > many
> > > >> > > >> > > >
> > > >> > > >> > > > people. Your JSSE provider can work fine even if it
> > doesn't
> > > >> > > >> implement a
> > > >> > > >> > > > custom algorithm.
> > > >> > > >> > > >
> > > >> > > >> > > > Maybe I'm missing something, but I don't understand the
> > > >> discussion
> > > >> > > >> of
> > > >> > > >> > > > Security.insertProviderAt() that you included.
> > > >> SslEngineBuilder
> > > >> > > >> doesn't
> > > >> > > >> > > >
> > > >> > > >> > > > use
> > > >> > > >> > > >
> > > >> > > >> > > > that API to get the security provider. Instead, it calls
> > > >> > > >> > > > "SSLContext.getInstance(protocol, provider)", where
> > > provider
> > > >> is
> > > >> > > the
> > > >> > > >> > > >
> > > >> > > >> > > > name
> > > >> > > >> > > >
> > > >> > > >> > > > of the provider.
> > > >> > > >> > > >
> > > >> > > >> > > > best,
> > > >> > > >> > > > Colin
> > > >> > > >> > > >
> > > >> > > >> > > > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
> > > >> > > >> > > >
> > > >> > > >> > > > On top of everything above I feel strongly to add the 4th
> > > >> point
> > > >> > > >> which
> > > >> > > >> > > >
> > > >> > > >> > > > is
> > > >> > > >> > > >
> > > >> > > >> > > > based on Java APIs for
> > TrustManagerFactory.init(KeyStore) (
> > > >> > > >> > > >
> > > >> > > >> > > > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> > > >> > > >> > > > TrustManagerFactory.html#init(java.security.KeyStore
> > > >> > > >> > > > <http://java.security.keystore/>
> > > >> > > >> > > > <http://java.security.keystore/>)
> > > >> > > >> > > > )
> > > >> > > >> > > >
> > > >> > > >> > > > and KeyManagerFactory.init(KeyStore, char[]) (
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> > > >> > >
> > > >> > > >> >
> > > >> > > >>
> > > >> > >
> > > >>
> > >
> > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory
> > > >> > > >> > > > .
> > > >> > > >> > > >
> > > >> > > >> > > > html#init(java.security.KeyStore <
> > > >> http://java.security.keystore/>
> > > >> > > >> > > <http://
> > > >> > > >> > > > java.security.keystore/
> > > >> > > >> > > > ,%20char[])
> > > >> > > >> > > >
> > > >> > > >> > > > ).
> > > >> > > >> > > >
> > > >> > > >> > > > 4. The above APIs are intended to support providing
> > > >> "trust/key
> > > >> > > >> > > >
> > > >> > > >> > > > material"
> > > >> > > >> > > >
> > > >> > > >> > > > from the user without having to write their own
> > > >> > > >> > > >
> > > >> > > >> > > > TrustManager/KeyManagers.
> > > >> > > >> > > >
> > > >> > > >> > > > To quote from the TrustManagerFactory.init()'s
> > > documentation
> > > >> > > >> > > >
> > > >> > > >> > > > "Initializes
> > > >> > > >> > > >
> > > >> > > >> > > > this factory with a source of certificate authorities and
> > > >> related
> > > >> > > >> trust
> > > >> > > >> > > > material."
> > > >> > > >> > > > To quote from the KeyManagerFactory.init()'s
> > documentation
> > > >> > > >> "Initializes
> > > >> > > >> > > > this factory with a source of key material."
> > > >> > > >> > > >
> > > >> > > >> > > > Based on this it is clear that there is a flexibility
> > > >> provided by
> > > >> > > >> Java
> > > >> > > >> > > >
> > > >> > > >> > > > to
> > > >> > > >> > > >
> > > >> > > >> > > > to enable developers to provide the required trust/key
> > > >> material
> > > >> > > >> loaded
> > > >> > > >> > > >
> > > >> > > >> > > > from
> > > >> > > >> > > >
> > > >> > > >> > > > "anywhere" without requiring them to write custom
> > provider
> > > OR
> > > >> > > >> trust/key
> > > >> > > >> > > > managers. This same flexibility is reflected in Kafka
> > code
> > > >> also
> > > >> > > >> where
> > > >> > > >> > > >
> > > >> > > >> > > > it
> > > >> > > >> > > >
> > > >> > > >> > > > loads the trust/keys from a local file and doesn't
> > require
> > > >> > > writing a
> > > >> > > >> > > > Provider necessarily. If we do NOT have a custom
> > algorithm,
> > > >> it
> > > >> > > makes
> > > >> > > >> > > >
> > > >> > > >> > > > less
> > > >> > > >> > > >
> > > >> > > >> > > > sense to write a Provider.
> > > >> > > >> > > >
> > > >> > > >> > > > Thanks
> > > >> > > >> > > > Maulin
> > > >> > > >> > > >
> > > >> > > >> > > > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
> > > >> > > >> > > >
> > > >> > > >> > > > maulin.vasavada@gmail.com>
> > > >> > > >> > > >
> > > >> > > >> > > > wrote:
> > > >> > > >> > > >
> > > >> > > >> > > > Hi Harsha/Colin
> > > >> > > >> > > >
> > > >> > > >> > > > I did the sample with a custom Provider for
> > > >> TrustStoreManager and
> > > >> > > >> tried
> > > >> > > >> > > > using ssl.provider Kafka config AND the way KIP-492 is
> > > >> suggesting
> > > >> > > >> (by
> > > >> > > >> > > > adding Provider programmatically instead of relying on
> > > >> > > >> > > >
> > > >> > > >> > > > ssl.provider+java.
> > > >> > > >> > > >
> > > >> > > >> > > > security. The below sample is followed by my detailed
> > > >> findings.
> > > >> > > I'll
> > > >> > > >> > > > appreciate if you can go through it carefully and see
> > > >> > > >> > > >
> > > >> > > >> > > > if you
> > > >> > > >> > > >
> > > >> > > >> > > > see my point.
> > > >> > > >> > > >
> > > >> > > >> > > > package providertest;
> > > >> > > >> > > >
> > > >> > > >> > > > import java.security.Provider <
> > > >> http://java.security.provider/>
> > > >> > > >> > <http://
> > > >> > > >> > > > java.security.provider/>
> > > >> > > >> > > >
> > > >> > > >> > > > <http://
> > > >> > > >> > > >
> > > >> > > >> > > > java.security.provider/>;
> > > >> > > >> > > >
> > > >> > > >> > > > public class MyProvider extends Provider {
> > > >> > > >> > > >
> > > >> > > >> > > > private static final String name = "MyProvider"; private
> > > >> static
> > > >> > > >> double
> > > >> > > >> > > > version = 1.0d;
> > > >> > > >> > > > private static String info = "Maulin's SSL Provider
> > > >> v"+version;
> > > >> > > >> > > >
> > > >> > > >> > > > public MyProvider() {
> > > >> > > >> > > > super(name, version, info);
> > > >> > > >> > > > this.put("TrustManagerFactory.PKIX",
> > > >> > > >> > > >
> > > >> > > >> > > > "providertest.MyTrustManagerFactory");
> > > >> > > >> > > >
> > > >> > > >> > > > }
> > > >> > > >> > > > }
> > > >> > > >> > > >
> > > >> > > >> > > > *Details:*
> > > >> > > >> > > >
> > > >> > > >> > > > KIP-492 documents that it will use Security.addProvider()
> > > >> assuming
> > > >> > > >> it
> > > >> > > >> > > >
> > > >> > > >> > > > will
> > > >> > > >> > > >
> > > >> > > >> > > > add it as position '0' which is not a correct assumption.
> > > The
> > > >> > > >> > > > addProvider()'s documentation says it will add it to the
> > > last
> > > >> > > >> available
> > > >> > > >> > > > position. You may want to correct that to say
> > > >> > > >> > > > Security.insertProviderAt(provider, 1).
> > > >> > > >> > > >
> > > >> > > >> > > > Now coming back to our specific discussion,
> > > >> > > >> > > >
> > > >> > > >> > > > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence
> > > when
> > > >> you
> > > >> > > add
> > > >> > > >> > > >
> > > >> > > >> > > > that
> > > >> > > >> > > >
> > > >> > > >> > > > provider in the provider list via Security.addProvider()
> > > the
> > > >> > > >> position
> > > >> > > >> > > >
> > > >> > > >> > > > where
> > > >> > > >> > > >
> > > >> > > >> > > > it gets added doesn't matter (even if you don't end up
> > > >> adding it
> > > >> > > as
> > > >> > > >> > > >
> > > >> > > >> > > > first
> > > >> > > >> > > >
> > > >> > > >> > > > entry) since that is the ONLY provider for SPIFFE
> > specific
> > > >> > > algorithm
> > > >> > > >> > > >
> > > >> > > >> > > > you
> > > >> > > >> > > >
> > > >> > > >> > > > might have.
> > > >> > > >> > > >
> > > >> > > >> > > > We do *not* have custom algorithm for Key/Trust
> > > StoreMangers.
> > > >> > > Which
> > > >> > > >> > > >
> > > >> > > >> > > > means
> > > >> > > >> > > >
> > > >> > > >> > > > we have to use X509, PKIX etc "Standard Algorithms" ((
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> > > >> > > >> > > > StandardNames.html
> > > >> > > >> > > > ))
> > > >> > > >> > > >
> > > >> > > >> > > > in our provider to override the TrustStoreManager (see my
> > > >> sample
> > > >> > > >> code)
> > > >> > > >> > > >
> > > >> > > >> > > > and
> > > >> > > >> > > >
> > > >> > > >> > > > KeyStoreManger and KeyManager. This creates another
> > > challenge
> > > >> > > >> > > >
> > > >> > > >> > > > mentioned in
> > > >> > > >> > > >
> > > >> > > >> > > > the below point.
> > > >> > > >> > > >
> > > >> > > >> > > > 2. In order to make our Provider for loading custom
> > > >> TrustStore
> > > >> > > >> work, we
> > > >> > > >> > > > have to add the provider as 'first' in the list since
> > there
> > > >> are
> > > >> > > >> others
> > > >> > > >> > > >
> > > >> > > >> > > > with
> > > >> > > >> > > >
> > > >> > > >> > > > the same algorithm.
> > > >> > > >> > > >
> > > >> > > >> > > > However, the programatic way of adding provider
> > > >> > > >> > > > (Security.insertProviderAt()) is *not* deterministic for
> > > >> ordering
> > > >> > > >> since
> > > >> > > >> > > > different code can call the method for a different
> > provider
> > > >> and
> > > >> > > >> > > >
> > > >> > > >> > > > depending
> > > >> > > >> > > >
> > > >> > > >> > > > upon the order of the call our provider can be first or
> > > >> pushed
> > > >> > > down
> > > >> > > >> the
> > > >> > > >> > > > list. This can happen very well in any client application
> > > >> using
> > > >> > > >> Kafka.
> > > >> > > >> > > >
> > > >> > > >> > > > This
> > > >> > > >> > > >
> > > >> > > >> > > > is specially problematic for a case when you want to
> > > >> guarantee
> > > >> > > order
> > > >> > > >> > > >
> > > >> > > >> > > > for a
> > > >> > > >> > > >
> > > >> > > >> > > > Provider having "Standard Algorithms".
> > > >> > > >> > > >
> > > >> > > >> > > > If we add our provider in java.security file that
> > > definitely
> > > >> > > >> guarantees
> > > >> > > >> > > > the order(unless somebody calls removeProvider() which is
> > > >> less
> > > >> > > >> > > >
> > > >> > > >> > > > likely). But
> > > >> > > >> > > >
> > > >> > > >> > > > if we add our provider in java.security file it will
> > defeat
> > > >> the
> > > >> > > >> > > >
> > > >> > > >> > > > purpose of
> > > >> > > >> > > >
> > > >> > > >> > > > the KIP-492.
> > > >> > > >> > > >
> > > >> > > >> > > > In the gist - Apache Kafka must not rely on
> > > >> "un-deterministic"
> > > >> > > >> method
> > > >> > > >> > > >
> > > >> > > >> > > > to
> > > >> > > >> > > >
> > > >> > > >> > > > rely on Provider ordering.
> > > >> > > >> > > >
> > > >> > > >> > > > 3. If we just use existing ssl.provider kafka
> > configuration
> > > >> then
> > > >> > > our
> > > >> > > >> > > > provider will be used in SSLContext.getInstance(protocol,
> > > >> > > provider)
> > > >> > > >> > > >
> > > >> > > >> > > > call in
> > > >> > > >> > > >
> > > >> > > >> > > > SslFactory.java <http://sslfactory.java/> <
> > > >> > > http://sslfactory.java/>
> > > >> > > >> <
> > > >> > > >> > > > http://sslfactory.java/>
> > > >> > > >> > > >
> > > >> > > >> > > > and
> > > >> > > >> > > >
> > > >> > > >> > > > if our provider does not have implementation for
> > > >> > > >> > > > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks
> > > >> > > >> > > >
> > > >> > > >> > > > (we
> > > >> > > >> > > >
> > > >> > > >> > > > tested it). Example:
> > > >> > > >> > > >
> > > >> > > >> > > > In
> > > >> > > >> > > >
> > > >> > > >> > > > MyProvider sample above you see that I didn't add
> > > >> SSLContext.TLSv1
> > > >> > > >> as
> > > >> > > >> > > > "Service+Algorithm" and that didn't work for me. In
> > SPIFFE
> > > >> > > provider
> > > >> > > >> you
> > > >> > > >> > > > don't have this challenge since you are planning to
> > bypass
> > > >> > > >> > > >
> > > >> > > >> > > > ssl.provider as
> > > >> > > >> > > >
> > > >> > > >> > > > you mention in the KIP-492.
> > > >> > > >> > > >
> > > >> > > >> > > > *Overall summary,*
> > > >> > > >> > > >
> > > >> > > >> > > > 1. Any provider based mechanisms- a) existing
> > ssl.provider
> > > >> and
> > > >> > > >> > > >
> > > >> > > >> > > > b)KIP-492,
> > > >> > > >> > > >
> > > >> > > >> > > > for loading key/trust store using "Standard Algorithms"
> > do
> > > >> not
> > > >> > > work
> > > >> > > >> > > >
> > > >> > > >> > > > 2. Approach suggested in our KIP-486 works without any
> > > issue
> > > >> and
> > > >> > > it
> > > >> > > >> is
> > > >> > > >> > > > *not* our context specific solve
> > > >> > > >> > > >
> > > >> > > >> > > > 3. Based on above we feel KIP-492 and KIP-486 are
> > > >> complimentary
> > > >> > > >> changes
> > > >> > > >> > > > and not contradicting or redundent.
> > > >> > > >> > > >
> > > >> > > >> > > > If you want we can do a joint session somehow to walk
> > > >> through the
> > > >> > > >> > > >
> > > >> > > >> > > > sample I
> > > >> > > >> > > >
> > > >> > > >> > > > have and various experiments I did. I would encourage you
> > > to
> > > >> do
> > > >> > > >> similar
> > > >> > > >> > > > exercise by writing a Provider for "Standard Algorithm"
> > for
> > > >> > > >> > > > TrustStoreManager (like our needs) and see what you find
> > > >> since
> > > >> > > only
> > > >> > > >> > > >
> > > >> > > >> > > > writing
> > > >> > > >> > > >
> > > >> > > >> > > > samples can bring out the complexity/challenges we face.
> > > >> > > >> > > >
> > > >> > > >> > > > Thanks
> > > >> > > >> > > > Maulin
> > > >> > > >> > > >
> > > >> > > >> > > > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <
> > > >> > > >> > > >
> > > >> > > >> > > > maulin.vasavada@gmail.
> > > >> > > >> > > >
> > > >> > > >> > > > com> wrote:
> > > >> > > >> > > >
> > > >> > > >> > > > Just to update - still working on it. Get to work only on
> > > >> and off
> > > >> > > on
> > > >> > > >> > > >
> > > >> > > >> > > > it :(
> > > >> > > >> > > >
> > > >> > > >> > > > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
> > > >> > > >> > > >
> > > >> > > >> > > > maulin.vasavada@gmail.com>
> > > >> > > >> > > >
> > > >> > > >> > > > wrote:
> > > >> > > >> > > >
> > > >> > > >> > > > Hi Harsha
> > > >> > > >> > > >
> > > >> > > >> > > > Let me try to write samples and will let you know.
> > > >> > > >> > > >
> > > >> > > >> > > > Thanks
> > > >> > > >> > > > Maulin
> > > >> > > >> > > >
> > > >> > > >> > > > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <
> > > >> harsha.ch@gmail.com>
> > > >> > > >> > > >
> > > >> > > >> > > > wrote:
> > > >> > > >> > > >
> > > >> > > >> > > > Hi Maulin,
> > > >> > > >> > > > With java security providers can be as custom you would
> > > >> > > >> > > >
> > > >> > > >> > > > like
> > > >> > > >> > > >
> > > >> > > >> > > > it to
> > > >> > > >> > > > be. If you only want to to implement a custom way of
> > > loading
> > > >> the
> > > >> > > >> > > >
> > > >> > > >> > > > keystore
> > > >> > > >> > > >
> > > >> > > >> > > > and truststore and not implement any protocol/encryption
> > > >> handling
> > > >> > > >> you
> > > >> > > >> > > >
> > > >> > > >> > > > can
> > > >> > > >> > > >
> > > >> > > >> > > > leave them empty and no need to implement. Have you
> > looked
> > > >> into
> > > >> > > the
> > > >> > > >> > > >
> > > >> > > >> > > > links I
> > > >> > > >> > > >
> > > >> > > >> > > > pasted before?
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> > > >> > >
> > > >> > > >> >
> > > >> > > >>
> > > >> > >
> > > >>
> > >
> > spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> > > >> > > >> > > >
> > > >> > > >> > > > java
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > >> > > >> > > >
> > spiffe-security-provider/src/main/java/spiffe/api/provider/
> > > >> > > >> > > > SpiffeTrustManager.java <http://spiffetrustmanager.java/
> > >
> > > >> > > <http://
> > > >> > > >> > > > spiffetrustmanager.java/>
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> > > >> > >
> > > >> > > >> >
> > > >> > > >>
> > > >> > >
> > > >>
> > >
> > spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > > >> > > >> > > >
> > > >> > > >> > > > java
> > > >> > > >> > > >
> > > >> > > >> > > > Can you please tell me which methods are too complex in
> > > >> above to
> > > >> > > >> > > >
> > > >> > > >> > > > implement
> > > >> > > >> > > >
> > > >> > > >> > > > or unnecessary? You are changing anything in SSL/TLS
> > > >> > > implementations
> > > >> > > >> > > > provided by
> > > >> > > >> > > >
> > > >> > > >> > > > All of the implementations delegating the checks to the
> > > >> default
> > > >> > > >> > > > implementation anyway.
> > > >> > > >> > > > Spire agent is an example, its nothing but a GRPC server
> > > >> listening
> > > >> > > >> > > >
> > > >> > > >> > > > on a
> > > >> > > >> > > >
> > > >> > > >> > > > unix domain socket . Above code is making a RPC call to
> > the
> > > >> local
> > > >> > > >> > > >
> > > >> > > >> > > > daemon
> > > >> > > >> > > >
> > > >> > > >> > > > to
> > > >> > > >> > > > get the certificate and keys. The mechanics are pretty
> > much
> > > >> same
> > > >> > > as
> > > >> > > >> > > >
> > > >> > > >> > > > what
> > > >> > > >> > > >
> > > >> > > >> > > > you are asking for.
> > > >> > > >> > > >
> > > >> > > >> > > > Thanks,
> > > >> > > >> > > > Harsha
> > > >> > > >> > > >
> > > >> > > >> > > > On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
> > > >> > > >> > > >
> > > >> > > >> > > > maulin.vasavada@gmail.com>
> > > >> > > >> > > >
> > > >> > > >> > > > wrote:
> > > >> > > >> > > >
> > > >> > > >> > > > Imagine a scenario like - We know we have a custom KMS
> > and
> > > >> as a
> > > >> > > >> > > >
> > > >> > > >> > > > Kafka
> > > >> > > >> > > >
> > > >> > > >> > > > owner
> > > >> > > >> > > >
> > > >> > > >> > > > we want to comply to using that KMS source to load
> > > >> keys/certs. As
> > > >> > > >> > > >
> > > >> > > >> > > > a
> > > >> > > >> > > >
> > > >> > > >> > > > Kafka
> > > >> > > >> > > >
> > > >> > > >> > > > owner we know how to integrate with KMS but doesn't
> > > >> necessarily
> > > >> > > >> > > >
> > > >> > > >> > > > have
> > > >> > > >> > > >
> > > >> > > >> > > > to
> > > >> > > >> > > >
> > > >> > > >> > > > know anything about cipher suites, algorithms, and
> > SSL/TLS
> > > >> > > >> > > >
> > > >> > > >> > > > implementation.
> > > >> > > >> > > >
> > > >> > > >> > > > Going the Provider way requires to know lot more than we
> > > >> should,
> > > >> > > >> > > >
> > > >> > > >> > > > isn't it?
> > > >> > > >> > > >
> > > >> > > >> > > > Not that we would have concern/shy-away knowing those
> > > >> details -
> > > >> > > >> > > >
> > > >> > > >> > > > but
> > > >> > > >> > > >
> > > >> > > >> > > > if we
> > > >> > > >> > > >
> > > >> > > >> > > > don't have to - why should we?
> > > >> > > >> > > >
> > > >> > > >> > > > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
> > > >> > > >> > > >
> > > >> > > >> > > > maulin.vasavada@gmail.com>
> > > >> > > >> > > >
> > > >> > > >> > > > wrote:
> > > >> > > >> > > >
> > > >> > > >> > > > Hi Harsha
> > > >> > > >> > > >
> > > >> > > >> > > > We don't have spire (or similar) agents and we do not
> > have
> > > >> > > >> > > >
> > > >> > > >> > > > keys/certs
> > > >> > > >> > > >
> > > >> > > >> > > > locally on any brokers. To elaborate more on my previous
> > > >> email,
> > > >> > > >> > > >
> > > >> > > >> > > > I agree that Java security Providers are used in much
> > > broader
> > > >> > > >> > > >
> > > >> > > >> > > > sense
> > > >> > > >> > > >
> > > >> > > >> > > > - to
> > > >> > > >> > > >
> > > >> > > >> > > > have a particular implementation of an algorithm, use
> > > >> specific
> > > >> > > >> > > >
> > > >> > > >> > > > cipher
> > > >> > > >> > > >
> > > >> > > >> > > > suites for SSL , OR in our current team's case have a
> > > >> > > >> > > >
> > > >> > > >> > > > particular
> > > >> > > >> > > >
> > > >> > > >> > > > way to
> > > >> > > >> > > >
> > > >> > > >> > > > leverage pre-generated SSL sessions. However, the scope
> > of
> > > >> our
> > > >> > > >> > > >
> > > >> > > >> > > > KIP
> > > >> > > >> > > >
> > > >> > > >> > > > (486)
> > > >> > > >> > > >
> > > >> > > >> > > > is
> > > >> > > >> > > >
> > > >> > > >> > > > much restricted than that. We merely intend to provide a
> > > >> custom
> > > >> > > >> > > > keystore/truststore for our SSL connections and not
> > really
> > > >> worry
> > > >> > > >> > > >
> > > >> > > >> > > > about
> > > >> > > >> > > >
> > > >> > > >> > > > underlying specific SSL/TLS implementation. This
> > simplifies
> > > >> it
> > > >> > > >> > > >
> > > >> > > >> > > > a
> > > >> > > >> > > >
> > > >> > > >> > > > lot for
> > > >> > > >> > > >
> > > >> > > >> > > > us to keep the concerns separate and clear.
> > > >> > > >> > > >
> > > >> > > >> > > > I feel our approach is more complimentary such that it
> > > allows
> > > >> > > >> > > >
> > > >> > > >> > > > for
> > > >> > > >> > > >
> > > >> > > >> > > > using
> > > >> > > >> > > >
> > > >> > > >> > > > keystores of choice while retaining the flexibility to
> > use
> > > >> any
> > > >> > > >> > > > underlying/available Provider for actually making the SSL
> > > >> call.
> > > >> > > >> > > >
> > > >> > > >> > > > We agree with KIP-492's approach based on Providers (and
> > > >> Java's
> > > >> > > >> > > > recommendation), but also strongly believe that our
> > > approach
> > > >> can
> > > >> > > >> > > >
> > > >> > > >> > > > compliment
> > > >> > > >> > > >
> > > >> > > >> > > > it very effectively for reasons explained above.
> > > >> > > >> > > >
> > > >> > > >> > > > Thanks
> > > >> > > >> > > > Maulin
> > > >> > > >> > > >
> > > >> > > >> > > > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <
> > > >> > > >> > > >
> > > >> > > >> > > > kafka@harsha.io
> > > >> > > >> > > >
> > > >> > > >> > > > wrote:
> > > >> > > >> > > >
> > > >> > > >> > > > Hi Maulin,
> > > >> > > >> > > >
> > > >> > > >> > > > On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
> > > >> > > >> > > >
> > > >> > > >> > > > maulin.vasavada@gmail.
> > > >> > > >> > > >
> > > >> > > >> > > > com>
> > > >> > > >> > > > wrote:
> > > >> > > >> > > >
> > > >> > > >> > > > Hi Harsha
> > > >> > > >> > > >
> > > >> > > >> > > > The reason we rejected the SslProvider route is that - we
> > > >> > > >> > > >
> > > >> > > >> > > > only
> > > >> > > >> > > >
> > > >> > > >> > > > needed
> > > >> > > >> > > >
> > > >> > > >> > > > a
> > > >> > > >> > > >
> > > >> > > >> > > > custom way to load keys/certs. Not touch any policy that
> > > >> > > >> > > >
> > > >> > > >> > > > existing
> > > >> > > >> > > >
> > > >> > > >> > > > Providers
> > > >> > > >> > > >
> > > >> > > >> > > > govern like SunJSSE Provider.
> > > >> > > >> > > >
> > > >> > > >> > > > We have exactly the same requirements to load certs and
> > > keys
> > > >> > > >> > > >
> > > >> > > >> > > > through
> > > >> > > >> > > >
> > > >> > > >> > > > spire
> > > >> > > >> > > >
> > > >> > > >> > > > agent. We used security.provider to do that exactly. I am
> > > not
> > > >> > > >> > > >
> > > >> > > >> > > > sure
> > > >> > > >> > > >
> > > >> > > >> > > > why
> > > >> > > >> > > >
> > > >> > > >> > > > you
> > > >> > > >> > > >
> > > >> > > >> > > > would be modifying any policies provided by default
> > SunJSSE
> > > >> > > >> > > >
> > > >> > > >> > > > provider.
> > > >> > > >> > > >
> > > >> > > >> > > > Can
> > > >> > > >> > > >
> > > >> > > >> > > > you give me an example of having custom provider that
> > will
> > > >> > > >> > > >
> > > >> > > >> > > > override an
> > > >> > > >> > > >
> > > >> > > >> > > > existing policy in SunJSSE provider.
> > > >> > > >> > > >
> > > >> > > >> > > > As pointed out earlier, this kip
> > > >> > > >> > > >
> > > >> > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > >> > > >> > > >
> > > >> KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> > > >> > > >> > > >
> > > >> > > >> > > > allows
> > > >> > > >> > > > you to load security.provider through config.
> > > >> > > >> > > > Take a look at the examples I gave before
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> > > >> > >
> > > >> > > >> >
> > > >> > > >>
> > > >> > >
> > > >>
> > >
> > spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > > >> > > >> > > >
> > > >> > > >> > > > java
> > > >> > > >> > > >
> > > >> > > >> > > > It registers KeyManagerFactory and TrustManagerFactory
> > and
> > > >> > > >> > > >
> > > >> > > >> > > > Keystore
> > > >> > > >> > > >
> > > >> > > >> > > > algorithm.
> > > >> > > >> > > > Implement your custom way of loading Keystore in here
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> > > >> > >
> > > >> > > >> >
> > > >> > > >>
> > > >> > >
> > > >>
> > >
> > spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> > > >> > > >> > > >
> > > >> > > >> > > > java
> > > >> > > >> > > >
> > > >> > > >> > > > and Trust manager like here
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > >> > > >> > > >
> > spiffe-security-provider/src/main/java/spiffe/api/provider/
> > > >> > > >> > > > SpiffeTrustManager.java <http://spiffetrustmanager.java/
> > >
> > > >> > > <http://
> > > >> > > >> > > > spiffetrustmanager.java/>
> > > >> > > >> > > >
> > > >> > > >> > > > In your Kafka client you can set the security.provider to
> > > >> your
> > > >> > > >> > > >
> > > >> > > >> > > > custom
> > > >> > > >> > > >
> > > >> > > >> > > > implementation and with this fix
> > > >> > > >> > > > https://issues.apache.org/jira/browse/KAFKA-8191 you can
> > > set
> > > >> > > >> > > > keyManagerAlgorigthm and trustManagerAlgorithm configs.
> > > >> > > >> > > >
> > > >> > > >> > > > All of this is in your clients and broker side and do not
> > > >> need
> > > >> > > >> > > >
> > > >> > > >> > > > to
> > > >> > > >> > > >
> > > >> > > >> > > > touch
> > > >> > > >> > > >
> > > >> > > >> > > > any
> > > >> > > >> > > > policy changes at JVM level. You'll register the
> > providers
> > > in
> > > >> > > >> > > >
> > > >> > > >> > > > the
> > > >> > > >> > > >
> > > >> > > >> > > > priority
> > > >> > > >> > > >
> > > >> > > >> > > > order and can still have SunJSSE provider and have your
> > > >> custom
> > > >> > > >> > > >
> > > >> > > >> > > > provider
> > > >> > > >> > > >
> > > >> > > >> > > > to
> > > >> > > >> > > >
> > > >> > > >> > > > implement the key and trust managers.
> > > >> > > >> > > >
> > > >> > > >> > > > The ask here is different than KIP-492. We don't have any
> > > >> need
> > > >> > > >> > > >
> > > >> > > >> > > > to
> > > >> > > >> > > >
> > > >> > > >> > > > modify/specify the algorithm parameter. Does that make
> > > sense?
> > > >> > > >> > > >
> > > >> > > >> > > > The ask in KIP is introducing new interfaces where the
> > > KIP's
> > > >> > > >> > > > goal/motivation can be achieved through the
> > > security.provider
> > > >> > > >> > > >
> > > >> > > >> > > > and
> > > >> > > >> > > >
> > > >> > > >> > > > we
> > > >> > > >> > > >
> > > >> > > >> > > > worked
> > > >> > > >> > > > on similar goal without touching any Keystore or
> > Truststore
> > > >> > > >> > > >
> > > >> > > >> > > > interfaces.
> > > >> > > >> > > >
> > > >> > > >> > > > My advise is against changing or introducing new
> > interfaces
> > > >> > > >> > > >
> > > >> > > >> > > > when
> > > >> > > >> > > >
> > > >> > > >> > > > it can
> > > >> > > >> > > >
> > > >> > > >> > > > work through security.provider.
> > > >> > > >> > > >
> > > >> > > >> > > > Thanks,
> > > >> > > >> > > > Harsha
> > > >> > > >> > > >
> > > >> > > >> > > > Thanks
> > > >> > > >> > > >
> > > >> > > >> > > > Maulin
> > > >> > > >> > > >
> > > >> > > >> > > > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
> > > >> > > >> > > >
> > > >> > > >> > > > kafka@harsha.io>
> > > >> > > >> > > >
> > > >> > > >> > > > wrote:
> > > >> > > >> > > >
> > > >> > > >> > > > In your KIP you added security. provider as rejected
> > > >> > > >> > > >
> > > >> > > >> > > > alternative
> > > >> > > >> > > >
> > > >> > > >> > > > and
> > > >> > > >> > > >
> > > >> > > >> > > > specified "its not the correct way". Do you mind
> > explaining
> > > >> > > >> > > >
> > > >> > > >> > > > why
> > > >> > > >> > > >
> > > >> > > >> > > > its
> > > >> > > >> > > >
> > > >> > > >> > > > not? I
> > > >> > > >> > > >
> > > >> > > >> > > > didn't find any evidence in Java docs to say so. Contrary
> > > to
> > > >> > > >> > > >
> > > >> > > >> > > > your
> > > >> > > >> > > >
> > > >> > > >> > > > statement
> > > >> > > >> > > >
> > > >> > > >> > > > it does say in the java docs
> > > >> > > >> > > > " However, please note that a provider can be used to
> > > >> > > >> > > >
> > > >> > > >> > > > implement
> > > >> > > >> > > >
> > > >> > > >> > > > any
> > > >> > > >> > > >
> > > >> > > >> > > > security service in Java that uses a pluggable
> > architecture
> > > >> > > >> > > >
> > > >> > > >> > > > with
> > > >> > > >> > > >
> > > >> > > >> > > > a
> > > >> > > >> > > >
> > > >> > > >> > > > choice
> > > >> > > >> > > >
> > > >> > > >> > > > of implementations that fit underneath."
> > > >> > > >> > > >
> > > >> > > >> > > > Java Security Providers have been used by other projects
> > to
> > > >> > > >> > > >
> > > >> > > >> > > > provide
> > > >> > > >> > > >
> > > >> > > >> > > > such
> > > >> > > >> > > >
> > > >> > > >> > > > integration . I am not sure if you looked into Spiffe
> > > >> > > >> > > >
> > > >> > > >> > > > project to
> > > >> > > >> > > >
> > > >> > > >> > > > efficiently distribute certificates but here is an
> > example
> > > of
> > > >> > > >> > > >
> > > >> > > >> > > > Java
> > > >> > > >> > > >
> > > >> > > >> > > > provider
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> > > >> > >
> > > >> > > >> >
> > > >> > > >>
> > > >> > >
> > > >>
> > >
> > spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > > >> > > >> > > >
> > > >> > > >> > > > java which
> > > >> > > >> > > > obtains certificates from local daemons.
> > > >> > > >> > > > These integrations are being used in Tomcat, Jetty etc..
> > We
> > > >> > > >> > > >
> > > >> > > >> > > > are
> > > >> > > >> > > >
> > > >> > > >> > > > also
> > > >> > > >> > > >
> > > >> > > >> > > > using
> > > >> > > >> > > >
> > > >> > > >> > > > Security provider to do the same in our Kafka clusters.
> > So
> > > >> > > >> > > >
> > > >> > > >> > > > unless I
> > > >> > > >> > > >
> > > >> > > >> > > > see
> > > >> > > >> > > >
> > > >> > > >> > > > more evidence why security.provider doesn't work for you
> > > >> > > >> > > >
> > > >> > > >> > > > adding
> > > >> > > >> > > >
> > > >> > > >> > > > new
> > > >> > > >> > > >
> > > >> > > >> > > > interfaces while there exists more cleaner way of
> > achieving
> > > >> > > >> > > >
> > > >> > > >> > > > the
> > > >> > > >> > > >
> > > >> > > >> > > > goals
> > > >> > > >> > > >
> > > >> > > >> > > > of
> > > >> > > >> > > >
> > > >> > > >> > > > this KIP is unnecessary and breaks the well known
> > security
> > > >> > > >> > > >
> > > >> > > >> > > > interfaces
> > > >> > > >> > > >
> > > >> > > >> > > > provided by Java itself.
> > > >> > > >> > > >
> > > >> > > >> > > > Thanks,
> > > >> > > >> > > > Harsha
> > > >> > > >> > > >
> > > >> > > >> > > > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
> > > >> > > >> > > >
> > > >> > > >> > > > kafka@harsha.io
> > > >> > > >> > > >
> > > >> > > >> > > > wrote:
> > > >> > > >> > > >
> > > >> > > >> > > > Hi Maulin,
> > > >> > > >> > > > Not sure if you looked at my previous replies. This
> > > >> > > >> > > >
> > > >> > > >> > > > changes
> > > >> > > >> > > >
> > > >> > > >> > > > are not required as there is already security Provider to
> > > do
> > > >> > > >> > > >
> > > >> > > >> > > > what you
> > > >> > > >> > > >
> > > >> > > >> > > > are
> > > >> > > >> > > >
> > > >> > > >> > > > proposing. This KIP
> > > >> > > >> > > >
> > > >> > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> > > >> > > >> > > >
> > > >> > > >> > > > also
> > > >> > > >> > > >
> > > >> > > >> > > > addresses easy registration of such providers.
> > > >> > > >> > > >
> > > >> > > >> > > > Thanks,
> > > >> > > >> > > > Harsha
> > > >> > > >> > > >
> > > >> > > >> > > > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
> > > >> > > >> > > >
> > > >> > > >> > > > <maulin.vasavada@gmail.
> > > >> > > >> > > >
> > > >> > > >> > > > com> wrote:
> > > >> > > >> > > >
> > > >> > > >> > > > Bump! Can somebody please review this?
> > > >> > > >> > > >
> > > >> > > >> > > > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
> > > >> > > >> > > >
> > > >> > > >> > > > maulin.vasavada@gmail.com>
> > > >> > > >> > > >
> > > >> > > >> > > > wrote:
> > > >> > > >> > > >
> > > >> > > >> > > > Bump! Can somebody please review this?
> > > >> > > >> > > >
> > > >> > > >> > > > --
> > > >> > > >> > > > Thanks,
> > > >> > > >> > > > M.Sai Sandeep
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> > > >> > >
> > > >> > > >> >
> > > >> > > >>
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Rajini Sivaram <ra...@gmail.com>.
I would expect SslEngineBuilder interface to look something like this,
perhaps with some tweaking:
public interface SslEngineBuilder extends Configurable, Closeable {
Set<String> reconfigurableConfigs();
boolean shouldBeRebuilt(Map<String, Object> nextConfigs);
SSLEngine createSslEngine(Mode mode, String peerHost, int
peerPort, String endpointIdentification);
}
The existing SslEngineBuilder class would be renamed and will implement
this interface. Loading of keystore/truststore will be in SslEngineBuilder
as it is now. The method `shouldBeRebuilt()` will validate configs during
reconfiguration and decide if reconfiguration is required because keystore
or truststore changed. SslFactory.reconfigurableConfigs() will return
SslEngineBuilder.reconfigurableConfigs() as well including any custom
configs of SslEngineBuilder, so no other changes will be required when we
eventually support custom SSL configs.
We don't want to make SslFactory the pluggable class since that contains
validation logic for SSL engines. Everything that we want to customise is
contained in SslEngineBuilder. Basically custom SslEngineBuilder will
validate custom configs during reconfiguration and create SSLEngine.
SslFactory will continue to perform validation of SSLEngines and this will
not be customizable. SslEngineBuilder will not be reconfigurable, instead
we create a new builder as we do now to avoid having to deal with
thread-safety and atomicity of updates. We could consider using a public
Reconfigurable interface as the pluggable interface for consistency, but I
think we would still want to create a new Builder on reconfiguration and
retain non-pluggable SSL engine validation in SslFactory.
On Fri, Aug 30, 2019 at 10:21 PM Maulin Vasavada <ma...@gmail.com>
wrote:
> Looking at SslFactory and SslEngineBuilder I feel the responsibilities are
> not clear. Both has public method for createSSLEngine for example. I feel
> the SslEngineBuilder was created to move out lot of code but it is not
> necessarily a public class (e.g. I don't think anybody calling
> SslEngineBuilder separately without SslFactory in between). I am currently
> inclined toward what Celement is suggesting - having pluggable SslFactory.
>
> Let me do this - let me refactor SslFactory and SslEngineBuilder and review
> what I can come up with you guys. Let us see if we can address all the
> objections raised previously for KIP-383's iterations. I'll need sometime
> though. Let me try to do it by next of next week.
>
> Thanks
> Maulin
>
> On Fri, Aug 30, 2019 at 12:25 PM Pellerin, Clement <
> Clement_Pellerin@ibi.com>
> wrote:
>
> > What is your solution to the objection that killed the second iteration
> of
> > KIP-383?
> > Mainly, how do you support validation of reconfiguration requests that
> > involve new custom properties implemented by the pluggable factory?
> > Custom properties do not exist yet, but that is very legitimate thing to
> > design for the future.
> >
> > That's why I favor a pluggable SslFactory instead of an SslEngineBuilder
> > factory.
> >
> > -----Original Message-----
> > From: Maulin Vasavada [mailto:maulin.vasavada@gmail.com]
> > Sent: Friday, August 30, 2019 3:07 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and
> > TrustStore
> >
> > +1 for making SslEngineBuilder configurable upon more thoughts.
> >
> > However, in the abstraction and default implementation we should make
> sure
> > when we do have a requirement to plugin custom key/trust store people
> don't
> > have to write lot more code which may not be related to it.
> >
> > Having said that, does this mean, we resurrect KIP-383 and update it with
> > latest context and go from there?
> >
> > We are willing to take up that work for making it configurable.
> >
> > Thanks
> > Maulin
> >
> >
> >
> >
> >
> > On Fri, Aug 30, 2019 at 10:34 AM Maulin Vasavada <
> > maulin.vasavada@gmail.com>
> > wrote:
> >
> > > Why don't we make SSLEngineBuilder code delegate the whole Key/Trust
> > store
> > > initialization to the interfaces we are proposing? Default
> implementation
> > > for those key/trust store loader interfaces will be "file based"
> loading
> > vs
> > > if somebody wants to customize any of it they can.
> > >
> > > Would that make sense?
> > >
> > > On Fri, Aug 30, 2019 at 10:03 AM Colin McCabe <cm...@apache.org>
> > wrote:
> > >
> > >> +1 for making SslEngineBuilder configurable. This would give
> > >> implementers a lot more flexibility-- to use key distribution methods
> > that
> > >> were not files, for example.
> > >>
> > >> best,
> > >> Colin
> > >>
> > >>
> > >> On Fri, Aug 30, 2019, at 02:03, Rajini Sivaram wrote:
> > >> > Just to make sure we are on the same page - KIP-383 was written
> before
> > >> > the
> > >> > code was refactored. The refactoring addressed some of the concerns
> of
> > >> > KIP-383. My suggestion was to make SslEngineBuilder configurable.
> The
> > >> > default implementation of this pluggable class would be
> > >> >
> > >>
> >
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/ssl/SslEngineBuilder.java
> > >> .
> > >> > That should give an idea of the size of the configurable part that a
> > >> > custom
> > >> > class needs to implement. A large part of that is about loading
> > >> > keystore/truststore. I agree it has slightly more code than KIP-486
> > >> > proposes, but since it lets you customize creation of SSLEngine, it
> > >> > would
> > >> > address every possible scenario.
> > >> >
> > >> > Thoughts?
> > >> >
> > >> >
> > >> > On Fri, Aug 30, 2019 at 2:02 AM Maulin Vasavada <
> > >> maulin.vasavada@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > I thought about it more. I feel that no matter how we refactor the
> > >> code
> > >> > > (with or without KIP-383 integrated), ultimately the need of
> > >> customizing
> > >> > > loading for keys and certs will still remain. Whenever that need
> > >> arises we
> > >> > > might end up thinking about the solution suggested by our KIP-486.
> > >> Hence
> > >> > > regardless of the other KIPs and configurations "if we do need to
> > >> customize
> > >> > > loading of keys/certs, we will need the code changes suggested by
> > this
> > >> > > KIP".
> > >> > >
> > >> > > Let me know what you guys think.
> > >> > >
> > >> > > Harsha, we are working on changing the interfaces for key/trust
> > store
> > >> > > loaders with Certificate and PrivateKey objects. Will probably be
> > >> able to
> > >> > > update it later today or tomorrow.
> > >> > >
> > >> > > Thanks
> > >> > > Maulin
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Thu, Aug 29, 2019 at 2:30 PM Maulin Vasavada <
> > >> maulin.vasavada@gmail.com
> > >> > > >
> > >> > > wrote:
> > >> > >
> > >> > > > On that, I actually looked at KIP-383 before briefly. However,
> > that
> > >> > > > sounded like lot of changes suggested.
> > >> > > >
> > >> > > > One "key" thing we have to keep in mind is - IF we need lot of
> > >> > > > customization Kafka already allows you to use your SslProvider
> via
> > >> > > > ssl.providers or the changes done by KIP-492 and
> > >> > > > SSLContext.getInstance(protocol, provider) call allows us to
> > return
> > >> the
> > >> > > > SSLContext with "ALL" the details we would like to customize.
> > Hence
> > >> I am
> > >> > > > not sure that customization suggested by KIP-383 would be worth
> > the
> > >> > > effort.
> > >> > > > We also have similar SSLContext customization outside of Kafka.
> > >> > > >
> > >> > > > Thanks
> > >> > > > Maulin
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On Thu, Aug 29, 2019 at 12:47 PM Pellerin, Clement <
> > >> > > > Clement_Pellerin@ibi.com> wrote:
> > >> > > >
> > >> > > >> KIP-383 in its present form was vetoed because it was not
> > possible
> > >> to
> > >> > > add
> > >> > > >> validation of custom properties in a future KIP. The solution
> to
> > >> that is
> > >> > > >> the first proposal I wrote for KIP-383 which made the whole
> > >> SslFactory
> > >> > > >> pluggable. That first solution was also vetoed hence the
> > deadlock.
> > >> > > >>
> > >> > > >> Replacing the whole factory was a much nicer solution. It was
> > >> vetoed
> > >> > > >> because doing this almost invariably meant the replacement lost
> > >> all the
> > >> > > >> complex validation code in the default SslFactory.
> > >> > > >>
> > >> > > >> My current idea is to extract the validation code into another
> > >> public
> > >> > > API
> > >> > > >> that SslFactory would call. I did not look at the newly
> > refactored
> > >> code
> > >> > > and
> > >> > > >> I did not study how to do this yet. KIP-383 was not popular at
> > the
> > >> time
> > >> > > and
> > >> > > >> designing a new solution is a lot of work.
> > >> > > >>
> > >> > > >> Is there interest from 3 binding voters for something like
> this?
> > >> > > >>
> > >> > > >> -----Original Message-----
> > >> > > >> From: Rajini Sivaram [mailto:rajinisivaram@gmail.com]
> > >> > > >> Sent: Thursday, August 29, 2019 2:57 PM
> > >> > > >> To: dev
> > >> > > >> Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore
> and
> > >> > > >> TrustStore
> > >> > > >>
> > >> > > >> Hi Maulin,
> > >> > > >>
> > >> > > >> In SSL scenarios, I imagine security providers introduced by
> > >> KIP-492 are
> > >> > > >> likely to be most useful when you want to use third party
> > >> providers. The
> > >> > > >> biggest advantage of the config from that KIP is that you don't
> > >> need to
> > >> > > >> write much code to integrate existing security providers into
> > Kafka
> > >> > > >> brokers
> > >> > > >> or clients. As I understand it, KIP-486 is a more convenient
> > >> option for
> > >> > > >> the
> > >> > > >> specific problem of loading keystores/truststores differently.
> It
> > >> can be
> > >> > > >> achieved in theory with KIP-492, but KIP-486 is a much simpler
> > >> option
> > >> > > for
> > >> > > >> this case.
> > >> > > >>
> > >> > > >> My concern about KIP-486 is that it introduces yet another
> > >> interface
> > >> > > into
> > >> > > >> our already complex security code, while only solving one
> > >> particular use
> > >> > > >> case. Have you looked at
> > >> > > >>
> > >> > > >>
> > >> > >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-383%3A++Pluggable+interface+for+SSL+Factory
> > >> > > >> ?
> > >> > > >> The goal was to make
> > >> > > >> org.apache.kafka.common.security.ssl.SslEngineBuilder
> pluggable.
> > >> > > >> The code has already been refactored by Colin after that KIP
> was
> > >> > > written,
> > >> > > >> making it easier to implement KIP-383. This should enable you
> to
> > >> load
> > >> > > your
> > >> > > >> keystores and truststores differently. Using a pluggable
> > >> > > SslEngineBuilder
> > >> > > >> will also solve several other use cases at the same time.
> KIP-383
> > >> hasn't
> > >> > > >> been voted through yet, but perhaps you could take a look and
> we
> > >> could
> > >> > > >> revive that instead if it solves your use case as well?
> > >> > > >>
> > >> > > >> Regards,
> > >> > > >>
> > >> > > >> Rajini
> > >> > > >>
> > >> > > >>
> > >> > > >> On Thu, Aug 29, 2019 at 6:42 PM Maulin Vasavada <
> > >> > > >> maulin.vasavada@gmail.com>
> > >> > > >> wrote:
> > >> > > >>
> > >> > > >> > Hi Harsha
> > >> > > >> >
> > >> > > >> > Thank you. Appreciate your time and support on this. Let me
> go
> > >> back do
> > >> > > >> some
> > >> > > >> > more research and get back to you on the KeyStore interface
> > part.
> > >> > > >> > Basically, if we return certs and keys in the interface then
> > >> Kafka
> > >> > > code
> > >> > > >> > will have to build KeyStore object - which is also
> reasonable.
> > >> > > >> >
> > >> > > >> > Thanks
> > >> > > >> > Maulin
> > >> > > >> >
> > >> > > >> > On Thu, Aug 29, 2019 at 10:01 AM Harsha Chintalapani <
> > >> kafka@harsha.io
> > >> > > >
> > >> > > >> > wrote:
> > >> > > >> >
> > >> > > >> > > Hi Maulin,
> > >> > > >> > > Use cases are clear now. I am +1 for
> > moving
> > >> > > >> forward
> > >> > > >> > > with the discussions on having such configurable option for
> > >> users.
> > >> > > But
> > >> > > >> > the
> > >> > > >> > > interfaces is proposed doesn't look right to me. We are
> still
> > >> > > talking
> > >> > > >> > about
> > >> > > >> > > keystore interfaces. Given keystore's are used as
> filebased
> > >> way of
> > >> > > >> > > transporting certificates I am not sure it will help the
> rest
> > >> of the
> > >> > > >> > > user-base.
> > >> > > >> > > In short, I am +1 on the KIP's motivation
> > >> and only
> > >> > > >> have
> > >> > > >> > > questions around returning keystores instead of returning
> > >> certs,
> > >> > > >> private
> > >> > > >> > > keys etc. . If others in the community are ok with such
> > >> interface we
> > >> > > >> can
> > >> > > >> > > move forward.
> > >> > > >> > >
> > >> > > >> > > Thanks,
> > >> > > >> > > Harsha
> > >> > > >> > >
> > >> > > >> > >
> > >> > > >> > > On Wed, Aug 28, 2019 at 1:51 PM, Maulin Vasavada <
> > >> > > >> > > maulin.vasavada@gmail.com>
> > >> > > >> > > wrote:
> > >> > > >> > >
> > >> > > >> > > > Hi Harsha
> > >> > > >> > > >
> > >> > > >> > > > As we synced-up offline on this topic, we hope you don't
> > >> have any
> > >> > > >> more
> > >> > > >> > > > clarifications that you are seeking. If that is the case,
> > >> can you
> > >> > > >> > please
> > >> > > >> > > > help us move this forward and discuss what changes you
> > would
> > >> > > expect
> > >> > > >> on
> > >> > > >> > > the
> > >> > > >> > > > KIP design in order to make it valuable contribution?
> > >> > > >> > > >
> > >> > > >> > > > Just FYI - we verified our primary design change with the
> > >> author
> > >> > > of
> > >> > > >> > Sun's
> > >> > > >> > > > X509 Trustmanager implementation and the outcome is that
> > >> what we
> > >> > > are
> > >> > > >> > > > proposing makes sense at the heart of it - "Instead of
> > >> writing
> > >> > > >> > > TrustManager
> > >> > > >> > > > just plugin the Trust store". We are open to discuss
> > >> additional
> > >> > > >> changes
> > >> > > >> > > > that you/anybody else would like to see on the
> > functionality
> > >> > > >> however.
> > >> > > >> > > >
> > >> > > >> > > > Thanks
> > >> > > >> > > > Maulin
> > >> > > >> > > >
> > >> > > >> > > > On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada <
> > >> > > >> > > maulin.vasavada@gmail.com>
> > >> > > >> > > > wrote:
> > >> > > >> > > >
> > >> > > >> > > > Hi Harsha
> > >> > > >> > > >
> > >> > > >> > > > Any response on my question? I feel this KIP is worth
> > >> > > accommodating.
> > >> > > >> > Your
> > >> > > >> > > > help is much appreciated.
> > >> > > >> > > >
> > >> > > >> > > > Thanks
> > >> > > >> > > > Maulin
> > >> > > >> > > >
> > >> > > >> > > > On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada <
> > >> > > >> > maulin.vasavada@gmail.
> > >> > > >> > > > com> wrote:
> > >> > > >> > > >
> > >> > > >> > > > Hi Harsha
> > >> > > >> > > >
> > >> > > >> > > > I've examined the SPIFFE provider more and have one
> > question
> > >> -
> > >> > > >> > > >
> > >> > > >> > > > If SPIFFE didn't have a need to do checkSpiffeId() call
> at
> > >> the
> > >> > > below
> > >> > > >> > > > location, would you really still write the Provider? *OR*
> > >> Would
> > >> > > you
> > >> > > >> > just
> > >> > > >> > > > use TrustManagerFactory.init(KeyStore) signature to pass
> > the
> > >> > > >> KeyStore
> > >> > > >> > > from
> > >> > > >> > > > set of certs returned by spiffeIdManager.
> > getTrustedCerts()?
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> >
> > >> > >
> > >>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > >> > > >> > > > provider/CertificateUtils.java#L100
> > >> > > >> > > >
> > >> > > >> > > > /**
> > >> > > >> > > >
> > >> > > >> > > > * Validates that the SPIFFE ID is present and matches the
> > >> SPIFFE
> > >> > > ID
> > >> > > >> > > > configured in
> > >> > > >> > > > * the java.security property ssl.spiffe.accept
> > >> > > >> > > > *
> > >> > > >> > > > * If the authorized spiffe ids list is empty any spiffe
> id
> > is
> > >> > > >> > authorized
> > >> > > >> > > > *
> > >> > > >> > > > * @param chain an array of X509Certificate that contains
> > the
> > >> > > Peer's
> > >> > > >> > SVID
> > >> > > >> > > > to be validated
> > >> > > >> > > > * @throws CertificateException when either the
> certificates
> > >> > > doesn't
> > >> > > >> > have
> > >> > > >> > > a
> > >> > > >> > > > SPIFFE ID or the SPIFFE ID is not authorized
> > >> > > >> > > > */
> > >> > > >> > > > static void checkSpiffeId(X509Certificate[] chain) throws
> > >> > > >> > > > CertificateException {
> > >> > > >> > > >
> > >> > > >> > > > Thanks
> > >> > > >> > > > Maulin
> > >> > > >> > > >
> > >> > > >> > > > On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani <
> > >> > > >> kafka@harsha.io>
> > >> > > >> > > > wrote:
> > >> > > >> > > >
> > >> > > >> > > > Maulin,
> > >> > > >> > > > The code parts you are pointing are specific for Spiffe
> and
> > >> if
> > >> > > >> > > > you are talking about validate method which uses PKIX
> check
> > >> like
> > >> > > any
> > >> > > >> > > other
> > >> > > >> > > > provider does.
> > >> > > >> > > > If you want to default to SunJSSE everywhere you can do
> so
> > by
> > >> > > >> > delegating
> > >> > > >> > > > the calls in these methods to SunJSSE provider.
> > >> > > >> > > >
> > >> > > >> > > > TrustManagerFactory tmf = TrustManagerFactory
> > >> > > >> > > >
> .getInstance(TrustManagerFactory.getDefaultAlgorithm());and
> > >> use
> > >> > > >> > > > tmf.chekServerTrusted()
> > >> > > >> > > > or use
> > >> > > >> > > > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> > >> > > >> > > > TrustManagerFactory.html#getInstance(java.lang.String)if
> > you
> > >> want
> > >> > > a
> > >> > > >> > > > specific provider.
> > >> > > >> > > >
> > >> > > >> > > > -Harsha
> > >> > > >> > > >
> > >> > > >> > > > On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada <
> > >> > > >> > maulin.vasavada@gmail.
> > >> > > >> > > > com>
> > >> > > >> > > > wrote:
> > >> > > >> > > >
> > >> > > >> > > > Okay, so I take that you guys agree that I have to write
> a
> > >> > > 'custom'
> > >> > > >> > > > algorithm and a provider to make it work , correct?
> > >> > > >> > > >
> > >> > > >> > > > Now, for Harsha's comment "Here the 'Custom' Algorithm is
> > >> not an
> > >> > > >> > > > implementation per say , ..." , I diagree. You can refer
> to
> > >> > > https://
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> >
> > >> > > >>
> > >> > >
> > >>
> > github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/
> > >> > > >> > > >
> > >> > > >> > > > SpiffeTrustManager.java#L91 <
> > >> http://spiffetrustmanager.java/#L91>
> > >> > > >> and
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> >
> > >> > >
> > >>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > >> > > >> > > >
> > >> > > >> > > > provider/CertificateUtils.java#L100
> > >> > > >> > > >
> > >> > > >> > > > "that code" is the customization you have for the custom
> > way
> > >> to
> > >> > > >> check
> > >> > > >> > > > something on top of regular checks. That method is NOT
> > doing
> > >> > > custom
> > >> > > >> > > > truststore loading. It is validating/verifying something
> in
> > >> the
> > >> > > >> > > >
> > >> > > >> > > > "custom"
> > >> > > >> > > >
> > >> > > >> > > > way with spiffeId.
> > >> > > >> > > > I bet that without that you won't have a need of the
> custom
> > >> > > >> algorithm
> > >> > > >> > > >
> > >> > > >> > > > in
> > >> > > >> > > >
> > >> > > >> > > > the first place.
> > >> > > >> > > >
> > >> > > >> > > > Let me know if you agree to this.
> > >> > > >> > > >
> > >> > > >> > > > Thanks
> > >> > > >> > > > Maulin
> > >> > > >> > > >
> > >> > > >> > > > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <
> > >> mprsai@gmail.com>
> > >> > > >> > > >
> > >> > > >> > > > wrote:
> > >> > > >> > > >
> > >> > > >> > > > Hi Maulin, thanks for the discussion. As Harsha pointed
> > out,
> > >> to
> > >> > > use
> > >> > > >> the
> > >> > > >> > > > KIP492, you need to create a new provider, register a
> *new*
> > >> custom
> > >> > > >> > > > algorithm for your keymanager and trustmanager factory
> > >> > > >> implementations.
> > >> > > >> > > > After this, the kafka server configuration can be done as
> > >> given
> > >> > > >> below
> > >> > > >> > > >
> > >> > > >> > > > # Register the provider class with custom algorithm, say
> > >> CUSTOM
> > >> > > >> > > >
> > >> > > >> > > > security.
> > >> > > >> > > >
> > >> > > >> > > > provider.classes=com.company.security.CustomProvider
> > >> > > >> > > > <http://provider.classes
> > >> =com.company.security.customprovider/>
> > >> > > >> > > > <http://security.provider.classes
> > >> > > >> > > >
> > >> > > >> > > > =com.company.security.customprovider/>
> > >> > > >> > > >
> > >> > > >> > > > # Register the keymanager and trustmanager algorithms
> > >> > > >> > > > # These algorithms indicate that the Keymanager and
> > >> Trustmanagers
> > >> > > >> > > > registered under the algorithm "CUSTOM" needs to be used
> > >> > > >> > > > ssl.trustmanager.algorithm=CUSTOM
> > >> > > >> > > > ssl.keymanager.algorithm=CUSTOM
> > >> > > >> > > >
> > >> > > >> > > > And the customprovider looks like this...
> > >> > > >> > > >
> > >> > > >> > > > public class CustomProvider extends Provider {
> > >> > > >> > > > public CustomProvider() {
> > >> > > >> > > > super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and
> > >> > > TrustStore");
> > >> > > >> > > > super.put("KeyManagerFactory.CUSTOM",
> > >> "customKeyManagerFactory");
> > >> > > >> > > > super.put("TrustManagerFactory.CUSTOM",
> > >> > > >> > > > "customTrustManagerFactory");
> > >> > > >> > > > }
> > >> > > >> > > > }
> > >> > > >> > > >
> > >> > > >> > > > The PR for this is in review and can be found here:
> > >> > > >> > > >
> > >> > > >> > > > https://github.com/
> > >> > > >> > > >
> > >> > > >> > > > apache/kafka/pull/7090
> > >> > > >> > > > This PR includes the fixed insertProviderAt function
> call.
> > >> > > >> > > >
> > >> > > >> > > > On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani <
> > >> > > >> kafka@harsha.io>
> > >> > > >> > > > wrote:
> > >> > > >> > > >
> > >> > > >> > > > Answers are in-line
> > >> > > >> > > >
> > >> > > >> > > > On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada <
> > >> > > >> > > >
> > >> > > >> > > > maulin.vasavada@gmail.
> > >> > > >> > > >
> > >> > > >> > > > com
> > >> > > >> > > >
> > >> > > >> > > > wrote:
> > >> > > >> > > >
> > >> > > >> > > > Hi Colin
> > >> > > >> > > >
> > >> > > >> > > > When I refer to "standard" or "custom" algorithms I am
> > >> following
> > >> > > >> Java
> > >> > > >> > > > security Provider Terminology. You can refer to
> > >> > > >> > > >
> > >> https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> > >> > > >> > > > StandardNames.html#TrustManagerFactory link I provided
> > >> earlier in
> > >> > > >> the
> > >> > > >> > > > emails. It says PKIX is the default Algorithm for
> > >> > > >> TrustManagerFactory.
> > >> > > >> > > >
> > >> > > >> > > > 1. For SPIFFE, I am not sure why you are saying 'it does
> > not
> > >> > > >> implement
> > >> > > >> > > > custom algorithms' because the following file clearly
> > >> indicates
> > >> > > >> that it
> > >> > > >> > > > does use custom algorithm-
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> >
> > >> > >
> > >>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > >> > > >> > > >
> > >> > > >> > > > provider/SpiffeProvider.java#L17
> > >> > > >> > > >
> > >> > > >> > > > Algorithm value:
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> >
> > >> > >
> > >>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > >> > > >> > > >
> > >> > > >> > > > provider/SpiffeProviderConstants.java#L6
> > >> > > >> > > >
> > >> > > >> > > > @Harsha do you want to chime in since you use that
> > provider?
> > >> > > >> > > >
> > >> > > >> > > > Here the "Custom" Algorithm is not an implementation per
> > say
> > >> ,
> > >> > > >> rather
> > >> > > >> > > >
> > >> > > >> > > > used
> > >> > > >> > > >
> > >> > > >> > > > to invoke the custom trust store factory and key manager
> > >> factory.
> > >> > > >> You
> > >> > > >> > > >
> > >> > > >> > > > are
> > >> > > >> > > >
> > >> > > >> > > > not moving away from "standard" alogrithms that are
> > >> available.
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> >
> > >> > >
> > >>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > >> > > >> > > >
> > >> > > >> > > > provider/SpiffeTrustManager.java
> > >> > > >> > > >
> > >> > > >> > > > As you can see it delegates all the calls of verification
> > of
> > >> > > >> > > >
> > >> > > >> > > > certificate
> > >> > > >> > > >
> > >> > > >> > > > to
> > >> > > >> > > >
> > >> > > >> > > > the default implementation available.
> > >> > > >> > > > So in our implementation we still use PKIX to verify the
> > >> > > certificate
> > >> > > >> > > > chain. So you are not losing anything here and Spiffe is
> > not
> > >> > > >> > > >
> > >> > > >> > > > reimplementing
> > >> > > >> > > >
> > >> > > >> > > > the verification process.
> > >> > > >> > > >
> > >> > > >> > > > 2. I already mentioned in my 3rd point, in my previous
> > post,
> > >> why
> > >> > > >> using
> > >> > > >> > > >
> > >> > > >> > > > ssl.provider does NOT work. I updated KIP-486 in
> "rejected
> > >> > > >> > > >
> > >> > > >> > > > alternatives"
> > >> > > >> > > >
> > >> > > >> > > > also why ssl.provider does not work.
> > >> > > >> > > >
> > >> > > >> > > > As mentioned before , provider is the right way to go. I
> am
> > >> still
> > >> > > >> not
> > >> > > >> > > > understanding the gap is.
> > >> > > >> > > > If I understand correctly your argument is , provider is
> > >> going to
> > >> > > >> ask
> > >> > > >> > > >
> > >> > > >> > > > to
> > >> > > >> > > >
> > >> > > >> > > > implement a custom algorithm.
> > >> > > >> > > > My answer to that is , no you are not re-implementing the
> > >> > > algorithm.
> > >> > > >> > > >
> > >> > > >> > > > Please
> > >> > > >> > > >
> > >> > > >> > > > check the above link , TrustManager implementation it
> > >> delegates
> > >> > > the
> > >> > > >> > > >
> > >> > > >> > > > calls
> > >> > > >> > > >
> > >> > > >> > > > back. There is no need to implement your own here.
> > >> > > >> > > >
> > >> > > >> > > > 3. Security.insertProviderAt() comments were based on
> > >> assumption
> > >> > > if
> > >> > > >> > > >
> > >> > > >> > > > KIP-492
> > >> > > >> > > >
> > >> > > >> > > > changes are done and we use that mechanism to configure
> > >> providers
> > >> > > >> > > >
> > >> > > >> > > > instead
> > >> > > >> > > >
> > >> > > >> > > > of ssl.provider configuration.
> > >> > > >> > > >
> > >> > > >> > > > KIP-492 has patch available and going through review.
> > >> > > >> > > >
> > >> > > >> > > > Can you read my all the points, I mentioned in my
> previous
> > >> post,
> > >> > > >> very
> > >> > > >> > > >
> > >> > > >> > > > carefully? I am covering all the aspects in explaining. I
> > am
> > >> open
> > >> > > to
> > >> > > >> > > >
> > >> > > >> > > > still
> > >> > > >> > > >
> > >> > > >> > > > discuss more to clarify any doubts.
> > >> > > >> > > >
> > >> > > >> > > > "3. If we just use existing ssl.provider kafka
> > configuration
> > >> then
> > >> > > >> our
> > >> > > >> > > > provider will be used in SSLContext.getInstance(protocol,
> > >> > > provider)
> > >> > > >> > > >
> > >> > > >> > > > call
> > >> > > >> > > >
> > >> > > >> > > > in
> > >> > > >> > > >
> > >> > > >> > > > SslFactory.java <http://sslfactory.java/> <
> > >> > > http://sslfactory.java/>
> > >> > > >> <
> > >> > > >> > > > http://sslfactory.java/>
> > >> > > >> > > >
> > >> > > >> > > > and
> > >> > > >> > > >
> > >> > > >> > > > if our provider does not have
> > >> > > >> > > > implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it
> > >> breaks
> > >> > > (we
> > >> > > >> > > >
> > >> > > >> > > > tested
> > >> > > >> > > >
> > >> > > >> > > > it). Example: In MyProvider sample above you see that I
> > >> didn't add
> > >> > > >> > > > SSLContext.TLSv1 as
> > >> > > >> > > > "Service+Algorithm" and that didn't work for me. In
> SPIFFE
> > >> > > provider
> > >> > > >> you
> > >> > > >> > > > don't have this challenge since you are planning to
> bypass
> > >> > > >> ssl.provider
> > >> > > >> > > >
> > >> > > >> > > > as
> > >> > > >> > > >
> > >> > > >> > > > you mention in the KIP-492."
> > >> > > >> > > >
> > >> > > >> > > > Yes here you need to pass the protocol that your
> > >> > > >> > > >
> > >> > > >> > > > KeyManager/TrustManager
> > >> > > >> > > >
> > >> > > >> > > > registered with and in no way its deviating from TLS RFC
> > >> spec.
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> >
> > >> > >
> > >>
> https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/
> > >> > > >> > > >
> > >> > > >> > > > javax/net/ssl/SSLContext.java#L134
> > >> > > >> > > >
> > >> > > >> > > > My suggestion here is for you to implement a simple
> > Security
> > >> > > >> Provider
> > >> > > >> > > >
> > >> > > >> > > > as
> > >> > > >> > > >
> > >> > > >> > > > you did before and register a Algorithm. You can use the
> > >> existing
> > >> > > >> > > > implementation in SpiffeProvider and plug in changes
> where
> > >> you
> > >> > > need
> > >> > > >> to
> > >> > > >> > > > retrieve the certificates from by making RPC call.
> > >> > > >> > > >
> > >> > > >> > > > Run an end-to-end test with Kafka broker coming up with
> > your
> > >> > > >> provider
> > >> > > >> > > > making calls to RPC call. You do need to pass the "custom
> > >> > > algorithm"
> > >> > > >> > > >
> > >> > > >> > > > that
> > >> > > >> > > >
> > >> > > >> > > > you registered in your key manager to invoke the
> provider.
> > I
> > >> think
> > >> > > >> your
> > >> > > >> > > > concern is this approach is replacing the existing known
> > >> > > >> ciphersuites
> > >> > > >> > > >
> > >> > > >> > > > and
> > >> > > >> > > >
> > >> > > >> > > > certificate validation provided by java. Which its not.
> > >> > > >> > > >
> > >> > > >> > > > Now test the TLS connection you can do so via openssl
> > >> -s_client
> > >> > > >> options
> > >> > > >> > > >
> > >> > > >> > > > to
> > >> > > >> > > >
> > >> > > >> > > > test if everything else is working.
> > >> > > >> > > >
> > >> > > >> > > > I am happy to share configs that we used if you are
> > >> interested.
> > >> > > >> > > >
> > >> > > >> > > > Thanks,
> > >> > > >> > > > Harsha
> > >> > > >> > > >
> > >> > > >> > > > Thanks
> > >> > > >> > > > Maulin
> > >> > > >> > > >
> > >> > > >> > > > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <
> > >> cmccabe@apache.org>
> > >> > > >> > > >
> > >> > > >> > > > wrote:
> > >> > > >> > > >
> > >> > > >> > > > Hi Maulin,
> > >> > > >> > > >
> > >> > > >> > > > A lot of JSSE providers don't implement custom
> algorithms.
> > >> Spire
> > >> > > is
> > >> > > >> a
> > >> > > >> > > >
> > >> > > >> > > > good
> > >> > > >> > > >
> > >> > > >> > > > example of a JSSE provider that doesn't, and yet is still
> > >> useful
> > >> > > to
> > >> > > >> > > >
> > >> > > >> > > > many
> > >> > > >> > > >
> > >> > > >> > > > people. Your JSSE provider can work fine even if it
> doesn't
> > >> > > >> implement a
> > >> > > >> > > > custom algorithm.
> > >> > > >> > > >
> > >> > > >> > > > Maybe I'm missing something, but I don't understand the
> > >> discussion
> > >> > > >> of
> > >> > > >> > > > Security.insertProviderAt() that you included.
> > >> SslEngineBuilder
> > >> > > >> doesn't
> > >> > > >> > > >
> > >> > > >> > > > use
> > >> > > >> > > >
> > >> > > >> > > > that API to get the security provider. Instead, it calls
> > >> > > >> > > > "SSLContext.getInstance(protocol, provider)", where
> > provider
> > >> is
> > >> > > the
> > >> > > >> > > >
> > >> > > >> > > > name
> > >> > > >> > > >
> > >> > > >> > > > of the provider.
> > >> > > >> > > >
> > >> > > >> > > > best,
> > >> > > >> > > > Colin
> > >> > > >> > > >
> > >> > > >> > > > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
> > >> > > >> > > >
> > >> > > >> > > > On top of everything above I feel strongly to add the 4th
> > >> point
> > >> > > >> which
> > >> > > >> > > >
> > >> > > >> > > > is
> > >> > > >> > > >
> > >> > > >> > > > based on Java APIs for
> TrustManagerFactory.init(KeyStore) (
> > >> > > >> > > >
> > >> > > >> > > > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> > >> > > >> > > > TrustManagerFactory.html#init(java.security.KeyStore
> > >> > > >> > > > <http://java.security.keystore/>
> > >> > > >> > > > <http://java.security.keystore/>)
> > >> > > >> > > > )
> > >> > > >> > > >
> > >> > > >> > > > and KeyManagerFactory.init(KeyStore, char[]) (
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > >
> > >> > > >> >
> > >> > > >>
> > >> > >
> > >>
> >
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory
> > >> > > >> > > > .
> > >> > > >> > > >
> > >> > > >> > > > html#init(java.security.KeyStore <
> > >> http://java.security.keystore/>
> > >> > > >> > > <http://
> > >> > > >> > > > java.security.keystore/
> > >> > > >> > > > ,%20char[])
> > >> > > >> > > >
> > >> > > >> > > > ).
> > >> > > >> > > >
> > >> > > >> > > > 4. The above APIs are intended to support providing
> > >> "trust/key
> > >> > > >> > > >
> > >> > > >> > > > material"
> > >> > > >> > > >
> > >> > > >> > > > from the user without having to write their own
> > >> > > >> > > >
> > >> > > >> > > > TrustManager/KeyManagers.
> > >> > > >> > > >
> > >> > > >> > > > To quote from the TrustManagerFactory.init()'s
> > documentation
> > >> > > >> > > >
> > >> > > >> > > > "Initializes
> > >> > > >> > > >
> > >> > > >> > > > this factory with a source of certificate authorities and
> > >> related
> > >> > > >> trust
> > >> > > >> > > > material."
> > >> > > >> > > > To quote from the KeyManagerFactory.init()'s
> documentation
> > >> > > >> "Initializes
> > >> > > >> > > > this factory with a source of key material."
> > >> > > >> > > >
> > >> > > >> > > > Based on this it is clear that there is a flexibility
> > >> provided by
> > >> > > >> Java
> > >> > > >> > > >
> > >> > > >> > > > to
> > >> > > >> > > >
> > >> > > >> > > > to enable developers to provide the required trust/key
> > >> material
> > >> > > >> loaded
> > >> > > >> > > >
> > >> > > >> > > > from
> > >> > > >> > > >
> > >> > > >> > > > "anywhere" without requiring them to write custom
> provider
> > OR
> > >> > > >> trust/key
> > >> > > >> > > > managers. This same flexibility is reflected in Kafka
> code
> > >> also
> > >> > > >> where
> > >> > > >> > > >
> > >> > > >> > > > it
> > >> > > >> > > >
> > >> > > >> > > > loads the trust/keys from a local file and doesn't
> require
> > >> > > writing a
> > >> > > >> > > > Provider necessarily. If we do NOT have a custom
> algorithm,
> > >> it
> > >> > > makes
> > >> > > >> > > >
> > >> > > >> > > > less
> > >> > > >> > > >
> > >> > > >> > > > sense to write a Provider.
> > >> > > >> > > >
> > >> > > >> > > > Thanks
> > >> > > >> > > > Maulin
> > >> > > >> > > >
> > >> > > >> > > > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
> > >> > > >> > > >
> > >> > > >> > > > maulin.vasavada@gmail.com>
> > >> > > >> > > >
> > >> > > >> > > > wrote:
> > >> > > >> > > >
> > >> > > >> > > > Hi Harsha/Colin
> > >> > > >> > > >
> > >> > > >> > > > I did the sample with a custom Provider for
> > >> TrustStoreManager and
> > >> > > >> tried
> > >> > > >> > > > using ssl.provider Kafka config AND the way KIP-492 is
> > >> suggesting
> > >> > > >> (by
> > >> > > >> > > > adding Provider programmatically instead of relying on
> > >> > > >> > > >
> > >> > > >> > > > ssl.provider+java.
> > >> > > >> > > >
> > >> > > >> > > > security. The below sample is followed by my detailed
> > >> findings.
> > >> > > I'll
> > >> > > >> > > > appreciate if you can go through it carefully and see
> > >> > > >> > > >
> > >> > > >> > > > if you
> > >> > > >> > > >
> > >> > > >> > > > see my point.
> > >> > > >> > > >
> > >> > > >> > > > package providertest;
> > >> > > >> > > >
> > >> > > >> > > > import java.security.Provider <
> > >> http://java.security.provider/>
> > >> > > >> > <http://
> > >> > > >> > > > java.security.provider/>
> > >> > > >> > > >
> > >> > > >> > > > <http://
> > >> > > >> > > >
> > >> > > >> > > > java.security.provider/>;
> > >> > > >> > > >
> > >> > > >> > > > public class MyProvider extends Provider {
> > >> > > >> > > >
> > >> > > >> > > > private static final String name = "MyProvider"; private
> > >> static
> > >> > > >> double
> > >> > > >> > > > version = 1.0d;
> > >> > > >> > > > private static String info = "Maulin's SSL Provider
> > >> v"+version;
> > >> > > >> > > >
> > >> > > >> > > > public MyProvider() {
> > >> > > >> > > > super(name, version, info);
> > >> > > >> > > > this.put("TrustManagerFactory.PKIX",
> > >> > > >> > > >
> > >> > > >> > > > "providertest.MyTrustManagerFactory");
> > >> > > >> > > >
> > >> > > >> > > > }
> > >> > > >> > > > }
> > >> > > >> > > >
> > >> > > >> > > > *Details:*
> > >> > > >> > > >
> > >> > > >> > > > KIP-492 documents that it will use Security.addProvider()
> > >> assuming
> > >> > > >> it
> > >> > > >> > > >
> > >> > > >> > > > will
> > >> > > >> > > >
> > >> > > >> > > > add it as position '0' which is not a correct assumption.
> > The
> > >> > > >> > > > addProvider()'s documentation says it will add it to the
> > last
> > >> > > >> available
> > >> > > >> > > > position. You may want to correct that to say
> > >> > > >> > > > Security.insertProviderAt(provider, 1).
> > >> > > >> > > >
> > >> > > >> > > > Now coming back to our specific discussion,
> > >> > > >> > > >
> > >> > > >> > > > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence
> > when
> > >> you
> > >> > > add
> > >> > > >> > > >
> > >> > > >> > > > that
> > >> > > >> > > >
> > >> > > >> > > > provider in the provider list via Security.addProvider()
> > the
> > >> > > >> position
> > >> > > >> > > >
> > >> > > >> > > > where
> > >> > > >> > > >
> > >> > > >> > > > it gets added doesn't matter (even if you don't end up
> > >> adding it
> > >> > > as
> > >> > > >> > > >
> > >> > > >> > > > first
> > >> > > >> > > >
> > >> > > >> > > > entry) since that is the ONLY provider for SPIFFE
> specific
> > >> > > algorithm
> > >> > > >> > > >
> > >> > > >> > > > you
> > >> > > >> > > >
> > >> > > >> > > > might have.
> > >> > > >> > > >
> > >> > > >> > > > We do *not* have custom algorithm for Key/Trust
> > StoreMangers.
> > >> > > Which
> > >> > > >> > > >
> > >> > > >> > > > means
> > >> > > >> > > >
> > >> > > >> > > > we have to use X509, PKIX etc "Standard Algorithms" ((
> > >> > > >> > > >
> > >> > > >> > > >
> > >> https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> > >> > > >> > > > StandardNames.html
> > >> > > >> > > > ))
> > >> > > >> > > >
> > >> > > >> > > > in our provider to override the TrustStoreManager (see my
> > >> sample
> > >> > > >> code)
> > >> > > >> > > >
> > >> > > >> > > > and
> > >> > > >> > > >
> > >> > > >> > > > KeyStoreManger and KeyManager. This creates another
> > challenge
> > >> > > >> > > >
> > >> > > >> > > > mentioned in
> > >> > > >> > > >
> > >> > > >> > > > the below point.
> > >> > > >> > > >
> > >> > > >> > > > 2. In order to make our Provider for loading custom
> > >> TrustStore
> > >> > > >> work, we
> > >> > > >> > > > have to add the provider as 'first' in the list since
> there
> > >> are
> > >> > > >> others
> > >> > > >> > > >
> > >> > > >> > > > with
> > >> > > >> > > >
> > >> > > >> > > > the same algorithm.
> > >> > > >> > > >
> > >> > > >> > > > However, the programatic way of adding provider
> > >> > > >> > > > (Security.insertProviderAt()) is *not* deterministic for
> > >> ordering
> > >> > > >> since
> > >> > > >> > > > different code can call the method for a different
> provider
> > >> and
> > >> > > >> > > >
> > >> > > >> > > > depending
> > >> > > >> > > >
> > >> > > >> > > > upon the order of the call our provider can be first or
> > >> pushed
> > >> > > down
> > >> > > >> the
> > >> > > >> > > > list. This can happen very well in any client application
> > >> using
> > >> > > >> Kafka.
> > >> > > >> > > >
> > >> > > >> > > > This
> > >> > > >> > > >
> > >> > > >> > > > is specially problematic for a case when you want to
> > >> guarantee
> > >> > > order
> > >> > > >> > > >
> > >> > > >> > > > for a
> > >> > > >> > > >
> > >> > > >> > > > Provider having "Standard Algorithms".
> > >> > > >> > > >
> > >> > > >> > > > If we add our provider in java.security file that
> > definitely
> > >> > > >> guarantees
> > >> > > >> > > > the order(unless somebody calls removeProvider() which is
> > >> less
> > >> > > >> > > >
> > >> > > >> > > > likely). But
> > >> > > >> > > >
> > >> > > >> > > > if we add our provider in java.security file it will
> defeat
> > >> the
> > >> > > >> > > >
> > >> > > >> > > > purpose of
> > >> > > >> > > >
> > >> > > >> > > > the KIP-492.
> > >> > > >> > > >
> > >> > > >> > > > In the gist - Apache Kafka must not rely on
> > >> "un-deterministic"
> > >> > > >> method
> > >> > > >> > > >
> > >> > > >> > > > to
> > >> > > >> > > >
> > >> > > >> > > > rely on Provider ordering.
> > >> > > >> > > >
> > >> > > >> > > > 3. If we just use existing ssl.provider kafka
> configuration
> > >> then
> > >> > > our
> > >> > > >> > > > provider will be used in SSLContext.getInstance(protocol,
> > >> > > provider)
> > >> > > >> > > >
> > >> > > >> > > > call in
> > >> > > >> > > >
> > >> > > >> > > > SslFactory.java <http://sslfactory.java/> <
> > >> > > http://sslfactory.java/>
> > >> > > >> <
> > >> > > >> > > > http://sslfactory.java/>
> > >> > > >> > > >
> > >> > > >> > > > and
> > >> > > >> > > >
> > >> > > >> > > > if our provider does not have implementation for
> > >> > > >> > > > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks
> > >> > > >> > > >
> > >> > > >> > > > (we
> > >> > > >> > > >
> > >> > > >> > > > tested it). Example:
> > >> > > >> > > >
> > >> > > >> > > > In
> > >> > > >> > > >
> > >> > > >> > > > MyProvider sample above you see that I didn't add
> > >> SSLContext.TLSv1
> > >> > > >> as
> > >> > > >> > > > "Service+Algorithm" and that didn't work for me. In
> SPIFFE
> > >> > > provider
> > >> > > >> you
> > >> > > >> > > > don't have this challenge since you are planning to
> bypass
> > >> > > >> > > >
> > >> > > >> > > > ssl.provider as
> > >> > > >> > > >
> > >> > > >> > > > you mention in the KIP-492.
> > >> > > >> > > >
> > >> > > >> > > > *Overall summary,*
> > >> > > >> > > >
> > >> > > >> > > > 1. Any provider based mechanisms- a) existing
> ssl.provider
> > >> and
> > >> > > >> > > >
> > >> > > >> > > > b)KIP-492,
> > >> > > >> > > >
> > >> > > >> > > > for loading key/trust store using "Standard Algorithms"
> do
> > >> not
> > >> > > work
> > >> > > >> > > >
> > >> > > >> > > > 2. Approach suggested in our KIP-486 works without any
> > issue
> > >> and
> > >> > > it
> > >> > > >> is
> > >> > > >> > > > *not* our context specific solve
> > >> > > >> > > >
> > >> > > >> > > > 3. Based on above we feel KIP-492 and KIP-486 are
> > >> complimentary
> > >> > > >> changes
> > >> > > >> > > > and not contradicting or redundent.
> > >> > > >> > > >
> > >> > > >> > > > If you want we can do a joint session somehow to walk
> > >> through the
> > >> > > >> > > >
> > >> > > >> > > > sample I
> > >> > > >> > > >
> > >> > > >> > > > have and various experiments I did. I would encourage you
> > to
> > >> do
> > >> > > >> similar
> > >> > > >> > > > exercise by writing a Provider for "Standard Algorithm"
> for
> > >> > > >> > > > TrustStoreManager (like our needs) and see what you find
> > >> since
> > >> > > only
> > >> > > >> > > >
> > >> > > >> > > > writing
> > >> > > >> > > >
> > >> > > >> > > > samples can bring out the complexity/challenges we face.
> > >> > > >> > > >
> > >> > > >> > > > Thanks
> > >> > > >> > > > Maulin
> > >> > > >> > > >
> > >> > > >> > > > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <
> > >> > > >> > > >
> > >> > > >> > > > maulin.vasavada@gmail.
> > >> > > >> > > >
> > >> > > >> > > > com> wrote:
> > >> > > >> > > >
> > >> > > >> > > > Just to update - still working on it. Get to work only on
> > >> and off
> > >> > > on
> > >> > > >> > > >
> > >> > > >> > > > it :(
> > >> > > >> > > >
> > >> > > >> > > > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
> > >> > > >> > > >
> > >> > > >> > > > maulin.vasavada@gmail.com>
> > >> > > >> > > >
> > >> > > >> > > > wrote:
> > >> > > >> > > >
> > >> > > >> > > > Hi Harsha
> > >> > > >> > > >
> > >> > > >> > > > Let me try to write samples and will let you know.
> > >> > > >> > > >
> > >> > > >> > > > Thanks
> > >> > > >> > > > Maulin
> > >> > > >> > > >
> > >> > > >> > > > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <
> > >> harsha.ch@gmail.com>
> > >> > > >> > > >
> > >> > > >> > > > wrote:
> > >> > > >> > > >
> > >> > > >> > > > Hi Maulin,
> > >> > > >> > > > With java security providers can be as custom you would
> > >> > > >> > > >
> > >> > > >> > > > like
> > >> > > >> > > >
> > >> > > >> > > > it to
> > >> > > >> > > > be. If you only want to to implement a custom way of
> > loading
> > >> the
> > >> > > >> > > >
> > >> > > >> > > > keystore
> > >> > > >> > > >
> > >> > > >> > > > and truststore and not implement any protocol/encryption
> > >> handling
> > >> > > >> you
> > >> > > >> > > >
> > >> > > >> > > > can
> > >> > > >> > > >
> > >> > > >> > > > leave them empty and no need to implement. Have you
> looked
> > >> into
> > >> > > the
> > >> > > >> > > >
> > >> > > >> > > > links I
> > >> > > >> > > >
> > >> > > >> > > > pasted before?
> > >> > > >> > > >
> > >> > > >> > > >
> > >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > >
> > >> > > >> >
> > >> > > >>
> > >> > >
> > >>
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> > >> > > >> > > >
> > >> > > >> > > > java
> > >> > > >> > > >
> > >> > > >> > > >
> > >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >> > > >> > > >
> spiffe-security-provider/src/main/java/spiffe/api/provider/
> > >> > > >> > > > SpiffeTrustManager.java <http://spiffetrustmanager.java/
> >
> > >> > > <http://
> > >> > > >> > > > spiffetrustmanager.java/>
> > >> > > >> > > >
> > >> > > >> > > >
> > >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > >
> > >> > > >> >
> > >> > > >>
> > >> > >
> > >>
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > >> > > >> > > >
> > >> > > >> > > > java
> > >> > > >> > > >
> > >> > > >> > > > Can you please tell me which methods are too complex in
> > >> above to
> > >> > > >> > > >
> > >> > > >> > > > implement
> > >> > > >> > > >
> > >> > > >> > > > or unnecessary? You are changing anything in SSL/TLS
> > >> > > implementations
> > >> > > >> > > > provided by
> > >> > > >> > > >
> > >> > > >> > > > All of the implementations delegating the checks to the
> > >> default
> > >> > > >> > > > implementation anyway.
> > >> > > >> > > > Spire agent is an example, its nothing but a GRPC server
> > >> listening
> > >> > > >> > > >
> > >> > > >> > > > on a
> > >> > > >> > > >
> > >> > > >> > > > unix domain socket . Above code is making a RPC call to
> the
> > >> local
> > >> > > >> > > >
> > >> > > >> > > > daemon
> > >> > > >> > > >
> > >> > > >> > > > to
> > >> > > >> > > > get the certificate and keys. The mechanics are pretty
> much
> > >> same
> > >> > > as
> > >> > > >> > > >
> > >> > > >> > > > what
> > >> > > >> > > >
> > >> > > >> > > > you are asking for.
> > >> > > >> > > >
> > >> > > >> > > > Thanks,
> > >> > > >> > > > Harsha
> > >> > > >> > > >
> > >> > > >> > > > On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
> > >> > > >> > > >
> > >> > > >> > > > maulin.vasavada@gmail.com>
> > >> > > >> > > >
> > >> > > >> > > > wrote:
> > >> > > >> > > >
> > >> > > >> > > > Imagine a scenario like - We know we have a custom KMS
> and
> > >> as a
> > >> > > >> > > >
> > >> > > >> > > > Kafka
> > >> > > >> > > >
> > >> > > >> > > > owner
> > >> > > >> > > >
> > >> > > >> > > > we want to comply to using that KMS source to load
> > >> keys/certs. As
> > >> > > >> > > >
> > >> > > >> > > > a
> > >> > > >> > > >
> > >> > > >> > > > Kafka
> > >> > > >> > > >
> > >> > > >> > > > owner we know how to integrate with KMS but doesn't
> > >> necessarily
> > >> > > >> > > >
> > >> > > >> > > > have
> > >> > > >> > > >
> > >> > > >> > > > to
> > >> > > >> > > >
> > >> > > >> > > > know anything about cipher suites, algorithms, and
> SSL/TLS
> > >> > > >> > > >
> > >> > > >> > > > implementation.
> > >> > > >> > > >
> > >> > > >> > > > Going the Provider way requires to know lot more than we
> > >> should,
> > >> > > >> > > >
> > >> > > >> > > > isn't it?
> > >> > > >> > > >
> > >> > > >> > > > Not that we would have concern/shy-away knowing those
> > >> details -
> > >> > > >> > > >
> > >> > > >> > > > but
> > >> > > >> > > >
> > >> > > >> > > > if we
> > >> > > >> > > >
> > >> > > >> > > > don't have to - why should we?
> > >> > > >> > > >
> > >> > > >> > > > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
> > >> > > >> > > >
> > >> > > >> > > > maulin.vasavada@gmail.com>
> > >> > > >> > > >
> > >> > > >> > > > wrote:
> > >> > > >> > > >
> > >> > > >> > > > Hi Harsha
> > >> > > >> > > >
> > >> > > >> > > > We don't have spire (or similar) agents and we do not
> have
> > >> > > >> > > >
> > >> > > >> > > > keys/certs
> > >> > > >> > > >
> > >> > > >> > > > locally on any brokers. To elaborate more on my previous
> > >> email,
> > >> > > >> > > >
> > >> > > >> > > > I agree that Java security Providers are used in much
> > broader
> > >> > > >> > > >
> > >> > > >> > > > sense
> > >> > > >> > > >
> > >> > > >> > > > - to
> > >> > > >> > > >
> > >> > > >> > > > have a particular implementation of an algorithm, use
> > >> specific
> > >> > > >> > > >
> > >> > > >> > > > cipher
> > >> > > >> > > >
> > >> > > >> > > > suites for SSL , OR in our current team's case have a
> > >> > > >> > > >
> > >> > > >> > > > particular
> > >> > > >> > > >
> > >> > > >> > > > way to
> > >> > > >> > > >
> > >> > > >> > > > leverage pre-generated SSL sessions. However, the scope
> of
> > >> our
> > >> > > >> > > >
> > >> > > >> > > > KIP
> > >> > > >> > > >
> > >> > > >> > > > (486)
> > >> > > >> > > >
> > >> > > >> > > > is
> > >> > > >> > > >
> > >> > > >> > > > much restricted than that. We merely intend to provide a
> > >> custom
> > >> > > >> > > > keystore/truststore for our SSL connections and not
> really
> > >> worry
> > >> > > >> > > >
> > >> > > >> > > > about
> > >> > > >> > > >
> > >> > > >> > > > underlying specific SSL/TLS implementation. This
> simplifies
> > >> it
> > >> > > >> > > >
> > >> > > >> > > > a
> > >> > > >> > > >
> > >> > > >> > > > lot for
> > >> > > >> > > >
> > >> > > >> > > > us to keep the concerns separate and clear.
> > >> > > >> > > >
> > >> > > >> > > > I feel our approach is more complimentary such that it
> > allows
> > >> > > >> > > >
> > >> > > >> > > > for
> > >> > > >> > > >
> > >> > > >> > > > using
> > >> > > >> > > >
> > >> > > >> > > > keystores of choice while retaining the flexibility to
> use
> > >> any
> > >> > > >> > > > underlying/available Provider for actually making the SSL
> > >> call.
> > >> > > >> > > >
> > >> > > >> > > > We agree with KIP-492's approach based on Providers (and
> > >> Java's
> > >> > > >> > > > recommendation), but also strongly believe that our
> > approach
> > >> can
> > >> > > >> > > >
> > >> > > >> > > > compliment
> > >> > > >> > > >
> > >> > > >> > > > it very effectively for reasons explained above.
> > >> > > >> > > >
> > >> > > >> > > > Thanks
> > >> > > >> > > > Maulin
> > >> > > >> > > >
> > >> > > >> > > > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <
> > >> > > >> > > >
> > >> > > >> > > > kafka@harsha.io
> > >> > > >> > > >
> > >> > > >> > > > wrote:
> > >> > > >> > > >
> > >> > > >> > > > Hi Maulin,
> > >> > > >> > > >
> > >> > > >> > > > On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
> > >> > > >> > > >
> > >> > > >> > > > maulin.vasavada@gmail.
> > >> > > >> > > >
> > >> > > >> > > > com>
> > >> > > >> > > > wrote:
> > >> > > >> > > >
> > >> > > >> > > > Hi Harsha
> > >> > > >> > > >
> > >> > > >> > > > The reason we rejected the SslProvider route is that - we
> > >> > > >> > > >
> > >> > > >> > > > only
> > >> > > >> > > >
> > >> > > >> > > > needed
> > >> > > >> > > >
> > >> > > >> > > > a
> > >> > > >> > > >
> > >> > > >> > > > custom way to load keys/certs. Not touch any policy that
> > >> > > >> > > >
> > >> > > >> > > > existing
> > >> > > >> > > >
> > >> > > >> > > > Providers
> > >> > > >> > > >
> > >> > > >> > > > govern like SunJSSE Provider.
> > >> > > >> > > >
> > >> > > >> > > > We have exactly the same requirements to load certs and
> > keys
> > >> > > >> > > >
> > >> > > >> > > > through
> > >> > > >> > > >
> > >> > > >> > > > spire
> > >> > > >> > > >
> > >> > > >> > > > agent. We used security.provider to do that exactly. I am
> > not
> > >> > > >> > > >
> > >> > > >> > > > sure
> > >> > > >> > > >
> > >> > > >> > > > why
> > >> > > >> > > >
> > >> > > >> > > > you
> > >> > > >> > > >
> > >> > > >> > > > would be modifying any policies provided by default
> SunJSSE
> > >> > > >> > > >
> > >> > > >> > > > provider.
> > >> > > >> > > >
> > >> > > >> > > > Can
> > >> > > >> > > >
> > >> > > >> > > > you give me an example of having custom provider that
> will
> > >> > > >> > > >
> > >> > > >> > > > override an
> > >> > > >> > > >
> > >> > > >> > > > existing policy in SunJSSE provider.
> > >> > > >> > > >
> > >> > > >> > > > As pointed out earlier, this kip
> > >> > > >> > > >
> > >> > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > >> > > >> > > >
> > >> KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> > >> > > >> > > >
> > >> > > >> > > > allows
> > >> > > >> > > > you to load security.provider through config.
> > >> > > >> > > > Take a look at the examples I gave before
> > >> > > >> > > >
> > >> > > >> > > >
> > >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > >
> > >> > > >> >
> > >> > > >>
> > >> > >
> > >>
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > >> > > >> > > >
> > >> > > >> > > > java
> > >> > > >> > > >
> > >> > > >> > > > It registers KeyManagerFactory and TrustManagerFactory
> and
> > >> > > >> > > >
> > >> > > >> > > > Keystore
> > >> > > >> > > >
> > >> > > >> > > > algorithm.
> > >> > > >> > > > Implement your custom way of loading Keystore in here
> > >> > > >> > > >
> > >> > > >> > > >
> > >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > >
> > >> > > >> >
> > >> > > >>
> > >> > >
> > >>
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> > >> > > >> > > >
> > >> > > >> > > > java
> > >> > > >> > > >
> > >> > > >> > > > and Trust manager like here
> > >> > > >> > > >
> > >> > > >> > > >
> > >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >> > > >> > > >
> spiffe-security-provider/src/main/java/spiffe/api/provider/
> > >> > > >> > > > SpiffeTrustManager.java <http://spiffetrustmanager.java/
> >
> > >> > > <http://
> > >> > > >> > > > spiffetrustmanager.java/>
> > >> > > >> > > >
> > >> > > >> > > > In your Kafka client you can set the security.provider to
> > >> your
> > >> > > >> > > >
> > >> > > >> > > > custom
> > >> > > >> > > >
> > >> > > >> > > > implementation and with this fix
> > >> > > >> > > > https://issues.apache.org/jira/browse/KAFKA-8191 you can
> > set
> > >> > > >> > > > keyManagerAlgorigthm and trustManagerAlgorithm configs.
> > >> > > >> > > >
> > >> > > >> > > > All of this is in your clients and broker side and do not
> > >> need
> > >> > > >> > > >
> > >> > > >> > > > to
> > >> > > >> > > >
> > >> > > >> > > > touch
> > >> > > >> > > >
> > >> > > >> > > > any
> > >> > > >> > > > policy changes at JVM level. You'll register the
> providers
> > in
> > >> > > >> > > >
> > >> > > >> > > > the
> > >> > > >> > > >
> > >> > > >> > > > priority
> > >> > > >> > > >
> > >> > > >> > > > order and can still have SunJSSE provider and have your
> > >> custom
> > >> > > >> > > >
> > >> > > >> > > > provider
> > >> > > >> > > >
> > >> > > >> > > > to
> > >> > > >> > > >
> > >> > > >> > > > implement the key and trust managers.
> > >> > > >> > > >
> > >> > > >> > > > The ask here is different than KIP-492. We don't have any
> > >> need
> > >> > > >> > > >
> > >> > > >> > > > to
> > >> > > >> > > >
> > >> > > >> > > > modify/specify the algorithm parameter. Does that make
> > sense?
> > >> > > >> > > >
> > >> > > >> > > > The ask in KIP is introducing new interfaces where the
> > KIP's
> > >> > > >> > > > goal/motivation can be achieved through the
> > security.provider
> > >> > > >> > > >
> > >> > > >> > > > and
> > >> > > >> > > >
> > >> > > >> > > > we
> > >> > > >> > > >
> > >> > > >> > > > worked
> > >> > > >> > > > on similar goal without touching any Keystore or
> Truststore
> > >> > > >> > > >
> > >> > > >> > > > interfaces.
> > >> > > >> > > >
> > >> > > >> > > > My advise is against changing or introducing new
> interfaces
> > >> > > >> > > >
> > >> > > >> > > > when
> > >> > > >> > > >
> > >> > > >> > > > it can
> > >> > > >> > > >
> > >> > > >> > > > work through security.provider.
> > >> > > >> > > >
> > >> > > >> > > > Thanks,
> > >> > > >> > > > Harsha
> > >> > > >> > > >
> > >> > > >> > > > Thanks
> > >> > > >> > > >
> > >> > > >> > > > Maulin
> > >> > > >> > > >
> > >> > > >> > > > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
> > >> > > >> > > >
> > >> > > >> > > > kafka@harsha.io>
> > >> > > >> > > >
> > >> > > >> > > > wrote:
> > >> > > >> > > >
> > >> > > >> > > > In your KIP you added security. provider as rejected
> > >> > > >> > > >
> > >> > > >> > > > alternative
> > >> > > >> > > >
> > >> > > >> > > > and
> > >> > > >> > > >
> > >> > > >> > > > specified "its not the correct way". Do you mind
> explaining
> > >> > > >> > > >
> > >> > > >> > > > why
> > >> > > >> > > >
> > >> > > >> > > > its
> > >> > > >> > > >
> > >> > > >> > > > not? I
> > >> > > >> > > >
> > >> > > >> > > > didn't find any evidence in Java docs to say so. Contrary
> > to
> > >> > > >> > > >
> > >> > > >> > > > your
> > >> > > >> > > >
> > >> > > >> > > > statement
> > >> > > >> > > >
> > >> > > >> > > > it does say in the java docs
> > >> > > >> > > > " However, please note that a provider can be used to
> > >> > > >> > > >
> > >> > > >> > > > implement
> > >> > > >> > > >
> > >> > > >> > > > any
> > >> > > >> > > >
> > >> > > >> > > > security service in Java that uses a pluggable
> architecture
> > >> > > >> > > >
> > >> > > >> > > > with
> > >> > > >> > > >
> > >> > > >> > > > a
> > >> > > >> > > >
> > >> > > >> > > > choice
> > >> > > >> > > >
> > >> > > >> > > > of implementations that fit underneath."
> > >> > > >> > > >
> > >> > > >> > > > Java Security Providers have been used by other projects
> to
> > >> > > >> > > >
> > >> > > >> > > > provide
> > >> > > >> > > >
> > >> > > >> > > > such
> > >> > > >> > > >
> > >> > > >> > > > integration . I am not sure if you looked into Spiffe
> > >> > > >> > > >
> > >> > > >> > > > project to
> > >> > > >> > > >
> > >> > > >> > > > efficiently distribute certificates but here is an
> example
> > of
> > >> > > >> > > >
> > >> > > >> > > > Java
> > >> > > >> > > >
> > >> > > >> > > > provider
> > >> > > >> > > >
> > >> > > >> > > >
> > >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > >
> > >> > > >> >
> > >> > > >>
> > >> > >
> > >>
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > >> > > >> > > >
> > >> > > >> > > > java which
> > >> > > >> > > > obtains certificates from local daemons.
> > >> > > >> > > > These integrations are being used in Tomcat, Jetty etc..
> We
> > >> > > >> > > >
> > >> > > >> > > > are
> > >> > > >> > > >
> > >> > > >> > > > also
> > >> > > >> > > >
> > >> > > >> > > > using
> > >> > > >> > > >
> > >> > > >> > > > Security provider to do the same in our Kafka clusters.
> So
> > >> > > >> > > >
> > >> > > >> > > > unless I
> > >> > > >> > > >
> > >> > > >> > > > see
> > >> > > >> > > >
> > >> > > >> > > > more evidence why security.provider doesn't work for you
> > >> > > >> > > >
> > >> > > >> > > > adding
> > >> > > >> > > >
> > >> > > >> > > > new
> > >> > > >> > > >
> > >> > > >> > > > interfaces while there exists more cleaner way of
> achieving
> > >> > > >> > > >
> > >> > > >> > > > the
> > >> > > >> > > >
> > >> > > >> > > > goals
> > >> > > >> > > >
> > >> > > >> > > > of
> > >> > > >> > > >
> > >> > > >> > > > this KIP is unnecessary and breaks the well known
> security
> > >> > > >> > > >
> > >> > > >> > > > interfaces
> > >> > > >> > > >
> > >> > > >> > > > provided by Java itself.
> > >> > > >> > > >
> > >> > > >> > > > Thanks,
> > >> > > >> > > > Harsha
> > >> > > >> > > >
> > >> > > >> > > > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
> > >> > > >> > > >
> > >> > > >> > > > kafka@harsha.io
> > >> > > >> > > >
> > >> > > >> > > > wrote:
> > >> > > >> > > >
> > >> > > >> > > > Hi Maulin,
> > >> > > >> > > > Not sure if you looked at my previous replies. This
> > >> > > >> > > >
> > >> > > >> > > > changes
> > >> > > >> > > >
> > >> > > >> > > > are not required as there is already security Provider to
> > do
> > >> > > >> > > >
> > >> > > >> > > > what you
> > >> > > >> > > >
> > >> > > >> > > > are
> > >> > > >> > > >
> > >> > > >> > > > proposing. This KIP
> > >> > > >> > > >
> > >> > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > >> > > >> > > >
> > >> > > >> > > >
> > >> KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> > >> > > >> > > >
> > >> > > >> > > > also
> > >> > > >> > > >
> > >> > > >> > > > addresses easy registration of such providers.
> > >> > > >> > > >
> > >> > > >> > > > Thanks,
> > >> > > >> > > > Harsha
> > >> > > >> > > >
> > >> > > >> > > > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
> > >> > > >> > > >
> > >> > > >> > > > <maulin.vasavada@gmail.
> > >> > > >> > > >
> > >> > > >> > > > com> wrote:
> > >> > > >> > > >
> > >> > > >> > > > Bump! Can somebody please review this?
> > >> > > >> > > >
> > >> > > >> > > > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
> > >> > > >> > > >
> > >> > > >> > > > maulin.vasavada@gmail.com>
> > >> > > >> > > >
> > >> > > >> > > > wrote:
> > >> > > >> > > >
> > >> > > >> > > > Bump! Can somebody please review this?
> > >> > > >> > > >
> > >> > > >> > > > --
> > >> > > >> > > > Thanks,
> > >> > > >> > > > M.Sai Sandeep
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > >
> > >> > > >> >
> > >> > > >>
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Maulin Vasavada <ma...@gmail.com>.
Looking at SslFactory and SslEngineBuilder I feel the responsibilities are
not clear. Both has public method for createSSLEngine for example. I feel
the SslEngineBuilder was created to move out lot of code but it is not
necessarily a public class (e.g. I don't think anybody calling
SslEngineBuilder separately without SslFactory in between). I am currently
inclined toward what Celement is suggesting - having pluggable SslFactory.
Let me do this - let me refactor SslFactory and SslEngineBuilder and review
what I can come up with you guys. Let us see if we can address all the
objections raised previously for KIP-383's iterations. I'll need sometime
though. Let me try to do it by next of next week.
Thanks
Maulin
On Fri, Aug 30, 2019 at 12:25 PM Pellerin, Clement <Cl...@ibi.com>
wrote:
> What is your solution to the objection that killed the second iteration of
> KIP-383?
> Mainly, how do you support validation of reconfiguration requests that
> involve new custom properties implemented by the pluggable factory?
> Custom properties do not exist yet, but that is very legitimate thing to
> design for the future.
>
> That's why I favor a pluggable SslFactory instead of an SslEngineBuilder
> factory.
>
> -----Original Message-----
> From: Maulin Vasavada [mailto:maulin.vasavada@gmail.com]
> Sent: Friday, August 30, 2019 3:07 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and
> TrustStore
>
> +1 for making SslEngineBuilder configurable upon more thoughts.
>
> However, in the abstraction and default implementation we should make sure
> when we do have a requirement to plugin custom key/trust store people don't
> have to write lot more code which may not be related to it.
>
> Having said that, does this mean, we resurrect KIP-383 and update it with
> latest context and go from there?
>
> We are willing to take up that work for making it configurable.
>
> Thanks
> Maulin
>
>
>
>
>
> On Fri, Aug 30, 2019 at 10:34 AM Maulin Vasavada <
> maulin.vasavada@gmail.com>
> wrote:
>
> > Why don't we make SSLEngineBuilder code delegate the whole Key/Trust
> store
> > initialization to the interfaces we are proposing? Default implementation
> > for those key/trust store loader interfaces will be "file based" loading
> vs
> > if somebody wants to customize any of it they can.
> >
> > Would that make sense?
> >
> > On Fri, Aug 30, 2019 at 10:03 AM Colin McCabe <cm...@apache.org>
> wrote:
> >
> >> +1 for making SslEngineBuilder configurable. This would give
> >> implementers a lot more flexibility-- to use key distribution methods
> that
> >> were not files, for example.
> >>
> >> best,
> >> Colin
> >>
> >>
> >> On Fri, Aug 30, 2019, at 02:03, Rajini Sivaram wrote:
> >> > Just to make sure we are on the same page - KIP-383 was written before
> >> > the
> >> > code was refactored. The refactoring addressed some of the concerns of
> >> > KIP-383. My suggestion was to make SslEngineBuilder configurable. The
> >> > default implementation of this pluggable class would be
> >> >
> >>
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/ssl/SslEngineBuilder.java
> >> .
> >> > That should give an idea of the size of the configurable part that a
> >> > custom
> >> > class needs to implement. A large part of that is about loading
> >> > keystore/truststore. I agree it has slightly more code than KIP-486
> >> > proposes, but since it lets you customize creation of SSLEngine, it
> >> > would
> >> > address every possible scenario.
> >> >
> >> > Thoughts?
> >> >
> >> >
> >> > On Fri, Aug 30, 2019 at 2:02 AM Maulin Vasavada <
> >> maulin.vasavada@gmail.com>
> >> > wrote:
> >> >
> >> > > I thought about it more. I feel that no matter how we refactor the
> >> code
> >> > > (with or without KIP-383 integrated), ultimately the need of
> >> customizing
> >> > > loading for keys and certs will still remain. Whenever that need
> >> arises we
> >> > > might end up thinking about the solution suggested by our KIP-486.
> >> Hence
> >> > > regardless of the other KIPs and configurations "if we do need to
> >> customize
> >> > > loading of keys/certs, we will need the code changes suggested by
> this
> >> > > KIP".
> >> > >
> >> > > Let me know what you guys think.
> >> > >
> >> > > Harsha, we are working on changing the interfaces for key/trust
> store
> >> > > loaders with Certificate and PrivateKey objects. Will probably be
> >> able to
> >> > > update it later today or tomorrow.
> >> > >
> >> > > Thanks
> >> > > Maulin
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > On Thu, Aug 29, 2019 at 2:30 PM Maulin Vasavada <
> >> maulin.vasavada@gmail.com
> >> > > >
> >> > > wrote:
> >> > >
> >> > > > On that, I actually looked at KIP-383 before briefly. However,
> that
> >> > > > sounded like lot of changes suggested.
> >> > > >
> >> > > > One "key" thing we have to keep in mind is - IF we need lot of
> >> > > > customization Kafka already allows you to use your SslProvider via
> >> > > > ssl.providers or the changes done by KIP-492 and
> >> > > > SSLContext.getInstance(protocol, provider) call allows us to
> return
> >> the
> >> > > > SSLContext with "ALL" the details we would like to customize.
> Hence
> >> I am
> >> > > > not sure that customization suggested by KIP-383 would be worth
> the
> >> > > effort.
> >> > > > We also have similar SSLContext customization outside of Kafka.
> >> > > >
> >> > > > Thanks
> >> > > > Maulin
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Thu, Aug 29, 2019 at 12:47 PM Pellerin, Clement <
> >> > > > Clement_Pellerin@ibi.com> wrote:
> >> > > >
> >> > > >> KIP-383 in its present form was vetoed because it was not
> possible
> >> to
> >> > > add
> >> > > >> validation of custom properties in a future KIP. The solution to
> >> that is
> >> > > >> the first proposal I wrote for KIP-383 which made the whole
> >> SslFactory
> >> > > >> pluggable. That first solution was also vetoed hence the
> deadlock.
> >> > > >>
> >> > > >> Replacing the whole factory was a much nicer solution. It was
> >> vetoed
> >> > > >> because doing this almost invariably meant the replacement lost
> >> all the
> >> > > >> complex validation code in the default SslFactory.
> >> > > >>
> >> > > >> My current idea is to extract the validation code into another
> >> public
> >> > > API
> >> > > >> that SslFactory would call. I did not look at the newly
> refactored
> >> code
> >> > > and
> >> > > >> I did not study how to do this yet. KIP-383 was not popular at
> the
> >> time
> >> > > and
> >> > > >> designing a new solution is a lot of work.
> >> > > >>
> >> > > >> Is there interest from 3 binding voters for something like this?
> >> > > >>
> >> > > >> -----Original Message-----
> >> > > >> From: Rajini Sivaram [mailto:rajinisivaram@gmail.com]
> >> > > >> Sent: Thursday, August 29, 2019 2:57 PM
> >> > > >> To: dev
> >> > > >> Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and
> >> > > >> TrustStore
> >> > > >>
> >> > > >> Hi Maulin,
> >> > > >>
> >> > > >> In SSL scenarios, I imagine security providers introduced by
> >> KIP-492 are
> >> > > >> likely to be most useful when you want to use third party
> >> providers. The
> >> > > >> biggest advantage of the config from that KIP is that you don't
> >> need to
> >> > > >> write much code to integrate existing security providers into
> Kafka
> >> > > >> brokers
> >> > > >> or clients. As I understand it, KIP-486 is a more convenient
> >> option for
> >> > > >> the
> >> > > >> specific problem of loading keystores/truststores differently. It
> >> can be
> >> > > >> achieved in theory with KIP-492, but KIP-486 is a much simpler
> >> option
> >> > > for
> >> > > >> this case.
> >> > > >>
> >> > > >> My concern about KIP-486 is that it introduces yet another
> >> interface
> >> > > into
> >> > > >> our already complex security code, while only solving one
> >> particular use
> >> > > >> case. Have you looked at
> >> > > >>
> >> > > >>
> >> > >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-383%3A++Pluggable+interface+for+SSL+Factory
> >> > > >> ?
> >> > > >> The goal was to make
> >> > > >> org.apache.kafka.common.security.ssl.SslEngineBuilder pluggable.
> >> > > >> The code has already been refactored by Colin after that KIP was
> >> > > written,
> >> > > >> making it easier to implement KIP-383. This should enable you to
> >> load
> >> > > your
> >> > > >> keystores and truststores differently. Using a pluggable
> >> > > SslEngineBuilder
> >> > > >> will also solve several other use cases at the same time. KIP-383
> >> hasn't
> >> > > >> been voted through yet, but perhaps you could take a look and we
> >> could
> >> > > >> revive that instead if it solves your use case as well?
> >> > > >>
> >> > > >> Regards,
> >> > > >>
> >> > > >> Rajini
> >> > > >>
> >> > > >>
> >> > > >> On Thu, Aug 29, 2019 at 6:42 PM Maulin Vasavada <
> >> > > >> maulin.vasavada@gmail.com>
> >> > > >> wrote:
> >> > > >>
> >> > > >> > Hi Harsha
> >> > > >> >
> >> > > >> > Thank you. Appreciate your time and support on this. Let me go
> >> back do
> >> > > >> some
> >> > > >> > more research and get back to you on the KeyStore interface
> part.
> >> > > >> > Basically, if we return certs and keys in the interface then
> >> Kafka
> >> > > code
> >> > > >> > will have to build KeyStore object - which is also reasonable.
> >> > > >> >
> >> > > >> > Thanks
> >> > > >> > Maulin
> >> > > >> >
> >> > > >> > On Thu, Aug 29, 2019 at 10:01 AM Harsha Chintalapani <
> >> kafka@harsha.io
> >> > > >
> >> > > >> > wrote:
> >> > > >> >
> >> > > >> > > Hi Maulin,
> >> > > >> > > Use cases are clear now. I am +1 for
> moving
> >> > > >> forward
> >> > > >> > > with the discussions on having such configurable option for
> >> users.
> >> > > But
> >> > > >> > the
> >> > > >> > > interfaces is proposed doesn't look right to me. We are still
> >> > > talking
> >> > > >> > about
> >> > > >> > > keystore interfaces. Given keystore's are used as filebased
> >> way of
> >> > > >> > > transporting certificates I am not sure it will help the rest
> >> of the
> >> > > >> > > user-base.
> >> > > >> > > In short, I am +1 on the KIP's motivation
> >> and only
> >> > > >> have
> >> > > >> > > questions around returning keystores instead of returning
> >> certs,
> >> > > >> private
> >> > > >> > > keys etc. . If others in the community are ok with such
> >> interface we
> >> > > >> can
> >> > > >> > > move forward.
> >> > > >> > >
> >> > > >> > > Thanks,
> >> > > >> > > Harsha
> >> > > >> > >
> >> > > >> > >
> >> > > >> > > On Wed, Aug 28, 2019 at 1:51 PM, Maulin Vasavada <
> >> > > >> > > maulin.vasavada@gmail.com>
> >> > > >> > > wrote:
> >> > > >> > >
> >> > > >> > > > Hi Harsha
> >> > > >> > > >
> >> > > >> > > > As we synced-up offline on this topic, we hope you don't
> >> have any
> >> > > >> more
> >> > > >> > > > clarifications that you are seeking. If that is the case,
> >> can you
> >> > > >> > please
> >> > > >> > > > help us move this forward and discuss what changes you
> would
> >> > > expect
> >> > > >> on
> >> > > >> > > the
> >> > > >> > > > KIP design in order to make it valuable contribution?
> >> > > >> > > >
> >> > > >> > > > Just FYI - we verified our primary design change with the
> >> author
> >> > > of
> >> > > >> > Sun's
> >> > > >> > > > X509 Trustmanager implementation and the outcome is that
> >> what we
> >> > > are
> >> > > >> > > > proposing makes sense at the heart of it - "Instead of
> >> writing
> >> > > >> > > TrustManager
> >> > > >> > > > just plugin the Trust store". We are open to discuss
> >> additional
> >> > > >> changes
> >> > > >> > > > that you/anybody else would like to see on the
> functionality
> >> > > >> however.
> >> > > >> > > >
> >> > > >> > > > Thanks
> >> > > >> > > > Maulin
> >> > > >> > > >
> >> > > >> > > > On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada <
> >> > > >> > > maulin.vasavada@gmail.com>
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Hi Harsha
> >> > > >> > > >
> >> > > >> > > > Any response on my question? I feel this KIP is worth
> >> > > accommodating.
> >> > > >> > Your
> >> > > >> > > > help is much appreciated.
> >> > > >> > > >
> >> > > >> > > > Thanks
> >> > > >> > > > Maulin
> >> > > >> > > >
> >> > > >> > > > On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada <
> >> > > >> > maulin.vasavada@gmail.
> >> > > >> > > > com> wrote:
> >> > > >> > > >
> >> > > >> > > > Hi Harsha
> >> > > >> > > >
> >> > > >> > > > I've examined the SPIFFE provider more and have one
> question
> >> -
> >> > > >> > > >
> >> > > >> > > > If SPIFFE didn't have a need to do checkSpiffeId() call at
> >> the
> >> > > below
> >> > > >> > > > location, would you really still write the Provider? *OR*
> >> Would
> >> > > you
> >> > > >> > just
> >> > > >> > > > use TrustManagerFactory.init(KeyStore) signature to pass
> the
> >> > > >> KeyStore
> >> > > >> > > from
> >> > > >> > > > set of certs returned by spiffeIdManager.
> getTrustedCerts()?
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> >
> >> > >
> >> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> >> > > >> > > > provider/CertificateUtils.java#L100
> >> > > >> > > >
> >> > > >> > > > /**
> >> > > >> > > >
> >> > > >> > > > * Validates that the SPIFFE ID is present and matches the
> >> SPIFFE
> >> > > ID
> >> > > >> > > > configured in
> >> > > >> > > > * the java.security property ssl.spiffe.accept
> >> > > >> > > > *
> >> > > >> > > > * If the authorized spiffe ids list is empty any spiffe id
> is
> >> > > >> > authorized
> >> > > >> > > > *
> >> > > >> > > > * @param chain an array of X509Certificate that contains
> the
> >> > > Peer's
> >> > > >> > SVID
> >> > > >> > > > to be validated
> >> > > >> > > > * @throws CertificateException when either the certificates
> >> > > doesn't
> >> > > >> > have
> >> > > >> > > a
> >> > > >> > > > SPIFFE ID or the SPIFFE ID is not authorized
> >> > > >> > > > */
> >> > > >> > > > static void checkSpiffeId(X509Certificate[] chain) throws
> >> > > >> > > > CertificateException {
> >> > > >> > > >
> >> > > >> > > > Thanks
> >> > > >> > > > Maulin
> >> > > >> > > >
> >> > > >> > > > On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani <
> >> > > >> kafka@harsha.io>
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Maulin,
> >> > > >> > > > The code parts you are pointing are specific for Spiffe and
> >> if
> >> > > >> > > > you are talking about validate method which uses PKIX check
> >> like
> >> > > any
> >> > > >> > > other
> >> > > >> > > > provider does.
> >> > > >> > > > If you want to default to SunJSSE everywhere you can do so
> by
> >> > > >> > delegating
> >> > > >> > > > the calls in these methods to SunJSSE provider.
> >> > > >> > > >
> >> > > >> > > > TrustManagerFactory tmf = TrustManagerFactory
> >> > > >> > > > .getInstance(TrustManagerFactory.getDefaultAlgorithm());and
> >> use
> >> > > >> > > > tmf.chekServerTrusted()
> >> > > >> > > > or use
> >> > > >> > > > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> >> > > >> > > > TrustManagerFactory.html#getInstance(java.lang.String)if
> you
> >> want
> >> > > a
> >> > > >> > > > specific provider.
> >> > > >> > > >
> >> > > >> > > > -Harsha
> >> > > >> > > >
> >> > > >> > > > On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada <
> >> > > >> > maulin.vasavada@gmail.
> >> > > >> > > > com>
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Okay, so I take that you guys agree that I have to write a
> >> > > 'custom'
> >> > > >> > > > algorithm and a provider to make it work , correct?
> >> > > >> > > >
> >> > > >> > > > Now, for Harsha's comment "Here the 'Custom' Algorithm is
> >> not an
> >> > > >> > > > implementation per say , ..." , I diagree. You can refer to
> >> > > https://
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> >
> >> > > >>
> >> > >
> >>
> github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/
> >> > > >> > > >
> >> > > >> > > > SpiffeTrustManager.java#L91 <
> >> http://spiffetrustmanager.java/#L91>
> >> > > >> and
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> >
> >> > >
> >> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> >> > > >> > > >
> >> > > >> > > > provider/CertificateUtils.java#L100
> >> > > >> > > >
> >> > > >> > > > "that code" is the customization you have for the custom
> way
> >> to
> >> > > >> check
> >> > > >> > > > something on top of regular checks. That method is NOT
> doing
> >> > > custom
> >> > > >> > > > truststore loading. It is validating/verifying something in
> >> the
> >> > > >> > > >
> >> > > >> > > > "custom"
> >> > > >> > > >
> >> > > >> > > > way with spiffeId.
> >> > > >> > > > I bet that without that you won't have a need of the custom
> >> > > >> algorithm
> >> > > >> > > >
> >> > > >> > > > in
> >> > > >> > > >
> >> > > >> > > > the first place.
> >> > > >> > > >
> >> > > >> > > > Let me know if you agree to this.
> >> > > >> > > >
> >> > > >> > > > Thanks
> >> > > >> > > > Maulin
> >> > > >> > > >
> >> > > >> > > > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <
> >> mprsai@gmail.com>
> >> > > >> > > >
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Hi Maulin, thanks for the discussion. As Harsha pointed
> out,
> >> to
> >> > > use
> >> > > >> the
> >> > > >> > > > KIP492, you need to create a new provider, register a *new*
> >> custom
> >> > > >> > > > algorithm for your keymanager and trustmanager factory
> >> > > >> implementations.
> >> > > >> > > > After this, the kafka server configuration can be done as
> >> given
> >> > > >> below
> >> > > >> > > >
> >> > > >> > > > # Register the provider class with custom algorithm, say
> >> CUSTOM
> >> > > >> > > >
> >> > > >> > > > security.
> >> > > >> > > >
> >> > > >> > > > provider.classes=com.company.security.CustomProvider
> >> > > >> > > > <http://provider.classes
> >> =com.company.security.customprovider/>
> >> > > >> > > > <http://security.provider.classes
> >> > > >> > > >
> >> > > >> > > > =com.company.security.customprovider/>
> >> > > >> > > >
> >> > > >> > > > # Register the keymanager and trustmanager algorithms
> >> > > >> > > > # These algorithms indicate that the Keymanager and
> >> Trustmanagers
> >> > > >> > > > registered under the algorithm "CUSTOM" needs to be used
> >> > > >> > > > ssl.trustmanager.algorithm=CUSTOM
> >> > > >> > > > ssl.keymanager.algorithm=CUSTOM
> >> > > >> > > >
> >> > > >> > > > And the customprovider looks like this...
> >> > > >> > > >
> >> > > >> > > > public class CustomProvider extends Provider {
> >> > > >> > > > public CustomProvider() {
> >> > > >> > > > super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and
> >> > > TrustStore");
> >> > > >> > > > super.put("KeyManagerFactory.CUSTOM",
> >> "customKeyManagerFactory");
> >> > > >> > > > super.put("TrustManagerFactory.CUSTOM",
> >> > > >> > > > "customTrustManagerFactory");
> >> > > >> > > > }
> >> > > >> > > > }
> >> > > >> > > >
> >> > > >> > > > The PR for this is in review and can be found here:
> >> > > >> > > >
> >> > > >> > > > https://github.com/
> >> > > >> > > >
> >> > > >> > > > apache/kafka/pull/7090
> >> > > >> > > > This PR includes the fixed insertProviderAt function call.
> >> > > >> > > >
> >> > > >> > > > On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani <
> >> > > >> kafka@harsha.io>
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Answers are in-line
> >> > > >> > > >
> >> > > >> > > > On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada <
> >> > > >> > > >
> >> > > >> > > > maulin.vasavada@gmail.
> >> > > >> > > >
> >> > > >> > > > com
> >> > > >> > > >
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Hi Colin
> >> > > >> > > >
> >> > > >> > > > When I refer to "standard" or "custom" algorithms I am
> >> following
> >> > > >> Java
> >> > > >> > > > security Provider Terminology. You can refer to
> >> > > >> > > >
> >> https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> >> > > >> > > > StandardNames.html#TrustManagerFactory link I provided
> >> earlier in
> >> > > >> the
> >> > > >> > > > emails. It says PKIX is the default Algorithm for
> >> > > >> TrustManagerFactory.
> >> > > >> > > >
> >> > > >> > > > 1. For SPIFFE, I am not sure why you are saying 'it does
> not
> >> > > >> implement
> >> > > >> > > > custom algorithms' because the following file clearly
> >> indicates
> >> > > >> that it
> >> > > >> > > > does use custom algorithm-
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> >
> >> > >
> >> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> >> > > >> > > >
> >> > > >> > > > provider/SpiffeProvider.java#L17
> >> > > >> > > >
> >> > > >> > > > Algorithm value:
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> >
> >> > >
> >> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> >> > > >> > > >
> >> > > >> > > > provider/SpiffeProviderConstants.java#L6
> >> > > >> > > >
> >> > > >> > > > @Harsha do you want to chime in since you use that
> provider?
> >> > > >> > > >
> >> > > >> > > > Here the "Custom" Algorithm is not an implementation per
> say
> >> ,
> >> > > >> rather
> >> > > >> > > >
> >> > > >> > > > used
> >> > > >> > > >
> >> > > >> > > > to invoke the custom trust store factory and key manager
> >> factory.
> >> > > >> You
> >> > > >> > > >
> >> > > >> > > > are
> >> > > >> > > >
> >> > > >> > > > not moving away from "standard" alogrithms that are
> >> available.
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> >
> >> > >
> >> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> >> > > >> > > >
> >> > > >> > > > provider/SpiffeTrustManager.java
> >> > > >> > > >
> >> > > >> > > > As you can see it delegates all the calls of verification
> of
> >> > > >> > > >
> >> > > >> > > > certificate
> >> > > >> > > >
> >> > > >> > > > to
> >> > > >> > > >
> >> > > >> > > > the default implementation available.
> >> > > >> > > > So in our implementation we still use PKIX to verify the
> >> > > certificate
> >> > > >> > > > chain. So you are not losing anything here and Spiffe is
> not
> >> > > >> > > >
> >> > > >> > > > reimplementing
> >> > > >> > > >
> >> > > >> > > > the verification process.
> >> > > >> > > >
> >> > > >> > > > 2. I already mentioned in my 3rd point, in my previous
> post,
> >> why
> >> > > >> using
> >> > > >> > > >
> >> > > >> > > > ssl.provider does NOT work. I updated KIP-486 in "rejected
> >> > > >> > > >
> >> > > >> > > > alternatives"
> >> > > >> > > >
> >> > > >> > > > also why ssl.provider does not work.
> >> > > >> > > >
> >> > > >> > > > As mentioned before , provider is the right way to go. I am
> >> still
> >> > > >> not
> >> > > >> > > > understanding the gap is.
> >> > > >> > > > If I understand correctly your argument is , provider is
> >> going to
> >> > > >> ask
> >> > > >> > > >
> >> > > >> > > > to
> >> > > >> > > >
> >> > > >> > > > implement a custom algorithm.
> >> > > >> > > > My answer to that is , no you are not re-implementing the
> >> > > algorithm.
> >> > > >> > > >
> >> > > >> > > > Please
> >> > > >> > > >
> >> > > >> > > > check the above link , TrustManager implementation it
> >> delegates
> >> > > the
> >> > > >> > > >
> >> > > >> > > > calls
> >> > > >> > > >
> >> > > >> > > > back. There is no need to implement your own here.
> >> > > >> > > >
> >> > > >> > > > 3. Security.insertProviderAt() comments were based on
> >> assumption
> >> > > if
> >> > > >> > > >
> >> > > >> > > > KIP-492
> >> > > >> > > >
> >> > > >> > > > changes are done and we use that mechanism to configure
> >> providers
> >> > > >> > > >
> >> > > >> > > > instead
> >> > > >> > > >
> >> > > >> > > > of ssl.provider configuration.
> >> > > >> > > >
> >> > > >> > > > KIP-492 has patch available and going through review.
> >> > > >> > > >
> >> > > >> > > > Can you read my all the points, I mentioned in my previous
> >> post,
> >> > > >> very
> >> > > >> > > >
> >> > > >> > > > carefully? I am covering all the aspects in explaining. I
> am
> >> open
> >> > > to
> >> > > >> > > >
> >> > > >> > > > still
> >> > > >> > > >
> >> > > >> > > > discuss more to clarify any doubts.
> >> > > >> > > >
> >> > > >> > > > "3. If we just use existing ssl.provider kafka
> configuration
> >> then
> >> > > >> our
> >> > > >> > > > provider will be used in SSLContext.getInstance(protocol,
> >> > > provider)
> >> > > >> > > >
> >> > > >> > > > call
> >> > > >> > > >
> >> > > >> > > > in
> >> > > >> > > >
> >> > > >> > > > SslFactory.java <http://sslfactory.java/> <
> >> > > http://sslfactory.java/>
> >> > > >> <
> >> > > >> > > > http://sslfactory.java/>
> >> > > >> > > >
> >> > > >> > > > and
> >> > > >> > > >
> >> > > >> > > > if our provider does not have
> >> > > >> > > > implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it
> >> breaks
> >> > > (we
> >> > > >> > > >
> >> > > >> > > > tested
> >> > > >> > > >
> >> > > >> > > > it). Example: In MyProvider sample above you see that I
> >> didn't add
> >> > > >> > > > SSLContext.TLSv1 as
> >> > > >> > > > "Service+Algorithm" and that didn't work for me. In SPIFFE
> >> > > provider
> >> > > >> you
> >> > > >> > > > don't have this challenge since you are planning to bypass
> >> > > >> ssl.provider
> >> > > >> > > >
> >> > > >> > > > as
> >> > > >> > > >
> >> > > >> > > > you mention in the KIP-492."
> >> > > >> > > >
> >> > > >> > > > Yes here you need to pass the protocol that your
> >> > > >> > > >
> >> > > >> > > > KeyManager/TrustManager
> >> > > >> > > >
> >> > > >> > > > registered with and in no way its deviating from TLS RFC
> >> spec.
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> >
> >> > >
> >> https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/
> >> > > >> > > >
> >> > > >> > > > javax/net/ssl/SSLContext.java#L134
> >> > > >> > > >
> >> > > >> > > > My suggestion here is for you to implement a simple
> Security
> >> > > >> Provider
> >> > > >> > > >
> >> > > >> > > > as
> >> > > >> > > >
> >> > > >> > > > you did before and register a Algorithm. You can use the
> >> existing
> >> > > >> > > > implementation in SpiffeProvider and plug in changes where
> >> you
> >> > > need
> >> > > >> to
> >> > > >> > > > retrieve the certificates from by making RPC call.
> >> > > >> > > >
> >> > > >> > > > Run an end-to-end test with Kafka broker coming up with
> your
> >> > > >> provider
> >> > > >> > > > making calls to RPC call. You do need to pass the "custom
> >> > > algorithm"
> >> > > >> > > >
> >> > > >> > > > that
> >> > > >> > > >
> >> > > >> > > > you registered in your key manager to invoke the provider.
> I
> >> think
> >> > > >> your
> >> > > >> > > > concern is this approach is replacing the existing known
> >> > > >> ciphersuites
> >> > > >> > > >
> >> > > >> > > > and
> >> > > >> > > >
> >> > > >> > > > certificate validation provided by java. Which its not.
> >> > > >> > > >
> >> > > >> > > > Now test the TLS connection you can do so via openssl
> >> -s_client
> >> > > >> options
> >> > > >> > > >
> >> > > >> > > > to
> >> > > >> > > >
> >> > > >> > > > test if everything else is working.
> >> > > >> > > >
> >> > > >> > > > I am happy to share configs that we used if you are
> >> interested.
> >> > > >> > > >
> >> > > >> > > > Thanks,
> >> > > >> > > > Harsha
> >> > > >> > > >
> >> > > >> > > > Thanks
> >> > > >> > > > Maulin
> >> > > >> > > >
> >> > > >> > > > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <
> >> cmccabe@apache.org>
> >> > > >> > > >
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Hi Maulin,
> >> > > >> > > >
> >> > > >> > > > A lot of JSSE providers don't implement custom algorithms.
> >> Spire
> >> > > is
> >> > > >> a
> >> > > >> > > >
> >> > > >> > > > good
> >> > > >> > > >
> >> > > >> > > > example of a JSSE provider that doesn't, and yet is still
> >> useful
> >> > > to
> >> > > >> > > >
> >> > > >> > > > many
> >> > > >> > > >
> >> > > >> > > > people. Your JSSE provider can work fine even if it doesn't
> >> > > >> implement a
> >> > > >> > > > custom algorithm.
> >> > > >> > > >
> >> > > >> > > > Maybe I'm missing something, but I don't understand the
> >> discussion
> >> > > >> of
> >> > > >> > > > Security.insertProviderAt() that you included.
> >> SslEngineBuilder
> >> > > >> doesn't
> >> > > >> > > >
> >> > > >> > > > use
> >> > > >> > > >
> >> > > >> > > > that API to get the security provider. Instead, it calls
> >> > > >> > > > "SSLContext.getInstance(protocol, provider)", where
> provider
> >> is
> >> > > the
> >> > > >> > > >
> >> > > >> > > > name
> >> > > >> > > >
> >> > > >> > > > of the provider.
> >> > > >> > > >
> >> > > >> > > > best,
> >> > > >> > > > Colin
> >> > > >> > > >
> >> > > >> > > > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
> >> > > >> > > >
> >> > > >> > > > On top of everything above I feel strongly to add the 4th
> >> point
> >> > > >> which
> >> > > >> > > >
> >> > > >> > > > is
> >> > > >> > > >
> >> > > >> > > > based on Java APIs for TrustManagerFactory.init(KeyStore) (
> >> > > >> > > >
> >> > > >> > > > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> >> > > >> > > > TrustManagerFactory.html#init(java.security.KeyStore
> >> > > >> > > > <http://java.security.keystore/>
> >> > > >> > > > <http://java.security.keystore/>)
> >> > > >> > > > )
> >> > > >> > > >
> >> > > >> > > > and KeyManagerFactory.init(KeyStore, char[]) (
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> > >
> >> > > >> >
> >> > > >>
> >> > >
> >>
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory
> >> > > >> > > > .
> >> > > >> > > >
> >> > > >> > > > html#init(java.security.KeyStore <
> >> http://java.security.keystore/>
> >> > > >> > > <http://
> >> > > >> > > > java.security.keystore/
> >> > > >> > > > ,%20char[])
> >> > > >> > > >
> >> > > >> > > > ).
> >> > > >> > > >
> >> > > >> > > > 4. The above APIs are intended to support providing
> >> "trust/key
> >> > > >> > > >
> >> > > >> > > > material"
> >> > > >> > > >
> >> > > >> > > > from the user without having to write their own
> >> > > >> > > >
> >> > > >> > > > TrustManager/KeyManagers.
> >> > > >> > > >
> >> > > >> > > > To quote from the TrustManagerFactory.init()'s
> documentation
> >> > > >> > > >
> >> > > >> > > > "Initializes
> >> > > >> > > >
> >> > > >> > > > this factory with a source of certificate authorities and
> >> related
> >> > > >> trust
> >> > > >> > > > material."
> >> > > >> > > > To quote from the KeyManagerFactory.init()'s documentation
> >> > > >> "Initializes
> >> > > >> > > > this factory with a source of key material."
> >> > > >> > > >
> >> > > >> > > > Based on this it is clear that there is a flexibility
> >> provided by
> >> > > >> Java
> >> > > >> > > >
> >> > > >> > > > to
> >> > > >> > > >
> >> > > >> > > > to enable developers to provide the required trust/key
> >> material
> >> > > >> loaded
> >> > > >> > > >
> >> > > >> > > > from
> >> > > >> > > >
> >> > > >> > > > "anywhere" without requiring them to write custom provider
> OR
> >> > > >> trust/key
> >> > > >> > > > managers. This same flexibility is reflected in Kafka code
> >> also
> >> > > >> where
> >> > > >> > > >
> >> > > >> > > > it
> >> > > >> > > >
> >> > > >> > > > loads the trust/keys from a local file and doesn't require
> >> > > writing a
> >> > > >> > > > Provider necessarily. If we do NOT have a custom algorithm,
> >> it
> >> > > makes
> >> > > >> > > >
> >> > > >> > > > less
> >> > > >> > > >
> >> > > >> > > > sense to write a Provider.
> >> > > >> > > >
> >> > > >> > > > Thanks
> >> > > >> > > > Maulin
> >> > > >> > > >
> >> > > >> > > > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
> >> > > >> > > >
> >> > > >> > > > maulin.vasavada@gmail.com>
> >> > > >> > > >
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Hi Harsha/Colin
> >> > > >> > > >
> >> > > >> > > > I did the sample with a custom Provider for
> >> TrustStoreManager and
> >> > > >> tried
> >> > > >> > > > using ssl.provider Kafka config AND the way KIP-492 is
> >> suggesting
> >> > > >> (by
> >> > > >> > > > adding Provider programmatically instead of relying on
> >> > > >> > > >
> >> > > >> > > > ssl.provider+java.
> >> > > >> > > >
> >> > > >> > > > security. The below sample is followed by my detailed
> >> findings.
> >> > > I'll
> >> > > >> > > > appreciate if you can go through it carefully and see
> >> > > >> > > >
> >> > > >> > > > if you
> >> > > >> > > >
> >> > > >> > > > see my point.
> >> > > >> > > >
> >> > > >> > > > package providertest;
> >> > > >> > > >
> >> > > >> > > > import java.security.Provider <
> >> http://java.security.provider/>
> >> > > >> > <http://
> >> > > >> > > > java.security.provider/>
> >> > > >> > > >
> >> > > >> > > > <http://
> >> > > >> > > >
> >> > > >> > > > java.security.provider/>;
> >> > > >> > > >
> >> > > >> > > > public class MyProvider extends Provider {
> >> > > >> > > >
> >> > > >> > > > private static final String name = "MyProvider"; private
> >> static
> >> > > >> double
> >> > > >> > > > version = 1.0d;
> >> > > >> > > > private static String info = "Maulin's SSL Provider
> >> v"+version;
> >> > > >> > > >
> >> > > >> > > > public MyProvider() {
> >> > > >> > > > super(name, version, info);
> >> > > >> > > > this.put("TrustManagerFactory.PKIX",
> >> > > >> > > >
> >> > > >> > > > "providertest.MyTrustManagerFactory");
> >> > > >> > > >
> >> > > >> > > > }
> >> > > >> > > > }
> >> > > >> > > >
> >> > > >> > > > *Details:*
> >> > > >> > > >
> >> > > >> > > > KIP-492 documents that it will use Security.addProvider()
> >> assuming
> >> > > >> it
> >> > > >> > > >
> >> > > >> > > > will
> >> > > >> > > >
> >> > > >> > > > add it as position '0' which is not a correct assumption.
> The
> >> > > >> > > > addProvider()'s documentation says it will add it to the
> last
> >> > > >> available
> >> > > >> > > > position. You may want to correct that to say
> >> > > >> > > > Security.insertProviderAt(provider, 1).
> >> > > >> > > >
> >> > > >> > > > Now coming back to our specific discussion,
> >> > > >> > > >
> >> > > >> > > > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence
> when
> >> you
> >> > > add
> >> > > >> > > >
> >> > > >> > > > that
> >> > > >> > > >
> >> > > >> > > > provider in the provider list via Security.addProvider()
> the
> >> > > >> position
> >> > > >> > > >
> >> > > >> > > > where
> >> > > >> > > >
> >> > > >> > > > it gets added doesn't matter (even if you don't end up
> >> adding it
> >> > > as
> >> > > >> > > >
> >> > > >> > > > first
> >> > > >> > > >
> >> > > >> > > > entry) since that is the ONLY provider for SPIFFE specific
> >> > > algorithm
> >> > > >> > > >
> >> > > >> > > > you
> >> > > >> > > >
> >> > > >> > > > might have.
> >> > > >> > > >
> >> > > >> > > > We do *not* have custom algorithm for Key/Trust
> StoreMangers.
> >> > > Which
> >> > > >> > > >
> >> > > >> > > > means
> >> > > >> > > >
> >> > > >> > > > we have to use X509, PKIX etc "Standard Algorithms" ((
> >> > > >> > > >
> >> > > >> > > >
> >> https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> >> > > >> > > > StandardNames.html
> >> > > >> > > > ))
> >> > > >> > > >
> >> > > >> > > > in our provider to override the TrustStoreManager (see my
> >> sample
> >> > > >> code)
> >> > > >> > > >
> >> > > >> > > > and
> >> > > >> > > >
> >> > > >> > > > KeyStoreManger and KeyManager. This creates another
> challenge
> >> > > >> > > >
> >> > > >> > > > mentioned in
> >> > > >> > > >
> >> > > >> > > > the below point.
> >> > > >> > > >
> >> > > >> > > > 2. In order to make our Provider for loading custom
> >> TrustStore
> >> > > >> work, we
> >> > > >> > > > have to add the provider as 'first' in the list since there
> >> are
> >> > > >> others
> >> > > >> > > >
> >> > > >> > > > with
> >> > > >> > > >
> >> > > >> > > > the same algorithm.
> >> > > >> > > >
> >> > > >> > > > However, the programatic way of adding provider
> >> > > >> > > > (Security.insertProviderAt()) is *not* deterministic for
> >> ordering
> >> > > >> since
> >> > > >> > > > different code can call the method for a different provider
> >> and
> >> > > >> > > >
> >> > > >> > > > depending
> >> > > >> > > >
> >> > > >> > > > upon the order of the call our provider can be first or
> >> pushed
> >> > > down
> >> > > >> the
> >> > > >> > > > list. This can happen very well in any client application
> >> using
> >> > > >> Kafka.
> >> > > >> > > >
> >> > > >> > > > This
> >> > > >> > > >
> >> > > >> > > > is specially problematic for a case when you want to
> >> guarantee
> >> > > order
> >> > > >> > > >
> >> > > >> > > > for a
> >> > > >> > > >
> >> > > >> > > > Provider having "Standard Algorithms".
> >> > > >> > > >
> >> > > >> > > > If we add our provider in java.security file that
> definitely
> >> > > >> guarantees
> >> > > >> > > > the order(unless somebody calls removeProvider() which is
> >> less
> >> > > >> > > >
> >> > > >> > > > likely). But
> >> > > >> > > >
> >> > > >> > > > if we add our provider in java.security file it will defeat
> >> the
> >> > > >> > > >
> >> > > >> > > > purpose of
> >> > > >> > > >
> >> > > >> > > > the KIP-492.
> >> > > >> > > >
> >> > > >> > > > In the gist - Apache Kafka must not rely on
> >> "un-deterministic"
> >> > > >> method
> >> > > >> > > >
> >> > > >> > > > to
> >> > > >> > > >
> >> > > >> > > > rely on Provider ordering.
> >> > > >> > > >
> >> > > >> > > > 3. If we just use existing ssl.provider kafka configuration
> >> then
> >> > > our
> >> > > >> > > > provider will be used in SSLContext.getInstance(protocol,
> >> > > provider)
> >> > > >> > > >
> >> > > >> > > > call in
> >> > > >> > > >
> >> > > >> > > > SslFactory.java <http://sslfactory.java/> <
> >> > > http://sslfactory.java/>
> >> > > >> <
> >> > > >> > > > http://sslfactory.java/>
> >> > > >> > > >
> >> > > >> > > > and
> >> > > >> > > >
> >> > > >> > > > if our provider does not have implementation for
> >> > > >> > > > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks
> >> > > >> > > >
> >> > > >> > > > (we
> >> > > >> > > >
> >> > > >> > > > tested it). Example:
> >> > > >> > > >
> >> > > >> > > > In
> >> > > >> > > >
> >> > > >> > > > MyProvider sample above you see that I didn't add
> >> SSLContext.TLSv1
> >> > > >> as
> >> > > >> > > > "Service+Algorithm" and that didn't work for me. In SPIFFE
> >> > > provider
> >> > > >> you
> >> > > >> > > > don't have this challenge since you are planning to bypass
> >> > > >> > > >
> >> > > >> > > > ssl.provider as
> >> > > >> > > >
> >> > > >> > > > you mention in the KIP-492.
> >> > > >> > > >
> >> > > >> > > > *Overall summary,*
> >> > > >> > > >
> >> > > >> > > > 1. Any provider based mechanisms- a) existing ssl.provider
> >> and
> >> > > >> > > >
> >> > > >> > > > b)KIP-492,
> >> > > >> > > >
> >> > > >> > > > for loading key/trust store using "Standard Algorithms" do
> >> not
> >> > > work
> >> > > >> > > >
> >> > > >> > > > 2. Approach suggested in our KIP-486 works without any
> issue
> >> and
> >> > > it
> >> > > >> is
> >> > > >> > > > *not* our context specific solve
> >> > > >> > > >
> >> > > >> > > > 3. Based on above we feel KIP-492 and KIP-486 are
> >> complimentary
> >> > > >> changes
> >> > > >> > > > and not contradicting or redundent.
> >> > > >> > > >
> >> > > >> > > > If you want we can do a joint session somehow to walk
> >> through the
> >> > > >> > > >
> >> > > >> > > > sample I
> >> > > >> > > >
> >> > > >> > > > have and various experiments I did. I would encourage you
> to
> >> do
> >> > > >> similar
> >> > > >> > > > exercise by writing a Provider for "Standard Algorithm" for
> >> > > >> > > > TrustStoreManager (like our needs) and see what you find
> >> since
> >> > > only
> >> > > >> > > >
> >> > > >> > > > writing
> >> > > >> > > >
> >> > > >> > > > samples can bring out the complexity/challenges we face.
> >> > > >> > > >
> >> > > >> > > > Thanks
> >> > > >> > > > Maulin
> >> > > >> > > >
> >> > > >> > > > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <
> >> > > >> > > >
> >> > > >> > > > maulin.vasavada@gmail.
> >> > > >> > > >
> >> > > >> > > > com> wrote:
> >> > > >> > > >
> >> > > >> > > > Just to update - still working on it. Get to work only on
> >> and off
> >> > > on
> >> > > >> > > >
> >> > > >> > > > it :(
> >> > > >> > > >
> >> > > >> > > > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
> >> > > >> > > >
> >> > > >> > > > maulin.vasavada@gmail.com>
> >> > > >> > > >
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Hi Harsha
> >> > > >> > > >
> >> > > >> > > > Let me try to write samples and will let you know.
> >> > > >> > > >
> >> > > >> > > > Thanks
> >> > > >> > > > Maulin
> >> > > >> > > >
> >> > > >> > > > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <
> >> harsha.ch@gmail.com>
> >> > > >> > > >
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Hi Maulin,
> >> > > >> > > > With java security providers can be as custom you would
> >> > > >> > > >
> >> > > >> > > > like
> >> > > >> > > >
> >> > > >> > > > it to
> >> > > >> > > > be. If you only want to to implement a custom way of
> loading
> >> the
> >> > > >> > > >
> >> > > >> > > > keystore
> >> > > >> > > >
> >> > > >> > > > and truststore and not implement any protocol/encryption
> >> handling
> >> > > >> you
> >> > > >> > > >
> >> > > >> > > > can
> >> > > >> > > >
> >> > > >> > > > leave them empty and no need to implement. Have you looked
> >> into
> >> > > the
> >> > > >> > > >
> >> > > >> > > > links I
> >> > > >> > > >
> >> > > >> > > > pasted before?
> >> > > >> > > >
> >> > > >> > > >
> >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> > >
> >> > > >> >
> >> > > >>
> >> > >
> >>
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> >> > > >> > > >
> >> > > >> > > > java
> >> > > >> > > >
> >> > > >> > > >
> >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >> > > >> > > > spiffe-security-provider/src/main/java/spiffe/api/provider/
> >> > > >> > > > SpiffeTrustManager.java <http://spiffetrustmanager.java/>
> >> > > <http://
> >> > > >> > > > spiffetrustmanager.java/>
> >> > > >> > > >
> >> > > >> > > >
> >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> > >
> >> > > >> >
> >> > > >>
> >> > >
> >>
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> >> > > >> > > >
> >> > > >> > > > java
> >> > > >> > > >
> >> > > >> > > > Can you please tell me which methods are too complex in
> >> above to
> >> > > >> > > >
> >> > > >> > > > implement
> >> > > >> > > >
> >> > > >> > > > or unnecessary? You are changing anything in SSL/TLS
> >> > > implementations
> >> > > >> > > > provided by
> >> > > >> > > >
> >> > > >> > > > All of the implementations delegating the checks to the
> >> default
> >> > > >> > > > implementation anyway.
> >> > > >> > > > Spire agent is an example, its nothing but a GRPC server
> >> listening
> >> > > >> > > >
> >> > > >> > > > on a
> >> > > >> > > >
> >> > > >> > > > unix domain socket . Above code is making a RPC call to the
> >> local
> >> > > >> > > >
> >> > > >> > > > daemon
> >> > > >> > > >
> >> > > >> > > > to
> >> > > >> > > > get the certificate and keys. The mechanics are pretty much
> >> same
> >> > > as
> >> > > >> > > >
> >> > > >> > > > what
> >> > > >> > > >
> >> > > >> > > > you are asking for.
> >> > > >> > > >
> >> > > >> > > > Thanks,
> >> > > >> > > > Harsha
> >> > > >> > > >
> >> > > >> > > > On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
> >> > > >> > > >
> >> > > >> > > > maulin.vasavada@gmail.com>
> >> > > >> > > >
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Imagine a scenario like - We know we have a custom KMS and
> >> as a
> >> > > >> > > >
> >> > > >> > > > Kafka
> >> > > >> > > >
> >> > > >> > > > owner
> >> > > >> > > >
> >> > > >> > > > we want to comply to using that KMS source to load
> >> keys/certs. As
> >> > > >> > > >
> >> > > >> > > > a
> >> > > >> > > >
> >> > > >> > > > Kafka
> >> > > >> > > >
> >> > > >> > > > owner we know how to integrate with KMS but doesn't
> >> necessarily
> >> > > >> > > >
> >> > > >> > > > have
> >> > > >> > > >
> >> > > >> > > > to
> >> > > >> > > >
> >> > > >> > > > know anything about cipher suites, algorithms, and SSL/TLS
> >> > > >> > > >
> >> > > >> > > > implementation.
> >> > > >> > > >
> >> > > >> > > > Going the Provider way requires to know lot more than we
> >> should,
> >> > > >> > > >
> >> > > >> > > > isn't it?
> >> > > >> > > >
> >> > > >> > > > Not that we would have concern/shy-away knowing those
> >> details -
> >> > > >> > > >
> >> > > >> > > > but
> >> > > >> > > >
> >> > > >> > > > if we
> >> > > >> > > >
> >> > > >> > > > don't have to - why should we?
> >> > > >> > > >
> >> > > >> > > > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
> >> > > >> > > >
> >> > > >> > > > maulin.vasavada@gmail.com>
> >> > > >> > > >
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Hi Harsha
> >> > > >> > > >
> >> > > >> > > > We don't have spire (or similar) agents and we do not have
> >> > > >> > > >
> >> > > >> > > > keys/certs
> >> > > >> > > >
> >> > > >> > > > locally on any brokers. To elaborate more on my previous
> >> email,
> >> > > >> > > >
> >> > > >> > > > I agree that Java security Providers are used in much
> broader
> >> > > >> > > >
> >> > > >> > > > sense
> >> > > >> > > >
> >> > > >> > > > - to
> >> > > >> > > >
> >> > > >> > > > have a particular implementation of an algorithm, use
> >> specific
> >> > > >> > > >
> >> > > >> > > > cipher
> >> > > >> > > >
> >> > > >> > > > suites for SSL , OR in our current team's case have a
> >> > > >> > > >
> >> > > >> > > > particular
> >> > > >> > > >
> >> > > >> > > > way to
> >> > > >> > > >
> >> > > >> > > > leverage pre-generated SSL sessions. However, the scope of
> >> our
> >> > > >> > > >
> >> > > >> > > > KIP
> >> > > >> > > >
> >> > > >> > > > (486)
> >> > > >> > > >
> >> > > >> > > > is
> >> > > >> > > >
> >> > > >> > > > much restricted than that. We merely intend to provide a
> >> custom
> >> > > >> > > > keystore/truststore for our SSL connections and not really
> >> worry
> >> > > >> > > >
> >> > > >> > > > about
> >> > > >> > > >
> >> > > >> > > > underlying specific SSL/TLS implementation. This simplifies
> >> it
> >> > > >> > > >
> >> > > >> > > > a
> >> > > >> > > >
> >> > > >> > > > lot for
> >> > > >> > > >
> >> > > >> > > > us to keep the concerns separate and clear.
> >> > > >> > > >
> >> > > >> > > > I feel our approach is more complimentary such that it
> allows
> >> > > >> > > >
> >> > > >> > > > for
> >> > > >> > > >
> >> > > >> > > > using
> >> > > >> > > >
> >> > > >> > > > keystores of choice while retaining the flexibility to use
> >> any
> >> > > >> > > > underlying/available Provider for actually making the SSL
> >> call.
> >> > > >> > > >
> >> > > >> > > > We agree with KIP-492's approach based on Providers (and
> >> Java's
> >> > > >> > > > recommendation), but also strongly believe that our
> approach
> >> can
> >> > > >> > > >
> >> > > >> > > > compliment
> >> > > >> > > >
> >> > > >> > > > it very effectively for reasons explained above.
> >> > > >> > > >
> >> > > >> > > > Thanks
> >> > > >> > > > Maulin
> >> > > >> > > >
> >> > > >> > > > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <
> >> > > >> > > >
> >> > > >> > > > kafka@harsha.io
> >> > > >> > > >
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Hi Maulin,
> >> > > >> > > >
> >> > > >> > > > On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
> >> > > >> > > >
> >> > > >> > > > maulin.vasavada@gmail.
> >> > > >> > > >
> >> > > >> > > > com>
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Hi Harsha
> >> > > >> > > >
> >> > > >> > > > The reason we rejected the SslProvider route is that - we
> >> > > >> > > >
> >> > > >> > > > only
> >> > > >> > > >
> >> > > >> > > > needed
> >> > > >> > > >
> >> > > >> > > > a
> >> > > >> > > >
> >> > > >> > > > custom way to load keys/certs. Not touch any policy that
> >> > > >> > > >
> >> > > >> > > > existing
> >> > > >> > > >
> >> > > >> > > > Providers
> >> > > >> > > >
> >> > > >> > > > govern like SunJSSE Provider.
> >> > > >> > > >
> >> > > >> > > > We have exactly the same requirements to load certs and
> keys
> >> > > >> > > >
> >> > > >> > > > through
> >> > > >> > > >
> >> > > >> > > > spire
> >> > > >> > > >
> >> > > >> > > > agent. We used security.provider to do that exactly. I am
> not
> >> > > >> > > >
> >> > > >> > > > sure
> >> > > >> > > >
> >> > > >> > > > why
> >> > > >> > > >
> >> > > >> > > > you
> >> > > >> > > >
> >> > > >> > > > would be modifying any policies provided by default SunJSSE
> >> > > >> > > >
> >> > > >> > > > provider.
> >> > > >> > > >
> >> > > >> > > > Can
> >> > > >> > > >
> >> > > >> > > > you give me an example of having custom provider that will
> >> > > >> > > >
> >> > > >> > > > override an
> >> > > >> > > >
> >> > > >> > > > existing policy in SunJSSE provider.
> >> > > >> > > >
> >> > > >> > > > As pointed out earlier, this kip
> >> > > >> > > >
> >> > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> >> > > >> > > >
> >> KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> >> > > >> > > >
> >> > > >> > > > allows
> >> > > >> > > > you to load security.provider through config.
> >> > > >> > > > Take a look at the examples I gave before
> >> > > >> > > >
> >> > > >> > > >
> >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> > >
> >> > > >> >
> >> > > >>
> >> > >
> >>
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> >> > > >> > > >
> >> > > >> > > > java
> >> > > >> > > >
> >> > > >> > > > It registers KeyManagerFactory and TrustManagerFactory and
> >> > > >> > > >
> >> > > >> > > > Keystore
> >> > > >> > > >
> >> > > >> > > > algorithm.
> >> > > >> > > > Implement your custom way of loading Keystore in here
> >> > > >> > > >
> >> > > >> > > >
> >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> > >
> >> > > >> >
> >> > > >>
> >> > >
> >>
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> >> > > >> > > >
> >> > > >> > > > java
> >> > > >> > > >
> >> > > >> > > > and Trust manager like here
> >> > > >> > > >
> >> > > >> > > >
> >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >> > > >> > > > spiffe-security-provider/src/main/java/spiffe/api/provider/
> >> > > >> > > > SpiffeTrustManager.java <http://spiffetrustmanager.java/>
> >> > > <http://
> >> > > >> > > > spiffetrustmanager.java/>
> >> > > >> > > >
> >> > > >> > > > In your Kafka client you can set the security.provider to
> >> your
> >> > > >> > > >
> >> > > >> > > > custom
> >> > > >> > > >
> >> > > >> > > > implementation and with this fix
> >> > > >> > > > https://issues.apache.org/jira/browse/KAFKA-8191 you can
> set
> >> > > >> > > > keyManagerAlgorigthm and trustManagerAlgorithm configs.
> >> > > >> > > >
> >> > > >> > > > All of this is in your clients and broker side and do not
> >> need
> >> > > >> > > >
> >> > > >> > > > to
> >> > > >> > > >
> >> > > >> > > > touch
> >> > > >> > > >
> >> > > >> > > > any
> >> > > >> > > > policy changes at JVM level. You'll register the providers
> in
> >> > > >> > > >
> >> > > >> > > > the
> >> > > >> > > >
> >> > > >> > > > priority
> >> > > >> > > >
> >> > > >> > > > order and can still have SunJSSE provider and have your
> >> custom
> >> > > >> > > >
> >> > > >> > > > provider
> >> > > >> > > >
> >> > > >> > > > to
> >> > > >> > > >
> >> > > >> > > > implement the key and trust managers.
> >> > > >> > > >
> >> > > >> > > > The ask here is different than KIP-492. We don't have any
> >> need
> >> > > >> > > >
> >> > > >> > > > to
> >> > > >> > > >
> >> > > >> > > > modify/specify the algorithm parameter. Does that make
> sense?
> >> > > >> > > >
> >> > > >> > > > The ask in KIP is introducing new interfaces where the
> KIP's
> >> > > >> > > > goal/motivation can be achieved through the
> security.provider
> >> > > >> > > >
> >> > > >> > > > and
> >> > > >> > > >
> >> > > >> > > > we
> >> > > >> > > >
> >> > > >> > > > worked
> >> > > >> > > > on similar goal without touching any Keystore or Truststore
> >> > > >> > > >
> >> > > >> > > > interfaces.
> >> > > >> > > >
> >> > > >> > > > My advise is against changing or introducing new interfaces
> >> > > >> > > >
> >> > > >> > > > when
> >> > > >> > > >
> >> > > >> > > > it can
> >> > > >> > > >
> >> > > >> > > > work through security.provider.
> >> > > >> > > >
> >> > > >> > > > Thanks,
> >> > > >> > > > Harsha
> >> > > >> > > >
> >> > > >> > > > Thanks
> >> > > >> > > >
> >> > > >> > > > Maulin
> >> > > >> > > >
> >> > > >> > > > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
> >> > > >> > > >
> >> > > >> > > > kafka@harsha.io>
> >> > > >> > > >
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > In your KIP you added security. provider as rejected
> >> > > >> > > >
> >> > > >> > > > alternative
> >> > > >> > > >
> >> > > >> > > > and
> >> > > >> > > >
> >> > > >> > > > specified "its not the correct way". Do you mind explaining
> >> > > >> > > >
> >> > > >> > > > why
> >> > > >> > > >
> >> > > >> > > > its
> >> > > >> > > >
> >> > > >> > > > not? I
> >> > > >> > > >
> >> > > >> > > > didn't find any evidence in Java docs to say so. Contrary
> to
> >> > > >> > > >
> >> > > >> > > > your
> >> > > >> > > >
> >> > > >> > > > statement
> >> > > >> > > >
> >> > > >> > > > it does say in the java docs
> >> > > >> > > > " However, please note that a provider can be used to
> >> > > >> > > >
> >> > > >> > > > implement
> >> > > >> > > >
> >> > > >> > > > any
> >> > > >> > > >
> >> > > >> > > > security service in Java that uses a pluggable architecture
> >> > > >> > > >
> >> > > >> > > > with
> >> > > >> > > >
> >> > > >> > > > a
> >> > > >> > > >
> >> > > >> > > > choice
> >> > > >> > > >
> >> > > >> > > > of implementations that fit underneath."
> >> > > >> > > >
> >> > > >> > > > Java Security Providers have been used by other projects to
> >> > > >> > > >
> >> > > >> > > > provide
> >> > > >> > > >
> >> > > >> > > > such
> >> > > >> > > >
> >> > > >> > > > integration . I am not sure if you looked into Spiffe
> >> > > >> > > >
> >> > > >> > > > project to
> >> > > >> > > >
> >> > > >> > > > efficiently distribute certificates but here is an example
> of
> >> > > >> > > >
> >> > > >> > > > Java
> >> > > >> > > >
> >> > > >> > > > provider
> >> > > >> > > >
> >> > > >> > > >
> >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> > >
> >> > > >> >
> >> > > >>
> >> > >
> >>
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> >> > > >> > > >
> >> > > >> > > > java which
> >> > > >> > > > obtains certificates from local daemons.
> >> > > >> > > > These integrations are being used in Tomcat, Jetty etc.. We
> >> > > >> > > >
> >> > > >> > > > are
> >> > > >> > > >
> >> > > >> > > > also
> >> > > >> > > >
> >> > > >> > > > using
> >> > > >> > > >
> >> > > >> > > > Security provider to do the same in our Kafka clusters. So
> >> > > >> > > >
> >> > > >> > > > unless I
> >> > > >> > > >
> >> > > >> > > > see
> >> > > >> > > >
> >> > > >> > > > more evidence why security.provider doesn't work for you
> >> > > >> > > >
> >> > > >> > > > adding
> >> > > >> > > >
> >> > > >> > > > new
> >> > > >> > > >
> >> > > >> > > > interfaces while there exists more cleaner way of achieving
> >> > > >> > > >
> >> > > >> > > > the
> >> > > >> > > >
> >> > > >> > > > goals
> >> > > >> > > >
> >> > > >> > > > of
> >> > > >> > > >
> >> > > >> > > > this KIP is unnecessary and breaks the well known security
> >> > > >> > > >
> >> > > >> > > > interfaces
> >> > > >> > > >
> >> > > >> > > > provided by Java itself.
> >> > > >> > > >
> >> > > >> > > > Thanks,
> >> > > >> > > > Harsha
> >> > > >> > > >
> >> > > >> > > > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
> >> > > >> > > >
> >> > > >> > > > kafka@harsha.io
> >> > > >> > > >
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Hi Maulin,
> >> > > >> > > > Not sure if you looked at my previous replies. This
> >> > > >> > > >
> >> > > >> > > > changes
> >> > > >> > > >
> >> > > >> > > > are not required as there is already security Provider to
> do
> >> > > >> > > >
> >> > > >> > > > what you
> >> > > >> > > >
> >> > > >> > > > are
> >> > > >> > > >
> >> > > >> > > > proposing. This KIP
> >> > > >> > > >
> >> > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> >> > > >> > > >
> >> > > >> > > >
> >> KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> >> > > >> > > >
> >> > > >> > > > also
> >> > > >> > > >
> >> > > >> > > > addresses easy registration of such providers.
> >> > > >> > > >
> >> > > >> > > > Thanks,
> >> > > >> > > > Harsha
> >> > > >> > > >
> >> > > >> > > > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
> >> > > >> > > >
> >> > > >> > > > <maulin.vasavada@gmail.
> >> > > >> > > >
> >> > > >> > > > com> wrote:
> >> > > >> > > >
> >> > > >> > > > Bump! Can somebody please review this?
> >> > > >> > > >
> >> > > >> > > > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
> >> > > >> > > >
> >> > > >> > > > maulin.vasavada@gmail.com>
> >> > > >> > > >
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Bump! Can somebody please review this?
> >> > > >> > > >
> >> > > >> > > > --
> >> > > >> > > > Thanks,
> >> > > >> > > > M.Sai Sandeep
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> > >
> >> > > >> >
> >> > > >>
> >> > > >
> >> > >
> >> >
> >>
> >
>
RE: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by "Pellerin, Clement" <Cl...@ibi.com>.
What is your solution to the objection that killed the second iteration of KIP-383?
Mainly, how do you support validation of reconfiguration requests that involve new custom properties implemented by the pluggable factory?
Custom properties do not exist yet, but that is very legitimate thing to design for the future.
That's why I favor a pluggable SslFactory instead of an SslEngineBuilder factory.
-----Original Message-----
From: Maulin Vasavada [mailto:maulin.vasavada@gmail.com]
Sent: Friday, August 30, 2019 3:07 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
+1 for making SslEngineBuilder configurable upon more thoughts.
However, in the abstraction and default implementation we should make sure
when we do have a requirement to plugin custom key/trust store people don't
have to write lot more code which may not be related to it.
Having said that, does this mean, we resurrect KIP-383 and update it with
latest context and go from there?
We are willing to take up that work for making it configurable.
Thanks
Maulin
On Fri, Aug 30, 2019 at 10:34 AM Maulin Vasavada <ma...@gmail.com>
wrote:
> Why don't we make SSLEngineBuilder code delegate the whole Key/Trust store
> initialization to the interfaces we are proposing? Default implementation
> for those key/trust store loader interfaces will be "file based" loading vs
> if somebody wants to customize any of it they can.
>
> Would that make sense?
>
> On Fri, Aug 30, 2019 at 10:03 AM Colin McCabe <cm...@apache.org> wrote:
>
>> +1 for making SslEngineBuilder configurable. This would give
>> implementers a lot more flexibility-- to use key distribution methods that
>> were not files, for example.
>>
>> best,
>> Colin
>>
>>
>> On Fri, Aug 30, 2019, at 02:03, Rajini Sivaram wrote:
>> > Just to make sure we are on the same page - KIP-383 was written before
>> > the
>> > code was refactored. The refactoring addressed some of the concerns of
>> > KIP-383. My suggestion was to make SslEngineBuilder configurable. The
>> > default implementation of this pluggable class would be
>> >
>> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/ssl/SslEngineBuilder.java
>> .
>> > That should give an idea of the size of the configurable part that a
>> > custom
>> > class needs to implement. A large part of that is about loading
>> > keystore/truststore. I agree it has slightly more code than KIP-486
>> > proposes, but since it lets you customize creation of SSLEngine, it
>> > would
>> > address every possible scenario.
>> >
>> > Thoughts?
>> >
>> >
>> > On Fri, Aug 30, 2019 at 2:02 AM Maulin Vasavada <
>> maulin.vasavada@gmail.com>
>> > wrote:
>> >
>> > > I thought about it more. I feel that no matter how we refactor the
>> code
>> > > (with or without KIP-383 integrated), ultimately the need of
>> customizing
>> > > loading for keys and certs will still remain. Whenever that need
>> arises we
>> > > might end up thinking about the solution suggested by our KIP-486.
>> Hence
>> > > regardless of the other KIPs and configurations "if we do need to
>> customize
>> > > loading of keys/certs, we will need the code changes suggested by this
>> > > KIP".
>> > >
>> > > Let me know what you guys think.
>> > >
>> > > Harsha, we are working on changing the interfaces for key/trust store
>> > > loaders with Certificate and PrivateKey objects. Will probably be
>> able to
>> > > update it later today or tomorrow.
>> > >
>> > > Thanks
>> > > Maulin
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > On Thu, Aug 29, 2019 at 2:30 PM Maulin Vasavada <
>> maulin.vasavada@gmail.com
>> > > >
>> > > wrote:
>> > >
>> > > > On that, I actually looked at KIP-383 before briefly. However, that
>> > > > sounded like lot of changes suggested.
>> > > >
>> > > > One "key" thing we have to keep in mind is - IF we need lot of
>> > > > customization Kafka already allows you to use your SslProvider via
>> > > > ssl.providers or the changes done by KIP-492 and
>> > > > SSLContext.getInstance(protocol, provider) call allows us to return
>> the
>> > > > SSLContext with "ALL" the details we would like to customize. Hence
>> I am
>> > > > not sure that customization suggested by KIP-383 would be worth the
>> > > effort.
>> > > > We also have similar SSLContext customization outside of Kafka.
>> > > >
>> > > > Thanks
>> > > > Maulin
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Thu, Aug 29, 2019 at 12:47 PM Pellerin, Clement <
>> > > > Clement_Pellerin@ibi.com> wrote:
>> > > >
>> > > >> KIP-383 in its present form was vetoed because it was not possible
>> to
>> > > add
>> > > >> validation of custom properties in a future KIP. The solution to
>> that is
>> > > >> the first proposal I wrote for KIP-383 which made the whole
>> SslFactory
>> > > >> pluggable. That first solution was also vetoed hence the deadlock.
>> > > >>
>> > > >> Replacing the whole factory was a much nicer solution. It was
>> vetoed
>> > > >> because doing this almost invariably meant the replacement lost
>> all the
>> > > >> complex validation code in the default SslFactory.
>> > > >>
>> > > >> My current idea is to extract the validation code into another
>> public
>> > > API
>> > > >> that SslFactory would call. I did not look at the newly refactored
>> code
>> > > and
>> > > >> I did not study how to do this yet. KIP-383 was not popular at the
>> time
>> > > and
>> > > >> designing a new solution is a lot of work.
>> > > >>
>> > > >> Is there interest from 3 binding voters for something like this?
>> > > >>
>> > > >> -----Original Message-----
>> > > >> From: Rajini Sivaram [mailto:rajinisivaram@gmail.com]
>> > > >> Sent: Thursday, August 29, 2019 2:57 PM
>> > > >> To: dev
>> > > >> Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and
>> > > >> TrustStore
>> > > >>
>> > > >> Hi Maulin,
>> > > >>
>> > > >> In SSL scenarios, I imagine security providers introduced by
>> KIP-492 are
>> > > >> likely to be most useful when you want to use third party
>> providers. The
>> > > >> biggest advantage of the config from that KIP is that you don't
>> need to
>> > > >> write much code to integrate existing security providers into Kafka
>> > > >> brokers
>> > > >> or clients. As I understand it, KIP-486 is a more convenient
>> option for
>> > > >> the
>> > > >> specific problem of loading keystores/truststores differently. It
>> can be
>> > > >> achieved in theory with KIP-492, but KIP-486 is a much simpler
>> option
>> > > for
>> > > >> this case.
>> > > >>
>> > > >> My concern about KIP-486 is that it introduces yet another
>> interface
>> > > into
>> > > >> our already complex security code, while only solving one
>> particular use
>> > > >> case. Have you looked at
>> > > >>
>> > > >>
>> > >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-383%3A++Pluggable+interface+for+SSL+Factory
>> > > >> ?
>> > > >> The goal was to make
>> > > >> org.apache.kafka.common.security.ssl.SslEngineBuilder pluggable.
>> > > >> The code has already been refactored by Colin after that KIP was
>> > > written,
>> > > >> making it easier to implement KIP-383. This should enable you to
>> load
>> > > your
>> > > >> keystores and truststores differently. Using a pluggable
>> > > SslEngineBuilder
>> > > >> will also solve several other use cases at the same time. KIP-383
>> hasn't
>> > > >> been voted through yet, but perhaps you could take a look and we
>> could
>> > > >> revive that instead if it solves your use case as well?
>> > > >>
>> > > >> Regards,
>> > > >>
>> > > >> Rajini
>> > > >>
>> > > >>
>> > > >> On Thu, Aug 29, 2019 at 6:42 PM Maulin Vasavada <
>> > > >> maulin.vasavada@gmail.com>
>> > > >> wrote:
>> > > >>
>> > > >> > Hi Harsha
>> > > >> >
>> > > >> > Thank you. Appreciate your time and support on this. Let me go
>> back do
>> > > >> some
>> > > >> > more research and get back to you on the KeyStore interface part.
>> > > >> > Basically, if we return certs and keys in the interface then
>> Kafka
>> > > code
>> > > >> > will have to build KeyStore object - which is also reasonable.
>> > > >> >
>> > > >> > Thanks
>> > > >> > Maulin
>> > > >> >
>> > > >> > On Thu, Aug 29, 2019 at 10:01 AM Harsha Chintalapani <
>> kafka@harsha.io
>> > > >
>> > > >> > wrote:
>> > > >> >
>> > > >> > > Hi Maulin,
>> > > >> > > Use cases are clear now. I am +1 for moving
>> > > >> forward
>> > > >> > > with the discussions on having such configurable option for
>> users.
>> > > But
>> > > >> > the
>> > > >> > > interfaces is proposed doesn't look right to me. We are still
>> > > talking
>> > > >> > about
>> > > >> > > keystore interfaces. Given keystore's are used as filebased
>> way of
>> > > >> > > transporting certificates I am not sure it will help the rest
>> of the
>> > > >> > > user-base.
>> > > >> > > In short, I am +1 on the KIP's motivation
>> and only
>> > > >> have
>> > > >> > > questions around returning keystores instead of returning
>> certs,
>> > > >> private
>> > > >> > > keys etc. . If others in the community are ok with such
>> interface we
>> > > >> can
>> > > >> > > move forward.
>> > > >> > >
>> > > >> > > Thanks,
>> > > >> > > Harsha
>> > > >> > >
>> > > >> > >
>> > > >> > > On Wed, Aug 28, 2019 at 1:51 PM, Maulin Vasavada <
>> > > >> > > maulin.vasavada@gmail.com>
>> > > >> > > wrote:
>> > > >> > >
>> > > >> > > > Hi Harsha
>> > > >> > > >
>> > > >> > > > As we synced-up offline on this topic, we hope you don't
>> have any
>> > > >> more
>> > > >> > > > clarifications that you are seeking. If that is the case,
>> can you
>> > > >> > please
>> > > >> > > > help us move this forward and discuss what changes you would
>> > > expect
>> > > >> on
>> > > >> > > the
>> > > >> > > > KIP design in order to make it valuable contribution?
>> > > >> > > >
>> > > >> > > > Just FYI - we verified our primary design change with the
>> author
>> > > of
>> > > >> > Sun's
>> > > >> > > > X509 Trustmanager implementation and the outcome is that
>> what we
>> > > are
>> > > >> > > > proposing makes sense at the heart of it - "Instead of
>> writing
>> > > >> > > TrustManager
>> > > >> > > > just plugin the Trust store". We are open to discuss
>> additional
>> > > >> changes
>> > > >> > > > that you/anybody else would like to see on the functionality
>> > > >> however.
>> > > >> > > >
>> > > >> > > > Thanks
>> > > >> > > > Maulin
>> > > >> > > >
>> > > >> > > > On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada <
>> > > >> > > maulin.vasavada@gmail.com>
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Hi Harsha
>> > > >> > > >
>> > > >> > > > Any response on my question? I feel this KIP is worth
>> > > accommodating.
>> > > >> > Your
>> > > >> > > > help is much appreciated.
>> > > >> > > >
>> > > >> > > > Thanks
>> > > >> > > > Maulin
>> > > >> > > >
>> > > >> > > > On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada <
>> > > >> > maulin.vasavada@gmail.
>> > > >> > > > com> wrote:
>> > > >> > > >
>> > > >> > > > Hi Harsha
>> > > >> > > >
>> > > >> > > > I've examined the SPIFFE provider more and have one question
>> -
>> > > >> > > >
>> > > >> > > > If SPIFFE didn't have a need to do checkSpiffeId() call at
>> the
>> > > below
>> > > >> > > > location, would you really still write the Provider? *OR*
>> Would
>> > > you
>> > > >> > just
>> > > >> > > > use TrustManagerFactory.init(KeyStore) signature to pass the
>> > > >> KeyStore
>> > > >> > > from
>> > > >> > > > set of certs returned by spiffeIdManager. getTrustedCerts()?
>> > > >> > > >
>> > > >> > > >
>> > > >> >
>> > >
>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>> > > >> > > > provider/CertificateUtils.java#L100
>> > > >> > > >
>> > > >> > > > /**
>> > > >> > > >
>> > > >> > > > * Validates that the SPIFFE ID is present and matches the
>> SPIFFE
>> > > ID
>> > > >> > > > configured in
>> > > >> > > > * the java.security property ssl.spiffe.accept
>> > > >> > > > *
>> > > >> > > > * If the authorized spiffe ids list is empty any spiffe id is
>> > > >> > authorized
>> > > >> > > > *
>> > > >> > > > * @param chain an array of X509Certificate that contains the
>> > > Peer's
>> > > >> > SVID
>> > > >> > > > to be validated
>> > > >> > > > * @throws CertificateException when either the certificates
>> > > doesn't
>> > > >> > have
>> > > >> > > a
>> > > >> > > > SPIFFE ID or the SPIFFE ID is not authorized
>> > > >> > > > */
>> > > >> > > > static void checkSpiffeId(X509Certificate[] chain) throws
>> > > >> > > > CertificateException {
>> > > >> > > >
>> > > >> > > > Thanks
>> > > >> > > > Maulin
>> > > >> > > >
>> > > >> > > > On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani <
>> > > >> kafka@harsha.io>
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Maulin,
>> > > >> > > > The code parts you are pointing are specific for Spiffe and
>> if
>> > > >> > > > you are talking about validate method which uses PKIX check
>> like
>> > > any
>> > > >> > > other
>> > > >> > > > provider does.
>> > > >> > > > If you want to default to SunJSSE everywhere you can do so by
>> > > >> > delegating
>> > > >> > > > the calls in these methods to SunJSSE provider.
>> > > >> > > >
>> > > >> > > > TrustManagerFactory tmf = TrustManagerFactory
>> > > >> > > > .getInstance(TrustManagerFactory.getDefaultAlgorithm());and
>> use
>> > > >> > > > tmf.chekServerTrusted()
>> > > >> > > > or use
>> > > >> > > > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
>> > > >> > > > TrustManagerFactory.html#getInstance(java.lang.String)if you
>> want
>> > > a
>> > > >> > > > specific provider.
>> > > >> > > >
>> > > >> > > > -Harsha
>> > > >> > > >
>> > > >> > > > On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada <
>> > > >> > maulin.vasavada@gmail.
>> > > >> > > > com>
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Okay, so I take that you guys agree that I have to write a
>> > > 'custom'
>> > > >> > > > algorithm and a provider to make it work , correct?
>> > > >> > > >
>> > > >> > > > Now, for Harsha's comment "Here the 'Custom' Algorithm is
>> not an
>> > > >> > > > implementation per say , ..." , I diagree. You can refer to
>> > > https://
>> > > >> > > >
>> > > >> > > >
>> > > >> >
>> > > >>
>> > >
>> github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/
>> > > >> > > >
>> > > >> > > > SpiffeTrustManager.java#L91 <
>> http://spiffetrustmanager.java/#L91>
>> > > >> and
>> > > >> > > >
>> > > >> > > >
>> > > >> >
>> > >
>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>> > > >> > > >
>> > > >> > > > provider/CertificateUtils.java#L100
>> > > >> > > >
>> > > >> > > > "that code" is the customization you have for the custom way
>> to
>> > > >> check
>> > > >> > > > something on top of regular checks. That method is NOT doing
>> > > custom
>> > > >> > > > truststore loading. It is validating/verifying something in
>> the
>> > > >> > > >
>> > > >> > > > "custom"
>> > > >> > > >
>> > > >> > > > way with spiffeId.
>> > > >> > > > I bet that without that you won't have a need of the custom
>> > > >> algorithm
>> > > >> > > >
>> > > >> > > > in
>> > > >> > > >
>> > > >> > > > the first place.
>> > > >> > > >
>> > > >> > > > Let me know if you agree to this.
>> > > >> > > >
>> > > >> > > > Thanks
>> > > >> > > > Maulin
>> > > >> > > >
>> > > >> > > > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <
>> mprsai@gmail.com>
>> > > >> > > >
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Hi Maulin, thanks for the discussion. As Harsha pointed out,
>> to
>> > > use
>> > > >> the
>> > > >> > > > KIP492, you need to create a new provider, register a *new*
>> custom
>> > > >> > > > algorithm for your keymanager and trustmanager factory
>> > > >> implementations.
>> > > >> > > > After this, the kafka server configuration can be done as
>> given
>> > > >> below
>> > > >> > > >
>> > > >> > > > # Register the provider class with custom algorithm, say
>> CUSTOM
>> > > >> > > >
>> > > >> > > > security.
>> > > >> > > >
>> > > >> > > > provider.classes=com.company.security.CustomProvider
>> > > >> > > > <http://provider.classes
>> =com.company.security.customprovider/>
>> > > >> > > > <http://security.provider.classes
>> > > >> > > >
>> > > >> > > > =com.company.security.customprovider/>
>> > > >> > > >
>> > > >> > > > # Register the keymanager and trustmanager algorithms
>> > > >> > > > # These algorithms indicate that the Keymanager and
>> Trustmanagers
>> > > >> > > > registered under the algorithm "CUSTOM" needs to be used
>> > > >> > > > ssl.trustmanager.algorithm=CUSTOM
>> > > >> > > > ssl.keymanager.algorithm=CUSTOM
>> > > >> > > >
>> > > >> > > > And the customprovider looks like this...
>> > > >> > > >
>> > > >> > > > public class CustomProvider extends Provider {
>> > > >> > > > public CustomProvider() {
>> > > >> > > > super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and
>> > > TrustStore");
>> > > >> > > > super.put("KeyManagerFactory.CUSTOM",
>> "customKeyManagerFactory");
>> > > >> > > > super.put("TrustManagerFactory.CUSTOM",
>> > > >> > > > "customTrustManagerFactory");
>> > > >> > > > }
>> > > >> > > > }
>> > > >> > > >
>> > > >> > > > The PR for this is in review and can be found here:
>> > > >> > > >
>> > > >> > > > https://github.com/
>> > > >> > > >
>> > > >> > > > apache/kafka/pull/7090
>> > > >> > > > This PR includes the fixed insertProviderAt function call.
>> > > >> > > >
>> > > >> > > > On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani <
>> > > >> kafka@harsha.io>
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Answers are in-line
>> > > >> > > >
>> > > >> > > > On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada <
>> > > >> > > >
>> > > >> > > > maulin.vasavada@gmail.
>> > > >> > > >
>> > > >> > > > com
>> > > >> > > >
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Hi Colin
>> > > >> > > >
>> > > >> > > > When I refer to "standard" or "custom" algorithms I am
>> following
>> > > >> Java
>> > > >> > > > security Provider Terminology. You can refer to
>> > > >> > > >
>> https://docs.oracle.com/javase/7/docs/technotes/guides/security/
>> > > >> > > > StandardNames.html#TrustManagerFactory link I provided
>> earlier in
>> > > >> the
>> > > >> > > > emails. It says PKIX is the default Algorithm for
>> > > >> TrustManagerFactory.
>> > > >> > > >
>> > > >> > > > 1. For SPIFFE, I am not sure why you are saying 'it does not
>> > > >> implement
>> > > >> > > > custom algorithms' because the following file clearly
>> indicates
>> > > >> that it
>> > > >> > > > does use custom algorithm-
>> > > >> > > >
>> > > >> > > >
>> > > >> >
>> > >
>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>> > > >> > > >
>> > > >> > > > provider/SpiffeProvider.java#L17
>> > > >> > > >
>> > > >> > > > Algorithm value:
>> > > >> > > >
>> > > >> > > >
>> > > >> >
>> > >
>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>> > > >> > > >
>> > > >> > > > provider/SpiffeProviderConstants.java#L6
>> > > >> > > >
>> > > >> > > > @Harsha do you want to chime in since you use that provider?
>> > > >> > > >
>> > > >> > > > Here the "Custom" Algorithm is not an implementation per say
>> ,
>> > > >> rather
>> > > >> > > >
>> > > >> > > > used
>> > > >> > > >
>> > > >> > > > to invoke the custom trust store factory and key manager
>> factory.
>> > > >> You
>> > > >> > > >
>> > > >> > > > are
>> > > >> > > >
>> > > >> > > > not moving away from "standard" alogrithms that are
>> available.
>> > > >> > > >
>> > > >> > > >
>> > > >> >
>> > >
>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>> > > >> > > >
>> > > >> > > > provider/SpiffeTrustManager.java
>> > > >> > > >
>> > > >> > > > As you can see it delegates all the calls of verification of
>> > > >> > > >
>> > > >> > > > certificate
>> > > >> > > >
>> > > >> > > > to
>> > > >> > > >
>> > > >> > > > the default implementation available.
>> > > >> > > > So in our implementation we still use PKIX to verify the
>> > > certificate
>> > > >> > > > chain. So you are not losing anything here and Spiffe is not
>> > > >> > > >
>> > > >> > > > reimplementing
>> > > >> > > >
>> > > >> > > > the verification process.
>> > > >> > > >
>> > > >> > > > 2. I already mentioned in my 3rd point, in my previous post,
>> why
>> > > >> using
>> > > >> > > >
>> > > >> > > > ssl.provider does NOT work. I updated KIP-486 in "rejected
>> > > >> > > >
>> > > >> > > > alternatives"
>> > > >> > > >
>> > > >> > > > also why ssl.provider does not work.
>> > > >> > > >
>> > > >> > > > As mentioned before , provider is the right way to go. I am
>> still
>> > > >> not
>> > > >> > > > understanding the gap is.
>> > > >> > > > If I understand correctly your argument is , provider is
>> going to
>> > > >> ask
>> > > >> > > >
>> > > >> > > > to
>> > > >> > > >
>> > > >> > > > implement a custom algorithm.
>> > > >> > > > My answer to that is , no you are not re-implementing the
>> > > algorithm.
>> > > >> > > >
>> > > >> > > > Please
>> > > >> > > >
>> > > >> > > > check the above link , TrustManager implementation it
>> delegates
>> > > the
>> > > >> > > >
>> > > >> > > > calls
>> > > >> > > >
>> > > >> > > > back. There is no need to implement your own here.
>> > > >> > > >
>> > > >> > > > 3. Security.insertProviderAt() comments were based on
>> assumption
>> > > if
>> > > >> > > >
>> > > >> > > > KIP-492
>> > > >> > > >
>> > > >> > > > changes are done and we use that mechanism to configure
>> providers
>> > > >> > > >
>> > > >> > > > instead
>> > > >> > > >
>> > > >> > > > of ssl.provider configuration.
>> > > >> > > >
>> > > >> > > > KIP-492 has patch available and going through review.
>> > > >> > > >
>> > > >> > > > Can you read my all the points, I mentioned in my previous
>> post,
>> > > >> very
>> > > >> > > >
>> > > >> > > > carefully? I am covering all the aspects in explaining. I am
>> open
>> > > to
>> > > >> > > >
>> > > >> > > > still
>> > > >> > > >
>> > > >> > > > discuss more to clarify any doubts.
>> > > >> > > >
>> > > >> > > > "3. If we just use existing ssl.provider kafka configuration
>> then
>> > > >> our
>> > > >> > > > provider will be used in SSLContext.getInstance(protocol,
>> > > provider)
>> > > >> > > >
>> > > >> > > > call
>> > > >> > > >
>> > > >> > > > in
>> > > >> > > >
>> > > >> > > > SslFactory.java <http://sslfactory.java/> <
>> > > http://sslfactory.java/>
>> > > >> <
>> > > >> > > > http://sslfactory.java/>
>> > > >> > > >
>> > > >> > > > and
>> > > >> > > >
>> > > >> > > > if our provider does not have
>> > > >> > > > implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it
>> breaks
>> > > (we
>> > > >> > > >
>> > > >> > > > tested
>> > > >> > > >
>> > > >> > > > it). Example: In MyProvider sample above you see that I
>> didn't add
>> > > >> > > > SSLContext.TLSv1 as
>> > > >> > > > "Service+Algorithm" and that didn't work for me. In SPIFFE
>> > > provider
>> > > >> you
>> > > >> > > > don't have this challenge since you are planning to bypass
>> > > >> ssl.provider
>> > > >> > > >
>> > > >> > > > as
>> > > >> > > >
>> > > >> > > > you mention in the KIP-492."
>> > > >> > > >
>> > > >> > > > Yes here you need to pass the protocol that your
>> > > >> > > >
>> > > >> > > > KeyManager/TrustManager
>> > > >> > > >
>> > > >> > > > registered with and in no way its deviating from TLS RFC
>> spec.
>> > > >> > > >
>> > > >> > > >
>> > > >> >
>> > >
>> https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/
>> > > >> > > >
>> > > >> > > > javax/net/ssl/SSLContext.java#L134
>> > > >> > > >
>> > > >> > > > My suggestion here is for you to implement a simple Security
>> > > >> Provider
>> > > >> > > >
>> > > >> > > > as
>> > > >> > > >
>> > > >> > > > you did before and register a Algorithm. You can use the
>> existing
>> > > >> > > > implementation in SpiffeProvider and plug in changes where
>> you
>> > > need
>> > > >> to
>> > > >> > > > retrieve the certificates from by making RPC call.
>> > > >> > > >
>> > > >> > > > Run an end-to-end test with Kafka broker coming up with your
>> > > >> provider
>> > > >> > > > making calls to RPC call. You do need to pass the "custom
>> > > algorithm"
>> > > >> > > >
>> > > >> > > > that
>> > > >> > > >
>> > > >> > > > you registered in your key manager to invoke the provider. I
>> think
>> > > >> your
>> > > >> > > > concern is this approach is replacing the existing known
>> > > >> ciphersuites
>> > > >> > > >
>> > > >> > > > and
>> > > >> > > >
>> > > >> > > > certificate validation provided by java. Which its not.
>> > > >> > > >
>> > > >> > > > Now test the TLS connection you can do so via openssl
>> -s_client
>> > > >> options
>> > > >> > > >
>> > > >> > > > to
>> > > >> > > >
>> > > >> > > > test if everything else is working.
>> > > >> > > >
>> > > >> > > > I am happy to share configs that we used if you are
>> interested.
>> > > >> > > >
>> > > >> > > > Thanks,
>> > > >> > > > Harsha
>> > > >> > > >
>> > > >> > > > Thanks
>> > > >> > > > Maulin
>> > > >> > > >
>> > > >> > > > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <
>> cmccabe@apache.org>
>> > > >> > > >
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Hi Maulin,
>> > > >> > > >
>> > > >> > > > A lot of JSSE providers don't implement custom algorithms.
>> Spire
>> > > is
>> > > >> a
>> > > >> > > >
>> > > >> > > > good
>> > > >> > > >
>> > > >> > > > example of a JSSE provider that doesn't, and yet is still
>> useful
>> > > to
>> > > >> > > >
>> > > >> > > > many
>> > > >> > > >
>> > > >> > > > people. Your JSSE provider can work fine even if it doesn't
>> > > >> implement a
>> > > >> > > > custom algorithm.
>> > > >> > > >
>> > > >> > > > Maybe I'm missing something, but I don't understand the
>> discussion
>> > > >> of
>> > > >> > > > Security.insertProviderAt() that you included.
>> SslEngineBuilder
>> > > >> doesn't
>> > > >> > > >
>> > > >> > > > use
>> > > >> > > >
>> > > >> > > > that API to get the security provider. Instead, it calls
>> > > >> > > > "SSLContext.getInstance(protocol, provider)", where provider
>> is
>> > > the
>> > > >> > > >
>> > > >> > > > name
>> > > >> > > >
>> > > >> > > > of the provider.
>> > > >> > > >
>> > > >> > > > best,
>> > > >> > > > Colin
>> > > >> > > >
>> > > >> > > > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
>> > > >> > > >
>> > > >> > > > On top of everything above I feel strongly to add the 4th
>> point
>> > > >> which
>> > > >> > > >
>> > > >> > > > is
>> > > >> > > >
>> > > >> > > > based on Java APIs for TrustManagerFactory.init(KeyStore) (
>> > > >> > > >
>> > > >> > > > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
>> > > >> > > > TrustManagerFactory.html#init(java.security.KeyStore
>> > > >> > > > <http://java.security.keystore/>
>> > > >> > > > <http://java.security.keystore/>)
>> > > >> > > > )
>> > > >> > > >
>> > > >> > > > and KeyManagerFactory.init(KeyStore, char[]) (
>> > > >> > > >
>> > > >> > > >
>> > > >> > >
>> > > >> >
>> > > >>
>> > >
>> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory
>> > > >> > > > .
>> > > >> > > >
>> > > >> > > > html#init(java.security.KeyStore <
>> http://java.security.keystore/>
>> > > >> > > <http://
>> > > >> > > > java.security.keystore/
>> > > >> > > > ,%20char[])
>> > > >> > > >
>> > > >> > > > ).
>> > > >> > > >
>> > > >> > > > 4. The above APIs are intended to support providing
>> "trust/key
>> > > >> > > >
>> > > >> > > > material"
>> > > >> > > >
>> > > >> > > > from the user without having to write their own
>> > > >> > > >
>> > > >> > > > TrustManager/KeyManagers.
>> > > >> > > >
>> > > >> > > > To quote from the TrustManagerFactory.init()'s documentation
>> > > >> > > >
>> > > >> > > > "Initializes
>> > > >> > > >
>> > > >> > > > this factory with a source of certificate authorities and
>> related
>> > > >> trust
>> > > >> > > > material."
>> > > >> > > > To quote from the KeyManagerFactory.init()'s documentation
>> > > >> "Initializes
>> > > >> > > > this factory with a source of key material."
>> > > >> > > >
>> > > >> > > > Based on this it is clear that there is a flexibility
>> provided by
>> > > >> Java
>> > > >> > > >
>> > > >> > > > to
>> > > >> > > >
>> > > >> > > > to enable developers to provide the required trust/key
>> material
>> > > >> loaded
>> > > >> > > >
>> > > >> > > > from
>> > > >> > > >
>> > > >> > > > "anywhere" without requiring them to write custom provider OR
>> > > >> trust/key
>> > > >> > > > managers. This same flexibility is reflected in Kafka code
>> also
>> > > >> where
>> > > >> > > >
>> > > >> > > > it
>> > > >> > > >
>> > > >> > > > loads the trust/keys from a local file and doesn't require
>> > > writing a
>> > > >> > > > Provider necessarily. If we do NOT have a custom algorithm,
>> it
>> > > makes
>> > > >> > > >
>> > > >> > > > less
>> > > >> > > >
>> > > >> > > > sense to write a Provider.
>> > > >> > > >
>> > > >> > > > Thanks
>> > > >> > > > Maulin
>> > > >> > > >
>> > > >> > > > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
>> > > >> > > >
>> > > >> > > > maulin.vasavada@gmail.com>
>> > > >> > > >
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Hi Harsha/Colin
>> > > >> > > >
>> > > >> > > > I did the sample with a custom Provider for
>> TrustStoreManager and
>> > > >> tried
>> > > >> > > > using ssl.provider Kafka config AND the way KIP-492 is
>> suggesting
>> > > >> (by
>> > > >> > > > adding Provider programmatically instead of relying on
>> > > >> > > >
>> > > >> > > > ssl.provider+java.
>> > > >> > > >
>> > > >> > > > security. The below sample is followed by my detailed
>> findings.
>> > > I'll
>> > > >> > > > appreciate if you can go through it carefully and see
>> > > >> > > >
>> > > >> > > > if you
>> > > >> > > >
>> > > >> > > > see my point.
>> > > >> > > >
>> > > >> > > > package providertest;
>> > > >> > > >
>> > > >> > > > import java.security.Provider <
>> http://java.security.provider/>
>> > > >> > <http://
>> > > >> > > > java.security.provider/>
>> > > >> > > >
>> > > >> > > > <http://
>> > > >> > > >
>> > > >> > > > java.security.provider/>;
>> > > >> > > >
>> > > >> > > > public class MyProvider extends Provider {
>> > > >> > > >
>> > > >> > > > private static final String name = "MyProvider"; private
>> static
>> > > >> double
>> > > >> > > > version = 1.0d;
>> > > >> > > > private static String info = "Maulin's SSL Provider
>> v"+version;
>> > > >> > > >
>> > > >> > > > public MyProvider() {
>> > > >> > > > super(name, version, info);
>> > > >> > > > this.put("TrustManagerFactory.PKIX",
>> > > >> > > >
>> > > >> > > > "providertest.MyTrustManagerFactory");
>> > > >> > > >
>> > > >> > > > }
>> > > >> > > > }
>> > > >> > > >
>> > > >> > > > *Details:*
>> > > >> > > >
>> > > >> > > > KIP-492 documents that it will use Security.addProvider()
>> assuming
>> > > >> it
>> > > >> > > >
>> > > >> > > > will
>> > > >> > > >
>> > > >> > > > add it as position '0' which is not a correct assumption. The
>> > > >> > > > addProvider()'s documentation says it will add it to the last
>> > > >> available
>> > > >> > > > position. You may want to correct that to say
>> > > >> > > > Security.insertProviderAt(provider, 1).
>> > > >> > > >
>> > > >> > > > Now coming back to our specific discussion,
>> > > >> > > >
>> > > >> > > > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when
>> you
>> > > add
>> > > >> > > >
>> > > >> > > > that
>> > > >> > > >
>> > > >> > > > provider in the provider list via Security.addProvider() the
>> > > >> position
>> > > >> > > >
>> > > >> > > > where
>> > > >> > > >
>> > > >> > > > it gets added doesn't matter (even if you don't end up
>> adding it
>> > > as
>> > > >> > > >
>> > > >> > > > first
>> > > >> > > >
>> > > >> > > > entry) since that is the ONLY provider for SPIFFE specific
>> > > algorithm
>> > > >> > > >
>> > > >> > > > you
>> > > >> > > >
>> > > >> > > > might have.
>> > > >> > > >
>> > > >> > > > We do *not* have custom algorithm for Key/Trust StoreMangers.
>> > > Which
>> > > >> > > >
>> > > >> > > > means
>> > > >> > > >
>> > > >> > > > we have to use X509, PKIX etc "Standard Algorithms" ((
>> > > >> > > >
>> > > >> > > >
>> https://docs.oracle.com/javase/7/docs/technotes/guides/security/
>> > > >> > > > StandardNames.html
>> > > >> > > > ))
>> > > >> > > >
>> > > >> > > > in our provider to override the TrustStoreManager (see my
>> sample
>> > > >> code)
>> > > >> > > >
>> > > >> > > > and
>> > > >> > > >
>> > > >> > > > KeyStoreManger and KeyManager. This creates another challenge
>> > > >> > > >
>> > > >> > > > mentioned in
>> > > >> > > >
>> > > >> > > > the below point.
>> > > >> > > >
>> > > >> > > > 2. In order to make our Provider for loading custom
>> TrustStore
>> > > >> work, we
>> > > >> > > > have to add the provider as 'first' in the list since there
>> are
>> > > >> others
>> > > >> > > >
>> > > >> > > > with
>> > > >> > > >
>> > > >> > > > the same algorithm.
>> > > >> > > >
>> > > >> > > > However, the programatic way of adding provider
>> > > >> > > > (Security.insertProviderAt()) is *not* deterministic for
>> ordering
>> > > >> since
>> > > >> > > > different code can call the method for a different provider
>> and
>> > > >> > > >
>> > > >> > > > depending
>> > > >> > > >
>> > > >> > > > upon the order of the call our provider can be first or
>> pushed
>> > > down
>> > > >> the
>> > > >> > > > list. This can happen very well in any client application
>> using
>> > > >> Kafka.
>> > > >> > > >
>> > > >> > > > This
>> > > >> > > >
>> > > >> > > > is specially problematic for a case when you want to
>> guarantee
>> > > order
>> > > >> > > >
>> > > >> > > > for a
>> > > >> > > >
>> > > >> > > > Provider having "Standard Algorithms".
>> > > >> > > >
>> > > >> > > > If we add our provider in java.security file that definitely
>> > > >> guarantees
>> > > >> > > > the order(unless somebody calls removeProvider() which is
>> less
>> > > >> > > >
>> > > >> > > > likely). But
>> > > >> > > >
>> > > >> > > > if we add our provider in java.security file it will defeat
>> the
>> > > >> > > >
>> > > >> > > > purpose of
>> > > >> > > >
>> > > >> > > > the KIP-492.
>> > > >> > > >
>> > > >> > > > In the gist - Apache Kafka must not rely on
>> "un-deterministic"
>> > > >> method
>> > > >> > > >
>> > > >> > > > to
>> > > >> > > >
>> > > >> > > > rely on Provider ordering.
>> > > >> > > >
>> > > >> > > > 3. If we just use existing ssl.provider kafka configuration
>> then
>> > > our
>> > > >> > > > provider will be used in SSLContext.getInstance(protocol,
>> > > provider)
>> > > >> > > >
>> > > >> > > > call in
>> > > >> > > >
>> > > >> > > > SslFactory.java <http://sslfactory.java/> <
>> > > http://sslfactory.java/>
>> > > >> <
>> > > >> > > > http://sslfactory.java/>
>> > > >> > > >
>> > > >> > > > and
>> > > >> > > >
>> > > >> > > > if our provider does not have implementation for
>> > > >> > > > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks
>> > > >> > > >
>> > > >> > > > (we
>> > > >> > > >
>> > > >> > > > tested it). Example:
>> > > >> > > >
>> > > >> > > > In
>> > > >> > > >
>> > > >> > > > MyProvider sample above you see that I didn't add
>> SSLContext.TLSv1
>> > > >> as
>> > > >> > > > "Service+Algorithm" and that didn't work for me. In SPIFFE
>> > > provider
>> > > >> you
>> > > >> > > > don't have this challenge since you are planning to bypass
>> > > >> > > >
>> > > >> > > > ssl.provider as
>> > > >> > > >
>> > > >> > > > you mention in the KIP-492.
>> > > >> > > >
>> > > >> > > > *Overall summary,*
>> > > >> > > >
>> > > >> > > > 1. Any provider based mechanisms- a) existing ssl.provider
>> and
>> > > >> > > >
>> > > >> > > > b)KIP-492,
>> > > >> > > >
>> > > >> > > > for loading key/trust store using "Standard Algorithms" do
>> not
>> > > work
>> > > >> > > >
>> > > >> > > > 2. Approach suggested in our KIP-486 works without any issue
>> and
>> > > it
>> > > >> is
>> > > >> > > > *not* our context specific solve
>> > > >> > > >
>> > > >> > > > 3. Based on above we feel KIP-492 and KIP-486 are
>> complimentary
>> > > >> changes
>> > > >> > > > and not contradicting or redundent.
>> > > >> > > >
>> > > >> > > > If you want we can do a joint session somehow to walk
>> through the
>> > > >> > > >
>> > > >> > > > sample I
>> > > >> > > >
>> > > >> > > > have and various experiments I did. I would encourage you to
>> do
>> > > >> similar
>> > > >> > > > exercise by writing a Provider for "Standard Algorithm" for
>> > > >> > > > TrustStoreManager (like our needs) and see what you find
>> since
>> > > only
>> > > >> > > >
>> > > >> > > > writing
>> > > >> > > >
>> > > >> > > > samples can bring out the complexity/challenges we face.
>> > > >> > > >
>> > > >> > > > Thanks
>> > > >> > > > Maulin
>> > > >> > > >
>> > > >> > > > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <
>> > > >> > > >
>> > > >> > > > maulin.vasavada@gmail.
>> > > >> > > >
>> > > >> > > > com> wrote:
>> > > >> > > >
>> > > >> > > > Just to update - still working on it. Get to work only on
>> and off
>> > > on
>> > > >> > > >
>> > > >> > > > it :(
>> > > >> > > >
>> > > >> > > > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
>> > > >> > > >
>> > > >> > > > maulin.vasavada@gmail.com>
>> > > >> > > >
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Hi Harsha
>> > > >> > > >
>> > > >> > > > Let me try to write samples and will let you know.
>> > > >> > > >
>> > > >> > > > Thanks
>> > > >> > > > Maulin
>> > > >> > > >
>> > > >> > > > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <
>> harsha.ch@gmail.com>
>> > > >> > > >
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Hi Maulin,
>> > > >> > > > With java security providers can be as custom you would
>> > > >> > > >
>> > > >> > > > like
>> > > >> > > >
>> > > >> > > > it to
>> > > >> > > > be. If you only want to to implement a custom way of loading
>> the
>> > > >> > > >
>> > > >> > > > keystore
>> > > >> > > >
>> > > >> > > > and truststore and not implement any protocol/encryption
>> handling
>> > > >> you
>> > > >> > > >
>> > > >> > > > can
>> > > >> > > >
>> > > >> > > > leave them empty and no need to implement. Have you looked
>> into
>> > > the
>> > > >> > > >
>> > > >> > > > links I
>> > > >> > > >
>> > > >> > > > pasted before?
>> > > >> > > >
>> > > >> > > >
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>> > > >> > > >
>> > > >> > > >
>> > > >> > >
>> > > >> >
>> > > >>
>> > >
>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
>> > > >> > > >
>> > > >> > > > java
>> > > >> > > >
>> > > >> > > >
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>> > > >> > > > spiffe-security-provider/src/main/java/spiffe/api/provider/
>> > > >> > > > SpiffeTrustManager.java <http://spiffetrustmanager.java/>
>> > > <http://
>> > > >> > > > spiffetrustmanager.java/>
>> > > >> > > >
>> > > >> > > >
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>> > > >> > > >
>> > > >> > > >
>> > > >> > >
>> > > >> >
>> > > >>
>> > >
>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>> > > >> > > >
>> > > >> > > > java
>> > > >> > > >
>> > > >> > > > Can you please tell me which methods are too complex in
>> above to
>> > > >> > > >
>> > > >> > > > implement
>> > > >> > > >
>> > > >> > > > or unnecessary? You are changing anything in SSL/TLS
>> > > implementations
>> > > >> > > > provided by
>> > > >> > > >
>> > > >> > > > All of the implementations delegating the checks to the
>> default
>> > > >> > > > implementation anyway.
>> > > >> > > > Spire agent is an example, its nothing but a GRPC server
>> listening
>> > > >> > > >
>> > > >> > > > on a
>> > > >> > > >
>> > > >> > > > unix domain socket . Above code is making a RPC call to the
>> local
>> > > >> > > >
>> > > >> > > > daemon
>> > > >> > > >
>> > > >> > > > to
>> > > >> > > > get the certificate and keys. The mechanics are pretty much
>> same
>> > > as
>> > > >> > > >
>> > > >> > > > what
>> > > >> > > >
>> > > >> > > > you are asking for.
>> > > >> > > >
>> > > >> > > > Thanks,
>> > > >> > > > Harsha
>> > > >> > > >
>> > > >> > > > On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
>> > > >> > > >
>> > > >> > > > maulin.vasavada@gmail.com>
>> > > >> > > >
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Imagine a scenario like - We know we have a custom KMS and
>> as a
>> > > >> > > >
>> > > >> > > > Kafka
>> > > >> > > >
>> > > >> > > > owner
>> > > >> > > >
>> > > >> > > > we want to comply to using that KMS source to load
>> keys/certs. As
>> > > >> > > >
>> > > >> > > > a
>> > > >> > > >
>> > > >> > > > Kafka
>> > > >> > > >
>> > > >> > > > owner we know how to integrate with KMS but doesn't
>> necessarily
>> > > >> > > >
>> > > >> > > > have
>> > > >> > > >
>> > > >> > > > to
>> > > >> > > >
>> > > >> > > > know anything about cipher suites, algorithms, and SSL/TLS
>> > > >> > > >
>> > > >> > > > implementation.
>> > > >> > > >
>> > > >> > > > Going the Provider way requires to know lot more than we
>> should,
>> > > >> > > >
>> > > >> > > > isn't it?
>> > > >> > > >
>> > > >> > > > Not that we would have concern/shy-away knowing those
>> details -
>> > > >> > > >
>> > > >> > > > but
>> > > >> > > >
>> > > >> > > > if we
>> > > >> > > >
>> > > >> > > > don't have to - why should we?
>> > > >> > > >
>> > > >> > > > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
>> > > >> > > >
>> > > >> > > > maulin.vasavada@gmail.com>
>> > > >> > > >
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Hi Harsha
>> > > >> > > >
>> > > >> > > > We don't have spire (or similar) agents and we do not have
>> > > >> > > >
>> > > >> > > > keys/certs
>> > > >> > > >
>> > > >> > > > locally on any brokers. To elaborate more on my previous
>> email,
>> > > >> > > >
>> > > >> > > > I agree that Java security Providers are used in much broader
>> > > >> > > >
>> > > >> > > > sense
>> > > >> > > >
>> > > >> > > > - to
>> > > >> > > >
>> > > >> > > > have a particular implementation of an algorithm, use
>> specific
>> > > >> > > >
>> > > >> > > > cipher
>> > > >> > > >
>> > > >> > > > suites for SSL , OR in our current team's case have a
>> > > >> > > >
>> > > >> > > > particular
>> > > >> > > >
>> > > >> > > > way to
>> > > >> > > >
>> > > >> > > > leverage pre-generated SSL sessions. However, the scope of
>> our
>> > > >> > > >
>> > > >> > > > KIP
>> > > >> > > >
>> > > >> > > > (486)
>> > > >> > > >
>> > > >> > > > is
>> > > >> > > >
>> > > >> > > > much restricted than that. We merely intend to provide a
>> custom
>> > > >> > > > keystore/truststore for our SSL connections and not really
>> worry
>> > > >> > > >
>> > > >> > > > about
>> > > >> > > >
>> > > >> > > > underlying specific SSL/TLS implementation. This simplifies
>> it
>> > > >> > > >
>> > > >> > > > a
>> > > >> > > >
>> > > >> > > > lot for
>> > > >> > > >
>> > > >> > > > us to keep the concerns separate and clear.
>> > > >> > > >
>> > > >> > > > I feel our approach is more complimentary such that it allows
>> > > >> > > >
>> > > >> > > > for
>> > > >> > > >
>> > > >> > > > using
>> > > >> > > >
>> > > >> > > > keystores of choice while retaining the flexibility to use
>> any
>> > > >> > > > underlying/available Provider for actually making the SSL
>> call.
>> > > >> > > >
>> > > >> > > > We agree with KIP-492's approach based on Providers (and
>> Java's
>> > > >> > > > recommendation), but also strongly believe that our approach
>> can
>> > > >> > > >
>> > > >> > > > compliment
>> > > >> > > >
>> > > >> > > > it very effectively for reasons explained above.
>> > > >> > > >
>> > > >> > > > Thanks
>> > > >> > > > Maulin
>> > > >> > > >
>> > > >> > > > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <
>> > > >> > > >
>> > > >> > > > kafka@harsha.io
>> > > >> > > >
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Hi Maulin,
>> > > >> > > >
>> > > >> > > > On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
>> > > >> > > >
>> > > >> > > > maulin.vasavada@gmail.
>> > > >> > > >
>> > > >> > > > com>
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Hi Harsha
>> > > >> > > >
>> > > >> > > > The reason we rejected the SslProvider route is that - we
>> > > >> > > >
>> > > >> > > > only
>> > > >> > > >
>> > > >> > > > needed
>> > > >> > > >
>> > > >> > > > a
>> > > >> > > >
>> > > >> > > > custom way to load keys/certs. Not touch any policy that
>> > > >> > > >
>> > > >> > > > existing
>> > > >> > > >
>> > > >> > > > Providers
>> > > >> > > >
>> > > >> > > > govern like SunJSSE Provider.
>> > > >> > > >
>> > > >> > > > We have exactly the same requirements to load certs and keys
>> > > >> > > >
>> > > >> > > > through
>> > > >> > > >
>> > > >> > > > spire
>> > > >> > > >
>> > > >> > > > agent. We used security.provider to do that exactly. I am not
>> > > >> > > >
>> > > >> > > > sure
>> > > >> > > >
>> > > >> > > > why
>> > > >> > > >
>> > > >> > > > you
>> > > >> > > >
>> > > >> > > > would be modifying any policies provided by default SunJSSE
>> > > >> > > >
>> > > >> > > > provider.
>> > > >> > > >
>> > > >> > > > Can
>> > > >> > > >
>> > > >> > > > you give me an example of having custom provider that will
>> > > >> > > >
>> > > >> > > > override an
>> > > >> > > >
>> > > >> > > > existing policy in SunJSSE provider.
>> > > >> > > >
>> > > >> > > > As pointed out earlier, this kip
>> > > >> > > >
>> > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/
>> > > >> > > >
>> KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>> > > >> > > >
>> > > >> > > > allows
>> > > >> > > > you to load security.provider through config.
>> > > >> > > > Take a look at the examples I gave before
>> > > >> > > >
>> > > >> > > >
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>> > > >> > > >
>> > > >> > > >
>> > > >> > >
>> > > >> >
>> > > >>
>> > >
>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>> > > >> > > >
>> > > >> > > > java
>> > > >> > > >
>> > > >> > > > It registers KeyManagerFactory and TrustManagerFactory and
>> > > >> > > >
>> > > >> > > > Keystore
>> > > >> > > >
>> > > >> > > > algorithm.
>> > > >> > > > Implement your custom way of loading Keystore in here
>> > > >> > > >
>> > > >> > > >
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>> > > >> > > >
>> > > >> > > >
>> > > >> > >
>> > > >> >
>> > > >>
>> > >
>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
>> > > >> > > >
>> > > >> > > > java
>> > > >> > > >
>> > > >> > > > and Trust manager like here
>> > > >> > > >
>> > > >> > > >
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>> > > >> > > > spiffe-security-provider/src/main/java/spiffe/api/provider/
>> > > >> > > > SpiffeTrustManager.java <http://spiffetrustmanager.java/>
>> > > <http://
>> > > >> > > > spiffetrustmanager.java/>
>> > > >> > > >
>> > > >> > > > In your Kafka client you can set the security.provider to
>> your
>> > > >> > > >
>> > > >> > > > custom
>> > > >> > > >
>> > > >> > > > implementation and with this fix
>> > > >> > > > https://issues.apache.org/jira/browse/KAFKA-8191 you can set
>> > > >> > > > keyManagerAlgorigthm and trustManagerAlgorithm configs.
>> > > >> > > >
>> > > >> > > > All of this is in your clients and broker side and do not
>> need
>> > > >> > > >
>> > > >> > > > to
>> > > >> > > >
>> > > >> > > > touch
>> > > >> > > >
>> > > >> > > > any
>> > > >> > > > policy changes at JVM level. You'll register the providers in
>> > > >> > > >
>> > > >> > > > the
>> > > >> > > >
>> > > >> > > > priority
>> > > >> > > >
>> > > >> > > > order and can still have SunJSSE provider and have your
>> custom
>> > > >> > > >
>> > > >> > > > provider
>> > > >> > > >
>> > > >> > > > to
>> > > >> > > >
>> > > >> > > > implement the key and trust managers.
>> > > >> > > >
>> > > >> > > > The ask here is different than KIP-492. We don't have any
>> need
>> > > >> > > >
>> > > >> > > > to
>> > > >> > > >
>> > > >> > > > modify/specify the algorithm parameter. Does that make sense?
>> > > >> > > >
>> > > >> > > > The ask in KIP is introducing new interfaces where the KIP's
>> > > >> > > > goal/motivation can be achieved through the security.provider
>> > > >> > > >
>> > > >> > > > and
>> > > >> > > >
>> > > >> > > > we
>> > > >> > > >
>> > > >> > > > worked
>> > > >> > > > on similar goal without touching any Keystore or Truststore
>> > > >> > > >
>> > > >> > > > interfaces.
>> > > >> > > >
>> > > >> > > > My advise is against changing or introducing new interfaces
>> > > >> > > >
>> > > >> > > > when
>> > > >> > > >
>> > > >> > > > it can
>> > > >> > > >
>> > > >> > > > work through security.provider.
>> > > >> > > >
>> > > >> > > > Thanks,
>> > > >> > > > Harsha
>> > > >> > > >
>> > > >> > > > Thanks
>> > > >> > > >
>> > > >> > > > Maulin
>> > > >> > > >
>> > > >> > > > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
>> > > >> > > >
>> > > >> > > > kafka@harsha.io>
>> > > >> > > >
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > In your KIP you added security. provider as rejected
>> > > >> > > >
>> > > >> > > > alternative
>> > > >> > > >
>> > > >> > > > and
>> > > >> > > >
>> > > >> > > > specified "its not the correct way". Do you mind explaining
>> > > >> > > >
>> > > >> > > > why
>> > > >> > > >
>> > > >> > > > its
>> > > >> > > >
>> > > >> > > > not? I
>> > > >> > > >
>> > > >> > > > didn't find any evidence in Java docs to say so. Contrary to
>> > > >> > > >
>> > > >> > > > your
>> > > >> > > >
>> > > >> > > > statement
>> > > >> > > >
>> > > >> > > > it does say in the java docs
>> > > >> > > > " However, please note that a provider can be used to
>> > > >> > > >
>> > > >> > > > implement
>> > > >> > > >
>> > > >> > > > any
>> > > >> > > >
>> > > >> > > > security service in Java that uses a pluggable architecture
>> > > >> > > >
>> > > >> > > > with
>> > > >> > > >
>> > > >> > > > a
>> > > >> > > >
>> > > >> > > > choice
>> > > >> > > >
>> > > >> > > > of implementations that fit underneath."
>> > > >> > > >
>> > > >> > > > Java Security Providers have been used by other projects to
>> > > >> > > >
>> > > >> > > > provide
>> > > >> > > >
>> > > >> > > > such
>> > > >> > > >
>> > > >> > > > integration . I am not sure if you looked into Spiffe
>> > > >> > > >
>> > > >> > > > project to
>> > > >> > > >
>> > > >> > > > efficiently distribute certificates but here is an example of
>> > > >> > > >
>> > > >> > > > Java
>> > > >> > > >
>> > > >> > > > provider
>> > > >> > > >
>> > > >> > > >
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>> > > >> > > >
>> > > >> > > >
>> > > >> > >
>> > > >> >
>> > > >>
>> > >
>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>> > > >> > > >
>> > > >> > > > java which
>> > > >> > > > obtains certificates from local daemons.
>> > > >> > > > These integrations are being used in Tomcat, Jetty etc.. We
>> > > >> > > >
>> > > >> > > > are
>> > > >> > > >
>> > > >> > > > also
>> > > >> > > >
>> > > >> > > > using
>> > > >> > > >
>> > > >> > > > Security provider to do the same in our Kafka clusters. So
>> > > >> > > >
>> > > >> > > > unless I
>> > > >> > > >
>> > > >> > > > see
>> > > >> > > >
>> > > >> > > > more evidence why security.provider doesn't work for you
>> > > >> > > >
>> > > >> > > > adding
>> > > >> > > >
>> > > >> > > > new
>> > > >> > > >
>> > > >> > > > interfaces while there exists more cleaner way of achieving
>> > > >> > > >
>> > > >> > > > the
>> > > >> > > >
>> > > >> > > > goals
>> > > >> > > >
>> > > >> > > > of
>> > > >> > > >
>> > > >> > > > this KIP is unnecessary and breaks the well known security
>> > > >> > > >
>> > > >> > > > interfaces
>> > > >> > > >
>> > > >> > > > provided by Java itself.
>> > > >> > > >
>> > > >> > > > Thanks,
>> > > >> > > > Harsha
>> > > >> > > >
>> > > >> > > > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
>> > > >> > > >
>> > > >> > > > kafka@harsha.io
>> > > >> > > >
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Hi Maulin,
>> > > >> > > > Not sure if you looked at my previous replies. This
>> > > >> > > >
>> > > >> > > > changes
>> > > >> > > >
>> > > >> > > > are not required as there is already security Provider to do
>> > > >> > > >
>> > > >> > > > what you
>> > > >> > > >
>> > > >> > > > are
>> > > >> > > >
>> > > >> > > > proposing. This KIP
>> > > >> > > >
>> > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/
>> > > >> > > >
>> > > >> > > >
>> KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>> > > >> > > >
>> > > >> > > > also
>> > > >> > > >
>> > > >> > > > addresses easy registration of such providers.
>> > > >> > > >
>> > > >> > > > Thanks,
>> > > >> > > > Harsha
>> > > >> > > >
>> > > >> > > > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
>> > > >> > > >
>> > > >> > > > <maulin.vasavada@gmail.
>> > > >> > > >
>> > > >> > > > com> wrote:
>> > > >> > > >
>> > > >> > > > Bump! Can somebody please review this?
>> > > >> > > >
>> > > >> > > > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
>> > > >> > > >
>> > > >> > > > maulin.vasavada@gmail.com>
>> > > >> > > >
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Bump! Can somebody please review this?
>> > > >> > > >
>> > > >> > > > --
>> > > >> > > > Thanks,
>> > > >> > > > M.Sai Sandeep
>> > > >> > > >
>> > > >> > > >
>> > > >> > >
>> > > >> >
>> > > >>
>> > > >
>> > >
>> >
>>
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Colin McCabe <co...@cmccabe.xyz>.
I think we could reuse KIP-383 for a KIP making SSLEngineBuilder pluggable. However, we need to be able to change this code in the future. That means it needs to be a very minimal interface. Ideally just the createSslEngine and shouldBeRebuilt methods.
best,
Colin
On Fri, Aug 30, 2019, at 12:06, Maulin Vasavada wrote:
> +1 for making SslEngineBuilder configurable upon more thoughts.
>
> However, in the abstraction and default implementation we should make sure
> when we do have a requirement to plugin custom key/trust store people don't
> have to write lot more code which may not be related to it.
>
> Having said that, does this mean, we resurrect KIP-383 and update it with
> latest context and go from there?
>
> We are willing to take up that work for making it configurable.
>
> Thanks
> Maulin
>
>
>
>
>
> On Fri, Aug 30, 2019 at 10:34 AM Maulin Vasavada <ma...@gmail.com>
> wrote:
>
> > Why don't we make SSLEngineBuilder code delegate the whole Key/Trust store
> > initialization to the interfaces we are proposing? Default implementation
> > for those key/trust store loader interfaces will be "file based" loading vs
> > if somebody wants to customize any of it they can.
> >
> > Would that make sense?
> >
> > On Fri, Aug 30, 2019 at 10:03 AM Colin McCabe <cm...@apache.org> wrote:
> >
> >> +1 for making SslEngineBuilder configurable. This would give
> >> implementers a lot more flexibility-- to use key distribution methods that
> >> were not files, for example.
> >>
> >> best,
> >> Colin
> >>
> >>
> >> On Fri, Aug 30, 2019, at 02:03, Rajini Sivaram wrote:
> >> > Just to make sure we are on the same page - KIP-383 was written before
> >> > the
> >> > code was refactored. The refactoring addressed some of the concerns of
> >> > KIP-383. My suggestion was to make SslEngineBuilder configurable. The
> >> > default implementation of this pluggable class would be
> >> >
> >> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/ssl/SslEngineBuilder.java
> >> .
> >> > That should give an idea of the size of the configurable part that a
> >> > custom
> >> > class needs to implement. A large part of that is about loading
> >> > keystore/truststore. I agree it has slightly more code than KIP-486
> >> > proposes, but since it lets you customize creation of SSLEngine, it
> >> > would
> >> > address every possible scenario.
> >> >
> >> > Thoughts?
> >> >
> >> >
> >> > On Fri, Aug 30, 2019 at 2:02 AM Maulin Vasavada <
> >> maulin.vasavada@gmail.com>
> >> > wrote:
> >> >
> >> > > I thought about it more. I feel that no matter how we refactor the
> >> code
> >> > > (with or without KIP-383 integrated), ultimately the need of
> >> customizing
> >> > > loading for keys and certs will still remain. Whenever that need
> >> arises we
> >> > > might end up thinking about the solution suggested by our KIP-486.
> >> Hence
> >> > > regardless of the other KIPs and configurations "if we do need to
> >> customize
> >> > > loading of keys/certs, we will need the code changes suggested by this
> >> > > KIP".
> >> > >
> >> > > Let me know what you guys think.
> >> > >
> >> > > Harsha, we are working on changing the interfaces for key/trust store
> >> > > loaders with Certificate and PrivateKey objects. Will probably be
> >> able to
> >> > > update it later today or tomorrow.
> >> > >
> >> > > Thanks
> >> > > Maulin
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > On Thu, Aug 29, 2019 at 2:30 PM Maulin Vasavada <
> >> maulin.vasavada@gmail.com
> >> > > >
> >> > > wrote:
> >> > >
> >> > > > On that, I actually looked at KIP-383 before briefly. However, that
> >> > > > sounded like lot of changes suggested.
> >> > > >
> >> > > > One "key" thing we have to keep in mind is - IF we need lot of
> >> > > > customization Kafka already allows you to use your SslProvider via
> >> > > > ssl.providers or the changes done by KIP-492 and
> >> > > > SSLContext.getInstance(protocol, provider) call allows us to return
> >> the
> >> > > > SSLContext with "ALL" the details we would like to customize. Hence
> >> I am
> >> > > > not sure that customization suggested by KIP-383 would be worth the
> >> > > effort.
> >> > > > We also have similar SSLContext customization outside of Kafka.
> >> > > >
> >> > > > Thanks
> >> > > > Maulin
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Thu, Aug 29, 2019 at 12:47 PM Pellerin, Clement <
> >> > > > Clement_Pellerin@ibi.com> wrote:
> >> > > >
> >> > > >> KIP-383 in its present form was vetoed because it was not possible
> >> to
> >> > > add
> >> > > >> validation of custom properties in a future KIP. The solution to
> >> that is
> >> > > >> the first proposal I wrote for KIP-383 which made the whole
> >> SslFactory
> >> > > >> pluggable. That first solution was also vetoed hence the deadlock.
> >> > > >>
> >> > > >> Replacing the whole factory was a much nicer solution. It was
> >> vetoed
> >> > > >> because doing this almost invariably meant the replacement lost
> >> all the
> >> > > >> complex validation code in the default SslFactory.
> >> > > >>
> >> > > >> My current idea is to extract the validation code into another
> >> public
> >> > > API
> >> > > >> that SslFactory would call. I did not look at the newly refactored
> >> code
> >> > > and
> >> > > >> I did not study how to do this yet. KIP-383 was not popular at the
> >> time
> >> > > and
> >> > > >> designing a new solution is a lot of work.
> >> > > >>
> >> > > >> Is there interest from 3 binding voters for something like this?
> >> > > >>
> >> > > >> -----Original Message-----
> >> > > >> From: Rajini Sivaram [mailto:rajinisivaram@gmail.com]
> >> > > >> Sent: Thursday, August 29, 2019 2:57 PM
> >> > > >> To: dev
> >> > > >> Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and
> >> > > >> TrustStore
> >> > > >>
> >> > > >> Hi Maulin,
> >> > > >>
> >> > > >> In SSL scenarios, I imagine security providers introduced by
> >> KIP-492 are
> >> > > >> likely to be most useful when you want to use third party
> >> providers. The
> >> > > >> biggest advantage of the config from that KIP is that you don't
> >> need to
> >> > > >> write much code to integrate existing security providers into Kafka
> >> > > >> brokers
> >> > > >> or clients. As I understand it, KIP-486 is a more convenient
> >> option for
> >> > > >> the
> >> > > >> specific problem of loading keystores/truststores differently. It
> >> can be
> >> > > >> achieved in theory with KIP-492, but KIP-486 is a much simpler
> >> option
> >> > > for
> >> > > >> this case.
> >> > > >>
> >> > > >> My concern about KIP-486 is that it introduces yet another
> >> interface
> >> > > into
> >> > > >> our already complex security code, while only solving one
> >> particular use
> >> > > >> case. Have you looked at
> >> > > >>
> >> > > >>
> >> > >
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-383%3A++Pluggable+interface+for+SSL+Factory
> >> > > >> ?
> >> > > >> The goal was to make
> >> > > >> org.apache.kafka.common.security.ssl.SslEngineBuilder pluggable.
> >> > > >> The code has already been refactored by Colin after that KIP was
> >> > > written,
> >> > > >> making it easier to implement KIP-383. This should enable you to
> >> load
> >> > > your
> >> > > >> keystores and truststores differently. Using a pluggable
> >> > > SslEngineBuilder
> >> > > >> will also solve several other use cases at the same time. KIP-383
> >> hasn't
> >> > > >> been voted through yet, but perhaps you could take a look and we
> >> could
> >> > > >> revive that instead if it solves your use case as well?
> >> > > >>
> >> > > >> Regards,
> >> > > >>
> >> > > >> Rajini
> >> > > >>
> >> > > >>
> >> > > >> On Thu, Aug 29, 2019 at 6:42 PM Maulin Vasavada <
> >> > > >> maulin.vasavada@gmail.com>
> >> > > >> wrote:
> >> > > >>
> >> > > >> > Hi Harsha
> >> > > >> >
> >> > > >> > Thank you. Appreciate your time and support on this. Let me go
> >> back do
> >> > > >> some
> >> > > >> > more research and get back to you on the KeyStore interface part.
> >> > > >> > Basically, if we return certs and keys in the interface then
> >> Kafka
> >> > > code
> >> > > >> > will have to build KeyStore object - which is also reasonable.
> >> > > >> >
> >> > > >> > Thanks
> >> > > >> > Maulin
> >> > > >> >
> >> > > >> > On Thu, Aug 29, 2019 at 10:01 AM Harsha Chintalapani <
> >> kafka@harsha.io
> >> > > >
> >> > > >> > wrote:
> >> > > >> >
> >> > > >> > > Hi Maulin,
> >> > > >> > > Use cases are clear now. I am +1 for moving
> >> > > >> forward
> >> > > >> > > with the discussions on having such configurable option for
> >> users.
> >> > > But
> >> > > >> > the
> >> > > >> > > interfaces is proposed doesn't look right to me. We are still
> >> > > talking
> >> > > >> > about
> >> > > >> > > keystore interfaces. Given keystore's are used as filebased
> >> way of
> >> > > >> > > transporting certificates I am not sure it will help the rest
> >> of the
> >> > > >> > > user-base.
> >> > > >> > > In short, I am +1 on the KIP's motivation
> >> and only
> >> > > >> have
> >> > > >> > > questions around returning keystores instead of returning
> >> certs,
> >> > > >> private
> >> > > >> > > keys etc. . If others in the community are ok with such
> >> interface we
> >> > > >> can
> >> > > >> > > move forward.
> >> > > >> > >
> >> > > >> > > Thanks,
> >> > > >> > > Harsha
> >> > > >> > >
> >> > > >> > >
> >> > > >> > > On Wed, Aug 28, 2019 at 1:51 PM, Maulin Vasavada <
> >> > > >> > > maulin.vasavada@gmail.com>
> >> > > >> > > wrote:
> >> > > >> > >
> >> > > >> > > > Hi Harsha
> >> > > >> > > >
> >> > > >> > > > As we synced-up offline on this topic, we hope you don't
> >> have any
> >> > > >> more
> >> > > >> > > > clarifications that you are seeking. If that is the case,
> >> can you
> >> > > >> > please
> >> > > >> > > > help us move this forward and discuss what changes you would
> >> > > expect
> >> > > >> on
> >> > > >> > > the
> >> > > >> > > > KIP design in order to make it valuable contribution?
> >> > > >> > > >
> >> > > >> > > > Just FYI - we verified our primary design change with the
> >> author
> >> > > of
> >> > > >> > Sun's
> >> > > >> > > > X509 Trustmanager implementation and the outcome is that
> >> what we
> >> > > are
> >> > > >> > > > proposing makes sense at the heart of it - "Instead of
> >> writing
> >> > > >> > > TrustManager
> >> > > >> > > > just plugin the Trust store". We are open to discuss
> >> additional
> >> > > >> changes
> >> > > >> > > > that you/anybody else would like to see on the functionality
> >> > > >> however.
> >> > > >> > > >
> >> > > >> > > > Thanks
> >> > > >> > > > Maulin
> >> > > >> > > >
> >> > > >> > > > On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada <
> >> > > >> > > maulin.vasavada@gmail.com>
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Hi Harsha
> >> > > >> > > >
> >> > > >> > > > Any response on my question? I feel this KIP is worth
> >> > > accommodating.
> >> > > >> > Your
> >> > > >> > > > help is much appreciated.
> >> > > >> > > >
> >> > > >> > > > Thanks
> >> > > >> > > > Maulin
> >> > > >> > > >
> >> > > >> > > > On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada <
> >> > > >> > maulin.vasavada@gmail.
> >> > > >> > > > com> wrote:
> >> > > >> > > >
> >> > > >> > > > Hi Harsha
> >> > > >> > > >
> >> > > >> > > > I've examined the SPIFFE provider more and have one question
> >> -
> >> > > >> > > >
> >> > > >> > > > If SPIFFE didn't have a need to do checkSpiffeId() call at
> >> the
> >> > > below
> >> > > >> > > > location, would you really still write the Provider? *OR*
> >> Would
> >> > > you
> >> > > >> > just
> >> > > >> > > > use TrustManagerFactory.init(KeyStore) signature to pass the
> >> > > >> KeyStore
> >> > > >> > > from
> >> > > >> > > > set of certs returned by spiffeIdManager. getTrustedCerts()?
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> >
> >> > >
> >> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> >> > > >> > > > provider/CertificateUtils.java#L100
> >> > > >> > > >
> >> > > >> > > > /**
> >> > > >> > > >
> >> > > >> > > > * Validates that the SPIFFE ID is present and matches the
> >> SPIFFE
> >> > > ID
> >> > > >> > > > configured in
> >> > > >> > > > * the java.security property ssl.spiffe.accept
> >> > > >> > > > *
> >> > > >> > > > * If the authorized spiffe ids list is empty any spiffe id is
> >> > > >> > authorized
> >> > > >> > > > *
> >> > > >> > > > * @param chain an array of X509Certificate that contains the
> >> > > Peer's
> >> > > >> > SVID
> >> > > >> > > > to be validated
> >> > > >> > > > * @throws CertificateException when either the certificates
> >> > > doesn't
> >> > > >> > have
> >> > > >> > > a
> >> > > >> > > > SPIFFE ID or the SPIFFE ID is not authorized
> >> > > >> > > > */
> >> > > >> > > > static void checkSpiffeId(X509Certificate[] chain) throws
> >> > > >> > > > CertificateException {
> >> > > >> > > >
> >> > > >> > > > Thanks
> >> > > >> > > > Maulin
> >> > > >> > > >
> >> > > >> > > > On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani <
> >> > > >> kafka@harsha.io>
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Maulin,
> >> > > >> > > > The code parts you are pointing are specific for Spiffe and
> >> if
> >> > > >> > > > you are talking about validate method which uses PKIX check
> >> like
> >> > > any
> >> > > >> > > other
> >> > > >> > > > provider does.
> >> > > >> > > > If you want to default to SunJSSE everywhere you can do so by
> >> > > >> > delegating
> >> > > >> > > > the calls in these methods to SunJSSE provider.
> >> > > >> > > >
> >> > > >> > > > TrustManagerFactory tmf = TrustManagerFactory
> >> > > >> > > > .getInstance(TrustManagerFactory.getDefaultAlgorithm());and
> >> use
> >> > > >> > > > tmf.chekServerTrusted()
> >> > > >> > > > or use
> >> > > >> > > > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> >> > > >> > > > TrustManagerFactory.html#getInstance(java.lang.String)if you
> >> want
> >> > > a
> >> > > >> > > > specific provider.
> >> > > >> > > >
> >> > > >> > > > -Harsha
> >> > > >> > > >
> >> > > >> > > > On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada <
> >> > > >> > maulin.vasavada@gmail.
> >> > > >> > > > com>
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Okay, so I take that you guys agree that I have to write a
> >> > > 'custom'
> >> > > >> > > > algorithm and a provider to make it work , correct?
> >> > > >> > > >
> >> > > >> > > > Now, for Harsha's comment "Here the 'Custom' Algorithm is
> >> not an
> >> > > >> > > > implementation per say , ..." , I diagree. You can refer to
> >> > > https://
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> >
> >> > > >>
> >> > >
> >> github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/
> >> > > >> > > >
> >> > > >> > > > SpiffeTrustManager.java#L91 <
> >> http://spiffetrustmanager.java/#L91>
> >> > > >> and
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> >
> >> > >
> >> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> >> > > >> > > >
> >> > > >> > > > provider/CertificateUtils.java#L100
> >> > > >> > > >
> >> > > >> > > > "that code" is the customization you have for the custom way
> >> to
> >> > > >> check
> >> > > >> > > > something on top of regular checks. That method is NOT doing
> >> > > custom
> >> > > >> > > > truststore loading. It is validating/verifying something in
> >> the
> >> > > >> > > >
> >> > > >> > > > "custom"
> >> > > >> > > >
> >> > > >> > > > way with spiffeId.
> >> > > >> > > > I bet that without that you won't have a need of the custom
> >> > > >> algorithm
> >> > > >> > > >
> >> > > >> > > > in
> >> > > >> > > >
> >> > > >> > > > the first place.
> >> > > >> > > >
> >> > > >> > > > Let me know if you agree to this.
> >> > > >> > > >
> >> > > >> > > > Thanks
> >> > > >> > > > Maulin
> >> > > >> > > >
> >> > > >> > > > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <
> >> mprsai@gmail.com>
> >> > > >> > > >
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Hi Maulin, thanks for the discussion. As Harsha pointed out,
> >> to
> >> > > use
> >> > > >> the
> >> > > >> > > > KIP492, you need to create a new provider, register a *new*
> >> custom
> >> > > >> > > > algorithm for your keymanager and trustmanager factory
> >> > > >> implementations.
> >> > > >> > > > After this, the kafka server configuration can be done as
> >> given
> >> > > >> below
> >> > > >> > > >
> >> > > >> > > > # Register the provider class with custom algorithm, say
> >> CUSTOM
> >> > > >> > > >
> >> > > >> > > > security.
> >> > > >> > > >
> >> > > >> > > > provider.classes=com.company.security.CustomProvider
> >> > > >> > > > <http://provider.classes
> >> =com.company.security.customprovider/>
> >> > > >> > > > <http://security.provider.classes
> >> > > >> > > >
> >> > > >> > > > =com.company.security.customprovider/>
> >> > > >> > > >
> >> > > >> > > > # Register the keymanager and trustmanager algorithms
> >> > > >> > > > # These algorithms indicate that the Keymanager and
> >> Trustmanagers
> >> > > >> > > > registered under the algorithm "CUSTOM" needs to be used
> >> > > >> > > > ssl.trustmanager.algorithm=CUSTOM
> >> > > >> > > > ssl.keymanager.algorithm=CUSTOM
> >> > > >> > > >
> >> > > >> > > > And the customprovider looks like this...
> >> > > >> > > >
> >> > > >> > > > public class CustomProvider extends Provider {
> >> > > >> > > > public CustomProvider() {
> >> > > >> > > > super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and
> >> > > TrustStore");
> >> > > >> > > > super.put("KeyManagerFactory.CUSTOM",
> >> "customKeyManagerFactory");
> >> > > >> > > > super.put("TrustManagerFactory.CUSTOM",
> >> > > >> > > > "customTrustManagerFactory");
> >> > > >> > > > }
> >> > > >> > > > }
> >> > > >> > > >
> >> > > >> > > > The PR for this is in review and can be found here:
> >> > > >> > > >
> >> > > >> > > > https://github.com/
> >> > > >> > > >
> >> > > >> > > > apache/kafka/pull/7090
> >> > > >> > > > This PR includes the fixed insertProviderAt function call.
> >> > > >> > > >
> >> > > >> > > > On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani <
> >> > > >> kafka@harsha.io>
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Answers are in-line
> >> > > >> > > >
> >> > > >> > > > On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada <
> >> > > >> > > >
> >> > > >> > > > maulin.vasavada@gmail.
> >> > > >> > > >
> >> > > >> > > > com
> >> > > >> > > >
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Hi Colin
> >> > > >> > > >
> >> > > >> > > > When I refer to "standard" or "custom" algorithms I am
> >> following
> >> > > >> Java
> >> > > >> > > > security Provider Terminology. You can refer to
> >> > > >> > > >
> >> https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> >> > > >> > > > StandardNames.html#TrustManagerFactory link I provided
> >> earlier in
> >> > > >> the
> >> > > >> > > > emails. It says PKIX is the default Algorithm for
> >> > > >> TrustManagerFactory.
> >> > > >> > > >
> >> > > >> > > > 1. For SPIFFE, I am not sure why you are saying 'it does not
> >> > > >> implement
> >> > > >> > > > custom algorithms' because the following file clearly
> >> indicates
> >> > > >> that it
> >> > > >> > > > does use custom algorithm-
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> >
> >> > >
> >> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> >> > > >> > > >
> >> > > >> > > > provider/SpiffeProvider.java#L17
> >> > > >> > > >
> >> > > >> > > > Algorithm value:
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> >
> >> > >
> >> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> >> > > >> > > >
> >> > > >> > > > provider/SpiffeProviderConstants.java#L6
> >> > > >> > > >
> >> > > >> > > > @Harsha do you want to chime in since you use that provider?
> >> > > >> > > >
> >> > > >> > > > Here the "Custom" Algorithm is not an implementation per say
> >> ,
> >> > > >> rather
> >> > > >> > > >
> >> > > >> > > > used
> >> > > >> > > >
> >> > > >> > > > to invoke the custom trust store factory and key manager
> >> factory.
> >> > > >> You
> >> > > >> > > >
> >> > > >> > > > are
> >> > > >> > > >
> >> > > >> > > > not moving away from "standard" alogrithms that are
> >> available.
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> >
> >> > >
> >> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> >> > > >> > > >
> >> > > >> > > > provider/SpiffeTrustManager.java
> >> > > >> > > >
> >> > > >> > > > As you can see it delegates all the calls of verification of
> >> > > >> > > >
> >> > > >> > > > certificate
> >> > > >> > > >
> >> > > >> > > > to
> >> > > >> > > >
> >> > > >> > > > the default implementation available.
> >> > > >> > > > So in our implementation we still use PKIX to verify the
> >> > > certificate
> >> > > >> > > > chain. So you are not losing anything here and Spiffe is not
> >> > > >> > > >
> >> > > >> > > > reimplementing
> >> > > >> > > >
> >> > > >> > > > the verification process.
> >> > > >> > > >
> >> > > >> > > > 2. I already mentioned in my 3rd point, in my previous post,
> >> why
> >> > > >> using
> >> > > >> > > >
> >> > > >> > > > ssl.provider does NOT work. I updated KIP-486 in "rejected
> >> > > >> > > >
> >> > > >> > > > alternatives"
> >> > > >> > > >
> >> > > >> > > > also why ssl.provider does not work.
> >> > > >> > > >
> >> > > >> > > > As mentioned before , provider is the right way to go. I am
> >> still
> >> > > >> not
> >> > > >> > > > understanding the gap is.
> >> > > >> > > > If I understand correctly your argument is , provider is
> >> going to
> >> > > >> ask
> >> > > >> > > >
> >> > > >> > > > to
> >> > > >> > > >
> >> > > >> > > > implement a custom algorithm.
> >> > > >> > > > My answer to that is , no you are not re-implementing the
> >> > > algorithm.
> >> > > >> > > >
> >> > > >> > > > Please
> >> > > >> > > >
> >> > > >> > > > check the above link , TrustManager implementation it
> >> delegates
> >> > > the
> >> > > >> > > >
> >> > > >> > > > calls
> >> > > >> > > >
> >> > > >> > > > back. There is no need to implement your own here.
> >> > > >> > > >
> >> > > >> > > > 3. Security.insertProviderAt() comments were based on
> >> assumption
> >> > > if
> >> > > >> > > >
> >> > > >> > > > KIP-492
> >> > > >> > > >
> >> > > >> > > > changes are done and we use that mechanism to configure
> >> providers
> >> > > >> > > >
> >> > > >> > > > instead
> >> > > >> > > >
> >> > > >> > > > of ssl.provider configuration.
> >> > > >> > > >
> >> > > >> > > > KIP-492 has patch available and going through review.
> >> > > >> > > >
> >> > > >> > > > Can you read my all the points, I mentioned in my previous
> >> post,
> >> > > >> very
> >> > > >> > > >
> >> > > >> > > > carefully? I am covering all the aspects in explaining. I am
> >> open
> >> > > to
> >> > > >> > > >
> >> > > >> > > > still
> >> > > >> > > >
> >> > > >> > > > discuss more to clarify any doubts.
> >> > > >> > > >
> >> > > >> > > > "3. If we just use existing ssl.provider kafka configuration
> >> then
> >> > > >> our
> >> > > >> > > > provider will be used in SSLContext.getInstance(protocol,
> >> > > provider)
> >> > > >> > > >
> >> > > >> > > > call
> >> > > >> > > >
> >> > > >> > > > in
> >> > > >> > > >
> >> > > >> > > > SslFactory.java <http://sslfactory.java/> <
> >> > > http://sslfactory.java/>
> >> > > >> <
> >> > > >> > > > http://sslfactory.java/>
> >> > > >> > > >
> >> > > >> > > > and
> >> > > >> > > >
> >> > > >> > > > if our provider does not have
> >> > > >> > > > implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it
> >> breaks
> >> > > (we
> >> > > >> > > >
> >> > > >> > > > tested
> >> > > >> > > >
> >> > > >> > > > it). Example: In MyProvider sample above you see that I
> >> didn't add
> >> > > >> > > > SSLContext.TLSv1 as
> >> > > >> > > > "Service+Algorithm" and that didn't work for me. In SPIFFE
> >> > > provider
> >> > > >> you
> >> > > >> > > > don't have this challenge since you are planning to bypass
> >> > > >> ssl.provider
> >> > > >> > > >
> >> > > >> > > > as
> >> > > >> > > >
> >> > > >> > > > you mention in the KIP-492."
> >> > > >> > > >
> >> > > >> > > > Yes here you need to pass the protocol that your
> >> > > >> > > >
> >> > > >> > > > KeyManager/TrustManager
> >> > > >> > > >
> >> > > >> > > > registered with and in no way its deviating from TLS RFC
> >> spec.
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> >
> >> > >
> >> https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/
> >> > > >> > > >
> >> > > >> > > > javax/net/ssl/SSLContext.java#L134
> >> > > >> > > >
> >> > > >> > > > My suggestion here is for you to implement a simple Security
> >> > > >> Provider
> >> > > >> > > >
> >> > > >> > > > as
> >> > > >> > > >
> >> > > >> > > > you did before and register a Algorithm. You can use the
> >> existing
> >> > > >> > > > implementation in SpiffeProvider and plug in changes where
> >> you
> >> > > need
> >> > > >> to
> >> > > >> > > > retrieve the certificates from by making RPC call.
> >> > > >> > > >
> >> > > >> > > > Run an end-to-end test with Kafka broker coming up with your
> >> > > >> provider
> >> > > >> > > > making calls to RPC call. You do need to pass the "custom
> >> > > algorithm"
> >> > > >> > > >
> >> > > >> > > > that
> >> > > >> > > >
> >> > > >> > > > you registered in your key manager to invoke the provider. I
> >> think
> >> > > >> your
> >> > > >> > > > concern is this approach is replacing the existing known
> >> > > >> ciphersuites
> >> > > >> > > >
> >> > > >> > > > and
> >> > > >> > > >
> >> > > >> > > > certificate validation provided by java. Which its not.
> >> > > >> > > >
> >> > > >> > > > Now test the TLS connection you can do so via openssl
> >> -s_client
> >> > > >> options
> >> > > >> > > >
> >> > > >> > > > to
> >> > > >> > > >
> >> > > >> > > > test if everything else is working.
> >> > > >> > > >
> >> > > >> > > > I am happy to share configs that we used if you are
> >> interested.
> >> > > >> > > >
> >> > > >> > > > Thanks,
> >> > > >> > > > Harsha
> >> > > >> > > >
> >> > > >> > > > Thanks
> >> > > >> > > > Maulin
> >> > > >> > > >
> >> > > >> > > > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <
> >> cmccabe@apache.org>
> >> > > >> > > >
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Hi Maulin,
> >> > > >> > > >
> >> > > >> > > > A lot of JSSE providers don't implement custom algorithms.
> >> Spire
> >> > > is
> >> > > >> a
> >> > > >> > > >
> >> > > >> > > > good
> >> > > >> > > >
> >> > > >> > > > example of a JSSE provider that doesn't, and yet is still
> >> useful
> >> > > to
> >> > > >> > > >
> >> > > >> > > > many
> >> > > >> > > >
> >> > > >> > > > people. Your JSSE provider can work fine even if it doesn't
> >> > > >> implement a
> >> > > >> > > > custom algorithm.
> >> > > >> > > >
> >> > > >> > > > Maybe I'm missing something, but I don't understand the
> >> discussion
> >> > > >> of
> >> > > >> > > > Security.insertProviderAt() that you included.
> >> SslEngineBuilder
> >> > > >> doesn't
> >> > > >> > > >
> >> > > >> > > > use
> >> > > >> > > >
> >> > > >> > > > that API to get the security provider. Instead, it calls
> >> > > >> > > > "SSLContext.getInstance(protocol, provider)", where provider
> >> is
> >> > > the
> >> > > >> > > >
> >> > > >> > > > name
> >> > > >> > > >
> >> > > >> > > > of the provider.
> >> > > >> > > >
> >> > > >> > > > best,
> >> > > >> > > > Colin
> >> > > >> > > >
> >> > > >> > > > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
> >> > > >> > > >
> >> > > >> > > > On top of everything above I feel strongly to add the 4th
> >> point
> >> > > >> which
> >> > > >> > > >
> >> > > >> > > > is
> >> > > >> > > >
> >> > > >> > > > based on Java APIs for TrustManagerFactory.init(KeyStore) (
> >> > > >> > > >
> >> > > >> > > > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> >> > > >> > > > TrustManagerFactory.html#init(java.security.KeyStore
> >> > > >> > > > <http://java.security.keystore/>
> >> > > >> > > > <http://java.security.keystore/>)
> >> > > >> > > > )
> >> > > >> > > >
> >> > > >> > > > and KeyManagerFactory.init(KeyStore, char[]) (
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> > >
> >> > > >> >
> >> > > >>
> >> > >
> >> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory
> >> > > >> > > > .
> >> > > >> > > >
> >> > > >> > > > html#init(java.security.KeyStore <
> >> http://java.security.keystore/>
> >> > > >> > > <http://
> >> > > >> > > > java.security.keystore/
> >> > > >> > > > ,%20char[])
> >> > > >> > > >
> >> > > >> > > > ).
> >> > > >> > > >
> >> > > >> > > > 4. The above APIs are intended to support providing
> >> "trust/key
> >> > > >> > > >
> >> > > >> > > > material"
> >> > > >> > > >
> >> > > >> > > > from the user without having to write their own
> >> > > >> > > >
> >> > > >> > > > TrustManager/KeyManagers.
> >> > > >> > > >
> >> > > >> > > > To quote from the TrustManagerFactory.init()'s documentation
> >> > > >> > > >
> >> > > >> > > > "Initializes
> >> > > >> > > >
> >> > > >> > > > this factory with a source of certificate authorities and
> >> related
> >> > > >> trust
> >> > > >> > > > material."
> >> > > >> > > > To quote from the KeyManagerFactory.init()'s documentation
> >> > > >> "Initializes
> >> > > >> > > > this factory with a source of key material."
> >> > > >> > > >
> >> > > >> > > > Based on this it is clear that there is a flexibility
> >> provided by
> >> > > >> Java
> >> > > >> > > >
> >> > > >> > > > to
> >> > > >> > > >
> >> > > >> > > > to enable developers to provide the required trust/key
> >> material
> >> > > >> loaded
> >> > > >> > > >
> >> > > >> > > > from
> >> > > >> > > >
> >> > > >> > > > "anywhere" without requiring them to write custom provider OR
> >> > > >> trust/key
> >> > > >> > > > managers. This same flexibility is reflected in Kafka code
> >> also
> >> > > >> where
> >> > > >> > > >
> >> > > >> > > > it
> >> > > >> > > >
> >> > > >> > > > loads the trust/keys from a local file and doesn't require
> >> > > writing a
> >> > > >> > > > Provider necessarily. If we do NOT have a custom algorithm,
> >> it
> >> > > makes
> >> > > >> > > >
> >> > > >> > > > less
> >> > > >> > > >
> >> > > >> > > > sense to write a Provider.
> >> > > >> > > >
> >> > > >> > > > Thanks
> >> > > >> > > > Maulin
> >> > > >> > > >
> >> > > >> > > > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
> >> > > >> > > >
> >> > > >> > > > maulin.vasavada@gmail.com>
> >> > > >> > > >
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Hi Harsha/Colin
> >> > > >> > > >
> >> > > >> > > > I did the sample with a custom Provider for
> >> TrustStoreManager and
> >> > > >> tried
> >> > > >> > > > using ssl.provider Kafka config AND the way KIP-492 is
> >> suggesting
> >> > > >> (by
> >> > > >> > > > adding Provider programmatically instead of relying on
> >> > > >> > > >
> >> > > >> > > > ssl.provider+java.
> >> > > >> > > >
> >> > > >> > > > security. The below sample is followed by my detailed
> >> findings.
> >> > > I'll
> >> > > >> > > > appreciate if you can go through it carefully and see
> >> > > >> > > >
> >> > > >> > > > if you
> >> > > >> > > >
> >> > > >> > > > see my point.
> >> > > >> > > >
> >> > > >> > > > package providertest;
> >> > > >> > > >
> >> > > >> > > > import java.security.Provider <
> >> http://java.security.provider/>
> >> > > >> > <http://
> >> > > >> > > > java.security.provider/>
> >> > > >> > > >
> >> > > >> > > > <http://
> >> > > >> > > >
> >> > > >> > > > java.security.provider/>;
> >> > > >> > > >
> >> > > >> > > > public class MyProvider extends Provider {
> >> > > >> > > >
> >> > > >> > > > private static final String name = "MyProvider"; private
> >> static
> >> > > >> double
> >> > > >> > > > version = 1.0d;
> >> > > >> > > > private static String info = "Maulin's SSL Provider
> >> v"+version;
> >> > > >> > > >
> >> > > >> > > > public MyProvider() {
> >> > > >> > > > super(name, version, info);
> >> > > >> > > > this.put("TrustManagerFactory.PKIX",
> >> > > >> > > >
> >> > > >> > > > "providertest.MyTrustManagerFactory");
> >> > > >> > > >
> >> > > >> > > > }
> >> > > >> > > > }
> >> > > >> > > >
> >> > > >> > > > *Details:*
> >> > > >> > > >
> >> > > >> > > > KIP-492 documents that it will use Security.addProvider()
> >> assuming
> >> > > >> it
> >> > > >> > > >
> >> > > >> > > > will
> >> > > >> > > >
> >> > > >> > > > add it as position '0' which is not a correct assumption. The
> >> > > >> > > > addProvider()'s documentation says it will add it to the last
> >> > > >> available
> >> > > >> > > > position. You may want to correct that to say
> >> > > >> > > > Security.insertProviderAt(provider, 1).
> >> > > >> > > >
> >> > > >> > > > Now coming back to our specific discussion,
> >> > > >> > > >
> >> > > >> > > > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when
> >> you
> >> > > add
> >> > > >> > > >
> >> > > >> > > > that
> >> > > >> > > >
> >> > > >> > > > provider in the provider list via Security.addProvider() the
> >> > > >> position
> >> > > >> > > >
> >> > > >> > > > where
> >> > > >> > > >
> >> > > >> > > > it gets added doesn't matter (even if you don't end up
> >> adding it
> >> > > as
> >> > > >> > > >
> >> > > >> > > > first
> >> > > >> > > >
> >> > > >> > > > entry) since that is the ONLY provider for SPIFFE specific
> >> > > algorithm
> >> > > >> > > >
> >> > > >> > > > you
> >> > > >> > > >
> >> > > >> > > > might have.
> >> > > >> > > >
> >> > > >> > > > We do *not* have custom algorithm for Key/Trust StoreMangers.
> >> > > Which
> >> > > >> > > >
> >> > > >> > > > means
> >> > > >> > > >
> >> > > >> > > > we have to use X509, PKIX etc "Standard Algorithms" ((
> >> > > >> > > >
> >> > > >> > > >
> >> https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> >> > > >> > > > StandardNames.html
> >> > > >> > > > ))
> >> > > >> > > >
> >> > > >> > > > in our provider to override the TrustStoreManager (see my
> >> sample
> >> > > >> code)
> >> > > >> > > >
> >> > > >> > > > and
> >> > > >> > > >
> >> > > >> > > > KeyStoreManger and KeyManager. This creates another challenge
> >> > > >> > > >
> >> > > >> > > > mentioned in
> >> > > >> > > >
> >> > > >> > > > the below point.
> >> > > >> > > >
> >> > > >> > > > 2. In order to make our Provider for loading custom
> >> TrustStore
> >> > > >> work, we
> >> > > >> > > > have to add the provider as 'first' in the list since there
> >> are
> >> > > >> others
> >> > > >> > > >
> >> > > >> > > > with
> >> > > >> > > >
> >> > > >> > > > the same algorithm.
> >> > > >> > > >
> >> > > >> > > > However, the programatic way of adding provider
> >> > > >> > > > (Security.insertProviderAt()) is *not* deterministic for
> >> ordering
> >> > > >> since
> >> > > >> > > > different code can call the method for a different provider
> >> and
> >> > > >> > > >
> >> > > >> > > > depending
> >> > > >> > > >
> >> > > >> > > > upon the order of the call our provider can be first or
> >> pushed
> >> > > down
> >> > > >> the
> >> > > >> > > > list. This can happen very well in any client application
> >> using
> >> > > >> Kafka.
> >> > > >> > > >
> >> > > >> > > > This
> >> > > >> > > >
> >> > > >> > > > is specially problematic for a case when you want to
> >> guarantee
> >> > > order
> >> > > >> > > >
> >> > > >> > > > for a
> >> > > >> > > >
> >> > > >> > > > Provider having "Standard Algorithms".
> >> > > >> > > >
> >> > > >> > > > If we add our provider in java.security file that definitely
> >> > > >> guarantees
> >> > > >> > > > the order(unless somebody calls removeProvider() which is
> >> less
> >> > > >> > > >
> >> > > >> > > > likely). But
> >> > > >> > > >
> >> > > >> > > > if we add our provider in java.security file it will defeat
> >> the
> >> > > >> > > >
> >> > > >> > > > purpose of
> >> > > >> > > >
> >> > > >> > > > the KIP-492.
> >> > > >> > > >
> >> > > >> > > > In the gist - Apache Kafka must not rely on
> >> "un-deterministic"
> >> > > >> method
> >> > > >> > > >
> >> > > >> > > > to
> >> > > >> > > >
> >> > > >> > > > rely on Provider ordering.
> >> > > >> > > >
> >> > > >> > > > 3. If we just use existing ssl.provider kafka configuration
> >> then
> >> > > our
> >> > > >> > > > provider will be used in SSLContext.getInstance(protocol,
> >> > > provider)
> >> > > >> > > >
> >> > > >> > > > call in
> >> > > >> > > >
> >> > > >> > > > SslFactory.java <http://sslfactory.java/> <
> >> > > http://sslfactory.java/>
> >> > > >> <
> >> > > >> > > > http://sslfactory.java/>
> >> > > >> > > >
> >> > > >> > > > and
> >> > > >> > > >
> >> > > >> > > > if our provider does not have implementation for
> >> > > >> > > > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks
> >> > > >> > > >
> >> > > >> > > > (we
> >> > > >> > > >
> >> > > >> > > > tested it). Example:
> >> > > >> > > >
> >> > > >> > > > In
> >> > > >> > > >
> >> > > >> > > > MyProvider sample above you see that I didn't add
> >> SSLContext.TLSv1
> >> > > >> as
> >> > > >> > > > "Service+Algorithm" and that didn't work for me. In SPIFFE
> >> > > provider
> >> > > >> you
> >> > > >> > > > don't have this challenge since you are planning to bypass
> >> > > >> > > >
> >> > > >> > > > ssl.provider as
> >> > > >> > > >
> >> > > >> > > > you mention in the KIP-492.
> >> > > >> > > >
> >> > > >> > > > *Overall summary,*
> >> > > >> > > >
> >> > > >> > > > 1. Any provider based mechanisms- a) existing ssl.provider
> >> and
> >> > > >> > > >
> >> > > >> > > > b)KIP-492,
> >> > > >> > > >
> >> > > >> > > > for loading key/trust store using "Standard Algorithms" do
> >> not
> >> > > work
> >> > > >> > > >
> >> > > >> > > > 2. Approach suggested in our KIP-486 works without any issue
> >> and
> >> > > it
> >> > > >> is
> >> > > >> > > > *not* our context specific solve
> >> > > >> > > >
> >> > > >> > > > 3. Based on above we feel KIP-492 and KIP-486 are
> >> complimentary
> >> > > >> changes
> >> > > >> > > > and not contradicting or redundent.
> >> > > >> > > >
> >> > > >> > > > If you want we can do a joint session somehow to walk
> >> through the
> >> > > >> > > >
> >> > > >> > > > sample I
> >> > > >> > > >
> >> > > >> > > > have and various experiments I did. I would encourage you to
> >> do
> >> > > >> similar
> >> > > >> > > > exercise by writing a Provider for "Standard Algorithm" for
> >> > > >> > > > TrustStoreManager (like our needs) and see what you find
> >> since
> >> > > only
> >> > > >> > > >
> >> > > >> > > > writing
> >> > > >> > > >
> >> > > >> > > > samples can bring out the complexity/challenges we face.
> >> > > >> > > >
> >> > > >> > > > Thanks
> >> > > >> > > > Maulin
> >> > > >> > > >
> >> > > >> > > > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <
> >> > > >> > > >
> >> > > >> > > > maulin.vasavada@gmail.
> >> > > >> > > >
> >> > > >> > > > com> wrote:
> >> > > >> > > >
> >> > > >> > > > Just to update - still working on it. Get to work only on
> >> and off
> >> > > on
> >> > > >> > > >
> >> > > >> > > > it :(
> >> > > >> > > >
> >> > > >> > > > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
> >> > > >> > > >
> >> > > >> > > > maulin.vasavada@gmail.com>
> >> > > >> > > >
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Hi Harsha
> >> > > >> > > >
> >> > > >> > > > Let me try to write samples and will let you know.
> >> > > >> > > >
> >> > > >> > > > Thanks
> >> > > >> > > > Maulin
> >> > > >> > > >
> >> > > >> > > > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <
> >> harsha.ch@gmail.com>
> >> > > >> > > >
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Hi Maulin,
> >> > > >> > > > With java security providers can be as custom you would
> >> > > >> > > >
> >> > > >> > > > like
> >> > > >> > > >
> >> > > >> > > > it to
> >> > > >> > > > be. If you only want to to implement a custom way of loading
> >> the
> >> > > >> > > >
> >> > > >> > > > keystore
> >> > > >> > > >
> >> > > >> > > > and truststore and not implement any protocol/encryption
> >> handling
> >> > > >> you
> >> > > >> > > >
> >> > > >> > > > can
> >> > > >> > > >
> >> > > >> > > > leave them empty and no need to implement. Have you looked
> >> into
> >> > > the
> >> > > >> > > >
> >> > > >> > > > links I
> >> > > >> > > >
> >> > > >> > > > pasted before?
> >> > > >> > > >
> >> > > >> > > >
> >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> > >
> >> > > >> >
> >> > > >>
> >> > >
> >> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> >> > > >> > > >
> >> > > >> > > > java
> >> > > >> > > >
> >> > > >> > > >
> >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >> > > >> > > > spiffe-security-provider/src/main/java/spiffe/api/provider/
> >> > > >> > > > SpiffeTrustManager.java <http://spiffetrustmanager.java/>
> >> > > <http://
> >> > > >> > > > spiffetrustmanager.java/>
> >> > > >> > > >
> >> > > >> > > >
> >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> > >
> >> > > >> >
> >> > > >>
> >> > >
> >> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> >> > > >> > > >
> >> > > >> > > > java
> >> > > >> > > >
> >> > > >> > > > Can you please tell me which methods are too complex in
> >> above to
> >> > > >> > > >
> >> > > >> > > > implement
> >> > > >> > > >
> >> > > >> > > > or unnecessary? You are changing anything in SSL/TLS
> >> > > implementations
> >> > > >> > > > provided by
> >> > > >> > > >
> >> > > >> > > > All of the implementations delegating the checks to the
> >> default
> >> > > >> > > > implementation anyway.
> >> > > >> > > > Spire agent is an example, its nothing but a GRPC server
> >> listening
> >> > > >> > > >
> >> > > >> > > > on a
> >> > > >> > > >
> >> > > >> > > > unix domain socket . Above code is making a RPC call to the
> >> local
> >> > > >> > > >
> >> > > >> > > > daemon
> >> > > >> > > >
> >> > > >> > > > to
> >> > > >> > > > get the certificate and keys. The mechanics are pretty much
> >> same
> >> > > as
> >> > > >> > > >
> >> > > >> > > > what
> >> > > >> > > >
> >> > > >> > > > you are asking for.
> >> > > >> > > >
> >> > > >> > > > Thanks,
> >> > > >> > > > Harsha
> >> > > >> > > >
> >> > > >> > > > On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
> >> > > >> > > >
> >> > > >> > > > maulin.vasavada@gmail.com>
> >> > > >> > > >
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Imagine a scenario like - We know we have a custom KMS and
> >> as a
> >> > > >> > > >
> >> > > >> > > > Kafka
> >> > > >> > > >
> >> > > >> > > > owner
> >> > > >> > > >
> >> > > >> > > > we want to comply to using that KMS source to load
> >> keys/certs. As
> >> > > >> > > >
> >> > > >> > > > a
> >> > > >> > > >
> >> > > >> > > > Kafka
> >> > > >> > > >
> >> > > >> > > > owner we know how to integrate with KMS but doesn't
> >> necessarily
> >> > > >> > > >
> >> > > >> > > > have
> >> > > >> > > >
> >> > > >> > > > to
> >> > > >> > > >
> >> > > >> > > > know anything about cipher suites, algorithms, and SSL/TLS
> >> > > >> > > >
> >> > > >> > > > implementation.
> >> > > >> > > >
> >> > > >> > > > Going the Provider way requires to know lot more than we
> >> should,
> >> > > >> > > >
> >> > > >> > > > isn't it?
> >> > > >> > > >
> >> > > >> > > > Not that we would have concern/shy-away knowing those
> >> details -
> >> > > >> > > >
> >> > > >> > > > but
> >> > > >> > > >
> >> > > >> > > > if we
> >> > > >> > > >
> >> > > >> > > > don't have to - why should we?
> >> > > >> > > >
> >> > > >> > > > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
> >> > > >> > > >
> >> > > >> > > > maulin.vasavada@gmail.com>
> >> > > >> > > >
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Hi Harsha
> >> > > >> > > >
> >> > > >> > > > We don't have spire (or similar) agents and we do not have
> >> > > >> > > >
> >> > > >> > > > keys/certs
> >> > > >> > > >
> >> > > >> > > > locally on any brokers. To elaborate more on my previous
> >> email,
> >> > > >> > > >
> >> > > >> > > > I agree that Java security Providers are used in much broader
> >> > > >> > > >
> >> > > >> > > > sense
> >> > > >> > > >
> >> > > >> > > > - to
> >> > > >> > > >
> >> > > >> > > > have a particular implementation of an algorithm, use
> >> specific
> >> > > >> > > >
> >> > > >> > > > cipher
> >> > > >> > > >
> >> > > >> > > > suites for SSL , OR in our current team's case have a
> >> > > >> > > >
> >> > > >> > > > particular
> >> > > >> > > >
> >> > > >> > > > way to
> >> > > >> > > >
> >> > > >> > > > leverage pre-generated SSL sessions. However, the scope of
> >> our
> >> > > >> > > >
> >> > > >> > > > KIP
> >> > > >> > > >
> >> > > >> > > > (486)
> >> > > >> > > >
> >> > > >> > > > is
> >> > > >> > > >
> >> > > >> > > > much restricted than that. We merely intend to provide a
> >> custom
> >> > > >> > > > keystore/truststore for our SSL connections and not really
> >> worry
> >> > > >> > > >
> >> > > >> > > > about
> >> > > >> > > >
> >> > > >> > > > underlying specific SSL/TLS implementation. This simplifies
> >> it
> >> > > >> > > >
> >> > > >> > > > a
> >> > > >> > > >
> >> > > >> > > > lot for
> >> > > >> > > >
> >> > > >> > > > us to keep the concerns separate and clear.
> >> > > >> > > >
> >> > > >> > > > I feel our approach is more complimentary such that it allows
> >> > > >> > > >
> >> > > >> > > > for
> >> > > >> > > >
> >> > > >> > > > using
> >> > > >> > > >
> >> > > >> > > > keystores of choice while retaining the flexibility to use
> >> any
> >> > > >> > > > underlying/available Provider for actually making the SSL
> >> call.
> >> > > >> > > >
> >> > > >> > > > We agree with KIP-492's approach based on Providers (and
> >> Java's
> >> > > >> > > > recommendation), but also strongly believe that our approach
> >> can
> >> > > >> > > >
> >> > > >> > > > compliment
> >> > > >> > > >
> >> > > >> > > > it very effectively for reasons explained above.
> >> > > >> > > >
> >> > > >> > > > Thanks
> >> > > >> > > > Maulin
> >> > > >> > > >
> >> > > >> > > > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <
> >> > > >> > > >
> >> > > >> > > > kafka@harsha.io
> >> > > >> > > >
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Hi Maulin,
> >> > > >> > > >
> >> > > >> > > > On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
> >> > > >> > > >
> >> > > >> > > > maulin.vasavada@gmail.
> >> > > >> > > >
> >> > > >> > > > com>
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Hi Harsha
> >> > > >> > > >
> >> > > >> > > > The reason we rejected the SslProvider route is that - we
> >> > > >> > > >
> >> > > >> > > > only
> >> > > >> > > >
> >> > > >> > > > needed
> >> > > >> > > >
> >> > > >> > > > a
> >> > > >> > > >
> >> > > >> > > > custom way to load keys/certs. Not touch any policy that
> >> > > >> > > >
> >> > > >> > > > existing
> >> > > >> > > >
> >> > > >> > > > Providers
> >> > > >> > > >
> >> > > >> > > > govern like SunJSSE Provider.
> >> > > >> > > >
> >> > > >> > > > We have exactly the same requirements to load certs and keys
> >> > > >> > > >
> >> > > >> > > > through
> >> > > >> > > >
> >> > > >> > > > spire
> >> > > >> > > >
> >> > > >> > > > agent. We used security.provider to do that exactly. I am not
> >> > > >> > > >
> >> > > >> > > > sure
> >> > > >> > > >
> >> > > >> > > > why
> >> > > >> > > >
> >> > > >> > > > you
> >> > > >> > > >
> >> > > >> > > > would be modifying any policies provided by default SunJSSE
> >> > > >> > > >
> >> > > >> > > > provider.
> >> > > >> > > >
> >> > > >> > > > Can
> >> > > >> > > >
> >> > > >> > > > you give me an example of having custom provider that will
> >> > > >> > > >
> >> > > >> > > > override an
> >> > > >> > > >
> >> > > >> > > > existing policy in SunJSSE provider.
> >> > > >> > > >
> >> > > >> > > > As pointed out earlier, this kip
> >> > > >> > > >
> >> > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> >> > > >> > > >
> >> KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> >> > > >> > > >
> >> > > >> > > > allows
> >> > > >> > > > you to load security.provider through config.
> >> > > >> > > > Take a look at the examples I gave before
> >> > > >> > > >
> >> > > >> > > >
> >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> > >
> >> > > >> >
> >> > > >>
> >> > >
> >> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> >> > > >> > > >
> >> > > >> > > > java
> >> > > >> > > >
> >> > > >> > > > It registers KeyManagerFactory and TrustManagerFactory and
> >> > > >> > > >
> >> > > >> > > > Keystore
> >> > > >> > > >
> >> > > >> > > > algorithm.
> >> > > >> > > > Implement your custom way of loading Keystore in here
> >> > > >> > > >
> >> > > >> > > >
> >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> > >
> >> > > >> >
> >> > > >>
> >> > >
> >> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> >> > > >> > > >
> >> > > >> > > > java
> >> > > >> > > >
> >> > > >> > > > and Trust manager like here
> >> > > >> > > >
> >> > > >> > > >
> >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >> > > >> > > > spiffe-security-provider/src/main/java/spiffe/api/provider/
> >> > > >> > > > SpiffeTrustManager.java <http://spiffetrustmanager.java/>
> >> > > <http://
> >> > > >> > > > spiffetrustmanager.java/>
> >> > > >> > > >
> >> > > >> > > > In your Kafka client you can set the security.provider to
> >> your
> >> > > >> > > >
> >> > > >> > > > custom
> >> > > >> > > >
> >> > > >> > > > implementation and with this fix
> >> > > >> > > > https://issues.apache.org/jira/browse/KAFKA-8191 you can set
> >> > > >> > > > keyManagerAlgorigthm and trustManagerAlgorithm configs.
> >> > > >> > > >
> >> > > >> > > > All of this is in your clients and broker side and do not
> >> need
> >> > > >> > > >
> >> > > >> > > > to
> >> > > >> > > >
> >> > > >> > > > touch
> >> > > >> > > >
> >> > > >> > > > any
> >> > > >> > > > policy changes at JVM level. You'll register the providers in
> >> > > >> > > >
> >> > > >> > > > the
> >> > > >> > > >
> >> > > >> > > > priority
> >> > > >> > > >
> >> > > >> > > > order and can still have SunJSSE provider and have your
> >> custom
> >> > > >> > > >
> >> > > >> > > > provider
> >> > > >> > > >
> >> > > >> > > > to
> >> > > >> > > >
> >> > > >> > > > implement the key and trust managers.
> >> > > >> > > >
> >> > > >> > > > The ask here is different than KIP-492. We don't have any
> >> need
> >> > > >> > > >
> >> > > >> > > > to
> >> > > >> > > >
> >> > > >> > > > modify/specify the algorithm parameter. Does that make sense?
> >> > > >> > > >
> >> > > >> > > > The ask in KIP is introducing new interfaces where the KIP's
> >> > > >> > > > goal/motivation can be achieved through the security.provider
> >> > > >> > > >
> >> > > >> > > > and
> >> > > >> > > >
> >> > > >> > > > we
> >> > > >> > > >
> >> > > >> > > > worked
> >> > > >> > > > on similar goal without touching any Keystore or Truststore
> >> > > >> > > >
> >> > > >> > > > interfaces.
> >> > > >> > > >
> >> > > >> > > > My advise is against changing or introducing new interfaces
> >> > > >> > > >
> >> > > >> > > > when
> >> > > >> > > >
> >> > > >> > > > it can
> >> > > >> > > >
> >> > > >> > > > work through security.provider.
> >> > > >> > > >
> >> > > >> > > > Thanks,
> >> > > >> > > > Harsha
> >> > > >> > > >
> >> > > >> > > > Thanks
> >> > > >> > > >
> >> > > >> > > > Maulin
> >> > > >> > > >
> >> > > >> > > > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
> >> > > >> > > >
> >> > > >> > > > kafka@harsha.io>
> >> > > >> > > >
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > In your KIP you added security. provider as rejected
> >> > > >> > > >
> >> > > >> > > > alternative
> >> > > >> > > >
> >> > > >> > > > and
> >> > > >> > > >
> >> > > >> > > > specified "its not the correct way". Do you mind explaining
> >> > > >> > > >
> >> > > >> > > > why
> >> > > >> > > >
> >> > > >> > > > its
> >> > > >> > > >
> >> > > >> > > > not? I
> >> > > >> > > >
> >> > > >> > > > didn't find any evidence in Java docs to say so. Contrary to
> >> > > >> > > >
> >> > > >> > > > your
> >> > > >> > > >
> >> > > >> > > > statement
> >> > > >> > > >
> >> > > >> > > > it does say in the java docs
> >> > > >> > > > " However, please note that a provider can be used to
> >> > > >> > > >
> >> > > >> > > > implement
> >> > > >> > > >
> >> > > >> > > > any
> >> > > >> > > >
> >> > > >> > > > security service in Java that uses a pluggable architecture
> >> > > >> > > >
> >> > > >> > > > with
> >> > > >> > > >
> >> > > >> > > > a
> >> > > >> > > >
> >> > > >> > > > choice
> >> > > >> > > >
> >> > > >> > > > of implementations that fit underneath."
> >> > > >> > > >
> >> > > >> > > > Java Security Providers have been used by other projects to
> >> > > >> > > >
> >> > > >> > > > provide
> >> > > >> > > >
> >> > > >> > > > such
> >> > > >> > > >
> >> > > >> > > > integration . I am not sure if you looked into Spiffe
> >> > > >> > > >
> >> > > >> > > > project to
> >> > > >> > > >
> >> > > >> > > > efficiently distribute certificates but here is an example of
> >> > > >> > > >
> >> > > >> > > > Java
> >> > > >> > > >
> >> > > >> > > > provider
> >> > > >> > > >
> >> > > >> > > >
> >> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> > >
> >> > > >> >
> >> > > >>
> >> > >
> >> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> >> > > >> > > >
> >> > > >> > > > java which
> >> > > >> > > > obtains certificates from local daemons.
> >> > > >> > > > These integrations are being used in Tomcat, Jetty etc.. We
> >> > > >> > > >
> >> > > >> > > > are
> >> > > >> > > >
> >> > > >> > > > also
> >> > > >> > > >
> >> > > >> > > > using
> >> > > >> > > >
> >> > > >> > > > Security provider to do the same in our Kafka clusters. So
> >> > > >> > > >
> >> > > >> > > > unless I
> >> > > >> > > >
> >> > > >> > > > see
> >> > > >> > > >
> >> > > >> > > > more evidence why security.provider doesn't work for you
> >> > > >> > > >
> >> > > >> > > > adding
> >> > > >> > > >
> >> > > >> > > > new
> >> > > >> > > >
> >> > > >> > > > interfaces while there exists more cleaner way of achieving
> >> > > >> > > >
> >> > > >> > > > the
> >> > > >> > > >
> >> > > >> > > > goals
> >> > > >> > > >
> >> > > >> > > > of
> >> > > >> > > >
> >> > > >> > > > this KIP is unnecessary and breaks the well known security
> >> > > >> > > >
> >> > > >> > > > interfaces
> >> > > >> > > >
> >> > > >> > > > provided by Java itself.
> >> > > >> > > >
> >> > > >> > > > Thanks,
> >> > > >> > > > Harsha
> >> > > >> > > >
> >> > > >> > > > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
> >> > > >> > > >
> >> > > >> > > > kafka@harsha.io
> >> > > >> > > >
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Hi Maulin,
> >> > > >> > > > Not sure if you looked at my previous replies. This
> >> > > >> > > >
> >> > > >> > > > changes
> >> > > >> > > >
> >> > > >> > > > are not required as there is already security Provider to do
> >> > > >> > > >
> >> > > >> > > > what you
> >> > > >> > > >
> >> > > >> > > > are
> >> > > >> > > >
> >> > > >> > > > proposing. This KIP
> >> > > >> > > >
> >> > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> >> > > >> > > >
> >> > > >> > > >
> >> KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> >> > > >> > > >
> >> > > >> > > > also
> >> > > >> > > >
> >> > > >> > > > addresses easy registration of such providers.
> >> > > >> > > >
> >> > > >> > > > Thanks,
> >> > > >> > > > Harsha
> >> > > >> > > >
> >> > > >> > > > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
> >> > > >> > > >
> >> > > >> > > > <maulin.vasavada@gmail.
> >> > > >> > > >
> >> > > >> > > > com> wrote:
> >> > > >> > > >
> >> > > >> > > > Bump! Can somebody please review this?
> >> > > >> > > >
> >> > > >> > > > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
> >> > > >> > > >
> >> > > >> > > > maulin.vasavada@gmail.com>
> >> > > >> > > >
> >> > > >> > > > wrote:
> >> > > >> > > >
> >> > > >> > > > Bump! Can somebody please review this?
> >> > > >> > > >
> >> > > >> > > > --
> >> > > >> > > > Thanks,
> >> > > >> > > > M.Sai Sandeep
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> > >
> >> > > >> >
> >> > > >>
> >> > > >
> >> > >
> >> >
> >>
> >
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Maulin Vasavada <ma...@gmail.com>.
+1 for making SslEngineBuilder configurable upon more thoughts.
However, in the abstraction and default implementation we should make sure
when we do have a requirement to plugin custom key/trust store people don't
have to write lot more code which may not be related to it.
Having said that, does this mean, we resurrect KIP-383 and update it with
latest context and go from there?
We are willing to take up that work for making it configurable.
Thanks
Maulin
On Fri, Aug 30, 2019 at 10:34 AM Maulin Vasavada <ma...@gmail.com>
wrote:
> Why don't we make SSLEngineBuilder code delegate the whole Key/Trust store
> initialization to the interfaces we are proposing? Default implementation
> for those key/trust store loader interfaces will be "file based" loading vs
> if somebody wants to customize any of it they can.
>
> Would that make sense?
>
> On Fri, Aug 30, 2019 at 10:03 AM Colin McCabe <cm...@apache.org> wrote:
>
>> +1 for making SslEngineBuilder configurable. This would give
>> implementers a lot more flexibility-- to use key distribution methods that
>> were not files, for example.
>>
>> best,
>> Colin
>>
>>
>> On Fri, Aug 30, 2019, at 02:03, Rajini Sivaram wrote:
>> > Just to make sure we are on the same page - KIP-383 was written before
>> > the
>> > code was refactored. The refactoring addressed some of the concerns of
>> > KIP-383. My suggestion was to make SslEngineBuilder configurable. The
>> > default implementation of this pluggable class would be
>> >
>> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/ssl/SslEngineBuilder.java
>> .
>> > That should give an idea of the size of the configurable part that a
>> > custom
>> > class needs to implement. A large part of that is about loading
>> > keystore/truststore. I agree it has slightly more code than KIP-486
>> > proposes, but since it lets you customize creation of SSLEngine, it
>> > would
>> > address every possible scenario.
>> >
>> > Thoughts?
>> >
>> >
>> > On Fri, Aug 30, 2019 at 2:02 AM Maulin Vasavada <
>> maulin.vasavada@gmail.com>
>> > wrote:
>> >
>> > > I thought about it more. I feel that no matter how we refactor the
>> code
>> > > (with or without KIP-383 integrated), ultimately the need of
>> customizing
>> > > loading for keys and certs will still remain. Whenever that need
>> arises we
>> > > might end up thinking about the solution suggested by our KIP-486.
>> Hence
>> > > regardless of the other KIPs and configurations "if we do need to
>> customize
>> > > loading of keys/certs, we will need the code changes suggested by this
>> > > KIP".
>> > >
>> > > Let me know what you guys think.
>> > >
>> > > Harsha, we are working on changing the interfaces for key/trust store
>> > > loaders with Certificate and PrivateKey objects. Will probably be
>> able to
>> > > update it later today or tomorrow.
>> > >
>> > > Thanks
>> > > Maulin
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > On Thu, Aug 29, 2019 at 2:30 PM Maulin Vasavada <
>> maulin.vasavada@gmail.com
>> > > >
>> > > wrote:
>> > >
>> > > > On that, I actually looked at KIP-383 before briefly. However, that
>> > > > sounded like lot of changes suggested.
>> > > >
>> > > > One "key" thing we have to keep in mind is - IF we need lot of
>> > > > customization Kafka already allows you to use your SslProvider via
>> > > > ssl.providers or the changes done by KIP-492 and
>> > > > SSLContext.getInstance(protocol, provider) call allows us to return
>> the
>> > > > SSLContext with "ALL" the details we would like to customize. Hence
>> I am
>> > > > not sure that customization suggested by KIP-383 would be worth the
>> > > effort.
>> > > > We also have similar SSLContext customization outside of Kafka.
>> > > >
>> > > > Thanks
>> > > > Maulin
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Thu, Aug 29, 2019 at 12:47 PM Pellerin, Clement <
>> > > > Clement_Pellerin@ibi.com> wrote:
>> > > >
>> > > >> KIP-383 in its present form was vetoed because it was not possible
>> to
>> > > add
>> > > >> validation of custom properties in a future KIP. The solution to
>> that is
>> > > >> the first proposal I wrote for KIP-383 which made the whole
>> SslFactory
>> > > >> pluggable. That first solution was also vetoed hence the deadlock.
>> > > >>
>> > > >> Replacing the whole factory was a much nicer solution. It was
>> vetoed
>> > > >> because doing this almost invariably meant the replacement lost
>> all the
>> > > >> complex validation code in the default SslFactory.
>> > > >>
>> > > >> My current idea is to extract the validation code into another
>> public
>> > > API
>> > > >> that SslFactory would call. I did not look at the newly refactored
>> code
>> > > and
>> > > >> I did not study how to do this yet. KIP-383 was not popular at the
>> time
>> > > and
>> > > >> designing a new solution is a lot of work.
>> > > >>
>> > > >> Is there interest from 3 binding voters for something like this?
>> > > >>
>> > > >> -----Original Message-----
>> > > >> From: Rajini Sivaram [mailto:rajinisivaram@gmail.com]
>> > > >> Sent: Thursday, August 29, 2019 2:57 PM
>> > > >> To: dev
>> > > >> Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and
>> > > >> TrustStore
>> > > >>
>> > > >> Hi Maulin,
>> > > >>
>> > > >> In SSL scenarios, I imagine security providers introduced by
>> KIP-492 are
>> > > >> likely to be most useful when you want to use third party
>> providers. The
>> > > >> biggest advantage of the config from that KIP is that you don't
>> need to
>> > > >> write much code to integrate existing security providers into Kafka
>> > > >> brokers
>> > > >> or clients. As I understand it, KIP-486 is a more convenient
>> option for
>> > > >> the
>> > > >> specific problem of loading keystores/truststores differently. It
>> can be
>> > > >> achieved in theory with KIP-492, but KIP-486 is a much simpler
>> option
>> > > for
>> > > >> this case.
>> > > >>
>> > > >> My concern about KIP-486 is that it introduces yet another
>> interface
>> > > into
>> > > >> our already complex security code, while only solving one
>> particular use
>> > > >> case. Have you looked at
>> > > >>
>> > > >>
>> > >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-383%3A++Pluggable+interface+for+SSL+Factory
>> > > >> ?
>> > > >> The goal was to make
>> > > >> org.apache.kafka.common.security.ssl.SslEngineBuilder pluggable.
>> > > >> The code has already been refactored by Colin after that KIP was
>> > > written,
>> > > >> making it easier to implement KIP-383. This should enable you to
>> load
>> > > your
>> > > >> keystores and truststores differently. Using a pluggable
>> > > SslEngineBuilder
>> > > >> will also solve several other use cases at the same time. KIP-383
>> hasn't
>> > > >> been voted through yet, but perhaps you could take a look and we
>> could
>> > > >> revive that instead if it solves your use case as well?
>> > > >>
>> > > >> Regards,
>> > > >>
>> > > >> Rajini
>> > > >>
>> > > >>
>> > > >> On Thu, Aug 29, 2019 at 6:42 PM Maulin Vasavada <
>> > > >> maulin.vasavada@gmail.com>
>> > > >> wrote:
>> > > >>
>> > > >> > Hi Harsha
>> > > >> >
>> > > >> > Thank you. Appreciate your time and support on this. Let me go
>> back do
>> > > >> some
>> > > >> > more research and get back to you on the KeyStore interface part.
>> > > >> > Basically, if we return certs and keys in the interface then
>> Kafka
>> > > code
>> > > >> > will have to build KeyStore object - which is also reasonable.
>> > > >> >
>> > > >> > Thanks
>> > > >> > Maulin
>> > > >> >
>> > > >> > On Thu, Aug 29, 2019 at 10:01 AM Harsha Chintalapani <
>> kafka@harsha.io
>> > > >
>> > > >> > wrote:
>> > > >> >
>> > > >> > > Hi Maulin,
>> > > >> > > Use cases are clear now. I am +1 for moving
>> > > >> forward
>> > > >> > > with the discussions on having such configurable option for
>> users.
>> > > But
>> > > >> > the
>> > > >> > > interfaces is proposed doesn't look right to me. We are still
>> > > talking
>> > > >> > about
>> > > >> > > keystore interfaces. Given keystore's are used as filebased
>> way of
>> > > >> > > transporting certificates I am not sure it will help the rest
>> of the
>> > > >> > > user-base.
>> > > >> > > In short, I am +1 on the KIP's motivation
>> and only
>> > > >> have
>> > > >> > > questions around returning keystores instead of returning
>> certs,
>> > > >> private
>> > > >> > > keys etc. . If others in the community are ok with such
>> interface we
>> > > >> can
>> > > >> > > move forward.
>> > > >> > >
>> > > >> > > Thanks,
>> > > >> > > Harsha
>> > > >> > >
>> > > >> > >
>> > > >> > > On Wed, Aug 28, 2019 at 1:51 PM, Maulin Vasavada <
>> > > >> > > maulin.vasavada@gmail.com>
>> > > >> > > wrote:
>> > > >> > >
>> > > >> > > > Hi Harsha
>> > > >> > > >
>> > > >> > > > As we synced-up offline on this topic, we hope you don't
>> have any
>> > > >> more
>> > > >> > > > clarifications that you are seeking. If that is the case,
>> can you
>> > > >> > please
>> > > >> > > > help us move this forward and discuss what changes you would
>> > > expect
>> > > >> on
>> > > >> > > the
>> > > >> > > > KIP design in order to make it valuable contribution?
>> > > >> > > >
>> > > >> > > > Just FYI - we verified our primary design change with the
>> author
>> > > of
>> > > >> > Sun's
>> > > >> > > > X509 Trustmanager implementation and the outcome is that
>> what we
>> > > are
>> > > >> > > > proposing makes sense at the heart of it - "Instead of
>> writing
>> > > >> > > TrustManager
>> > > >> > > > just plugin the Trust store". We are open to discuss
>> additional
>> > > >> changes
>> > > >> > > > that you/anybody else would like to see on the functionality
>> > > >> however.
>> > > >> > > >
>> > > >> > > > Thanks
>> > > >> > > > Maulin
>> > > >> > > >
>> > > >> > > > On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada <
>> > > >> > > maulin.vasavada@gmail.com>
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Hi Harsha
>> > > >> > > >
>> > > >> > > > Any response on my question? I feel this KIP is worth
>> > > accommodating.
>> > > >> > Your
>> > > >> > > > help is much appreciated.
>> > > >> > > >
>> > > >> > > > Thanks
>> > > >> > > > Maulin
>> > > >> > > >
>> > > >> > > > On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada <
>> > > >> > maulin.vasavada@gmail.
>> > > >> > > > com> wrote:
>> > > >> > > >
>> > > >> > > > Hi Harsha
>> > > >> > > >
>> > > >> > > > I've examined the SPIFFE provider more and have one question
>> -
>> > > >> > > >
>> > > >> > > > If SPIFFE didn't have a need to do checkSpiffeId() call at
>> the
>> > > below
>> > > >> > > > location, would you really still write the Provider? *OR*
>> Would
>> > > you
>> > > >> > just
>> > > >> > > > use TrustManagerFactory.init(KeyStore) signature to pass the
>> > > >> KeyStore
>> > > >> > > from
>> > > >> > > > set of certs returned by spiffeIdManager. getTrustedCerts()?
>> > > >> > > >
>> > > >> > > >
>> > > >> >
>> > >
>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>> > > >> > > > provider/CertificateUtils.java#L100
>> > > >> > > >
>> > > >> > > > /**
>> > > >> > > >
>> > > >> > > > * Validates that the SPIFFE ID is present and matches the
>> SPIFFE
>> > > ID
>> > > >> > > > configured in
>> > > >> > > > * the java.security property ssl.spiffe.accept
>> > > >> > > > *
>> > > >> > > > * If the authorized spiffe ids list is empty any spiffe id is
>> > > >> > authorized
>> > > >> > > > *
>> > > >> > > > * @param chain an array of X509Certificate that contains the
>> > > Peer's
>> > > >> > SVID
>> > > >> > > > to be validated
>> > > >> > > > * @throws CertificateException when either the certificates
>> > > doesn't
>> > > >> > have
>> > > >> > > a
>> > > >> > > > SPIFFE ID or the SPIFFE ID is not authorized
>> > > >> > > > */
>> > > >> > > > static void checkSpiffeId(X509Certificate[] chain) throws
>> > > >> > > > CertificateException {
>> > > >> > > >
>> > > >> > > > Thanks
>> > > >> > > > Maulin
>> > > >> > > >
>> > > >> > > > On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani <
>> > > >> kafka@harsha.io>
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Maulin,
>> > > >> > > > The code parts you are pointing are specific for Spiffe and
>> if
>> > > >> > > > you are talking about validate method which uses PKIX check
>> like
>> > > any
>> > > >> > > other
>> > > >> > > > provider does.
>> > > >> > > > If you want to default to SunJSSE everywhere you can do so by
>> > > >> > delegating
>> > > >> > > > the calls in these methods to SunJSSE provider.
>> > > >> > > >
>> > > >> > > > TrustManagerFactory tmf = TrustManagerFactory
>> > > >> > > > .getInstance(TrustManagerFactory.getDefaultAlgorithm());and
>> use
>> > > >> > > > tmf.chekServerTrusted()
>> > > >> > > > or use
>> > > >> > > > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
>> > > >> > > > TrustManagerFactory.html#getInstance(java.lang.String)if you
>> want
>> > > a
>> > > >> > > > specific provider.
>> > > >> > > >
>> > > >> > > > -Harsha
>> > > >> > > >
>> > > >> > > > On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada <
>> > > >> > maulin.vasavada@gmail.
>> > > >> > > > com>
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Okay, so I take that you guys agree that I have to write a
>> > > 'custom'
>> > > >> > > > algorithm and a provider to make it work , correct?
>> > > >> > > >
>> > > >> > > > Now, for Harsha's comment "Here the 'Custom' Algorithm is
>> not an
>> > > >> > > > implementation per say , ..." , I diagree. You can refer to
>> > > https://
>> > > >> > > >
>> > > >> > > >
>> > > >> >
>> > > >>
>> > >
>> github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/
>> > > >> > > >
>> > > >> > > > SpiffeTrustManager.java#L91 <
>> http://spiffetrustmanager.java/#L91>
>> > > >> and
>> > > >> > > >
>> > > >> > > >
>> > > >> >
>> > >
>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>> > > >> > > >
>> > > >> > > > provider/CertificateUtils.java#L100
>> > > >> > > >
>> > > >> > > > "that code" is the customization you have for the custom way
>> to
>> > > >> check
>> > > >> > > > something on top of regular checks. That method is NOT doing
>> > > custom
>> > > >> > > > truststore loading. It is validating/verifying something in
>> the
>> > > >> > > >
>> > > >> > > > "custom"
>> > > >> > > >
>> > > >> > > > way with spiffeId.
>> > > >> > > > I bet that without that you won't have a need of the custom
>> > > >> algorithm
>> > > >> > > >
>> > > >> > > > in
>> > > >> > > >
>> > > >> > > > the first place.
>> > > >> > > >
>> > > >> > > > Let me know if you agree to this.
>> > > >> > > >
>> > > >> > > > Thanks
>> > > >> > > > Maulin
>> > > >> > > >
>> > > >> > > > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <
>> mprsai@gmail.com>
>> > > >> > > >
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Hi Maulin, thanks for the discussion. As Harsha pointed out,
>> to
>> > > use
>> > > >> the
>> > > >> > > > KIP492, you need to create a new provider, register a *new*
>> custom
>> > > >> > > > algorithm for your keymanager and trustmanager factory
>> > > >> implementations.
>> > > >> > > > After this, the kafka server configuration can be done as
>> given
>> > > >> below
>> > > >> > > >
>> > > >> > > > # Register the provider class with custom algorithm, say
>> CUSTOM
>> > > >> > > >
>> > > >> > > > security.
>> > > >> > > >
>> > > >> > > > provider.classes=com.company.security.CustomProvider
>> > > >> > > > <http://provider.classes
>> =com.company.security.customprovider/>
>> > > >> > > > <http://security.provider.classes
>> > > >> > > >
>> > > >> > > > =com.company.security.customprovider/>
>> > > >> > > >
>> > > >> > > > # Register the keymanager and trustmanager algorithms
>> > > >> > > > # These algorithms indicate that the Keymanager and
>> Trustmanagers
>> > > >> > > > registered under the algorithm "CUSTOM" needs to be used
>> > > >> > > > ssl.trustmanager.algorithm=CUSTOM
>> > > >> > > > ssl.keymanager.algorithm=CUSTOM
>> > > >> > > >
>> > > >> > > > And the customprovider looks like this...
>> > > >> > > >
>> > > >> > > > public class CustomProvider extends Provider {
>> > > >> > > > public CustomProvider() {
>> > > >> > > > super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and
>> > > TrustStore");
>> > > >> > > > super.put("KeyManagerFactory.CUSTOM",
>> "customKeyManagerFactory");
>> > > >> > > > super.put("TrustManagerFactory.CUSTOM",
>> > > >> > > > "customTrustManagerFactory");
>> > > >> > > > }
>> > > >> > > > }
>> > > >> > > >
>> > > >> > > > The PR for this is in review and can be found here:
>> > > >> > > >
>> > > >> > > > https://github.com/
>> > > >> > > >
>> > > >> > > > apache/kafka/pull/7090
>> > > >> > > > This PR includes the fixed insertProviderAt function call.
>> > > >> > > >
>> > > >> > > > On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani <
>> > > >> kafka@harsha.io>
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Answers are in-line
>> > > >> > > >
>> > > >> > > > On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada <
>> > > >> > > >
>> > > >> > > > maulin.vasavada@gmail.
>> > > >> > > >
>> > > >> > > > com
>> > > >> > > >
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Hi Colin
>> > > >> > > >
>> > > >> > > > When I refer to "standard" or "custom" algorithms I am
>> following
>> > > >> Java
>> > > >> > > > security Provider Terminology. You can refer to
>> > > >> > > >
>> https://docs.oracle.com/javase/7/docs/technotes/guides/security/
>> > > >> > > > StandardNames.html#TrustManagerFactory link I provided
>> earlier in
>> > > >> the
>> > > >> > > > emails. It says PKIX is the default Algorithm for
>> > > >> TrustManagerFactory.
>> > > >> > > >
>> > > >> > > > 1. For SPIFFE, I am not sure why you are saying 'it does not
>> > > >> implement
>> > > >> > > > custom algorithms' because the following file clearly
>> indicates
>> > > >> that it
>> > > >> > > > does use custom algorithm-
>> > > >> > > >
>> > > >> > > >
>> > > >> >
>> > >
>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>> > > >> > > >
>> > > >> > > > provider/SpiffeProvider.java#L17
>> > > >> > > >
>> > > >> > > > Algorithm value:
>> > > >> > > >
>> > > >> > > >
>> > > >> >
>> > >
>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>> > > >> > > >
>> > > >> > > > provider/SpiffeProviderConstants.java#L6
>> > > >> > > >
>> > > >> > > > @Harsha do you want to chime in since you use that provider?
>> > > >> > > >
>> > > >> > > > Here the "Custom" Algorithm is not an implementation per say
>> ,
>> > > >> rather
>> > > >> > > >
>> > > >> > > > used
>> > > >> > > >
>> > > >> > > > to invoke the custom trust store factory and key manager
>> factory.
>> > > >> You
>> > > >> > > >
>> > > >> > > > are
>> > > >> > > >
>> > > >> > > > not moving away from "standard" alogrithms that are
>> available.
>> > > >> > > >
>> > > >> > > >
>> > > >> >
>> > >
>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>> > > >> > > >
>> > > >> > > > provider/SpiffeTrustManager.java
>> > > >> > > >
>> > > >> > > > As you can see it delegates all the calls of verification of
>> > > >> > > >
>> > > >> > > > certificate
>> > > >> > > >
>> > > >> > > > to
>> > > >> > > >
>> > > >> > > > the default implementation available.
>> > > >> > > > So in our implementation we still use PKIX to verify the
>> > > certificate
>> > > >> > > > chain. So you are not losing anything here and Spiffe is not
>> > > >> > > >
>> > > >> > > > reimplementing
>> > > >> > > >
>> > > >> > > > the verification process.
>> > > >> > > >
>> > > >> > > > 2. I already mentioned in my 3rd point, in my previous post,
>> why
>> > > >> using
>> > > >> > > >
>> > > >> > > > ssl.provider does NOT work. I updated KIP-486 in "rejected
>> > > >> > > >
>> > > >> > > > alternatives"
>> > > >> > > >
>> > > >> > > > also why ssl.provider does not work.
>> > > >> > > >
>> > > >> > > > As mentioned before , provider is the right way to go. I am
>> still
>> > > >> not
>> > > >> > > > understanding the gap is.
>> > > >> > > > If I understand correctly your argument is , provider is
>> going to
>> > > >> ask
>> > > >> > > >
>> > > >> > > > to
>> > > >> > > >
>> > > >> > > > implement a custom algorithm.
>> > > >> > > > My answer to that is , no you are not re-implementing the
>> > > algorithm.
>> > > >> > > >
>> > > >> > > > Please
>> > > >> > > >
>> > > >> > > > check the above link , TrustManager implementation it
>> delegates
>> > > the
>> > > >> > > >
>> > > >> > > > calls
>> > > >> > > >
>> > > >> > > > back. There is no need to implement your own here.
>> > > >> > > >
>> > > >> > > > 3. Security.insertProviderAt() comments were based on
>> assumption
>> > > if
>> > > >> > > >
>> > > >> > > > KIP-492
>> > > >> > > >
>> > > >> > > > changes are done and we use that mechanism to configure
>> providers
>> > > >> > > >
>> > > >> > > > instead
>> > > >> > > >
>> > > >> > > > of ssl.provider configuration.
>> > > >> > > >
>> > > >> > > > KIP-492 has patch available and going through review.
>> > > >> > > >
>> > > >> > > > Can you read my all the points, I mentioned in my previous
>> post,
>> > > >> very
>> > > >> > > >
>> > > >> > > > carefully? I am covering all the aspects in explaining. I am
>> open
>> > > to
>> > > >> > > >
>> > > >> > > > still
>> > > >> > > >
>> > > >> > > > discuss more to clarify any doubts.
>> > > >> > > >
>> > > >> > > > "3. If we just use existing ssl.provider kafka configuration
>> then
>> > > >> our
>> > > >> > > > provider will be used in SSLContext.getInstance(protocol,
>> > > provider)
>> > > >> > > >
>> > > >> > > > call
>> > > >> > > >
>> > > >> > > > in
>> > > >> > > >
>> > > >> > > > SslFactory.java <http://sslfactory.java/> <
>> > > http://sslfactory.java/>
>> > > >> <
>> > > >> > > > http://sslfactory.java/>
>> > > >> > > >
>> > > >> > > > and
>> > > >> > > >
>> > > >> > > > if our provider does not have
>> > > >> > > > implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it
>> breaks
>> > > (we
>> > > >> > > >
>> > > >> > > > tested
>> > > >> > > >
>> > > >> > > > it). Example: In MyProvider sample above you see that I
>> didn't add
>> > > >> > > > SSLContext.TLSv1 as
>> > > >> > > > "Service+Algorithm" and that didn't work for me. In SPIFFE
>> > > provider
>> > > >> you
>> > > >> > > > don't have this challenge since you are planning to bypass
>> > > >> ssl.provider
>> > > >> > > >
>> > > >> > > > as
>> > > >> > > >
>> > > >> > > > you mention in the KIP-492."
>> > > >> > > >
>> > > >> > > > Yes here you need to pass the protocol that your
>> > > >> > > >
>> > > >> > > > KeyManager/TrustManager
>> > > >> > > >
>> > > >> > > > registered with and in no way its deviating from TLS RFC
>> spec.
>> > > >> > > >
>> > > >> > > >
>> > > >> >
>> > >
>> https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/
>> > > >> > > >
>> > > >> > > > javax/net/ssl/SSLContext.java#L134
>> > > >> > > >
>> > > >> > > > My suggestion here is for you to implement a simple Security
>> > > >> Provider
>> > > >> > > >
>> > > >> > > > as
>> > > >> > > >
>> > > >> > > > you did before and register a Algorithm. You can use the
>> existing
>> > > >> > > > implementation in SpiffeProvider and plug in changes where
>> you
>> > > need
>> > > >> to
>> > > >> > > > retrieve the certificates from by making RPC call.
>> > > >> > > >
>> > > >> > > > Run an end-to-end test with Kafka broker coming up with your
>> > > >> provider
>> > > >> > > > making calls to RPC call. You do need to pass the "custom
>> > > algorithm"
>> > > >> > > >
>> > > >> > > > that
>> > > >> > > >
>> > > >> > > > you registered in your key manager to invoke the provider. I
>> think
>> > > >> your
>> > > >> > > > concern is this approach is replacing the existing known
>> > > >> ciphersuites
>> > > >> > > >
>> > > >> > > > and
>> > > >> > > >
>> > > >> > > > certificate validation provided by java. Which its not.
>> > > >> > > >
>> > > >> > > > Now test the TLS connection you can do so via openssl
>> -s_client
>> > > >> options
>> > > >> > > >
>> > > >> > > > to
>> > > >> > > >
>> > > >> > > > test if everything else is working.
>> > > >> > > >
>> > > >> > > > I am happy to share configs that we used if you are
>> interested.
>> > > >> > > >
>> > > >> > > > Thanks,
>> > > >> > > > Harsha
>> > > >> > > >
>> > > >> > > > Thanks
>> > > >> > > > Maulin
>> > > >> > > >
>> > > >> > > > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <
>> cmccabe@apache.org>
>> > > >> > > >
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Hi Maulin,
>> > > >> > > >
>> > > >> > > > A lot of JSSE providers don't implement custom algorithms.
>> Spire
>> > > is
>> > > >> a
>> > > >> > > >
>> > > >> > > > good
>> > > >> > > >
>> > > >> > > > example of a JSSE provider that doesn't, and yet is still
>> useful
>> > > to
>> > > >> > > >
>> > > >> > > > many
>> > > >> > > >
>> > > >> > > > people. Your JSSE provider can work fine even if it doesn't
>> > > >> implement a
>> > > >> > > > custom algorithm.
>> > > >> > > >
>> > > >> > > > Maybe I'm missing something, but I don't understand the
>> discussion
>> > > >> of
>> > > >> > > > Security.insertProviderAt() that you included.
>> SslEngineBuilder
>> > > >> doesn't
>> > > >> > > >
>> > > >> > > > use
>> > > >> > > >
>> > > >> > > > that API to get the security provider. Instead, it calls
>> > > >> > > > "SSLContext.getInstance(protocol, provider)", where provider
>> is
>> > > the
>> > > >> > > >
>> > > >> > > > name
>> > > >> > > >
>> > > >> > > > of the provider.
>> > > >> > > >
>> > > >> > > > best,
>> > > >> > > > Colin
>> > > >> > > >
>> > > >> > > > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
>> > > >> > > >
>> > > >> > > > On top of everything above I feel strongly to add the 4th
>> point
>> > > >> which
>> > > >> > > >
>> > > >> > > > is
>> > > >> > > >
>> > > >> > > > based on Java APIs for TrustManagerFactory.init(KeyStore) (
>> > > >> > > >
>> > > >> > > > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
>> > > >> > > > TrustManagerFactory.html#init(java.security.KeyStore
>> > > >> > > > <http://java.security.keystore/>
>> > > >> > > > <http://java.security.keystore/>)
>> > > >> > > > )
>> > > >> > > >
>> > > >> > > > and KeyManagerFactory.init(KeyStore, char[]) (
>> > > >> > > >
>> > > >> > > >
>> > > >> > >
>> > > >> >
>> > > >>
>> > >
>> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory
>> > > >> > > > .
>> > > >> > > >
>> > > >> > > > html#init(java.security.KeyStore <
>> http://java.security.keystore/>
>> > > >> > > <http://
>> > > >> > > > java.security.keystore/
>> > > >> > > > ,%20char[])
>> > > >> > > >
>> > > >> > > > ).
>> > > >> > > >
>> > > >> > > > 4. The above APIs are intended to support providing
>> "trust/key
>> > > >> > > >
>> > > >> > > > material"
>> > > >> > > >
>> > > >> > > > from the user without having to write their own
>> > > >> > > >
>> > > >> > > > TrustManager/KeyManagers.
>> > > >> > > >
>> > > >> > > > To quote from the TrustManagerFactory.init()'s documentation
>> > > >> > > >
>> > > >> > > > "Initializes
>> > > >> > > >
>> > > >> > > > this factory with a source of certificate authorities and
>> related
>> > > >> trust
>> > > >> > > > material."
>> > > >> > > > To quote from the KeyManagerFactory.init()'s documentation
>> > > >> "Initializes
>> > > >> > > > this factory with a source of key material."
>> > > >> > > >
>> > > >> > > > Based on this it is clear that there is a flexibility
>> provided by
>> > > >> Java
>> > > >> > > >
>> > > >> > > > to
>> > > >> > > >
>> > > >> > > > to enable developers to provide the required trust/key
>> material
>> > > >> loaded
>> > > >> > > >
>> > > >> > > > from
>> > > >> > > >
>> > > >> > > > "anywhere" without requiring them to write custom provider OR
>> > > >> trust/key
>> > > >> > > > managers. This same flexibility is reflected in Kafka code
>> also
>> > > >> where
>> > > >> > > >
>> > > >> > > > it
>> > > >> > > >
>> > > >> > > > loads the trust/keys from a local file and doesn't require
>> > > writing a
>> > > >> > > > Provider necessarily. If we do NOT have a custom algorithm,
>> it
>> > > makes
>> > > >> > > >
>> > > >> > > > less
>> > > >> > > >
>> > > >> > > > sense to write a Provider.
>> > > >> > > >
>> > > >> > > > Thanks
>> > > >> > > > Maulin
>> > > >> > > >
>> > > >> > > > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
>> > > >> > > >
>> > > >> > > > maulin.vasavada@gmail.com>
>> > > >> > > >
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Hi Harsha/Colin
>> > > >> > > >
>> > > >> > > > I did the sample with a custom Provider for
>> TrustStoreManager and
>> > > >> tried
>> > > >> > > > using ssl.provider Kafka config AND the way KIP-492 is
>> suggesting
>> > > >> (by
>> > > >> > > > adding Provider programmatically instead of relying on
>> > > >> > > >
>> > > >> > > > ssl.provider+java.
>> > > >> > > >
>> > > >> > > > security. The below sample is followed by my detailed
>> findings.
>> > > I'll
>> > > >> > > > appreciate if you can go through it carefully and see
>> > > >> > > >
>> > > >> > > > if you
>> > > >> > > >
>> > > >> > > > see my point.
>> > > >> > > >
>> > > >> > > > package providertest;
>> > > >> > > >
>> > > >> > > > import java.security.Provider <
>> http://java.security.provider/>
>> > > >> > <http://
>> > > >> > > > java.security.provider/>
>> > > >> > > >
>> > > >> > > > <http://
>> > > >> > > >
>> > > >> > > > java.security.provider/>;
>> > > >> > > >
>> > > >> > > > public class MyProvider extends Provider {
>> > > >> > > >
>> > > >> > > > private static final String name = "MyProvider"; private
>> static
>> > > >> double
>> > > >> > > > version = 1.0d;
>> > > >> > > > private static String info = "Maulin's SSL Provider
>> v"+version;
>> > > >> > > >
>> > > >> > > > public MyProvider() {
>> > > >> > > > super(name, version, info);
>> > > >> > > > this.put("TrustManagerFactory.PKIX",
>> > > >> > > >
>> > > >> > > > "providertest.MyTrustManagerFactory");
>> > > >> > > >
>> > > >> > > > }
>> > > >> > > > }
>> > > >> > > >
>> > > >> > > > *Details:*
>> > > >> > > >
>> > > >> > > > KIP-492 documents that it will use Security.addProvider()
>> assuming
>> > > >> it
>> > > >> > > >
>> > > >> > > > will
>> > > >> > > >
>> > > >> > > > add it as position '0' which is not a correct assumption. The
>> > > >> > > > addProvider()'s documentation says it will add it to the last
>> > > >> available
>> > > >> > > > position. You may want to correct that to say
>> > > >> > > > Security.insertProviderAt(provider, 1).
>> > > >> > > >
>> > > >> > > > Now coming back to our specific discussion,
>> > > >> > > >
>> > > >> > > > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when
>> you
>> > > add
>> > > >> > > >
>> > > >> > > > that
>> > > >> > > >
>> > > >> > > > provider in the provider list via Security.addProvider() the
>> > > >> position
>> > > >> > > >
>> > > >> > > > where
>> > > >> > > >
>> > > >> > > > it gets added doesn't matter (even if you don't end up
>> adding it
>> > > as
>> > > >> > > >
>> > > >> > > > first
>> > > >> > > >
>> > > >> > > > entry) since that is the ONLY provider for SPIFFE specific
>> > > algorithm
>> > > >> > > >
>> > > >> > > > you
>> > > >> > > >
>> > > >> > > > might have.
>> > > >> > > >
>> > > >> > > > We do *not* have custom algorithm for Key/Trust StoreMangers.
>> > > Which
>> > > >> > > >
>> > > >> > > > means
>> > > >> > > >
>> > > >> > > > we have to use X509, PKIX etc "Standard Algorithms" ((
>> > > >> > > >
>> > > >> > > >
>> https://docs.oracle.com/javase/7/docs/technotes/guides/security/
>> > > >> > > > StandardNames.html
>> > > >> > > > ))
>> > > >> > > >
>> > > >> > > > in our provider to override the TrustStoreManager (see my
>> sample
>> > > >> code)
>> > > >> > > >
>> > > >> > > > and
>> > > >> > > >
>> > > >> > > > KeyStoreManger and KeyManager. This creates another challenge
>> > > >> > > >
>> > > >> > > > mentioned in
>> > > >> > > >
>> > > >> > > > the below point.
>> > > >> > > >
>> > > >> > > > 2. In order to make our Provider for loading custom
>> TrustStore
>> > > >> work, we
>> > > >> > > > have to add the provider as 'first' in the list since there
>> are
>> > > >> others
>> > > >> > > >
>> > > >> > > > with
>> > > >> > > >
>> > > >> > > > the same algorithm.
>> > > >> > > >
>> > > >> > > > However, the programatic way of adding provider
>> > > >> > > > (Security.insertProviderAt()) is *not* deterministic for
>> ordering
>> > > >> since
>> > > >> > > > different code can call the method for a different provider
>> and
>> > > >> > > >
>> > > >> > > > depending
>> > > >> > > >
>> > > >> > > > upon the order of the call our provider can be first or
>> pushed
>> > > down
>> > > >> the
>> > > >> > > > list. This can happen very well in any client application
>> using
>> > > >> Kafka.
>> > > >> > > >
>> > > >> > > > This
>> > > >> > > >
>> > > >> > > > is specially problematic for a case when you want to
>> guarantee
>> > > order
>> > > >> > > >
>> > > >> > > > for a
>> > > >> > > >
>> > > >> > > > Provider having "Standard Algorithms".
>> > > >> > > >
>> > > >> > > > If we add our provider in java.security file that definitely
>> > > >> guarantees
>> > > >> > > > the order(unless somebody calls removeProvider() which is
>> less
>> > > >> > > >
>> > > >> > > > likely). But
>> > > >> > > >
>> > > >> > > > if we add our provider in java.security file it will defeat
>> the
>> > > >> > > >
>> > > >> > > > purpose of
>> > > >> > > >
>> > > >> > > > the KIP-492.
>> > > >> > > >
>> > > >> > > > In the gist - Apache Kafka must not rely on
>> "un-deterministic"
>> > > >> method
>> > > >> > > >
>> > > >> > > > to
>> > > >> > > >
>> > > >> > > > rely on Provider ordering.
>> > > >> > > >
>> > > >> > > > 3. If we just use existing ssl.provider kafka configuration
>> then
>> > > our
>> > > >> > > > provider will be used in SSLContext.getInstance(protocol,
>> > > provider)
>> > > >> > > >
>> > > >> > > > call in
>> > > >> > > >
>> > > >> > > > SslFactory.java <http://sslfactory.java/> <
>> > > http://sslfactory.java/>
>> > > >> <
>> > > >> > > > http://sslfactory.java/>
>> > > >> > > >
>> > > >> > > > and
>> > > >> > > >
>> > > >> > > > if our provider does not have implementation for
>> > > >> > > > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks
>> > > >> > > >
>> > > >> > > > (we
>> > > >> > > >
>> > > >> > > > tested it). Example:
>> > > >> > > >
>> > > >> > > > In
>> > > >> > > >
>> > > >> > > > MyProvider sample above you see that I didn't add
>> SSLContext.TLSv1
>> > > >> as
>> > > >> > > > "Service+Algorithm" and that didn't work for me. In SPIFFE
>> > > provider
>> > > >> you
>> > > >> > > > don't have this challenge since you are planning to bypass
>> > > >> > > >
>> > > >> > > > ssl.provider as
>> > > >> > > >
>> > > >> > > > you mention in the KIP-492.
>> > > >> > > >
>> > > >> > > > *Overall summary,*
>> > > >> > > >
>> > > >> > > > 1. Any provider based mechanisms- a) existing ssl.provider
>> and
>> > > >> > > >
>> > > >> > > > b)KIP-492,
>> > > >> > > >
>> > > >> > > > for loading key/trust store using "Standard Algorithms" do
>> not
>> > > work
>> > > >> > > >
>> > > >> > > > 2. Approach suggested in our KIP-486 works without any issue
>> and
>> > > it
>> > > >> is
>> > > >> > > > *not* our context specific solve
>> > > >> > > >
>> > > >> > > > 3. Based on above we feel KIP-492 and KIP-486 are
>> complimentary
>> > > >> changes
>> > > >> > > > and not contradicting or redundent.
>> > > >> > > >
>> > > >> > > > If you want we can do a joint session somehow to walk
>> through the
>> > > >> > > >
>> > > >> > > > sample I
>> > > >> > > >
>> > > >> > > > have and various experiments I did. I would encourage you to
>> do
>> > > >> similar
>> > > >> > > > exercise by writing a Provider for "Standard Algorithm" for
>> > > >> > > > TrustStoreManager (like our needs) and see what you find
>> since
>> > > only
>> > > >> > > >
>> > > >> > > > writing
>> > > >> > > >
>> > > >> > > > samples can bring out the complexity/challenges we face.
>> > > >> > > >
>> > > >> > > > Thanks
>> > > >> > > > Maulin
>> > > >> > > >
>> > > >> > > > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <
>> > > >> > > >
>> > > >> > > > maulin.vasavada@gmail.
>> > > >> > > >
>> > > >> > > > com> wrote:
>> > > >> > > >
>> > > >> > > > Just to update - still working on it. Get to work only on
>> and off
>> > > on
>> > > >> > > >
>> > > >> > > > it :(
>> > > >> > > >
>> > > >> > > > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
>> > > >> > > >
>> > > >> > > > maulin.vasavada@gmail.com>
>> > > >> > > >
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Hi Harsha
>> > > >> > > >
>> > > >> > > > Let me try to write samples and will let you know.
>> > > >> > > >
>> > > >> > > > Thanks
>> > > >> > > > Maulin
>> > > >> > > >
>> > > >> > > > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <
>> harsha.ch@gmail.com>
>> > > >> > > >
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Hi Maulin,
>> > > >> > > > With java security providers can be as custom you would
>> > > >> > > >
>> > > >> > > > like
>> > > >> > > >
>> > > >> > > > it to
>> > > >> > > > be. If you only want to to implement a custom way of loading
>> the
>> > > >> > > >
>> > > >> > > > keystore
>> > > >> > > >
>> > > >> > > > and truststore and not implement any protocol/encryption
>> handling
>> > > >> you
>> > > >> > > >
>> > > >> > > > can
>> > > >> > > >
>> > > >> > > > leave them empty and no need to implement. Have you looked
>> into
>> > > the
>> > > >> > > >
>> > > >> > > > links I
>> > > >> > > >
>> > > >> > > > pasted before?
>> > > >> > > >
>> > > >> > > >
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>> > > >> > > >
>> > > >> > > >
>> > > >> > >
>> > > >> >
>> > > >>
>> > >
>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
>> > > >> > > >
>> > > >> > > > java
>> > > >> > > >
>> > > >> > > >
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>> > > >> > > > spiffe-security-provider/src/main/java/spiffe/api/provider/
>> > > >> > > > SpiffeTrustManager.java <http://spiffetrustmanager.java/>
>> > > <http://
>> > > >> > > > spiffetrustmanager.java/>
>> > > >> > > >
>> > > >> > > >
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>> > > >> > > >
>> > > >> > > >
>> > > >> > >
>> > > >> >
>> > > >>
>> > >
>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>> > > >> > > >
>> > > >> > > > java
>> > > >> > > >
>> > > >> > > > Can you please tell me which methods are too complex in
>> above to
>> > > >> > > >
>> > > >> > > > implement
>> > > >> > > >
>> > > >> > > > or unnecessary? You are changing anything in SSL/TLS
>> > > implementations
>> > > >> > > > provided by
>> > > >> > > >
>> > > >> > > > All of the implementations delegating the checks to the
>> default
>> > > >> > > > implementation anyway.
>> > > >> > > > Spire agent is an example, its nothing but a GRPC server
>> listening
>> > > >> > > >
>> > > >> > > > on a
>> > > >> > > >
>> > > >> > > > unix domain socket . Above code is making a RPC call to the
>> local
>> > > >> > > >
>> > > >> > > > daemon
>> > > >> > > >
>> > > >> > > > to
>> > > >> > > > get the certificate and keys. The mechanics are pretty much
>> same
>> > > as
>> > > >> > > >
>> > > >> > > > what
>> > > >> > > >
>> > > >> > > > you are asking for.
>> > > >> > > >
>> > > >> > > > Thanks,
>> > > >> > > > Harsha
>> > > >> > > >
>> > > >> > > > On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
>> > > >> > > >
>> > > >> > > > maulin.vasavada@gmail.com>
>> > > >> > > >
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Imagine a scenario like - We know we have a custom KMS and
>> as a
>> > > >> > > >
>> > > >> > > > Kafka
>> > > >> > > >
>> > > >> > > > owner
>> > > >> > > >
>> > > >> > > > we want to comply to using that KMS source to load
>> keys/certs. As
>> > > >> > > >
>> > > >> > > > a
>> > > >> > > >
>> > > >> > > > Kafka
>> > > >> > > >
>> > > >> > > > owner we know how to integrate with KMS but doesn't
>> necessarily
>> > > >> > > >
>> > > >> > > > have
>> > > >> > > >
>> > > >> > > > to
>> > > >> > > >
>> > > >> > > > know anything about cipher suites, algorithms, and SSL/TLS
>> > > >> > > >
>> > > >> > > > implementation.
>> > > >> > > >
>> > > >> > > > Going the Provider way requires to know lot more than we
>> should,
>> > > >> > > >
>> > > >> > > > isn't it?
>> > > >> > > >
>> > > >> > > > Not that we would have concern/shy-away knowing those
>> details -
>> > > >> > > >
>> > > >> > > > but
>> > > >> > > >
>> > > >> > > > if we
>> > > >> > > >
>> > > >> > > > don't have to - why should we?
>> > > >> > > >
>> > > >> > > > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
>> > > >> > > >
>> > > >> > > > maulin.vasavada@gmail.com>
>> > > >> > > >
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Hi Harsha
>> > > >> > > >
>> > > >> > > > We don't have spire (or similar) agents and we do not have
>> > > >> > > >
>> > > >> > > > keys/certs
>> > > >> > > >
>> > > >> > > > locally on any brokers. To elaborate more on my previous
>> email,
>> > > >> > > >
>> > > >> > > > I agree that Java security Providers are used in much broader
>> > > >> > > >
>> > > >> > > > sense
>> > > >> > > >
>> > > >> > > > - to
>> > > >> > > >
>> > > >> > > > have a particular implementation of an algorithm, use
>> specific
>> > > >> > > >
>> > > >> > > > cipher
>> > > >> > > >
>> > > >> > > > suites for SSL , OR in our current team's case have a
>> > > >> > > >
>> > > >> > > > particular
>> > > >> > > >
>> > > >> > > > way to
>> > > >> > > >
>> > > >> > > > leverage pre-generated SSL sessions. However, the scope of
>> our
>> > > >> > > >
>> > > >> > > > KIP
>> > > >> > > >
>> > > >> > > > (486)
>> > > >> > > >
>> > > >> > > > is
>> > > >> > > >
>> > > >> > > > much restricted than that. We merely intend to provide a
>> custom
>> > > >> > > > keystore/truststore for our SSL connections and not really
>> worry
>> > > >> > > >
>> > > >> > > > about
>> > > >> > > >
>> > > >> > > > underlying specific SSL/TLS implementation. This simplifies
>> it
>> > > >> > > >
>> > > >> > > > a
>> > > >> > > >
>> > > >> > > > lot for
>> > > >> > > >
>> > > >> > > > us to keep the concerns separate and clear.
>> > > >> > > >
>> > > >> > > > I feel our approach is more complimentary such that it allows
>> > > >> > > >
>> > > >> > > > for
>> > > >> > > >
>> > > >> > > > using
>> > > >> > > >
>> > > >> > > > keystores of choice while retaining the flexibility to use
>> any
>> > > >> > > > underlying/available Provider for actually making the SSL
>> call.
>> > > >> > > >
>> > > >> > > > We agree with KIP-492's approach based on Providers (and
>> Java's
>> > > >> > > > recommendation), but also strongly believe that our approach
>> can
>> > > >> > > >
>> > > >> > > > compliment
>> > > >> > > >
>> > > >> > > > it very effectively for reasons explained above.
>> > > >> > > >
>> > > >> > > > Thanks
>> > > >> > > > Maulin
>> > > >> > > >
>> > > >> > > > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <
>> > > >> > > >
>> > > >> > > > kafka@harsha.io
>> > > >> > > >
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Hi Maulin,
>> > > >> > > >
>> > > >> > > > On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
>> > > >> > > >
>> > > >> > > > maulin.vasavada@gmail.
>> > > >> > > >
>> > > >> > > > com>
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Hi Harsha
>> > > >> > > >
>> > > >> > > > The reason we rejected the SslProvider route is that - we
>> > > >> > > >
>> > > >> > > > only
>> > > >> > > >
>> > > >> > > > needed
>> > > >> > > >
>> > > >> > > > a
>> > > >> > > >
>> > > >> > > > custom way to load keys/certs. Not touch any policy that
>> > > >> > > >
>> > > >> > > > existing
>> > > >> > > >
>> > > >> > > > Providers
>> > > >> > > >
>> > > >> > > > govern like SunJSSE Provider.
>> > > >> > > >
>> > > >> > > > We have exactly the same requirements to load certs and keys
>> > > >> > > >
>> > > >> > > > through
>> > > >> > > >
>> > > >> > > > spire
>> > > >> > > >
>> > > >> > > > agent. We used security.provider to do that exactly. I am not
>> > > >> > > >
>> > > >> > > > sure
>> > > >> > > >
>> > > >> > > > why
>> > > >> > > >
>> > > >> > > > you
>> > > >> > > >
>> > > >> > > > would be modifying any policies provided by default SunJSSE
>> > > >> > > >
>> > > >> > > > provider.
>> > > >> > > >
>> > > >> > > > Can
>> > > >> > > >
>> > > >> > > > you give me an example of having custom provider that will
>> > > >> > > >
>> > > >> > > > override an
>> > > >> > > >
>> > > >> > > > existing policy in SunJSSE provider.
>> > > >> > > >
>> > > >> > > > As pointed out earlier, this kip
>> > > >> > > >
>> > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/
>> > > >> > > >
>> KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>> > > >> > > >
>> > > >> > > > allows
>> > > >> > > > you to load security.provider through config.
>> > > >> > > > Take a look at the examples I gave before
>> > > >> > > >
>> > > >> > > >
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>> > > >> > > >
>> > > >> > > >
>> > > >> > >
>> > > >> >
>> > > >>
>> > >
>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>> > > >> > > >
>> > > >> > > > java
>> > > >> > > >
>> > > >> > > > It registers KeyManagerFactory and TrustManagerFactory and
>> > > >> > > >
>> > > >> > > > Keystore
>> > > >> > > >
>> > > >> > > > algorithm.
>> > > >> > > > Implement your custom way of loading Keystore in here
>> > > >> > > >
>> > > >> > > >
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>> > > >> > > >
>> > > >> > > >
>> > > >> > >
>> > > >> >
>> > > >>
>> > >
>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
>> > > >> > > >
>> > > >> > > > java
>> > > >> > > >
>> > > >> > > > and Trust manager like here
>> > > >> > > >
>> > > >> > > >
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>> > > >> > > > spiffe-security-provider/src/main/java/spiffe/api/provider/
>> > > >> > > > SpiffeTrustManager.java <http://spiffetrustmanager.java/>
>> > > <http://
>> > > >> > > > spiffetrustmanager.java/>
>> > > >> > > >
>> > > >> > > > In your Kafka client you can set the security.provider to
>> your
>> > > >> > > >
>> > > >> > > > custom
>> > > >> > > >
>> > > >> > > > implementation and with this fix
>> > > >> > > > https://issues.apache.org/jira/browse/KAFKA-8191 you can set
>> > > >> > > > keyManagerAlgorigthm and trustManagerAlgorithm configs.
>> > > >> > > >
>> > > >> > > > All of this is in your clients and broker side and do not
>> need
>> > > >> > > >
>> > > >> > > > to
>> > > >> > > >
>> > > >> > > > touch
>> > > >> > > >
>> > > >> > > > any
>> > > >> > > > policy changes at JVM level. You'll register the providers in
>> > > >> > > >
>> > > >> > > > the
>> > > >> > > >
>> > > >> > > > priority
>> > > >> > > >
>> > > >> > > > order and can still have SunJSSE provider and have your
>> custom
>> > > >> > > >
>> > > >> > > > provider
>> > > >> > > >
>> > > >> > > > to
>> > > >> > > >
>> > > >> > > > implement the key and trust managers.
>> > > >> > > >
>> > > >> > > > The ask here is different than KIP-492. We don't have any
>> need
>> > > >> > > >
>> > > >> > > > to
>> > > >> > > >
>> > > >> > > > modify/specify the algorithm parameter. Does that make sense?
>> > > >> > > >
>> > > >> > > > The ask in KIP is introducing new interfaces where the KIP's
>> > > >> > > > goal/motivation can be achieved through the security.provider
>> > > >> > > >
>> > > >> > > > and
>> > > >> > > >
>> > > >> > > > we
>> > > >> > > >
>> > > >> > > > worked
>> > > >> > > > on similar goal without touching any Keystore or Truststore
>> > > >> > > >
>> > > >> > > > interfaces.
>> > > >> > > >
>> > > >> > > > My advise is against changing or introducing new interfaces
>> > > >> > > >
>> > > >> > > > when
>> > > >> > > >
>> > > >> > > > it can
>> > > >> > > >
>> > > >> > > > work through security.provider.
>> > > >> > > >
>> > > >> > > > Thanks,
>> > > >> > > > Harsha
>> > > >> > > >
>> > > >> > > > Thanks
>> > > >> > > >
>> > > >> > > > Maulin
>> > > >> > > >
>> > > >> > > > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
>> > > >> > > >
>> > > >> > > > kafka@harsha.io>
>> > > >> > > >
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > In your KIP you added security. provider as rejected
>> > > >> > > >
>> > > >> > > > alternative
>> > > >> > > >
>> > > >> > > > and
>> > > >> > > >
>> > > >> > > > specified "its not the correct way". Do you mind explaining
>> > > >> > > >
>> > > >> > > > why
>> > > >> > > >
>> > > >> > > > its
>> > > >> > > >
>> > > >> > > > not? I
>> > > >> > > >
>> > > >> > > > didn't find any evidence in Java docs to say so. Contrary to
>> > > >> > > >
>> > > >> > > > your
>> > > >> > > >
>> > > >> > > > statement
>> > > >> > > >
>> > > >> > > > it does say in the java docs
>> > > >> > > > " However, please note that a provider can be used to
>> > > >> > > >
>> > > >> > > > implement
>> > > >> > > >
>> > > >> > > > any
>> > > >> > > >
>> > > >> > > > security service in Java that uses a pluggable architecture
>> > > >> > > >
>> > > >> > > > with
>> > > >> > > >
>> > > >> > > > a
>> > > >> > > >
>> > > >> > > > choice
>> > > >> > > >
>> > > >> > > > of implementations that fit underneath."
>> > > >> > > >
>> > > >> > > > Java Security Providers have been used by other projects to
>> > > >> > > >
>> > > >> > > > provide
>> > > >> > > >
>> > > >> > > > such
>> > > >> > > >
>> > > >> > > > integration . I am not sure if you looked into Spiffe
>> > > >> > > >
>> > > >> > > > project to
>> > > >> > > >
>> > > >> > > > efficiently distribute certificates but here is an example of
>> > > >> > > >
>> > > >> > > > Java
>> > > >> > > >
>> > > >> > > > provider
>> > > >> > > >
>> > > >> > > >
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>> > > >> > > >
>> > > >> > > >
>> > > >> > >
>> > > >> >
>> > > >>
>> > >
>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>> > > >> > > >
>> > > >> > > > java which
>> > > >> > > > obtains certificates from local daemons.
>> > > >> > > > These integrations are being used in Tomcat, Jetty etc.. We
>> > > >> > > >
>> > > >> > > > are
>> > > >> > > >
>> > > >> > > > also
>> > > >> > > >
>> > > >> > > > using
>> > > >> > > >
>> > > >> > > > Security provider to do the same in our Kafka clusters. So
>> > > >> > > >
>> > > >> > > > unless I
>> > > >> > > >
>> > > >> > > > see
>> > > >> > > >
>> > > >> > > > more evidence why security.provider doesn't work for you
>> > > >> > > >
>> > > >> > > > adding
>> > > >> > > >
>> > > >> > > > new
>> > > >> > > >
>> > > >> > > > interfaces while there exists more cleaner way of achieving
>> > > >> > > >
>> > > >> > > > the
>> > > >> > > >
>> > > >> > > > goals
>> > > >> > > >
>> > > >> > > > of
>> > > >> > > >
>> > > >> > > > this KIP is unnecessary and breaks the well known security
>> > > >> > > >
>> > > >> > > > interfaces
>> > > >> > > >
>> > > >> > > > provided by Java itself.
>> > > >> > > >
>> > > >> > > > Thanks,
>> > > >> > > > Harsha
>> > > >> > > >
>> > > >> > > > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
>> > > >> > > >
>> > > >> > > > kafka@harsha.io
>> > > >> > > >
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Hi Maulin,
>> > > >> > > > Not sure if you looked at my previous replies. This
>> > > >> > > >
>> > > >> > > > changes
>> > > >> > > >
>> > > >> > > > are not required as there is already security Provider to do
>> > > >> > > >
>> > > >> > > > what you
>> > > >> > > >
>> > > >> > > > are
>> > > >> > > >
>> > > >> > > > proposing. This KIP
>> > > >> > > >
>> > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/
>> > > >> > > >
>> > > >> > > >
>> KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>> > > >> > > >
>> > > >> > > > also
>> > > >> > > >
>> > > >> > > > addresses easy registration of such providers.
>> > > >> > > >
>> > > >> > > > Thanks,
>> > > >> > > > Harsha
>> > > >> > > >
>> > > >> > > > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
>> > > >> > > >
>> > > >> > > > <maulin.vasavada@gmail.
>> > > >> > > >
>> > > >> > > > com> wrote:
>> > > >> > > >
>> > > >> > > > Bump! Can somebody please review this?
>> > > >> > > >
>> > > >> > > > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
>> > > >> > > >
>> > > >> > > > maulin.vasavada@gmail.com>
>> > > >> > > >
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > Bump! Can somebody please review this?
>> > > >> > > >
>> > > >> > > > --
>> > > >> > > > Thanks,
>> > > >> > > > M.Sai Sandeep
>> > > >> > > >
>> > > >> > > >
>> > > >> > >
>> > > >> >
>> > > >>
>> > > >
>> > >
>> >
>>
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Maulin Vasavada <ma...@gmail.com>.
Why don't we make SSLEngineBuilder code delegate the whole Key/Trust store
initialization to the interfaces we are proposing? Default implementation
for those key/trust store loader interfaces will be "file based" loading vs
if somebody wants to customize any of it they can.
Would that make sense?
On Fri, Aug 30, 2019 at 10:03 AM Colin McCabe <cm...@apache.org> wrote:
> +1 for making SslEngineBuilder configurable. This would give implementers
> a lot more flexibility-- to use key distribution methods that were not
> files, for example.
>
> best,
> Colin
>
>
> On Fri, Aug 30, 2019, at 02:03, Rajini Sivaram wrote:
> > Just to make sure we are on the same page - KIP-383 was written before
> > the
> > code was refactored. The refactoring addressed some of the concerns of
> > KIP-383. My suggestion was to make SslEngineBuilder configurable. The
> > default implementation of this pluggable class would be
> >
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/ssl/SslEngineBuilder.java
> .
> > That should give an idea of the size of the configurable part that a
> > custom
> > class needs to implement. A large part of that is about loading
> > keystore/truststore. I agree it has slightly more code than KIP-486
> > proposes, but since it lets you customize creation of SSLEngine, it
> > would
> > address every possible scenario.
> >
> > Thoughts?
> >
> >
> > On Fri, Aug 30, 2019 at 2:02 AM Maulin Vasavada <
> maulin.vasavada@gmail.com>
> > wrote:
> >
> > > I thought about it more. I feel that no matter how we refactor the code
> > > (with or without KIP-383 integrated), ultimately the need of
> customizing
> > > loading for keys and certs will still remain. Whenever that need
> arises we
> > > might end up thinking about the solution suggested by our KIP-486.
> Hence
> > > regardless of the other KIPs and configurations "if we do need to
> customize
> > > loading of keys/certs, we will need the code changes suggested by this
> > > KIP".
> > >
> > > Let me know what you guys think.
> > >
> > > Harsha, we are working on changing the interfaces for key/trust store
> > > loaders with Certificate and PrivateKey objects. Will probably be able
> to
> > > update it later today or tomorrow.
> > >
> > > Thanks
> > > Maulin
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Aug 29, 2019 at 2:30 PM Maulin Vasavada <
> maulin.vasavada@gmail.com
> > > >
> > > wrote:
> > >
> > > > On that, I actually looked at KIP-383 before briefly. However, that
> > > > sounded like lot of changes suggested.
> > > >
> > > > One "key" thing we have to keep in mind is - IF we need lot of
> > > > customization Kafka already allows you to use your SslProvider via
> > > > ssl.providers or the changes done by KIP-492 and
> > > > SSLContext.getInstance(protocol, provider) call allows us to return
> the
> > > > SSLContext with "ALL" the details we would like to customize. Hence
> I am
> > > > not sure that customization suggested by KIP-383 would be worth the
> > > effort.
> > > > We also have similar SSLContext customization outside of Kafka.
> > > >
> > > > Thanks
> > > > Maulin
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Aug 29, 2019 at 12:47 PM Pellerin, Clement <
> > > > Clement_Pellerin@ibi.com> wrote:
> > > >
> > > >> KIP-383 in its present form was vetoed because it was not possible
> to
> > > add
> > > >> validation of custom properties in a future KIP. The solution to
> that is
> > > >> the first proposal I wrote for KIP-383 which made the whole
> SslFactory
> > > >> pluggable. That first solution was also vetoed hence the deadlock.
> > > >>
> > > >> Replacing the whole factory was a much nicer solution. It was vetoed
> > > >> because doing this almost invariably meant the replacement lost all
> the
> > > >> complex validation code in the default SslFactory.
> > > >>
> > > >> My current idea is to extract the validation code into another
> public
> > > API
> > > >> that SslFactory would call. I did not look at the newly refactored
> code
> > > and
> > > >> I did not study how to do this yet. KIP-383 was not popular at the
> time
> > > and
> > > >> designing a new solution is a lot of work.
> > > >>
> > > >> Is there interest from 3 binding voters for something like this?
> > > >>
> > > >> -----Original Message-----
> > > >> From: Rajini Sivaram [mailto:rajinisivaram@gmail.com]
> > > >> Sent: Thursday, August 29, 2019 2:57 PM
> > > >> To: dev
> > > >> Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and
> > > >> TrustStore
> > > >>
> > > >> Hi Maulin,
> > > >>
> > > >> In SSL scenarios, I imagine security providers introduced by
> KIP-492 are
> > > >> likely to be most useful when you want to use third party
> providers. The
> > > >> biggest advantage of the config from that KIP is that you don't
> need to
> > > >> write much code to integrate existing security providers into Kafka
> > > >> brokers
> > > >> or clients. As I understand it, KIP-486 is a more convenient option
> for
> > > >> the
> > > >> specific problem of loading keystores/truststores differently. It
> can be
> > > >> achieved in theory with KIP-492, but KIP-486 is a much simpler
> option
> > > for
> > > >> this case.
> > > >>
> > > >> My concern about KIP-486 is that it introduces yet another interface
> > > into
> > > >> our already complex security code, while only solving one
> particular use
> > > >> case. Have you looked at
> > > >>
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-383%3A++Pluggable+interface+for+SSL+Factory
> > > >> ?
> > > >> The goal was to make
> > > >> org.apache.kafka.common.security.ssl.SslEngineBuilder pluggable.
> > > >> The code has already been refactored by Colin after that KIP was
> > > written,
> > > >> making it easier to implement KIP-383. This should enable you to
> load
> > > your
> > > >> keystores and truststores differently. Using a pluggable
> > > SslEngineBuilder
> > > >> will also solve several other use cases at the same time. KIP-383
> hasn't
> > > >> been voted through yet, but perhaps you could take a look and we
> could
> > > >> revive that instead if it solves your use case as well?
> > > >>
> > > >> Regards,
> > > >>
> > > >> Rajini
> > > >>
> > > >>
> > > >> On Thu, Aug 29, 2019 at 6:42 PM Maulin Vasavada <
> > > >> maulin.vasavada@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > Hi Harsha
> > > >> >
> > > >> > Thank you. Appreciate your time and support on this. Let me go
> back do
> > > >> some
> > > >> > more research and get back to you on the KeyStore interface part.
> > > >> > Basically, if we return certs and keys in the interface then Kafka
> > > code
> > > >> > will have to build KeyStore object - which is also reasonable.
> > > >> >
> > > >> > Thanks
> > > >> > Maulin
> > > >> >
> > > >> > On Thu, Aug 29, 2019 at 10:01 AM Harsha Chintalapani <
> kafka@harsha.io
> > > >
> > > >> > wrote:
> > > >> >
> > > >> > > Hi Maulin,
> > > >> > > Use cases are clear now. I am +1 for moving
> > > >> forward
> > > >> > > with the discussions on having such configurable option for
> users.
> > > But
> > > >> > the
> > > >> > > interfaces is proposed doesn't look right to me. We are still
> > > talking
> > > >> > about
> > > >> > > keystore interfaces. Given keystore's are used as filebased
> way of
> > > >> > > transporting certificates I am not sure it will help the rest
> of the
> > > >> > > user-base.
> > > >> > > In short, I am +1 on the KIP's motivation and
> only
> > > >> have
> > > >> > > questions around returning keystores instead of returning certs,
> > > >> private
> > > >> > > keys etc. . If others in the community are ok with such
> interface we
> > > >> can
> > > >> > > move forward.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Harsha
> > > >> > >
> > > >> > >
> > > >> > > On Wed, Aug 28, 2019 at 1:51 PM, Maulin Vasavada <
> > > >> > > maulin.vasavada@gmail.com>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Hi Harsha
> > > >> > > >
> > > >> > > > As we synced-up offline on this topic, we hope you don't have
> any
> > > >> more
> > > >> > > > clarifications that you are seeking. If that is the case, can
> you
> > > >> > please
> > > >> > > > help us move this forward and discuss what changes you would
> > > expect
> > > >> on
> > > >> > > the
> > > >> > > > KIP design in order to make it valuable contribution?
> > > >> > > >
> > > >> > > > Just FYI - we verified our primary design change with the
> author
> > > of
> > > >> > Sun's
> > > >> > > > X509 Trustmanager implementation and the outcome is that what
> we
> > > are
> > > >> > > > proposing makes sense at the heart of it - "Instead of writing
> > > >> > > TrustManager
> > > >> > > > just plugin the Trust store". We are open to discuss
> additional
> > > >> changes
> > > >> > > > that you/anybody else would like to see on the functionality
> > > >> however.
> > > >> > > >
> > > >> > > > Thanks
> > > >> > > > Maulin
> > > >> > > >
> > > >> > > > On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada <
> > > >> > > maulin.vasavada@gmail.com>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > Hi Harsha
> > > >> > > >
> > > >> > > > Any response on my question? I feel this KIP is worth
> > > accommodating.
> > > >> > Your
> > > >> > > > help is much appreciated.
> > > >> > > >
> > > >> > > > Thanks
> > > >> > > > Maulin
> > > >> > > >
> > > >> > > > On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada <
> > > >> > maulin.vasavada@gmail.
> > > >> > > > com> wrote:
> > > >> > > >
> > > >> > > > Hi Harsha
> > > >> > > >
> > > >> > > > I've examined the SPIFFE provider more and have one question -
> > > >> > > >
> > > >> > > > If SPIFFE didn't have a need to do checkSpiffeId() call at the
> > > below
> > > >> > > > location, would you really still write the Provider? *OR*
> Would
> > > you
> > > >> > just
> > > >> > > > use TrustManagerFactory.init(KeyStore) signature to pass the
> > > >> KeyStore
> > > >> > > from
> > > >> > > > set of certs returned by spiffeIdManager. getTrustedCerts()?
> > > >> > > >
> > > >> > > >
> > > >> >
> > >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > >> > > > provider/CertificateUtils.java#L100
> > > >> > > >
> > > >> > > > /**
> > > >> > > >
> > > >> > > > * Validates that the SPIFFE ID is present and matches the
> SPIFFE
> > > ID
> > > >> > > > configured in
> > > >> > > > * the java.security property ssl.spiffe.accept
> > > >> > > > *
> > > >> > > > * If the authorized spiffe ids list is empty any spiffe id is
> > > >> > authorized
> > > >> > > > *
> > > >> > > > * @param chain an array of X509Certificate that contains the
> > > Peer's
> > > >> > SVID
> > > >> > > > to be validated
> > > >> > > > * @throws CertificateException when either the certificates
> > > doesn't
> > > >> > have
> > > >> > > a
> > > >> > > > SPIFFE ID or the SPIFFE ID is not authorized
> > > >> > > > */
> > > >> > > > static void checkSpiffeId(X509Certificate[] chain) throws
> > > >> > > > CertificateException {
> > > >> > > >
> > > >> > > > Thanks
> > > >> > > > Maulin
> > > >> > > >
> > > >> > > > On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani <
> > > >> kafka@harsha.io>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > Maulin,
> > > >> > > > The code parts you are pointing are specific for Spiffe and if
> > > >> > > > you are talking about validate method which uses PKIX check
> like
> > > any
> > > >> > > other
> > > >> > > > provider does.
> > > >> > > > If you want to default to SunJSSE everywhere you can do so by
> > > >> > delegating
> > > >> > > > the calls in these methods to SunJSSE provider.
> > > >> > > >
> > > >> > > > TrustManagerFactory tmf = TrustManagerFactory
> > > >> > > > .getInstance(TrustManagerFactory.getDefaultAlgorithm());and
> use
> > > >> > > > tmf.chekServerTrusted()
> > > >> > > > or use
> > > >> > > > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> > > >> > > > TrustManagerFactory.html#getInstance(java.lang.String)if you
> want
> > > a
> > > >> > > > specific provider.
> > > >> > > >
> > > >> > > > -Harsha
> > > >> > > >
> > > >> > > > On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada <
> > > >> > maulin.vasavada@gmail.
> > > >> > > > com>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > Okay, so I take that you guys agree that I have to write a
> > > 'custom'
> > > >> > > > algorithm and a provider to make it work , correct?
> > > >> > > >
> > > >> > > > Now, for Harsha's comment "Here the 'Custom' Algorithm is not
> an
> > > >> > > > implementation per say , ..." , I diagree. You can refer to
> > > https://
> > > >> > > >
> > > >> > > >
> > > >> >
> > > >>
> > >
> github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/
> > > >> > > >
> > > >> > > > SpiffeTrustManager.java#L91 <
> http://spiffetrustmanager.java/#L91>
> > > >> and
> > > >> > > >
> > > >> > > >
> > > >> >
> > >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > >> > > >
> > > >> > > > provider/CertificateUtils.java#L100
> > > >> > > >
> > > >> > > > "that code" is the customization you have for the custom way
> to
> > > >> check
> > > >> > > > something on top of regular checks. That method is NOT doing
> > > custom
> > > >> > > > truststore loading. It is validating/verifying something in
> the
> > > >> > > >
> > > >> > > > "custom"
> > > >> > > >
> > > >> > > > way with spiffeId.
> > > >> > > > I bet that without that you won't have a need of the custom
> > > >> algorithm
> > > >> > > >
> > > >> > > > in
> > > >> > > >
> > > >> > > > the first place.
> > > >> > > >
> > > >> > > > Let me know if you agree to this.
> > > >> > > >
> > > >> > > > Thanks
> > > >> > > > Maulin
> > > >> > > >
> > > >> > > > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <
> mprsai@gmail.com>
> > > >> > > >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > Hi Maulin, thanks for the discussion. As Harsha pointed out,
> to
> > > use
> > > >> the
> > > >> > > > KIP492, you need to create a new provider, register a *new*
> custom
> > > >> > > > algorithm for your keymanager and trustmanager factory
> > > >> implementations.
> > > >> > > > After this, the kafka server configuration can be done as
> given
> > > >> below
> > > >> > > >
> > > >> > > > # Register the provider class with custom algorithm, say
> CUSTOM
> > > >> > > >
> > > >> > > > security.
> > > >> > > >
> > > >> > > > provider.classes=com.company.security.CustomProvider
> > > >> > > > <http://provider.classes
> =com.company.security.customprovider/>
> > > >> > > > <http://security.provider.classes
> > > >> > > >
> > > >> > > > =com.company.security.customprovider/>
> > > >> > > >
> > > >> > > > # Register the keymanager and trustmanager algorithms
> > > >> > > > # These algorithms indicate that the Keymanager and
> Trustmanagers
> > > >> > > > registered under the algorithm "CUSTOM" needs to be used
> > > >> > > > ssl.trustmanager.algorithm=CUSTOM
> > > >> > > > ssl.keymanager.algorithm=CUSTOM
> > > >> > > >
> > > >> > > > And the customprovider looks like this...
> > > >> > > >
> > > >> > > > public class CustomProvider extends Provider {
> > > >> > > > public CustomProvider() {
> > > >> > > > super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and
> > > TrustStore");
> > > >> > > > super.put("KeyManagerFactory.CUSTOM",
> "customKeyManagerFactory");
> > > >> > > > super.put("TrustManagerFactory.CUSTOM",
> > > >> > > > "customTrustManagerFactory");
> > > >> > > > }
> > > >> > > > }
> > > >> > > >
> > > >> > > > The PR for this is in review and can be found here:
> > > >> > > >
> > > >> > > > https://github.com/
> > > >> > > >
> > > >> > > > apache/kafka/pull/7090
> > > >> > > > This PR includes the fixed insertProviderAt function call.
> > > >> > > >
> > > >> > > > On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani <
> > > >> kafka@harsha.io>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > Answers are in-line
> > > >> > > >
> > > >> > > > On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada <
> > > >> > > >
> > > >> > > > maulin.vasavada@gmail.
> > > >> > > >
> > > >> > > > com
> > > >> > > >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > Hi Colin
> > > >> > > >
> > > >> > > > When I refer to "standard" or "custom" algorithms I am
> following
> > > >> Java
> > > >> > > > security Provider Terminology. You can refer to
> > > >> > > >
> https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> > > >> > > > StandardNames.html#TrustManagerFactory link I provided
> earlier in
> > > >> the
> > > >> > > > emails. It says PKIX is the default Algorithm for
> > > >> TrustManagerFactory.
> > > >> > > >
> > > >> > > > 1. For SPIFFE, I am not sure why you are saying 'it does not
> > > >> implement
> > > >> > > > custom algorithms' because the following file clearly
> indicates
> > > >> that it
> > > >> > > > does use custom algorithm-
> > > >> > > >
> > > >> > > >
> > > >> >
> > >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > >> > > >
> > > >> > > > provider/SpiffeProvider.java#L17
> > > >> > > >
> > > >> > > > Algorithm value:
> > > >> > > >
> > > >> > > >
> > > >> >
> > >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > >> > > >
> > > >> > > > provider/SpiffeProviderConstants.java#L6
> > > >> > > >
> > > >> > > > @Harsha do you want to chime in since you use that provider?
> > > >> > > >
> > > >> > > > Here the "Custom" Algorithm is not an implementation per say ,
> > > >> rather
> > > >> > > >
> > > >> > > > used
> > > >> > > >
> > > >> > > > to invoke the custom trust store factory and key manager
> factory.
> > > >> You
> > > >> > > >
> > > >> > > > are
> > > >> > > >
> > > >> > > > not moving away from "standard" alogrithms that are available.
> > > >> > > >
> > > >> > > >
> > > >> >
> > >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > >> > > >
> > > >> > > > provider/SpiffeTrustManager.java
> > > >> > > >
> > > >> > > > As you can see it delegates all the calls of verification of
> > > >> > > >
> > > >> > > > certificate
> > > >> > > >
> > > >> > > > to
> > > >> > > >
> > > >> > > > the default implementation available.
> > > >> > > > So in our implementation we still use PKIX to verify the
> > > certificate
> > > >> > > > chain. So you are not losing anything here and Spiffe is not
> > > >> > > >
> > > >> > > > reimplementing
> > > >> > > >
> > > >> > > > the verification process.
> > > >> > > >
> > > >> > > > 2. I already mentioned in my 3rd point, in my previous post,
> why
> > > >> using
> > > >> > > >
> > > >> > > > ssl.provider does NOT work. I updated KIP-486 in "rejected
> > > >> > > >
> > > >> > > > alternatives"
> > > >> > > >
> > > >> > > > also why ssl.provider does not work.
> > > >> > > >
> > > >> > > > As mentioned before , provider is the right way to go. I am
> still
> > > >> not
> > > >> > > > understanding the gap is.
> > > >> > > > If I understand correctly your argument is , provider is
> going to
> > > >> ask
> > > >> > > >
> > > >> > > > to
> > > >> > > >
> > > >> > > > implement a custom algorithm.
> > > >> > > > My answer to that is , no you are not re-implementing the
> > > algorithm.
> > > >> > > >
> > > >> > > > Please
> > > >> > > >
> > > >> > > > check the above link , TrustManager implementation it
> delegates
> > > the
> > > >> > > >
> > > >> > > > calls
> > > >> > > >
> > > >> > > > back. There is no need to implement your own here.
> > > >> > > >
> > > >> > > > 3. Security.insertProviderAt() comments were based on
> assumption
> > > if
> > > >> > > >
> > > >> > > > KIP-492
> > > >> > > >
> > > >> > > > changes are done and we use that mechanism to configure
> providers
> > > >> > > >
> > > >> > > > instead
> > > >> > > >
> > > >> > > > of ssl.provider configuration.
> > > >> > > >
> > > >> > > > KIP-492 has patch available and going through review.
> > > >> > > >
> > > >> > > > Can you read my all the points, I mentioned in my previous
> post,
> > > >> very
> > > >> > > >
> > > >> > > > carefully? I am covering all the aspects in explaining. I am
> open
> > > to
> > > >> > > >
> > > >> > > > still
> > > >> > > >
> > > >> > > > discuss more to clarify any doubts.
> > > >> > > >
> > > >> > > > "3. If we just use existing ssl.provider kafka configuration
> then
> > > >> our
> > > >> > > > provider will be used in SSLContext.getInstance(protocol,
> > > provider)
> > > >> > > >
> > > >> > > > call
> > > >> > > >
> > > >> > > > in
> > > >> > > >
> > > >> > > > SslFactory.java <http://sslfactory.java/> <
> > > http://sslfactory.java/>
> > > >> <
> > > >> > > > http://sslfactory.java/>
> > > >> > > >
> > > >> > > > and
> > > >> > > >
> > > >> > > > if our provider does not have
> > > >> > > > implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it
> breaks
> > > (we
> > > >> > > >
> > > >> > > > tested
> > > >> > > >
> > > >> > > > it). Example: In MyProvider sample above you see that I
> didn't add
> > > >> > > > SSLContext.TLSv1 as
> > > >> > > > "Service+Algorithm" and that didn't work for me. In SPIFFE
> > > provider
> > > >> you
> > > >> > > > don't have this challenge since you are planning to bypass
> > > >> ssl.provider
> > > >> > > >
> > > >> > > > as
> > > >> > > >
> > > >> > > > you mention in the KIP-492."
> > > >> > > >
> > > >> > > > Yes here you need to pass the protocol that your
> > > >> > > >
> > > >> > > > KeyManager/TrustManager
> > > >> > > >
> > > >> > > > registered with and in no way its deviating from TLS RFC spec.
> > > >> > > >
> > > >> > > >
> > > >> >
> > >
> https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/
> > > >> > > >
> > > >> > > > javax/net/ssl/SSLContext.java#L134
> > > >> > > >
> > > >> > > > My suggestion here is for you to implement a simple Security
> > > >> Provider
> > > >> > > >
> > > >> > > > as
> > > >> > > >
> > > >> > > > you did before and register a Algorithm. You can use the
> existing
> > > >> > > > implementation in SpiffeProvider and plug in changes where you
> > > need
> > > >> to
> > > >> > > > retrieve the certificates from by making RPC call.
> > > >> > > >
> > > >> > > > Run an end-to-end test with Kafka broker coming up with your
> > > >> provider
> > > >> > > > making calls to RPC call. You do need to pass the "custom
> > > algorithm"
> > > >> > > >
> > > >> > > > that
> > > >> > > >
> > > >> > > > you registered in your key manager to invoke the provider. I
> think
> > > >> your
> > > >> > > > concern is this approach is replacing the existing known
> > > >> ciphersuites
> > > >> > > >
> > > >> > > > and
> > > >> > > >
> > > >> > > > certificate validation provided by java. Which its not.
> > > >> > > >
> > > >> > > > Now test the TLS connection you can do so via openssl
> -s_client
> > > >> options
> > > >> > > >
> > > >> > > > to
> > > >> > > >
> > > >> > > > test if everything else is working.
> > > >> > > >
> > > >> > > > I am happy to share configs that we used if you are
> interested.
> > > >> > > >
> > > >> > > > Thanks,
> > > >> > > > Harsha
> > > >> > > >
> > > >> > > > Thanks
> > > >> > > > Maulin
> > > >> > > >
> > > >> > > > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <
> cmccabe@apache.org>
> > > >> > > >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > Hi Maulin,
> > > >> > > >
> > > >> > > > A lot of JSSE providers don't implement custom algorithms.
> Spire
> > > is
> > > >> a
> > > >> > > >
> > > >> > > > good
> > > >> > > >
> > > >> > > > example of a JSSE provider that doesn't, and yet is still
> useful
> > > to
> > > >> > > >
> > > >> > > > many
> > > >> > > >
> > > >> > > > people. Your JSSE provider can work fine even if it doesn't
> > > >> implement a
> > > >> > > > custom algorithm.
> > > >> > > >
> > > >> > > > Maybe I'm missing something, but I don't understand the
> discussion
> > > >> of
> > > >> > > > Security.insertProviderAt() that you included.
> SslEngineBuilder
> > > >> doesn't
> > > >> > > >
> > > >> > > > use
> > > >> > > >
> > > >> > > > that API to get the security provider. Instead, it calls
> > > >> > > > "SSLContext.getInstance(protocol, provider)", where provider
> is
> > > the
> > > >> > > >
> > > >> > > > name
> > > >> > > >
> > > >> > > > of the provider.
> > > >> > > >
> > > >> > > > best,
> > > >> > > > Colin
> > > >> > > >
> > > >> > > > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
> > > >> > > >
> > > >> > > > On top of everything above I feel strongly to add the 4th
> point
> > > >> which
> > > >> > > >
> > > >> > > > is
> > > >> > > >
> > > >> > > > based on Java APIs for TrustManagerFactory.init(KeyStore) (
> > > >> > > >
> > > >> > > > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> > > >> > > > TrustManagerFactory.html#init(java.security.KeyStore
> > > >> > > > <http://java.security.keystore/>
> > > >> > > > <http://java.security.keystore/>)
> > > >> > > > )
> > > >> > > >
> > > >> > > > and KeyManagerFactory.init(KeyStore, char[]) (
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory
> > > >> > > > .
> > > >> > > >
> > > >> > > > html#init(java.security.KeyStore <
> http://java.security.keystore/>
> > > >> > > <http://
> > > >> > > > java.security.keystore/
> > > >> > > > ,%20char[])
> > > >> > > >
> > > >> > > > ).
> > > >> > > >
> > > >> > > > 4. The above APIs are intended to support providing "trust/key
> > > >> > > >
> > > >> > > > material"
> > > >> > > >
> > > >> > > > from the user without having to write their own
> > > >> > > >
> > > >> > > > TrustManager/KeyManagers.
> > > >> > > >
> > > >> > > > To quote from the TrustManagerFactory.init()'s documentation
> > > >> > > >
> > > >> > > > "Initializes
> > > >> > > >
> > > >> > > > this factory with a source of certificate authorities and
> related
> > > >> trust
> > > >> > > > material."
> > > >> > > > To quote from the KeyManagerFactory.init()'s documentation
> > > >> "Initializes
> > > >> > > > this factory with a source of key material."
> > > >> > > >
> > > >> > > > Based on this it is clear that there is a flexibility
> provided by
> > > >> Java
> > > >> > > >
> > > >> > > > to
> > > >> > > >
> > > >> > > > to enable developers to provide the required trust/key
> material
> > > >> loaded
> > > >> > > >
> > > >> > > > from
> > > >> > > >
> > > >> > > > "anywhere" without requiring them to write custom provider OR
> > > >> trust/key
> > > >> > > > managers. This same flexibility is reflected in Kafka code
> also
> > > >> where
> > > >> > > >
> > > >> > > > it
> > > >> > > >
> > > >> > > > loads the trust/keys from a local file and doesn't require
> > > writing a
> > > >> > > > Provider necessarily. If we do NOT have a custom algorithm, it
> > > makes
> > > >> > > >
> > > >> > > > less
> > > >> > > >
> > > >> > > > sense to write a Provider.
> > > >> > > >
> > > >> > > > Thanks
> > > >> > > > Maulin
> > > >> > > >
> > > >> > > > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
> > > >> > > >
> > > >> > > > maulin.vasavada@gmail.com>
> > > >> > > >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > Hi Harsha/Colin
> > > >> > > >
> > > >> > > > I did the sample with a custom Provider for TrustStoreManager
> and
> > > >> tried
> > > >> > > > using ssl.provider Kafka config AND the way KIP-492 is
> suggesting
> > > >> (by
> > > >> > > > adding Provider programmatically instead of relying on
> > > >> > > >
> > > >> > > > ssl.provider+java.
> > > >> > > >
> > > >> > > > security. The below sample is followed by my detailed
> findings.
> > > I'll
> > > >> > > > appreciate if you can go through it carefully and see
> > > >> > > >
> > > >> > > > if you
> > > >> > > >
> > > >> > > > see my point.
> > > >> > > >
> > > >> > > > package providertest;
> > > >> > > >
> > > >> > > > import java.security.Provider <http://java.security.provider/
> >
> > > >> > <http://
> > > >> > > > java.security.provider/>
> > > >> > > >
> > > >> > > > <http://
> > > >> > > >
> > > >> > > > java.security.provider/>;
> > > >> > > >
> > > >> > > > public class MyProvider extends Provider {
> > > >> > > >
> > > >> > > > private static final String name = "MyProvider"; private
> static
> > > >> double
> > > >> > > > version = 1.0d;
> > > >> > > > private static String info = "Maulin's SSL Provider
> v"+version;
> > > >> > > >
> > > >> > > > public MyProvider() {
> > > >> > > > super(name, version, info);
> > > >> > > > this.put("TrustManagerFactory.PKIX",
> > > >> > > >
> > > >> > > > "providertest.MyTrustManagerFactory");
> > > >> > > >
> > > >> > > > }
> > > >> > > > }
> > > >> > > >
> > > >> > > > *Details:*
> > > >> > > >
> > > >> > > > KIP-492 documents that it will use Security.addProvider()
> assuming
> > > >> it
> > > >> > > >
> > > >> > > > will
> > > >> > > >
> > > >> > > > add it as position '0' which is not a correct assumption. The
> > > >> > > > addProvider()'s documentation says it will add it to the last
> > > >> available
> > > >> > > > position. You may want to correct that to say
> > > >> > > > Security.insertProviderAt(provider, 1).
> > > >> > > >
> > > >> > > > Now coming back to our specific discussion,
> > > >> > > >
> > > >> > > > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when
> you
> > > add
> > > >> > > >
> > > >> > > > that
> > > >> > > >
> > > >> > > > provider in the provider list via Security.addProvider() the
> > > >> position
> > > >> > > >
> > > >> > > > where
> > > >> > > >
> > > >> > > > it gets added doesn't matter (even if you don't end up adding
> it
> > > as
> > > >> > > >
> > > >> > > > first
> > > >> > > >
> > > >> > > > entry) since that is the ONLY provider for SPIFFE specific
> > > algorithm
> > > >> > > >
> > > >> > > > you
> > > >> > > >
> > > >> > > > might have.
> > > >> > > >
> > > >> > > > We do *not* have custom algorithm for Key/Trust StoreMangers.
> > > Which
> > > >> > > >
> > > >> > > > means
> > > >> > > >
> > > >> > > > we have to use X509, PKIX etc "Standard Algorithms" ((
> > > >> > > >
> > > >> > > >
> https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> > > >> > > > StandardNames.html
> > > >> > > > ))
> > > >> > > >
> > > >> > > > in our provider to override the TrustStoreManager (see my
> sample
> > > >> code)
> > > >> > > >
> > > >> > > > and
> > > >> > > >
> > > >> > > > KeyStoreManger and KeyManager. This creates another challenge
> > > >> > > >
> > > >> > > > mentioned in
> > > >> > > >
> > > >> > > > the below point.
> > > >> > > >
> > > >> > > > 2. In order to make our Provider for loading custom TrustStore
> > > >> work, we
> > > >> > > > have to add the provider as 'first' in the list since there
> are
> > > >> others
> > > >> > > >
> > > >> > > > with
> > > >> > > >
> > > >> > > > the same algorithm.
> > > >> > > >
> > > >> > > > However, the programatic way of adding provider
> > > >> > > > (Security.insertProviderAt()) is *not* deterministic for
> ordering
> > > >> since
> > > >> > > > different code can call the method for a different provider
> and
> > > >> > > >
> > > >> > > > depending
> > > >> > > >
> > > >> > > > upon the order of the call our provider can be first or pushed
> > > down
> > > >> the
> > > >> > > > list. This can happen very well in any client application
> using
> > > >> Kafka.
> > > >> > > >
> > > >> > > > This
> > > >> > > >
> > > >> > > > is specially problematic for a case when you want to guarantee
> > > order
> > > >> > > >
> > > >> > > > for a
> > > >> > > >
> > > >> > > > Provider having "Standard Algorithms".
> > > >> > > >
> > > >> > > > If we add our provider in java.security file that definitely
> > > >> guarantees
> > > >> > > > the order(unless somebody calls removeProvider() which is less
> > > >> > > >
> > > >> > > > likely). But
> > > >> > > >
> > > >> > > > if we add our provider in java.security file it will defeat
> the
> > > >> > > >
> > > >> > > > purpose of
> > > >> > > >
> > > >> > > > the KIP-492.
> > > >> > > >
> > > >> > > > In the gist - Apache Kafka must not rely on "un-deterministic"
> > > >> method
> > > >> > > >
> > > >> > > > to
> > > >> > > >
> > > >> > > > rely on Provider ordering.
> > > >> > > >
> > > >> > > > 3. If we just use existing ssl.provider kafka configuration
> then
> > > our
> > > >> > > > provider will be used in SSLContext.getInstance(protocol,
> > > provider)
> > > >> > > >
> > > >> > > > call in
> > > >> > > >
> > > >> > > > SslFactory.java <http://sslfactory.java/> <
> > > http://sslfactory.java/>
> > > >> <
> > > >> > > > http://sslfactory.java/>
> > > >> > > >
> > > >> > > > and
> > > >> > > >
> > > >> > > > if our provider does not have implementation for
> > > >> > > > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks
> > > >> > > >
> > > >> > > > (we
> > > >> > > >
> > > >> > > > tested it). Example:
> > > >> > > >
> > > >> > > > In
> > > >> > > >
> > > >> > > > MyProvider sample above you see that I didn't add
> SSLContext.TLSv1
> > > >> as
> > > >> > > > "Service+Algorithm" and that didn't work for me. In SPIFFE
> > > provider
> > > >> you
> > > >> > > > don't have this challenge since you are planning to bypass
> > > >> > > >
> > > >> > > > ssl.provider as
> > > >> > > >
> > > >> > > > you mention in the KIP-492.
> > > >> > > >
> > > >> > > > *Overall summary,*
> > > >> > > >
> > > >> > > > 1. Any provider based mechanisms- a) existing ssl.provider and
> > > >> > > >
> > > >> > > > b)KIP-492,
> > > >> > > >
> > > >> > > > for loading key/trust store using "Standard Algorithms" do not
> > > work
> > > >> > > >
> > > >> > > > 2. Approach suggested in our KIP-486 works without any issue
> and
> > > it
> > > >> is
> > > >> > > > *not* our context specific solve
> > > >> > > >
> > > >> > > > 3. Based on above we feel KIP-492 and KIP-486 are
> complimentary
> > > >> changes
> > > >> > > > and not contradicting or redundent.
> > > >> > > >
> > > >> > > > If you want we can do a joint session somehow to walk through
> the
> > > >> > > >
> > > >> > > > sample I
> > > >> > > >
> > > >> > > > have and various experiments I did. I would encourage you to
> do
> > > >> similar
> > > >> > > > exercise by writing a Provider for "Standard Algorithm" for
> > > >> > > > TrustStoreManager (like our needs) and see what you find since
> > > only
> > > >> > > >
> > > >> > > > writing
> > > >> > > >
> > > >> > > > samples can bring out the complexity/challenges we face.
> > > >> > > >
> > > >> > > > Thanks
> > > >> > > > Maulin
> > > >> > > >
> > > >> > > > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <
> > > >> > > >
> > > >> > > > maulin.vasavada@gmail.
> > > >> > > >
> > > >> > > > com> wrote:
> > > >> > > >
> > > >> > > > Just to update - still working on it. Get to work only on and
> off
> > > on
> > > >> > > >
> > > >> > > > it :(
> > > >> > > >
> > > >> > > > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
> > > >> > > >
> > > >> > > > maulin.vasavada@gmail.com>
> > > >> > > >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > Hi Harsha
> > > >> > > >
> > > >> > > > Let me try to write samples and will let you know.
> > > >> > > >
> > > >> > > > Thanks
> > > >> > > > Maulin
> > > >> > > >
> > > >> > > > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <harsha.ch@gmail.com
> >
> > > >> > > >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > Hi Maulin,
> > > >> > > > With java security providers can be as custom you would
> > > >> > > >
> > > >> > > > like
> > > >> > > >
> > > >> > > > it to
> > > >> > > > be. If you only want to to implement a custom way of loading
> the
> > > >> > > >
> > > >> > > > keystore
> > > >> > > >
> > > >> > > > and truststore and not implement any protocol/encryption
> handling
> > > >> you
> > > >> > > >
> > > >> > > > can
> > > >> > > >
> > > >> > > > leave them empty and no need to implement. Have you looked
> into
> > > the
> > > >> > > >
> > > >> > > > links I
> > > >> > > >
> > > >> > > > pasted before?
> > > >> > > >
> > > >> > > >
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> > > >> > > >
> > > >> > > > java
> > > >> > > >
> > > >> > > >
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > >> > > > spiffe-security-provider/src/main/java/spiffe/api/provider/
> > > >> > > > SpiffeTrustManager.java <http://spiffetrustmanager.java/>
> > > <http://
> > > >> > > > spiffetrustmanager.java/>
> > > >> > > >
> > > >> > > >
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > > >> > > >
> > > >> > > > java
> > > >> > > >
> > > >> > > > Can you please tell me which methods are too complex in above
> to
> > > >> > > >
> > > >> > > > implement
> > > >> > > >
> > > >> > > > or unnecessary? You are changing anything in SSL/TLS
> > > implementations
> > > >> > > > provided by
> > > >> > > >
> > > >> > > > All of the implementations delegating the checks to the
> default
> > > >> > > > implementation anyway.
> > > >> > > > Spire agent is an example, its nothing but a GRPC server
> listening
> > > >> > > >
> > > >> > > > on a
> > > >> > > >
> > > >> > > > unix domain socket . Above code is making a RPC call to the
> local
> > > >> > > >
> > > >> > > > daemon
> > > >> > > >
> > > >> > > > to
> > > >> > > > get the certificate and keys. The mechanics are pretty much
> same
> > > as
> > > >> > > >
> > > >> > > > what
> > > >> > > >
> > > >> > > > you are asking for.
> > > >> > > >
> > > >> > > > Thanks,
> > > >> > > > Harsha
> > > >> > > >
> > > >> > > > On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
> > > >> > > >
> > > >> > > > maulin.vasavada@gmail.com>
> > > >> > > >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > Imagine a scenario like - We know we have a custom KMS and as
> a
> > > >> > > >
> > > >> > > > Kafka
> > > >> > > >
> > > >> > > > owner
> > > >> > > >
> > > >> > > > we want to comply to using that KMS source to load
> keys/certs. As
> > > >> > > >
> > > >> > > > a
> > > >> > > >
> > > >> > > > Kafka
> > > >> > > >
> > > >> > > > owner we know how to integrate with KMS but doesn't
> necessarily
> > > >> > > >
> > > >> > > > have
> > > >> > > >
> > > >> > > > to
> > > >> > > >
> > > >> > > > know anything about cipher suites, algorithms, and SSL/TLS
> > > >> > > >
> > > >> > > > implementation.
> > > >> > > >
> > > >> > > > Going the Provider way requires to know lot more than we
> should,
> > > >> > > >
> > > >> > > > isn't it?
> > > >> > > >
> > > >> > > > Not that we would have concern/shy-away knowing those details
> -
> > > >> > > >
> > > >> > > > but
> > > >> > > >
> > > >> > > > if we
> > > >> > > >
> > > >> > > > don't have to - why should we?
> > > >> > > >
> > > >> > > > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
> > > >> > > >
> > > >> > > > maulin.vasavada@gmail.com>
> > > >> > > >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > Hi Harsha
> > > >> > > >
> > > >> > > > We don't have spire (or similar) agents and we do not have
> > > >> > > >
> > > >> > > > keys/certs
> > > >> > > >
> > > >> > > > locally on any brokers. To elaborate more on my previous
> email,
> > > >> > > >
> > > >> > > > I agree that Java security Providers are used in much broader
> > > >> > > >
> > > >> > > > sense
> > > >> > > >
> > > >> > > > - to
> > > >> > > >
> > > >> > > > have a particular implementation of an algorithm, use specific
> > > >> > > >
> > > >> > > > cipher
> > > >> > > >
> > > >> > > > suites for SSL , OR in our current team's case have a
> > > >> > > >
> > > >> > > > particular
> > > >> > > >
> > > >> > > > way to
> > > >> > > >
> > > >> > > > leverage pre-generated SSL sessions. However, the scope of our
> > > >> > > >
> > > >> > > > KIP
> > > >> > > >
> > > >> > > > (486)
> > > >> > > >
> > > >> > > > is
> > > >> > > >
> > > >> > > > much restricted than that. We merely intend to provide a
> custom
> > > >> > > > keystore/truststore for our SSL connections and not really
> worry
> > > >> > > >
> > > >> > > > about
> > > >> > > >
> > > >> > > > underlying specific SSL/TLS implementation. This simplifies it
> > > >> > > >
> > > >> > > > a
> > > >> > > >
> > > >> > > > lot for
> > > >> > > >
> > > >> > > > us to keep the concerns separate and clear.
> > > >> > > >
> > > >> > > > I feel our approach is more complimentary such that it allows
> > > >> > > >
> > > >> > > > for
> > > >> > > >
> > > >> > > > using
> > > >> > > >
> > > >> > > > keystores of choice while retaining the flexibility to use any
> > > >> > > > underlying/available Provider for actually making the SSL
> call.
> > > >> > > >
> > > >> > > > We agree with KIP-492's approach based on Providers (and
> Java's
> > > >> > > > recommendation), but also strongly believe that our approach
> can
> > > >> > > >
> > > >> > > > compliment
> > > >> > > >
> > > >> > > > it very effectively for reasons explained above.
> > > >> > > >
> > > >> > > > Thanks
> > > >> > > > Maulin
> > > >> > > >
> > > >> > > > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <
> > > >> > > >
> > > >> > > > kafka@harsha.io
> > > >> > > >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > Hi Maulin,
> > > >> > > >
> > > >> > > > On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
> > > >> > > >
> > > >> > > > maulin.vasavada@gmail.
> > > >> > > >
> > > >> > > > com>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > Hi Harsha
> > > >> > > >
> > > >> > > > The reason we rejected the SslProvider route is that - we
> > > >> > > >
> > > >> > > > only
> > > >> > > >
> > > >> > > > needed
> > > >> > > >
> > > >> > > > a
> > > >> > > >
> > > >> > > > custom way to load keys/certs. Not touch any policy that
> > > >> > > >
> > > >> > > > existing
> > > >> > > >
> > > >> > > > Providers
> > > >> > > >
> > > >> > > > govern like SunJSSE Provider.
> > > >> > > >
> > > >> > > > We have exactly the same requirements to load certs and keys
> > > >> > > >
> > > >> > > > through
> > > >> > > >
> > > >> > > > spire
> > > >> > > >
> > > >> > > > agent. We used security.provider to do that exactly. I am not
> > > >> > > >
> > > >> > > > sure
> > > >> > > >
> > > >> > > > why
> > > >> > > >
> > > >> > > > you
> > > >> > > >
> > > >> > > > would be modifying any policies provided by default SunJSSE
> > > >> > > >
> > > >> > > > provider.
> > > >> > > >
> > > >> > > > Can
> > > >> > > >
> > > >> > > > you give me an example of having custom provider that will
> > > >> > > >
> > > >> > > > override an
> > > >> > > >
> > > >> > > > existing policy in SunJSSE provider.
> > > >> > > >
> > > >> > > > As pointed out earlier, this kip
> > > >> > > >
> > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > >> > > >
> KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> > > >> > > >
> > > >> > > > allows
> > > >> > > > you to load security.provider through config.
> > > >> > > > Take a look at the examples I gave before
> > > >> > > >
> > > >> > > >
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > > >> > > >
> > > >> > > > java
> > > >> > > >
> > > >> > > > It registers KeyManagerFactory and TrustManagerFactory and
> > > >> > > >
> > > >> > > > Keystore
> > > >> > > >
> > > >> > > > algorithm.
> > > >> > > > Implement your custom way of loading Keystore in here
> > > >> > > >
> > > >> > > >
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> > > >> > > >
> > > >> > > > java
> > > >> > > >
> > > >> > > > and Trust manager like here
> > > >> > > >
> > > >> > > >
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > >> > > > spiffe-security-provider/src/main/java/spiffe/api/provider/
> > > >> > > > SpiffeTrustManager.java <http://spiffetrustmanager.java/>
> > > <http://
> > > >> > > > spiffetrustmanager.java/>
> > > >> > > >
> > > >> > > > In your Kafka client you can set the security.provider to your
> > > >> > > >
> > > >> > > > custom
> > > >> > > >
> > > >> > > > implementation and with this fix
> > > >> > > > https://issues.apache.org/jira/browse/KAFKA-8191 you can set
> > > >> > > > keyManagerAlgorigthm and trustManagerAlgorithm configs.
> > > >> > > >
> > > >> > > > All of this is in your clients and broker side and do not need
> > > >> > > >
> > > >> > > > to
> > > >> > > >
> > > >> > > > touch
> > > >> > > >
> > > >> > > > any
> > > >> > > > policy changes at JVM level. You'll register the providers in
> > > >> > > >
> > > >> > > > the
> > > >> > > >
> > > >> > > > priority
> > > >> > > >
> > > >> > > > order and can still have SunJSSE provider and have your custom
> > > >> > > >
> > > >> > > > provider
> > > >> > > >
> > > >> > > > to
> > > >> > > >
> > > >> > > > implement the key and trust managers.
> > > >> > > >
> > > >> > > > The ask here is different than KIP-492. We don't have any need
> > > >> > > >
> > > >> > > > to
> > > >> > > >
> > > >> > > > modify/specify the algorithm parameter. Does that make sense?
> > > >> > > >
> > > >> > > > The ask in KIP is introducing new interfaces where the KIP's
> > > >> > > > goal/motivation can be achieved through the security.provider
> > > >> > > >
> > > >> > > > and
> > > >> > > >
> > > >> > > > we
> > > >> > > >
> > > >> > > > worked
> > > >> > > > on similar goal without touching any Keystore or Truststore
> > > >> > > >
> > > >> > > > interfaces.
> > > >> > > >
> > > >> > > > My advise is against changing or introducing new interfaces
> > > >> > > >
> > > >> > > > when
> > > >> > > >
> > > >> > > > it can
> > > >> > > >
> > > >> > > > work through security.provider.
> > > >> > > >
> > > >> > > > Thanks,
> > > >> > > > Harsha
> > > >> > > >
> > > >> > > > Thanks
> > > >> > > >
> > > >> > > > Maulin
> > > >> > > >
> > > >> > > > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
> > > >> > > >
> > > >> > > > kafka@harsha.io>
> > > >> > > >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > In your KIP you added security. provider as rejected
> > > >> > > >
> > > >> > > > alternative
> > > >> > > >
> > > >> > > > and
> > > >> > > >
> > > >> > > > specified "its not the correct way". Do you mind explaining
> > > >> > > >
> > > >> > > > why
> > > >> > > >
> > > >> > > > its
> > > >> > > >
> > > >> > > > not? I
> > > >> > > >
> > > >> > > > didn't find any evidence in Java docs to say so. Contrary to
> > > >> > > >
> > > >> > > > your
> > > >> > > >
> > > >> > > > statement
> > > >> > > >
> > > >> > > > it does say in the java docs
> > > >> > > > " However, please note that a provider can be used to
> > > >> > > >
> > > >> > > > implement
> > > >> > > >
> > > >> > > > any
> > > >> > > >
> > > >> > > > security service in Java that uses a pluggable architecture
> > > >> > > >
> > > >> > > > with
> > > >> > > >
> > > >> > > > a
> > > >> > > >
> > > >> > > > choice
> > > >> > > >
> > > >> > > > of implementations that fit underneath."
> > > >> > > >
> > > >> > > > Java Security Providers have been used by other projects to
> > > >> > > >
> > > >> > > > provide
> > > >> > > >
> > > >> > > > such
> > > >> > > >
> > > >> > > > integration . I am not sure if you looked into Spiffe
> > > >> > > >
> > > >> > > > project to
> > > >> > > >
> > > >> > > > efficiently distribute certificates but here is an example of
> > > >> > > >
> > > >> > > > Java
> > > >> > > >
> > > >> > > > provider
> > > >> > > >
> > > >> > > >
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > > >> > > >
> > > >> > > > java which
> > > >> > > > obtains certificates from local daemons.
> > > >> > > > These integrations are being used in Tomcat, Jetty etc.. We
> > > >> > > >
> > > >> > > > are
> > > >> > > >
> > > >> > > > also
> > > >> > > >
> > > >> > > > using
> > > >> > > >
> > > >> > > > Security provider to do the same in our Kafka clusters. So
> > > >> > > >
> > > >> > > > unless I
> > > >> > > >
> > > >> > > > see
> > > >> > > >
> > > >> > > > more evidence why security.provider doesn't work for you
> > > >> > > >
> > > >> > > > adding
> > > >> > > >
> > > >> > > > new
> > > >> > > >
> > > >> > > > interfaces while there exists more cleaner way of achieving
> > > >> > > >
> > > >> > > > the
> > > >> > > >
> > > >> > > > goals
> > > >> > > >
> > > >> > > > of
> > > >> > > >
> > > >> > > > this KIP is unnecessary and breaks the well known security
> > > >> > > >
> > > >> > > > interfaces
> > > >> > > >
> > > >> > > > provided by Java itself.
> > > >> > > >
> > > >> > > > Thanks,
> > > >> > > > Harsha
> > > >> > > >
> > > >> > > > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
> > > >> > > >
> > > >> > > > kafka@harsha.io
> > > >> > > >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > Hi Maulin,
> > > >> > > > Not sure if you looked at my previous replies. This
> > > >> > > >
> > > >> > > > changes
> > > >> > > >
> > > >> > > > are not required as there is already security Provider to do
> > > >> > > >
> > > >> > > > what you
> > > >> > > >
> > > >> > > > are
> > > >> > > >
> > > >> > > > proposing. This KIP
> > > >> > > >
> > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > >> > > >
> > > >> > > >
> KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> > > >> > > >
> > > >> > > > also
> > > >> > > >
> > > >> > > > addresses easy registration of such providers.
> > > >> > > >
> > > >> > > > Thanks,
> > > >> > > > Harsha
> > > >> > > >
> > > >> > > > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
> > > >> > > >
> > > >> > > > <maulin.vasavada@gmail.
> > > >> > > >
> > > >> > > > com> wrote:
> > > >> > > >
> > > >> > > > Bump! Can somebody please review this?
> > > >> > > >
> > > >> > > > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
> > > >> > > >
> > > >> > > > maulin.vasavada@gmail.com>
> > > >> > > >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > Bump! Can somebody please review this?
> > > >> > > >
> > > >> > > > --
> > > >> > > > Thanks,
> > > >> > > > M.Sai Sandeep
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Colin McCabe <cm...@apache.org>.
+1 for making SslEngineBuilder configurable. This would give implementers a lot more flexibility-- to use key distribution methods that were not files, for example.
best,
Colin
On Fri, Aug 30, 2019, at 02:03, Rajini Sivaram wrote:
> Just to make sure we are on the same page - KIP-383 was written before
> the
> code was refactored. The refactoring addressed some of the concerns of
> KIP-383. My suggestion was to make SslEngineBuilder configurable. The
> default implementation of this pluggable class would be
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/ssl/SslEngineBuilder.java.
> That should give an idea of the size of the configurable part that a
> custom
> class needs to implement. A large part of that is about loading
> keystore/truststore. I agree it has slightly more code than KIP-486
> proposes, but since it lets you customize creation of SSLEngine, it
> would
> address every possible scenario.
>
> Thoughts?
>
>
> On Fri, Aug 30, 2019 at 2:02 AM Maulin Vasavada <ma...@gmail.com>
> wrote:
>
> > I thought about it more. I feel that no matter how we refactor the code
> > (with or without KIP-383 integrated), ultimately the need of customizing
> > loading for keys and certs will still remain. Whenever that need arises we
> > might end up thinking about the solution suggested by our KIP-486. Hence
> > regardless of the other KIPs and configurations "if we do need to customize
> > loading of keys/certs, we will need the code changes suggested by this
> > KIP".
> >
> > Let me know what you guys think.
> >
> > Harsha, we are working on changing the interfaces for key/trust store
> > loaders with Certificate and PrivateKey objects. Will probably be able to
> > update it later today or tomorrow.
> >
> > Thanks
> > Maulin
> >
> >
> >
> >
> >
> > On Thu, Aug 29, 2019 at 2:30 PM Maulin Vasavada <maulin.vasavada@gmail.com
> > >
> > wrote:
> >
> > > On that, I actually looked at KIP-383 before briefly. However, that
> > > sounded like lot of changes suggested.
> > >
> > > One "key" thing we have to keep in mind is - IF we need lot of
> > > customization Kafka already allows you to use your SslProvider via
> > > ssl.providers or the changes done by KIP-492 and
> > > SSLContext.getInstance(protocol, provider) call allows us to return the
> > > SSLContext with "ALL" the details we would like to customize. Hence I am
> > > not sure that customization suggested by KIP-383 would be worth the
> > effort.
> > > We also have similar SSLContext customization outside of Kafka.
> > >
> > > Thanks
> > > Maulin
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Aug 29, 2019 at 12:47 PM Pellerin, Clement <
> > > Clement_Pellerin@ibi.com> wrote:
> > >
> > >> KIP-383 in its present form was vetoed because it was not possible to
> > add
> > >> validation of custom properties in a future KIP. The solution to that is
> > >> the first proposal I wrote for KIP-383 which made the whole SslFactory
> > >> pluggable. That first solution was also vetoed hence the deadlock.
> > >>
> > >> Replacing the whole factory was a much nicer solution. It was vetoed
> > >> because doing this almost invariably meant the replacement lost all the
> > >> complex validation code in the default SslFactory.
> > >>
> > >> My current idea is to extract the validation code into another public
> > API
> > >> that SslFactory would call. I did not look at the newly refactored code
> > and
> > >> I did not study how to do this yet. KIP-383 was not popular at the time
> > and
> > >> designing a new solution is a lot of work.
> > >>
> > >> Is there interest from 3 binding voters for something like this?
> > >>
> > >> -----Original Message-----
> > >> From: Rajini Sivaram [mailto:rajinisivaram@gmail.com]
> > >> Sent: Thursday, August 29, 2019 2:57 PM
> > >> To: dev
> > >> Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and
> > >> TrustStore
> > >>
> > >> Hi Maulin,
> > >>
> > >> In SSL scenarios, I imagine security providers introduced by KIP-492 are
> > >> likely to be most useful when you want to use third party providers. The
> > >> biggest advantage of the config from that KIP is that you don't need to
> > >> write much code to integrate existing security providers into Kafka
> > >> brokers
> > >> or clients. As I understand it, KIP-486 is a more convenient option for
> > >> the
> > >> specific problem of loading keystores/truststores differently. It can be
> > >> achieved in theory with KIP-492, but KIP-486 is a much simpler option
> > for
> > >> this case.
> > >>
> > >> My concern about KIP-486 is that it introduces yet another interface
> > into
> > >> our already complex security code, while only solving one particular use
> > >> case. Have you looked at
> > >>
> > >>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-383%3A++Pluggable+interface+for+SSL+Factory
> > >> ?
> > >> The goal was to make
> > >> org.apache.kafka.common.security.ssl.SslEngineBuilder pluggable.
> > >> The code has already been refactored by Colin after that KIP was
> > written,
> > >> making it easier to implement KIP-383. This should enable you to load
> > your
> > >> keystores and truststores differently. Using a pluggable
> > SslEngineBuilder
> > >> will also solve several other use cases at the same time. KIP-383 hasn't
> > >> been voted through yet, but perhaps you could take a look and we could
> > >> revive that instead if it solves your use case as well?
> > >>
> > >> Regards,
> > >>
> > >> Rajini
> > >>
> > >>
> > >> On Thu, Aug 29, 2019 at 6:42 PM Maulin Vasavada <
> > >> maulin.vasavada@gmail.com>
> > >> wrote:
> > >>
> > >> > Hi Harsha
> > >> >
> > >> > Thank you. Appreciate your time and support on this. Let me go back do
> > >> some
> > >> > more research and get back to you on the KeyStore interface part.
> > >> > Basically, if we return certs and keys in the interface then Kafka
> > code
> > >> > will have to build KeyStore object - which is also reasonable.
> > >> >
> > >> > Thanks
> > >> > Maulin
> > >> >
> > >> > On Thu, Aug 29, 2019 at 10:01 AM Harsha Chintalapani <kafka@harsha.io
> > >
> > >> > wrote:
> > >> >
> > >> > > Hi Maulin,
> > >> > > Use cases are clear now. I am +1 for moving
> > >> forward
> > >> > > with the discussions on having such configurable option for users.
> > But
> > >> > the
> > >> > > interfaces is proposed doesn't look right to me. We are still
> > talking
> > >> > about
> > >> > > keystore interfaces. Given keystore's are used as filebased way of
> > >> > > transporting certificates I am not sure it will help the rest of the
> > >> > > user-base.
> > >> > > In short, I am +1 on the KIP's motivation and only
> > >> have
> > >> > > questions around returning keystores instead of returning certs,
> > >> private
> > >> > > keys etc. . If others in the community are ok with such interface we
> > >> can
> > >> > > move forward.
> > >> > >
> > >> > > Thanks,
> > >> > > Harsha
> > >> > >
> > >> > >
> > >> > > On Wed, Aug 28, 2019 at 1:51 PM, Maulin Vasavada <
> > >> > > maulin.vasavada@gmail.com>
> > >> > > wrote:
> > >> > >
> > >> > > > Hi Harsha
> > >> > > >
> > >> > > > As we synced-up offline on this topic, we hope you don't have any
> > >> more
> > >> > > > clarifications that you are seeking. If that is the case, can you
> > >> > please
> > >> > > > help us move this forward and discuss what changes you would
> > expect
> > >> on
> > >> > > the
> > >> > > > KIP design in order to make it valuable contribution?
> > >> > > >
> > >> > > > Just FYI - we verified our primary design change with the author
> > of
> > >> > Sun's
> > >> > > > X509 Trustmanager implementation and the outcome is that what we
> > are
> > >> > > > proposing makes sense at the heart of it - "Instead of writing
> > >> > > TrustManager
> > >> > > > just plugin the Trust store". We are open to discuss additional
> > >> changes
> > >> > > > that you/anybody else would like to see on the functionality
> > >> however.
> > >> > > >
> > >> > > > Thanks
> > >> > > > Maulin
> > >> > > >
> > >> > > > On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada <
> > >> > > maulin.vasavada@gmail.com>
> > >> > > > wrote:
> > >> > > >
> > >> > > > Hi Harsha
> > >> > > >
> > >> > > > Any response on my question? I feel this KIP is worth
> > accommodating.
> > >> > Your
> > >> > > > help is much appreciated.
> > >> > > >
> > >> > > > Thanks
> > >> > > > Maulin
> > >> > > >
> > >> > > > On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada <
> > >> > maulin.vasavada@gmail.
> > >> > > > com> wrote:
> > >> > > >
> > >> > > > Hi Harsha
> > >> > > >
> > >> > > > I've examined the SPIFFE provider more and have one question -
> > >> > > >
> > >> > > > If SPIFFE didn't have a need to do checkSpiffeId() call at the
> > below
> > >> > > > location, would you really still write the Provider? *OR* Would
> > you
> > >> > just
> > >> > > > use TrustManagerFactory.init(KeyStore) signature to pass the
> > >> KeyStore
> > >> > > from
> > >> > > > set of certs returned by spiffeIdManager. getTrustedCerts()?
> > >> > > >
> > >> > > >
> > >> >
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > >> > > > provider/CertificateUtils.java#L100
> > >> > > >
> > >> > > > /**
> > >> > > >
> > >> > > > * Validates that the SPIFFE ID is present and matches the SPIFFE
> > ID
> > >> > > > configured in
> > >> > > > * the java.security property ssl.spiffe.accept
> > >> > > > *
> > >> > > > * If the authorized spiffe ids list is empty any spiffe id is
> > >> > authorized
> > >> > > > *
> > >> > > > * @param chain an array of X509Certificate that contains the
> > Peer's
> > >> > SVID
> > >> > > > to be validated
> > >> > > > * @throws CertificateException when either the certificates
> > doesn't
> > >> > have
> > >> > > a
> > >> > > > SPIFFE ID or the SPIFFE ID is not authorized
> > >> > > > */
> > >> > > > static void checkSpiffeId(X509Certificate[] chain) throws
> > >> > > > CertificateException {
> > >> > > >
> > >> > > > Thanks
> > >> > > > Maulin
> > >> > > >
> > >> > > > On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani <
> > >> kafka@harsha.io>
> > >> > > > wrote:
> > >> > > >
> > >> > > > Maulin,
> > >> > > > The code parts you are pointing are specific for Spiffe and if
> > >> > > > you are talking about validate method which uses PKIX check like
> > any
> > >> > > other
> > >> > > > provider does.
> > >> > > > If you want to default to SunJSSE everywhere you can do so by
> > >> > delegating
> > >> > > > the calls in these methods to SunJSSE provider.
> > >> > > >
> > >> > > > TrustManagerFactory tmf = TrustManagerFactory
> > >> > > > .getInstance(TrustManagerFactory.getDefaultAlgorithm());and use
> > >> > > > tmf.chekServerTrusted()
> > >> > > > or use
> > >> > > > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> > >> > > > TrustManagerFactory.html#getInstance(java.lang.String)if you want
> > a
> > >> > > > specific provider.
> > >> > > >
> > >> > > > -Harsha
> > >> > > >
> > >> > > > On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada <
> > >> > maulin.vasavada@gmail.
> > >> > > > com>
> > >> > > > wrote:
> > >> > > >
> > >> > > > Okay, so I take that you guys agree that I have to write a
> > 'custom'
> > >> > > > algorithm and a provider to make it work , correct?
> > >> > > >
> > >> > > > Now, for Harsha's comment "Here the 'Custom' Algorithm is not an
> > >> > > > implementation per say , ..." , I diagree. You can refer to
> > https://
> > >> > > >
> > >> > > >
> > >> >
> > >>
> > github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/
> > >> > > >
> > >> > > > SpiffeTrustManager.java#L91 <http://spiffetrustmanager.java/#L91>
> > >> and
> > >> > > >
> > >> > > >
> > >> >
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > >> > > >
> > >> > > > provider/CertificateUtils.java#L100
> > >> > > >
> > >> > > > "that code" is the customization you have for the custom way to
> > >> check
> > >> > > > something on top of regular checks. That method is NOT doing
> > custom
> > >> > > > truststore loading. It is validating/verifying something in the
> > >> > > >
> > >> > > > "custom"
> > >> > > >
> > >> > > > way with spiffeId.
> > >> > > > I bet that without that you won't have a need of the custom
> > >> algorithm
> > >> > > >
> > >> > > > in
> > >> > > >
> > >> > > > the first place.
> > >> > > >
> > >> > > > Let me know if you agree to this.
> > >> > > >
> > >> > > > Thanks
> > >> > > > Maulin
> > >> > > >
> > >> > > > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <mp...@gmail.com>
> > >> > > >
> > >> > > > wrote:
> > >> > > >
> > >> > > > Hi Maulin, thanks for the discussion. As Harsha pointed out, to
> > use
> > >> the
> > >> > > > KIP492, you need to create a new provider, register a *new* custom
> > >> > > > algorithm for your keymanager and trustmanager factory
> > >> implementations.
> > >> > > > After this, the kafka server configuration can be done as given
> > >> below
> > >> > > >
> > >> > > > # Register the provider class with custom algorithm, say CUSTOM
> > >> > > >
> > >> > > > security.
> > >> > > >
> > >> > > > provider.classes=com.company.security.CustomProvider
> > >> > > > <http://provider.classes=com.company.security.customprovider/>
> > >> > > > <http://security.provider.classes
> > >> > > >
> > >> > > > =com.company.security.customprovider/>
> > >> > > >
> > >> > > > # Register the keymanager and trustmanager algorithms
> > >> > > > # These algorithms indicate that the Keymanager and Trustmanagers
> > >> > > > registered under the algorithm "CUSTOM" needs to be used
> > >> > > > ssl.trustmanager.algorithm=CUSTOM
> > >> > > > ssl.keymanager.algorithm=CUSTOM
> > >> > > >
> > >> > > > And the customprovider looks like this...
> > >> > > >
> > >> > > > public class CustomProvider extends Provider {
> > >> > > > public CustomProvider() {
> > >> > > > super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and
> > TrustStore");
> > >> > > > super.put("KeyManagerFactory.CUSTOM", "customKeyManagerFactory");
> > >> > > > super.put("TrustManagerFactory.CUSTOM",
> > >> > > > "customTrustManagerFactory");
> > >> > > > }
> > >> > > > }
> > >> > > >
> > >> > > > The PR for this is in review and can be found here:
> > >> > > >
> > >> > > > https://github.com/
> > >> > > >
> > >> > > > apache/kafka/pull/7090
> > >> > > > This PR includes the fixed insertProviderAt function call.
> > >> > > >
> > >> > > > On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani <
> > >> kafka@harsha.io>
> > >> > > > wrote:
> > >> > > >
> > >> > > > Answers are in-line
> > >> > > >
> > >> > > > On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada <
> > >> > > >
> > >> > > > maulin.vasavada@gmail.
> > >> > > >
> > >> > > > com
> > >> > > >
> > >> > > > wrote:
> > >> > > >
> > >> > > > Hi Colin
> > >> > > >
> > >> > > > When I refer to "standard" or "custom" algorithms I am following
> > >> Java
> > >> > > > security Provider Terminology. You can refer to
> > >> > > > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> > >> > > > StandardNames.html#TrustManagerFactory link I provided earlier in
> > >> the
> > >> > > > emails. It says PKIX is the default Algorithm for
> > >> TrustManagerFactory.
> > >> > > >
> > >> > > > 1. For SPIFFE, I am not sure why you are saying 'it does not
> > >> implement
> > >> > > > custom algorithms' because the following file clearly indicates
> > >> that it
> > >> > > > does use custom algorithm-
> > >> > > >
> > >> > > >
> > >> >
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > >> > > >
> > >> > > > provider/SpiffeProvider.java#L17
> > >> > > >
> > >> > > > Algorithm value:
> > >> > > >
> > >> > > >
> > >> >
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > >> > > >
> > >> > > > provider/SpiffeProviderConstants.java#L6
> > >> > > >
> > >> > > > @Harsha do you want to chime in since you use that provider?
> > >> > > >
> > >> > > > Here the "Custom" Algorithm is not an implementation per say ,
> > >> rather
> > >> > > >
> > >> > > > used
> > >> > > >
> > >> > > > to invoke the custom trust store factory and key manager factory.
> > >> You
> > >> > > >
> > >> > > > are
> > >> > > >
> > >> > > > not moving away from "standard" alogrithms that are available.
> > >> > > >
> > >> > > >
> > >> >
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > >> > > >
> > >> > > > provider/SpiffeTrustManager.java
> > >> > > >
> > >> > > > As you can see it delegates all the calls of verification of
> > >> > > >
> > >> > > > certificate
> > >> > > >
> > >> > > > to
> > >> > > >
> > >> > > > the default implementation available.
> > >> > > > So in our implementation we still use PKIX to verify the
> > certificate
> > >> > > > chain. So you are not losing anything here and Spiffe is not
> > >> > > >
> > >> > > > reimplementing
> > >> > > >
> > >> > > > the verification process.
> > >> > > >
> > >> > > > 2. I already mentioned in my 3rd point, in my previous post, why
> > >> using
> > >> > > >
> > >> > > > ssl.provider does NOT work. I updated KIP-486 in "rejected
> > >> > > >
> > >> > > > alternatives"
> > >> > > >
> > >> > > > also why ssl.provider does not work.
> > >> > > >
> > >> > > > As mentioned before , provider is the right way to go. I am still
> > >> not
> > >> > > > understanding the gap is.
> > >> > > > If I understand correctly your argument is , provider is going to
> > >> ask
> > >> > > >
> > >> > > > to
> > >> > > >
> > >> > > > implement a custom algorithm.
> > >> > > > My answer to that is , no you are not re-implementing the
> > algorithm.
> > >> > > >
> > >> > > > Please
> > >> > > >
> > >> > > > check the above link , TrustManager implementation it delegates
> > the
> > >> > > >
> > >> > > > calls
> > >> > > >
> > >> > > > back. There is no need to implement your own here.
> > >> > > >
> > >> > > > 3. Security.insertProviderAt() comments were based on assumption
> > if
> > >> > > >
> > >> > > > KIP-492
> > >> > > >
> > >> > > > changes are done and we use that mechanism to configure providers
> > >> > > >
> > >> > > > instead
> > >> > > >
> > >> > > > of ssl.provider configuration.
> > >> > > >
> > >> > > > KIP-492 has patch available and going through review.
> > >> > > >
> > >> > > > Can you read my all the points, I mentioned in my previous post,
> > >> very
> > >> > > >
> > >> > > > carefully? I am covering all the aspects in explaining. I am open
> > to
> > >> > > >
> > >> > > > still
> > >> > > >
> > >> > > > discuss more to clarify any doubts.
> > >> > > >
> > >> > > > "3. If we just use existing ssl.provider kafka configuration then
> > >> our
> > >> > > > provider will be used in SSLContext.getInstance(protocol,
> > provider)
> > >> > > >
> > >> > > > call
> > >> > > >
> > >> > > > in
> > >> > > >
> > >> > > > SslFactory.java <http://sslfactory.java/> <
> > http://sslfactory.java/>
> > >> <
> > >> > > > http://sslfactory.java/>
> > >> > > >
> > >> > > > and
> > >> > > >
> > >> > > > if our provider does not have
> > >> > > > implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks
> > (we
> > >> > > >
> > >> > > > tested
> > >> > > >
> > >> > > > it). Example: In MyProvider sample above you see that I didn't add
> > >> > > > SSLContext.TLSv1 as
> > >> > > > "Service+Algorithm" and that didn't work for me. In SPIFFE
> > provider
> > >> you
> > >> > > > don't have this challenge since you are planning to bypass
> > >> ssl.provider
> > >> > > >
> > >> > > > as
> > >> > > >
> > >> > > > you mention in the KIP-492."
> > >> > > >
> > >> > > > Yes here you need to pass the protocol that your
> > >> > > >
> > >> > > > KeyManager/TrustManager
> > >> > > >
> > >> > > > registered with and in no way its deviating from TLS RFC spec.
> > >> > > >
> > >> > > >
> > >> >
> > https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/
> > >> > > >
> > >> > > > javax/net/ssl/SSLContext.java#L134
> > >> > > >
> > >> > > > My suggestion here is for you to implement a simple Security
> > >> Provider
> > >> > > >
> > >> > > > as
> > >> > > >
> > >> > > > you did before and register a Algorithm. You can use the existing
> > >> > > > implementation in SpiffeProvider and plug in changes where you
> > need
> > >> to
> > >> > > > retrieve the certificates from by making RPC call.
> > >> > > >
> > >> > > > Run an end-to-end test with Kafka broker coming up with your
> > >> provider
> > >> > > > making calls to RPC call. You do need to pass the "custom
> > algorithm"
> > >> > > >
> > >> > > > that
> > >> > > >
> > >> > > > you registered in your key manager to invoke the provider. I think
> > >> your
> > >> > > > concern is this approach is replacing the existing known
> > >> ciphersuites
> > >> > > >
> > >> > > > and
> > >> > > >
> > >> > > > certificate validation provided by java. Which its not.
> > >> > > >
> > >> > > > Now test the TLS connection you can do so via openssl -s_client
> > >> options
> > >> > > >
> > >> > > > to
> > >> > > >
> > >> > > > test if everything else is working.
> > >> > > >
> > >> > > > I am happy to share configs that we used if you are interested.
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Harsha
> > >> > > >
> > >> > > > Thanks
> > >> > > > Maulin
> > >> > > >
> > >> > > > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <cm...@apache.org>
> > >> > > >
> > >> > > > wrote:
> > >> > > >
> > >> > > > Hi Maulin,
> > >> > > >
> > >> > > > A lot of JSSE providers don't implement custom algorithms. Spire
> > is
> > >> a
> > >> > > >
> > >> > > > good
> > >> > > >
> > >> > > > example of a JSSE provider that doesn't, and yet is still useful
> > to
> > >> > > >
> > >> > > > many
> > >> > > >
> > >> > > > people. Your JSSE provider can work fine even if it doesn't
> > >> implement a
> > >> > > > custom algorithm.
> > >> > > >
> > >> > > > Maybe I'm missing something, but I don't understand the discussion
> > >> of
> > >> > > > Security.insertProviderAt() that you included. SslEngineBuilder
> > >> doesn't
> > >> > > >
> > >> > > > use
> > >> > > >
> > >> > > > that API to get the security provider. Instead, it calls
> > >> > > > "SSLContext.getInstance(protocol, provider)", where provider is
> > the
> > >> > > >
> > >> > > > name
> > >> > > >
> > >> > > > of the provider.
> > >> > > >
> > >> > > > best,
> > >> > > > Colin
> > >> > > >
> > >> > > > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
> > >> > > >
> > >> > > > On top of everything above I feel strongly to add the 4th point
> > >> which
> > >> > > >
> > >> > > > is
> > >> > > >
> > >> > > > based on Java APIs for TrustManagerFactory.init(KeyStore) (
> > >> > > >
> > >> > > > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> > >> > > > TrustManagerFactory.html#init(java.security.KeyStore
> > >> > > > <http://java.security.keystore/>
> > >> > > > <http://java.security.keystore/>)
> > >> > > > )
> > >> > > >
> > >> > > > and KeyManagerFactory.init(KeyStore, char[]) (
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory
> > >> > > > .
> > >> > > >
> > >> > > > html#init(java.security.KeyStore <http://java.security.keystore/>
> > >> > > <http://
> > >> > > > java.security.keystore/
> > >> > > > ,%20char[])
> > >> > > >
> > >> > > > ).
> > >> > > >
> > >> > > > 4. The above APIs are intended to support providing "trust/key
> > >> > > >
> > >> > > > material"
> > >> > > >
> > >> > > > from the user without having to write their own
> > >> > > >
> > >> > > > TrustManager/KeyManagers.
> > >> > > >
> > >> > > > To quote from the TrustManagerFactory.init()'s documentation
> > >> > > >
> > >> > > > "Initializes
> > >> > > >
> > >> > > > this factory with a source of certificate authorities and related
> > >> trust
> > >> > > > material."
> > >> > > > To quote from the KeyManagerFactory.init()'s documentation
> > >> "Initializes
> > >> > > > this factory with a source of key material."
> > >> > > >
> > >> > > > Based on this it is clear that there is a flexibility provided by
> > >> Java
> > >> > > >
> > >> > > > to
> > >> > > >
> > >> > > > to enable developers to provide the required trust/key material
> > >> loaded
> > >> > > >
> > >> > > > from
> > >> > > >
> > >> > > > "anywhere" without requiring them to write custom provider OR
> > >> trust/key
> > >> > > > managers. This same flexibility is reflected in Kafka code also
> > >> where
> > >> > > >
> > >> > > > it
> > >> > > >
> > >> > > > loads the trust/keys from a local file and doesn't require
> > writing a
> > >> > > > Provider necessarily. If we do NOT have a custom algorithm, it
> > makes
> > >> > > >
> > >> > > > less
> > >> > > >
> > >> > > > sense to write a Provider.
> > >> > > >
> > >> > > > Thanks
> > >> > > > Maulin
> > >> > > >
> > >> > > > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
> > >> > > >
> > >> > > > maulin.vasavada@gmail.com>
> > >> > > >
> > >> > > > wrote:
> > >> > > >
> > >> > > > Hi Harsha/Colin
> > >> > > >
> > >> > > > I did the sample with a custom Provider for TrustStoreManager and
> > >> tried
> > >> > > > using ssl.provider Kafka config AND the way KIP-492 is suggesting
> > >> (by
> > >> > > > adding Provider programmatically instead of relying on
> > >> > > >
> > >> > > > ssl.provider+java.
> > >> > > >
> > >> > > > security. The below sample is followed by my detailed findings.
> > I'll
> > >> > > > appreciate if you can go through it carefully and see
> > >> > > >
> > >> > > > if you
> > >> > > >
> > >> > > > see my point.
> > >> > > >
> > >> > > > package providertest;
> > >> > > >
> > >> > > > import java.security.Provider <http://java.security.provider/>
> > >> > <http://
> > >> > > > java.security.provider/>
> > >> > > >
> > >> > > > <http://
> > >> > > >
> > >> > > > java.security.provider/>;
> > >> > > >
> > >> > > > public class MyProvider extends Provider {
> > >> > > >
> > >> > > > private static final String name = "MyProvider"; private static
> > >> double
> > >> > > > version = 1.0d;
> > >> > > > private static String info = "Maulin's SSL Provider v"+version;
> > >> > > >
> > >> > > > public MyProvider() {
> > >> > > > super(name, version, info);
> > >> > > > this.put("TrustManagerFactory.PKIX",
> > >> > > >
> > >> > > > "providertest.MyTrustManagerFactory");
> > >> > > >
> > >> > > > }
> > >> > > > }
> > >> > > >
> > >> > > > *Details:*
> > >> > > >
> > >> > > > KIP-492 documents that it will use Security.addProvider() assuming
> > >> it
> > >> > > >
> > >> > > > will
> > >> > > >
> > >> > > > add it as position '0' which is not a correct assumption. The
> > >> > > > addProvider()'s documentation says it will add it to the last
> > >> available
> > >> > > > position. You may want to correct that to say
> > >> > > > Security.insertProviderAt(provider, 1).
> > >> > > >
> > >> > > > Now coming back to our specific discussion,
> > >> > > >
> > >> > > > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you
> > add
> > >> > > >
> > >> > > > that
> > >> > > >
> > >> > > > provider in the provider list via Security.addProvider() the
> > >> position
> > >> > > >
> > >> > > > where
> > >> > > >
> > >> > > > it gets added doesn't matter (even if you don't end up adding it
> > as
> > >> > > >
> > >> > > > first
> > >> > > >
> > >> > > > entry) since that is the ONLY provider for SPIFFE specific
> > algorithm
> > >> > > >
> > >> > > > you
> > >> > > >
> > >> > > > might have.
> > >> > > >
> > >> > > > We do *not* have custom algorithm for Key/Trust StoreMangers.
> > Which
> > >> > > >
> > >> > > > means
> > >> > > >
> > >> > > > we have to use X509, PKIX etc "Standard Algorithms" ((
> > >> > > >
> > >> > > > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> > >> > > > StandardNames.html
> > >> > > > ))
> > >> > > >
> > >> > > > in our provider to override the TrustStoreManager (see my sample
> > >> code)
> > >> > > >
> > >> > > > and
> > >> > > >
> > >> > > > KeyStoreManger and KeyManager. This creates another challenge
> > >> > > >
> > >> > > > mentioned in
> > >> > > >
> > >> > > > the below point.
> > >> > > >
> > >> > > > 2. In order to make our Provider for loading custom TrustStore
> > >> work, we
> > >> > > > have to add the provider as 'first' in the list since there are
> > >> others
> > >> > > >
> > >> > > > with
> > >> > > >
> > >> > > > the same algorithm.
> > >> > > >
> > >> > > > However, the programatic way of adding provider
> > >> > > > (Security.insertProviderAt()) is *not* deterministic for ordering
> > >> since
> > >> > > > different code can call the method for a different provider and
> > >> > > >
> > >> > > > depending
> > >> > > >
> > >> > > > upon the order of the call our provider can be first or pushed
> > down
> > >> the
> > >> > > > list. This can happen very well in any client application using
> > >> Kafka.
> > >> > > >
> > >> > > > This
> > >> > > >
> > >> > > > is specially problematic for a case when you want to guarantee
> > order
> > >> > > >
> > >> > > > for a
> > >> > > >
> > >> > > > Provider having "Standard Algorithms".
> > >> > > >
> > >> > > > If we add our provider in java.security file that definitely
> > >> guarantees
> > >> > > > the order(unless somebody calls removeProvider() which is less
> > >> > > >
> > >> > > > likely). But
> > >> > > >
> > >> > > > if we add our provider in java.security file it will defeat the
> > >> > > >
> > >> > > > purpose of
> > >> > > >
> > >> > > > the KIP-492.
> > >> > > >
> > >> > > > In the gist - Apache Kafka must not rely on "un-deterministic"
> > >> method
> > >> > > >
> > >> > > > to
> > >> > > >
> > >> > > > rely on Provider ordering.
> > >> > > >
> > >> > > > 3. If we just use existing ssl.provider kafka configuration then
> > our
> > >> > > > provider will be used in SSLContext.getInstance(protocol,
> > provider)
> > >> > > >
> > >> > > > call in
> > >> > > >
> > >> > > > SslFactory.java <http://sslfactory.java/> <
> > http://sslfactory.java/>
> > >> <
> > >> > > > http://sslfactory.java/>
> > >> > > >
> > >> > > > and
> > >> > > >
> > >> > > > if our provider does not have implementation for
> > >> > > > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks
> > >> > > >
> > >> > > > (we
> > >> > > >
> > >> > > > tested it). Example:
> > >> > > >
> > >> > > > In
> > >> > > >
> > >> > > > MyProvider sample above you see that I didn't add SSLContext.TLSv1
> > >> as
> > >> > > > "Service+Algorithm" and that didn't work for me. In SPIFFE
> > provider
> > >> you
> > >> > > > don't have this challenge since you are planning to bypass
> > >> > > >
> > >> > > > ssl.provider as
> > >> > > >
> > >> > > > you mention in the KIP-492.
> > >> > > >
> > >> > > > *Overall summary,*
> > >> > > >
> > >> > > > 1. Any provider based mechanisms- a) existing ssl.provider and
> > >> > > >
> > >> > > > b)KIP-492,
> > >> > > >
> > >> > > > for loading key/trust store using "Standard Algorithms" do not
> > work
> > >> > > >
> > >> > > > 2. Approach suggested in our KIP-486 works without any issue and
> > it
> > >> is
> > >> > > > *not* our context specific solve
> > >> > > >
> > >> > > > 3. Based on above we feel KIP-492 and KIP-486 are complimentary
> > >> changes
> > >> > > > and not contradicting or redundent.
> > >> > > >
> > >> > > > If you want we can do a joint session somehow to walk through the
> > >> > > >
> > >> > > > sample I
> > >> > > >
> > >> > > > have and various experiments I did. I would encourage you to do
> > >> similar
> > >> > > > exercise by writing a Provider for "Standard Algorithm" for
> > >> > > > TrustStoreManager (like our needs) and see what you find since
> > only
> > >> > > >
> > >> > > > writing
> > >> > > >
> > >> > > > samples can bring out the complexity/challenges we face.
> > >> > > >
> > >> > > > Thanks
> > >> > > > Maulin
> > >> > > >
> > >> > > > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <
> > >> > > >
> > >> > > > maulin.vasavada@gmail.
> > >> > > >
> > >> > > > com> wrote:
> > >> > > >
> > >> > > > Just to update - still working on it. Get to work only on and off
> > on
> > >> > > >
> > >> > > > it :(
> > >> > > >
> > >> > > > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
> > >> > > >
> > >> > > > maulin.vasavada@gmail.com>
> > >> > > >
> > >> > > > wrote:
> > >> > > >
> > >> > > > Hi Harsha
> > >> > > >
> > >> > > > Let me try to write samples and will let you know.
> > >> > > >
> > >> > > > Thanks
> > >> > > > Maulin
> > >> > > >
> > >> > > > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <ha...@gmail.com>
> > >> > > >
> > >> > > > wrote:
> > >> > > >
> > >> > > > Hi Maulin,
> > >> > > > With java security providers can be as custom you would
> > >> > > >
> > >> > > > like
> > >> > > >
> > >> > > > it to
> > >> > > > be. If you only want to to implement a custom way of loading the
> > >> > > >
> > >> > > > keystore
> > >> > > >
> > >> > > > and truststore and not implement any protocol/encryption handling
> > >> you
> > >> > > >
> > >> > > > can
> > >> > > >
> > >> > > > leave them empty and no need to implement. Have you looked into
> > the
> > >> > > >
> > >> > > > links I
> > >> > > >
> > >> > > > pasted before?
> > >> > > >
> > >> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> > >> > > >
> > >> > > > java
> > >> > > >
> > >> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >> > > > spiffe-security-provider/src/main/java/spiffe/api/provider/
> > >> > > > SpiffeTrustManager.java <http://spiffetrustmanager.java/>
> > <http://
> > >> > > > spiffetrustmanager.java/>
> > >> > > >
> > >> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > >> > > >
> > >> > > > java
> > >> > > >
> > >> > > > Can you please tell me which methods are too complex in above to
> > >> > > >
> > >> > > > implement
> > >> > > >
> > >> > > > or unnecessary? You are changing anything in SSL/TLS
> > implementations
> > >> > > > provided by
> > >> > > >
> > >> > > > All of the implementations delegating the checks to the default
> > >> > > > implementation anyway.
> > >> > > > Spire agent is an example, its nothing but a GRPC server listening
> > >> > > >
> > >> > > > on a
> > >> > > >
> > >> > > > unix domain socket . Above code is making a RPC call to the local
> > >> > > >
> > >> > > > daemon
> > >> > > >
> > >> > > > to
> > >> > > > get the certificate and keys. The mechanics are pretty much same
> > as
> > >> > > >
> > >> > > > what
> > >> > > >
> > >> > > > you are asking for.
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Harsha
> > >> > > >
> > >> > > > On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
> > >> > > >
> > >> > > > maulin.vasavada@gmail.com>
> > >> > > >
> > >> > > > wrote:
> > >> > > >
> > >> > > > Imagine a scenario like - We know we have a custom KMS and as a
> > >> > > >
> > >> > > > Kafka
> > >> > > >
> > >> > > > owner
> > >> > > >
> > >> > > > we want to comply to using that KMS source to load keys/certs. As
> > >> > > >
> > >> > > > a
> > >> > > >
> > >> > > > Kafka
> > >> > > >
> > >> > > > owner we know how to integrate with KMS but doesn't necessarily
> > >> > > >
> > >> > > > have
> > >> > > >
> > >> > > > to
> > >> > > >
> > >> > > > know anything about cipher suites, algorithms, and SSL/TLS
> > >> > > >
> > >> > > > implementation.
> > >> > > >
> > >> > > > Going the Provider way requires to know lot more than we should,
> > >> > > >
> > >> > > > isn't it?
> > >> > > >
> > >> > > > Not that we would have concern/shy-away knowing those details -
> > >> > > >
> > >> > > > but
> > >> > > >
> > >> > > > if we
> > >> > > >
> > >> > > > don't have to - why should we?
> > >> > > >
> > >> > > > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
> > >> > > >
> > >> > > > maulin.vasavada@gmail.com>
> > >> > > >
> > >> > > > wrote:
> > >> > > >
> > >> > > > Hi Harsha
> > >> > > >
> > >> > > > We don't have spire (or similar) agents and we do not have
> > >> > > >
> > >> > > > keys/certs
> > >> > > >
> > >> > > > locally on any brokers. To elaborate more on my previous email,
> > >> > > >
> > >> > > > I agree that Java security Providers are used in much broader
> > >> > > >
> > >> > > > sense
> > >> > > >
> > >> > > > - to
> > >> > > >
> > >> > > > have a particular implementation of an algorithm, use specific
> > >> > > >
> > >> > > > cipher
> > >> > > >
> > >> > > > suites for SSL , OR in our current team's case have a
> > >> > > >
> > >> > > > particular
> > >> > > >
> > >> > > > way to
> > >> > > >
> > >> > > > leverage pre-generated SSL sessions. However, the scope of our
> > >> > > >
> > >> > > > KIP
> > >> > > >
> > >> > > > (486)
> > >> > > >
> > >> > > > is
> > >> > > >
> > >> > > > much restricted than that. We merely intend to provide a custom
> > >> > > > keystore/truststore for our SSL connections and not really worry
> > >> > > >
> > >> > > > about
> > >> > > >
> > >> > > > underlying specific SSL/TLS implementation. This simplifies it
> > >> > > >
> > >> > > > a
> > >> > > >
> > >> > > > lot for
> > >> > > >
> > >> > > > us to keep the concerns separate and clear.
> > >> > > >
> > >> > > > I feel our approach is more complimentary such that it allows
> > >> > > >
> > >> > > > for
> > >> > > >
> > >> > > > using
> > >> > > >
> > >> > > > keystores of choice while retaining the flexibility to use any
> > >> > > > underlying/available Provider for actually making the SSL call.
> > >> > > >
> > >> > > > We agree with KIP-492's approach based on Providers (and Java's
> > >> > > > recommendation), but also strongly believe that our approach can
> > >> > > >
> > >> > > > compliment
> > >> > > >
> > >> > > > it very effectively for reasons explained above.
> > >> > > >
> > >> > > > Thanks
> > >> > > > Maulin
> > >> > > >
> > >> > > > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <
> > >> > > >
> > >> > > > kafka@harsha.io
> > >> > > >
> > >> > > > wrote:
> > >> > > >
> > >> > > > Hi Maulin,
> > >> > > >
> > >> > > > On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
> > >> > > >
> > >> > > > maulin.vasavada@gmail.
> > >> > > >
> > >> > > > com>
> > >> > > > wrote:
> > >> > > >
> > >> > > > Hi Harsha
> > >> > > >
> > >> > > > The reason we rejected the SslProvider route is that - we
> > >> > > >
> > >> > > > only
> > >> > > >
> > >> > > > needed
> > >> > > >
> > >> > > > a
> > >> > > >
> > >> > > > custom way to load keys/certs. Not touch any policy that
> > >> > > >
> > >> > > > existing
> > >> > > >
> > >> > > > Providers
> > >> > > >
> > >> > > > govern like SunJSSE Provider.
> > >> > > >
> > >> > > > We have exactly the same requirements to load certs and keys
> > >> > > >
> > >> > > > through
> > >> > > >
> > >> > > > spire
> > >> > > >
> > >> > > > agent. We used security.provider to do that exactly. I am not
> > >> > > >
> > >> > > > sure
> > >> > > >
> > >> > > > why
> > >> > > >
> > >> > > > you
> > >> > > >
> > >> > > > would be modifying any policies provided by default SunJSSE
> > >> > > >
> > >> > > > provider.
> > >> > > >
> > >> > > > Can
> > >> > > >
> > >> > > > you give me an example of having custom provider that will
> > >> > > >
> > >> > > > override an
> > >> > > >
> > >> > > > existing policy in SunJSSE provider.
> > >> > > >
> > >> > > > As pointed out earlier, this kip
> > >> > > >
> > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > >> > > > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> > >> > > >
> > >> > > > allows
> > >> > > > you to load security.provider through config.
> > >> > > > Take a look at the examples I gave before
> > >> > > >
> > >> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > >> > > >
> > >> > > > java
> > >> > > >
> > >> > > > It registers KeyManagerFactory and TrustManagerFactory and
> > >> > > >
> > >> > > > Keystore
> > >> > > >
> > >> > > > algorithm.
> > >> > > > Implement your custom way of loading Keystore in here
> > >> > > >
> > >> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> > >> > > >
> > >> > > > java
> > >> > > >
> > >> > > > and Trust manager like here
> > >> > > >
> > >> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >> > > > spiffe-security-provider/src/main/java/spiffe/api/provider/
> > >> > > > SpiffeTrustManager.java <http://spiffetrustmanager.java/>
> > <http://
> > >> > > > spiffetrustmanager.java/>
> > >> > > >
> > >> > > > In your Kafka client you can set the security.provider to your
> > >> > > >
> > >> > > > custom
> > >> > > >
> > >> > > > implementation and with this fix
> > >> > > > https://issues.apache.org/jira/browse/KAFKA-8191 you can set
> > >> > > > keyManagerAlgorigthm and trustManagerAlgorithm configs.
> > >> > > >
> > >> > > > All of this is in your clients and broker side and do not need
> > >> > > >
> > >> > > > to
> > >> > > >
> > >> > > > touch
> > >> > > >
> > >> > > > any
> > >> > > > policy changes at JVM level. You'll register the providers in
> > >> > > >
> > >> > > > the
> > >> > > >
> > >> > > > priority
> > >> > > >
> > >> > > > order and can still have SunJSSE provider and have your custom
> > >> > > >
> > >> > > > provider
> > >> > > >
> > >> > > > to
> > >> > > >
> > >> > > > implement the key and trust managers.
> > >> > > >
> > >> > > > The ask here is different than KIP-492. We don't have any need
> > >> > > >
> > >> > > > to
> > >> > > >
> > >> > > > modify/specify the algorithm parameter. Does that make sense?
> > >> > > >
> > >> > > > The ask in KIP is introducing new interfaces where the KIP's
> > >> > > > goal/motivation can be achieved through the security.provider
> > >> > > >
> > >> > > > and
> > >> > > >
> > >> > > > we
> > >> > > >
> > >> > > > worked
> > >> > > > on similar goal without touching any Keystore or Truststore
> > >> > > >
> > >> > > > interfaces.
> > >> > > >
> > >> > > > My advise is against changing or introducing new interfaces
> > >> > > >
> > >> > > > when
> > >> > > >
> > >> > > > it can
> > >> > > >
> > >> > > > work through security.provider.
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Harsha
> > >> > > >
> > >> > > > Thanks
> > >> > > >
> > >> > > > Maulin
> > >> > > >
> > >> > > > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
> > >> > > >
> > >> > > > kafka@harsha.io>
> > >> > > >
> > >> > > > wrote:
> > >> > > >
> > >> > > > In your KIP you added security. provider as rejected
> > >> > > >
> > >> > > > alternative
> > >> > > >
> > >> > > > and
> > >> > > >
> > >> > > > specified "its not the correct way". Do you mind explaining
> > >> > > >
> > >> > > > why
> > >> > > >
> > >> > > > its
> > >> > > >
> > >> > > > not? I
> > >> > > >
> > >> > > > didn't find any evidence in Java docs to say so. Contrary to
> > >> > > >
> > >> > > > your
> > >> > > >
> > >> > > > statement
> > >> > > >
> > >> > > > it does say in the java docs
> > >> > > > " However, please note that a provider can be used to
> > >> > > >
> > >> > > > implement
> > >> > > >
> > >> > > > any
> > >> > > >
> > >> > > > security service in Java that uses a pluggable architecture
> > >> > > >
> > >> > > > with
> > >> > > >
> > >> > > > a
> > >> > > >
> > >> > > > choice
> > >> > > >
> > >> > > > of implementations that fit underneath."
> > >> > > >
> > >> > > > Java Security Providers have been used by other projects to
> > >> > > >
> > >> > > > provide
> > >> > > >
> > >> > > > such
> > >> > > >
> > >> > > > integration . I am not sure if you looked into Spiffe
> > >> > > >
> > >> > > > project to
> > >> > > >
> > >> > > > efficiently distribute certificates but here is an example of
> > >> > > >
> > >> > > > Java
> > >> > > >
> > >> > > > provider
> > >> > > >
> > >> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > >> > > >
> > >> > > > java which
> > >> > > > obtains certificates from local daemons.
> > >> > > > These integrations are being used in Tomcat, Jetty etc.. We
> > >> > > >
> > >> > > > are
> > >> > > >
> > >> > > > also
> > >> > > >
> > >> > > > using
> > >> > > >
> > >> > > > Security provider to do the same in our Kafka clusters. So
> > >> > > >
> > >> > > > unless I
> > >> > > >
> > >> > > > see
> > >> > > >
> > >> > > > more evidence why security.provider doesn't work for you
> > >> > > >
> > >> > > > adding
> > >> > > >
> > >> > > > new
> > >> > > >
> > >> > > > interfaces while there exists more cleaner way of achieving
> > >> > > >
> > >> > > > the
> > >> > > >
> > >> > > > goals
> > >> > > >
> > >> > > > of
> > >> > > >
> > >> > > > this KIP is unnecessary and breaks the well known security
> > >> > > >
> > >> > > > interfaces
> > >> > > >
> > >> > > > provided by Java itself.
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Harsha
> > >> > > >
> > >> > > > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
> > >> > > >
> > >> > > > kafka@harsha.io
> > >> > > >
> > >> > > > wrote:
> > >> > > >
> > >> > > > Hi Maulin,
> > >> > > > Not sure if you looked at my previous replies. This
> > >> > > >
> > >> > > > changes
> > >> > > >
> > >> > > > are not required as there is already security Provider to do
> > >> > > >
> > >> > > > what you
> > >> > > >
> > >> > > > are
> > >> > > >
> > >> > > > proposing. This KIP
> > >> > > >
> > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > >> > > >
> > >> > > > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> > >> > > >
> > >> > > > also
> > >> > > >
> > >> > > > addresses easy registration of such providers.
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Harsha
> > >> > > >
> > >> > > > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
> > >> > > >
> > >> > > > <maulin.vasavada@gmail.
> > >> > > >
> > >> > > > com> wrote:
> > >> > > >
> > >> > > > Bump! Can somebody please review this?
> > >> > > >
> > >> > > > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
> > >> > > >
> > >> > > > maulin.vasavada@gmail.com>
> > >> > > >
> > >> > > > wrote:
> > >> > > >
> > >> > > > Bump! Can somebody please review this?
> > >> > > >
> > >> > > > --
> > >> > > > Thanks,
> > >> > > > M.Sai Sandeep
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Rajini Sivaram <ra...@gmail.com>.
Just to make sure we are on the same page - KIP-383 was written before the
code was refactored. The refactoring addressed some of the concerns of
KIP-383. My suggestion was to make SslEngineBuilder configurable. The
default implementation of this pluggable class would be
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/ssl/SslEngineBuilder.java.
That should give an idea of the size of the configurable part that a custom
class needs to implement. A large part of that is about loading
keystore/truststore. I agree it has slightly more code than KIP-486
proposes, but since it lets you customize creation of SSLEngine, it would
address every possible scenario.
Thoughts?
On Fri, Aug 30, 2019 at 2:02 AM Maulin Vasavada <ma...@gmail.com>
wrote:
> I thought about it more. I feel that no matter how we refactor the code
> (with or without KIP-383 integrated), ultimately the need of customizing
> loading for keys and certs will still remain. Whenever that need arises we
> might end up thinking about the solution suggested by our KIP-486. Hence
> regardless of the other KIPs and configurations "if we do need to customize
> loading of keys/certs, we will need the code changes suggested by this
> KIP".
>
> Let me know what you guys think.
>
> Harsha, we are working on changing the interfaces for key/trust store
> loaders with Certificate and PrivateKey objects. Will probably be able to
> update it later today or tomorrow.
>
> Thanks
> Maulin
>
>
>
>
>
> On Thu, Aug 29, 2019 at 2:30 PM Maulin Vasavada <maulin.vasavada@gmail.com
> >
> wrote:
>
> > On that, I actually looked at KIP-383 before briefly. However, that
> > sounded like lot of changes suggested.
> >
> > One "key" thing we have to keep in mind is - IF we need lot of
> > customization Kafka already allows you to use your SslProvider via
> > ssl.providers or the changes done by KIP-492 and
> > SSLContext.getInstance(protocol, provider) call allows us to return the
> > SSLContext with "ALL" the details we would like to customize. Hence I am
> > not sure that customization suggested by KIP-383 would be worth the
> effort.
> > We also have similar SSLContext customization outside of Kafka.
> >
> > Thanks
> > Maulin
> >
> >
> >
> >
> >
> > On Thu, Aug 29, 2019 at 12:47 PM Pellerin, Clement <
> > Clement_Pellerin@ibi.com> wrote:
> >
> >> KIP-383 in its present form was vetoed because it was not possible to
> add
> >> validation of custom properties in a future KIP. The solution to that is
> >> the first proposal I wrote for KIP-383 which made the whole SslFactory
> >> pluggable. That first solution was also vetoed hence the deadlock.
> >>
> >> Replacing the whole factory was a much nicer solution. It was vetoed
> >> because doing this almost invariably meant the replacement lost all the
> >> complex validation code in the default SslFactory.
> >>
> >> My current idea is to extract the validation code into another public
> API
> >> that SslFactory would call. I did not look at the newly refactored code
> and
> >> I did not study how to do this yet. KIP-383 was not popular at the time
> and
> >> designing a new solution is a lot of work.
> >>
> >> Is there interest from 3 binding voters for something like this?
> >>
> >> -----Original Message-----
> >> From: Rajini Sivaram [mailto:rajinisivaram@gmail.com]
> >> Sent: Thursday, August 29, 2019 2:57 PM
> >> To: dev
> >> Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and
> >> TrustStore
> >>
> >> Hi Maulin,
> >>
> >> In SSL scenarios, I imagine security providers introduced by KIP-492 are
> >> likely to be most useful when you want to use third party providers. The
> >> biggest advantage of the config from that KIP is that you don't need to
> >> write much code to integrate existing security providers into Kafka
> >> brokers
> >> or clients. As I understand it, KIP-486 is a more convenient option for
> >> the
> >> specific problem of loading keystores/truststores differently. It can be
> >> achieved in theory with KIP-492, but KIP-486 is a much simpler option
> for
> >> this case.
> >>
> >> My concern about KIP-486 is that it introduces yet another interface
> into
> >> our already complex security code, while only solving one particular use
> >> case. Have you looked at
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-383%3A++Pluggable+interface+for+SSL+Factory
> >> ?
> >> The goal was to make
> >> org.apache.kafka.common.security.ssl.SslEngineBuilder pluggable.
> >> The code has already been refactored by Colin after that KIP was
> written,
> >> making it easier to implement KIP-383. This should enable you to load
> your
> >> keystores and truststores differently. Using a pluggable
> SslEngineBuilder
> >> will also solve several other use cases at the same time. KIP-383 hasn't
> >> been voted through yet, but perhaps you could take a look and we could
> >> revive that instead if it solves your use case as well?
> >>
> >> Regards,
> >>
> >> Rajini
> >>
> >>
> >> On Thu, Aug 29, 2019 at 6:42 PM Maulin Vasavada <
> >> maulin.vasavada@gmail.com>
> >> wrote:
> >>
> >> > Hi Harsha
> >> >
> >> > Thank you. Appreciate your time and support on this. Let me go back do
> >> some
> >> > more research and get back to you on the KeyStore interface part.
> >> > Basically, if we return certs and keys in the interface then Kafka
> code
> >> > will have to build KeyStore object - which is also reasonable.
> >> >
> >> > Thanks
> >> > Maulin
> >> >
> >> > On Thu, Aug 29, 2019 at 10:01 AM Harsha Chintalapani <kafka@harsha.io
> >
> >> > wrote:
> >> >
> >> > > Hi Maulin,
> >> > > Use cases are clear now. I am +1 for moving
> >> forward
> >> > > with the discussions on having such configurable option for users.
> But
> >> > the
> >> > > interfaces is proposed doesn't look right to me. We are still
> talking
> >> > about
> >> > > keystore interfaces. Given keystore's are used as filebased way of
> >> > > transporting certificates I am not sure it will help the rest of the
> >> > > user-base.
> >> > > In short, I am +1 on the KIP's motivation and only
> >> have
> >> > > questions around returning keystores instead of returning certs,
> >> private
> >> > > keys etc. . If others in the community are ok with such interface we
> >> can
> >> > > move forward.
> >> > >
> >> > > Thanks,
> >> > > Harsha
> >> > >
> >> > >
> >> > > On Wed, Aug 28, 2019 at 1:51 PM, Maulin Vasavada <
> >> > > maulin.vasavada@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > Hi Harsha
> >> > > >
> >> > > > As we synced-up offline on this topic, we hope you don't have any
> >> more
> >> > > > clarifications that you are seeking. If that is the case, can you
> >> > please
> >> > > > help us move this forward and discuss what changes you would
> expect
> >> on
> >> > > the
> >> > > > KIP design in order to make it valuable contribution?
> >> > > >
> >> > > > Just FYI - we verified our primary design change with the author
> of
> >> > Sun's
> >> > > > X509 Trustmanager implementation and the outcome is that what we
> are
> >> > > > proposing makes sense at the heart of it - "Instead of writing
> >> > > TrustManager
> >> > > > just plugin the Trust store". We are open to discuss additional
> >> changes
> >> > > > that you/anybody else would like to see on the functionality
> >> however.
> >> > > >
> >> > > > Thanks
> >> > > > Maulin
> >> > > >
> >> > > > On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada <
> >> > > maulin.vasavada@gmail.com>
> >> > > > wrote:
> >> > > >
> >> > > > Hi Harsha
> >> > > >
> >> > > > Any response on my question? I feel this KIP is worth
> accommodating.
> >> > Your
> >> > > > help is much appreciated.
> >> > > >
> >> > > > Thanks
> >> > > > Maulin
> >> > > >
> >> > > > On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada <
> >> > maulin.vasavada@gmail.
> >> > > > com> wrote:
> >> > > >
> >> > > > Hi Harsha
> >> > > >
> >> > > > I've examined the SPIFFE provider more and have one question -
> >> > > >
> >> > > > If SPIFFE didn't have a need to do checkSpiffeId() call at the
> below
> >> > > > location, would you really still write the Provider? *OR* Would
> you
> >> > just
> >> > > > use TrustManagerFactory.init(KeyStore) signature to pass the
> >> KeyStore
> >> > > from
> >> > > > set of certs returned by spiffeIdManager. getTrustedCerts()?
> >> > > >
> >> > > >
> >> >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> >> > > > provider/CertificateUtils.java#L100
> >> > > >
> >> > > > /**
> >> > > >
> >> > > > * Validates that the SPIFFE ID is present and matches the SPIFFE
> ID
> >> > > > configured in
> >> > > > * the java.security property ssl.spiffe.accept
> >> > > > *
> >> > > > * If the authorized spiffe ids list is empty any spiffe id is
> >> > authorized
> >> > > > *
> >> > > > * @param chain an array of X509Certificate that contains the
> Peer's
> >> > SVID
> >> > > > to be validated
> >> > > > * @throws CertificateException when either the certificates
> doesn't
> >> > have
> >> > > a
> >> > > > SPIFFE ID or the SPIFFE ID is not authorized
> >> > > > */
> >> > > > static void checkSpiffeId(X509Certificate[] chain) throws
> >> > > > CertificateException {
> >> > > >
> >> > > > Thanks
> >> > > > Maulin
> >> > > >
> >> > > > On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani <
> >> kafka@harsha.io>
> >> > > > wrote:
> >> > > >
> >> > > > Maulin,
> >> > > > The code parts you are pointing are specific for Spiffe and if
> >> > > > you are talking about validate method which uses PKIX check like
> any
> >> > > other
> >> > > > provider does.
> >> > > > If you want to default to SunJSSE everywhere you can do so by
> >> > delegating
> >> > > > the calls in these methods to SunJSSE provider.
> >> > > >
> >> > > > TrustManagerFactory tmf = TrustManagerFactory
> >> > > > .getInstance(TrustManagerFactory.getDefaultAlgorithm());and use
> >> > > > tmf.chekServerTrusted()
> >> > > > or use
> >> > > > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> >> > > > TrustManagerFactory.html#getInstance(java.lang.String)if you want
> a
> >> > > > specific provider.
> >> > > >
> >> > > > -Harsha
> >> > > >
> >> > > > On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada <
> >> > maulin.vasavada@gmail.
> >> > > > com>
> >> > > > wrote:
> >> > > >
> >> > > > Okay, so I take that you guys agree that I have to write a
> 'custom'
> >> > > > algorithm and a provider to make it work , correct?
> >> > > >
> >> > > > Now, for Harsha's comment "Here the 'Custom' Algorithm is not an
> >> > > > implementation per say , ..." , I diagree. You can refer to
> https://
> >> > > >
> >> > > >
> >> >
> >>
> github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/
> >> > > >
> >> > > > SpiffeTrustManager.java#L91 <http://spiffetrustmanager.java/#L91>
> >> and
> >> > > >
> >> > > >
> >> >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> >> > > >
> >> > > > provider/CertificateUtils.java#L100
> >> > > >
> >> > > > "that code" is the customization you have for the custom way to
> >> check
> >> > > > something on top of regular checks. That method is NOT doing
> custom
> >> > > > truststore loading. It is validating/verifying something in the
> >> > > >
> >> > > > "custom"
> >> > > >
> >> > > > way with spiffeId.
> >> > > > I bet that without that you won't have a need of the custom
> >> algorithm
> >> > > >
> >> > > > in
> >> > > >
> >> > > > the first place.
> >> > > >
> >> > > > Let me know if you agree to this.
> >> > > >
> >> > > > Thanks
> >> > > > Maulin
> >> > > >
> >> > > > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <mp...@gmail.com>
> >> > > >
> >> > > > wrote:
> >> > > >
> >> > > > Hi Maulin, thanks for the discussion. As Harsha pointed out, to
> use
> >> the
> >> > > > KIP492, you need to create a new provider, register a *new* custom
> >> > > > algorithm for your keymanager and trustmanager factory
> >> implementations.
> >> > > > After this, the kafka server configuration can be done as given
> >> below
> >> > > >
> >> > > > # Register the provider class with custom algorithm, say CUSTOM
> >> > > >
> >> > > > security.
> >> > > >
> >> > > > provider.classes=com.company.security.CustomProvider
> >> > > > <http://provider.classes=com.company.security.customprovider/>
> >> > > > <http://security.provider.classes
> >> > > >
> >> > > > =com.company.security.customprovider/>
> >> > > >
> >> > > > # Register the keymanager and trustmanager algorithms
> >> > > > # These algorithms indicate that the Keymanager and Trustmanagers
> >> > > > registered under the algorithm "CUSTOM" needs to be used
> >> > > > ssl.trustmanager.algorithm=CUSTOM
> >> > > > ssl.keymanager.algorithm=CUSTOM
> >> > > >
> >> > > > And the customprovider looks like this...
> >> > > >
> >> > > > public class CustomProvider extends Provider {
> >> > > > public CustomProvider() {
> >> > > > super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and
> TrustStore");
> >> > > > super.put("KeyManagerFactory.CUSTOM", "customKeyManagerFactory");
> >> > > > super.put("TrustManagerFactory.CUSTOM",
> >> > > > "customTrustManagerFactory");
> >> > > > }
> >> > > > }
> >> > > >
> >> > > > The PR for this is in review and can be found here:
> >> > > >
> >> > > > https://github.com/
> >> > > >
> >> > > > apache/kafka/pull/7090
> >> > > > This PR includes the fixed insertProviderAt function call.
> >> > > >
> >> > > > On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani <
> >> kafka@harsha.io>
> >> > > > wrote:
> >> > > >
> >> > > > Answers are in-line
> >> > > >
> >> > > > On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada <
> >> > > >
> >> > > > maulin.vasavada@gmail.
> >> > > >
> >> > > > com
> >> > > >
> >> > > > wrote:
> >> > > >
> >> > > > Hi Colin
> >> > > >
> >> > > > When I refer to "standard" or "custom" algorithms I am following
> >> Java
> >> > > > security Provider Terminology. You can refer to
> >> > > > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> >> > > > StandardNames.html#TrustManagerFactory link I provided earlier in
> >> the
> >> > > > emails. It says PKIX is the default Algorithm for
> >> TrustManagerFactory.
> >> > > >
> >> > > > 1. For SPIFFE, I am not sure why you are saying 'it does not
> >> implement
> >> > > > custom algorithms' because the following file clearly indicates
> >> that it
> >> > > > does use custom algorithm-
> >> > > >
> >> > > >
> >> >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> >> > > >
> >> > > > provider/SpiffeProvider.java#L17
> >> > > >
> >> > > > Algorithm value:
> >> > > >
> >> > > >
> >> >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> >> > > >
> >> > > > provider/SpiffeProviderConstants.java#L6
> >> > > >
> >> > > > @Harsha do you want to chime in since you use that provider?
> >> > > >
> >> > > > Here the "Custom" Algorithm is not an implementation per say ,
> >> rather
> >> > > >
> >> > > > used
> >> > > >
> >> > > > to invoke the custom trust store factory and key manager factory.
> >> You
> >> > > >
> >> > > > are
> >> > > >
> >> > > > not moving away from "standard" alogrithms that are available.
> >> > > >
> >> > > >
> >> >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> >> > > >
> >> > > > provider/SpiffeTrustManager.java
> >> > > >
> >> > > > As you can see it delegates all the calls of verification of
> >> > > >
> >> > > > certificate
> >> > > >
> >> > > > to
> >> > > >
> >> > > > the default implementation available.
> >> > > > So in our implementation we still use PKIX to verify the
> certificate
> >> > > > chain. So you are not losing anything here and Spiffe is not
> >> > > >
> >> > > > reimplementing
> >> > > >
> >> > > > the verification process.
> >> > > >
> >> > > > 2. I already mentioned in my 3rd point, in my previous post, why
> >> using
> >> > > >
> >> > > > ssl.provider does NOT work. I updated KIP-486 in "rejected
> >> > > >
> >> > > > alternatives"
> >> > > >
> >> > > > also why ssl.provider does not work.
> >> > > >
> >> > > > As mentioned before , provider is the right way to go. I am still
> >> not
> >> > > > understanding the gap is.
> >> > > > If I understand correctly your argument is , provider is going to
> >> ask
> >> > > >
> >> > > > to
> >> > > >
> >> > > > implement a custom algorithm.
> >> > > > My answer to that is , no you are not re-implementing the
> algorithm.
> >> > > >
> >> > > > Please
> >> > > >
> >> > > > check the above link , TrustManager implementation it delegates
> the
> >> > > >
> >> > > > calls
> >> > > >
> >> > > > back. There is no need to implement your own here.
> >> > > >
> >> > > > 3. Security.insertProviderAt() comments were based on assumption
> if
> >> > > >
> >> > > > KIP-492
> >> > > >
> >> > > > changes are done and we use that mechanism to configure providers
> >> > > >
> >> > > > instead
> >> > > >
> >> > > > of ssl.provider configuration.
> >> > > >
> >> > > > KIP-492 has patch available and going through review.
> >> > > >
> >> > > > Can you read my all the points, I mentioned in my previous post,
> >> very
> >> > > >
> >> > > > carefully? I am covering all the aspects in explaining. I am open
> to
> >> > > >
> >> > > > still
> >> > > >
> >> > > > discuss more to clarify any doubts.
> >> > > >
> >> > > > "3. If we just use existing ssl.provider kafka configuration then
> >> our
> >> > > > provider will be used in SSLContext.getInstance(protocol,
> provider)
> >> > > >
> >> > > > call
> >> > > >
> >> > > > in
> >> > > >
> >> > > > SslFactory.java <http://sslfactory.java/> <
> http://sslfactory.java/>
> >> <
> >> > > > http://sslfactory.java/>
> >> > > >
> >> > > > and
> >> > > >
> >> > > > if our provider does not have
> >> > > > implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks
> (we
> >> > > >
> >> > > > tested
> >> > > >
> >> > > > it). Example: In MyProvider sample above you see that I didn't add
> >> > > > SSLContext.TLSv1 as
> >> > > > "Service+Algorithm" and that didn't work for me. In SPIFFE
> provider
> >> you
> >> > > > don't have this challenge since you are planning to bypass
> >> ssl.provider
> >> > > >
> >> > > > as
> >> > > >
> >> > > > you mention in the KIP-492."
> >> > > >
> >> > > > Yes here you need to pass the protocol that your
> >> > > >
> >> > > > KeyManager/TrustManager
> >> > > >
> >> > > > registered with and in no way its deviating from TLS RFC spec.
> >> > > >
> >> > > >
> >> >
> https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/
> >> > > >
> >> > > > javax/net/ssl/SSLContext.java#L134
> >> > > >
> >> > > > My suggestion here is for you to implement a simple Security
> >> Provider
> >> > > >
> >> > > > as
> >> > > >
> >> > > > you did before and register a Algorithm. You can use the existing
> >> > > > implementation in SpiffeProvider and plug in changes where you
> need
> >> to
> >> > > > retrieve the certificates from by making RPC call.
> >> > > >
> >> > > > Run an end-to-end test with Kafka broker coming up with your
> >> provider
> >> > > > making calls to RPC call. You do need to pass the "custom
> algorithm"
> >> > > >
> >> > > > that
> >> > > >
> >> > > > you registered in your key manager to invoke the provider. I think
> >> your
> >> > > > concern is this approach is replacing the existing known
> >> ciphersuites
> >> > > >
> >> > > > and
> >> > > >
> >> > > > certificate validation provided by java. Which its not.
> >> > > >
> >> > > > Now test the TLS connection you can do so via openssl -s_client
> >> options
> >> > > >
> >> > > > to
> >> > > >
> >> > > > test if everything else is working.
> >> > > >
> >> > > > I am happy to share configs that we used if you are interested.
> >> > > >
> >> > > > Thanks,
> >> > > > Harsha
> >> > > >
> >> > > > Thanks
> >> > > > Maulin
> >> > > >
> >> > > > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <cm...@apache.org>
> >> > > >
> >> > > > wrote:
> >> > > >
> >> > > > Hi Maulin,
> >> > > >
> >> > > > A lot of JSSE providers don't implement custom algorithms. Spire
> is
> >> a
> >> > > >
> >> > > > good
> >> > > >
> >> > > > example of a JSSE provider that doesn't, and yet is still useful
> to
> >> > > >
> >> > > > many
> >> > > >
> >> > > > people. Your JSSE provider can work fine even if it doesn't
> >> implement a
> >> > > > custom algorithm.
> >> > > >
> >> > > > Maybe I'm missing something, but I don't understand the discussion
> >> of
> >> > > > Security.insertProviderAt() that you included. SslEngineBuilder
> >> doesn't
> >> > > >
> >> > > > use
> >> > > >
> >> > > > that API to get the security provider. Instead, it calls
> >> > > > "SSLContext.getInstance(protocol, provider)", where provider is
> the
> >> > > >
> >> > > > name
> >> > > >
> >> > > > of the provider.
> >> > > >
> >> > > > best,
> >> > > > Colin
> >> > > >
> >> > > > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
> >> > > >
> >> > > > On top of everything above I feel strongly to add the 4th point
> >> which
> >> > > >
> >> > > > is
> >> > > >
> >> > > > based on Java APIs for TrustManagerFactory.init(KeyStore) (
> >> > > >
> >> > > > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> >> > > > TrustManagerFactory.html#init(java.security.KeyStore
> >> > > > <http://java.security.keystore/>
> >> > > > <http://java.security.keystore/>)
> >> > > > )
> >> > > >
> >> > > > and KeyManagerFactory.init(KeyStore, char[]) (
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory
> >> > > > .
> >> > > >
> >> > > > html#init(java.security.KeyStore <http://java.security.keystore/>
> >> > > <http://
> >> > > > java.security.keystore/
> >> > > > ,%20char[])
> >> > > >
> >> > > > ).
> >> > > >
> >> > > > 4. The above APIs are intended to support providing "trust/key
> >> > > >
> >> > > > material"
> >> > > >
> >> > > > from the user without having to write their own
> >> > > >
> >> > > > TrustManager/KeyManagers.
> >> > > >
> >> > > > To quote from the TrustManagerFactory.init()'s documentation
> >> > > >
> >> > > > "Initializes
> >> > > >
> >> > > > this factory with a source of certificate authorities and related
> >> trust
> >> > > > material."
> >> > > > To quote from the KeyManagerFactory.init()'s documentation
> >> "Initializes
> >> > > > this factory with a source of key material."
> >> > > >
> >> > > > Based on this it is clear that there is a flexibility provided by
> >> Java
> >> > > >
> >> > > > to
> >> > > >
> >> > > > to enable developers to provide the required trust/key material
> >> loaded
> >> > > >
> >> > > > from
> >> > > >
> >> > > > "anywhere" without requiring them to write custom provider OR
> >> trust/key
> >> > > > managers. This same flexibility is reflected in Kafka code also
> >> where
> >> > > >
> >> > > > it
> >> > > >
> >> > > > loads the trust/keys from a local file and doesn't require
> writing a
> >> > > > Provider necessarily. If we do NOT have a custom algorithm, it
> makes
> >> > > >
> >> > > > less
> >> > > >
> >> > > > sense to write a Provider.
> >> > > >
> >> > > > Thanks
> >> > > > Maulin
> >> > > >
> >> > > > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
> >> > > >
> >> > > > maulin.vasavada@gmail.com>
> >> > > >
> >> > > > wrote:
> >> > > >
> >> > > > Hi Harsha/Colin
> >> > > >
> >> > > > I did the sample with a custom Provider for TrustStoreManager and
> >> tried
> >> > > > using ssl.provider Kafka config AND the way KIP-492 is suggesting
> >> (by
> >> > > > adding Provider programmatically instead of relying on
> >> > > >
> >> > > > ssl.provider+java.
> >> > > >
> >> > > > security. The below sample is followed by my detailed findings.
> I'll
> >> > > > appreciate if you can go through it carefully and see
> >> > > >
> >> > > > if you
> >> > > >
> >> > > > see my point.
> >> > > >
> >> > > > package providertest;
> >> > > >
> >> > > > import java.security.Provider <http://java.security.provider/>
> >> > <http://
> >> > > > java.security.provider/>
> >> > > >
> >> > > > <http://
> >> > > >
> >> > > > java.security.provider/>;
> >> > > >
> >> > > > public class MyProvider extends Provider {
> >> > > >
> >> > > > private static final String name = "MyProvider"; private static
> >> double
> >> > > > version = 1.0d;
> >> > > > private static String info = "Maulin's SSL Provider v"+version;
> >> > > >
> >> > > > public MyProvider() {
> >> > > > super(name, version, info);
> >> > > > this.put("TrustManagerFactory.PKIX",
> >> > > >
> >> > > > "providertest.MyTrustManagerFactory");
> >> > > >
> >> > > > }
> >> > > > }
> >> > > >
> >> > > > *Details:*
> >> > > >
> >> > > > KIP-492 documents that it will use Security.addProvider() assuming
> >> it
> >> > > >
> >> > > > will
> >> > > >
> >> > > > add it as position '0' which is not a correct assumption. The
> >> > > > addProvider()'s documentation says it will add it to the last
> >> available
> >> > > > position. You may want to correct that to say
> >> > > > Security.insertProviderAt(provider, 1).
> >> > > >
> >> > > > Now coming back to our specific discussion,
> >> > > >
> >> > > > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you
> add
> >> > > >
> >> > > > that
> >> > > >
> >> > > > provider in the provider list via Security.addProvider() the
> >> position
> >> > > >
> >> > > > where
> >> > > >
> >> > > > it gets added doesn't matter (even if you don't end up adding it
> as
> >> > > >
> >> > > > first
> >> > > >
> >> > > > entry) since that is the ONLY provider for SPIFFE specific
> algorithm
> >> > > >
> >> > > > you
> >> > > >
> >> > > > might have.
> >> > > >
> >> > > > We do *not* have custom algorithm for Key/Trust StoreMangers.
> Which
> >> > > >
> >> > > > means
> >> > > >
> >> > > > we have to use X509, PKIX etc "Standard Algorithms" ((
> >> > > >
> >> > > > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> >> > > > StandardNames.html
> >> > > > ))
> >> > > >
> >> > > > in our provider to override the TrustStoreManager (see my sample
> >> code)
> >> > > >
> >> > > > and
> >> > > >
> >> > > > KeyStoreManger and KeyManager. This creates another challenge
> >> > > >
> >> > > > mentioned in
> >> > > >
> >> > > > the below point.
> >> > > >
> >> > > > 2. In order to make our Provider for loading custom TrustStore
> >> work, we
> >> > > > have to add the provider as 'first' in the list since there are
> >> others
> >> > > >
> >> > > > with
> >> > > >
> >> > > > the same algorithm.
> >> > > >
> >> > > > However, the programatic way of adding provider
> >> > > > (Security.insertProviderAt()) is *not* deterministic for ordering
> >> since
> >> > > > different code can call the method for a different provider and
> >> > > >
> >> > > > depending
> >> > > >
> >> > > > upon the order of the call our provider can be first or pushed
> down
> >> the
> >> > > > list. This can happen very well in any client application using
> >> Kafka.
> >> > > >
> >> > > > This
> >> > > >
> >> > > > is specially problematic for a case when you want to guarantee
> order
> >> > > >
> >> > > > for a
> >> > > >
> >> > > > Provider having "Standard Algorithms".
> >> > > >
> >> > > > If we add our provider in java.security file that definitely
> >> guarantees
> >> > > > the order(unless somebody calls removeProvider() which is less
> >> > > >
> >> > > > likely). But
> >> > > >
> >> > > > if we add our provider in java.security file it will defeat the
> >> > > >
> >> > > > purpose of
> >> > > >
> >> > > > the KIP-492.
> >> > > >
> >> > > > In the gist - Apache Kafka must not rely on "un-deterministic"
> >> method
> >> > > >
> >> > > > to
> >> > > >
> >> > > > rely on Provider ordering.
> >> > > >
> >> > > > 3. If we just use existing ssl.provider kafka configuration then
> our
> >> > > > provider will be used in SSLContext.getInstance(protocol,
> provider)
> >> > > >
> >> > > > call in
> >> > > >
> >> > > > SslFactory.java <http://sslfactory.java/> <
> http://sslfactory.java/>
> >> <
> >> > > > http://sslfactory.java/>
> >> > > >
> >> > > > and
> >> > > >
> >> > > > if our provider does not have implementation for
> >> > > > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks
> >> > > >
> >> > > > (we
> >> > > >
> >> > > > tested it). Example:
> >> > > >
> >> > > > In
> >> > > >
> >> > > > MyProvider sample above you see that I didn't add SSLContext.TLSv1
> >> as
> >> > > > "Service+Algorithm" and that didn't work for me. In SPIFFE
> provider
> >> you
> >> > > > don't have this challenge since you are planning to bypass
> >> > > >
> >> > > > ssl.provider as
> >> > > >
> >> > > > you mention in the KIP-492.
> >> > > >
> >> > > > *Overall summary,*
> >> > > >
> >> > > > 1. Any provider based mechanisms- a) existing ssl.provider and
> >> > > >
> >> > > > b)KIP-492,
> >> > > >
> >> > > > for loading key/trust store using "Standard Algorithms" do not
> work
> >> > > >
> >> > > > 2. Approach suggested in our KIP-486 works without any issue and
> it
> >> is
> >> > > > *not* our context specific solve
> >> > > >
> >> > > > 3. Based on above we feel KIP-492 and KIP-486 are complimentary
> >> changes
> >> > > > and not contradicting or redundent.
> >> > > >
> >> > > > If you want we can do a joint session somehow to walk through the
> >> > > >
> >> > > > sample I
> >> > > >
> >> > > > have and various experiments I did. I would encourage you to do
> >> similar
> >> > > > exercise by writing a Provider for "Standard Algorithm" for
> >> > > > TrustStoreManager (like our needs) and see what you find since
> only
> >> > > >
> >> > > > writing
> >> > > >
> >> > > > samples can bring out the complexity/challenges we face.
> >> > > >
> >> > > > Thanks
> >> > > > Maulin
> >> > > >
> >> > > > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <
> >> > > >
> >> > > > maulin.vasavada@gmail.
> >> > > >
> >> > > > com> wrote:
> >> > > >
> >> > > > Just to update - still working on it. Get to work only on and off
> on
> >> > > >
> >> > > > it :(
> >> > > >
> >> > > > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
> >> > > >
> >> > > > maulin.vasavada@gmail.com>
> >> > > >
> >> > > > wrote:
> >> > > >
> >> > > > Hi Harsha
> >> > > >
> >> > > > Let me try to write samples and will let you know.
> >> > > >
> >> > > > Thanks
> >> > > > Maulin
> >> > > >
> >> > > > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <ha...@gmail.com>
> >> > > >
> >> > > > wrote:
> >> > > >
> >> > > > Hi Maulin,
> >> > > > With java security providers can be as custom you would
> >> > > >
> >> > > > like
> >> > > >
> >> > > > it to
> >> > > > be. If you only want to to implement a custom way of loading the
> >> > > >
> >> > > > keystore
> >> > > >
> >> > > > and truststore and not implement any protocol/encryption handling
> >> you
> >> > > >
> >> > > > can
> >> > > >
> >> > > > leave them empty and no need to implement. Have you looked into
> the
> >> > > >
> >> > > > links I
> >> > > >
> >> > > > pasted before?
> >> > > >
> >> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> >> > > >
> >> > > > java
> >> > > >
> >> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >> > > > spiffe-security-provider/src/main/java/spiffe/api/provider/
> >> > > > SpiffeTrustManager.java <http://spiffetrustmanager.java/>
> <http://
> >> > > > spiffetrustmanager.java/>
> >> > > >
> >> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> >> > > >
> >> > > > java
> >> > > >
> >> > > > Can you please tell me which methods are too complex in above to
> >> > > >
> >> > > > implement
> >> > > >
> >> > > > or unnecessary? You are changing anything in SSL/TLS
> implementations
> >> > > > provided by
> >> > > >
> >> > > > All of the implementations delegating the checks to the default
> >> > > > implementation anyway.
> >> > > > Spire agent is an example, its nothing but a GRPC server listening
> >> > > >
> >> > > > on a
> >> > > >
> >> > > > unix domain socket . Above code is making a RPC call to the local
> >> > > >
> >> > > > daemon
> >> > > >
> >> > > > to
> >> > > > get the certificate and keys. The mechanics are pretty much same
> as
> >> > > >
> >> > > > what
> >> > > >
> >> > > > you are asking for.
> >> > > >
> >> > > > Thanks,
> >> > > > Harsha
> >> > > >
> >> > > > On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
> >> > > >
> >> > > > maulin.vasavada@gmail.com>
> >> > > >
> >> > > > wrote:
> >> > > >
> >> > > > Imagine a scenario like - We know we have a custom KMS and as a
> >> > > >
> >> > > > Kafka
> >> > > >
> >> > > > owner
> >> > > >
> >> > > > we want to comply to using that KMS source to load keys/certs. As
> >> > > >
> >> > > > a
> >> > > >
> >> > > > Kafka
> >> > > >
> >> > > > owner we know how to integrate with KMS but doesn't necessarily
> >> > > >
> >> > > > have
> >> > > >
> >> > > > to
> >> > > >
> >> > > > know anything about cipher suites, algorithms, and SSL/TLS
> >> > > >
> >> > > > implementation.
> >> > > >
> >> > > > Going the Provider way requires to know lot more than we should,
> >> > > >
> >> > > > isn't it?
> >> > > >
> >> > > > Not that we would have concern/shy-away knowing those details -
> >> > > >
> >> > > > but
> >> > > >
> >> > > > if we
> >> > > >
> >> > > > don't have to - why should we?
> >> > > >
> >> > > > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
> >> > > >
> >> > > > maulin.vasavada@gmail.com>
> >> > > >
> >> > > > wrote:
> >> > > >
> >> > > > Hi Harsha
> >> > > >
> >> > > > We don't have spire (or similar) agents and we do not have
> >> > > >
> >> > > > keys/certs
> >> > > >
> >> > > > locally on any brokers. To elaborate more on my previous email,
> >> > > >
> >> > > > I agree that Java security Providers are used in much broader
> >> > > >
> >> > > > sense
> >> > > >
> >> > > > - to
> >> > > >
> >> > > > have a particular implementation of an algorithm, use specific
> >> > > >
> >> > > > cipher
> >> > > >
> >> > > > suites for SSL , OR in our current team's case have a
> >> > > >
> >> > > > particular
> >> > > >
> >> > > > way to
> >> > > >
> >> > > > leverage pre-generated SSL sessions. However, the scope of our
> >> > > >
> >> > > > KIP
> >> > > >
> >> > > > (486)
> >> > > >
> >> > > > is
> >> > > >
> >> > > > much restricted than that. We merely intend to provide a custom
> >> > > > keystore/truststore for our SSL connections and not really worry
> >> > > >
> >> > > > about
> >> > > >
> >> > > > underlying specific SSL/TLS implementation. This simplifies it
> >> > > >
> >> > > > a
> >> > > >
> >> > > > lot for
> >> > > >
> >> > > > us to keep the concerns separate and clear.
> >> > > >
> >> > > > I feel our approach is more complimentary such that it allows
> >> > > >
> >> > > > for
> >> > > >
> >> > > > using
> >> > > >
> >> > > > keystores of choice while retaining the flexibility to use any
> >> > > > underlying/available Provider for actually making the SSL call.
> >> > > >
> >> > > > We agree with KIP-492's approach based on Providers (and Java's
> >> > > > recommendation), but also strongly believe that our approach can
> >> > > >
> >> > > > compliment
> >> > > >
> >> > > > it very effectively for reasons explained above.
> >> > > >
> >> > > > Thanks
> >> > > > Maulin
> >> > > >
> >> > > > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <
> >> > > >
> >> > > > kafka@harsha.io
> >> > > >
> >> > > > wrote:
> >> > > >
> >> > > > Hi Maulin,
> >> > > >
> >> > > > On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
> >> > > >
> >> > > > maulin.vasavada@gmail.
> >> > > >
> >> > > > com>
> >> > > > wrote:
> >> > > >
> >> > > > Hi Harsha
> >> > > >
> >> > > > The reason we rejected the SslProvider route is that - we
> >> > > >
> >> > > > only
> >> > > >
> >> > > > needed
> >> > > >
> >> > > > a
> >> > > >
> >> > > > custom way to load keys/certs. Not touch any policy that
> >> > > >
> >> > > > existing
> >> > > >
> >> > > > Providers
> >> > > >
> >> > > > govern like SunJSSE Provider.
> >> > > >
> >> > > > We have exactly the same requirements to load certs and keys
> >> > > >
> >> > > > through
> >> > > >
> >> > > > spire
> >> > > >
> >> > > > agent. We used security.provider to do that exactly. I am not
> >> > > >
> >> > > > sure
> >> > > >
> >> > > > why
> >> > > >
> >> > > > you
> >> > > >
> >> > > > would be modifying any policies provided by default SunJSSE
> >> > > >
> >> > > > provider.
> >> > > >
> >> > > > Can
> >> > > >
> >> > > > you give me an example of having custom provider that will
> >> > > >
> >> > > > override an
> >> > > >
> >> > > > existing policy in SunJSSE provider.
> >> > > >
> >> > > > As pointed out earlier, this kip
> >> > > >
> >> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> >> > > > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> >> > > >
> >> > > > allows
> >> > > > you to load security.provider through config.
> >> > > > Take a look at the examples I gave before
> >> > > >
> >> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> >> > > >
> >> > > > java
> >> > > >
> >> > > > It registers KeyManagerFactory and TrustManagerFactory and
> >> > > >
> >> > > > Keystore
> >> > > >
> >> > > > algorithm.
> >> > > > Implement your custom way of loading Keystore in here
> >> > > >
> >> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> >> > > >
> >> > > > java
> >> > > >
> >> > > > and Trust manager like here
> >> > > >
> >> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >> > > > spiffe-security-provider/src/main/java/spiffe/api/provider/
> >> > > > SpiffeTrustManager.java <http://spiffetrustmanager.java/>
> <http://
> >> > > > spiffetrustmanager.java/>
> >> > > >
> >> > > > In your Kafka client you can set the security.provider to your
> >> > > >
> >> > > > custom
> >> > > >
> >> > > > implementation and with this fix
> >> > > > https://issues.apache.org/jira/browse/KAFKA-8191 you can set
> >> > > > keyManagerAlgorigthm and trustManagerAlgorithm configs.
> >> > > >
> >> > > > All of this is in your clients and broker side and do not need
> >> > > >
> >> > > > to
> >> > > >
> >> > > > touch
> >> > > >
> >> > > > any
> >> > > > policy changes at JVM level. You'll register the providers in
> >> > > >
> >> > > > the
> >> > > >
> >> > > > priority
> >> > > >
> >> > > > order and can still have SunJSSE provider and have your custom
> >> > > >
> >> > > > provider
> >> > > >
> >> > > > to
> >> > > >
> >> > > > implement the key and trust managers.
> >> > > >
> >> > > > The ask here is different than KIP-492. We don't have any need
> >> > > >
> >> > > > to
> >> > > >
> >> > > > modify/specify the algorithm parameter. Does that make sense?
> >> > > >
> >> > > > The ask in KIP is introducing new interfaces where the KIP's
> >> > > > goal/motivation can be achieved through the security.provider
> >> > > >
> >> > > > and
> >> > > >
> >> > > > we
> >> > > >
> >> > > > worked
> >> > > > on similar goal without touching any Keystore or Truststore
> >> > > >
> >> > > > interfaces.
> >> > > >
> >> > > > My advise is against changing or introducing new interfaces
> >> > > >
> >> > > > when
> >> > > >
> >> > > > it can
> >> > > >
> >> > > > work through security.provider.
> >> > > >
> >> > > > Thanks,
> >> > > > Harsha
> >> > > >
> >> > > > Thanks
> >> > > >
> >> > > > Maulin
> >> > > >
> >> > > > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
> >> > > >
> >> > > > kafka@harsha.io>
> >> > > >
> >> > > > wrote:
> >> > > >
> >> > > > In your KIP you added security. provider as rejected
> >> > > >
> >> > > > alternative
> >> > > >
> >> > > > and
> >> > > >
> >> > > > specified "its not the correct way". Do you mind explaining
> >> > > >
> >> > > > why
> >> > > >
> >> > > > its
> >> > > >
> >> > > > not? I
> >> > > >
> >> > > > didn't find any evidence in Java docs to say so. Contrary to
> >> > > >
> >> > > > your
> >> > > >
> >> > > > statement
> >> > > >
> >> > > > it does say in the java docs
> >> > > > " However, please note that a provider can be used to
> >> > > >
> >> > > > implement
> >> > > >
> >> > > > any
> >> > > >
> >> > > > security service in Java that uses a pluggable architecture
> >> > > >
> >> > > > with
> >> > > >
> >> > > > a
> >> > > >
> >> > > > choice
> >> > > >
> >> > > > of implementations that fit underneath."
> >> > > >
> >> > > > Java Security Providers have been used by other projects to
> >> > > >
> >> > > > provide
> >> > > >
> >> > > > such
> >> > > >
> >> > > > integration . I am not sure if you looked into Spiffe
> >> > > >
> >> > > > project to
> >> > > >
> >> > > > efficiently distribute certificates but here is an example of
> >> > > >
> >> > > > Java
> >> > > >
> >> > > > provider
> >> > > >
> >> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> >> > > >
> >> > > > java which
> >> > > > obtains certificates from local daemons.
> >> > > > These integrations are being used in Tomcat, Jetty etc.. We
> >> > > >
> >> > > > are
> >> > > >
> >> > > > also
> >> > > >
> >> > > > using
> >> > > >
> >> > > > Security provider to do the same in our Kafka clusters. So
> >> > > >
> >> > > > unless I
> >> > > >
> >> > > > see
> >> > > >
> >> > > > more evidence why security.provider doesn't work for you
> >> > > >
> >> > > > adding
> >> > > >
> >> > > > new
> >> > > >
> >> > > > interfaces while there exists more cleaner way of achieving
> >> > > >
> >> > > > the
> >> > > >
> >> > > > goals
> >> > > >
> >> > > > of
> >> > > >
> >> > > > this KIP is unnecessary and breaks the well known security
> >> > > >
> >> > > > interfaces
> >> > > >
> >> > > > provided by Java itself.
> >> > > >
> >> > > > Thanks,
> >> > > > Harsha
> >> > > >
> >> > > > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
> >> > > >
> >> > > > kafka@harsha.io
> >> > > >
> >> > > > wrote:
> >> > > >
> >> > > > Hi Maulin,
> >> > > > Not sure if you looked at my previous replies. This
> >> > > >
> >> > > > changes
> >> > > >
> >> > > > are not required as there is already security Provider to do
> >> > > >
> >> > > > what you
> >> > > >
> >> > > > are
> >> > > >
> >> > > > proposing. This KIP
> >> > > >
> >> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> >> > > >
> >> > > > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> >> > > >
> >> > > > also
> >> > > >
> >> > > > addresses easy registration of such providers.
> >> > > >
> >> > > > Thanks,
> >> > > > Harsha
> >> > > >
> >> > > > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
> >> > > >
> >> > > > <maulin.vasavada@gmail.
> >> > > >
> >> > > > com> wrote:
> >> > > >
> >> > > > Bump! Can somebody please review this?
> >> > > >
> >> > > > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
> >> > > >
> >> > > > maulin.vasavada@gmail.com>
> >> > > >
> >> > > > wrote:
> >> > > >
> >> > > > Bump! Can somebody please review this?
> >> > > >
> >> > > > --
> >> > > > Thanks,
> >> > > > M.Sai Sandeep
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Maulin Vasavada <ma...@gmail.com>.
I thought about it more. I feel that no matter how we refactor the code
(with or without KIP-383 integrated), ultimately the need of customizing
loading for keys and certs will still remain. Whenever that need arises we
might end up thinking about the solution suggested by our KIP-486. Hence
regardless of the other KIPs and configurations "if we do need to customize
loading of keys/certs, we will need the code changes suggested by this KIP".
Let me know what you guys think.
Harsha, we are working on changing the interfaces for key/trust store
loaders with Certificate and PrivateKey objects. Will probably be able to
update it later today or tomorrow.
Thanks
Maulin
On Thu, Aug 29, 2019 at 2:30 PM Maulin Vasavada <ma...@gmail.com>
wrote:
> On that, I actually looked at KIP-383 before briefly. However, that
> sounded like lot of changes suggested.
>
> One "key" thing we have to keep in mind is - IF we need lot of
> customization Kafka already allows you to use your SslProvider via
> ssl.providers or the changes done by KIP-492 and
> SSLContext.getInstance(protocol, provider) call allows us to return the
> SSLContext with "ALL" the details we would like to customize. Hence I am
> not sure that customization suggested by KIP-383 would be worth the effort.
> We also have similar SSLContext customization outside of Kafka.
>
> Thanks
> Maulin
>
>
>
>
>
> On Thu, Aug 29, 2019 at 12:47 PM Pellerin, Clement <
> Clement_Pellerin@ibi.com> wrote:
>
>> KIP-383 in its present form was vetoed because it was not possible to add
>> validation of custom properties in a future KIP. The solution to that is
>> the first proposal I wrote for KIP-383 which made the whole SslFactory
>> pluggable. That first solution was also vetoed hence the deadlock.
>>
>> Replacing the whole factory was a much nicer solution. It was vetoed
>> because doing this almost invariably meant the replacement lost all the
>> complex validation code in the default SslFactory.
>>
>> My current idea is to extract the validation code into another public API
>> that SslFactory would call. I did not look at the newly refactored code and
>> I did not study how to do this yet. KIP-383 was not popular at the time and
>> designing a new solution is a lot of work.
>>
>> Is there interest from 3 binding voters for something like this?
>>
>> -----Original Message-----
>> From: Rajini Sivaram [mailto:rajinisivaram@gmail.com]
>> Sent: Thursday, August 29, 2019 2:57 PM
>> To: dev
>> Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and
>> TrustStore
>>
>> Hi Maulin,
>>
>> In SSL scenarios, I imagine security providers introduced by KIP-492 are
>> likely to be most useful when you want to use third party providers. The
>> biggest advantage of the config from that KIP is that you don't need to
>> write much code to integrate existing security providers into Kafka
>> brokers
>> or clients. As I understand it, KIP-486 is a more convenient option for
>> the
>> specific problem of loading keystores/truststores differently. It can be
>> achieved in theory with KIP-492, but KIP-486 is a much simpler option for
>> this case.
>>
>> My concern about KIP-486 is that it introduces yet another interface into
>> our already complex security code, while only solving one particular use
>> case. Have you looked at
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-383%3A++Pluggable+interface+for+SSL+Factory
>> ?
>> The goal was to make
>> org.apache.kafka.common.security.ssl.SslEngineBuilder pluggable.
>> The code has already been refactored by Colin after that KIP was written,
>> making it easier to implement KIP-383. This should enable you to load your
>> keystores and truststores differently. Using a pluggable SslEngineBuilder
>> will also solve several other use cases at the same time. KIP-383 hasn't
>> been voted through yet, but perhaps you could take a look and we could
>> revive that instead if it solves your use case as well?
>>
>> Regards,
>>
>> Rajini
>>
>>
>> On Thu, Aug 29, 2019 at 6:42 PM Maulin Vasavada <
>> maulin.vasavada@gmail.com>
>> wrote:
>>
>> > Hi Harsha
>> >
>> > Thank you. Appreciate your time and support on this. Let me go back do
>> some
>> > more research and get back to you on the KeyStore interface part.
>> > Basically, if we return certs and keys in the interface then Kafka code
>> > will have to build KeyStore object - which is also reasonable.
>> >
>> > Thanks
>> > Maulin
>> >
>> > On Thu, Aug 29, 2019 at 10:01 AM Harsha Chintalapani <ka...@harsha.io>
>> > wrote:
>> >
>> > > Hi Maulin,
>> > > Use cases are clear now. I am +1 for moving
>> forward
>> > > with the discussions on having such configurable option for users. But
>> > the
>> > > interfaces is proposed doesn't look right to me. We are still talking
>> > about
>> > > keystore interfaces. Given keystore's are used as filebased way of
>> > > transporting certificates I am not sure it will help the rest of the
>> > > user-base.
>> > > In short, I am +1 on the KIP's motivation and only
>> have
>> > > questions around returning keystores instead of returning certs,
>> private
>> > > keys etc. . If others in the community are ok with such interface we
>> can
>> > > move forward.
>> > >
>> > > Thanks,
>> > > Harsha
>> > >
>> > >
>> > > On Wed, Aug 28, 2019 at 1:51 PM, Maulin Vasavada <
>> > > maulin.vasavada@gmail.com>
>> > > wrote:
>> > >
>> > > > Hi Harsha
>> > > >
>> > > > As we synced-up offline on this topic, we hope you don't have any
>> more
>> > > > clarifications that you are seeking. If that is the case, can you
>> > please
>> > > > help us move this forward and discuss what changes you would expect
>> on
>> > > the
>> > > > KIP design in order to make it valuable contribution?
>> > > >
>> > > > Just FYI - we verified our primary design change with the author of
>> > Sun's
>> > > > X509 Trustmanager implementation and the outcome is that what we are
>> > > > proposing makes sense at the heart of it - "Instead of writing
>> > > TrustManager
>> > > > just plugin the Trust store". We are open to discuss additional
>> changes
>> > > > that you/anybody else would like to see on the functionality
>> however.
>> > > >
>> > > > Thanks
>> > > > Maulin
>> > > >
>> > > > On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada <
>> > > maulin.vasavada@gmail.com>
>> > > > wrote:
>> > > >
>> > > > Hi Harsha
>> > > >
>> > > > Any response on my question? I feel this KIP is worth accommodating.
>> > Your
>> > > > help is much appreciated.
>> > > >
>> > > > Thanks
>> > > > Maulin
>> > > >
>> > > > On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada <
>> > maulin.vasavada@gmail.
>> > > > com> wrote:
>> > > >
>> > > > Hi Harsha
>> > > >
>> > > > I've examined the SPIFFE provider more and have one question -
>> > > >
>> > > > If SPIFFE didn't have a need to do checkSpiffeId() call at the below
>> > > > location, would you really still write the Provider? *OR* Would you
>> > just
>> > > > use TrustManagerFactory.init(KeyStore) signature to pass the
>> KeyStore
>> > > from
>> > > > set of certs returned by spiffeIdManager. getTrustedCerts()?
>> > > >
>> > > >
>> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>> > > > provider/CertificateUtils.java#L100
>> > > >
>> > > > /**
>> > > >
>> > > > * Validates that the SPIFFE ID is present and matches the SPIFFE ID
>> > > > configured in
>> > > > * the java.security property ssl.spiffe.accept
>> > > > *
>> > > > * If the authorized spiffe ids list is empty any spiffe id is
>> > authorized
>> > > > *
>> > > > * @param chain an array of X509Certificate that contains the Peer's
>> > SVID
>> > > > to be validated
>> > > > * @throws CertificateException when either the certificates doesn't
>> > have
>> > > a
>> > > > SPIFFE ID or the SPIFFE ID is not authorized
>> > > > */
>> > > > static void checkSpiffeId(X509Certificate[] chain) throws
>> > > > CertificateException {
>> > > >
>> > > > Thanks
>> > > > Maulin
>> > > >
>> > > > On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani <
>> kafka@harsha.io>
>> > > > wrote:
>> > > >
>> > > > Maulin,
>> > > > The code parts you are pointing are specific for Spiffe and if
>> > > > you are talking about validate method which uses PKIX check like any
>> > > other
>> > > > provider does.
>> > > > If you want to default to SunJSSE everywhere you can do so by
>> > delegating
>> > > > the calls in these methods to SunJSSE provider.
>> > > >
>> > > > TrustManagerFactory tmf = TrustManagerFactory
>> > > > .getInstance(TrustManagerFactory.getDefaultAlgorithm());and use
>> > > > tmf.chekServerTrusted()
>> > > > or use
>> > > > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
>> > > > TrustManagerFactory.html#getInstance(java.lang.String)if you want a
>> > > > specific provider.
>> > > >
>> > > > -Harsha
>> > > >
>> > > > On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada <
>> > maulin.vasavada@gmail.
>> > > > com>
>> > > > wrote:
>> > > >
>> > > > Okay, so I take that you guys agree that I have to write a 'custom'
>> > > > algorithm and a provider to make it work , correct?
>> > > >
>> > > > Now, for Harsha's comment "Here the 'Custom' Algorithm is not an
>> > > > implementation per say , ..." , I diagree. You can refer to https://
>> > > >
>> > > >
>> >
>> github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/
>> > > >
>> > > > SpiffeTrustManager.java#L91 <http://spiffetrustmanager.java/#L91>
>> and
>> > > >
>> > > >
>> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>> > > >
>> > > > provider/CertificateUtils.java#L100
>> > > >
>> > > > "that code" is the customization you have for the custom way to
>> check
>> > > > something on top of regular checks. That method is NOT doing custom
>> > > > truststore loading. It is validating/verifying something in the
>> > > >
>> > > > "custom"
>> > > >
>> > > > way with spiffeId.
>> > > > I bet that without that you won't have a need of the custom
>> algorithm
>> > > >
>> > > > in
>> > > >
>> > > > the first place.
>> > > >
>> > > > Let me know if you agree to this.
>> > > >
>> > > > Thanks
>> > > > Maulin
>> > > >
>> > > > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <mp...@gmail.com>
>> > > >
>> > > > wrote:
>> > > >
>> > > > Hi Maulin, thanks for the discussion. As Harsha pointed out, to use
>> the
>> > > > KIP492, you need to create a new provider, register a *new* custom
>> > > > algorithm for your keymanager and trustmanager factory
>> implementations.
>> > > > After this, the kafka server configuration can be done as given
>> below
>> > > >
>> > > > # Register the provider class with custom algorithm, say CUSTOM
>> > > >
>> > > > security.
>> > > >
>> > > > provider.classes=com.company.security.CustomProvider
>> > > > <http://provider.classes=com.company.security.customprovider/>
>> > > > <http://security.provider.classes
>> > > >
>> > > > =com.company.security.customprovider/>
>> > > >
>> > > > # Register the keymanager and trustmanager algorithms
>> > > > # These algorithms indicate that the Keymanager and Trustmanagers
>> > > > registered under the algorithm "CUSTOM" needs to be used
>> > > > ssl.trustmanager.algorithm=CUSTOM
>> > > > ssl.keymanager.algorithm=CUSTOM
>> > > >
>> > > > And the customprovider looks like this...
>> > > >
>> > > > public class CustomProvider extends Provider {
>> > > > public CustomProvider() {
>> > > > super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and TrustStore");
>> > > > super.put("KeyManagerFactory.CUSTOM", "customKeyManagerFactory");
>> > > > super.put("TrustManagerFactory.CUSTOM",
>> > > > "customTrustManagerFactory");
>> > > > }
>> > > > }
>> > > >
>> > > > The PR for this is in review and can be found here:
>> > > >
>> > > > https://github.com/
>> > > >
>> > > > apache/kafka/pull/7090
>> > > > This PR includes the fixed insertProviderAt function call.
>> > > >
>> > > > On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani <
>> kafka@harsha.io>
>> > > > wrote:
>> > > >
>> > > > Answers are in-line
>> > > >
>> > > > On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada <
>> > > >
>> > > > maulin.vasavada@gmail.
>> > > >
>> > > > com
>> > > >
>> > > > wrote:
>> > > >
>> > > > Hi Colin
>> > > >
>> > > > When I refer to "standard" or "custom" algorithms I am following
>> Java
>> > > > security Provider Terminology. You can refer to
>> > > > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
>> > > > StandardNames.html#TrustManagerFactory link I provided earlier in
>> the
>> > > > emails. It says PKIX is the default Algorithm for
>> TrustManagerFactory.
>> > > >
>> > > > 1. For SPIFFE, I am not sure why you are saying 'it does not
>> implement
>> > > > custom algorithms' because the following file clearly indicates
>> that it
>> > > > does use custom algorithm-
>> > > >
>> > > >
>> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>> > > >
>> > > > provider/SpiffeProvider.java#L17
>> > > >
>> > > > Algorithm value:
>> > > >
>> > > >
>> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>> > > >
>> > > > provider/SpiffeProviderConstants.java#L6
>> > > >
>> > > > @Harsha do you want to chime in since you use that provider?
>> > > >
>> > > > Here the "Custom" Algorithm is not an implementation per say ,
>> rather
>> > > >
>> > > > used
>> > > >
>> > > > to invoke the custom trust store factory and key manager factory.
>> You
>> > > >
>> > > > are
>> > > >
>> > > > not moving away from "standard" alogrithms that are available.
>> > > >
>> > > >
>> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>> > > >
>> > > > provider/SpiffeTrustManager.java
>> > > >
>> > > > As you can see it delegates all the calls of verification of
>> > > >
>> > > > certificate
>> > > >
>> > > > to
>> > > >
>> > > > the default implementation available.
>> > > > So in our implementation we still use PKIX to verify the certificate
>> > > > chain. So you are not losing anything here and Spiffe is not
>> > > >
>> > > > reimplementing
>> > > >
>> > > > the verification process.
>> > > >
>> > > > 2. I already mentioned in my 3rd point, in my previous post, why
>> using
>> > > >
>> > > > ssl.provider does NOT work. I updated KIP-486 in "rejected
>> > > >
>> > > > alternatives"
>> > > >
>> > > > also why ssl.provider does not work.
>> > > >
>> > > > As mentioned before , provider is the right way to go. I am still
>> not
>> > > > understanding the gap is.
>> > > > If I understand correctly your argument is , provider is going to
>> ask
>> > > >
>> > > > to
>> > > >
>> > > > implement a custom algorithm.
>> > > > My answer to that is , no you are not re-implementing the algorithm.
>> > > >
>> > > > Please
>> > > >
>> > > > check the above link , TrustManager implementation it delegates the
>> > > >
>> > > > calls
>> > > >
>> > > > back. There is no need to implement your own here.
>> > > >
>> > > > 3. Security.insertProviderAt() comments were based on assumption if
>> > > >
>> > > > KIP-492
>> > > >
>> > > > changes are done and we use that mechanism to configure providers
>> > > >
>> > > > instead
>> > > >
>> > > > of ssl.provider configuration.
>> > > >
>> > > > KIP-492 has patch available and going through review.
>> > > >
>> > > > Can you read my all the points, I mentioned in my previous post,
>> very
>> > > >
>> > > > carefully? I am covering all the aspects in explaining. I am open to
>> > > >
>> > > > still
>> > > >
>> > > > discuss more to clarify any doubts.
>> > > >
>> > > > "3. If we just use existing ssl.provider kafka configuration then
>> our
>> > > > provider will be used in SSLContext.getInstance(protocol, provider)
>> > > >
>> > > > call
>> > > >
>> > > > in
>> > > >
>> > > > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/>
>> <
>> > > > http://sslfactory.java/>
>> > > >
>> > > > and
>> > > >
>> > > > if our provider does not have
>> > > > implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we
>> > > >
>> > > > tested
>> > > >
>> > > > it). Example: In MyProvider sample above you see that I didn't add
>> > > > SSLContext.TLSv1 as
>> > > > "Service+Algorithm" and that didn't work for me. In SPIFFE provider
>> you
>> > > > don't have this challenge since you are planning to bypass
>> ssl.provider
>> > > >
>> > > > as
>> > > >
>> > > > you mention in the KIP-492."
>> > > >
>> > > > Yes here you need to pass the protocol that your
>> > > >
>> > > > KeyManager/TrustManager
>> > > >
>> > > > registered with and in no way its deviating from TLS RFC spec.
>> > > >
>> > > >
>> > https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/
>> > > >
>> > > > javax/net/ssl/SSLContext.java#L134
>> > > >
>> > > > My suggestion here is for you to implement a simple Security
>> Provider
>> > > >
>> > > > as
>> > > >
>> > > > you did before and register a Algorithm. You can use the existing
>> > > > implementation in SpiffeProvider and plug in changes where you need
>> to
>> > > > retrieve the certificates from by making RPC call.
>> > > >
>> > > > Run an end-to-end test with Kafka broker coming up with your
>> provider
>> > > > making calls to RPC call. You do need to pass the "custom algorithm"
>> > > >
>> > > > that
>> > > >
>> > > > you registered in your key manager to invoke the provider. I think
>> your
>> > > > concern is this approach is replacing the existing known
>> ciphersuites
>> > > >
>> > > > and
>> > > >
>> > > > certificate validation provided by java. Which its not.
>> > > >
>> > > > Now test the TLS connection you can do so via openssl -s_client
>> options
>> > > >
>> > > > to
>> > > >
>> > > > test if everything else is working.
>> > > >
>> > > > I am happy to share configs that we used if you are interested.
>> > > >
>> > > > Thanks,
>> > > > Harsha
>> > > >
>> > > > Thanks
>> > > > Maulin
>> > > >
>> > > > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <cm...@apache.org>
>> > > >
>> > > > wrote:
>> > > >
>> > > > Hi Maulin,
>> > > >
>> > > > A lot of JSSE providers don't implement custom algorithms. Spire is
>> a
>> > > >
>> > > > good
>> > > >
>> > > > example of a JSSE provider that doesn't, and yet is still useful to
>> > > >
>> > > > many
>> > > >
>> > > > people. Your JSSE provider can work fine even if it doesn't
>> implement a
>> > > > custom algorithm.
>> > > >
>> > > > Maybe I'm missing something, but I don't understand the discussion
>> of
>> > > > Security.insertProviderAt() that you included. SslEngineBuilder
>> doesn't
>> > > >
>> > > > use
>> > > >
>> > > > that API to get the security provider. Instead, it calls
>> > > > "SSLContext.getInstance(protocol, provider)", where provider is the
>> > > >
>> > > > name
>> > > >
>> > > > of the provider.
>> > > >
>> > > > best,
>> > > > Colin
>> > > >
>> > > > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
>> > > >
>> > > > On top of everything above I feel strongly to add the 4th point
>> which
>> > > >
>> > > > is
>> > > >
>> > > > based on Java APIs for TrustManagerFactory.init(KeyStore) (
>> > > >
>> > > > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
>> > > > TrustManagerFactory.html#init(java.security.KeyStore
>> > > > <http://java.security.keystore/>
>> > > > <http://java.security.keystore/>)
>> > > > )
>> > > >
>> > > > and KeyManagerFactory.init(KeyStore, char[]) (
>> > > >
>> > > >
>> > >
>> >
>> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory
>> > > > .
>> > > >
>> > > > html#init(java.security.KeyStore <http://java.security.keystore/>
>> > > <http://
>> > > > java.security.keystore/
>> > > > ,%20char[])
>> > > >
>> > > > ).
>> > > >
>> > > > 4. The above APIs are intended to support providing "trust/key
>> > > >
>> > > > material"
>> > > >
>> > > > from the user without having to write their own
>> > > >
>> > > > TrustManager/KeyManagers.
>> > > >
>> > > > To quote from the TrustManagerFactory.init()'s documentation
>> > > >
>> > > > "Initializes
>> > > >
>> > > > this factory with a source of certificate authorities and related
>> trust
>> > > > material."
>> > > > To quote from the KeyManagerFactory.init()'s documentation
>> "Initializes
>> > > > this factory with a source of key material."
>> > > >
>> > > > Based on this it is clear that there is a flexibility provided by
>> Java
>> > > >
>> > > > to
>> > > >
>> > > > to enable developers to provide the required trust/key material
>> loaded
>> > > >
>> > > > from
>> > > >
>> > > > "anywhere" without requiring them to write custom provider OR
>> trust/key
>> > > > managers. This same flexibility is reflected in Kafka code also
>> where
>> > > >
>> > > > it
>> > > >
>> > > > loads the trust/keys from a local file and doesn't require writing a
>> > > > Provider necessarily. If we do NOT have a custom algorithm, it makes
>> > > >
>> > > > less
>> > > >
>> > > > sense to write a Provider.
>> > > >
>> > > > Thanks
>> > > > Maulin
>> > > >
>> > > > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
>> > > >
>> > > > maulin.vasavada@gmail.com>
>> > > >
>> > > > wrote:
>> > > >
>> > > > Hi Harsha/Colin
>> > > >
>> > > > I did the sample with a custom Provider for TrustStoreManager and
>> tried
>> > > > using ssl.provider Kafka config AND the way KIP-492 is suggesting
>> (by
>> > > > adding Provider programmatically instead of relying on
>> > > >
>> > > > ssl.provider+java.
>> > > >
>> > > > security. The below sample is followed by my detailed findings. I'll
>> > > > appreciate if you can go through it carefully and see
>> > > >
>> > > > if you
>> > > >
>> > > > see my point.
>> > > >
>> > > > package providertest;
>> > > >
>> > > > import java.security.Provider <http://java.security.provider/>
>> > <http://
>> > > > java.security.provider/>
>> > > >
>> > > > <http://
>> > > >
>> > > > java.security.provider/>;
>> > > >
>> > > > public class MyProvider extends Provider {
>> > > >
>> > > > private static final String name = "MyProvider"; private static
>> double
>> > > > version = 1.0d;
>> > > > private static String info = "Maulin's SSL Provider v"+version;
>> > > >
>> > > > public MyProvider() {
>> > > > super(name, version, info);
>> > > > this.put("TrustManagerFactory.PKIX",
>> > > >
>> > > > "providertest.MyTrustManagerFactory");
>> > > >
>> > > > }
>> > > > }
>> > > >
>> > > > *Details:*
>> > > >
>> > > > KIP-492 documents that it will use Security.addProvider() assuming
>> it
>> > > >
>> > > > will
>> > > >
>> > > > add it as position '0' which is not a correct assumption. The
>> > > > addProvider()'s documentation says it will add it to the last
>> available
>> > > > position. You may want to correct that to say
>> > > > Security.insertProviderAt(provider, 1).
>> > > >
>> > > > Now coming back to our specific discussion,
>> > > >
>> > > > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add
>> > > >
>> > > > that
>> > > >
>> > > > provider in the provider list via Security.addProvider() the
>> position
>> > > >
>> > > > where
>> > > >
>> > > > it gets added doesn't matter (even if you don't end up adding it as
>> > > >
>> > > > first
>> > > >
>> > > > entry) since that is the ONLY provider for SPIFFE specific algorithm
>> > > >
>> > > > you
>> > > >
>> > > > might have.
>> > > >
>> > > > We do *not* have custom algorithm for Key/Trust StoreMangers. Which
>> > > >
>> > > > means
>> > > >
>> > > > we have to use X509, PKIX etc "Standard Algorithms" ((
>> > > >
>> > > > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
>> > > > StandardNames.html
>> > > > ))
>> > > >
>> > > > in our provider to override the TrustStoreManager (see my sample
>> code)
>> > > >
>> > > > and
>> > > >
>> > > > KeyStoreManger and KeyManager. This creates another challenge
>> > > >
>> > > > mentioned in
>> > > >
>> > > > the below point.
>> > > >
>> > > > 2. In order to make our Provider for loading custom TrustStore
>> work, we
>> > > > have to add the provider as 'first' in the list since there are
>> others
>> > > >
>> > > > with
>> > > >
>> > > > the same algorithm.
>> > > >
>> > > > However, the programatic way of adding provider
>> > > > (Security.insertProviderAt()) is *not* deterministic for ordering
>> since
>> > > > different code can call the method for a different provider and
>> > > >
>> > > > depending
>> > > >
>> > > > upon the order of the call our provider can be first or pushed down
>> the
>> > > > list. This can happen very well in any client application using
>> Kafka.
>> > > >
>> > > > This
>> > > >
>> > > > is specially problematic for a case when you want to guarantee order
>> > > >
>> > > > for a
>> > > >
>> > > > Provider having "Standard Algorithms".
>> > > >
>> > > > If we add our provider in java.security file that definitely
>> guarantees
>> > > > the order(unless somebody calls removeProvider() which is less
>> > > >
>> > > > likely). But
>> > > >
>> > > > if we add our provider in java.security file it will defeat the
>> > > >
>> > > > purpose of
>> > > >
>> > > > the KIP-492.
>> > > >
>> > > > In the gist - Apache Kafka must not rely on "un-deterministic"
>> method
>> > > >
>> > > > to
>> > > >
>> > > > rely on Provider ordering.
>> > > >
>> > > > 3. If we just use existing ssl.provider kafka configuration then our
>> > > > provider will be used in SSLContext.getInstance(protocol, provider)
>> > > >
>> > > > call in
>> > > >
>> > > > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/>
>> <
>> > > > http://sslfactory.java/>
>> > > >
>> > > > and
>> > > >
>> > > > if our provider does not have implementation for
>> > > > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks
>> > > >
>> > > > (we
>> > > >
>> > > > tested it). Example:
>> > > >
>> > > > In
>> > > >
>> > > > MyProvider sample above you see that I didn't add SSLContext.TLSv1
>> as
>> > > > "Service+Algorithm" and that didn't work for me. In SPIFFE provider
>> you
>> > > > don't have this challenge since you are planning to bypass
>> > > >
>> > > > ssl.provider as
>> > > >
>> > > > you mention in the KIP-492.
>> > > >
>> > > > *Overall summary,*
>> > > >
>> > > > 1. Any provider based mechanisms- a) existing ssl.provider and
>> > > >
>> > > > b)KIP-492,
>> > > >
>> > > > for loading key/trust store using "Standard Algorithms" do not work
>> > > >
>> > > > 2. Approach suggested in our KIP-486 works without any issue and it
>> is
>> > > > *not* our context specific solve
>> > > >
>> > > > 3. Based on above we feel KIP-492 and KIP-486 are complimentary
>> changes
>> > > > and not contradicting or redundent.
>> > > >
>> > > > If you want we can do a joint session somehow to walk through the
>> > > >
>> > > > sample I
>> > > >
>> > > > have and various experiments I did. I would encourage you to do
>> similar
>> > > > exercise by writing a Provider for "Standard Algorithm" for
>> > > > TrustStoreManager (like our needs) and see what you find since only
>> > > >
>> > > > writing
>> > > >
>> > > > samples can bring out the complexity/challenges we face.
>> > > >
>> > > > Thanks
>> > > > Maulin
>> > > >
>> > > > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <
>> > > >
>> > > > maulin.vasavada@gmail.
>> > > >
>> > > > com> wrote:
>> > > >
>> > > > Just to update - still working on it. Get to work only on and off on
>> > > >
>> > > > it :(
>> > > >
>> > > > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
>> > > >
>> > > > maulin.vasavada@gmail.com>
>> > > >
>> > > > wrote:
>> > > >
>> > > > Hi Harsha
>> > > >
>> > > > Let me try to write samples and will let you know.
>> > > >
>> > > > Thanks
>> > > > Maulin
>> > > >
>> > > > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <ha...@gmail.com>
>> > > >
>> > > > wrote:
>> > > >
>> > > > Hi Maulin,
>> > > > With java security providers can be as custom you would
>> > > >
>> > > > like
>> > > >
>> > > > it to
>> > > > be. If you only want to to implement a custom way of loading the
>> > > >
>> > > > keystore
>> > > >
>> > > > and truststore and not implement any protocol/encryption handling
>> you
>> > > >
>> > > > can
>> > > >
>> > > > leave them empty and no need to implement. Have you looked into the
>> > > >
>> > > > links I
>> > > >
>> > > > pasted before?
>> > > >
>> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>> > > >
>> > > >
>> > >
>> >
>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
>> > > >
>> > > > java
>> > > >
>> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>> > > > spiffe-security-provider/src/main/java/spiffe/api/provider/
>> > > > SpiffeTrustManager.java <http://spiffetrustmanager.java/> <http://
>> > > > spiffetrustmanager.java/>
>> > > >
>> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>> > > >
>> > > >
>> > >
>> >
>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>> > > >
>> > > > java
>> > > >
>> > > > Can you please tell me which methods are too complex in above to
>> > > >
>> > > > implement
>> > > >
>> > > > or unnecessary? You are changing anything in SSL/TLS implementations
>> > > > provided by
>> > > >
>> > > > All of the implementations delegating the checks to the default
>> > > > implementation anyway.
>> > > > Spire agent is an example, its nothing but a GRPC server listening
>> > > >
>> > > > on a
>> > > >
>> > > > unix domain socket . Above code is making a RPC call to the local
>> > > >
>> > > > daemon
>> > > >
>> > > > to
>> > > > get the certificate and keys. The mechanics are pretty much same as
>> > > >
>> > > > what
>> > > >
>> > > > you are asking for.
>> > > >
>> > > > Thanks,
>> > > > Harsha
>> > > >
>> > > > On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
>> > > >
>> > > > maulin.vasavada@gmail.com>
>> > > >
>> > > > wrote:
>> > > >
>> > > > Imagine a scenario like - We know we have a custom KMS and as a
>> > > >
>> > > > Kafka
>> > > >
>> > > > owner
>> > > >
>> > > > we want to comply to using that KMS source to load keys/certs. As
>> > > >
>> > > > a
>> > > >
>> > > > Kafka
>> > > >
>> > > > owner we know how to integrate with KMS but doesn't necessarily
>> > > >
>> > > > have
>> > > >
>> > > > to
>> > > >
>> > > > know anything about cipher suites, algorithms, and SSL/TLS
>> > > >
>> > > > implementation.
>> > > >
>> > > > Going the Provider way requires to know lot more than we should,
>> > > >
>> > > > isn't it?
>> > > >
>> > > > Not that we would have concern/shy-away knowing those details -
>> > > >
>> > > > but
>> > > >
>> > > > if we
>> > > >
>> > > > don't have to - why should we?
>> > > >
>> > > > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
>> > > >
>> > > > maulin.vasavada@gmail.com>
>> > > >
>> > > > wrote:
>> > > >
>> > > > Hi Harsha
>> > > >
>> > > > We don't have spire (or similar) agents and we do not have
>> > > >
>> > > > keys/certs
>> > > >
>> > > > locally on any brokers. To elaborate more on my previous email,
>> > > >
>> > > > I agree that Java security Providers are used in much broader
>> > > >
>> > > > sense
>> > > >
>> > > > - to
>> > > >
>> > > > have a particular implementation of an algorithm, use specific
>> > > >
>> > > > cipher
>> > > >
>> > > > suites for SSL , OR in our current team's case have a
>> > > >
>> > > > particular
>> > > >
>> > > > way to
>> > > >
>> > > > leverage pre-generated SSL sessions. However, the scope of our
>> > > >
>> > > > KIP
>> > > >
>> > > > (486)
>> > > >
>> > > > is
>> > > >
>> > > > much restricted than that. We merely intend to provide a custom
>> > > > keystore/truststore for our SSL connections and not really worry
>> > > >
>> > > > about
>> > > >
>> > > > underlying specific SSL/TLS implementation. This simplifies it
>> > > >
>> > > > a
>> > > >
>> > > > lot for
>> > > >
>> > > > us to keep the concerns separate and clear.
>> > > >
>> > > > I feel our approach is more complimentary such that it allows
>> > > >
>> > > > for
>> > > >
>> > > > using
>> > > >
>> > > > keystores of choice while retaining the flexibility to use any
>> > > > underlying/available Provider for actually making the SSL call.
>> > > >
>> > > > We agree with KIP-492's approach based on Providers (and Java's
>> > > > recommendation), but also strongly believe that our approach can
>> > > >
>> > > > compliment
>> > > >
>> > > > it very effectively for reasons explained above.
>> > > >
>> > > > Thanks
>> > > > Maulin
>> > > >
>> > > > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <
>> > > >
>> > > > kafka@harsha.io
>> > > >
>> > > > wrote:
>> > > >
>> > > > Hi Maulin,
>> > > >
>> > > > On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
>> > > >
>> > > > maulin.vasavada@gmail.
>> > > >
>> > > > com>
>> > > > wrote:
>> > > >
>> > > > Hi Harsha
>> > > >
>> > > > The reason we rejected the SslProvider route is that - we
>> > > >
>> > > > only
>> > > >
>> > > > needed
>> > > >
>> > > > a
>> > > >
>> > > > custom way to load keys/certs. Not touch any policy that
>> > > >
>> > > > existing
>> > > >
>> > > > Providers
>> > > >
>> > > > govern like SunJSSE Provider.
>> > > >
>> > > > We have exactly the same requirements to load certs and keys
>> > > >
>> > > > through
>> > > >
>> > > > spire
>> > > >
>> > > > agent. We used security.provider to do that exactly. I am not
>> > > >
>> > > > sure
>> > > >
>> > > > why
>> > > >
>> > > > you
>> > > >
>> > > > would be modifying any policies provided by default SunJSSE
>> > > >
>> > > > provider.
>> > > >
>> > > > Can
>> > > >
>> > > > you give me an example of having custom provider that will
>> > > >
>> > > > override an
>> > > >
>> > > > existing policy in SunJSSE provider.
>> > > >
>> > > > As pointed out earlier, this kip
>> > > >
>> > > > https://cwiki.apache.org/confluence/display/KAFKA/
>> > > > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>> > > >
>> > > > allows
>> > > > you to load security.provider through config.
>> > > > Take a look at the examples I gave before
>> > > >
>> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>> > > >
>> > > >
>> > >
>> >
>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>> > > >
>> > > > java
>> > > >
>> > > > It registers KeyManagerFactory and TrustManagerFactory and
>> > > >
>> > > > Keystore
>> > > >
>> > > > algorithm.
>> > > > Implement your custom way of loading Keystore in here
>> > > >
>> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>> > > >
>> > > >
>> > >
>> >
>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
>> > > >
>> > > > java
>> > > >
>> > > > and Trust manager like here
>> > > >
>> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>> > > > spiffe-security-provider/src/main/java/spiffe/api/provider/
>> > > > SpiffeTrustManager.java <http://spiffetrustmanager.java/> <http://
>> > > > spiffetrustmanager.java/>
>> > > >
>> > > > In your Kafka client you can set the security.provider to your
>> > > >
>> > > > custom
>> > > >
>> > > > implementation and with this fix
>> > > > https://issues.apache.org/jira/browse/KAFKA-8191 you can set
>> > > > keyManagerAlgorigthm and trustManagerAlgorithm configs.
>> > > >
>> > > > All of this is in your clients and broker side and do not need
>> > > >
>> > > > to
>> > > >
>> > > > touch
>> > > >
>> > > > any
>> > > > policy changes at JVM level. You'll register the providers in
>> > > >
>> > > > the
>> > > >
>> > > > priority
>> > > >
>> > > > order and can still have SunJSSE provider and have your custom
>> > > >
>> > > > provider
>> > > >
>> > > > to
>> > > >
>> > > > implement the key and trust managers.
>> > > >
>> > > > The ask here is different than KIP-492. We don't have any need
>> > > >
>> > > > to
>> > > >
>> > > > modify/specify the algorithm parameter. Does that make sense?
>> > > >
>> > > > The ask in KIP is introducing new interfaces where the KIP's
>> > > > goal/motivation can be achieved through the security.provider
>> > > >
>> > > > and
>> > > >
>> > > > we
>> > > >
>> > > > worked
>> > > > on similar goal without touching any Keystore or Truststore
>> > > >
>> > > > interfaces.
>> > > >
>> > > > My advise is against changing or introducing new interfaces
>> > > >
>> > > > when
>> > > >
>> > > > it can
>> > > >
>> > > > work through security.provider.
>> > > >
>> > > > Thanks,
>> > > > Harsha
>> > > >
>> > > > Thanks
>> > > >
>> > > > Maulin
>> > > >
>> > > > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
>> > > >
>> > > > kafka@harsha.io>
>> > > >
>> > > > wrote:
>> > > >
>> > > > In your KIP you added security. provider as rejected
>> > > >
>> > > > alternative
>> > > >
>> > > > and
>> > > >
>> > > > specified "its not the correct way". Do you mind explaining
>> > > >
>> > > > why
>> > > >
>> > > > its
>> > > >
>> > > > not? I
>> > > >
>> > > > didn't find any evidence in Java docs to say so. Contrary to
>> > > >
>> > > > your
>> > > >
>> > > > statement
>> > > >
>> > > > it does say in the java docs
>> > > > " However, please note that a provider can be used to
>> > > >
>> > > > implement
>> > > >
>> > > > any
>> > > >
>> > > > security service in Java that uses a pluggable architecture
>> > > >
>> > > > with
>> > > >
>> > > > a
>> > > >
>> > > > choice
>> > > >
>> > > > of implementations that fit underneath."
>> > > >
>> > > > Java Security Providers have been used by other projects to
>> > > >
>> > > > provide
>> > > >
>> > > > such
>> > > >
>> > > > integration . I am not sure if you looked into Spiffe
>> > > >
>> > > > project to
>> > > >
>> > > > efficiently distribute certificates but here is an example of
>> > > >
>> > > > Java
>> > > >
>> > > > provider
>> > > >
>> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>> > > >
>> > > >
>> > >
>> >
>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>> > > >
>> > > > java which
>> > > > obtains certificates from local daemons.
>> > > > These integrations are being used in Tomcat, Jetty etc.. We
>> > > >
>> > > > are
>> > > >
>> > > > also
>> > > >
>> > > > using
>> > > >
>> > > > Security provider to do the same in our Kafka clusters. So
>> > > >
>> > > > unless I
>> > > >
>> > > > see
>> > > >
>> > > > more evidence why security.provider doesn't work for you
>> > > >
>> > > > adding
>> > > >
>> > > > new
>> > > >
>> > > > interfaces while there exists more cleaner way of achieving
>> > > >
>> > > > the
>> > > >
>> > > > goals
>> > > >
>> > > > of
>> > > >
>> > > > this KIP is unnecessary and breaks the well known security
>> > > >
>> > > > interfaces
>> > > >
>> > > > provided by Java itself.
>> > > >
>> > > > Thanks,
>> > > > Harsha
>> > > >
>> > > > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
>> > > >
>> > > > kafka@harsha.io
>> > > >
>> > > > wrote:
>> > > >
>> > > > Hi Maulin,
>> > > > Not sure if you looked at my previous replies. This
>> > > >
>> > > > changes
>> > > >
>> > > > are not required as there is already security Provider to do
>> > > >
>> > > > what you
>> > > >
>> > > > are
>> > > >
>> > > > proposing. This KIP
>> > > >
>> > > > https://cwiki.apache.org/confluence/display/KAFKA/
>> > > >
>> > > > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>> > > >
>> > > > also
>> > > >
>> > > > addresses easy registration of such providers.
>> > > >
>> > > > Thanks,
>> > > > Harsha
>> > > >
>> > > > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
>> > > >
>> > > > <maulin.vasavada@gmail.
>> > > >
>> > > > com> wrote:
>> > > >
>> > > > Bump! Can somebody please review this?
>> > > >
>> > > > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
>> > > >
>> > > > maulin.vasavada@gmail.com>
>> > > >
>> > > > wrote:
>> > > >
>> > > > Bump! Can somebody please review this?
>> > > >
>> > > > --
>> > > > Thanks,
>> > > > M.Sai Sandeep
>> > > >
>> > > >
>> > >
>> >
>>
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Maulin Vasavada <ma...@gmail.com>.
On that, I actually looked at KIP-383 before briefly. However, that
sounded like lot of changes suggested.
One "key" thing we have to keep in mind is - IF we need lot of
customization Kafka already allows you to use your SslProvider via
ssl.providers or the changes done by KIP-492 and
SSLContext.getInstance(protocol, provider) call allows us to return the
SSLContext with "ALL" the details we would like to customize. Hence I am
not sure that customization suggested by KIP-383 would be worth the effort.
We also have similar SSLContext customization outside of Kafka.
Thanks
Maulin
On Thu, Aug 29, 2019 at 12:47 PM Pellerin, Clement <Cl...@ibi.com>
wrote:
> KIP-383 in its present form was vetoed because it was not possible to add
> validation of custom properties in a future KIP. The solution to that is
> the first proposal I wrote for KIP-383 which made the whole SslFactory
> pluggable. That first solution was also vetoed hence the deadlock.
>
> Replacing the whole factory was a much nicer solution. It was vetoed
> because doing this almost invariably meant the replacement lost all the
> complex validation code in the default SslFactory.
>
> My current idea is to extract the validation code into another public API
> that SslFactory would call. I did not look at the newly refactored code and
> I did not study how to do this yet. KIP-383 was not popular at the time and
> designing a new solution is a lot of work.
>
> Is there interest from 3 binding voters for something like this?
>
> -----Original Message-----
> From: Rajini Sivaram [mailto:rajinisivaram@gmail.com]
> Sent: Thursday, August 29, 2019 2:57 PM
> To: dev
> Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and
> TrustStore
>
> Hi Maulin,
>
> In SSL scenarios, I imagine security providers introduced by KIP-492 are
> likely to be most useful when you want to use third party providers. The
> biggest advantage of the config from that KIP is that you don't need to
> write much code to integrate existing security providers into Kafka brokers
> or clients. As I understand it, KIP-486 is a more convenient option for the
> specific problem of loading keystores/truststores differently. It can be
> achieved in theory with KIP-492, but KIP-486 is a much simpler option for
> this case.
>
> My concern about KIP-486 is that it introduces yet another interface into
> our already complex security code, while only solving one particular use
> case. Have you looked at
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-383%3A++Pluggable+interface+for+SSL+Factory
> ?
> The goal was to make
> org.apache.kafka.common.security.ssl.SslEngineBuilder pluggable.
> The code has already been refactored by Colin after that KIP was written,
> making it easier to implement KIP-383. This should enable you to load your
> keystores and truststores differently. Using a pluggable SslEngineBuilder
> will also solve several other use cases at the same time. KIP-383 hasn't
> been voted through yet, but perhaps you could take a look and we could
> revive that instead if it solves your use case as well?
>
> Regards,
>
> Rajini
>
>
> On Thu, Aug 29, 2019 at 6:42 PM Maulin Vasavada <maulin.vasavada@gmail.com
> >
> wrote:
>
> > Hi Harsha
> >
> > Thank you. Appreciate your time and support on this. Let me go back do
> some
> > more research and get back to you on the KeyStore interface part.
> > Basically, if we return certs and keys in the interface then Kafka code
> > will have to build KeyStore object - which is also reasonable.
> >
> > Thanks
> > Maulin
> >
> > On Thu, Aug 29, 2019 at 10:01 AM Harsha Chintalapani <ka...@harsha.io>
> > wrote:
> >
> > > Hi Maulin,
> > > Use cases are clear now. I am +1 for moving forward
> > > with the discussions on having such configurable option for users. But
> > the
> > > interfaces is proposed doesn't look right to me. We are still talking
> > about
> > > keystore interfaces. Given keystore's are used as filebased way of
> > > transporting certificates I am not sure it will help the rest of the
> > > user-base.
> > > In short, I am +1 on the KIP's motivation and only
> have
> > > questions around returning keystores instead of returning certs,
> private
> > > keys etc. . If others in the community are ok with such interface we
> can
> > > move forward.
> > >
> > > Thanks,
> > > Harsha
> > >
> > >
> > > On Wed, Aug 28, 2019 at 1:51 PM, Maulin Vasavada <
> > > maulin.vasavada@gmail.com>
> > > wrote:
> > >
> > > > Hi Harsha
> > > >
> > > > As we synced-up offline on this topic, we hope you don't have any
> more
> > > > clarifications that you are seeking. If that is the case, can you
> > please
> > > > help us move this forward and discuss what changes you would expect
> on
> > > the
> > > > KIP design in order to make it valuable contribution?
> > > >
> > > > Just FYI - we verified our primary design change with the author of
> > Sun's
> > > > X509 Trustmanager implementation and the outcome is that what we are
> > > > proposing makes sense at the heart of it - "Instead of writing
> > > TrustManager
> > > > just plugin the Trust store". We are open to discuss additional
> changes
> > > > that you/anybody else would like to see on the functionality however.
> > > >
> > > > Thanks
> > > > Maulin
> > > >
> > > > On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada <
> > > maulin.vasavada@gmail.com>
> > > > wrote:
> > > >
> > > > Hi Harsha
> > > >
> > > > Any response on my question? I feel this KIP is worth accommodating.
> > Your
> > > > help is much appreciated.
> > > >
> > > > Thanks
> > > > Maulin
> > > >
> > > > On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada <
> > maulin.vasavada@gmail.
> > > > com> wrote:
> > > >
> > > > Hi Harsha
> > > >
> > > > I've examined the SPIFFE provider more and have one question -
> > > >
> > > > If SPIFFE didn't have a need to do checkSpiffeId() call at the below
> > > > location, would you really still write the Provider? *OR* Would you
> > just
> > > > use TrustManagerFactory.init(KeyStore) signature to pass the KeyStore
> > > from
> > > > set of certs returned by spiffeIdManager. getTrustedCerts()?
> > > >
> > > >
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > > provider/CertificateUtils.java#L100
> > > >
> > > > /**
> > > >
> > > > * Validates that the SPIFFE ID is present and matches the SPIFFE ID
> > > > configured in
> > > > * the java.security property ssl.spiffe.accept
> > > > *
> > > > * If the authorized spiffe ids list is empty any spiffe id is
> > authorized
> > > > *
> > > > * @param chain an array of X509Certificate that contains the Peer's
> > SVID
> > > > to be validated
> > > > * @throws CertificateException when either the certificates doesn't
> > have
> > > a
> > > > SPIFFE ID or the SPIFFE ID is not authorized
> > > > */
> > > > static void checkSpiffeId(X509Certificate[] chain) throws
> > > > CertificateException {
> > > >
> > > > Thanks
> > > > Maulin
> > > >
> > > > On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani <kafka@harsha.io
> >
> > > > wrote:
> > > >
> > > > Maulin,
> > > > The code parts you are pointing are specific for Spiffe and if
> > > > you are talking about validate method which uses PKIX check like any
> > > other
> > > > provider does.
> > > > If you want to default to SunJSSE everywhere you can do so by
> > delegating
> > > > the calls in these methods to SunJSSE provider.
> > > >
> > > > TrustManagerFactory tmf = TrustManagerFactory
> > > > .getInstance(TrustManagerFactory.getDefaultAlgorithm());and use
> > > > tmf.chekServerTrusted()
> > > > or use
> > > > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> > > > TrustManagerFactory.html#getInstance(java.lang.String)if you want a
> > > > specific provider.
> > > >
> > > > -Harsha
> > > >
> > > > On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada <
> > maulin.vasavada@gmail.
> > > > com>
> > > > wrote:
> > > >
> > > > Okay, so I take that you guys agree that I have to write a 'custom'
> > > > algorithm and a provider to make it work , correct?
> > > >
> > > > Now, for Harsha's comment "Here the 'Custom' Algorithm is not an
> > > > implementation per say , ..." , I diagree. You can refer to https://
> > > >
> > > >
> > github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/
> > > >
> > > > SpiffeTrustManager.java#L91 <http://spiffetrustmanager.java/#L91>
> and
> > > >
> > > >
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > >
> > > > provider/CertificateUtils.java#L100
> > > >
> > > > "that code" is the customization you have for the custom way to check
> > > > something on top of regular checks. That method is NOT doing custom
> > > > truststore loading. It is validating/verifying something in the
> > > >
> > > > "custom"
> > > >
> > > > way with spiffeId.
> > > > I bet that without that you won't have a need of the custom algorithm
> > > >
> > > > in
> > > >
> > > > the first place.
> > > >
> > > > Let me know if you agree to this.
> > > >
> > > > Thanks
> > > > Maulin
> > > >
> > > > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <mp...@gmail.com>
> > > >
> > > > wrote:
> > > >
> > > > Hi Maulin, thanks for the discussion. As Harsha pointed out, to use
> the
> > > > KIP492, you need to create a new provider, register a *new* custom
> > > > algorithm for your keymanager and trustmanager factory
> implementations.
> > > > After this, the kafka server configuration can be done as given below
> > > >
> > > > # Register the provider class with custom algorithm, say CUSTOM
> > > >
> > > > security.
> > > >
> > > > provider.classes=com.company.security.CustomProvider
> > > > <http://provider.classes=com.company.security.customprovider/>
> > > > <http://security.provider.classes
> > > >
> > > > =com.company.security.customprovider/>
> > > >
> > > > # Register the keymanager and trustmanager algorithms
> > > > # These algorithms indicate that the Keymanager and Trustmanagers
> > > > registered under the algorithm "CUSTOM" needs to be used
> > > > ssl.trustmanager.algorithm=CUSTOM
> > > > ssl.keymanager.algorithm=CUSTOM
> > > >
> > > > And the customprovider looks like this...
> > > >
> > > > public class CustomProvider extends Provider {
> > > > public CustomProvider() {
> > > > super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and TrustStore");
> > > > super.put("KeyManagerFactory.CUSTOM", "customKeyManagerFactory");
> > > > super.put("TrustManagerFactory.CUSTOM",
> > > > "customTrustManagerFactory");
> > > > }
> > > > }
> > > >
> > > > The PR for this is in review and can be found here:
> > > >
> > > > https://github.com/
> > > >
> > > > apache/kafka/pull/7090
> > > > This PR includes the fixed insertProviderAt function call.
> > > >
> > > > On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani <kafka@harsha.io
> >
> > > > wrote:
> > > >
> > > > Answers are in-line
> > > >
> > > > On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada <
> > > >
> > > > maulin.vasavada@gmail.
> > > >
> > > > com
> > > >
> > > > wrote:
> > > >
> > > > Hi Colin
> > > >
> > > > When I refer to "standard" or "custom" algorithms I am following Java
> > > > security Provider Terminology. You can refer to
> > > > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> > > > StandardNames.html#TrustManagerFactory link I provided earlier in the
> > > > emails. It says PKIX is the default Algorithm for
> TrustManagerFactory.
> > > >
> > > > 1. For SPIFFE, I am not sure why you are saying 'it does not
> implement
> > > > custom algorithms' because the following file clearly indicates that
> it
> > > > does use custom algorithm-
> > > >
> > > >
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > >
> > > > provider/SpiffeProvider.java#L17
> > > >
> > > > Algorithm value:
> > > >
> > > >
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > >
> > > > provider/SpiffeProviderConstants.java#L6
> > > >
> > > > @Harsha do you want to chime in since you use that provider?
> > > >
> > > > Here the "Custom" Algorithm is not an implementation per say , rather
> > > >
> > > > used
> > > >
> > > > to invoke the custom trust store factory and key manager factory. You
> > > >
> > > > are
> > > >
> > > > not moving away from "standard" alogrithms that are available.
> > > >
> > > >
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > >
> > > > provider/SpiffeTrustManager.java
> > > >
> > > > As you can see it delegates all the calls of verification of
> > > >
> > > > certificate
> > > >
> > > > to
> > > >
> > > > the default implementation available.
> > > > So in our implementation we still use PKIX to verify the certificate
> > > > chain. So you are not losing anything here and Spiffe is not
> > > >
> > > > reimplementing
> > > >
> > > > the verification process.
> > > >
> > > > 2. I already mentioned in my 3rd point, in my previous post, why
> using
> > > >
> > > > ssl.provider does NOT work. I updated KIP-486 in "rejected
> > > >
> > > > alternatives"
> > > >
> > > > also why ssl.provider does not work.
> > > >
> > > > As mentioned before , provider is the right way to go. I am still not
> > > > understanding the gap is.
> > > > If I understand correctly your argument is , provider is going to ask
> > > >
> > > > to
> > > >
> > > > implement a custom algorithm.
> > > > My answer to that is , no you are not re-implementing the algorithm.
> > > >
> > > > Please
> > > >
> > > > check the above link , TrustManager implementation it delegates the
> > > >
> > > > calls
> > > >
> > > > back. There is no need to implement your own here.
> > > >
> > > > 3. Security.insertProviderAt() comments were based on assumption if
> > > >
> > > > KIP-492
> > > >
> > > > changes are done and we use that mechanism to configure providers
> > > >
> > > > instead
> > > >
> > > > of ssl.provider configuration.
> > > >
> > > > KIP-492 has patch available and going through review.
> > > >
> > > > Can you read my all the points, I mentioned in my previous post, very
> > > >
> > > > carefully? I am covering all the aspects in explaining. I am open to
> > > >
> > > > still
> > > >
> > > > discuss more to clarify any doubts.
> > > >
> > > > "3. If we just use existing ssl.provider kafka configuration then our
> > > > provider will be used in SSLContext.getInstance(protocol, provider)
> > > >
> > > > call
> > > >
> > > > in
> > > >
> > > > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/>
> <
> > > > http://sslfactory.java/>
> > > >
> > > > and
> > > >
> > > > if our provider does not have
> > > > implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we
> > > >
> > > > tested
> > > >
> > > > it). Example: In MyProvider sample above you see that I didn't add
> > > > SSLContext.TLSv1 as
> > > > "Service+Algorithm" and that didn't work for me. In SPIFFE provider
> you
> > > > don't have this challenge since you are planning to bypass
> ssl.provider
> > > >
> > > > as
> > > >
> > > > you mention in the KIP-492."
> > > >
> > > > Yes here you need to pass the protocol that your
> > > >
> > > > KeyManager/TrustManager
> > > >
> > > > registered with and in no way its deviating from TLS RFC spec.
> > > >
> > > >
> > https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/
> > > >
> > > > javax/net/ssl/SSLContext.java#L134
> > > >
> > > > My suggestion here is for you to implement a simple Security Provider
> > > >
> > > > as
> > > >
> > > > you did before and register a Algorithm. You can use the existing
> > > > implementation in SpiffeProvider and plug in changes where you need
> to
> > > > retrieve the certificates from by making RPC call.
> > > >
> > > > Run an end-to-end test with Kafka broker coming up with your provider
> > > > making calls to RPC call. You do need to pass the "custom algorithm"
> > > >
> > > > that
> > > >
> > > > you registered in your key manager to invoke the provider. I think
> your
> > > > concern is this approach is replacing the existing known ciphersuites
> > > >
> > > > and
> > > >
> > > > certificate validation provided by java. Which its not.
> > > >
> > > > Now test the TLS connection you can do so via openssl -s_client
> options
> > > >
> > > > to
> > > >
> > > > test if everything else is working.
> > > >
> > > > I am happy to share configs that we used if you are interested.
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > > Thanks
> > > > Maulin
> > > >
> > > > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <cm...@apache.org>
> > > >
> > > > wrote:
> > > >
> > > > Hi Maulin,
> > > >
> > > > A lot of JSSE providers don't implement custom algorithms. Spire is a
> > > >
> > > > good
> > > >
> > > > example of a JSSE provider that doesn't, and yet is still useful to
> > > >
> > > > many
> > > >
> > > > people. Your JSSE provider can work fine even if it doesn't
> implement a
> > > > custom algorithm.
> > > >
> > > > Maybe I'm missing something, but I don't understand the discussion of
> > > > Security.insertProviderAt() that you included. SslEngineBuilder
> doesn't
> > > >
> > > > use
> > > >
> > > > that API to get the security provider. Instead, it calls
> > > > "SSLContext.getInstance(protocol, provider)", where provider is the
> > > >
> > > > name
> > > >
> > > > of the provider.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
> > > >
> > > > On top of everything above I feel strongly to add the 4th point which
> > > >
> > > > is
> > > >
> > > > based on Java APIs for TrustManagerFactory.init(KeyStore) (
> > > >
> > > > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> > > > TrustManagerFactory.html#init(java.security.KeyStore
> > > > <http://java.security.keystore/>
> > > > <http://java.security.keystore/>)
> > > > )
> > > >
> > > > and KeyManagerFactory.init(KeyStore, char[]) (
> > > >
> > > >
> > >
> >
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory
> > > > .
> > > >
> > > > html#init(java.security.KeyStore <http://java.security.keystore/>
> > > <http://
> > > > java.security.keystore/
> > > > ,%20char[])
> > > >
> > > > ).
> > > >
> > > > 4. The above APIs are intended to support providing "trust/key
> > > >
> > > > material"
> > > >
> > > > from the user without having to write their own
> > > >
> > > > TrustManager/KeyManagers.
> > > >
> > > > To quote from the TrustManagerFactory.init()'s documentation
> > > >
> > > > "Initializes
> > > >
> > > > this factory with a source of certificate authorities and related
> trust
> > > > material."
> > > > To quote from the KeyManagerFactory.init()'s documentation
> "Initializes
> > > > this factory with a source of key material."
> > > >
> > > > Based on this it is clear that there is a flexibility provided by
> Java
> > > >
> > > > to
> > > >
> > > > to enable developers to provide the required trust/key material
> loaded
> > > >
> > > > from
> > > >
> > > > "anywhere" without requiring them to write custom provider OR
> trust/key
> > > > managers. This same flexibility is reflected in Kafka code also where
> > > >
> > > > it
> > > >
> > > > loads the trust/keys from a local file and doesn't require writing a
> > > > Provider necessarily. If we do NOT have a custom algorithm, it makes
> > > >
> > > > less
> > > >
> > > > sense to write a Provider.
> > > >
> > > > Thanks
> > > > Maulin
> > > >
> > > > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
> > > >
> > > > maulin.vasavada@gmail.com>
> > > >
> > > > wrote:
> > > >
> > > > Hi Harsha/Colin
> > > >
> > > > I did the sample with a custom Provider for TrustStoreManager and
> tried
> > > > using ssl.provider Kafka config AND the way KIP-492 is suggesting (by
> > > > adding Provider programmatically instead of relying on
> > > >
> > > > ssl.provider+java.
> > > >
> > > > security. The below sample is followed by my detailed findings. I'll
> > > > appreciate if you can go through it carefully and see
> > > >
> > > > if you
> > > >
> > > > see my point.
> > > >
> > > > package providertest;
> > > >
> > > > import java.security.Provider <http://java.security.provider/>
> > <http://
> > > > java.security.provider/>
> > > >
> > > > <http://
> > > >
> > > > java.security.provider/>;
> > > >
> > > > public class MyProvider extends Provider {
> > > >
> > > > private static final String name = "MyProvider"; private static
> double
> > > > version = 1.0d;
> > > > private static String info = "Maulin's SSL Provider v"+version;
> > > >
> > > > public MyProvider() {
> > > > super(name, version, info);
> > > > this.put("TrustManagerFactory.PKIX",
> > > >
> > > > "providertest.MyTrustManagerFactory");
> > > >
> > > > }
> > > > }
> > > >
> > > > *Details:*
> > > >
> > > > KIP-492 documents that it will use Security.addProvider() assuming it
> > > >
> > > > will
> > > >
> > > > add it as position '0' which is not a correct assumption. The
> > > > addProvider()'s documentation says it will add it to the last
> available
> > > > position. You may want to correct that to say
> > > > Security.insertProviderAt(provider, 1).
> > > >
> > > > Now coming back to our specific discussion,
> > > >
> > > > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add
> > > >
> > > > that
> > > >
> > > > provider in the provider list via Security.addProvider() the position
> > > >
> > > > where
> > > >
> > > > it gets added doesn't matter (even if you don't end up adding it as
> > > >
> > > > first
> > > >
> > > > entry) since that is the ONLY provider for SPIFFE specific algorithm
> > > >
> > > > you
> > > >
> > > > might have.
> > > >
> > > > We do *not* have custom algorithm for Key/Trust StoreMangers. Which
> > > >
> > > > means
> > > >
> > > > we have to use X509, PKIX etc "Standard Algorithms" ((
> > > >
> > > > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> > > > StandardNames.html
> > > > ))
> > > >
> > > > in our provider to override the TrustStoreManager (see my sample
> code)
> > > >
> > > > and
> > > >
> > > > KeyStoreManger and KeyManager. This creates another challenge
> > > >
> > > > mentioned in
> > > >
> > > > the below point.
> > > >
> > > > 2. In order to make our Provider for loading custom TrustStore work,
> we
> > > > have to add the provider as 'first' in the list since there are
> others
> > > >
> > > > with
> > > >
> > > > the same algorithm.
> > > >
> > > > However, the programatic way of adding provider
> > > > (Security.insertProviderAt()) is *not* deterministic for ordering
> since
> > > > different code can call the method for a different provider and
> > > >
> > > > depending
> > > >
> > > > upon the order of the call our provider can be first or pushed down
> the
> > > > list. This can happen very well in any client application using
> Kafka.
> > > >
> > > > This
> > > >
> > > > is specially problematic for a case when you want to guarantee order
> > > >
> > > > for a
> > > >
> > > > Provider having "Standard Algorithms".
> > > >
> > > > If we add our provider in java.security file that definitely
> guarantees
> > > > the order(unless somebody calls removeProvider() which is less
> > > >
> > > > likely). But
> > > >
> > > > if we add our provider in java.security file it will defeat the
> > > >
> > > > purpose of
> > > >
> > > > the KIP-492.
> > > >
> > > > In the gist - Apache Kafka must not rely on "un-deterministic" method
> > > >
> > > > to
> > > >
> > > > rely on Provider ordering.
> > > >
> > > > 3. If we just use existing ssl.provider kafka configuration then our
> > > > provider will be used in SSLContext.getInstance(protocol, provider)
> > > >
> > > > call in
> > > >
> > > > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/>
> <
> > > > http://sslfactory.java/>
> > > >
> > > > and
> > > >
> > > > if our provider does not have implementation for
> > > > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks
> > > >
> > > > (we
> > > >
> > > > tested it). Example:
> > > >
> > > > In
> > > >
> > > > MyProvider sample above you see that I didn't add SSLContext.TLSv1 as
> > > > "Service+Algorithm" and that didn't work for me. In SPIFFE provider
> you
> > > > don't have this challenge since you are planning to bypass
> > > >
> > > > ssl.provider as
> > > >
> > > > you mention in the KIP-492.
> > > >
> > > > *Overall summary,*
> > > >
> > > > 1. Any provider based mechanisms- a) existing ssl.provider and
> > > >
> > > > b)KIP-492,
> > > >
> > > > for loading key/trust store using "Standard Algorithms" do not work
> > > >
> > > > 2. Approach suggested in our KIP-486 works without any issue and it
> is
> > > > *not* our context specific solve
> > > >
> > > > 3. Based on above we feel KIP-492 and KIP-486 are complimentary
> changes
> > > > and not contradicting or redundent.
> > > >
> > > > If you want we can do a joint session somehow to walk through the
> > > >
> > > > sample I
> > > >
> > > > have and various experiments I did. I would encourage you to do
> similar
> > > > exercise by writing a Provider for "Standard Algorithm" for
> > > > TrustStoreManager (like our needs) and see what you find since only
> > > >
> > > > writing
> > > >
> > > > samples can bring out the complexity/challenges we face.
> > > >
> > > > Thanks
> > > > Maulin
> > > >
> > > > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <
> > > >
> > > > maulin.vasavada@gmail.
> > > >
> > > > com> wrote:
> > > >
> > > > Just to update - still working on it. Get to work only on and off on
> > > >
> > > > it :(
> > > >
> > > > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
> > > >
> > > > maulin.vasavada@gmail.com>
> > > >
> > > > wrote:
> > > >
> > > > Hi Harsha
> > > >
> > > > Let me try to write samples and will let you know.
> > > >
> > > > Thanks
> > > > Maulin
> > > >
> > > > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <ha...@gmail.com>
> > > >
> > > > wrote:
> > > >
> > > > Hi Maulin,
> > > > With java security providers can be as custom you would
> > > >
> > > > like
> > > >
> > > > it to
> > > > be. If you only want to to implement a custom way of loading the
> > > >
> > > > keystore
> > > >
> > > > and truststore and not implement any protocol/encryption handling you
> > > >
> > > > can
> > > >
> > > > leave them empty and no need to implement. Have you looked into the
> > > >
> > > > links I
> > > >
> > > > pasted before?
> > > >
> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > >
> > > >
> > >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> > > >
> > > > java
> > > >
> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > > spiffe-security-provider/src/main/java/spiffe/api/provider/
> > > > SpiffeTrustManager.java <http://spiffetrustmanager.java/> <http://
> > > > spiffetrustmanager.java/>
> > > >
> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > >
> > > >
> > >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > > >
> > > > java
> > > >
> > > > Can you please tell me which methods are too complex in above to
> > > >
> > > > implement
> > > >
> > > > or unnecessary? You are changing anything in SSL/TLS implementations
> > > > provided by
> > > >
> > > > All of the implementations delegating the checks to the default
> > > > implementation anyway.
> > > > Spire agent is an example, its nothing but a GRPC server listening
> > > >
> > > > on a
> > > >
> > > > unix domain socket . Above code is making a RPC call to the local
> > > >
> > > > daemon
> > > >
> > > > to
> > > > get the certificate and keys. The mechanics are pretty much same as
> > > >
> > > > what
> > > >
> > > > you are asking for.
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > > On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
> > > >
> > > > maulin.vasavada@gmail.com>
> > > >
> > > > wrote:
> > > >
> > > > Imagine a scenario like - We know we have a custom KMS and as a
> > > >
> > > > Kafka
> > > >
> > > > owner
> > > >
> > > > we want to comply to using that KMS source to load keys/certs. As
> > > >
> > > > a
> > > >
> > > > Kafka
> > > >
> > > > owner we know how to integrate with KMS but doesn't necessarily
> > > >
> > > > have
> > > >
> > > > to
> > > >
> > > > know anything about cipher suites, algorithms, and SSL/TLS
> > > >
> > > > implementation.
> > > >
> > > > Going the Provider way requires to know lot more than we should,
> > > >
> > > > isn't it?
> > > >
> > > > Not that we would have concern/shy-away knowing those details -
> > > >
> > > > but
> > > >
> > > > if we
> > > >
> > > > don't have to - why should we?
> > > >
> > > > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
> > > >
> > > > maulin.vasavada@gmail.com>
> > > >
> > > > wrote:
> > > >
> > > > Hi Harsha
> > > >
> > > > We don't have spire (or similar) agents and we do not have
> > > >
> > > > keys/certs
> > > >
> > > > locally on any brokers. To elaborate more on my previous email,
> > > >
> > > > I agree that Java security Providers are used in much broader
> > > >
> > > > sense
> > > >
> > > > - to
> > > >
> > > > have a particular implementation of an algorithm, use specific
> > > >
> > > > cipher
> > > >
> > > > suites for SSL , OR in our current team's case have a
> > > >
> > > > particular
> > > >
> > > > way to
> > > >
> > > > leverage pre-generated SSL sessions. However, the scope of our
> > > >
> > > > KIP
> > > >
> > > > (486)
> > > >
> > > > is
> > > >
> > > > much restricted than that. We merely intend to provide a custom
> > > > keystore/truststore for our SSL connections and not really worry
> > > >
> > > > about
> > > >
> > > > underlying specific SSL/TLS implementation. This simplifies it
> > > >
> > > > a
> > > >
> > > > lot for
> > > >
> > > > us to keep the concerns separate and clear.
> > > >
> > > > I feel our approach is more complimentary such that it allows
> > > >
> > > > for
> > > >
> > > > using
> > > >
> > > > keystores of choice while retaining the flexibility to use any
> > > > underlying/available Provider for actually making the SSL call.
> > > >
> > > > We agree with KIP-492's approach based on Providers (and Java's
> > > > recommendation), but also strongly believe that our approach can
> > > >
> > > > compliment
> > > >
> > > > it very effectively for reasons explained above.
> > > >
> > > > Thanks
> > > > Maulin
> > > >
> > > > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <
> > > >
> > > > kafka@harsha.io
> > > >
> > > > wrote:
> > > >
> > > > Hi Maulin,
> > > >
> > > > On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
> > > >
> > > > maulin.vasavada@gmail.
> > > >
> > > > com>
> > > > wrote:
> > > >
> > > > Hi Harsha
> > > >
> > > > The reason we rejected the SslProvider route is that - we
> > > >
> > > > only
> > > >
> > > > needed
> > > >
> > > > a
> > > >
> > > > custom way to load keys/certs. Not touch any policy that
> > > >
> > > > existing
> > > >
> > > > Providers
> > > >
> > > > govern like SunJSSE Provider.
> > > >
> > > > We have exactly the same requirements to load certs and keys
> > > >
> > > > through
> > > >
> > > > spire
> > > >
> > > > agent. We used security.provider to do that exactly. I am not
> > > >
> > > > sure
> > > >
> > > > why
> > > >
> > > > you
> > > >
> > > > would be modifying any policies provided by default SunJSSE
> > > >
> > > > provider.
> > > >
> > > > Can
> > > >
> > > > you give me an example of having custom provider that will
> > > >
> > > > override an
> > > >
> > > > existing policy in SunJSSE provider.
> > > >
> > > > As pointed out earlier, this kip
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> > > >
> > > > allows
> > > > you to load security.provider through config.
> > > > Take a look at the examples I gave before
> > > >
> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > >
> > > >
> > >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > > >
> > > > java
> > > >
> > > > It registers KeyManagerFactory and TrustManagerFactory and
> > > >
> > > > Keystore
> > > >
> > > > algorithm.
> > > > Implement your custom way of loading Keystore in here
> > > >
> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > >
> > > >
> > >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> > > >
> > > > java
> > > >
> > > > and Trust manager like here
> > > >
> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > > spiffe-security-provider/src/main/java/spiffe/api/provider/
> > > > SpiffeTrustManager.java <http://spiffetrustmanager.java/> <http://
> > > > spiffetrustmanager.java/>
> > > >
> > > > In your Kafka client you can set the security.provider to your
> > > >
> > > > custom
> > > >
> > > > implementation and with this fix
> > > > https://issues.apache.org/jira/browse/KAFKA-8191 you can set
> > > > keyManagerAlgorigthm and trustManagerAlgorithm configs.
> > > >
> > > > All of this is in your clients and broker side and do not need
> > > >
> > > > to
> > > >
> > > > touch
> > > >
> > > > any
> > > > policy changes at JVM level. You'll register the providers in
> > > >
> > > > the
> > > >
> > > > priority
> > > >
> > > > order and can still have SunJSSE provider and have your custom
> > > >
> > > > provider
> > > >
> > > > to
> > > >
> > > > implement the key and trust managers.
> > > >
> > > > The ask here is different than KIP-492. We don't have any need
> > > >
> > > > to
> > > >
> > > > modify/specify the algorithm parameter. Does that make sense?
> > > >
> > > > The ask in KIP is introducing new interfaces where the KIP's
> > > > goal/motivation can be achieved through the security.provider
> > > >
> > > > and
> > > >
> > > > we
> > > >
> > > > worked
> > > > on similar goal without touching any Keystore or Truststore
> > > >
> > > > interfaces.
> > > >
> > > > My advise is against changing or introducing new interfaces
> > > >
> > > > when
> > > >
> > > > it can
> > > >
> > > > work through security.provider.
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > > Thanks
> > > >
> > > > Maulin
> > > >
> > > > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
> > > >
> > > > kafka@harsha.io>
> > > >
> > > > wrote:
> > > >
> > > > In your KIP you added security. provider as rejected
> > > >
> > > > alternative
> > > >
> > > > and
> > > >
> > > > specified "its not the correct way". Do you mind explaining
> > > >
> > > > why
> > > >
> > > > its
> > > >
> > > > not? I
> > > >
> > > > didn't find any evidence in Java docs to say so. Contrary to
> > > >
> > > > your
> > > >
> > > > statement
> > > >
> > > > it does say in the java docs
> > > > " However, please note that a provider can be used to
> > > >
> > > > implement
> > > >
> > > > any
> > > >
> > > > security service in Java that uses a pluggable architecture
> > > >
> > > > with
> > > >
> > > > a
> > > >
> > > > choice
> > > >
> > > > of implementations that fit underneath."
> > > >
> > > > Java Security Providers have been used by other projects to
> > > >
> > > > provide
> > > >
> > > > such
> > > >
> > > > integration . I am not sure if you looked into Spiffe
> > > >
> > > > project to
> > > >
> > > > efficiently distribute certificates but here is an example of
> > > >
> > > > Java
> > > >
> > > > provider
> > > >
> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > >
> > > >
> > >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > > >
> > > > java which
> > > > obtains certificates from local daemons.
> > > > These integrations are being used in Tomcat, Jetty etc.. We
> > > >
> > > > are
> > > >
> > > > also
> > > >
> > > > using
> > > >
> > > > Security provider to do the same in our Kafka clusters. So
> > > >
> > > > unless I
> > > >
> > > > see
> > > >
> > > > more evidence why security.provider doesn't work for you
> > > >
> > > > adding
> > > >
> > > > new
> > > >
> > > > interfaces while there exists more cleaner way of achieving
> > > >
> > > > the
> > > >
> > > > goals
> > > >
> > > > of
> > > >
> > > > this KIP is unnecessary and breaks the well known security
> > > >
> > > > interfaces
> > > >
> > > > provided by Java itself.
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
> > > >
> > > > kafka@harsha.io
> > > >
> > > > wrote:
> > > >
> > > > Hi Maulin,
> > > > Not sure if you looked at my previous replies. This
> > > >
> > > > changes
> > > >
> > > > are not required as there is already security Provider to do
> > > >
> > > > what you
> > > >
> > > > are
> > > >
> > > > proposing. This KIP
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > >
> > > > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> > > >
> > > > also
> > > >
> > > > addresses easy registration of such providers.
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
> > > >
> > > > <maulin.vasavada@gmail.
> > > >
> > > > com> wrote:
> > > >
> > > > Bump! Can somebody please review this?
> > > >
> > > > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
> > > >
> > > > maulin.vasavada@gmail.com>
> > > >
> > > > wrote:
> > > >
> > > > Bump! Can somebody please review this?
> > > >
> > > > --
> > > > Thanks,
> > > > M.Sai Sandeep
> > > >
> > > >
> > >
> >
>
RE: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by "Pellerin, Clement" <Cl...@ibi.com>.
KIP-383 in its present form was vetoed because it was not possible to add validation of custom properties in a future KIP. The solution to that is the first proposal I wrote for KIP-383 which made the whole SslFactory pluggable. That first solution was also vetoed hence the deadlock.
Replacing the whole factory was a much nicer solution. It was vetoed because doing this almost invariably meant the replacement lost all the complex validation code in the default SslFactory.
My current idea is to extract the validation code into another public API that SslFactory would call. I did not look at the newly refactored code and I did not study how to do this yet. KIP-383 was not popular at the time and designing a new solution is a lot of work.
Is there interest from 3 binding voters for something like this?
-----Original Message-----
From: Rajini Sivaram [mailto:rajinisivaram@gmail.com]
Sent: Thursday, August 29, 2019 2:57 PM
To: dev
Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Hi Maulin,
In SSL scenarios, I imagine security providers introduced by KIP-492 are
likely to be most useful when you want to use third party providers. The
biggest advantage of the config from that KIP is that you don't need to
write much code to integrate existing security providers into Kafka brokers
or clients. As I understand it, KIP-486 is a more convenient option for the
specific problem of loading keystores/truststores differently. It can be
achieved in theory with KIP-492, but KIP-486 is a much simpler option for
this case.
My concern about KIP-486 is that it introduces yet another interface into
our already complex security code, while only solving one particular use
case. Have you looked at
https://cwiki.apache.org/confluence/display/KAFKA/KIP-383%3A++Pluggable+interface+for+SSL+Factory?
The goal was to make
org.apache.kafka.common.security.ssl.SslEngineBuilder pluggable.
The code has already been refactored by Colin after that KIP was written,
making it easier to implement KIP-383. This should enable you to load your
keystores and truststores differently. Using a pluggable SslEngineBuilder
will also solve several other use cases at the same time. KIP-383 hasn't
been voted through yet, but perhaps you could take a look and we could
revive that instead if it solves your use case as well?
Regards,
Rajini
On Thu, Aug 29, 2019 at 6:42 PM Maulin Vasavada <ma...@gmail.com>
wrote:
> Hi Harsha
>
> Thank you. Appreciate your time and support on this. Let me go back do some
> more research and get back to you on the KeyStore interface part.
> Basically, if we return certs and keys in the interface then Kafka code
> will have to build KeyStore object - which is also reasonable.
>
> Thanks
> Maulin
>
> On Thu, Aug 29, 2019 at 10:01 AM Harsha Chintalapani <ka...@harsha.io>
> wrote:
>
> > Hi Maulin,
> > Use cases are clear now. I am +1 for moving forward
> > with the discussions on having such configurable option for users. But
> the
> > interfaces is proposed doesn't look right to me. We are still talking
> about
> > keystore interfaces. Given keystore's are used as filebased way of
> > transporting certificates I am not sure it will help the rest of the
> > user-base.
> > In short, I am +1 on the KIP's motivation and only have
> > questions around returning keystores instead of returning certs, private
> > keys etc. . If others in the community are ok with such interface we can
> > move forward.
> >
> > Thanks,
> > Harsha
> >
> >
> > On Wed, Aug 28, 2019 at 1:51 PM, Maulin Vasavada <
> > maulin.vasavada@gmail.com>
> > wrote:
> >
> > > Hi Harsha
> > >
> > > As we synced-up offline on this topic, we hope you don't have any more
> > > clarifications that you are seeking. If that is the case, can you
> please
> > > help us move this forward and discuss what changes you would expect on
> > the
> > > KIP design in order to make it valuable contribution?
> > >
> > > Just FYI - we verified our primary design change with the author of
> Sun's
> > > X509 Trustmanager implementation and the outcome is that what we are
> > > proposing makes sense at the heart of it - "Instead of writing
> > TrustManager
> > > just plugin the Trust store". We are open to discuss additional changes
> > > that you/anybody else would like to see on the functionality however.
> > >
> > > Thanks
> > > Maulin
> > >
> > > On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada <
> > maulin.vasavada@gmail.com>
> > > wrote:
> > >
> > > Hi Harsha
> > >
> > > Any response on my question? I feel this KIP is worth accommodating.
> Your
> > > help is much appreciated.
> > >
> > > Thanks
> > > Maulin
> > >
> > > On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada <
> maulin.vasavada@gmail.
> > > com> wrote:
> > >
> > > Hi Harsha
> > >
> > > I've examined the SPIFFE provider more and have one question -
> > >
> > > If SPIFFE didn't have a need to do checkSpiffeId() call at the below
> > > location, would you really still write the Provider? *OR* Would you
> just
> > > use TrustManagerFactory.init(KeyStore) signature to pass the KeyStore
> > from
> > > set of certs returned by spiffeIdManager. getTrustedCerts()?
> > >
> > >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > provider/CertificateUtils.java#L100
> > >
> > > /**
> > >
> > > * Validates that the SPIFFE ID is present and matches the SPIFFE ID
> > > configured in
> > > * the java.security property ssl.spiffe.accept
> > > *
> > > * If the authorized spiffe ids list is empty any spiffe id is
> authorized
> > > *
> > > * @param chain an array of X509Certificate that contains the Peer's
> SVID
> > > to be validated
> > > * @throws CertificateException when either the certificates doesn't
> have
> > a
> > > SPIFFE ID or the SPIFFE ID is not authorized
> > > */
> > > static void checkSpiffeId(X509Certificate[] chain) throws
> > > CertificateException {
> > >
> > > Thanks
> > > Maulin
> > >
> > > On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani <ka...@harsha.io>
> > > wrote:
> > >
> > > Maulin,
> > > The code parts you are pointing are specific for Spiffe and if
> > > you are talking about validate method which uses PKIX check like any
> > other
> > > provider does.
> > > If you want to default to SunJSSE everywhere you can do so by
> delegating
> > > the calls in these methods to SunJSSE provider.
> > >
> > > TrustManagerFactory tmf = TrustManagerFactory
> > > .getInstance(TrustManagerFactory.getDefaultAlgorithm());and use
> > > tmf.chekServerTrusted()
> > > or use
> > > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> > > TrustManagerFactory.html#getInstance(java.lang.String)if you want a
> > > specific provider.
> > >
> > > -Harsha
> > >
> > > On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada <
> maulin.vasavada@gmail.
> > > com>
> > > wrote:
> > >
> > > Okay, so I take that you guys agree that I have to write a 'custom'
> > > algorithm and a provider to make it work , correct?
> > >
> > > Now, for Harsha's comment "Here the 'Custom' Algorithm is not an
> > > implementation per say , ..." , I diagree. You can refer to https://
> > >
> > >
> github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/
> > >
> > > SpiffeTrustManager.java#L91 <http://spiffetrustmanager.java/#L91> and
> > >
> > >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > >
> > > provider/CertificateUtils.java#L100
> > >
> > > "that code" is the customization you have for the custom way to check
> > > something on top of regular checks. That method is NOT doing custom
> > > truststore loading. It is validating/verifying something in the
> > >
> > > "custom"
> > >
> > > way with spiffeId.
> > > I bet that without that you won't have a need of the custom algorithm
> > >
> > > in
> > >
> > > the first place.
> > >
> > > Let me know if you agree to this.
> > >
> > > Thanks
> > > Maulin
> > >
> > > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <mp...@gmail.com>
> > >
> > > wrote:
> > >
> > > Hi Maulin, thanks for the discussion. As Harsha pointed out, to use the
> > > KIP492, you need to create a new provider, register a *new* custom
> > > algorithm for your keymanager and trustmanager factory implementations.
> > > After this, the kafka server configuration can be done as given below
> > >
> > > # Register the provider class with custom algorithm, say CUSTOM
> > >
> > > security.
> > >
> > > provider.classes=com.company.security.CustomProvider
> > > <http://provider.classes=com.company.security.customprovider/>
> > > <http://security.provider.classes
> > >
> > > =com.company.security.customprovider/>
> > >
> > > # Register the keymanager and trustmanager algorithms
> > > # These algorithms indicate that the Keymanager and Trustmanagers
> > > registered under the algorithm "CUSTOM" needs to be used
> > > ssl.trustmanager.algorithm=CUSTOM
> > > ssl.keymanager.algorithm=CUSTOM
> > >
> > > And the customprovider looks like this...
> > >
> > > public class CustomProvider extends Provider {
> > > public CustomProvider() {
> > > super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and TrustStore");
> > > super.put("KeyManagerFactory.CUSTOM", "customKeyManagerFactory");
> > > super.put("TrustManagerFactory.CUSTOM",
> > > "customTrustManagerFactory");
> > > }
> > > }
> > >
> > > The PR for this is in review and can be found here:
> > >
> > > https://github.com/
> > >
> > > apache/kafka/pull/7090
> > > This PR includes the fixed insertProviderAt function call.
> > >
> > > On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani <ka...@harsha.io>
> > > wrote:
> > >
> > > Answers are in-line
> > >
> > > On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada <
> > >
> > > maulin.vasavada@gmail.
> > >
> > > com
> > >
> > > wrote:
> > >
> > > Hi Colin
> > >
> > > When I refer to "standard" or "custom" algorithms I am following Java
> > > security Provider Terminology. You can refer to
> > > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> > > StandardNames.html#TrustManagerFactory link I provided earlier in the
> > > emails. It says PKIX is the default Algorithm for TrustManagerFactory.
> > >
> > > 1. For SPIFFE, I am not sure why you are saying 'it does not implement
> > > custom algorithms' because the following file clearly indicates that it
> > > does use custom algorithm-
> > >
> > >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > >
> > > provider/SpiffeProvider.java#L17
> > >
> > > Algorithm value:
> > >
> > >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > >
> > > provider/SpiffeProviderConstants.java#L6
> > >
> > > @Harsha do you want to chime in since you use that provider?
> > >
> > > Here the "Custom" Algorithm is not an implementation per say , rather
> > >
> > > used
> > >
> > > to invoke the custom trust store factory and key manager factory. You
> > >
> > > are
> > >
> > > not moving away from "standard" alogrithms that are available.
> > >
> > >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > >
> > > provider/SpiffeTrustManager.java
> > >
> > > As you can see it delegates all the calls of verification of
> > >
> > > certificate
> > >
> > > to
> > >
> > > the default implementation available.
> > > So in our implementation we still use PKIX to verify the certificate
> > > chain. So you are not losing anything here and Spiffe is not
> > >
> > > reimplementing
> > >
> > > the verification process.
> > >
> > > 2. I already mentioned in my 3rd point, in my previous post, why using
> > >
> > > ssl.provider does NOT work. I updated KIP-486 in "rejected
> > >
> > > alternatives"
> > >
> > > also why ssl.provider does not work.
> > >
> > > As mentioned before , provider is the right way to go. I am still not
> > > understanding the gap is.
> > > If I understand correctly your argument is , provider is going to ask
> > >
> > > to
> > >
> > > implement a custom algorithm.
> > > My answer to that is , no you are not re-implementing the algorithm.
> > >
> > > Please
> > >
> > > check the above link , TrustManager implementation it delegates the
> > >
> > > calls
> > >
> > > back. There is no need to implement your own here.
> > >
> > > 3. Security.insertProviderAt() comments were based on assumption if
> > >
> > > KIP-492
> > >
> > > changes are done and we use that mechanism to configure providers
> > >
> > > instead
> > >
> > > of ssl.provider configuration.
> > >
> > > KIP-492 has patch available and going through review.
> > >
> > > Can you read my all the points, I mentioned in my previous post, very
> > >
> > > carefully? I am covering all the aspects in explaining. I am open to
> > >
> > > still
> > >
> > > discuss more to clarify any doubts.
> > >
> > > "3. If we just use existing ssl.provider kafka configuration then our
> > > provider will be used in SSLContext.getInstance(protocol, provider)
> > >
> > > call
> > >
> > > in
> > >
> > > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/> <
> > > http://sslfactory.java/>
> > >
> > > and
> > >
> > > if our provider does not have
> > > implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we
> > >
> > > tested
> > >
> > > it). Example: In MyProvider sample above you see that I didn't add
> > > SSLContext.TLSv1 as
> > > "Service+Algorithm" and that didn't work for me. In SPIFFE provider you
> > > don't have this challenge since you are planning to bypass ssl.provider
> > >
> > > as
> > >
> > > you mention in the KIP-492."
> > >
> > > Yes here you need to pass the protocol that your
> > >
> > > KeyManager/TrustManager
> > >
> > > registered with and in no way its deviating from TLS RFC spec.
> > >
> > >
> https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/
> > >
> > > javax/net/ssl/SSLContext.java#L134
> > >
> > > My suggestion here is for you to implement a simple Security Provider
> > >
> > > as
> > >
> > > you did before and register a Algorithm. You can use the existing
> > > implementation in SpiffeProvider and plug in changes where you need to
> > > retrieve the certificates from by making RPC call.
> > >
> > > Run an end-to-end test with Kafka broker coming up with your provider
> > > making calls to RPC call. You do need to pass the "custom algorithm"
> > >
> > > that
> > >
> > > you registered in your key manager to invoke the provider. I think your
> > > concern is this approach is replacing the existing known ciphersuites
> > >
> > > and
> > >
> > > certificate validation provided by java. Which its not.
> > >
> > > Now test the TLS connection you can do so via openssl -s_client options
> > >
> > > to
> > >
> > > test if everything else is working.
> > >
> > > I am happy to share configs that we used if you are interested.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > Thanks
> > > Maulin
> > >
> > > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <cm...@apache.org>
> > >
> > > wrote:
> > >
> > > Hi Maulin,
> > >
> > > A lot of JSSE providers don't implement custom algorithms. Spire is a
> > >
> > > good
> > >
> > > example of a JSSE provider that doesn't, and yet is still useful to
> > >
> > > many
> > >
> > > people. Your JSSE provider can work fine even if it doesn't implement a
> > > custom algorithm.
> > >
> > > Maybe I'm missing something, but I don't understand the discussion of
> > > Security.insertProviderAt() that you included. SslEngineBuilder doesn't
> > >
> > > use
> > >
> > > that API to get the security provider. Instead, it calls
> > > "SSLContext.getInstance(protocol, provider)", where provider is the
> > >
> > > name
> > >
> > > of the provider.
> > >
> > > best,
> > > Colin
> > >
> > > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
> > >
> > > On top of everything above I feel strongly to add the 4th point which
> > >
> > > is
> > >
> > > based on Java APIs for TrustManagerFactory.init(KeyStore) (
> > >
> > > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> > > TrustManagerFactory.html#init(java.security.KeyStore
> > > <http://java.security.keystore/>
> > > <http://java.security.keystore/>)
> > > )
> > >
> > > and KeyManagerFactory.init(KeyStore, char[]) (
> > >
> > >
> >
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory
> > > .
> > >
> > > html#init(java.security.KeyStore <http://java.security.keystore/>
> > <http://
> > > java.security.keystore/
> > > ,%20char[])
> > >
> > > ).
> > >
> > > 4. The above APIs are intended to support providing "trust/key
> > >
> > > material"
> > >
> > > from the user without having to write their own
> > >
> > > TrustManager/KeyManagers.
> > >
> > > To quote from the TrustManagerFactory.init()'s documentation
> > >
> > > "Initializes
> > >
> > > this factory with a source of certificate authorities and related trust
> > > material."
> > > To quote from the KeyManagerFactory.init()'s documentation "Initializes
> > > this factory with a source of key material."
> > >
> > > Based on this it is clear that there is a flexibility provided by Java
> > >
> > > to
> > >
> > > to enable developers to provide the required trust/key material loaded
> > >
> > > from
> > >
> > > "anywhere" without requiring them to write custom provider OR trust/key
> > > managers. This same flexibility is reflected in Kafka code also where
> > >
> > > it
> > >
> > > loads the trust/keys from a local file and doesn't require writing a
> > > Provider necessarily. If we do NOT have a custom algorithm, it makes
> > >
> > > less
> > >
> > > sense to write a Provider.
> > >
> > > Thanks
> > > Maulin
> > >
> > > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
> > >
> > > maulin.vasavada@gmail.com>
> > >
> > > wrote:
> > >
> > > Hi Harsha/Colin
> > >
> > > I did the sample with a custom Provider for TrustStoreManager and tried
> > > using ssl.provider Kafka config AND the way KIP-492 is suggesting (by
> > > adding Provider programmatically instead of relying on
> > >
> > > ssl.provider+java.
> > >
> > > security. The below sample is followed by my detailed findings. I'll
> > > appreciate if you can go through it carefully and see
> > >
> > > if you
> > >
> > > see my point.
> > >
> > > package providertest;
> > >
> > > import java.security.Provider <http://java.security.provider/>
> <http://
> > > java.security.provider/>
> > >
> > > <http://
> > >
> > > java.security.provider/>;
> > >
> > > public class MyProvider extends Provider {
> > >
> > > private static final String name = "MyProvider"; private static double
> > > version = 1.0d;
> > > private static String info = "Maulin's SSL Provider v"+version;
> > >
> > > public MyProvider() {
> > > super(name, version, info);
> > > this.put("TrustManagerFactory.PKIX",
> > >
> > > "providertest.MyTrustManagerFactory");
> > >
> > > }
> > > }
> > >
> > > *Details:*
> > >
> > > KIP-492 documents that it will use Security.addProvider() assuming it
> > >
> > > will
> > >
> > > add it as position '0' which is not a correct assumption. The
> > > addProvider()'s documentation says it will add it to the last available
> > > position. You may want to correct that to say
> > > Security.insertProviderAt(provider, 1).
> > >
> > > Now coming back to our specific discussion,
> > >
> > > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add
> > >
> > > that
> > >
> > > provider in the provider list via Security.addProvider() the position
> > >
> > > where
> > >
> > > it gets added doesn't matter (even if you don't end up adding it as
> > >
> > > first
> > >
> > > entry) since that is the ONLY provider for SPIFFE specific algorithm
> > >
> > > you
> > >
> > > might have.
> > >
> > > We do *not* have custom algorithm for Key/Trust StoreMangers. Which
> > >
> > > means
> > >
> > > we have to use X509, PKIX etc "Standard Algorithms" ((
> > >
> > > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> > > StandardNames.html
> > > ))
> > >
> > > in our provider to override the TrustStoreManager (see my sample code)
> > >
> > > and
> > >
> > > KeyStoreManger and KeyManager. This creates another challenge
> > >
> > > mentioned in
> > >
> > > the below point.
> > >
> > > 2. In order to make our Provider for loading custom TrustStore work, we
> > > have to add the provider as 'first' in the list since there are others
> > >
> > > with
> > >
> > > the same algorithm.
> > >
> > > However, the programatic way of adding provider
> > > (Security.insertProviderAt()) is *not* deterministic for ordering since
> > > different code can call the method for a different provider and
> > >
> > > depending
> > >
> > > upon the order of the call our provider can be first or pushed down the
> > > list. This can happen very well in any client application using Kafka.
> > >
> > > This
> > >
> > > is specially problematic for a case when you want to guarantee order
> > >
> > > for a
> > >
> > > Provider having "Standard Algorithms".
> > >
> > > If we add our provider in java.security file that definitely guarantees
> > > the order(unless somebody calls removeProvider() which is less
> > >
> > > likely). But
> > >
> > > if we add our provider in java.security file it will defeat the
> > >
> > > purpose of
> > >
> > > the KIP-492.
> > >
> > > In the gist - Apache Kafka must not rely on "un-deterministic" method
> > >
> > > to
> > >
> > > rely on Provider ordering.
> > >
> > > 3. If we just use existing ssl.provider kafka configuration then our
> > > provider will be used in SSLContext.getInstance(protocol, provider)
> > >
> > > call in
> > >
> > > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/> <
> > > http://sslfactory.java/>
> > >
> > > and
> > >
> > > if our provider does not have implementation for
> > > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks
> > >
> > > (we
> > >
> > > tested it). Example:
> > >
> > > In
> > >
> > > MyProvider sample above you see that I didn't add SSLContext.TLSv1 as
> > > "Service+Algorithm" and that didn't work for me. In SPIFFE provider you
> > > don't have this challenge since you are planning to bypass
> > >
> > > ssl.provider as
> > >
> > > you mention in the KIP-492.
> > >
> > > *Overall summary,*
> > >
> > > 1. Any provider based mechanisms- a) existing ssl.provider and
> > >
> > > b)KIP-492,
> > >
> > > for loading key/trust store using "Standard Algorithms" do not work
> > >
> > > 2. Approach suggested in our KIP-486 works without any issue and it is
> > > *not* our context specific solve
> > >
> > > 3. Based on above we feel KIP-492 and KIP-486 are complimentary changes
> > > and not contradicting or redundent.
> > >
> > > If you want we can do a joint session somehow to walk through the
> > >
> > > sample I
> > >
> > > have and various experiments I did. I would encourage you to do similar
> > > exercise by writing a Provider for "Standard Algorithm" for
> > > TrustStoreManager (like our needs) and see what you find since only
> > >
> > > writing
> > >
> > > samples can bring out the complexity/challenges we face.
> > >
> > > Thanks
> > > Maulin
> > >
> > > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <
> > >
> > > maulin.vasavada@gmail.
> > >
> > > com> wrote:
> > >
> > > Just to update - still working on it. Get to work only on and off on
> > >
> > > it :(
> > >
> > > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
> > >
> > > maulin.vasavada@gmail.com>
> > >
> > > wrote:
> > >
> > > Hi Harsha
> > >
> > > Let me try to write samples and will let you know.
> > >
> > > Thanks
> > > Maulin
> > >
> > > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <ha...@gmail.com>
> > >
> > > wrote:
> > >
> > > Hi Maulin,
> > > With java security providers can be as custom you would
> > >
> > > like
> > >
> > > it to
> > > be. If you only want to to implement a custom way of loading the
> > >
> > > keystore
> > >
> > > and truststore and not implement any protocol/encryption handling you
> > >
> > > can
> > >
> > > leave them empty and no need to implement. Have you looked into the
> > >
> > > links I
> > >
> > > pasted before?
> > >
> > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >
> > >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> > >
> > > java
> > >
> > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > spiffe-security-provider/src/main/java/spiffe/api/provider/
> > > SpiffeTrustManager.java <http://spiffetrustmanager.java/> <http://
> > > spiffetrustmanager.java/>
> > >
> > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >
> > >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > >
> > > java
> > >
> > > Can you please tell me which methods are too complex in above to
> > >
> > > implement
> > >
> > > or unnecessary? You are changing anything in SSL/TLS implementations
> > > provided by
> > >
> > > All of the implementations delegating the checks to the default
> > > implementation anyway.
> > > Spire agent is an example, its nothing but a GRPC server listening
> > >
> > > on a
> > >
> > > unix domain socket . Above code is making a RPC call to the local
> > >
> > > daemon
> > >
> > > to
> > > get the certificate and keys. The mechanics are pretty much same as
> > >
> > > what
> > >
> > > you are asking for.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
> > >
> > > maulin.vasavada@gmail.com>
> > >
> > > wrote:
> > >
> > > Imagine a scenario like - We know we have a custom KMS and as a
> > >
> > > Kafka
> > >
> > > owner
> > >
> > > we want to comply to using that KMS source to load keys/certs. As
> > >
> > > a
> > >
> > > Kafka
> > >
> > > owner we know how to integrate with KMS but doesn't necessarily
> > >
> > > have
> > >
> > > to
> > >
> > > know anything about cipher suites, algorithms, and SSL/TLS
> > >
> > > implementation.
> > >
> > > Going the Provider way requires to know lot more than we should,
> > >
> > > isn't it?
> > >
> > > Not that we would have concern/shy-away knowing those details -
> > >
> > > but
> > >
> > > if we
> > >
> > > don't have to - why should we?
> > >
> > > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
> > >
> > > maulin.vasavada@gmail.com>
> > >
> > > wrote:
> > >
> > > Hi Harsha
> > >
> > > We don't have spire (or similar) agents and we do not have
> > >
> > > keys/certs
> > >
> > > locally on any brokers. To elaborate more on my previous email,
> > >
> > > I agree that Java security Providers are used in much broader
> > >
> > > sense
> > >
> > > - to
> > >
> > > have a particular implementation of an algorithm, use specific
> > >
> > > cipher
> > >
> > > suites for SSL , OR in our current team's case have a
> > >
> > > particular
> > >
> > > way to
> > >
> > > leverage pre-generated SSL sessions. However, the scope of our
> > >
> > > KIP
> > >
> > > (486)
> > >
> > > is
> > >
> > > much restricted than that. We merely intend to provide a custom
> > > keystore/truststore for our SSL connections and not really worry
> > >
> > > about
> > >
> > > underlying specific SSL/TLS implementation. This simplifies it
> > >
> > > a
> > >
> > > lot for
> > >
> > > us to keep the concerns separate and clear.
> > >
> > > I feel our approach is more complimentary such that it allows
> > >
> > > for
> > >
> > > using
> > >
> > > keystores of choice while retaining the flexibility to use any
> > > underlying/available Provider for actually making the SSL call.
> > >
> > > We agree with KIP-492's approach based on Providers (and Java's
> > > recommendation), but also strongly believe that our approach can
> > >
> > > compliment
> > >
> > > it very effectively for reasons explained above.
> > >
> > > Thanks
> > > Maulin
> > >
> > > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <
> > >
> > > kafka@harsha.io
> > >
> > > wrote:
> > >
> > > Hi Maulin,
> > >
> > > On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
> > >
> > > maulin.vasavada@gmail.
> > >
> > > com>
> > > wrote:
> > >
> > > Hi Harsha
> > >
> > > The reason we rejected the SslProvider route is that - we
> > >
> > > only
> > >
> > > needed
> > >
> > > a
> > >
> > > custom way to load keys/certs. Not touch any policy that
> > >
> > > existing
> > >
> > > Providers
> > >
> > > govern like SunJSSE Provider.
> > >
> > > We have exactly the same requirements to load certs and keys
> > >
> > > through
> > >
> > > spire
> > >
> > > agent. We used security.provider to do that exactly. I am not
> > >
> > > sure
> > >
> > > why
> > >
> > > you
> > >
> > > would be modifying any policies provided by default SunJSSE
> > >
> > > provider.
> > >
> > > Can
> > >
> > > you give me an example of having custom provider that will
> > >
> > > override an
> > >
> > > existing policy in SunJSSE provider.
> > >
> > > As pointed out earlier, this kip
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> > >
> > > allows
> > > you to load security.provider through config.
> > > Take a look at the examples I gave before
> > >
> > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >
> > >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > >
> > > java
> > >
> > > It registers KeyManagerFactory and TrustManagerFactory and
> > >
> > > Keystore
> > >
> > > algorithm.
> > > Implement your custom way of loading Keystore in here
> > >
> > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >
> > >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> > >
> > > java
> > >
> > > and Trust manager like here
> > >
> > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > spiffe-security-provider/src/main/java/spiffe/api/provider/
> > > SpiffeTrustManager.java <http://spiffetrustmanager.java/> <http://
> > > spiffetrustmanager.java/>
> > >
> > > In your Kafka client you can set the security.provider to your
> > >
> > > custom
> > >
> > > implementation and with this fix
> > > https://issues.apache.org/jira/browse/KAFKA-8191 you can set
> > > keyManagerAlgorigthm and trustManagerAlgorithm configs.
> > >
> > > All of this is in your clients and broker side and do not need
> > >
> > > to
> > >
> > > touch
> > >
> > > any
> > > policy changes at JVM level. You'll register the providers in
> > >
> > > the
> > >
> > > priority
> > >
> > > order and can still have SunJSSE provider and have your custom
> > >
> > > provider
> > >
> > > to
> > >
> > > implement the key and trust managers.
> > >
> > > The ask here is different than KIP-492. We don't have any need
> > >
> > > to
> > >
> > > modify/specify the algorithm parameter. Does that make sense?
> > >
> > > The ask in KIP is introducing new interfaces where the KIP's
> > > goal/motivation can be achieved through the security.provider
> > >
> > > and
> > >
> > > we
> > >
> > > worked
> > > on similar goal without touching any Keystore or Truststore
> > >
> > > interfaces.
> > >
> > > My advise is against changing or introducing new interfaces
> > >
> > > when
> > >
> > > it can
> > >
> > > work through security.provider.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > Thanks
> > >
> > > Maulin
> > >
> > > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
> > >
> > > kafka@harsha.io>
> > >
> > > wrote:
> > >
> > > In your KIP you added security. provider as rejected
> > >
> > > alternative
> > >
> > > and
> > >
> > > specified "its not the correct way". Do you mind explaining
> > >
> > > why
> > >
> > > its
> > >
> > > not? I
> > >
> > > didn't find any evidence in Java docs to say so. Contrary to
> > >
> > > your
> > >
> > > statement
> > >
> > > it does say in the java docs
> > > " However, please note that a provider can be used to
> > >
> > > implement
> > >
> > > any
> > >
> > > security service in Java that uses a pluggable architecture
> > >
> > > with
> > >
> > > a
> > >
> > > choice
> > >
> > > of implementations that fit underneath."
> > >
> > > Java Security Providers have been used by other projects to
> > >
> > > provide
> > >
> > > such
> > >
> > > integration . I am not sure if you looked into Spiffe
> > >
> > > project to
> > >
> > > efficiently distribute certificates but here is an example of
> > >
> > > Java
> > >
> > > provider
> > >
> > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >
> > >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > >
> > > java which
> > > obtains certificates from local daemons.
> > > These integrations are being used in Tomcat, Jetty etc.. We
> > >
> > > are
> > >
> > > also
> > >
> > > using
> > >
> > > Security provider to do the same in our Kafka clusters. So
> > >
> > > unless I
> > >
> > > see
> > >
> > > more evidence why security.provider doesn't work for you
> > >
> > > adding
> > >
> > > new
> > >
> > > interfaces while there exists more cleaner way of achieving
> > >
> > > the
> > >
> > > goals
> > >
> > > of
> > >
> > > this KIP is unnecessary and breaks the well known security
> > >
> > > interfaces
> > >
> > > provided by Java itself.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
> > >
> > > kafka@harsha.io
> > >
> > > wrote:
> > >
> > > Hi Maulin,
> > > Not sure if you looked at my previous replies. This
> > >
> > > changes
> > >
> > > are not required as there is already security Provider to do
> > >
> > > what you
> > >
> > > are
> > >
> > > proposing. This KIP
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/
> > >
> > > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> > >
> > > also
> > >
> > > addresses easy registration of such providers.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
> > >
> > > <maulin.vasavada@gmail.
> > >
> > > com> wrote:
> > >
> > > Bump! Can somebody please review this?
> > >
> > > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
> > >
> > > maulin.vasavada@gmail.com>
> > >
> > > wrote:
> > >
> > > Bump! Can somebody please review this?
> > >
> > > --
> > > Thanks,
> > > M.Sai Sandeep
> > >
> > >
> >
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi Maulin,
In SSL scenarios, I imagine security providers introduced by KIP-492 are
likely to be most useful when you want to use third party providers. The
biggest advantage of the config from that KIP is that you don't need to
write much code to integrate existing security providers into Kafka brokers
or clients. As I understand it, KIP-486 is a more convenient option for the
specific problem of loading keystores/truststores differently. It can be
achieved in theory with KIP-492, but KIP-486 is a much simpler option for
this case.
My concern about KIP-486 is that it introduces yet another interface into
our already complex security code, while only solving one particular use
case. Have you looked at
https://cwiki.apache.org/confluence/display/KAFKA/KIP-383%3A++Pluggable+interface+for+SSL+Factory?
The goal was to make
org.apache.kafka.common.security.ssl.SslEngineBuilder pluggable.
The code has already been refactored by Colin after that KIP was written,
making it easier to implement KIP-383. This should enable you to load your
keystores and truststores differently. Using a pluggable SslEngineBuilder
will also solve several other use cases at the same time. KIP-383 hasn't
been voted through yet, but perhaps you could take a look and we could
revive that instead if it solves your use case as well?
Regards,
Rajini
On Thu, Aug 29, 2019 at 6:42 PM Maulin Vasavada <ma...@gmail.com>
wrote:
> Hi Harsha
>
> Thank you. Appreciate your time and support on this. Let me go back do some
> more research and get back to you on the KeyStore interface part.
> Basically, if we return certs and keys in the interface then Kafka code
> will have to build KeyStore object - which is also reasonable.
>
> Thanks
> Maulin
>
> On Thu, Aug 29, 2019 at 10:01 AM Harsha Chintalapani <ka...@harsha.io>
> wrote:
>
> > Hi Maulin,
> > Use cases are clear now. I am +1 for moving forward
> > with the discussions on having such configurable option for users. But
> the
> > interfaces is proposed doesn't look right to me. We are still talking
> about
> > keystore interfaces. Given keystore's are used as filebased way of
> > transporting certificates I am not sure it will help the rest of the
> > user-base.
> > In short, I am +1 on the KIP's motivation and only have
> > questions around returning keystores instead of returning certs, private
> > keys etc. . If others in the community are ok with such interface we can
> > move forward.
> >
> > Thanks,
> > Harsha
> >
> >
> > On Wed, Aug 28, 2019 at 1:51 PM, Maulin Vasavada <
> > maulin.vasavada@gmail.com>
> > wrote:
> >
> > > Hi Harsha
> > >
> > > As we synced-up offline on this topic, we hope you don't have any more
> > > clarifications that you are seeking. If that is the case, can you
> please
> > > help us move this forward and discuss what changes you would expect on
> > the
> > > KIP design in order to make it valuable contribution?
> > >
> > > Just FYI - we verified our primary design change with the author of
> Sun's
> > > X509 Trustmanager implementation and the outcome is that what we are
> > > proposing makes sense at the heart of it - "Instead of writing
> > TrustManager
> > > just plugin the Trust store". We are open to discuss additional changes
> > > that you/anybody else would like to see on the functionality however.
> > >
> > > Thanks
> > > Maulin
> > >
> > > On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada <
> > maulin.vasavada@gmail.com>
> > > wrote:
> > >
> > > Hi Harsha
> > >
> > > Any response on my question? I feel this KIP is worth accommodating.
> Your
> > > help is much appreciated.
> > >
> > > Thanks
> > > Maulin
> > >
> > > On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada <
> maulin.vasavada@gmail.
> > > com> wrote:
> > >
> > > Hi Harsha
> > >
> > > I've examined the SPIFFE provider more and have one question -
> > >
> > > If SPIFFE didn't have a need to do checkSpiffeId() call at the below
> > > location, would you really still write the Provider? *OR* Would you
> just
> > > use TrustManagerFactory.init(KeyStore) signature to pass the KeyStore
> > from
> > > set of certs returned by spiffeIdManager. getTrustedCerts()?
> > >
> > >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > provider/CertificateUtils.java#L100
> > >
> > > /**
> > >
> > > * Validates that the SPIFFE ID is present and matches the SPIFFE ID
> > > configured in
> > > * the java.security property ssl.spiffe.accept
> > > *
> > > * If the authorized spiffe ids list is empty any spiffe id is
> authorized
> > > *
> > > * @param chain an array of X509Certificate that contains the Peer's
> SVID
> > > to be validated
> > > * @throws CertificateException when either the certificates doesn't
> have
> > a
> > > SPIFFE ID or the SPIFFE ID is not authorized
> > > */
> > > static void checkSpiffeId(X509Certificate[] chain) throws
> > > CertificateException {
> > >
> > > Thanks
> > > Maulin
> > >
> > > On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani <ka...@harsha.io>
> > > wrote:
> > >
> > > Maulin,
> > > The code parts you are pointing are specific for Spiffe and if
> > > you are talking about validate method which uses PKIX check like any
> > other
> > > provider does.
> > > If you want to default to SunJSSE everywhere you can do so by
> delegating
> > > the calls in these methods to SunJSSE provider.
> > >
> > > TrustManagerFactory tmf = TrustManagerFactory
> > > .getInstance(TrustManagerFactory.getDefaultAlgorithm());and use
> > > tmf.chekServerTrusted()
> > > or use
> > > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> > > TrustManagerFactory.html#getInstance(java.lang.String)if you want a
> > > specific provider.
> > >
> > > -Harsha
> > >
> > > On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada <
> maulin.vasavada@gmail.
> > > com>
> > > wrote:
> > >
> > > Okay, so I take that you guys agree that I have to write a 'custom'
> > > algorithm and a provider to make it work , correct?
> > >
> > > Now, for Harsha's comment "Here the 'Custom' Algorithm is not an
> > > implementation per say , ..." , I diagree. You can refer to https://
> > >
> > >
> github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/
> > >
> > > SpiffeTrustManager.java#L91 <http://spiffetrustmanager.java/#L91> and
> > >
> > >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > >
> > > provider/CertificateUtils.java#L100
> > >
> > > "that code" is the customization you have for the custom way to check
> > > something on top of regular checks. That method is NOT doing custom
> > > truststore loading. It is validating/verifying something in the
> > >
> > > "custom"
> > >
> > > way with spiffeId.
> > > I bet that without that you won't have a need of the custom algorithm
> > >
> > > in
> > >
> > > the first place.
> > >
> > > Let me know if you agree to this.
> > >
> > > Thanks
> > > Maulin
> > >
> > > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <mp...@gmail.com>
> > >
> > > wrote:
> > >
> > > Hi Maulin, thanks for the discussion. As Harsha pointed out, to use the
> > > KIP492, you need to create a new provider, register a *new* custom
> > > algorithm for your keymanager and trustmanager factory implementations.
> > > After this, the kafka server configuration can be done as given below
> > >
> > > # Register the provider class with custom algorithm, say CUSTOM
> > >
> > > security.
> > >
> > > provider.classes=com.company.security.CustomProvider
> > > <http://provider.classes=com.company.security.customprovider/>
> > > <http://security.provider.classes
> > >
> > > =com.company.security.customprovider/>
> > >
> > > # Register the keymanager and trustmanager algorithms
> > > # These algorithms indicate that the Keymanager and Trustmanagers
> > > registered under the algorithm "CUSTOM" needs to be used
> > > ssl.trustmanager.algorithm=CUSTOM
> > > ssl.keymanager.algorithm=CUSTOM
> > >
> > > And the customprovider looks like this...
> > >
> > > public class CustomProvider extends Provider {
> > > public CustomProvider() {
> > > super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and TrustStore");
> > > super.put("KeyManagerFactory.CUSTOM", "customKeyManagerFactory");
> > > super.put("TrustManagerFactory.CUSTOM",
> > > "customTrustManagerFactory");
> > > }
> > > }
> > >
> > > The PR for this is in review and can be found here:
> > >
> > > https://github.com/
> > >
> > > apache/kafka/pull/7090
> > > This PR includes the fixed insertProviderAt function call.
> > >
> > > On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani <ka...@harsha.io>
> > > wrote:
> > >
> > > Answers are in-line
> > >
> > > On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada <
> > >
> > > maulin.vasavada@gmail.
> > >
> > > com
> > >
> > > wrote:
> > >
> > > Hi Colin
> > >
> > > When I refer to "standard" or "custom" algorithms I am following Java
> > > security Provider Terminology. You can refer to
> > > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> > > StandardNames.html#TrustManagerFactory link I provided earlier in the
> > > emails. It says PKIX is the default Algorithm for TrustManagerFactory.
> > >
> > > 1. For SPIFFE, I am not sure why you are saying 'it does not implement
> > > custom algorithms' because the following file clearly indicates that it
> > > does use custom algorithm-
> > >
> > >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > >
> > > provider/SpiffeProvider.java#L17
> > >
> > > Algorithm value:
> > >
> > >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > >
> > > provider/SpiffeProviderConstants.java#L6
> > >
> > > @Harsha do you want to chime in since you use that provider?
> > >
> > > Here the "Custom" Algorithm is not an implementation per say , rather
> > >
> > > used
> > >
> > > to invoke the custom trust store factory and key manager factory. You
> > >
> > > are
> > >
> > > not moving away from "standard" alogrithms that are available.
> > >
> > >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > >
> > > provider/SpiffeTrustManager.java
> > >
> > > As you can see it delegates all the calls of verification of
> > >
> > > certificate
> > >
> > > to
> > >
> > > the default implementation available.
> > > So in our implementation we still use PKIX to verify the certificate
> > > chain. So you are not losing anything here and Spiffe is not
> > >
> > > reimplementing
> > >
> > > the verification process.
> > >
> > > 2. I already mentioned in my 3rd point, in my previous post, why using
> > >
> > > ssl.provider does NOT work. I updated KIP-486 in "rejected
> > >
> > > alternatives"
> > >
> > > also why ssl.provider does not work.
> > >
> > > As mentioned before , provider is the right way to go. I am still not
> > > understanding the gap is.
> > > If I understand correctly your argument is , provider is going to ask
> > >
> > > to
> > >
> > > implement a custom algorithm.
> > > My answer to that is , no you are not re-implementing the algorithm.
> > >
> > > Please
> > >
> > > check the above link , TrustManager implementation it delegates the
> > >
> > > calls
> > >
> > > back. There is no need to implement your own here.
> > >
> > > 3. Security.insertProviderAt() comments were based on assumption if
> > >
> > > KIP-492
> > >
> > > changes are done and we use that mechanism to configure providers
> > >
> > > instead
> > >
> > > of ssl.provider configuration.
> > >
> > > KIP-492 has patch available and going through review.
> > >
> > > Can you read my all the points, I mentioned in my previous post, very
> > >
> > > carefully? I am covering all the aspects in explaining. I am open to
> > >
> > > still
> > >
> > > discuss more to clarify any doubts.
> > >
> > > "3. If we just use existing ssl.provider kafka configuration then our
> > > provider will be used in SSLContext.getInstance(protocol, provider)
> > >
> > > call
> > >
> > > in
> > >
> > > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/> <
> > > http://sslfactory.java/>
> > >
> > > and
> > >
> > > if our provider does not have
> > > implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we
> > >
> > > tested
> > >
> > > it). Example: In MyProvider sample above you see that I didn't add
> > > SSLContext.TLSv1 as
> > > "Service+Algorithm" and that didn't work for me. In SPIFFE provider you
> > > don't have this challenge since you are planning to bypass ssl.provider
> > >
> > > as
> > >
> > > you mention in the KIP-492."
> > >
> > > Yes here you need to pass the protocol that your
> > >
> > > KeyManager/TrustManager
> > >
> > > registered with and in no way its deviating from TLS RFC spec.
> > >
> > >
> https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/
> > >
> > > javax/net/ssl/SSLContext.java#L134
> > >
> > > My suggestion here is for you to implement a simple Security Provider
> > >
> > > as
> > >
> > > you did before and register a Algorithm. You can use the existing
> > > implementation in SpiffeProvider and plug in changes where you need to
> > > retrieve the certificates from by making RPC call.
> > >
> > > Run an end-to-end test with Kafka broker coming up with your provider
> > > making calls to RPC call. You do need to pass the "custom algorithm"
> > >
> > > that
> > >
> > > you registered in your key manager to invoke the provider. I think your
> > > concern is this approach is replacing the existing known ciphersuites
> > >
> > > and
> > >
> > > certificate validation provided by java. Which its not.
> > >
> > > Now test the TLS connection you can do so via openssl -s_client options
> > >
> > > to
> > >
> > > test if everything else is working.
> > >
> > > I am happy to share configs that we used if you are interested.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > Thanks
> > > Maulin
> > >
> > > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <cm...@apache.org>
> > >
> > > wrote:
> > >
> > > Hi Maulin,
> > >
> > > A lot of JSSE providers don't implement custom algorithms. Spire is a
> > >
> > > good
> > >
> > > example of a JSSE provider that doesn't, and yet is still useful to
> > >
> > > many
> > >
> > > people. Your JSSE provider can work fine even if it doesn't implement a
> > > custom algorithm.
> > >
> > > Maybe I'm missing something, but I don't understand the discussion of
> > > Security.insertProviderAt() that you included. SslEngineBuilder doesn't
> > >
> > > use
> > >
> > > that API to get the security provider. Instead, it calls
> > > "SSLContext.getInstance(protocol, provider)", where provider is the
> > >
> > > name
> > >
> > > of the provider.
> > >
> > > best,
> > > Colin
> > >
> > > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
> > >
> > > On top of everything above I feel strongly to add the 4th point which
> > >
> > > is
> > >
> > > based on Java APIs for TrustManagerFactory.init(KeyStore) (
> > >
> > > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> > > TrustManagerFactory.html#init(java.security.KeyStore
> > > <http://java.security.keystore/>
> > > <http://java.security.keystore/>)
> > > )
> > >
> > > and KeyManagerFactory.init(KeyStore, char[]) (
> > >
> > >
> >
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory
> > > .
> > >
> > > html#init(java.security.KeyStore <http://java.security.keystore/>
> > <http://
> > > java.security.keystore/
> > > ,%20char[])
> > >
> > > ).
> > >
> > > 4. The above APIs are intended to support providing "trust/key
> > >
> > > material"
> > >
> > > from the user without having to write their own
> > >
> > > TrustManager/KeyManagers.
> > >
> > > To quote from the TrustManagerFactory.init()'s documentation
> > >
> > > "Initializes
> > >
> > > this factory with a source of certificate authorities and related trust
> > > material."
> > > To quote from the KeyManagerFactory.init()'s documentation "Initializes
> > > this factory with a source of key material."
> > >
> > > Based on this it is clear that there is a flexibility provided by Java
> > >
> > > to
> > >
> > > to enable developers to provide the required trust/key material loaded
> > >
> > > from
> > >
> > > "anywhere" without requiring them to write custom provider OR trust/key
> > > managers. This same flexibility is reflected in Kafka code also where
> > >
> > > it
> > >
> > > loads the trust/keys from a local file and doesn't require writing a
> > > Provider necessarily. If we do NOT have a custom algorithm, it makes
> > >
> > > less
> > >
> > > sense to write a Provider.
> > >
> > > Thanks
> > > Maulin
> > >
> > > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
> > >
> > > maulin.vasavada@gmail.com>
> > >
> > > wrote:
> > >
> > > Hi Harsha/Colin
> > >
> > > I did the sample with a custom Provider for TrustStoreManager and tried
> > > using ssl.provider Kafka config AND the way KIP-492 is suggesting (by
> > > adding Provider programmatically instead of relying on
> > >
> > > ssl.provider+java.
> > >
> > > security. The below sample is followed by my detailed findings. I'll
> > > appreciate if you can go through it carefully and see
> > >
> > > if you
> > >
> > > see my point.
> > >
> > > package providertest;
> > >
> > > import java.security.Provider <http://java.security.provider/>
> <http://
> > > java.security.provider/>
> > >
> > > <http://
> > >
> > > java.security.provider/>;
> > >
> > > public class MyProvider extends Provider {
> > >
> > > private static final String name = "MyProvider"; private static double
> > > version = 1.0d;
> > > private static String info = "Maulin's SSL Provider v"+version;
> > >
> > > public MyProvider() {
> > > super(name, version, info);
> > > this.put("TrustManagerFactory.PKIX",
> > >
> > > "providertest.MyTrustManagerFactory");
> > >
> > > }
> > > }
> > >
> > > *Details:*
> > >
> > > KIP-492 documents that it will use Security.addProvider() assuming it
> > >
> > > will
> > >
> > > add it as position '0' which is not a correct assumption. The
> > > addProvider()'s documentation says it will add it to the last available
> > > position. You may want to correct that to say
> > > Security.insertProviderAt(provider, 1).
> > >
> > > Now coming back to our specific discussion,
> > >
> > > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add
> > >
> > > that
> > >
> > > provider in the provider list via Security.addProvider() the position
> > >
> > > where
> > >
> > > it gets added doesn't matter (even if you don't end up adding it as
> > >
> > > first
> > >
> > > entry) since that is the ONLY provider for SPIFFE specific algorithm
> > >
> > > you
> > >
> > > might have.
> > >
> > > We do *not* have custom algorithm for Key/Trust StoreMangers. Which
> > >
> > > means
> > >
> > > we have to use X509, PKIX etc "Standard Algorithms" ((
> > >
> > > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> > > StandardNames.html
> > > ))
> > >
> > > in our provider to override the TrustStoreManager (see my sample code)
> > >
> > > and
> > >
> > > KeyStoreManger and KeyManager. This creates another challenge
> > >
> > > mentioned in
> > >
> > > the below point.
> > >
> > > 2. In order to make our Provider for loading custom TrustStore work, we
> > > have to add the provider as 'first' in the list since there are others
> > >
> > > with
> > >
> > > the same algorithm.
> > >
> > > However, the programatic way of adding provider
> > > (Security.insertProviderAt()) is *not* deterministic for ordering since
> > > different code can call the method for a different provider and
> > >
> > > depending
> > >
> > > upon the order of the call our provider can be first or pushed down the
> > > list. This can happen very well in any client application using Kafka.
> > >
> > > This
> > >
> > > is specially problematic for a case when you want to guarantee order
> > >
> > > for a
> > >
> > > Provider having "Standard Algorithms".
> > >
> > > If we add our provider in java.security file that definitely guarantees
> > > the order(unless somebody calls removeProvider() which is less
> > >
> > > likely). But
> > >
> > > if we add our provider in java.security file it will defeat the
> > >
> > > purpose of
> > >
> > > the KIP-492.
> > >
> > > In the gist - Apache Kafka must not rely on "un-deterministic" method
> > >
> > > to
> > >
> > > rely on Provider ordering.
> > >
> > > 3. If we just use existing ssl.provider kafka configuration then our
> > > provider will be used in SSLContext.getInstance(protocol, provider)
> > >
> > > call in
> > >
> > > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/> <
> > > http://sslfactory.java/>
> > >
> > > and
> > >
> > > if our provider does not have implementation for
> > > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks
> > >
> > > (we
> > >
> > > tested it). Example:
> > >
> > > In
> > >
> > > MyProvider sample above you see that I didn't add SSLContext.TLSv1 as
> > > "Service+Algorithm" and that didn't work for me. In SPIFFE provider you
> > > don't have this challenge since you are planning to bypass
> > >
> > > ssl.provider as
> > >
> > > you mention in the KIP-492.
> > >
> > > *Overall summary,*
> > >
> > > 1. Any provider based mechanisms- a) existing ssl.provider and
> > >
> > > b)KIP-492,
> > >
> > > for loading key/trust store using "Standard Algorithms" do not work
> > >
> > > 2. Approach suggested in our KIP-486 works without any issue and it is
> > > *not* our context specific solve
> > >
> > > 3. Based on above we feel KIP-492 and KIP-486 are complimentary changes
> > > and not contradicting or redundent.
> > >
> > > If you want we can do a joint session somehow to walk through the
> > >
> > > sample I
> > >
> > > have and various experiments I did. I would encourage you to do similar
> > > exercise by writing a Provider for "Standard Algorithm" for
> > > TrustStoreManager (like our needs) and see what you find since only
> > >
> > > writing
> > >
> > > samples can bring out the complexity/challenges we face.
> > >
> > > Thanks
> > > Maulin
> > >
> > > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <
> > >
> > > maulin.vasavada@gmail.
> > >
> > > com> wrote:
> > >
> > > Just to update - still working on it. Get to work only on and off on
> > >
> > > it :(
> > >
> > > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
> > >
> > > maulin.vasavada@gmail.com>
> > >
> > > wrote:
> > >
> > > Hi Harsha
> > >
> > > Let me try to write samples and will let you know.
> > >
> > > Thanks
> > > Maulin
> > >
> > > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <ha...@gmail.com>
> > >
> > > wrote:
> > >
> > > Hi Maulin,
> > > With java security providers can be as custom you would
> > >
> > > like
> > >
> > > it to
> > > be. If you only want to to implement a custom way of loading the
> > >
> > > keystore
> > >
> > > and truststore and not implement any protocol/encryption handling you
> > >
> > > can
> > >
> > > leave them empty and no need to implement. Have you looked into the
> > >
> > > links I
> > >
> > > pasted before?
> > >
> > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >
> > >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> > >
> > > java
> > >
> > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > spiffe-security-provider/src/main/java/spiffe/api/provider/
> > > SpiffeTrustManager.java <http://spiffetrustmanager.java/> <http://
> > > spiffetrustmanager.java/>
> > >
> > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >
> > >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > >
> > > java
> > >
> > > Can you please tell me which methods are too complex in above to
> > >
> > > implement
> > >
> > > or unnecessary? You are changing anything in SSL/TLS implementations
> > > provided by
> > >
> > > All of the implementations delegating the checks to the default
> > > implementation anyway.
> > > Spire agent is an example, its nothing but a GRPC server listening
> > >
> > > on a
> > >
> > > unix domain socket . Above code is making a RPC call to the local
> > >
> > > daemon
> > >
> > > to
> > > get the certificate and keys. The mechanics are pretty much same as
> > >
> > > what
> > >
> > > you are asking for.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
> > >
> > > maulin.vasavada@gmail.com>
> > >
> > > wrote:
> > >
> > > Imagine a scenario like - We know we have a custom KMS and as a
> > >
> > > Kafka
> > >
> > > owner
> > >
> > > we want to comply to using that KMS source to load keys/certs. As
> > >
> > > a
> > >
> > > Kafka
> > >
> > > owner we know how to integrate with KMS but doesn't necessarily
> > >
> > > have
> > >
> > > to
> > >
> > > know anything about cipher suites, algorithms, and SSL/TLS
> > >
> > > implementation.
> > >
> > > Going the Provider way requires to know lot more than we should,
> > >
> > > isn't it?
> > >
> > > Not that we would have concern/shy-away knowing those details -
> > >
> > > but
> > >
> > > if we
> > >
> > > don't have to - why should we?
> > >
> > > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
> > >
> > > maulin.vasavada@gmail.com>
> > >
> > > wrote:
> > >
> > > Hi Harsha
> > >
> > > We don't have spire (or similar) agents and we do not have
> > >
> > > keys/certs
> > >
> > > locally on any brokers. To elaborate more on my previous email,
> > >
> > > I agree that Java security Providers are used in much broader
> > >
> > > sense
> > >
> > > - to
> > >
> > > have a particular implementation of an algorithm, use specific
> > >
> > > cipher
> > >
> > > suites for SSL , OR in our current team's case have a
> > >
> > > particular
> > >
> > > way to
> > >
> > > leverage pre-generated SSL sessions. However, the scope of our
> > >
> > > KIP
> > >
> > > (486)
> > >
> > > is
> > >
> > > much restricted than that. We merely intend to provide a custom
> > > keystore/truststore for our SSL connections and not really worry
> > >
> > > about
> > >
> > > underlying specific SSL/TLS implementation. This simplifies it
> > >
> > > a
> > >
> > > lot for
> > >
> > > us to keep the concerns separate and clear.
> > >
> > > I feel our approach is more complimentary such that it allows
> > >
> > > for
> > >
> > > using
> > >
> > > keystores of choice while retaining the flexibility to use any
> > > underlying/available Provider for actually making the SSL call.
> > >
> > > We agree with KIP-492's approach based on Providers (and Java's
> > > recommendation), but also strongly believe that our approach can
> > >
> > > compliment
> > >
> > > it very effectively for reasons explained above.
> > >
> > > Thanks
> > > Maulin
> > >
> > > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <
> > >
> > > kafka@harsha.io
> > >
> > > wrote:
> > >
> > > Hi Maulin,
> > >
> > > On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
> > >
> > > maulin.vasavada@gmail.
> > >
> > > com>
> > > wrote:
> > >
> > > Hi Harsha
> > >
> > > The reason we rejected the SslProvider route is that - we
> > >
> > > only
> > >
> > > needed
> > >
> > > a
> > >
> > > custom way to load keys/certs. Not touch any policy that
> > >
> > > existing
> > >
> > > Providers
> > >
> > > govern like SunJSSE Provider.
> > >
> > > We have exactly the same requirements to load certs and keys
> > >
> > > through
> > >
> > > spire
> > >
> > > agent. We used security.provider to do that exactly. I am not
> > >
> > > sure
> > >
> > > why
> > >
> > > you
> > >
> > > would be modifying any policies provided by default SunJSSE
> > >
> > > provider.
> > >
> > > Can
> > >
> > > you give me an example of having custom provider that will
> > >
> > > override an
> > >
> > > existing policy in SunJSSE provider.
> > >
> > > As pointed out earlier, this kip
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> > >
> > > allows
> > > you to load security.provider through config.
> > > Take a look at the examples I gave before
> > >
> > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >
> > >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > >
> > > java
> > >
> > > It registers KeyManagerFactory and TrustManagerFactory and
> > >
> > > Keystore
> > >
> > > algorithm.
> > > Implement your custom way of loading Keystore in here
> > >
> > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >
> > >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> > >
> > > java
> > >
> > > and Trust manager like here
> > >
> > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > spiffe-security-provider/src/main/java/spiffe/api/provider/
> > > SpiffeTrustManager.java <http://spiffetrustmanager.java/> <http://
> > > spiffetrustmanager.java/>
> > >
> > > In your Kafka client you can set the security.provider to your
> > >
> > > custom
> > >
> > > implementation and with this fix
> > > https://issues.apache.org/jira/browse/KAFKA-8191 you can set
> > > keyManagerAlgorigthm and trustManagerAlgorithm configs.
> > >
> > > All of this is in your clients and broker side and do not need
> > >
> > > to
> > >
> > > touch
> > >
> > > any
> > > policy changes at JVM level. You'll register the providers in
> > >
> > > the
> > >
> > > priority
> > >
> > > order and can still have SunJSSE provider and have your custom
> > >
> > > provider
> > >
> > > to
> > >
> > > implement the key and trust managers.
> > >
> > > The ask here is different than KIP-492. We don't have any need
> > >
> > > to
> > >
> > > modify/specify the algorithm parameter. Does that make sense?
> > >
> > > The ask in KIP is introducing new interfaces where the KIP's
> > > goal/motivation can be achieved through the security.provider
> > >
> > > and
> > >
> > > we
> > >
> > > worked
> > > on similar goal without touching any Keystore or Truststore
> > >
> > > interfaces.
> > >
> > > My advise is against changing or introducing new interfaces
> > >
> > > when
> > >
> > > it can
> > >
> > > work through security.provider.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > Thanks
> > >
> > > Maulin
> > >
> > > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
> > >
> > > kafka@harsha.io>
> > >
> > > wrote:
> > >
> > > In your KIP you added security. provider as rejected
> > >
> > > alternative
> > >
> > > and
> > >
> > > specified "its not the correct way". Do you mind explaining
> > >
> > > why
> > >
> > > its
> > >
> > > not? I
> > >
> > > didn't find any evidence in Java docs to say so. Contrary to
> > >
> > > your
> > >
> > > statement
> > >
> > > it does say in the java docs
> > > " However, please note that a provider can be used to
> > >
> > > implement
> > >
> > > any
> > >
> > > security service in Java that uses a pluggable architecture
> > >
> > > with
> > >
> > > a
> > >
> > > choice
> > >
> > > of implementations that fit underneath."
> > >
> > > Java Security Providers have been used by other projects to
> > >
> > > provide
> > >
> > > such
> > >
> > > integration . I am not sure if you looked into Spiffe
> > >
> > > project to
> > >
> > > efficiently distribute certificates but here is an example of
> > >
> > > Java
> > >
> > > provider
> > >
> > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >
> > >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > >
> > > java which
> > > obtains certificates from local daemons.
> > > These integrations are being used in Tomcat, Jetty etc.. We
> > >
> > > are
> > >
> > > also
> > >
> > > using
> > >
> > > Security provider to do the same in our Kafka clusters. So
> > >
> > > unless I
> > >
> > > see
> > >
> > > more evidence why security.provider doesn't work for you
> > >
> > > adding
> > >
> > > new
> > >
> > > interfaces while there exists more cleaner way of achieving
> > >
> > > the
> > >
> > > goals
> > >
> > > of
> > >
> > > this KIP is unnecessary and breaks the well known security
> > >
> > > interfaces
> > >
> > > provided by Java itself.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
> > >
> > > kafka@harsha.io
> > >
> > > wrote:
> > >
> > > Hi Maulin,
> > > Not sure if you looked at my previous replies. This
> > >
> > > changes
> > >
> > > are not required as there is already security Provider to do
> > >
> > > what you
> > >
> > > are
> > >
> > > proposing. This KIP
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/
> > >
> > > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> > >
> > > also
> > >
> > > addresses easy registration of such providers.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
> > >
> > > <maulin.vasavada@gmail.
> > >
> > > com> wrote:
> > >
> > > Bump! Can somebody please review this?
> > >
> > > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
> > >
> > > maulin.vasavada@gmail.com>
> > >
> > > wrote:
> > >
> > > Bump! Can somebody please review this?
> > >
> > > --
> > > Thanks,
> > > M.Sai Sandeep
> > >
> > >
> >
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Maulin Vasavada <ma...@gmail.com>.
Hi Harsha
Thank you. Appreciate your time and support on this. Let me go back do some
more research and get back to you on the KeyStore interface part.
Basically, if we return certs and keys in the interface then Kafka code
will have to build KeyStore object - which is also reasonable.
Thanks
Maulin
On Thu, Aug 29, 2019 at 10:01 AM Harsha Chintalapani <ka...@harsha.io>
wrote:
> Hi Maulin,
> Use cases are clear now. I am +1 for moving forward
> with the discussions on having such configurable option for users. But the
> interfaces is proposed doesn't look right to me. We are still talking about
> keystore interfaces. Given keystore's are used as filebased way of
> transporting certificates I am not sure it will help the rest of the
> user-base.
> In short, I am +1 on the KIP's motivation and only have
> questions around returning keystores instead of returning certs, private
> keys etc. . If others in the community are ok with such interface we can
> move forward.
>
> Thanks,
> Harsha
>
>
> On Wed, Aug 28, 2019 at 1:51 PM, Maulin Vasavada <
> maulin.vasavada@gmail.com>
> wrote:
>
> > Hi Harsha
> >
> > As we synced-up offline on this topic, we hope you don't have any more
> > clarifications that you are seeking. If that is the case, can you please
> > help us move this forward and discuss what changes you would expect on
> the
> > KIP design in order to make it valuable contribution?
> >
> > Just FYI - we verified our primary design change with the author of Sun's
> > X509 Trustmanager implementation and the outcome is that what we are
> > proposing makes sense at the heart of it - "Instead of writing
> TrustManager
> > just plugin the Trust store". We are open to discuss additional changes
> > that you/anybody else would like to see on the functionality however.
> >
> > Thanks
> > Maulin
> >
> > On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada <
> maulin.vasavada@gmail.com>
> > wrote:
> >
> > Hi Harsha
> >
> > Any response on my question? I feel this KIP is worth accommodating. Your
> > help is much appreciated.
> >
> > Thanks
> > Maulin
> >
> > On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada < maulin.vasavada@gmail.
> > com> wrote:
> >
> > Hi Harsha
> >
> > I've examined the SPIFFE provider more and have one question -
> >
> > If SPIFFE didn't have a need to do checkSpiffeId() call at the below
> > location, would you really still write the Provider? *OR* Would you just
> > use TrustManagerFactory.init(KeyStore) signature to pass the KeyStore
> from
> > set of certs returned by spiffeIdManager. getTrustedCerts()?
> >
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > provider/CertificateUtils.java#L100
> >
> > /**
> >
> > * Validates that the SPIFFE ID is present and matches the SPIFFE ID
> > configured in
> > * the java.security property ssl.spiffe.accept
> > *
> > * If the authorized spiffe ids list is empty any spiffe id is authorized
> > *
> > * @param chain an array of X509Certificate that contains the Peer's SVID
> > to be validated
> > * @throws CertificateException when either the certificates doesn't have
> a
> > SPIFFE ID or the SPIFFE ID is not authorized
> > */
> > static void checkSpiffeId(X509Certificate[] chain) throws
> > CertificateException {
> >
> > Thanks
> > Maulin
> >
> > On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani <ka...@harsha.io>
> > wrote:
> >
> > Maulin,
> > The code parts you are pointing are specific for Spiffe and if
> > you are talking about validate method which uses PKIX check like any
> other
> > provider does.
> > If you want to default to SunJSSE everywhere you can do so by delegating
> > the calls in these methods to SunJSSE provider.
> >
> > TrustManagerFactory tmf = TrustManagerFactory
> > .getInstance(TrustManagerFactory.getDefaultAlgorithm());and use
> > tmf.chekServerTrusted()
> > or use
> > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> > TrustManagerFactory.html#getInstance(java.lang.String)if you want a
> > specific provider.
> >
> > -Harsha
> >
> > On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada < maulin.vasavada@gmail.
> > com>
> > wrote:
> >
> > Okay, so I take that you guys agree that I have to write a 'custom'
> > algorithm and a provider to make it work , correct?
> >
> > Now, for Harsha's comment "Here the 'Custom' Algorithm is not an
> > implementation per say , ..." , I diagree. You can refer to https://
> >
> > github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/
> >
> > SpiffeTrustManager.java#L91 <http://spiffetrustmanager.java/#L91> and
> >
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> >
> > provider/CertificateUtils.java#L100
> >
> > "that code" is the customization you have for the custom way to check
> > something on top of regular checks. That method is NOT doing custom
> > truststore loading. It is validating/verifying something in the
> >
> > "custom"
> >
> > way with spiffeId.
> > I bet that without that you won't have a need of the custom algorithm
> >
> > in
> >
> > the first place.
> >
> > Let me know if you agree to this.
> >
> > Thanks
> > Maulin
> >
> > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <mp...@gmail.com>
> >
> > wrote:
> >
> > Hi Maulin, thanks for the discussion. As Harsha pointed out, to use the
> > KIP492, you need to create a new provider, register a *new* custom
> > algorithm for your keymanager and trustmanager factory implementations.
> > After this, the kafka server configuration can be done as given below
> >
> > # Register the provider class with custom algorithm, say CUSTOM
> >
> > security.
> >
> > provider.classes=com.company.security.CustomProvider
> > <http://provider.classes=com.company.security.customprovider/>
> > <http://security.provider.classes
> >
> > =com.company.security.customprovider/>
> >
> > # Register the keymanager and trustmanager algorithms
> > # These algorithms indicate that the Keymanager and Trustmanagers
> > registered under the algorithm "CUSTOM" needs to be used
> > ssl.trustmanager.algorithm=CUSTOM
> > ssl.keymanager.algorithm=CUSTOM
> >
> > And the customprovider looks like this...
> >
> > public class CustomProvider extends Provider {
> > public CustomProvider() {
> > super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and TrustStore");
> > super.put("KeyManagerFactory.CUSTOM", "customKeyManagerFactory");
> > super.put("TrustManagerFactory.CUSTOM",
> > "customTrustManagerFactory");
> > }
> > }
> >
> > The PR for this is in review and can be found here:
> >
> > https://github.com/
> >
> > apache/kafka/pull/7090
> > This PR includes the fixed insertProviderAt function call.
> >
> > On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani <ka...@harsha.io>
> > wrote:
> >
> > Answers are in-line
> >
> > On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada <
> >
> > maulin.vasavada@gmail.
> >
> > com
> >
> > wrote:
> >
> > Hi Colin
> >
> > When I refer to "standard" or "custom" algorithms I am following Java
> > security Provider Terminology. You can refer to
> > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> > StandardNames.html#TrustManagerFactory link I provided earlier in the
> > emails. It says PKIX is the default Algorithm for TrustManagerFactory.
> >
> > 1. For SPIFFE, I am not sure why you are saying 'it does not implement
> > custom algorithms' because the following file clearly indicates that it
> > does use custom algorithm-
> >
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> >
> > provider/SpiffeProvider.java#L17
> >
> > Algorithm value:
> >
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> >
> > provider/SpiffeProviderConstants.java#L6
> >
> > @Harsha do you want to chime in since you use that provider?
> >
> > Here the "Custom" Algorithm is not an implementation per say , rather
> >
> > used
> >
> > to invoke the custom trust store factory and key manager factory. You
> >
> > are
> >
> > not moving away from "standard" alogrithms that are available.
> >
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> >
> > provider/SpiffeTrustManager.java
> >
> > As you can see it delegates all the calls of verification of
> >
> > certificate
> >
> > to
> >
> > the default implementation available.
> > So in our implementation we still use PKIX to verify the certificate
> > chain. So you are not losing anything here and Spiffe is not
> >
> > reimplementing
> >
> > the verification process.
> >
> > 2. I already mentioned in my 3rd point, in my previous post, why using
> >
> > ssl.provider does NOT work. I updated KIP-486 in "rejected
> >
> > alternatives"
> >
> > also why ssl.provider does not work.
> >
> > As mentioned before , provider is the right way to go. I am still not
> > understanding the gap is.
> > If I understand correctly your argument is , provider is going to ask
> >
> > to
> >
> > implement a custom algorithm.
> > My answer to that is , no you are not re-implementing the algorithm.
> >
> > Please
> >
> > check the above link , TrustManager implementation it delegates the
> >
> > calls
> >
> > back. There is no need to implement your own here.
> >
> > 3. Security.insertProviderAt() comments were based on assumption if
> >
> > KIP-492
> >
> > changes are done and we use that mechanism to configure providers
> >
> > instead
> >
> > of ssl.provider configuration.
> >
> > KIP-492 has patch available and going through review.
> >
> > Can you read my all the points, I mentioned in my previous post, very
> >
> > carefully? I am covering all the aspects in explaining. I am open to
> >
> > still
> >
> > discuss more to clarify any doubts.
> >
> > "3. If we just use existing ssl.provider kafka configuration then our
> > provider will be used in SSLContext.getInstance(protocol, provider)
> >
> > call
> >
> > in
> >
> > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/> <
> > http://sslfactory.java/>
> >
> > and
> >
> > if our provider does not have
> > implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we
> >
> > tested
> >
> > it). Example: In MyProvider sample above you see that I didn't add
> > SSLContext.TLSv1 as
> > "Service+Algorithm" and that didn't work for me. In SPIFFE provider you
> > don't have this challenge since you are planning to bypass ssl.provider
> >
> > as
> >
> > you mention in the KIP-492."
> >
> > Yes here you need to pass the protocol that your
> >
> > KeyManager/TrustManager
> >
> > registered with and in no way its deviating from TLS RFC spec.
> >
> > https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/
> >
> > javax/net/ssl/SSLContext.java#L134
> >
> > My suggestion here is for you to implement a simple Security Provider
> >
> > as
> >
> > you did before and register a Algorithm. You can use the existing
> > implementation in SpiffeProvider and plug in changes where you need to
> > retrieve the certificates from by making RPC call.
> >
> > Run an end-to-end test with Kafka broker coming up with your provider
> > making calls to RPC call. You do need to pass the "custom algorithm"
> >
> > that
> >
> > you registered in your key manager to invoke the provider. I think your
> > concern is this approach is replacing the existing known ciphersuites
> >
> > and
> >
> > certificate validation provided by java. Which its not.
> >
> > Now test the TLS connection you can do so via openssl -s_client options
> >
> > to
> >
> > test if everything else is working.
> >
> > I am happy to share configs that we used if you are interested.
> >
> > Thanks,
> > Harsha
> >
> > Thanks
> > Maulin
> >
> > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <cm...@apache.org>
> >
> > wrote:
> >
> > Hi Maulin,
> >
> > A lot of JSSE providers don't implement custom algorithms. Spire is a
> >
> > good
> >
> > example of a JSSE provider that doesn't, and yet is still useful to
> >
> > many
> >
> > people. Your JSSE provider can work fine even if it doesn't implement a
> > custom algorithm.
> >
> > Maybe I'm missing something, but I don't understand the discussion of
> > Security.insertProviderAt() that you included. SslEngineBuilder doesn't
> >
> > use
> >
> > that API to get the security provider. Instead, it calls
> > "SSLContext.getInstance(protocol, provider)", where provider is the
> >
> > name
> >
> > of the provider.
> >
> > best,
> > Colin
> >
> > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
> >
> > On top of everything above I feel strongly to add the 4th point which
> >
> > is
> >
> > based on Java APIs for TrustManagerFactory.init(KeyStore) (
> >
> > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> > TrustManagerFactory.html#init(java.security.KeyStore
> > <http://java.security.keystore/>
> > <http://java.security.keystore/>)
> > )
> >
> > and KeyManagerFactory.init(KeyStore, char[]) (
> >
> >
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory
> > .
> >
> > html#init(java.security.KeyStore <http://java.security.keystore/>
> <http://
> > java.security.keystore/
> > ,%20char[])
> >
> > ).
> >
> > 4. The above APIs are intended to support providing "trust/key
> >
> > material"
> >
> > from the user without having to write their own
> >
> > TrustManager/KeyManagers.
> >
> > To quote from the TrustManagerFactory.init()'s documentation
> >
> > "Initializes
> >
> > this factory with a source of certificate authorities and related trust
> > material."
> > To quote from the KeyManagerFactory.init()'s documentation "Initializes
> > this factory with a source of key material."
> >
> > Based on this it is clear that there is a flexibility provided by Java
> >
> > to
> >
> > to enable developers to provide the required trust/key material loaded
> >
> > from
> >
> > "anywhere" without requiring them to write custom provider OR trust/key
> > managers. This same flexibility is reflected in Kafka code also where
> >
> > it
> >
> > loads the trust/keys from a local file and doesn't require writing a
> > Provider necessarily. If we do NOT have a custom algorithm, it makes
> >
> > less
> >
> > sense to write a Provider.
> >
> > Thanks
> > Maulin
> >
> > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
> >
> > maulin.vasavada@gmail.com>
> >
> > wrote:
> >
> > Hi Harsha/Colin
> >
> > I did the sample with a custom Provider for TrustStoreManager and tried
> > using ssl.provider Kafka config AND the way KIP-492 is suggesting (by
> > adding Provider programmatically instead of relying on
> >
> > ssl.provider+java.
> >
> > security. The below sample is followed by my detailed findings. I'll
> > appreciate if you can go through it carefully and see
> >
> > if you
> >
> > see my point.
> >
> > package providertest;
> >
> > import java.security.Provider <http://java.security.provider/> <http://
> > java.security.provider/>
> >
> > <http://
> >
> > java.security.provider/>;
> >
> > public class MyProvider extends Provider {
> >
> > private static final String name = "MyProvider"; private static double
> > version = 1.0d;
> > private static String info = "Maulin's SSL Provider v"+version;
> >
> > public MyProvider() {
> > super(name, version, info);
> > this.put("TrustManagerFactory.PKIX",
> >
> > "providertest.MyTrustManagerFactory");
> >
> > }
> > }
> >
> > *Details:*
> >
> > KIP-492 documents that it will use Security.addProvider() assuming it
> >
> > will
> >
> > add it as position '0' which is not a correct assumption. The
> > addProvider()'s documentation says it will add it to the last available
> > position. You may want to correct that to say
> > Security.insertProviderAt(provider, 1).
> >
> > Now coming back to our specific discussion,
> >
> > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add
> >
> > that
> >
> > provider in the provider list via Security.addProvider() the position
> >
> > where
> >
> > it gets added doesn't matter (even if you don't end up adding it as
> >
> > first
> >
> > entry) since that is the ONLY provider for SPIFFE specific algorithm
> >
> > you
> >
> > might have.
> >
> > We do *not* have custom algorithm for Key/Trust StoreMangers. Which
> >
> > means
> >
> > we have to use X509, PKIX etc "Standard Algorithms" ((
> >
> > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> > StandardNames.html
> > ))
> >
> > in our provider to override the TrustStoreManager (see my sample code)
> >
> > and
> >
> > KeyStoreManger and KeyManager. This creates another challenge
> >
> > mentioned in
> >
> > the below point.
> >
> > 2. In order to make our Provider for loading custom TrustStore work, we
> > have to add the provider as 'first' in the list since there are others
> >
> > with
> >
> > the same algorithm.
> >
> > However, the programatic way of adding provider
> > (Security.insertProviderAt()) is *not* deterministic for ordering since
> > different code can call the method for a different provider and
> >
> > depending
> >
> > upon the order of the call our provider can be first or pushed down the
> > list. This can happen very well in any client application using Kafka.
> >
> > This
> >
> > is specially problematic for a case when you want to guarantee order
> >
> > for a
> >
> > Provider having "Standard Algorithms".
> >
> > If we add our provider in java.security file that definitely guarantees
> > the order(unless somebody calls removeProvider() which is less
> >
> > likely). But
> >
> > if we add our provider in java.security file it will defeat the
> >
> > purpose of
> >
> > the KIP-492.
> >
> > In the gist - Apache Kafka must not rely on "un-deterministic" method
> >
> > to
> >
> > rely on Provider ordering.
> >
> > 3. If we just use existing ssl.provider kafka configuration then our
> > provider will be used in SSLContext.getInstance(protocol, provider)
> >
> > call in
> >
> > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/> <
> > http://sslfactory.java/>
> >
> > and
> >
> > if our provider does not have implementation for
> > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks
> >
> > (we
> >
> > tested it). Example:
> >
> > In
> >
> > MyProvider sample above you see that I didn't add SSLContext.TLSv1 as
> > "Service+Algorithm" and that didn't work for me. In SPIFFE provider you
> > don't have this challenge since you are planning to bypass
> >
> > ssl.provider as
> >
> > you mention in the KIP-492.
> >
> > *Overall summary,*
> >
> > 1. Any provider based mechanisms- a) existing ssl.provider and
> >
> > b)KIP-492,
> >
> > for loading key/trust store using "Standard Algorithms" do not work
> >
> > 2. Approach suggested in our KIP-486 works without any issue and it is
> > *not* our context specific solve
> >
> > 3. Based on above we feel KIP-492 and KIP-486 are complimentary changes
> > and not contradicting or redundent.
> >
> > If you want we can do a joint session somehow to walk through the
> >
> > sample I
> >
> > have and various experiments I did. I would encourage you to do similar
> > exercise by writing a Provider for "Standard Algorithm" for
> > TrustStoreManager (like our needs) and see what you find since only
> >
> > writing
> >
> > samples can bring out the complexity/challenges we face.
> >
> > Thanks
> > Maulin
> >
> > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <
> >
> > maulin.vasavada@gmail.
> >
> > com> wrote:
> >
> > Just to update - still working on it. Get to work only on and off on
> >
> > it :(
> >
> > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
> >
> > maulin.vasavada@gmail.com>
> >
> > wrote:
> >
> > Hi Harsha
> >
> > Let me try to write samples and will let you know.
> >
> > Thanks
> > Maulin
> >
> > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <ha...@gmail.com>
> >
> > wrote:
> >
> > Hi Maulin,
> > With java security providers can be as custom you would
> >
> > like
> >
> > it to
> > be. If you only want to to implement a custom way of loading the
> >
> > keystore
> >
> > and truststore and not implement any protocol/encryption handling you
> >
> > can
> >
> > leave them empty and no need to implement. Have you looked into the
> >
> > links I
> >
> > pasted before?
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> >
> > java
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > spiffe-security-provider/src/main/java/spiffe/api/provider/
> > SpiffeTrustManager.java <http://spiffetrustmanager.java/> <http://
> > spiffetrustmanager.java/>
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> >
> > java
> >
> > Can you please tell me which methods are too complex in above to
> >
> > implement
> >
> > or unnecessary? You are changing anything in SSL/TLS implementations
> > provided by
> >
> > All of the implementations delegating the checks to the default
> > implementation anyway.
> > Spire agent is an example, its nothing but a GRPC server listening
> >
> > on a
> >
> > unix domain socket . Above code is making a RPC call to the local
> >
> > daemon
> >
> > to
> > get the certificate and keys. The mechanics are pretty much same as
> >
> > what
> >
> > you are asking for.
> >
> > Thanks,
> > Harsha
> >
> > On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
> >
> > maulin.vasavada@gmail.com>
> >
> > wrote:
> >
> > Imagine a scenario like - We know we have a custom KMS and as a
> >
> > Kafka
> >
> > owner
> >
> > we want to comply to using that KMS source to load keys/certs. As
> >
> > a
> >
> > Kafka
> >
> > owner we know how to integrate with KMS but doesn't necessarily
> >
> > have
> >
> > to
> >
> > know anything about cipher suites, algorithms, and SSL/TLS
> >
> > implementation.
> >
> > Going the Provider way requires to know lot more than we should,
> >
> > isn't it?
> >
> > Not that we would have concern/shy-away knowing those details -
> >
> > but
> >
> > if we
> >
> > don't have to - why should we?
> >
> > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
> >
> > maulin.vasavada@gmail.com>
> >
> > wrote:
> >
> > Hi Harsha
> >
> > We don't have spire (or similar) agents and we do not have
> >
> > keys/certs
> >
> > locally on any brokers. To elaborate more on my previous email,
> >
> > I agree that Java security Providers are used in much broader
> >
> > sense
> >
> > - to
> >
> > have a particular implementation of an algorithm, use specific
> >
> > cipher
> >
> > suites for SSL , OR in our current team's case have a
> >
> > particular
> >
> > way to
> >
> > leverage pre-generated SSL sessions. However, the scope of our
> >
> > KIP
> >
> > (486)
> >
> > is
> >
> > much restricted than that. We merely intend to provide a custom
> > keystore/truststore for our SSL connections and not really worry
> >
> > about
> >
> > underlying specific SSL/TLS implementation. This simplifies it
> >
> > a
> >
> > lot for
> >
> > us to keep the concerns separate and clear.
> >
> > I feel our approach is more complimentary such that it allows
> >
> > for
> >
> > using
> >
> > keystores of choice while retaining the flexibility to use any
> > underlying/available Provider for actually making the SSL call.
> >
> > We agree with KIP-492's approach based on Providers (and Java's
> > recommendation), but also strongly believe that our approach can
> >
> > compliment
> >
> > it very effectively for reasons explained above.
> >
> > Thanks
> > Maulin
> >
> > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <
> >
> > kafka@harsha.io
> >
> > wrote:
> >
> > Hi Maulin,
> >
> > On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
> >
> > maulin.vasavada@gmail.
> >
> > com>
> > wrote:
> >
> > Hi Harsha
> >
> > The reason we rejected the SslProvider route is that - we
> >
> > only
> >
> > needed
> >
> > a
> >
> > custom way to load keys/certs. Not touch any policy that
> >
> > existing
> >
> > Providers
> >
> > govern like SunJSSE Provider.
> >
> > We have exactly the same requirements to load certs and keys
> >
> > through
> >
> > spire
> >
> > agent. We used security.provider to do that exactly. I am not
> >
> > sure
> >
> > why
> >
> > you
> >
> > would be modifying any policies provided by default SunJSSE
> >
> > provider.
> >
> > Can
> >
> > you give me an example of having custom provider that will
> >
> > override an
> >
> > existing policy in SunJSSE provider.
> >
> > As pointed out earlier, this kip
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/
> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> >
> > allows
> > you to load security.provider through config.
> > Take a look at the examples I gave before
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> >
> > java
> >
> > It registers KeyManagerFactory and TrustManagerFactory and
> >
> > Keystore
> >
> > algorithm.
> > Implement your custom way of loading Keystore in here
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> >
> > java
> >
> > and Trust manager like here
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > spiffe-security-provider/src/main/java/spiffe/api/provider/
> > SpiffeTrustManager.java <http://spiffetrustmanager.java/> <http://
> > spiffetrustmanager.java/>
> >
> > In your Kafka client you can set the security.provider to your
> >
> > custom
> >
> > implementation and with this fix
> > https://issues.apache.org/jira/browse/KAFKA-8191 you can set
> > keyManagerAlgorigthm and trustManagerAlgorithm configs.
> >
> > All of this is in your clients and broker side and do not need
> >
> > to
> >
> > touch
> >
> > any
> > policy changes at JVM level. You'll register the providers in
> >
> > the
> >
> > priority
> >
> > order and can still have SunJSSE provider and have your custom
> >
> > provider
> >
> > to
> >
> > implement the key and trust managers.
> >
> > The ask here is different than KIP-492. We don't have any need
> >
> > to
> >
> > modify/specify the algorithm parameter. Does that make sense?
> >
> > The ask in KIP is introducing new interfaces where the KIP's
> > goal/motivation can be achieved through the security.provider
> >
> > and
> >
> > we
> >
> > worked
> > on similar goal without touching any Keystore or Truststore
> >
> > interfaces.
> >
> > My advise is against changing or introducing new interfaces
> >
> > when
> >
> > it can
> >
> > work through security.provider.
> >
> > Thanks,
> > Harsha
> >
> > Thanks
> >
> > Maulin
> >
> > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
> >
> > kafka@harsha.io>
> >
> > wrote:
> >
> > In your KIP you added security. provider as rejected
> >
> > alternative
> >
> > and
> >
> > specified "its not the correct way". Do you mind explaining
> >
> > why
> >
> > its
> >
> > not? I
> >
> > didn't find any evidence in Java docs to say so. Contrary to
> >
> > your
> >
> > statement
> >
> > it does say in the java docs
> > " However, please note that a provider can be used to
> >
> > implement
> >
> > any
> >
> > security service in Java that uses a pluggable architecture
> >
> > with
> >
> > a
> >
> > choice
> >
> > of implementations that fit underneath."
> >
> > Java Security Providers have been used by other projects to
> >
> > provide
> >
> > such
> >
> > integration . I am not sure if you looked into Spiffe
> >
> > project to
> >
> > efficiently distribute certificates but here is an example of
> >
> > Java
> >
> > provider
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> >
> > java which
> > obtains certificates from local daemons.
> > These integrations are being used in Tomcat, Jetty etc.. We
> >
> > are
> >
> > also
> >
> > using
> >
> > Security provider to do the same in our Kafka clusters. So
> >
> > unless I
> >
> > see
> >
> > more evidence why security.provider doesn't work for you
> >
> > adding
> >
> > new
> >
> > interfaces while there exists more cleaner way of achieving
> >
> > the
> >
> > goals
> >
> > of
> >
> > this KIP is unnecessary and breaks the well known security
> >
> > interfaces
> >
> > provided by Java itself.
> >
> > Thanks,
> > Harsha
> >
> > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
> >
> > kafka@harsha.io
> >
> > wrote:
> >
> > Hi Maulin,
> > Not sure if you looked at my previous replies. This
> >
> > changes
> >
> > are not required as there is already security Provider to do
> >
> > what you
> >
> > are
> >
> > proposing. This KIP
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/
> >
> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> >
> > also
> >
> > addresses easy registration of such providers.
> >
> > Thanks,
> > Harsha
> >
> > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
> >
> > <maulin.vasavada@gmail.
> >
> > com> wrote:
> >
> > Bump! Can somebody please review this?
> >
> > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
> >
> > maulin.vasavada@gmail.com>
> >
> > wrote:
> >
> > Bump! Can somebody please review this?
> >
> > --
> > Thanks,
> > M.Sai Sandeep
> >
> >
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Harsha Chintalapani <ka...@harsha.io>.
Hi Maulin,
Use cases are clear now. I am +1 for moving forward
with the discussions on having such configurable option for users. But the
interfaces is proposed doesn't look right to me. We are still talking about
keystore interfaces. Given keystore's are used as filebased way of
transporting certificates I am not sure it will help the rest of the
user-base.
In short, I am +1 on the KIP's motivation and only have
questions around returning keystores instead of returning certs, private
keys etc. . If others in the community are ok with such interface we can
move forward.
Thanks,
Harsha
On Wed, Aug 28, 2019 at 1:51 PM, Maulin Vasavada <ma...@gmail.com>
wrote:
> Hi Harsha
>
> As we synced-up offline on this topic, we hope you don't have any more
> clarifications that you are seeking. If that is the case, can you please
> help us move this forward and discuss what changes you would expect on the
> KIP design in order to make it valuable contribution?
>
> Just FYI - we verified our primary design change with the author of Sun's
> X509 Trustmanager implementation and the outcome is that what we are
> proposing makes sense at the heart of it - "Instead of writing TrustManager
> just plugin the Trust store". We are open to discuss additional changes
> that you/anybody else would like to see on the functionality however.
>
> Thanks
> Maulin
>
> On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada <ma...@gmail.com>
> wrote:
>
> Hi Harsha
>
> Any response on my question? I feel this KIP is worth accommodating. Your
> help is much appreciated.
>
> Thanks
> Maulin
>
> On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada < maulin.vasavada@gmail.
> com> wrote:
>
> Hi Harsha
>
> I've examined the SPIFFE provider more and have one question -
>
> If SPIFFE didn't have a need to do checkSpiffeId() call at the below
> location, would you really still write the Provider? *OR* Would you just
> use TrustManagerFactory.init(KeyStore) signature to pass the KeyStore from
> set of certs returned by spiffeIdManager. getTrustedCerts()?
>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> provider/CertificateUtils.java#L100
>
> /**
>
> * Validates that the SPIFFE ID is present and matches the SPIFFE ID
> configured in
> * the java.security property ssl.spiffe.accept
> *
> * If the authorized spiffe ids list is empty any spiffe id is authorized
> *
> * @param chain an array of X509Certificate that contains the Peer's SVID
> to be validated
> * @throws CertificateException when either the certificates doesn't have a
> SPIFFE ID or the SPIFFE ID is not authorized
> */
> static void checkSpiffeId(X509Certificate[] chain) throws
> CertificateException {
>
> Thanks
> Maulin
>
> On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani <ka...@harsha.io>
> wrote:
>
> Maulin,
> The code parts you are pointing are specific for Spiffe and if
> you are talking about validate method which uses PKIX check like any other
> provider does.
> If you want to default to SunJSSE everywhere you can do so by delegating
> the calls in these methods to SunJSSE provider.
>
> TrustManagerFactory tmf = TrustManagerFactory
> .getInstance(TrustManagerFactory.getDefaultAlgorithm());and use
> tmf.chekServerTrusted()
> or use
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> TrustManagerFactory.html#getInstance(java.lang.String)if you want a
> specific provider.
>
> -Harsha
>
> On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada < maulin.vasavada@gmail.
> com>
> wrote:
>
> Okay, so I take that you guys agree that I have to write a 'custom'
> algorithm and a provider to make it work , correct?
>
> Now, for Harsha's comment "Here the 'Custom' Algorithm is not an
> implementation per say , ..." , I diagree. You can refer to https://
>
> github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/
>
> SpiffeTrustManager.java#L91 <http://spiffetrustmanager.java/#L91> and
>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>
> provider/CertificateUtils.java#L100
>
> "that code" is the customization you have for the custom way to check
> something on top of regular checks. That method is NOT doing custom
> truststore loading. It is validating/verifying something in the
>
> "custom"
>
> way with spiffeId.
> I bet that without that you won't have a need of the custom algorithm
>
> in
>
> the first place.
>
> Let me know if you agree to this.
>
> Thanks
> Maulin
>
> On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <mp...@gmail.com>
>
> wrote:
>
> Hi Maulin, thanks for the discussion. As Harsha pointed out, to use the
> KIP492, you need to create a new provider, register a *new* custom
> algorithm for your keymanager and trustmanager factory implementations.
> After this, the kafka server configuration can be done as given below
>
> # Register the provider class with custom algorithm, say CUSTOM
>
> security.
>
> provider.classes=com.company.security.CustomProvider
> <http://provider.classes=com.company.security.customprovider/>
> <http://security.provider.classes
>
> =com.company.security.customprovider/>
>
> # Register the keymanager and trustmanager algorithms
> # These algorithms indicate that the Keymanager and Trustmanagers
> registered under the algorithm "CUSTOM" needs to be used
> ssl.trustmanager.algorithm=CUSTOM
> ssl.keymanager.algorithm=CUSTOM
>
> And the customprovider looks like this...
>
> public class CustomProvider extends Provider {
> public CustomProvider() {
> super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and TrustStore");
> super.put("KeyManagerFactory.CUSTOM", "customKeyManagerFactory");
> super.put("TrustManagerFactory.CUSTOM",
> "customTrustManagerFactory");
> }
> }
>
> The PR for this is in review and can be found here:
>
> https://github.com/
>
> apache/kafka/pull/7090
> This PR includes the fixed insertProviderAt function call.
>
> On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani <ka...@harsha.io>
> wrote:
>
> Answers are in-line
>
> On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada <
>
> maulin.vasavada@gmail.
>
> com
>
> wrote:
>
> Hi Colin
>
> When I refer to "standard" or "custom" algorithms I am following Java
> security Provider Terminology. You can refer to
> https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> StandardNames.html#TrustManagerFactory link I provided earlier in the
> emails. It says PKIX is the default Algorithm for TrustManagerFactory.
>
> 1. For SPIFFE, I am not sure why you are saying 'it does not implement
> custom algorithms' because the following file clearly indicates that it
> does use custom algorithm-
>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>
> provider/SpiffeProvider.java#L17
>
> Algorithm value:
>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>
> provider/SpiffeProviderConstants.java#L6
>
> @Harsha do you want to chime in since you use that provider?
>
> Here the "Custom" Algorithm is not an implementation per say , rather
>
> used
>
> to invoke the custom trust store factory and key manager factory. You
>
> are
>
> not moving away from "standard" alogrithms that are available.
>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>
> provider/SpiffeTrustManager.java
>
> As you can see it delegates all the calls of verification of
>
> certificate
>
> to
>
> the default implementation available.
> So in our implementation we still use PKIX to verify the certificate
> chain. So you are not losing anything here and Spiffe is not
>
> reimplementing
>
> the verification process.
>
> 2. I already mentioned in my 3rd point, in my previous post, why using
>
> ssl.provider does NOT work. I updated KIP-486 in "rejected
>
> alternatives"
>
> also why ssl.provider does not work.
>
> As mentioned before , provider is the right way to go. I am still not
> understanding the gap is.
> If I understand correctly your argument is , provider is going to ask
>
> to
>
> implement a custom algorithm.
> My answer to that is , no you are not re-implementing the algorithm.
>
> Please
>
> check the above link , TrustManager implementation it delegates the
>
> calls
>
> back. There is no need to implement your own here.
>
> 3. Security.insertProviderAt() comments were based on assumption if
>
> KIP-492
>
> changes are done and we use that mechanism to configure providers
>
> instead
>
> of ssl.provider configuration.
>
> KIP-492 has patch available and going through review.
>
> Can you read my all the points, I mentioned in my previous post, very
>
> carefully? I am covering all the aspects in explaining. I am open to
>
> still
>
> discuss more to clarify any doubts.
>
> "3. If we just use existing ssl.provider kafka configuration then our
> provider will be used in SSLContext.getInstance(protocol, provider)
>
> call
>
> in
>
> SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/> <
> http://sslfactory.java/>
>
> and
>
> if our provider does not have
> implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we
>
> tested
>
> it). Example: In MyProvider sample above you see that I didn't add
> SSLContext.TLSv1 as
> "Service+Algorithm" and that didn't work for me. In SPIFFE provider you
> don't have this challenge since you are planning to bypass ssl.provider
>
> as
>
> you mention in the KIP-492."
>
> Yes here you need to pass the protocol that your
>
> KeyManager/TrustManager
>
> registered with and in no way its deviating from TLS RFC spec.
>
> https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/
>
> javax/net/ssl/SSLContext.java#L134
>
> My suggestion here is for you to implement a simple Security Provider
>
> as
>
> you did before and register a Algorithm. You can use the existing
> implementation in SpiffeProvider and plug in changes where you need to
> retrieve the certificates from by making RPC call.
>
> Run an end-to-end test with Kafka broker coming up with your provider
> making calls to RPC call. You do need to pass the "custom algorithm"
>
> that
>
> you registered in your key manager to invoke the provider. I think your
> concern is this approach is replacing the existing known ciphersuites
>
> and
>
> certificate validation provided by java. Which its not.
>
> Now test the TLS connection you can do so via openssl -s_client options
>
> to
>
> test if everything else is working.
>
> I am happy to share configs that we used if you are interested.
>
> Thanks,
> Harsha
>
> Thanks
> Maulin
>
> On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <cm...@apache.org>
>
> wrote:
>
> Hi Maulin,
>
> A lot of JSSE providers don't implement custom algorithms. Spire is a
>
> good
>
> example of a JSSE provider that doesn't, and yet is still useful to
>
> many
>
> people. Your JSSE provider can work fine even if it doesn't implement a
> custom algorithm.
>
> Maybe I'm missing something, but I don't understand the discussion of
> Security.insertProviderAt() that you included. SslEngineBuilder doesn't
>
> use
>
> that API to get the security provider. Instead, it calls
> "SSLContext.getInstance(protocol, provider)", where provider is the
>
> name
>
> of the provider.
>
> best,
> Colin
>
> On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
>
> On top of everything above I feel strongly to add the 4th point which
>
> is
>
> based on Java APIs for TrustManagerFactory.init(KeyStore) (
>
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> TrustManagerFactory.html#init(java.security.KeyStore
> <http://java.security.keystore/>
> <http://java.security.keystore/>)
> )
>
> and KeyManagerFactory.init(KeyStore, char[]) (
>
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory
> .
>
> html#init(java.security.KeyStore <http://java.security.keystore/> <http://
> java.security.keystore/
> ,%20char[])
>
> ).
>
> 4. The above APIs are intended to support providing "trust/key
>
> material"
>
> from the user without having to write their own
>
> TrustManager/KeyManagers.
>
> To quote from the TrustManagerFactory.init()'s documentation
>
> "Initializes
>
> this factory with a source of certificate authorities and related trust
> material."
> To quote from the KeyManagerFactory.init()'s documentation "Initializes
> this factory with a source of key material."
>
> Based on this it is clear that there is a flexibility provided by Java
>
> to
>
> to enable developers to provide the required trust/key material loaded
>
> from
>
> "anywhere" without requiring them to write custom provider OR trust/key
> managers. This same flexibility is reflected in Kafka code also where
>
> it
>
> loads the trust/keys from a local file and doesn't require writing a
> Provider necessarily. If we do NOT have a custom algorithm, it makes
>
> less
>
> sense to write a Provider.
>
> Thanks
> Maulin
>
> On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
>
> maulin.vasavada@gmail.com>
>
> wrote:
>
> Hi Harsha/Colin
>
> I did the sample with a custom Provider for TrustStoreManager and tried
> using ssl.provider Kafka config AND the way KIP-492 is suggesting (by
> adding Provider programmatically instead of relying on
>
> ssl.provider+java.
>
> security. The below sample is followed by my detailed findings. I'll
> appreciate if you can go through it carefully and see
>
> if you
>
> see my point.
>
> package providertest;
>
> import java.security.Provider <http://java.security.provider/> <http://
> java.security.provider/>
>
> <http://
>
> java.security.provider/>;
>
> public class MyProvider extends Provider {
>
> private static final String name = "MyProvider"; private static double
> version = 1.0d;
> private static String info = "Maulin's SSL Provider v"+version;
>
> public MyProvider() {
> super(name, version, info);
> this.put("TrustManagerFactory.PKIX",
>
> "providertest.MyTrustManagerFactory");
>
> }
> }
>
> *Details:*
>
> KIP-492 documents that it will use Security.addProvider() assuming it
>
> will
>
> add it as position '0' which is not a correct assumption. The
> addProvider()'s documentation says it will add it to the last available
> position. You may want to correct that to say
> Security.insertProviderAt(provider, 1).
>
> Now coming back to our specific discussion,
>
> 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add
>
> that
>
> provider in the provider list via Security.addProvider() the position
>
> where
>
> it gets added doesn't matter (even if you don't end up adding it as
>
> first
>
> entry) since that is the ONLY provider for SPIFFE specific algorithm
>
> you
>
> might have.
>
> We do *not* have custom algorithm for Key/Trust StoreMangers. Which
>
> means
>
> we have to use X509, PKIX etc "Standard Algorithms" ((
>
> https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> StandardNames.html
> ))
>
> in our provider to override the TrustStoreManager (see my sample code)
>
> and
>
> KeyStoreManger and KeyManager. This creates another challenge
>
> mentioned in
>
> the below point.
>
> 2. In order to make our Provider for loading custom TrustStore work, we
> have to add the provider as 'first' in the list since there are others
>
> with
>
> the same algorithm.
>
> However, the programatic way of adding provider
> (Security.insertProviderAt()) is *not* deterministic for ordering since
> different code can call the method for a different provider and
>
> depending
>
> upon the order of the call our provider can be first or pushed down the
> list. This can happen very well in any client application using Kafka.
>
> This
>
> is specially problematic for a case when you want to guarantee order
>
> for a
>
> Provider having "Standard Algorithms".
>
> If we add our provider in java.security file that definitely guarantees
> the order(unless somebody calls removeProvider() which is less
>
> likely). But
>
> if we add our provider in java.security file it will defeat the
>
> purpose of
>
> the KIP-492.
>
> In the gist - Apache Kafka must not rely on "un-deterministic" method
>
> to
>
> rely on Provider ordering.
>
> 3. If we just use existing ssl.provider kafka configuration then our
> provider will be used in SSLContext.getInstance(protocol, provider)
>
> call in
>
> SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/> <
> http://sslfactory.java/>
>
> and
>
> if our provider does not have implementation for
> SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks
>
> (we
>
> tested it). Example:
>
> In
>
> MyProvider sample above you see that I didn't add SSLContext.TLSv1 as
> "Service+Algorithm" and that didn't work for me. In SPIFFE provider you
> don't have this challenge since you are planning to bypass
>
> ssl.provider as
>
> you mention in the KIP-492.
>
> *Overall summary,*
>
> 1. Any provider based mechanisms- a) existing ssl.provider and
>
> b)KIP-492,
>
> for loading key/trust store using "Standard Algorithms" do not work
>
> 2. Approach suggested in our KIP-486 works without any issue and it is
> *not* our context specific solve
>
> 3. Based on above we feel KIP-492 and KIP-486 are complimentary changes
> and not contradicting or redundent.
>
> If you want we can do a joint session somehow to walk through the
>
> sample I
>
> have and various experiments I did. I would encourage you to do similar
> exercise by writing a Provider for "Standard Algorithm" for
> TrustStoreManager (like our needs) and see what you find since only
>
> writing
>
> samples can bring out the complexity/challenges we face.
>
> Thanks
> Maulin
>
> On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <
>
> maulin.vasavada@gmail.
>
> com> wrote:
>
> Just to update - still working on it. Get to work only on and off on
>
> it :(
>
> On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
>
> maulin.vasavada@gmail.com>
>
> wrote:
>
> Hi Harsha
>
> Let me try to write samples and will let you know.
>
> Thanks
> Maulin
>
> On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <ha...@gmail.com>
>
> wrote:
>
> Hi Maulin,
> With java security providers can be as custom you would
>
> like
>
> it to
> be. If you only want to to implement a custom way of loading the
>
> keystore
>
> and truststore and not implement any protocol/encryption handling you
>
> can
>
> leave them empty and no need to implement. Have you looked into the
>
> links I
>
> pasted before?
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
>
> java
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> spiffe-security-provider/src/main/java/spiffe/api/provider/
> SpiffeTrustManager.java <http://spiffetrustmanager.java/> <http://
> spiffetrustmanager.java/>
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>
> java
>
> Can you please tell me which methods are too complex in above to
>
> implement
>
> or unnecessary? You are changing anything in SSL/TLS implementations
> provided by
>
> All of the implementations delegating the checks to the default
> implementation anyway.
> Spire agent is an example, its nothing but a GRPC server listening
>
> on a
>
> unix domain socket . Above code is making a RPC call to the local
>
> daemon
>
> to
> get the certificate and keys. The mechanics are pretty much same as
>
> what
>
> you are asking for.
>
> Thanks,
> Harsha
>
> On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
>
> maulin.vasavada@gmail.com>
>
> wrote:
>
> Imagine a scenario like - We know we have a custom KMS and as a
>
> Kafka
>
> owner
>
> we want to comply to using that KMS source to load keys/certs. As
>
> a
>
> Kafka
>
> owner we know how to integrate with KMS but doesn't necessarily
>
> have
>
> to
>
> know anything about cipher suites, algorithms, and SSL/TLS
>
> implementation.
>
> Going the Provider way requires to know lot more than we should,
>
> isn't it?
>
> Not that we would have concern/shy-away knowing those details -
>
> but
>
> if we
>
> don't have to - why should we?
>
> On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
>
> maulin.vasavada@gmail.com>
>
> wrote:
>
> Hi Harsha
>
> We don't have spire (or similar) agents and we do not have
>
> keys/certs
>
> locally on any brokers. To elaborate more on my previous email,
>
> I agree that Java security Providers are used in much broader
>
> sense
>
> - to
>
> have a particular implementation of an algorithm, use specific
>
> cipher
>
> suites for SSL , OR in our current team's case have a
>
> particular
>
> way to
>
> leverage pre-generated SSL sessions. However, the scope of our
>
> KIP
>
> (486)
>
> is
>
> much restricted than that. We merely intend to provide a custom
> keystore/truststore for our SSL connections and not really worry
>
> about
>
> underlying specific SSL/TLS implementation. This simplifies it
>
> a
>
> lot for
>
> us to keep the concerns separate and clear.
>
> I feel our approach is more complimentary such that it allows
>
> for
>
> using
>
> keystores of choice while retaining the flexibility to use any
> underlying/available Provider for actually making the SSL call.
>
> We agree with KIP-492's approach based on Providers (and Java's
> recommendation), but also strongly believe that our approach can
>
> compliment
>
> it very effectively for reasons explained above.
>
> Thanks
> Maulin
>
> On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <
>
> kafka@harsha.io
>
> wrote:
>
> Hi Maulin,
>
> On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
>
> maulin.vasavada@gmail.
>
> com>
> wrote:
>
> Hi Harsha
>
> The reason we rejected the SslProvider route is that - we
>
> only
>
> needed
>
> a
>
> custom way to load keys/certs. Not touch any policy that
>
> existing
>
> Providers
>
> govern like SunJSSE Provider.
>
> We have exactly the same requirements to load certs and keys
>
> through
>
> spire
>
> agent. We used security.provider to do that exactly. I am not
>
> sure
>
> why
>
> you
>
> would be modifying any policies provided by default SunJSSE
>
> provider.
>
> Can
>
> you give me an example of having custom provider that will
>
> override an
>
> existing policy in SunJSSE provider.
>
> As pointed out earlier, this kip
>
> https://cwiki.apache.org/confluence/display/KAFKA/
> KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>
> allows
> you to load security.provider through config.
> Take a look at the examples I gave before
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>
> java
>
> It registers KeyManagerFactory and TrustManagerFactory and
>
> Keystore
>
> algorithm.
> Implement your custom way of loading Keystore in here
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
>
> java
>
> and Trust manager like here
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> spiffe-security-provider/src/main/java/spiffe/api/provider/
> SpiffeTrustManager.java <http://spiffetrustmanager.java/> <http://
> spiffetrustmanager.java/>
>
> In your Kafka client you can set the security.provider to your
>
> custom
>
> implementation and with this fix
> https://issues.apache.org/jira/browse/KAFKA-8191 you can set
> keyManagerAlgorigthm and trustManagerAlgorithm configs.
>
> All of this is in your clients and broker side and do not need
>
> to
>
> touch
>
> any
> policy changes at JVM level. You'll register the providers in
>
> the
>
> priority
>
> order and can still have SunJSSE provider and have your custom
>
> provider
>
> to
>
> implement the key and trust managers.
>
> The ask here is different than KIP-492. We don't have any need
>
> to
>
> modify/specify the algorithm parameter. Does that make sense?
>
> The ask in KIP is introducing new interfaces where the KIP's
> goal/motivation can be achieved through the security.provider
>
> and
>
> we
>
> worked
> on similar goal without touching any Keystore or Truststore
>
> interfaces.
>
> My advise is against changing or introducing new interfaces
>
> when
>
> it can
>
> work through security.provider.
>
> Thanks,
> Harsha
>
> Thanks
>
> Maulin
>
> On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
>
> kafka@harsha.io>
>
> wrote:
>
> In your KIP you added security. provider as rejected
>
> alternative
>
> and
>
> specified "its not the correct way". Do you mind explaining
>
> why
>
> its
>
> not? I
>
> didn't find any evidence in Java docs to say so. Contrary to
>
> your
>
> statement
>
> it does say in the java docs
> " However, please note that a provider can be used to
>
> implement
>
> any
>
> security service in Java that uses a pluggable architecture
>
> with
>
> a
>
> choice
>
> of implementations that fit underneath."
>
> Java Security Providers have been used by other projects to
>
> provide
>
> such
>
> integration . I am not sure if you looked into Spiffe
>
> project to
>
> efficiently distribute certificates but here is an example of
>
> Java
>
> provider
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>
> java which
> obtains certificates from local daemons.
> These integrations are being used in Tomcat, Jetty etc.. We
>
> are
>
> also
>
> using
>
> Security provider to do the same in our Kafka clusters. So
>
> unless I
>
> see
>
> more evidence why security.provider doesn't work for you
>
> adding
>
> new
>
> interfaces while there exists more cleaner way of achieving
>
> the
>
> goals
>
> of
>
> this KIP is unnecessary and breaks the well known security
>
> interfaces
>
> provided by Java itself.
>
> Thanks,
> Harsha
>
> On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
>
> kafka@harsha.io
>
> wrote:
>
> Hi Maulin,
> Not sure if you looked at my previous replies. This
>
> changes
>
> are not required as there is already security Provider to do
>
> what you
>
> are
>
> proposing. This KIP
>
> https://cwiki.apache.org/confluence/display/KAFKA/
>
> KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>
> also
>
> addresses easy registration of such providers.
>
> Thanks,
> Harsha
>
> On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
>
> <maulin.vasavada@gmail.
>
> com> wrote:
>
> Bump! Can somebody please review this?
>
> On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
>
> maulin.vasavada@gmail.com>
>
> wrote:
>
> Bump! Can somebody please review this?
>
> --
> Thanks,
> M.Sai Sandeep
>
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Maulin Vasavada <ma...@gmail.com>.
Hi Rajan
Your email format doesn't show up correctly. Can you repost to make it more
readable?
Thanks
Maulin
On Wed, Aug 28, 2019 at 2:32 PM Rajan Dhabalia <rd...@apache.org> wrote:
> *Hi Harsha/Maulin,I am following KIP-486 and KIP-492 and it seems
> https://github.com/apache/kafka/pull/7090
> <https://github.com/apache/kafka/pull/7090> is the right solution when one
> wants to register custom factory class for KeyManager and TrustManager.
> User can easily configure custom implementation of TrustManager and
> KeyManager using factory Provider class.Configuration of the provider is
> also simple and straightforward: 1. Create custom provider which defines
> factory classes for KeyManager and TrustManagerpublic class CustomProvider
> extends Provider { public CustomProvider()
> { super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and
> TrustStore"); super.put("KeyManagerFactory.CUSTOM",
> "customKeyManagerFactory");
> super.put("TrustManagerFactory.CUSTOM","customTrustManagerFactory");
> }}
> 1. Register provider at broker
> startupjava.security.Security.addProvider(new CustomProvider());However,
> this approach is useful when one wants to implement custom TrustManager for
> X509 certs by extending X509ExtendedKeyManager and implement various
> abstract methods such as: checkClientTrusted, checkServerTrusted, etc..In
> JDK, default implementation class of X509ExtendedKeyManager is
> X509TrustManagerImpl and one can’t extend or delegate calls to this class
> because it’s final and same is applicable for other available providers
> such as : BouncyCastleProviderTurstManager/KeyManager mainly serve two
> purposes: 1. Provide certs/key2. Perform validationX509TrustManagerImpl
> performs various RFC specified validations.#7090 can be helpful when user
> has both above asks. However, problem defined at KIP-486 has different ask
> where user wants to provide certs/key without copying/implementing Manager
> class because all the available Manager classes are final and can’t be
> extended or delegated. And security team in most of the companies don’t
> allow custom/copying provider in order to get up to date with various RFC
> validations provided into standard jdk provider.Many users manage keys and
> certs into KMS and sometimes it’s not feasible to copy them to file-system
> instead directly use them from the memory. So, KIP-486 provides a custom
> way to load keys/certs without implementing security-provider.Thanks,Rajan*
>
> On Wed, Aug 28, 2019 at 2:18 PM Maulin Vasavada <maulin.vasavada@gmail.com
> >
> wrote:
>
> >
> >
> > Hi Harsha
> >
> > As we synced-up offline on this topic, we hope you don't have any more
> > clarifications that you are seeking. If that is the case, can you please
> > help us move this forward and discuss what changes you would expect on
> the
> > KIP design in order to make it valuable contribution?
> >
> > Just FYI - we verified our primary design change with the author of Sun's
> > X509 Trustmanager implementation and the outcome is that what we are
> > proposing makes sense at the heart of it - "Instead of writing
> TrustManager
> > just plugin the Trust store". We are open to discuss additional changes
> > that you/anybody else would like to see on the functionality however.
> >
> >
> > Thanks
> > Maulin
> >
> > On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada <
> maulin.vasavada@gmail.com>
> > wrote:
> >
> >> Hi Harsha
> >>
> >> Any response on my question? I feel this KIP is worth accommodating.
> Your
> >> help is much appreciated.
> >>
> >> Thanks
> >> Maulin
> >>
> >> On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada <
> >> maulin.vasavada@gmail.com> wrote:
> >>
> >>> Hi Harsha
> >>>
> >>> I've examined the SPIFFE provider more and have one question -
> >>>
> >>> If SPIFFE didn't have a need to do checkSpiffeId() call at the below
> >>> location, would you really still write the Provider? *OR*
> >>> Would you just use TrustManagerFactory.init(KeyStore) signature to pass
> >>> the KeyStore from set of certs returned by spiffeIdManager.
> >>> getTrustedCerts()?
> >>>
> >>>
> >>>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/CertificateUtils.java#L100
> >>>
> >>>
> >>> /**
> >>>> * Validates that the SPIFFE ID is present and matches the SPIFFE ID
> >>>> configured in
> >>>> * the java.security property ssl.spiffe.accept
> >>>> *
> >>>> * If the authorized spiffe ids list is empty any spiffe id is
> authorized
> >>>> *
> >>>> * @param chain an array of X509Certificate that contains the Peer's
> >>>> SVID to be validated
> >>>> * @throws CertificateException when either the certificates doesn't
> >>>> have a SPIFFE ID or the SPIFFE ID is not authorized
> >>>> */
> >>>> static void checkSpiffeId(X509Certificate[] chain) throws
> >>>> CertificateException {
> >>>
> >>>
> >>>
> >>> Thanks
> >>> Maulin
> >>>
> >>>
> >>> On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani <ka...@harsha.io>
> >>> wrote:
> >>>
> >>>> Maulin,
> >>>> The code parts you are pointing are specific for Spiffe
> >>>> and if
> >>>> you are talking about validate method which uses PKIX check like any
> >>>> other provider does.
> >>>> If you want to default to SunJSSE everywhere you can do so by
> delegating
> >>>> the calls in these methods to SunJSSE provider.
> >>>>
> >>>> TrustManagerFactory tmf = TrustManagerFactory
> >>>> .getInstance(TrustManagerFactory.getDefaultAlgorithm());and use
> >>>> tmf.chekServerTrusted()
> >>>> or use
> >>>>
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/TrustManagerFactory.html#getInstance(java.lang.String)if
> >>>> you want a specific provider.
> >>>>
> >>>> -Harsha
> >>>>
> >>>>
> >>>> On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada <
> >>>> maulin.vasavada@gmail.com>
> >>>> wrote:
> >>>>
> >>>> > Okay, so I take that you guys agree that I have to write a 'custom'
> >>>> > algorithm and a provider to make it work , correct?
> >>>> >
> >>>> > Now, for Harsha's comment "Here the 'Custom' Algorithm is not an
> >>>> > implementation per say , ..." , I diagree. You can refer to https://
> >>>> >
> >>>>
> github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/
> >>>> > SpiffeTrustManager.java#L91 and
> >>>> >
> >>>>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> >>>> > provider/CertificateUtils.java#L100
> >>>> >
> >>>> > "that code" is the customization you have for the custom way to
> check
> >>>> > something on top of regular checks. That method is NOT doing custom
> >>>> > truststore loading. It is validating/verifying something in the
> >>>> "custom"
> >>>> > way with spiffeId.
> >>>> > I bet that without that you won't have a need of the custom
> algorithm
> >>>> in
> >>>> > the first place.
> >>>> >
> >>>> > Let me know if you agree to this.
> >>>> >
> >>>> > Thanks
> >>>> > Maulin
> >>>> >
> >>>> > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <mp...@gmail.com>
> >>>> wrote:
> >>>> >
> >>>> > Hi Maulin, thanks for the discussion. As Harsha pointed out, to use
> >>>> the
> >>>> > KIP492, you need to create a new provider, register a *new* custom
> >>>> > algorithm for your keymanager and trustmanager factory
> >>>> implementations.
> >>>> > After this, the kafka server configuration can be done as given
> below
> >>>> >
> >>>> > # Register the provider class with custom algorithm, say CUSTOM
> >>>> security.
> >>>> > provider.classes=com.company.security.CustomProvider
> >>>> > <http://security.provider.classes
> >>>> =com.company.security.customprovider/>
> >>>> >
> >>>> > # Register the keymanager and trustmanager algorithms
> >>>> > # These algorithms indicate that the Keymanager and Trustmanagers
> >>>> > registered under the algorithm "CUSTOM" needs to be used
> >>>> > ssl.trustmanager.algorithm=CUSTOM
> >>>> > ssl.keymanager.algorithm=CUSTOM
> >>>> >
> >>>> > And the customprovider looks like this...
> >>>> >
> >>>> > public class CustomProvider extends Provider {
> >>>> > public CustomProvider() {
> >>>> > super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and TrustStore");
> >>>> > super.put("KeyManagerFactory.CUSTOM", "customKeyManagerFactory");
> >>>> > super.put("TrustManagerFactory.CUSTOM",
> >>>> > "customTrustManagerFactory");
> >>>> > }
> >>>> > }
> >>>> >
> >>>> > The PR for this is in review and can be found here:
> >>>> https://github.com/
> >>>> > apache/kafka/pull/7090
> >>>> > This PR includes the fixed insertProviderAt function call.
> >>>> >
> >>>> > On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani <
> kafka@harsha.io>
> >>>> > wrote:
> >>>> >
> >>>> > Answers are in-line
> >>>> >
> >>>> > On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada <
> >>>> maulin.vasavada@gmail.
> >>>> > com
> >>>> >
> >>>> > wrote:
> >>>> >
> >>>> > Hi Colin
> >>>> >
> >>>> > When I refer to "standard" or "custom" algorithms I am following
> Java
> >>>> > security Provider Terminology. You can refer to
> >>>> > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> >>>> > StandardNames.html#TrustManagerFactory link I provided earlier in
> the
> >>>> > emails. It says PKIX is the default Algorithm for
> TrustManagerFactory.
> >>>> >
> >>>> > 1. For SPIFFE, I am not sure why you are saying 'it does not
> implement
> >>>> > custom algorithms' because the following file clearly indicates that
> >>>> it
> >>>> > does use custom algorithm-
> >>>> >
> >>>> >
> >>>>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> >>>> >
> >>>> > provider/SpiffeProvider.java#L17
> >>>> >
> >>>> > Algorithm value:
> >>>> >
> >>>> >
> >>>>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> >>>> >
> >>>> > provider/SpiffeProviderConstants.java#L6
> >>>> >
> >>>> > @Harsha do you want to chime in since you use that provider?
> >>>> >
> >>>> > Here the "Custom" Algorithm is not an implementation per say ,
> rather
> >>>> >
> >>>> > used
> >>>> >
> >>>> > to invoke the custom trust store factory and key manager factory.
> You
> >>>> are
> >>>> > not moving away from "standard" alogrithms that are available.
> >>>> >
> >>>> >
> >>>>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> >>>> > provider/SpiffeTrustManager.java
> >>>> >
> >>>> > As you can see it delegates all the calls of verification of
> >>>> certificate
> >>>> >
> >>>> > to
> >>>> >
> >>>> > the default implementation available.
> >>>> > So in our implementation we still use PKIX to verify the certificate
> >>>> > chain. So you are not losing anything here and Spiffe is not
> >>>> reimplementing
> >>>> > the verification process.
> >>>> >
> >>>> > 2. I already mentioned in my 3rd point, in my previous post, why
> using
> >>>> >
> >>>> > ssl.provider does NOT work. I updated KIP-486 in "rejected
> >>>> >
> >>>> > alternatives"
> >>>> >
> >>>> > also why ssl.provider does not work.
> >>>> >
> >>>> > As mentioned before , provider is the right way to go. I am still
> not
> >>>> > understanding the gap is.
> >>>> > If I understand correctly your argument is , provider is going to
> ask
> >>>> to
> >>>> > implement a custom algorithm.
> >>>> > My answer to that is , no you are not re-implementing the algorithm.
> >>>> >
> >>>> > Please
> >>>> >
> >>>> > check the above link , TrustManager implementation it delegates the
> >>>> calls
> >>>> > back. There is no need to implement your own here.
> >>>> >
> >>>> > 3. Security.insertProviderAt() comments were based on assumption if
> >>>> >
> >>>> > KIP-492
> >>>> >
> >>>> > changes are done and we use that mechanism to configure providers
> >>>> >
> >>>> > instead
> >>>> >
> >>>> > of ssl.provider configuration.
> >>>> >
> >>>> > KIP-492 has patch available and going through review.
> >>>> >
> >>>> > Can you read my all the points, I mentioned in my previous post,
> very
> >>>> >
> >>>> > carefully? I am covering all the aspects in explaining. I am open to
> >>>> >
> >>>> > still
> >>>> >
> >>>> > discuss more to clarify any doubts.
> >>>> >
> >>>> > "3. If we just use existing ssl.provider kafka configuration then
> our
> >>>> > provider will be used in SSLContext.getInstance(protocol, provider)
> >>>> call
> >>>> >
> >>>> > in
> >>>> >
> >>>> > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/>
> >>>> and
> >>>> > if our provider does not have
> >>>> > implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we
> >>>> >
> >>>> > tested
> >>>> >
> >>>> > it). Example: In MyProvider sample above you see that I didn't add
> >>>> > SSLContext.TLSv1 as
> >>>> > "Service+Algorithm" and that didn't work for me. In SPIFFE provider
> >>>> you
> >>>> > don't have this challenge since you are planning to bypass
> >>>> ssl.provider
> >>>> >
> >>>> > as
> >>>> >
> >>>> > you mention in the KIP-492."
> >>>> >
> >>>> > Yes here you need to pass the protocol that your
> >>>> KeyManager/TrustManager
> >>>> > registered with and in no way its deviating from TLS RFC spec.
> >>>> >
> >>>> >
> >>>>
> https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/
> >>>> > javax/net/ssl/SSLContext.java#L134
> >>>> >
> >>>> > My suggestion here is for you to implement a simple Security
> Provider
> >>>> as
> >>>> > you did before and register a Algorithm. You can use the existing
> >>>> > implementation in SpiffeProvider and plug in changes where you need
> to
> >>>> > retrieve the certificates from by making RPC call.
> >>>> >
> >>>> > Run an end-to-end test with Kafka broker coming up with your
> provider
> >>>> > making calls to RPC call. You do need to pass the "custom algorithm"
> >>>> that
> >>>> > you registered in your key manager to invoke the provider. I think
> >>>> your
> >>>> > concern is this approach is replacing the existing known
> ciphersuites
> >>>> and
> >>>> > certificate validation provided by java. Which its not.
> >>>> >
> >>>> > Now test the TLS connection you can do so via openssl -s_client
> >>>> options
> >>>> >
> >>>> > to
> >>>> >
> >>>> > test if everything else is working.
> >>>> >
> >>>> > I am happy to share configs that we used if you are interested.
> >>>> >
> >>>> > Thanks,
> >>>> > Harsha
> >>>> >
> >>>> > Thanks
> >>>> > Maulin
> >>>> >
> >>>> > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <cm...@apache.org>
> >>>> >
> >>>> > wrote:
> >>>> >
> >>>> > Hi Maulin,
> >>>> >
> >>>> > A lot of JSSE providers don't implement custom algorithms. Spire is
> a
> >>>> >
> >>>> > good
> >>>> >
> >>>> > example of a JSSE provider that doesn't, and yet is still useful to
> >>>> >
> >>>> > many
> >>>> >
> >>>> > people. Your JSSE provider can work fine even if it doesn't
> implement
> >>>> a
> >>>> > custom algorithm.
> >>>> >
> >>>> > Maybe I'm missing something, but I don't understand the discussion
> of
> >>>> > Security.insertProviderAt() that you included. SslEngineBuilder
> >>>> doesn't
> >>>> >
> >>>> > use
> >>>> >
> >>>> > that API to get the security provider. Instead, it calls
> >>>> > "SSLContext.getInstance(protocol, provider)", where provider is the
> >>>> >
> >>>> > name
> >>>> >
> >>>> > of the provider.
> >>>> >
> >>>> > best,
> >>>> > Colin
> >>>> >
> >>>> > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
> >>>> >
> >>>> > On top of everything above I feel strongly to add the 4th point
> which
> >>>> >
> >>>> > is
> >>>> >
> >>>> > based on Java APIs for TrustManagerFactory.init(KeyStore) (
> >>>> >
> >>>> > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> >>>> > TrustManagerFactory.html#init(java.security.KeyStore
> >>>> > <http://java.security.keystore/>)
> >>>> > )
> >>>> >
> >>>> > and KeyManagerFactory.init(KeyStore, char[]) (
> >>>> >
> >>>> >
> >>>>
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory
> >>>> .
> >>>> >
> >>>> >
> >>>> > html#init(java.security.KeyStore <http://java.security.keystore/
> >>>> >,%20char[])
> >>>> >
> >>>> >
> >>>> > ).
> >>>> >
> >>>> > 4. The above APIs are intended to support providing "trust/key
> >>>> >
> >>>> > material"
> >>>> >
> >>>> > from the user without having to write their own
> >>>> >
> >>>> > TrustManager/KeyManagers.
> >>>> >
> >>>> > To quote from the TrustManagerFactory.init()'s documentation
> >>>> >
> >>>> > "Initializes
> >>>> >
> >>>> > this factory with a source of certificate authorities and related
> >>>> trust
> >>>> > material."
> >>>> > To quote from the KeyManagerFactory.init()'s documentation
> >>>> "Initializes
> >>>> > this factory with a source of key material."
> >>>> >
> >>>> > Based on this it is clear that there is a flexibility provided by
> Java
> >>>> >
> >>>> > to
> >>>> >
> >>>> > to enable developers to provide the required trust/key material
> loaded
> >>>> >
> >>>> > from
> >>>> >
> >>>> > "anywhere" without requiring them to write custom provider OR
> >>>> trust/key
> >>>> > managers. This same flexibility is reflected in Kafka code also
> where
> >>>> >
> >>>> > it
> >>>> >
> >>>> > loads the trust/keys from a local file and doesn't require writing a
> >>>> > Provider necessarily. If we do NOT have a custom algorithm, it makes
> >>>> >
> >>>> > less
> >>>> >
> >>>> > sense to write a Provider.
> >>>> >
> >>>> > Thanks
> >>>> > Maulin
> >>>> >
> >>>> > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
> >>>> >
> >>>> > maulin.vasavada@gmail.com>
> >>>> >
> >>>> > wrote:
> >>>> >
> >>>> > Hi Harsha/Colin
> >>>> >
> >>>> > I did the sample with a custom Provider for TrustStoreManager and
> >>>> tried
> >>>> > using ssl.provider Kafka config AND the way KIP-492 is suggesting
> (by
> >>>> > adding Provider programmatically instead of relying on
> >>>> >
> >>>> > ssl.provider+java.
> >>>> >
> >>>> > security. The below sample is followed by my detailed findings. I'll
> >>>> > appreciate if you can go through it carefully and see
> >>>> >
> >>>> > if you
> >>>> >
> >>>> > see my point.
> >>>> >
> >>>> > package providertest;
> >>>> >
> >>>> > import java.security.Provider <http://java.security.provider/>
> >>>> <http://
> >>>> > java.security.provider/>;
> >>>> >
> >>>> > public class MyProvider extends Provider {
> >>>> >
> >>>> > private static final String name = "MyProvider"; private static
> double
> >>>> > version = 1.0d;
> >>>> > private static String info = "Maulin's SSL Provider v"+version;
> >>>> >
> >>>> > public MyProvider() {
> >>>> > super(name, version, info);
> >>>> > this.put("TrustManagerFactory.PKIX",
> >>>> >
> >>>> > "providertest.MyTrustManagerFactory");
> >>>> >
> >>>> > }
> >>>> > }
> >>>> >
> >>>> > *Details:*
> >>>> >
> >>>> > KIP-492 documents that it will use Security.addProvider() assuming
> it
> >>>> >
> >>>> > will
> >>>> >
> >>>> > add it as position '0' which is not a correct assumption. The
> >>>> > addProvider()'s documentation says it will add it to the last
> >>>> available
> >>>> > position. You may want to correct that to say
> >>>> > Security.insertProviderAt(provider, 1).
> >>>> >
> >>>> > Now coming back to our specific discussion,
> >>>> >
> >>>> > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add
> >>>> >
> >>>> > that
> >>>> >
> >>>> > provider in the provider list via Security.addProvider() the
> position
> >>>> >
> >>>> > where
> >>>> >
> >>>> > it gets added doesn't matter (even if you don't end up adding it as
> >>>> >
> >>>> > first
> >>>> >
> >>>> > entry) since that is the ONLY provider for SPIFFE specific algorithm
> >>>> >
> >>>> > you
> >>>> >
> >>>> > might have.
> >>>> >
> >>>> > We do *not* have custom algorithm for Key/Trust StoreMangers. Which
> >>>> >
> >>>> > means
> >>>> >
> >>>> > we have to use X509, PKIX etc "Standard Algorithms" ((
> >>>> >
> >>>> > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> >>>> > StandardNames.html
> >>>> > ))
> >>>> >
> >>>> > in our provider to override the TrustStoreManager (see my sample
> code)
> >>>> >
> >>>> > and
> >>>> >
> >>>> > KeyStoreManger and KeyManager. This creates another challenge
> >>>> >
> >>>> > mentioned in
> >>>> >
> >>>> > the below point.
> >>>> >
> >>>> > 2. In order to make our Provider for loading custom TrustStore work,
> >>>> we
> >>>> > have to add the provider as 'first' in the list since there are
> others
> >>>> >
> >>>> > with
> >>>> >
> >>>> > the same algorithm.
> >>>> >
> >>>> > However, the programatic way of adding provider
> >>>> > (Security.insertProviderAt()) is *not* deterministic for ordering
> >>>> since
> >>>> > different code can call the method for a different provider and
> >>>> >
> >>>> > depending
> >>>> >
> >>>> > upon the order of the call our provider can be first or pushed down
> >>>> the
> >>>> > list. This can happen very well in any client application using
> Kafka.
> >>>> >
> >>>> > This
> >>>> >
> >>>> > is specially problematic for a case when you want to guarantee order
> >>>> >
> >>>> > for a
> >>>> >
> >>>> > Provider having "Standard Algorithms".
> >>>> >
> >>>> > If we add our provider in java.security file that definitely
> >>>> guarantees
> >>>> > the order(unless somebody calls removeProvider() which is less
> >>>> >
> >>>> > likely). But
> >>>> >
> >>>> > if we add our provider in java.security file it will defeat the
> >>>> >
> >>>> > purpose of
> >>>> >
> >>>> > the KIP-492.
> >>>> >
> >>>> > In the gist - Apache Kafka must not rely on "un-deterministic"
> method
> >>>> >
> >>>> > to
> >>>> >
> >>>> > rely on Provider ordering.
> >>>> >
> >>>> > 3. If we just use existing ssl.provider kafka configuration then our
> >>>> > provider will be used in SSLContext.getInstance(protocol, provider)
> >>>> >
> >>>> > call in
> >>>> >
> >>>> > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/>
> >>>> and
> >>>> > if our provider does not have implementation for
> >>>> > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks
> >>>> >
> >>>> > (we
> >>>> >
> >>>> > tested it). Example:
> >>>> >
> >>>> > In
> >>>> >
> >>>> > MyProvider sample above you see that I didn't add SSLContext.TLSv1
> as
> >>>> > "Service+Algorithm" and that didn't work for me. In SPIFFE provider
> >>>> you
> >>>> > don't have this challenge since you are planning to bypass
> >>>> >
> >>>> > ssl.provider as
> >>>> >
> >>>> > you mention in the KIP-492.
> >>>> >
> >>>> > *Overall summary,*
> >>>> >
> >>>> > 1. Any provider based mechanisms- a) existing ssl.provider and
> >>>> >
> >>>> > b)KIP-492,
> >>>> >
> >>>> > for loading key/trust store using "Standard Algorithms" do not work
> >>>> >
> >>>> > 2. Approach suggested in our KIP-486 works without any issue and it
> is
> >>>> > *not* our context specific solve
> >>>> >
> >>>> > 3. Based on above we feel KIP-492 and KIP-486 are complimentary
> >>>> changes
> >>>> > and not contradicting or redundent.
> >>>> >
> >>>> > If you want we can do a joint session somehow to walk through the
> >>>> >
> >>>> > sample I
> >>>> >
> >>>> > have and various experiments I did. I would encourage you to do
> >>>> similar
> >>>> > exercise by writing a Provider for "Standard Algorithm" for
> >>>> > TrustStoreManager (like our needs) and see what you find since only
> >>>> >
> >>>> > writing
> >>>> >
> >>>> > samples can bring out the complexity/challenges we face.
> >>>> >
> >>>> > Thanks
> >>>> > Maulin
> >>>> >
> >>>> > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <
> >>>> >
> >>>> > maulin.vasavada@gmail.
> >>>> >
> >>>> > com> wrote:
> >>>> >
> >>>> > Just to update - still working on it. Get to work only on and off on
> >>>> >
> >>>> > it :(
> >>>> >
> >>>> > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
> >>>> >
> >>>> > maulin.vasavada@gmail.com>
> >>>> >
> >>>> > wrote:
> >>>> >
> >>>> > Hi Harsha
> >>>> >
> >>>> > Let me try to write samples and will let you know.
> >>>> >
> >>>> > Thanks
> >>>> > Maulin
> >>>> >
> >>>> > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <ha...@gmail.com>
> >>>> >
> >>>> > wrote:
> >>>> >
> >>>> > Hi Maulin,
> >>>> > With java security providers can be as custom you would
> >>>> >
> >>>> > like
> >>>> >
> >>>> > it to
> >>>> > be. If you only want to to implement a custom way of loading the
> >>>> >
> >>>> > keystore
> >>>> >
> >>>> > and truststore and not implement any protocol/encryption handling
> you
> >>>> can
> >>>> > leave them empty and no need to implement. Have you looked into the
> >>>> links I
> >>>> > pasted before?
> >>>> >
> >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >>>> >
> >>>> >
> >>>>
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> >>>> >
> >>>> > java
> >>>> >
> >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >>>> > spiffe-security-provider/src/main/java/spiffe/api/provider/
> >>>> > SpiffeTrustManager.java <http://spiffetrustmanager.java/>
> >>>> >
> >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >>>> >
> >>>> >
> >>>>
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> >>>> >
> >>>> > java
> >>>> >
> >>>> > Can you please tell me which methods are too complex in above to
> >>>> >
> >>>> > implement
> >>>> >
> >>>> > or unnecessary? You are changing anything in SSL/TLS implementations
> >>>> > provided by
> >>>> >
> >>>> > All of the implementations delegating the checks to the default
> >>>> > implementation anyway.
> >>>> > Spire agent is an example, its nothing but a GRPC server listening
> >>>> >
> >>>> > on a
> >>>> >
> >>>> > unix domain socket . Above code is making a RPC call to the local
> >>>> >
> >>>> > daemon
> >>>> >
> >>>> > to
> >>>> > get the certificate and keys. The mechanics are pretty much same as
> >>>> >
> >>>> > what
> >>>> >
> >>>> > you are asking for.
> >>>> >
> >>>> > Thanks,
> >>>> > Harsha
> >>>> >
> >>>> > On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
> >>>> >
> >>>> > maulin.vasavada@gmail.com>
> >>>> >
> >>>> > wrote:
> >>>> >
> >>>> > Imagine a scenario like - We know we have a custom KMS and as a
> >>>> >
> >>>> > Kafka
> >>>> >
> >>>> > owner
> >>>> >
> >>>> > we want to comply to using that KMS source to load keys/certs. As
> >>>> >
> >>>> > a
> >>>> >
> >>>> > Kafka
> >>>> >
> >>>> > owner we know how to integrate with KMS but doesn't necessarily
> >>>> >
> >>>> > have
> >>>> >
> >>>> > to
> >>>> >
> >>>> > know anything about cipher suites, algorithms, and SSL/TLS
> >>>> >
> >>>> > implementation.
> >>>> >
> >>>> > Going the Provider way requires to know lot more than we should,
> >>>> >
> >>>> > isn't it?
> >>>> >
> >>>> > Not that we would have concern/shy-away knowing those details -
> >>>> >
> >>>> > but
> >>>> >
> >>>> > if we
> >>>> >
> >>>> > don't have to - why should we?
> >>>> >
> >>>> > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
> >>>> >
> >>>> > maulin.vasavada@gmail.com>
> >>>> >
> >>>> > wrote:
> >>>> >
> >>>> > Hi Harsha
> >>>> >
> >>>> > We don't have spire (or similar) agents and we do not have
> >>>> >
> >>>> > keys/certs
> >>>> >
> >>>> > locally on any brokers. To elaborate more on my previous email,
> >>>> >
> >>>> > I agree that Java security Providers are used in much broader
> >>>> >
> >>>> > sense
> >>>> >
> >>>> > - to
> >>>> >
> >>>> > have a particular implementation of an algorithm, use specific
> >>>> >
> >>>> > cipher
> >>>> >
> >>>> > suites for SSL , OR in our current team's case have a
> >>>> >
> >>>> > particular
> >>>> >
> >>>> > way to
> >>>> >
> >>>> > leverage pre-generated SSL sessions. However, the scope of our
> >>>> >
> >>>> > KIP
> >>>> >
> >>>> > (486)
> >>>> >
> >>>> > is
> >>>> >
> >>>> > much restricted than that. We merely intend to provide a custom
> >>>> > keystore/truststore for our SSL connections and not really worry
> >>>> >
> >>>> > about
> >>>> >
> >>>> > underlying specific SSL/TLS implementation. This simplifies it
> >>>> >
> >>>> > a
> >>>> >
> >>>> > lot for
> >>>> >
> >>>> > us to keep the concerns separate and clear.
> >>>> >
> >>>> > I feel our approach is more complimentary such that it allows
> >>>> >
> >>>> > for
> >>>> >
> >>>> > using
> >>>> >
> >>>> > keystores of choice while retaining the flexibility to use any
> >>>> > underlying/available Provider for actually making the SSL call.
> >>>> >
> >>>> > We agree with KIP-492's approach based on Providers (and Java's
> >>>> > recommendation), but also strongly believe that our approach can
> >>>> >
> >>>> > compliment
> >>>> >
> >>>> > it very effectively for reasons explained above.
> >>>> >
> >>>> > Thanks
> >>>> > Maulin
> >>>> >
> >>>> > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <
> >>>> >
> >>>> > kafka@harsha.io
> >>>> >
> >>>> > wrote:
> >>>> >
> >>>> > Hi Maulin,
> >>>> >
> >>>> > On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
> >>>> >
> >>>> > maulin.vasavada@gmail.
> >>>> >
> >>>> > com>
> >>>> > wrote:
> >>>> >
> >>>> > Hi Harsha
> >>>> >
> >>>> > The reason we rejected the SslProvider route is that - we
> >>>> >
> >>>> > only
> >>>> >
> >>>> > needed
> >>>> >
> >>>> > a
> >>>> >
> >>>> > custom way to load keys/certs. Not touch any policy that
> >>>> >
> >>>> > existing
> >>>> >
> >>>> > Providers
> >>>> >
> >>>> > govern like SunJSSE Provider.
> >>>> >
> >>>> > We have exactly the same requirements to load certs and keys
> >>>> >
> >>>> > through
> >>>> >
> >>>> > spire
> >>>> >
> >>>> > agent. We used security.provider to do that exactly. I am not
> >>>> >
> >>>> > sure
> >>>> >
> >>>> > why
> >>>> >
> >>>> > you
> >>>> >
> >>>> > would be modifying any policies provided by default SunJSSE
> >>>> >
> >>>> > provider.
> >>>> >
> >>>> > Can
> >>>> >
> >>>> > you give me an example of having custom provider that will
> >>>> >
> >>>> > override an
> >>>> >
> >>>> > existing policy in SunJSSE provider.
> >>>> >
> >>>> > As pointed out earlier, this kip
> >>>> >
> >>>> > https://cwiki.apache.org/confluence/display/KAFKA/
> >>>> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> >>>> >
> >>>> > allows
> >>>> > you to load security.provider through config.
> >>>> > Take a look at the examples I gave before
> >>>> >
> >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >>>> >
> >>>> >
> >>>>
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> >>>> >
> >>>> > java
> >>>> >
> >>>> > It registers KeyManagerFactory and TrustManagerFactory and
> >>>> >
> >>>> > Keystore
> >>>> >
> >>>> > algorithm.
> >>>> > Implement your custom way of loading Keystore in here
> >>>> >
> >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >>>> >
> >>>> >
> >>>>
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> >>>> >
> >>>> > java
> >>>> >
> >>>> > and Trust manager like here
> >>>> >
> >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >>>> > spiffe-security-provider/src/main/java/spiffe/api/provider/
> >>>> > SpiffeTrustManager.java <http://spiffetrustmanager.java/>
> >>>> >
> >>>> > In your Kafka client you can set the security.provider to your
> >>>> >
> >>>> > custom
> >>>> >
> >>>> > implementation and with this fix
> >>>> > https://issues.apache.org/jira/browse/KAFKA-8191 you can set
> >>>> > keyManagerAlgorigthm and trustManagerAlgorithm configs.
> >>>> >
> >>>> > All of this is in your clients and broker side and do not need
> >>>> >
> >>>> > to
> >>>> >
> >>>> > touch
> >>>> >
> >>>> > any
> >>>> > policy changes at JVM level. You'll register the providers in
> >>>> >
> >>>> > the
> >>>> >
> >>>> > priority
> >>>> >
> >>>> > order and can still have SunJSSE provider and have your custom
> >>>> >
> >>>> > provider
> >>>> >
> >>>> > to
> >>>> >
> >>>> > implement the key and trust managers.
> >>>> >
> >>>> > The ask here is different than KIP-492. We don't have any need
> >>>> >
> >>>> > to
> >>>> >
> >>>> > modify/specify the algorithm parameter. Does that make sense?
> >>>> >
> >>>> > The ask in KIP is introducing new interfaces where the KIP's
> >>>> > goal/motivation can be achieved through the security.provider
> >>>> >
> >>>> > and
> >>>> >
> >>>> > we
> >>>> >
> >>>> > worked
> >>>> > on similar goal without touching any Keystore or Truststore
> >>>> >
> >>>> > interfaces.
> >>>> >
> >>>> > My advise is against changing or introducing new interfaces
> >>>> >
> >>>> > when
> >>>> >
> >>>> > it can
> >>>> >
> >>>> > work through security.provider.
> >>>> >
> >>>> > Thanks,
> >>>> > Harsha
> >>>> >
> >>>> > Thanks
> >>>> >
> >>>> > Maulin
> >>>> >
> >>>> > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
> >>>> >
> >>>> > kafka@harsha.io>
> >>>> >
> >>>> > wrote:
> >>>> >
> >>>> > In your KIP you added security. provider as rejected
> >>>> >
> >>>> > alternative
> >>>> >
> >>>> > and
> >>>> >
> >>>> > specified "its not the correct way". Do you mind explaining
> >>>> >
> >>>> > why
> >>>> >
> >>>> > its
> >>>> >
> >>>> > not? I
> >>>> >
> >>>> > didn't find any evidence in Java docs to say so. Contrary to
> >>>> >
> >>>> > your
> >>>> >
> >>>> > statement
> >>>> >
> >>>> > it does say in the java docs
> >>>> > " However, please note that a provider can be used to
> >>>> >
> >>>> > implement
> >>>> >
> >>>> > any
> >>>> >
> >>>> > security service in Java that uses a pluggable architecture
> >>>> >
> >>>> > with
> >>>> >
> >>>> > a
> >>>> >
> >>>> > choice
> >>>> >
> >>>> > of implementations that fit underneath."
> >>>> >
> >>>> > Java Security Providers have been used by other projects to
> >>>> >
> >>>> > provide
> >>>> >
> >>>> > such
> >>>> >
> >>>> > integration . I am not sure if you looked into Spiffe
> >>>> >
> >>>> > project to
> >>>> >
> >>>> > efficiently distribute certificates but here is an example of
> >>>> >
> >>>> > Java
> >>>> >
> >>>> > provider
> >>>> >
> >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >>>> >
> >>>> >
> >>>>
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> >>>> >
> >>>> > java which
> >>>> > obtains certificates from local daemons.
> >>>> > These integrations are being used in Tomcat, Jetty etc.. We
> >>>> >
> >>>> > are
> >>>> >
> >>>> > also
> >>>> >
> >>>> > using
> >>>> >
> >>>> > Security provider to do the same in our Kafka clusters. So
> >>>> >
> >>>> > unless I
> >>>> >
> >>>> > see
> >>>> >
> >>>> > more evidence why security.provider doesn't work for you
> >>>> >
> >>>> > adding
> >>>> >
> >>>> > new
> >>>> >
> >>>> > interfaces while there exists more cleaner way of achieving
> >>>> >
> >>>> > the
> >>>> >
> >>>> > goals
> >>>> >
> >>>> > of
> >>>> >
> >>>> > this KIP is unnecessary and breaks the well known security
> >>>> >
> >>>> > interfaces
> >>>> >
> >>>> > provided by Java itself.
> >>>> >
> >>>> > Thanks,
> >>>> > Harsha
> >>>> >
> >>>> > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
> >>>> >
> >>>> > kafka@harsha.io
> >>>> >
> >>>> > wrote:
> >>>> >
> >>>> > Hi Maulin,
> >>>> > Not sure if you looked at my previous replies. This
> >>>> >
> >>>> > changes
> >>>> >
> >>>> > are not required as there is already security Provider to do
> >>>> >
> >>>> > what you
> >>>> >
> >>>> > are
> >>>> >
> >>>> > proposing. This KIP
> >>>> >
> >>>> > https://cwiki.apache.org/confluence/display/KAFKA/
> >>>> >
> >>>> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> >>>> >
> >>>> > also
> >>>> >
> >>>> > addresses easy registration of such providers.
> >>>> >
> >>>> > Thanks,
> >>>> > Harsha
> >>>> >
> >>>> > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
> >>>> >
> >>>> > <maulin.vasavada@gmail.
> >>>> >
> >>>> > com> wrote:
> >>>> >
> >>>> > Bump! Can somebody please review this?
> >>>> >
> >>>> > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
> >>>> >
> >>>> > maulin.vasavada@gmail.com>
> >>>> >
> >>>> > wrote:
> >>>> >
> >>>> > Bump! Can somebody please review this?
> >>>> >
> >>>> > --
> >>>> > Thanks,
> >>>> > M.Sai Sandeep
> >>>> >
> >>>> >
> >>>>
> >>>
> On Wed, Aug 28, 2019 at 2:18 PM Maulin Vasavada <maulin.vasavada@gmail.com
> >
> wrote:
>
> >
> >
> > ---------- Forwarded message ---------
> > From: Maulin Vasavada <ma...@gmail.com>
> > Date: Wed, Aug 28, 2019 at 1:51 PM
> > Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and
> > TrustStore
> > To: <de...@kafka.apache.org>
> >
> >
> > Hi Harsha
> >
> > As we synced-up offline on this topic, we hope you don't have any more
> > clarifications that you are seeking. If that is the case, can you please
> > help us move this forward and discuss what changes you would expect on
> the
> > KIP design in order to make it valuable contribution?
> >
> > Just FYI - we verified our primary design change with the author of Sun's
> > X509 Trustmanager implementation and the outcome is that what we are
> > proposing makes sense at the heart of it - "Instead of writing
> TrustManager
> > just plugin the Trust store". We are open to discuss additional changes
> > that you/anybody else would like to see on the functionality however.
> >
> >
> > Thanks
> > Maulin
> >
> > On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada <
> maulin.vasavada@gmail.com>
> > wrote:
> >
> >> Hi Harsha
> >>
> >> Any response on my question? I feel this KIP is worth accommodating.
> Your
> >> help is much appreciated.
> >>
> >> Thanks
> >> Maulin
> >>
> >> On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada <
> >> maulin.vasavada@gmail.com> wrote:
> >>
> >>> Hi Harsha
> >>>
> >>> I've examined the SPIFFE provider more and have one question -
> >>>
> >>> If SPIFFE didn't have a need to do checkSpiffeId() call at the below
> >>> location, would you really still write the Provider? *OR*
> >>> Would you just use TrustManagerFactory.init(KeyStore) signature to pass
> >>> the KeyStore from set of certs returned by spiffeIdManager.
> >>> getTrustedCerts()?
> >>>
> >>>
> >>>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/CertificateUtils.java#L100
> >>>
> >>>
> >>> /**
> >>>> * Validates that the SPIFFE ID is present and matches the SPIFFE ID
> >>>> configured in
> >>>> * the java.security property ssl.spiffe.accept
> >>>> *
> >>>> * If the authorized spiffe ids list is empty any spiffe id is
> authorized
> >>>> *
> >>>> * @param chain an array of X509Certificate that contains the Peer's
> >>>> SVID to be validated
> >>>> * @throws CertificateException when either the certificates doesn't
> >>>> have a SPIFFE ID or the SPIFFE ID is not authorized
> >>>> */
> >>>> static void checkSpiffeId(X509Certificate[] chain) throws
> >>>> CertificateException {
> >>>
> >>>
> >>>
> >>> Thanks
> >>> Maulin
> >>>
> >>>
> >>> On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani <ka...@harsha.io>
> >>> wrote:
> >>>
> >>>> Maulin,
> >>>> The code parts you are pointing are specific for Spiffe
> >>>> and if
> >>>> you are talking about validate method which uses PKIX check like any
> >>>> other provider does.
> >>>> If you want to default to SunJSSE everywhere you can do so by
> delegating
> >>>> the calls in these methods to SunJSSE provider.
> >>>>
> >>>> TrustManagerFactory tmf = TrustManagerFactory
> >>>> .getInstance(TrustManagerFactory.getDefaultAlgorithm());and use
> >>>> tmf.chekServerTrusted()
> >>>> or use
> >>>>
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/TrustManagerFactory.html#getInstance(java.lang.String)if
> >>>> you want a specific provider.
> >>>>
> >>>> -Harsha
> >>>>
> >>>>
> >>>> On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada <
> >>>> maulin.vasavada@gmail.com>
> >>>> wrote:
> >>>>
> >>>> > Okay, so I take that you guys agree that I have to write a 'custom'
> >>>> > algorithm and a provider to make it work , correct?
> >>>> >
> >>>> > Now, for Harsha's comment "Here the 'Custom' Algorithm is not an
> >>>> > implementation per say , ..." , I diagree. You can refer to https://
> >>>> >
> >>>>
> github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/
> >>>> > SpiffeTrustManager.java#L91 and
> >>>> >
> >>>>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> >>>> > provider/CertificateUtils.java#L100
> >>>> >
> >>>> > "that code" is the customization you have for the custom way to
> check
> >>>> > something on top of regular checks. That method is NOT doing custom
> >>>> > truststore loading. It is validating/verifying something in the
> >>>> "custom"
> >>>> > way with spiffeId.
> >>>> > I bet that without that you won't have a need of the custom
> algorithm
> >>>> in
> >>>> > the first place.
> >>>> >
> >>>> > Let me know if you agree to this.
> >>>> >
> >>>> > Thanks
> >>>> > Maulin
> >>>> >
> >>>> > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <mp...@gmail.com>
> >>>> wrote:
> >>>> >
> >>>> > Hi Maulin, thanks for the discussion. As Harsha pointed out, to use
> >>>> the
> >>>> > KIP492, you need to create a new provider, register a *new* custom
> >>>> > algorithm for your keymanager and trustmanager factory
> >>>> implementations.
> >>>> > After this, the kafka server configuration can be done as given
> below
> >>>> >
> >>>> > # Register the provider class with custom algorithm, say CUSTOM
> >>>> security.
> >>>> > provider.classes=com.company.security.CustomProvider
> >>>> > <http://security.provider.classes
> >>>> =com.company.security.customprovider/>
> >>>> >
> >>>> > # Register the keymanager and trustmanager algorithms
> >>>> > # These algorithms indicate that the Keymanager and Trustmanagers
> >>>> > registered under the algorithm "CUSTOM" needs to be used
> >>>> > ssl.trustmanager.algorithm=CUSTOM
> >>>> > ssl.keymanager.algorithm=CUSTOM
> >>>> >
> >>>> > And the customprovider looks like this...
> >>>> >
> >>>> > public class CustomProvider extends Provider {
> >>>> > public CustomProvider() {
> >>>> > super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and TrustStore");
> >>>> > super.put("KeyManagerFactory.CUSTOM", "customKeyManagerFactory");
> >>>> > super.put("TrustManagerFactory.CUSTOM",
> >>>> > "customTrustManagerFactory");
> >>>> > }
> >>>> > }
> >>>> >
> >>>> > The PR for this is in review and can be found here:
> >>>> https://github.com/
> >>>> > apache/kafka/pull/7090
> >>>> > This PR includes the fixed insertProviderAt function call.
> >>>> >
> >>>> > On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani <
> kafka@harsha.io>
> >>>> > wrote:
> >>>> >
> >>>> > Answers are in-line
> >>>> >
> >>>> > On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada <
> >>>> maulin.vasavada@gmail.
> >>>> > com
> >>>> >
> >>>> > wrote:
> >>>> >
> >>>> > Hi Colin
> >>>> >
> >>>> > When I refer to "standard" or "custom" algorithms I am following
> Java
> >>>> > security Provider Terminology. You can refer to
> >>>> > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> >>>> > StandardNames.html#TrustManagerFactory link I provided earlier in
> the
> >>>> > emails. It says PKIX is the default Algorithm for
> TrustManagerFactory.
> >>>> >
> >>>> > 1. For SPIFFE, I am not sure why you are saying 'it does not
> implement
> >>>> > custom algorithms' because the following file clearly indicates that
> >>>> it
> >>>> > does use custom algorithm-
> >>>> >
> >>>> >
> >>>>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> >>>> >
> >>>> > provider/SpiffeProvider.java#L17
> >>>> >
> >>>> > Algorithm value:
> >>>> >
> >>>> >
> >>>>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> >>>> >
> >>>> > provider/SpiffeProviderConstants.java#L6
> >>>> >
> >>>> > @Harsha do you want to chime in since you use that provider?
> >>>> >
> >>>> > Here the "Custom" Algorithm is not an implementation per say ,
> rather
> >>>> >
> >>>> > used
> >>>> >
> >>>> > to invoke the custom trust store factory and key manager factory.
> You
> >>>> are
> >>>> > not moving away from "standard" alogrithms that are available.
> >>>> >
> >>>> >
> >>>>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> >>>> > provider/SpiffeTrustManager.java
> >>>> >
> >>>> > As you can see it delegates all the calls of verification of
> >>>> certificate
> >>>> >
> >>>> > to
> >>>> >
> >>>> > the default implementation available.
> >>>> > So in our implementation we still use PKIX to verify the certificate
> >>>> > chain. So you are not losing anything here and Spiffe is not
> >>>> reimplementing
> >>>> > the verification process.
> >>>> >
> >>>> > 2. I already mentioned in my 3rd point, in my previous post, why
> using
> >>>> >
> >>>> > ssl.provider does NOT work. I updated KIP-486 in "rejected
> >>>> >
> >>>> > alternatives"
> >>>> >
> >>>> > also why ssl.provider does not work.
> >>>> >
> >>>> > As mentioned before , provider is the right way to go. I am still
> not
> >>>> > understanding the gap is.
> >>>> > If I understand correctly your argument is , provider is going to
> ask
> >>>> to
> >>>> > implement a custom algorithm.
> >>>> > My answer to that is , no you are not re-implementing the algorithm.
> >>>> >
> >>>> > Please
> >>>> >
> >>>> > check the above link , TrustManager implementation it delegates the
> >>>> calls
> >>>> > back. There is no need to implement your own here.
> >>>> >
> >>>> > 3. Security.insertProviderAt() comments were based on assumption if
> >>>> >
> >>>> > KIP-492
> >>>> >
> >>>> > changes are done and we use that mechanism to configure providers
> >>>> >
> >>>> > instead
> >>>> >
> >>>> > of ssl.provider configuration.
> >>>> >
> >>>> > KIP-492 has patch available and going through review.
> >>>> >
> >>>> > Can you read my all the points, I mentioned in my previous post,
> very
> >>>> >
> >>>> > carefully? I am covering all the aspects in explaining. I am open to
> >>>> >
> >>>> > still
> >>>> >
> >>>> > discuss more to clarify any doubts.
> >>>> >
> >>>> > "3. If we just use existing ssl.provider kafka configuration then
> our
> >>>> > provider will be used in SSLContext.getInstance(protocol, provider)
> >>>> call
> >>>> >
> >>>> > in
> >>>> >
> >>>> > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/>
> >>>> and
> >>>> > if our provider does not have
> >>>> > implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we
> >>>> >
> >>>> > tested
> >>>> >
> >>>> > it). Example: In MyProvider sample above you see that I didn't add
> >>>> > SSLContext.TLSv1 as
> >>>> > "Service+Algorithm" and that didn't work for me. In SPIFFE provider
> >>>> you
> >>>> > don't have this challenge since you are planning to bypass
> >>>> ssl.provider
> >>>> >
> >>>> > as
> >>>> >
> >>>> > you mention in the KIP-492."
> >>>> >
> >>>> > Yes here you need to pass the protocol that your
> >>>> KeyManager/TrustManager
> >>>> > registered with and in no way its deviating from TLS RFC spec.
> >>>> >
> >>>> >
> >>>>
> https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/
> >>>> > javax/net/ssl/SSLContext.java#L134
> >>>> >
> >>>> > My suggestion here is for you to implement a simple Security
> Provider
> >>>> as
> >>>> > you did before and register a Algorithm. You can use the existing
> >>>> > implementation in SpiffeProvider and plug in changes where you need
> to
> >>>> > retrieve the certificates from by making RPC call.
> >>>> >
> >>>> > Run an end-to-end test with Kafka broker coming up with your
> provider
> >>>> > making calls to RPC call. You do need to pass the "custom algorithm"
> >>>> that
> >>>> > you registered in your key manager to invoke the provider. I think
> >>>> your
> >>>> > concern is this approach is replacing the existing known
> ciphersuites
> >>>> and
> >>>> > certificate validation provided by java. Which its not.
> >>>> >
> >>>> > Now test the TLS connection you can do so via openssl -s_client
> >>>> options
> >>>> >
> >>>> > to
> >>>> >
> >>>> > test if everything else is working.
> >>>> >
> >>>> > I am happy to share configs that we used if you are interested.
> >>>> >
> >>>> > Thanks,
> >>>> > Harsha
> >>>> >
> >>>> > Thanks
> >>>> > Maulin
> >>>> >
> >>>> > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <cm...@apache.org>
> >>>> >
> >>>> > wrote:
> >>>> >
> >>>> > Hi Maulin,
> >>>> >
> >>>> > A lot of JSSE providers don't implement custom algorithms. Spire is
> a
> >>>> >
> >>>> > good
> >>>> >
> >>>> > example of a JSSE provider that doesn't, and yet is still useful to
> >>>> >
> >>>> > many
> >>>> >
> >>>> > people. Your JSSE provider can work fine even if it doesn't
> implement
> >>>> a
> >>>> > custom algorithm.
> >>>> >
> >>>> > Maybe I'm missing something, but I don't understand the discussion
> of
> >>>> > Security.insertProviderAt() that you included. SslEngineBuilder
> >>>> doesn't
> >>>> >
> >>>> > use
> >>>> >
> >>>> > that API to get the security provider. Instead, it calls
> >>>> > "SSLContext.getInstance(protocol, provider)", where provider is the
> >>>> >
> >>>> > name
> >>>> >
> >>>> > of the provider.
> >>>> >
> >>>> > best,
> >>>> > Colin
> >>>> >
> >>>> > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
> >>>> >
> >>>> > On top of everything above I feel strongly to add the 4th point
> which
> >>>> >
> >>>> > is
> >>>> >
> >>>> > based on Java APIs for TrustManagerFactory.init(KeyStore) (
> >>>> >
> >>>> > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> >>>> > TrustManagerFactory.html#init(java.security.KeyStore
> >>>> > <http://java.security.keystore/>)
> >>>> > )
> >>>> >
> >>>> > and KeyManagerFactory.init(KeyStore, char[]) (
> >>>> >
> >>>> >
> >>>>
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory
> >>>> .
> >>>> >
> >>>> >
> >>>> > html#init(java.security.KeyStore <http://java.security.keystore/
> >>>> >,%20char[])
> >>>> >
> >>>> >
> >>>> > ).
> >>>> >
> >>>> > 4. The above APIs are intended to support providing "trust/key
> >>>> >
> >>>> > material"
> >>>> >
> >>>> > from the user without having to write their own
> >>>> >
> >>>> > TrustManager/KeyManagers.
> >>>> >
> >>>> > To quote from the TrustManagerFactory.init()'s documentation
> >>>> >
> >>>> > "Initializes
> >>>> >
> >>>> > this factory with a source of certificate authorities and related
> >>>> trust
> >>>> > material."
> >>>> > To quote from the KeyManagerFactory.init()'s documentation
> >>>> "Initializes
> >>>> > this factory with a source of key material."
> >>>> >
> >>>> > Based on this it is clear that there is a flexibility provided by
> Java
> >>>> >
> >>>> > to
> >>>> >
> >>>> > to enable developers to provide the required trust/key material
> loaded
> >>>> >
> >>>> > from
> >>>> >
> >>>> > "anywhere" without requiring them to write custom provider OR
> >>>> trust/key
> >>>> > managers. This same flexibility is reflected in Kafka code also
> where
> >>>> >
> >>>> > it
> >>>> >
> >>>> > loads the trust/keys from a local file and doesn't require writing a
> >>>> > Provider necessarily. If we do NOT have a custom algorithm, it makes
> >>>> >
> >>>> > less
> >>>> >
> >>>> > sense to write a Provider.
> >>>> >
> >>>> > Thanks
> >>>> > Maulin
> >>>> >
> >>>> > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
> >>>> >
> >>>> > maulin.vasavada@gmail.com>
> >>>> >
> >>>> > wrote:
> >>>> >
> >>>> > Hi Harsha/Colin
> >>>> >
> >>>> > I did the sample with a custom Provider for TrustStoreManager and
> >>>> tried
> >>>> > using ssl.provider Kafka config AND the way KIP-492 is suggesting
> (by
> >>>> > adding Provider programmatically instead of relying on
> >>>> >
> >>>> > ssl.provider+java.
> >>>> >
> >>>> > security. The below sample is followed by my detailed findings. I'll
> >>>> > appreciate if you can go through it carefully and see
> >>>> >
> >>>> > if you
> >>>> >
> >>>> > see my point.
> >>>> >
> >>>> > package providertest;
> >>>> >
> >>>> > import java.security.Provider <http://java.security.provider/>
> >>>> <http://
> >>>> > java.security.provider/>;
> >>>> >
> >>>> > public class MyProvider extends Provider {
> >>>> >
> >>>> > private static final String name = "MyProvider"; private static
> double
> >>>> > version = 1.0d;
> >>>> > private static String info = "Maulin's SSL Provider v"+version;
> >>>> >
> >>>> > public MyProvider() {
> >>>> > super(name, version, info);
> >>>> > this.put("TrustManagerFactory.PKIX",
> >>>> >
> >>>> > "providertest.MyTrustManagerFactory");
> >>>> >
> >>>> > }
> >>>> > }
> >>>> >
> >>>> > *Details:*
> >>>> >
> >>>> > KIP-492 documents that it will use Security.addProvider() assuming
> it
> >>>> >
> >>>> > will
> >>>> >
> >>>> > add it as position '0' which is not a correct assumption. The
> >>>> > addProvider()'s documentation says it will add it to the last
> >>>> available
> >>>> > position. You may want to correct that to say
> >>>> > Security.insertProviderAt(provider, 1).
> >>>> >
> >>>> > Now coming back to our specific discussion,
> >>>> >
> >>>> > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add
> >>>> >
> >>>> > that
> >>>> >
> >>>> > provider in the provider list via Security.addProvider() the
> position
> >>>> >
> >>>> > where
> >>>> >
> >>>> > it gets added doesn't matter (even if you don't end up adding it as
> >>>> >
> >>>> > first
> >>>> >
> >>>> > entry) since that is the ONLY provider for SPIFFE specific algorithm
> >>>> >
> >>>> > you
> >>>> >
> >>>> > might have.
> >>>> >
> >>>> > We do *not* have custom algorithm for Key/Trust StoreMangers. Which
> >>>> >
> >>>> > means
> >>>> >
> >>>> > we have to use X509, PKIX etc "Standard Algorithms" ((
> >>>> >
> >>>> > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> >>>> > StandardNames.html
> >>>> > ))
> >>>> >
> >>>> > in our provider to override the TrustStoreManager (see my sample
> code)
> >>>> >
> >>>> > and
> >>>> >
> >>>> > KeyStoreManger and KeyManager. This creates another challenge
> >>>> >
> >>>> > mentioned in
> >>>> >
> >>>> > the below point.
> >>>> >
> >>>> > 2. In order to make our Provider for loading custom TrustStore work,
> >>>> we
> >>>> > have to add the provider as 'first' in the list since there are
> others
> >>>> >
> >>>> > with
> >>>> >
> >>>> > the same algorithm.
> >>>> >
> >>>> > However, the programatic way of adding provider
> >>>> > (Security.insertProviderAt()) is *not* deterministic for ordering
> >>>> since
> >>>> > different code can call the method for a different provider and
> >>>> >
> >>>> > depending
> >>>> >
> >>>> > upon the order of the call our provider can be first or pushed down
> >>>> the
> >>>> > list. This can happen very well in any client application using
> Kafka.
> >>>> >
> >>>> > This
> >>>> >
> >>>> > is specially problematic for a case when you want to guarantee order
> >>>> >
> >>>> > for a
> >>>> >
> >>>> > Provider having "Standard Algorithms".
> >>>> >
> >>>> > If we add our provider in java.security file that definitely
> >>>> guarantees
> >>>> > the order(unless somebody calls removeProvider() which is less
> >>>> >
> >>>> > likely). But
> >>>> >
> >>>> > if we add our provider in java.security file it will defeat the
> >>>> >
> >>>> > purpose of
> >>>> >
> >>>> > the KIP-492.
> >>>> >
> >>>> > In the gist - Apache Kafka must not rely on "un-deterministic"
> method
> >>>> >
> >>>> > to
> >>>> >
> >>>> > rely on Provider ordering.
> >>>> >
> >>>> > 3. If we just use existing ssl.provider kafka configuration then our
> >>>> > provider will be used in SSLContext.getInstance(protocol, provider)
> >>>> >
> >>>> > call in
> >>>> >
> >>>> > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/>
> >>>> and
> >>>> > if our provider does not have implementation for
> >>>> > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks
> >>>> >
> >>>> > (we
> >>>> >
> >>>> > tested it). Example:
> >>>> >
> >>>> > In
> >>>> >
> >>>> > MyProvider sample above you see that I didn't add SSLContext.TLSv1
> as
> >>>> > "Service+Algorithm" and that didn't work for me. In SPIFFE provider
> >>>> you
> >>>> > don't have this challenge since you are planning to bypass
> >>>> >
> >>>> > ssl.provider as
> >>>> >
> >>>> > you mention in the KIP-492.
> >>>> >
> >>>> > *Overall summary,*
> >>>> >
> >>>> > 1. Any provider based mechanisms- a) existing ssl.provider and
> >>>> >
> >>>> > b)KIP-492,
> >>>> >
> >>>> > for loading key/trust store using "Standard Algorithms" do not work
> >>>> >
> >>>> > 2. Approach suggested in our KIP-486 works without any issue and it
> is
> >>>> > *not* our context specific solve
> >>>> >
> >>>> > 3. Based on above we feel KIP-492 and KIP-486 are complimentary
> >>>> changes
> >>>> > and not contradicting or redundent.
> >>>> >
> >>>> > If you want we can do a joint session somehow to walk through the
> >>>> >
> >>>> > sample I
> >>>> >
> >>>> > have and various experiments I did. I would encourage you to do
> >>>> similar
> >>>> > exercise by writing a Provider for "Standard Algorithm" for
> >>>> > TrustStoreManager (like our needs) and see what you find since only
> >>>> >
> >>>> > writing
> >>>> >
> >>>> > samples can bring out the complexity/challenges we face.
> >>>> >
> >>>> > Thanks
> >>>> > Maulin
> >>>> >
> >>>> > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <
> >>>> >
> >>>> > maulin.vasavada@gmail.
> >>>> >
> >>>> > com> wrote:
> >>>> >
> >>>> > Just to update - still working on it. Get to work only on and off on
> >>>> >
> >>>> > it :(
> >>>> >
> >>>> > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
> >>>> >
> >>>> > maulin.vasavada@gmail.com>
> >>>> >
> >>>> > wrote:
> >>>> >
> >>>> > Hi Harsha
> >>>> >
> >>>> > Let me try to write samples and will let you know.
> >>>> >
> >>>> > Thanks
> >>>> > Maulin
> >>>> >
> >>>> > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <ha...@gmail.com>
> >>>> >
> >>>> > wrote:
> >>>> >
> >>>> > Hi Maulin,
> >>>> > With java security providers can be as custom you would
> >>>> >
> >>>> > like
> >>>> >
> >>>> > it to
> >>>> > be. If you only want to to implement a custom way of loading the
> >>>> >
> >>>> > keystore
> >>>> >
> >>>> > and truststore and not implement any protocol/encryption handling
> you
> >>>> can
> >>>> > leave them empty and no need to implement. Have you looked into the
> >>>> links I
> >>>> > pasted before?
> >>>> >
> >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >>>> >
> >>>> >
> >>>>
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> >>>> >
> >>>> > java
> >>>> >
> >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >>>> > spiffe-security-provider/src/main/java/spiffe/api/provider/
> >>>> > SpiffeTrustManager.java <http://spiffetrustmanager.java/>
> >>>> >
> >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >>>> >
> >>>> >
> >>>>
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> >>>> >
> >>>> > java
> >>>> >
> >>>> > Can you please tell me which methods are too complex in above to
> >>>> >
> >>>> > implement
> >>>> >
> >>>> > or unnecessary? You are changing anything in SSL/TLS implementations
> >>>> > provided by
> >>>> >
> >>>> > All of the implementations delegating the checks to the default
> >>>> > implementation anyway.
> >>>> > Spire agent is an example, its nothing but a GRPC server listening
> >>>> >
> >>>> > on a
> >>>> >
> >>>> > unix domain socket . Above code is making a RPC call to the local
> >>>> >
> >>>> > daemon
> >>>> >
> >>>> > to
> >>>> > get the certificate and keys. The mechanics are pretty much same as
> >>>> >
> >>>> > what
> >>>> >
> >>>> > you are asking for.
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Rajan Dhabalia <rd...@apache.org>.
*Hi Harsha/Maulin,I am following KIP-486 and KIP-492 and it seems
https://github.com/apache/kafka/pull/7090
<https://github.com/apache/kafka/pull/7090> is the right solution when one
wants to register custom factory class for KeyManager and TrustManager.
User can easily configure custom implementation of TrustManager and
KeyManager using factory Provider class.Configuration of the provider is
also simple and straightforward: 1. Create custom provider which defines
factory classes for KeyManager and TrustManagerpublic class CustomProvider
extends Provider { public CustomProvider()
{ super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and
TrustStore"); super.put("KeyManagerFactory.CUSTOM",
"customKeyManagerFactory");
super.put("TrustManagerFactory.CUSTOM","customTrustManagerFactory");
}}
1. Register provider at broker
startupjava.security.Security.addProvider(new CustomProvider());However,
this approach is useful when one wants to implement custom TrustManager for
X509 certs by extending X509ExtendedKeyManager and implement various
abstract methods such as: checkClientTrusted, checkServerTrusted, etc..In
JDK, default implementation class of X509ExtendedKeyManager is
X509TrustManagerImpl and one can’t extend or delegate calls to this class
because it’s final and same is applicable for other available providers
such as : BouncyCastleProviderTurstManager/KeyManager mainly serve two
purposes: 1. Provide certs/key2. Perform validationX509TrustManagerImpl
performs various RFC specified validations.#7090 can be helpful when user
has both above asks. However, problem defined at KIP-486 has different ask
where user wants to provide certs/key without copying/implementing Manager
class because all the available Manager classes are final and can’t be
extended or delegated. And security team in most of the companies don’t
allow custom/copying provider in order to get up to date with various RFC
validations provided into standard jdk provider.Many users manage keys and
certs into KMS and sometimes it’s not feasible to copy them to file-system
instead directly use them from the memory. So, KIP-486 provides a custom
way to load keys/certs without implementing security-provider.Thanks,Rajan*
On Wed, Aug 28, 2019 at 2:18 PM Maulin Vasavada <ma...@gmail.com>
wrote:
>
>
> Hi Harsha
>
> As we synced-up offline on this topic, we hope you don't have any more
> clarifications that you are seeking. If that is the case, can you please
> help us move this forward and discuss what changes you would expect on the
> KIP design in order to make it valuable contribution?
>
> Just FYI - we verified our primary design change with the author of Sun's
> X509 Trustmanager implementation and the outcome is that what we are
> proposing makes sense at the heart of it - "Instead of writing TrustManager
> just plugin the Trust store". We are open to discuss additional changes
> that you/anybody else would like to see on the functionality however.
>
>
> Thanks
> Maulin
>
> On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada <ma...@gmail.com>
> wrote:
>
>> Hi Harsha
>>
>> Any response on my question? I feel this KIP is worth accommodating. Your
>> help is much appreciated.
>>
>> Thanks
>> Maulin
>>
>> On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada <
>> maulin.vasavada@gmail.com> wrote:
>>
>>> Hi Harsha
>>>
>>> I've examined the SPIFFE provider more and have one question -
>>>
>>> If SPIFFE didn't have a need to do checkSpiffeId() call at the below
>>> location, would you really still write the Provider? *OR*
>>> Would you just use TrustManagerFactory.init(KeyStore) signature to pass
>>> the KeyStore from set of certs returned by spiffeIdManager.
>>> getTrustedCerts()?
>>>
>>>
>>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/CertificateUtils.java#L100
>>>
>>>
>>> /**
>>>> * Validates that the SPIFFE ID is present and matches the SPIFFE ID
>>>> configured in
>>>> * the java.security property ssl.spiffe.accept
>>>> *
>>>> * If the authorized spiffe ids list is empty any spiffe id is authorized
>>>> *
>>>> * @param chain an array of X509Certificate that contains the Peer's
>>>> SVID to be validated
>>>> * @throws CertificateException when either the certificates doesn't
>>>> have a SPIFFE ID or the SPIFFE ID is not authorized
>>>> */
>>>> static void checkSpiffeId(X509Certificate[] chain) throws
>>>> CertificateException {
>>>
>>>
>>>
>>> Thanks
>>> Maulin
>>>
>>>
>>> On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani <ka...@harsha.io>
>>> wrote:
>>>
>>>> Maulin,
>>>> The code parts you are pointing are specific for Spiffe
>>>> and if
>>>> you are talking about validate method which uses PKIX check like any
>>>> other provider does.
>>>> If you want to default to SunJSSE everywhere you can do so by delegating
>>>> the calls in these methods to SunJSSE provider.
>>>>
>>>> TrustManagerFactory tmf = TrustManagerFactory
>>>> .getInstance(TrustManagerFactory.getDefaultAlgorithm());and use
>>>> tmf.chekServerTrusted()
>>>> or use
>>>> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/TrustManagerFactory.html#getInstance(java.lang.String)if
>>>> you want a specific provider.
>>>>
>>>> -Harsha
>>>>
>>>>
>>>> On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada <
>>>> maulin.vasavada@gmail.com>
>>>> wrote:
>>>>
>>>> > Okay, so I take that you guys agree that I have to write a 'custom'
>>>> > algorithm and a provider to make it work , correct?
>>>> >
>>>> > Now, for Harsha's comment "Here the 'Custom' Algorithm is not an
>>>> > implementation per say , ..." , I diagree. You can refer to https://
>>>> >
>>>> github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/
>>>> > SpiffeTrustManager.java#L91 and
>>>> >
>>>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>>>> > provider/CertificateUtils.java#L100
>>>> >
>>>> > "that code" is the customization you have for the custom way to check
>>>> > something on top of regular checks. That method is NOT doing custom
>>>> > truststore loading. It is validating/verifying something in the
>>>> "custom"
>>>> > way with spiffeId.
>>>> > I bet that without that you won't have a need of the custom algorithm
>>>> in
>>>> > the first place.
>>>> >
>>>> > Let me know if you agree to this.
>>>> >
>>>> > Thanks
>>>> > Maulin
>>>> >
>>>> > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <mp...@gmail.com>
>>>> wrote:
>>>> >
>>>> > Hi Maulin, thanks for the discussion. As Harsha pointed out, to use
>>>> the
>>>> > KIP492, you need to create a new provider, register a *new* custom
>>>> > algorithm for your keymanager and trustmanager factory
>>>> implementations.
>>>> > After this, the kafka server configuration can be done as given below
>>>> >
>>>> > # Register the provider class with custom algorithm, say CUSTOM
>>>> security.
>>>> > provider.classes=com.company.security.CustomProvider
>>>> > <http://security.provider.classes
>>>> =com.company.security.customprovider/>
>>>> >
>>>> > # Register the keymanager and trustmanager algorithms
>>>> > # These algorithms indicate that the Keymanager and Trustmanagers
>>>> > registered under the algorithm "CUSTOM" needs to be used
>>>> > ssl.trustmanager.algorithm=CUSTOM
>>>> > ssl.keymanager.algorithm=CUSTOM
>>>> >
>>>> > And the customprovider looks like this...
>>>> >
>>>> > public class CustomProvider extends Provider {
>>>> > public CustomProvider() {
>>>> > super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and TrustStore");
>>>> > super.put("KeyManagerFactory.CUSTOM", "customKeyManagerFactory");
>>>> > super.put("TrustManagerFactory.CUSTOM",
>>>> > "customTrustManagerFactory");
>>>> > }
>>>> > }
>>>> >
>>>> > The PR for this is in review and can be found here:
>>>> https://github.com/
>>>> > apache/kafka/pull/7090
>>>> > This PR includes the fixed insertProviderAt function call.
>>>> >
>>>> > On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani <ka...@harsha.io>
>>>> > wrote:
>>>> >
>>>> > Answers are in-line
>>>> >
>>>> > On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada <
>>>> maulin.vasavada@gmail.
>>>> > com
>>>> >
>>>> > wrote:
>>>> >
>>>> > Hi Colin
>>>> >
>>>> > When I refer to "standard" or "custom" algorithms I am following Java
>>>> > security Provider Terminology. You can refer to
>>>> > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
>>>> > StandardNames.html#TrustManagerFactory link I provided earlier in the
>>>> > emails. It says PKIX is the default Algorithm for TrustManagerFactory.
>>>> >
>>>> > 1. For SPIFFE, I am not sure why you are saying 'it does not implement
>>>> > custom algorithms' because the following file clearly indicates that
>>>> it
>>>> > does use custom algorithm-
>>>> >
>>>> >
>>>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>>>> >
>>>> > provider/SpiffeProvider.java#L17
>>>> >
>>>> > Algorithm value:
>>>> >
>>>> >
>>>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>>>> >
>>>> > provider/SpiffeProviderConstants.java#L6
>>>> >
>>>> > @Harsha do you want to chime in since you use that provider?
>>>> >
>>>> > Here the "Custom" Algorithm is not an implementation per say , rather
>>>> >
>>>> > used
>>>> >
>>>> > to invoke the custom trust store factory and key manager factory. You
>>>> are
>>>> > not moving away from "standard" alogrithms that are available.
>>>> >
>>>> >
>>>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>>>> > provider/SpiffeTrustManager.java
>>>> >
>>>> > As you can see it delegates all the calls of verification of
>>>> certificate
>>>> >
>>>> > to
>>>> >
>>>> > the default implementation available.
>>>> > So in our implementation we still use PKIX to verify the certificate
>>>> > chain. So you are not losing anything here and Spiffe is not
>>>> reimplementing
>>>> > the verification process.
>>>> >
>>>> > 2. I already mentioned in my 3rd point, in my previous post, why using
>>>> >
>>>> > ssl.provider does NOT work. I updated KIP-486 in "rejected
>>>> >
>>>> > alternatives"
>>>> >
>>>> > also why ssl.provider does not work.
>>>> >
>>>> > As mentioned before , provider is the right way to go. I am still not
>>>> > understanding the gap is.
>>>> > If I understand correctly your argument is , provider is going to ask
>>>> to
>>>> > implement a custom algorithm.
>>>> > My answer to that is , no you are not re-implementing the algorithm.
>>>> >
>>>> > Please
>>>> >
>>>> > check the above link , TrustManager implementation it delegates the
>>>> calls
>>>> > back. There is no need to implement your own here.
>>>> >
>>>> > 3. Security.insertProviderAt() comments were based on assumption if
>>>> >
>>>> > KIP-492
>>>> >
>>>> > changes are done and we use that mechanism to configure providers
>>>> >
>>>> > instead
>>>> >
>>>> > of ssl.provider configuration.
>>>> >
>>>> > KIP-492 has patch available and going through review.
>>>> >
>>>> > Can you read my all the points, I mentioned in my previous post, very
>>>> >
>>>> > carefully? I am covering all the aspects in explaining. I am open to
>>>> >
>>>> > still
>>>> >
>>>> > discuss more to clarify any doubts.
>>>> >
>>>> > "3. If we just use existing ssl.provider kafka configuration then our
>>>> > provider will be used in SSLContext.getInstance(protocol, provider)
>>>> call
>>>> >
>>>> > in
>>>> >
>>>> > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/>
>>>> and
>>>> > if our provider does not have
>>>> > implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we
>>>> >
>>>> > tested
>>>> >
>>>> > it). Example: In MyProvider sample above you see that I didn't add
>>>> > SSLContext.TLSv1 as
>>>> > "Service+Algorithm" and that didn't work for me. In SPIFFE provider
>>>> you
>>>> > don't have this challenge since you are planning to bypass
>>>> ssl.provider
>>>> >
>>>> > as
>>>> >
>>>> > you mention in the KIP-492."
>>>> >
>>>> > Yes here you need to pass the protocol that your
>>>> KeyManager/TrustManager
>>>> > registered with and in no way its deviating from TLS RFC spec.
>>>> >
>>>> >
>>>> https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/
>>>> > javax/net/ssl/SSLContext.java#L134
>>>> >
>>>> > My suggestion here is for you to implement a simple Security Provider
>>>> as
>>>> > you did before and register a Algorithm. You can use the existing
>>>> > implementation in SpiffeProvider and plug in changes where you need to
>>>> > retrieve the certificates from by making RPC call.
>>>> >
>>>> > Run an end-to-end test with Kafka broker coming up with your provider
>>>> > making calls to RPC call. You do need to pass the "custom algorithm"
>>>> that
>>>> > you registered in your key manager to invoke the provider. I think
>>>> your
>>>> > concern is this approach is replacing the existing known ciphersuites
>>>> and
>>>> > certificate validation provided by java. Which its not.
>>>> >
>>>> > Now test the TLS connection you can do so via openssl -s_client
>>>> options
>>>> >
>>>> > to
>>>> >
>>>> > test if everything else is working.
>>>> >
>>>> > I am happy to share configs that we used if you are interested.
>>>> >
>>>> > Thanks,
>>>> > Harsha
>>>> >
>>>> > Thanks
>>>> > Maulin
>>>> >
>>>> > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <cm...@apache.org>
>>>> >
>>>> > wrote:
>>>> >
>>>> > Hi Maulin,
>>>> >
>>>> > A lot of JSSE providers don't implement custom algorithms. Spire is a
>>>> >
>>>> > good
>>>> >
>>>> > example of a JSSE provider that doesn't, and yet is still useful to
>>>> >
>>>> > many
>>>> >
>>>> > people. Your JSSE provider can work fine even if it doesn't implement
>>>> a
>>>> > custom algorithm.
>>>> >
>>>> > Maybe I'm missing something, but I don't understand the discussion of
>>>> > Security.insertProviderAt() that you included. SslEngineBuilder
>>>> doesn't
>>>> >
>>>> > use
>>>> >
>>>> > that API to get the security provider. Instead, it calls
>>>> > "SSLContext.getInstance(protocol, provider)", where provider is the
>>>> >
>>>> > name
>>>> >
>>>> > of the provider.
>>>> >
>>>> > best,
>>>> > Colin
>>>> >
>>>> > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
>>>> >
>>>> > On top of everything above I feel strongly to add the 4th point which
>>>> >
>>>> > is
>>>> >
>>>> > based on Java APIs for TrustManagerFactory.init(KeyStore) (
>>>> >
>>>> > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
>>>> > TrustManagerFactory.html#init(java.security.KeyStore
>>>> > <http://java.security.keystore/>)
>>>> > )
>>>> >
>>>> > and KeyManagerFactory.init(KeyStore, char[]) (
>>>> >
>>>> >
>>>> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory
>>>> .
>>>> >
>>>> >
>>>> > html#init(java.security.KeyStore <http://java.security.keystore/
>>>> >,%20char[])
>>>> >
>>>> >
>>>> > ).
>>>> >
>>>> > 4. The above APIs are intended to support providing "trust/key
>>>> >
>>>> > material"
>>>> >
>>>> > from the user without having to write their own
>>>> >
>>>> > TrustManager/KeyManagers.
>>>> >
>>>> > To quote from the TrustManagerFactory.init()'s documentation
>>>> >
>>>> > "Initializes
>>>> >
>>>> > this factory with a source of certificate authorities and related
>>>> trust
>>>> > material."
>>>> > To quote from the KeyManagerFactory.init()'s documentation
>>>> "Initializes
>>>> > this factory with a source of key material."
>>>> >
>>>> > Based on this it is clear that there is a flexibility provided by Java
>>>> >
>>>> > to
>>>> >
>>>> > to enable developers to provide the required trust/key material loaded
>>>> >
>>>> > from
>>>> >
>>>> > "anywhere" without requiring them to write custom provider OR
>>>> trust/key
>>>> > managers. This same flexibility is reflected in Kafka code also where
>>>> >
>>>> > it
>>>> >
>>>> > loads the trust/keys from a local file and doesn't require writing a
>>>> > Provider necessarily. If we do NOT have a custom algorithm, it makes
>>>> >
>>>> > less
>>>> >
>>>> > sense to write a Provider.
>>>> >
>>>> > Thanks
>>>> > Maulin
>>>> >
>>>> > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
>>>> >
>>>> > maulin.vasavada@gmail.com>
>>>> >
>>>> > wrote:
>>>> >
>>>> > Hi Harsha/Colin
>>>> >
>>>> > I did the sample with a custom Provider for TrustStoreManager and
>>>> tried
>>>> > using ssl.provider Kafka config AND the way KIP-492 is suggesting (by
>>>> > adding Provider programmatically instead of relying on
>>>> >
>>>> > ssl.provider+java.
>>>> >
>>>> > security. The below sample is followed by my detailed findings. I'll
>>>> > appreciate if you can go through it carefully and see
>>>> >
>>>> > if you
>>>> >
>>>> > see my point.
>>>> >
>>>> > package providertest;
>>>> >
>>>> > import java.security.Provider <http://java.security.provider/>
>>>> <http://
>>>> > java.security.provider/>;
>>>> >
>>>> > public class MyProvider extends Provider {
>>>> >
>>>> > private static final String name = "MyProvider"; private static double
>>>> > version = 1.0d;
>>>> > private static String info = "Maulin's SSL Provider v"+version;
>>>> >
>>>> > public MyProvider() {
>>>> > super(name, version, info);
>>>> > this.put("TrustManagerFactory.PKIX",
>>>> >
>>>> > "providertest.MyTrustManagerFactory");
>>>> >
>>>> > }
>>>> > }
>>>> >
>>>> > *Details:*
>>>> >
>>>> > KIP-492 documents that it will use Security.addProvider() assuming it
>>>> >
>>>> > will
>>>> >
>>>> > add it as position '0' which is not a correct assumption. The
>>>> > addProvider()'s documentation says it will add it to the last
>>>> available
>>>> > position. You may want to correct that to say
>>>> > Security.insertProviderAt(provider, 1).
>>>> >
>>>> > Now coming back to our specific discussion,
>>>> >
>>>> > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add
>>>> >
>>>> > that
>>>> >
>>>> > provider in the provider list via Security.addProvider() the position
>>>> >
>>>> > where
>>>> >
>>>> > it gets added doesn't matter (even if you don't end up adding it as
>>>> >
>>>> > first
>>>> >
>>>> > entry) since that is the ONLY provider for SPIFFE specific algorithm
>>>> >
>>>> > you
>>>> >
>>>> > might have.
>>>> >
>>>> > We do *not* have custom algorithm for Key/Trust StoreMangers. Which
>>>> >
>>>> > means
>>>> >
>>>> > we have to use X509, PKIX etc "Standard Algorithms" ((
>>>> >
>>>> > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
>>>> > StandardNames.html
>>>> > ))
>>>> >
>>>> > in our provider to override the TrustStoreManager (see my sample code)
>>>> >
>>>> > and
>>>> >
>>>> > KeyStoreManger and KeyManager. This creates another challenge
>>>> >
>>>> > mentioned in
>>>> >
>>>> > the below point.
>>>> >
>>>> > 2. In order to make our Provider for loading custom TrustStore work,
>>>> we
>>>> > have to add the provider as 'first' in the list since there are others
>>>> >
>>>> > with
>>>> >
>>>> > the same algorithm.
>>>> >
>>>> > However, the programatic way of adding provider
>>>> > (Security.insertProviderAt()) is *not* deterministic for ordering
>>>> since
>>>> > different code can call the method for a different provider and
>>>> >
>>>> > depending
>>>> >
>>>> > upon the order of the call our provider can be first or pushed down
>>>> the
>>>> > list. This can happen very well in any client application using Kafka.
>>>> >
>>>> > This
>>>> >
>>>> > is specially problematic for a case when you want to guarantee order
>>>> >
>>>> > for a
>>>> >
>>>> > Provider having "Standard Algorithms".
>>>> >
>>>> > If we add our provider in java.security file that definitely
>>>> guarantees
>>>> > the order(unless somebody calls removeProvider() which is less
>>>> >
>>>> > likely). But
>>>> >
>>>> > if we add our provider in java.security file it will defeat the
>>>> >
>>>> > purpose of
>>>> >
>>>> > the KIP-492.
>>>> >
>>>> > In the gist - Apache Kafka must not rely on "un-deterministic" method
>>>> >
>>>> > to
>>>> >
>>>> > rely on Provider ordering.
>>>> >
>>>> > 3. If we just use existing ssl.provider kafka configuration then our
>>>> > provider will be used in SSLContext.getInstance(protocol, provider)
>>>> >
>>>> > call in
>>>> >
>>>> > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/>
>>>> and
>>>> > if our provider does not have implementation for
>>>> > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks
>>>> >
>>>> > (we
>>>> >
>>>> > tested it). Example:
>>>> >
>>>> > In
>>>> >
>>>> > MyProvider sample above you see that I didn't add SSLContext.TLSv1 as
>>>> > "Service+Algorithm" and that didn't work for me. In SPIFFE provider
>>>> you
>>>> > don't have this challenge since you are planning to bypass
>>>> >
>>>> > ssl.provider as
>>>> >
>>>> > you mention in the KIP-492.
>>>> >
>>>> > *Overall summary,*
>>>> >
>>>> > 1. Any provider based mechanisms- a) existing ssl.provider and
>>>> >
>>>> > b)KIP-492,
>>>> >
>>>> > for loading key/trust store using "Standard Algorithms" do not work
>>>> >
>>>> > 2. Approach suggested in our KIP-486 works without any issue and it is
>>>> > *not* our context specific solve
>>>> >
>>>> > 3. Based on above we feel KIP-492 and KIP-486 are complimentary
>>>> changes
>>>> > and not contradicting or redundent.
>>>> >
>>>> > If you want we can do a joint session somehow to walk through the
>>>> >
>>>> > sample I
>>>> >
>>>> > have and various experiments I did. I would encourage you to do
>>>> similar
>>>> > exercise by writing a Provider for "Standard Algorithm" for
>>>> > TrustStoreManager (like our needs) and see what you find since only
>>>> >
>>>> > writing
>>>> >
>>>> > samples can bring out the complexity/challenges we face.
>>>> >
>>>> > Thanks
>>>> > Maulin
>>>> >
>>>> > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <
>>>> >
>>>> > maulin.vasavada@gmail.
>>>> >
>>>> > com> wrote:
>>>> >
>>>> > Just to update - still working on it. Get to work only on and off on
>>>> >
>>>> > it :(
>>>> >
>>>> > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
>>>> >
>>>> > maulin.vasavada@gmail.com>
>>>> >
>>>> > wrote:
>>>> >
>>>> > Hi Harsha
>>>> >
>>>> > Let me try to write samples and will let you know.
>>>> >
>>>> > Thanks
>>>> > Maulin
>>>> >
>>>> > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <ha...@gmail.com>
>>>> >
>>>> > wrote:
>>>> >
>>>> > Hi Maulin,
>>>> > With java security providers can be as custom you would
>>>> >
>>>> > like
>>>> >
>>>> > it to
>>>> > be. If you only want to to implement a custom way of loading the
>>>> >
>>>> > keystore
>>>> >
>>>> > and truststore and not implement any protocol/encryption handling you
>>>> can
>>>> > leave them empty and no need to implement. Have you looked into the
>>>> links I
>>>> > pasted before?
>>>> >
>>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>>>> >
>>>> >
>>>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
>>>> >
>>>> > java
>>>> >
>>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>>>> > spiffe-security-provider/src/main/java/spiffe/api/provider/
>>>> > SpiffeTrustManager.java <http://spiffetrustmanager.java/>
>>>> >
>>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>>>> >
>>>> >
>>>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>>>> >
>>>> > java
>>>> >
>>>> > Can you please tell me which methods are too complex in above to
>>>> >
>>>> > implement
>>>> >
>>>> > or unnecessary? You are changing anything in SSL/TLS implementations
>>>> > provided by
>>>> >
>>>> > All of the implementations delegating the checks to the default
>>>> > implementation anyway.
>>>> > Spire agent is an example, its nothing but a GRPC server listening
>>>> >
>>>> > on a
>>>> >
>>>> > unix domain socket . Above code is making a RPC call to the local
>>>> >
>>>> > daemon
>>>> >
>>>> > to
>>>> > get the certificate and keys. The mechanics are pretty much same as
>>>> >
>>>> > what
>>>> >
>>>> > you are asking for.
>>>> >
>>>> > Thanks,
>>>> > Harsha
>>>> >
>>>> > On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
>>>> >
>>>> > maulin.vasavada@gmail.com>
>>>> >
>>>> > wrote:
>>>> >
>>>> > Imagine a scenario like - We know we have a custom KMS and as a
>>>> >
>>>> > Kafka
>>>> >
>>>> > owner
>>>> >
>>>> > we want to comply to using that KMS source to load keys/certs. As
>>>> >
>>>> > a
>>>> >
>>>> > Kafka
>>>> >
>>>> > owner we know how to integrate with KMS but doesn't necessarily
>>>> >
>>>> > have
>>>> >
>>>> > to
>>>> >
>>>> > know anything about cipher suites, algorithms, and SSL/TLS
>>>> >
>>>> > implementation.
>>>> >
>>>> > Going the Provider way requires to know lot more than we should,
>>>> >
>>>> > isn't it?
>>>> >
>>>> > Not that we would have concern/shy-away knowing those details -
>>>> >
>>>> > but
>>>> >
>>>> > if we
>>>> >
>>>> > don't have to - why should we?
>>>> >
>>>> > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
>>>> >
>>>> > maulin.vasavada@gmail.com>
>>>> >
>>>> > wrote:
>>>> >
>>>> > Hi Harsha
>>>> >
>>>> > We don't have spire (or similar) agents and we do not have
>>>> >
>>>> > keys/certs
>>>> >
>>>> > locally on any brokers. To elaborate more on my previous email,
>>>> >
>>>> > I agree that Java security Providers are used in much broader
>>>> >
>>>> > sense
>>>> >
>>>> > - to
>>>> >
>>>> > have a particular implementation of an algorithm, use specific
>>>> >
>>>> > cipher
>>>> >
>>>> > suites for SSL , OR in our current team's case have a
>>>> >
>>>> > particular
>>>> >
>>>> > way to
>>>> >
>>>> > leverage pre-generated SSL sessions. However, the scope of our
>>>> >
>>>> > KIP
>>>> >
>>>> > (486)
>>>> >
>>>> > is
>>>> >
>>>> > much restricted than that. We merely intend to provide a custom
>>>> > keystore/truststore for our SSL connections and not really worry
>>>> >
>>>> > about
>>>> >
>>>> > underlying specific SSL/TLS implementation. This simplifies it
>>>> >
>>>> > a
>>>> >
>>>> > lot for
>>>> >
>>>> > us to keep the concerns separate and clear.
>>>> >
>>>> > I feel our approach is more complimentary such that it allows
>>>> >
>>>> > for
>>>> >
>>>> > using
>>>> >
>>>> > keystores of choice while retaining the flexibility to use any
>>>> > underlying/available Provider for actually making the SSL call.
>>>> >
>>>> > We agree with KIP-492's approach based on Providers (and Java's
>>>> > recommendation), but also strongly believe that our approach can
>>>> >
>>>> > compliment
>>>> >
>>>> > it very effectively for reasons explained above.
>>>> >
>>>> > Thanks
>>>> > Maulin
>>>> >
>>>> > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <
>>>> >
>>>> > kafka@harsha.io
>>>> >
>>>> > wrote:
>>>> >
>>>> > Hi Maulin,
>>>> >
>>>> > On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
>>>> >
>>>> > maulin.vasavada@gmail.
>>>> >
>>>> > com>
>>>> > wrote:
>>>> >
>>>> > Hi Harsha
>>>> >
>>>> > The reason we rejected the SslProvider route is that - we
>>>> >
>>>> > only
>>>> >
>>>> > needed
>>>> >
>>>> > a
>>>> >
>>>> > custom way to load keys/certs. Not touch any policy that
>>>> >
>>>> > existing
>>>> >
>>>> > Providers
>>>> >
>>>> > govern like SunJSSE Provider.
>>>> >
>>>> > We have exactly the same requirements to load certs and keys
>>>> >
>>>> > through
>>>> >
>>>> > spire
>>>> >
>>>> > agent. We used security.provider to do that exactly. I am not
>>>> >
>>>> > sure
>>>> >
>>>> > why
>>>> >
>>>> > you
>>>> >
>>>> > would be modifying any policies provided by default SunJSSE
>>>> >
>>>> > provider.
>>>> >
>>>> > Can
>>>> >
>>>> > you give me an example of having custom provider that will
>>>> >
>>>> > override an
>>>> >
>>>> > existing policy in SunJSSE provider.
>>>> >
>>>> > As pointed out earlier, this kip
>>>> >
>>>> > https://cwiki.apache.org/confluence/display/KAFKA/
>>>> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>>>> >
>>>> > allows
>>>> > you to load security.provider through config.
>>>> > Take a look at the examples I gave before
>>>> >
>>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>>>> >
>>>> >
>>>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>>>> >
>>>> > java
>>>> >
>>>> > It registers KeyManagerFactory and TrustManagerFactory and
>>>> >
>>>> > Keystore
>>>> >
>>>> > algorithm.
>>>> > Implement your custom way of loading Keystore in here
>>>> >
>>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>>>> >
>>>> >
>>>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
>>>> >
>>>> > java
>>>> >
>>>> > and Trust manager like here
>>>> >
>>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>>>> > spiffe-security-provider/src/main/java/spiffe/api/provider/
>>>> > SpiffeTrustManager.java <http://spiffetrustmanager.java/>
>>>> >
>>>> > In your Kafka client you can set the security.provider to your
>>>> >
>>>> > custom
>>>> >
>>>> > implementation and with this fix
>>>> > https://issues.apache.org/jira/browse/KAFKA-8191 you can set
>>>> > keyManagerAlgorigthm and trustManagerAlgorithm configs.
>>>> >
>>>> > All of this is in your clients and broker side and do not need
>>>> >
>>>> > to
>>>> >
>>>> > touch
>>>> >
>>>> > any
>>>> > policy changes at JVM level. You'll register the providers in
>>>> >
>>>> > the
>>>> >
>>>> > priority
>>>> >
>>>> > order and can still have SunJSSE provider and have your custom
>>>> >
>>>> > provider
>>>> >
>>>> > to
>>>> >
>>>> > implement the key and trust managers.
>>>> >
>>>> > The ask here is different than KIP-492. We don't have any need
>>>> >
>>>> > to
>>>> >
>>>> > modify/specify the algorithm parameter. Does that make sense?
>>>> >
>>>> > The ask in KIP is introducing new interfaces where the KIP's
>>>> > goal/motivation can be achieved through the security.provider
>>>> >
>>>> > and
>>>> >
>>>> > we
>>>> >
>>>> > worked
>>>> > on similar goal without touching any Keystore or Truststore
>>>> >
>>>> > interfaces.
>>>> >
>>>> > My advise is against changing or introducing new interfaces
>>>> >
>>>> > when
>>>> >
>>>> > it can
>>>> >
>>>> > work through security.provider.
>>>> >
>>>> > Thanks,
>>>> > Harsha
>>>> >
>>>> > Thanks
>>>> >
>>>> > Maulin
>>>> >
>>>> > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
>>>> >
>>>> > kafka@harsha.io>
>>>> >
>>>> > wrote:
>>>> >
>>>> > In your KIP you added security. provider as rejected
>>>> >
>>>> > alternative
>>>> >
>>>> > and
>>>> >
>>>> > specified "its not the correct way". Do you mind explaining
>>>> >
>>>> > why
>>>> >
>>>> > its
>>>> >
>>>> > not? I
>>>> >
>>>> > didn't find any evidence in Java docs to say so. Contrary to
>>>> >
>>>> > your
>>>> >
>>>> > statement
>>>> >
>>>> > it does say in the java docs
>>>> > " However, please note that a provider can be used to
>>>> >
>>>> > implement
>>>> >
>>>> > any
>>>> >
>>>> > security service in Java that uses a pluggable architecture
>>>> >
>>>> > with
>>>> >
>>>> > a
>>>> >
>>>> > choice
>>>> >
>>>> > of implementations that fit underneath."
>>>> >
>>>> > Java Security Providers have been used by other projects to
>>>> >
>>>> > provide
>>>> >
>>>> > such
>>>> >
>>>> > integration . I am not sure if you looked into Spiffe
>>>> >
>>>> > project to
>>>> >
>>>> > efficiently distribute certificates but here is an example of
>>>> >
>>>> > Java
>>>> >
>>>> > provider
>>>> >
>>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>>>> >
>>>> >
>>>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>>>> >
>>>> > java which
>>>> > obtains certificates from local daemons.
>>>> > These integrations are being used in Tomcat, Jetty etc.. We
>>>> >
>>>> > are
>>>> >
>>>> > also
>>>> >
>>>> > using
>>>> >
>>>> > Security provider to do the same in our Kafka clusters. So
>>>> >
>>>> > unless I
>>>> >
>>>> > see
>>>> >
>>>> > more evidence why security.provider doesn't work for you
>>>> >
>>>> > adding
>>>> >
>>>> > new
>>>> >
>>>> > interfaces while there exists more cleaner way of achieving
>>>> >
>>>> > the
>>>> >
>>>> > goals
>>>> >
>>>> > of
>>>> >
>>>> > this KIP is unnecessary and breaks the well known security
>>>> >
>>>> > interfaces
>>>> >
>>>> > provided by Java itself.
>>>> >
>>>> > Thanks,
>>>> > Harsha
>>>> >
>>>> > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
>>>> >
>>>> > kafka@harsha.io
>>>> >
>>>> > wrote:
>>>> >
>>>> > Hi Maulin,
>>>> > Not sure if you looked at my previous replies. This
>>>> >
>>>> > changes
>>>> >
>>>> > are not required as there is already security Provider to do
>>>> >
>>>> > what you
>>>> >
>>>> > are
>>>> >
>>>> > proposing. This KIP
>>>> >
>>>> > https://cwiki.apache.org/confluence/display/KAFKA/
>>>> >
>>>> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>>>> >
>>>> > also
>>>> >
>>>> > addresses easy registration of such providers.
>>>> >
>>>> > Thanks,
>>>> > Harsha
>>>> >
>>>> > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
>>>> >
>>>> > <maulin.vasavada@gmail.
>>>> >
>>>> > com> wrote:
>>>> >
>>>> > Bump! Can somebody please review this?
>>>> >
>>>> > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
>>>> >
>>>> > maulin.vasavada@gmail.com>
>>>> >
>>>> > wrote:
>>>> >
>>>> > Bump! Can somebody please review this?
>>>> >
>>>> > --
>>>> > Thanks,
>>>> > M.Sai Sandeep
>>>> >
>>>> >
>>>>
>>>
On Wed, Aug 28, 2019 at 2:18 PM Maulin Vasavada <ma...@gmail.com>
wrote:
>
>
> ---------- Forwarded message ---------
> From: Maulin Vasavada <ma...@gmail.com>
> Date: Wed, Aug 28, 2019 at 1:51 PM
> Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and
> TrustStore
> To: <de...@kafka.apache.org>
>
>
> Hi Harsha
>
> As we synced-up offline on this topic, we hope you don't have any more
> clarifications that you are seeking. If that is the case, can you please
> help us move this forward and discuss what changes you would expect on the
> KIP design in order to make it valuable contribution?
>
> Just FYI - we verified our primary design change with the author of Sun's
> X509 Trustmanager implementation and the outcome is that what we are
> proposing makes sense at the heart of it - "Instead of writing TrustManager
> just plugin the Trust store". We are open to discuss additional changes
> that you/anybody else would like to see on the functionality however.
>
>
> Thanks
> Maulin
>
> On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada <ma...@gmail.com>
> wrote:
>
>> Hi Harsha
>>
>> Any response on my question? I feel this KIP is worth accommodating. Your
>> help is much appreciated.
>>
>> Thanks
>> Maulin
>>
>> On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada <
>> maulin.vasavada@gmail.com> wrote:
>>
>>> Hi Harsha
>>>
>>> I've examined the SPIFFE provider more and have one question -
>>>
>>> If SPIFFE didn't have a need to do checkSpiffeId() call at the below
>>> location, would you really still write the Provider? *OR*
>>> Would you just use TrustManagerFactory.init(KeyStore) signature to pass
>>> the KeyStore from set of certs returned by spiffeIdManager.
>>> getTrustedCerts()?
>>>
>>>
>>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/CertificateUtils.java#L100
>>>
>>>
>>> /**
>>>> * Validates that the SPIFFE ID is present and matches the SPIFFE ID
>>>> configured in
>>>> * the java.security property ssl.spiffe.accept
>>>> *
>>>> * If the authorized spiffe ids list is empty any spiffe id is authorized
>>>> *
>>>> * @param chain an array of X509Certificate that contains the Peer's
>>>> SVID to be validated
>>>> * @throws CertificateException when either the certificates doesn't
>>>> have a SPIFFE ID or the SPIFFE ID is not authorized
>>>> */
>>>> static void checkSpiffeId(X509Certificate[] chain) throws
>>>> CertificateException {
>>>
>>>
>>>
>>> Thanks
>>> Maulin
>>>
>>>
>>> On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani <ka...@harsha.io>
>>> wrote:
>>>
>>>> Maulin,
>>>> The code parts you are pointing are specific for Spiffe
>>>> and if
>>>> you are talking about validate method which uses PKIX check like any
>>>> other provider does.
>>>> If you want to default to SunJSSE everywhere you can do so by delegating
>>>> the calls in these methods to SunJSSE provider.
>>>>
>>>> TrustManagerFactory tmf = TrustManagerFactory
>>>> .getInstance(TrustManagerFactory.getDefaultAlgorithm());and use
>>>> tmf.chekServerTrusted()
>>>> or use
>>>> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/TrustManagerFactory.html#getInstance(java.lang.String)if
>>>> you want a specific provider.
>>>>
>>>> -Harsha
>>>>
>>>>
>>>> On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada <
>>>> maulin.vasavada@gmail.com>
>>>> wrote:
>>>>
>>>> > Okay, so I take that you guys agree that I have to write a 'custom'
>>>> > algorithm and a provider to make it work , correct?
>>>> >
>>>> > Now, for Harsha's comment "Here the 'Custom' Algorithm is not an
>>>> > implementation per say , ..." , I diagree. You can refer to https://
>>>> >
>>>> github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/
>>>> > SpiffeTrustManager.java#L91 and
>>>> >
>>>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>>>> > provider/CertificateUtils.java#L100
>>>> >
>>>> > "that code" is the customization you have for the custom way to check
>>>> > something on top of regular checks. That method is NOT doing custom
>>>> > truststore loading. It is validating/verifying something in the
>>>> "custom"
>>>> > way with spiffeId.
>>>> > I bet that without that you won't have a need of the custom algorithm
>>>> in
>>>> > the first place.
>>>> >
>>>> > Let me know if you agree to this.
>>>> >
>>>> > Thanks
>>>> > Maulin
>>>> >
>>>> > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <mp...@gmail.com>
>>>> wrote:
>>>> >
>>>> > Hi Maulin, thanks for the discussion. As Harsha pointed out, to use
>>>> the
>>>> > KIP492, you need to create a new provider, register a *new* custom
>>>> > algorithm for your keymanager and trustmanager factory
>>>> implementations.
>>>> > After this, the kafka server configuration can be done as given below
>>>> >
>>>> > # Register the provider class with custom algorithm, say CUSTOM
>>>> security.
>>>> > provider.classes=com.company.security.CustomProvider
>>>> > <http://security.provider.classes
>>>> =com.company.security.customprovider/>
>>>> >
>>>> > # Register the keymanager and trustmanager algorithms
>>>> > # These algorithms indicate that the Keymanager and Trustmanagers
>>>> > registered under the algorithm "CUSTOM" needs to be used
>>>> > ssl.trustmanager.algorithm=CUSTOM
>>>> > ssl.keymanager.algorithm=CUSTOM
>>>> >
>>>> > And the customprovider looks like this...
>>>> >
>>>> > public class CustomProvider extends Provider {
>>>> > public CustomProvider() {
>>>> > super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and TrustStore");
>>>> > super.put("KeyManagerFactory.CUSTOM", "customKeyManagerFactory");
>>>> > super.put("TrustManagerFactory.CUSTOM",
>>>> > "customTrustManagerFactory");
>>>> > }
>>>> > }
>>>> >
>>>> > The PR for this is in review and can be found here:
>>>> https://github.com/
>>>> > apache/kafka/pull/7090
>>>> > This PR includes the fixed insertProviderAt function call.
>>>> >
>>>> > On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani <ka...@harsha.io>
>>>> > wrote:
>>>> >
>>>> > Answers are in-line
>>>> >
>>>> > On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada <
>>>> maulin.vasavada@gmail.
>>>> > com
>>>> >
>>>> > wrote:
>>>> >
>>>> > Hi Colin
>>>> >
>>>> > When I refer to "standard" or "custom" algorithms I am following Java
>>>> > security Provider Terminology. You can refer to
>>>> > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
>>>> > StandardNames.html#TrustManagerFactory link I provided earlier in the
>>>> > emails. It says PKIX is the default Algorithm for TrustManagerFactory.
>>>> >
>>>> > 1. For SPIFFE, I am not sure why you are saying 'it does not implement
>>>> > custom algorithms' because the following file clearly indicates that
>>>> it
>>>> > does use custom algorithm-
>>>> >
>>>> >
>>>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>>>> >
>>>> > provider/SpiffeProvider.java#L17
>>>> >
>>>> > Algorithm value:
>>>> >
>>>> >
>>>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>>>> >
>>>> > provider/SpiffeProviderConstants.java#L6
>>>> >
>>>> > @Harsha do you want to chime in since you use that provider?
>>>> >
>>>> > Here the "Custom" Algorithm is not an implementation per say , rather
>>>> >
>>>> > used
>>>> >
>>>> > to invoke the custom trust store factory and key manager factory. You
>>>> are
>>>> > not moving away from "standard" alogrithms that are available.
>>>> >
>>>> >
>>>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>>>> > provider/SpiffeTrustManager.java
>>>> >
>>>> > As you can see it delegates all the calls of verification of
>>>> certificate
>>>> >
>>>> > to
>>>> >
>>>> > the default implementation available.
>>>> > So in our implementation we still use PKIX to verify the certificate
>>>> > chain. So you are not losing anything here and Spiffe is not
>>>> reimplementing
>>>> > the verification process.
>>>> >
>>>> > 2. I already mentioned in my 3rd point, in my previous post, why using
>>>> >
>>>> > ssl.provider does NOT work. I updated KIP-486 in "rejected
>>>> >
>>>> > alternatives"
>>>> >
>>>> > also why ssl.provider does not work.
>>>> >
>>>> > As mentioned before , provider is the right way to go. I am still not
>>>> > understanding the gap is.
>>>> > If I understand correctly your argument is , provider is going to ask
>>>> to
>>>> > implement a custom algorithm.
>>>> > My answer to that is , no you are not re-implementing the algorithm.
>>>> >
>>>> > Please
>>>> >
>>>> > check the above link , TrustManager implementation it delegates the
>>>> calls
>>>> > back. There is no need to implement your own here.
>>>> >
>>>> > 3. Security.insertProviderAt() comments were based on assumption if
>>>> >
>>>> > KIP-492
>>>> >
>>>> > changes are done and we use that mechanism to configure providers
>>>> >
>>>> > instead
>>>> >
>>>> > of ssl.provider configuration.
>>>> >
>>>> > KIP-492 has patch available and going through review.
>>>> >
>>>> > Can you read my all the points, I mentioned in my previous post, very
>>>> >
>>>> > carefully? I am covering all the aspects in explaining. I am open to
>>>> >
>>>> > still
>>>> >
>>>> > discuss more to clarify any doubts.
>>>> >
>>>> > "3. If we just use existing ssl.provider kafka configuration then our
>>>> > provider will be used in SSLContext.getInstance(protocol, provider)
>>>> call
>>>> >
>>>> > in
>>>> >
>>>> > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/>
>>>> and
>>>> > if our provider does not have
>>>> > implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we
>>>> >
>>>> > tested
>>>> >
>>>> > it). Example: In MyProvider sample above you see that I didn't add
>>>> > SSLContext.TLSv1 as
>>>> > "Service+Algorithm" and that didn't work for me. In SPIFFE provider
>>>> you
>>>> > don't have this challenge since you are planning to bypass
>>>> ssl.provider
>>>> >
>>>> > as
>>>> >
>>>> > you mention in the KIP-492."
>>>> >
>>>> > Yes here you need to pass the protocol that your
>>>> KeyManager/TrustManager
>>>> > registered with and in no way its deviating from TLS RFC spec.
>>>> >
>>>> >
>>>> https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/
>>>> > javax/net/ssl/SSLContext.java#L134
>>>> >
>>>> > My suggestion here is for you to implement a simple Security Provider
>>>> as
>>>> > you did before and register a Algorithm. You can use the existing
>>>> > implementation in SpiffeProvider and plug in changes where you need to
>>>> > retrieve the certificates from by making RPC call.
>>>> >
>>>> > Run an end-to-end test with Kafka broker coming up with your provider
>>>> > making calls to RPC call. You do need to pass the "custom algorithm"
>>>> that
>>>> > you registered in your key manager to invoke the provider. I think
>>>> your
>>>> > concern is this approach is replacing the existing known ciphersuites
>>>> and
>>>> > certificate validation provided by java. Which its not.
>>>> >
>>>> > Now test the TLS connection you can do so via openssl -s_client
>>>> options
>>>> >
>>>> > to
>>>> >
>>>> > test if everything else is working.
>>>> >
>>>> > I am happy to share configs that we used if you are interested.
>>>> >
>>>> > Thanks,
>>>> > Harsha
>>>> >
>>>> > Thanks
>>>> > Maulin
>>>> >
>>>> > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <cm...@apache.org>
>>>> >
>>>> > wrote:
>>>> >
>>>> > Hi Maulin,
>>>> >
>>>> > A lot of JSSE providers don't implement custom algorithms. Spire is a
>>>> >
>>>> > good
>>>> >
>>>> > example of a JSSE provider that doesn't, and yet is still useful to
>>>> >
>>>> > many
>>>> >
>>>> > people. Your JSSE provider can work fine even if it doesn't implement
>>>> a
>>>> > custom algorithm.
>>>> >
>>>> > Maybe I'm missing something, but I don't understand the discussion of
>>>> > Security.insertProviderAt() that you included. SslEngineBuilder
>>>> doesn't
>>>> >
>>>> > use
>>>> >
>>>> > that API to get the security provider. Instead, it calls
>>>> > "SSLContext.getInstance(protocol, provider)", where provider is the
>>>> >
>>>> > name
>>>> >
>>>> > of the provider.
>>>> >
>>>> > best,
>>>> > Colin
>>>> >
>>>> > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
>>>> >
>>>> > On top of everything above I feel strongly to add the 4th point which
>>>> >
>>>> > is
>>>> >
>>>> > based on Java APIs for TrustManagerFactory.init(KeyStore) (
>>>> >
>>>> > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
>>>> > TrustManagerFactory.html#init(java.security.KeyStore
>>>> > <http://java.security.keystore/>)
>>>> > )
>>>> >
>>>> > and KeyManagerFactory.init(KeyStore, char[]) (
>>>> >
>>>> >
>>>> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory
>>>> .
>>>> >
>>>> >
>>>> > html#init(java.security.KeyStore <http://java.security.keystore/
>>>> >,%20char[])
>>>> >
>>>> >
>>>> > ).
>>>> >
>>>> > 4. The above APIs are intended to support providing "trust/key
>>>> >
>>>> > material"
>>>> >
>>>> > from the user without having to write their own
>>>> >
>>>> > TrustManager/KeyManagers.
>>>> >
>>>> > To quote from the TrustManagerFactory.init()'s documentation
>>>> >
>>>> > "Initializes
>>>> >
>>>> > this factory with a source of certificate authorities and related
>>>> trust
>>>> > material."
>>>> > To quote from the KeyManagerFactory.init()'s documentation
>>>> "Initializes
>>>> > this factory with a source of key material."
>>>> >
>>>> > Based on this it is clear that there is a flexibility provided by Java
>>>> >
>>>> > to
>>>> >
>>>> > to enable developers to provide the required trust/key material loaded
>>>> >
>>>> > from
>>>> >
>>>> > "anywhere" without requiring them to write custom provider OR
>>>> trust/key
>>>> > managers. This same flexibility is reflected in Kafka code also where
>>>> >
>>>> > it
>>>> >
>>>> > loads the trust/keys from a local file and doesn't require writing a
>>>> > Provider necessarily. If we do NOT have a custom algorithm, it makes
>>>> >
>>>> > less
>>>> >
>>>> > sense to write a Provider.
>>>> >
>>>> > Thanks
>>>> > Maulin
>>>> >
>>>> > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
>>>> >
>>>> > maulin.vasavada@gmail.com>
>>>> >
>>>> > wrote:
>>>> >
>>>> > Hi Harsha/Colin
>>>> >
>>>> > I did the sample with a custom Provider for TrustStoreManager and
>>>> tried
>>>> > using ssl.provider Kafka config AND the way KIP-492 is suggesting (by
>>>> > adding Provider programmatically instead of relying on
>>>> >
>>>> > ssl.provider+java.
>>>> >
>>>> > security. The below sample is followed by my detailed findings. I'll
>>>> > appreciate if you can go through it carefully and see
>>>> >
>>>> > if you
>>>> >
>>>> > see my point.
>>>> >
>>>> > package providertest;
>>>> >
>>>> > import java.security.Provider <http://java.security.provider/>
>>>> <http://
>>>> > java.security.provider/>;
>>>> >
>>>> > public class MyProvider extends Provider {
>>>> >
>>>> > private static final String name = "MyProvider"; private static double
>>>> > version = 1.0d;
>>>> > private static String info = "Maulin's SSL Provider v"+version;
>>>> >
>>>> > public MyProvider() {
>>>> > super(name, version, info);
>>>> > this.put("TrustManagerFactory.PKIX",
>>>> >
>>>> > "providertest.MyTrustManagerFactory");
>>>> >
>>>> > }
>>>> > }
>>>> >
>>>> > *Details:*
>>>> >
>>>> > KIP-492 documents that it will use Security.addProvider() assuming it
>>>> >
>>>> > will
>>>> >
>>>> > add it as position '0' which is not a correct assumption. The
>>>> > addProvider()'s documentation says it will add it to the last
>>>> available
>>>> > position. You may want to correct that to say
>>>> > Security.insertProviderAt(provider, 1).
>>>> >
>>>> > Now coming back to our specific discussion,
>>>> >
>>>> > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add
>>>> >
>>>> > that
>>>> >
>>>> > provider in the provider list via Security.addProvider() the position
>>>> >
>>>> > where
>>>> >
>>>> > it gets added doesn't matter (even if you don't end up adding it as
>>>> >
>>>> > first
>>>> >
>>>> > entry) since that is the ONLY provider for SPIFFE specific algorithm
>>>> >
>>>> > you
>>>> >
>>>> > might have.
>>>> >
>>>> > We do *not* have custom algorithm for Key/Trust StoreMangers. Which
>>>> >
>>>> > means
>>>> >
>>>> > we have to use X509, PKIX etc "Standard Algorithms" ((
>>>> >
>>>> > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
>>>> > StandardNames.html
>>>> > ))
>>>> >
>>>> > in our provider to override the TrustStoreManager (see my sample code)
>>>> >
>>>> > and
>>>> >
>>>> > KeyStoreManger and KeyManager. This creates another challenge
>>>> >
>>>> > mentioned in
>>>> >
>>>> > the below point.
>>>> >
>>>> > 2. In order to make our Provider for loading custom TrustStore work,
>>>> we
>>>> > have to add the provider as 'first' in the list since there are others
>>>> >
>>>> > with
>>>> >
>>>> > the same algorithm.
>>>> >
>>>> > However, the programatic way of adding provider
>>>> > (Security.insertProviderAt()) is *not* deterministic for ordering
>>>> since
>>>> > different code can call the method for a different provider and
>>>> >
>>>> > depending
>>>> >
>>>> > upon the order of the call our provider can be first or pushed down
>>>> the
>>>> > list. This can happen very well in any client application using Kafka.
>>>> >
>>>> > This
>>>> >
>>>> > is specially problematic for a case when you want to guarantee order
>>>> >
>>>> > for a
>>>> >
>>>> > Provider having "Standard Algorithms".
>>>> >
>>>> > If we add our provider in java.security file that definitely
>>>> guarantees
>>>> > the order(unless somebody calls removeProvider() which is less
>>>> >
>>>> > likely). But
>>>> >
>>>> > if we add our provider in java.security file it will defeat the
>>>> >
>>>> > purpose of
>>>> >
>>>> > the KIP-492.
>>>> >
>>>> > In the gist - Apache Kafka must not rely on "un-deterministic" method
>>>> >
>>>> > to
>>>> >
>>>> > rely on Provider ordering.
>>>> >
>>>> > 3. If we just use existing ssl.provider kafka configuration then our
>>>> > provider will be used in SSLContext.getInstance(protocol, provider)
>>>> >
>>>> > call in
>>>> >
>>>> > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/>
>>>> and
>>>> > if our provider does not have implementation for
>>>> > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks
>>>> >
>>>> > (we
>>>> >
>>>> > tested it). Example:
>>>> >
>>>> > In
>>>> >
>>>> > MyProvider sample above you see that I didn't add SSLContext.TLSv1 as
>>>> > "Service+Algorithm" and that didn't work for me. In SPIFFE provider
>>>> you
>>>> > don't have this challenge since you are planning to bypass
>>>> >
>>>> > ssl.provider as
>>>> >
>>>> > you mention in the KIP-492.
>>>> >
>>>> > *Overall summary,*
>>>> >
>>>> > 1. Any provider based mechanisms- a) existing ssl.provider and
>>>> >
>>>> > b)KIP-492,
>>>> >
>>>> > for loading key/trust store using "Standard Algorithms" do not work
>>>> >
>>>> > 2. Approach suggested in our KIP-486 works without any issue and it is
>>>> > *not* our context specific solve
>>>> >
>>>> > 3. Based on above we feel KIP-492 and KIP-486 are complimentary
>>>> changes
>>>> > and not contradicting or redundent.
>>>> >
>>>> > If you want we can do a joint session somehow to walk through the
>>>> >
>>>> > sample I
>>>> >
>>>> > have and various experiments I did. I would encourage you to do
>>>> similar
>>>> > exercise by writing a Provider for "Standard Algorithm" for
>>>> > TrustStoreManager (like our needs) and see what you find since only
>>>> >
>>>> > writing
>>>> >
>>>> > samples can bring out the complexity/challenges we face.
>>>> >
>>>> > Thanks
>>>> > Maulin
>>>> >
>>>> > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <
>>>> >
>>>> > maulin.vasavada@gmail.
>>>> >
>>>> > com> wrote:
>>>> >
>>>> > Just to update - still working on it. Get to work only on and off on
>>>> >
>>>> > it :(
>>>> >
>>>> > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
>>>> >
>>>> > maulin.vasavada@gmail.com>
>>>> >
>>>> > wrote:
>>>> >
>>>> > Hi Harsha
>>>> >
>>>> > Let me try to write samples and will let you know.
>>>> >
>>>> > Thanks
>>>> > Maulin
>>>> >
>>>> > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <ha...@gmail.com>
>>>> >
>>>> > wrote:
>>>> >
>>>> > Hi Maulin,
>>>> > With java security providers can be as custom you would
>>>> >
>>>> > like
>>>> >
>>>> > it to
>>>> > be. If you only want to to implement a custom way of loading the
>>>> >
>>>> > keystore
>>>> >
>>>> > and truststore and not implement any protocol/encryption handling you
>>>> can
>>>> > leave them empty and no need to implement. Have you looked into the
>>>> links I
>>>> > pasted before?
>>>> >
>>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>>>> >
>>>> >
>>>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
>>>> >
>>>> > java
>>>> >
>>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>>>> > spiffe-security-provider/src/main/java/spiffe/api/provider/
>>>> > SpiffeTrustManager.java <http://spiffetrustmanager.java/>
>>>> >
>>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>>>> >
>>>> >
>>>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>>>> >
>>>> > java
>>>> >
>>>> > Can you please tell me which methods are too complex in above to
>>>> >
>>>> > implement
>>>> >
>>>> > or unnecessary? You are changing anything in SSL/TLS implementations
>>>> > provided by
>>>> >
>>>> > All of the implementations delegating the checks to the default
>>>> > implementation anyway.
>>>> > Spire agent is an example, its nothing but a GRPC server listening
>>>> >
>>>> > on a
>>>> >
>>>> > unix domain socket . Above code is making a RPC call to the local
>>>> >
>>>> > daemon
>>>> >
>>>> > to
>>>> > get the certificate and keys. The mechanics are pretty much same as
>>>> >
>>>> > what
>>>> >
>>>> > you are asking for.
>>>> >
>>>> > Thanks,
>>>> > Harsha
>>>> >
>>>> > On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
>>>> >
>>>> > maulin.vasavada@gmail.com>
>>>> >
>>>> > wrote:
>>>> >
>>>> > Imagine a scenario like - We know we have a custom KMS and as a
>>>> >
>>>> > Kafka
>>>> >
>>>> > owner
>>>> >
>>>> > we want to comply to using that KMS source to load keys/certs. As
>>>> >
>>>> > a
>>>> >
>>>> > Kafka
>>>> >
>>>> > owner we know how to integrate with KMS but doesn't necessarily
>>>> >
>>>> > have
>>>> >
>>>> > to
>>>> >
>>>> > know anything about cipher suites, algorithms, and SSL/TLS
>>>> >
>>>> > implementation.
>>>> >
>>>> > Going the Provider way requires to know lot more than we should,
>>>> >
>>>> > isn't it?
>>>> >
>>>> > Not that we would have concern/shy-away knowing those details -
>>>> >
>>>> > but
>>>> >
>>>> > if we
>>>> >
>>>> > don't have to - why should we?
>>>> >
>>>> > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
>>>> >
>>>> > maulin.vasavada@gmail.com>
>>>> >
>>>> > wrote:
>>>> >
>>>> > Hi Harsha
>>>> >
>>>> > We don't have spire (or similar) agents and we do not have
>>>> >
>>>> > keys/certs
>>>> >
>>>> > locally on any brokers. To elaborate more on my previous email,
>>>> >
>>>> > I agree that Java security Providers are used in much broader
>>>> >
>>>> > sense
>>>> >
>>>> > - to
>>>> >
>>>> > have a particular implementation of an algorithm, use specific
>>>> >
>>>> > cipher
>>>> >
>>>> > suites for SSL , OR in our current team's case have a
>>>> >
>>>> > particular
>>>> >
>>>> > way to
>>>> >
>>>> > leverage pre-generated SSL sessions. However, the scope of our
>>>> >
>>>> > KIP
>>>> >
>>>> > (486)
>>>> >
>>>> > is
>>>> >
>>>> > much restricted than that. We merely intend to provide a custom
>>>> > keystore/truststore for our SSL connections and not really worry
>>>> >
>>>> > about
>>>> >
>>>> > underlying specific SSL/TLS implementation. This simplifies it
>>>> >
>>>> > a
>>>> >
>>>> > lot for
>>>> >
>>>> > us to keep the concerns separate and clear.
>>>> >
>>>> > I feel our approach is more complimentary such that it allows
>>>> >
>>>> > for
>>>> >
>>>> > using
>>>> >
>>>> > keystores of choice while retaining the flexibility to use any
>>>> > underlying/available Provider for actually making the SSL call.
>>>> >
>>>> > We agree with KIP-492's approach based on Providers (and Java's
>>>> > recommendation), but also strongly believe that our approach can
>>>> >
>>>> > compliment
>>>> >
>>>> > it very effectively for reasons explained above.
>>>> >
>>>> > Thanks
>>>> > Maulin
>>>> >
>>>> > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <
>>>> >
>>>> > kafka@harsha.io
>>>> >
>>>> > wrote:
>>>> >
>>>> > Hi Maulin,
>>>> >
>>>> > On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
>>>> >
>>>> > maulin.vasavada@gmail.
>>>> >
>>>> > com>
>>>> > wrote:
>>>> >
>>>> > Hi Harsha
>>>> >
>>>> > The reason we rejected the SslProvider route is that - we
>>>> >
>>>> > only
>>>> >
>>>> > needed
>>>> >
>>>> > a
>>>> >
>>>> > custom way to load keys/certs. Not touch any policy that
>>>> >
>>>> > existing
>>>> >
>>>> > Providers
>>>> >
>>>> > govern like SunJSSE Provider.
>>>> >
>>>> > We have exactly the same requirements to load certs and keys
>>>> >
>>>> > through
>>>> >
>>>> > spire
>>>> >
>>>> > agent. We used security.provider to do that exactly. I am not
>>>> >
>>>> > sure
>>>> >
>>>> > why
>>>> >
>>>> > you
>>>> >
>>>> > would be modifying any policies provided by default SunJSSE
>>>> >
>>>> > provider.
>>>> >
>>>> > Can
>>>> >
>>>> > you give me an example of having custom provider that will
>>>> >
>>>> > override an
>>>> >
>>>> > existing policy in SunJSSE provider.
>>>> >
>>>> > As pointed out earlier, this kip
>>>> >
>>>> > https://cwiki.apache.org/confluence/display/KAFKA/
>>>> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>>>> >
>>>> > allows
>>>> > you to load security.provider through config.
>>>> > Take a look at the examples I gave before
>>>> >
>>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>>>> >
>>>> >
>>>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>>>> >
>>>> > java
>>>> >
>>>> > It registers KeyManagerFactory and TrustManagerFactory and
>>>> >
>>>> > Keystore
>>>> >
>>>> > algorithm.
>>>> > Implement your custom way of loading Keystore in here
>>>> >
>>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>>>> >
>>>> >
>>>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
>>>> >
>>>> > java
>>>> >
>>>> > and Trust manager like here
>>>> >
>>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>>>> > spiffe-security-provider/src/main/java/spiffe/api/provider/
>>>> > SpiffeTrustManager.java <http://spiffetrustmanager.java/>
>>>> >
>>>> > In your Kafka client you can set the security.provider to your
>>>> >
>>>> > custom
>>>> >
>>>> > implementation and with this fix
>>>> > https://issues.apache.org/jira/browse/KAFKA-8191 you can set
>>>> > keyManagerAlgorigthm and trustManagerAlgorithm configs.
>>>> >
>>>> > All of this is in your clients and broker side and do not need
>>>> >
>>>> > to
>>>> >
>>>> > touch
>>>> >
>>>> > any
>>>> > policy changes at JVM level. You'll register the providers in
>>>> >
>>>> > the
>>>> >
>>>> > priority
>>>> >
>>>> > order and can still have SunJSSE provider and have your custom
>>>> >
>>>> > provider
>>>> >
>>>> > to
>>>> >
>>>> > implement the key and trust managers.
>>>> >
>>>> > The ask here is different than KIP-492. We don't have any need
>>>> >
>>>> > to
>>>> >
>>>> > modify/specify the algorithm parameter. Does that make sense?
>>>> >
>>>> > The ask in KIP is introducing new interfaces where the KIP's
>>>> > goal/motivation can be achieved through the security.provider
>>>> >
>>>> > and
>>>> >
>>>> > we
>>>> >
>>>> > worked
>>>> > on similar goal without touching any Keystore or Truststore
>>>> >
>>>> > interfaces.
>>>> >
>>>> > My advise is against changing or introducing new interfaces
>>>> >
>>>> > when
>>>> >
>>>> > it can
>>>> >
>>>> > work through security.provider.
>>>> >
>>>> > Thanks,
>>>> > Harsha
>>>> >
>>>> > Thanks
>>>> >
>>>> > Maulin
>>>> >
>>>> > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
>>>> >
>>>> > kafka@harsha.io>
>>>> >
>>>> > wrote:
>>>> >
>>>> > In your KIP you added security. provider as rejected
>>>> >
>>>> > alternative
>>>> >
>>>> > and
>>>> >
>>>> > specified "its not the correct way". Do you mind explaining
>>>> >
>>>> > why
>>>> >
>>>> > its
>>>> >
>>>> > not? I
>>>> >
>>>> > didn't find any evidence in Java docs to say so. Contrary to
>>>> >
>>>> > your
>>>> >
>>>> > statement
>>>> >
>>>> > it does say in the java docs
>>>> > " However, please note that a provider can be used to
>>>> >
>>>> > implement
>>>> >
>>>> > any
>>>> >
>>>> > security service in Java that uses a pluggable architecture
>>>> >
>>>> > with
>>>> >
>>>> > a
>>>> >
>>>> > choice
>>>> >
>>>> > of implementations that fit underneath."
>>>> >
>>>> > Java Security Providers have been used by other projects to
>>>> >
>>>> > provide
>>>> >
>>>> > such
>>>> >
>>>> > integration . I am not sure if you looked into Spiffe
>>>> >
>>>> > project to
>>>> >
>>>> > efficiently distribute certificates but here is an example of
>>>> >
>>>> > Java
>>>> >
>>>> > provider
>>>> >
>>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>>>> >
>>>> >
>>>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>>>> >
>>>> > java which
>>>> > obtains certificates from local daemons.
>>>> > These integrations are being used in Tomcat, Jetty etc.. We
>>>> >
>>>> > are
>>>> >
>>>> > also
>>>> >
>>>> > using
>>>> >
>>>> > Security provider to do the same in our Kafka clusters. So
>>>> >
>>>> > unless I
>>>> >
>>>> > see
>>>> >
>>>> > more evidence why security.provider doesn't work for you
>>>> >
>>>> > adding
>>>> >
>>>> > new
>>>> >
>>>> > interfaces while there exists more cleaner way of achieving
>>>> >
>>>> > the
>>>> >
>>>> > goals
>>>> >
>>>> > of
>>>> >
>>>> > this KIP is unnecessary and breaks the well known security
>>>> >
>>>> > interfaces
>>>> >
>>>> > provided by Java itself.
>>>> >
>>>> > Thanks,
>>>> > Harsha
>>>> >
>>>> > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
>>>> >
>>>> > kafka@harsha.io
>>>> >
>>>> > wrote:
>>>> >
>>>> > Hi Maulin,
>>>> > Not sure if you looked at my previous replies. This
>>>> >
>>>> > changes
>>>> >
>>>> > are not required as there is already security Provider to do
>>>> >
>>>> > what you
>>>> >
>>>> > are
>>>> >
>>>> > proposing. This KIP
>>>> >
>>>> > https://cwiki.apache.org/confluence/display/KAFKA/
>>>> >
>>>> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>>>> >
>>>> > also
>>>> >
>>>> > addresses easy registration of such providers.
>>>> >
>>>> > Thanks,
>>>> > Harsha
>>>> >
>>>> > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
>>>> >
>>>> > <maulin.vasavada@gmail.
>>>> >
>>>> > com> wrote:
>>>> >
>>>> > Bump! Can somebody please review this?
>>>> >
>>>> > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
>>>> >
>>>> > maulin.vasavada@gmail.com>
>>>> >
>>>> > wrote:
>>>> >
>>>> > Bump! Can somebody please review this?
>>>> >
>>>> > --
>>>> > Thanks,
>>>> > M.Sai Sandeep
>>>> >
>>>> >
>>>>
>>>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Maulin Vasavada <ma...@gmail.com>.
Hi Harsha
As we synced-up offline on this topic, we hope you don't have any more
clarifications that you are seeking. If that is the case, can you please
help us move this forward and discuss what changes you would expect on the
KIP design in order to make it valuable contribution?
Just FYI - we verified our primary design change with the author of Sun's
X509 Trustmanager implementation and the outcome is that what we are
proposing makes sense at the heart of it - "Instead of writing TrustManager
just plugin the Trust store". We are open to discuss additional changes
that you/anybody else would like to see on the functionality however.
Thanks
Maulin
On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada <ma...@gmail.com>
wrote:
> Hi Harsha
>
> Any response on my question? I feel this KIP is worth accommodating. Your
> help is much appreciated.
>
> Thanks
> Maulin
>
> On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada <
> maulin.vasavada@gmail.com> wrote:
>
>> Hi Harsha
>>
>> I've examined the SPIFFE provider more and have one question -
>>
>> If SPIFFE didn't have a need to do checkSpiffeId() call at the below
>> location, would you really still write the Provider? *OR*
>> Would you just use TrustManagerFactory.init(KeyStore) signature to pass
>> the KeyStore from set of certs returned by spiffeIdManager.
>> getTrustedCerts()?
>>
>>
>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/CertificateUtils.java#L100
>>
>>
>> /**
>>> * Validates that the SPIFFE ID is present and matches the SPIFFE ID
>>> configured in
>>> * the java.security property ssl.spiffe.accept
>>> *
>>> * If the authorized spiffe ids list is empty any spiffe id is authorized
>>> *
>>> * @param chain an array of X509Certificate that contains the Peer's
>>> SVID to be validated
>>> * @throws CertificateException when either the certificates doesn't
>>> have a SPIFFE ID or the SPIFFE ID is not authorized
>>> */
>>> static void checkSpiffeId(X509Certificate[] chain) throws
>>> CertificateException {
>>
>>
>>
>> Thanks
>> Maulin
>>
>>
>> On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani <ka...@harsha.io>
>> wrote:
>>
>>> Maulin,
>>> The code parts you are pointing are specific for Spiffe and
>>> if
>>> you are talking about validate method which uses PKIX check like any
>>> other provider does.
>>> If you want to default to SunJSSE everywhere you can do so by delegating
>>> the calls in these methods to SunJSSE provider.
>>>
>>> TrustManagerFactory tmf = TrustManagerFactory
>>> .getInstance(TrustManagerFactory.getDefaultAlgorithm());and use
>>> tmf.chekServerTrusted()
>>> or use
>>> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/TrustManagerFactory.html#getInstance(java.lang.String)if
>>> you want a specific provider.
>>>
>>> -Harsha
>>>
>>>
>>> On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada <
>>> maulin.vasavada@gmail.com>
>>> wrote:
>>>
>>> > Okay, so I take that you guys agree that I have to write a 'custom'
>>> > algorithm and a provider to make it work , correct?
>>> >
>>> > Now, for Harsha's comment "Here the 'Custom' Algorithm is not an
>>> > implementation per say , ..." , I diagree. You can refer to https://
>>> >
>>> github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/
>>> > SpiffeTrustManager.java#L91 and
>>> >
>>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>>> > provider/CertificateUtils.java#L100
>>> >
>>> > "that code" is the customization you have for the custom way to check
>>> > something on top of regular checks. That method is NOT doing custom
>>> > truststore loading. It is validating/verifying something in the
>>> "custom"
>>> > way with spiffeId.
>>> > I bet that without that you won't have a need of the custom algorithm
>>> in
>>> > the first place.
>>> >
>>> > Let me know if you agree to this.
>>> >
>>> > Thanks
>>> > Maulin
>>> >
>>> > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <mp...@gmail.com>
>>> wrote:
>>> >
>>> > Hi Maulin, thanks for the discussion. As Harsha pointed out, to use the
>>> > KIP492, you need to create a new provider, register a *new* custom
>>> > algorithm for your keymanager and trustmanager factory implementations.
>>> > After this, the kafka server configuration can be done as given below
>>> >
>>> > # Register the provider class with custom algorithm, say CUSTOM
>>> security.
>>> > provider.classes=com.company.security.CustomProvider
>>> > <http://security.provider.classes
>>> =com.company.security.customprovider/>
>>> >
>>> > # Register the keymanager and trustmanager algorithms
>>> > # These algorithms indicate that the Keymanager and Trustmanagers
>>> > registered under the algorithm "CUSTOM" needs to be used
>>> > ssl.trustmanager.algorithm=CUSTOM
>>> > ssl.keymanager.algorithm=CUSTOM
>>> >
>>> > And the customprovider looks like this...
>>> >
>>> > public class CustomProvider extends Provider {
>>> > public CustomProvider() {
>>> > super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and TrustStore");
>>> > super.put("KeyManagerFactory.CUSTOM", "customKeyManagerFactory");
>>> > super.put("TrustManagerFactory.CUSTOM",
>>> > "customTrustManagerFactory");
>>> > }
>>> > }
>>> >
>>> > The PR for this is in review and can be found here:
>>> https://github.com/
>>> > apache/kafka/pull/7090
>>> > This PR includes the fixed insertProviderAt function call.
>>> >
>>> > On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani <ka...@harsha.io>
>>> > wrote:
>>> >
>>> > Answers are in-line
>>> >
>>> > On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada <
>>> maulin.vasavada@gmail.
>>> > com
>>> >
>>> > wrote:
>>> >
>>> > Hi Colin
>>> >
>>> > When I refer to "standard" or "custom" algorithms I am following Java
>>> > security Provider Terminology. You can refer to
>>> > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
>>> > StandardNames.html#TrustManagerFactory link I provided earlier in the
>>> > emails. It says PKIX is the default Algorithm for TrustManagerFactory.
>>> >
>>> > 1. For SPIFFE, I am not sure why you are saying 'it does not implement
>>> > custom algorithms' because the following file clearly indicates that it
>>> > does use custom algorithm-
>>> >
>>> >
>>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>>> >
>>> > provider/SpiffeProvider.java#L17
>>> >
>>> > Algorithm value:
>>> >
>>> >
>>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>>> >
>>> > provider/SpiffeProviderConstants.java#L6
>>> >
>>> > @Harsha do you want to chime in since you use that provider?
>>> >
>>> > Here the "Custom" Algorithm is not an implementation per say , rather
>>> >
>>> > used
>>> >
>>> > to invoke the custom trust store factory and key manager factory. You
>>> are
>>> > not moving away from "standard" alogrithms that are available.
>>> >
>>> >
>>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>>> > provider/SpiffeTrustManager.java
>>> >
>>> > As you can see it delegates all the calls of verification of
>>> certificate
>>> >
>>> > to
>>> >
>>> > the default implementation available.
>>> > So in our implementation we still use PKIX to verify the certificate
>>> > chain. So you are not losing anything here and Spiffe is not
>>> reimplementing
>>> > the verification process.
>>> >
>>> > 2. I already mentioned in my 3rd point, in my previous post, why using
>>> >
>>> > ssl.provider does NOT work. I updated KIP-486 in "rejected
>>> >
>>> > alternatives"
>>> >
>>> > also why ssl.provider does not work.
>>> >
>>> > As mentioned before , provider is the right way to go. I am still not
>>> > understanding the gap is.
>>> > If I understand correctly your argument is , provider is going to ask
>>> to
>>> > implement a custom algorithm.
>>> > My answer to that is , no you are not re-implementing the algorithm.
>>> >
>>> > Please
>>> >
>>> > check the above link , TrustManager implementation it delegates the
>>> calls
>>> > back. There is no need to implement your own here.
>>> >
>>> > 3. Security.insertProviderAt() comments were based on assumption if
>>> >
>>> > KIP-492
>>> >
>>> > changes are done and we use that mechanism to configure providers
>>> >
>>> > instead
>>> >
>>> > of ssl.provider configuration.
>>> >
>>> > KIP-492 has patch available and going through review.
>>> >
>>> > Can you read my all the points, I mentioned in my previous post, very
>>> >
>>> > carefully? I am covering all the aspects in explaining. I am open to
>>> >
>>> > still
>>> >
>>> > discuss more to clarify any doubts.
>>> >
>>> > "3. If we just use existing ssl.provider kafka configuration then our
>>> > provider will be used in SSLContext.getInstance(protocol, provider)
>>> call
>>> >
>>> > in
>>> >
>>> > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/>
>>> and
>>> > if our provider does not have
>>> > implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we
>>> >
>>> > tested
>>> >
>>> > it). Example: In MyProvider sample above you see that I didn't add
>>> > SSLContext.TLSv1 as
>>> > "Service+Algorithm" and that didn't work for me. In SPIFFE provider you
>>> > don't have this challenge since you are planning to bypass ssl.provider
>>> >
>>> > as
>>> >
>>> > you mention in the KIP-492."
>>> >
>>> > Yes here you need to pass the protocol that your
>>> KeyManager/TrustManager
>>> > registered with and in no way its deviating from TLS RFC spec.
>>> >
>>> >
>>> https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/
>>> > javax/net/ssl/SSLContext.java#L134
>>> >
>>> > My suggestion here is for you to implement a simple Security Provider
>>> as
>>> > you did before and register a Algorithm. You can use the existing
>>> > implementation in SpiffeProvider and plug in changes where you need to
>>> > retrieve the certificates from by making RPC call.
>>> >
>>> > Run an end-to-end test with Kafka broker coming up with your provider
>>> > making calls to RPC call. You do need to pass the "custom algorithm"
>>> that
>>> > you registered in your key manager to invoke the provider. I think your
>>> > concern is this approach is replacing the existing known ciphersuites
>>> and
>>> > certificate validation provided by java. Which its not.
>>> >
>>> > Now test the TLS connection you can do so via openssl -s_client options
>>> >
>>> > to
>>> >
>>> > test if everything else is working.
>>> >
>>> > I am happy to share configs that we used if you are interested.
>>> >
>>> > Thanks,
>>> > Harsha
>>> >
>>> > Thanks
>>> > Maulin
>>> >
>>> > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <cm...@apache.org>
>>> >
>>> > wrote:
>>> >
>>> > Hi Maulin,
>>> >
>>> > A lot of JSSE providers don't implement custom algorithms. Spire is a
>>> >
>>> > good
>>> >
>>> > example of a JSSE provider that doesn't, and yet is still useful to
>>> >
>>> > many
>>> >
>>> > people. Your JSSE provider can work fine even if it doesn't implement a
>>> > custom algorithm.
>>> >
>>> > Maybe I'm missing something, but I don't understand the discussion of
>>> > Security.insertProviderAt() that you included. SslEngineBuilder doesn't
>>> >
>>> > use
>>> >
>>> > that API to get the security provider. Instead, it calls
>>> > "SSLContext.getInstance(protocol, provider)", where provider is the
>>> >
>>> > name
>>> >
>>> > of the provider.
>>> >
>>> > best,
>>> > Colin
>>> >
>>> > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
>>> >
>>> > On top of everything above I feel strongly to add the 4th point which
>>> >
>>> > is
>>> >
>>> > based on Java APIs for TrustManagerFactory.init(KeyStore) (
>>> >
>>> > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
>>> > TrustManagerFactory.html#init(java.security.KeyStore
>>> > <http://java.security.keystore/>)
>>> > )
>>> >
>>> > and KeyManagerFactory.init(KeyStore, char[]) (
>>> >
>>> >
>>> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory
>>> .
>>> >
>>> >
>>> > html#init(java.security.KeyStore <http://java.security.keystore/
>>> >,%20char[])
>>> >
>>> >
>>> > ).
>>> >
>>> > 4. The above APIs are intended to support providing "trust/key
>>> >
>>> > material"
>>> >
>>> > from the user without having to write their own
>>> >
>>> > TrustManager/KeyManagers.
>>> >
>>> > To quote from the TrustManagerFactory.init()'s documentation
>>> >
>>> > "Initializes
>>> >
>>> > this factory with a source of certificate authorities and related trust
>>> > material."
>>> > To quote from the KeyManagerFactory.init()'s documentation "Initializes
>>> > this factory with a source of key material."
>>> >
>>> > Based on this it is clear that there is a flexibility provided by Java
>>> >
>>> > to
>>> >
>>> > to enable developers to provide the required trust/key material loaded
>>> >
>>> > from
>>> >
>>> > "anywhere" without requiring them to write custom provider OR trust/key
>>> > managers. This same flexibility is reflected in Kafka code also where
>>> >
>>> > it
>>> >
>>> > loads the trust/keys from a local file and doesn't require writing a
>>> > Provider necessarily. If we do NOT have a custom algorithm, it makes
>>> >
>>> > less
>>> >
>>> > sense to write a Provider.
>>> >
>>> > Thanks
>>> > Maulin
>>> >
>>> > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
>>> >
>>> > maulin.vasavada@gmail.com>
>>> >
>>> > wrote:
>>> >
>>> > Hi Harsha/Colin
>>> >
>>> > I did the sample with a custom Provider for TrustStoreManager and tried
>>> > using ssl.provider Kafka config AND the way KIP-492 is suggesting (by
>>> > adding Provider programmatically instead of relying on
>>> >
>>> > ssl.provider+java.
>>> >
>>> > security. The below sample is followed by my detailed findings. I'll
>>> > appreciate if you can go through it carefully and see
>>> >
>>> > if you
>>> >
>>> > see my point.
>>> >
>>> > package providertest;
>>> >
>>> > import java.security.Provider <http://java.security.provider/>
>>> <http://
>>> > java.security.provider/>;
>>> >
>>> > public class MyProvider extends Provider {
>>> >
>>> > private static final String name = "MyProvider"; private static double
>>> > version = 1.0d;
>>> > private static String info = "Maulin's SSL Provider v"+version;
>>> >
>>> > public MyProvider() {
>>> > super(name, version, info);
>>> > this.put("TrustManagerFactory.PKIX",
>>> >
>>> > "providertest.MyTrustManagerFactory");
>>> >
>>> > }
>>> > }
>>> >
>>> > *Details:*
>>> >
>>> > KIP-492 documents that it will use Security.addProvider() assuming it
>>> >
>>> > will
>>> >
>>> > add it as position '0' which is not a correct assumption. The
>>> > addProvider()'s documentation says it will add it to the last available
>>> > position. You may want to correct that to say
>>> > Security.insertProviderAt(provider, 1).
>>> >
>>> > Now coming back to our specific discussion,
>>> >
>>> > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add
>>> >
>>> > that
>>> >
>>> > provider in the provider list via Security.addProvider() the position
>>> >
>>> > where
>>> >
>>> > it gets added doesn't matter (even if you don't end up adding it as
>>> >
>>> > first
>>> >
>>> > entry) since that is the ONLY provider for SPIFFE specific algorithm
>>> >
>>> > you
>>> >
>>> > might have.
>>> >
>>> > We do *not* have custom algorithm for Key/Trust StoreMangers. Which
>>> >
>>> > means
>>> >
>>> > we have to use X509, PKIX etc "Standard Algorithms" ((
>>> >
>>> > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
>>> > StandardNames.html
>>> > ))
>>> >
>>> > in our provider to override the TrustStoreManager (see my sample code)
>>> >
>>> > and
>>> >
>>> > KeyStoreManger and KeyManager. This creates another challenge
>>> >
>>> > mentioned in
>>> >
>>> > the below point.
>>> >
>>> > 2. In order to make our Provider for loading custom TrustStore work, we
>>> > have to add the provider as 'first' in the list since there are others
>>> >
>>> > with
>>> >
>>> > the same algorithm.
>>> >
>>> > However, the programatic way of adding provider
>>> > (Security.insertProviderAt()) is *not* deterministic for ordering since
>>> > different code can call the method for a different provider and
>>> >
>>> > depending
>>> >
>>> > upon the order of the call our provider can be first or pushed down the
>>> > list. This can happen very well in any client application using Kafka.
>>> >
>>> > This
>>> >
>>> > is specially problematic for a case when you want to guarantee order
>>> >
>>> > for a
>>> >
>>> > Provider having "Standard Algorithms".
>>> >
>>> > If we add our provider in java.security file that definitely guarantees
>>> > the order(unless somebody calls removeProvider() which is less
>>> >
>>> > likely). But
>>> >
>>> > if we add our provider in java.security file it will defeat the
>>> >
>>> > purpose of
>>> >
>>> > the KIP-492.
>>> >
>>> > In the gist - Apache Kafka must not rely on "un-deterministic" method
>>> >
>>> > to
>>> >
>>> > rely on Provider ordering.
>>> >
>>> > 3. If we just use existing ssl.provider kafka configuration then our
>>> > provider will be used in SSLContext.getInstance(protocol, provider)
>>> >
>>> > call in
>>> >
>>> > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/>
>>> and
>>> > if our provider does not have implementation for
>>> > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks
>>> >
>>> > (we
>>> >
>>> > tested it). Example:
>>> >
>>> > In
>>> >
>>> > MyProvider sample above you see that I didn't add SSLContext.TLSv1 as
>>> > "Service+Algorithm" and that didn't work for me. In SPIFFE provider you
>>> > don't have this challenge since you are planning to bypass
>>> >
>>> > ssl.provider as
>>> >
>>> > you mention in the KIP-492.
>>> >
>>> > *Overall summary,*
>>> >
>>> > 1. Any provider based mechanisms- a) existing ssl.provider and
>>> >
>>> > b)KIP-492,
>>> >
>>> > for loading key/trust store using "Standard Algorithms" do not work
>>> >
>>> > 2. Approach suggested in our KIP-486 works without any issue and it is
>>> > *not* our context specific solve
>>> >
>>> > 3. Based on above we feel KIP-492 and KIP-486 are complimentary changes
>>> > and not contradicting or redundent.
>>> >
>>> > If you want we can do a joint session somehow to walk through the
>>> >
>>> > sample I
>>> >
>>> > have and various experiments I did. I would encourage you to do similar
>>> > exercise by writing a Provider for "Standard Algorithm" for
>>> > TrustStoreManager (like our needs) and see what you find since only
>>> >
>>> > writing
>>> >
>>> > samples can bring out the complexity/challenges we face.
>>> >
>>> > Thanks
>>> > Maulin
>>> >
>>> > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <
>>> >
>>> > maulin.vasavada@gmail.
>>> >
>>> > com> wrote:
>>> >
>>> > Just to update - still working on it. Get to work only on and off on
>>> >
>>> > it :(
>>> >
>>> > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
>>> >
>>> > maulin.vasavada@gmail.com>
>>> >
>>> > wrote:
>>> >
>>> > Hi Harsha
>>> >
>>> > Let me try to write samples and will let you know.
>>> >
>>> > Thanks
>>> > Maulin
>>> >
>>> > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <ha...@gmail.com>
>>> >
>>> > wrote:
>>> >
>>> > Hi Maulin,
>>> > With java security providers can be as custom you would
>>> >
>>> > like
>>> >
>>> > it to
>>> > be. If you only want to to implement a custom way of loading the
>>> >
>>> > keystore
>>> >
>>> > and truststore and not implement any protocol/encryption handling you
>>> can
>>> > leave them empty and no need to implement. Have you looked into the
>>> links I
>>> > pasted before?
>>> >
>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>>> >
>>> >
>>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
>>> >
>>> > java
>>> >
>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>>> > spiffe-security-provider/src/main/java/spiffe/api/provider/
>>> > SpiffeTrustManager.java <http://spiffetrustmanager.java/>
>>> >
>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>>> >
>>> >
>>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>>> >
>>> > java
>>> >
>>> > Can you please tell me which methods are too complex in above to
>>> >
>>> > implement
>>> >
>>> > or unnecessary? You are changing anything in SSL/TLS implementations
>>> > provided by
>>> >
>>> > All of the implementations delegating the checks to the default
>>> > implementation anyway.
>>> > Spire agent is an example, its nothing but a GRPC server listening
>>> >
>>> > on a
>>> >
>>> > unix domain socket . Above code is making a RPC call to the local
>>> >
>>> > daemon
>>> >
>>> > to
>>> > get the certificate and keys. The mechanics are pretty much same as
>>> >
>>> > what
>>> >
>>> > you are asking for.
>>> >
>>> > Thanks,
>>> > Harsha
>>> >
>>> > On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
>>> >
>>> > maulin.vasavada@gmail.com>
>>> >
>>> > wrote:
>>> >
>>> > Imagine a scenario like - We know we have a custom KMS and as a
>>> >
>>> > Kafka
>>> >
>>> > owner
>>> >
>>> > we want to comply to using that KMS source to load keys/certs. As
>>> >
>>> > a
>>> >
>>> > Kafka
>>> >
>>> > owner we know how to integrate with KMS but doesn't necessarily
>>> >
>>> > have
>>> >
>>> > to
>>> >
>>> > know anything about cipher suites, algorithms, and SSL/TLS
>>> >
>>> > implementation.
>>> >
>>> > Going the Provider way requires to know lot more than we should,
>>> >
>>> > isn't it?
>>> >
>>> > Not that we would have concern/shy-away knowing those details -
>>> >
>>> > but
>>> >
>>> > if we
>>> >
>>> > don't have to - why should we?
>>> >
>>> > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
>>> >
>>> > maulin.vasavada@gmail.com>
>>> >
>>> > wrote:
>>> >
>>> > Hi Harsha
>>> >
>>> > We don't have spire (or similar) agents and we do not have
>>> >
>>> > keys/certs
>>> >
>>> > locally on any brokers. To elaborate more on my previous email,
>>> >
>>> > I agree that Java security Providers are used in much broader
>>> >
>>> > sense
>>> >
>>> > - to
>>> >
>>> > have a particular implementation of an algorithm, use specific
>>> >
>>> > cipher
>>> >
>>> > suites for SSL , OR in our current team's case have a
>>> >
>>> > particular
>>> >
>>> > way to
>>> >
>>> > leverage pre-generated SSL sessions. However, the scope of our
>>> >
>>> > KIP
>>> >
>>> > (486)
>>> >
>>> > is
>>> >
>>> > much restricted than that. We merely intend to provide a custom
>>> > keystore/truststore for our SSL connections and not really worry
>>> >
>>> > about
>>> >
>>> > underlying specific SSL/TLS implementation. This simplifies it
>>> >
>>> > a
>>> >
>>> > lot for
>>> >
>>> > us to keep the concerns separate and clear.
>>> >
>>> > I feel our approach is more complimentary such that it allows
>>> >
>>> > for
>>> >
>>> > using
>>> >
>>> > keystores of choice while retaining the flexibility to use any
>>> > underlying/available Provider for actually making the SSL call.
>>> >
>>> > We agree with KIP-492's approach based on Providers (and Java's
>>> > recommendation), but also strongly believe that our approach can
>>> >
>>> > compliment
>>> >
>>> > it very effectively for reasons explained above.
>>> >
>>> > Thanks
>>> > Maulin
>>> >
>>> > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <
>>> >
>>> > kafka@harsha.io
>>> >
>>> > wrote:
>>> >
>>> > Hi Maulin,
>>> >
>>> > On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
>>> >
>>> > maulin.vasavada@gmail.
>>> >
>>> > com>
>>> > wrote:
>>> >
>>> > Hi Harsha
>>> >
>>> > The reason we rejected the SslProvider route is that - we
>>> >
>>> > only
>>> >
>>> > needed
>>> >
>>> > a
>>> >
>>> > custom way to load keys/certs. Not touch any policy that
>>> >
>>> > existing
>>> >
>>> > Providers
>>> >
>>> > govern like SunJSSE Provider.
>>> >
>>> > We have exactly the same requirements to load certs and keys
>>> >
>>> > through
>>> >
>>> > spire
>>> >
>>> > agent. We used security.provider to do that exactly. I am not
>>> >
>>> > sure
>>> >
>>> > why
>>> >
>>> > you
>>> >
>>> > would be modifying any policies provided by default SunJSSE
>>> >
>>> > provider.
>>> >
>>> > Can
>>> >
>>> > you give me an example of having custom provider that will
>>> >
>>> > override an
>>> >
>>> > existing policy in SunJSSE provider.
>>> >
>>> > As pointed out earlier, this kip
>>> >
>>> > https://cwiki.apache.org/confluence/display/KAFKA/
>>> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>>> >
>>> > allows
>>> > you to load security.provider through config.
>>> > Take a look at the examples I gave before
>>> >
>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>>> >
>>> >
>>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>>> >
>>> > java
>>> >
>>> > It registers KeyManagerFactory and TrustManagerFactory and
>>> >
>>> > Keystore
>>> >
>>> > algorithm.
>>> > Implement your custom way of loading Keystore in here
>>> >
>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>>> >
>>> >
>>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
>>> >
>>> > java
>>> >
>>> > and Trust manager like here
>>> >
>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>>> > spiffe-security-provider/src/main/java/spiffe/api/provider/
>>> > SpiffeTrustManager.java <http://spiffetrustmanager.java/>
>>> >
>>> > In your Kafka client you can set the security.provider to your
>>> >
>>> > custom
>>> >
>>> > implementation and with this fix
>>> > https://issues.apache.org/jira/browse/KAFKA-8191 you can set
>>> > keyManagerAlgorigthm and trustManagerAlgorithm configs.
>>> >
>>> > All of this is in your clients and broker side and do not need
>>> >
>>> > to
>>> >
>>> > touch
>>> >
>>> > any
>>> > policy changes at JVM level. You'll register the providers in
>>> >
>>> > the
>>> >
>>> > priority
>>> >
>>> > order and can still have SunJSSE provider and have your custom
>>> >
>>> > provider
>>> >
>>> > to
>>> >
>>> > implement the key and trust managers.
>>> >
>>> > The ask here is different than KIP-492. We don't have any need
>>> >
>>> > to
>>> >
>>> > modify/specify the algorithm parameter. Does that make sense?
>>> >
>>> > The ask in KIP is introducing new interfaces where the KIP's
>>> > goal/motivation can be achieved through the security.provider
>>> >
>>> > and
>>> >
>>> > we
>>> >
>>> > worked
>>> > on similar goal without touching any Keystore or Truststore
>>> >
>>> > interfaces.
>>> >
>>> > My advise is against changing or introducing new interfaces
>>> >
>>> > when
>>> >
>>> > it can
>>> >
>>> > work through security.provider.
>>> >
>>> > Thanks,
>>> > Harsha
>>> >
>>> > Thanks
>>> >
>>> > Maulin
>>> >
>>> > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
>>> >
>>> > kafka@harsha.io>
>>> >
>>> > wrote:
>>> >
>>> > In your KIP you added security. provider as rejected
>>> >
>>> > alternative
>>> >
>>> > and
>>> >
>>> > specified "its not the correct way". Do you mind explaining
>>> >
>>> > why
>>> >
>>> > its
>>> >
>>> > not? I
>>> >
>>> > didn't find any evidence in Java docs to say so. Contrary to
>>> >
>>> > your
>>> >
>>> > statement
>>> >
>>> > it does say in the java docs
>>> > " However, please note that a provider can be used to
>>> >
>>> > implement
>>> >
>>> > any
>>> >
>>> > security service in Java that uses a pluggable architecture
>>> >
>>> > with
>>> >
>>> > a
>>> >
>>> > choice
>>> >
>>> > of implementations that fit underneath."
>>> >
>>> > Java Security Providers have been used by other projects to
>>> >
>>> > provide
>>> >
>>> > such
>>> >
>>> > integration . I am not sure if you looked into Spiffe
>>> >
>>> > project to
>>> >
>>> > efficiently distribute certificates but here is an example of
>>> >
>>> > Java
>>> >
>>> > provider
>>> >
>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>>> >
>>> >
>>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>>> >
>>> > java which
>>> > obtains certificates from local daemons.
>>> > These integrations are being used in Tomcat, Jetty etc.. We
>>> >
>>> > are
>>> >
>>> > also
>>> >
>>> > using
>>> >
>>> > Security provider to do the same in our Kafka clusters. So
>>> >
>>> > unless I
>>> >
>>> > see
>>> >
>>> > more evidence why security.provider doesn't work for you
>>> >
>>> > adding
>>> >
>>> > new
>>> >
>>> > interfaces while there exists more cleaner way of achieving
>>> >
>>> > the
>>> >
>>> > goals
>>> >
>>> > of
>>> >
>>> > this KIP is unnecessary and breaks the well known security
>>> >
>>> > interfaces
>>> >
>>> > provided by Java itself.
>>> >
>>> > Thanks,
>>> > Harsha
>>> >
>>> > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
>>> >
>>> > kafka@harsha.io
>>> >
>>> > wrote:
>>> >
>>> > Hi Maulin,
>>> > Not sure if you looked at my previous replies. This
>>> >
>>> > changes
>>> >
>>> > are not required as there is already security Provider to do
>>> >
>>> > what you
>>> >
>>> > are
>>> >
>>> > proposing. This KIP
>>> >
>>> > https://cwiki.apache.org/confluence/display/KAFKA/
>>> >
>>> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>>> >
>>> > also
>>> >
>>> > addresses easy registration of such providers.
>>> >
>>> > Thanks,
>>> > Harsha
>>> >
>>> > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
>>> >
>>> > <maulin.vasavada@gmail.
>>> >
>>> > com> wrote:
>>> >
>>> > Bump! Can somebody please review this?
>>> >
>>> > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
>>> >
>>> > maulin.vasavada@gmail.com>
>>> >
>>> > wrote:
>>> >
>>> > Bump! Can somebody please review this?
>>> >
>>> > --
>>> > Thanks,
>>> > M.Sai Sandeep
>>> >
>>> >
>>>
>>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Maulin Vasavada <ma...@gmail.com>.
Hi Harsha
Any response on my question? I feel this KIP is worth accommodating. Your
help is much appreciated.
Thanks
Maulin
On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada <ma...@gmail.com>
wrote:
> Hi Harsha
>
> I've examined the SPIFFE provider more and have one question -
>
> If SPIFFE didn't have a need to do checkSpiffeId() call at the below
> location, would you really still write the Provider? *OR*
> Would you just use TrustManagerFactory.init(KeyStore) signature to pass
> the KeyStore from set of certs returned by spiffeIdManager.
> getTrustedCerts()?
>
>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/CertificateUtils.java#L100
>
>
> /**
>> * Validates that the SPIFFE ID is present and matches the SPIFFE ID
>> configured in
>> * the java.security property ssl.spiffe.accept
>> *
>> * If the authorized spiffe ids list is empty any spiffe id is authorized
>> *
>> * @param chain an array of X509Certificate that contains the Peer's SVID
>> to be validated
>> * @throws CertificateException when either the certificates doesn't have
>> a SPIFFE ID or the SPIFFE ID is not authorized
>> */
>> static void checkSpiffeId(X509Certificate[] chain) throws
>> CertificateException {
>
>
>
> Thanks
> Maulin
>
>
> On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani <ka...@harsha.io>
> wrote:
>
>> Maulin,
>> The code parts you are pointing are specific for Spiffe and
>> if
>> you are talking about validate method which uses PKIX check like any
>> other provider does.
>> If you want to default to SunJSSE everywhere you can do so by delegating
>> the calls in these methods to SunJSSE provider.
>>
>> TrustManagerFactory tmf = TrustManagerFactory
>> .getInstance(TrustManagerFactory.getDefaultAlgorithm());and use
>> tmf.chekServerTrusted()
>> or use
>> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/TrustManagerFactory.html#getInstance(java.lang.String)if
>> you want a specific provider.
>>
>> -Harsha
>>
>>
>> On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada <
>> maulin.vasavada@gmail.com>
>> wrote:
>>
>> > Okay, so I take that you guys agree that I have to write a 'custom'
>> > algorithm and a provider to make it work , correct?
>> >
>> > Now, for Harsha's comment "Here the 'Custom' Algorithm is not an
>> > implementation per say , ..." , I diagree. You can refer to https://
>> >
>> github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/
>> > SpiffeTrustManager.java#L91 and
>> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>> > provider/CertificateUtils.java#L100
>> >
>> > "that code" is the customization you have for the custom way to check
>> > something on top of regular checks. That method is NOT doing custom
>> > truststore loading. It is validating/verifying something in the "custom"
>> > way with spiffeId.
>> > I bet that without that you won't have a need of the custom algorithm in
>> > the first place.
>> >
>> > Let me know if you agree to this.
>> >
>> > Thanks
>> > Maulin
>> >
>> > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <mp...@gmail.com>
>> wrote:
>> >
>> > Hi Maulin, thanks for the discussion. As Harsha pointed out, to use the
>> > KIP492, you need to create a new provider, register a *new* custom
>> > algorithm for your keymanager and trustmanager factory implementations.
>> > After this, the kafka server configuration can be done as given below
>> >
>> > # Register the provider class with custom algorithm, say CUSTOM
>> security.
>> > provider.classes=com.company.security.CustomProvider
>> > <http://security.provider.classes=com.company.security.customprovider/>
>> >
>> > # Register the keymanager and trustmanager algorithms
>> > # These algorithms indicate that the Keymanager and Trustmanagers
>> > registered under the algorithm "CUSTOM" needs to be used
>> > ssl.trustmanager.algorithm=CUSTOM
>> > ssl.keymanager.algorithm=CUSTOM
>> >
>> > And the customprovider looks like this...
>> >
>> > public class CustomProvider extends Provider {
>> > public CustomProvider() {
>> > super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and TrustStore");
>> > super.put("KeyManagerFactory.CUSTOM", "customKeyManagerFactory");
>> > super.put("TrustManagerFactory.CUSTOM",
>> > "customTrustManagerFactory");
>> > }
>> > }
>> >
>> > The PR for this is in review and can be found here: https://github.com/
>> > apache/kafka/pull/7090
>> > This PR includes the fixed insertProviderAt function call.
>> >
>> > On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani <ka...@harsha.io>
>> > wrote:
>> >
>> > Answers are in-line
>> >
>> > On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada <
>> maulin.vasavada@gmail.
>> > com
>> >
>> > wrote:
>> >
>> > Hi Colin
>> >
>> > When I refer to "standard" or "custom" algorithms I am following Java
>> > security Provider Terminology. You can refer to
>> > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
>> > StandardNames.html#TrustManagerFactory link I provided earlier in the
>> > emails. It says PKIX is the default Algorithm for TrustManagerFactory.
>> >
>> > 1. For SPIFFE, I am not sure why you are saying 'it does not implement
>> > custom algorithms' because the following file clearly indicates that it
>> > does use custom algorithm-
>> >
>> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>> >
>> > provider/SpiffeProvider.java#L17
>> >
>> > Algorithm value:
>> >
>> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>> >
>> > provider/SpiffeProviderConstants.java#L6
>> >
>> > @Harsha do you want to chime in since you use that provider?
>> >
>> > Here the "Custom" Algorithm is not an implementation per say , rather
>> >
>> > used
>> >
>> > to invoke the custom trust store factory and key manager factory. You
>> are
>> > not moving away from "standard" alogrithms that are available.
>> >
>> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>> > provider/SpiffeTrustManager.java
>> >
>> > As you can see it delegates all the calls of verification of certificate
>> >
>> > to
>> >
>> > the default implementation available.
>> > So in our implementation we still use PKIX to verify the certificate
>> > chain. So you are not losing anything here and Spiffe is not
>> reimplementing
>> > the verification process.
>> >
>> > 2. I already mentioned in my 3rd point, in my previous post, why using
>> >
>> > ssl.provider does NOT work. I updated KIP-486 in "rejected
>> >
>> > alternatives"
>> >
>> > also why ssl.provider does not work.
>> >
>> > As mentioned before , provider is the right way to go. I am still not
>> > understanding the gap is.
>> > If I understand correctly your argument is , provider is going to ask to
>> > implement a custom algorithm.
>> > My answer to that is , no you are not re-implementing the algorithm.
>> >
>> > Please
>> >
>> > check the above link , TrustManager implementation it delegates the
>> calls
>> > back. There is no need to implement your own here.
>> >
>> > 3. Security.insertProviderAt() comments were based on assumption if
>> >
>> > KIP-492
>> >
>> > changes are done and we use that mechanism to configure providers
>> >
>> > instead
>> >
>> > of ssl.provider configuration.
>> >
>> > KIP-492 has patch available and going through review.
>> >
>> > Can you read my all the points, I mentioned in my previous post, very
>> >
>> > carefully? I am covering all the aspects in explaining. I am open to
>> >
>> > still
>> >
>> > discuss more to clarify any doubts.
>> >
>> > "3. If we just use existing ssl.provider kafka configuration then our
>> > provider will be used in SSLContext.getInstance(protocol, provider) call
>> >
>> > in
>> >
>> > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/> and
>> > if our provider does not have
>> > implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we
>> >
>> > tested
>> >
>> > it). Example: In MyProvider sample above you see that I didn't add
>> > SSLContext.TLSv1 as
>> > "Service+Algorithm" and that didn't work for me. In SPIFFE provider you
>> > don't have this challenge since you are planning to bypass ssl.provider
>> >
>> > as
>> >
>> > you mention in the KIP-492."
>> >
>> > Yes here you need to pass the protocol that your KeyManager/TrustManager
>> > registered with and in no way its deviating from TLS RFC spec.
>> >
>> > https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/
>> > javax/net/ssl/SSLContext.java#L134
>> >
>> > My suggestion here is for you to implement a simple Security Provider as
>> > you did before and register a Algorithm. You can use the existing
>> > implementation in SpiffeProvider and plug in changes where you need to
>> > retrieve the certificates from by making RPC call.
>> >
>> > Run an end-to-end test with Kafka broker coming up with your provider
>> > making calls to RPC call. You do need to pass the "custom algorithm"
>> that
>> > you registered in your key manager to invoke the provider. I think your
>> > concern is this approach is replacing the existing known ciphersuites
>> and
>> > certificate validation provided by java. Which its not.
>> >
>> > Now test the TLS connection you can do so via openssl -s_client options
>> >
>> > to
>> >
>> > test if everything else is working.
>> >
>> > I am happy to share configs that we used if you are interested.
>> >
>> > Thanks,
>> > Harsha
>> >
>> > Thanks
>> > Maulin
>> >
>> > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <cm...@apache.org>
>> >
>> > wrote:
>> >
>> > Hi Maulin,
>> >
>> > A lot of JSSE providers don't implement custom algorithms. Spire is a
>> >
>> > good
>> >
>> > example of a JSSE provider that doesn't, and yet is still useful to
>> >
>> > many
>> >
>> > people. Your JSSE provider can work fine even if it doesn't implement a
>> > custom algorithm.
>> >
>> > Maybe I'm missing something, but I don't understand the discussion of
>> > Security.insertProviderAt() that you included. SslEngineBuilder doesn't
>> >
>> > use
>> >
>> > that API to get the security provider. Instead, it calls
>> > "SSLContext.getInstance(protocol, provider)", where provider is the
>> >
>> > name
>> >
>> > of the provider.
>> >
>> > best,
>> > Colin
>> >
>> > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
>> >
>> > On top of everything above I feel strongly to add the 4th point which
>> >
>> > is
>> >
>> > based on Java APIs for TrustManagerFactory.init(KeyStore) (
>> >
>> > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
>> > TrustManagerFactory.html#init(java.security.KeyStore
>> > <http://java.security.keystore/>)
>> > )
>> >
>> > and KeyManagerFactory.init(KeyStore, char[]) (
>> >
>> >
>> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory
>> .
>> >
>> >
>> > html#init(java.security.KeyStore <http://java.security.keystore/
>> >,%20char[])
>> >
>> >
>> > ).
>> >
>> > 4. The above APIs are intended to support providing "trust/key
>> >
>> > material"
>> >
>> > from the user without having to write their own
>> >
>> > TrustManager/KeyManagers.
>> >
>> > To quote from the TrustManagerFactory.init()'s documentation
>> >
>> > "Initializes
>> >
>> > this factory with a source of certificate authorities and related trust
>> > material."
>> > To quote from the KeyManagerFactory.init()'s documentation "Initializes
>> > this factory with a source of key material."
>> >
>> > Based on this it is clear that there is a flexibility provided by Java
>> >
>> > to
>> >
>> > to enable developers to provide the required trust/key material loaded
>> >
>> > from
>> >
>> > "anywhere" without requiring them to write custom provider OR trust/key
>> > managers. This same flexibility is reflected in Kafka code also where
>> >
>> > it
>> >
>> > loads the trust/keys from a local file and doesn't require writing a
>> > Provider necessarily. If we do NOT have a custom algorithm, it makes
>> >
>> > less
>> >
>> > sense to write a Provider.
>> >
>> > Thanks
>> > Maulin
>> >
>> > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
>> >
>> > maulin.vasavada@gmail.com>
>> >
>> > wrote:
>> >
>> > Hi Harsha/Colin
>> >
>> > I did the sample with a custom Provider for TrustStoreManager and tried
>> > using ssl.provider Kafka config AND the way KIP-492 is suggesting (by
>> > adding Provider programmatically instead of relying on
>> >
>> > ssl.provider+java.
>> >
>> > security. The below sample is followed by my detailed findings. I'll
>> > appreciate if you can go through it carefully and see
>> >
>> > if you
>> >
>> > see my point.
>> >
>> > package providertest;
>> >
>> > import java.security.Provider <http://java.security.provider/> <http://
>> > java.security.provider/>;
>> >
>> > public class MyProvider extends Provider {
>> >
>> > private static final String name = "MyProvider"; private static double
>> > version = 1.0d;
>> > private static String info = "Maulin's SSL Provider v"+version;
>> >
>> > public MyProvider() {
>> > super(name, version, info);
>> > this.put("TrustManagerFactory.PKIX",
>> >
>> > "providertest.MyTrustManagerFactory");
>> >
>> > }
>> > }
>> >
>> > *Details:*
>> >
>> > KIP-492 documents that it will use Security.addProvider() assuming it
>> >
>> > will
>> >
>> > add it as position '0' which is not a correct assumption. The
>> > addProvider()'s documentation says it will add it to the last available
>> > position. You may want to correct that to say
>> > Security.insertProviderAt(provider, 1).
>> >
>> > Now coming back to our specific discussion,
>> >
>> > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add
>> >
>> > that
>> >
>> > provider in the provider list via Security.addProvider() the position
>> >
>> > where
>> >
>> > it gets added doesn't matter (even if you don't end up adding it as
>> >
>> > first
>> >
>> > entry) since that is the ONLY provider for SPIFFE specific algorithm
>> >
>> > you
>> >
>> > might have.
>> >
>> > We do *not* have custom algorithm for Key/Trust StoreMangers. Which
>> >
>> > means
>> >
>> > we have to use X509, PKIX etc "Standard Algorithms" ((
>> >
>> > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
>> > StandardNames.html
>> > ))
>> >
>> > in our provider to override the TrustStoreManager (see my sample code)
>> >
>> > and
>> >
>> > KeyStoreManger and KeyManager. This creates another challenge
>> >
>> > mentioned in
>> >
>> > the below point.
>> >
>> > 2. In order to make our Provider for loading custom TrustStore work, we
>> > have to add the provider as 'first' in the list since there are others
>> >
>> > with
>> >
>> > the same algorithm.
>> >
>> > However, the programatic way of adding provider
>> > (Security.insertProviderAt()) is *not* deterministic for ordering since
>> > different code can call the method for a different provider and
>> >
>> > depending
>> >
>> > upon the order of the call our provider can be first or pushed down the
>> > list. This can happen very well in any client application using Kafka.
>> >
>> > This
>> >
>> > is specially problematic for a case when you want to guarantee order
>> >
>> > for a
>> >
>> > Provider having "Standard Algorithms".
>> >
>> > If we add our provider in java.security file that definitely guarantees
>> > the order(unless somebody calls removeProvider() which is less
>> >
>> > likely). But
>> >
>> > if we add our provider in java.security file it will defeat the
>> >
>> > purpose of
>> >
>> > the KIP-492.
>> >
>> > In the gist - Apache Kafka must not rely on "un-deterministic" method
>> >
>> > to
>> >
>> > rely on Provider ordering.
>> >
>> > 3. If we just use existing ssl.provider kafka configuration then our
>> > provider will be used in SSLContext.getInstance(protocol, provider)
>> >
>> > call in
>> >
>> > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/> and
>> > if our provider does not have implementation for
>> > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks
>> >
>> > (we
>> >
>> > tested it). Example:
>> >
>> > In
>> >
>> > MyProvider sample above you see that I didn't add SSLContext.TLSv1 as
>> > "Service+Algorithm" and that didn't work for me. In SPIFFE provider you
>> > don't have this challenge since you are planning to bypass
>> >
>> > ssl.provider as
>> >
>> > you mention in the KIP-492.
>> >
>> > *Overall summary,*
>> >
>> > 1. Any provider based mechanisms- a) existing ssl.provider and
>> >
>> > b)KIP-492,
>> >
>> > for loading key/trust store using "Standard Algorithms" do not work
>> >
>> > 2. Approach suggested in our KIP-486 works without any issue and it is
>> > *not* our context specific solve
>> >
>> > 3. Based on above we feel KIP-492 and KIP-486 are complimentary changes
>> > and not contradicting or redundent.
>> >
>> > If you want we can do a joint session somehow to walk through the
>> >
>> > sample I
>> >
>> > have and various experiments I did. I would encourage you to do similar
>> > exercise by writing a Provider for "Standard Algorithm" for
>> > TrustStoreManager (like our needs) and see what you find since only
>> >
>> > writing
>> >
>> > samples can bring out the complexity/challenges we face.
>> >
>> > Thanks
>> > Maulin
>> >
>> > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <
>> >
>> > maulin.vasavada@gmail.
>> >
>> > com> wrote:
>> >
>> > Just to update - still working on it. Get to work only on and off on
>> >
>> > it :(
>> >
>> > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
>> >
>> > maulin.vasavada@gmail.com>
>> >
>> > wrote:
>> >
>> > Hi Harsha
>> >
>> > Let me try to write samples and will let you know.
>> >
>> > Thanks
>> > Maulin
>> >
>> > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <ha...@gmail.com>
>> >
>> > wrote:
>> >
>> > Hi Maulin,
>> > With java security providers can be as custom you would
>> >
>> > like
>> >
>> > it to
>> > be. If you only want to to implement a custom way of loading the
>> >
>> > keystore
>> >
>> > and truststore and not implement any protocol/encryption handling you
>> can
>> > leave them empty and no need to implement. Have you looked into the
>> links I
>> > pasted before?
>> >
>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>> >
>> >
>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
>> >
>> > java
>> >
>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>> > spiffe-security-provider/src/main/java/spiffe/api/provider/
>> > SpiffeTrustManager.java <http://spiffetrustmanager.java/>
>> >
>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>> >
>> >
>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>> >
>> > java
>> >
>> > Can you please tell me which methods are too complex in above to
>> >
>> > implement
>> >
>> > or unnecessary? You are changing anything in SSL/TLS implementations
>> > provided by
>> >
>> > All of the implementations delegating the checks to the default
>> > implementation anyway.
>> > Spire agent is an example, its nothing but a GRPC server listening
>> >
>> > on a
>> >
>> > unix domain socket . Above code is making a RPC call to the local
>> >
>> > daemon
>> >
>> > to
>> > get the certificate and keys. The mechanics are pretty much same as
>> >
>> > what
>> >
>> > you are asking for.
>> >
>> > Thanks,
>> > Harsha
>> >
>> > On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
>> >
>> > maulin.vasavada@gmail.com>
>> >
>> > wrote:
>> >
>> > Imagine a scenario like - We know we have a custom KMS and as a
>> >
>> > Kafka
>> >
>> > owner
>> >
>> > we want to comply to using that KMS source to load keys/certs. As
>> >
>> > a
>> >
>> > Kafka
>> >
>> > owner we know how to integrate with KMS but doesn't necessarily
>> >
>> > have
>> >
>> > to
>> >
>> > know anything about cipher suites, algorithms, and SSL/TLS
>> >
>> > implementation.
>> >
>> > Going the Provider way requires to know lot more than we should,
>> >
>> > isn't it?
>> >
>> > Not that we would have concern/shy-away knowing those details -
>> >
>> > but
>> >
>> > if we
>> >
>> > don't have to - why should we?
>> >
>> > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
>> >
>> > maulin.vasavada@gmail.com>
>> >
>> > wrote:
>> >
>> > Hi Harsha
>> >
>> > We don't have spire (or similar) agents and we do not have
>> >
>> > keys/certs
>> >
>> > locally on any brokers. To elaborate more on my previous email,
>> >
>> > I agree that Java security Providers are used in much broader
>> >
>> > sense
>> >
>> > - to
>> >
>> > have a particular implementation of an algorithm, use specific
>> >
>> > cipher
>> >
>> > suites for SSL , OR in our current team's case have a
>> >
>> > particular
>> >
>> > way to
>> >
>> > leverage pre-generated SSL sessions. However, the scope of our
>> >
>> > KIP
>> >
>> > (486)
>> >
>> > is
>> >
>> > much restricted than that. We merely intend to provide a custom
>> > keystore/truststore for our SSL connections and not really worry
>> >
>> > about
>> >
>> > underlying specific SSL/TLS implementation. This simplifies it
>> >
>> > a
>> >
>> > lot for
>> >
>> > us to keep the concerns separate and clear.
>> >
>> > I feel our approach is more complimentary such that it allows
>> >
>> > for
>> >
>> > using
>> >
>> > keystores of choice while retaining the flexibility to use any
>> > underlying/available Provider for actually making the SSL call.
>> >
>> > We agree with KIP-492's approach based on Providers (and Java's
>> > recommendation), but also strongly believe that our approach can
>> >
>> > compliment
>> >
>> > it very effectively for reasons explained above.
>> >
>> > Thanks
>> > Maulin
>> >
>> > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <
>> >
>> > kafka@harsha.io
>> >
>> > wrote:
>> >
>> > Hi Maulin,
>> >
>> > On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
>> >
>> > maulin.vasavada@gmail.
>> >
>> > com>
>> > wrote:
>> >
>> > Hi Harsha
>> >
>> > The reason we rejected the SslProvider route is that - we
>> >
>> > only
>> >
>> > needed
>> >
>> > a
>> >
>> > custom way to load keys/certs. Not touch any policy that
>> >
>> > existing
>> >
>> > Providers
>> >
>> > govern like SunJSSE Provider.
>> >
>> > We have exactly the same requirements to load certs and keys
>> >
>> > through
>> >
>> > spire
>> >
>> > agent. We used security.provider to do that exactly. I am not
>> >
>> > sure
>> >
>> > why
>> >
>> > you
>> >
>> > would be modifying any policies provided by default SunJSSE
>> >
>> > provider.
>> >
>> > Can
>> >
>> > you give me an example of having custom provider that will
>> >
>> > override an
>> >
>> > existing policy in SunJSSE provider.
>> >
>> > As pointed out earlier, this kip
>> >
>> > https://cwiki.apache.org/confluence/display/KAFKA/
>> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>> >
>> > allows
>> > you to load security.provider through config.
>> > Take a look at the examples I gave before
>> >
>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>> >
>> >
>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>> >
>> > java
>> >
>> > It registers KeyManagerFactory and TrustManagerFactory and
>> >
>> > Keystore
>> >
>> > algorithm.
>> > Implement your custom way of loading Keystore in here
>> >
>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>> >
>> >
>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
>> >
>> > java
>> >
>> > and Trust manager like here
>> >
>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>> > spiffe-security-provider/src/main/java/spiffe/api/provider/
>> > SpiffeTrustManager.java <http://spiffetrustmanager.java/>
>> >
>> > In your Kafka client you can set the security.provider to your
>> >
>> > custom
>> >
>> > implementation and with this fix
>> > https://issues.apache.org/jira/browse/KAFKA-8191 you can set
>> > keyManagerAlgorigthm and trustManagerAlgorithm configs.
>> >
>> > All of this is in your clients and broker side and do not need
>> >
>> > to
>> >
>> > touch
>> >
>> > any
>> > policy changes at JVM level. You'll register the providers in
>> >
>> > the
>> >
>> > priority
>> >
>> > order and can still have SunJSSE provider and have your custom
>> >
>> > provider
>> >
>> > to
>> >
>> > implement the key and trust managers.
>> >
>> > The ask here is different than KIP-492. We don't have any need
>> >
>> > to
>> >
>> > modify/specify the algorithm parameter. Does that make sense?
>> >
>> > The ask in KIP is introducing new interfaces where the KIP's
>> > goal/motivation can be achieved through the security.provider
>> >
>> > and
>> >
>> > we
>> >
>> > worked
>> > on similar goal without touching any Keystore or Truststore
>> >
>> > interfaces.
>> >
>> > My advise is against changing or introducing new interfaces
>> >
>> > when
>> >
>> > it can
>> >
>> > work through security.provider.
>> >
>> > Thanks,
>> > Harsha
>> >
>> > Thanks
>> >
>> > Maulin
>> >
>> > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
>> >
>> > kafka@harsha.io>
>> >
>> > wrote:
>> >
>> > In your KIP you added security. provider as rejected
>> >
>> > alternative
>> >
>> > and
>> >
>> > specified "its not the correct way". Do you mind explaining
>> >
>> > why
>> >
>> > its
>> >
>> > not? I
>> >
>> > didn't find any evidence in Java docs to say so. Contrary to
>> >
>> > your
>> >
>> > statement
>> >
>> > it does say in the java docs
>> > " However, please note that a provider can be used to
>> >
>> > implement
>> >
>> > any
>> >
>> > security service in Java that uses a pluggable architecture
>> >
>> > with
>> >
>> > a
>> >
>> > choice
>> >
>> > of implementations that fit underneath."
>> >
>> > Java Security Providers have been used by other projects to
>> >
>> > provide
>> >
>> > such
>> >
>> > integration . I am not sure if you looked into Spiffe
>> >
>> > project to
>> >
>> > efficiently distribute certificates but here is an example of
>> >
>> > Java
>> >
>> > provider
>> >
>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>> >
>> >
>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>> >
>> > java which
>> > obtains certificates from local daemons.
>> > These integrations are being used in Tomcat, Jetty etc.. We
>> >
>> > are
>> >
>> > also
>> >
>> > using
>> >
>> > Security provider to do the same in our Kafka clusters. So
>> >
>> > unless I
>> >
>> > see
>> >
>> > more evidence why security.provider doesn't work for you
>> >
>> > adding
>> >
>> > new
>> >
>> > interfaces while there exists more cleaner way of achieving
>> >
>> > the
>> >
>> > goals
>> >
>> > of
>> >
>> > this KIP is unnecessary and breaks the well known security
>> >
>> > interfaces
>> >
>> > provided by Java itself.
>> >
>> > Thanks,
>> > Harsha
>> >
>> > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
>> >
>> > kafka@harsha.io
>> >
>> > wrote:
>> >
>> > Hi Maulin,
>> > Not sure if you looked at my previous replies. This
>> >
>> > changes
>> >
>> > are not required as there is already security Provider to do
>> >
>> > what you
>> >
>> > are
>> >
>> > proposing. This KIP
>> >
>> > https://cwiki.apache.org/confluence/display/KAFKA/
>> >
>> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>> >
>> > also
>> >
>> > addresses easy registration of such providers.
>> >
>> > Thanks,
>> > Harsha
>> >
>> > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
>> >
>> > <maulin.vasavada@gmail.
>> >
>> > com> wrote:
>> >
>> > Bump! Can somebody please review this?
>> >
>> > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
>> >
>> > maulin.vasavada@gmail.com>
>> >
>> > wrote:
>> >
>> > Bump! Can somebody please review this?
>> >
>> > --
>> > Thanks,
>> > M.Sai Sandeep
>> >
>> >
>>
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Maulin Vasavada <ma...@gmail.com>.
Hi Harsha
I've examined the SPIFFE provider more and have one question -
If SPIFFE didn't have a need to do checkSpiffeId() call at the below
location, would you really still write the Provider? *OR*
Would you just use TrustManagerFactory.init(KeyStore) signature to pass the
KeyStore from set of certs returned by spiffeIdManager.getTrustedCerts()?
https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/CertificateUtils.java#L100
/**
> * Validates that the SPIFFE ID is present and matches the SPIFFE ID
> configured in
> * the java.security property ssl.spiffe.accept
> *
> * If the authorized spiffe ids list is empty any spiffe id is authorized
> *
> * @param chain an array of X509Certificate that contains the Peer's SVID
> to be validated
> * @throws CertificateException when either the certificates doesn't have
> a SPIFFE ID or the SPIFFE ID is not authorized
> */
> static void checkSpiffeId(X509Certificate[] chain) throws
> CertificateException {
Thanks
Maulin
On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani <ka...@harsha.io> wrote:
> Maulin,
> The code parts you are pointing are specific for Spiffe and if
> you are talking about validate method which uses PKIX check like any
> other provider does.
> If you want to default to SunJSSE everywhere you can do so by delegating
> the calls in these methods to SunJSSE provider.
>
> TrustManagerFactory tmf = TrustManagerFactory
> .getInstance(TrustManagerFactory.getDefaultAlgorithm());and use
> tmf.chekServerTrusted()
> or use
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/TrustManagerFactory.html#getInstance(java.lang.String)if
> you want a specific provider.
>
> -Harsha
>
>
> On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada <
> maulin.vasavada@gmail.com>
> wrote:
>
> > Okay, so I take that you guys agree that I have to write a 'custom'
> > algorithm and a provider to make it work , correct?
> >
> > Now, for Harsha's comment "Here the 'Custom' Algorithm is not an
> > implementation per say , ..." , I diagree. You can refer to https://
> > github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/
> > SpiffeTrustManager.java#L91 and
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > provider/CertificateUtils.java#L100
> >
> > "that code" is the customization you have for the custom way to check
> > something on top of regular checks. That method is NOT doing custom
> > truststore loading. It is validating/verifying something in the "custom"
> > way with spiffeId.
> > I bet that without that you won't have a need of the custom algorithm in
> > the first place.
> >
> > Let me know if you agree to this.
> >
> > Thanks
> > Maulin
> >
> > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <mp...@gmail.com> wrote:
> >
> > Hi Maulin, thanks for the discussion. As Harsha pointed out, to use the
> > KIP492, you need to create a new provider, register a *new* custom
> > algorithm for your keymanager and trustmanager factory implementations.
> > After this, the kafka server configuration can be done as given below
> >
> > # Register the provider class with custom algorithm, say CUSTOM security.
> > provider.classes=com.company.security.CustomProvider
> > <http://security.provider.classes=com.company.security.customprovider/>
> >
> > # Register the keymanager and trustmanager algorithms
> > # These algorithms indicate that the Keymanager and Trustmanagers
> > registered under the algorithm "CUSTOM" needs to be used
> > ssl.trustmanager.algorithm=CUSTOM
> > ssl.keymanager.algorithm=CUSTOM
> >
> > And the customprovider looks like this...
> >
> > public class CustomProvider extends Provider {
> > public CustomProvider() {
> > super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and TrustStore");
> > super.put("KeyManagerFactory.CUSTOM", "customKeyManagerFactory");
> > super.put("TrustManagerFactory.CUSTOM",
> > "customTrustManagerFactory");
> > }
> > }
> >
> > The PR for this is in review and can be found here: https://github.com/
> > apache/kafka/pull/7090
> > This PR includes the fixed insertProviderAt function call.
> >
> > On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani <ka...@harsha.io>
> > wrote:
> >
> > Answers are in-line
> >
> > On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada <
> maulin.vasavada@gmail.
> > com
> >
> > wrote:
> >
> > Hi Colin
> >
> > When I refer to "standard" or "custom" algorithms I am following Java
> > security Provider Terminology. You can refer to
> > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> > StandardNames.html#TrustManagerFactory link I provided earlier in the
> > emails. It says PKIX is the default Algorithm for TrustManagerFactory.
> >
> > 1. For SPIFFE, I am not sure why you are saying 'it does not implement
> > custom algorithms' because the following file clearly indicates that it
> > does use custom algorithm-
> >
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> >
> > provider/SpiffeProvider.java#L17
> >
> > Algorithm value:
> >
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> >
> > provider/SpiffeProviderConstants.java#L6
> >
> > @Harsha do you want to chime in since you use that provider?
> >
> > Here the "Custom" Algorithm is not an implementation per say , rather
> >
> > used
> >
> > to invoke the custom trust store factory and key manager factory. You are
> > not moving away from "standard" alogrithms that are available.
> >
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > provider/SpiffeTrustManager.java
> >
> > As you can see it delegates all the calls of verification of certificate
> >
> > to
> >
> > the default implementation available.
> > So in our implementation we still use PKIX to verify the certificate
> > chain. So you are not losing anything here and Spiffe is not
> reimplementing
> > the verification process.
> >
> > 2. I already mentioned in my 3rd point, in my previous post, why using
> >
> > ssl.provider does NOT work. I updated KIP-486 in "rejected
> >
> > alternatives"
> >
> > also why ssl.provider does not work.
> >
> > As mentioned before , provider is the right way to go. I am still not
> > understanding the gap is.
> > If I understand correctly your argument is , provider is going to ask to
> > implement a custom algorithm.
> > My answer to that is , no you are not re-implementing the algorithm.
> >
> > Please
> >
> > check the above link , TrustManager implementation it delegates the calls
> > back. There is no need to implement your own here.
> >
> > 3. Security.insertProviderAt() comments were based on assumption if
> >
> > KIP-492
> >
> > changes are done and we use that mechanism to configure providers
> >
> > instead
> >
> > of ssl.provider configuration.
> >
> > KIP-492 has patch available and going through review.
> >
> > Can you read my all the points, I mentioned in my previous post, very
> >
> > carefully? I am covering all the aspects in explaining. I am open to
> >
> > still
> >
> > discuss more to clarify any doubts.
> >
> > "3. If we just use existing ssl.provider kafka configuration then our
> > provider will be used in SSLContext.getInstance(protocol, provider) call
> >
> > in
> >
> > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/> and
> > if our provider does not have
> > implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we
> >
> > tested
> >
> > it). Example: In MyProvider sample above you see that I didn't add
> > SSLContext.TLSv1 as
> > "Service+Algorithm" and that didn't work for me. In SPIFFE provider you
> > don't have this challenge since you are planning to bypass ssl.provider
> >
> > as
> >
> > you mention in the KIP-492."
> >
> > Yes here you need to pass the protocol that your KeyManager/TrustManager
> > registered with and in no way its deviating from TLS RFC spec.
> >
> > https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/
> > javax/net/ssl/SSLContext.java#L134
> >
> > My suggestion here is for you to implement a simple Security Provider as
> > you did before and register a Algorithm. You can use the existing
> > implementation in SpiffeProvider and plug in changes where you need to
> > retrieve the certificates from by making RPC call.
> >
> > Run an end-to-end test with Kafka broker coming up with your provider
> > making calls to RPC call. You do need to pass the "custom algorithm" that
> > you registered in your key manager to invoke the provider. I think your
> > concern is this approach is replacing the existing known ciphersuites and
> > certificate validation provided by java. Which its not.
> >
> > Now test the TLS connection you can do so via openssl -s_client options
> >
> > to
> >
> > test if everything else is working.
> >
> > I am happy to share configs that we used if you are interested.
> >
> > Thanks,
> > Harsha
> >
> > Thanks
> > Maulin
> >
> > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <cm...@apache.org>
> >
> > wrote:
> >
> > Hi Maulin,
> >
> > A lot of JSSE providers don't implement custom algorithms. Spire is a
> >
> > good
> >
> > example of a JSSE provider that doesn't, and yet is still useful to
> >
> > many
> >
> > people. Your JSSE provider can work fine even if it doesn't implement a
> > custom algorithm.
> >
> > Maybe I'm missing something, but I don't understand the discussion of
> > Security.insertProviderAt() that you included. SslEngineBuilder doesn't
> >
> > use
> >
> > that API to get the security provider. Instead, it calls
> > "SSLContext.getInstance(protocol, provider)", where provider is the
> >
> > name
> >
> > of the provider.
> >
> > best,
> > Colin
> >
> > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
> >
> > On top of everything above I feel strongly to add the 4th point which
> >
> > is
> >
> > based on Java APIs for TrustManagerFactory.init(KeyStore) (
> >
> > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> > TrustManagerFactory.html#init(java.security.KeyStore
> > <http://java.security.keystore/>)
> > )
> >
> > and KeyManagerFactory.init(KeyStore, char[]) (
> >
> >
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory.
> >
> >
> > html#init(java.security.KeyStore <http://java.security.keystore/
> >,%20char[])
> >
> >
> > ).
> >
> > 4. The above APIs are intended to support providing "trust/key
> >
> > material"
> >
> > from the user without having to write their own
> >
> > TrustManager/KeyManagers.
> >
> > To quote from the TrustManagerFactory.init()'s documentation
> >
> > "Initializes
> >
> > this factory with a source of certificate authorities and related trust
> > material."
> > To quote from the KeyManagerFactory.init()'s documentation "Initializes
> > this factory with a source of key material."
> >
> > Based on this it is clear that there is a flexibility provided by Java
> >
> > to
> >
> > to enable developers to provide the required trust/key material loaded
> >
> > from
> >
> > "anywhere" without requiring them to write custom provider OR trust/key
> > managers. This same flexibility is reflected in Kafka code also where
> >
> > it
> >
> > loads the trust/keys from a local file and doesn't require writing a
> > Provider necessarily. If we do NOT have a custom algorithm, it makes
> >
> > less
> >
> > sense to write a Provider.
> >
> > Thanks
> > Maulin
> >
> > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
> >
> > maulin.vasavada@gmail.com>
> >
> > wrote:
> >
> > Hi Harsha/Colin
> >
> > I did the sample with a custom Provider for TrustStoreManager and tried
> > using ssl.provider Kafka config AND the way KIP-492 is suggesting (by
> > adding Provider programmatically instead of relying on
> >
> > ssl.provider+java.
> >
> > security. The below sample is followed by my detailed findings. I'll
> > appreciate if you can go through it carefully and see
> >
> > if you
> >
> > see my point.
> >
> > package providertest;
> >
> > import java.security.Provider <http://java.security.provider/> <http://
> > java.security.provider/>;
> >
> > public class MyProvider extends Provider {
> >
> > private static final String name = "MyProvider"; private static double
> > version = 1.0d;
> > private static String info = "Maulin's SSL Provider v"+version;
> >
> > public MyProvider() {
> > super(name, version, info);
> > this.put("TrustManagerFactory.PKIX",
> >
> > "providertest.MyTrustManagerFactory");
> >
> > }
> > }
> >
> > *Details:*
> >
> > KIP-492 documents that it will use Security.addProvider() assuming it
> >
> > will
> >
> > add it as position '0' which is not a correct assumption. The
> > addProvider()'s documentation says it will add it to the last available
> > position. You may want to correct that to say
> > Security.insertProviderAt(provider, 1).
> >
> > Now coming back to our specific discussion,
> >
> > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add
> >
> > that
> >
> > provider in the provider list via Security.addProvider() the position
> >
> > where
> >
> > it gets added doesn't matter (even if you don't end up adding it as
> >
> > first
> >
> > entry) since that is the ONLY provider for SPIFFE specific algorithm
> >
> > you
> >
> > might have.
> >
> > We do *not* have custom algorithm for Key/Trust StoreMangers. Which
> >
> > means
> >
> > we have to use X509, PKIX etc "Standard Algorithms" ((
> >
> > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> > StandardNames.html
> > ))
> >
> > in our provider to override the TrustStoreManager (see my sample code)
> >
> > and
> >
> > KeyStoreManger and KeyManager. This creates another challenge
> >
> > mentioned in
> >
> > the below point.
> >
> > 2. In order to make our Provider for loading custom TrustStore work, we
> > have to add the provider as 'first' in the list since there are others
> >
> > with
> >
> > the same algorithm.
> >
> > However, the programatic way of adding provider
> > (Security.insertProviderAt()) is *not* deterministic for ordering since
> > different code can call the method for a different provider and
> >
> > depending
> >
> > upon the order of the call our provider can be first or pushed down the
> > list. This can happen very well in any client application using Kafka.
> >
> > This
> >
> > is specially problematic for a case when you want to guarantee order
> >
> > for a
> >
> > Provider having "Standard Algorithms".
> >
> > If we add our provider in java.security file that definitely guarantees
> > the order(unless somebody calls removeProvider() which is less
> >
> > likely). But
> >
> > if we add our provider in java.security file it will defeat the
> >
> > purpose of
> >
> > the KIP-492.
> >
> > In the gist - Apache Kafka must not rely on "un-deterministic" method
> >
> > to
> >
> > rely on Provider ordering.
> >
> > 3. If we just use existing ssl.provider kafka configuration then our
> > provider will be used in SSLContext.getInstance(protocol, provider)
> >
> > call in
> >
> > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/> and
> > if our provider does not have implementation for
> > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks
> >
> > (we
> >
> > tested it). Example:
> >
> > In
> >
> > MyProvider sample above you see that I didn't add SSLContext.TLSv1 as
> > "Service+Algorithm" and that didn't work for me. In SPIFFE provider you
> > don't have this challenge since you are planning to bypass
> >
> > ssl.provider as
> >
> > you mention in the KIP-492.
> >
> > *Overall summary,*
> >
> > 1. Any provider based mechanisms- a) existing ssl.provider and
> >
> > b)KIP-492,
> >
> > for loading key/trust store using "Standard Algorithms" do not work
> >
> > 2. Approach suggested in our KIP-486 works without any issue and it is
> > *not* our context specific solve
> >
> > 3. Based on above we feel KIP-492 and KIP-486 are complimentary changes
> > and not contradicting or redundent.
> >
> > If you want we can do a joint session somehow to walk through the
> >
> > sample I
> >
> > have and various experiments I did. I would encourage you to do similar
> > exercise by writing a Provider for "Standard Algorithm" for
> > TrustStoreManager (like our needs) and see what you find since only
> >
> > writing
> >
> > samples can bring out the complexity/challenges we face.
> >
> > Thanks
> > Maulin
> >
> > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <
> >
> > maulin.vasavada@gmail.
> >
> > com> wrote:
> >
> > Just to update - still working on it. Get to work only on and off on
> >
> > it :(
> >
> > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
> >
> > maulin.vasavada@gmail.com>
> >
> > wrote:
> >
> > Hi Harsha
> >
> > Let me try to write samples and will let you know.
> >
> > Thanks
> > Maulin
> >
> > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <ha...@gmail.com>
> >
> > wrote:
> >
> > Hi Maulin,
> > With java security providers can be as custom you would
> >
> > like
> >
> > it to
> > be. If you only want to to implement a custom way of loading the
> >
> > keystore
> >
> > and truststore and not implement any protocol/encryption handling you can
> > leave them empty and no need to implement. Have you looked into the
> links I
> > pasted before?
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> >
> > java
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > spiffe-security-provider/src/main/java/spiffe/api/provider/
> > SpiffeTrustManager.java <http://spiffetrustmanager.java/>
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> >
> > java
> >
> > Can you please tell me which methods are too complex in above to
> >
> > implement
> >
> > or unnecessary? You are changing anything in SSL/TLS implementations
> > provided by
> >
> > All of the implementations delegating the checks to the default
> > implementation anyway.
> > Spire agent is an example, its nothing but a GRPC server listening
> >
> > on a
> >
> > unix domain socket . Above code is making a RPC call to the local
> >
> > daemon
> >
> > to
> > get the certificate and keys. The mechanics are pretty much same as
> >
> > what
> >
> > you are asking for.
> >
> > Thanks,
> > Harsha
> >
> > On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
> >
> > maulin.vasavada@gmail.com>
> >
> > wrote:
> >
> > Imagine a scenario like - We know we have a custom KMS and as a
> >
> > Kafka
> >
> > owner
> >
> > we want to comply to using that KMS source to load keys/certs. As
> >
> > a
> >
> > Kafka
> >
> > owner we know how to integrate with KMS but doesn't necessarily
> >
> > have
> >
> > to
> >
> > know anything about cipher suites, algorithms, and SSL/TLS
> >
> > implementation.
> >
> > Going the Provider way requires to know lot more than we should,
> >
> > isn't it?
> >
> > Not that we would have concern/shy-away knowing those details -
> >
> > but
> >
> > if we
> >
> > don't have to - why should we?
> >
> > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
> >
> > maulin.vasavada@gmail.com>
> >
> > wrote:
> >
> > Hi Harsha
> >
> > We don't have spire (or similar) agents and we do not have
> >
> > keys/certs
> >
> > locally on any brokers. To elaborate more on my previous email,
> >
> > I agree that Java security Providers are used in much broader
> >
> > sense
> >
> > - to
> >
> > have a particular implementation of an algorithm, use specific
> >
> > cipher
> >
> > suites for SSL , OR in our current team's case have a
> >
> > particular
> >
> > way to
> >
> > leverage pre-generated SSL sessions. However, the scope of our
> >
> > KIP
> >
> > (486)
> >
> > is
> >
> > much restricted than that. We merely intend to provide a custom
> > keystore/truststore for our SSL connections and not really worry
> >
> > about
> >
> > underlying specific SSL/TLS implementation. This simplifies it
> >
> > a
> >
> > lot for
> >
> > us to keep the concerns separate and clear.
> >
> > I feel our approach is more complimentary such that it allows
> >
> > for
> >
> > using
> >
> > keystores of choice while retaining the flexibility to use any
> > underlying/available Provider for actually making the SSL call.
> >
> > We agree with KIP-492's approach based on Providers (and Java's
> > recommendation), but also strongly believe that our approach can
> >
> > compliment
> >
> > it very effectively for reasons explained above.
> >
> > Thanks
> > Maulin
> >
> > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <
> >
> > kafka@harsha.io
> >
> > wrote:
> >
> > Hi Maulin,
> >
> > On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
> >
> > maulin.vasavada@gmail.
> >
> > com>
> > wrote:
> >
> > Hi Harsha
> >
> > The reason we rejected the SslProvider route is that - we
> >
> > only
> >
> > needed
> >
> > a
> >
> > custom way to load keys/certs. Not touch any policy that
> >
> > existing
> >
> > Providers
> >
> > govern like SunJSSE Provider.
> >
> > We have exactly the same requirements to load certs and keys
> >
> > through
> >
> > spire
> >
> > agent. We used security.provider to do that exactly. I am not
> >
> > sure
> >
> > why
> >
> > you
> >
> > would be modifying any policies provided by default SunJSSE
> >
> > provider.
> >
> > Can
> >
> > you give me an example of having custom provider that will
> >
> > override an
> >
> > existing policy in SunJSSE provider.
> >
> > As pointed out earlier, this kip
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/
> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> >
> > allows
> > you to load security.provider through config.
> > Take a look at the examples I gave before
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> >
> > java
> >
> > It registers KeyManagerFactory and TrustManagerFactory and
> >
> > Keystore
> >
> > algorithm.
> > Implement your custom way of loading Keystore in here
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> >
> > java
> >
> > and Trust manager like here
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > spiffe-security-provider/src/main/java/spiffe/api/provider/
> > SpiffeTrustManager.java <http://spiffetrustmanager.java/>
> >
> > In your Kafka client you can set the security.provider to your
> >
> > custom
> >
> > implementation and with this fix
> > https://issues.apache.org/jira/browse/KAFKA-8191 you can set
> > keyManagerAlgorigthm and trustManagerAlgorithm configs.
> >
> > All of this is in your clients and broker side and do not need
> >
> > to
> >
> > touch
> >
> > any
> > policy changes at JVM level. You'll register the providers in
> >
> > the
> >
> > priority
> >
> > order and can still have SunJSSE provider and have your custom
> >
> > provider
> >
> > to
> >
> > implement the key and trust managers.
> >
> > The ask here is different than KIP-492. We don't have any need
> >
> > to
> >
> > modify/specify the algorithm parameter. Does that make sense?
> >
> > The ask in KIP is introducing new interfaces where the KIP's
> > goal/motivation can be achieved through the security.provider
> >
> > and
> >
> > we
> >
> > worked
> > on similar goal without touching any Keystore or Truststore
> >
> > interfaces.
> >
> > My advise is against changing or introducing new interfaces
> >
> > when
> >
> > it can
> >
> > work through security.provider.
> >
> > Thanks,
> > Harsha
> >
> > Thanks
> >
> > Maulin
> >
> > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
> >
> > kafka@harsha.io>
> >
> > wrote:
> >
> > In your KIP you added security. provider as rejected
> >
> > alternative
> >
> > and
> >
> > specified "its not the correct way". Do you mind explaining
> >
> > why
> >
> > its
> >
> > not? I
> >
> > didn't find any evidence in Java docs to say so. Contrary to
> >
> > your
> >
> > statement
> >
> > it does say in the java docs
> > " However, please note that a provider can be used to
> >
> > implement
> >
> > any
> >
> > security service in Java that uses a pluggable architecture
> >
> > with
> >
> > a
> >
> > choice
> >
> > of implementations that fit underneath."
> >
> > Java Security Providers have been used by other projects to
> >
> > provide
> >
> > such
> >
> > integration . I am not sure if you looked into Spiffe
> >
> > project to
> >
> > efficiently distribute certificates but here is an example of
> >
> > Java
> >
> > provider
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> >
> > java which
> > obtains certificates from local daemons.
> > These integrations are being used in Tomcat, Jetty etc.. We
> >
> > are
> >
> > also
> >
> > using
> >
> > Security provider to do the same in our Kafka clusters. So
> >
> > unless I
> >
> > see
> >
> > more evidence why security.provider doesn't work for you
> >
> > adding
> >
> > new
> >
> > interfaces while there exists more cleaner way of achieving
> >
> > the
> >
> > goals
> >
> > of
> >
> > this KIP is unnecessary and breaks the well known security
> >
> > interfaces
> >
> > provided by Java itself.
> >
> > Thanks,
> > Harsha
> >
> > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
> >
> > kafka@harsha.io
> >
> > wrote:
> >
> > Hi Maulin,
> > Not sure if you looked at my previous replies. This
> >
> > changes
> >
> > are not required as there is already security Provider to do
> >
> > what you
> >
> > are
> >
> > proposing. This KIP
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/
> >
> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> >
> > also
> >
> > addresses easy registration of such providers.
> >
> > Thanks,
> > Harsha
> >
> > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
> >
> > <maulin.vasavada@gmail.
> >
> > com> wrote:
> >
> > Bump! Can somebody please review this?
> >
> > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
> >
> > maulin.vasavada@gmail.com>
> >
> > wrote:
> >
> > Bump! Can somebody please review this?
> >
> > --
> > Thanks,
> > M.Sai Sandeep
> >
> >
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Harsha Chintalapani <ka...@harsha.io>.
Maulin,
The code parts you are pointing are specific for Spiffe and if
you are talking about validate method which uses PKIX check like any
other provider does.
If you want to default to SunJSSE everywhere you can do so by delegating
the calls in these methods to SunJSSE provider.
TrustManagerFactory tmf = TrustManagerFactory
.getInstance(TrustManagerFactory.getDefaultAlgorithm());and use
tmf.chekServerTrusted()
or use https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/TrustManagerFactory.html#getInstance(java.lang.String)if
you want a specific provider.
-Harsha
On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada <ma...@gmail.com>
wrote:
> Okay, so I take that you guys agree that I have to write a 'custom'
> algorithm and a provider to make it work , correct?
>
> Now, for Harsha's comment "Here the 'Custom' Algorithm is not an
> implementation per say , ..." , I diagree. You can refer to https://
> github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/
> SpiffeTrustManager.java#L91 and
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> provider/CertificateUtils.java#L100
>
> "that code" is the customization you have for the custom way to check
> something on top of regular checks. That method is NOT doing custom
> truststore loading. It is validating/verifying something in the "custom"
> way with spiffeId.
> I bet that without that you won't have a need of the custom algorithm in
> the first place.
>
> Let me know if you agree to this.
>
> Thanks
> Maulin
>
> On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <mp...@gmail.com> wrote:
>
> Hi Maulin, thanks for the discussion. As Harsha pointed out, to use the
> KIP492, you need to create a new provider, register a *new* custom
> algorithm for your keymanager and trustmanager factory implementations.
> After this, the kafka server configuration can be done as given below
>
> # Register the provider class with custom algorithm, say CUSTOM security.
> provider.classes=com.company.security.CustomProvider
> <http://security.provider.classes=com.company.security.customprovider/>
>
> # Register the keymanager and trustmanager algorithms
> # These algorithms indicate that the Keymanager and Trustmanagers
> registered under the algorithm "CUSTOM" needs to be used
> ssl.trustmanager.algorithm=CUSTOM
> ssl.keymanager.algorithm=CUSTOM
>
> And the customprovider looks like this...
>
> public class CustomProvider extends Provider {
> public CustomProvider() {
> super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and TrustStore");
> super.put("KeyManagerFactory.CUSTOM", "customKeyManagerFactory");
> super.put("TrustManagerFactory.CUSTOM",
> "customTrustManagerFactory");
> }
> }
>
> The PR for this is in review and can be found here: https://github.com/
> apache/kafka/pull/7090
> This PR includes the fixed insertProviderAt function call.
>
> On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani <ka...@harsha.io>
> wrote:
>
> Answers are in-line
>
> On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada < maulin.vasavada@gmail.
> com
>
> wrote:
>
> Hi Colin
>
> When I refer to "standard" or "custom" algorithms I am following Java
> security Provider Terminology. You can refer to
> https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> StandardNames.html#TrustManagerFactory link I provided earlier in the
> emails. It says PKIX is the default Algorithm for TrustManagerFactory.
>
> 1. For SPIFFE, I am not sure why you are saying 'it does not implement
> custom algorithms' because the following file clearly indicates that it
> does use custom algorithm-
>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>
> provider/SpiffeProvider.java#L17
>
> Algorithm value:
>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
>
> provider/SpiffeProviderConstants.java#L6
>
> @Harsha do you want to chime in since you use that provider?
>
> Here the "Custom" Algorithm is not an implementation per say , rather
>
> used
>
> to invoke the custom trust store factory and key manager factory. You are
> not moving away from "standard" alogrithms that are available.
>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> provider/SpiffeTrustManager.java
>
> As you can see it delegates all the calls of verification of certificate
>
> to
>
> the default implementation available.
> So in our implementation we still use PKIX to verify the certificate
> chain. So you are not losing anything here and Spiffe is not reimplementing
> the verification process.
>
> 2. I already mentioned in my 3rd point, in my previous post, why using
>
> ssl.provider does NOT work. I updated KIP-486 in "rejected
>
> alternatives"
>
> also why ssl.provider does not work.
>
> As mentioned before , provider is the right way to go. I am still not
> understanding the gap is.
> If I understand correctly your argument is , provider is going to ask to
> implement a custom algorithm.
> My answer to that is , no you are not re-implementing the algorithm.
>
> Please
>
> check the above link , TrustManager implementation it delegates the calls
> back. There is no need to implement your own here.
>
> 3. Security.insertProviderAt() comments were based on assumption if
>
> KIP-492
>
> changes are done and we use that mechanism to configure providers
>
> instead
>
> of ssl.provider configuration.
>
> KIP-492 has patch available and going through review.
>
> Can you read my all the points, I mentioned in my previous post, very
>
> carefully? I am covering all the aspects in explaining. I am open to
>
> still
>
> discuss more to clarify any doubts.
>
> "3. If we just use existing ssl.provider kafka configuration then our
> provider will be used in SSLContext.getInstance(protocol, provider) call
>
> in
>
> SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/> and
> if our provider does not have
> implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we
>
> tested
>
> it). Example: In MyProvider sample above you see that I didn't add
> SSLContext.TLSv1 as
> "Service+Algorithm" and that didn't work for me. In SPIFFE provider you
> don't have this challenge since you are planning to bypass ssl.provider
>
> as
>
> you mention in the KIP-492."
>
> Yes here you need to pass the protocol that your KeyManager/TrustManager
> registered with and in no way its deviating from TLS RFC spec.
>
> https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/
> javax/net/ssl/SSLContext.java#L134
>
> My suggestion here is for you to implement a simple Security Provider as
> you did before and register a Algorithm. You can use the existing
> implementation in SpiffeProvider and plug in changes where you need to
> retrieve the certificates from by making RPC call.
>
> Run an end-to-end test with Kafka broker coming up with your provider
> making calls to RPC call. You do need to pass the "custom algorithm" that
> you registered in your key manager to invoke the provider. I think your
> concern is this approach is replacing the existing known ciphersuites and
> certificate validation provided by java. Which its not.
>
> Now test the TLS connection you can do so via openssl -s_client options
>
> to
>
> test if everything else is working.
>
> I am happy to share configs that we used if you are interested.
>
> Thanks,
> Harsha
>
> Thanks
> Maulin
>
> On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <cm...@apache.org>
>
> wrote:
>
> Hi Maulin,
>
> A lot of JSSE providers don't implement custom algorithms. Spire is a
>
> good
>
> example of a JSSE provider that doesn't, and yet is still useful to
>
> many
>
> people. Your JSSE provider can work fine even if it doesn't implement a
> custom algorithm.
>
> Maybe I'm missing something, but I don't understand the discussion of
> Security.insertProviderAt() that you included. SslEngineBuilder doesn't
>
> use
>
> that API to get the security provider. Instead, it calls
> "SSLContext.getInstance(protocol, provider)", where provider is the
>
> name
>
> of the provider.
>
> best,
> Colin
>
> On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
>
> On top of everything above I feel strongly to add the 4th point which
>
> is
>
> based on Java APIs for TrustManagerFactory.init(KeyStore) (
>
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> TrustManagerFactory.html#init(java.security.KeyStore
> <http://java.security.keystore/>)
> )
>
> and KeyManagerFactory.init(KeyStore, char[]) (
>
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory.
>
>
> html#init(java.security.KeyStore <http://java.security.keystore/>,%20char[])
>
>
> ).
>
> 4. The above APIs are intended to support providing "trust/key
>
> material"
>
> from the user without having to write their own
>
> TrustManager/KeyManagers.
>
> To quote from the TrustManagerFactory.init()'s documentation
>
> "Initializes
>
> this factory with a source of certificate authorities and related trust
> material."
> To quote from the KeyManagerFactory.init()'s documentation "Initializes
> this factory with a source of key material."
>
> Based on this it is clear that there is a flexibility provided by Java
>
> to
>
> to enable developers to provide the required trust/key material loaded
>
> from
>
> "anywhere" without requiring them to write custom provider OR trust/key
> managers. This same flexibility is reflected in Kafka code also where
>
> it
>
> loads the trust/keys from a local file and doesn't require writing a
> Provider necessarily. If we do NOT have a custom algorithm, it makes
>
> less
>
> sense to write a Provider.
>
> Thanks
> Maulin
>
> On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
>
> maulin.vasavada@gmail.com>
>
> wrote:
>
> Hi Harsha/Colin
>
> I did the sample with a custom Provider for TrustStoreManager and tried
> using ssl.provider Kafka config AND the way KIP-492 is suggesting (by
> adding Provider programmatically instead of relying on
>
> ssl.provider+java.
>
> security. The below sample is followed by my detailed findings. I'll
> appreciate if you can go through it carefully and see
>
> if you
>
> see my point.
>
> package providertest;
>
> import java.security.Provider <http://java.security.provider/> <http://
> java.security.provider/>;
>
> public class MyProvider extends Provider {
>
> private static final String name = "MyProvider"; private static double
> version = 1.0d;
> private static String info = "Maulin's SSL Provider v"+version;
>
> public MyProvider() {
> super(name, version, info);
> this.put("TrustManagerFactory.PKIX",
>
> "providertest.MyTrustManagerFactory");
>
> }
> }
>
> *Details:*
>
> KIP-492 documents that it will use Security.addProvider() assuming it
>
> will
>
> add it as position '0' which is not a correct assumption. The
> addProvider()'s documentation says it will add it to the last available
> position. You may want to correct that to say
> Security.insertProviderAt(provider, 1).
>
> Now coming back to our specific discussion,
>
> 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add
>
> that
>
> provider in the provider list via Security.addProvider() the position
>
> where
>
> it gets added doesn't matter (even if you don't end up adding it as
>
> first
>
> entry) since that is the ONLY provider for SPIFFE specific algorithm
>
> you
>
> might have.
>
> We do *not* have custom algorithm for Key/Trust StoreMangers. Which
>
> means
>
> we have to use X509, PKIX etc "Standard Algorithms" ((
>
> https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> StandardNames.html
> ))
>
> in our provider to override the TrustStoreManager (see my sample code)
>
> and
>
> KeyStoreManger and KeyManager. This creates another challenge
>
> mentioned in
>
> the below point.
>
> 2. In order to make our Provider for loading custom TrustStore work, we
> have to add the provider as 'first' in the list since there are others
>
> with
>
> the same algorithm.
>
> However, the programatic way of adding provider
> (Security.insertProviderAt()) is *not* deterministic for ordering since
> different code can call the method for a different provider and
>
> depending
>
> upon the order of the call our provider can be first or pushed down the
> list. This can happen very well in any client application using Kafka.
>
> This
>
> is specially problematic for a case when you want to guarantee order
>
> for a
>
> Provider having "Standard Algorithms".
>
> If we add our provider in java.security file that definitely guarantees
> the order(unless somebody calls removeProvider() which is less
>
> likely). But
>
> if we add our provider in java.security file it will defeat the
>
> purpose of
>
> the KIP-492.
>
> In the gist - Apache Kafka must not rely on "un-deterministic" method
>
> to
>
> rely on Provider ordering.
>
> 3. If we just use existing ssl.provider kafka configuration then our
> provider will be used in SSLContext.getInstance(protocol, provider)
>
> call in
>
> SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/> and
> if our provider does not have implementation for
> SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks
>
> (we
>
> tested it). Example:
>
> In
>
> MyProvider sample above you see that I didn't add SSLContext.TLSv1 as
> "Service+Algorithm" and that didn't work for me. In SPIFFE provider you
> don't have this challenge since you are planning to bypass
>
> ssl.provider as
>
> you mention in the KIP-492.
>
> *Overall summary,*
>
> 1. Any provider based mechanisms- a) existing ssl.provider and
>
> b)KIP-492,
>
> for loading key/trust store using "Standard Algorithms" do not work
>
> 2. Approach suggested in our KIP-486 works without any issue and it is
> *not* our context specific solve
>
> 3. Based on above we feel KIP-492 and KIP-486 are complimentary changes
> and not contradicting or redundent.
>
> If you want we can do a joint session somehow to walk through the
>
> sample I
>
> have and various experiments I did. I would encourage you to do similar
> exercise by writing a Provider for "Standard Algorithm" for
> TrustStoreManager (like our needs) and see what you find since only
>
> writing
>
> samples can bring out the complexity/challenges we face.
>
> Thanks
> Maulin
>
> On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <
>
> maulin.vasavada@gmail.
>
> com> wrote:
>
> Just to update - still working on it. Get to work only on and off on
>
> it :(
>
> On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
>
> maulin.vasavada@gmail.com>
>
> wrote:
>
> Hi Harsha
>
> Let me try to write samples and will let you know.
>
> Thanks
> Maulin
>
> On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <ha...@gmail.com>
>
> wrote:
>
> Hi Maulin,
> With java security providers can be as custom you would
>
> like
>
> it to
> be. If you only want to to implement a custom way of loading the
>
> keystore
>
> and truststore and not implement any protocol/encryption handling you can
> leave them empty and no need to implement. Have you looked into the links I
> pasted before?
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
>
> java
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> spiffe-security-provider/src/main/java/spiffe/api/provider/
> SpiffeTrustManager.java <http://spiffetrustmanager.java/>
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>
> java
>
> Can you please tell me which methods are too complex in above to
>
> implement
>
> or unnecessary? You are changing anything in SSL/TLS implementations
> provided by
>
> All of the implementations delegating the checks to the default
> implementation anyway.
> Spire agent is an example, its nothing but a GRPC server listening
>
> on a
>
> unix domain socket . Above code is making a RPC call to the local
>
> daemon
>
> to
> get the certificate and keys. The mechanics are pretty much same as
>
> what
>
> you are asking for.
>
> Thanks,
> Harsha
>
> On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
>
> maulin.vasavada@gmail.com>
>
> wrote:
>
> Imagine a scenario like - We know we have a custom KMS and as a
>
> Kafka
>
> owner
>
> we want to comply to using that KMS source to load keys/certs. As
>
> a
>
> Kafka
>
> owner we know how to integrate with KMS but doesn't necessarily
>
> have
>
> to
>
> know anything about cipher suites, algorithms, and SSL/TLS
>
> implementation.
>
> Going the Provider way requires to know lot more than we should,
>
> isn't it?
>
> Not that we would have concern/shy-away knowing those details -
>
> but
>
> if we
>
> don't have to - why should we?
>
> On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
>
> maulin.vasavada@gmail.com>
>
> wrote:
>
> Hi Harsha
>
> We don't have spire (or similar) agents and we do not have
>
> keys/certs
>
> locally on any brokers. To elaborate more on my previous email,
>
> I agree that Java security Providers are used in much broader
>
> sense
>
> - to
>
> have a particular implementation of an algorithm, use specific
>
> cipher
>
> suites for SSL , OR in our current team's case have a
>
> particular
>
> way to
>
> leverage pre-generated SSL sessions. However, the scope of our
>
> KIP
>
> (486)
>
> is
>
> much restricted than that. We merely intend to provide a custom
> keystore/truststore for our SSL connections and not really worry
>
> about
>
> underlying specific SSL/TLS implementation. This simplifies it
>
> a
>
> lot for
>
> us to keep the concerns separate and clear.
>
> I feel our approach is more complimentary such that it allows
>
> for
>
> using
>
> keystores of choice while retaining the flexibility to use any
> underlying/available Provider for actually making the SSL call.
>
> We agree with KIP-492's approach based on Providers (and Java's
> recommendation), but also strongly believe that our approach can
>
> compliment
>
> it very effectively for reasons explained above.
>
> Thanks
> Maulin
>
> On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <
>
> kafka@harsha.io
>
> wrote:
>
> Hi Maulin,
>
> On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
>
> maulin.vasavada@gmail.
>
> com>
> wrote:
>
> Hi Harsha
>
> The reason we rejected the SslProvider route is that - we
>
> only
>
> needed
>
> a
>
> custom way to load keys/certs. Not touch any policy that
>
> existing
>
> Providers
>
> govern like SunJSSE Provider.
>
> We have exactly the same requirements to load certs and keys
>
> through
>
> spire
>
> agent. We used security.provider to do that exactly. I am not
>
> sure
>
> why
>
> you
>
> would be modifying any policies provided by default SunJSSE
>
> provider.
>
> Can
>
> you give me an example of having custom provider that will
>
> override an
>
> existing policy in SunJSSE provider.
>
> As pointed out earlier, this kip
>
> https://cwiki.apache.org/confluence/display/KAFKA/
> KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>
> allows
> you to load security.provider through config.
> Take a look at the examples I gave before
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>
> java
>
> It registers KeyManagerFactory and TrustManagerFactory and
>
> Keystore
>
> algorithm.
> Implement your custom way of loading Keystore in here
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
>
> java
>
> and Trust manager like here
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> spiffe-security-provider/src/main/java/spiffe/api/provider/
> SpiffeTrustManager.java <http://spiffetrustmanager.java/>
>
> In your Kafka client you can set the security.provider to your
>
> custom
>
> implementation and with this fix
> https://issues.apache.org/jira/browse/KAFKA-8191 you can set
> keyManagerAlgorigthm and trustManagerAlgorithm configs.
>
> All of this is in your clients and broker side and do not need
>
> to
>
> touch
>
> any
> policy changes at JVM level. You'll register the providers in
>
> the
>
> priority
>
> order and can still have SunJSSE provider and have your custom
>
> provider
>
> to
>
> implement the key and trust managers.
>
> The ask here is different than KIP-492. We don't have any need
>
> to
>
> modify/specify the algorithm parameter. Does that make sense?
>
> The ask in KIP is introducing new interfaces where the KIP's
> goal/motivation can be achieved through the security.provider
>
> and
>
> we
>
> worked
> on similar goal without touching any Keystore or Truststore
>
> interfaces.
>
> My advise is against changing or introducing new interfaces
>
> when
>
> it can
>
> work through security.provider.
>
> Thanks,
> Harsha
>
> Thanks
>
> Maulin
>
> On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
>
> kafka@harsha.io>
>
> wrote:
>
> In your KIP you added security. provider as rejected
>
> alternative
>
> and
>
> specified "its not the correct way". Do you mind explaining
>
> why
>
> its
>
> not? I
>
> didn't find any evidence in Java docs to say so. Contrary to
>
> your
>
> statement
>
> it does say in the java docs
> " However, please note that a provider can be used to
>
> implement
>
> any
>
> security service in Java that uses a pluggable architecture
>
> with
>
> a
>
> choice
>
> of implementations that fit underneath."
>
> Java Security Providers have been used by other projects to
>
> provide
>
> such
>
> integration . I am not sure if you looked into Spiffe
>
> project to
>
> efficiently distribute certificates but here is an example of
>
> Java
>
> provider
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>
> java which
> obtains certificates from local daemons.
> These integrations are being used in Tomcat, Jetty etc.. We
>
> are
>
> also
>
> using
>
> Security provider to do the same in our Kafka clusters. So
>
> unless I
>
> see
>
> more evidence why security.provider doesn't work for you
>
> adding
>
> new
>
> interfaces while there exists more cleaner way of achieving
>
> the
>
> goals
>
> of
>
> this KIP is unnecessary and breaks the well known security
>
> interfaces
>
> provided by Java itself.
>
> Thanks,
> Harsha
>
> On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
>
> kafka@harsha.io
>
> wrote:
>
> Hi Maulin,
> Not sure if you looked at my previous replies. This
>
> changes
>
> are not required as there is already security Provider to do
>
> what you
>
> are
>
> proposing. This KIP
>
> https://cwiki.apache.org/confluence/display/KAFKA/
>
> KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>
> also
>
> addresses easy registration of such providers.
>
> Thanks,
> Harsha
>
> On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
>
> <maulin.vasavada@gmail.
>
> com> wrote:
>
> Bump! Can somebody please review this?
>
> On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
>
> maulin.vasavada@gmail.com>
>
> wrote:
>
> Bump! Can somebody please review this?
>
> --
> Thanks,
> M.Sai Sandeep
>
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Maulin Vasavada <ma...@gmail.com>.
Okay, so I take that you guys agree that I have to write a 'custom'
algorithm and a provider to make it work , correct?
Now, for Harsha's comment "Here the 'Custom' Algorithm is not an
implementation per say , ..." , I diagree. You can refer to
https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/SpiffeTrustManager.java#L91
and
https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/CertificateUtils.java#L100
"that code" is the customization you have for the custom way to check
something on top of regular checks. That method is NOT doing custom
truststore loading. It is validating/verifying something in the "custom"
way with spiffeId.
I bet that without that you won't have a need of the custom algorithm in
the first place.
Let me know if you agree to this.
Thanks
Maulin
On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <mp...@gmail.com> wrote:
> Hi Maulin, thanks for the discussion. As Harsha pointed out, to use the
> KIP492, you need to create a new provider, register a *new* custom
> algorithm for your keymanager and trustmanager factory implementations.
> After this, the kafka server configuration can be done as given below
>
> # Register the provider class with custom algorithm, say CUSTOM
> security.provider.classes=com.company.security.CustomProvider
>
> # Register the keymanager and trustmanager algorithms
> # These algorithms indicate that the Keymanager and Trustmanagers
> registered under the algorithm "CUSTOM" needs to be used
> ssl.trustmanager.algorithm=CUSTOM
> ssl.keymanager.algorithm=CUSTOM
>
>
> And the customprovider looks like this...
>
> public class CustomProvider extends Provider {
> public CustomProvider() {
> super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and
> TrustStore");
> super.put("KeyManagerFactory.CUSTOM", "customKeyManagerFactory");
> super.put("TrustManagerFactory.CUSTOM",
> "customTrustManagerFactory");
> }
> }
>
>
> The PR for this is in review and can be found here:
> https://github.com/apache/kafka/pull/7090
> This PR includes the fixed insertProviderAt function call.
>
> On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani <ka...@harsha.io>
> wrote:
>
> > Answers are in-line
> >
> >
> >
> > On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada <
> > maulin.vasavada@gmail.com
> > > wrote:
> >
> > > Hi Colin
> > >
> > >
> > > When I refer to "standard" or "custom" algorithms I am following Java
> > > security Provider Terminology. You can refer to
> > > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> > > StandardNames.html#TrustManagerFactory link I provided earlier in the
> > > emails. It says PKIX is the default Algorithm for TrustManagerFactory.
> > >
> > >
> > > 1. For SPIFFE, I am not sure why you are saying 'it does not implement
> > > custom algorithms' because the following file clearly indicates that it
> > > does use custom algorithm-
> > >
> > >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > provider/SpiffeProvider.java#L17
> > >
> > >
> > > Algorithm value:
> > >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > > provider/SpiffeProviderConstants.java#L6
> > >
> > >
> > > @Harsha do you want to chime in since you use that provider?
> > >
> >
> > Here the "Custom" Algorithm is not an implementation per say , rather
> used
> > to invoke the custom trust store factory and key manager factory. You are
> > not moving away from "standard" alogrithms that are available.
> >
> >
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/SpiffeTrustManager.java
> > As you can see it delegates all the calls of verification of certificate
> to
> > the default implementation available.
> > So in our implementation we still use PKIX to verify the certificate
> > chain. So you are not losing anything here and Spiffe is not
> > reimplementing the verification process.
> >
> >
> > 2. I already mentioned in my 3rd point, in my previous post, why using
> > > ssl.provider does NOT work. I updated KIP-486 in "rejected
> alternatives"
> > > also why ssl.provider does not work.
> > >
> >
> > As mentioned before , provider is the right way to go. I am still not
> > understanding the gap is.
> > If I understand correctly your argument is , provider is going to ask to
> > implement a custom algorithm.
> > My answer to that is , no you are not re-implementing the algorithm.
> Please
> > check the above link , TrustManager implementation it delegates the calls
> > back. There is no need to implement your own here.
> >
> >
> >
> > 3. Security.insertProviderAt() comments were based on assumption if
> KIP-492
> > > changes are done and we use that mechanism to configure providers
> instead
> > > of ssl.provider configuration.
> > >
> >
> > KIP-492 has patch available and going through review.
> >
> >
> > Can you read my all the points, I mentioned in my previous post, very
> > > carefully? I am covering all the aspects in explaining. I am open to
> > still
> > > discuss more to clarify any doubts.
> > >
> >
> >
> > "3. If we just use existing ssl.provider kafka configuration then our
> > provider will be used in SSLContext.getInstance(protocol, provider) call
> in
> > SslFactory.java <http://sslfactory.java/> and if our provider does not
> > have
> > implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we
> tested
> > it). Example: In MyProvider sample above you see that I didn't add
> > SSLContext.TLSv1 as
> > "Service+Algorithm" and that didn't work for me. In SPIFFE provider you
> > don't have this challenge since you are planning to bypass ssl.provider
> as
> > you mention in the KIP-492."
> >
> > Yes here you need to pass the protocol that your KeyManager/TrustManager
> > registered with and in no way its deviating from TLS RFC spec.
> >
> >
> https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/javax/net/ssl/SSLContext.java#L134
> >
> >
> > My suggestion here is for you to implement a simple Security Provider as
> > you did before and register a Algorithm. You can use the existing
> > implementation in SpiffeProvider and plug in changes where you need to
> > retrieve the certificates from by making RPC call.
> >
> > Run an end-to-end test with Kafka broker coming up with your provider
> > making calls to RPC call. You do need to pass the "custom algorithm" that
> > you registered in your key manager to invoke the provider.
> > I think your concern is this approach is replacing the existing known
> > ciphersuites and certificate validation provided by java. Which its not.
> >
> > Now test the TLS connection you can do so via openssl -s_client options
> to
> > test if everything else is working.
> >
> > I am happy to share configs that we used if you are interested.
> >
> > Thanks,
> > Harsha
> >
> >
> >
> > > Thanks
> > > Maulin
> > >
> > >
> > > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <cm...@apache.org>
> wrote:
> > >
> > > Hi Maulin,
> > >
> > > A lot of JSSE providers don't implement custom algorithms. Spire is a
> > good
> > > example of a JSSE provider that doesn't, and yet is still useful to
> many
> > > people. Your JSSE provider can work fine even if it doesn't implement a
> > > custom algorithm.
> > >
> > >
> > > Maybe I'm missing something, but I don't understand the discussion of
> > > Security.insertProviderAt() that you included. SslEngineBuilder doesn't
> > use
> > > that API to get the security provider. Instead, it calls
> > > "SSLContext.getInstance(protocol, provider)", where provider is the
> name
> > > of the provider.
> > >
> > >
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
> > >
> > > On top of everything above I feel strongly to add the 4th point which
> is
> > > based on Java APIs for TrustManagerFactory.init(KeyStore) (
> > >
> > >
> > > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> > > TrustManagerFactory.html#init(java.security.KeyStore)
> > > )
> > >
> > >
> > > and KeyManagerFactory.init(KeyStore, char[]) (
> > >
> > >
> >
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory.
> > > html#init(java.security.KeyStore,%20char[])
> > >
> > > ).
> > >
> > >
> > > 4. The above APIs are intended to support providing "trust/key
> material"
> > > from the user without having to write their own
> TrustManager/KeyManagers.
> > > To quote from the TrustManagerFactory.init()'s documentation
> "Initializes
> > > this factory with a source of certificate authorities and related trust
> > > material."
> > > To quote from the KeyManagerFactory.init()'s documentation "Initializes
> > > this factory with a source of key material."
> > >
> > >
> > > Based on this it is clear that there is a flexibility provided by Java
> to
> > > to enable developers to provide the required trust/key material loaded
> > >
> > > from
> > >
> > > "anywhere" without requiring them to write custom provider OR trust/key
> > > managers. This same flexibility is reflected in Kafka code also where
> it
> > > loads the trust/keys from a local file and doesn't require writing a
> > > Provider necessarily. If we do NOT have a custom algorithm, it makes
> less
> > > sense to write a Provider.
> > >
> > >
> > > Thanks
> > > Maulin
> > >
> > >
> > > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
> > >
> > > maulin.vasavada@gmail.com>
> > >
> > > wrote:
> > >
> > > Hi Harsha/Colin
> > >
> > > I did the sample with a custom Provider for TrustStoreManager and tried
> > > using ssl.provider Kafka config AND the way KIP-492 is suggesting (by
> > > adding Provider programmatically instead of relying on
> ssl.provider+java.
> > > security. The below sample is followed by my detailed findings. I'll
> > > appreciate if you can go through it carefully and see
> > >
> > > if you
> > >
> > > see my point.
> > >
> > > package providertest;
> > >
> > > import java.security.Provider <http://java.security.provider/>;
> > >
> > > public class MyProvider extends Provider {
> > >
> > >
> > > private static final String name = "MyProvider"; private static double
> > > version = 1.0d;
> > > private static String info = "Maulin's SSL Provider v"+version;
> > >
> > >
> > >
> > > public MyProvider() {
> > > super(name, version, info);
> > > this.put("TrustManagerFactory.PKIX",
> > >
> > >
> > > "providertest.MyTrustManagerFactory");
> > >
> > >
> > > }
> > > }
> > >
> > >
> > > *Details:*
> > >
> > > KIP-492 documents that it will use Security.addProvider() assuming it
> > >
> > > will
> > >
> > >
> > > add it as position '0' which is not a correct assumption. The
> > > addProvider()'s documentation says it will add it to the last available
> > > position. You may want to correct that to say
> > > Security.insertProviderAt(provider, 1).
> > >
> > >
> > > Now coming back to our specific discussion,
> > >
> > > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add
> > >
> > > that
> > >
> > > provider in the provider list via Security.addProvider() the position
> > >
> > > where
> > >
> > > it gets added doesn't matter (even if you don't end up adding it as
> > >
> > > first
> > >
> > > entry) since that is the ONLY provider for SPIFFE specific algorithm
> > >
> > > you
> > >
> > > might have.
> > >
> > > We do *not* have custom algorithm for Key/Trust StoreMangers. Which
> > >
> > > means
> > >
> > > we have to use X509, PKIX etc "Standard Algorithms" ((
> > >
> > >
> > > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> > > StandardNames.html
> > > ))
> > >
> > >
> > > in our provider to override the TrustStoreManager (see my sample code)
> > >
> > > and
> > >
> > > KeyStoreManger and KeyManager. This creates another challenge
> > >
> > > mentioned in
> > >
> > > the below point.
> > >
> > > 2. In order to make our Provider for loading custom TrustStore work, we
> > > have to add the provider as 'first' in the list since there are others
> > >
> > > with
> > >
> > > the same algorithm.
> > >
> > >
> > > However, the programatic way of adding provider
> > > (Security.insertProviderAt()) is *not* deterministic for ordering since
> > > different code can call the method for a different provider and
> > >
> > >
> > > depending
> > >
> > > upon the order of the call our provider can be first or pushed down the
> > > list. This can happen very well in any client application using Kafka.
> > >
> > > This
> > >
> > > is specially problematic for a case when you want to guarantee order
> > >
> > > for a
> > >
> > > Provider having "Standard Algorithms".
> > >
> > > If we add our provider in java.security file that definitely guarantees
> > > the order(unless somebody calls removeProvider() which is less
> > >
> > > likely). But
> > >
> > > if we add our provider in java.security file it will defeat the
> > >
> > > purpose of
> > >
> > > the KIP-492.
> > >
> > > In the gist - Apache Kafka must not rely on "un-deterministic" method
> > >
> > > to
> > >
> > > rely on Provider ordering.
> > >
> > > 3. If we just use existing ssl.provider kafka configuration then our
> > > provider will be used in SSLContext.getInstance(protocol, provider)
> > >
> > > call in
> > >
> > > SslFactory.java <http://sslfactory.java/> and if our provider does not
> > > have implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks
> (we
> > > tested it). Example:
> > >
> > > In
> > >
> > >
> > > MyProvider sample above you see that I didn't add SSLContext.TLSv1 as
> > > "Service+Algorithm" and that didn't work for me. In SPIFFE provider you
> > > don't have this challenge since you are planning to bypass
> > >
> > >
> > > ssl.provider as
> > >
> > > you mention in the KIP-492.
> > >
> > > *Overall summary,*
> > >
> > > 1. Any provider based mechanisms- a) existing ssl.provider and
> > >
> > > b)KIP-492,
> > >
> > > for loading key/trust store using "Standard Algorithms" do not work
> > >
> > >
> > > 2. Approach suggested in our KIP-486 works without any issue and it is
> > > *not* our context specific solve
> > >
> > >
> > > 3. Based on above we feel KIP-492 and KIP-486 are complimentary changes
> > > and not contradicting or redundent.
> > >
> > > If you want we can do a joint session somehow to walk through the
> > >
> > > sample I
> > >
> > > have and various experiments I did. I would encourage you to do similar
> > > exercise by writing a Provider for "Standard Algorithm" for
> > > TrustStoreManager (like our needs) and see what you find since only
> > >
> > > writing
> > >
> > > samples can bring out the complexity/challenges we face.
> > >
> > >
> > > Thanks
> > > Maulin
> > >
> > >
> > > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <
> maulin.vasavada@gmail.
> > > com> wrote:
> > >
> > > Just to update - still working on it. Get to work only on and off on
> > >
> > > it :(
> > >
> > > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
> > >
> > > maulin.vasavada@gmail.com>
> > >
> > > wrote:
> > >
> > > Hi Harsha
> > >
> > > Let me try to write samples and will let you know.
> > >
> > >
> > > Thanks
> > > Maulin
> > >
> > >
> > > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <ha...@gmail.com>
> > >
> > > wrote:
> > >
> > >
> > > Hi Maulin,
> > > With java security providers can be as custom you would
> > >
> > >
> > > like
> > >
> > >
> > > it to
> > > be. If you only want to to implement a custom way of loading the
> keystore
> > > and truststore and not implement any protocol/encryption handling
> > > you can leave them empty and no need to implement. Have you looked into
> > > the links I pasted before?
> > >
> > >
> > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> > > java
> > >
> > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > spiffe-security-provider/src/main/java/spiffe/api/provider/
> > > SpiffeTrustManager.java
> > >
> > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > > java
> > >
> > >
> > > Can you please tell me which methods are too complex in above to
> > implement
> > > or unnecessary? You are changing anything in SSL/TLS implementations
> > > provided by
> > >
> > >
> > >
> > > All of the implementations delegating the checks to the default
> > > implementation anyway.
> > > Spire agent is an example, its nothing but a GRPC server listening
> > >
> > >
> > > on a
> > >
> > >
> > > unix domain socket . Above code is making a RPC call to the local
> daemon
> > > to
> > > get the certificate and keys. The mechanics are pretty much same as
> > >
> > >
> > > what
> > >
> > > you are asking for.
> > >
> > >
> > > Thanks,
> > > Harsha
> > >
> > >
> > >
> > > On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
> > maulin.vasavada@gmail.com>
> > >
> > > wrote:
> > >
> > >
> > > Imagine a scenario like - We know we have a custom KMS and as a
> > >
> > > Kafka
> > >
> > > owner
> > >
> > > we want to comply to using that KMS source to load keys/certs. As
> > >
> > > a
> > >
> > > Kafka
> > >
> > > owner we know how to integrate with KMS but doesn't necessarily
> > >
> > > have
> > >
> > > to
> > >
> > > know anything about cipher suites, algorithms, and SSL/TLS
> > >
> > > implementation.
> > >
> > > Going the Provider way requires to know lot more than we should,
> > >
> > > isn't it?
> > >
> > > Not that we would have concern/shy-away knowing those details -
> > >
> > > but
> > >
> > > if we
> > >
> > > don't have to - why should we?
> > >
> > > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
> > >
> > > maulin.vasavada@gmail.com>
> > >
> > > wrote:
> > >
> > > Hi Harsha
> > >
> > > We don't have spire (or similar) agents and we do not have
> > >
> > > keys/certs
> > >
> > > locally on any brokers. To elaborate more on my previous email,
> > >
> > > I agree that Java security Providers are used in much broader
> > >
> > > sense
> > >
> > > - to
> > >
> > > have a particular implementation of an algorithm, use specific
> > >
> > > cipher
> > >
> > > suites for SSL , OR in our current team's case have a
> > >
> > > particular
> > >
> > > way to
> > >
> > > leverage pre-generated SSL sessions. However, the scope of our
> > >
> > > KIP
> > >
> > > (486)
> > >
> > > is
> > >
> > > much restricted than that. We merely intend to provide a custom
> > > keystore/truststore for our SSL connections and not really worry
> > >
> > > about
> > >
> > > underlying specific SSL/TLS implementation. This simplifies it
> > >
> > > a
> > >
> > > lot for
> > >
> > > us to keep the concerns separate and clear.
> > >
> > > I feel our approach is more complimentary such that it allows
> > >
> > > for
> > >
> > > using
> > >
> > > keystores of choice while retaining the flexibility to use any
> > > underlying/available Provider for actually making the SSL call.
> > >
> > > We agree with KIP-492's approach based on Providers (and Java's
> > > recommendation), but also strongly believe that our approach can
> > >
> > > compliment
> > >
> > > it very effectively for reasons explained above.
> > >
> > >
> > > Thanks
> > > Maulin
> > >
> > >
> > > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <
> > >
> > > kafka@harsha.io
> > >
> > > wrote:
> > >
> > > Hi Maulin,
> > >
> > >
> > > On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
> maulin.vasavada@gmail.
> > > com>
> > > wrote:
> > >
> > >
> > > Hi Harsha
> > >
> > > The reason we rejected the SslProvider route is that - we
> > >
> > > only
> > >
> > > needed
> > >
> > > a
> > >
> > > custom way to load keys/certs. Not touch any policy that
> > >
> > > existing
> > >
> > > Providers
> > >
> > > govern like SunJSSE Provider.
> > >
> > > We have exactly the same requirements to load certs and keys
> > >
> > > through
> > >
> > > spire
> > >
> > > agent. We used security.provider to do that exactly. I am not
> > >
> > > sure
> > >
> > > why
> > >
> > > you
> > >
> > > would be modifying any policies provided by default SunJSSE
> > >
> > > provider.
> > >
> > > Can
> > >
> > > you give me an example of having custom provider that will
> > >
> > > override an
> > >
> > > existing policy in SunJSSE provider.
> > >
> > > As pointed out earlier, this kip
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> > >
> > >
> > > allows
> > > you to load security.provider through config.
> > > Take a look at the examples I gave before
> > >
> > >
> > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > > java
> > >
> > > It registers KeyManagerFactory and TrustManagerFactory and
> > >
> > > Keystore
> > >
> > >
> > > algorithm.
> > > Implement your custom way of loading Keystore in here
> > >
> > >
> > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> > > java
> > >
> > > and Trust manager like here
> > >
> > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > > spiffe-security-provider/src/main/java/spiffe/api/provider/
> > > SpiffeTrustManager.java
> > >
> > > In your Kafka client you can set the security.provider to your
> > >
> > > custom
> > >
> > >
> > > implementation and with this fix
> > > https://issues.apache.org/jira/browse/KAFKA-8191 you can set
> > > keyManagerAlgorigthm and trustManagerAlgorithm configs.
> > >
> > >
> > > All of this is in your clients and broker side and do not need
> > >
> > > to
> > >
> > > touch
> > >
> > >
> > > any
> > > policy changes at JVM level. You'll register the providers in
> > >
> > >
> > > the
> > >
> > > priority
> > >
> > > order and can still have SunJSSE provider and have your custom
> > >
> > > provider
> > >
> > > to
> > >
> > > implement the key and trust managers.
> > >
> > > The ask here is different than KIP-492. We don't have any need
> > >
> > > to
> > >
> > > modify/specify the algorithm parameter. Does that make sense?
> > >
> > > The ask in KIP is introducing new interfaces where the KIP's
> > > goal/motivation can be achieved through the security.provider
> > >
> > > and
> > >
> > > we
> > >
> > >
> > > worked
> > > on similar goal without touching any Keystore or Truststore
> > >
> > >
> > > interfaces.
> > >
> > > My advise is against changing or introducing new interfaces
> > >
> > > when
> > >
> > > it can
> > >
> > > work through security.provider.
> > >
> > >
> > > Thanks,
> > > Harsha
> > >
> > >
> > > Thanks
> > >
> > > Maulin
> > >
> > > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
> > >
> > > kafka@harsha.io>
> > >
> > > wrote:
> > >
> > > In your KIP you added security. provider as rejected
> > >
> > > alternative
> > >
> > > and
> > >
> > > specified "its not the correct way". Do you mind explaining
> > >
> > > why
> > >
> > > its
> > >
> > > not? I
> > >
> > > didn't find any evidence in Java docs to say so. Contrary to
> > >
> > > your
> > >
> > > statement
> > >
> > >
> > > it does say in the java docs
> > > " However, please note that a provider can be used to
> > >
> > >
> > > implement
> > >
> > > any
> > >
> > > security service in Java that uses a pluggable architecture
> > >
> > > with
> > >
> > > a
> > >
> > > choice
> > >
> > > of implementations that fit underneath."
> > >
> > > Java Security Providers have been used by other projects to
> > >
> > > provide
> > >
> > > such
> > >
> > > integration . I am not sure if you looked into Spiffe
> > >
> > > project to
> > >
> > > efficiently distribute certificates but here is an example of
> > >
> > > Java
> > >
> > > provider
> > >
> > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >
> > >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > >
> > >
> > > java which
> > > obtains certificates from local daemons.
> > > These integrations are being used in Tomcat, Jetty etc.. We
> > >
> > >
> > > are
> > >
> > > also
> > >
> > > using
> > >
> > > Security provider to do the same in our Kafka clusters. So
> > >
> > > unless I
> > >
> > > see
> > >
> > > more evidence why security.provider doesn't work for you
> > >
> > > adding
> > >
> > > new
> > >
> > > interfaces while there exists more cleaner way of achieving
> > >
> > > the
> > >
> > > goals
> > >
> > > of
> > >
> > > this KIP is unnecessary and breaks the well known security
> > >
> > > interfaces
> > >
> > > provided by Java itself.
> > >
> > >
> > > Thanks,
> > > Harsha
> > >
> > >
> > > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
> > >
> > > kafka@harsha.io
> > >
> > > wrote:
> > >
> > >
> > > Hi Maulin,
> > > Not sure if you looked at my previous replies. This
> > >
> > >
> > > changes
> > >
> > > are not required as there is already security Provider to do
> > >
> > > what you
> > >
> > > are
> > >
> > > proposing. This KIP
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/
> > >
> > > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> > >
> > > also
> > >
> > > addresses easy registration of such providers.
> > >
> > >
> > > Thanks,
> > > Harsha
> > >
> > >
> > > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
> > >
> > > <maulin.vasavada@gmail.
> > >
> > > com> wrote:
> > >
> > > Bump! Can somebody please review this?
> > >
> > > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
> > >
> > > maulin.vasavada@gmail.com>
> > >
> > > wrote:
> > >
> > > Bump! Can somebody please review this?
> > >
> > >
> >
>
>
> --
> Thanks,
> M.Sai Sandeep
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Sandeep Mopuri <mp...@gmail.com>.
Hi Maulin, thanks for the discussion. As Harsha pointed out, to use the
KIP492, you need to create a new provider, register a *new* custom
algorithm for your keymanager and trustmanager factory implementations.
After this, the kafka server configuration can be done as given below
# Register the provider class with custom algorithm, say CUSTOM
security.provider.classes=com.company.security.CustomProvider
# Register the keymanager and trustmanager algorithms
# These algorithms indicate that the Keymanager and Trustmanagers
registered under the algorithm "CUSTOM" needs to be used
ssl.trustmanager.algorithm=CUSTOM
ssl.keymanager.algorithm=CUSTOM
And the customprovider looks like this...
public class CustomProvider extends Provider {
public CustomProvider() {
super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and TrustStore");
super.put("KeyManagerFactory.CUSTOM", "customKeyManagerFactory");
super.put("TrustManagerFactory.CUSTOM",
"customTrustManagerFactory");
}
}
The PR for this is in review and can be found here:
https://github.com/apache/kafka/pull/7090
This PR includes the fixed insertProviderAt function call.
On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani <ka...@harsha.io> wrote:
> Answers are in-line
>
>
>
> On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada <
> maulin.vasavada@gmail.com
> > wrote:
>
> > Hi Colin
> >
> >
> > When I refer to "standard" or "custom" algorithms I am following Java
> > security Provider Terminology. You can refer to
> > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> > StandardNames.html#TrustManagerFactory link I provided earlier in the
> > emails. It says PKIX is the default Algorithm for TrustManagerFactory.
> >
> >
> > 1. For SPIFFE, I am not sure why you are saying 'it does not implement
> > custom algorithms' because the following file clearly indicates that it
> > does use custom algorithm-
> >
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > provider/SpiffeProvider.java#L17
> >
> >
> > Algorithm value:
> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> > provider/SpiffeProviderConstants.java#L6
> >
> >
> > @Harsha do you want to chime in since you use that provider?
> >
>
> Here the "Custom" Algorithm is not an implementation per say , rather used
> to invoke the custom trust store factory and key manager factory. You are
> not moving away from "standard" alogrithms that are available.
>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/SpiffeTrustManager.java
> As you can see it delegates all the calls of verification of certificate to
> the default implementation available.
> So in our implementation we still use PKIX to verify the certificate
> chain. So you are not losing anything here and Spiffe is not
> reimplementing the verification process.
>
>
> 2. I already mentioned in my 3rd point, in my previous post, why using
> > ssl.provider does NOT work. I updated KIP-486 in "rejected alternatives"
> > also why ssl.provider does not work.
> >
>
> As mentioned before , provider is the right way to go. I am still not
> understanding the gap is.
> If I understand correctly your argument is , provider is going to ask to
> implement a custom algorithm.
> My answer to that is , no you are not re-implementing the algorithm. Please
> check the above link , TrustManager implementation it delegates the calls
> back. There is no need to implement your own here.
>
>
>
> 3. Security.insertProviderAt() comments were based on assumption if KIP-492
> > changes are done and we use that mechanism to configure providers instead
> > of ssl.provider configuration.
> >
>
> KIP-492 has patch available and going through review.
>
>
> Can you read my all the points, I mentioned in my previous post, very
> > carefully? I am covering all the aspects in explaining. I am open to
> still
> > discuss more to clarify any doubts.
> >
>
>
> "3. If we just use existing ssl.provider kafka configuration then our
> provider will be used in SSLContext.getInstance(protocol, provider) call in
> SslFactory.java <http://sslfactory.java/> and if our provider does not
> have
> implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we tested
> it). Example: In MyProvider sample above you see that I didn't add
> SSLContext.TLSv1 as
> "Service+Algorithm" and that didn't work for me. In SPIFFE provider you
> don't have this challenge since you are planning to bypass ssl.provider as
> you mention in the KIP-492."
>
> Yes here you need to pass the protocol that your KeyManager/TrustManager
> registered with and in no way its deviating from TLS RFC spec.
>
> https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/javax/net/ssl/SSLContext.java#L134
>
>
> My suggestion here is for you to implement a simple Security Provider as
> you did before and register a Algorithm. You can use the existing
> implementation in SpiffeProvider and plug in changes where you need to
> retrieve the certificates from by making RPC call.
>
> Run an end-to-end test with Kafka broker coming up with your provider
> making calls to RPC call. You do need to pass the "custom algorithm" that
> you registered in your key manager to invoke the provider.
> I think your concern is this approach is replacing the existing known
> ciphersuites and certificate validation provided by java. Which its not.
>
> Now test the TLS connection you can do so via openssl -s_client options to
> test if everything else is working.
>
> I am happy to share configs that we used if you are interested.
>
> Thanks,
> Harsha
>
>
>
> > Thanks
> > Maulin
> >
> >
> > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <cm...@apache.org> wrote:
> >
> > Hi Maulin,
> >
> > A lot of JSSE providers don't implement custom algorithms. Spire is a
> good
> > example of a JSSE provider that doesn't, and yet is still useful to many
> > people. Your JSSE provider can work fine even if it doesn't implement a
> > custom algorithm.
> >
> >
> > Maybe I'm missing something, but I don't understand the discussion of
> > Security.insertProviderAt() that you included. SslEngineBuilder doesn't
> use
> > that API to get the security provider. Instead, it calls
> > "SSLContext.getInstance(protocol, provider)", where provider is the name
> > of the provider.
> >
> >
> >
> > best,
> > Colin
> >
> >
> > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
> >
> > On top of everything above I feel strongly to add the 4th point which is
> > based on Java APIs for TrustManagerFactory.init(KeyStore) (
> >
> >
> > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> > TrustManagerFactory.html#init(java.security.KeyStore)
> > )
> >
> >
> > and KeyManagerFactory.init(KeyStore, char[]) (
> >
> >
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory.
> > html#init(java.security.KeyStore,%20char[])
> >
> > ).
> >
> >
> > 4. The above APIs are intended to support providing "trust/key material"
> > from the user without having to write their own TrustManager/KeyManagers.
> > To quote from the TrustManagerFactory.init()'s documentation "Initializes
> > this factory with a source of certificate authorities and related trust
> > material."
> > To quote from the KeyManagerFactory.init()'s documentation "Initializes
> > this factory with a source of key material."
> >
> >
> > Based on this it is clear that there is a flexibility provided by Java to
> > to enable developers to provide the required trust/key material loaded
> >
> > from
> >
> > "anywhere" without requiring them to write custom provider OR trust/key
> > managers. This same flexibility is reflected in Kafka code also where it
> > loads the trust/keys from a local file and doesn't require writing a
> > Provider necessarily. If we do NOT have a custom algorithm, it makes less
> > sense to write a Provider.
> >
> >
> > Thanks
> > Maulin
> >
> >
> > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
> >
> > maulin.vasavada@gmail.com>
> >
> > wrote:
> >
> > Hi Harsha/Colin
> >
> > I did the sample with a custom Provider for TrustStoreManager and tried
> > using ssl.provider Kafka config AND the way KIP-492 is suggesting (by
> > adding Provider programmatically instead of relying on ssl.provider+java.
> > security. The below sample is followed by my detailed findings. I'll
> > appreciate if you can go through it carefully and see
> >
> > if you
> >
> > see my point.
> >
> > package providertest;
> >
> > import java.security.Provider <http://java.security.provider/>;
> >
> > public class MyProvider extends Provider {
> >
> >
> > private static final String name = "MyProvider"; private static double
> > version = 1.0d;
> > private static String info = "Maulin's SSL Provider v"+version;
> >
> >
> >
> > public MyProvider() {
> > super(name, version, info);
> > this.put("TrustManagerFactory.PKIX",
> >
> >
> > "providertest.MyTrustManagerFactory");
> >
> >
> > }
> > }
> >
> >
> > *Details:*
> >
> > KIP-492 documents that it will use Security.addProvider() assuming it
> >
> > will
> >
> >
> > add it as position '0' which is not a correct assumption. The
> > addProvider()'s documentation says it will add it to the last available
> > position. You may want to correct that to say
> > Security.insertProviderAt(provider, 1).
> >
> >
> > Now coming back to our specific discussion,
> >
> > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add
> >
> > that
> >
> > provider in the provider list via Security.addProvider() the position
> >
> > where
> >
> > it gets added doesn't matter (even if you don't end up adding it as
> >
> > first
> >
> > entry) since that is the ONLY provider for SPIFFE specific algorithm
> >
> > you
> >
> > might have.
> >
> > We do *not* have custom algorithm for Key/Trust StoreMangers. Which
> >
> > means
> >
> > we have to use X509, PKIX etc "Standard Algorithms" ((
> >
> >
> > https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> > StandardNames.html
> > ))
> >
> >
> > in our provider to override the TrustStoreManager (see my sample code)
> >
> > and
> >
> > KeyStoreManger and KeyManager. This creates another challenge
> >
> > mentioned in
> >
> > the below point.
> >
> > 2. In order to make our Provider for loading custom TrustStore work, we
> > have to add the provider as 'first' in the list since there are others
> >
> > with
> >
> > the same algorithm.
> >
> >
> > However, the programatic way of adding provider
> > (Security.insertProviderAt()) is *not* deterministic for ordering since
> > different code can call the method for a different provider and
> >
> >
> > depending
> >
> > upon the order of the call our provider can be first or pushed down the
> > list. This can happen very well in any client application using Kafka.
> >
> > This
> >
> > is specially problematic for a case when you want to guarantee order
> >
> > for a
> >
> > Provider having "Standard Algorithms".
> >
> > If we add our provider in java.security file that definitely guarantees
> > the order(unless somebody calls removeProvider() which is less
> >
> > likely). But
> >
> > if we add our provider in java.security file it will defeat the
> >
> > purpose of
> >
> > the KIP-492.
> >
> > In the gist - Apache Kafka must not rely on "un-deterministic" method
> >
> > to
> >
> > rely on Provider ordering.
> >
> > 3. If we just use existing ssl.provider kafka configuration then our
> > provider will be used in SSLContext.getInstance(protocol, provider)
> >
> > call in
> >
> > SslFactory.java <http://sslfactory.java/> and if our provider does not
> > have implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we
> > tested it). Example:
> >
> > In
> >
> >
> > MyProvider sample above you see that I didn't add SSLContext.TLSv1 as
> > "Service+Algorithm" and that didn't work for me. In SPIFFE provider you
> > don't have this challenge since you are planning to bypass
> >
> >
> > ssl.provider as
> >
> > you mention in the KIP-492.
> >
> > *Overall summary,*
> >
> > 1. Any provider based mechanisms- a) existing ssl.provider and
> >
> > b)KIP-492,
> >
> > for loading key/trust store using "Standard Algorithms" do not work
> >
> >
> > 2. Approach suggested in our KIP-486 works without any issue and it is
> > *not* our context specific solve
> >
> >
> > 3. Based on above we feel KIP-492 and KIP-486 are complimentary changes
> > and not contradicting or redundent.
> >
> > If you want we can do a joint session somehow to walk through the
> >
> > sample I
> >
> > have and various experiments I did. I would encourage you to do similar
> > exercise by writing a Provider for "Standard Algorithm" for
> > TrustStoreManager (like our needs) and see what you find since only
> >
> > writing
> >
> > samples can bring out the complexity/challenges we face.
> >
> >
> > Thanks
> > Maulin
> >
> >
> > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada < maulin.vasavada@gmail.
> > com> wrote:
> >
> > Just to update - still working on it. Get to work only on and off on
> >
> > it :(
> >
> > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
> >
> > maulin.vasavada@gmail.com>
> >
> > wrote:
> >
> > Hi Harsha
> >
> > Let me try to write samples and will let you know.
> >
> >
> > Thanks
> > Maulin
> >
> >
> > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <ha...@gmail.com>
> >
> > wrote:
> >
> >
> > Hi Maulin,
> > With java security providers can be as custom you would
> >
> >
> > like
> >
> >
> > it to
> > be. If you only want to to implement a custom way of loading the keystore
> > and truststore and not implement any protocol/encryption handling
> > you can leave them empty and no need to implement. Have you looked into
> > the links I pasted before?
> >
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> > java
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > spiffe-security-provider/src/main/java/spiffe/api/provider/
> > SpiffeTrustManager.java
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > java
> >
> >
> > Can you please tell me which methods are too complex in above to
> implement
> > or unnecessary? You are changing anything in SSL/TLS implementations
> > provided by
> >
> >
> >
> > All of the implementations delegating the checks to the default
> > implementation anyway.
> > Spire agent is an example, its nothing but a GRPC server listening
> >
> >
> > on a
> >
> >
> > unix domain socket . Above code is making a RPC call to the local daemon
> > to
> > get the certificate and keys. The mechanics are pretty much same as
> >
> >
> > what
> >
> > you are asking for.
> >
> >
> > Thanks,
> > Harsha
> >
> >
> >
> > On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
> maulin.vasavada@gmail.com>
> >
> > wrote:
> >
> >
> > Imagine a scenario like - We know we have a custom KMS and as a
> >
> > Kafka
> >
> > owner
> >
> > we want to comply to using that KMS source to load keys/certs. As
> >
> > a
> >
> > Kafka
> >
> > owner we know how to integrate with KMS but doesn't necessarily
> >
> > have
> >
> > to
> >
> > know anything about cipher suites, algorithms, and SSL/TLS
> >
> > implementation.
> >
> > Going the Provider way requires to know lot more than we should,
> >
> > isn't it?
> >
> > Not that we would have concern/shy-away knowing those details -
> >
> > but
> >
> > if we
> >
> > don't have to - why should we?
> >
> > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
> >
> > maulin.vasavada@gmail.com>
> >
> > wrote:
> >
> > Hi Harsha
> >
> > We don't have spire (or similar) agents and we do not have
> >
> > keys/certs
> >
> > locally on any brokers. To elaborate more on my previous email,
> >
> > I agree that Java security Providers are used in much broader
> >
> > sense
> >
> > - to
> >
> > have a particular implementation of an algorithm, use specific
> >
> > cipher
> >
> > suites for SSL , OR in our current team's case have a
> >
> > particular
> >
> > way to
> >
> > leverage pre-generated SSL sessions. However, the scope of our
> >
> > KIP
> >
> > (486)
> >
> > is
> >
> > much restricted than that. We merely intend to provide a custom
> > keystore/truststore for our SSL connections and not really worry
> >
> > about
> >
> > underlying specific SSL/TLS implementation. This simplifies it
> >
> > a
> >
> > lot for
> >
> > us to keep the concerns separate and clear.
> >
> > I feel our approach is more complimentary such that it allows
> >
> > for
> >
> > using
> >
> > keystores of choice while retaining the flexibility to use any
> > underlying/available Provider for actually making the SSL call.
> >
> > We agree with KIP-492's approach based on Providers (and Java's
> > recommendation), but also strongly believe that our approach can
> >
> > compliment
> >
> > it very effectively for reasons explained above.
> >
> >
> > Thanks
> > Maulin
> >
> >
> > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <
> >
> > kafka@harsha.io
> >
> > wrote:
> >
> > Hi Maulin,
> >
> >
> > On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada < maulin.vasavada@gmail.
> > com>
> > wrote:
> >
> >
> > Hi Harsha
> >
> > The reason we rejected the SslProvider route is that - we
> >
> > only
> >
> > needed
> >
> > a
> >
> > custom way to load keys/certs. Not touch any policy that
> >
> > existing
> >
> > Providers
> >
> > govern like SunJSSE Provider.
> >
> > We have exactly the same requirements to load certs and keys
> >
> > through
> >
> > spire
> >
> > agent. We used security.provider to do that exactly. I am not
> >
> > sure
> >
> > why
> >
> > you
> >
> > would be modifying any policies provided by default SunJSSE
> >
> > provider.
> >
> > Can
> >
> > you give me an example of having custom provider that will
> >
> > override an
> >
> > existing policy in SunJSSE provider.
> >
> > As pointed out earlier, this kip
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/
> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> >
> >
> > allows
> > you to load security.provider through config.
> > Take a look at the examples I gave before
> >
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > java
> >
> > It registers KeyManagerFactory and TrustManagerFactory and
> >
> > Keystore
> >
> >
> > algorithm.
> > Implement your custom way of loading Keystore in here
> >
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> > java
> >
> > and Trust manager like here
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > spiffe-security-provider/src/main/java/spiffe/api/provider/
> > SpiffeTrustManager.java
> >
> > In your Kafka client you can set the security.provider to your
> >
> > custom
> >
> >
> > implementation and with this fix
> > https://issues.apache.org/jira/browse/KAFKA-8191 you can set
> > keyManagerAlgorigthm and trustManagerAlgorithm configs.
> >
> >
> > All of this is in your clients and broker side and do not need
> >
> > to
> >
> > touch
> >
> >
> > any
> > policy changes at JVM level. You'll register the providers in
> >
> >
> > the
> >
> > priority
> >
> > order and can still have SunJSSE provider and have your custom
> >
> > provider
> >
> > to
> >
> > implement the key and trust managers.
> >
> > The ask here is different than KIP-492. We don't have any need
> >
> > to
> >
> > modify/specify the algorithm parameter. Does that make sense?
> >
> > The ask in KIP is introducing new interfaces where the KIP's
> > goal/motivation can be achieved through the security.provider
> >
> > and
> >
> > we
> >
> >
> > worked
> > on similar goal without touching any Keystore or Truststore
> >
> >
> > interfaces.
> >
> > My advise is against changing or introducing new interfaces
> >
> > when
> >
> > it can
> >
> > work through security.provider.
> >
> >
> > Thanks,
> > Harsha
> >
> >
> > Thanks
> >
> > Maulin
> >
> > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
> >
> > kafka@harsha.io>
> >
> > wrote:
> >
> > In your KIP you added security. provider as rejected
> >
> > alternative
> >
> > and
> >
> > specified "its not the correct way". Do you mind explaining
> >
> > why
> >
> > its
> >
> > not? I
> >
> > didn't find any evidence in Java docs to say so. Contrary to
> >
> > your
> >
> > statement
> >
> >
> > it does say in the java docs
> > " However, please note that a provider can be used to
> >
> >
> > implement
> >
> > any
> >
> > security service in Java that uses a pluggable architecture
> >
> > with
> >
> > a
> >
> > choice
> >
> > of implementations that fit underneath."
> >
> > Java Security Providers have been used by other projects to
> >
> > provide
> >
> > such
> >
> > integration . I am not sure if you looked into Spiffe
> >
> > project to
> >
> > efficiently distribute certificates but here is an example of
> >
> > Java
> >
> > provider
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> >
> >
> > java which
> > obtains certificates from local daemons.
> > These integrations are being used in Tomcat, Jetty etc.. We
> >
> >
> > are
> >
> > also
> >
> > using
> >
> > Security provider to do the same in our Kafka clusters. So
> >
> > unless I
> >
> > see
> >
> > more evidence why security.provider doesn't work for you
> >
> > adding
> >
> > new
> >
> > interfaces while there exists more cleaner way of achieving
> >
> > the
> >
> > goals
> >
> > of
> >
> > this KIP is unnecessary and breaks the well known security
> >
> > interfaces
> >
> > provided by Java itself.
> >
> >
> > Thanks,
> > Harsha
> >
> >
> > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
> >
> > kafka@harsha.io
> >
> > wrote:
> >
> >
> > Hi Maulin,
> > Not sure if you looked at my previous replies. This
> >
> >
> > changes
> >
> > are not required as there is already security Provider to do
> >
> > what you
> >
> > are
> >
> > proposing. This KIP
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/
> >
> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> >
> > also
> >
> > addresses easy registration of such providers.
> >
> >
> > Thanks,
> > Harsha
> >
> >
> > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
> >
> > <maulin.vasavada@gmail.
> >
> > com> wrote:
> >
> > Bump! Can somebody please review this?
> >
> > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
> >
> > maulin.vasavada@gmail.com>
> >
> > wrote:
> >
> > Bump! Can somebody please review this?
> >
> >
>
--
Thanks,
M.Sai Sandeep
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Harsha Chintalapani <ka...@harsha.io>.
Answers are in-line
On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada <maulin.vasavada@gmail.com
> wrote:
> Hi Colin
>
>
> When I refer to "standard" or "custom" algorithms I am following Java
> security Provider Terminology. You can refer to
> https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> StandardNames.html#TrustManagerFactory link I provided earlier in the
> emails. It says PKIX is the default Algorithm for TrustManagerFactory.
>
>
> 1. For SPIFFE, I am not sure why you are saying 'it does not implement
> custom algorithms' because the following file clearly indicates that it
> does use custom algorithm-
>
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> provider/SpiffeProvider.java#L17
>
>
> Algorithm value:
> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/
> provider/SpiffeProviderConstants.java#L6
>
>
> @Harsha do you want to chime in since you use that provider?
>
Here the "Custom" Algorithm is not an implementation per say , rather used
to invoke the custom trust store factory and key manager factory. You are
not moving away from "standard" alogrithms that are available.
https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/SpiffeTrustManager.java
As you can see it delegates all the calls of verification of certificate to
the default implementation available.
So in our implementation we still use PKIX to verify the certificate
chain. So you are not losing anything here and Spiffe is not
reimplementing the verification process.
2. I already mentioned in my 3rd point, in my previous post, why using
> ssl.provider does NOT work. I updated KIP-486 in "rejected alternatives"
> also why ssl.provider does not work.
>
As mentioned before , provider is the right way to go. I am still not
understanding the gap is.
If I understand correctly your argument is , provider is going to ask to
implement a custom algorithm.
My answer to that is , no you are not re-implementing the algorithm. Please
check the above link , TrustManager implementation it delegates the calls
back. There is no need to implement your own here.
3. Security.insertProviderAt() comments were based on assumption if KIP-492
> changes are done and we use that mechanism to configure providers instead
> of ssl.provider configuration.
>
KIP-492 has patch available and going through review.
Can you read my all the points, I mentioned in my previous post, very
> carefully? I am covering all the aspects in explaining. I am open to still
> discuss more to clarify any doubts.
>
"3. If we just use existing ssl.provider kafka configuration then our
provider will be used in SSLContext.getInstance(protocol, provider) call in
SslFactory.java <http://sslfactory.java/> and if our provider does not have
implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we tested
it). Example: In MyProvider sample above you see that I didn't add
SSLContext.TLSv1 as
"Service+Algorithm" and that didn't work for me. In SPIFFE provider you
don't have this challenge since you are planning to bypass ssl.provider as
you mention in the KIP-492."
Yes here you need to pass the protocol that your KeyManager/TrustManager
registered with and in no way its deviating from TLS RFC spec.
https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/javax/net/ssl/SSLContext.java#L134
My suggestion here is for you to implement a simple Security Provider as
you did before and register a Algorithm. You can use the existing
implementation in SpiffeProvider and plug in changes where you need to
retrieve the certificates from by making RPC call.
Run an end-to-end test with Kafka broker coming up with your provider
making calls to RPC call. You do need to pass the "custom algorithm" that
you registered in your key manager to invoke the provider.
I think your concern is this approach is replacing the existing known
ciphersuites and certificate validation provided by java. Which its not.
Now test the TLS connection you can do so via openssl -s_client options to
test if everything else is working.
I am happy to share configs that we used if you are interested.
Thanks,
Harsha
> Thanks
> Maulin
>
>
> On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <cm...@apache.org> wrote:
>
> Hi Maulin,
>
> A lot of JSSE providers don't implement custom algorithms. Spire is a good
> example of a JSSE provider that doesn't, and yet is still useful to many
> people. Your JSSE provider can work fine even if it doesn't implement a
> custom algorithm.
>
>
> Maybe I'm missing something, but I don't understand the discussion of
> Security.insertProviderAt() that you included. SslEngineBuilder doesn't use
> that API to get the security provider. Instead, it calls
> "SSLContext.getInstance(protocol, provider)", where provider is the name
> of the provider.
>
>
>
> best,
> Colin
>
>
> On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
>
> On top of everything above I feel strongly to add the 4th point which is
> based on Java APIs for TrustManagerFactory.init(KeyStore) (
>
>
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/
> TrustManagerFactory.html#init(java.security.KeyStore)
> )
>
>
> and KeyManagerFactory.init(KeyStore, char[]) (
>
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory.
> html#init(java.security.KeyStore,%20char[])
>
> ).
>
>
> 4. The above APIs are intended to support providing "trust/key material"
> from the user without having to write their own TrustManager/KeyManagers.
> To quote from the TrustManagerFactory.init()'s documentation "Initializes
> this factory with a source of certificate authorities and related trust
> material."
> To quote from the KeyManagerFactory.init()'s documentation "Initializes
> this factory with a source of key material."
>
>
> Based on this it is clear that there is a flexibility provided by Java to
> to enable developers to provide the required trust/key material loaded
>
> from
>
> "anywhere" without requiring them to write custom provider OR trust/key
> managers. This same flexibility is reflected in Kafka code also where it
> loads the trust/keys from a local file and doesn't require writing a
> Provider necessarily. If we do NOT have a custom algorithm, it makes less
> sense to write a Provider.
>
>
> Thanks
> Maulin
>
>
> On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
>
> maulin.vasavada@gmail.com>
>
> wrote:
>
> Hi Harsha/Colin
>
> I did the sample with a custom Provider for TrustStoreManager and tried
> using ssl.provider Kafka config AND the way KIP-492 is suggesting (by
> adding Provider programmatically instead of relying on ssl.provider+java.
> security. The below sample is followed by my detailed findings. I'll
> appreciate if you can go through it carefully and see
>
> if you
>
> see my point.
>
> package providertest;
>
> import java.security.Provider <http://java.security.provider/>;
>
> public class MyProvider extends Provider {
>
>
> private static final String name = "MyProvider"; private static double
> version = 1.0d;
> private static String info = "Maulin's SSL Provider v"+version;
>
>
>
> public MyProvider() {
> super(name, version, info);
> this.put("TrustManagerFactory.PKIX",
>
>
> "providertest.MyTrustManagerFactory");
>
>
> }
> }
>
>
> *Details:*
>
> KIP-492 documents that it will use Security.addProvider() assuming it
>
> will
>
>
> add it as position '0' which is not a correct assumption. The
> addProvider()'s documentation says it will add it to the last available
> position. You may want to correct that to say
> Security.insertProviderAt(provider, 1).
>
>
> Now coming back to our specific discussion,
>
> 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add
>
> that
>
> provider in the provider list via Security.addProvider() the position
>
> where
>
> it gets added doesn't matter (even if you don't end up adding it as
>
> first
>
> entry) since that is the ONLY provider for SPIFFE specific algorithm
>
> you
>
> might have.
>
> We do *not* have custom algorithm for Key/Trust StoreMangers. Which
>
> means
>
> we have to use X509, PKIX etc "Standard Algorithms" ((
>
>
> https://docs.oracle.com/javase/7/docs/technotes/guides/security/
> StandardNames.html
> ))
>
>
> in our provider to override the TrustStoreManager (see my sample code)
>
> and
>
> KeyStoreManger and KeyManager. This creates another challenge
>
> mentioned in
>
> the below point.
>
> 2. In order to make our Provider for loading custom TrustStore work, we
> have to add the provider as 'first' in the list since there are others
>
> with
>
> the same algorithm.
>
>
> However, the programatic way of adding provider
> (Security.insertProviderAt()) is *not* deterministic for ordering since
> different code can call the method for a different provider and
>
>
> depending
>
> upon the order of the call our provider can be first or pushed down the
> list. This can happen very well in any client application using Kafka.
>
> This
>
> is specially problematic for a case when you want to guarantee order
>
> for a
>
> Provider having "Standard Algorithms".
>
> If we add our provider in java.security file that definitely guarantees
> the order(unless somebody calls removeProvider() which is less
>
> likely). But
>
> if we add our provider in java.security file it will defeat the
>
> purpose of
>
> the KIP-492.
>
> In the gist - Apache Kafka must not rely on "un-deterministic" method
>
> to
>
> rely on Provider ordering.
>
> 3. If we just use existing ssl.provider kafka configuration then our
> provider will be used in SSLContext.getInstance(protocol, provider)
>
> call in
>
> SslFactory.java <http://sslfactory.java/> and if our provider does not
> have implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we
> tested it). Example:
>
> In
>
>
> MyProvider sample above you see that I didn't add SSLContext.TLSv1 as
> "Service+Algorithm" and that didn't work for me. In SPIFFE provider you
> don't have this challenge since you are planning to bypass
>
>
> ssl.provider as
>
> you mention in the KIP-492.
>
> *Overall summary,*
>
> 1. Any provider based mechanisms- a) existing ssl.provider and
>
> b)KIP-492,
>
> for loading key/trust store using "Standard Algorithms" do not work
>
>
> 2. Approach suggested in our KIP-486 works without any issue and it is
> *not* our context specific solve
>
>
> 3. Based on above we feel KIP-492 and KIP-486 are complimentary changes
> and not contradicting or redundent.
>
> If you want we can do a joint session somehow to walk through the
>
> sample I
>
> have and various experiments I did. I would encourage you to do similar
> exercise by writing a Provider for "Standard Algorithm" for
> TrustStoreManager (like our needs) and see what you find since only
>
> writing
>
> samples can bring out the complexity/challenges we face.
>
>
> Thanks
> Maulin
>
>
> On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada < maulin.vasavada@gmail.
> com> wrote:
>
> Just to update - still working on it. Get to work only on and off on
>
> it :(
>
> On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
>
> maulin.vasavada@gmail.com>
>
> wrote:
>
> Hi Harsha
>
> Let me try to write samples and will let you know.
>
>
> Thanks
> Maulin
>
>
> On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <ha...@gmail.com>
>
> wrote:
>
>
> Hi Maulin,
> With java security providers can be as custom you would
>
>
> like
>
>
> it to
> be. If you only want to to implement a custom way of loading the keystore
> and truststore and not implement any protocol/encryption handling
> you can leave them empty and no need to implement. Have you looked into
> the links I pasted before?
>
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> java
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> spiffe-security-provider/src/main/java/spiffe/api/provider/
> SpiffeTrustManager.java
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> java
>
>
> Can you please tell me which methods are too complex in above to implement
> or unnecessary? You are changing anything in SSL/TLS implementations
> provided by
>
>
>
> All of the implementations delegating the checks to the default
> implementation anyway.
> Spire agent is an example, its nothing but a GRPC server listening
>
>
> on a
>
>
> unix domain socket . Above code is making a RPC call to the local daemon
> to
> get the certificate and keys. The mechanics are pretty much same as
>
>
> what
>
> you are asking for.
>
>
> Thanks,
> Harsha
>
>
>
> On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada < maulin.vasavada@gmail.com>
>
> wrote:
>
>
> Imagine a scenario like - We know we have a custom KMS and as a
>
> Kafka
>
> owner
>
> we want to comply to using that KMS source to load keys/certs. As
>
> a
>
> Kafka
>
> owner we know how to integrate with KMS but doesn't necessarily
>
> have
>
> to
>
> know anything about cipher suites, algorithms, and SSL/TLS
>
> implementation.
>
> Going the Provider way requires to know lot more than we should,
>
> isn't it?
>
> Not that we would have concern/shy-away knowing those details -
>
> but
>
> if we
>
> don't have to - why should we?
>
> On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
>
> maulin.vasavada@gmail.com>
>
> wrote:
>
> Hi Harsha
>
> We don't have spire (or similar) agents and we do not have
>
> keys/certs
>
> locally on any brokers. To elaborate more on my previous email,
>
> I agree that Java security Providers are used in much broader
>
> sense
>
> - to
>
> have a particular implementation of an algorithm, use specific
>
> cipher
>
> suites for SSL , OR in our current team's case have a
>
> particular
>
> way to
>
> leverage pre-generated SSL sessions. However, the scope of our
>
> KIP
>
> (486)
>
> is
>
> much restricted than that. We merely intend to provide a custom
> keystore/truststore for our SSL connections and not really worry
>
> about
>
> underlying specific SSL/TLS implementation. This simplifies it
>
> a
>
> lot for
>
> us to keep the concerns separate and clear.
>
> I feel our approach is more complimentary such that it allows
>
> for
>
> using
>
> keystores of choice while retaining the flexibility to use any
> underlying/available Provider for actually making the SSL call.
>
> We agree with KIP-492's approach based on Providers (and Java's
> recommendation), but also strongly believe that our approach can
>
> compliment
>
> it very effectively for reasons explained above.
>
>
> Thanks
> Maulin
>
>
> On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <
>
> kafka@harsha.io
>
> wrote:
>
> Hi Maulin,
>
>
> On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada < maulin.vasavada@gmail.
> com>
> wrote:
>
>
> Hi Harsha
>
> The reason we rejected the SslProvider route is that - we
>
> only
>
> needed
>
> a
>
> custom way to load keys/certs. Not touch any policy that
>
> existing
>
> Providers
>
> govern like SunJSSE Provider.
>
> We have exactly the same requirements to load certs and keys
>
> through
>
> spire
>
> agent. We used security.provider to do that exactly. I am not
>
> sure
>
> why
>
> you
>
> would be modifying any policies provided by default SunJSSE
>
> provider.
>
> Can
>
> you give me an example of having custom provider that will
>
> override an
>
> existing policy in SunJSSE provider.
>
> As pointed out earlier, this kip
>
> https://cwiki.apache.org/confluence/display/KAFKA/
> KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>
>
> allows
> you to load security.provider through config.
> Take a look at the examples I gave before
>
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> java
>
> It registers KeyManagerFactory and TrustManagerFactory and
>
> Keystore
>
>
> algorithm.
> Implement your custom way of loading Keystore in here
>
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.
> java
>
> and Trust manager like here
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> spiffe-security-provider/src/main/java/spiffe/api/provider/
> SpiffeTrustManager.java
>
> In your Kafka client you can set the security.provider to your
>
> custom
>
>
> implementation and with this fix
> https://issues.apache.org/jira/browse/KAFKA-8191 you can set
> keyManagerAlgorigthm and trustManagerAlgorithm configs.
>
>
> All of this is in your clients and broker side and do not need
>
> to
>
> touch
>
>
> any
> policy changes at JVM level. You'll register the providers in
>
>
> the
>
> priority
>
> order and can still have SunJSSE provider and have your custom
>
> provider
>
> to
>
> implement the key and trust managers.
>
> The ask here is different than KIP-492. We don't have any need
>
> to
>
> modify/specify the algorithm parameter. Does that make sense?
>
> The ask in KIP is introducing new interfaces where the KIP's
> goal/motivation can be achieved through the security.provider
>
> and
>
> we
>
>
> worked
> on similar goal without touching any Keystore or Truststore
>
>
> interfaces.
>
> My advise is against changing or introducing new interfaces
>
> when
>
> it can
>
> work through security.provider.
>
>
> Thanks,
> Harsha
>
>
> Thanks
>
> Maulin
>
> On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
>
> kafka@harsha.io>
>
> wrote:
>
> In your KIP you added security. provider as rejected
>
> alternative
>
> and
>
> specified "its not the correct way". Do you mind explaining
>
> why
>
> its
>
> not? I
>
> didn't find any evidence in Java docs to say so. Contrary to
>
> your
>
> statement
>
>
> it does say in the java docs
> " However, please note that a provider can be used to
>
>
> implement
>
> any
>
> security service in Java that uses a pluggable architecture
>
> with
>
> a
>
> choice
>
> of implementations that fit underneath."
>
> Java Security Providers have been used by other projects to
>
> provide
>
> such
>
> integration . I am not sure if you looked into Spiffe
>
> project to
>
> efficiently distribute certificates but here is an example of
>
> Java
>
> provider
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>
>
> java which
> obtains certificates from local daemons.
> These integrations are being used in Tomcat, Jetty etc.. We
>
>
> are
>
> also
>
> using
>
> Security provider to do the same in our Kafka clusters. So
>
> unless I
>
> see
>
> more evidence why security.provider doesn't work for you
>
> adding
>
> new
>
> interfaces while there exists more cleaner way of achieving
>
> the
>
> goals
>
> of
>
> this KIP is unnecessary and breaks the well known security
>
> interfaces
>
> provided by Java itself.
>
>
> Thanks,
> Harsha
>
>
> On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
>
> kafka@harsha.io
>
> wrote:
>
>
> Hi Maulin,
> Not sure if you looked at my previous replies. This
>
>
> changes
>
> are not required as there is already security Provider to do
>
> what you
>
> are
>
> proposing. This KIP
>
> https://cwiki.apache.org/confluence/display/KAFKA/
>
> KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>
> also
>
> addresses easy registration of such providers.
>
>
> Thanks,
> Harsha
>
>
> On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
>
> <maulin.vasavada@gmail.
>
> com> wrote:
>
> Bump! Can somebody please review this?
>
> On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
>
> maulin.vasavada@gmail.com>
>
> wrote:
>
> Bump! Can somebody please review this?
>
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Maulin Vasavada <ma...@gmail.com>.
Hi Colin
When I refer to "standard" or "custom" algorithms I am following Java
security Provider Terminology. You can refer to
https://docs.oracle.com/javase/7/docs/technotes/guides/security/StandardNames.html#TrustManagerFactory
link I provided earlier in the emails. It says PKIX is the default
Algorithm for TrustManagerFactory.
1. For SPIFFE, I am not sure why you are saying 'it does not implement
custom algorithms' because the following file clearly indicates that it
does use custom algorithm-
https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/SpiffeProvider.java#L17
Algorithm value:
https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/SpiffeProviderConstants.java#L6
@Harsha do you want to chime in since you use that provider?
2. I already mentioned in my 3rd point, in my previous post, why using
ssl.provider does NOT work. I updated KIP-486 in "rejected alternatives"
also why ssl.provider does not work.
3. Security.insertProviderAt() comments were based on assumption if KIP-492
changes are done and we use that mechanism to configure providers instead
of ssl.provider configuration.
Can you read my all the points, I mentioned in my previous post, very
carefully? I am covering all the aspects in explaining. I am open to still
discuss more to clarify any doubts.
Thanks
Maulin
On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <cm...@apache.org> wrote:
> Hi Maulin,
>
> A lot of JSSE providers don't implement custom algorithms. Spire is a
> good example of a JSSE provider that doesn't, and yet is still useful to
> many people. Your JSSE provider can work fine even if it doesn't implement
> a custom algorithm.
>
> Maybe I'm missing something, but I don't understand the discussion of
> Security.insertProviderAt() that you included. SslEngineBuilder doesn't
> use that API to get the security provider. Instead, it calls
> "SSLContext.getInstance(protocol, provider)", where provider is the name of
> the provider.
>
> best,
> Colin
>
>
> On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
> > On top of everything above I feel strongly to add the 4th point which is
> > based on Java APIs for TrustManagerFactory.init(KeyStore) (
> >
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/TrustManagerFactory.html#init(java.security.KeyStore)
> )
> > and KeyManagerFactory.init(KeyStore, char[]) (
> >
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory.html#init(java.security.KeyStore,%20char[])
> > ).
> >
> > 4. The above APIs are intended to support providing "trust/key material"
> > from the user without having to write their own TrustManager/KeyManagers.
> > To quote from the TrustManagerFactory.init()'s documentation "Initializes
> > this factory with a source of certificate authorities and related trust
> > material."
> > To quote from the KeyManagerFactory.init()'s documentation "Initializes
> > this factory with a source of key material."
> >
> > Based on this it is clear that there is a flexibility provided by Java to
> > to enable developers to provide the required trust/key material loaded
> from
> > "anywhere" without requiring them to write custom provider OR trust/key
> > managers. This same flexibility is reflected in Kafka code also where it
> > loads the trust/keys from a local file and doesn't require writing a
> > Provider necessarily. If we do NOT have a custom algorithm, it makes less
> > sense to write a Provider.
> >
> > Thanks
> > Maulin
> >
> >
> >
> >
> >
> >
> > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <
> maulin.vasavada@gmail.com>
> > wrote:
> >
> > > Hi Harsha/Colin
> > >
> > > I did the sample with a custom Provider for TrustStoreManager and tried
> > > using ssl.provider Kafka config AND the way KIP-492 is suggesting (by
> > > adding Provider programmatically instead of relying on
> > > ssl.provider+java.security. The below sample is followed by my detailed
> > > findings. I'll appreciate if you can go through it carefully and see
> if you
> > > see my point.
> > >
> > > package providertest;
> > >
> > > import java.security.Provider;
> > >
> > > public class MyProvider extends Provider {
> > >
> > > private static final String name = "MyProvider";
> > > private static double version = 1.0d;
> > > private static String info = "Maulin's SSL Provider v"+version;
> > >
> > > public MyProvider() {
> > > super(name, version, info);
> > > this.put("TrustManagerFactory.PKIX",
> "providertest.MyTrustManagerFactory");
> > > }
> > > }
> > >
> > >
> > >
> > > *Details:*
> > >
> > > KIP-492 documents that it will use Security.addProvider() assuming it
> will
> > > add it as position '0' which is not a correct assumption. The
> > > addProvider()'s documentation says it will add it to the last available
> > > position. You may want to correct that to say
> > > Security.insertProviderAt(provider, 1).
> > >
> > > Now coming back to our specific discussion,
> > >
> > > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add
> that
> > > provider in the provider list via Security.addProvider() the position
> where
> > > it gets added doesn't matter (even if you don't end up adding it as
> first
> > > entry) since that is the ONLY provider for SPIFFE specific algorithm
> you
> > > might have.
> > >
> > > We do *not* have custom algorithm for Key/Trust StoreMangers. Which
> means
> > > we have to use X509, PKIX etc "Standard Algorithms" ((
> > >
> https://docs.oracle.com/javase/7/docs/technotes/guides/security/StandardNames.html
> ))
> > > in our provider to override the TrustStoreManager (see my sample code)
> and
> > > KeyStoreManger and KeyManager. This creates another challenge
> mentioned in
> > > the below point.
> > >
> > > 2. In order to make our Provider for loading custom TrustStore work, we
> > > have to add the provider as 'first' in the list since there are others
> with
> > > the same algorithm.
> > >
> > > However, the programatic way of adding provider
> > > (Security.insertProviderAt()) is *not* deterministic for ordering since
> > > different code can call the method for a different provider and
> depending
> > > upon the order of the call our provider can be first or pushed down the
> > > list. This can happen very well in any client application using Kafka.
> This
> > > is specially problematic for a case when you want to guarantee order
> for a
> > > Provider having "Standard Algorithms".
> > >
> > > If we add our provider in java.security file that definitely guarantees
> > > the order(unless somebody calls removeProvider() which is less
> likely). But
> > > if we add our provider in java.security file it will defeat the
> purpose of
> > > the KIP-492.
> > >
> > > In the gist - Apache Kafka must not rely on "un-deterministic" method
> to
> > > rely on Provider ordering.
> > >
> > > 3. If we just use existing ssl.provider kafka configuration then our
> > > provider will be used in SSLContext.getInstance(protocol, provider)
> call in
> > > SslFactory.java and if our provider does not have implementation for
> > > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we tested it). Example:
> In
> > > MyProvider sample above you see that I didn't add SSLContext.TLSv1 as
> > > "Service+Algorithm" and that didn't work for me. In SPIFFE provider you
> > > don't have this challenge since you are planning to bypass
> ssl.provider as
> > > you mention in the KIP-492.
> > >
> > >
> > > *Overall summary,*
> > >
> > > 1. Any provider based mechanisms- a) existing ssl.provider and
> b)KIP-492,
> > > for loading key/trust store using "Standard Algorithms" do not work
> > >
> > > 2. Approach suggested in our KIP-486 works without any issue and it is
> > > *not* our context specific solve
> > >
> > > 3. Based on above we feel KIP-492 and KIP-486 are complimentary changes
> > > and not contradicting or redundent.
> > >
> > > If you want we can do a joint session somehow to walk through the
> sample I
> > > have and various experiments I did. I would encourage you to do similar
> > > exercise by writing a Provider for "Standard Algorithm" for
> > > TrustStoreManager (like our needs) and see what you find since only
> writing
> > > samples can bring out the complexity/challenges we face.
> > >
> > > Thanks
> > > Maulin
> > >
> > > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <
> > > maulin.vasavada@gmail.com> wrote:
> > >
> > >> Just to update - still working on it. Get to work only on and off on
> it :(
> > >>
> > >> On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <
> maulin.vasavada@gmail.com>
> > >> wrote:
> > >>
> > >>> Hi Harsha
> > >>>
> > >>> Let me try to write samples and will let you know.
> > >>>
> > >>> Thanks
> > >>> Maulin
> > >>>
> > >>> On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <ha...@gmail.com>
> wrote:
> > >>>
> > >>>> Hi Maulin,
> > >>>> With java security providers can be as custom you would
> like
> > >>>> it to
> > >>>> be. If you only want to to implement a custom way of loading the
> > >>>> keystore and truststore and not implement any protocol/encryption
> > >>>> handling
> > >>>> you can leave them empty and no need to implement.
> > >>>> Have you looked into the links I pasted before?
> > >>>>
> > >>>>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java
> > >>>>
> > >>>>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java
> > >>>>
> > >>>>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
> > >>>>
> > >>>> Can you please tell me which methods are too complex in above to
> > >>>> implement
> > >>>> or unnecessary? You are changing anything in SSL/TLS implementations
> > >>>> provided by
> > >>>>
> > >>>> All of the implementations delegating the checks to the default
> > >>>> implementation anyway.
> > >>>> Spire agent is an example, its nothing but a GRPC server listening
> on a
> > >>>> unix domain socket . Above code is making a RPC call to the local
> > >>>> daemon to
> > >>>> get the certificate and keys. The mechanics are pretty much same as
> what
> > >>>> you are asking for.
> > >>>>
> > >>>> Thanks,
> > >>>> Harsha
> > >>>>
> > >>>> On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
> > >>>> maulin.vasavada@gmail.com>
> > >>>> wrote:
> > >>>>
> > >>>> > Imagine a scenario like - We know we have a custom KMS and as a
> Kafka
> > >>>> owner
> > >>>> > we want to comply to using that KMS source to load keys/certs. As
> a
> > >>>> Kafka
> > >>>> > owner we know how to integrate with KMS but doesn't necessarily
> have
> > >>>> to
> > >>>> > know anything about cipher suites, algorithms, and SSL/TLS
> > >>>> implementation.
> > >>>> > Going the Provider way requires to know lot more than we should,
> > >>>> isn't it?
> > >>>> > Not that we would have concern/shy-away knowing those details -
> but
> > >>>> if we
> > >>>> > don't have to - why should we?
> > >>>> >
> > >>>> > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
> > >>>> maulin.vasavada@gmail.com>
> > >>>> > wrote:
> > >>>> >
> > >>>> > > Hi Harsha
> > >>>> > >
> > >>>> > > We don't have spire (or similar) agents and we do not have
> > >>>> keys/certs
> > >>>> > > locally on any brokers. To elaborate more on my previous email,
> > >>>> > >
> > >>>> > > I agree that Java security Providers are used in much broader
> sense
> > >>>> - to
> > >>>> > > have a particular implementation of an algorithm, use specific
> > >>>> cipher
> > >>>> > > suites for SSL , OR in our current team's case have a
> particular
> > >>>> way to
> > >>>> > > leverage pre-generated SSL sessions. However, the scope of our
> KIP
> > >>>> (486)
> > >>>> > is
> > >>>> > > much restricted than that. We merely intend to provide a custom
> > >>>> > > keystore/truststore for our SSL connections and not really worry
> > >>>> about
> > >>>> > > underlying specific SSL/TLS implementation. This simplifies it
> a
> > >>>> lot for
> > >>>> > > us to keep the concerns separate and clear.
> > >>>> > >
> > >>>> > > I feel our approach is more complimentary such that it allows
> for
> > >>>> using
> > >>>> > > keystores of choice while retaining the flexibility to use any
> > >>>> > > underlying/available Provider for actually making the SSL call.
> > >>>> > >
> > >>>> > > We agree with KIP-492's approach based on Providers (and Java's
> > >>>> > > recommendation), but also strongly believe that our approach can
> > >>>> > compliment
> > >>>> > > it very effectively for reasons explained above.
> > >>>> > >
> > >>>> > > Thanks
> > >>>> > > Maulin
> > >>>> > >
> > >>>> > > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <
> kafka@harsha.io
> > >>>> >
> > >>>> > > wrote:
> > >>>> > >
> > >>>> > >> Hi Maulin,
> > >>>> > >>
> > >>>> > >> On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
> > >>>> > >> maulin.vasavada@gmail.com>
> > >>>> > >> wrote:
> > >>>> > >>
> > >>>> > >> > Hi Harsha
> > >>>> > >> >
> > >>>> > >> > The reason we rejected the SslProvider route is that - we
> only
> > >>>> needed
> > >>>> > a
> > >>>> > >> > custom way to load keys/certs. Not touch any policy that
> existing
> > >>>> > >> Providers
> > >>>> > >> > govern like SunJSSE Provider.
> > >>>> > >> >
> > >>>> > >>
> > >>>> > >> We have exactly the same requirements to load certs and keys
> > >>>> through
> > >>>> > spire
> > >>>> > >> agent. We used security.provider to do that exactly. I am not
> sure
> > >>>> why
> > >>>> > you
> > >>>> > >> would be modifying any policies provided by default SunJSSE
> > >>>> provider.
> > >>>> > Can
> > >>>> > >> you give me an example of having custom provider that will
> > >>>> override an
> > >>>> > >> existing policy in SunJSSE provider.
> > >>>> > >>
> > >>>> > >> As pointed out earlier, this kip
> > >>>> > >>
> > >>>> > >>
> > >>>> >
> > >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> > >>>> > >> allows
> > >>>> > >> you to load security.provider through config.
> > >>>> > >> Take a look at the examples I gave before
> > >>>> > >>
> > >>>> > >>
> > >>>> >
> > >>>>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
> > >>>> > >> It registers KeyManagerFactory and TrustManagerFactory and
> Keystore
> > >>>> > >> algorithm.
> > >>>> > >> Implement your custom way of loading Keystore in here
> > >>>> > >>
> > >>>> > >>
> > >>>> >
> > >>>>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java
> > >>>> > >>
> > >>>> > >> and Trust manager like here
> > >>>> > >>
> > >>>> > >>
> > >>>> >
> > >>>>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java
> > >>>> > >>
> > >>>> > >> In your Kafka client you can set the security.provider to your
> > >>>> custom
> > >>>> > >> implementation and with this fix
> > >>>> > >> https://issues.apache.org/jira/browse/KAFKA-8191 you can set
> > >>>> > >> keyManagerAlgorigthm and trustManagerAlgorithm configs.
> > >>>> > >>
> > >>>> > >> All of this is in your clients and broker side and do not need
> to
> > >>>> touch
> > >>>> > >> any
> > >>>> > >> policy changes at JVM level. You'll register the providers in
> the
> > >>>> > priority
> > >>>> > >> order and can still have SunJSSE provider and have your custom
> > >>>> provider
> > >>>> > to
> > >>>> > >> implement the key and trust managers.
> > >>>> > >>
> > >>>> > >>
> > >>>> > >>
> > >>>> > >> The ask here is different than KIP-492. We don't have any need
> to
> > >>>> > >> > modify/specify the algorithm parameter. Does that make sense?
> > >>>> > >> >
> > >>>> > >>
> > >>>> > >> The ask in KIP is introducing new interfaces where the KIP's
> > >>>> > >> goal/motivation can be achieved through the security.provider
> and
> > >>>> we
> > >>>> > >> worked
> > >>>> > >> on similar goal without touching any Keystore or Truststore
> > >>>> interfaces.
> > >>>> > >> My advise is against changing or introducing new interfaces
> when
> > >>>> it can
> > >>>> > >> work through security.provider.
> > >>>> > >>
> > >>>> > >> Thanks,
> > >>>> > >> Harsha
> > >>>> > >>
> > >>>> > >>
> > >>>> > >> Thanks
> > >>>> > >> > Maulin
> > >>>> > >> >
> > >>>> > >> > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
> > >>>> kafka@harsha.io>
> > >>>> > >> > wrote:
> > >>>> > >> >
> > >>>> > >> > In your KIP you added security. provider as rejected
> alternative
> > >>>> and
> > >>>> > >> > specified "its not the correct way". Do you mind explaining
> why
> > >>>> its
> > >>>> > >> not? I
> > >>>> > >> > didn't find any evidence in Java docs to say so. Contrary to
> your
> > >>>> > >> statement
> > >>>> > >> > it does say in the java docs
> > >>>> > >> > " However, please note that a provider can be used to
> implement
> > >>>> any
> > >>>> > >> > security service in Java that uses a pluggable architecture
> with
> > >>>> a
> > >>>> > >> choice
> > >>>> > >> > of implementations that fit underneath."
> > >>>> > >> >
> > >>>> > >> > Java Security Providers have been used by other projects to
> > >>>> provide
> > >>>> > such
> > >>>> > >> > integration . I am not sure if you looked into Spiffe
> project to
> > >>>> > >> > efficiently distribute certificates but here is an example of
> > >>>> Java
> > >>>> > >> provider
> > >>>> > >> >
> > >>>> > >> >
> > >>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >>>> > >> >
> > >>>> > >>
> > >>>> >
> > >>>>
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > >>>> > >> > java which
> > >>>> > >> > obtains certificates from local daemons.
> > >>>> > >> > These integrations are being used in Tomcat, Jetty etc.. We
> are
> > >>>> also
> > >>>> > >> using
> > >>>> > >> > Security provider to do the same in our Kafka clusters. So
> > >>>> unless I
> > >>>> > see
> > >>>> > >> > more evidence why security.provider doesn't work for you
> adding
> > >>>> new
> > >>>> > >> > interfaces while there exists more cleaner way of achieving
> the
> > >>>> goals
> > >>>> > of
> > >>>> > >> > this KIP is unnecessary and breaks the well known security
> > >>>> interfaces
> > >>>> > >> > provided by Java itself.
> > >>>> > >> >
> > >>>> > >> > Thanks,
> > >>>> > >> > Harsha
> > >>>> > >> >
> > >>>> > >> > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
> > >>>> kafka@harsha.io
> > >>>> > >
> > >>>> > >> > wrote:
> > >>>> > >> >
> > >>>> > >> > Hi Maulin,
> > >>>> > >> > Not sure if you looked at my previous replies. This
> > >>>> > >> >
> > >>>> > >> > changes
> > >>>> > >> >
> > >>>> > >> > are not required as there is already security Provider to do
> > >>>> what you
> > >>>> > >> are
> > >>>> > >> > proposing. This KIP
> > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/
> > >>>> > >> >
> KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> > >>>> also
> > >>>> > >> > addresses easy registration of such providers.
> > >>>> > >> >
> > >>>> > >> > Thanks,
> > >>>> > >> > Harsha
> > >>>> > >> >
> > >>>> > >> > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
> > >>>> > >> <maulin.vasavada@gmail.
> > >>>> > >> > com> wrote:
> > >>>> > >> >
> > >>>> > >> > Bump! Can somebody please review this?
> > >>>> > >> >
> > >>>> > >> > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
> > >>>> > >> >
> > >>>> > >> > maulin.vasavada@gmail.com>
> > >>>> > >> >
> > >>>> > >> > wrote:
> > >>>> > >> >
> > >>>> > >> > Bump! Can somebody please review this?
> > >>>> > >> >
> > >>>> > >> >
> > >>>> > >>
> > >>>> > >
> > >>>> >
> > >>>>
> > >>>
> >
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Colin McCabe <cm...@apache.org>.
Hi Maulin,
A lot of JSSE providers don't implement custom algorithms. Spire is a good example of a JSSE provider that doesn't, and yet is still useful to many people. Your JSSE provider can work fine even if it doesn't implement a custom algorithm.
Maybe I'm missing something, but I don't understand the discussion of Security.insertProviderAt() that you included. SslEngineBuilder doesn't use that API to get the security provider. Instead, it calls "SSLContext.getInstance(protocol, provider)", where provider is the name of the provider.
best,
Colin
On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote:
> On top of everything above I feel strongly to add the 4th point which is
> based on Java APIs for TrustManagerFactory.init(KeyStore) (
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/TrustManagerFactory.html#init(java.security.KeyStore))
> and KeyManagerFactory.init(KeyStore, char[]) (
> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory.html#init(java.security.KeyStore,%20char[])
> ).
>
> 4. The above APIs are intended to support providing "trust/key material"
> from the user without having to write their own TrustManager/KeyManagers.
> To quote from the TrustManagerFactory.init()'s documentation "Initializes
> this factory with a source of certificate authorities and related trust
> material."
> To quote from the KeyManagerFactory.init()'s documentation "Initializes
> this factory with a source of key material."
>
> Based on this it is clear that there is a flexibility provided by Java to
> to enable developers to provide the required trust/key material loaded from
> "anywhere" without requiring them to write custom provider OR trust/key
> managers. This same flexibility is reflected in Kafka code also where it
> loads the trust/keys from a local file and doesn't require writing a
> Provider necessarily. If we do NOT have a custom algorithm, it makes less
> sense to write a Provider.
>
> Thanks
> Maulin
>
>
>
>
>
>
> On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <ma...@gmail.com>
> wrote:
>
> > Hi Harsha/Colin
> >
> > I did the sample with a custom Provider for TrustStoreManager and tried
> > using ssl.provider Kafka config AND the way KIP-492 is suggesting (by
> > adding Provider programmatically instead of relying on
> > ssl.provider+java.security. The below sample is followed by my detailed
> > findings. I'll appreciate if you can go through it carefully and see if you
> > see my point.
> >
> > package providertest;
> >
> > import java.security.Provider;
> >
> > public class MyProvider extends Provider {
> >
> > private static final String name = "MyProvider";
> > private static double version = 1.0d;
> > private static String info = "Maulin's SSL Provider v"+version;
> >
> > public MyProvider() {
> > super(name, version, info);
> > this.put("TrustManagerFactory.PKIX", "providertest.MyTrustManagerFactory");
> > }
> > }
> >
> >
> >
> > *Details:*
> >
> > KIP-492 documents that it will use Security.addProvider() assuming it will
> > add it as position '0' which is not a correct assumption. The
> > addProvider()'s documentation says it will add it to the last available
> > position. You may want to correct that to say
> > Security.insertProviderAt(provider, 1).
> >
> > Now coming back to our specific discussion,
> >
> > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add that
> > provider in the provider list via Security.addProvider() the position where
> > it gets added doesn't matter (even if you don't end up adding it as first
> > entry) since that is the ONLY provider for SPIFFE specific algorithm you
> > might have.
> >
> > We do *not* have custom algorithm for Key/Trust StoreMangers. Which means
> > we have to use X509, PKIX etc "Standard Algorithms" ((
> > https://docs.oracle.com/javase/7/docs/technotes/guides/security/StandardNames.html))
> > in our provider to override the TrustStoreManager (see my sample code) and
> > KeyStoreManger and KeyManager. This creates another challenge mentioned in
> > the below point.
> >
> > 2. In order to make our Provider for loading custom TrustStore work, we
> > have to add the provider as 'first' in the list since there are others with
> > the same algorithm.
> >
> > However, the programatic way of adding provider
> > (Security.insertProviderAt()) is *not* deterministic for ordering since
> > different code can call the method for a different provider and depending
> > upon the order of the call our provider can be first or pushed down the
> > list. This can happen very well in any client application using Kafka. This
> > is specially problematic for a case when you want to guarantee order for a
> > Provider having "Standard Algorithms".
> >
> > If we add our provider in java.security file that definitely guarantees
> > the order(unless somebody calls removeProvider() which is less likely). But
> > if we add our provider in java.security file it will defeat the purpose of
> > the KIP-492.
> >
> > In the gist - Apache Kafka must not rely on "un-deterministic" method to
> > rely on Provider ordering.
> >
> > 3. If we just use existing ssl.provider kafka configuration then our
> > provider will be used in SSLContext.getInstance(protocol, provider) call in
> > SslFactory.java and if our provider does not have implementation for
> > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we tested it). Example: In
> > MyProvider sample above you see that I didn't add SSLContext.TLSv1 as
> > "Service+Algorithm" and that didn't work for me. In SPIFFE provider you
> > don't have this challenge since you are planning to bypass ssl.provider as
> > you mention in the KIP-492.
> >
> >
> > *Overall summary,*
> >
> > 1. Any provider based mechanisms- a) existing ssl.provider and b)KIP-492,
> > for loading key/trust store using "Standard Algorithms" do not work
> >
> > 2. Approach suggested in our KIP-486 works without any issue and it is
> > *not* our context specific solve
> >
> > 3. Based on above we feel KIP-492 and KIP-486 are complimentary changes
> > and not contradicting or redundent.
> >
> > If you want we can do a joint session somehow to walk through the sample I
> > have and various experiments I did. I would encourage you to do similar
> > exercise by writing a Provider for "Standard Algorithm" for
> > TrustStoreManager (like our needs) and see what you find since only writing
> > samples can bring out the complexity/challenges we face.
> >
> > Thanks
> > Maulin
> >
> > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <
> > maulin.vasavada@gmail.com> wrote:
> >
> >> Just to update - still working on it. Get to work only on and off on it :(
> >>
> >> On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <ma...@gmail.com>
> >> wrote:
> >>
> >>> Hi Harsha
> >>>
> >>> Let me try to write samples and will let you know.
> >>>
> >>> Thanks
> >>> Maulin
> >>>
> >>> On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <ha...@gmail.com> wrote:
> >>>
> >>>> Hi Maulin,
> >>>> With java security providers can be as custom you would like
> >>>> it to
> >>>> be. If you only want to to implement a custom way of loading the
> >>>> keystore and truststore and not implement any protocol/encryption
> >>>> handling
> >>>> you can leave them empty and no need to implement.
> >>>> Have you looked into the links I pasted before?
> >>>>
> >>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java
> >>>>
> >>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java
> >>>>
> >>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
> >>>>
> >>>> Can you please tell me which methods are too complex in above to
> >>>> implement
> >>>> or unnecessary? You are changing anything in SSL/TLS implementations
> >>>> provided by
> >>>>
> >>>> All of the implementations delegating the checks to the default
> >>>> implementation anyway.
> >>>> Spire agent is an example, its nothing but a GRPC server listening on a
> >>>> unix domain socket . Above code is making a RPC call to the local
> >>>> daemon to
> >>>> get the certificate and keys. The mechanics are pretty much same as what
> >>>> you are asking for.
> >>>>
> >>>> Thanks,
> >>>> Harsha
> >>>>
> >>>> On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
> >>>> maulin.vasavada@gmail.com>
> >>>> wrote:
> >>>>
> >>>> > Imagine a scenario like - We know we have a custom KMS and as a Kafka
> >>>> owner
> >>>> > we want to comply to using that KMS source to load keys/certs. As a
> >>>> Kafka
> >>>> > owner we know how to integrate with KMS but doesn't necessarily have
> >>>> to
> >>>> > know anything about cipher suites, algorithms, and SSL/TLS
> >>>> implementation.
> >>>> > Going the Provider way requires to know lot more than we should,
> >>>> isn't it?
> >>>> > Not that we would have concern/shy-away knowing those details - but
> >>>> if we
> >>>> > don't have to - why should we?
> >>>> >
> >>>> > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
> >>>> maulin.vasavada@gmail.com>
> >>>> > wrote:
> >>>> >
> >>>> > > Hi Harsha
> >>>> > >
> >>>> > > We don't have spire (or similar) agents and we do not have
> >>>> keys/certs
> >>>> > > locally on any brokers. To elaborate more on my previous email,
> >>>> > >
> >>>> > > I agree that Java security Providers are used in much broader sense
> >>>> - to
> >>>> > > have a particular implementation of an algorithm, use specific
> >>>> cipher
> >>>> > > suites for SSL , OR in our current team's case have a particular
> >>>> way to
> >>>> > > leverage pre-generated SSL sessions. However, the scope of our KIP
> >>>> (486)
> >>>> > is
> >>>> > > much restricted than that. We merely intend to provide a custom
> >>>> > > keystore/truststore for our SSL connections and not really worry
> >>>> about
> >>>> > > underlying specific SSL/TLS implementation. This simplifies it a
> >>>> lot for
> >>>> > > us to keep the concerns separate and clear.
> >>>> > >
> >>>> > > I feel our approach is more complimentary such that it allows for
> >>>> using
> >>>> > > keystores of choice while retaining the flexibility to use any
> >>>> > > underlying/available Provider for actually making the SSL call.
> >>>> > >
> >>>> > > We agree with KIP-492's approach based on Providers (and Java's
> >>>> > > recommendation), but also strongly believe that our approach can
> >>>> > compliment
> >>>> > > it very effectively for reasons explained above.
> >>>> > >
> >>>> > > Thanks
> >>>> > > Maulin
> >>>> > >
> >>>> > > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <kafka@harsha.io
> >>>> >
> >>>> > > wrote:
> >>>> > >
> >>>> > >> Hi Maulin,
> >>>> > >>
> >>>> > >> On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
> >>>> > >> maulin.vasavada@gmail.com>
> >>>> > >> wrote:
> >>>> > >>
> >>>> > >> > Hi Harsha
> >>>> > >> >
> >>>> > >> > The reason we rejected the SslProvider route is that - we only
> >>>> needed
> >>>> > a
> >>>> > >> > custom way to load keys/certs. Not touch any policy that existing
> >>>> > >> Providers
> >>>> > >> > govern like SunJSSE Provider.
> >>>> > >> >
> >>>> > >>
> >>>> > >> We have exactly the same requirements to load certs and keys
> >>>> through
> >>>> > spire
> >>>> > >> agent. We used security.provider to do that exactly. I am not sure
> >>>> why
> >>>> > you
> >>>> > >> would be modifying any policies provided by default SunJSSE
> >>>> provider.
> >>>> > Can
> >>>> > >> you give me an example of having custom provider that will
> >>>> override an
> >>>> > >> existing policy in SunJSSE provider.
> >>>> > >>
> >>>> > >> As pointed out earlier, this kip
> >>>> > >>
> >>>> > >>
> >>>> >
> >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> >>>> > >> allows
> >>>> > >> you to load security.provider through config.
> >>>> > >> Take a look at the examples I gave before
> >>>> > >>
> >>>> > >>
> >>>> >
> >>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
> >>>> > >> It registers KeyManagerFactory and TrustManagerFactory and Keystore
> >>>> > >> algorithm.
> >>>> > >> Implement your custom way of loading Keystore in here
> >>>> > >>
> >>>> > >>
> >>>> >
> >>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java
> >>>> > >>
> >>>> > >> and Trust manager like here
> >>>> > >>
> >>>> > >>
> >>>> >
> >>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java
> >>>> > >>
> >>>> > >> In your Kafka client you can set the security.provider to your
> >>>> custom
> >>>> > >> implementation and with this fix
> >>>> > >> https://issues.apache.org/jira/browse/KAFKA-8191 you can set
> >>>> > >> keyManagerAlgorigthm and trustManagerAlgorithm configs.
> >>>> > >>
> >>>> > >> All of this is in your clients and broker side and do not need to
> >>>> touch
> >>>> > >> any
> >>>> > >> policy changes at JVM level. You'll register the providers in the
> >>>> > priority
> >>>> > >> order and can still have SunJSSE provider and have your custom
> >>>> provider
> >>>> > to
> >>>> > >> implement the key and trust managers.
> >>>> > >>
> >>>> > >>
> >>>> > >>
> >>>> > >> The ask here is different than KIP-492. We don't have any need to
> >>>> > >> > modify/specify the algorithm parameter. Does that make sense?
> >>>> > >> >
> >>>> > >>
> >>>> > >> The ask in KIP is introducing new interfaces where the KIP's
> >>>> > >> goal/motivation can be achieved through the security.provider and
> >>>> we
> >>>> > >> worked
> >>>> > >> on similar goal without touching any Keystore or Truststore
> >>>> interfaces.
> >>>> > >> My advise is against changing or introducing new interfaces when
> >>>> it can
> >>>> > >> work through security.provider.
> >>>> > >>
> >>>> > >> Thanks,
> >>>> > >> Harsha
> >>>> > >>
> >>>> > >>
> >>>> > >> Thanks
> >>>> > >> > Maulin
> >>>> > >> >
> >>>> > >> > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
> >>>> kafka@harsha.io>
> >>>> > >> > wrote:
> >>>> > >> >
> >>>> > >> > In your KIP you added security. provider as rejected alternative
> >>>> and
> >>>> > >> > specified "its not the correct way". Do you mind explaining why
> >>>> its
> >>>> > >> not? I
> >>>> > >> > didn't find any evidence in Java docs to say so. Contrary to your
> >>>> > >> statement
> >>>> > >> > it does say in the java docs
> >>>> > >> > " However, please note that a provider can be used to implement
> >>>> any
> >>>> > >> > security service in Java that uses a pluggable architecture with
> >>>> a
> >>>> > >> choice
> >>>> > >> > of implementations that fit underneath."
> >>>> > >> >
> >>>> > >> > Java Security Providers have been used by other projects to
> >>>> provide
> >>>> > such
> >>>> > >> > integration . I am not sure if you looked into Spiffe project to
> >>>> > >> > efficiently distribute certificates but here is an example of
> >>>> Java
> >>>> > >> provider
> >>>> > >> >
> >>>> > >> >
> >>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >>>> > >> >
> >>>> > >>
> >>>> >
> >>>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> >>>> > >> > java which
> >>>> > >> > obtains certificates from local daemons.
> >>>> > >> > These integrations are being used in Tomcat, Jetty etc.. We are
> >>>> also
> >>>> > >> using
> >>>> > >> > Security provider to do the same in our Kafka clusters. So
> >>>> unless I
> >>>> > see
> >>>> > >> > more evidence why security.provider doesn't work for you adding
> >>>> new
> >>>> > >> > interfaces while there exists more cleaner way of achieving the
> >>>> goals
> >>>> > of
> >>>> > >> > this KIP is unnecessary and breaks the well known security
> >>>> interfaces
> >>>> > >> > provided by Java itself.
> >>>> > >> >
> >>>> > >> > Thanks,
> >>>> > >> > Harsha
> >>>> > >> >
> >>>> > >> > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
> >>>> kafka@harsha.io
> >>>> > >
> >>>> > >> > wrote:
> >>>> > >> >
> >>>> > >> > Hi Maulin,
> >>>> > >> > Not sure if you looked at my previous replies. This
> >>>> > >> >
> >>>> > >> > changes
> >>>> > >> >
> >>>> > >> > are not required as there is already security Provider to do
> >>>> what you
> >>>> > >> are
> >>>> > >> > proposing. This KIP
> >>>> > https://cwiki.apache.org/confluence/display/KAFKA/
> >>>> > >> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> >>>> also
> >>>> > >> > addresses easy registration of such providers.
> >>>> > >> >
> >>>> > >> > Thanks,
> >>>> > >> > Harsha
> >>>> > >> >
> >>>> > >> > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
> >>>> > >> <maulin.vasavada@gmail.
> >>>> > >> > com> wrote:
> >>>> > >> >
> >>>> > >> > Bump! Can somebody please review this?
> >>>> > >> >
> >>>> > >> > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
> >>>> > >> >
> >>>> > >> > maulin.vasavada@gmail.com>
> >>>> > >> >
> >>>> > >> > wrote:
> >>>> > >> >
> >>>> > >> > Bump! Can somebody please review this?
> >>>> > >> >
> >>>> > >> >
> >>>> > >>
> >>>> > >
> >>>> >
> >>>>
> >>>
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Maulin Vasavada <ma...@gmail.com>.
On top of everything above I feel strongly to add the 4th point which is
based on Java APIs for TrustManagerFactory.init(KeyStore) (
https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/TrustManagerFactory.html#init(java.security.KeyStore))
and KeyManagerFactory.init(KeyStore, char[]) (
https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory.html#init(java.security.KeyStore,%20char[])
).
4. The above APIs are intended to support providing "trust/key material"
from the user without having to write their own TrustManager/KeyManagers.
To quote from the TrustManagerFactory.init()'s documentation "Initializes
this factory with a source of certificate authorities and related trust
material."
To quote from the KeyManagerFactory.init()'s documentation "Initializes
this factory with a source of key material."
Based on this it is clear that there is a flexibility provided by Java to
to enable developers to provide the required trust/key material loaded from
"anywhere" without requiring them to write custom provider OR trust/key
managers. This same flexibility is reflected in Kafka code also where it
loads the trust/keys from a local file and doesn't require writing a
Provider necessarily. If we do NOT have a custom algorithm, it makes less
sense to write a Provider.
Thanks
Maulin
On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada <ma...@gmail.com>
wrote:
> Hi Harsha/Colin
>
> I did the sample with a custom Provider for TrustStoreManager and tried
> using ssl.provider Kafka config AND the way KIP-492 is suggesting (by
> adding Provider programmatically instead of relying on
> ssl.provider+java.security. The below sample is followed by my detailed
> findings. I'll appreciate if you can go through it carefully and see if you
> see my point.
>
> package providertest;
>
> import java.security.Provider;
>
> public class MyProvider extends Provider {
>
> private static final String name = "MyProvider";
> private static double version = 1.0d;
> private static String info = "Maulin's SSL Provider v"+version;
>
> public MyProvider() {
> super(name, version, info);
> this.put("TrustManagerFactory.PKIX", "providertest.MyTrustManagerFactory");
> }
> }
>
>
>
> *Details:*
>
> KIP-492 documents that it will use Security.addProvider() assuming it will
> add it as position '0' which is not a correct assumption. The
> addProvider()'s documentation says it will add it to the last available
> position. You may want to correct that to say
> Security.insertProviderAt(provider, 1).
>
> Now coming back to our specific discussion,
>
> 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add that
> provider in the provider list via Security.addProvider() the position where
> it gets added doesn't matter (even if you don't end up adding it as first
> entry) since that is the ONLY provider for SPIFFE specific algorithm you
> might have.
>
> We do *not* have custom algorithm for Key/Trust StoreMangers. Which means
> we have to use X509, PKIX etc "Standard Algorithms" ((
> https://docs.oracle.com/javase/7/docs/technotes/guides/security/StandardNames.html))
> in our provider to override the TrustStoreManager (see my sample code) and
> KeyStoreManger and KeyManager. This creates another challenge mentioned in
> the below point.
>
> 2. In order to make our Provider for loading custom TrustStore work, we
> have to add the provider as 'first' in the list since there are others with
> the same algorithm.
>
> However, the programatic way of adding provider
> (Security.insertProviderAt()) is *not* deterministic for ordering since
> different code can call the method for a different provider and depending
> upon the order of the call our provider can be first or pushed down the
> list. This can happen very well in any client application using Kafka. This
> is specially problematic for a case when you want to guarantee order for a
> Provider having "Standard Algorithms".
>
> If we add our provider in java.security file that definitely guarantees
> the order(unless somebody calls removeProvider() which is less likely). But
> if we add our provider in java.security file it will defeat the purpose of
> the KIP-492.
>
> In the gist - Apache Kafka must not rely on "un-deterministic" method to
> rely on Provider ordering.
>
> 3. If we just use existing ssl.provider kafka configuration then our
> provider will be used in SSLContext.getInstance(protocol, provider) call in
> SslFactory.java and if our provider does not have implementation for
> SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we tested it). Example: In
> MyProvider sample above you see that I didn't add SSLContext.TLSv1 as
> "Service+Algorithm" and that didn't work for me. In SPIFFE provider you
> don't have this challenge since you are planning to bypass ssl.provider as
> you mention in the KIP-492.
>
>
> *Overall summary,*
>
> 1. Any provider based mechanisms- a) existing ssl.provider and b)KIP-492,
> for loading key/trust store using "Standard Algorithms" do not work
>
> 2. Approach suggested in our KIP-486 works without any issue and it is
> *not* our context specific solve
>
> 3. Based on above we feel KIP-492 and KIP-486 are complimentary changes
> and not contradicting or redundent.
>
> If you want we can do a joint session somehow to walk through the sample I
> have and various experiments I did. I would encourage you to do similar
> exercise by writing a Provider for "Standard Algorithm" for
> TrustStoreManager (like our needs) and see what you find since only writing
> samples can bring out the complexity/challenges we face.
>
> Thanks
> Maulin
>
> On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <
> maulin.vasavada@gmail.com> wrote:
>
>> Just to update - still working on it. Get to work only on and off on it :(
>>
>> On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <ma...@gmail.com>
>> wrote:
>>
>>> Hi Harsha
>>>
>>> Let me try to write samples and will let you know.
>>>
>>> Thanks
>>> Maulin
>>>
>>> On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <ha...@gmail.com> wrote:
>>>
>>>> Hi Maulin,
>>>> With java security providers can be as custom you would like
>>>> it to
>>>> be. If you only want to to implement a custom way of loading the
>>>> keystore and truststore and not implement any protocol/encryption
>>>> handling
>>>> you can leave them empty and no need to implement.
>>>> Have you looked into the links I pasted before?
>>>>
>>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java
>>>>
>>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java
>>>>
>>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
>>>>
>>>> Can you please tell me which methods are too complex in above to
>>>> implement
>>>> or unnecessary? You are changing anything in SSL/TLS implementations
>>>> provided by
>>>>
>>>> All of the implementations delegating the checks to the default
>>>> implementation anyway.
>>>> Spire agent is an example, its nothing but a GRPC server listening on a
>>>> unix domain socket . Above code is making a RPC call to the local
>>>> daemon to
>>>> get the certificate and keys. The mechanics are pretty much same as what
>>>> you are asking for.
>>>>
>>>> Thanks,
>>>> Harsha
>>>>
>>>> On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
>>>> maulin.vasavada@gmail.com>
>>>> wrote:
>>>>
>>>> > Imagine a scenario like - We know we have a custom KMS and as a Kafka
>>>> owner
>>>> > we want to comply to using that KMS source to load keys/certs. As a
>>>> Kafka
>>>> > owner we know how to integrate with KMS but doesn't necessarily have
>>>> to
>>>> > know anything about cipher suites, algorithms, and SSL/TLS
>>>> implementation.
>>>> > Going the Provider way requires to know lot more than we should,
>>>> isn't it?
>>>> > Not that we would have concern/shy-away knowing those details - but
>>>> if we
>>>> > don't have to - why should we?
>>>> >
>>>> > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
>>>> maulin.vasavada@gmail.com>
>>>> > wrote:
>>>> >
>>>> > > Hi Harsha
>>>> > >
>>>> > > We don't have spire (or similar) agents and we do not have
>>>> keys/certs
>>>> > > locally on any brokers. To elaborate more on my previous email,
>>>> > >
>>>> > > I agree that Java security Providers are used in much broader sense
>>>> - to
>>>> > > have a particular implementation of an algorithm, use specific
>>>> cipher
>>>> > > suites for SSL , OR in our current team's case have a particular
>>>> way to
>>>> > > leverage pre-generated SSL sessions. However, the scope of our KIP
>>>> (486)
>>>> > is
>>>> > > much restricted than that. We merely intend to provide a custom
>>>> > > keystore/truststore for our SSL connections and not really worry
>>>> about
>>>> > > underlying specific SSL/TLS implementation. This simplifies it a
>>>> lot for
>>>> > > us to keep the concerns separate and clear.
>>>> > >
>>>> > > I feel our approach is more complimentary such that it allows for
>>>> using
>>>> > > keystores of choice while retaining the flexibility to use any
>>>> > > underlying/available Provider for actually making the SSL call.
>>>> > >
>>>> > > We agree with KIP-492's approach based on Providers (and Java's
>>>> > > recommendation), but also strongly believe that our approach can
>>>> > compliment
>>>> > > it very effectively for reasons explained above.
>>>> > >
>>>> > > Thanks
>>>> > > Maulin
>>>> > >
>>>> > > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <kafka@harsha.io
>>>> >
>>>> > > wrote:
>>>> > >
>>>> > >> Hi Maulin,
>>>> > >>
>>>> > >> On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
>>>> > >> maulin.vasavada@gmail.com>
>>>> > >> wrote:
>>>> > >>
>>>> > >> > Hi Harsha
>>>> > >> >
>>>> > >> > The reason we rejected the SslProvider route is that - we only
>>>> needed
>>>> > a
>>>> > >> > custom way to load keys/certs. Not touch any policy that existing
>>>> > >> Providers
>>>> > >> > govern like SunJSSE Provider.
>>>> > >> >
>>>> > >>
>>>> > >> We have exactly the same requirements to load certs and keys
>>>> through
>>>> > spire
>>>> > >> agent. We used security.provider to do that exactly. I am not sure
>>>> why
>>>> > you
>>>> > >> would be modifying any policies provided by default SunJSSE
>>>> provider.
>>>> > Can
>>>> > >> you give me an example of having custom provider that will
>>>> override an
>>>> > >> existing policy in SunJSSE provider.
>>>> > >>
>>>> > >> As pointed out earlier, this kip
>>>> > >>
>>>> > >>
>>>> >
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>>>> > >> allows
>>>> > >> you to load security.provider through config.
>>>> > >> Take a look at the examples I gave before
>>>> > >>
>>>> > >>
>>>> >
>>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
>>>> > >> It registers KeyManagerFactory and TrustManagerFactory and Keystore
>>>> > >> algorithm.
>>>> > >> Implement your custom way of loading Keystore in here
>>>> > >>
>>>> > >>
>>>> >
>>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java
>>>> > >>
>>>> > >> and Trust manager like here
>>>> > >>
>>>> > >>
>>>> >
>>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java
>>>> > >>
>>>> > >> In your Kafka client you can set the security.provider to your
>>>> custom
>>>> > >> implementation and with this fix
>>>> > >> https://issues.apache.org/jira/browse/KAFKA-8191 you can set
>>>> > >> keyManagerAlgorigthm and trustManagerAlgorithm configs.
>>>> > >>
>>>> > >> All of this is in your clients and broker side and do not need to
>>>> touch
>>>> > >> any
>>>> > >> policy changes at JVM level. You'll register the providers in the
>>>> > priority
>>>> > >> order and can still have SunJSSE provider and have your custom
>>>> provider
>>>> > to
>>>> > >> implement the key and trust managers.
>>>> > >>
>>>> > >>
>>>> > >>
>>>> > >> The ask here is different than KIP-492. We don't have any need to
>>>> > >> > modify/specify the algorithm parameter. Does that make sense?
>>>> > >> >
>>>> > >>
>>>> > >> The ask in KIP is introducing new interfaces where the KIP's
>>>> > >> goal/motivation can be achieved through the security.provider and
>>>> we
>>>> > >> worked
>>>> > >> on similar goal without touching any Keystore or Truststore
>>>> interfaces.
>>>> > >> My advise is against changing or introducing new interfaces when
>>>> it can
>>>> > >> work through security.provider.
>>>> > >>
>>>> > >> Thanks,
>>>> > >> Harsha
>>>> > >>
>>>> > >>
>>>> > >> Thanks
>>>> > >> > Maulin
>>>> > >> >
>>>> > >> > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
>>>> kafka@harsha.io>
>>>> > >> > wrote:
>>>> > >> >
>>>> > >> > In your KIP you added security. provider as rejected alternative
>>>> and
>>>> > >> > specified "its not the correct way". Do you mind explaining why
>>>> its
>>>> > >> not? I
>>>> > >> > didn't find any evidence in Java docs to say so. Contrary to your
>>>> > >> statement
>>>> > >> > it does say in the java docs
>>>> > >> > " However, please note that a provider can be used to implement
>>>> any
>>>> > >> > security service in Java that uses a pluggable architecture with
>>>> a
>>>> > >> choice
>>>> > >> > of implementations that fit underneath."
>>>> > >> >
>>>> > >> > Java Security Providers have been used by other projects to
>>>> provide
>>>> > such
>>>> > >> > integration . I am not sure if you looked into Spiffe project to
>>>> > >> > efficiently distribute certificates but here is an example of
>>>> Java
>>>> > >> provider
>>>> > >> >
>>>> > >> >
>>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>>>> > >> >
>>>> > >>
>>>> >
>>>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>>>> > >> > java which
>>>> > >> > obtains certificates from local daemons.
>>>> > >> > These integrations are being used in Tomcat, Jetty etc.. We are
>>>> also
>>>> > >> using
>>>> > >> > Security provider to do the same in our Kafka clusters. So
>>>> unless I
>>>> > see
>>>> > >> > more evidence why security.provider doesn't work for you adding
>>>> new
>>>> > >> > interfaces while there exists more cleaner way of achieving the
>>>> goals
>>>> > of
>>>> > >> > this KIP is unnecessary and breaks the well known security
>>>> interfaces
>>>> > >> > provided by Java itself.
>>>> > >> >
>>>> > >> > Thanks,
>>>> > >> > Harsha
>>>> > >> >
>>>> > >> > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
>>>> kafka@harsha.io
>>>> > >
>>>> > >> > wrote:
>>>> > >> >
>>>> > >> > Hi Maulin,
>>>> > >> > Not sure if you looked at my previous replies. This
>>>> > >> >
>>>> > >> > changes
>>>> > >> >
>>>> > >> > are not required as there is already security Provider to do
>>>> what you
>>>> > >> are
>>>> > >> > proposing. This KIP
>>>> > https://cwiki.apache.org/confluence/display/KAFKA/
>>>> > >> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>>>> also
>>>> > >> > addresses easy registration of such providers.
>>>> > >> >
>>>> > >> > Thanks,
>>>> > >> > Harsha
>>>> > >> >
>>>> > >> > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
>>>> > >> <maulin.vasavada@gmail.
>>>> > >> > com> wrote:
>>>> > >> >
>>>> > >> > Bump! Can somebody please review this?
>>>> > >> >
>>>> > >> > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
>>>> > >> >
>>>> > >> > maulin.vasavada@gmail.com>
>>>> > >> >
>>>> > >> > wrote:
>>>> > >> >
>>>> > >> > Bump! Can somebody please review this?
>>>> > >> >
>>>> > >> >
>>>> > >>
>>>> > >
>>>> >
>>>>
>>>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Maulin Vasavada <ma...@gmail.com>.
Hi Harsha/Colin
I did the sample with a custom Provider for TrustStoreManager and tried
using ssl.provider Kafka config AND the way KIP-492 is suggesting (by
adding Provider programmatically instead of relying on
ssl.provider+java.security. The below sample is followed by my detailed
findings. I'll appreciate if you can go through it carefully and see if you
see my point.
package providertest;
import java.security.Provider;
public class MyProvider extends Provider {
private static final String name = "MyProvider";
private static double version = 1.0d;
private static String info = "Maulin's SSL Provider v"+version;
public MyProvider() {
super(name, version, info);
this.put("TrustManagerFactory.PKIX",
"providertest.MyTrustManagerFactory");
}
}
*Details:*
KIP-492 documents that it will use Security.addProvider() assuming it will
add it as position '0' which is not a correct assumption. The
addProvider()'s documentation says it will add it to the last available
position. You may want to correct that to say
Security.insertProviderAt(provider, 1).
Now coming back to our specific discussion,
1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add that
provider in the provider list via Security.addProvider() the position where
it gets added doesn't matter (even if you don't end up adding it as first
entry) since that is the ONLY provider for SPIFFE specific algorithm you
might have.
We do *not* have custom algorithm for Key/Trust StoreMangers. Which means
we have to use X509, PKIX etc "Standard Algorithms" ((
https://docs.oracle.com/javase/7/docs/technotes/guides/security/StandardNames.html))
in our provider to override the TrustStoreManager (see my sample code) and
KeyStoreManger and KeyManager. This creates another challenge mentioned in
the below point.
2. In order to make our Provider for loading custom TrustStore work, we
have to add the provider as 'first' in the list since there are others with
the same algorithm.
However, the programatic way of adding provider
(Security.insertProviderAt()) is *not* deterministic for ordering since
different code can call the method for a different provider and depending
upon the order of the call our provider can be first or pushed down the
list. This can happen very well in any client application using Kafka. This
is specially problematic for a case when you want to guarantee order for a
Provider having "Standard Algorithms".
If we add our provider in java.security file that definitely guarantees the
order(unless somebody calls removeProvider() which is less likely). But if
we add our provider in java.security file it will defeat the purpose of the
KIP-492.
In the gist - Apache Kafka must not rely on "un-deterministic" method to
rely on Provider ordering.
3. If we just use existing ssl.provider kafka configuration then our
provider will be used in SSLContext.getInstance(protocol, provider) call in
SslFactory.java and if our provider does not have implementation for
SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we tested it). Example: In
MyProvider sample above you see that I didn't add SSLContext.TLSv1 as
"Service+Algorithm" and that didn't work for me. In SPIFFE provider you
don't have this challenge since you are planning to bypass ssl.provider as
you mention in the KIP-492.
*Overall summary,*
1. Any provider based mechanisms- a) existing ssl.provider and b)KIP-492,
for loading key/trust store using "Standard Algorithms" do not work
2. Approach suggested in our KIP-486 works without any issue and it is *not*
our context specific solve
3. Based on above we feel KIP-492 and KIP-486 are complimentary changes and
not contradicting or redundent.
If you want we can do a joint session somehow to walk through the sample I
have and various experiments I did. I would encourage you to do similar
exercise by writing a Provider for "Standard Algorithm" for
TrustStoreManager (like our needs) and see what you find since only writing
samples can bring out the complexity/challenges we face.
Thanks
Maulin
On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <ma...@gmail.com>
wrote:
> Just to update - still working on it. Get to work only on and off on it :(
>
> On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <ma...@gmail.com>
> wrote:
>
>> Hi Harsha
>>
>> Let me try to write samples and will let you know.
>>
>> Thanks
>> Maulin
>>
>> On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <ha...@gmail.com> wrote:
>>
>>> Hi Maulin,
>>> With java security providers can be as custom you would like it
>>> to
>>> be. If you only want to to implement a custom way of loading the
>>> keystore and truststore and not implement any protocol/encryption
>>> handling
>>> you can leave them empty and no need to implement.
>>> Have you looked into the links I pasted before?
>>>
>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java
>>>
>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java
>>>
>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
>>>
>>> Can you please tell me which methods are too complex in above to
>>> implement
>>> or unnecessary? You are changing anything in SSL/TLS implementations
>>> provided by
>>>
>>> All of the implementations delegating the checks to the default
>>> implementation anyway.
>>> Spire agent is an example, its nothing but a GRPC server listening on a
>>> unix domain socket . Above code is making a RPC call to the local daemon
>>> to
>>> get the certificate and keys. The mechanics are pretty much same as what
>>> you are asking for.
>>>
>>> Thanks,
>>> Harsha
>>>
>>> On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
>>> maulin.vasavada@gmail.com>
>>> wrote:
>>>
>>> > Imagine a scenario like - We know we have a custom KMS and as a Kafka
>>> owner
>>> > we want to comply to using that KMS source to load keys/certs. As a
>>> Kafka
>>> > owner we know how to integrate with KMS but doesn't necessarily have to
>>> > know anything about cipher suites, algorithms, and SSL/TLS
>>> implementation.
>>> > Going the Provider way requires to know lot more than we should, isn't
>>> it?
>>> > Not that we would have concern/shy-away knowing those details - but if
>>> we
>>> > don't have to - why should we?
>>> >
>>> > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
>>> maulin.vasavada@gmail.com>
>>> > wrote:
>>> >
>>> > > Hi Harsha
>>> > >
>>> > > We don't have spire (or similar) agents and we do not have keys/certs
>>> > > locally on any brokers. To elaborate more on my previous email,
>>> > >
>>> > > I agree that Java security Providers are used in much broader sense
>>> - to
>>> > > have a particular implementation of an algorithm, use specific cipher
>>> > > suites for SSL , OR in our current team's case have a particular
>>> way to
>>> > > leverage pre-generated SSL sessions. However, the scope of our KIP
>>> (486)
>>> > is
>>> > > much restricted than that. We merely intend to provide a custom
>>> > > keystore/truststore for our SSL connections and not really worry
>>> about
>>> > > underlying specific SSL/TLS implementation. This simplifies it a
>>> lot for
>>> > > us to keep the concerns separate and clear.
>>> > >
>>> > > I feel our approach is more complimentary such that it allows for
>>> using
>>> > > keystores of choice while retaining the flexibility to use any
>>> > > underlying/available Provider for actually making the SSL call.
>>> > >
>>> > > We agree with KIP-492's approach based on Providers (and Java's
>>> > > recommendation), but also strongly believe that our approach can
>>> > compliment
>>> > > it very effectively for reasons explained above.
>>> > >
>>> > > Thanks
>>> > > Maulin
>>> > >
>>> > > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <ka...@harsha.io>
>>> > > wrote:
>>> > >
>>> > >> Hi Maulin,
>>> > >>
>>> > >> On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
>>> > >> maulin.vasavada@gmail.com>
>>> > >> wrote:
>>> > >>
>>> > >> > Hi Harsha
>>> > >> >
>>> > >> > The reason we rejected the SslProvider route is that - we only
>>> needed
>>> > a
>>> > >> > custom way to load keys/certs. Not touch any policy that existing
>>> > >> Providers
>>> > >> > govern like SunJSSE Provider.
>>> > >> >
>>> > >>
>>> > >> We have exactly the same requirements to load certs and keys through
>>> > spire
>>> > >> agent. We used security.provider to do that exactly. I am not sure
>>> why
>>> > you
>>> > >> would be modifying any policies provided by default SunJSSE
>>> provider.
>>> > Can
>>> > >> you give me an example of having custom provider that will override
>>> an
>>> > >> existing policy in SunJSSE provider.
>>> > >>
>>> > >> As pointed out earlier, this kip
>>> > >>
>>> > >>
>>> >
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>>> > >> allows
>>> > >> you to load security.provider through config.
>>> > >> Take a look at the examples I gave before
>>> > >>
>>> > >>
>>> >
>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
>>> > >> It registers KeyManagerFactory and TrustManagerFactory and Keystore
>>> > >> algorithm.
>>> > >> Implement your custom way of loading Keystore in here
>>> > >>
>>> > >>
>>> >
>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java
>>> > >>
>>> > >> and Trust manager like here
>>> > >>
>>> > >>
>>> >
>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java
>>> > >>
>>> > >> In your Kafka client you can set the security.provider to your
>>> custom
>>> > >> implementation and with this fix
>>> > >> https://issues.apache.org/jira/browse/KAFKA-8191 you can set
>>> > >> keyManagerAlgorigthm and trustManagerAlgorithm configs.
>>> > >>
>>> > >> All of this is in your clients and broker side and do not need to
>>> touch
>>> > >> any
>>> > >> policy changes at JVM level. You'll register the providers in the
>>> > priority
>>> > >> order and can still have SunJSSE provider and have your custom
>>> provider
>>> > to
>>> > >> implement the key and trust managers.
>>> > >>
>>> > >>
>>> > >>
>>> > >> The ask here is different than KIP-492. We don't have any need to
>>> > >> > modify/specify the algorithm parameter. Does that make sense?
>>> > >> >
>>> > >>
>>> > >> The ask in KIP is introducing new interfaces where the KIP's
>>> > >> goal/motivation can be achieved through the security.provider and we
>>> > >> worked
>>> > >> on similar goal without touching any Keystore or Truststore
>>> interfaces.
>>> > >> My advise is against changing or introducing new interfaces when it
>>> can
>>> > >> work through security.provider.
>>> > >>
>>> > >> Thanks,
>>> > >> Harsha
>>> > >>
>>> > >>
>>> > >> Thanks
>>> > >> > Maulin
>>> > >> >
>>> > >> > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
>>> kafka@harsha.io>
>>> > >> > wrote:
>>> > >> >
>>> > >> > In your KIP you added security. provider as rejected alternative
>>> and
>>> > >> > specified "its not the correct way". Do you mind explaining why
>>> its
>>> > >> not? I
>>> > >> > didn't find any evidence in Java docs to say so. Contrary to your
>>> > >> statement
>>> > >> > it does say in the java docs
>>> > >> > " However, please note that a provider can be used to implement
>>> any
>>> > >> > security service in Java that uses a pluggable architecture with a
>>> > >> choice
>>> > >> > of implementations that fit underneath."
>>> > >> >
>>> > >> > Java Security Providers have been used by other projects to
>>> provide
>>> > such
>>> > >> > integration . I am not sure if you looked into Spiffe project to
>>> > >> > efficiently distribute certificates but here is an example of Java
>>> > >> provider
>>> > >> >
>>> > >> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>>> > >> >
>>> > >>
>>> >
>>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>>> > >> > java which
>>> > >> > obtains certificates from local daemons.
>>> > >> > These integrations are being used in Tomcat, Jetty etc.. We are
>>> also
>>> > >> using
>>> > >> > Security provider to do the same in our Kafka clusters. So unless
>>> I
>>> > see
>>> > >> > more evidence why security.provider doesn't work for you adding
>>> new
>>> > >> > interfaces while there exists more cleaner way of achieving the
>>> goals
>>> > of
>>> > >> > this KIP is unnecessary and breaks the well known security
>>> interfaces
>>> > >> > provided by Java itself.
>>> > >> >
>>> > >> > Thanks,
>>> > >> > Harsha
>>> > >> >
>>> > >> > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
>>> kafka@harsha.io
>>> > >
>>> > >> > wrote:
>>> > >> >
>>> > >> > Hi Maulin,
>>> > >> > Not sure if you looked at my previous replies. This
>>> > >> >
>>> > >> > changes
>>> > >> >
>>> > >> > are not required as there is already security Provider to do what
>>> you
>>> > >> are
>>> > >> > proposing. This KIP
>>> > https://cwiki.apache.org/confluence/display/KAFKA/
>>> > >> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>>> also
>>> > >> > addresses easy registration of such providers.
>>> > >> >
>>> > >> > Thanks,
>>> > >> > Harsha
>>> > >> >
>>> > >> > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
>>> > >> <maulin.vasavada@gmail.
>>> > >> > com> wrote:
>>> > >> >
>>> > >> > Bump! Can somebody please review this?
>>> > >> >
>>> > >> > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
>>> > >> >
>>> > >> > maulin.vasavada@gmail.com>
>>> > >> >
>>> > >> > wrote:
>>> > >> >
>>> > >> > Bump! Can somebody please review this?
>>> > >> >
>>> > >> >
>>> > >>
>>> > >
>>> >
>>>
>>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Maulin Vasavada <ma...@gmail.com>.
Just to update - still working on it. Get to work only on and off on it :(
On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <ma...@gmail.com>
wrote:
> Hi Harsha
>
> Let me try to write samples and will let you know.
>
> Thanks
> Maulin
>
> On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <ha...@gmail.com> wrote:
>
>> Hi Maulin,
>> With java security providers can be as custom you would like it
>> to
>> be. If you only want to to implement a custom way of loading the
>> keystore and truststore and not implement any protocol/encryption handling
>> you can leave them empty and no need to implement.
>> Have you looked into the links I pasted before?
>>
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java
>>
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java
>>
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
>>
>> Can you please tell me which methods are too complex in above to implement
>> or unnecessary? You are changing anything in SSL/TLS implementations
>> provided by
>>
>> All of the implementations delegating the checks to the default
>> implementation anyway.
>> Spire agent is an example, its nothing but a GRPC server listening on a
>> unix domain socket . Above code is making a RPC call to the local daemon
>> to
>> get the certificate and keys. The mechanics are pretty much same as what
>> you are asking for.
>>
>> Thanks,
>> Harsha
>>
>> On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <maulin.vasavada@gmail.com
>> >
>> wrote:
>>
>> > Imagine a scenario like - We know we have a custom KMS and as a Kafka
>> owner
>> > we want to comply to using that KMS source to load keys/certs. As a
>> Kafka
>> > owner we know how to integrate with KMS but doesn't necessarily have to
>> > know anything about cipher suites, algorithms, and SSL/TLS
>> implementation.
>> > Going the Provider way requires to know lot more than we should, isn't
>> it?
>> > Not that we would have concern/shy-away knowing those details - but if
>> we
>> > don't have to - why should we?
>> >
>> > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
>> maulin.vasavada@gmail.com>
>> > wrote:
>> >
>> > > Hi Harsha
>> > >
>> > > We don't have spire (or similar) agents and we do not have keys/certs
>> > > locally on any brokers. To elaborate more on my previous email,
>> > >
>> > > I agree that Java security Providers are used in much broader sense -
>> to
>> > > have a particular implementation of an algorithm, use specific cipher
>> > > suites for SSL , OR in our current team's case have a particular way
>> to
>> > > leverage pre-generated SSL sessions. However, the scope of our KIP
>> (486)
>> > is
>> > > much restricted than that. We merely intend to provide a custom
>> > > keystore/truststore for our SSL connections and not really worry about
>> > > underlying specific SSL/TLS implementation. This simplifies it a lot
>> for
>> > > us to keep the concerns separate and clear.
>> > >
>> > > I feel our approach is more complimentary such that it allows for
>> using
>> > > keystores of choice while retaining the flexibility to use any
>> > > underlying/available Provider for actually making the SSL call.
>> > >
>> > > We agree with KIP-492's approach based on Providers (and Java's
>> > > recommendation), but also strongly believe that our approach can
>> > compliment
>> > > it very effectively for reasons explained above.
>> > >
>> > > Thanks
>> > > Maulin
>> > >
>> > > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <ka...@harsha.io>
>> > > wrote:
>> > >
>> > >> Hi Maulin,
>> > >>
>> > >> On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
>> > >> maulin.vasavada@gmail.com>
>> > >> wrote:
>> > >>
>> > >> > Hi Harsha
>> > >> >
>> > >> > The reason we rejected the SslProvider route is that - we only
>> needed
>> > a
>> > >> > custom way to load keys/certs. Not touch any policy that existing
>> > >> Providers
>> > >> > govern like SunJSSE Provider.
>> > >> >
>> > >>
>> > >> We have exactly the same requirements to load certs and keys through
>> > spire
>> > >> agent. We used security.provider to do that exactly. I am not sure
>> why
>> > you
>> > >> would be modifying any policies provided by default SunJSSE provider.
>> > Can
>> > >> you give me an example of having custom provider that will override
>> an
>> > >> existing policy in SunJSSE provider.
>> > >>
>> > >> As pointed out earlier, this kip
>> > >>
>> > >>
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>> > >> allows
>> > >> you to load security.provider through config.
>> > >> Take a look at the examples I gave before
>> > >>
>> > >>
>> >
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
>> > >> It registers KeyManagerFactory and TrustManagerFactory and Keystore
>> > >> algorithm.
>> > >> Implement your custom way of loading Keystore in here
>> > >>
>> > >>
>> >
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java
>> > >>
>> > >> and Trust manager like here
>> > >>
>> > >>
>> >
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java
>> > >>
>> > >> In your Kafka client you can set the security.provider to your
>> custom
>> > >> implementation and with this fix
>> > >> https://issues.apache.org/jira/browse/KAFKA-8191 you can set
>> > >> keyManagerAlgorigthm and trustManagerAlgorithm configs.
>> > >>
>> > >> All of this is in your clients and broker side and do not need to
>> touch
>> > >> any
>> > >> policy changes at JVM level. You'll register the providers in the
>> > priority
>> > >> order and can still have SunJSSE provider and have your custom
>> provider
>> > to
>> > >> implement the key and trust managers.
>> > >>
>> > >>
>> > >>
>> > >> The ask here is different than KIP-492. We don't have any need to
>> > >> > modify/specify the algorithm parameter. Does that make sense?
>> > >> >
>> > >>
>> > >> The ask in KIP is introducing new interfaces where the KIP's
>> > >> goal/motivation can be achieved through the security.provider and we
>> > >> worked
>> > >> on similar goal without touching any Keystore or Truststore
>> interfaces.
>> > >> My advise is against changing or introducing new interfaces when it
>> can
>> > >> work through security.provider.
>> > >>
>> > >> Thanks,
>> > >> Harsha
>> > >>
>> > >>
>> > >> Thanks
>> > >> > Maulin
>> > >> >
>> > >> > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
>> kafka@harsha.io>
>> > >> > wrote:
>> > >> >
>> > >> > In your KIP you added security. provider as rejected alternative
>> and
>> > >> > specified "its not the correct way". Do you mind explaining why its
>> > >> not? I
>> > >> > didn't find any evidence in Java docs to say so. Contrary to your
>> > >> statement
>> > >> > it does say in the java docs
>> > >> > " However, please note that a provider can be used to implement any
>> > >> > security service in Java that uses a pluggable architecture with a
>> > >> choice
>> > >> > of implementations that fit underneath."
>> > >> >
>> > >> > Java Security Providers have been used by other projects to provide
>> > such
>> > >> > integration . I am not sure if you looked into Spiffe project to
>> > >> > efficiently distribute certificates but here is an example of Java
>> > >> provider
>> > >> >
>> > >> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>> > >> >
>> > >>
>> >
>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>> > >> > java which
>> > >> > obtains certificates from local daemons.
>> > >> > These integrations are being used in Tomcat, Jetty etc.. We are
>> also
>> > >> using
>> > >> > Security provider to do the same in our Kafka clusters. So unless I
>> > see
>> > >> > more evidence why security.provider doesn't work for you adding new
>> > >> > interfaces while there exists more cleaner way of achieving the
>> goals
>> > of
>> > >> > this KIP is unnecessary and breaks the well known security
>> interfaces
>> > >> > provided by Java itself.
>> > >> >
>> > >> > Thanks,
>> > >> > Harsha
>> > >> >
>> > >> > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
>> kafka@harsha.io
>> > >
>> > >> > wrote:
>> > >> >
>> > >> > Hi Maulin,
>> > >> > Not sure if you looked at my previous replies. This
>> > >> >
>> > >> > changes
>> > >> >
>> > >> > are not required as there is already security Provider to do what
>> you
>> > >> are
>> > >> > proposing. This KIP
>> > https://cwiki.apache.org/confluence/display/KAFKA/
>> > >> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>> also
>> > >> > addresses easy registration of such providers.
>> > >> >
>> > >> > Thanks,
>> > >> > Harsha
>> > >> >
>> > >> > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
>> > >> <maulin.vasavada@gmail.
>> > >> > com> wrote:
>> > >> >
>> > >> > Bump! Can somebody please review this?
>> > >> >
>> > >> > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
>> > >> >
>> > >> > maulin.vasavada@gmail.com>
>> > >> >
>> > >> > wrote:
>> > >> >
>> > >> > Bump! Can somebody please review this?
>> > >> >
>> > >> >
>> > >>
>> > >
>> >
>>
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Maulin Vasavada <ma...@gmail.com>.
Hi Harsha
Let me try to write samples and will let you know.
Thanks
Maulin
On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <ha...@gmail.com> wrote:
> Hi Maulin,
> With java security providers can be as custom you would like it to
> be. If you only want to to implement a custom way of loading the
> keystore and truststore and not implement any protocol/encryption handling
> you can leave them empty and no need to implement.
> Have you looked into the links I pasted before?
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
>
> Can you please tell me which methods are too complex in above to implement
> or unnecessary? You are changing anything in SSL/TLS implementations
> provided by
>
> All of the implementations delegating the checks to the default
> implementation anyway.
> Spire agent is an example, its nothing but a GRPC server listening on a
> unix domain socket . Above code is making a RPC call to the local daemon to
> get the certificate and keys. The mechanics are pretty much same as what
> you are asking for.
>
> Thanks,
> Harsha
>
> On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <ma...@gmail.com>
> wrote:
>
> > Imagine a scenario like - We know we have a custom KMS and as a Kafka
> owner
> > we want to comply to using that KMS source to load keys/certs. As a Kafka
> > owner we know how to integrate with KMS but doesn't necessarily have to
> > know anything about cipher suites, algorithms, and SSL/TLS
> implementation.
> > Going the Provider way requires to know lot more than we should, isn't
> it?
> > Not that we would have concern/shy-away knowing those details - but if we
> > don't have to - why should we?
> >
> > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
> maulin.vasavada@gmail.com>
> > wrote:
> >
> > > Hi Harsha
> > >
> > > We don't have spire (or similar) agents and we do not have keys/certs
> > > locally on any brokers. To elaborate more on my previous email,
> > >
> > > I agree that Java security Providers are used in much broader sense -
> to
> > > have a particular implementation of an algorithm, use specific cipher
> > > suites for SSL , OR in our current team's case have a particular way
> to
> > > leverage pre-generated SSL sessions. However, the scope of our KIP
> (486)
> > is
> > > much restricted than that. We merely intend to provide a custom
> > > keystore/truststore for our SSL connections and not really worry about
> > > underlying specific SSL/TLS implementation. This simplifies it a lot
> for
> > > us to keep the concerns separate and clear.
> > >
> > > I feel our approach is more complimentary such that it allows for using
> > > keystores of choice while retaining the flexibility to use any
> > > underlying/available Provider for actually making the SSL call.
> > >
> > > We agree with KIP-492's approach based on Providers (and Java's
> > > recommendation), but also strongly believe that our approach can
> > compliment
> > > it very effectively for reasons explained above.
> > >
> > > Thanks
> > > Maulin
> > >
> > > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <ka...@harsha.io>
> > > wrote:
> > >
> > >> Hi Maulin,
> > >>
> > >> On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
> > >> maulin.vasavada@gmail.com>
> > >> wrote:
> > >>
> > >> > Hi Harsha
> > >> >
> > >> > The reason we rejected the SslProvider route is that - we only
> needed
> > a
> > >> > custom way to load keys/certs. Not touch any policy that existing
> > >> Providers
> > >> > govern like SunJSSE Provider.
> > >> >
> > >>
> > >> We have exactly the same requirements to load certs and keys through
> > spire
> > >> agent. We used security.provider to do that exactly. I am not sure why
> > you
> > >> would be modifying any policies provided by default SunJSSE provider.
> > Can
> > >> you give me an example of having custom provider that will override an
> > >> existing policy in SunJSSE provider.
> > >>
> > >> As pointed out earlier, this kip
> > >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> > >> allows
> > >> you to load security.provider through config.
> > >> Take a look at the examples I gave before
> > >>
> > >>
> >
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
> > >> It registers KeyManagerFactory and TrustManagerFactory and Keystore
> > >> algorithm.
> > >> Implement your custom way of loading Keystore in here
> > >>
> > >>
> >
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java
> > >>
> > >> and Trust manager like here
> > >>
> > >>
> >
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java
> > >>
> > >> In your Kafka client you can set the security.provider to your custom
> > >> implementation and with this fix
> > >> https://issues.apache.org/jira/browse/KAFKA-8191 you can set
> > >> keyManagerAlgorigthm and trustManagerAlgorithm configs.
> > >>
> > >> All of this is in your clients and broker side and do not need to
> touch
> > >> any
> > >> policy changes at JVM level. You'll register the providers in the
> > priority
> > >> order and can still have SunJSSE provider and have your custom
> provider
> > to
> > >> implement the key and trust managers.
> > >>
> > >>
> > >>
> > >> The ask here is different than KIP-492. We don't have any need to
> > >> > modify/specify the algorithm parameter. Does that make sense?
> > >> >
> > >>
> > >> The ask in KIP is introducing new interfaces where the KIP's
> > >> goal/motivation can be achieved through the security.provider and we
> > >> worked
> > >> on similar goal without touching any Keystore or Truststore
> interfaces.
> > >> My advise is against changing or introducing new interfaces when it
> can
> > >> work through security.provider.
> > >>
> > >> Thanks,
> > >> Harsha
> > >>
> > >>
> > >> Thanks
> > >> > Maulin
> > >> >
> > >> > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <kafka@harsha.io
> >
> > >> > wrote:
> > >> >
> > >> > In your KIP you added security. provider as rejected alternative and
> > >> > specified "its not the correct way". Do you mind explaining why its
> > >> not? I
> > >> > didn't find any evidence in Java docs to say so. Contrary to your
> > >> statement
> > >> > it does say in the java docs
> > >> > " However, please note that a provider can be used to implement any
> > >> > security service in Java that uses a pluggable architecture with a
> > >> choice
> > >> > of implementations that fit underneath."
> > >> >
> > >> > Java Security Providers have been used by other projects to provide
> > such
> > >> > integration . I am not sure if you looked into Spiffe project to
> > >> > efficiently distribute certificates but here is an example of Java
> > >> provider
> > >> >
> > >> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> > >> >
> > >>
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > >> > java which
> > >> > obtains certificates from local daemons.
> > >> > These integrations are being used in Tomcat, Jetty etc.. We are also
> > >> using
> > >> > Security provider to do the same in our Kafka clusters. So unless I
> > see
> > >> > more evidence why security.provider doesn't work for you adding new
> > >> > interfaces while there exists more cleaner way of achieving the
> goals
> > of
> > >> > this KIP is unnecessary and breaks the well known security
> interfaces
> > >> > provided by Java itself.
> > >> >
> > >> > Thanks,
> > >> > Harsha
> > >> >
> > >> > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
> kafka@harsha.io
> > >
> > >> > wrote:
> > >> >
> > >> > Hi Maulin,
> > >> > Not sure if you looked at my previous replies. This
> > >> >
> > >> > changes
> > >> >
> > >> > are not required as there is already security Provider to do what
> you
> > >> are
> > >> > proposing. This KIP
> > https://cwiki.apache.org/confluence/display/KAFKA/
> > >> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config also
> > >> > addresses easy registration of such providers.
> > >> >
> > >> > Thanks,
> > >> > Harsha
> > >> >
> > >> > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
> > >> <maulin.vasavada@gmail.
> > >> > com> wrote:
> > >> >
> > >> > Bump! Can somebody please review this?
> > >> >
> > >> > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
> > >> >
> > >> > maulin.vasavada@gmail.com>
> > >> >
> > >> > wrote:
> > >> >
> > >> > Bump! Can somebody please review this?
> > >> >
> > >> >
> > >>
> > >
> >
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Harsha Ch <ha...@gmail.com>.
Hi Maulin,
With java security providers can be as custom you would like it to
be. If you only want to to implement a custom way of loading the
keystore and truststore and not implement any protocol/encryption handling
you can leave them empty and no need to implement.
Have you looked into the links I pasted before?
https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java
https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java
https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
Can you please tell me which methods are too complex in above to implement
or unnecessary? You are changing anything in SSL/TLS implementations
provided by
All of the implementations delegating the checks to the default
implementation anyway.
Spire agent is an example, its nothing but a GRPC server listening on a
unix domain socket . Above code is making a RPC call to the local daemon to
get the certificate and keys. The mechanics are pretty much same as what
you are asking for.
Thanks,
Harsha
On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <ma...@gmail.com>
wrote:
> Imagine a scenario like - We know we have a custom KMS and as a Kafka owner
> we want to comply to using that KMS source to load keys/certs. As a Kafka
> owner we know how to integrate with KMS but doesn't necessarily have to
> know anything about cipher suites, algorithms, and SSL/TLS implementation.
> Going the Provider way requires to know lot more than we should, isn't it?
> Not that we would have concern/shy-away knowing those details - but if we
> don't have to - why should we?
>
> On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <ma...@gmail.com>
> wrote:
>
> > Hi Harsha
> >
> > We don't have spire (or similar) agents and we do not have keys/certs
> > locally on any brokers. To elaborate more on my previous email,
> >
> > I agree that Java security Providers are used in much broader sense - to
> > have a particular implementation of an algorithm, use specific cipher
> > suites for SSL , OR in our current team's case have a particular way to
> > leverage pre-generated SSL sessions. However, the scope of our KIP (486)
> is
> > much restricted than that. We merely intend to provide a custom
> > keystore/truststore for our SSL connections and not really worry about
> > underlying specific SSL/TLS implementation. This simplifies it a lot for
> > us to keep the concerns separate and clear.
> >
> > I feel our approach is more complimentary such that it allows for using
> > keystores of choice while retaining the flexibility to use any
> > underlying/available Provider for actually making the SSL call.
> >
> > We agree with KIP-492's approach based on Providers (and Java's
> > recommendation), but also strongly believe that our approach can
> compliment
> > it very effectively for reasons explained above.
> >
> > Thanks
> > Maulin
> >
> > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <ka...@harsha.io>
> > wrote:
> >
> >> Hi Maulin,
> >>
> >> On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
> >> maulin.vasavada@gmail.com>
> >> wrote:
> >>
> >> > Hi Harsha
> >> >
> >> > The reason we rejected the SslProvider route is that - we only needed
> a
> >> > custom way to load keys/certs. Not touch any policy that existing
> >> Providers
> >> > govern like SunJSSE Provider.
> >> >
> >>
> >> We have exactly the same requirements to load certs and keys through
> spire
> >> agent. We used security.provider to do that exactly. I am not sure why
> you
> >> would be modifying any policies provided by default SunJSSE provider.
> Can
> >> you give me an example of having custom provider that will override an
> >> existing policy in SunJSSE provider.
> >>
> >> As pointed out earlier, this kip
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> >> allows
> >> you to load security.provider through config.
> >> Take a look at the examples I gave before
> >>
> >>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
> >> It registers KeyManagerFactory and TrustManagerFactory and Keystore
> >> algorithm.
> >> Implement your custom way of loading Keystore in here
> >>
> >>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java
> >>
> >> and Trust manager like here
> >>
> >>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java
> >>
> >> In your Kafka client you can set the security.provider to your custom
> >> implementation and with this fix
> >> https://issues.apache.org/jira/browse/KAFKA-8191 you can set
> >> keyManagerAlgorigthm and trustManagerAlgorithm configs.
> >>
> >> All of this is in your clients and broker side and do not need to touch
> >> any
> >> policy changes at JVM level. You'll register the providers in the
> priority
> >> order and can still have SunJSSE provider and have your custom provider
> to
> >> implement the key and trust managers.
> >>
> >>
> >>
> >> The ask here is different than KIP-492. We don't have any need to
> >> > modify/specify the algorithm parameter. Does that make sense?
> >> >
> >>
> >> The ask in KIP is introducing new interfaces where the KIP's
> >> goal/motivation can be achieved through the security.provider and we
> >> worked
> >> on similar goal without touching any Keystore or Truststore interfaces.
> >> My advise is against changing or introducing new interfaces when it can
> >> work through security.provider.
> >>
> >> Thanks,
> >> Harsha
> >>
> >>
> >> Thanks
> >> > Maulin
> >> >
> >> > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <ka...@harsha.io>
> >> > wrote:
> >> >
> >> > In your KIP you added security. provider as rejected alternative and
> >> > specified "its not the correct way". Do you mind explaining why its
> >> not? I
> >> > didn't find any evidence in Java docs to say so. Contrary to your
> >> statement
> >> > it does say in the java docs
> >> > " However, please note that a provider can be used to implement any
> >> > security service in Java that uses a pluggable architecture with a
> >> choice
> >> > of implementations that fit underneath."
> >> >
> >> > Java Security Providers have been used by other projects to provide
> such
> >> > integration . I am not sure if you looked into Spiffe project to
> >> > efficiently distribute certificates but here is an example of Java
> >> provider
> >> >
> >> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >> >
> >>
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> >> > java which
> >> > obtains certificates from local daemons.
> >> > These integrations are being used in Tomcat, Jetty etc.. We are also
> >> using
> >> > Security provider to do the same in our Kafka clusters. So unless I
> see
> >> > more evidence why security.provider doesn't work for you adding new
> >> > interfaces while there exists more cleaner way of achieving the goals
> of
> >> > this KIP is unnecessary and breaks the well known security interfaces
> >> > provided by Java itself.
> >> >
> >> > Thanks,
> >> > Harsha
> >> >
> >> > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <kafka@harsha.io
> >
> >> > wrote:
> >> >
> >> > Hi Maulin,
> >> > Not sure if you looked at my previous replies. This
> >> >
> >> > changes
> >> >
> >> > are not required as there is already security Provider to do what you
> >> are
> >> > proposing. This KIP
> https://cwiki.apache.org/confluence/display/KAFKA/
> >> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config also
> >> > addresses easy registration of such providers.
> >> >
> >> > Thanks,
> >> > Harsha
> >> >
> >> > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
> >> <maulin.vasavada@gmail.
> >> > com> wrote:
> >> >
> >> > Bump! Can somebody please review this?
> >> >
> >> > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
> >> >
> >> > maulin.vasavada@gmail.com>
> >> >
> >> > wrote:
> >> >
> >> > Bump! Can somebody please review this?
> >> >
> >> >
> >>
> >
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Maulin Vasavada <ma...@gmail.com>.
Imagine a scenario like - We know we have a custom KMS and as a Kafka owner
we want to comply to using that KMS source to load keys/certs. As a Kafka
owner we know how to integrate with KMS but doesn't necessarily have to
know anything about cipher suites, algorithms, and SSL/TLS implementation.
Going the Provider way requires to know lot more than we should, isn't it?
Not that we would have concern/shy-away knowing those details - but if we
don't have to - why should we?
On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <ma...@gmail.com>
wrote:
> Hi Harsha
>
> We don't have spire (or similar) agents and we do not have keys/certs
> locally on any brokers. To elaborate more on my previous email,
>
> I agree that Java security Providers are used in much broader sense - to
> have a particular implementation of an algorithm, use specific cipher
> suites for SSL , OR in our current team's case have a particular way to
> leverage pre-generated SSL sessions. However, the scope of our KIP (486) is
> much restricted than that. We merely intend to provide a custom
> keystore/truststore for our SSL connections and not really worry about
> underlying specific SSL/TLS implementation. This simplifies it a lot for
> us to keep the concerns separate and clear.
>
> I feel our approach is more complimentary such that it allows for using
> keystores of choice while retaining the flexibility to use any
> underlying/available Provider for actually making the SSL call.
>
> We agree with KIP-492's approach based on Providers (and Java's
> recommendation), but also strongly believe that our approach can compliment
> it very effectively for reasons explained above.
>
> Thanks
> Maulin
>
> On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <ka...@harsha.io>
> wrote:
>
>> Hi Maulin,
>>
>> On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
>> maulin.vasavada@gmail.com>
>> wrote:
>>
>> > Hi Harsha
>> >
>> > The reason we rejected the SslProvider route is that - we only needed a
>> > custom way to load keys/certs. Not touch any policy that existing
>> Providers
>> > govern like SunJSSE Provider.
>> >
>>
>> We have exactly the same requirements to load certs and keys through spire
>> agent. We used security.provider to do that exactly. I am not sure why you
>> would be modifying any policies provided by default SunJSSE provider. Can
>> you give me an example of having custom provider that will override an
>> existing policy in SunJSSE provider.
>>
>> As pointed out earlier, this kip
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>> allows
>> you to load security.provider through config.
>> Take a look at the examples I gave before
>>
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
>> It registers KeyManagerFactory and TrustManagerFactory and Keystore
>> algorithm.
>> Implement your custom way of loading Keystore in here
>>
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java
>>
>> and Trust manager like here
>>
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java
>>
>> In your Kafka client you can set the security.provider to your custom
>> implementation and with this fix
>> https://issues.apache.org/jira/browse/KAFKA-8191 you can set
>> keyManagerAlgorigthm and trustManagerAlgorithm configs.
>>
>> All of this is in your clients and broker side and do not need to touch
>> any
>> policy changes at JVM level. You'll register the providers in the priority
>> order and can still have SunJSSE provider and have your custom provider to
>> implement the key and trust managers.
>>
>>
>>
>> The ask here is different than KIP-492. We don't have any need to
>> > modify/specify the algorithm parameter. Does that make sense?
>> >
>>
>> The ask in KIP is introducing new interfaces where the KIP's
>> goal/motivation can be achieved through the security.provider and we
>> worked
>> on similar goal without touching any Keystore or Truststore interfaces.
>> My advise is against changing or introducing new interfaces when it can
>> work through security.provider.
>>
>> Thanks,
>> Harsha
>>
>>
>> Thanks
>> > Maulin
>> >
>> > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <ka...@harsha.io>
>> > wrote:
>> >
>> > In your KIP you added security. provider as rejected alternative and
>> > specified "its not the correct way". Do you mind explaining why its
>> not? I
>> > didn't find any evidence in Java docs to say so. Contrary to your
>> statement
>> > it does say in the java docs
>> > " However, please note that a provider can be used to implement any
>> > security service in Java that uses a pluggable architecture with a
>> choice
>> > of implementations that fit underneath."
>> >
>> > Java Security Providers have been used by other projects to provide such
>> > integration . I am not sure if you looked into Spiffe project to
>> > efficiently distribute certificates but here is an example of Java
>> provider
>> >
>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>> >
>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>> > java which
>> > obtains certificates from local daemons.
>> > These integrations are being used in Tomcat, Jetty etc.. We are also
>> using
>> > Security provider to do the same in our Kafka clusters. So unless I see
>> > more evidence why security.provider doesn't work for you adding new
>> > interfaces while there exists more cleaner way of achieving the goals of
>> > this KIP is unnecessary and breaks the well known security interfaces
>> > provided by Java itself.
>> >
>> > Thanks,
>> > Harsha
>> >
>> > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <ka...@harsha.io>
>> > wrote:
>> >
>> > Hi Maulin,
>> > Not sure if you looked at my previous replies. This
>> >
>> > changes
>> >
>> > are not required as there is already security Provider to do what you
>> are
>> > proposing. This KIP https://cwiki.apache.org/confluence/display/KAFKA/
>> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config also
>> > addresses easy registration of such providers.
>> >
>> > Thanks,
>> > Harsha
>> >
>> > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
>> <maulin.vasavada@gmail.
>> > com> wrote:
>> >
>> > Bump! Can somebody please review this?
>> >
>> > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
>> >
>> > maulin.vasavada@gmail.com>
>> >
>> > wrote:
>> >
>> > Bump! Can somebody please review this?
>> >
>> >
>>
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Maulin Vasavada <ma...@gmail.com>.
Hi Harsha
We don't have spire (or similar) agents and we do not have keys/certs
locally on any brokers. To elaborate more on my previous email,
I agree that Java security Providers are used in much broader sense - to
have a particular implementation of an algorithm, use specific cipher
suites for SSL , OR in our current team's case have a particular way to
leverage pre-generated SSL sessions. However, the scope of our KIP (486) is
much restricted than that. We merely intend to provide a custom
keystore/truststore for our SSL connections and not really worry about
underlying specific SSL/TLS implementation. This simplifies it a lot for
us to keep the concerns separate and clear.
I feel our approach is more complimentary such that it allows for using
keystores of choice while retaining the flexibility to use any
underlying/available Provider for actually making the SSL call.
We agree with KIP-492's approach based on Providers (and Java's
recommendation), but also strongly believe that our approach can compliment
it very effectively for reasons explained above.
Thanks
Maulin
On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <ka...@harsha.io> wrote:
> Hi Maulin,
>
> On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
> maulin.vasavada@gmail.com>
> wrote:
>
> > Hi Harsha
> >
> > The reason we rejected the SslProvider route is that - we only needed a
> > custom way to load keys/certs. Not touch any policy that existing
> Providers
> > govern like SunJSSE Provider.
> >
>
> We have exactly the same requirements to load certs and keys through spire
> agent. We used security.provider to do that exactly. I am not sure why you
> would be modifying any policies provided by default SunJSSE provider. Can
> you give me an example of having custom provider that will override an
> existing policy in SunJSSE provider.
>
> As pointed out earlier, this kip
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
> allows
> you to load security.provider through config.
> Take a look at the examples I gave before
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
> It registers KeyManagerFactory and TrustManagerFactory and Keystore
> algorithm.
> Implement your custom way of loading Keystore in here
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java
>
> and Trust manager like here
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java
>
> In your Kafka client you can set the security.provider to your custom
> implementation and with this fix
> https://issues.apache.org/jira/browse/KAFKA-8191 you can set
> keyManagerAlgorigthm and trustManagerAlgorithm configs.
>
> All of this is in your clients and broker side and do not need to touch any
> policy changes at JVM level. You'll register the providers in the priority
> order and can still have SunJSSE provider and have your custom provider to
> implement the key and trust managers.
>
>
>
> The ask here is different than KIP-492. We don't have any need to
> > modify/specify the algorithm parameter. Does that make sense?
> >
>
> The ask in KIP is introducing new interfaces where the KIP's
> goal/motivation can be achieved through the security.provider and we worked
> on similar goal without touching any Keystore or Truststore interfaces.
> My advise is against changing or introducing new interfaces when it can
> work through security.provider.
>
> Thanks,
> Harsha
>
>
> Thanks
> > Maulin
> >
> > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <ka...@harsha.io>
> > wrote:
> >
> > In your KIP you added security. provider as rejected alternative and
> > specified "its not the correct way". Do you mind explaining why its not?
> I
> > didn't find any evidence in Java docs to say so. Contrary to your
> statement
> > it does say in the java docs
> > " However, please note that a provider can be used to implement any
> > security service in Java that uses a pluggable architecture with a choice
> > of implementations that fit underneath."
> >
> > Java Security Providers have been used by other projects to provide such
> > integration . I am not sure if you looked into Spiffe project to
> > efficiently distribute certificates but here is an example of Java
> provider
> >
> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> >
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> > java which
> > obtains certificates from local daemons.
> > These integrations are being used in Tomcat, Jetty etc.. We are also
> using
> > Security provider to do the same in our Kafka clusters. So unless I see
> > more evidence why security.provider doesn't work for you adding new
> > interfaces while there exists more cleaner way of achieving the goals of
> > this KIP is unnecessary and breaks the well known security interfaces
> > provided by Java itself.
> >
> > Thanks,
> > Harsha
> >
> > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <ka...@harsha.io>
> > wrote:
> >
> > Hi Maulin,
> > Not sure if you looked at my previous replies. This
> >
> > changes
> >
> > are not required as there is already security Provider to do what you are
> > proposing. This KIP https://cwiki.apache.org/confluence/display/KAFKA/
> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config also
> > addresses easy registration of such providers.
> >
> > Thanks,
> > Harsha
> >
> > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada <maulin.vasavada@gmail.
> > com> wrote:
> >
> > Bump! Can somebody please review this?
> >
> > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
> >
> > maulin.vasavada@gmail.com>
> >
> > wrote:
> >
> > Bump! Can somebody please review this?
> >
> >
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Harsha Chintalapani <ka...@harsha.io>.
Hi Maulin,
On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <ma...@gmail.com>
wrote:
> Hi Harsha
>
> The reason we rejected the SslProvider route is that - we only needed a
> custom way to load keys/certs. Not touch any policy that existing Providers
> govern like SunJSSE Provider.
>
We have exactly the same requirements to load certs and keys through spire
agent. We used security.provider to do that exactly. I am not sure why you
would be modifying any policies provided by default SunJSSE provider. Can
you give me an example of having custom provider that will override an
existing policy in SunJSSE provider.
As pointed out earlier, this kip
https://cwiki.apache.org/confluence/display/KAFKA/KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
allows
you to load security.provider through config.
Take a look at the examples I gave before
https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
It registers KeyManagerFactory and TrustManagerFactory and Keystore
algorithm.
Implement your custom way of loading Keystore in here
https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java
and Trust manager like here
https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java
In your Kafka client you can set the security.provider to your custom
implementation and with this fix
https://issues.apache.org/jira/browse/KAFKA-8191 you can set
keyManagerAlgorigthm and trustManagerAlgorithm configs.
All of this is in your clients and broker side and do not need to touch any
policy changes at JVM level. You'll register the providers in the priority
order and can still have SunJSSE provider and have your custom provider to
implement the key and trust managers.
The ask here is different than KIP-492. We don't have any need to
> modify/specify the algorithm parameter. Does that make sense?
>
The ask in KIP is introducing new interfaces where the KIP's
goal/motivation can be achieved through the security.provider and we worked
on similar goal without touching any Keystore or Truststore interfaces.
My advise is against changing or introducing new interfaces when it can
work through security.provider.
Thanks,
Harsha
Thanks
> Maulin
>
> On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <ka...@harsha.io>
> wrote:
>
> In your KIP you added security. provider as rejected alternative and
> specified "its not the correct way". Do you mind explaining why its not? I
> didn't find any evidence in Java docs to say so. Contrary to your statement
> it does say in the java docs
> " However, please note that a provider can be used to implement any
> security service in Java that uses a pluggable architecture with a choice
> of implementations that fit underneath."
>
> Java Security Providers have been used by other projects to provide such
> integration . I am not sure if you looked into Spiffe project to
> efficiently distribute certificates but here is an example of Java provider
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
> java which
> obtains certificates from local daemons.
> These integrations are being used in Tomcat, Jetty etc.. We are also using
> Security provider to do the same in our Kafka clusters. So unless I see
> more evidence why security.provider doesn't work for you adding new
> interfaces while there exists more cleaner way of achieving the goals of
> this KIP is unnecessary and breaks the well known security interfaces
> provided by Java itself.
>
> Thanks,
> Harsha
>
> On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <ka...@harsha.io>
> wrote:
>
> Hi Maulin,
> Not sure if you looked at my previous replies. This
>
> changes
>
> are not required as there is already security Provider to do what you are
> proposing. This KIP https://cwiki.apache.org/confluence/display/KAFKA/
> KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config also
> addresses easy registration of such providers.
>
> Thanks,
> Harsha
>
> On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada <maulin.vasavada@gmail.
> com> wrote:
>
> Bump! Can somebody please review this?
>
> On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
>
> maulin.vasavada@gmail.com>
>
> wrote:
>
> Bump! Can somebody please review this?
>
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Maulin Vasavada <ma...@gmail.com>.
Hi Harsha
The reason we rejected the SslProvider route is that - we only needed a
custom way to load keys/certs. Not touch any policy that existing Providers
govern like SunJSSE Provider.
The ask here is different than KIP-492. We don't have any need to
modify/specify the algorithm parameter. Does that make sense?
Thanks
Maulin
On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <ka...@harsha.io> wrote:
> In your KIP you added security. provider as rejected alternative and
> specified "its not the correct way". Do you mind explaining why its not? I
> didn't find any evidence in Java docs to say so. Contrary to your statement
> it does say in the java docs
> " However, please note that a provider can be used to implement any
> security service in Java that uses a pluggable architecture with a choice
> of implementations that fit underneath."
>
> Java Security Providers have been used by other projects to provide such
> integration . I am not sure if you looked into Spiffe project to
> efficiently distribute certificates but here is an example of Java provider
>
> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
> which
> obtains certificates from local daemons.
> These integrations are being used in Tomcat, Jetty etc.. We are also using
> Security provider to do the same in our Kafka clusters. So unless I see
> more evidence why security.provider doesn't work for you
> adding new interfaces while there exists more cleaner way of achieving the
> goals of this KIP is unnecessary and breaks the well known security
> interfaces provided by Java itself.
>
> Thanks,
> Harsha
>
>
> On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <ka...@harsha.io>
> wrote:
>
> > Hi Maulin,
> > Not sure if you looked at my previous replies. This
> changes
> > are not required as there is already security Provider to do what you are
> > proposing. This KIP https://cwiki.apache.org/confluence/display/KAFKA/
> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config also
> > addresses easy registration of such providers.
> >
> > Thanks,
> > Harsha
> >
> >
> > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada <maulin.vasavada@gmail.
> > com> wrote:
> >
> > Bump! Can somebody please review this?
> >
> > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
> maulin.vasavada@gmail.com>
> > wrote:
> >
> > Bump! Can somebody please review this?
> >
> >
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Harsha Chintalapani <ka...@harsha.io>.
In your KIP you added security. provider as rejected alternative and
specified "its not the correct way". Do you mind explaining why its not? I
didn't find any evidence in Java docs to say so. Contrary to your statement
it does say in the java docs
" However, please note that a provider can be used to implement any
security service in Java that uses a pluggable architecture with a choice
of implementations that fit underneath."
Java Security Providers have been used by other projects to provide such
integration . I am not sure if you looked into Spiffe project to
efficiently distribute certificates but here is an example of Java provider
https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
which
obtains certificates from local daemons.
These integrations are being used in Tomcat, Jetty etc.. We are also using
Security provider to do the same in our Kafka clusters. So unless I see
more evidence why security.provider doesn't work for you
adding new interfaces while there exists more cleaner way of achieving the
goals of this KIP is unnecessary and breaks the well known security
interfaces provided by Java itself.
Thanks,
Harsha
On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <ka...@harsha.io>
wrote:
> Hi Maulin,
> Not sure if you looked at my previous replies. This changes
> are not required as there is already security Provider to do what you are
> proposing. This KIP https://cwiki.apache.org/confluence/display/KAFKA/
> KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config also
> addresses easy registration of such providers.
>
> Thanks,
> Harsha
>
>
> On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada <maulin.vasavada@gmail.
> com> wrote:
>
> Bump! Can somebody please review this?
>
> On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <ma...@gmail.com>
> wrote:
>
> Bump! Can somebody please review this?
>
>
Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore
Posted by Harsha Chintalapani <ka...@harsha.io>.
Hi Maulin,
Not sure if you looked at my previous replies. This changes
are not required as there is already security Provider to do what you are
proposing. This KIP
https://cwiki.apache.org/confluence/display/KAFKA/KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
also
addresses easy registration of such providers.
Thanks,
Harsha
On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada <maulin.vasavada@gmail.com
> wrote:
> Bump! Can somebody please review this?
>
> On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <ma...@gmail.com>
> wrote:
>
> Bump! Can somebody please review this?
>
>