You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by Sanjay Pujare <sa...@datatorrent.com> on 2017/04/21 07:00:15 UTC
Enhancement to support custom SSL configuration
Currently StrAM supports only the default Hadoop SSL configuration because
it uses org.apache.hadoop.yarn.webapp.WebApps helper class which has the
limitation of only using the default Hadoop SSL config that is read from
Hadoop's ssl-server.xml resource file. Some users have run into a situation
where Hadoops' SSL keystore is not available on most cluster nodes or the
Stram process doesn't have read access to the keystore even when present.
So there is a need for the Stram to use a custom SSL keystore and
configuration that does not suffer from these limitations.
I am planning to fix this by first fixing WebApps in Hadoop and then
enhancing Stram to use this new fix in Hadoop. I have already submitted a
PR https://github.com/apache/hadoop/pull/213 to Hadoop and one of the the
Hadoop distributors has agreed to accept this fix so I expect it to be
merged very soon.
After that I will enhance Stram to accept the location of a custom
ssl-server.xml file (supplied by the client via a DAG attribute or
property) and use the values from that file to set up the config object to
be passed to WebApps which will end up using the custom SSL configuration.
I have already verified this approach in a prototype.
We will also enhance the Apex client/launcher to distribute the custom SSL
files (XML and the keystore) along with the application jars/resources so
the user does not need to pre-distribute the custom SSL files.
Please let me know your comments.
Sanjay
Re: Enhancement to support custom SSL configuration
Posted by Aniruddha Thombare <an...@datatorrent.com>.
+1 for custom keystore support.
Thanks,
Aniruddha
_____________________________________________
Always finding your faults, just like your Mom!
#QA
On Fri, Apr 21, 2017 at 12:47 PM, Sanjay Pujare <sa...@datatorrent.com>
wrote:
> Regarding distributing the SSL files by us, I agree it is not ideal because
> we are taking the responsibility of distributing sensitive security
> material. But it would also be very convenient for certain users. Also as
> some users have pointed out there is a precedent: H2O does it as per
> http://docs.h2o.ai/h2o/latest-stable/h2o-docs/security.html#
> ssl-internode-security
> . It says "This will tell h2odriver to automatically generate all the
> necessary files and distribute them to all mappers. This distribution may
> be secure depending on your YARN configuration."
>
> On Fri, Apr 21, 2017 at 12:08 AM, Priyanka Gugale <pr...@apache.org>
> wrote:
>
> > +1 for this support, it's important to let users use their own own
> keystore
> > files.
> >
> > Is it okay to distribute files inside our package along with
> jar/resources,
> > are there any security restrictions? Or we should use other medium like
> > HDFS or other shared file system to host these key files?
> >
> > -Priyanka
> >
> >
> >
> > On Fri, Apr 21, 2017 at 12:30 PM, Sanjay Pujare <sa...@datatorrent.com>
> > wrote:
> >
> > > Currently StrAM supports only the default Hadoop SSL configuration
> > because
> > > it uses org.apache.hadoop.yarn.webapp.WebApps helper class which has
> the
> > > limitation of only using the default Hadoop SSL config that is read
> from
> > > Hadoop's ssl-server.xml resource file. Some users have run into a
> > situation
> > > where Hadoops' SSL keystore is not available on most cluster nodes or
> the
> > > Stram process doesn't have read access to the keystore even when
> present.
> > > So there is a need for the Stram to use a custom SSL keystore and
> > > configuration that does not suffer from these limitations.
> > >
> > > I am planning to fix this by first fixing WebApps in Hadoop and then
> > > enhancing Stram to use this new fix in Hadoop. I have already
> submitted a
> > > PR https://github.com/apache/hadoop/pull/213 to Hadoop and one of the
> > the
> > > Hadoop distributors has agreed to accept this fix so I expect it to be
> > > merged very soon.
> > >
> > > After that I will enhance Stram to accept the location of a custom
> > > ssl-server.xml file (supplied by the client via a DAG attribute or
> > > property) and use the values from that file to set up the config object
> > to
> > > be passed to WebApps which will end up using the custom SSL
> > configuration.
> > > I have already verified this approach in a prototype.
> > >
> > > We will also enhance the Apex client/launcher to distribute the custom
> > SSL
> > > files (XML and the keystore) along with the application jars/resources
> so
> > > the user does not need to pre-distribute the custom SSL files.
> > >
> > > Please let me know your comments.
> > >
> > > Sanjay
> > >
> >
>
Re: Enhancement to support custom SSL configuration
Posted by Sanjay Pujare <sa...@datatorrent.com>.
Regarding distributing the SSL files by us, I agree it is not ideal because
we are taking the responsibility of distributing sensitive security
material. But it would also be very convenient for certain users. Also as
some users have pointed out there is a precedent: H2O does it as per
http://docs.h2o.ai/h2o/latest-stable/h2o-docs/security.html#ssl-internode-security
. It says "This will tell h2odriver to automatically generate all the
necessary files and distribute them to all mappers. This distribution may
be secure depending on your YARN configuration."
On Fri, Apr 21, 2017 at 12:08 AM, Priyanka Gugale <pr...@apache.org> wrote:
> +1 for this support, it's important to let users use their own own keystore
> files.
>
> Is it okay to distribute files inside our package along with jar/resources,
> are there any security restrictions? Or we should use other medium like
> HDFS or other shared file system to host these key files?
>
> -Priyanka
>
>
>
> On Fri, Apr 21, 2017 at 12:30 PM, Sanjay Pujare <sa...@datatorrent.com>
> wrote:
>
> > Currently StrAM supports only the default Hadoop SSL configuration
> because
> > it uses org.apache.hadoop.yarn.webapp.WebApps helper class which has the
> > limitation of only using the default Hadoop SSL config that is read from
> > Hadoop's ssl-server.xml resource file. Some users have run into a
> situation
> > where Hadoops' SSL keystore is not available on most cluster nodes or the
> > Stram process doesn't have read access to the keystore even when present.
> > So there is a need for the Stram to use a custom SSL keystore and
> > configuration that does not suffer from these limitations.
> >
> > I am planning to fix this by first fixing WebApps in Hadoop and then
> > enhancing Stram to use this new fix in Hadoop. I have already submitted a
> > PR https://github.com/apache/hadoop/pull/213 to Hadoop and one of the
> the
> > Hadoop distributors has agreed to accept this fix so I expect it to be
> > merged very soon.
> >
> > After that I will enhance Stram to accept the location of a custom
> > ssl-server.xml file (supplied by the client via a DAG attribute or
> > property) and use the values from that file to set up the config object
> to
> > be passed to WebApps which will end up using the custom SSL
> configuration.
> > I have already verified this approach in a prototype.
> >
> > We will also enhance the Apex client/launcher to distribute the custom
> SSL
> > files (XML and the keystore) along with the application jars/resources so
> > the user does not need to pre-distribute the custom SSL files.
> >
> > Please let me know your comments.
> >
> > Sanjay
> >
>
Re: Enhancement to support custom SSL configuration
Posted by Priyanka Gugale <pr...@apache.org>.
+1 for this support, it's important to let users use their own own keystore
files.
Is it okay to distribute files inside our package along with jar/resources,
are there any security restrictions? Or we should use other medium like
HDFS or other shared file system to host these key files?
-Priyanka
On Fri, Apr 21, 2017 at 12:30 PM, Sanjay Pujare <sa...@datatorrent.com>
wrote:
> Currently StrAM supports only the default Hadoop SSL configuration because
> it uses org.apache.hadoop.yarn.webapp.WebApps helper class which has the
> limitation of only using the default Hadoop SSL config that is read from
> Hadoop's ssl-server.xml resource file. Some users have run into a situation
> where Hadoops' SSL keystore is not available on most cluster nodes or the
> Stram process doesn't have read access to the keystore even when present.
> So there is a need for the Stram to use a custom SSL keystore and
> configuration that does not suffer from these limitations.
>
> I am planning to fix this by first fixing WebApps in Hadoop and then
> enhancing Stram to use this new fix in Hadoop. I have already submitted a
> PR https://github.com/apache/hadoop/pull/213 to Hadoop and one of the the
> Hadoop distributors has agreed to accept this fix so I expect it to be
> merged very soon.
>
> After that I will enhance Stram to accept the location of a custom
> ssl-server.xml file (supplied by the client via a DAG attribute or
> property) and use the values from that file to set up the config object to
> be passed to WebApps which will end up using the custom SSL configuration.
> I have already verified this approach in a prototype.
>
> We will also enhance the Apex client/launcher to distribute the custom SSL
> files (XML and the keystore) along with the application jars/resources so
> the user does not need to pre-distribute the custom SSL files.
>
> Please let me know your comments.
>
> Sanjay
>