You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by turp1twin <tu...@gmail.com> on 2015/03/07 01:20:09 UTC
Block Transfer Service encryption support
Is there a plan to implement SSL support for the Block Transfer Service
(specifically, the NettyBlockTransferService implementation)? I can
volunteer if needed...
Jeff
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Block-Transfer-Service-encryption-support-tp10934.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Block Transfer Service encryption support
Posted by Tim Preece <te...@mail.com>.
So it appears the tests fail because of an SSLHandshakeException.
Tracing the failure I see:
3,0001,Using SSLEngineImpl.\0A
3,0001,\0AIs initial handshake: true\0A
3,0001,Ignoring unsupported cipher suite: SSL_RSA_WITH_DES_CBC_SHA for
TLSv1.2\0A
3,0001,No available cipher suite for TLSv1.2\0A
3,0001,shuffle-client-4\2C fatal error: 40: Couldn't kickstart
handshaking\0Ajavax.net.ssl.SSLHandshakeException: No appropriate
protocol\2C may be no appropriate cipher suite specified or protocols are
deactivated\0A
3,0001,shuffle-client-4
3,0001,\2C SEND TLSv1.2 ALERT:
3,0001,fatal\2C
3,0001,description = handshake_failure\0A
3,0001,shuffle-client-4\2C WRITE: TLSv1.2 Alert\2C length = 2\0A
3,0001,Using SSLEngineImpl.\0A
3,0001,shuffle-client-4\2C called closeOutbound()\0A
3,0001,shuffle-client-4\2C closeOutboundInternal()\0A
3,0001,[Raw write]: length = 7\0A
3,0001,0000: 15 03 03 00 02 02 28
.......\0A\0A
3,0001,\0AIs initial handshake: true\0A
3,0001,Ignoring unsupported cipher suite: SSL_RSA_WITH_DES_CBC_SHA for
TLSv1.2\0A
3,0001,No available cipher suite for TLSv1.2\0A
3,0001,shuffle-server-5\2C fatal error: 80: problem unwrapping net
record\0Ajavax.net.ssl.SSLHandshakeException: No appropriate protocol\2C may
be no appropriate cipher suite specified or protocols are deactivated\0A
3,0001,shuffle-server-5
3,0001,\2C SEND TLSv1.2 ALERT:
3,0001,fatal\2C
3,0001,description = internal_error\0A
3,0001,shuffle-server-5\2C WRITE: TLSv1.2 Alert\2C length = 2\0A
3,0001,shuffle-server-5\2C called closeOutbound()\0A
3,0001,shuffle-server-5\2C closeOutboundInternal()\0A
3,0001,shuffle-server-5\2C called closeInbound()\0A
3,0001,shuffle-server-5\2C closeInboundInternal()\0A
3,0001,shuffle-client-4\2C called closeOutbound()\0A
3,0001,shuffle-client-4\2C closeOutboundInternal()\0A
3,0001,shuffle-client-4\2C called closeInbound()\0A
3,0001,shuffle-client-4\2C closeInboundInternal()\0A
3,0001,shuffle-server-5\2C called closeOutbound()\0A
3,0001,shuffle-server-5\2C closeOutboundInternal()\0A
3,0001,shuffle-server-5\2C called closeInbound()\0A
3,0001,shuffle-server-5\2C closeInboundInternal()\0A
So this fails because of the use of DES. From
https://www-01.ibm.com/support/knowledgecenter/SSYKE2_7.0.0/com.ibm.java.security.component.71.doc/security-component/jsse2Docs/ciphersuites.html
I see: 2 RFC 5246 TLS 1.2 forbids the use of these suites. These can be used
in the SSLv3/TLS1.0/TLS1.1 protocols, but cannot be used in TLS 1.2 and
later.
Note. I'm using the IBM Java SDK.
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Block-Transfer-Service-encryption-support-tp10934p15116.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Block Transfer Service encryption support
Posted by Tim Preece <te...@mail.com>.
Nb. I did notice some test failures when I ran a quick test on the pull
request ( not sure if it is related - I haven't looked in any detail at the
cause ).
Failed tests:
SslChunkFetchIntegrationSuite>ChunkFetchIntegrationSuite.fetchBothChunks:201
expected:<[]> but was:<[0, 1]>
SslChunkFetchIntegrationSuite>ChunkFetchIntegrationSuite.fetchBufferChunk:175
expected:<[]> but was:<[0]>
SslChunkFetchIntegrationSuite>ChunkFetchIntegrationSuite.fetchChunkAndNonExistent:210
expected:<[]> but was:<[0]>
SslChunkFetchIntegrationSuite>ChunkFetchIntegrationSuite.fetchFileChunk:184
expected:<[]> but was:<[1]>
SslTransportClientFactorySuite>TransportClientFactorySuite.neverReturnInactiveClients:165
null
SslTransportClientFactorySuite>TransportClientFactorySuite.returnDifferentClientsForDifferentServers:145
null
Tim
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Block-Transfer-Service-encryption-support-tp10934p15114.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Block Transfer Service encryption support
Posted by turp1twin <tu...@gmail.com>.
I created a pull request for issue SPARK-6373
<https://issues.apache.org/jira/browse/SPARK-6373> .... Any feedback would
be appreciated... https://github.com/apache/spark/pull/9416
Jeff
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Block-Transfer-Service-encryption-support-tp10934p15098.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Block Transfer Service encryption support
Posted by turp1twin <tu...@gmail.com>.
Still looking for feedback... I opened the ticket as a minor, thinking of
changing it to a major? Anyone object. I did see that a related ticket
(https://issues.apache.org/jira/browse/SPARK-6229) is marked as a major...
Cheers!
Jeff
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Block-Transfer-Service-encryption-support-tp10934p11110.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Block Transfer Service encryption support
Posted by turp1twin <tu...@gmail.com>.
Hey Aaron,
That is what I do, except I add the Netty SslHandler in the TransportServer
and the TransportClientFactory.... I do this because the Server pipeline is
a bit different as I have to add a Netty ChunkedWriteHandler... Again, this
is a "rough" prototype, just to get something working... Cheers!
Jeff
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Block-Transfer-Service-encryption-support-tp10934p11091.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Block Transfer Service encryption support
Posted by Aaron Davidson <il...@gmail.com>.
Out of curiosity, why could we not use Netty's SslHandler injected into the
TransportContext pipeline?
On Mon, Mar 16, 2015 at 7:56 PM, turp1twin <tu...@gmail.com> wrote:
> Hey Patrick,
>
> Sorry for the delay, I was at Elastic{ON} last week and well, my day job
> has
> been keeping me busy... I went ahead and opened a Jira feature request,
> https://issues.apache.org/jira/browse/SPARK-6373. In it I reference a
> commit
> I made in my fork which is a "rough" implementation, definitely still a
> WIP.
> Would like to iterate the design if possible, as there are some performance
> trade offs for using SSL for sure.. Zero copy will not be possible with
> SSL,
> so there will definitely be a hit there.. That being said, for my use case,
> which is health care related and involves processing personal health
> information, I have no choice, as all data must be encrypted in transit and
> at rest... Cheers!
>
> Jeff
>
>
> https://github.com/turp1twin/spark/commit/024b559f27945eb63068d1badf7f82e4e7c3621c
>
>
>
>
> --
> View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/Block-Transfer-Service-encryption-support-tp10934p11089.html
> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>
Re: Block Transfer Service encryption support
Posted by turp1twin <tu...@gmail.com>.
Hey Patrick,
Sorry for the delay, I was at Elastic{ON} last week and well, my day job has
been keeping me busy... I went ahead and opened a Jira feature request,
https://issues.apache.org/jira/browse/SPARK-6373. In it I reference a commit
I made in my fork which is a "rough" implementation, definitely still a WIP.
Would like to iterate the design if possible, as there are some performance
trade offs for using SSL for sure.. Zero copy will not be possible with SSL,
so there will definitely be a hit there.. That being said, for my use case,
which is health care related and involves processing personal health
information, I have no choice, as all data must be encrypted in transit and
at rest... Cheers!
Jeff
https://github.com/turp1twin/spark/commit/024b559f27945eb63068d1badf7f82e4e7c3621c
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Block-Transfer-Service-encryption-support-tp10934p11089.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Block Transfer Service encryption support
Posted by Jeff Turpin <tu...@gmail.com>.
Hey Patrick,
Yes, I will open a Jira tomorrow... For now my implementation is a basic
SSL implementation for the TransportServer and TransportClient.. I will
type up the design and at the same time look at the Hadoop impl for
possible improvements... Cheers!
Jeff
On Sun, Mar 8, 2015 at 5:51 PM, Patrick Wendell <pw...@gmail.com> wrote:
> I think that yes, longer term we want to have encryption of all
> communicated data. However Jeff, can you open a JIRA to discuss the
> design before opening a pull request (it's fine to link to a WIP
> branch if you'd like)? I'd like to better understand the performance
> and operational complexity of using SSL for this in comparison with
> alternatives. It would also be good to look at how the Hadoop
> encryption works for their shuffle service, in terms of the design
> decisions made there.
>
> - Patrick
>
> On Sun, Mar 8, 2015 at 5:42 PM, Jeff Turpin <tu...@gmail.com> wrote:
> > I have already written most of the code, just finishing up the unit tests
> > right now...
> >
> > Jeff
> >
> >
> > On Sun, Mar 8, 2015 at 5:39 PM, Andrew Ash <an...@andrewash.com> wrote:
> >
> >> I'm interested in seeing this data transfer occurring over encrypted
> >> communication channels as well. Many customers require that all network
> >> transfer occur encrypted to prevent the "soft underbelly" that's often
> >> found inside a corporate network.
> >>
> >> On Fri, Mar 6, 2015 at 4:20 PM, turp1twin <tu...@gmail.com> wrote:
> >>
> >>> Is there a plan to implement SSL support for the Block Transfer Service
> >>> (specifically, the NettyBlockTransferService implementation)? I can
> >>> volunteer if needed...
> >>>
> >>> Jeff
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> View this message in context:
> >>>
> http://apache-spark-developers-list.1001551.n3.nabble.com/Block-Transfer-Service-encryption-support-tp10934.html
> >>> Sent from the Apache Spark Developers List mailing list archive at
> >>> Nabble.com.
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> >>> For additional commands, e-mail: dev-help@spark.apache.org
> >>>
> >>>
> >>
>
Re: Block Transfer Service encryption support
Posted by Patrick Wendell <pw...@gmail.com>.
I think that yes, longer term we want to have encryption of all
communicated data. However Jeff, can you open a JIRA to discuss the
design before opening a pull request (it's fine to link to a WIP
branch if you'd like)? I'd like to better understand the performance
and operational complexity of using SSL for this in comparison with
alternatives. It would also be good to look at how the Hadoop
encryption works for their shuffle service, in terms of the design
decisions made there.
- Patrick
On Sun, Mar 8, 2015 at 5:42 PM, Jeff Turpin <tu...@gmail.com> wrote:
> I have already written most of the code, just finishing up the unit tests
> right now...
>
> Jeff
>
>
> On Sun, Mar 8, 2015 at 5:39 PM, Andrew Ash <an...@andrewash.com> wrote:
>
>> I'm interested in seeing this data transfer occurring over encrypted
>> communication channels as well. Many customers require that all network
>> transfer occur encrypted to prevent the "soft underbelly" that's often
>> found inside a corporate network.
>>
>> On Fri, Mar 6, 2015 at 4:20 PM, turp1twin <tu...@gmail.com> wrote:
>>
>>> Is there a plan to implement SSL support for the Block Transfer Service
>>> (specifically, the NettyBlockTransferService implementation)? I can
>>> volunteer if needed...
>>>
>>> Jeff
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-developers-list.1001551.n3.nabble.com/Block-Transfer-Service-encryption-support-tp10934.html
>>> Sent from the Apache Spark Developers List mailing list archive at
>>> Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>
>>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Block Transfer Service encryption support
Posted by Jeff Turpin <tu...@gmail.com>.
I have already written most of the code, just finishing up the unit tests
right now...
Jeff
On Sun, Mar 8, 2015 at 5:39 PM, Andrew Ash <an...@andrewash.com> wrote:
> I'm interested in seeing this data transfer occurring over encrypted
> communication channels as well. Many customers require that all network
> transfer occur encrypted to prevent the "soft underbelly" that's often
> found inside a corporate network.
>
> On Fri, Mar 6, 2015 at 4:20 PM, turp1twin <tu...@gmail.com> wrote:
>
>> Is there a plan to implement SSL support for the Block Transfer Service
>> (specifically, the NettyBlockTransferService implementation)? I can
>> volunteer if needed...
>>
>> Jeff
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-developers-list.1001551.n3.nabble.com/Block-Transfer-Service-encryption-support-tp10934.html
>> Sent from the Apache Spark Developers List mailing list archive at
>> Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>>
>
Re: Block Transfer Service encryption support
Posted by Andrew Ash <an...@andrewash.com>.
I'm interested in seeing this data transfer occurring over encrypted
communication channels as well. Many customers require that all network
transfer occur encrypted to prevent the "soft underbelly" that's often
found inside a corporate network.
On Fri, Mar 6, 2015 at 4:20 PM, turp1twin <tu...@gmail.com> wrote:
> Is there a plan to implement SSL support for the Block Transfer Service
> (specifically, the NettyBlockTransferService implementation)? I can
> volunteer if needed...
>
> Jeff
>
>
>
>
> --
> View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/Block-Transfer-Service-encryption-support-tp10934.html
> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>