You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Brahma Reddy Battula <br...@huawei.com> on 2015/09/09 14:24:59 UTC

Even after HDFS-2856 JSVC References are require..?

Hi All,

AFAIK JSVC added secure the block tokens(..?).

Since block tokens are secure now (SASL used to secure the DataTransferProtocol, which transfers file block content between HDFS clients and DataNodes),then can we remove jsvc now (script files)..?



Thanks & Regards

 Brahma Reddy Battula




Re: Even after HDFS-2856 JSVC References are require..?

Posted by Chris Nauroth <cn...@hortonworks.com>.
I have no specific numbers to offer on this.  Like you, I expect the
impact is non-zero.  I also expect the impact would depend on the setting
of dfs.data.transfer.protection.  For "authentication", it might just be a
small up-front cost for the SASL authentication handshake.  For
"integrity" or "privacy", it becomes more a function of the payload (the
block data transferred).  For "privacy", I expect a similar hit to
enabling data transfer encryption using dfs.encrypt.data.transfer, which
is already known to be significant.

I think it's a fair point to require a performance comparison before
proceeding with any proposal of deprecating the jsvc deployment option.

--Chris Nauroth




On 9/14/15, 4:23 PM, "Colin P. McCabe" <cm...@apache.org> wrote:

>Has anyone measured the overhead of running SASL on
>DataTransferProtocol?  I would expect it to be non-zero compared with
>simply running on a low port.  The CPU overhead especially could
>regress performance on a typical Hadoop cluster.
>
>best,
>Colin
>
>On Thu, Sep 10, 2015 at 9:55 AM, Chris Nauroth <cn...@hortonworks.com>
>wrote:
>> Yes, I have a paragraph in the docs describing how someone would go
>>about
>> migrating a jsvc-based deployment to a SASL-based deployment.
>>
>> 
>>http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/Se
>>cu
>> reMode.html#Secure_DataNode
>>
>>
>> It's a non-trivial operation that starts by making sure everyone is on
>>2.6
>> first.  This includes client deployments, which are notoriously more
>> difficult to control than server deployments.
>>
>> --Chris Nauroth
>>
>>
>>
>>
>> On 9/10/15, 1:21 AM, "Steve Loughran" <st...@hortonworks.com> wrote:
>>
>>>SASL authenticates the DN on Hadoop 2.6+, but it requires the clients to
>>>be using the 2.6+ JARs; you can't use it on the 2.2-2.5 artifacts.
>>>
>>>> On 9 Sep 2015, at 18:45, Allen Wittenauer <aw...@altiscale.com> wrote:
>>>>
>>>>
>>>> FWIW, I still use and prefer jsvc, esp with the sudo trick in place.
>>>>
>>>> On Sep 9, 2015, at 9:35 AM, Chris Nauroth <cn...@hortonworks.com>
>>>>wrote:
>>>>
>>>>> AFAIK, the majority of existing deployments still use jsvc to run a
>>>>> secured DataNode.  It would be a backwards-incompatible change to
>>>>>remove
>>>>> support for this deployment model.  For that reason, I would be -1
>>>>>for
>>>>> removing jsvc support, at least in the 2.x line.
>>>>>
>>>>>
>>>>> It's something that could be considered for 3.x if we think the
>>>>>clean-up
>>>>> benefit outweighs the incompatibility cost.  Before we do that, I'd
>>>>>prefer
>>>>> to hear if end users are having success with the SASL deployment
>>>>>model.
>>>>> Brahma, are you asking because you run clusters with the SASL
>>>>>approach?
>>>>> If so, has it been working well?
>>>>>
>>>>> --Chris Nauroth
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 9/9/15, 9:25 AM, "Haohui Mai" <wh...@apache.org> wrote:
>>>>>
>>>>>> JSVC is no longer required. It causes a lot of headaches in
>>>>>> deployments. It's definitely a good target for clean ups.
>>>>>>
>>>>>> ~Haohui
>>>>>>
>>>>>> On Wed, Sep 9, 2015 at 5:24 AM, Brahma Reddy Battula
>>>>>> <br...@huawei.com> wrote:
>>>>>>> Hi All,
>>>>>>>
>>>>>>> AFAIK JSVC added secure the block tokens(..?).
>>>>>>>
>>>>>>> Since block tokens are secure now (SASL used to secure the
>>>>>>> DataTransferProtocol, which transfers file block content between
>>>>>>>HDFS
>>>>>>> clients and DataNodes),then can we remove jsvc now (script
>>>>>>>files)..?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks & Regards
>>>>>>>
>>>>>>> Brahma Reddy Battula
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>


Re: Even after HDFS-2856 JSVC References are require..?

Posted by "Colin P. McCabe" <cm...@apache.org>.
Has anyone measured the overhead of running SASL on
DataTransferProtocol?  I would expect it to be non-zero compared with
simply running on a low port.  The CPU overhead especially could
regress performance on a typical Hadoop cluster.

best,
Colin

On Thu, Sep 10, 2015 at 9:55 AM, Chris Nauroth <cn...@hortonworks.com> wrote:
> Yes, I have a paragraph in the docs describing how someone would go about
> migrating a jsvc-based deployment to a SASL-based deployment.
>
> http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/Secu
> reMode.html#Secure_DataNode
>
>
> It's a non-trivial operation that starts by making sure everyone is on 2.6
> first.  This includes client deployments, which are notoriously more
> difficult to control than server deployments.
>
> --Chris Nauroth
>
>
>
>
> On 9/10/15, 1:21 AM, "Steve Loughran" <st...@hortonworks.com> wrote:
>
>>SASL authenticates the DN on Hadoop 2.6+, but it requires the clients to
>>be using the 2.6+ JARs; you can't use it on the 2.2-2.5 artifacts.
>>
>>> On 9 Sep 2015, at 18:45, Allen Wittenauer <aw...@altiscale.com> wrote:
>>>
>>>
>>> FWIW, I still use and prefer jsvc, esp with the sudo trick in place.
>>>
>>> On Sep 9, 2015, at 9:35 AM, Chris Nauroth <cn...@hortonworks.com>
>>>wrote:
>>>
>>>> AFAIK, the majority of existing deployments still use jsvc to run a
>>>> secured DataNode.  It would be a backwards-incompatible change to
>>>>remove
>>>> support for this deployment model.  For that reason, I would be -1 for
>>>> removing jsvc support, at least in the 2.x line.
>>>>
>>>>
>>>> It's something that could be considered for 3.x if we think the
>>>>clean-up
>>>> benefit outweighs the incompatibility cost.  Before we do that, I'd
>>>>prefer
>>>> to hear if end users are having success with the SASL deployment model.
>>>> Brahma, are you asking because you run clusters with the SASL approach?
>>>> If so, has it been working well?
>>>>
>>>> --Chris Nauroth
>>>>
>>>>
>>>>
>>>>
>>>> On 9/9/15, 9:25 AM, "Haohui Mai" <wh...@apache.org> wrote:
>>>>
>>>>> JSVC is no longer required. It causes a lot of headaches in
>>>>> deployments. It's definitely a good target for clean ups.
>>>>>
>>>>> ~Haohui
>>>>>
>>>>> On Wed, Sep 9, 2015 at 5:24 AM, Brahma Reddy Battula
>>>>> <br...@huawei.com> wrote:
>>>>>> Hi All,
>>>>>>
>>>>>> AFAIK JSVC added secure the block tokens(..?).
>>>>>>
>>>>>> Since block tokens are secure now (SASL used to secure the
>>>>>> DataTransferProtocol, which transfers file block content between HDFS
>>>>>> clients and DataNodes),then can we remove jsvc now (script files)..?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks & Regards
>>>>>>
>>>>>> Brahma Reddy Battula
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>
>>
>

Re: Even after HDFS-2856 JSVC References are require..?

Posted by Chris Nauroth <cn...@hortonworks.com>.
Yes, I have a paragraph in the docs describing how someone would go about
migrating a jsvc-based deployment to a SASL-based deployment.

http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/Secu
reMode.html#Secure_DataNode


It's a non-trivial operation that starts by making sure everyone is on 2.6
first.  This includes client deployments, which are notoriously more
difficult to control than server deployments.

--Chris Nauroth




On 9/10/15, 1:21 AM, "Steve Loughran" <st...@hortonworks.com> wrote:

>SASL authenticates the DN on Hadoop 2.6+, but it requires the clients to
>be using the 2.6+ JARs; you can't use it on the 2.2-2.5 artifacts.
>
>> On 9 Sep 2015, at 18:45, Allen Wittenauer <aw...@altiscale.com> wrote:
>> 
>> 
>> FWIW, I still use and prefer jsvc, esp with the sudo trick in place.
>> 
>> On Sep 9, 2015, at 9:35 AM, Chris Nauroth <cn...@hortonworks.com>
>>wrote:
>> 
>>> AFAIK, the majority of existing deployments still use jsvc to run a
>>> secured DataNode.  It would be a backwards-incompatible change to
>>>remove
>>> support for this deployment model.  For that reason, I would be -1 for
>>> removing jsvc support, at least in the 2.x line.
>>> 
>>> 
>>> It's something that could be considered for 3.x if we think the
>>>clean-up
>>> benefit outweighs the incompatibility cost.  Before we do that, I'd
>>>prefer
>>> to hear if end users are having success with the SASL deployment model.
>>> Brahma, are you asking because you run clusters with the SASL approach?
>>> If so, has it been working well?
>>> 
>>> --Chris Nauroth
>>> 
>>> 
>>> 
>>> 
>>> On 9/9/15, 9:25 AM, "Haohui Mai" <wh...@apache.org> wrote:
>>> 
>>>> JSVC is no longer required. It causes a lot of headaches in
>>>> deployments. It's definitely a good target for clean ups.
>>>> 
>>>> ~Haohui
>>>> 
>>>> On Wed, Sep 9, 2015 at 5:24 AM, Brahma Reddy Battula
>>>> <br...@huawei.com> wrote:
>>>>> Hi All,
>>>>> 
>>>>> AFAIK JSVC added secure the block tokens(..?).
>>>>> 
>>>>> Since block tokens are secure now (SASL used to secure the
>>>>> DataTransferProtocol, which transfers file block content between HDFS
>>>>> clients and DataNodes),then can we remove jsvc now (script files)..?
>>>>> 
>>>>> 
>>>>> 
>>>>> Thanks & Regards
>>>>> 
>>>>> Brahma Reddy Battula
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> 
>
>


Re: Even after HDFS-2856 JSVC References are require..?

Posted by Steve Loughran <st...@hortonworks.com>.
SASL authenticates the DN on Hadoop 2.6+, but it requires the clients to be using the 2.6+ JARs; you can't use it on the 2.2-2.5 artifacts. 

> On 9 Sep 2015, at 18:45, Allen Wittenauer <aw...@altiscale.com> wrote:
> 
> 
> FWIW, I still use and prefer jsvc, esp with the sudo trick in place.
> 
> On Sep 9, 2015, at 9:35 AM, Chris Nauroth <cn...@hortonworks.com> wrote:
> 
>> AFAIK, the majority of existing deployments still use jsvc to run a
>> secured DataNode.  It would be a backwards-incompatible change to remove
>> support for this deployment model.  For that reason, I would be -1 for
>> removing jsvc support, at least in the 2.x line.
>> 
>> 
>> It's something that could be considered for 3.x if we think the clean-up
>> benefit outweighs the incompatibility cost.  Before we do that, I'd prefer
>> to hear if end users are having success with the SASL deployment model.
>> Brahma, are you asking because you run clusters with the SASL approach?
>> If so, has it been working well?
>> 
>> --Chris Nauroth
>> 
>> 
>> 
>> 
>> On 9/9/15, 9:25 AM, "Haohui Mai" <wh...@apache.org> wrote:
>> 
>>> JSVC is no longer required. It causes a lot of headaches in
>>> deployments. It's definitely a good target for clean ups.
>>> 
>>> ~Haohui
>>> 
>>> On Wed, Sep 9, 2015 at 5:24 AM, Brahma Reddy Battula
>>> <br...@huawei.com> wrote:
>>>> Hi All,
>>>> 
>>>> AFAIK JSVC added secure the block tokens(..?).
>>>> 
>>>> Since block tokens are secure now (SASL used to secure the
>>>> DataTransferProtocol, which transfers file block content between HDFS
>>>> clients and DataNodes),then can we remove jsvc now (script files)..?
>>>> 
>>>> 
>>>> 
>>>> Thanks & Regards
>>>> 
>>>> Brahma Reddy Battula
>>>> 
>>>> 
>>>> 
>>> 
>> 
> 
> 


Re: Even after HDFS-2856 JSVC References are require..?

Posted by Allen Wittenauer <aw...@altiscale.com>.
FWIW, I still use and prefer jsvc, esp with the sudo trick in place.

On Sep 9, 2015, at 9:35 AM, Chris Nauroth <cn...@hortonworks.com> wrote:

> AFAIK, the majority of existing deployments still use jsvc to run a
> secured DataNode.  It would be a backwards-incompatible change to remove
> support for this deployment model.  For that reason, I would be -1 for
> removing jsvc support, at least in the 2.x line.
> 
> 
> It's something that could be considered for 3.x if we think the clean-up
> benefit outweighs the incompatibility cost.  Before we do that, I'd prefer
> to hear if end users are having success with the SASL deployment model.
> Brahma, are you asking because you run clusters with the SASL approach?
> If so, has it been working well?
> 
> --Chris Nauroth
> 
> 
> 
> 
> On 9/9/15, 9:25 AM, "Haohui Mai" <wh...@apache.org> wrote:
> 
>> JSVC is no longer required. It causes a lot of headaches in
>> deployments. It's definitely a good target for clean ups.
>> 
>> ~Haohui
>> 
>> On Wed, Sep 9, 2015 at 5:24 AM, Brahma Reddy Battula
>> <br...@huawei.com> wrote:
>>> Hi All,
>>> 
>>> AFAIK JSVC added secure the block tokens(..?).
>>> 
>>> Since block tokens are secure now (SASL used to secure the
>>> DataTransferProtocol, which transfers file block content between HDFS
>>> clients and DataNodes),then can we remove jsvc now (script files)..?
>>> 
>>> 
>>> 
>>> Thanks & Regards
>>> 
>>> Brahma Reddy Battula
>>> 
>>> 
>>> 
>> 
> 


Re: Even after HDFS-2856 JSVC References are require..?

Posted by Chris Nauroth <cn...@hortonworks.com>.
AFAIK, the majority of existing deployments still use jsvc to run a
secured DataNode.  It would be a backwards-incompatible change to remove
support for this deployment model.  For that reason, I would be -1 for
removing jsvc support, at least in the 2.x line.


It's something that could be considered for 3.x if we think the clean-up
benefit outweighs the incompatibility cost.  Before we do that, I'd prefer
to hear if end users are having success with the SASL deployment model.
Brahma, are you asking because you run clusters with the SASL approach?
If so, has it been working well?

--Chris Nauroth




On 9/9/15, 9:25 AM, "Haohui Mai" <wh...@apache.org> wrote:

>JSVC is no longer required. It causes a lot of headaches in
>deployments. It's definitely a good target for clean ups.
>
>~Haohui
>
>On Wed, Sep 9, 2015 at 5:24 AM, Brahma Reddy Battula
><br...@huawei.com> wrote:
>> Hi All,
>>
>> AFAIK JSVC added secure the block tokens(..?).
>>
>> Since block tokens are secure now (SASL used to secure the
>>DataTransferProtocol, which transfers file block content between HDFS
>>clients and DataNodes),then can we remove jsvc now (script files)..?
>>
>>
>>
>> Thanks & Regards
>>
>>  Brahma Reddy Battula
>>
>>
>>
>


Re: Even after HDFS-2856 JSVC References are require..?

Posted by Haohui Mai <wh...@apache.org>.
JSVC is no longer required. It causes a lot of headaches in
deployments. It's definitely a good target for clean ups.

~Haohui

On Wed, Sep 9, 2015 at 5:24 AM, Brahma Reddy Battula
<br...@huawei.com> wrote:
> Hi All,
>
> AFAIK JSVC added secure the block tokens(..?).
>
> Since block tokens are secure now (SASL used to secure the DataTransferProtocol, which transfers file block content between HDFS clients and DataNodes),then can we remove jsvc now (script files)..?
>
>
>
> Thanks & Regards
>
>  Brahma Reddy Battula
>
>
>