HTTPFS without impersonation

[Nathaniel Braun <n....@criteo.com>]

Hello all,

We need to run several HTTPFS instances on our Hadoop cluster, with different users (basically, one HTTPFS per team).

In our setup, each HTTPFS instance runs as a team user and is allowed write access to that user's directory only (so, HTTPFS does not run as the httpfs user).

However, this setup does not work, as we get exceptions related to impersonation, such as this one:

{"RemoteException":{"message":"User: team_user is not allowed to impersonate team_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}

So, it seems that HTTPFS unconditionally tries to impersonate a user, even though it's running as that same user. Is there a way to somehow disable impersonation?

Thanks for your help.

Regards,
Nathaniel

Re: HTTPFS without impersonation

[Wellington Chevreuil <we...@gmail.com>]

If that doesn't work, u may need to define one entry for these properties
to each user running an httpfs instance.

See below:

http://hadoop.apache.org/docs/current/hadoop-hdfs-httpfs/ServerSetup.html
Em 03/06/2015 12:40, "Wellington Chevreuil" <we...@gmail.com>
escreveu:

> Hi, do u have below property on core-site.xml file used by your hdfs?
>
> <property>
>     <name>hadoop.proxyuser.HTTP.hosts</name>
>     <value>*</value>
>   </property>
>   <property>
>     <name>hadoop.proxyuser.HTTP.groups</name>
>     <value>*</value>
>   </property>
>
> Hello all,
>
>
>
> We need to run several HTTPFS instances on our Hadoop cluster, with
> different users (basically, one HTTPFS per team).
>
>
>
> In our setup, each HTTPFS instance runs as a team user and is allowed
> write access to that user’s directory only (so, HTTPFS does not run as the
> httpfs user).
>
>
>
> However, this setup does not work, as we get exceptions related to
> impersonation, such as this one:
>
>
>
> {"RemoteException":{"message":"User: *team_user* is not allowed to
> impersonate *team_user*
> ","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}
>
>
>
> *So, it seems that HTTPFS unconditionally tries to impersonate a user,
> even though it’s running as that same user*. Is there a way to somehow
> disable impersonation?
>
>
>
> Thanks for your help.
>
>
>
> Regards,
>
> Nathaniel
>
>
>

Re: HTTPFS without impersonation

[Wellington Chevreuil <we...@gmail.com>]

If that doesn't work, u may need to define one entry for these properties
to each user running an httpfs instance.

See below:

http://hadoop.apache.org/docs/current/hadoop-hdfs-httpfs/ServerSetup.html
Em 03/06/2015 12:40, "Wellington Chevreuil" <we...@gmail.com>
escreveu:

> Hi, do u have below property on core-site.xml file used by your hdfs?
>
> <property>
>     <name>hadoop.proxyuser.HTTP.hosts</name>
>     <value>*</value>
>   </property>
>   <property>
>     <name>hadoop.proxyuser.HTTP.groups</name>
>     <value>*</value>
>   </property>
>
> Hello all,
>
>
>
> We need to run several HTTPFS instances on our Hadoop cluster, with
> different users (basically, one HTTPFS per team).
>
>
>
> In our setup, each HTTPFS instance runs as a team user and is allowed
> write access to that user’s directory only (so, HTTPFS does not run as the
> httpfs user).
>
>
>
> However, this setup does not work, as we get exceptions related to
> impersonation, such as this one:
>
>
>
> {"RemoteException":{"message":"User: *team_user* is not allowed to
> impersonate *team_user*
> ","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}
>
>
>
> *So, it seems that HTTPFS unconditionally tries to impersonate a user,
> even though it’s running as that same user*. Is there a way to somehow
> disable impersonation?
>
>
>
> Thanks for your help.
>
>
>
> Regards,
>
> Nathaniel
>
>
>

Re: HTTPFS without impersonation

[Larry McCay <lm...@hortonworks.com>]

Sorry.
No, I don’t think that this is possible and I don’t think that you should try and manipulate the proxy settings in such a way that team users are configured as trusted proxies.
That would introduce risk of exactly the sort of things that you are trying to avoid.

On Jun 3, 2015, at 9:06 AM, Nathaniel Braun <n....@criteo.com>> wrote:

Hi,

Thanks for your answer. I’m not sure I understand it all, though.

Of course, you could send a request to another team’s HTTPFS instance. But you won’t be necessarily be granted access to every operations (based on Kerberos authentication, for instance).

Anyway, my objective was to use HTTPFS without the impersonation mechanisms. Thus, a given HTTPFS instance would be only granted the right of the users under which it runs. Do you think this is possible?

Thanks & regards,
Nathaniel



From: Larry McCay [mailto:lmccay@hortonworks.com]
Sent: mercredi 3 juin 2015 14:28
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation

inline...

On Jun 3, 2015, at 8:03 AM, Nathaniel Braun <n....@criteo.com>> wrote:


Hi,

We want to let users & teams be able to run their HTTPFS in order to isolate instances. One team thus cannot crash another team’s HTTPFS instance.


For my own clarity...
How does this keep them from using instances that are running as another team user?
If the instances are running locally to the user or on a team user specific gateway machine then it should be able to run as http and have the same benefit of physical isolation - no?
If they are running on edge nodes of the cluster then can’t a user send a request to any HttpFs instance?


Now, I make the following request:

curl "localhost:14000/webhdfs/v1/user/team_user?op=LISTSTATUS&user.name=team_user"

And I get the following response:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}


We provide the concept of trusted proxies in Hadoop.
The number of these trusted entities should ideally be kept to a minimum.
A proliferation of such trust relationships can lead to unexpected results and a management headache.

I wouldn’t want to see teamUser1 be trusted to impersonate teamUser2 or HDFS for instance - avoid using ‘*’ for the groups property.

When using a gateway like HttpFs or Knox you want that server to be trusted to act on behalf of other users not every user that uses it to be trusted.

My suggestion is to use HttpFs as a proxy server running as a single user that can be configured as trusted.
Physical isolation can be used across tenants so that they don’t have access to the others instances.


Thanks,
Nathaniel

From: Larry McCay [mailto:lmccay@hortonworks.com]
Sent: mercredi 3 juin 2015 13:57
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation

Out of curiosity, what is the added benefit of having HttpFs run as separate team users give you?
If the APIs are invoked with SPNEGO or a user.name of the appropriate user don’t you get the same permissions based protections?

Generally speaking, gateways such as HttpFs provide access on behalf of endusers.

On Jun 3, 2015, at 7:44 AM, Nathaniel Braun <n....@criteo.com>> wrote:



Hi,

Thanks for your answer.

With this setup, only the HTTP user will be able to impersonate other users, so HTTPFS has to run with the HTTP user.

Instead, I need users to run HTTPFS with their own user, not with the HTTP user.

Thanks

From: Wellington Chevreuil [mailto:wellington.chevreuil@gmail.com]
Sent: mercredi 3 juin 2015 13:41
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation


Hi, do u have below property on core-site.xml file used by your hdfs?

<property>
    <name>hadoop.proxyuser.HTTP.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.HTTP.groups</name>
    <value>*</value>
  </property>

Hello all,

We need to run several HTTPFS instances on our Hadoop cluster, with different users (basically, one HTTPFS per team).

In our setup, each HTTPFS instance runs as a team user and is allowed write access to that user’s directory only (so, HTTPFS does not run as the httpfs user).

However, this setup does not work, as we get exceptions related to impersonation, such as this one:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}

So, it seems that HTTPFS unconditionally tries to impersonate a user, even though it’s running as that same user. Is there a way to somehow disable impersonation?

Thanks for your help.

Regards,
Nathaniel

Re: HTTPFS without impersonation

[Larry McCay <lm...@hortonworks.com>]

Sorry.
No, I don’t think that this is possible and I don’t think that you should try and manipulate the proxy settings in such a way that team users are configured as trusted proxies.
That would introduce risk of exactly the sort of things that you are trying to avoid.

On Jun 3, 2015, at 9:06 AM, Nathaniel Braun <n....@criteo.com>> wrote:

Hi,

Thanks for your answer. I’m not sure I understand it all, though.

Of course, you could send a request to another team’s HTTPFS instance. But you won’t be necessarily be granted access to every operations (based on Kerberos authentication, for instance).

Anyway, my objective was to use HTTPFS without the impersonation mechanisms. Thus, a given HTTPFS instance would be only granted the right of the users under which it runs. Do you think this is possible?

Thanks & regards,
Nathaniel



From: Larry McCay [mailto:lmccay@hortonworks.com]
Sent: mercredi 3 juin 2015 14:28
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation

inline...

On Jun 3, 2015, at 8:03 AM, Nathaniel Braun <n....@criteo.com>> wrote:


Hi,

We want to let users & teams be able to run their HTTPFS in order to isolate instances. One team thus cannot crash another team’s HTTPFS instance.


For my own clarity...
How does this keep them from using instances that are running as another team user?
If the instances are running locally to the user or on a team user specific gateway machine then it should be able to run as http and have the same benefit of physical isolation - no?
If they are running on edge nodes of the cluster then can’t a user send a request to any HttpFs instance?


Now, I make the following request:

curl "localhost:14000/webhdfs/v1/user/team_user?op=LISTSTATUS&user.name=team_user"

And I get the following response:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}


We provide the concept of trusted proxies in Hadoop.
The number of these trusted entities should ideally be kept to a minimum.
A proliferation of such trust relationships can lead to unexpected results and a management headache.

I wouldn’t want to see teamUser1 be trusted to impersonate teamUser2 or HDFS for instance - avoid using ‘*’ for the groups property.

When using a gateway like HttpFs or Knox you want that server to be trusted to act on behalf of other users not every user that uses it to be trusted.

My suggestion is to use HttpFs as a proxy server running as a single user that can be configured as trusted.
Physical isolation can be used across tenants so that they don’t have access to the others instances.


Thanks,
Nathaniel

From: Larry McCay [mailto:lmccay@hortonworks.com]
Sent: mercredi 3 juin 2015 13:57
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation

Out of curiosity, what is the added benefit of having HttpFs run as separate team users give you?
If the APIs are invoked with SPNEGO or a user.name of the appropriate user don’t you get the same permissions based protections?

Generally speaking, gateways such as HttpFs provide access on behalf of endusers.

On Jun 3, 2015, at 7:44 AM, Nathaniel Braun <n....@criteo.com>> wrote:



Hi,

Thanks for your answer.

With this setup, only the HTTP user will be able to impersonate other users, so HTTPFS has to run with the HTTP user.

Instead, I need users to run HTTPFS with their own user, not with the HTTP user.

Thanks

From: Wellington Chevreuil [mailto:wellington.chevreuil@gmail.com]
Sent: mercredi 3 juin 2015 13:41
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation


Hi, do u have below property on core-site.xml file used by your hdfs?

<property>
    <name>hadoop.proxyuser.HTTP.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.HTTP.groups</name>
    <value>*</value>
  </property>

Hello all,

We need to run several HTTPFS instances on our Hadoop cluster, with different users (basically, one HTTPFS per team).

In our setup, each HTTPFS instance runs as a team user and is allowed write access to that user’s directory only (so, HTTPFS does not run as the httpfs user).

However, this setup does not work, as we get exceptions related to impersonation, such as this one:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}

So, it seems that HTTPFS unconditionally tries to impersonate a user, even though it’s running as that same user. Is there a way to somehow disable impersonation?

Thanks for your help.

Regards,
Nathaniel

Re: HTTPFS without impersonation

[Larry McCay <lm...@hortonworks.com>]

Sorry.
No, I don’t think that this is possible and I don’t think that you should try and manipulate the proxy settings in such a way that team users are configured as trusted proxies.
That would introduce risk of exactly the sort of things that you are trying to avoid.

On Jun 3, 2015, at 9:06 AM, Nathaniel Braun <n....@criteo.com>> wrote:

Hi,

Thanks for your answer. I’m not sure I understand it all, though.

Of course, you could send a request to another team’s HTTPFS instance. But you won’t be necessarily be granted access to every operations (based on Kerberos authentication, for instance).

Anyway, my objective was to use HTTPFS without the impersonation mechanisms. Thus, a given HTTPFS instance would be only granted the right of the users under which it runs. Do you think this is possible?

Thanks & regards,
Nathaniel



From: Larry McCay [mailto:lmccay@hortonworks.com]
Sent: mercredi 3 juin 2015 14:28
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation

inline...

On Jun 3, 2015, at 8:03 AM, Nathaniel Braun <n....@criteo.com>> wrote:


Hi,

We want to let users & teams be able to run their HTTPFS in order to isolate instances. One team thus cannot crash another team’s HTTPFS instance.


For my own clarity...
How does this keep them from using instances that are running as another team user?
If the instances are running locally to the user or on a team user specific gateway machine then it should be able to run as http and have the same benefit of physical isolation - no?
If they are running on edge nodes of the cluster then can’t a user send a request to any HttpFs instance?


Now, I make the following request:

curl "localhost:14000/webhdfs/v1/user/team_user?op=LISTSTATUS&user.name=team_user"

And I get the following response:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}


We provide the concept of trusted proxies in Hadoop.
The number of these trusted entities should ideally be kept to a minimum.
A proliferation of such trust relationships can lead to unexpected results and a management headache.

I wouldn’t want to see teamUser1 be trusted to impersonate teamUser2 or HDFS for instance - avoid using ‘*’ for the groups property.

When using a gateway like HttpFs or Knox you want that server to be trusted to act on behalf of other users not every user that uses it to be trusted.

My suggestion is to use HttpFs as a proxy server running as a single user that can be configured as trusted.
Physical isolation can be used across tenants so that they don’t have access to the others instances.


Thanks,
Nathaniel

From: Larry McCay [mailto:lmccay@hortonworks.com]
Sent: mercredi 3 juin 2015 13:57
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation

Out of curiosity, what is the added benefit of having HttpFs run as separate team users give you?
If the APIs are invoked with SPNEGO or a user.name of the appropriate user don’t you get the same permissions based protections?

Generally speaking, gateways such as HttpFs provide access on behalf of endusers.

On Jun 3, 2015, at 7:44 AM, Nathaniel Braun <n....@criteo.com>> wrote:



Hi,

Thanks for your answer.

With this setup, only the HTTP user will be able to impersonate other users, so HTTPFS has to run with the HTTP user.

Instead, I need users to run HTTPFS with their own user, not with the HTTP user.

Thanks

From: Wellington Chevreuil [mailto:wellington.chevreuil@gmail.com]
Sent: mercredi 3 juin 2015 13:41
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation


Hi, do u have below property on core-site.xml file used by your hdfs?

<property>
    <name>hadoop.proxyuser.HTTP.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.HTTP.groups</name>
    <value>*</value>
  </property>

Hello all,

We need to run several HTTPFS instances on our Hadoop cluster, with different users (basically, one HTTPFS per team).

In our setup, each HTTPFS instance runs as a team user and is allowed write access to that user’s directory only (so, HTTPFS does not run as the httpfs user).

However, this setup does not work, as we get exceptions related to impersonation, such as this one:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}

So, it seems that HTTPFS unconditionally tries to impersonate a user, even though it’s running as that same user. Is there a way to somehow disable impersonation?

Thanks for your help.

Regards,
Nathaniel

Re: HTTPFS without impersonation

[Larry McCay <lm...@hortonworks.com>]

Sorry.
No, I don’t think that this is possible and I don’t think that you should try and manipulate the proxy settings in such a way that team users are configured as trusted proxies.
That would introduce risk of exactly the sort of things that you are trying to avoid.

On Jun 3, 2015, at 9:06 AM, Nathaniel Braun <n....@criteo.com>> wrote:

Hi,

Thanks for your answer. I’m not sure I understand it all, though.

Of course, you could send a request to another team’s HTTPFS instance. But you won’t be necessarily be granted access to every operations (based on Kerberos authentication, for instance).

Anyway, my objective was to use HTTPFS without the impersonation mechanisms. Thus, a given HTTPFS instance would be only granted the right of the users under which it runs. Do you think this is possible?

Thanks & regards,
Nathaniel



From: Larry McCay [mailto:lmccay@hortonworks.com]
Sent: mercredi 3 juin 2015 14:28
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation

inline...

On Jun 3, 2015, at 8:03 AM, Nathaniel Braun <n....@criteo.com>> wrote:


Hi,

We want to let users & teams be able to run their HTTPFS in order to isolate instances. One team thus cannot crash another team’s HTTPFS instance.


For my own clarity...
How does this keep them from using instances that are running as another team user?
If the instances are running locally to the user or on a team user specific gateway machine then it should be able to run as http and have the same benefit of physical isolation - no?
If they are running on edge nodes of the cluster then can’t a user send a request to any HttpFs instance?


Now, I make the following request:

curl "localhost:14000/webhdfs/v1/user/team_user?op=LISTSTATUS&user.name=team_user"

And I get the following response:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}


We provide the concept of trusted proxies in Hadoop.
The number of these trusted entities should ideally be kept to a minimum.
A proliferation of such trust relationships can lead to unexpected results and a management headache.

I wouldn’t want to see teamUser1 be trusted to impersonate teamUser2 or HDFS for instance - avoid using ‘*’ for the groups property.

When using a gateway like HttpFs or Knox you want that server to be trusted to act on behalf of other users not every user that uses it to be trusted.

My suggestion is to use HttpFs as a proxy server running as a single user that can be configured as trusted.
Physical isolation can be used across tenants so that they don’t have access to the others instances.


Thanks,
Nathaniel

From: Larry McCay [mailto:lmccay@hortonworks.com]
Sent: mercredi 3 juin 2015 13:57
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation

Out of curiosity, what is the added benefit of having HttpFs run as separate team users give you?
If the APIs are invoked with SPNEGO or a user.name of the appropriate user don’t you get the same permissions based protections?

Generally speaking, gateways such as HttpFs provide access on behalf of endusers.

On Jun 3, 2015, at 7:44 AM, Nathaniel Braun <n....@criteo.com>> wrote:



Hi,

Thanks for your answer.

With this setup, only the HTTP user will be able to impersonate other users, so HTTPFS has to run with the HTTP user.

Instead, I need users to run HTTPFS with their own user, not with the HTTP user.

Thanks

From: Wellington Chevreuil [mailto:wellington.chevreuil@gmail.com]
Sent: mercredi 3 juin 2015 13:41
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation


Hi, do u have below property on core-site.xml file used by your hdfs?

<property>
    <name>hadoop.proxyuser.HTTP.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.HTTP.groups</name>
    <value>*</value>
  </property>

Hello all,

We need to run several HTTPFS instances on our Hadoop cluster, with different users (basically, one HTTPFS per team).

In our setup, each HTTPFS instance runs as a team user and is allowed write access to that user’s directory only (so, HTTPFS does not run as the httpfs user).

However, this setup does not work, as we get exceptions related to impersonation, such as this one:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}

So, it seems that HTTPFS unconditionally tries to impersonate a user, even though it’s running as that same user. Is there a way to somehow disable impersonation?

Thanks for your help.

Regards,
Nathaniel

RE: HTTPFS without impersonation

[Nathaniel Braun <n....@criteo.com>]

Hi,

Thanks for your answer. I'm not sure I understand it all, though.

Of course, you could send a request to another team's HTTPFS instance. But you won't be necessarily be granted access to every operations (based on Kerberos authentication, for instance).

Anyway, my objective was to use HTTPFS without the impersonation mechanisms. Thus, a given HTTPFS instance would be only granted the right of the users under which it runs. Do you think this is possible?

Thanks & regards,
Nathaniel



From: Larry McCay [mailto:lmccay@hortonworks.com]
Sent: mercredi 3 juin 2015 14:28
To: user@hadoop.apache.org
Subject: Re: HTTPFS without impersonation

inline...

On Jun 3, 2015, at 8:03 AM, Nathaniel Braun <n....@criteo.com>> wrote:


Hi,

We want to let users & teams be able to run their HTTPFS in order to isolate instances. One team thus cannot crash another team's HTTPFS instance.


For my own clarity...
How does this keep them from using instances that are running as another team user?
If the instances are running locally to the user or on a team user specific gateway machine then it should be able to run as http and have the same benefit of physical isolation - no?
If they are running on edge nodes of the cluster then can't a user send a request to any HttpFs instance?


Now, I make the following request:

curl "localhost:14000/webhdfs/v1/user/team_user?op=LISTSTATUS&user.name=team_user"

And I get the following response:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}


We provide the concept of trusted proxies in Hadoop.
The number of these trusted entities should ideally be kept to a minimum.
A proliferation of such trust relationships can lead to unexpected results and a management headache.

I wouldn't want to see teamUser1 be trusted to impersonate teamUser2 or HDFS for instance - avoid using '*' for the groups property.

When using a gateway like HttpFs or Knox you want that server to be trusted to act on behalf of other users not every user that uses it to be trusted.

My suggestion is to use HttpFs as a proxy server running as a single user that can be configured as trusted.
Physical isolation can be used across tenants so that they don't have access to the others instances.


Thanks,
Nathaniel

From: Larry McCay [mailto:lmccay@hortonworks.com]
Sent: mercredi 3 juin 2015 13:57
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation

Out of curiosity, what is the added benefit of having HttpFs run as separate team users give you?
If the APIs are invoked with SPNEGO or a user.name of the appropriate user don't you get the same permissions based protections?

Generally speaking, gateways such as HttpFs provide access on behalf of endusers.

On Jun 3, 2015, at 7:44 AM, Nathaniel Braun <n....@criteo.com>> wrote:



Hi,

Thanks for your answer.

With this setup, only the HTTP user will be able to impersonate other users, so HTTPFS has to run with the HTTP user.

Instead, I need users to run HTTPFS with their own user, not with the HTTP user.

Thanks

From: Wellington Chevreuil [mailto:wellington.chevreuil@gmail.com]
Sent: mercredi 3 juin 2015 13:41
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation


Hi, do u have below property on core-site.xml file used by your hdfs?

<property>
    <name>hadoop.proxyuser.HTTP.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.HTTP.groups</name>
    <value>*</value>
  </property>
Hello all,

We need to run several HTTPFS instances on our Hadoop cluster, with different users (basically, one HTTPFS per team).

In our setup, each HTTPFS instance runs as a team user and is allowed write access to that user's directory only (so, HTTPFS does not run as the httpfs user).

However, this setup does not work, as we get exceptions related to impersonation, such as this one:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}

So, it seems that HTTPFS unconditionally tries to impersonate a user, even though it's running as that same user. Is there a way to somehow disable impersonation?

Thanks for your help.

Regards,
Nathaniel

RE: HTTPFS without impersonation

[Nathaniel Braun <n....@criteo.com>]

Hi,

Thanks for your answer. I'm not sure I understand it all, though.

Of course, you could send a request to another team's HTTPFS instance. But you won't be necessarily be granted access to every operations (based on Kerberos authentication, for instance).

Anyway, my objective was to use HTTPFS without the impersonation mechanisms. Thus, a given HTTPFS instance would be only granted the right of the users under which it runs. Do you think this is possible?

Thanks & regards,
Nathaniel



From: Larry McCay [mailto:lmccay@hortonworks.com]
Sent: mercredi 3 juin 2015 14:28
To: user@hadoop.apache.org
Subject: Re: HTTPFS without impersonation

inline...

On Jun 3, 2015, at 8:03 AM, Nathaniel Braun <n....@criteo.com>> wrote:


Hi,

We want to let users & teams be able to run their HTTPFS in order to isolate instances. One team thus cannot crash another team's HTTPFS instance.


For my own clarity...
How does this keep them from using instances that are running as another team user?
If the instances are running locally to the user or on a team user specific gateway machine then it should be able to run as http and have the same benefit of physical isolation - no?
If they are running on edge nodes of the cluster then can't a user send a request to any HttpFs instance?


Now, I make the following request:

curl "localhost:14000/webhdfs/v1/user/team_user?op=LISTSTATUS&user.name=team_user"

And I get the following response:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}


We provide the concept of trusted proxies in Hadoop.
The number of these trusted entities should ideally be kept to a minimum.
A proliferation of such trust relationships can lead to unexpected results and a management headache.

I wouldn't want to see teamUser1 be trusted to impersonate teamUser2 or HDFS for instance - avoid using '*' for the groups property.

When using a gateway like HttpFs or Knox you want that server to be trusted to act on behalf of other users not every user that uses it to be trusted.

My suggestion is to use HttpFs as a proxy server running as a single user that can be configured as trusted.
Physical isolation can be used across tenants so that they don't have access to the others instances.


Thanks,
Nathaniel

From: Larry McCay [mailto:lmccay@hortonworks.com]
Sent: mercredi 3 juin 2015 13:57
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation

Out of curiosity, what is the added benefit of having HttpFs run as separate team users give you?
If the APIs are invoked with SPNEGO or a user.name of the appropriate user don't you get the same permissions based protections?

Generally speaking, gateways such as HttpFs provide access on behalf of endusers.

On Jun 3, 2015, at 7:44 AM, Nathaniel Braun <n....@criteo.com>> wrote:



Hi,

Thanks for your answer.

With this setup, only the HTTP user will be able to impersonate other users, so HTTPFS has to run with the HTTP user.

Instead, I need users to run HTTPFS with their own user, not with the HTTP user.

Thanks

From: Wellington Chevreuil [mailto:wellington.chevreuil@gmail.com]
Sent: mercredi 3 juin 2015 13:41
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation


Hi, do u have below property on core-site.xml file used by your hdfs?

<property>
    <name>hadoop.proxyuser.HTTP.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.HTTP.groups</name>
    <value>*</value>
  </property>
Hello all,

We need to run several HTTPFS instances on our Hadoop cluster, with different users (basically, one HTTPFS per team).

In our setup, each HTTPFS instance runs as a team user and is allowed write access to that user's directory only (so, HTTPFS does not run as the httpfs user).

However, this setup does not work, as we get exceptions related to impersonation, such as this one:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}

So, it seems that HTTPFS unconditionally tries to impersonate a user, even though it's running as that same user. Is there a way to somehow disable impersonation?

Thanks for your help.

Regards,
Nathaniel

RE: HTTPFS without impersonation

[Nathaniel Braun <n....@criteo.com>]

Hi,

Thanks for your answer. I'm not sure I understand it all, though.

Of course, you could send a request to another team's HTTPFS instance. But you won't be necessarily be granted access to every operations (based on Kerberos authentication, for instance).

Anyway, my objective was to use HTTPFS without the impersonation mechanisms. Thus, a given HTTPFS instance would be only granted the right of the users under which it runs. Do you think this is possible?

Thanks & regards,
Nathaniel



From: Larry McCay [mailto:lmccay@hortonworks.com]
Sent: mercredi 3 juin 2015 14:28
To: user@hadoop.apache.org
Subject: Re: HTTPFS without impersonation

inline...

On Jun 3, 2015, at 8:03 AM, Nathaniel Braun <n....@criteo.com>> wrote:


Hi,

We want to let users & teams be able to run their HTTPFS in order to isolate instances. One team thus cannot crash another team's HTTPFS instance.


For my own clarity...
How does this keep them from using instances that are running as another team user?
If the instances are running locally to the user or on a team user specific gateway machine then it should be able to run as http and have the same benefit of physical isolation - no?
If they are running on edge nodes of the cluster then can't a user send a request to any HttpFs instance?


Now, I make the following request:

curl "localhost:14000/webhdfs/v1/user/team_user?op=LISTSTATUS&user.name=team_user"

And I get the following response:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}


We provide the concept of trusted proxies in Hadoop.
The number of these trusted entities should ideally be kept to a minimum.
A proliferation of such trust relationships can lead to unexpected results and a management headache.

I wouldn't want to see teamUser1 be trusted to impersonate teamUser2 or HDFS for instance - avoid using '*' for the groups property.

When using a gateway like HttpFs or Knox you want that server to be trusted to act on behalf of other users not every user that uses it to be trusted.

My suggestion is to use HttpFs as a proxy server running as a single user that can be configured as trusted.
Physical isolation can be used across tenants so that they don't have access to the others instances.


Thanks,
Nathaniel

From: Larry McCay [mailto:lmccay@hortonworks.com]
Sent: mercredi 3 juin 2015 13:57
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation

Out of curiosity, what is the added benefit of having HttpFs run as separate team users give you?
If the APIs are invoked with SPNEGO or a user.name of the appropriate user don't you get the same permissions based protections?

Generally speaking, gateways such as HttpFs provide access on behalf of endusers.

On Jun 3, 2015, at 7:44 AM, Nathaniel Braun <n....@criteo.com>> wrote:



Hi,

Thanks for your answer.

With this setup, only the HTTP user will be able to impersonate other users, so HTTPFS has to run with the HTTP user.

Instead, I need users to run HTTPFS with their own user, not with the HTTP user.

Thanks

From: Wellington Chevreuil [mailto:wellington.chevreuil@gmail.com]
Sent: mercredi 3 juin 2015 13:41
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation


Hi, do u have below property on core-site.xml file used by your hdfs?

<property>
    <name>hadoop.proxyuser.HTTP.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.HTTP.groups</name>
    <value>*</value>
  </property>
Hello all,

We need to run several HTTPFS instances on our Hadoop cluster, with different users (basically, one HTTPFS per team).

In our setup, each HTTPFS instance runs as a team user and is allowed write access to that user's directory only (so, HTTPFS does not run as the httpfs user).

However, this setup does not work, as we get exceptions related to impersonation, such as this one:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}

So, it seems that HTTPFS unconditionally tries to impersonate a user, even though it's running as that same user. Is there a way to somehow disable impersonation?

Thanks for your help.

Regards,
Nathaniel

RE: HTTPFS without impersonation

[Nathaniel Braun <n....@criteo.com>]

Hi,

Thanks for your answer. I'm not sure I understand it all, though.

Of course, you could send a request to another team's HTTPFS instance. But you won't be necessarily be granted access to every operations (based on Kerberos authentication, for instance).

Anyway, my objective was to use HTTPFS without the impersonation mechanisms. Thus, a given HTTPFS instance would be only granted the right of the users under which it runs. Do you think this is possible?

Thanks & regards,
Nathaniel



From: Larry McCay [mailto:lmccay@hortonworks.com]
Sent: mercredi 3 juin 2015 14:28
To: user@hadoop.apache.org
Subject: Re: HTTPFS without impersonation

inline...

On Jun 3, 2015, at 8:03 AM, Nathaniel Braun <n....@criteo.com>> wrote:


Hi,

We want to let users & teams be able to run their HTTPFS in order to isolate instances. One team thus cannot crash another team's HTTPFS instance.


For my own clarity...
How does this keep them from using instances that are running as another team user?
If the instances are running locally to the user or on a team user specific gateway machine then it should be able to run as http and have the same benefit of physical isolation - no?
If they are running on edge nodes of the cluster then can't a user send a request to any HttpFs instance?


Now, I make the following request:

curl "localhost:14000/webhdfs/v1/user/team_user?op=LISTSTATUS&user.name=team_user"

And I get the following response:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}


We provide the concept of trusted proxies in Hadoop.
The number of these trusted entities should ideally be kept to a minimum.
A proliferation of such trust relationships can lead to unexpected results and a management headache.

I wouldn't want to see teamUser1 be trusted to impersonate teamUser2 or HDFS for instance - avoid using '*' for the groups property.

When using a gateway like HttpFs or Knox you want that server to be trusted to act on behalf of other users not every user that uses it to be trusted.

My suggestion is to use HttpFs as a proxy server running as a single user that can be configured as trusted.
Physical isolation can be used across tenants so that they don't have access to the others instances.


Thanks,
Nathaniel

From: Larry McCay [mailto:lmccay@hortonworks.com]
Sent: mercredi 3 juin 2015 13:57
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation

Out of curiosity, what is the added benefit of having HttpFs run as separate team users give you?
If the APIs are invoked with SPNEGO or a user.name of the appropriate user don't you get the same permissions based protections?

Generally speaking, gateways such as HttpFs provide access on behalf of endusers.

On Jun 3, 2015, at 7:44 AM, Nathaniel Braun <n....@criteo.com>> wrote:



Hi,

Thanks for your answer.

With this setup, only the HTTP user will be able to impersonate other users, so HTTPFS has to run with the HTTP user.

Instead, I need users to run HTTPFS with their own user, not with the HTTP user.

Thanks

From: Wellington Chevreuil [mailto:wellington.chevreuil@gmail.com]
Sent: mercredi 3 juin 2015 13:41
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation


Hi, do u have below property on core-site.xml file used by your hdfs?

<property>
    <name>hadoop.proxyuser.HTTP.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.HTTP.groups</name>
    <value>*</value>
  </property>
Hello all,

We need to run several HTTPFS instances on our Hadoop cluster, with different users (basically, one HTTPFS per team).

In our setup, each HTTPFS instance runs as a team user and is allowed write access to that user's directory only (so, HTTPFS does not run as the httpfs user).

However, this setup does not work, as we get exceptions related to impersonation, such as this one:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}

So, it seems that HTTPFS unconditionally tries to impersonate a user, even though it's running as that same user. Is there a way to somehow disable impersonation?

Thanks for your help.

Regards,
Nathaniel

Re: HTTPFS without impersonation

[Larry McCay <lm...@hortonworks.com>]

inline...

On Jun 3, 2015, at 8:03 AM, Nathaniel Braun <n....@criteo.com>> wrote:

Hi,

We want to let users & teams be able to run their HTTPFS in order to isolate instances. One team thus cannot crash another team’s HTTPFS instance.


For my own clarity...
How does this keep them from using instances that are running as another team user?
If the instances are running locally to the user or on a team user specific gateway machine then it should be able to run as http and have the same benefit of physical isolation - no?
If they are running on edge nodes of the cluster then can’t a user send a request to any HttpFs instance?

Now, I make the following request:

curl "localhost:14000/webhdfs/v1/user/team_user?op=LISTSTATUS&user.name=team_user"

And I get the following response:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}


We provide the concept of trusted proxies in Hadoop.
The number of these trusted entities should ideally be kept to a minimum.
A proliferation of such trust relationships can lead to unexpected results and a management headache.

I wouldn’t want to see teamUser1 be trusted to impersonate teamUser2 or HDFS for instance - avoid using ‘*’ for the groups property.

When using a gateway like HttpFs or Knox you want that server to be trusted to act on behalf of other users not every user that uses it to be trusted.

My suggestion is to use HttpFs as a proxy server running as a single user that can be configured as trusted.
Physical isolation can be used across tenants so that they don’t have access to the others instances.

Thanks,
Nathaniel

From: Larry McCay [mailto:lmccay@hortonworks.com]
Sent: mercredi 3 juin 2015 13:57
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation

Out of curiosity, what is the added benefit of having HttpFs run as separate team users give you?
If the APIs are invoked with SPNEGO or a user.name of the appropriate user don’t you get the same permissions based protections?

Generally speaking, gateways such as HttpFs provide access on behalf of endusers.

On Jun 3, 2015, at 7:44 AM, Nathaniel Braun <n....@criteo.com>> wrote:


Hi,

Thanks for your answer.

With this setup, only the HTTP user will be able to impersonate other users, so HTTPFS has to run with the HTTP user.

Instead, I need users to run HTTPFS with their own user, not with the HTTP user.

Thanks

From: Wellington Chevreuil [mailto:wellington.chevreuil@gmail.com]
Sent: mercredi 3 juin 2015 13:41
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation


Hi, do u have below property on core-site.xml file used by your hdfs?

<property>
    <name>hadoop.proxyuser.HTTP.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.HTTP.groups</name>
    <value>*</value>
  </property>

Hello all,

We need to run several HTTPFS instances on our Hadoop cluster, with different users (basically, one HTTPFS per team).

In our setup, each HTTPFS instance runs as a team user and is allowed write access to that user’s directory only (so, HTTPFS does not run as the httpfs user).

However, this setup does not work, as we get exceptions related to impersonation, such as this one:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}

So, it seems that HTTPFS unconditionally tries to impersonate a user, even though it’s running as that same user. Is there a way to somehow disable impersonation?

Thanks for your help.

Regards,
Nathaniel

Re: HTTPFS without impersonation

[Larry McCay <lm...@hortonworks.com>]

inline...

On Jun 3, 2015, at 8:03 AM, Nathaniel Braun <n....@criteo.com>> wrote:

Hi,

We want to let users & teams be able to run their HTTPFS in order to isolate instances. One team thus cannot crash another team’s HTTPFS instance.


For my own clarity...
How does this keep them from using instances that are running as another team user?
If the instances are running locally to the user or on a team user specific gateway machine then it should be able to run as http and have the same benefit of physical isolation - no?
If they are running on edge nodes of the cluster then can’t a user send a request to any HttpFs instance?

Now, I make the following request:

curl "localhost:14000/webhdfs/v1/user/team_user?op=LISTSTATUS&user.name=team_user"

And I get the following response:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}


We provide the concept of trusted proxies in Hadoop.
The number of these trusted entities should ideally be kept to a minimum.
A proliferation of such trust relationships can lead to unexpected results and a management headache.

I wouldn’t want to see teamUser1 be trusted to impersonate teamUser2 or HDFS for instance - avoid using ‘*’ for the groups property.

When using a gateway like HttpFs or Knox you want that server to be trusted to act on behalf of other users not every user that uses it to be trusted.

My suggestion is to use HttpFs as a proxy server running as a single user that can be configured as trusted.
Physical isolation can be used across tenants so that they don’t have access to the others instances.

Thanks,
Nathaniel

From: Larry McCay [mailto:lmccay@hortonworks.com]
Sent: mercredi 3 juin 2015 13:57
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation

Out of curiosity, what is the added benefit of having HttpFs run as separate team users give you?
If the APIs are invoked with SPNEGO or a user.name of the appropriate user don’t you get the same permissions based protections?

Generally speaking, gateways such as HttpFs provide access on behalf of endusers.

On Jun 3, 2015, at 7:44 AM, Nathaniel Braun <n....@criteo.com>> wrote:


Hi,

Thanks for your answer.

With this setup, only the HTTP user will be able to impersonate other users, so HTTPFS has to run with the HTTP user.

Instead, I need users to run HTTPFS with their own user, not with the HTTP user.

Thanks

From: Wellington Chevreuil [mailto:wellington.chevreuil@gmail.com]
Sent: mercredi 3 juin 2015 13:41
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation


Hi, do u have below property on core-site.xml file used by your hdfs?

<property>
    <name>hadoop.proxyuser.HTTP.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.HTTP.groups</name>
    <value>*</value>
  </property>

Hello all,

We need to run several HTTPFS instances on our Hadoop cluster, with different users (basically, one HTTPFS per team).

In our setup, each HTTPFS instance runs as a team user and is allowed write access to that user’s directory only (so, HTTPFS does not run as the httpfs user).

However, this setup does not work, as we get exceptions related to impersonation, such as this one:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}

So, it seems that HTTPFS unconditionally tries to impersonate a user, even though it’s running as that same user. Is there a way to somehow disable impersonation?

Thanks for your help.

Regards,
Nathaniel

Re: HTTPFS without impersonation

[Larry McCay <lm...@hortonworks.com>]

inline...

On Jun 3, 2015, at 8:03 AM, Nathaniel Braun <n....@criteo.com>> wrote:

Hi,

We want to let users & teams be able to run their HTTPFS in order to isolate instances. One team thus cannot crash another team’s HTTPFS instance.


For my own clarity...
How does this keep them from using instances that are running as another team user?
If the instances are running locally to the user or on a team user specific gateway machine then it should be able to run as http and have the same benefit of physical isolation - no?
If they are running on edge nodes of the cluster then can’t a user send a request to any HttpFs instance?

Now, I make the following request:

curl "localhost:14000/webhdfs/v1/user/team_user?op=LISTSTATUS&user.name=team_user"

And I get the following response:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}


We provide the concept of trusted proxies in Hadoop.
The number of these trusted entities should ideally be kept to a minimum.
A proliferation of such trust relationships can lead to unexpected results and a management headache.

I wouldn’t want to see teamUser1 be trusted to impersonate teamUser2 or HDFS for instance - avoid using ‘*’ for the groups property.

When using a gateway like HttpFs or Knox you want that server to be trusted to act on behalf of other users not every user that uses it to be trusted.

My suggestion is to use HttpFs as a proxy server running as a single user that can be configured as trusted.
Physical isolation can be used across tenants so that they don’t have access to the others instances.

Thanks,
Nathaniel

From: Larry McCay [mailto:lmccay@hortonworks.com]
Sent: mercredi 3 juin 2015 13:57
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation

Out of curiosity, what is the added benefit of having HttpFs run as separate team users give you?
If the APIs are invoked with SPNEGO or a user.name of the appropriate user don’t you get the same permissions based protections?

Generally speaking, gateways such as HttpFs provide access on behalf of endusers.

On Jun 3, 2015, at 7:44 AM, Nathaniel Braun <n....@criteo.com>> wrote:


Hi,

Thanks for your answer.

With this setup, only the HTTP user will be able to impersonate other users, so HTTPFS has to run with the HTTP user.

Instead, I need users to run HTTPFS with their own user, not with the HTTP user.

Thanks

From: Wellington Chevreuil [mailto:wellington.chevreuil@gmail.com]
Sent: mercredi 3 juin 2015 13:41
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation


Hi, do u have below property on core-site.xml file used by your hdfs?

<property>
    <name>hadoop.proxyuser.HTTP.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.HTTP.groups</name>
    <value>*</value>
  </property>

Hello all,

We need to run several HTTPFS instances on our Hadoop cluster, with different users (basically, one HTTPFS per team).

In our setup, each HTTPFS instance runs as a team user and is allowed write access to that user’s directory only (so, HTTPFS does not run as the httpfs user).

However, this setup does not work, as we get exceptions related to impersonation, such as this one:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}

So, it seems that HTTPFS unconditionally tries to impersonate a user, even though it’s running as that same user. Is there a way to somehow disable impersonation?

Thanks for your help.

Regards,
Nathaniel

Re: HTTPFS without impersonation

[Larry McCay <lm...@hortonworks.com>]

inline...

On Jun 3, 2015, at 8:03 AM, Nathaniel Braun <n....@criteo.com>> wrote:

Hi,

We want to let users & teams be able to run their HTTPFS in order to isolate instances. One team thus cannot crash another team’s HTTPFS instance.


For my own clarity...
How does this keep them from using instances that are running as another team user?
If the instances are running locally to the user or on a team user specific gateway machine then it should be able to run as http and have the same benefit of physical isolation - no?
If they are running on edge nodes of the cluster then can’t a user send a request to any HttpFs instance?

Now, I make the following request:

curl "localhost:14000/webhdfs/v1/user/team_user?op=LISTSTATUS&user.name=team_user"

And I get the following response:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}


We provide the concept of trusted proxies in Hadoop.
The number of these trusted entities should ideally be kept to a minimum.
A proliferation of such trust relationships can lead to unexpected results and a management headache.

I wouldn’t want to see teamUser1 be trusted to impersonate teamUser2 or HDFS for instance - avoid using ‘*’ for the groups property.

When using a gateway like HttpFs or Knox you want that server to be trusted to act on behalf of other users not every user that uses it to be trusted.

My suggestion is to use HttpFs as a proxy server running as a single user that can be configured as trusted.
Physical isolation can be used across tenants so that they don’t have access to the others instances.

Thanks,
Nathaniel

From: Larry McCay [mailto:lmccay@hortonworks.com]
Sent: mercredi 3 juin 2015 13:57
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation

Out of curiosity, what is the added benefit of having HttpFs run as separate team users give you?
If the APIs are invoked with SPNEGO or a user.name of the appropriate user don’t you get the same permissions based protections?

Generally speaking, gateways such as HttpFs provide access on behalf of endusers.

On Jun 3, 2015, at 7:44 AM, Nathaniel Braun <n....@criteo.com>> wrote:


Hi,

Thanks for your answer.

With this setup, only the HTTP user will be able to impersonate other users, so HTTPFS has to run with the HTTP user.

Instead, I need users to run HTTPFS with their own user, not with the HTTP user.

Thanks

From: Wellington Chevreuil [mailto:wellington.chevreuil@gmail.com]
Sent: mercredi 3 juin 2015 13:41
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation


Hi, do u have below property on core-site.xml file used by your hdfs?

<property>
    <name>hadoop.proxyuser.HTTP.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.HTTP.groups</name>
    <value>*</value>
  </property>

Hello all,

We need to run several HTTPFS instances on our Hadoop cluster, with different users (basically, one HTTPFS per team).

In our setup, each HTTPFS instance runs as a team user and is allowed write access to that user’s directory only (so, HTTPFS does not run as the httpfs user).

However, this setup does not work, as we get exceptions related to impersonation, such as this one:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}

So, it seems that HTTPFS unconditionally tries to impersonate a user, even though it’s running as that same user. Is there a way to somehow disable impersonation?

Thanks for your help.

Regards,
Nathaniel

RE: HTTPFS without impersonation

[Nathaniel Braun <n....@criteo.com>]

Hi,

We want to let users & teams be able to run their HTTPFS in order to isolate instances. One team thus cannot crash another team's HTTPFS instance.

Now, I make the following request:

curl "localhost:14000/webhdfs/v1/user/team_user?op=LISTSTATUS&user.name=team_user"

And I get the following response:

{"RemoteException":{"message":"User: team_user is not allowed to impersonate team_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}

Thanks,
Nathaniel

From: Larry McCay [mailto:lmccay@hortonworks.com]
Sent: mercredi 3 juin 2015 13:57
To: user@hadoop.apache.org
Subject: Re: HTTPFS without impersonation

Out of curiosity, what is the added benefit of having HttpFs run as separate team users give you?
If the APIs are invoked with SPNEGO or a user.name of the appropriate user don't you get the same permissions based protections?

Generally speaking, gateways such as HttpFs provide access on behalf of endusers.

On Jun 3, 2015, at 7:44 AM, Nathaniel Braun <n....@criteo.com>> wrote:


Hi,

Thanks for your answer.

With this setup, only the HTTP user will be able to impersonate other users, so HTTPFS has to run with the HTTP user.

Instead, I need users to run HTTPFS with their own user, not with the HTTP user.

Thanks

From: Wellington Chevreuil [mailto:wellington.chevreuil@gmail.com]
Sent: mercredi 3 juin 2015 13:41
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation


Hi, do u have below property on core-site.xml file used by your hdfs?

<property>
    <name>hadoop.proxyuser.HTTP.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.HTTP.groups</name>
    <value>*</value>
  </property>
Hello all,

We need to run several HTTPFS instances on our Hadoop cluster, with different users (basically, one HTTPFS per team).

In our setup, each HTTPFS instance runs as a team user and is allowed write access to that user's directory only (so, HTTPFS does not run as the httpfs user).

However, this setup does not work, as we get exceptions related to impersonation, such as this one:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}

So, it seems that HTTPFS unconditionally tries to impersonate a user, even though it's running as that same user. Is there a way to somehow disable impersonation?

Thanks for your help.

Regards,
Nathaniel

RE: HTTPFS without impersonation

[Nathaniel Braun <n....@criteo.com>]

Hi,

We want to let users & teams be able to run their HTTPFS in order to isolate instances. One team thus cannot crash another team's HTTPFS instance.

Now, I make the following request:

curl "localhost:14000/webhdfs/v1/user/team_user?op=LISTSTATUS&user.name=team_user"

And I get the following response:

{"RemoteException":{"message":"User: team_user is not allowed to impersonate team_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}

Thanks,
Nathaniel

From: Larry McCay [mailto:lmccay@hortonworks.com]
Sent: mercredi 3 juin 2015 13:57
To: user@hadoop.apache.org
Subject: Re: HTTPFS without impersonation

Out of curiosity, what is the added benefit of having HttpFs run as separate team users give you?
If the APIs are invoked with SPNEGO or a user.name of the appropriate user don't you get the same permissions based protections?

Generally speaking, gateways such as HttpFs provide access on behalf of endusers.

On Jun 3, 2015, at 7:44 AM, Nathaniel Braun <n....@criteo.com>> wrote:


Hi,

Thanks for your answer.

With this setup, only the HTTP user will be able to impersonate other users, so HTTPFS has to run with the HTTP user.

Instead, I need users to run HTTPFS with their own user, not with the HTTP user.

Thanks

From: Wellington Chevreuil [mailto:wellington.chevreuil@gmail.com]
Sent: mercredi 3 juin 2015 13:41
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation


Hi, do u have below property on core-site.xml file used by your hdfs?

<property>
    <name>hadoop.proxyuser.HTTP.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.HTTP.groups</name>
    <value>*</value>
  </property>
Hello all,

We need to run several HTTPFS instances on our Hadoop cluster, with different users (basically, one HTTPFS per team).

In our setup, each HTTPFS instance runs as a team user and is allowed write access to that user's directory only (so, HTTPFS does not run as the httpfs user).

However, this setup does not work, as we get exceptions related to impersonation, such as this one:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}

So, it seems that HTTPFS unconditionally tries to impersonate a user, even though it's running as that same user. Is there a way to somehow disable impersonation?

Thanks for your help.

Regards,
Nathaniel

RE: HTTPFS without impersonation

[Nathaniel Braun <n....@criteo.com>]

Hi,

We want to let users & teams be able to run their HTTPFS in order to isolate instances. One team thus cannot crash another team's HTTPFS instance.

Now, I make the following request:

curl "localhost:14000/webhdfs/v1/user/team_user?op=LISTSTATUS&user.name=team_user"

And I get the following response:

{"RemoteException":{"message":"User: team_user is not allowed to impersonate team_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}

Thanks,
Nathaniel

From: Larry McCay [mailto:lmccay@hortonworks.com]
Sent: mercredi 3 juin 2015 13:57
To: user@hadoop.apache.org
Subject: Re: HTTPFS without impersonation

Out of curiosity, what is the added benefit of having HttpFs run as separate team users give you?
If the APIs are invoked with SPNEGO or a user.name of the appropriate user don't you get the same permissions based protections?

Generally speaking, gateways such as HttpFs provide access on behalf of endusers.

On Jun 3, 2015, at 7:44 AM, Nathaniel Braun <n....@criteo.com>> wrote:


Hi,

Thanks for your answer.

With this setup, only the HTTP user will be able to impersonate other users, so HTTPFS has to run with the HTTP user.

Instead, I need users to run HTTPFS with their own user, not with the HTTP user.

Thanks

From: Wellington Chevreuil [mailto:wellington.chevreuil@gmail.com]
Sent: mercredi 3 juin 2015 13:41
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation


Hi, do u have below property on core-site.xml file used by your hdfs?

<property>
    <name>hadoop.proxyuser.HTTP.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.HTTP.groups</name>
    <value>*</value>
  </property>
Hello all,

We need to run several HTTPFS instances on our Hadoop cluster, with different users (basically, one HTTPFS per team).

In our setup, each HTTPFS instance runs as a team user and is allowed write access to that user's directory only (so, HTTPFS does not run as the httpfs user).

However, this setup does not work, as we get exceptions related to impersonation, such as this one:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}

So, it seems that HTTPFS unconditionally tries to impersonate a user, even though it's running as that same user. Is there a way to somehow disable impersonation?

Thanks for your help.

Regards,
Nathaniel

RE: HTTPFS without impersonation

[Nathaniel Braun <n....@criteo.com>]

Hi,

We want to let users & teams be able to run their HTTPFS in order to isolate instances. One team thus cannot crash another team's HTTPFS instance.

Now, I make the following request:

curl "localhost:14000/webhdfs/v1/user/team_user?op=LISTSTATUS&user.name=team_user"

And I get the following response:

{"RemoteException":{"message":"User: team_user is not allowed to impersonate team_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}

Thanks,
Nathaniel

From: Larry McCay [mailto:lmccay@hortonworks.com]
Sent: mercredi 3 juin 2015 13:57
To: user@hadoop.apache.org
Subject: Re: HTTPFS without impersonation

Out of curiosity, what is the added benefit of having HttpFs run as separate team users give you?
If the APIs are invoked with SPNEGO or a user.name of the appropriate user don't you get the same permissions based protections?

Generally speaking, gateways such as HttpFs provide access on behalf of endusers.

On Jun 3, 2015, at 7:44 AM, Nathaniel Braun <n....@criteo.com>> wrote:


Hi,

Thanks for your answer.

With this setup, only the HTTP user will be able to impersonate other users, so HTTPFS has to run with the HTTP user.

Instead, I need users to run HTTPFS with their own user, not with the HTTP user.

Thanks

From: Wellington Chevreuil [mailto:wellington.chevreuil@gmail.com]
Sent: mercredi 3 juin 2015 13:41
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation


Hi, do u have below property on core-site.xml file used by your hdfs?

<property>
    <name>hadoop.proxyuser.HTTP.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.HTTP.groups</name>
    <value>*</value>
  </property>
Hello all,

We need to run several HTTPFS instances on our Hadoop cluster, with different users (basically, one HTTPFS per team).

In our setup, each HTTPFS instance runs as a team user and is allowed write access to that user's directory only (so, HTTPFS does not run as the httpfs user).

However, this setup does not work, as we get exceptions related to impersonation, such as this one:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}

So, it seems that HTTPFS unconditionally tries to impersonate a user, even though it's running as that same user. Is there a way to somehow disable impersonation?

Thanks for your help.

Regards,
Nathaniel

Re: HTTPFS without impersonation

[Larry McCay <lm...@hortonworks.com>]

Out of curiosity, what is the added benefit of having HttpFs run as separate team users give you?
If the APIs are invoked with SPNEGO or a user.name of the appropriate user don’t you get the same permissions based protections?

Generally speaking, gateways such as HttpFs provide access on behalf of endusers.

On Jun 3, 2015, at 7:44 AM, Nathaniel Braun <n....@criteo.com>> wrote:

Hi,

Thanks for your answer.

With this setup, only the HTTP user will be able to impersonate other users, so HTTPFS has to run with the HTTP user.

Instead, I need users to run HTTPFS with their own user, not with the HTTP user.

Thanks

From: Wellington Chevreuil [mailto:wellington.chevreuil@gmail.com]
Sent: mercredi 3 juin 2015 13:41
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation


Hi, do u have below property on core-site.xml file used by your hdfs?

<property>
    <name>hadoop.proxyuser.HTTP.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.HTTP.groups</name>
    <value>*</value>
  </property>

Hello all,

We need to run several HTTPFS instances on our Hadoop cluster, with different users (basically, one HTTPFS per team).

In our setup, each HTTPFS instance runs as a team user and is allowed write access to that user’s directory only (so, HTTPFS does not run as the httpfs user).

However, this setup does not work, as we get exceptions related to impersonation, such as this one:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}

So, it seems that HTTPFS unconditionally tries to impersonate a user, even though it’s running as that same user. Is there a way to somehow disable impersonation?

Thanks for your help.

Regards,
Nathaniel

Re: HTTPFS without impersonation

[Larry McCay <lm...@hortonworks.com>]

Out of curiosity, what is the added benefit of having HttpFs run as separate team users give you?
If the APIs are invoked with SPNEGO or a user.name of the appropriate user don’t you get the same permissions based protections?

Generally speaking, gateways such as HttpFs provide access on behalf of endusers.

On Jun 3, 2015, at 7:44 AM, Nathaniel Braun <n....@criteo.com>> wrote:

Hi,

Thanks for your answer.

With this setup, only the HTTP user will be able to impersonate other users, so HTTPFS has to run with the HTTP user.

Instead, I need users to run HTTPFS with their own user, not with the HTTP user.

Thanks

From: Wellington Chevreuil [mailto:wellington.chevreuil@gmail.com]
Sent: mercredi 3 juin 2015 13:41
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation


Hi, do u have below property on core-site.xml file used by your hdfs?

<property>
    <name>hadoop.proxyuser.HTTP.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.HTTP.groups</name>
    <value>*</value>
  </property>

Hello all,

We need to run several HTTPFS instances on our Hadoop cluster, with different users (basically, one HTTPFS per team).

In our setup, each HTTPFS instance runs as a team user and is allowed write access to that user’s directory only (so, HTTPFS does not run as the httpfs user).

However, this setup does not work, as we get exceptions related to impersonation, such as this one:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}

So, it seems that HTTPFS unconditionally tries to impersonate a user, even though it’s running as that same user. Is there a way to somehow disable impersonation?

Thanks for your help.

Regards,
Nathaniel

Re: HTTPFS without impersonation

[Larry McCay <lm...@hortonworks.com>]

Out of curiosity, what is the added benefit of having HttpFs run as separate team users give you?
If the APIs are invoked with SPNEGO or a user.name of the appropriate user don’t you get the same permissions based protections?

Generally speaking, gateways such as HttpFs provide access on behalf of endusers.

On Jun 3, 2015, at 7:44 AM, Nathaniel Braun <n....@criteo.com>> wrote:

Hi,

Thanks for your answer.

With this setup, only the HTTP user will be able to impersonate other users, so HTTPFS has to run with the HTTP user.

Instead, I need users to run HTTPFS with their own user, not with the HTTP user.

Thanks

From: Wellington Chevreuil [mailto:wellington.chevreuil@gmail.com]
Sent: mercredi 3 juin 2015 13:41
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation


Hi, do u have below property on core-site.xml file used by your hdfs?

<property>
    <name>hadoop.proxyuser.HTTP.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.HTTP.groups</name>
    <value>*</value>
  </property>

Hello all,

We need to run several HTTPFS instances on our Hadoop cluster, with different users (basically, one HTTPFS per team).

In our setup, each HTTPFS instance runs as a team user and is allowed write access to that user’s directory only (so, HTTPFS does not run as the httpfs user).

However, this setup does not work, as we get exceptions related to impersonation, such as this one:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}

So, it seems that HTTPFS unconditionally tries to impersonate a user, even though it’s running as that same user. Is there a way to somehow disable impersonation?

Thanks for your help.

Regards,
Nathaniel

Re: HTTPFS without impersonation

[Larry McCay <lm...@hortonworks.com>]

Out of curiosity, what is the added benefit of having HttpFs run as separate team users give you?
If the APIs are invoked with SPNEGO or a user.name of the appropriate user don’t you get the same permissions based protections?

Generally speaking, gateways such as HttpFs provide access on behalf of endusers.

On Jun 3, 2015, at 7:44 AM, Nathaniel Braun <n....@criteo.com>> wrote:

Hi,

Thanks for your answer.

With this setup, only the HTTP user will be able to impersonate other users, so HTTPFS has to run with the HTTP user.

Instead, I need users to run HTTPFS with their own user, not with the HTTP user.

Thanks

From: Wellington Chevreuil [mailto:wellington.chevreuil@gmail.com]
Sent: mercredi 3 juin 2015 13:41
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HTTPFS without impersonation


Hi, do u have below property on core-site.xml file used by your hdfs?

<property>
    <name>hadoop.proxyuser.HTTP.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.HTTP.groups</name>
    <value>*</value>
  </property>

Hello all,

We need to run several HTTPFS instances on our Hadoop cluster, with different users (basically, one HTTPFS per team).

In our setup, each HTTPFS instance runs as a team user and is allowed write access to that user’s directory only (so, HTTPFS does not run as the httpfs user).

However, this setup does not work, as we get exceptions related to impersonation, such as this one:

{"RemoteException":{"message":"User: team_user is not allowed to impersonateteam_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}

So, it seems that HTTPFS unconditionally tries to impersonate a user, even though it’s running as that same user. Is there a way to somehow disable impersonation?

Thanks for your help.

Regards,
Nathaniel

RE: HTTPFS without impersonation

[Nathaniel Braun <n....@criteo.com>]

Hi,

Thanks for your answer.

With this setup, only the HTTP user will be able to impersonate other users, so HTTPFS has to run with the HTTP user.

Instead, I need users to run HTTPFS with their own user, not with the HTTP user.

Thanks

From: Wellington Chevreuil [mailto:wellington.chevreuil@gmail.com]
Sent: mercredi 3 juin 2015 13:41
To: user@hadoop.apache.org
Subject: Re: HTTPFS without impersonation


Hi, do u have below property on core-site.xml file used by your hdfs?

<property>
    <name>hadoop.proxyuser.HTTP.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.HTTP.groups</name>
    <value>*</value>
  </property>
Hello all,

We need to run several HTTPFS instances on our Hadoop cluster, with different users (basically, one HTTPFS per team).

In our setup, each HTTPFS instance runs as a team user and is allowed write access to that user’s directory only (so, HTTPFS does not run as the httpfs user).

However, this setup does not work, as we get exceptions related to impersonation, such as this one:

{"RemoteException":{"message":"User: team_user is not allowed to impersonate team_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}

So, it seems that HTTPFS unconditionally tries to impersonate a user, even though it’s running as that same user. Is there a way to somehow disable impersonation?

Thanks for your help.

Regards,
Nathaniel

Re: HTTPFS without impersonation

[Wellington Chevreuil <we...@gmail.com>]

If that doesn't work, u may need to define one entry for these properties
to each user running an httpfs instance.

See below:

http://hadoop.apache.org/docs/current/hadoop-hdfs-httpfs/ServerSetup.html
Em 03/06/2015 12:40, "Wellington Chevreuil" <we...@gmail.com>
escreveu:

> Hi, do u have below property on core-site.xml file used by your hdfs?
>
> <property>
>     <name>hadoop.proxyuser.HTTP.hosts</name>
>     <value>*</value>
>   </property>
>   <property>
>     <name>hadoop.proxyuser.HTTP.groups</name>
>     <value>*</value>
>   </property>
>
> Hello all,
>
>
>
> We need to run several HTTPFS instances on our Hadoop cluster, with
> different users (basically, one HTTPFS per team).
>
>
>
> In our setup, each HTTPFS instance runs as a team user and is allowed
> write access to that user’s directory only (so, HTTPFS does not run as the
> httpfs user).
>
>
>
> However, this setup does not work, as we get exceptions related to
> impersonation, such as this one:
>
>
>
> {"RemoteException":{"message":"User: *team_user* is not allowed to
> impersonate *team_user*
> ","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}
>
>
>
> *So, it seems that HTTPFS unconditionally tries to impersonate a user,
> even though it’s running as that same user*. Is there a way to somehow
> disable impersonation?
>
>
>
> Thanks for your help.
>
>
>
> Regards,
>
> Nathaniel
>
>
>

Re: HTTPFS without impersonation

[Wellington Chevreuil <we...@gmail.com>]

If that doesn't work, u may need to define one entry for these properties
to each user running an httpfs instance.

See below:

http://hadoop.apache.org/docs/current/hadoop-hdfs-httpfs/ServerSetup.html
Em 03/06/2015 12:40, "Wellington Chevreuil" <we...@gmail.com>
escreveu:

> Hi, do u have below property on core-site.xml file used by your hdfs?
>
> <property>
>     <name>hadoop.proxyuser.HTTP.hosts</name>
>     <value>*</value>
>   </property>
>   <property>
>     <name>hadoop.proxyuser.HTTP.groups</name>
>     <value>*</value>
>   </property>
>
> Hello all,
>
>
>
> We need to run several HTTPFS instances on our Hadoop cluster, with
> different users (basically, one HTTPFS per team).
>
>
>
> In our setup, each HTTPFS instance runs as a team user and is allowed
> write access to that user’s directory only (so, HTTPFS does not run as the
> httpfs user).
>
>
>
> However, this setup does not work, as we get exceptions related to
> impersonation, such as this one:
>
>
>
> {"RemoteException":{"message":"User: *team_user* is not allowed to
> impersonate *team_user*
> ","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}
>
>
>
> *So, it seems that HTTPFS unconditionally tries to impersonate a user,
> even though it’s running as that same user*. Is there a way to somehow
> disable impersonation?
>
>
>
> Thanks for your help.
>
>
>
> Regards,
>
> Nathaniel
>
>
>

RE: HTTPFS without impersonation

[Nathaniel Braun <n....@criteo.com>]

Hi,

Thanks for your answer.

With this setup, only the HTTP user will be able to impersonate other users, so HTTPFS has to run with the HTTP user.

Instead, I need users to run HTTPFS with their own user, not with the HTTP user.

Thanks

From: Wellington Chevreuil [mailto:wellington.chevreuil@gmail.com]
Sent: mercredi 3 juin 2015 13:41
To: user@hadoop.apache.org
Subject: Re: HTTPFS without impersonation


Hi, do u have below property on core-site.xml file used by your hdfs?

<property>
    <name>hadoop.proxyuser.HTTP.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.HTTP.groups</name>
    <value>*</value>
  </property>
Hello all,

We need to run several HTTPFS instances on our Hadoop cluster, with different users (basically, one HTTPFS per team).

In our setup, each HTTPFS instance runs as a team user and is allowed write access to that user’s directory only (so, HTTPFS does not run as the httpfs user).

However, this setup does not work, as we get exceptions related to impersonation, such as this one:

{"RemoteException":{"message":"User: team_user is not allowed to impersonate team_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}

So, it seems that HTTPFS unconditionally tries to impersonate a user, even though it’s running as that same user. Is there a way to somehow disable impersonation?

Thanks for your help.

Regards,
Nathaniel

RE: HTTPFS without impersonation

[Nathaniel Braun <n....@criteo.com>]

Hi,

Thanks for your answer.

With this setup, only the HTTP user will be able to impersonate other users, so HTTPFS has to run with the HTTP user.

Instead, I need users to run HTTPFS with their own user, not with the HTTP user.

Thanks

From: Wellington Chevreuil [mailto:wellington.chevreuil@gmail.com]
Sent: mercredi 3 juin 2015 13:41
To: user@hadoop.apache.org
Subject: Re: HTTPFS without impersonation


Hi, do u have below property on core-site.xml file used by your hdfs?

<property>
    <name>hadoop.proxyuser.HTTP.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.HTTP.groups</name>
    <value>*</value>
  </property>
Hello all,

We need to run several HTTPFS instances on our Hadoop cluster, with different users (basically, one HTTPFS per team).

In our setup, each HTTPFS instance runs as a team user and is allowed write access to that user’s directory only (so, HTTPFS does not run as the httpfs user).

However, this setup does not work, as we get exceptions related to impersonation, such as this one:

{"RemoteException":{"message":"User: team_user is not allowed to impersonate team_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}

So, it seems that HTTPFS unconditionally tries to impersonate a user, even though it’s running as that same user. Is there a way to somehow disable impersonation?

Thanks for your help.

Regards,
Nathaniel

RE: HTTPFS without impersonation

[Nathaniel Braun <n....@criteo.com>]

Hi,

Thanks for your answer.

With this setup, only the HTTP user will be able to impersonate other users, so HTTPFS has to run with the HTTP user.

Instead, I need users to run HTTPFS with their own user, not with the HTTP user.

Thanks

From: Wellington Chevreuil [mailto:wellington.chevreuil@gmail.com]
Sent: mercredi 3 juin 2015 13:41
To: user@hadoop.apache.org
Subject: Re: HTTPFS without impersonation


Hi, do u have below property on core-site.xml file used by your hdfs?

<property>
    <name>hadoop.proxyuser.HTTP.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.HTTP.groups</name>
    <value>*</value>
  </property>
Hello all,

We need to run several HTTPFS instances on our Hadoop cluster, with different users (basically, one HTTPFS per team).

In our setup, each HTTPFS instance runs as a team user and is allowed write access to that user’s directory only (so, HTTPFS does not run as the httpfs user).

However, this setup does not work, as we get exceptions related to impersonation, such as this one:

{"RemoteException":{"message":"User: team_user is not allowed to impersonate team_user","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}

So, it seems that HTTPFS unconditionally tries to impersonate a user, even though it’s running as that same user. Is there a way to somehow disable impersonation?

Thanks for your help.

Regards,
Nathaniel

Re: HTTPFS without impersonation

[Wellington Chevreuil <we...@gmail.com>]

Hi, do u have below property on core-site.xml file used by your hdfs?

<property>
    <name>hadoop.proxyuser.HTTP.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.HTTP.groups</name>
    <value>*</value>
  </property>

Hello all,



We need to run several HTTPFS instances on our Hadoop cluster, with
different users (basically, one HTTPFS per team).



In our setup, each HTTPFS instance runs as a team user and is allowed write
access to that user’s directory only (so, HTTPFS does not run as the httpfs
user).



However, this setup does not work, as we get exceptions related to
impersonation, such as this one:



{"RemoteException":{"message":"User: *team_user* is not allowed to
impersonate *team_user*
","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}



*So, it seems that HTTPFS unconditionally tries to impersonate a user, even
though it’s running as that same user*. Is there a way to somehow disable
impersonation?



Thanks for your help.



Regards,

Nathaniel

Re: HTTPFS without impersonation

[Wellington Chevreuil <we...@gmail.com>]

Hi, do u have below property on core-site.xml file used by your hdfs?

<property>
    <name>hadoop.proxyuser.HTTP.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.HTTP.groups</name>
    <value>*</value>
  </property>

Hello all,



We need to run several HTTPFS instances on our Hadoop cluster, with
different users (basically, one HTTPFS per team).



In our setup, each HTTPFS instance runs as a team user and is allowed write
access to that user’s directory only (so, HTTPFS does not run as the httpfs
user).



However, this setup does not work, as we get exceptions related to
impersonation, such as this one:



{"RemoteException":{"message":"User: *team_user* is not allowed to
impersonate *team_user*
","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}



*So, it seems that HTTPFS unconditionally tries to impersonate a user, even
though it’s running as that same user*. Is there a way to somehow disable
impersonation?



Thanks for your help.



Regards,

Nathaniel

Re: HTTPFS without impersonation

[Wellington Chevreuil <we...@gmail.com>]

Hi, do u have below property on core-site.xml file used by your hdfs?

<property>
    <name>hadoop.proxyuser.HTTP.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.HTTP.groups</name>
    <value>*</value>
  </property>

Hello all,



We need to run several HTTPFS instances on our Hadoop cluster, with
different users (basically, one HTTPFS per team).



In our setup, each HTTPFS instance runs as a team user and is allowed write
access to that user’s directory only (so, HTTPFS does not run as the httpfs
user).



However, this setup does not work, as we get exceptions related to
impersonation, such as this one:



{"RemoteException":{"message":"User: *team_user* is not allowed to
impersonate *team_user*
","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}



*So, it seems that HTTPFS unconditionally tries to impersonate a user, even
though it’s running as that same user*. Is there a way to somehow disable
impersonation?



Thanks for your help.



Regards,

Nathaniel

Re: HTTPFS without impersonation

[Wellington Chevreuil <we...@gmail.com>]

Hi, do u have below property on core-site.xml file used by your hdfs?

<property>
    <name>hadoop.proxyuser.HTTP.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.HTTP.groups</name>
    <value>*</value>
  </property>

Hello all,



We need to run several HTTPFS instances on our Hadoop cluster, with
different users (basically, one HTTPFS per team).



In our setup, each HTTPFS instance runs as a team user and is allowed write
access to that user’s directory only (so, HTTPFS does not run as the httpfs
user).



However, this setup does not work, as we get exceptions related to
impersonation, such as this one:



{"RemoteException":{"message":"User: *team_user* is not allowed to
impersonate *team_user*
","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}



*So, it seems that HTTPFS unconditionally tries to impersonate a user, even
though it’s running as that same user*. Is there a way to somehow disable
impersonation?



Thanks for your help.



Regards,

Nathaniel