Need a way to run TestCLI under non-hdfs users

[Dasha Boudnik <da...@wandisco.com>]

Hi all,

Currently, TestCLI (and TestDFSCLI and TestHDFSCLI) fails because it needs
to be run under user hdfs. Since this isn't the case for the rest of the
hadoop-smokes, it seems like ideally we'd have a way to run *all* the tests
under the same user by making the CLIs runnable under non-hdfs users.
Unfortunately, it also seems like this is very difficult to do.

Does anyone happen to have any suggestions? Would be greatly appreciated!

Best,

-- 
*DASHA BOUDNIK* ENGINEERING INTERN

*E* dasha@wandisco.com

*www.wandisco.com <http://www.wandisco.com/>*

-- 


5 reasons your Hadoop needs WANdisco 
<http://www.wandisco.com/system/files/documentation/5-Reasons.pdf>

Listed on the London Stock Exchange: WAND 
<http://www.bloomberg.com/quote/WAND:LN>

THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE 
PRIVILEGED.  If this message was misdirected, WANdisco, Inc. and its 
subsidiaries, ("WANdisco") does not waive any confidentiality or privilege. 
 If you are not the intended recipient, please notify us immediately and 
destroy the message without disclosing its contents to anyone.  Any 
distribution, use or copying of this e-mail or the information it contains 
by other than an intended recipient is unauthorized.  The views and 
opinions expressed in this e-mail message are the author's own and may not 
reflect the views and opinions of WANdisco, unless the author is authorized 
by WANdisco to express such views or opinions on its behalf.  All email 
sent to or from this address is subject to electronic storage and review by 
WANdisco.  Although WANdisco operates anti-virus programs, it does not 
accept responsibility for any damage whatsoever caused by viruses being 
passed.

Re: Need a way to run TestCLI under non-hdfs users

[jay vyas <ja...@gmail.com>]

Good idea to run non-elevated user tests.  this what most of the tests
contents are concerned with iirc.


On Tue, Mar 10, 2015 at 8:12 PM, Konstantin Boudnik <co...@apache.org> wrote:

> Good point Mark. And that exactly what CLI tests - or at least most of
> them do
> not require - a super user privilege. User hdfs is needed to only perform
> super-user ops like put stuff under '/'; do dfsadmin actions, etc. Doing
> everything as hdfs is a big no-no in the real world.
>
> And I believe issue at stake is more significant, namely: running different
> tests under different uid creates a nightmare for test automation people.
> Besides of dealing with diff. OSes, JDKs, and so on now they need to track
> which tests are running under what users. In other words - the executing
> system all of a sudden gets too much knowledge about the innards of the
> tests.
> Violation of 'separation of concerns' is always a very bad idea.
>
> The correct approach would be to run the tests under non-elevated user
> (bigtop, jenkins) that has an ability to do chuser or similar op. at will.
> Then the test driver process will carry the knowledge or a matrix of
> relations
> between the tests and the users. I agree with Dasha here - HDFS tests are
> becoming a mess and that needs to be fixed.
>
> Cos
>
> On Tue, Mar 10, 2015 at 04:39PM, Mark Grover wrote:
> > Good question, Dasha.
> >
> > I am curious to hear your thoughts on why all tests should be run under
> the
> > same user. I am of the opinion that our tests should mimic how most
> people
> > use our software. And, for that, they use 'hive' user for running hive
> > queries, 'hdfs' user for writing to hdfs, etc. I'd personally like our
> > tests to follow the same, if it's not too much work.
> >
> > On Mon, Mar 9, 2015 at 6:18 PM, Dasha Boudnik <da...@wandisco.com>
> wrote:
> >
> > > Hi all,
> > >
> > > Currently, TestCLI (and TestDFSCLI and TestHDFSCLI) fails because it
> needs
> > > to be run under user hdfs. Since this isn't the case for the rest of
> the
> > > hadoop-smokes, it seems like ideally we'd have a way to run *all* the
> tests
> > > under the same user by making the CLIs runnable under non-hdfs users.
> > > Unfortunately, it also seems like this is very difficult to do.
> > >
> > > Does anyone happen to have any suggestions? Would be greatly
> appreciated!
> > >
> > > Best,
> > >
> > > --
> > > *DASHA BOUDNIK* ENGINEERING INTERN
> > >
> > > *E* dasha@wandisco.com
> > >
> > > *www.wandisco.com <http://www.wandisco.com/>*
> > >
> > > --
> > >
> > >
> > > 5 reasons your Hadoop needs WANdisco
> > > <http://www.wandisco.com/system/files/documentation/5-Reasons.pdf>
> > >
> > > Listed on the London Stock Exchange: WAND
> > > <http://www.bloomberg.com/quote/WAND:LN>
> > >
> > > THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND
> MAY BE
> > > PRIVILEGED.  If this message was misdirected, WANdisco, Inc. and its
> > > subsidiaries, ("WANdisco") does not waive any confidentiality or
> privilege.
> > >  If you are not the intended recipient, please notify us immediately
> and
> > > destroy the message without disclosing its contents to anyone.  Any
> > > distribution, use or copying of this e-mail or the information it
> contains
> > > by other than an intended recipient is unauthorized.  The views and
> > > opinions expressed in this e-mail message are the author's own and may
> not
> > > reflect the views and opinions of WANdisco, unless the author is
> authorized
> > > by WANdisco to express such views or opinions on its behalf.  All email
> > > sent to or from this address is subject to electronic storage and
> review by
> > > WANdisco.  Although WANdisco operates anti-virus programs, it does not
> > > accept responsibility for any damage whatsoever caused by viruses being
> > > passed.
> > >
>



-- 
jay vyas

Re: Need a way to run TestCLI under non-hdfs users

[Konstantin Boudnik <co...@apache.org>]

Good point Mark. And that exactly what CLI tests - or at least most of them do
not require - a super user privilege. User hdfs is needed to only perform
super-user ops like put stuff under '/'; do dfsadmin actions, etc. Doing
everything as hdfs is a big no-no in the real world.

And I believe issue at stake is more significant, namely: running different
tests under different uid creates a nightmare for test automation people.
Besides of dealing with diff. OSes, JDKs, and so on now they need to track
which tests are running under what users. In other words - the executing
system all of a sudden gets too much knowledge about the innards of the tests.
Violation of 'separation of concerns' is always a very bad idea.

The correct approach would be to run the tests under non-elevated user
(bigtop, jenkins) that has an ability to do chuser or similar op. at will.
Then the test driver process will carry the knowledge or a matrix of relations
between the tests and the users. I agree with Dasha here - HDFS tests are
becoming a mess and that needs to be fixed.

Cos

On Tue, Mar 10, 2015 at 04:39PM, Mark Grover wrote:
> Good question, Dasha.
> 
> I am curious to hear your thoughts on why all tests should be run under the
> same user. I am of the opinion that our tests should mimic how most people
> use our software. And, for that, they use 'hive' user for running hive
> queries, 'hdfs' user for writing to hdfs, etc. I'd personally like our
> tests to follow the same, if it's not too much work.
> 
> On Mon, Mar 9, 2015 at 6:18 PM, Dasha Boudnik <da...@wandisco.com> wrote:
> 
> > Hi all,
> >
> > Currently, TestCLI (and TestDFSCLI and TestHDFSCLI) fails because it needs
> > to be run under user hdfs. Since this isn't the case for the rest of the
> > hadoop-smokes, it seems like ideally we'd have a way to run *all* the tests
> > under the same user by making the CLIs runnable under non-hdfs users.
> > Unfortunately, it also seems like this is very difficult to do.
> >
> > Does anyone happen to have any suggestions? Would be greatly appreciated!
> >
> > Best,
> >
> > --
> > *DASHA BOUDNIK* ENGINEERING INTERN
> >
> > *E* dasha@wandisco.com
> >
> > *www.wandisco.com <http://www.wandisco.com/>*
> >
> > --
> >
> >
> > 5 reasons your Hadoop needs WANdisco
> > <http://www.wandisco.com/system/files/documentation/5-Reasons.pdf>
> >
> > Listed on the London Stock Exchange: WAND
> > <http://www.bloomberg.com/quote/WAND:LN>
> >
> > THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE
> > PRIVILEGED.  If this message was misdirected, WANdisco, Inc. and its
> > subsidiaries, ("WANdisco") does not waive any confidentiality or privilege.
> >  If you are not the intended recipient, please notify us immediately and
> > destroy the message without disclosing its contents to anyone.  Any
> > distribution, use or copying of this e-mail or the information it contains
> > by other than an intended recipient is unauthorized.  The views and
> > opinions expressed in this e-mail message are the author's own and may not
> > reflect the views and opinions of WANdisco, unless the author is authorized
> > by WANdisco to express such views or opinions on its behalf.  All email
> > sent to or from this address is subject to electronic storage and review by
> > WANdisco.  Although WANdisco operates anti-virus programs, it does not
> > accept responsibility for any damage whatsoever caused by viruses being
> > passed.
> >

Re: Need a way to run TestCLI under non-hdfs users

[Mark Grover <ma...@apache.org>]

Good question, Dasha.

I am curious to hear your thoughts on why all tests should be run under the
same user. I am of the opinion that our tests should mimic how most people
use our software. And, for that, they use 'hive' user for running hive
queries, 'hdfs' user for writing to hdfs, etc. I'd personally like our
tests to follow the same, if it's not too much work.

On Mon, Mar 9, 2015 at 6:18 PM, Dasha Boudnik <da...@wandisco.com> wrote:

> Hi all,
>
> Currently, TestCLI (and TestDFSCLI and TestHDFSCLI) fails because it needs
> to be run under user hdfs. Since this isn't the case for the rest of the
> hadoop-smokes, it seems like ideally we'd have a way to run *all* the tests
> under the same user by making the CLIs runnable under non-hdfs users.
> Unfortunately, it also seems like this is very difficult to do.
>
> Does anyone happen to have any suggestions? Would be greatly appreciated!
>
> Best,
>
> --
> *DASHA BOUDNIK* ENGINEERING INTERN
>
> *E* dasha@wandisco.com
>
> *www.wandisco.com <http://www.wandisco.com/>*
>
> --
>
>
> 5 reasons your Hadoop needs WANdisco
> <http://www.wandisco.com/system/files/documentation/5-Reasons.pdf>
>
> Listed on the London Stock Exchange: WAND
> <http://www.bloomberg.com/quote/WAND:LN>
>
> THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE
> PRIVILEGED.  If this message was misdirected, WANdisco, Inc. and its
> subsidiaries, ("WANdisco") does not waive any confidentiality or privilege.
>  If you are not the intended recipient, please notify us immediately and
> destroy the message without disclosing its contents to anyone.  Any
> distribution, use or copying of this e-mail or the information it contains
> by other than an intended recipient is unauthorized.  The views and
> opinions expressed in this e-mail message are the author's own and may not
> reflect the views and opinions of WANdisco, unless the author is authorized
> by WANdisco to express such views or opinions on its behalf.  All email
> sent to or from this address is subject to electronic storage and review by
> WANdisco.  Although WANdisco operates anti-virus programs, it does not
> accept responsibility for any damage whatsoever caused by viruses being
> passed.
>