You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Jay Vyas <ja...@gmail.com> on 2013/04/10 19:16:35 UTC
testHDFSConf.xml
Hello HDFS brethren !
I've noticed that the testHDFSConf.xml has alot of references to
supergroup.
https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
1) I wonder why this is hardcoded in the testHDFSConf.xml
2) Also, Im wondering if there are any good ideas for extending/modifying
this file for a extention of the FileSystem implementation.
Right now im doing some global find replace statements - but was thinking
that maybe parameterizing the file would be a good JIRA - so that people
could use this as a base test for FileSystem implementations
Depending on feedback im certainly willing to submit and put in a first
pass at a more modular version of this file.
Its in many ways a very generalizable component of the hdfs trunk.
Thanks!
--
Jay Vyas
http://jayunit100.blogspot.com
Re: testHDFSConf.xml
Posted by Konstantin Boudnik <co...@apache.org>.
The test cases weren't split - there's only testHDFS suite. However, the
drivers available for common and mapreduce, so all you need to do is to add
your tests cases a-la hdfs.
Cos
On Thu, Apr 11, 2013 at 09:14AM, Jay Vyas wrote:
> Thats great. Yes this is certainly something we could benefit from.
>
> But the "trunk" github appears to only have
> "./src/test/hdfs/org/apache/hadoop/cli/testHDFSConf.xml" in the source tree.
>
> Also, I don't see it in branch-0.22:
>
> # git checkout branch-0.22
> Branch branch-0.22 set up to track remote branch branch-0.22 from origin.
> Switched to a new branch 'branch-0.22'
> # find ./ -name *Conf.xml
> ./src/test/hdfs/org/apache/hadoop/cli/testHDFSConf.xml
>
>
> Maybe I'm missing something?
>
>
> On Thu, Apr 11, 2013 at 1:28 AM, Konstantin Boudnik <co...@apache.org> wrote:
>
> > I have split CLI test infrastructure into hierarchical pieces that allow to
> > have different configurations for different components. E.g. you can have
> > one
> > for YARN that would exist independently of HDFS, etc. The change has been
> > in
> > since like 0.22 and committed to 0.203.x as well IIRC, hence should be
> > usable
> > across the Hadoop versions. May be it something you would benefit from.
> >
> > Cos
> >
> > On Wed, Apr 10, 2013 at 10:43AM, Colin McCabe wrote:
> > > On Wed, Apr 10, 2013 at 10:16 AM, Jay Vyas <ja...@gmail.com> wrote:
> > >
> > > > Hello HDFS brethren !
> > > >
> > > > I've noticed that the testHDFSConf.xml has alot of references to
> > > > supergroup.
> > > >
> > > >
> > > >
> > https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
> > > >
> > > > 1) I wonder why this is hardcoded in the testHDFSConf.xml
> > > >
> > > >
> > > "supergroup" is the default supergroup in HDFS. Check
> > DFSConfigKeys.java:
> > >
> > > public static final String DFS_PERMISSIONS_SUPERUSERGROUP_KEY =
> > > "dfs.permissions.superusergroup";
> > > public static final String DFS_PERMISSIONS_SUPERUSERGROUP_DEFAULT =
> > > "supergroup";
> > >
> > > It seems fine to use "supergroup" in a test. after all, we do control
> > the
> > > configuration we pass into the test.
> > >
> > >
> > > > 2) Also, Im wondering if there are any good ideas for
> > extending/modifying
> > > > this file for a extention of the FileSystem implementation.
> > > >
> > > >
> > > It would be interesting to think about pulling the non-hdfs-specific
> > > components of TestHDFSCLI into another test; perhaps one in common.
> > > Theoretically, what we print on the console should be really similar, no
> > > matter whether HDFS or some other filesystem is being used. In practice,
> > > there may be some differences, however...
> > >
> > > I find it a little bit challenging to modify TestHDFSCLI because the test
> > > is really long and executes as a single unit. Breaking it down into
> > > multiple units would probably be another good improvement, at least in my
> > > opinion.
> > >
> > > best,
> > > Colin
> > >
> > >
> > > Right now im doing some global find replace statements - but was
> > thinking
> > > > that maybe parameterizing the file would be a good JIRA - so that
> > people
> > > > could use this as a base test for FileSystem implementations
> > > >
> > > > Depending on feedback im certainly willing to submit and put in a first
> > > > pass at a more modular version of this file.
> > > >
> > > > Its in many ways a very generalizable component of the hdfs trunk.
> > > >
> > > > Thanks!
> > > > --
> > > > Jay Vyas
> > > > http://jayunit100.blogspot.com
> > > >
> >
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
Re: testHDFSConf.xml
Posted by Jay Vyas <ja...@gmail.com>.
Thats great. Yes this is certainly something we could benefit from.
But the "trunk" github appears to only have
"./src/test/hdfs/org/apache/hadoop/cli/testHDFSConf.xml" in the source tree.
Also, I don't see it in branch-0.22:
# git checkout branch-0.22
Branch branch-0.22 set up to track remote branch branch-0.22 from origin.
Switched to a new branch 'branch-0.22'
# find ./ -name *Conf.xml
./src/test/hdfs/org/apache/hadoop/cli/testHDFSConf.xml
Maybe I'm missing something?
On Thu, Apr 11, 2013 at 1:28 AM, Konstantin Boudnik <co...@apache.org> wrote:
> I have split CLI test infrastructure into hierarchical pieces that allow to
> have different configurations for different components. E.g. you can have
> one
> for YARN that would exist independently of HDFS, etc. The change has been
> in
> since like 0.22 and committed to 0.203.x as well IIRC, hence should be
> usable
> across the Hadoop versions. May be it something you would benefit from.
>
> Cos
>
> On Wed, Apr 10, 2013 at 10:43AM, Colin McCabe wrote:
> > On Wed, Apr 10, 2013 at 10:16 AM, Jay Vyas <ja...@gmail.com> wrote:
> >
> > > Hello HDFS brethren !
> > >
> > > I've noticed that the testHDFSConf.xml has alot of references to
> > > supergroup.
> > >
> > >
> > >
> https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
> > >
> > > 1) I wonder why this is hardcoded in the testHDFSConf.xml
> > >
> > >
> > "supergroup" is the default supergroup in HDFS. Check
> DFSConfigKeys.java:
> >
> > public static final String DFS_PERMISSIONS_SUPERUSERGROUP_KEY =
> > "dfs.permissions.superusergroup";
> > public static final String DFS_PERMISSIONS_SUPERUSERGROUP_DEFAULT =
> > "supergroup";
> >
> > It seems fine to use "supergroup" in a test. after all, we do control
> the
> > configuration we pass into the test.
> >
> >
> > > 2) Also, Im wondering if there are any good ideas for
> extending/modifying
> > > this file for a extention of the FileSystem implementation.
> > >
> > >
> > It would be interesting to think about pulling the non-hdfs-specific
> > components of TestHDFSCLI into another test; perhaps one in common.
> > Theoretically, what we print on the console should be really similar, no
> > matter whether HDFS or some other filesystem is being used. In practice,
> > there may be some differences, however...
> >
> > I find it a little bit challenging to modify TestHDFSCLI because the test
> > is really long and executes as a single unit. Breaking it down into
> > multiple units would probably be another good improvement, at least in my
> > opinion.
> >
> > best,
> > Colin
> >
> >
> > Right now im doing some global find replace statements - but was
> thinking
> > > that maybe parameterizing the file would be a good JIRA - so that
> people
> > > could use this as a base test for FileSystem implementations
> > >
> > > Depending on feedback im certainly willing to submit and put in a first
> > > pass at a more modular version of this file.
> > >
> > > Its in many ways a very generalizable component of the hdfs trunk.
> > >
> > > Thanks!
> > > --
> > > Jay Vyas
> > > http://jayunit100.blogspot.com
> > >
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: testHDFSConf.xml
Posted by Konstantin Boudnik <co...@apache.org>.
I have split CLI test infrastructure into hierarchical pieces that allow to
have different configurations for different components. E.g. you can have one
for YARN that would exist independently of HDFS, etc. The change has been in
since like 0.22 and committed to 0.203.x as well IIRC, hence should be usable
across the Hadoop versions. May be it something you would benefit from.
Cos
On Wed, Apr 10, 2013 at 10:43AM, Colin McCabe wrote:
> On Wed, Apr 10, 2013 at 10:16 AM, Jay Vyas <ja...@gmail.com> wrote:
>
> > Hello HDFS brethren !
> >
> > I've noticed that the testHDFSConf.xml has alot of references to
> > supergroup.
> >
> >
> > https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
> >
> > 1) I wonder why this is hardcoded in the testHDFSConf.xml
> >
> >
> "supergroup" is the default supergroup in HDFS. Check DFSConfigKeys.java:
>
> public static final String DFS_PERMISSIONS_SUPERUSERGROUP_KEY =
> "dfs.permissions.superusergroup";
> public static final String DFS_PERMISSIONS_SUPERUSERGROUP_DEFAULT =
> "supergroup";
>
> It seems fine to use "supergroup" in a test. after all, we do control the
> configuration we pass into the test.
>
>
> > 2) Also, Im wondering if there are any good ideas for extending/modifying
> > this file for a extention of the FileSystem implementation.
> >
> >
> It would be interesting to think about pulling the non-hdfs-specific
> components of TestHDFSCLI into another test; perhaps one in common.
> Theoretically, what we print on the console should be really similar, no
> matter whether HDFS or some other filesystem is being used. In practice,
> there may be some differences, however...
>
> I find it a little bit challenging to modify TestHDFSCLI because the test
> is really long and executes as a single unit. Breaking it down into
> multiple units would probably be another good improvement, at least in my
> opinion.
>
> best,
> Colin
>
>
> Right now im doing some global find replace statements - but was thinking
> > that maybe parameterizing the file would be a good JIRA - so that people
> > could use this as a base test for FileSystem implementations
> >
> > Depending on feedback im certainly willing to submit and put in a first
> > pass at a more modular version of this file.
> >
> > Its in many ways a very generalizable component of the hdfs trunk.
> >
> > Thanks!
> > --
> > Jay Vyas
> > http://jayunit100.blogspot.com
> >
Re: testHDFSConf.xml
Posted by Colin McCabe <cm...@alumni.cmu.edu>.
On Wed, Apr 10, 2013 at 10:16 AM, Jay Vyas <ja...@gmail.com> wrote:
> Hello HDFS brethren !
>
> I've noticed that the testHDFSConf.xml has alot of references to
> supergroup.
>
>
> https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
>
> 1) I wonder why this is hardcoded in the testHDFSConf.xml
>
>
"supergroup" is the default supergroup in HDFS. Check DFSConfigKeys.java:
public static final String DFS_PERMISSIONS_SUPERUSERGROUP_KEY =
"dfs.permissions.superusergroup";
public static final String DFS_PERMISSIONS_SUPERUSERGROUP_DEFAULT =
"supergroup";
It seems fine to use "supergroup" in a test. after all, we do control the
configuration we pass into the test.
> 2) Also, Im wondering if there are any good ideas for extending/modifying
> this file for a extention of the FileSystem implementation.
>
>
It would be interesting to think about pulling the non-hdfs-specific
components of TestHDFSCLI into another test; perhaps one in common.
Theoretically, what we print on the console should be really similar, no
matter whether HDFS or some other filesystem is being used. In practice,
there may be some differences, however...
I find it a little bit challenging to modify TestHDFSCLI because the test
is really long and executes as a single unit. Breaking it down into
multiple units would probably be another good improvement, at least in my
opinion.
best,
Colin
Right now im doing some global find replace statements - but was thinking
> that maybe parameterizing the file would be a good JIRA - so that people
> could use this as a base test for FileSystem implementations
>
> Depending on feedback im certainly willing to submit and put in a first
> pass at a more modular version of this file.
>
> Its in many ways a very generalizable component of the hdfs trunk.
>
> Thanks!
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>