You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@knox.apache.org by Kevin Minder <ke...@hortonworks.com> on 2016/01/05 20:59:35 UTC
Sample Hadoop test using HDFS and MiniKDC
Hey Everyone,
I ran across this test in Hadoop the other day.
https://github.com/apache/hadoop/blob/2f623fb8cc3dc49221216c3b46b6f51144811904/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestSecureNNWithQJM.java
I was thinking it could be the bases for a secure functional test. We are sorely missing those. My basic thought would be that this type of test would have a separate maven profile and would run as a separate jenkins job.
Kevin.
Re: Sample Hadoop test using HDFS and MiniKDC
Posted by Sumit Gupta <su...@hortonworks.com>.
Thanks Kevin,
Just to wrap up this thread here, the work for this is being done in
KNOX-651. There is some further discussion to be had in terms of potential
dev process changes with respect to running of various test suite types.
I’ll send a review request for this JIRA and we can discuss it there.
Sumit.
On 1/18/16, 1:56 PM, "Kevin Minder" <ke...@hortonworks.com> wrote:
>This is awesome.
>
>As far as project structure I was wondering if a bit of a deeper tree
>might be in order. I was thinking about this in particular because we
>might want to have separate suites for individual components and our top
>level module list is already to long. We might need individual per
>component integration suites to keep the resource requirements in check.
>
>Could something like this be made to work? It really comes down to what
>Maven might require.
>./gateway-test-integ/webhdfs-kerb
>./gateway-test-integ/webhdfs (possibly)
>./gateway-test-integ/hbase-kerb (eventually)
>
>Also note that I sort of tried to start a test partitioning mechanism for
>this already.
>Check this out:
>http://www.agile-engineering.net/2012/04/unit-and-integration-tests-with-m
>aven.html
>
>And take a look here:
>gateway-test-utils/src/main/java/org/apache/hadoop/test/category
>Not sure if we should add a KerberosTests category.
>
>If we go that route we might need to go back and add @Category
>annotations to tests that don’t have them.
>
>Right now the unit tests do this via the root pom.xml.
>
><excludedGroups>
>
>org.apache.hadoop.test.category.SlowTests,org.apache.hadoop.test.category.
>ManualTests,org.apache.hadoop.test.category.IntegrationTests
> </excludedGroups>
>
>
>
>
>On 1/18/16, 12:18 PM, "Sumit Gupta" <su...@hortonworks.com> wrote:
>
>>Hi everyone,
>>
>>I took a stab at getting the pieces together for a very basic knox test.
>>I
>>put it up here for a preview https://github.com/sumitg/knox-minikdc-test.
>>Please note that it will be cleaned up if we decide we want to put
>>something like this in knox project.
>>
>>The way I understand Kevin¹s thoughts, I would add a maven module
>>(something like Œgateway-test-secure¹ or Œgateway-test-integration¹?) and
>>a maven profile that triggers the test suite run, so that we can have a
>>separate job for this or run it locally by passing in the profile name.
>>
>>I will be happy to add a bunch of the webhdfs tests to start this off. I
>>imagine we¹ll iterate over this over time and keep adding more support
>>for
>>various services and the associated tests for them.
>>
>>Sumit.
>>
>>
>>On 1/5/16, 3:33 PM, "larry mccay" <la...@gmail.com> wrote:
>>
>>>+1 on adding secure tests!
>>>
>>>What functionality do we want to test as part of this suite initially?
>>>
>>>
>>>On Tue, Jan 5, 2016 at 2:59 PM, Kevin Minder
>>><ke...@hortonworks.com>
>>>wrote:
>>>
>>>> Hey Everyone,
>>>> I ran across this test in Hadoop the other day.
>>>>
>>>>
>>>>https://github.com/apache/hadoop/blob/2f623fb8cc3dc49221216c3b46b6f5114
>>>>48
>>>>11904/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/h
>>>>df
>>>>s/qjournal/TestSecureNNWithQJM.java
>>>> I was thinking it could be the bases for a secure functional test. We
>>>>are
>>>> sorely missing those. My basic thought would be that this type of
>>>>test
>>>> would have a separate maven profile and would run as a separate
>>>>jenkins
>>>>job.
>>>> Kevin.
>>>>
>>
>>
Re: Sample Hadoop test using HDFS and MiniKDC
Posted by Kevin Minder <ke...@hortonworks.com>.
This is awesome.
As far as project structure I was wondering if a bit of a deeper tree might be in order. I was thinking about this in particular because we might want to have separate suites for individual components and our top level module list is already to long. We might need individual per component integration suites to keep the resource requirements in check.
Could something like this be made to work? It really comes down to what Maven might require.
./gateway-test-integ/webhdfs-kerb
./gateway-test-integ/webhdfs (possibly)
./gateway-test-integ/hbase-kerb (eventually)
Also note that I sort of tried to start a test partitioning mechanism for this already.
Check this out:
http://www.agile-engineering.net/2012/04/unit-and-integration-tests-with-maven.html
And take a look here:
gateway-test-utils/src/main/java/org/apache/hadoop/test/category
Not sure if we should add a KerberosTests category.
If we go that route we might need to go back and add @Category annotations to tests that don’t have them.
Right now the unit tests do this via the root pom.xml.
<excludedGroups>
org.apache.hadoop.test.category.SlowTests,org.apache.hadoop.test.category.ManualTests,org.apache.hadoop.test.category.IntegrationTests
</excludedGroups>
On 1/18/16, 12:18 PM, "Sumit Gupta" <su...@hortonworks.com> wrote:
>Hi everyone,
>
>I took a stab at getting the pieces together for a very basic knox test. I
>put it up here for a preview https://github.com/sumitg/knox-minikdc-test.
>Please note that it will be cleaned up if we decide we want to put
>something like this in knox project.
>
>The way I understand Kevin¹s thoughts, I would add a maven module
>(something like Œgateway-test-secure¹ or Œgateway-test-integration¹?) and
>a maven profile that triggers the test suite run, so that we can have a
>separate job for this or run it locally by passing in the profile name.
>
>I will be happy to add a bunch of the webhdfs tests to start this off. I
>imagine we¹ll iterate over this over time and keep adding more support for
>various services and the associated tests for them.
>
>Sumit.
>
>
>On 1/5/16, 3:33 PM, "larry mccay" <la...@gmail.com> wrote:
>
>>+1 on adding secure tests!
>>
>>What functionality do we want to test as part of this suite initially?
>>
>>
>>On Tue, Jan 5, 2016 at 2:59 PM, Kevin Minder
>><ke...@hortonworks.com>
>>wrote:
>>
>>> Hey Everyone,
>>> I ran across this test in Hadoop the other day.
>>>
>>>
>>>https://github.com/apache/hadoop/blob/2f623fb8cc3dc49221216c3b46b6f511448
>>>11904/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdf
>>>s/qjournal/TestSecureNNWithQJM.java
>>> I was thinking it could be the bases for a secure functional test. We
>>>are
>>> sorely missing those. My basic thought would be that this type of test
>>> would have a separate maven profile and would run as a separate jenkins
>>>job.
>>> Kevin.
>>>
>
>
Re: Sample Hadoop test using HDFS and MiniKDC
Posted by Sumit Gupta <su...@hortonworks.com>.
Hi everyone,
I took a stab at getting the pieces together for a very basic knox test. I
put it up here for a preview https://github.com/sumitg/knox-minikdc-test.
Please note that it will be cleaned up if we decide we want to put
something like this in knox project.
The way I understand Kevin¹s thoughts, I would add a maven module
(something like Œgateway-test-secure¹ or Œgateway-test-integration¹?) and
a maven profile that triggers the test suite run, so that we can have a
separate job for this or run it locally by passing in the profile name.
I will be happy to add a bunch of the webhdfs tests to start this off. I
imagine we¹ll iterate over this over time and keep adding more support for
various services and the associated tests for them.
Sumit.
On 1/5/16, 3:33 PM, "larry mccay" <la...@gmail.com> wrote:
>+1 on adding secure tests!
>
>What functionality do we want to test as part of this suite initially?
>
>
>On Tue, Jan 5, 2016 at 2:59 PM, Kevin Minder
><ke...@hortonworks.com>
>wrote:
>
>> Hey Everyone,
>> I ran across this test in Hadoop the other day.
>>
>>
>>https://github.com/apache/hadoop/blob/2f623fb8cc3dc49221216c3b46b6f511448
>>11904/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdf
>>s/qjournal/TestSecureNNWithQJM.java
>> I was thinking it could be the bases for a secure functional test. We
>>are
>> sorely missing those. My basic thought would be that this type of test
>> would have a separate maven profile and would run as a separate jenkins
>>job.
>> Kevin.
>>
Re: Sample Hadoop test using HDFS and MiniKDC
Posted by larry mccay <la...@gmail.com>.
+1 on adding secure tests!
What functionality do we want to test as part of this suite initially?
On Tue, Jan 5, 2016 at 2:59 PM, Kevin Minder <ke...@hortonworks.com>
wrote:
> Hey Everyone,
> I ran across this test in Hadoop the other day.
>
> https://github.com/apache/hadoop/blob/2f623fb8cc3dc49221216c3b46b6f51144811904/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestSecureNNWithQJM.java
> I was thinking it could be the bases for a secure functional test. We are
> sorely missing those. My basic thought would be that this type of test
> would have a separate maven profile and would run as a separate jenkins job.
> Kevin.
>