You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Vasu Kulkarni <va...@redhat.com> on 2017/05/26 16:00:05 UTC
Fwd: s3a hadoop 2.8.0 tests vs 2.7.3
Sorry resending because i had problems with group subscribe
Hi,
I am trying to run hadoop s3a unit tests on 2.8.0 release(using ceph
radosgw), I notice that many tests that ran in 2.7.3 have been
skipped in 2.8.0 hadoop release,
I am following the configuration options from here that worked here
for 2.7.3: https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
Has any of the configuration options changed for 2.8.0 which are not
yet documented or the test structure has changed ? Thanks
on 2.8.0:
Tests: (mvn test -Dtest=S3a*,TestS3A* )
Running org.apache.hadoop.fs.s3a.TestS3AExceptionTranslation
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.124
sec - in org.apache.hadoop.fs.s3a.TestS3AExceptionTranslation
Running org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.385
sec - in org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider
Running org.apache.hadoop.fs.s3a.TestS3AInputPolicies
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.146
sec - in org.apache.hadoop.fs.s3a.TestS3AInputPolicies
Running org.apache.hadoop.fs.s3a.TestS3AGetFileStatus
Tests run: 40, Failures: 0, Errors: 0, Skipped: 0
on 2.7.3:
Tests: ( mvn test -Dtest=S3a*,TestS3A*)
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
.
.
Running org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles
Tests run: 88, Failures: 0, Errors: 0, Skipped: 48
Thanks
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: s3a hadoop 2.8.0 tests vs 2.7.3
Posted by Vasu Kulkarni <va...@redhat.com>.
Thanks Steve
On Wed, May 31, 2017 at 4:18 AM, Steve Loughran <st...@hortonworks.com>
wrote:
> see also
>
> https://github.com/apache/hadoop/blob/trunk/hadoop-
> tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
>
>
> On 26 May 2017, at 23:22, Vasu Kulkarni <va...@redhat.com> wrote:
>
> Thanks Liu
>
> On Fri, May 26, 2017 at 2:42 PM, Mingliang Liu <li...@apache.org> wrote:
>
> Hi,
>
> Many tests of S3A have been moved to “integration tests”, whose names start
> with “ITestS3A”. Moreover, the phase of Maven for those tests are “verify”
> instead of “test” now.
>
> So, you can specify the "mvn -Dit.test=‘ ITestS3A*’ verify" for integration
> tests (and unit tests). “mvn test” will run unit tests only. Make sure the
> credentials are provided.
>
> By the way, upgrading from 2.7 to 2.8 is a smart choice from S3A point of
> view.
>
> https://github.com/apache/hadoop/blob/branch-2.8.1/
> hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-
> aws/index.md#running-the-tests
>
> L
>
> On May 26, 2017, at 9:00 AM, Vasu Kulkarni <va...@redhat.com> wrote:
>
> Sorry resending because i had problems with group subscribe
>
> Hi,
>
> I am trying to run hadoop s3a unit tests on 2.8.0 release(using ceph
> radosgw), I notice that many tests that ran in 2.7.3 have been
> skipped in 2.8.0 hadoop release,
> I am following the configuration options from here that worked here
> for 2.7.3:
> https://github.com/apache/hadoop/blob/trunk/hadoop-
> tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
>
> Has any of the configuration options changed for 2.8.0 which are not
> yet documented or the test structure has changed ? Thanks
>
> on 2.8.0:
>
> Tests: (mvn test -Dtest=S3a*,TestS3A* )
> Running org.apache.hadoop.fs.s3a.TestS3AExceptionTranslation
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.124
> sec - in org.apache.hadoop.fs.s3a.TestS3AExceptionTranslation
> Running org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider
> Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.385
> sec - in org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider
> Running org.apache.hadoop.fs.s3a.TestS3AInputPolicies
> Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.146
> sec - in org.apache.hadoop.fs.s3a.TestS3AInputPolicies
> Running org.apache.hadoop.fs.s3a.TestS3AGetFileStatus
>
>
> Tests run: 40, Failures: 0, Errors: 0, Skipped: 0
>
>
>
> on 2.7.3:
>
> Tests: ( mvn test -Dtest=S3a*,TestS3A*)
>
> Running org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir
> Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
> Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
> .
> .
> Running org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles
>
> Tests run: 88, Failures: 0, Errors: 0, Skipped: 48
>
> Thanks
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>
>
>
Re: s3a hadoop 2.8.0 tests vs 2.7.3
Posted by Steve Loughran <st...@hortonworks.com>.
see also
https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
On 26 May 2017, at 23:22, Vasu Kulkarni <va...@redhat.com>> wrote:
Thanks Liu
On Fri, May 26, 2017 at 2:42 PM, Mingliang Liu <li...@apache.org>> wrote:
Hi,
Many tests of S3A have been moved to “integration tests”, whose names start
with “ITestS3A”. Moreover, the phase of Maven for those tests are “verify”
instead of “test” now.
So, you can specify the "mvn -Dit.test=‘ ITestS3A*’ verify" for integration
tests (and unit tests). “mvn test” will run unit tests only. Make sure the
credentials are provided.
By the way, upgrading from 2.7 to 2.8 is a smart choice from S3A point of
view.
https://github.com/apache/hadoop/blob/branch-2.8.1/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md#running-the-tests
L
On May 26, 2017, at 9:00 AM, Vasu Kulkarni <va...@redhat.com> wrote:
Sorry resending because i had problems with group subscribe
Hi,
I am trying to run hadoop s3a unit tests on 2.8.0 release(using ceph
radosgw), I notice that many tests that ran in 2.7.3 have been
skipped in 2.8.0 hadoop release,
I am following the configuration options from here that worked here
for 2.7.3:
https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
Has any of the configuration options changed for 2.8.0 which are not
yet documented or the test structure has changed ? Thanks
on 2.8.0:
Tests: (mvn test -Dtest=S3a*,TestS3A* )
Running org.apache.hadoop.fs.s3a.TestS3AExceptionTranslation
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.124
sec - in org.apache.hadoop.fs.s3a.TestS3AExceptionTranslation
Running org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.385
sec - in org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider
Running org.apache.hadoop.fs.s3a.TestS3AInputPolicies
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.146
sec - in org.apache.hadoop.fs.s3a.TestS3AInputPolicies
Running org.apache.hadoop.fs.s3a.TestS3AGetFileStatus
Tests run: 40, Failures: 0, Errors: 0, Skipped: 0
on 2.7.3:
Tests: ( mvn test -Dtest=S3a*,TestS3A*)
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
.
.
Running org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles
Tests run: 88, Failures: 0, Errors: 0, Skipped: 48
Thanks
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org<ma...@hadoop.apache.org>
For additional commands, e-mail: common-dev-help@hadoop.apache.org<ma...@hadoop.apache.org>
Re: s3a hadoop 2.8.0 tests vs 2.7.3
Posted by Vasu Kulkarni <va...@redhat.com>.
Thanks Liu
On Fri, May 26, 2017 at 2:42 PM, Mingliang Liu <li...@apache.org> wrote:
> Hi,
>
> Many tests of S3A have been moved to “integration tests”, whose names start
> with “ITestS3A”. Moreover, the phase of Maven for those tests are “verify”
> instead of “test” now.
>
> So, you can specify the "mvn -Dit.test=‘ ITestS3A*’ verify" for integration
> tests (and unit tests). “mvn test” will run unit tests only. Make sure the
> credentials are provided.
>
> By the way, upgrading from 2.7 to 2.8 is a smart choice from S3A point of
> view.
>
> https://github.com/apache/hadoop/blob/branch-2.8.1/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md#running-the-tests
>
> L
>
> On May 26, 2017, at 9:00 AM, Vasu Kulkarni <va...@redhat.com> wrote:
>
> Sorry resending because i had problems with group subscribe
>
> Hi,
>
> I am trying to run hadoop s3a unit tests on 2.8.0 release(using ceph
> radosgw), I notice that many tests that ran in 2.7.3 have been
> skipped in 2.8.0 hadoop release,
> I am following the configuration options from here that worked here
> for 2.7.3:
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
>
> Has any of the configuration options changed for 2.8.0 which are not
> yet documented or the test structure has changed ? Thanks
>
> on 2.8.0:
>
> Tests: (mvn test -Dtest=S3a*,TestS3A* )
> Running org.apache.hadoop.fs.s3a.TestS3AExceptionTranslation
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.124
> sec - in org.apache.hadoop.fs.s3a.TestS3AExceptionTranslation
> Running org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider
> Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.385
> sec - in org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider
> Running org.apache.hadoop.fs.s3a.TestS3AInputPolicies
> Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.146
> sec - in org.apache.hadoop.fs.s3a.TestS3AInputPolicies
> Running org.apache.hadoop.fs.s3a.TestS3AGetFileStatus
>
>
> Tests run: 40, Failures: 0, Errors: 0, Skipped: 0
>
>
>
> on 2.7.3:
>
> Tests: ( mvn test -Dtest=S3a*,TestS3A*)
>
> Running org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir
> Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
> Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
> .
> .
> Running org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles
>
> Tests run: 88, Failures: 0, Errors: 0, Skipped: 48
>
> Thanks
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: s3a hadoop 2.8.0 tests vs 2.7.3
Posted by Mingliang Liu <li...@apache.org>.
Hi,
Many tests of S3A have been moved to “integration tests”, whose names start with “ITestS3A”. Moreover, the phase of Maven for those tests are “verify” instead of “test” now.
So, you can specify the "mvn -Dit.test=‘ITestS3A*’ verify" for integration tests (and unit tests). “mvn test” will run unit tests only. Make sure the credentials are provided.
By the way, upgrading from 2.7 to 2.8 is a smart choice from S3A point of view.
https://github.com/apache/hadoop/blob/branch-2.8.1/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md#running-the-tests <https://github.com/apache/hadoop/blob/branch-2.8.1/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md#running-the-tests>
L
> On May 26, 2017, at 9:00 AM, Vasu Kulkarni <va...@redhat.com> wrote:
>
> Sorry resending because i had problems with group subscribe
>
> Hi,
>
> I am trying to run hadoop s3a unit tests on 2.8.0 release(using ceph
> radosgw), I notice that many tests that ran in 2.7.3 have been
> skipped in 2.8.0 hadoop release,
> I am following the configuration options from here that worked here
> for 2.7.3: https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
>
> Has any of the configuration options changed for 2.8.0 which are not
> yet documented or the test structure has changed ? Thanks
>
> on 2.8.0:
>
> Tests: (mvn test -Dtest=S3a*,TestS3A* )
> Running org.apache.hadoop.fs.s3a.TestS3AExceptionTranslation
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.124
> sec - in org.apache.hadoop.fs.s3a.TestS3AExceptionTranslation
> Running org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider
> Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.385
> sec - in org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider
> Running org.apache.hadoop.fs.s3a.TestS3AInputPolicies
> Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.146
> sec - in org.apache.hadoop.fs.s3a.TestS3AInputPolicies
> Running org.apache.hadoop.fs.s3a.TestS3AGetFileStatus
>
>
> Tests run: 40, Failures: 0, Errors: 0, Skipped: 0
>
>
>
> on 2.7.3:
>
> Tests: ( mvn test -Dtest=S3a*,TestS3A*)
>
> Running org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir
> Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
> Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
> .
> .
> Running org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles
>
> Tests run: 88, Failures: 0, Errors: 0, Skipped: 48
>
> Thanks
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>