You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Jarek Cecho <ja...@apache.org> on 2015/02/02 17:03:29 UTC
Review Request 30500: Preliminary patch - SQOOP-2059 Provide
exhaustive data type integration tests
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30500/
-----------------------------------------------------------
Review request for Sqoop.
Bugs: SQOOP-2059
https://issues.apache.org/jira/browse/SQOOP-2059
Repository: sqoop-sqoop2
Description
-------
I'm attaching preliminary patch how would I envision to attack the problem.
I've added new facility into the DatabaseProvider that can iterate over all types that given database supports with examples of data for each of the type (using various encodings). Each provider will have to define their own types as each database might have different types. Subsequently I'm using this new facility in new test that will iterate over all types and run both From and To transfer verifying that we got what we expected.
This patch contains a lot of unrelated changes that I want to cut into standalone patches. I've added a few types from Derby (not exhaustive yet) and I'm soliciting feedback :)
Diffs
-----
common-test/src/main/java/org/apache/sqoop/common/test/asserts/ProviderAsserts.java fb4e7af
common-test/src/main/java/org/apache/sqoop/common/test/db/DatabaseProvider.java 82289e8
common-test/src/main/java/org/apache/sqoop/common/test/db/DerbyProvider.java 98591a3
test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java adc638a
test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java 1124cd3
test/src/main/java/org/apache/sqoop/test/utils/ParametrizedUtils.java 656bc34
test/src/test/java/org/apache/sqoop/integration/connector/jdbc/generic/AllTypesTest.java PRE-CREATION
Diff: https://reviews.apache.org/r/30500/diff/
Testing
-------
Thanks,
Jarek Cecho
Re: Review Request 30500: Preliminary patch - SQOOP-2059 Provide
exhaustive data type integration tests
Posted by Abraham Elmahrek <ab...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30500/#review71543
-----------------------------------------------------------
Good idea!
I like the fact that we're explicit, but I think it's a bit misplaced. The DatabaseProvider seems more about facilitating database queries than providing test data.
common-test/src/main/java/org/apache/sqoop/common/test/db/DatabaseProvider.java
<https://reviews.apache.org/r/30500/#comment117284>
Think we could re-use the DataSet class instead? Or perhaps just move this out of DatabaseProvider into it's own class.
common-test/src/main/java/org/apache/sqoop/common/test/db/DatabaseProvider.java
<https://reviews.apache.org/r/30500/#comment117285>
Cool idea!
test/src/test/java/org/apache/sqoop/integration/connector/jdbc/generic/AllTypesTest.java
<https://reviews.apache.org/r/30500/#comment117287>
Be wary of process versus thread parallelization. I haven't set any of this up yet. The largest cost is not the time it takes to run a test, but the time it takes to start a cluster. Running clusters in parallel is definitely not going to work yet.
test/src/test/java/org/apache/sqoop/integration/connector/jdbc/generic/AllTypesTest.java
<https://reviews.apache.org/r/30500/#comment117286>
alwaysRun = true
- Abraham Elmahrek
On Feb. 2, 2015, 4:03 p.m., Jarek Cecho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30500/
> -----------------------------------------------------------
>
> (Updated Feb. 2, 2015, 4:03 p.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-2059
> https://issues.apache.org/jira/browse/SQOOP-2059
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> I'm attaching preliminary patch how would I envision to attack the problem.
>
> I've added new facility into the DatabaseProvider that can iterate over all types that given database supports with examples of data for each of the type (using various encodings). Each provider will have to define their own types as each database might have different types. Subsequently I'm using this new facility in new test that will iterate over all types and run both From and To transfer verifying that we got what we expected.
>
> This patch contains a lot of unrelated changes that I want to cut into standalone patches. I've added a few types from Derby (not exhaustive yet) and I'm soliciting feedback :)
>
>
> Diffs
> -----
>
> common-test/src/main/java/org/apache/sqoop/common/test/asserts/ProviderAsserts.java fb4e7af
> common-test/src/main/java/org/apache/sqoop/common/test/db/DatabaseProvider.java 82289e8
> common-test/src/main/java/org/apache/sqoop/common/test/db/DerbyProvider.java 98591a3
> test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java adc638a
> test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java 1124cd3
> test/src/main/java/org/apache/sqoop/test/utils/ParametrizedUtils.java 656bc34
> test/src/test/java/org/apache/sqoop/integration/connector/jdbc/generic/AllTypesTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/30500/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jarek Cecho
>
>