You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Nick White <nw...@palantir.com> on 2013/01/03 01:36:09 UTC
Review Request: SQOOP-749: Exports Using Stored Procedures (Functions)
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8806/
-----------------------------------------------------------
Review request for Sqoop and Jarek Cecho.
Description
-------
It'd be useful if you could use stored procedures (or functions) to insert data - currently you can only use insert or update statements (or upsert / merges, depending on the SqlManager you're using). This would help sqoop adoption / migration into environments which have existing, SQL-based data import workflows.
The attached patch adds a --call argument to the export tool that can be used in place of --table.
Note that this patch depends on HSQLDB 2.x, whereas trunk currently depends on 1.8, so it can't be committed until that situation's resolved (see SQOOP-519 and linked tickets).
This addresses bug SQOOP-749.
https://issues.apache.org/jira/browse/SQOOP-749
Diffs
-----
src/java/org/apache/sqoop/SqoopOptions.java 613f797
src/java/org/apache/sqoop/manager/ConnManager.java 21eea93
src/java/org/apache/sqoop/manager/SqlManager.java 3a52c6d
src/java/org/apache/sqoop/mapreduce/ExportCallOutputFormat.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/JdbcCallExportJob.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java 7c52110
src/java/org/apache/sqoop/orm/ClassWriter.java b73711e
src/java/org/apache/sqoop/tool/BaseSqoopTool.java d795646
src/java/org/apache/sqoop/tool/ExportTool.java acd296d
src/test/com/cloudera/sqoop/SmokeTests.java 76df6cf
src/test/com/cloudera/sqoop/TestConnFactory.java 893b388
src/test/com/cloudera/sqoop/TestExport.java eba10aa
src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java 4f6fd37
src/test/org/apache/sqoop/TestExportUsingProcedure.java PRE-CREATION
Diff: https://reviews.apache.org/r/8806/diff/
Testing
-------
I've added a subclass of TestExport to make sure all the usual export functionality works. I've also used it on live data exporting to postgres.
Thanks,
Nick White
Re: Review Request: SQOOP-749: Exports Using Stored Procedures (Functions)
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8806/#review15101
-----------------------------------------------------------
Hi Nick,
thank you very much for your hard work on this issue. The high level functionality looks good to me. Would you mind providing also documentation for user? (src/doc/user)
I'm afraid that we can't upgrade to HSQLDB 2.0 as it would break internal metastore. As an alternative we might use derby or create tests in third party "module" (e.g separate tests for MySQL, PostgreSQL, Microsoft SQL Server, Oracle, ...).
I would personally really like to see the stored procedure tests for majority of third party tests to be sure that it will work correctly on given databases.
Jarcec
- Jarek Cecho
On Jan. 3, 2013, 12:36 a.m., Nick White wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8806/
> -----------------------------------------------------------
>
> (Updated Jan. 3, 2013, 12:36 a.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Description
> -------
>
> It'd be useful if you could use stored procedures (or functions) to insert data - currently you can only use insert or update statements (or upsert / merges, depending on the SqlManager you're using). This would help sqoop adoption / migration into environments which have existing, SQL-based data import workflows.
>
> The attached patch adds a --call argument to the export tool that can be used in place of --table.
>
> Note that this patch depends on HSQLDB 2.x, whereas trunk currently depends on 1.8, so it can't be committed until that situation's resolved (see SQOOP-519 and linked tickets).
>
>
> This addresses bug SQOOP-749.
> https://issues.apache.org/jira/browse/SQOOP-749
>
>
> Diffs
> -----
>
> src/java/org/apache/sqoop/SqoopOptions.java 613f797
> src/java/org/apache/sqoop/manager/ConnManager.java 21eea93
> src/java/org/apache/sqoop/manager/SqlManager.java 3a52c6d
> src/java/org/apache/sqoop/mapreduce/ExportCallOutputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/JdbcCallExportJob.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java 7c52110
> src/java/org/apache/sqoop/orm/ClassWriter.java b73711e
> src/java/org/apache/sqoop/tool/BaseSqoopTool.java d795646
> src/java/org/apache/sqoop/tool/ExportTool.java acd296d
> src/test/com/cloudera/sqoop/SmokeTests.java 76df6cf
> src/test/com/cloudera/sqoop/TestConnFactory.java 893b388
> src/test/com/cloudera/sqoop/TestExport.java eba10aa
> src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java 4f6fd37
> src/test/org/apache/sqoop/TestExportUsingProcedure.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/8806/diff/
>
>
> Testing
> -------
>
> I've added a subclass of TestExport to make sure all the usual export functionality works. I've also used it on live data exporting to postgres.
>
>
> Thanks,
>
> Nick White
>
>
Re: Review Request: SQOOP-749: Exports Using Stored Procedures (Functions)
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8806/#review15183
-----------------------------------------------------------
Hi Nick,
I think that you've done excellent job putting this together. I believe that we're almost there.
What bothers me is that stored procedures are heavily dependent on target database. I'm hesitating to press the "Ship It" button when there are no third party tests for MySQL, PostgreSQL, Microsoft SQL Server nor Oracle. Looking at current Microsoft SQL Server tests (SQLServerManagerExportManualTest.java), I think that extending them to stored procedures should not be a big deal. The challenging part is to set up all third party databases so that you can actually run the tests somewhere. I can help with that if you want me to.
ivy.xml
<https://reviews.apache.org/r/8806/#comment32753>
Nit: Trailing white space characters
src/java/org/apache/sqoop/mapreduce/JdbcCallExportJob.java
<https://reviews.apache.org/r/8806/#comment32754>
I'm thinking whether it do make sense to escape the procedure name in all cases. We've recently find out that some third party connectors do have issues if we will automatically escape the table name (SQOOP-820), but that is probably not the case for stored procedures as this is completely new thing.
Jarcec
- Jarek Cecho
On Jan. 7, 2013, 7:47 p.m., Nick White wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8806/
> -----------------------------------------------------------
>
> (Updated Jan. 7, 2013, 7:47 p.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Description
> -------
>
> It'd be useful if you could use stored procedures (or functions) to insert data - currently you can only use insert or update statements (or upsert / merges, depending on the SqlManager you're using). This would help sqoop adoption / migration into environments which have existing, SQL-based data import workflows.
>
> The attached patch adds a --call argument to the export tool that can be used in place of --table.
>
> Note that this patch depends on HSQLDB 2.x, whereas trunk currently depends on 1.8, so it can't be committed until that situation's resolved (see SQOOP-519 and linked tickets).
>
>
> This addresses bug SQOOP-749.
> https://issues.apache.org/jira/browse/SQOOP-749
>
>
> Diffs
> -----
>
> ivy.xml 1ee60df
> ivy/libraries.properties 4c9e37d
> src/docs/user/export-purpose.txt c26eaa7
> src/docs/user/export.txt 9f600fe
> src/java/org/apache/sqoop/SqoopOptions.java 3e0ec3e
> src/java/org/apache/sqoop/manager/ConnManager.java 21eea93
> src/java/org/apache/sqoop/manager/SqlManager.java 3a52c6d
> src/java/org/apache/sqoop/mapreduce/ExportCallOutputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/JdbcCallExportJob.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java 7c52110
> src/java/org/apache/sqoop/orm/ClassWriter.java b73711e
> src/java/org/apache/sqoop/tool/BaseSqoopTool.java c0221c9
> src/java/org/apache/sqoop/tool/ExportTool.java acd296d
> src/test/com/cloudera/sqoop/SmokeTests.java 76df6cf
> src/test/com/cloudera/sqoop/TestConnFactory.java 893b388
> src/test/com/cloudera/sqoop/TestExport.java eba10aa
> src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java 4f6fd37
> src/test/org/apache/sqoop/TestExportUsingProcedure.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/8806/diff/
>
>
> Testing
> -------
>
> I've added a subclass of TestExport to make sure all the usual export functionality works. I've also used it on live data exporting to postgres.
>
>
> Thanks,
>
> Nick White
>
>
Re: Review Request: SQOOP-749: Exports Using Stored Procedures (Functions)
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8806/#review15340
-----------------------------------------------------------
Hi Nick,
Thank you for introducing explicit PostgreSQL third party test. I think that test coverage is reasonable for newly introduced functionality - let's create follow up JIRAs to add similar tests to other third party tests once this gets committed.
I've noticed that your patch is changing order of import statements in a lot of files. Such changes are complicating development as they are introducing unnecessary changes. Would you mind putting the order back? I've highlighted all of them for you in the review notes below.
Other than than I just realized that adding new abstract methods to ConnManager is dangerous for backward compatibility - it might break already existing connectors that inherits from ConnManager and do not implement such methods. As sqoop 1 is considered stable, would you mind changing those methods to non abstract and throw an exception in case that they will be called?
src/java/org/apache/sqoop/SqoopOptions.java
<https://reviews.apache.org/r/8806/#comment32992>
Would you mind putting the import statements in original order?
src/java/org/apache/sqoop/manager/ConnManager.java
<https://reviews.apache.org/r/8806/#comment32990>
Would you mind changing this abstract method to a normal method returning Exception that it's not supported?
I'm concerned here with backward compatibly for connectors that are already out there.
src/java/org/apache/sqoop/manager/ConnManager.java
<https://reviews.apache.org/r/8806/#comment32991>
Would you mind changing this abstract method to a normal method returning Exception that it's not supported?
I'm concerned here with backward compatibly for connectors that are already out there.
src/java/org/apache/sqoop/manager/SqlManager.java
<https://reviews.apache.org/r/8806/#comment32993>
Would you mind putting the import statements in original order?
src/java/org/apache/sqoop/mapreduce/ExportCallOutputFormat.java
<https://reviews.apache.org/r/8806/#comment33004>
This method seems to be unused. Would you mind remove it if it's really the case?
src/java/org/apache/sqoop/orm/ClassWriter.java
<https://reviews.apache.org/r/8806/#comment32994>
Would you mind putting the import statements in original order?
src/test/com/cloudera/sqoop/TestConnFactory.java
<https://reviews.apache.org/r/8806/#comment32995>
Would you mind putting the import statements in original order?
src/test/com/cloudera/sqoop/TestExport.java
<https://reviews.apache.org/r/8806/#comment32996>
Would you mind putting the import statements in original order?
src/test/com/cloudera/sqoop/manager/PostgresqlExportTest.java
<https://reviews.apache.org/r/8806/#comment32997>
Would you mind putting the import statements in original order?
src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java
<https://reviews.apache.org/r/8806/#comment32998>
Would you mind putting the import statements in original order?
Thank you very much for all your hard work, I really do appreciate that!
Jarcec
- Jarek Cecho
On Jan. 15, 2013, 3:20 a.m., Nick White wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8806/
> -----------------------------------------------------------
>
> (Updated Jan. 15, 2013, 3:20 a.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Description
> -------
>
> It'd be useful if you could use stored procedures (or functions) to insert data - currently you can only use insert or update statements (or upsert / merges, depending on the SqlManager you're using). This would help sqoop adoption / migration into environments which have existing, SQL-based data import workflows.
>
> The attached patch adds a --call argument to the export tool that can be used in place of --table.
>
> Note that this patch depends on HSQLDB 2.x, whereas trunk currently depends on 1.8, so it can't be committed until that situation's resolved (see SQOOP-519 and linked tickets).
>
>
> This addresses bug SQOOP-749.
> https://issues.apache.org/jira/browse/SQOOP-749
>
>
> Diffs
> -----
>
> ivy.xml 1ee60df
> ivy/libraries.properties 4c9e37d
> src/docs/user/export-purpose.txt c26eaa7
> src/docs/user/export.txt 9f600fe
> src/java/org/apache/sqoop/SqoopOptions.java 3e0ec3e
> src/java/org/apache/sqoop/manager/ConnManager.java 21eea93
> src/java/org/apache/sqoop/manager/SqlManager.java 3a52c6d
> src/java/org/apache/sqoop/mapreduce/ExportCallOutputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/JdbcCallExportJob.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java 7c52110
> src/java/org/apache/sqoop/orm/ClassWriter.java b73711e
> src/java/org/apache/sqoop/tool/BaseSqoopTool.java c0221c9
> src/java/org/apache/sqoop/tool/ExportTool.java acd296d
> src/test/com/cloudera/sqoop/SmokeTests.java 76df6cf
> src/test/com/cloudera/sqoop/TestConnFactory.java 893b388
> src/test/com/cloudera/sqoop/TestExport.java eba10aa
> src/test/com/cloudera/sqoop/manager/PostgresqlExportTest.java be449e4
> src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java 4f6fd37
> src/test/org/apache/sqoop/TestExportUsingProcedure.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/8806/diff/
>
>
> Testing
> -------
>
> I've added a subclass of TestExport to make sure all the usual export functionality works. I've also used it on live data exporting to postgres.
>
>
> Thanks,
>
> Nick White
>
>
Re: Review Request: SQOOP-749: Exports Using Stored Procedures (Functions)
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8806/#review15580
-----------------------------------------------------------
Ship it!
Ship It!
- Jarek Cecho
On Jan. 17, 2013, 2:25 p.m., Nick White wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8806/
> -----------------------------------------------------------
>
> (Updated Jan. 17, 2013, 2:25 p.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Description
> -------
>
> It'd be useful if you could use stored procedures (or functions) to insert data - currently you can only use insert or update statements (or upsert / merges, depending on the SqlManager you're using). This would help sqoop adoption / migration into environments which have existing, SQL-based data import workflows.
>
> The attached patch adds a --call argument to the export tool that can be used in place of --table.
>
> Note that this patch depends on HSQLDB 2.x, whereas trunk currently depends on 1.8, so it can't be committed until that situation's resolved (see SQOOP-519 and linked tickets).
>
>
> This addresses bug SQOOP-749.
> https://issues.apache.org/jira/browse/SQOOP-749
>
>
> Diffs
> -----
>
> ivy.xml 1ee60df
> ivy/libraries.properties 4c9e37d
> src/docs/user/export-purpose.txt c26eaa7
> src/docs/user/export.txt 9f600fe
> src/java/org/apache/sqoop/SqoopOptions.java b0fdfa0
> src/java/org/apache/sqoop/manager/ConnManager.java 115186f
> src/java/org/apache/sqoop/manager/SqlManager.java 3a52c6d
> src/java/org/apache/sqoop/mapreduce/ExportCallOutputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/JdbcCallExportJob.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java 00bd910
> src/java/org/apache/sqoop/orm/ClassWriter.java 47e1221
> src/java/org/apache/sqoop/tool/BaseSqoopTool.java b4b2213
> src/java/org/apache/sqoop/tool/ExportTool.java acd296d
> src/test/com/cloudera/sqoop/SmokeTests.java 76df6cf
> src/test/com/cloudera/sqoop/TestConnFactory.java 893b388
> src/test/com/cloudera/sqoop/TestExport.java eba10aa
> src/test/com/cloudera/sqoop/manager/PostgresqlExportTest.java be449e4
> src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java 4f6fd37
> src/test/org/apache/sqoop/TestExportUsingProcedure.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/8806/diff/
>
>
> Testing
> -------
>
> I've added a subclass of TestExport to make sure all the usual export functionality works. I've also used it on live data exporting to postgres.
>
>
> Thanks,
>
> Nick White
>
>
Re: Review Request: SQOOP-749: Exports Using Stored Procedures (Functions)
Posted by Nick White <nw...@palantir.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8806/
-----------------------------------------------------------
(Updated Jan. 17, 2013, 2:25 p.m.)
Review request for Sqoop and Jarek Cecho.
Changes
-------
Sorry about the import stuff - I haven't figured out how to make eclipse remove unused imports without sorting the remaining ones alphabetically (or how to be as productive in a different editor :). The attached patch fixes all the issues you've raised. Thanks!
Description
-------
It'd be useful if you could use stored procedures (or functions) to insert data - currently you can only use insert or update statements (or upsert / merges, depending on the SqlManager you're using). This would help sqoop adoption / migration into environments which have existing, SQL-based data import workflows.
The attached patch adds a --call argument to the export tool that can be used in place of --table.
Note that this patch depends on HSQLDB 2.x, whereas trunk currently depends on 1.8, so it can't be committed until that situation's resolved (see SQOOP-519 and linked tickets).
This addresses bug SQOOP-749.
https://issues.apache.org/jira/browse/SQOOP-749
Diffs (updated)
-----
ivy.xml 1ee60df
ivy/libraries.properties 4c9e37d
src/docs/user/export-purpose.txt c26eaa7
src/docs/user/export.txt 9f600fe
src/java/org/apache/sqoop/SqoopOptions.java b0fdfa0
src/java/org/apache/sqoop/manager/ConnManager.java 115186f
src/java/org/apache/sqoop/manager/SqlManager.java 3a52c6d
src/java/org/apache/sqoop/mapreduce/ExportCallOutputFormat.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/JdbcCallExportJob.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java 00bd910
src/java/org/apache/sqoop/orm/ClassWriter.java 47e1221
src/java/org/apache/sqoop/tool/BaseSqoopTool.java b4b2213
src/java/org/apache/sqoop/tool/ExportTool.java acd296d
src/test/com/cloudera/sqoop/SmokeTests.java 76df6cf
src/test/com/cloudera/sqoop/TestConnFactory.java 893b388
src/test/com/cloudera/sqoop/TestExport.java eba10aa
src/test/com/cloudera/sqoop/manager/PostgresqlExportTest.java be449e4
src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java 4f6fd37
src/test/org/apache/sqoop/TestExportUsingProcedure.java PRE-CREATION
Diff: https://reviews.apache.org/r/8806/diff/
Testing
-------
I've added a subclass of TestExport to make sure all the usual export functionality works. I've also used it on live data exporting to postgres.
Thanks,
Nick White
Re: Review Request: SQOOP-749: Exports Using Stored Procedures (Functions)
Posted by Nick White <nw...@palantir.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8806/
-----------------------------------------------------------
(Updated Jan. 15, 2013, 3:20 a.m.)
Review request for Sqoop and Jarek Cecho.
Changes
-------
Remove the runtime dependency on commons-lang 2.4 (StringUtils.repeat).
Description
-------
It'd be useful if you could use stored procedures (or functions) to insert data - currently you can only use insert or update statements (or upsert / merges, depending on the SqlManager you're using). This would help sqoop adoption / migration into environments which have existing, SQL-based data import workflows.
The attached patch adds a --call argument to the export tool that can be used in place of --table.
Note that this patch depends on HSQLDB 2.x, whereas trunk currently depends on 1.8, so it can't be committed until that situation's resolved (see SQOOP-519 and linked tickets).
This addresses bug SQOOP-749.
https://issues.apache.org/jira/browse/SQOOP-749
Diffs (updated)
-----
ivy.xml 1ee60df
ivy/libraries.properties 4c9e37d
src/docs/user/export-purpose.txt c26eaa7
src/docs/user/export.txt 9f600fe
src/java/org/apache/sqoop/SqoopOptions.java 3e0ec3e
src/java/org/apache/sqoop/manager/ConnManager.java 21eea93
src/java/org/apache/sqoop/manager/SqlManager.java 3a52c6d
src/java/org/apache/sqoop/mapreduce/ExportCallOutputFormat.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/JdbcCallExportJob.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java 7c52110
src/java/org/apache/sqoop/orm/ClassWriter.java b73711e
src/java/org/apache/sqoop/tool/BaseSqoopTool.java c0221c9
src/java/org/apache/sqoop/tool/ExportTool.java acd296d
src/test/com/cloudera/sqoop/SmokeTests.java 76df6cf
src/test/com/cloudera/sqoop/TestConnFactory.java 893b388
src/test/com/cloudera/sqoop/TestExport.java eba10aa
src/test/com/cloudera/sqoop/manager/PostgresqlExportTest.java be449e4
src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java 4f6fd37
src/test/org/apache/sqoop/TestExportUsingProcedure.java PRE-CREATION
Diff: https://reviews.apache.org/r/8806/diff/
Testing
-------
I've added a subclass of TestExport to make sure all the usual export functionality works. I've also used it on live data exporting to postgres.
Thanks,
Nick White
Re: Review Request: SQOOP-749: Exports Using Stored Procedures (Functions)
Posted by Nick White <nw...@palantir.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8806/
-----------------------------------------------------------
(Updated Jan. 15, 2013, 1:39 a.m.)
Review request for Sqoop and Jarek Cecho.
Changes
-------
I've fixed one of the imports - I meant it to come from Google Guava.
Description
-------
It'd be useful if you could use stored procedures (or functions) to insert data - currently you can only use insert or update statements (or upsert / merges, depending on the SqlManager you're using). This would help sqoop adoption / migration into environments which have existing, SQL-based data import workflows.
The attached patch adds a --call argument to the export tool that can be used in place of --table.
Note that this patch depends on HSQLDB 2.x, whereas trunk currently depends on 1.8, so it can't be committed until that situation's resolved (see SQOOP-519 and linked tickets).
This addresses bug SQOOP-749.
https://issues.apache.org/jira/browse/SQOOP-749
Diffs (updated)
-----
ivy.xml 1ee60df
ivy/libraries.properties 4c9e37d
src/docs/user/export-purpose.txt c26eaa7
src/docs/user/export.txt 9f600fe
src/java/org/apache/sqoop/SqoopOptions.java 3e0ec3e
src/java/org/apache/sqoop/manager/ConnManager.java 21eea93
src/java/org/apache/sqoop/manager/SqlManager.java 3a52c6d
src/java/org/apache/sqoop/mapreduce/ExportCallOutputFormat.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/JdbcCallExportJob.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java 7c52110
src/java/org/apache/sqoop/orm/ClassWriter.java b73711e
src/java/org/apache/sqoop/tool/BaseSqoopTool.java c0221c9
src/java/org/apache/sqoop/tool/ExportTool.java acd296d
src/test/com/cloudera/sqoop/SmokeTests.java 76df6cf
src/test/com/cloudera/sqoop/TestConnFactory.java 893b388
src/test/com/cloudera/sqoop/TestExport.java eba10aa
src/test/com/cloudera/sqoop/manager/PostgresqlExportTest.java be449e4
src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java 4f6fd37
src/test/org/apache/sqoop/TestExportUsingProcedure.java PRE-CREATION
Diff: https://reviews.apache.org/r/8806/diff/
Testing
-------
I've added a subclass of TestExport to make sure all the usual export functionality works. I've also used it on live data exporting to postgres.
Thanks,
Nick White
Re: Review Request: SQOOP-749: Exports Using Stored Procedures (Functions)
Posted by Nick White <nw...@palantir.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8806/
-----------------------------------------------------------
(Updated Jan. 10, 2013, 6:25 p.m.)
Review request for Sqoop and Jarek Cecho.
Changes
-------
I've added third-party postgres tests to this latest revision. I don't have a MSSQL installation around to test on, but if I add a --call test case to TestExport it'll get picked up by the Oracle, JdbcMySQL & DirectMySQL tests. Would this be a good next step?
Description
-------
It'd be useful if you could use stored procedures (or functions) to insert data - currently you can only use insert or update statements (or upsert / merges, depending on the SqlManager you're using). This would help sqoop adoption / migration into environments which have existing, SQL-based data import workflows.
The attached patch adds a --call argument to the export tool that can be used in place of --table.
Note that this patch depends on HSQLDB 2.x, whereas trunk currently depends on 1.8, so it can't be committed until that situation's resolved (see SQOOP-519 and linked tickets).
This addresses bug SQOOP-749.
https://issues.apache.org/jira/browse/SQOOP-749
Diffs (updated)
-----
ivy.xml 1ee60df
ivy/libraries.properties 4c9e37d
src/docs/user/export-purpose.txt c26eaa7
src/docs/user/export.txt 9f600fe
src/java/org/apache/sqoop/SqoopOptions.java 3e0ec3e
src/java/org/apache/sqoop/manager/ConnManager.java 21eea93
src/java/org/apache/sqoop/manager/SqlManager.java 3a52c6d
src/java/org/apache/sqoop/mapreduce/ExportCallOutputFormat.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/JdbcCallExportJob.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java 7c52110
src/java/org/apache/sqoop/orm/ClassWriter.java b73711e
src/java/org/apache/sqoop/tool/BaseSqoopTool.java c0221c9
src/java/org/apache/sqoop/tool/ExportTool.java acd296d
src/test/com/cloudera/sqoop/SmokeTests.java 76df6cf
src/test/com/cloudera/sqoop/TestConnFactory.java 893b388
src/test/com/cloudera/sqoop/TestExport.java eba10aa
src/test/com/cloudera/sqoop/manager/PostgresqlExportTest.java be449e4
src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java 4f6fd37
src/test/org/apache/sqoop/TestExportUsingProcedure.java PRE-CREATION
Diff: https://reviews.apache.org/r/8806/diff/
Testing
-------
I've added a subclass of TestExport to make sure all the usual export functionality works. I've also used it on live data exporting to postgres.
Thanks,
Nick White
Re: Review Request: SQOOP-749: Exports Using Stored Procedures (Functions)
Posted by Nick White <nw...@palantir.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8806/
-----------------------------------------------------------
(Updated Jan. 7, 2013, 7:47 p.m.)
Review request for Sqoop and Jarek Cecho.
Changes
-------
I've revised the patch so that it includes H2 as a test dependency, and uses that in the unit tests (I'm more familiar with H2 than Derby). I've also fixed the checkstyle issues and added some documentation.
Description
-------
It'd be useful if you could use stored procedures (or functions) to insert data - currently you can only use insert or update statements (or upsert / merges, depending on the SqlManager you're using). This would help sqoop adoption / migration into environments which have existing, SQL-based data import workflows.
The attached patch adds a --call argument to the export tool that can be used in place of --table.
Note that this patch depends on HSQLDB 2.x, whereas trunk currently depends on 1.8, so it can't be committed until that situation's resolved (see SQOOP-519 and linked tickets).
This addresses bug SQOOP-749.
https://issues.apache.org/jira/browse/SQOOP-749
Diffs (updated)
-----
ivy.xml 1ee60df
ivy/libraries.properties 4c9e37d
src/docs/user/export-purpose.txt c26eaa7
src/docs/user/export.txt 9f600fe
src/java/org/apache/sqoop/SqoopOptions.java 3e0ec3e
src/java/org/apache/sqoop/manager/ConnManager.java 21eea93
src/java/org/apache/sqoop/manager/SqlManager.java 3a52c6d
src/java/org/apache/sqoop/mapreduce/ExportCallOutputFormat.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/JdbcCallExportJob.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java 7c52110
src/java/org/apache/sqoop/orm/ClassWriter.java b73711e
src/java/org/apache/sqoop/tool/BaseSqoopTool.java c0221c9
src/java/org/apache/sqoop/tool/ExportTool.java acd296d
src/test/com/cloudera/sqoop/SmokeTests.java 76df6cf
src/test/com/cloudera/sqoop/TestConnFactory.java 893b388
src/test/com/cloudera/sqoop/TestExport.java eba10aa
src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java 4f6fd37
src/test/org/apache/sqoop/TestExportUsingProcedure.java PRE-CREATION
Diff: https://reviews.apache.org/r/8806/diff/
Testing
-------
I've added a subclass of TestExport to make sure all the usual export functionality works. I've also used it on live data exporting to postgres.
Thanks,
Nick White