You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Jarek Cecho <ja...@apache.org> on 2012/08/10 09:20:47 UTC
Review Request: SQOOP-544: Split metadata between import and export related
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6508/
-----------------------------------------------------------
Review request for Sqoop, Arvind Prabhakar and Bilung Lee.
Description
-------
Sqoop framework needs to support multiple job types - mainly import and export job. As we're using word "job" for different purpose in the framework already, I've decided to introduce keyword "operation" for those job types. E.g. import and export operation. I do not insist on this particular word, so feel free to suggest better one.
This patch provide all necessary changes on both client, server and repository side.
This addresses bug SQOOP-544.
https://issues.apache.org/jira/browse/SQOOP-544
Diffs
-----
/branches/sqoop2/client/src/main/java/org/apache/sqoop/client/shell/ShowConnectorFunction.java 1371581
/branches/sqoop2/common/src/main/java/org/apache/sqoop/job/etl/Operation.java PRE-CREATION
/branches/sqoop2/common/src/main/java/org/apache/sqoop/json/ConnectorBean.java 1371581
/branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MConnector.java 1371581
/branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MFramework.java 1371581
/branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MOperation.java PRE-CREATION
/branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MStringInput.java 1371581
/branches/sqoop2/common/src/test/java/org/apache/sqoop/json/TestConnectorBean.java PRE-CREATION
/branches/sqoop2/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java 1371581
/branches/sqoop2/connector/connector-mysql-jdbc/src/main/java/org/apache/sqoop/connector/mysqljdbc/MySqlJdbcConnector.java 1371581
/branches/sqoop2/core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1371581
/branches/sqoop2/core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java 1371581
/branches/sqoop2/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java 1371581
/branches/sqoop2/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaConstants.java 1371581
/branches/sqoop2/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaQuery.java 1371581
/branches/sqoop2/spi/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 1371581
Diff: https://reviews.apache.org/r/6508/diff/
Testing
-------
* I've added new unit test for testing ConnectorBean serialization&deserialization
* I've manually verified (derby database, client side)
Expected output for "show connector -a" is:
sqoop:000> show connector -a
2 connector(s) to show:
Connector with id 1:
Name: mysql-jdbc-connector
Class: org.apache.sqoop.connector.mysqljdbc.MySqlJdbcConnector
Supported operations: []
Connector with id 2:
Name: generic-jdbc-connector
Class: org.apache.sqoop.connector.jdbc.GenericJdbcConnector
Supported operations: [EXPORT, IMPORT]
Forms for operation EXPORT:
Connection form 1:
Name: form-connection
Input 1:
Name: inp-conn-jdbcdriver
Type: STRING
Mask: false
Size: 128
Input 2:
Name: inp-conn-connectstring
Type: STRING
Mask: false
Size: 128
Input 3:
Name: inp-conn-username
Type: STRING
Mask: false
Size: 36
Input 4:
Name: inp-conn-password
Type: STRING
Mask: true
Size: 10
Input 5:
Name: inp-conn-jdbc-properties
Type: MAP
Forms for operation IMPORT:
Connection form 1:
Name: form-connection
Input 1:
Name: inp-conn-jdbcdriver
Type: STRING
Mask: false
Size: 128
Input 2:
Name: inp-conn-connectstring
Type: STRING
Mask: false
Size: 128
Input 3:
Name: inp-conn-username
Type: STRING
Mask: false
Size: 36
Input 4:
Name: inp-conn-password
Type: STRING
Mask: true
Size: 10
Input 5:
Name: inp-conn-jdbc-properties
Type: MAP
sqoop:000>
Thanks,
Jarek Cecho
Re: Review Request: SQOOP-544: Split metadata between import and export
related
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6508/#review10143
-----------------------------------------------------------
-1.
I've got second thoughts and I'm planning to reorganize this little bit - I believe that it makes more sense to have one connection metadata for all job types. The reasoning here is that administrator will create connection object (for example will set JDBC Url, password and username). Operator than should be able to reuse this single connection for different jobs (even for combination of imports and exports).
I'll rework my patch and update it shortly.
Jarcec
- Jarek Cecho
On Aug. 10, 2012, 7:20 a.m., Jarek Cecho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6508/
> -----------------------------------------------------------
>
> (Updated Aug. 10, 2012, 7:20 a.m.)
>
>
> Review request for Sqoop, Arvind Prabhakar and Bilung Lee.
>
>
> Description
> -------
>
> Sqoop framework needs to support multiple job types - mainly import and export job. As we're using word "job" for different purpose in the framework already, I've decided to introduce keyword "operation" for those job types. E.g. import and export operation. I do not insist on this particular word, so feel free to suggest better one.
>
> This patch provide all necessary changes on both client, server and repository side.
>
>
> This addresses bug SQOOP-544.
> https://issues.apache.org/jira/browse/SQOOP-544
>
>
> Diffs
> -----
>
> /branches/sqoop2/client/src/main/java/org/apache/sqoop/client/shell/ShowConnectorFunction.java 1371581
> /branches/sqoop2/common/src/main/java/org/apache/sqoop/job/etl/Operation.java PRE-CREATION
> /branches/sqoop2/common/src/main/java/org/apache/sqoop/json/ConnectorBean.java 1371581
> /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MConnector.java 1371581
> /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MFramework.java 1371581
> /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MOperation.java PRE-CREATION
> /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MStringInput.java 1371581
> /branches/sqoop2/common/src/test/java/org/apache/sqoop/json/TestConnectorBean.java PRE-CREATION
> /branches/sqoop2/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java 1371581
> /branches/sqoop2/connector/connector-mysql-jdbc/src/main/java/org/apache/sqoop/connector/mysqljdbc/MySqlJdbcConnector.java 1371581
> /branches/sqoop2/core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1371581
> /branches/sqoop2/core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java 1371581
> /branches/sqoop2/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java 1371581
> /branches/sqoop2/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaConstants.java 1371581
> /branches/sqoop2/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaQuery.java 1371581
> /branches/sqoop2/spi/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 1371581
>
> Diff: https://reviews.apache.org/r/6508/diff/
>
>
> Testing
> -------
>
> * I've added new unit test for testing ConnectorBean serialization&deserialization
> * I've manually verified (derby database, client side)
>
> Expected output for "show connector -a" is:
> sqoop:000> show connector -a
> 2 connector(s) to show:
> Connector with id 1:
> Name: mysql-jdbc-connector
> Class: org.apache.sqoop.connector.mysqljdbc.MySqlJdbcConnector
> Supported operations: []
> Connector with id 2:
> Name: generic-jdbc-connector
> Class: org.apache.sqoop.connector.jdbc.GenericJdbcConnector
> Supported operations: [EXPORT, IMPORT]
> Forms for operation EXPORT:
> Connection form 1:
> Name: form-connection
> Input 1:
> Name: inp-conn-jdbcdriver
> Type: STRING
> Mask: false
> Size: 128
> Input 2:
> Name: inp-conn-connectstring
> Type: STRING
> Mask: false
> Size: 128
> Input 3:
> Name: inp-conn-username
> Type: STRING
> Mask: false
> Size: 36
> Input 4:
> Name: inp-conn-password
> Type: STRING
> Mask: true
> Size: 10
> Input 5:
> Name: inp-conn-jdbc-properties
> Type: MAP
> Forms for operation IMPORT:
> Connection form 1:
> Name: form-connection
> Input 1:
> Name: inp-conn-jdbcdriver
> Type: STRING
> Mask: false
> Size: 128
> Input 2:
> Name: inp-conn-connectstring
> Type: STRING
> Mask: false
> Size: 128
> Input 3:
> Name: inp-conn-username
> Type: STRING
> Mask: false
> Size: 36
> Input 4:
> Name: inp-conn-password
> Type: STRING
> Mask: true
> Size: 10
> Input 5:
> Name: inp-conn-jdbc-properties
> Type: MAP
>
> sqoop:000>
>
>
> Thanks,
>
> Jarek Cecho
>
>