You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Szabolcs Vasas <va...@gmail.com> on 2017/11/17 16:27:04 UTC
Review Request 63920: Sqoop ignores metastore properties defined in
sqoop-site.xml
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63920/
-----------------------------------------------------------
Review request for Sqoop.
Bugs: SQOOP-3255
https://issues.apache.org/jira/browse/SQOOP-3255
Repository: sqoop-trunk
Description
-------
This patch restores the following metastore behavior:
If --meta-connect parameter is present then Sqoop will try to connect to the metastore database specified in this parameter value. It will use the username and password specified in --meta-username and --meta-password parameters if they are not present Sqoop will use empty username/password. If the database in the connection string is not supported, then Sqoop will default to HSQLDB metastore (See: org.apache.sqoop.metastore.GenericJobStorage#canAccept).
If --meta-connect parameter is not present Sqoop checks if sqoop.metastore.client.enable.autoconnect configuartion parameter is set to false. If the parameter is false then Sqoop throws an error since there are no applicable metastore implementations.
The next step is to check sqoop.metastore.client.autoconnect.url configuration parameter. In this parameter users can specify a JDBC connection string to a metastore database (HSQLDB, MySQL, Oracle, PostgreSQL, DB2 and SQL Server are supported) if it is not present Sqoop will create an embedded HSQLDB metastore in the home directory of the OS user which executes the sqoop command. The username and password can be specified in sqoop.metastore.client.autoconnect.username (default value: SA) and sqoop.metastore.client.autoconnect.password (default value is the empty string).
SqoopOptions:
- It turned out that the default value initialisation did not work, because the configuration parameters are not present in system properties just in the configuration object
- getLocalAutoConnectString is moved to AutoGenericJobStorage.getHomeDirFileConnectStr
AutoGenericJobStorage:
- The class is restored version of AutoHsqldbStorage.
- it extends GenericJobStorage
GenericJobStorage:
- the auto metastore parameters are moved to AutoGenericJobStorage
- driverClass field is removed since it can be derived from the metastoreConnectStr
- canAccept is modified
JobTool:
- SUPPORTED_DRIVERS is moved to GenericJobStorage since JobTool just delegates to job storages so the supported databases depend on the job storage implementation.
- applyMetastoreOptions now copies the parameter values from the CommandLine object to the SqoopObject instance.
- the storage descriptor map initialized in initializeStorageDescriptor now
Diffs
-----
src/java/org/apache/sqoop/SqoopOptions.java 587d4e1dd4012692ca7c742bcd518f55c10d1f8f
src/java/org/apache/sqoop/metastore/AutoGenericJobStorage.java PRE-CREATION
src/java/org/apache/sqoop/metastore/GenericJobStorage.java 9e1b18b6dad08991e11e41435bf62ea87b534f2f
src/java/org/apache/sqoop/metastore/JobStorageFactory.java 9a348d5181d20498dc89f68261302d65e041959a
src/java/org/apache/sqoop/tool/JobTool.java dbe89349c17ca86c3985c279cb3fe150e3bcab0e
src/test/com/cloudera/sqoop/TestIncrementalImport.java 166792b516008f34f7d20f899a0910ed19155f6b
src/test/com/cloudera/sqoop/metastore/SavedJobsTestBase.java 81789e76f2a5c9f85c38fed3e6fb7b791ed7299f
src/test/com/cloudera/sqoop/metastore/TestMetastoreConfigurationParameters.java PRE-CREATION
src/test/com/cloudera/sqoop/testutil/HsqldbTestServer.java 8d0a30db8e3ab0b364f8ccf43e643269d4eb0df0
src/test/org/apache/sqoop/metastore/TestAutoGenericJobStorageCanAccept.java PRE-CREATION
src/test/org/apache/sqoop/metastore/TestGenericJobStorageCanAccept.java PRE-CREATION
Diff: https://reviews.apache.org/r/63920/diff/1/
Testing
-------
Executed unit and third party test suite, created new test cases.
Thanks,
Szabolcs Vasas
Re: Review Request 63920: Sqoop ignores metastore properties defined
in sqoop-site.xml
Posted by Boglarka Egyed <bo...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63920/#review193388
-----------------------------------------------------------
Fix it, then Ship it!
Hi Szabi,
I checked your patch and it seems to be good however I have some minor findings, please find them below.
I ran unit and 3rd party tests with your patch succesfully.
Thanks,
Bogi
src/java/org/apache/sqoop/SqoopOptions.java
Lines 1087-1093 (original)
<https://reviews.apache.org/r/63920/#comment271948>
GenericJobStorage import is no longer used in this class after this change, please remove it too.
src/test/com/cloudera/sqoop/metastore/TestMetastoreConfigurationParameters.java
Lines 1 (patched)
<https://reviews.apache.org/r/63920/#comment271949>
Please add license info header to this new file too.
src/test/com/cloudera/sqoop/metastore/TestMetastoreConfigurationParameters.java
Lines 58-70 (patched)
<https://reviews.apache.org/r/63920/#comment271952>
Typo in test case and method names: "AutoAuto"
- Boglarka Egyed
On Dec. 8, 2017, 3:31 p.m., Szabolcs Vasas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63920/
> -----------------------------------------------------------
>
> (Updated Dec. 8, 2017, 3:31 p.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-3255
> https://issues.apache.org/jira/browse/SQOOP-3255
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This patch restores the following metastore behavior:
> If --meta-connect parameter is present then Sqoop will try to connect to the metastore database specified in this parameter value. It will use the username and password specified in --meta-username and --meta-password parameters if they are not present Sqoop will use empty username/password. If the database in the connection string is not supported, then Sqoop will default to HSQLDB metastore (See: org.apache.sqoop.metastore.GenericJobStorage#canAccept).
>
> If --meta-connect parameter is not present Sqoop checks if sqoop.metastore.client.enable.autoconnect configuartion parameter is set to false. If the parameter is false then Sqoop throws an error since there are no applicable metastore implementations.
> The next step is to check sqoop.metastore.client.autoconnect.url configuration parameter. In this parameter users can specify a JDBC connection string to a metastore database (HSQLDB, MySQL, Oracle, PostgreSQL, DB2 and SQL Server are supported) if it is not present Sqoop will create an embedded HSQLDB metastore in the home directory of the OS user which executes the sqoop command. The username and password can be specified in sqoop.metastore.client.autoconnect.username (default value: SA) and sqoop.metastore.client.autoconnect.password (default value is the empty string).
>
>
> SqoopOptions:
> - It turned out that the default value initialisation did not work, because the configuration parameters are not present in system properties just in the configuration object
> - getLocalAutoConnectString is moved to AutoGenericJobStorage.getHomeDirFileConnectStr
>
> AutoGenericJobStorage:
> - The class is restored version of AutoHsqldbStorage.
> - it extends GenericJobStorage
>
> GenericJobStorage:
> - the auto metastore parameters are moved to AutoGenericJobStorage
> - driverClass field is removed since it can be derived from the metastoreConnectStr
> - canAccept is modified
>
>
> JobTool:
> - SUPPORTED_DRIVERS is moved to GenericJobStorage since JobTool just delegates to job storages so the supported databases depend on the job storage implementation.
> - applyMetastoreOptions now copies the parameter values from the CommandLine object to the SqoopObject instance.
> - the storage descriptor map initialized in initializeStorageDescriptor now
>
>
> Diffs
> -----
>
> src/java/org/apache/sqoop/SqoopOptions.java d5fdfba14f5a2a22ae41ce4627b4e9afd13d5d28
> src/java/org/apache/sqoop/metastore/AutoGenericJobStorage.java PRE-CREATION
> src/java/org/apache/sqoop/metastore/GenericJobStorage.java e4ffde210d6b93eec4cb4d3c5b20e388971bf564
> src/java/org/apache/sqoop/metastore/JobStorageFactory.java 9a348d5181d20498dc89f68261302d65e041959a
> src/java/org/apache/sqoop/tool/JobTool.java 5b95c7d00643ed76d3e15799648ca40bda7f0b90
> src/test/com/cloudera/sqoop/TestIncrementalImport.java 166792b516008f34f7d20f899a0910ed19155f6b
> src/test/com/cloudera/sqoop/metastore/SavedJobsTestBase.java 81789e76f2a5c9f85c38fed3e6fb7b791ed7299f
> src/test/com/cloudera/sqoop/metastore/TestMetastoreConfigurationParameters.java PRE-CREATION
> src/test/com/cloudera/sqoop/testutil/HsqldbTestServer.java 8d0a30db8e3ab0b364f8ccf43e643269d4eb0df0
> src/test/org/apache/sqoop/metastore/TestAutoGenericJobStorageCanAccept.java PRE-CREATION
> src/test/org/apache/sqoop/metastore/TestGenericJobStorageCanAccept.java PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/63920/diff/2/
>
>
> Testing
> -------
>
> Executed unit and third party test suite, created new test cases.
>
>
> Thanks,
>
> Szabolcs Vasas
>
>
Re: Review Request 63920: Sqoop ignores metastore properties defined
in sqoop-site.xml
Posted by Boglarka Egyed <bo...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63920/#review194863
-----------------------------------------------------------
Ship it!
Thanks Szabi for improving the patch! Unit and 3rd party tests still pass.
- Boglarka Egyed
On Jan. 5, 2018, 10:05 a.m., Szabolcs Vasas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63920/
> -----------------------------------------------------------
>
> (Updated Jan. 5, 2018, 10:05 a.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-3255
> https://issues.apache.org/jira/browse/SQOOP-3255
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This patch restores the following metastore behavior:
> If --meta-connect parameter is present then Sqoop will try to connect to the metastore database specified in this parameter value. It will use the username and password specified in --meta-username and --meta-password parameters if they are not present Sqoop will use empty username/password. If the database in the connection string is not supported then Sqoop will throw an exception.
>
> If --meta-connect parameter is not present Sqoop checks if sqoop.metastore.client.enable.autoconnect configuartion parameter is set to false. If the parameter is false then Sqoop throws an error since there are no applicable metastore implementations.
> The next step is to check sqoop.metastore.client.autoconnect.url configuration parameter. In this parameter users can specify a JDBC connection string to a metastore database (HSQLDB, MySQL, Oracle, PostgreSQL, DB2 and SQL Server are supported) if it is not present Sqoop will create an embedded HSQLDB metastore in the home directory of the OS user which executes the sqoop command. The username and password can be specified in sqoop.metastore.client.autoconnect.username (default value: SA) and sqoop.metastore.client.autoconnect.password (default value is the empty string). If the connection string in sqoop.metastore.client.autoconnect.url is not supported then Sqoop will throw an exception.
>
>
> SqoopOptions:
> - It turned out that the default value initialisation did not work, because the configuration parameters are not present in system properties just in the configuration object
> - getLocalAutoConnectString is moved to AutoGenericJobStorage.getHomeDirFileConnectStr
>
> AutoGenericJobStorage:
> - The class is restored version of AutoHsqldbStorage.
> - it extends GenericJobStorage
>
> GenericJobStorage:
> - the auto metastore parameters are moved to AutoGenericJobStorage
> - driverClass field is removed since it can be derived from the metastoreConnectStr
> - canAccept is modified
>
>
> JobTool:
> - SUPPORTED_DRIVERS is moved to GenericJobStorage since JobTool just delegates to job storages so the supported databases depend on the job storage implementation.
> - applyMetastoreOptions now copies the parameter values from the CommandLine object to the SqoopObject instance.
> - the storage descriptor map initialized in initializeStorageDescriptor now
>
>
> Diffs
> -----
>
> src/java/org/apache/sqoop/SqoopOptions.java d5fdfba14
> src/java/org/apache/sqoop/metastore/AutoGenericJobStorage.java PRE-CREATION
> src/java/org/apache/sqoop/metastore/GenericJobStorage.java e4ffde210
> src/java/org/apache/sqoop/metastore/JobStorageFactory.java 9a348d518
> src/java/org/apache/sqoop/tool/JobTool.java 5b95c7d00
> src/test/com/cloudera/sqoop/TestIncrementalImport.java 166792b51
> src/test/com/cloudera/sqoop/metastore/SavedJobsTestBase.java 81789e76f
> src/test/com/cloudera/sqoop/metastore/TestMetastoreConfigurationParameters.java PRE-CREATION
> src/test/com/cloudera/sqoop/testutil/HsqldbTestServer.java 8d0a30db8
> src/test/org/apache/sqoop/metastore/TestAutoGenericJobStorage.java PRE-CREATION
> src/test/org/apache/sqoop/metastore/TestGenericJobStorage.java PRE-CREATION
> src/test/org/apache/sqoop/metastore/TestGenericJobStorageValidate.java PRE-CREATION
> src/test/org/apache/sqoop/testutil/Argument.java PRE-CREATION
> src/test/org/apache/sqoop/testutil/ArgumentUtils.java PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/63920/diff/6/
>
>
> Testing
> -------
>
> Executed unit and third party test suite, created new test cases.
>
>
> Thanks,
>
> Szabolcs Vasas
>
>
Re: Review Request 63920: Sqoop ignores metastore properties defined
in sqoop-site.xml
Posted by Fero Szabo via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63920/#review194847
-----------------------------------------------------------
Ship it!
Ship It!
- Fero Szabo
On Jan. 5, 2018, 10:05 a.m., Szabolcs Vasas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63920/
> -----------------------------------------------------------
>
> (Updated Jan. 5, 2018, 10:05 a.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-3255
> https://issues.apache.org/jira/browse/SQOOP-3255
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This patch restores the following metastore behavior:
> If --meta-connect parameter is present then Sqoop will try to connect to the metastore database specified in this parameter value. It will use the username and password specified in --meta-username and --meta-password parameters if they are not present Sqoop will use empty username/password. If the database in the connection string is not supported then Sqoop will throw an exception.
>
> If --meta-connect parameter is not present Sqoop checks if sqoop.metastore.client.enable.autoconnect configuartion parameter is set to false. If the parameter is false then Sqoop throws an error since there are no applicable metastore implementations.
> The next step is to check sqoop.metastore.client.autoconnect.url configuration parameter. In this parameter users can specify a JDBC connection string to a metastore database (HSQLDB, MySQL, Oracle, PostgreSQL, DB2 and SQL Server are supported) if it is not present Sqoop will create an embedded HSQLDB metastore in the home directory of the OS user which executes the sqoop command. The username and password can be specified in sqoop.metastore.client.autoconnect.username (default value: SA) and sqoop.metastore.client.autoconnect.password (default value is the empty string). If the connection string in sqoop.metastore.client.autoconnect.url is not supported then Sqoop will throw an exception.
>
>
> SqoopOptions:
> - It turned out that the default value initialisation did not work, because the configuration parameters are not present in system properties just in the configuration object
> - getLocalAutoConnectString is moved to AutoGenericJobStorage.getHomeDirFileConnectStr
>
> AutoGenericJobStorage:
> - The class is restored version of AutoHsqldbStorage.
> - it extends GenericJobStorage
>
> GenericJobStorage:
> - the auto metastore parameters are moved to AutoGenericJobStorage
> - driverClass field is removed since it can be derived from the metastoreConnectStr
> - canAccept is modified
>
>
> JobTool:
> - SUPPORTED_DRIVERS is moved to GenericJobStorage since JobTool just delegates to job storages so the supported databases depend on the job storage implementation.
> - applyMetastoreOptions now copies the parameter values from the CommandLine object to the SqoopObject instance.
> - the storage descriptor map initialized in initializeStorageDescriptor now
>
>
> Diffs
> -----
>
> src/java/org/apache/sqoop/SqoopOptions.java d5fdfba14
> src/java/org/apache/sqoop/metastore/AutoGenericJobStorage.java PRE-CREATION
> src/java/org/apache/sqoop/metastore/GenericJobStorage.java e4ffde210
> src/java/org/apache/sqoop/metastore/JobStorageFactory.java 9a348d518
> src/java/org/apache/sqoop/tool/JobTool.java 5b95c7d00
> src/test/com/cloudera/sqoop/TestIncrementalImport.java 166792b51
> src/test/com/cloudera/sqoop/metastore/SavedJobsTestBase.java 81789e76f
> src/test/com/cloudera/sqoop/metastore/TestMetastoreConfigurationParameters.java PRE-CREATION
> src/test/com/cloudera/sqoop/testutil/HsqldbTestServer.java 8d0a30db8
> src/test/org/apache/sqoop/metastore/TestAutoGenericJobStorage.java PRE-CREATION
> src/test/org/apache/sqoop/metastore/TestGenericJobStorage.java PRE-CREATION
> src/test/org/apache/sqoop/metastore/TestGenericJobStorageValidate.java PRE-CREATION
> src/test/org/apache/sqoop/testutil/Argument.java PRE-CREATION
> src/test/org/apache/sqoop/testutil/ArgumentUtils.java PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/63920/diff/6/
>
>
> Testing
> -------
>
> Executed unit and third party test suite, created new test cases.
>
>
> Thanks,
>
> Szabolcs Vasas
>
>
Re: Review Request 63920: Sqoop ignores metastore properties defined
in sqoop-site.xml
Posted by Szabolcs Vasas <va...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63920/
-----------------------------------------------------------
(Updated Jan. 5, 2018, 10:05 a.m.)
Review request for Sqoop.
Bugs: SQOOP-3255
https://issues.apache.org/jira/browse/SQOOP-3255
Repository: sqoop-trunk
Description
-------
This patch restores the following metastore behavior:
If --meta-connect parameter is present then Sqoop will try to connect to the metastore database specified in this parameter value. It will use the username and password specified in --meta-username and --meta-password parameters if they are not present Sqoop will use empty username/password. If the database in the connection string is not supported then Sqoop will throw an exception.
If --meta-connect parameter is not present Sqoop checks if sqoop.metastore.client.enable.autoconnect configuartion parameter is set to false. If the parameter is false then Sqoop throws an error since there are no applicable metastore implementations.
The next step is to check sqoop.metastore.client.autoconnect.url configuration parameter. In this parameter users can specify a JDBC connection string to a metastore database (HSQLDB, MySQL, Oracle, PostgreSQL, DB2 and SQL Server are supported) if it is not present Sqoop will create an embedded HSQLDB metastore in the home directory of the OS user which executes the sqoop command. The username and password can be specified in sqoop.metastore.client.autoconnect.username (default value: SA) and sqoop.metastore.client.autoconnect.password (default value is the empty string). If the connection string in sqoop.metastore.client.autoconnect.url is not supported then Sqoop will throw an exception.
SqoopOptions:
- It turned out that the default value initialisation did not work, because the configuration parameters are not present in system properties just in the configuration object
- getLocalAutoConnectString is moved to AutoGenericJobStorage.getHomeDirFileConnectStr
AutoGenericJobStorage:
- The class is restored version of AutoHsqldbStorage.
- it extends GenericJobStorage
GenericJobStorage:
- the auto metastore parameters are moved to AutoGenericJobStorage
- driverClass field is removed since it can be derived from the metastoreConnectStr
- canAccept is modified
JobTool:
- SUPPORTED_DRIVERS is moved to GenericJobStorage since JobTool just delegates to job storages so the supported databases depend on the job storage implementation.
- applyMetastoreOptions now copies the parameter values from the CommandLine object to the SqoopObject instance.
- the storage descriptor map initialized in initializeStorageDescriptor now
Diffs (updated)
-----
src/java/org/apache/sqoop/SqoopOptions.java d5fdfba14
src/java/org/apache/sqoop/metastore/AutoGenericJobStorage.java PRE-CREATION
src/java/org/apache/sqoop/metastore/GenericJobStorage.java e4ffde210
src/java/org/apache/sqoop/metastore/JobStorageFactory.java 9a348d518
src/java/org/apache/sqoop/tool/JobTool.java 5b95c7d00
src/test/com/cloudera/sqoop/TestIncrementalImport.java 166792b51
src/test/com/cloudera/sqoop/metastore/SavedJobsTestBase.java 81789e76f
src/test/com/cloudera/sqoop/metastore/TestMetastoreConfigurationParameters.java PRE-CREATION
src/test/com/cloudera/sqoop/testutil/HsqldbTestServer.java 8d0a30db8
src/test/org/apache/sqoop/metastore/TestAutoGenericJobStorage.java PRE-CREATION
src/test/org/apache/sqoop/metastore/TestGenericJobStorage.java PRE-CREATION
src/test/org/apache/sqoop/metastore/TestGenericJobStorageValidate.java PRE-CREATION
src/test/org/apache/sqoop/testutil/Argument.java PRE-CREATION
src/test/org/apache/sqoop/testutil/ArgumentUtils.java PRE-CREATION
Diff: https://reviews.apache.org/r/63920/diff/6/
Changes: https://reviews.apache.org/r/63920/diff/5-6/
Testing
-------
Executed unit and third party test suite, created new test cases.
Thanks,
Szabolcs Vasas
Re: Review Request 63920: Sqoop ignores metastore properties defined
in sqoop-site.xml
Posted by Szabolcs Vasas <va...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63920/
-----------------------------------------------------------
(Updated Jan. 5, 2018, 9:46 a.m.)
Review request for Sqoop.
Bugs: SQOOP-3255
https://issues.apache.org/jira/browse/SQOOP-3255
Repository: sqoop-trunk
Description
-------
This patch restores the following metastore behavior:
If --meta-connect parameter is present then Sqoop will try to connect to the metastore database specified in this parameter value. It will use the username and password specified in --meta-username and --meta-password parameters if they are not present Sqoop will use empty username/password. If the database in the connection string is not supported then Sqoop will throw an exception.
If --meta-connect parameter is not present Sqoop checks if sqoop.metastore.client.enable.autoconnect configuartion parameter is set to false. If the parameter is false then Sqoop throws an error since there are no applicable metastore implementations.
The next step is to check sqoop.metastore.client.autoconnect.url configuration parameter. In this parameter users can specify a JDBC connection string to a metastore database (HSQLDB, MySQL, Oracle, PostgreSQL, DB2 and SQL Server are supported) if it is not present Sqoop will create an embedded HSQLDB metastore in the home directory of the OS user which executes the sqoop command. The username and password can be specified in sqoop.metastore.client.autoconnect.username (default value: SA) and sqoop.metastore.client.autoconnect.password (default value is the empty string). If the connection string in sqoop.metastore.client.autoconnect.url is not supported then Sqoop will throw an exception.
SqoopOptions:
- It turned out that the default value initialisation did not work, because the configuration parameters are not present in system properties just in the configuration object
- getLocalAutoConnectString is moved to AutoGenericJobStorage.getHomeDirFileConnectStr
AutoGenericJobStorage:
- The class is restored version of AutoHsqldbStorage.
- it extends GenericJobStorage
GenericJobStorage:
- the auto metastore parameters are moved to AutoGenericJobStorage
- driverClass field is removed since it can be derived from the metastoreConnectStr
- canAccept is modified
JobTool:
- SUPPORTED_DRIVERS is moved to GenericJobStorage since JobTool just delegates to job storages so the supported databases depend on the job storage implementation.
- applyMetastoreOptions now copies the parameter values from the CommandLine object to the SqoopObject instance.
- the storage descriptor map initialized in initializeStorageDescriptor now
Diffs (updated)
-----
src/java/org/apache/sqoop/SqoopOptions.java d5fdfba14
src/java/org/apache/sqoop/metastore/AutoGenericJobStorage.java PRE-CREATION
src/java/org/apache/sqoop/metastore/GenericJobStorage.java e4ffde210
src/java/org/apache/sqoop/metastore/JobStorageFactory.java 9a348d518
src/java/org/apache/sqoop/tool/JobTool.java 5b95c7d00
src/test/com/cloudera/sqoop/TestIncrementalImport.java 166792b51
src/test/com/cloudera/sqoop/metastore/SavedJobsTestBase.java 81789e76f
src/test/com/cloudera/sqoop/metastore/TestMetastoreConfigurationParameters.java PRE-CREATION
src/test/com/cloudera/sqoop/testutil/HsqldbTestServer.java 8d0a30db8
src/test/org/apache/sqoop/metastore/TestAutoGenericJobStorage.java PRE-CREATION
src/test/org/apache/sqoop/metastore/TestGenericJobStorage.java PRE-CREATION
src/test/org/apache/sqoop/metastore/TestGenericJobStorageValidate.java PRE-CREATION
src/test/org/apache/sqoop/testutil/Argument.java PRE-CREATION
src/test/org/apache/sqoop/testutil/ArgumentUtils.java PRE-CREATION
Diff: https://reviews.apache.org/r/63920/diff/5/
Changes: https://reviews.apache.org/r/63920/diff/4-5/
Testing
-------
Executed unit and third party test suite, created new test cases.
Thanks,
Szabolcs Vasas
Re: Review Request 63920: Sqoop ignores metastore properties defined
in sqoop-site.xml
Posted by Szabolcs Vasas <va...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63920/
-----------------------------------------------------------
(Updated Jan. 3, 2018, 5:10 p.m.)
Review request for Sqoop.
Bugs: SQOOP-3255
https://issues.apache.org/jira/browse/SQOOP-3255
Repository: sqoop-trunk
Description (updated)
-------
This patch restores the following metastore behavior:
If --meta-connect parameter is present then Sqoop will try to connect to the metastore database specified in this parameter value. It will use the username and password specified in --meta-username and --meta-password parameters if they are not present Sqoop will use empty username/password. If the database in the connection string is not supported then Sqoop will throw an exception.
If --meta-connect parameter is not present Sqoop checks if sqoop.metastore.client.enable.autoconnect configuartion parameter is set to false. If the parameter is false then Sqoop throws an error since there are no applicable metastore implementations.
The next step is to check sqoop.metastore.client.autoconnect.url configuration parameter. In this parameter users can specify a JDBC connection string to a metastore database (HSQLDB, MySQL, Oracle, PostgreSQL, DB2 and SQL Server are supported) if it is not present Sqoop will create an embedded HSQLDB metastore in the home directory of the OS user which executes the sqoop command. The username and password can be specified in sqoop.metastore.client.autoconnect.username (default value: SA) and sqoop.metastore.client.autoconnect.password (default value is the empty string). If the connection string in sqoop.metastore.client.autoconnect.url is not supported then Sqoop will throw an exception.
SqoopOptions:
- It turned out that the default value initialisation did not work, because the configuration parameters are not present in system properties just in the configuration object
- getLocalAutoConnectString is moved to AutoGenericJobStorage.getHomeDirFileConnectStr
AutoGenericJobStorage:
- The class is restored version of AutoHsqldbStorage.
- it extends GenericJobStorage
GenericJobStorage:
- the auto metastore parameters are moved to AutoGenericJobStorage
- driverClass field is removed since it can be derived from the metastoreConnectStr
- canAccept is modified
JobTool:
- SUPPORTED_DRIVERS is moved to GenericJobStorage since JobTool just delegates to job storages so the supported databases depend on the job storage implementation.
- applyMetastoreOptions now copies the parameter values from the CommandLine object to the SqoopObject instance.
- the storage descriptor map initialized in initializeStorageDescriptor now
Diffs (updated)
-----
src/java/org/apache/sqoop/SqoopOptions.java d5fdfba14
src/java/org/apache/sqoop/metastore/AutoGenericJobStorage.java PRE-CREATION
src/java/org/apache/sqoop/metastore/GenericJobStorage.java e4ffde210
src/java/org/apache/sqoop/metastore/JobStorageFactory.java 9a348d518
src/java/org/apache/sqoop/tool/JobTool.java 5b95c7d00
src/test/com/cloudera/sqoop/TestIncrementalImport.java 166792b51
src/test/com/cloudera/sqoop/metastore/SavedJobsTestBase.java 81789e76f
src/test/com/cloudera/sqoop/metastore/TestMetastoreConfigurationParameters.java PRE-CREATION
src/test/com/cloudera/sqoop/testutil/HsqldbTestServer.java 8d0a30db8
src/test/org/apache/sqoop/metastore/TestAutoGenericJobStorage.java PRE-CREATION
src/test/org/apache/sqoop/metastore/TestGenericJobStorage.java PRE-CREATION
src/test/org/apache/sqoop/metastore/TestGenericJobStorageValidate.java PRE-CREATION
Diff: https://reviews.apache.org/r/63920/diff/4/
Changes: https://reviews.apache.org/r/63920/diff/3-4/
Testing
-------
Executed unit and third party test suite, created new test cases.
Thanks,
Szabolcs Vasas
Re: Review Request 63920: Sqoop ignores metastore properties defined
in sqoop-site.xml
Posted by Szabolcs Vasas <va...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63920/
-----------------------------------------------------------
(Updated Dec. 12, 2017, 2:31 p.m.)
Review request for Sqoop.
Bugs: SQOOP-3255
https://issues.apache.org/jira/browse/SQOOP-3255
Repository: sqoop-trunk
Description
-------
This patch restores the following metastore behavior:
If --meta-connect parameter is present then Sqoop will try to connect to the metastore database specified in this parameter value. It will use the username and password specified in --meta-username and --meta-password parameters if they are not present Sqoop will use empty username/password. If the database in the connection string is not supported, then Sqoop will default to HSQLDB metastore (See: org.apache.sqoop.metastore.GenericJobStorage#canAccept).
If --meta-connect parameter is not present Sqoop checks if sqoop.metastore.client.enable.autoconnect configuartion parameter is set to false. If the parameter is false then Sqoop throws an error since there are no applicable metastore implementations.
The next step is to check sqoop.metastore.client.autoconnect.url configuration parameter. In this parameter users can specify a JDBC connection string to a metastore database (HSQLDB, MySQL, Oracle, PostgreSQL, DB2 and SQL Server are supported) if it is not present Sqoop will create an embedded HSQLDB metastore in the home directory of the OS user which executes the sqoop command. The username and password can be specified in sqoop.metastore.client.autoconnect.username (default value: SA) and sqoop.metastore.client.autoconnect.password (default value is the empty string).
SqoopOptions:
- It turned out that the default value initialisation did not work, because the configuration parameters are not present in system properties just in the configuration object
- getLocalAutoConnectString is moved to AutoGenericJobStorage.getHomeDirFileConnectStr
AutoGenericJobStorage:
- The class is restored version of AutoHsqldbStorage.
- it extends GenericJobStorage
GenericJobStorage:
- the auto metastore parameters are moved to AutoGenericJobStorage
- driverClass field is removed since it can be derived from the metastoreConnectStr
- canAccept is modified
JobTool:
- SUPPORTED_DRIVERS is moved to GenericJobStorage since JobTool just delegates to job storages so the supported databases depend on the job storage implementation.
- applyMetastoreOptions now copies the parameter values from the CommandLine object to the SqoopObject instance.
- the storage descriptor map initialized in initializeStorageDescriptor now
Diffs (updated)
-----
src/java/org/apache/sqoop/SqoopOptions.java d5fdfba14f5a2a22ae41ce4627b4e9afd13d5d28
src/java/org/apache/sqoop/metastore/AutoGenericJobStorage.java PRE-CREATION
src/java/org/apache/sqoop/metastore/GenericJobStorage.java e4ffde210d6b93eec4cb4d3c5b20e388971bf564
src/java/org/apache/sqoop/metastore/JobStorageFactory.java 9a348d5181d20498dc89f68261302d65e041959a
src/java/org/apache/sqoop/tool/JobTool.java 5b95c7d00643ed76d3e15799648ca40bda7f0b90
src/test/com/cloudera/sqoop/TestIncrementalImport.java 166792b516008f34f7d20f899a0910ed19155f6b
src/test/com/cloudera/sqoop/metastore/SavedJobsTestBase.java 81789e76f2a5c9f85c38fed3e6fb7b791ed7299f
src/test/com/cloudera/sqoop/metastore/TestMetastoreConfigurationParameters.java PRE-CREATION
src/test/com/cloudera/sqoop/testutil/HsqldbTestServer.java 8d0a30db8e3ab0b364f8ccf43e643269d4eb0df0
src/test/org/apache/sqoop/metastore/TestAutoGenericJobStorageCanAccept.java PRE-CREATION
src/test/org/apache/sqoop/metastore/TestGenericJobStorageCanAccept.java PRE-CREATION
Diff: https://reviews.apache.org/r/63920/diff/3/
Changes: https://reviews.apache.org/r/63920/diff/2-3/
Testing
-------
Executed unit and third party test suite, created new test cases.
Thanks,
Szabolcs Vasas
Re: Review Request 63920: Sqoop ignores metastore properties defined
in sqoop-site.xml
Posted by Szabolcs Vasas <va...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63920/
-----------------------------------------------------------
(Updated Dec. 8, 2017, 3:31 p.m.)
Review request for Sqoop.
Changes
-------
White space errors are fixed, rebased to the latest trunk.
Bugs: SQOOP-3255
https://issues.apache.org/jira/browse/SQOOP-3255
Repository: sqoop-trunk
Description
-------
This patch restores the following metastore behavior:
If --meta-connect parameter is present then Sqoop will try to connect to the metastore database specified in this parameter value. It will use the username and password specified in --meta-username and --meta-password parameters if they are not present Sqoop will use empty username/password. If the database in the connection string is not supported, then Sqoop will default to HSQLDB metastore (See: org.apache.sqoop.metastore.GenericJobStorage#canAccept).
If --meta-connect parameter is not present Sqoop checks if sqoop.metastore.client.enable.autoconnect configuartion parameter is set to false. If the parameter is false then Sqoop throws an error since there are no applicable metastore implementations.
The next step is to check sqoop.metastore.client.autoconnect.url configuration parameter. In this parameter users can specify a JDBC connection string to a metastore database (HSQLDB, MySQL, Oracle, PostgreSQL, DB2 and SQL Server are supported) if it is not present Sqoop will create an embedded HSQLDB metastore in the home directory of the OS user which executes the sqoop command. The username and password can be specified in sqoop.metastore.client.autoconnect.username (default value: SA) and sqoop.metastore.client.autoconnect.password (default value is the empty string).
SqoopOptions:
- It turned out that the default value initialisation did not work, because the configuration parameters are not present in system properties just in the configuration object
- getLocalAutoConnectString is moved to AutoGenericJobStorage.getHomeDirFileConnectStr
AutoGenericJobStorage:
- The class is restored version of AutoHsqldbStorage.
- it extends GenericJobStorage
GenericJobStorage:
- the auto metastore parameters are moved to AutoGenericJobStorage
- driverClass field is removed since it can be derived from the metastoreConnectStr
- canAccept is modified
JobTool:
- SUPPORTED_DRIVERS is moved to GenericJobStorage since JobTool just delegates to job storages so the supported databases depend on the job storage implementation.
- applyMetastoreOptions now copies the parameter values from the CommandLine object to the SqoopObject instance.
- the storage descriptor map initialized in initializeStorageDescriptor now
Diffs (updated)
-----
src/java/org/apache/sqoop/SqoopOptions.java d5fdfba14f5a2a22ae41ce4627b4e9afd13d5d28
src/java/org/apache/sqoop/metastore/AutoGenericJobStorage.java PRE-CREATION
src/java/org/apache/sqoop/metastore/GenericJobStorage.java e4ffde210d6b93eec4cb4d3c5b20e388971bf564
src/java/org/apache/sqoop/metastore/JobStorageFactory.java 9a348d5181d20498dc89f68261302d65e041959a
src/java/org/apache/sqoop/tool/JobTool.java 5b95c7d00643ed76d3e15799648ca40bda7f0b90
src/test/com/cloudera/sqoop/TestIncrementalImport.java 166792b516008f34f7d20f899a0910ed19155f6b
src/test/com/cloudera/sqoop/metastore/SavedJobsTestBase.java 81789e76f2a5c9f85c38fed3e6fb7b791ed7299f
src/test/com/cloudera/sqoop/metastore/TestMetastoreConfigurationParameters.java PRE-CREATION
src/test/com/cloudera/sqoop/testutil/HsqldbTestServer.java 8d0a30db8e3ab0b364f8ccf43e643269d4eb0df0
src/test/org/apache/sqoop/metastore/TestAutoGenericJobStorageCanAccept.java PRE-CREATION
src/test/org/apache/sqoop/metastore/TestGenericJobStorageCanAccept.java PRE-CREATION
Diff: https://reviews.apache.org/r/63920/diff/2/
Changes: https://reviews.apache.org/r/63920/diff/1-2/
Testing
-------
Executed unit and third party test suite, created new test cases.
Thanks,
Szabolcs Vasas
Re: Review Request 63920: Sqoop ignores metastore properties defined
in sqoop-site.xml
Posted by Boglarka Egyed <bo...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63920/#review193249
-----------------------------------------------------------
Hi Szabi,
I was not able to apply your patch, please see the details below. Could you please regenerate it?
Thanks,
Bogi
src/java/org/apache/sqoop/metastore/GenericJobStorage.java
Line 858 (original), 837 (patched)
<https://reviews.apache.org/r/63920/#comment271799>
Your patch can not be applied because of this blank line at EOF.
- Boglarka Egyed
On Nov. 17, 2017, 4:27 p.m., Szabolcs Vasas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63920/
> -----------------------------------------------------------
>
> (Updated Nov. 17, 2017, 4:27 p.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-3255
> https://issues.apache.org/jira/browse/SQOOP-3255
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This patch restores the following metastore behavior:
> If --meta-connect parameter is present then Sqoop will try to connect to the metastore database specified in this parameter value. It will use the username and password specified in --meta-username and --meta-password parameters if they are not present Sqoop will use empty username/password. If the database in the connection string is not supported, then Sqoop will default to HSQLDB metastore (See: org.apache.sqoop.metastore.GenericJobStorage#canAccept).
>
> If --meta-connect parameter is not present Sqoop checks if sqoop.metastore.client.enable.autoconnect configuartion parameter is set to false. If the parameter is false then Sqoop throws an error since there are no applicable metastore implementations.
> The next step is to check sqoop.metastore.client.autoconnect.url configuration parameter. In this parameter users can specify a JDBC connection string to a metastore database (HSQLDB, MySQL, Oracle, PostgreSQL, DB2 and SQL Server are supported) if it is not present Sqoop will create an embedded HSQLDB metastore in the home directory of the OS user which executes the sqoop command. The username and password can be specified in sqoop.metastore.client.autoconnect.username (default value: SA) and sqoop.metastore.client.autoconnect.password (default value is the empty string).
>
>
> SqoopOptions:
> - It turned out that the default value initialisation did not work, because the configuration parameters are not present in system properties just in the configuration object
> - getLocalAutoConnectString is moved to AutoGenericJobStorage.getHomeDirFileConnectStr
>
> AutoGenericJobStorage:
> - The class is restored version of AutoHsqldbStorage.
> - it extends GenericJobStorage
>
> GenericJobStorage:
> - the auto metastore parameters are moved to AutoGenericJobStorage
> - driverClass field is removed since it can be derived from the metastoreConnectStr
> - canAccept is modified
>
>
> JobTool:
> - SUPPORTED_DRIVERS is moved to GenericJobStorage since JobTool just delegates to job storages so the supported databases depend on the job storage implementation.
> - applyMetastoreOptions now copies the parameter values from the CommandLine object to the SqoopObject instance.
> - the storage descriptor map initialized in initializeStorageDescriptor now
>
>
> Diffs
> -----
>
> src/java/org/apache/sqoop/SqoopOptions.java 587d4e1dd4012692ca7c742bcd518f55c10d1f8f
> src/java/org/apache/sqoop/metastore/AutoGenericJobStorage.java PRE-CREATION
> src/java/org/apache/sqoop/metastore/GenericJobStorage.java 9e1b18b6dad08991e11e41435bf62ea87b534f2f
> src/java/org/apache/sqoop/metastore/JobStorageFactory.java 9a348d5181d20498dc89f68261302d65e041959a
> src/java/org/apache/sqoop/tool/JobTool.java dbe89349c17ca86c3985c279cb3fe150e3bcab0e
> src/test/com/cloudera/sqoop/TestIncrementalImport.java 166792b516008f34f7d20f899a0910ed19155f6b
> src/test/com/cloudera/sqoop/metastore/SavedJobsTestBase.java 81789e76f2a5c9f85c38fed3e6fb7b791ed7299f
> src/test/com/cloudera/sqoop/metastore/TestMetastoreConfigurationParameters.java PRE-CREATION
> src/test/com/cloudera/sqoop/testutil/HsqldbTestServer.java 8d0a30db8e3ab0b364f8ccf43e643269d4eb0df0
> src/test/org/apache/sqoop/metastore/TestAutoGenericJobStorageCanAccept.java PRE-CREATION
> src/test/org/apache/sqoop/metastore/TestGenericJobStorageCanAccept.java PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/63920/diff/1/
>
>
> Testing
> -------
>
> Executed unit and third party test suite, created new test cases.
>
>
> Thanks,
>
> Szabolcs Vasas
>
>