You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by zachb1217 <gi...@git.apache.org> on 2017/07/24 12:15:34 UTC
[GitHub] sqoop pull request #38: Metastore
GitHub user zachb1217 opened a pull request:
https://github.com/apache/sqoop/pull/38
Metastore
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zachb1217/sqoop metastore
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/sqoop/pull/38.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #38
----
commit 5a23574f21a4030388d5fcdf4d4b53bfedea665a
Author: Zach Berkowitz <za...@cloudera.com>
Date: 2017-07-18T08:03:30Z
Changed HsqldbJobStorage class to handle six types of databases, created a parameterized test
commit 4d52c3a55ff9115b103beddaec2030e735b44da5
Author: Zach Berkowitz <za...@cloudera.com>
Date: 2017-07-18T12:16:03Z
Removed Cloudera DBs from test, renamed test to fit with convention, refactored db schema and driver classes to constants, added driver selection method into JobTool, removed TestIncremental import dependencies on SavedJobsTest
commit 4ae8ed039bf574512bd25bb97759cebc6d1bf604
Author: Zach Berkowitz <za...@cloudera.com>
Date: 2017-07-19T15:51:16Z
AutoHsqldbStorage is configurable for different databases, fixed bug with oracle connect string, added command-line args to specify meta-connect username and password, created command-line test for metastore
commit 3531d58b434ae41d3e68cc3066d2474091759c78
Author: Zach Berkowitz <za...@cloudera.com>
Date: 2017-07-20T14:51:52Z
Added a test for Exec Job
commit 4ee4d1e1195523b2df82d9d706e50cec03e67745
Author: Zach Berkowitz <za...@cloudera.com>
Date: 2017-07-24T12:12:31Z
Final Commit
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] sqoop pull request #38: Metastore
Posted by szvasas <gi...@git.apache.org>.
Github user szvasas commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/38#discussion_r130806000
--- Diff: src/docs/user/metastore-purpose.txt ---
@@ -22,6 +22,7 @@ Multiple users and/or remote users can define and execute saved jobs (created
with +sqoop job+) defined in this metastore.
Clients must be configured to connect to the metastore in +sqoop-site.xml+ or
-with the +--meta-connect+ argument.
+with the +--meta-connect+ argument. Supports MySql, Hsqldb, PostgreSql, Oracle, DB2,
+and SqlServer databses.
--- End diff --
Typo.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] sqoop pull request #38: Metastore
Posted by szvasas <gi...@git.apache.org>.
Github user szvasas commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/38#discussion_r130059198
--- Diff: src/java/org/apache/sqoop/metastore/GenericJobStorage.java ---
@@ -113,11 +121,14 @@
private static final String SQOOP_TOOL_KEY = "sqoop.tool";
--- End diff --
Can we remove these extra empty lines?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] sqoop pull request #38: Metastore
Posted by szvasas <gi...@git.apache.org>.
Github user szvasas commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/38#discussion_r130062375
--- Diff: src/java/org/apache/sqoop/tool/JobTool.java ---
@@ -346,8 +361,21 @@ public void applyOptions(CommandLine in, SqoopOptions out)
this.storageDescriptor = new TreeMap<String, String>();
if (in.hasOption(STORAGE_METASTORE_ARG)) {
- this.storageDescriptor.put(HsqldbJobStorage.META_CONNECT_KEY,
- in.getOptionValue(STORAGE_METASTORE_ARG));
+ String metaConnectString = in.getOptionValue(STORAGE_METASTORE_ARG);
+ this.storageDescriptor.put(GenericJobStorage.META_CONNECT_KEY, metaConnectString);
+
+ String metaUserString = in.getOptionValue(METASTORE_USER_ARG, null);
+ this.storageDescriptor.put(GenericJobStorage.META_USERNAME_KEY, metaUserString);
+
+ String metaPassString = in.getOptionValue(METASTORE_PASS_ARG, null);
+ this.storageDescriptor.put(GenericJobStorage.META_PASSWORD_KEY, metaPassString);
+
+ String driverString = chooseDriverType(metaConnectString);
+ this.storageDescriptor.put(GenericJobStorage.META_DRIVER_KEY, driverString);
+ }
+ else if (in.hasOption(METASTORE_USER_ARG) || in.hasOption(METASTORE_PASS_ARG)) {
--- End diff --
I think this validation should go in validateOptions method.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] sqoop pull request #38: Metastore
Posted by szvasas <gi...@git.apache.org>.
Github user szvasas commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/38#discussion_r130063002
--- Diff: src/test/com/cloudera/sqoop/metastore/SavedJobsTest.java ---
@@ -0,0 +1,379 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.cloudera.sqoop.metastore;
+
+import java.sql.SQLException;
+import java.sql.Statement;
+
+import java.util.*;
+
+import com.cloudera.sqoop.manager.ConnManager;
+import com.cloudera.sqoop.manager.MySQLTestUtils;
+import com.cloudera.sqoop.manager.OracleUtils;
+import org.apache.hadoop.conf.Configuration;
+
+import com.cloudera.sqoop.SqoopOptions;
+import com.cloudera.sqoop.tool.VersionTool;
+import org.apache.sqoop.manager.*;
+import org.apache.sqoop.manager.sqlserver.MSSQLTestUtils;
+import org.apache.sqoop.tool.ImportTool;
+import org.junit.*;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.sql.Connection;
+
+import static org.apache.sqoop.metastore.GenericJobStorage.*;
+import static org.hamcrest.core.IsCollectionContaining.hasItems;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertThat;
+
+/**
+ * Test the metastore and job-handling features.
+ *
+ * These all make use of the auto-connect hsqldb-based metastore.
+ * The metastore URL is configured to be in-memory, and drop all
+ * state between individual tests.
+ */
+@RunWith(Parameterized.class)
+public class SavedJobsTest {
+
+ private static MySQLTestUtils mySQLTestUtils = new MySQLTestUtils();
+ private static MSSQLTestUtils msSQLTestUtils = new MSSQLTestUtils();
+
+ @Parameterized.Parameters(name = "metaConnect = {0}, metaUser = {1}, metaPassword = {2}, driverClass = {3}")
+ public static Iterable<? extends Object> dbConnectParameters() {
+ return Arrays.asList(
+ new Object[] {
+ mySQLTestUtils.getHostUrl(), mySQLTestUtils.getUserName(),
+ mySQLTestUtils.getUserPass(), MySQLManager.DRIVER_CLASS
+ },
+ new Object[] {
+ OracleUtils.CONNECT_STRING, OracleUtils.ORACLE_USER_NAME,
+ OracleUtils.ORACLE_USER_PASS, OracleManager.DRIVER_CLASS
+ },
+ new Object[] {
+ msSQLTestUtils.getDBConnectString(), msSQLTestUtils.getDBUserName(),
+ msSQLTestUtils.getDBPassWord(), SQLServerManager.DRIVER_CLASS
+ },
+ new Object[] {
+ System.getProperty(
+ "sqoop.test.postgresql.connectstring.host_url",
+ "jdbc:postgresql://localhost/"),
+ System.getProperty(
+ "sqoop.test.postgresql.connectstring.username",
+ "sqooptest"),
+ System.getProperty(
+ "sqoop.test.postgresql.connectstring.password"),
+ PostgresqlManager.DRIVER_CLASS
+ },
+ new Object[] {
+ System.getProperty(
+ "sqoop.test.db2.connectstring.host_url",
+ "jdbc:db2://db2host:50000"),
+ System.getProperty(
+ "sqoop.test.db2.connectstring.username",
+ "SQOOP"),
+ System.getProperty(
+ "sqoop.test.db2.connectstring.password",
+ "SQOOP"),
+ Db2Manager.DRIVER_CLASS
+ },
+ new Object[] { "jdbc:hsqldb:mem:sqoopmetastore", "SA" , "", HsqldbManager.DRIVER_CLASS }
+ );
+ }
+
+ private String metaConnect;
+ private String metaUser;
+ private String metaPassword;
+ private String driverClass;
+ private JobStorage storage;
+
+ private Configuration conf;
+ private Map<String, String> descriptor;
+
+ public String INVALID_KEY = "INVALID_KEY";
+
+
+
+ public SavedJobsTest(String metaConnect, String metaUser, String metaPassword, String driverClass){
+ this.metaConnect = metaConnect;
+ this.metaUser = metaUser;
+ this.metaPassword = metaPassword;
+ this.driverClass = driverClass;
+ descriptor = new TreeMap<String, String>();
--- End diff --
We could use the diamond operator here, String, String is redundant.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] sqoop pull request #38: Metastore
Posted by szvasas <gi...@git.apache.org>.
Github user szvasas commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/38#discussion_r130084482
--- Diff: src/java/org/apache/sqoop/metastore/GenericJobStorage.java ---
@@ -666,6 +681,14 @@ private void createJobTable() throws SQLException {
*/
private void initV0Schema() throws SQLException {
this.jobTableName = getRootProperty(SESSION_TABLE_KEY, 0);
+
+ /** Checks to see if there is an existing job table under HsqldbJobStorage. **/
--- End diff --
This looks like it could be extracted in another method.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] sqoop pull request #38: Metastore
Posted by szvasas <gi...@git.apache.org>.
Github user szvasas commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/38#discussion_r130807571
--- Diff: src/java/org/apache/sqoop/manager/CubridManager.java ---
@@ -44,7 +44,7 @@
// driver class to ensure is loaded when making db connection.
private static final String DRIVER_CLASS =
--- End diff --
I think we could eliminate the DRIVER_CLASS constants from the managers and use JdbcDrivers enum directly in the constructors.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] sqoop pull request #38: Metastore
Posted by szvasas <gi...@git.apache.org>.
Github user szvasas commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/38#discussion_r130062813
--- Diff: src/test/com/cloudera/sqoop/metastore/SavedJobsTest.java ---
@@ -0,0 +1,379 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.cloudera.sqoop.metastore;
+
+import java.sql.SQLException;
+import java.sql.Statement;
+
+import java.util.*;
+
+import com.cloudera.sqoop.manager.ConnManager;
+import com.cloudera.sqoop.manager.MySQLTestUtils;
+import com.cloudera.sqoop.manager.OracleUtils;
+import org.apache.hadoop.conf.Configuration;
+
+import com.cloudera.sqoop.SqoopOptions;
+import com.cloudera.sqoop.tool.VersionTool;
+import org.apache.sqoop.manager.*;
+import org.apache.sqoop.manager.sqlserver.MSSQLTestUtils;
+import org.apache.sqoop.tool.ImportTool;
+import org.junit.*;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.sql.Connection;
+
+import static org.apache.sqoop.metastore.GenericJobStorage.*;
+import static org.hamcrest.core.IsCollectionContaining.hasItems;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertThat;
+
+/**
+ * Test the metastore and job-handling features.
+ *
+ * These all make use of the auto-connect hsqldb-based metastore.
+ * The metastore URL is configured to be in-memory, and drop all
+ * state between individual tests.
+ */
+@RunWith(Parameterized.class)
+public class SavedJobsTest {
+
+ private static MySQLTestUtils mySQLTestUtils = new MySQLTestUtils();
+ private static MSSQLTestUtils msSQLTestUtils = new MSSQLTestUtils();
+
+ @Parameterized.Parameters(name = "metaConnect = {0}, metaUser = {1}, metaPassword = {2}, driverClass = {3}")
+ public static Iterable<? extends Object> dbConnectParameters() {
+ return Arrays.asList(
+ new Object[] {
+ mySQLTestUtils.getHostUrl(), mySQLTestUtils.getUserName(),
+ mySQLTestUtils.getUserPass(), MySQLManager.DRIVER_CLASS
+ },
+ new Object[] {
+ OracleUtils.CONNECT_STRING, OracleUtils.ORACLE_USER_NAME,
+ OracleUtils.ORACLE_USER_PASS, OracleManager.DRIVER_CLASS
+ },
+ new Object[] {
+ msSQLTestUtils.getDBConnectString(), msSQLTestUtils.getDBUserName(),
+ msSQLTestUtils.getDBPassWord(), SQLServerManager.DRIVER_CLASS
+ },
+ new Object[] {
+ System.getProperty(
+ "sqoop.test.postgresql.connectstring.host_url",
+ "jdbc:postgresql://localhost/"),
+ System.getProperty(
+ "sqoop.test.postgresql.connectstring.username",
+ "sqooptest"),
+ System.getProperty(
+ "sqoop.test.postgresql.connectstring.password"),
+ PostgresqlManager.DRIVER_CLASS
+ },
+ new Object[] {
+ System.getProperty(
+ "sqoop.test.db2.connectstring.host_url",
+ "jdbc:db2://db2host:50000"),
+ System.getProperty(
+ "sqoop.test.db2.connectstring.username",
+ "SQOOP"),
+ System.getProperty(
+ "sqoop.test.db2.connectstring.password",
+ "SQOOP"),
+ Db2Manager.DRIVER_CLASS
+ },
+ new Object[] { "jdbc:hsqldb:mem:sqoopmetastore", "SA" , "", HsqldbManager.DRIVER_CLASS }
+ );
+ }
+
+ private String metaConnect;
+ private String metaUser;
+ private String metaPassword;
+ private String driverClass;
+ private JobStorage storage;
+
+ private Configuration conf;
+ private Map<String, String> descriptor;
+
+ public String INVALID_KEY = "INVALID_KEY";
+
+
--- End diff --
Please remove the extra new lines here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] sqoop pull request #38: Metastore
Posted by szvasas <gi...@git.apache.org>.
Github user szvasas commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/38#discussion_r129511691
--- Diff: src/java/com/cloudera/sqoop/metastore/GeneralJobStorage.java ---
@@ -15,22 +15,22 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package com.cloudera.sqoop.metastore.hsqldb;
+package com.cloudera.sqoop.metastore;
/**
* @deprecated Moving to use org.apache.sqoop namespace.
*/
-public class HsqldbJobStorage
- extends org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage {
+public class GeneralJobStorage
--- End diff --
Can we use the GenericJobStorage?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] sqoop pull request #38: Metastore
Posted by szvasas <gi...@git.apache.org>.
Github user szvasas commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/38#discussion_r130810721
--- Diff: src/java/org/apache/sqoop/metastore/GenericJobStorage.java ---
@@ -666,6 +722,11 @@ private void createJobTable() throws SQLException {
*/
private void initV0Schema() throws SQLException {
this.jobTableName = getRootProperty(SESSION_TABLE_KEY, 0);
+
+ /** Checks to see if there is an existing job table under the old root table schema
--- End diff --
Since this is a javadoc comment it should go to the method definition.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] sqoop pull request #38: Metastore
Posted by szvasas <gi...@git.apache.org>.
Github user szvasas commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/38#discussion_r130082337
--- Diff: src/test/com/cloudera/sqoop/metastore/SavedJobsTest.java ---
@@ -0,0 +1,379 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.cloudera.sqoop.metastore;
+
+import java.sql.SQLException;
+import java.sql.Statement;
+
+import java.util.*;
+
+import com.cloudera.sqoop.manager.ConnManager;
+import com.cloudera.sqoop.manager.MySQLTestUtils;
+import com.cloudera.sqoop.manager.OracleUtils;
+import org.apache.hadoop.conf.Configuration;
+
+import com.cloudera.sqoop.SqoopOptions;
+import com.cloudera.sqoop.tool.VersionTool;
+import org.apache.sqoop.manager.*;
+import org.apache.sqoop.manager.sqlserver.MSSQLTestUtils;
+import org.apache.sqoop.tool.ImportTool;
+import org.junit.*;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.sql.Connection;
+
+import static org.apache.sqoop.metastore.GenericJobStorage.*;
+import static org.hamcrest.core.IsCollectionContaining.hasItems;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertThat;
+
+/**
+ * Test the metastore and job-handling features.
+ *
+ * These all make use of the auto-connect hsqldb-based metastore.
+ * The metastore URL is configured to be in-memory, and drop all
+ * state between individual tests.
+ */
+@RunWith(Parameterized.class)
+public class SavedJobsTest {
+
+ private static MySQLTestUtils mySQLTestUtils = new MySQLTestUtils();
+ private static MSSQLTestUtils msSQLTestUtils = new MSSQLTestUtils();
+
+ @Parameterized.Parameters(name = "metaConnect = {0}, metaUser = {1}, metaPassword = {2}, driverClass = {3}")
+ public static Iterable<? extends Object> dbConnectParameters() {
+ return Arrays.asList(
+ new Object[] {
+ mySQLTestUtils.getHostUrl(), mySQLTestUtils.getUserName(),
+ mySQLTestUtils.getUserPass(), MySQLManager.DRIVER_CLASS
+ },
+ new Object[] {
+ OracleUtils.CONNECT_STRING, OracleUtils.ORACLE_USER_NAME,
+ OracleUtils.ORACLE_USER_PASS, OracleManager.DRIVER_CLASS
+ },
+ new Object[] {
+ msSQLTestUtils.getDBConnectString(), msSQLTestUtils.getDBUserName(),
+ msSQLTestUtils.getDBPassWord(), SQLServerManager.DRIVER_CLASS
+ },
+ new Object[] {
+ System.getProperty(
+ "sqoop.test.postgresql.connectstring.host_url",
+ "jdbc:postgresql://localhost/"),
+ System.getProperty(
+ "sqoop.test.postgresql.connectstring.username",
+ "sqooptest"),
+ System.getProperty(
+ "sqoop.test.postgresql.connectstring.password"),
+ PostgresqlManager.DRIVER_CLASS
+ },
+ new Object[] {
+ System.getProperty(
+ "sqoop.test.db2.connectstring.host_url",
+ "jdbc:db2://db2host:50000"),
+ System.getProperty(
+ "sqoop.test.db2.connectstring.username",
+ "SQOOP"),
+ System.getProperty(
+ "sqoop.test.db2.connectstring.password",
+ "SQOOP"),
+ Db2Manager.DRIVER_CLASS
+ },
+ new Object[] { "jdbc:hsqldb:mem:sqoopmetastore", "SA" , "", HsqldbManager.DRIVER_CLASS }
+ );
+ }
+
+ private String metaConnect;
+ private String metaUser;
+ private String metaPassword;
+ private String driverClass;
+ private JobStorage storage;
+
+ private Configuration conf;
+ private Map<String, String> descriptor;
+
+ public String INVALID_KEY = "INVALID_KEY";
+
+
+
+ public SavedJobsTest(String metaConnect, String metaUser, String metaPassword, String driverClass){
+ this.metaConnect = metaConnect;
+ this.metaUser = metaUser;
+ this.metaPassword = metaPassword;
+ this.driverClass = driverClass;
+ descriptor = new TreeMap<String, String>();
+ }
+
+ @Rule
+ public ExpectedException thrown = ExpectedException.none();
+
+ @Before
+ public void setUp() throws Exception {
+ // Delete db state between tests.
+ resetJobSchema();
+ conf = newConf();
+
+ descriptor.put(META_CONNECT_KEY, metaConnect);
+ descriptor.put(META_USERNAME_KEY, metaUser);
+ descriptor.put(META_PASSWORD_KEY, metaPassword);
+ descriptor.put(META_DRIVER_KEY, driverClass);
+
+ JobStorageFactory ssf = new JobStorageFactory(conf);
+ storage = ssf.getJobStorage(descriptor);
+ storage.open(descriptor);
+ }
+
+ @After
+ public void tearDown() throws Exception {
+ descriptor.clear();
+ storage.close();
+ }
+
+ public void resetJobSchema()
+ throws SQLException {
+ SqoopOptions options = new SqoopOptions();
+ options.setConnectString(metaConnect);
+ options.setUsername(metaUser);
+ options.setPassword(metaPassword);
+ options.setDriverClassName(driverClass);
+
+ resetSchema(options);
+ }
+
+ /**
+ * Drop all tables in the configured HSQLDB-based schema/user/pass.
+ */
+ public static void resetSchema(SqoopOptions options) throws SQLException {
+ JobData jd = new JobData();
+ jd.setSqoopOptions(options);
+ DefaultManagerFactory dmf = new DefaultManagerFactory();
+ ConnManager manager = dmf.accept(jd);
+ Connection c = manager.getConnection();
+ Statement s = c.createStatement();
+ try {
+ String [] tables = manager.listTables();
+ for (String table : tables) {
+ if(table.equalsIgnoreCase("SQOOP_ROOT") || table.equalsIgnoreCase("SQOOP_SESSIONS")){
+ s.execute("DROP TABLE " + table);
+ }
+ }
+
+ c.commit();
+ } finally {
+ s.close();
+ }
+ }
+
+ public Configuration newConf() {
+ Configuration conf = new Configuration();
+ conf.set(META_CONNECT_KEY, metaConnect);
+ conf.set(META_USERNAME_KEY, metaUser);
+ conf.set(META_PASSWORD_KEY, metaPassword);
+ conf.set(META_DRIVER_KEY, driverClass);
+
+ return conf;
+ }
+
+ @Test
+ public void canAcceptInvalidKeyFalseTest() throws Exception {
+ TreeMap<String,String> t = new TreeMap<>();
+ t.put(INVALID_KEY, "abc");
+
+ assertEquals("canAccept() should not accept invalid key",
+ storage.canAccept(t), false);
+ }
+
+ @Test
+ public void canAcceptValidKeyTrueTest() throws Exception {
+ TreeMap<String,String> t = new TreeMap<>();
--- End diff --
Use the superinterface in the variable decalaration.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
Re: [GitHub] sqoop pull request #38: Metastore
Posted by Szabolcs Vasas <va...@cloudera.com>.
Hi Priyanshi,
I am happy that you would like to contribute! There is a wiki page (
https://cwiki.apache.org/confluence/display/SQOOP/How+to+Contribute) about
how you can start working on Sqoop I think reading it is a good first step.
If you have any further questions you can always contact the dev list.
Regards,
Szabolcs
On Mon, Jul 31, 2017 at 6:14 PM, Priyanshi Sharma <
priyanshisharma1506@gmail.com> wrote:
> Hey everyone,
> I'm Priyanshi Sharma.
> 3rd year B.tech student studying Computer Science.
> I'd like to contribute.
> Can someone please help me get started?
>
>
> On Fri, Jul 28, 2017 at 7:41 PM, szvasas <gi...@git.apache.org> wrote:
>
> > Github user szvasas commented on a diff in the pull request:
> >
> > https://github.com/apache/sqoop/pull/38#discussion_r130098876
> >
> > --- Diff: src/test/com/cloudera/sqoop/JobToolTest.java ---
> > @@ -0,0 +1,258 @@
> > +/**
> > + * Licensed to the Apache Software Foundation (ASF) under one
> > + * or more contributor license agreements. See the NOTICE file
> > + * distributed with this work for additional information
> > + * regarding copyright ownership. The ASF licenses this file
> > + * to you under the Apache License, Version 2.0 (the
> > + * "License"); you may not use this file except in compliance
> > + * with the License. You may obtain a copy of the License at
> > + *
> > + * http://www.apache.org/licenses/LICENSE-2.0
> > + *
> > + * Unless required by applicable law or agreed to in writing,
> software
> > + * distributed under the License is distributed on an "AS IS" BASIS,
> > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> > implied.
> > + * See the License for the specific language governing permissions
> and
> > + * limitations under the License.
> > + */
> > +
> > +package com.cloudera.sqoop;
> > +
> > +import com.cloudera.sqoop.manager.MySQLTestUtils;
> > +import com.cloudera.sqoop.manager.OracleUtils;
> > +import com.cloudera.sqoop.testutil.BaseSqoopTestCase;
> > +import com.cloudera.sqoop.testutil.CommonArgs;
> > +import org.apache.commons.logging.Log;
> > +import org.apache.commons.logging.LogFactory;
> > +import org.apache.hadoop.conf.Configuration;
> > +import org.apache.sqoop.Sqoop;
> > +import org.apache.sqoop.manager.ConnManager;
> > +import org.apache.sqoop.manager.DefaultManagerFactory;
> > +import org.apache.sqoop.manager.sqlserver.MSSQLTestUtils;
> > +import org.apache.sqoop.tool.JobTool;
> > +import org.junit.After;
> > +import org.junit.Before;
> > +import org.junit.Test;
> > +import org.junit.runner.RunWith;
> > +import org.junit.runners.Parameterized;
> > +
> > +import java.io.IOException;
> > +import java.sql.Connection;
> > +import java.sql.SQLException;
> > +import java.sql.Statement;
> > +import java.util.ArrayList;
> > +import java.util.Arrays;
> > +
> > +import static org.junit.Assert.assertEquals;
> > +
> > +@RunWith(Parameterized.class)
> > +public class JobToolTest extends BaseSqoopTestCase {
> > +
> > + public static final Log LOG = LogFactory
> > + .getLog(MetaConnectIncrementalImportTe
> > st.class.getName());
> > +
> > + private static MySQLTestUtils mySQLTestUtils = new
> > MySQLTestUtils();
> > + private static MSSQLTestUtils msSQLTestUtils = new
> > MSSQLTestUtils();
> > + ConnManager cm;
> > +
> > + @Parameterized.Parameters(name = "metaConnectString = {0},
> > metaUser = {1}, metaPassword = {2}")
> > + public static Iterable<? extends Object> dbConnectParameters() {
> > + return Arrays.asList(
> > + new Object[] {
> > + mySQLTestUtils.getHostUrl(),
> > mySQLTestUtils.getUserName(),
> > + mySQLTestUtils.getUserPass()
> > + },
> > + new Object[] {
> > + OracleUtils.CONNECT_STRING,
> > OracleUtils.ORACLE_USER_NAME,
> > + OracleUtils.ORACLE_USER_PASS
> > + },
> > + new Object[] {
> > + msSQLTestUtils.getDBConnectString(),
> > msSQLTestUtils.getDBUserName(),
> > + msSQLTestUtils.getDBPassWord()
> > + },
> > + new Object[] {
> > + System.getProperty(
> > + "sqoop.test.postgresql.
> > connectstring.host_url",
> > + "jdbc:postgresql://localhost/"),
> > + System.getProperty(
> > + "sqoop.test.postgresql.
> > connectstring.username",
> > + "sqooptest"),
> > + System.getProperty(
> > + "sqoop.test.postgresql.
> > connectstring.password"),
> > + },
> > + new Object[] {
> > + System.getProperty(
> > + "sqoop.test.db2.connectstring.
> > host_url",
> > + "jdbc:db2://db2host:50000"),
> > + System.getProperty(
> > + "sqoop.test.db2.connectstring.
> > username",
> > + "SQOOP"),
> > + System.getProperty(
> > + "sqoop.test.db2.connectstring.
> > password",
> > + "SQOOP"),
> > + },
> > + new Object[] { "jdbc:hsqldb:mem:sqoopmetastore",
> > "SA" , "" }
> > + );
> > + }
> > +
> > + @Before
> > + public void setUp() {
> > + super.setUp();
> > +
> > + SqoopOptions options = getSqoopOptions();
> > +
> > + Connection conn = getConnection(options);
> > +
> > + try {
> > + Statement statement = conn.createStatement();
> > + statement.execute("DROP TABLE SQOOP_ROOT");
> > + statement.execute("DROP TABLE SQOOP_SESSIONS");
> > + conn.commit();
> > + } catch (Exception e) {
> > + LOG.error("Failed to clear metastore database");
> > + }
> > + //Methods from BaseSqoopTestClass reference the test Hsqldb
> > database, not the metastore
> > + try{
> > + dropTableIfExists("CarLocations");
> > + } catch (SQLException e) {
> > + LOG.error("Failed to drop table CarLocations");
> > + }
> > + setCurTableName("CarLocations");
> > + createTableWithColTypesAndNames(
> > + new String [] {"carId", "Locations"},
> > + new String [] {"INTEGER", "VARCHAR"},
> > + new String [] {"1", "'Lexus'"});
> > + }
> > +
> > + private Connection getConnection(SqoopOptions options) {
> > + try {
> > + com.cloudera.sqoop.metastore.JobData jd = new
> > com.cloudera.sqoop.metastore.JobData(options, null);
> > + DefaultManagerFactory dmf = new DefaultManagerFactory();
> > + cm = dmf.accept(jd);
> > + return cm.getConnection();
> > + } catch (SQLException e) {
> > + LOG.error("Failed to create a connection to the
> > Metastore");
> > + return null;
> > + }
> > + }
> > +
> > + private SqoopOptions getSqoopOptions() {
> > + SqoopOptions options = new SqoopOptions();
> > + options.setConnectString(metaConnectString);
> > + options.setUsername(metaUser);
> > + options.setPassword(metaPass);
> > + return options;
> > + }
> > +
> > + @After
> > + public void tearDown() {
> > + super.tearDown();
> > +
> > + try {
> > + cm.close();
> > + } catch (SQLException e) {
> > + LOG.error("Failed to close ConnManager");
> > + }
> > +
> > + }
> > +
> > + private String metaConnectString;
> > + private String metaUser;
> > + private String metaPass;
> > +
> > +
> > + public JobToolTest(String metaConnectString, String metaUser,
> > String metaPass) {
> > + this.metaConnectString = metaConnectString;
> > + this.metaUser = metaUser;
> > + this.metaPass = metaPass;
> > + }
> > +
> > + protected String[] getCreateJob(String metaConnectString, String
> > metaUser, String metaPass) {
> > + ArrayList<String> args = new ArrayList<String>();
> > + CommonArgs.addHadoopFlags(args);
> > + args.add("--create");
> > + args.add("testJob");
> > + args.add("--meta-connect");
> > + args.add(metaConnectString);
> > + args.add("--meta-username");
> > + args.add(metaUser);
> > + args.add("--meta-password");
> > + args.add(metaPass);
> > + args.add("--");
> > + args.add("list-tables");
> > + args.add("--connect");
> > + args.add(getConnectString());
> > +
> > + return args.toArray(new String[0]);
> > + }
> > +
> > + protected String[] getExecJob(String metaConnectString, String
> > metaUser, String metaPass) {
> > + ArrayList<String> args = new ArrayList<String>();
> > --- End diff --
> >
> > Use superinterface instead of concrete implementation.
> >
> >
> > ---
> > If your project is set up for it, you can reply to this email and have
> your
> > reply appear on GitHub as well. If your project does not have this
> feature
> > enabled and wishes so, or if the feature is enabled but not working,
> please
> > contact infrastructure at infrastructure@apache.org or file a JIRA
> ticket
> > with INFRA.
> > ---
> >
>
--
Szabolcs Vasas
Software Engineer
<http://www.cloudera.com>
Re: [GitHub] sqoop pull request #38: Metastore
Posted by Priyanshi Sharma <pr...@gmail.com>.
Hey everyone,
I'm Priyanshi Sharma.
3rd year B.tech student studying Computer Science.
I'd like to contribute.
Can someone please help me get started?
On Fri, Jul 28, 2017 at 7:41 PM, szvasas <gi...@git.apache.org> wrote:
> Github user szvasas commented on a diff in the pull request:
>
> https://github.com/apache/sqoop/pull/38#discussion_r130098876
>
> --- Diff: src/test/com/cloudera/sqoop/JobToolTest.java ---
> @@ -0,0 +1,258 @@
> +/**
> + * Licensed to the Apache Software Foundation (ASF) under one
> + * or more contributor license agreements. See the NOTICE file
> + * distributed with this work for additional information
> + * regarding copyright ownership. The ASF licenses this file
> + * to you under the Apache License, Version 2.0 (the
> + * "License"); you may not use this file except in compliance
> + * with the License. You may obtain a copy of the License at
> + *
> + * http://www.apache.org/licenses/LICENSE-2.0
> + *
> + * Unless required by applicable law or agreed to in writing, software
> + * distributed under the License is distributed on an "AS IS" BASIS,
> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> implied.
> + * See the License for the specific language governing permissions and
> + * limitations under the License.
> + */
> +
> +package com.cloudera.sqoop;
> +
> +import com.cloudera.sqoop.manager.MySQLTestUtils;
> +import com.cloudera.sqoop.manager.OracleUtils;
> +import com.cloudera.sqoop.testutil.BaseSqoopTestCase;
> +import com.cloudera.sqoop.testutil.CommonArgs;
> +import org.apache.commons.logging.Log;
> +import org.apache.commons.logging.LogFactory;
> +import org.apache.hadoop.conf.Configuration;
> +import org.apache.sqoop.Sqoop;
> +import org.apache.sqoop.manager.ConnManager;
> +import org.apache.sqoop.manager.DefaultManagerFactory;
> +import org.apache.sqoop.manager.sqlserver.MSSQLTestUtils;
> +import org.apache.sqoop.tool.JobTool;
> +import org.junit.After;
> +import org.junit.Before;
> +import org.junit.Test;
> +import org.junit.runner.RunWith;
> +import org.junit.runners.Parameterized;
> +
> +import java.io.IOException;
> +import java.sql.Connection;
> +import java.sql.SQLException;
> +import java.sql.Statement;
> +import java.util.ArrayList;
> +import java.util.Arrays;
> +
> +import static org.junit.Assert.assertEquals;
> +
> +@RunWith(Parameterized.class)
> +public class JobToolTest extends BaseSqoopTestCase {
> +
> + public static final Log LOG = LogFactory
> + .getLog(MetaConnectIncrementalImportTe
> st.class.getName());
> +
> + private static MySQLTestUtils mySQLTestUtils = new
> MySQLTestUtils();
> + private static MSSQLTestUtils msSQLTestUtils = new
> MSSQLTestUtils();
> + ConnManager cm;
> +
> + @Parameterized.Parameters(name = "metaConnectString = {0},
> metaUser = {1}, metaPassword = {2}")
> + public static Iterable<? extends Object> dbConnectParameters() {
> + return Arrays.asList(
> + new Object[] {
> + mySQLTestUtils.getHostUrl(),
> mySQLTestUtils.getUserName(),
> + mySQLTestUtils.getUserPass()
> + },
> + new Object[] {
> + OracleUtils.CONNECT_STRING,
> OracleUtils.ORACLE_USER_NAME,
> + OracleUtils.ORACLE_USER_PASS
> + },
> + new Object[] {
> + msSQLTestUtils.getDBConnectString(),
> msSQLTestUtils.getDBUserName(),
> + msSQLTestUtils.getDBPassWord()
> + },
> + new Object[] {
> + System.getProperty(
> + "sqoop.test.postgresql.
> connectstring.host_url",
> + "jdbc:postgresql://localhost/"),
> + System.getProperty(
> + "sqoop.test.postgresql.
> connectstring.username",
> + "sqooptest"),
> + System.getProperty(
> + "sqoop.test.postgresql.
> connectstring.password"),
> + },
> + new Object[] {
> + System.getProperty(
> + "sqoop.test.db2.connectstring.
> host_url",
> + "jdbc:db2://db2host:50000"),
> + System.getProperty(
> + "sqoop.test.db2.connectstring.
> username",
> + "SQOOP"),
> + System.getProperty(
> + "sqoop.test.db2.connectstring.
> password",
> + "SQOOP"),
> + },
> + new Object[] { "jdbc:hsqldb:mem:sqoopmetastore",
> "SA" , "" }
> + );
> + }
> +
> + @Before
> + public void setUp() {
> + super.setUp();
> +
> + SqoopOptions options = getSqoopOptions();
> +
> + Connection conn = getConnection(options);
> +
> + try {
> + Statement statement = conn.createStatement();
> + statement.execute("DROP TABLE SQOOP_ROOT");
> + statement.execute("DROP TABLE SQOOP_SESSIONS");
> + conn.commit();
> + } catch (Exception e) {
> + LOG.error("Failed to clear metastore database");
> + }
> + //Methods from BaseSqoopTestClass reference the test Hsqldb
> database, not the metastore
> + try{
> + dropTableIfExists("CarLocations");
> + } catch (SQLException e) {
> + LOG.error("Failed to drop table CarLocations");
> + }
> + setCurTableName("CarLocations");
> + createTableWithColTypesAndNames(
> + new String [] {"carId", "Locations"},
> + new String [] {"INTEGER", "VARCHAR"},
> + new String [] {"1", "'Lexus'"});
> + }
> +
> + private Connection getConnection(SqoopOptions options) {
> + try {
> + com.cloudera.sqoop.metastore.JobData jd = new
> com.cloudera.sqoop.metastore.JobData(options, null);
> + DefaultManagerFactory dmf = new DefaultManagerFactory();
> + cm = dmf.accept(jd);
> + return cm.getConnection();
> + } catch (SQLException e) {
> + LOG.error("Failed to create a connection to the
> Metastore");
> + return null;
> + }
> + }
> +
> + private SqoopOptions getSqoopOptions() {
> + SqoopOptions options = new SqoopOptions();
> + options.setConnectString(metaConnectString);
> + options.setUsername(metaUser);
> + options.setPassword(metaPass);
> + return options;
> + }
> +
> + @After
> + public void tearDown() {
> + super.tearDown();
> +
> + try {
> + cm.close();
> + } catch (SQLException e) {
> + LOG.error("Failed to close ConnManager");
> + }
> +
> + }
> +
> + private String metaConnectString;
> + private String metaUser;
> + private String metaPass;
> +
> +
> + public JobToolTest(String metaConnectString, String metaUser,
> String metaPass) {
> + this.metaConnectString = metaConnectString;
> + this.metaUser = metaUser;
> + this.metaPass = metaPass;
> + }
> +
> + protected String[] getCreateJob(String metaConnectString, String
> metaUser, String metaPass) {
> + ArrayList<String> args = new ArrayList<String>();
> + CommonArgs.addHadoopFlags(args);
> + args.add("--create");
> + args.add("testJob");
> + args.add("--meta-connect");
> + args.add(metaConnectString);
> + args.add("--meta-username");
> + args.add(metaUser);
> + args.add("--meta-password");
> + args.add(metaPass);
> + args.add("--");
> + args.add("list-tables");
> + args.add("--connect");
> + args.add(getConnectString());
> +
> + return args.toArray(new String[0]);
> + }
> +
> + protected String[] getExecJob(String metaConnectString, String
> metaUser, String metaPass) {
> + ArrayList<String> args = new ArrayList<String>();
> --- End diff --
>
> Use superinterface instead of concrete implementation.
>
>
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastructure@apache.org or file a JIRA ticket
> with INFRA.
> ---
>
[GitHub] sqoop pull request #38: Metastore
Posted by szvasas <gi...@git.apache.org>.
Github user szvasas commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/38#discussion_r130098876
--- Diff: src/test/com/cloudera/sqoop/JobToolTest.java ---
@@ -0,0 +1,258 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.cloudera.sqoop;
+
+import com.cloudera.sqoop.manager.MySQLTestUtils;
+import com.cloudera.sqoop.manager.OracleUtils;
+import com.cloudera.sqoop.testutil.BaseSqoopTestCase;
+import com.cloudera.sqoop.testutil.CommonArgs;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.sqoop.Sqoop;
+import org.apache.sqoop.manager.ConnManager;
+import org.apache.sqoop.manager.DefaultManagerFactory;
+import org.apache.sqoop.manager.sqlserver.MSSQLTestUtils;
+import org.apache.sqoop.tool.JobTool;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.Arrays;
+
+import static org.junit.Assert.assertEquals;
+
+@RunWith(Parameterized.class)
+public class JobToolTest extends BaseSqoopTestCase {
+
+ public static final Log LOG = LogFactory
+ .getLog(MetaConnectIncrementalImportTest.class.getName());
+
+ private static MySQLTestUtils mySQLTestUtils = new MySQLTestUtils();
+ private static MSSQLTestUtils msSQLTestUtils = new MSSQLTestUtils();
+ ConnManager cm;
+
+ @Parameterized.Parameters(name = "metaConnectString = {0}, metaUser = {1}, metaPassword = {2}")
+ public static Iterable<? extends Object> dbConnectParameters() {
+ return Arrays.asList(
+ new Object[] {
+ mySQLTestUtils.getHostUrl(), mySQLTestUtils.getUserName(),
+ mySQLTestUtils.getUserPass()
+ },
+ new Object[] {
+ OracleUtils.CONNECT_STRING, OracleUtils.ORACLE_USER_NAME,
+ OracleUtils.ORACLE_USER_PASS
+ },
+ new Object[] {
+ msSQLTestUtils.getDBConnectString(), msSQLTestUtils.getDBUserName(),
+ msSQLTestUtils.getDBPassWord()
+ },
+ new Object[] {
+ System.getProperty(
+ "sqoop.test.postgresql.connectstring.host_url",
+ "jdbc:postgresql://localhost/"),
+ System.getProperty(
+ "sqoop.test.postgresql.connectstring.username",
+ "sqooptest"),
+ System.getProperty(
+ "sqoop.test.postgresql.connectstring.password"),
+ },
+ new Object[] {
+ System.getProperty(
+ "sqoop.test.db2.connectstring.host_url",
+ "jdbc:db2://db2host:50000"),
+ System.getProperty(
+ "sqoop.test.db2.connectstring.username",
+ "SQOOP"),
+ System.getProperty(
+ "sqoop.test.db2.connectstring.password",
+ "SQOOP"),
+ },
+ new Object[] { "jdbc:hsqldb:mem:sqoopmetastore", "SA" , "" }
+ );
+ }
+
+ @Before
+ public void setUp() {
+ super.setUp();
+
+ SqoopOptions options = getSqoopOptions();
+
+ Connection conn = getConnection(options);
+
+ try {
+ Statement statement = conn.createStatement();
+ statement.execute("DROP TABLE SQOOP_ROOT");
+ statement.execute("DROP TABLE SQOOP_SESSIONS");
+ conn.commit();
+ } catch (Exception e) {
+ LOG.error("Failed to clear metastore database");
+ }
+ //Methods from BaseSqoopTestClass reference the test Hsqldb database, not the metastore
+ try{
+ dropTableIfExists("CarLocations");
+ } catch (SQLException e) {
+ LOG.error("Failed to drop table CarLocations");
+ }
+ setCurTableName("CarLocations");
+ createTableWithColTypesAndNames(
+ new String [] {"carId", "Locations"},
+ new String [] {"INTEGER", "VARCHAR"},
+ new String [] {"1", "'Lexus'"});
+ }
+
+ private Connection getConnection(SqoopOptions options) {
+ try {
+ com.cloudera.sqoop.metastore.JobData jd = new com.cloudera.sqoop.metastore.JobData(options, null);
+ DefaultManagerFactory dmf = new DefaultManagerFactory();
+ cm = dmf.accept(jd);
+ return cm.getConnection();
+ } catch (SQLException e) {
+ LOG.error("Failed to create a connection to the Metastore");
+ return null;
+ }
+ }
+
+ private SqoopOptions getSqoopOptions() {
+ SqoopOptions options = new SqoopOptions();
+ options.setConnectString(metaConnectString);
+ options.setUsername(metaUser);
+ options.setPassword(metaPass);
+ return options;
+ }
+
+ @After
+ public void tearDown() {
+ super.tearDown();
+
+ try {
+ cm.close();
+ } catch (SQLException e) {
+ LOG.error("Failed to close ConnManager");
+ }
+
+ }
+
+ private String metaConnectString;
+ private String metaUser;
+ private String metaPass;
+
+
+ public JobToolTest(String metaConnectString, String metaUser, String metaPass) {
+ this.metaConnectString = metaConnectString;
+ this.metaUser = metaUser;
+ this.metaPass = metaPass;
+ }
+
+ protected String[] getCreateJob(String metaConnectString, String metaUser, String metaPass) {
+ ArrayList<String> args = new ArrayList<String>();
+ CommonArgs.addHadoopFlags(args);
+ args.add("--create");
+ args.add("testJob");
+ args.add("--meta-connect");
+ args.add(metaConnectString);
+ args.add("--meta-username");
+ args.add(metaUser);
+ args.add("--meta-password");
+ args.add(metaPass);
+ args.add("--");
+ args.add("list-tables");
+ args.add("--connect");
+ args.add(getConnectString());
+
+ return args.toArray(new String[0]);
+ }
+
+ protected String[] getExecJob(String metaConnectString, String metaUser, String metaPass) {
+ ArrayList<String> args = new ArrayList<String>();
--- End diff --
Use superinterface instead of concrete implementation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] sqoop pull request #38: Metastore
Posted by szvasas <gi...@git.apache.org>.
Github user szvasas commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/38#discussion_r130812016
--- Diff: src/java/org/apache/sqoop/tool/JobTool.java ---
@@ -369,6 +366,61 @@ public void applyOptions(CommandLine in, SqoopOptions out)
}
}
+ private void applyMetastoreOptions(CommandLine in, SqoopOptions out) throws InvalidOptionsException {
+ String metaConnectString;
+ String metaUsernameString;
+ String metaPasswordString;
+ if (in.hasOption(STORAGE_METASTORE_ARG)) {
+ metaConnectString = in.getOptionValue(STORAGE_METASTORE_ARG);
+ this.storageDescriptor.put(GenericJobStorage.META_DRIVER_KEY, chooseDriverType(metaConnectString));
+ this.storageDescriptor.put(GenericJobStorage.META_CONNECT_KEY, metaConnectString);
+ } else {
+ metaConnectString = out.getMetaConnectStr();
+ this.storageDescriptor.put(GenericJobStorage.META_DRIVER_KEY, chooseDriverType(metaConnectString));
+ this.storageDescriptor.put(GenericJobStorage.META_CONNECT_KEY, metaConnectString);
+ }
+ if (in.hasOption(METASTORE_USER_ARG)) {
+ metaUsernameString = in.getOptionValue(METASTORE_USER_ARG);
+ this.storageDescriptor.put(GenericJobStorage.META_USERNAME_KEY, metaUsernameString);
+ } else {
+ metaUsernameString = out.getMetaUsername();
+ this.storageDescriptor.put(GenericJobStorage.META_USERNAME_KEY, metaUsernameString);
+ }
+ if (in.hasOption(METASTORE_PASS_ARG)) {
+ metaPasswordString = in.getOptionValue(METASTORE_PASS_ARG);
+ this.storageDescriptor.put(GenericJobStorage.META_PASSWORD_KEY, metaPasswordString);
+ } else {
+ metaPasswordString = out.getMetaPassword();
+ this.storageDescriptor.put(GenericJobStorage.META_PASSWORD_KEY, metaPasswordString);
+ }
+ }
+
+ public static String chooseDriverType(String metaConnectString) throws InvalidOptionsException {
--- End diff --
This method should be private and non-static since it is only used in this class.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] sqoop pull request #38: Metastore
Posted by szvasas <gi...@git.apache.org>.
Github user szvasas commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/38#discussion_r130057718
--- Diff: src/docs/user/metastore-purpose.txt ---
@@ -22,6 +22,8 @@ Multiple users and/or remote users can define and execute saved jobs (created
with +sqoop job+) defined in this metastore.
Clients must be configured to connect to the metastore in +sqoop-site.xml+ or
-with the +--meta-connect+ argument.
+with the +--meta-connect+ argument. The +sqoop-site.xml+ configuration supports
+Hsqldb databases only, while +--meta-connect+ supports MySql, PostgreSql, Oracle, DB2,
--- End diff --
What does it mean that sqoop-site.xml configuration supports Hsqldb only?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] sqoop pull request #38: Metastore
Posted by szvasas <gi...@git.apache.org>.
Github user szvasas commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/38#discussion_r130082318
--- Diff: src/test/com/cloudera/sqoop/metastore/SavedJobsTest.java ---
@@ -0,0 +1,379 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.cloudera.sqoop.metastore;
+
+import java.sql.SQLException;
+import java.sql.Statement;
+
+import java.util.*;
+
+import com.cloudera.sqoop.manager.ConnManager;
+import com.cloudera.sqoop.manager.MySQLTestUtils;
+import com.cloudera.sqoop.manager.OracleUtils;
+import org.apache.hadoop.conf.Configuration;
+
+import com.cloudera.sqoop.SqoopOptions;
+import com.cloudera.sqoop.tool.VersionTool;
+import org.apache.sqoop.manager.*;
+import org.apache.sqoop.manager.sqlserver.MSSQLTestUtils;
+import org.apache.sqoop.tool.ImportTool;
+import org.junit.*;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.sql.Connection;
+
+import static org.apache.sqoop.metastore.GenericJobStorage.*;
+import static org.hamcrest.core.IsCollectionContaining.hasItems;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertThat;
+
+/**
+ * Test the metastore and job-handling features.
+ *
+ * These all make use of the auto-connect hsqldb-based metastore.
+ * The metastore URL is configured to be in-memory, and drop all
+ * state between individual tests.
+ */
+@RunWith(Parameterized.class)
+public class SavedJobsTest {
+
+ private static MySQLTestUtils mySQLTestUtils = new MySQLTestUtils();
+ private static MSSQLTestUtils msSQLTestUtils = new MSSQLTestUtils();
+
+ @Parameterized.Parameters(name = "metaConnect = {0}, metaUser = {1}, metaPassword = {2}, driverClass = {3}")
+ public static Iterable<? extends Object> dbConnectParameters() {
+ return Arrays.asList(
+ new Object[] {
+ mySQLTestUtils.getHostUrl(), mySQLTestUtils.getUserName(),
+ mySQLTestUtils.getUserPass(), MySQLManager.DRIVER_CLASS
+ },
+ new Object[] {
+ OracleUtils.CONNECT_STRING, OracleUtils.ORACLE_USER_NAME,
+ OracleUtils.ORACLE_USER_PASS, OracleManager.DRIVER_CLASS
+ },
+ new Object[] {
+ msSQLTestUtils.getDBConnectString(), msSQLTestUtils.getDBUserName(),
+ msSQLTestUtils.getDBPassWord(), SQLServerManager.DRIVER_CLASS
+ },
+ new Object[] {
+ System.getProperty(
+ "sqoop.test.postgresql.connectstring.host_url",
+ "jdbc:postgresql://localhost/"),
+ System.getProperty(
+ "sqoop.test.postgresql.connectstring.username",
+ "sqooptest"),
+ System.getProperty(
+ "sqoop.test.postgresql.connectstring.password"),
+ PostgresqlManager.DRIVER_CLASS
+ },
+ new Object[] {
+ System.getProperty(
+ "sqoop.test.db2.connectstring.host_url",
+ "jdbc:db2://db2host:50000"),
+ System.getProperty(
+ "sqoop.test.db2.connectstring.username",
+ "SQOOP"),
+ System.getProperty(
+ "sqoop.test.db2.connectstring.password",
+ "SQOOP"),
+ Db2Manager.DRIVER_CLASS
+ },
+ new Object[] { "jdbc:hsqldb:mem:sqoopmetastore", "SA" , "", HsqldbManager.DRIVER_CLASS }
+ );
+ }
+
+ private String metaConnect;
+ private String metaUser;
+ private String metaPassword;
+ private String driverClass;
+ private JobStorage storage;
+
+ private Configuration conf;
+ private Map<String, String> descriptor;
+
+ public String INVALID_KEY = "INVALID_KEY";
+
+
+
+ public SavedJobsTest(String metaConnect, String metaUser, String metaPassword, String driverClass){
+ this.metaConnect = metaConnect;
+ this.metaUser = metaUser;
+ this.metaPassword = metaPassword;
+ this.driverClass = driverClass;
+ descriptor = new TreeMap<String, String>();
+ }
+
+ @Rule
+ public ExpectedException thrown = ExpectedException.none();
+
+ @Before
+ public void setUp() throws Exception {
+ // Delete db state between tests.
+ resetJobSchema();
+ conf = newConf();
+
+ descriptor.put(META_CONNECT_KEY, metaConnect);
+ descriptor.put(META_USERNAME_KEY, metaUser);
+ descriptor.put(META_PASSWORD_KEY, metaPassword);
+ descriptor.put(META_DRIVER_KEY, driverClass);
+
+ JobStorageFactory ssf = new JobStorageFactory(conf);
+ storage = ssf.getJobStorage(descriptor);
+ storage.open(descriptor);
+ }
+
+ @After
+ public void tearDown() throws Exception {
+ descriptor.clear();
+ storage.close();
+ }
+
+ public void resetJobSchema()
+ throws SQLException {
+ SqoopOptions options = new SqoopOptions();
+ options.setConnectString(metaConnect);
+ options.setUsername(metaUser);
+ options.setPassword(metaPassword);
+ options.setDriverClassName(driverClass);
+
+ resetSchema(options);
+ }
+
+ /**
+ * Drop all tables in the configured HSQLDB-based schema/user/pass.
+ */
+ public static void resetSchema(SqoopOptions options) throws SQLException {
+ JobData jd = new JobData();
+ jd.setSqoopOptions(options);
+ DefaultManagerFactory dmf = new DefaultManagerFactory();
+ ConnManager manager = dmf.accept(jd);
+ Connection c = manager.getConnection();
+ Statement s = c.createStatement();
+ try {
+ String [] tables = manager.listTables();
+ for (String table : tables) {
+ if(table.equalsIgnoreCase("SQOOP_ROOT") || table.equalsIgnoreCase("SQOOP_SESSIONS")){
+ s.execute("DROP TABLE " + table);
+ }
+ }
+
+ c.commit();
+ } finally {
+ s.close();
+ }
+ }
+
+ public Configuration newConf() {
+ Configuration conf = new Configuration();
+ conf.set(META_CONNECT_KEY, metaConnect);
+ conf.set(META_USERNAME_KEY, metaUser);
+ conf.set(META_PASSWORD_KEY, metaPassword);
+ conf.set(META_DRIVER_KEY, driverClass);
+
+ return conf;
+ }
+
+ @Test
+ public void canAcceptInvalidKeyFalseTest() throws Exception {
+ TreeMap<String,String> t = new TreeMap<>();
--- End diff --
Use the superinterface in the variable decalaration.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] sqoop pull request #38: Metastore
Posted by szvasas <gi...@git.apache.org>.
Github user szvasas commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/38#discussion_r130057397
--- Diff: src/docs/Makefile ---
@@ -24,8 +24,7 @@ VERSION=Unknown
# Building documentation under CentOS causes xmlto to have issues
# with validation. Disable this step.
-VALIDATION=$(shell ( lsb_release -d | grep CentOS > /dev/null 2>&1 ) \
- && echo "--skip-validation" )
+
--- End diff --
Why did we need to remove this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] sqoop pull request #38: Metastore
Posted by szvasas <gi...@git.apache.org>.
Github user szvasas commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/38#discussion_r130062575
--- Diff: src/test/com/cloudera/sqoop/metastore/SavedJobsTest.java ---
@@ -0,0 +1,379 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.cloudera.sqoop.metastore;
+
+import java.sql.SQLException;
+import java.sql.Statement;
+
+import java.util.*;
+
+import com.cloudera.sqoop.manager.ConnManager;
+import com.cloudera.sqoop.manager.MySQLTestUtils;
+import com.cloudera.sqoop.manager.OracleUtils;
+import org.apache.hadoop.conf.Configuration;
+
+import com.cloudera.sqoop.SqoopOptions;
+import com.cloudera.sqoop.tool.VersionTool;
+import org.apache.sqoop.manager.*;
+import org.apache.sqoop.manager.sqlserver.MSSQLTestUtils;
+import org.apache.sqoop.tool.ImportTool;
+import org.junit.*;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.sql.Connection;
+
+import static org.apache.sqoop.metastore.GenericJobStorage.*;
--- End diff --
Please remove * import.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] sqoop pull request #38: Metastore
Posted by szvasas <gi...@git.apache.org>.
Github user szvasas commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/38#discussion_r130811043
--- Diff: src/java/org/apache/sqoop/metastore/GenericJobStorage.java ---
@@ -675,6 +736,14 @@ private void initV0Schema() throws SQLException {
}
}
+ private void checkForOldSchema() throws SQLException {
--- End diff --
I would call this checkForOldRootProperties or something because the schema itself did not change just the ROOT table and a property in it, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] sqoop pull request #38: Metastore
Posted by szvasas <gi...@git.apache.org>.
Github user szvasas commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/38#discussion_r130062627
--- Diff: src/test/com/cloudera/sqoop/metastore/SavedJobsTest.java ---
@@ -0,0 +1,379 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.cloudera.sqoop.metastore;
+
+import java.sql.SQLException;
+import java.sql.Statement;
+
+import java.util.*;
+
+import com.cloudera.sqoop.manager.ConnManager;
+import com.cloudera.sqoop.manager.MySQLTestUtils;
+import com.cloudera.sqoop.manager.OracleUtils;
+import org.apache.hadoop.conf.Configuration;
+
+import com.cloudera.sqoop.SqoopOptions;
+import com.cloudera.sqoop.tool.VersionTool;
+import org.apache.sqoop.manager.*;
+import org.apache.sqoop.manager.sqlserver.MSSQLTestUtils;
+import org.apache.sqoop.tool.ImportTool;
+import org.junit.*;
--- End diff --
Please remove * import.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] sqoop pull request #38: Metastore
Posted by szvasas <gi...@git.apache.org>.
Github user szvasas commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/38#discussion_r129515098
--- Diff: src/docs/user/metastore-purpose.txt ---
@@ -22,6 +22,8 @@ Multiple users and/or remote users can define and execute saved jobs (created
with +sqoop job+) defined in this metastore.
Clients must be configured to connect to the metastore in +sqoop-site.xml+ or
-with the +--meta-connect+ argument.
+with the +--meta-connect+ argument. The +sqoop-site.xml+ configuration supports
--- End diff --
Thank you for adding documentation!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] sqoop pull request #38: Metastore
Posted by szvasas <gi...@git.apache.org>.
Github user szvasas commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/38#discussion_r130098767
--- Diff: src/test/com/cloudera/sqoop/JobToolTest.java ---
@@ -0,0 +1,258 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.cloudera.sqoop;
+
+import com.cloudera.sqoop.manager.MySQLTestUtils;
+import com.cloudera.sqoop.manager.OracleUtils;
+import com.cloudera.sqoop.testutil.BaseSqoopTestCase;
+import com.cloudera.sqoop.testutil.CommonArgs;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.sqoop.Sqoop;
+import org.apache.sqoop.manager.ConnManager;
+import org.apache.sqoop.manager.DefaultManagerFactory;
+import org.apache.sqoop.manager.sqlserver.MSSQLTestUtils;
+import org.apache.sqoop.tool.JobTool;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.Arrays;
+
+import static org.junit.Assert.assertEquals;
+
+@RunWith(Parameterized.class)
+public class JobToolTest extends BaseSqoopTestCase {
+
+ public static final Log LOG = LogFactory
+ .getLog(MetaConnectIncrementalImportTest.class.getName());
+
+ private static MySQLTestUtils mySQLTestUtils = new MySQLTestUtils();
+ private static MSSQLTestUtils msSQLTestUtils = new MSSQLTestUtils();
+ ConnManager cm;
+
+ @Parameterized.Parameters(name = "metaConnectString = {0}, metaUser = {1}, metaPassword = {2}")
+ public static Iterable<? extends Object> dbConnectParameters() {
+ return Arrays.asList(
+ new Object[] {
+ mySQLTestUtils.getHostUrl(), mySQLTestUtils.getUserName(),
+ mySQLTestUtils.getUserPass()
+ },
+ new Object[] {
+ OracleUtils.CONNECT_STRING, OracleUtils.ORACLE_USER_NAME,
+ OracleUtils.ORACLE_USER_PASS
+ },
+ new Object[] {
+ msSQLTestUtils.getDBConnectString(), msSQLTestUtils.getDBUserName(),
+ msSQLTestUtils.getDBPassWord()
+ },
+ new Object[] {
+ System.getProperty(
+ "sqoop.test.postgresql.connectstring.host_url",
+ "jdbc:postgresql://localhost/"),
+ System.getProperty(
+ "sqoop.test.postgresql.connectstring.username",
+ "sqooptest"),
+ System.getProperty(
+ "sqoop.test.postgresql.connectstring.password"),
+ },
+ new Object[] {
+ System.getProperty(
+ "sqoop.test.db2.connectstring.host_url",
+ "jdbc:db2://db2host:50000"),
+ System.getProperty(
+ "sqoop.test.db2.connectstring.username",
+ "SQOOP"),
+ System.getProperty(
+ "sqoop.test.db2.connectstring.password",
+ "SQOOP"),
+ },
+ new Object[] { "jdbc:hsqldb:mem:sqoopmetastore", "SA" , "" }
+ );
+ }
+
+ @Before
+ public void setUp() {
+ super.setUp();
+
+ SqoopOptions options = getSqoopOptions();
+
+ Connection conn = getConnection(options);
+
+ try {
+ Statement statement = conn.createStatement();
+ statement.execute("DROP TABLE SQOOP_ROOT");
+ statement.execute("DROP TABLE SQOOP_SESSIONS");
+ conn.commit();
+ } catch (Exception e) {
+ LOG.error("Failed to clear metastore database");
+ }
+ //Methods from BaseSqoopTestClass reference the test Hsqldb database, not the metastore
+ try{
+ dropTableIfExists("CarLocations");
+ } catch (SQLException e) {
+ LOG.error("Failed to drop table CarLocations");
+ }
+ setCurTableName("CarLocations");
+ createTableWithColTypesAndNames(
+ new String [] {"carId", "Locations"},
+ new String [] {"INTEGER", "VARCHAR"},
+ new String [] {"1", "'Lexus'"});
+ }
+
+ private Connection getConnection(SqoopOptions options) {
+ try {
+ com.cloudera.sqoop.metastore.JobData jd = new com.cloudera.sqoop.metastore.JobData(options, null);
+ DefaultManagerFactory dmf = new DefaultManagerFactory();
+ cm = dmf.accept(jd);
+ return cm.getConnection();
+ } catch (SQLException e) {
+ LOG.error("Failed to create a connection to the Metastore");
+ return null;
+ }
+ }
+
+ private SqoopOptions getSqoopOptions() {
+ SqoopOptions options = new SqoopOptions();
+ options.setConnectString(metaConnectString);
+ options.setUsername(metaUser);
+ options.setPassword(metaPass);
+ return options;
+ }
+
+ @After
+ public void tearDown() {
+ super.tearDown();
+
+ try {
+ cm.close();
+ } catch (SQLException e) {
+ LOG.error("Failed to close ConnManager");
+ }
+
+ }
+
+ private String metaConnectString;
--- End diff --
Please move this above all of the methods.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] sqoop pull request #38: Metastore
Posted by szvasas <gi...@git.apache.org>.
Github user szvasas commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/38#discussion_r130062567
--- Diff: src/test/com/cloudera/sqoop/metastore/SavedJobsTest.java ---
@@ -0,0 +1,379 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.cloudera.sqoop.metastore;
+
+import java.sql.SQLException;
+import java.sql.Statement;
+
+import java.util.*;
+
+import com.cloudera.sqoop.manager.ConnManager;
+import com.cloudera.sqoop.manager.MySQLTestUtils;
+import com.cloudera.sqoop.manager.OracleUtils;
+import org.apache.hadoop.conf.Configuration;
+
+import com.cloudera.sqoop.SqoopOptions;
+import com.cloudera.sqoop.tool.VersionTool;
+import org.apache.sqoop.manager.*;
--- End diff --
Please remove * import.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---