You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by "Veena Basavaraj (JIRA)" <ji...@apache.org> on 2014/11/15 01:49:35 UTC
[jira] [Comment Edited] (SQOOP-1736) Sqoop2: Driver missing during
upgrade
[ https://issues.apache.org/jira/browse/SQOOP-1736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14213186#comment-14213186 ]
Veena Basavaraj edited comment on SQOOP-1736 at 11/15/14 12:49 AM:
-------------------------------------------------------------------
tested with the data set attached. still needs confirmation
[root@vbsqoop-1 bin]# ./sqoop.sh tool upgrade
Sqoop home directory: /root/projects/sqoop/sqoop-1.99.4-bin-hadoop200
Setting SQOOP_HTTP_PORT: 12000
Setting SQOOP_ADMIN_PORT: 12001
Using CATALINA_OPTS:
Adding to CATALINA_OPTS: -Dsqoop.http.port=12000 -Dsqoop.admin.port=12001
Nov 14, 2014 4:45:11 PM org.apache.catalina.startup.ClassLoaderFactory validateFile
WARNING: Problem with directory [/root/projects/sqoop/sqoop-1.99.4-bin-hadoop200/lib], exists: [false], isDirectory: [false], canRead: [false]
Sqoop tool executor:
Version: 1.99.4
Revision: 68d5beeab40b9d9da9f1c00da544185b161fdce6
Compiled on Fri Nov 14 15:46:21 PST 2014 by vbasavaraj
Running tool: class org.apache.sqoop.tools.tool.UpgradeTool
2014-11-14 16:45:12,123 INFO [main] core.PropertiesConfigurationProvider (PropertiesConfigurationProvider.java:initialize(96)) - Starting config file poller thread
log4j: Parsing for [root] with value=[WARN, file].
log4j: Level token is [WARN].
log4j: Category root set to WARN
log4j: Parsing appender named "file".
log4j: Parsing layout options for "file".
log4j: Setting property [conversionPattern] to [%d{ISO8601} %-5p %c{2} [%l] %m%n].
log4j: End of parsing for "file".
log4j: Setting property [file] to [@LOGDIR@/sqoop.log].
log4j: Setting property [maxBackupIndex] to [5].
log4j: Setting property [maxFileSize] to [25MB].
log4j: setFile called: @LOGDIR@/sqoop.log, true
log4j: setFile ended
log4j: Parsed "file" options.
log4j: Parsing for [org.apache.sqoop] with value=[DEBUG].
log4j: Level token is [DEBUG].
log4j: Category org.apache.sqoop set to DEBUG
log4j: Handling log4j.additivity.org.apache.sqoop=[null]
log4j: Parsing for [org.apache.derby] with value=[INFO].
log4j: Level token is [INFO].
log4j: Category org.apache.derby set to INFO
log4j: Handling log4j.additivity.org.apache.derby=[null]
log4j: Finished configuring.
Tool class org.apache.sqoop.tools.tool.UpgradeTool has finished correctly.
14-11-14 16:45:23,850 DEBUG derby.DerbyRepositoryHandler [org.apache.sqoop.repository.derby.DerbyRepositoryHandler.findDriver(DerbyRepositoryHandler.java:1073)] Looking up Driver and config
2014-11-14 16:45:23,852 DEBUG derby.DerbyRepositoryHandler [org.apache.sqoop.repository.derby.DerbyRepositoryHandler.findDriver(DerbyRepositoryHandler.java:1129)] Looked up Driver and config
2014-11-14 16:45:23,896 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)] Tx count-commit: 3, rollback: false
2014-11-14 16:45:23,897 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)] Tx count-close: 2, rollback: false
2014-11-14 16:45:23,897 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)] Tx count-begin: 3, rollback: false
2014-11-14 16:45:24,035 INFO repository.Repository [org.apache.sqoop.repository.Repository.upgradeConnector(Repository.java:443)] Link upgrade for link:hdfs-connector for connector:hdfs-connector
2014-11-14 16:45:24,134 INFO repository.Repository [org.apache.sqoop.repository.Repository.upgradeConnector(Repository.java:467)] All Link and configs for this connector processed
2014-11-14 16:45:24,135 INFO repository.Repository [org.apache.sqoop.repository.Repository.upgradeConnector(Repository.java:473)] Job upgrade for job:m2h for connector:hdfs-connector
2014-11-14 16:45:24,138 WARN configurable.ConfigurableUpgradeUtil [org.apache.sqoop.configurable.ConfigurableUpgradeUtil.doUpgrade(ConfigurableUpgradeUtil.java:56)] Input: 'toJobConfig.customCompression' not present in old configurable. So it will not be transferred by the upgrader.
2014-11-14 16:45:24,179 INFO repository.Repository [org.apache.sqoop.repository.Repository.upgradeConnector(Repository.java:523)] All Job and configs for this connector processed
2014-11-14 16:45:24,179 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)] Tx count-commit: 3, rollback: false
2014-11-14 16:45:24,179 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)] Tx count-close: 2, rollback: false
2014-11-14 16:45:24,180 INFO repository.Repository [org.apache.sqoop.repository.Repository.upgradeConnector(Repository.java:543)] Connector upgrade finished for: hdfs-connector
2014-11-14 16:45:24,180 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)] Tx count-commit: 2, rollback: false
2014-11-14 16:45:24,180 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)] Tx count-close: 1, rollback: false
2014-11-14 16:45:24,180 DEBUG connector.ConnectorManager [org.apache.sqoop.connector.ConnectorManager.registerConnectors(ConnectorManager.java:215)] Registered connector: connector-hdfs-connector:2:org.apache.sqoop.connector.hdfs.HdfsConnector, Link: Configs: config-linkConfig:10:[input-linkConfig.uri:30:STRING:255]Type: LINK, From: Configs: config-fromJobConfig:11:[input-fromJobConfig.inputDirectory:31:STRING:255]Type: JOB, To: Configs: config-toJobConfig:12:[input-toJobConfig.outputFormat:32:ENUM:TEXT_FILE,SEQUENCE_FILE, input-toJobConfig.compression:33:ENUM:NONE,DEFAULT,DEFLATE,GZIP,BZIP2,LZO,LZ4,SNAPPY,CUSTOM, input-toJobConfig.customCompression:34:STRING:255, input-toJobConfig.outputDirectory:35:STRING:255]Type: JOB
2014-11-14 16:45:24,181 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)] Tx count-begin: 2, rollback: false
2014-11-14 16:45:24,181 DEBUG derby.DerbyRepositoryHandler [org.apache.sqoop.repository.derby.DerbyRepositoryHandler.findConnector(DerbyRepositoryHandler.java:988)] Looking up connector: generic-jdbc-connector
2014-11-14 16:45:24,190 DEBUG derby.DerbyRepositoryHandler [org.apache.sqoop.repository.derby.DerbyRepositoryHandler.findConnector(DerbyRepositoryHandler.java:1002)] Looking up connector: generic-jdbc-connector, found: null
2014-11-14 16:45:24,190 INFO repository.Repository [org.apache.sqoop.repository.Repository.upgradeConnector(Repository.java:413)] Upgrading connector: generic-jdbc-connector
2014-11-14 16:45:24,191 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)] Tx count-begin: 3, rollback: false
2014-11-14 16:45:24,215 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)] Tx count-commit: 3, rollback: false
2014-11-14 16:45:24,215 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)] Tx count-close: 2, rollback: false
2014-11-14 16:45:24,215 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)] Tx count-begin: 3, rollback: false
2014-11-14 16:45:24,216 DEBUG derby.DerbyRepositoryHandler [org.apache.sqoop.repository.derby.DerbyRepositoryHandler.findDriver(DerbyRepositoryHandler.java:1073)] Looking up Driver and config
2014-11-14 16:45:24,218 DEBUG derby.DerbyRepositoryHandler [org.apache.sqoop.repository.derby.DerbyRepositoryHandler.findDriver(DerbyRepositoryHandler.java:1129)] Looked up Driver and config
2014-11-14 16:45:24,234 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)] Tx count-commit: 3, rollback: false
2014-11-14 16:45:24,235 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)] Tx count-close: 2, rollback: false
2014-11-14 16:45:24,235 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)] Tx count-begin: 3, rollback: false
2014-11-14 16:45:24,312 INFO repository.Repository [org.apache.sqoop.repository.Repository.upgradeConnector(Repository.java:443)] Link upgrade for link:mysql for connector:generic-jdbc-connector
2014-11-14 16:45:24,313 WARN configurable.ConfigurableUpgradeUtil [org.apache.sqoop.configurable.ConfigurableUpgradeUtil.doUpgrade(ConfigurableUpgradeUtil.java:47)] Config: 'linkConfig' not present in old configurable. So it and its inputs will not be transferred by the upgrader.
2014-11-14 16:45:24,613 INFO repository.Repository [org.apache.sqoop.repository.Repository.upgradeConnector(Repository.java:443)] Link upgrade for link:mysql2 for connector:generic-jdbc-connector
2014-11-14 16:45:24,614 WARN configurable.ConfigurableUpgradeUtil [org.apache.sqoop.configurable.ConfigurableUpgradeUtil.doUpgrade(ConfigurableUpgradeUtil.java:47)] Config: 'linkConfig' not present in old configurable. So it and its inputs will not be transferred by the upgrader.
2014-11-14 16:45:24,649 INFO repository.Repository [org.apache.sqoop.repository.Repository.upgradeConnector(Repository.java:443)] Link upgrade for link:postgresql for connector:generic-jdbc-connector
2014-11-14 16:45:24,651 WARN configurable.ConfigurableUpgradeUtil [org.apache.sqoop.configurable.ConfigurableUpgradeUtil.doUpgrade(ConfigurableUpgradeUtil.java:47)] Config: 'linkConfig' not present in old configurable. So it and its inputs will not be transferred by the upgrader.
2014-11-14 16:45:24,741 INFO repository.Repository [org.apache.sqoop.repository.Repository.upgradeConnector(Repository.java:467)] All Link and configs for this connector processed
2014-11-14 16:45:24,742 INFO repository.Repository [org.apache.sqoop.repository.Repository.upgradeConnector(Repository.java:473)] Job upgrade for job:m2h for connector:generic-jdbc-connector
2014-11-14 16:45:24,742 WARN configurable.ConfigurableUpgradeUtil [org.apache.sqoop.configurable.ConfigurableUpgradeUtil.doUpgrade(ConfigurableUpgradeUtil.java:47)] Config: 'fromJobConfig' not present in old configurable. So it and its inputs will not be transferred by the upgrader.
2014-11-14 16:45:24,753 INFO repository.Repository [org.apache.sqoop.repository.Repository.upgradeConnector(Repository.java:523)] All Job and configs for this connector processed
2014-11-14 16:45:24,753 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)] Tx count-commit: 3, rollback: false
2014-11-14 16:45:24,753 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)] Tx count-close: 2, rollback: false
2014-11-14 16:45:24,754 INFO repository.Repository [org.apache.sqoop.repository.Repository.upgradeConnector(Repository.java:543)] Connector upgrade finished for: generic-jdbc-connector
2014-11-14 16:45:24,754 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)] Tx count-commit: 2, rollback: false
2014-11-14 16:45:24,754 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)] Tx count-close: 1, rollback: false
2014-11-14 16:45:24,755 DEBUG connector.ConnectorManager [org.apache.sqoop.connector.ConnectorManager.registerConnectors(ConnectorManager.java:215)] Registered connector: connector-generic-jdbc-connector:1:org.apache.sqoop.connector.jdbc.GenericJdbcConnector, Link: Configs: config-linkConfig:13:[input-linkConfig.jdbcDriver:36:STRING:128, input-linkConfig.connectionString:37:STRING:128, input-linkConfig.username:38:STRING:40, input-linkConfig.password:39:STRING:40, input-linkConfig.jdbcProperties:40:MAP]Type: LINK, From: Configs: config-fromJobConfig:14:[input-fromJobConfig.schemaName:41:STRING:50, input-fromJobConfig.tableName:42:STRING:50, input-fromJobConfig.sql:43:STRING:2000, input-fromJobConfig.columns:44:STRING:50, input-fromJobConfig.partitionColumn:45:STRING:50, input-fromJobConfig.allowNullValueInPartitionColumn:46:BOOLEAN, input-fromJobConfig.boundaryQuery:47:STRING:50]Type: JOB, To: Configs: config-toJobConfig:15:[input-toJobConfig.schemaName:48:STRING:50, input-toJobConfig.tableName:49:STRING:2000, input-toJobConfig.sql:50:STRING:50, input-toJobConfig.columns:51:STRING:50, input-toJobConfig.stageTableName:52:STRING:2000, input-toJobConfig.shouldClearStageTable:53:BOOLEAN]Type: JOB
2014-11-14 16:45:24,755 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)] Tx count-commit: 1, rollback: false
2014-11-14 16:45:24,755 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)] Tx count-close: 0, rollback: false
2014-11-14 16:45:24,755 INFO repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:111)] Attempting transaction commit
2014-11-14 16:45:24,776 INFO connector.ConnectorManager [org.apache.sqoop.connector.ConnectorManager.initialize(ConnectorManager.java:185)] Connectors loaded: {hdfs-connector={hdfs-connector:org.apache.sqoop.connector.hdfs.HdfsConnector:jar:file:/root/projects/sqoop/sqoop-1.99.4-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/sqoop-connector-hdfs-1.99.4.jar!/sqoopconnector.properties}, generic-jdbc-connector={generic-jdbc-connector:org.apache.sqoop.connector.jdbc.GenericJdbcConnector:jar:file:/root/projects/sqoop/sqoop-1.99.4-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/sqoop-connector-generic-jdbc-1.99.4.jar!/sqoopconnector.properties}}
2014-11-14 16:45:24,776 INFO tool.UpgradeTool [org.apache.sqoop.tools.tool.UpgradeTool.runToolWithConfiguration(UpgradeTool.java:44)] Initializing the Driver with upgrade option turned on.
2014-11-14 16:45:24,780 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)] Tx count-begin: 1, rollback: false
2014-11-14 16:45:24,780 DEBUG derby.DerbyRepositoryHandler [org.apache.sqoop.repository.derby.DerbyRepositoryHandler.findDriver(DerbyRepositoryHandler.java:1073)] Looking up Driver and config
2014-11-14 16:45:24,782 DEBUG derby.DerbyRepositoryHandler [org.apache.sqoop.repository.derby.DerbyRepositoryHandler.findDriver(DerbyRepositoryHandler.java:1129)] Looked up Driver and config
2014-11-14 16:45:24,783 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)] Tx count-commit: 1, rollback: false
2014-11-14 16:45:24,783 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)] Tx count-close: 0, rollback: false
2014-11-14 16:45:24,783 INFO repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:111)] Attempting transaction commit
2014-11-14 16:45:24,784 INFO driver.Driver [org.apache.sqoop.driver.Driver.initialize(Driver.java:134)] Driver initialized: OK
2014-11-14 16:45:24,784 INFO tool.UpgradeTool [org.apache.sqoop.tools.tool.UpgradeTool.runToolWithConfiguration(UpgradeTool.java:47)] Upgrade completed successfully.
2014-11-14 16
was (Author: vybs):
tested with the data set attached. still needs confirmation
[root@vbsqoop-1 bin]# ./sqoop.sh tool upgrade
Sqoop home directory: /root/projects/sqoop/sqoop-1.99.4-bin-hadoop200
Setting SQOOP_HTTP_PORT: 12000
Setting SQOOP_ADMIN_PORT: 12001
Using CATALINA_OPTS:
Adding to CATALINA_OPTS: -Dsqoop.http.port=12000 -Dsqoop.admin.port=12001
Nov 14, 2014 4:45:11 PM org.apache.catalina.startup.ClassLoaderFactory validateFile
WARNING: Problem with directory [/root/projects/sqoop/sqoop-1.99.4-bin-hadoop200/lib], exists: [false], isDirectory: [false], canRead: [false]
Sqoop tool executor:
Version: 1.99.4
Revision: 68d5beeab40b9d9da9f1c00da544185b161fdce6
Compiled on Fri Nov 14 15:46:21 PST 2014 by vbasavaraj
Running tool: class org.apache.sqoop.tools.tool.UpgradeTool
2014-11-14 16:45:12,123 INFO [main] core.PropertiesConfigurationProvider (PropertiesConfigurationProvider.java:initialize(96)) - Starting config file poller thread
log4j: Parsing for [root] with value=[WARN, file].
log4j: Level token is [WARN].
log4j: Category root set to WARN
log4j: Parsing appender named "file".
log4j: Parsing layout options for "file".
log4j: Setting property [conversionPattern] to [%d{ISO8601} %-5p %c{2} [%l] %m%n].
log4j: End of parsing for "file".
log4j: Setting property [file] to [@LOGDIR@/sqoop.log].
log4j: Setting property [maxBackupIndex] to [5].
log4j: Setting property [maxFileSize] to [25MB].
log4j: setFile called: @LOGDIR@/sqoop.log, true
log4j: setFile ended
log4j: Parsed "file" options.
log4j: Parsing for [org.apache.sqoop] with value=[DEBUG].
log4j: Level token is [DEBUG].
log4j: Category org.apache.sqoop set to DEBUG
log4j: Handling log4j.additivity.org.apache.sqoop=[null]
log4j: Parsing for [org.apache.derby] with value=[INFO].
log4j: Level token is [INFO].
log4j: Category org.apache.derby set to INFO
log4j: Handling log4j.additivity.org.apache.derby=[null]
log4j: Finished configuring.
Tool class org.apache.sqoop.tools.tool.UpgradeTool has finished correctly.
> Sqoop2: Driver missing during upgrade
> -------------------------------------
>
> Key: SQOOP-1736
> URL: https://issues.apache.org/jira/browse/SQOOP-1736
> Project: Sqoop
> Issue Type: Bug
> Components: sqoop2-framework
> Affects Versions: 1.99.4
> Reporter: Abraham Elmahrek
> Assignee: Veena Basavaraj
> Fix For: 1.99.4
>
> Attachments: 1993.tar.gz, SQOOP-1736-1.99.4-v2.patch, SQOOP-1736-1.99.4-v3.patch, SQOOP-1736-1.99.4.patch
>
>
> There are two issues:
> # The driver doesn't exist for existing jobs and links to reference when upgrading. During repository upgrade, a driver should be created (not done).
> # The original framework forms should be transformed to driver configs (done) and reference the driver configurable (not done).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)