You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by "Prashant Gokhale (Created) (JIRA)" <ji...@apache.org> on 2012/03/30 02:31:27 UTC
[jira] [Created] (SQOOP-473) Sqoop import-all-tables fails with
MySQL connector
Sqoop import-all-tables fails with MySQL connector
--------------------------------------------------
Key: SQOOP-473
URL: https://issues.apache.org/jira/browse/SQOOP-473
Project: Sqoop
Issue Type: Bug
Components: connectors
Affects Versions: 1.4.1-incubating
Reporter: Prashant Gokhale
Sqoop command
sqoop import-all-tables --connect jdbc:mysql://xxxx/mysqltestdb2 --username=xxxx --warehouse-dir output-dir/alltables
Output:
{noformat}
12/03/29 10:42:20 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
12/03/29 10:42:21 INFO tool.CodeGenTool: Beginning code generation
12/03/29 10:42:21 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `testtable` AS t LIMIT 1
12/03/29 10:42:21 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.NullPointerException
java.lang.NullPointerException
at org.apache.sqoop.manager.ConnManager.getColumnTypeNames(ConnManager.java:306)
at org.apache.sqoop.manager.MySQLManager.toJavaType(MySQLManager.java:237)
at org.apache.sqoop.orm.ClassWriter.toJavaType(ClassWriter.java:253)
at org.apache.sqoop.orm.ClassWriter.generateFields(ClassWriter.java:480)
at org.apache.sqoop.orm.ClassWriter.generateClassForColumns(ClassWriter.java:1306)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1138)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:82)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:390)
at org.apache.sqoop.tool.ImportAllTablesTool.run(ImportAllTablesTool.java:64)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
at com.cloudera.sqoop.Sqoop.main(Sqoop.java:57)
{noformat}
I have not tested with other connectors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-473) Sqoop import-all-tables fails with
MySQL connector
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13245777#comment-13245777 ]
jiraposter@reviews.apache.org commented on SQOOP-473:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4584/#review6665
-----------------------------------------------------------
Ship it!
- Bilung
On 2012-04-03 19:13:20, Cheolsoo Park wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4584/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-04-03 19:13:20)
bq.
bq.
bq. Review request for Sqoop.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. All-tables import fails with MySQL:
bq.
bq. https://issues.apache.org/jira/browse/SQOOP-473
bq.
bq. The change include:
bq.
bq. 1) Modify toJavaType(), toHiveType(), and toAvroType() methods.
bq. 2) Add new unit test for MySQL all-tables import (subclassing TestAllTables).
bq. 3) Minor changes to TestAllTables class to make it available for non-Hsqldb servers.
bq.
bq.
bq. This addresses bug SQOOP-473.
bq. https://issues.apache.org/jira/browse/SQOOP-473
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/java/org/apache/sqoop/hive/TableDefWriter.java bfac391
bq. src/java/org/apache/sqoop/manager/ConnManager.java 26a2a82
bq. src/java/org/apache/sqoop/manager/MySQLManager.java 3254a2c
bq. src/java/org/apache/sqoop/manager/OracleManager.java 30401c5
bq. src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java a7ea834
bq. src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java ff39801
bq. src/java/org/apache/sqoop/orm/ClassWriter.java 986934d
bq. src/test/com/cloudera/sqoop/TestAllTables.java e6561e9
bq. src/test/com/cloudera/sqoop/manager/MySQLAllTablesTest.java e69de29
bq.
bq. Diff: https://reviews.apache.org/r/4584/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. 1) Verified that MySQL all-tables import is fixed using new unit test.
bq. 2) Ran ant test, ant test -Dthirdparty=true, and ant checkstyle.
bq.
bq.
bq. Thanks,
bq.
bq. Cheolsoo
bq.
bq.
> Sqoop import-all-tables fails with MySQL connector
> --------------------------------------------------
>
> Key: SQOOP-473
> URL: https://issues.apache.org/jira/browse/SQOOP-473
> Project: Sqoop
> Issue Type: Bug
> Components: connectors
> Affects Versions: 1.4.1-incubating
> Reporter: Prashant Gokhale
> Assignee: Cheolsoo Park
> Attachments: SQOOP-473.patch
>
>
> Sqoop command
> sqoop import-all-tables --connect jdbc:mysql://xxxx/mysqltestdb2 --username=xxxx --warehouse-dir output-dir/alltables
> Output:
> {noformat}
> 12/03/29 10:42:20 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
> 12/03/29 10:42:21 INFO tool.CodeGenTool: Beginning code generation
> 12/03/29 10:42:21 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `testtable` AS t LIMIT 1
> 12/03/29 10:42:21 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.NullPointerException
> java.lang.NullPointerException
> at org.apache.sqoop.manager.ConnManager.getColumnTypeNames(ConnManager.java:306)
> at org.apache.sqoop.manager.MySQLManager.toJavaType(MySQLManager.java:237)
> at org.apache.sqoop.orm.ClassWriter.toJavaType(ClassWriter.java:253)
> at org.apache.sqoop.orm.ClassWriter.generateFields(ClassWriter.java:480)
> at org.apache.sqoop.orm.ClassWriter.generateClassForColumns(ClassWriter.java:1306)
> at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1138)
> at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:82)
> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:390)
> at org.apache.sqoop.tool.ImportAllTablesTool.run(ImportAllTablesTool.java:64)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
> at com.cloudera.sqoop.Sqoop.main(Sqoop.java:57)
> {noformat}
> I have not tested with other connectors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (SQOOP-473) Sqoop import-all-tables fails with
MySQL connector
Posted by "Bilung Lee (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bilung Lee reassigned SQOOP-473:
--------------------------------
Assignee: Cheolsoo Park
> Sqoop import-all-tables fails with MySQL connector
> --------------------------------------------------
>
> Key: SQOOP-473
> URL: https://issues.apache.org/jira/browse/SQOOP-473
> Project: Sqoop
> Issue Type: Bug
> Components: connectors
> Affects Versions: 1.4.1-incubating
> Reporter: Prashant Gokhale
> Assignee: Cheolsoo Park
>
> Sqoop command
> sqoop import-all-tables --connect jdbc:mysql://xxxx/mysqltestdb2 --username=xxxx --warehouse-dir output-dir/alltables
> Output:
> {noformat}
> 12/03/29 10:42:20 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
> 12/03/29 10:42:21 INFO tool.CodeGenTool: Beginning code generation
> 12/03/29 10:42:21 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `testtable` AS t LIMIT 1
> 12/03/29 10:42:21 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.NullPointerException
> java.lang.NullPointerException
> at org.apache.sqoop.manager.ConnManager.getColumnTypeNames(ConnManager.java:306)
> at org.apache.sqoop.manager.MySQLManager.toJavaType(MySQLManager.java:237)
> at org.apache.sqoop.orm.ClassWriter.toJavaType(ClassWriter.java:253)
> at org.apache.sqoop.orm.ClassWriter.generateFields(ClassWriter.java:480)
> at org.apache.sqoop.orm.ClassWriter.generateClassForColumns(ClassWriter.java:1306)
> at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1138)
> at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:82)
> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:390)
> at org.apache.sqoop.tool.ImportAllTablesTool.run(ImportAllTablesTool.java:64)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
> at com.cloudera.sqoop.Sqoop.main(Sqoop.java:57)
> {noformat}
> I have not tested with other connectors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-473) Sqoop import-all-tables fails with
MySQL connector
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13245583#comment-13245583 ]
jiraposter@reviews.apache.org commented on SQOOP-473:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4584/#review6650
-----------------------------------------------------------
Thanks for the patch. Some comments below.
src/java/org/apache/sqoop/manager/MySQLManager.java
<https://reviews.apache.org/r/4584/#comment14346>
Also need to use the tableName parameter here.
src/java/org/apache/sqoop/manager/MySQLManager.java
<https://reviews.apache.org/r/4584/#comment14347>
Ditto.
- Bilung
On 2012-04-02 17:50:28, Cheolsoo Park wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4584/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-04-02 17:50:28)
bq.
bq.
bq. Review request for Sqoop.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. All-tables import fails with MySQL:
bq.
bq. https://issues.apache.org/jira/browse/SQOOP-473
bq.
bq. The change include:
bq.
bq. 1) Modify toJavaType(), toHiveType(), and toAvroType() methods.
bq. 2) Add new unit test for MySQL all-tables import (subclassing TestAllTables).
bq. 3) Minor changes to TestAllTables class to make it available for non-Hsqldb servers.
bq.
bq.
bq. This addresses bug SQOOP-473.
bq. https://issues.apache.org/jira/browse/SQOOP-473
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/java/org/apache/sqoop/hive/TableDefWriter.java bfac391
bq. src/java/org/apache/sqoop/manager/ConnManager.java 26a2a82
bq. src/java/org/apache/sqoop/manager/MySQLManager.java 3254a2c
bq. src/java/org/apache/sqoop/manager/OracleManager.java 30401c5
bq. src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java a7ea834
bq. src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java ff39801
bq. src/java/org/apache/sqoop/orm/ClassWriter.java 986934d
bq. src/test/com/cloudera/sqoop/TestAllTables.java e6561e9
bq. src/test/com/cloudera/sqoop/manager/MySQLAllTablesTest.java e69de29
bq.
bq. Diff: https://reviews.apache.org/r/4584/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. 1) Verified that MySQL all-tables import is fixed using new unit test.
bq. 2) Ran ant test, ant test -Dthirdparty=true, and ant checkstyle.
bq.
bq.
bq. Thanks,
bq.
bq. Cheolsoo
bq.
bq.
> Sqoop import-all-tables fails with MySQL connector
> --------------------------------------------------
>
> Key: SQOOP-473
> URL: https://issues.apache.org/jira/browse/SQOOP-473
> Project: Sqoop
> Issue Type: Bug
> Components: connectors
> Affects Versions: 1.4.1-incubating
> Reporter: Prashant Gokhale
> Assignee: Cheolsoo Park
>
> Sqoop command
> sqoop import-all-tables --connect jdbc:mysql://xxxx/mysqltestdb2 --username=xxxx --warehouse-dir output-dir/alltables
> Output:
> {noformat}
> 12/03/29 10:42:20 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
> 12/03/29 10:42:21 INFO tool.CodeGenTool: Beginning code generation
> 12/03/29 10:42:21 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `testtable` AS t LIMIT 1
> 12/03/29 10:42:21 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.NullPointerException
> java.lang.NullPointerException
> at org.apache.sqoop.manager.ConnManager.getColumnTypeNames(ConnManager.java:306)
> at org.apache.sqoop.manager.MySQLManager.toJavaType(MySQLManager.java:237)
> at org.apache.sqoop.orm.ClassWriter.toJavaType(ClassWriter.java:253)
> at org.apache.sqoop.orm.ClassWriter.generateFields(ClassWriter.java:480)
> at org.apache.sqoop.orm.ClassWriter.generateClassForColumns(ClassWriter.java:1306)
> at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1138)
> at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:82)
> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:390)
> at org.apache.sqoop.tool.ImportAllTablesTool.run(ImportAllTablesTool.java:64)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
> at com.cloudera.sqoop.Sqoop.main(Sqoop.java:57)
> {noformat}
> I have not tested with other connectors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-473) Sqoop import-all-tables fails with
MySQL connector
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13245822#comment-13245822 ]
Hudson commented on SQOOP-473:
------------------------------
Integrated in Sqoop-ant-jdk-1.6 #105 (See [https://builds.apache.org/job/Sqoop-ant-jdk-1.6/105/])
SQOOP-473 Sqoop import-all-tables fails with MySQL connector (Revision 1309163)
Result = SUCCESS
blee :
Files :
* /sqoop/trunk/src/java/org/apache/sqoop/hive/TableDefWriter.java
* /sqoop/trunk/src/java/org/apache/sqoop/manager/ConnManager.java
* /sqoop/trunk/src/java/org/apache/sqoop/manager/MySQLManager.java
* /sqoop/trunk/src/java/org/apache/sqoop/manager/OracleManager.java
* /sqoop/trunk/src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java
* /sqoop/trunk/src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java
* /sqoop/trunk/src/java/org/apache/sqoop/orm/ClassWriter.java
* /sqoop/trunk/src/test/com/cloudera/sqoop/TestAllTables.java
* /sqoop/trunk/src/test/com/cloudera/sqoop/manager/MySQLAllTablesTest.java
> Sqoop import-all-tables fails with MySQL connector
> --------------------------------------------------
>
> Key: SQOOP-473
> URL: https://issues.apache.org/jira/browse/SQOOP-473
> Project: Sqoop
> Issue Type: Bug
> Components: connectors
> Affects Versions: 1.4.1-incubating
> Reporter: Prashant Gokhale
> Assignee: Cheolsoo Park
> Fix For: 1.4.2-incubating
>
> Attachments: SQOOP-473.patch
>
>
> Sqoop command
> sqoop import-all-tables --connect jdbc:mysql://xxxx/mysqltestdb2 --username=xxxx --warehouse-dir output-dir/alltables
> Output:
> {noformat}
> 12/03/29 10:42:20 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
> 12/03/29 10:42:21 INFO tool.CodeGenTool: Beginning code generation
> 12/03/29 10:42:21 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `testtable` AS t LIMIT 1
> 12/03/29 10:42:21 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.NullPointerException
> java.lang.NullPointerException
> at org.apache.sqoop.manager.ConnManager.getColumnTypeNames(ConnManager.java:306)
> at org.apache.sqoop.manager.MySQLManager.toJavaType(MySQLManager.java:237)
> at org.apache.sqoop.orm.ClassWriter.toJavaType(ClassWriter.java:253)
> at org.apache.sqoop.orm.ClassWriter.generateFields(ClassWriter.java:480)
> at org.apache.sqoop.orm.ClassWriter.generateClassForColumns(ClassWriter.java:1306)
> at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1138)
> at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:82)
> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:390)
> at org.apache.sqoop.tool.ImportAllTablesTool.run(ImportAllTablesTool.java:64)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
> at com.cloudera.sqoop.Sqoop.main(Sqoop.java:57)
> {noformat}
> I have not tested with other connectors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-473) Sqoop import-all-tables fails with
MySQL connector
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13245619#comment-13245619 ]
jiraposter@reviews.apache.org commented on SQOOP-473:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4584/
-----------------------------------------------------------
(Updated 2012-04-03 19:13:20.381070)
Review request for Sqoop.
Changes
-------
1) Incorporate Bilung's comments in toHiveType() and to AvroType().
2) Remove a unnecessary method from MySQLAllTablesTest.
Summary
-------
All-tables import fails with MySQL:
https://issues.apache.org/jira/browse/SQOOP-473
The change include:
1) Modify toJavaType(), toHiveType(), and toAvroType() methods.
2) Add new unit test for MySQL all-tables import (subclassing TestAllTables).
3) Minor changes to TestAllTables class to make it available for non-Hsqldb servers.
This addresses bug SQOOP-473.
https://issues.apache.org/jira/browse/SQOOP-473
Diffs (updated)
-----
src/java/org/apache/sqoop/hive/TableDefWriter.java bfac391
src/java/org/apache/sqoop/manager/ConnManager.java 26a2a82
src/java/org/apache/sqoop/manager/MySQLManager.java 3254a2c
src/java/org/apache/sqoop/manager/OracleManager.java 30401c5
src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java a7ea834
src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java ff39801
src/java/org/apache/sqoop/orm/ClassWriter.java 986934d
src/test/com/cloudera/sqoop/TestAllTables.java e6561e9
src/test/com/cloudera/sqoop/manager/MySQLAllTablesTest.java e69de29
Diff: https://reviews.apache.org/r/4584/diff
Testing
-------
1) Verified that MySQL all-tables import is fixed using new unit test.
2) Ran ant test, ant test -Dthirdparty=true, and ant checkstyle.
Thanks,
Cheolsoo
> Sqoop import-all-tables fails with MySQL connector
> --------------------------------------------------
>
> Key: SQOOP-473
> URL: https://issues.apache.org/jira/browse/SQOOP-473
> Project: Sqoop
> Issue Type: Bug
> Components: connectors
> Affects Versions: 1.4.1-incubating
> Reporter: Prashant Gokhale
> Assignee: Cheolsoo Park
>
> Sqoop command
> sqoop import-all-tables --connect jdbc:mysql://xxxx/mysqltestdb2 --username=xxxx --warehouse-dir output-dir/alltables
> Output:
> {noformat}
> 12/03/29 10:42:20 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
> 12/03/29 10:42:21 INFO tool.CodeGenTool: Beginning code generation
> 12/03/29 10:42:21 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `testtable` AS t LIMIT 1
> 12/03/29 10:42:21 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.NullPointerException
> java.lang.NullPointerException
> at org.apache.sqoop.manager.ConnManager.getColumnTypeNames(ConnManager.java:306)
> at org.apache.sqoop.manager.MySQLManager.toJavaType(MySQLManager.java:237)
> at org.apache.sqoop.orm.ClassWriter.toJavaType(ClassWriter.java:253)
> at org.apache.sqoop.orm.ClassWriter.generateFields(ClassWriter.java:480)
> at org.apache.sqoop.orm.ClassWriter.generateClassForColumns(ClassWriter.java:1306)
> at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1138)
> at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:82)
> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:390)
> at org.apache.sqoop.tool.ImportAllTablesTool.run(ImportAllTablesTool.java:64)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
> at com.cloudera.sqoop.Sqoop.main(Sqoop.java:57)
> {noformat}
> I have not tested with other connectors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-473) Sqoop import-all-tables fails with
MySQL connector
Posted by "Cheolsoo Park (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242926#comment-13242926 ]
Cheolsoo Park commented on SQOOP-473:
-------------------------------------
While working on my patch, I also noticed that all-tables import does not work with Oracle at all. But I am not going to address Oracle in this Jira because it is a totally different issue. According to a test that I quickly wrote, it fails because the query that is executed to obtain all tables in the db returns more tables than what the user has actually created. A new Jira will be opened to track this issue.
Here is a call stack that I get from my test with Oracle for whom may be curious:
{code}
java.lang.IllegalArgumentException: Attempted to generate class with no columns!
at org.apache.sqoop.orm.ClassWriter.generateClassForColumns(ClassWriter.java:1251)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1139)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:82)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:390)
at org.apache.sqoop.tool.ImportAllTablesTool.run(ImportAllTablesTool.java:64)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at com.cloudera.sqoop.Sqoop.runSqoop(Sqoop.java:45)
at com.cloudera.sqoop.testutil.ImportJobTestCase.runImport(ImportJobTestCase.java:215)
at com.cloudera.sqoop.TestAllTables.testMultiTableImport(TestAllTables.java:106)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:243)
at junit.framework.TestSuite.run(TestSuite.java:238)
at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911)
at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768)
{code}
> Sqoop import-all-tables fails with MySQL connector
> --------------------------------------------------
>
> Key: SQOOP-473
> URL: https://issues.apache.org/jira/browse/SQOOP-473
> Project: Sqoop
> Issue Type: Bug
> Components: connectors
> Affects Versions: 1.4.1-incubating
> Reporter: Prashant Gokhale
> Assignee: Cheolsoo Park
>
> Sqoop command
> sqoop import-all-tables --connect jdbc:mysql://xxxx/mysqltestdb2 --username=xxxx --warehouse-dir output-dir/alltables
> Output:
> {noformat}
> 12/03/29 10:42:20 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
> 12/03/29 10:42:21 INFO tool.CodeGenTool: Beginning code generation
> 12/03/29 10:42:21 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `testtable` AS t LIMIT 1
> 12/03/29 10:42:21 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.NullPointerException
> java.lang.NullPointerException
> at org.apache.sqoop.manager.ConnManager.getColumnTypeNames(ConnManager.java:306)
> at org.apache.sqoop.manager.MySQLManager.toJavaType(MySQLManager.java:237)
> at org.apache.sqoop.orm.ClassWriter.toJavaType(ClassWriter.java:253)
> at org.apache.sqoop.orm.ClassWriter.generateFields(ClassWriter.java:480)
> at org.apache.sqoop.orm.ClassWriter.generateClassForColumns(ClassWriter.java:1306)
> at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1138)
> at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:82)
> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:390)
> at org.apache.sqoop.tool.ImportAllTablesTool.run(ImportAllTablesTool.java:64)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
> at com.cloudera.sqoop.Sqoop.main(Sqoop.java:57)
> {noformat}
> I have not tested with other connectors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-473) Sqoop import-all-tables fails with
MySQL connector
Posted by "Cheolsoo Park (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheolsoo Park updated SQOOP-473:
--------------------------------
Attachment: SQOOP-473.patch
> Sqoop import-all-tables fails with MySQL connector
> --------------------------------------------------
>
> Key: SQOOP-473
> URL: https://issues.apache.org/jira/browse/SQOOP-473
> Project: Sqoop
> Issue Type: Bug
> Components: connectors
> Affects Versions: 1.4.1-incubating
> Reporter: Prashant Gokhale
> Assignee: Cheolsoo Park
> Attachments: SQOOP-473.patch
>
>
> Sqoop command
> sqoop import-all-tables --connect jdbc:mysql://xxxx/mysqltestdb2 --username=xxxx --warehouse-dir output-dir/alltables
> Output:
> {noformat}
> 12/03/29 10:42:20 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
> 12/03/29 10:42:21 INFO tool.CodeGenTool: Beginning code generation
> 12/03/29 10:42:21 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `testtable` AS t LIMIT 1
> 12/03/29 10:42:21 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.NullPointerException
> java.lang.NullPointerException
> at org.apache.sqoop.manager.ConnManager.getColumnTypeNames(ConnManager.java:306)
> at org.apache.sqoop.manager.MySQLManager.toJavaType(MySQLManager.java:237)
> at org.apache.sqoop.orm.ClassWriter.toJavaType(ClassWriter.java:253)
> at org.apache.sqoop.orm.ClassWriter.generateFields(ClassWriter.java:480)
> at org.apache.sqoop.orm.ClassWriter.generateClassForColumns(ClassWriter.java:1306)
> at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1138)
> at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:82)
> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:390)
> at org.apache.sqoop.tool.ImportAllTablesTool.run(ImportAllTablesTool.java:64)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
> at com.cloudera.sqoop.Sqoop.main(Sqoop.java:57)
> {noformat}
> I have not tested with other connectors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-473) Sqoop import-all-tables fails with
MySQL connector
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242918#comment-13242918 ]
jiraposter@reviews.apache.org commented on SQOOP-473:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4584/
-----------------------------------------------------------
Review request for Sqoop.
Summary
-------
All-tables import fails with MySQL:
https://issues.apache.org/jira/browse/SQOOP-473
The change include:
1) Modify toJavaType(), toHiveType(), and toAvroType() methods.
2) Add new unit test for MySQL all-tables import (subclassing TestAllTables).
3) Minor changes to TestAllTables class to make it available for non-Hsqldb servers.
This addresses bug SQOOP-473.
https://issues.apache.org/jira/browse/SQOOP-473
Diffs
-----
src/java/org/apache/sqoop/hive/TableDefWriter.java bfac391
src/java/org/apache/sqoop/manager/ConnManager.java 26a2a82
src/java/org/apache/sqoop/manager/MySQLManager.java 3254a2c
src/java/org/apache/sqoop/manager/OracleManager.java 30401c5
src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java a7ea834
src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java ff39801
src/java/org/apache/sqoop/orm/ClassWriter.java 986934d
src/test/com/cloudera/sqoop/TestAllTables.java e6561e9
src/test/com/cloudera/sqoop/manager/MySQLAllTablesTest.java e69de29
Diff: https://reviews.apache.org/r/4584/diff
Testing
-------
1) Verified that MySQL all-tables import is fixed using new unit test.
2) Ran ant test, ant test -Dthirdparty=true, and ant checkstyle.
Thanks,
Cheolsoo
> Sqoop import-all-tables fails with MySQL connector
> --------------------------------------------------
>
> Key: SQOOP-473
> URL: https://issues.apache.org/jira/browse/SQOOP-473
> Project: Sqoop
> Issue Type: Bug
> Components: connectors
> Affects Versions: 1.4.1-incubating
> Reporter: Prashant Gokhale
> Assignee: Cheolsoo Park
>
> Sqoop command
> sqoop import-all-tables --connect jdbc:mysql://xxxx/mysqltestdb2 --username=xxxx --warehouse-dir output-dir/alltables
> Output:
> {noformat}
> 12/03/29 10:42:20 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
> 12/03/29 10:42:21 INFO tool.CodeGenTool: Beginning code generation
> 12/03/29 10:42:21 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `testtable` AS t LIMIT 1
> 12/03/29 10:42:21 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.NullPointerException
> java.lang.NullPointerException
> at org.apache.sqoop.manager.ConnManager.getColumnTypeNames(ConnManager.java:306)
> at org.apache.sqoop.manager.MySQLManager.toJavaType(MySQLManager.java:237)
> at org.apache.sqoop.orm.ClassWriter.toJavaType(ClassWriter.java:253)
> at org.apache.sqoop.orm.ClassWriter.generateFields(ClassWriter.java:480)
> at org.apache.sqoop.orm.ClassWriter.generateClassForColumns(ClassWriter.java:1306)
> at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1138)
> at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:82)
> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:390)
> at org.apache.sqoop.tool.ImportAllTablesTool.run(ImportAllTablesTool.java:64)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
> at com.cloudera.sqoop.Sqoop.main(Sqoop.java:57)
> {noformat}
> I have not tested with other connectors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-473) Sqoop import-all-tables fails with
MySQL connector
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13244375#comment-13244375 ]
jiraposter@reviews.apache.org commented on SQOOP-473:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4584/
-----------------------------------------------------------
(Updated 2012-04-02 17:50:28.671860)
Review request for Sqoop.
Summary
-------
All-tables import fails with MySQL:
https://issues.apache.org/jira/browse/SQOOP-473
The change include:
1) Modify toJavaType(), toHiveType(), and toAvroType() methods.
2) Add new unit test for MySQL all-tables import (subclassing TestAllTables).
3) Minor changes to TestAllTables class to make it available for non-Hsqldb servers.
This addresses bug SQOOP-473.
https://issues.apache.org/jira/browse/SQOOP-473
Diffs (updated)
-----
src/java/org/apache/sqoop/hive/TableDefWriter.java bfac391
src/java/org/apache/sqoop/manager/ConnManager.java 26a2a82
src/java/org/apache/sqoop/manager/MySQLManager.java 3254a2c
src/java/org/apache/sqoop/manager/OracleManager.java 30401c5
src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java a7ea834
src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java ff39801
src/java/org/apache/sqoop/orm/ClassWriter.java 986934d
src/test/com/cloudera/sqoop/TestAllTables.java e6561e9
src/test/com/cloudera/sqoop/manager/MySQLAllTablesTest.java e69de29
Diff: https://reviews.apache.org/r/4584/diff
Testing
-------
1) Verified that MySQL all-tables import is fixed using new unit test.
2) Ran ant test, ant test -Dthirdparty=true, and ant checkstyle.
Thanks,
Cheolsoo
> Sqoop import-all-tables fails with MySQL connector
> --------------------------------------------------
>
> Key: SQOOP-473
> URL: https://issues.apache.org/jira/browse/SQOOP-473
> Project: Sqoop
> Issue Type: Bug
> Components: connectors
> Affects Versions: 1.4.1-incubating
> Reporter: Prashant Gokhale
> Assignee: Cheolsoo Park
>
> Sqoop command
> sqoop import-all-tables --connect jdbc:mysql://xxxx/mysqltestdb2 --username=xxxx --warehouse-dir output-dir/alltables
> Output:
> {noformat}
> 12/03/29 10:42:20 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
> 12/03/29 10:42:21 INFO tool.CodeGenTool: Beginning code generation
> 12/03/29 10:42:21 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `testtable` AS t LIMIT 1
> 12/03/29 10:42:21 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.NullPointerException
> java.lang.NullPointerException
> at org.apache.sqoop.manager.ConnManager.getColumnTypeNames(ConnManager.java:306)
> at org.apache.sqoop.manager.MySQLManager.toJavaType(MySQLManager.java:237)
> at org.apache.sqoop.orm.ClassWriter.toJavaType(ClassWriter.java:253)
> at org.apache.sqoop.orm.ClassWriter.generateFields(ClassWriter.java:480)
> at org.apache.sqoop.orm.ClassWriter.generateClassForColumns(ClassWriter.java:1306)
> at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1138)
> at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:82)
> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:390)
> at org.apache.sqoop.tool.ImportAllTablesTool.run(ImportAllTablesTool.java:64)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
> at com.cloudera.sqoop.Sqoop.main(Sqoop.java:57)
> {noformat}
> I have not tested with other connectors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-473) Sqoop import-all-tables fails with
MySQL connector
Posted by "Cheolsoo Park (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242878#comment-13242878 ]
Cheolsoo Park commented on SQOOP-473:
-------------------------------------
Bilung suggested the following fix for this problem, and I like his suggestion.
For all-tables import, the table name is not specified, so in the getColumnTypeNames() method of ConnManager class, the 'else' path is taken in the following code:
{code}
if (null != tableName) {
// We're generating a class based on a table import.
columnTypeNames = getColumnTypeNamesForTable(tableName);
} else {
// This is based on an arbitrary query.
String query = sqlQuery;
if (query.indexOf(SqlManager.SUBSTITUTE_TOKEN) == -1) {
throw new RuntimeException("Query [" + query + "] must contain '"
+ SqlManager.SUBSTITUTE_TOKEN + "' in WHERE clause.");
}
columnTypeNames = getColumnTypeNamesForQuery(query);
}
{code}
However, this is wrong since all-tables import is not free form query import.
To fix the problem, I am going to modify the getColumnTypeNames() method so that tableName can be given as an argument in addition to columnName and sqlType. This change requires to modify the toJavaType() method as well, but at all the places where toJavaType() is called, tableName is available so only minor changes are needed.
In addition, I am going to modify the toHiveType() and toAvroType() methods so that the signature of these methods can stay consistent with that of toJavaType() (although this is not required).
> Sqoop import-all-tables fails with MySQL connector
> --------------------------------------------------
>
> Key: SQOOP-473
> URL: https://issues.apache.org/jira/browse/SQOOP-473
> Project: Sqoop
> Issue Type: Bug
> Components: connectors
> Affects Versions: 1.4.1-incubating
> Reporter: Prashant Gokhale
> Assignee: Cheolsoo Park
>
> Sqoop command
> sqoop import-all-tables --connect jdbc:mysql://xxxx/mysqltestdb2 --username=xxxx --warehouse-dir output-dir/alltables
> Output:
> {noformat}
> 12/03/29 10:42:20 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
> 12/03/29 10:42:21 INFO tool.CodeGenTool: Beginning code generation
> 12/03/29 10:42:21 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `testtable` AS t LIMIT 1
> 12/03/29 10:42:21 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.NullPointerException
> java.lang.NullPointerException
> at org.apache.sqoop.manager.ConnManager.getColumnTypeNames(ConnManager.java:306)
> at org.apache.sqoop.manager.MySQLManager.toJavaType(MySQLManager.java:237)
> at org.apache.sqoop.orm.ClassWriter.toJavaType(ClassWriter.java:253)
> at org.apache.sqoop.orm.ClassWriter.generateFields(ClassWriter.java:480)
> at org.apache.sqoop.orm.ClassWriter.generateClassForColumns(ClassWriter.java:1306)
> at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1138)
> at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:82)
> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:390)
> at org.apache.sqoop.tool.ImportAllTablesTool.run(ImportAllTablesTool.java:64)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
> at com.cloudera.sqoop.Sqoop.main(Sqoop.java:57)
> {noformat}
> I have not tested with other connectors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (SQOOP-473) Sqoop import-all-tables
fails with MySQL connector
Posted by "Cheolsoo Park (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242878#comment-13242878 ]
Cheolsoo Park edited comment on SQOOP-473 at 3/31/12 6:02 AM:
--------------------------------------------------------------
Bilung suggested the following fix for this problem, and I like his suggestion.
For all-tables import, the table name is not specified, so in the getColumnTypeNames() method of ConnManager, the 'else' path is taken in the following code:
{code}
if (null != tableName) {
// We're generating a class based on a table import.
columnTypeNames = getColumnTypeNamesForTable(tableName);
} else {
// This is based on an arbitrary query.
String query = sqlQuery;
if (query.indexOf(SqlManager.SUBSTITUTE_TOKEN) == -1) {
throw new RuntimeException("Query [" + query + "] must contain '"
+ SqlManager.SUBSTITUTE_TOKEN + "' in WHERE clause.");
}
columnTypeNames = getColumnTypeNamesForQuery(query);
}
{code}
However, this is wrong since all-tables import is not free form query import.
To fix the problem, I am going to modify the signature of the toJavaType() method so that tableName can be given as an argument in addition to columnName and sqlType and passed down to getColumnTypeNames(). Since tableName is available at all the places where toJavaType() is called, only minor changes are needed.
In addition, I am going to modify the toHiveType() and toAvroType() methods so that the signature of these methods can stay consistent with that of toJavaType() (although this is not required).
was (Author: cheolsoo):
Bilung suggested the following fix for this problem, and I like his suggestion.
For all-tables import, the table name is not specified, so in the getColumnTypeNames() method of ConnManager class, the 'else' path is taken in the following code:
{code}
if (null != tableName) {
// We're generating a class based on a table import.
columnTypeNames = getColumnTypeNamesForTable(tableName);
} else {
// This is based on an arbitrary query.
String query = sqlQuery;
if (query.indexOf(SqlManager.SUBSTITUTE_TOKEN) == -1) {
throw new RuntimeException("Query [" + query + "] must contain '"
+ SqlManager.SUBSTITUTE_TOKEN + "' in WHERE clause.");
}
columnTypeNames = getColumnTypeNamesForQuery(query);
}
{code}
However, this is wrong since all-tables import is not free form query import.
To fix the problem, I am going to modify the getColumnTypeNames() method so that tableName can be given as an argument in addition to columnName and sqlType. This change requires to modify the toJavaType() method as well, but at all the places where toJavaType() is called, tableName is available so only minor changes are needed.
In addition, I am going to modify the toHiveType() and toAvroType() methods so that the signature of these methods can stay consistent with that of toJavaType() (although this is not required).
> Sqoop import-all-tables fails with MySQL connector
> --------------------------------------------------
>
> Key: SQOOP-473
> URL: https://issues.apache.org/jira/browse/SQOOP-473
> Project: Sqoop
> Issue Type: Bug
> Components: connectors
> Affects Versions: 1.4.1-incubating
> Reporter: Prashant Gokhale
> Assignee: Cheolsoo Park
>
> Sqoop command
> sqoop import-all-tables --connect jdbc:mysql://xxxx/mysqltestdb2 --username=xxxx --warehouse-dir output-dir/alltables
> Output:
> {noformat}
> 12/03/29 10:42:20 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
> 12/03/29 10:42:21 INFO tool.CodeGenTool: Beginning code generation
> 12/03/29 10:42:21 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `testtable` AS t LIMIT 1
> 12/03/29 10:42:21 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.NullPointerException
> java.lang.NullPointerException
> at org.apache.sqoop.manager.ConnManager.getColumnTypeNames(ConnManager.java:306)
> at org.apache.sqoop.manager.MySQLManager.toJavaType(MySQLManager.java:237)
> at org.apache.sqoop.orm.ClassWriter.toJavaType(ClassWriter.java:253)
> at org.apache.sqoop.orm.ClassWriter.generateFields(ClassWriter.java:480)
> at org.apache.sqoop.orm.ClassWriter.generateClassForColumns(ClassWriter.java:1306)
> at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1138)
> at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:82)
> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:390)
> at org.apache.sqoop.tool.ImportAllTablesTool.run(ImportAllTablesTool.java:64)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
> at com.cloudera.sqoop.Sqoop.main(Sqoop.java:57)
> {noformat}
> I have not tested with other connectors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira