You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by "wenzhi wang (JIRA)" <ji...@apache.org> on 2013/07/23 11:02:49 UTC

[jira] [Commented] (SQOOP-1054) New feature to support importing MySQL table via binary type splitter

    [ https://issues.apache.org/jira/browse/SQOOP-1054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13716246#comment-13716246 ] 

wenzhi wang commented on SQOOP-1054:
------------------------------------

Hi,

I tried your suggestion, but it doesn't work, could you have a check? thanks.

bl1253{wenzhiw}/local/wenzhiw/sqoop2/sqoop-1.4.3.bin__hadoop-0.23/bin>./sqoop --options-file ./import_3col_imsi.txt
Warning: /usr/lib/hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: $HADOOP_HOME is deprecated.

13/07/23 16:57:57 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
13/07/23 16:57:58 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
13/07/23 16:57:58 INFO tool.CodeGenTool: Beginning code generation
13/07/23 16:57:59 INFO manager.SqlManager: Executing SQL statement: select convert(mSubIdentificationNumberId using utf8), mobileNetworkCodeId,mobileCountryCodeId from IMSI where  (1 = 0)
13/07/23 16:57:59 INFO manager.SqlManager: Executing SQL statement: select convert(mSubIdentificationNumberId using utf8), mobileNetworkCodeId,mobileCountryCodeId from IMSI where  (1 = 0)
13/07/23 16:57:59 INFO manager.SqlManager: Executing SQL statement: select convert(mSubIdentificationNumberId using utf8), mobileNetworkCodeId,mobileCountryCodeId from IMSI where  (1 = 0)
13/07/23 16:57:59 ERROR orm.ClassWriter: No parser available for Java type org.apache.hadoop.io.BytesWritable
13/07/23 16:57:59 ERROR orm.ClassWriter: No parser available for Java type org.apache.hadoop.io.BytesWritable
13/07/23 16:57:59 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /local/wenzhiw/hadoop-1.0.4
Note: /tmp/sqoop-wenzhiw/compile/279635f8e9ce0376053e3f5c30674f2e/QueryResult.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
13/07/23 16:58:05 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-wenzhiw/compile/279635f8e9ce0376053e3f5c30674f2e/QueryResult.jar
13/07/23 16:58:06 INFO mapreduce.ImportJobBase: Beginning query import.
13/07/23 16:58:10 INFO mapred.JobClient: Cleaning up the staging area hdfs://135.252.181.253:50080/local/wenzhiw/hadoop-1.0.4/tmp/mapred/staging/wenzhiw/.staging/job_201307221026_0002
Exception in thread "main" java.lang.IncompatibleClassChangeError: Found class org.apache.hadoop.mapreduce.JobContext, but interface was expected
        at org.apache.sqoop.config.ConfigurationHelper.getJobNumMaps(ConfigurationHelper.java:53)
        at com.cloudera.sqoop.config.ConfigurationHelper.getJobNumMaps(ConfigurationHelper.java:36)
        at org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat.getSplits(DataDrivenDBInputFormat.java:121)
        at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:962)
        at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:979)
        at org.apache.hadoop.mapred.JobClient.access$600(JobClient.java:174)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:897)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:500)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530)
        at org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:173)
        at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:151)
        at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:221)
        at org.apache.sqoop.manager.SqlManager.importQuery(SqlManager.java:590)
        at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:405)
        at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
bl1253{wenzhiw}/local/wenzhiw/sqoop2/sqoop-1.4.3.bin__hadoop-0.23/bin>./sqoop --options-file ./import_3col_imsi.txt
Warning: /usr/lib/hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: $HADOOP_HOME is deprecated.

13/07/23 16:58:15 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
13/07/23 16:58:16 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
13/07/23 16:58:16 INFO tool.CodeGenTool: Beginning code generation
13/07/23 16:58:17 INFO manager.SqlManager: Executing SQL statement: select convert(mSubIdentificationNumberId using utf8), mobileNetworkCodeId,mobileCountryCodeId from IMSI where  (1 = 0)
13/07/23 16:58:17 INFO manager.SqlManager: Executing SQL statement: select convert(mSubIdentificationNumberId using utf8), mobileNetworkCodeId,mobileCountryCodeId from IMSI where  (1 = 0)
13/07/23 16:58:17 INFO manager.SqlManager: Executing SQL statement: select convert(mSubIdentificationNumberId using utf8), mobileNetworkCodeId,mobileCountryCodeId from IMSI where  (1 = 0)
13/07/23 16:58:17 ERROR orm.ClassWriter: No parser available for Java type org.apache.hadoop.io.BytesWritable
13/07/23 16:58:17 ERROR orm.ClassWriter: No parser available for Java type org.apache.hadoop.io.BytesWritable
13/07/23 16:58:17 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /local/wenzhiw/hadoop-1.0.4
Note: /tmp/sqoop-wenzhiw/compile/fd98a50ab15fb67c0b5b3322a7dd50d9/QueryResult.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
13/07/23 16:58:23 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-wenzhiw/compile/fd98a50ab15fb67c0b5b3322a7dd50d9/QueryResult.jar
13/07/23 16:58:23 INFO mapreduce.ImportJobBase: Beginning query import.
13/07/23 16:58:29 INFO mapred.JobClient: Cleaning up the staging area hdfs://135.252.181.253:50080/local/wenzhiw/hadoop-1.0.4/tmp/mapred/staging/wenzhiw/.staging/job_201307221026_0003
Exception in thread "main" java.lang.IncompatibleClassChangeError: Found class org.apache.hadoop.mapreduce.JobContext, but interface was expected
        at org.apache.sqoop.config.ConfigurationHelper.getJobNumMaps(ConfigurationHelper.java:53)
        at com.cloudera.sqoop.config.ConfigurationHelper.getJobNumMaps(ConfigurationHelper.java:36)
        at org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat.getSplits(DataDrivenDBInputFormat.java:121)
        at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:962)
        at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:979)
        at org.apache.hadoop.mapred.JobClient.access$600(JobClient.java:174)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:897)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:500)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530)
        at org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:173)
        at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:151)
        at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:221)
        at org.apache.sqoop.manager.SqlManager.importQuery(SqlManager.java:590)
        at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:405)
        at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
bl1253{wenzhiw}/local/wenzhiw/sqoop2/sqoop-1.4.3.bin__hadoop-0.23/bin>cat import_3col_imsi.txt
import
--connect
jdbc:mysql://135.252.158.250/hlr
--username
root
--password
admin
--query
'select convert(mSubIdentificationNumberId using utf8), mobileNetworkCodeId,mobileCountryCodeId from IMSI where $CONDITIONS'
--split-by
mSubIdentificationNumberId
--target-dir
/local/wenzhiw/3col_imsi

Best Regards,
Wenzhi

                
> New feature to support importing MySQL table via binary type splitter 
> ----------------------------------------------------------------------
>
>                 Key: SQOOP-1054
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1054
>             Project: Sqoop
>          Issue Type: New Feature
>          Components: connectors/mysql, tools
>    Affects Versions: 1.4.3
>         Environment: $uname -a
> Linux bl1253.cn.alcatel-lucent.com 2.6.18-128.7.1.el5PAE #1 SMP Mon Aug 24 08:41:30 EDT 2009 i686 athlon i386 GNU/Linux
>            Reporter: wenzhi wang
>
> I'm using (Sqoop 1.4.3 + Hadoop 1.0.4) handle data of our product, which employs MySQL 5.1.6 and most tables are defined with primary key on binary columns. Sqoop does not support binary type column as splitter now, and I have to import tables by restricting mapper number to 1 which lowers the performance. So, I open this feature to request Sqoop support binary type splitter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira