You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Rakesh Sharma <ra...@expedia.com> on 2014/09/23 06:59:36 UTC
Stable release for sqoop
Hi,
We are planning to use Sqoop at Expedia but we might have to do some enhancements and customizations into Sqoop. We were planning to base our work with 1.4.5 branch but we found a couple of issues with code from that branch.
eg., We were not able to import data from mysql db using incremental import with 1.4.5 branch but it is running properly with latest code and even in 1.4.4 branch as well.
Could you please suggest the stable version after 1.4.5 which we can pick to use as base for our development.
Thanks and Regards,
Rakesh.
Re: Stable release for sqoop
Posted by Rakesh Sharma <ra...@expedia.com>.
Hi,
We are trying to generate an rpm for sqoop but it is taking time for us for the same.
If anyone has already created an rpm for sqoop, it will be of great help. Could you please provide us the spec file and send us step by step process to create rpm.
We are working on a tight schedule and any help will be really appreciated.
Thanks and Regards,
Rakesh.
From: Abraham Elmahrek <ab...@cloudera.com>>
Date: Friday, September 26, 2014 at 5:25 AM
To: Rakesh Sharma <ra...@expedia.com>>
Cc: Atul Gupta <at...@expedia.com>>, "dev@sqoop.apache.org<ma...@sqoop.apache.org>" <de...@sqoop.apache.org>>, Shashank Tandon <st...@expedia.com>>
Subject: Re: Stable release for sqoop
I'd still recommend Sqoop 1.4.5 or use a version of Sqoop that's provided by one of the vendors.
For reference though, could you provide the Schema of this table?
On Thu, Sep 25, 2014 at 3:41 AM, Rakesh Sharma <ra...@expedia.com>> wrote:
Hi Abe,
Thanks for the prompt reply and support. We are using the following sqoop command for code from 1.4.5 branch:
sqoop import -Dmapred.job.queue.name<http://Dmapred.job.queue.name>=default --connect jdbc:mysql://localhost/test --username root --password '' --table Temp_Employee --incremental lastmodified --check-column last_updated --last-value "2014-001-01 00:00:00" --hive-import --hive-table="Temp_Employee" -m 1
And we are getting the following exception :
14/09/25 03:35:11 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `Temp_Employee` AS t LIMIT 1
14/09/25 03:35:11 ERROR manager.SqlManager: Error reading from database: java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@69222fc3 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@69222fc3 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:934)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:931)
at com.mysql.jdbc.MysqlIO.checkForOutstandingStreamingData(MysqlIO.java:2735)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1899)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2619)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2569)
at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1524)
at com.mysql.jdbc.ConnectionImpl.getMaxBytesPerChar(ConnectionImpl.java:3003)
at com.mysql.jdbc.Field.getMaxBytesPerCharacter(Field.java:602)
at com.mysql.jdbc.ResultSetMetaData.getPrecision(ResultSetMetaData.java:445)
at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:285)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:240)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:226)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1773)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1578)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:601)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
14/09/25 03:35:11 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: No columns to generate for ClassWriter
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1584)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:601)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
The same command works on 1.4.6(trunk) and even on 1.4.4 version of sqoop. Could you please suggest what code is stable and we can make that as base of our development.
And Regarding the feature list, I will send you a separate mail with list of features. A couple of them are very generic and we are planning to contribute them back to community. May be we will need your help for that.
Thanks and Regards,
Rakesh.
From: Atul Gupta <at...@expedia.com>>
Date: Tuesday, September 23, 2014 at 12:42 PM
To: Abraham Elmahrek <ab...@cloudera.com>>
Cc: "dev@sqoop.apache.org<ma...@sqoop.apache.org>" <de...@sqoop.apache.org>>, Shashank Tandon <st...@expedia.com>>, Rakesh Sharma <ra...@expedia.com>>
Subject: Re: Stable release for sqoop
++rakesh
Regards,
Atul
On 23-Sep-2014, at 12:16 pm, Abraham Elmahrek <ab...@cloudera.com>> wrote:
Hey there,
Could you provide your incremental import command? Also, what security enhancements and customizations are you looking into? It might be something that fits generically into Sqoop. Sqoop 1.4.5 is the latest community release of Sqoop.
-Abe
On Mon, Sep 22, 2014 at 9:59 PM, Rakesh Sharma <ra...@expedia.com>> wrote:
Hi,
We are planning to use Sqoop at Expedia but we might have to do some enhancements and customizations into Sqoop. We were planning to base our work with 1.4.5 branch but we found a couple of issues with code from that branch.
eg., We were not able to import data from mysql db using incremental import with 1.4.5 branch but it is running properly with latest code and even in 1.4.4 branch as well.
Could you please suggest the stable version after 1.4.5 which we can pick to use as base for our development.
Thanks and Regards,
Rakesh.
Re: Stable release for sqoop
Posted by Venkat Ranganathan <vr...@hortonworks.com>.
Sorry for jumping on this late. What is the version of mysql server and
mysql java connector you are using. I think some older versions of server
and client (5.1.17 or earlier of client and some similar older versions of
server) exhibit the problem.
Thanks
Venkat
On Thu, Sep 25, 2014 at 8:48 PM, Rakesh Sharma <ra...@expedia.com>
wrote:
> Hi Abe,
>
> We have tried it with different tables but had same issue. For your
> reference, the table details are below :
>
> Hive table :
> =========
> hive> show create table test_employee;
> OK
> CREATE EXTERNAL TABLE `test_employee`(
> `id` int,
> `name` string,
> `salary` int,
> `country` string,
> `last_updated` timestamp)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> LOCATION
> 'hdfs://sandbox.hortonworks.com:8020/data/hive-ext/Test_Employee'
> TBLPROPERTIES (
> 'numFiles'='0',
> 'transient_lastDdlTime'='1411566509',
> 'COLUMN_STATS_ACCURATE'='false',
> 'totalSize'='0',
> 'numRows'='-1',
> 'rawDataSize'='-1')
> Time taken: 0.7 seconds, Fetched: 21 row(s)
>
> Mysql Table:
> ==========
> show create table Test_Employee;
>
> +---------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> | Table | Create Table
>
>
>
> |
>
> +---------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> | Test_Employee | CREATE TABLE `Test_Employee` (
> `id` int(11) DEFAULT NULL,
> `name` varchar(20) DEFAULT NULL,
> `salary` int(11) DEFAULT NULL,
> `country` varchar(20) DEFAULT NULL,
> `LAST_UPDATED` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE
> CURRENT_TIMESTAMP
> ) ENGINE=MyISAM DEFAULT CHARSET=latin1 |
>
> +---------------+————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————+
>
> Any help will be highly appreciated.
>
> Thanks and Regards,
> Rakesh.
>
>
> From: Abraham Elmahrek <ab...@cloudera.com>>
> Date: Friday, September 26, 2014 at 5:25 AM
> To: Rakesh Sharma <ra...@expedia.com>>
> Cc: Atul Gupta <at...@expedia.com>>, "
> dev@sqoop.apache.org<ma...@sqoop.apache.org>" <dev@sqoop.apache.org
> <ma...@sqoop.apache.org>>, Shashank Tandon <standon@expedia.com
> <ma...@expedia.com>>
> Subject: Re: Stable release for sqoop
>
> I'd still recommend Sqoop 1.4.5 or use a version of Sqoop that's provided
> by one of the vendors.
>
> For reference though, could you provide the Schema of this table?
>
> On Thu, Sep 25, 2014 at 3:41 AM, Rakesh Sharma <raksharma@expedia.com
> <ma...@expedia.com>> wrote:
> Hi Abe,
>
> Thanks for the prompt reply and support. We are using the following sqoop
> command for code from 1.4.5 branch:
>
> sqoop import -Dmapred.job.queue.name<http://Dmapred.job.queue.name>=default
> --connect jdbc:mysql://localhost/test --username root --password '' --table
> Temp_Employee --incremental lastmodified --check-column last_updated
> --last-value "2014-001-01 00:00:00" --hive-import
> --hive-table="Temp_Employee" -m 1
>
> And we are getting the following exception :
> 14/09/25 03:35:11 INFO manager.SqlManager: Executing SQL statement: SELECT
> t.* FROM `Temp_Employee` AS t LIMIT 1
> 14/09/25 03:35:11 ERROR manager.SqlManager: Error reading from database:
> java.sql.SQLException: Streaming result set
> com.mysql.jdbc.RowDataDynamic@69222fc3 is still active. No statements may
> be issued when any streaming result sets are open and in use on a given
> connection. Ensure that you have called .close() on any active streaming
> result sets before attempting more queries.
> java.sql.SQLException: Streaming result set
> com.mysql.jdbc.RowDataDynamic@69222fc3 is still active. No statements may
> be issued when any streaming result sets are open and in use on a given
> connection. Ensure that you have called .close() on any active streaming
> result sets before attempting more queries.
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:934)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:931)
> at
> com.mysql.jdbc.MysqlIO.checkForOutstandingStreamingData(MysqlIO.java:2735)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1899)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2619)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2569)
> at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1524)
> at
> com.mysql.jdbc.ConnectionImpl.getMaxBytesPerChar(ConnectionImpl.java:3003)
> at com.mysql.jdbc.Field.getMaxBytesPerCharacter(Field.java:602)
> at
> com.mysql.jdbc.ResultSetMetaData.getPrecision(ResultSetMetaData.java:445)
> at
> org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:285)
> at
> org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:240)
> at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:226)
> at
> org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
> at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1773)
> at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1578)
> at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:601)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
> 14/09/25 03:35:11 ERROR tool.ImportTool: Encountered IOException running
> import job: java.io.IOException: No columns to generate for ClassWriter
> at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1584)
> at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:601)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
>
> The same command works on 1.4.6(trunk) and even on 1.4.4 version of sqoop.
> Could you please suggest what code is stable and we can make that as base
> of our development.
>
> And Regarding the feature list, I will send you a separate mail with list
> of features. A couple of them are very generic and we are planning to
> contribute them back to community. May be we will need your help for that.
>
> Thanks and Regards,
> Rakesh.
>
>
>
> From: Atul Gupta <at...@expedia.com>>
> Date: Tuesday, September 23, 2014 at 12:42 PM
> To: Abraham Elmahrek <ab...@cloudera.com>>
> Cc: "dev@sqoop.apache.org<ma...@sqoop.apache.org>" <
> dev@sqoop.apache.org<ma...@sqoop.apache.org>>, Shashank Tandon <
> standon@expedia.com<ma...@expedia.com>>, Rakesh Sharma <
> raksharma@expedia.com<ma...@expedia.com>>
> Subject: Re: Stable release for sqoop
>
> ++rakesh
>
> Regards,
> Atul
>
> On 23-Sep-2014, at 12:16 pm, Abraham Elmahrek <abe@cloudera.com<mailto:
> abe@cloudera.com>> wrote:
>
> Hey there,
>
> Could you provide your incremental import command? Also, what security
> enhancements and customizations are you looking into? It might be something
> that fits generically into Sqoop. Sqoop 1.4.5 is the latest community
> release of Sqoop.
>
> -Abe
>
> On Mon, Sep 22, 2014 at 9:59 PM, Rakesh Sharma <raksharma@expedia.com
> <ma...@expedia.com>> wrote:
> Hi,
>
> We are planning to use Sqoop at Expedia but we might have to do some
> enhancements and customizations into Sqoop. We were planning to base our
> work with 1.4.5 branch but we found a couple of issues with code from that
> branch.
> eg., We were not able to import data from mysql db using incremental
> import with 1.4.5 branch but it is running properly with latest code and
> even in 1.4.4 branch as well.
>
> Could you please suggest the stable version after 1.4.5 which we can pick
> to use as base for our development.
>
> Thanks and Regards,
> Rakesh.
>
>
>
>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Stable release for sqoop
Posted by Rakesh Sharma <ra...@expedia.com>.
Hi Abe,
We have tried it with different tables but had same issue. For your reference, the table details are below :
Hive table :
=========
hive> show create table test_employee;
OK
CREATE EXTERNAL TABLE `test_employee`(
`id` int,
`name` string,
`salary` int,
`country` string,
`last_updated` timestamp)
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
'hdfs://sandbox.hortonworks.com:8020/data/hive-ext/Test_Employee'
TBLPROPERTIES (
'numFiles'='0',
'transient_lastDdlTime'='1411566509',
'COLUMN_STATS_ACCURATE'='false',
'totalSize'='0',
'numRows'='-1',
'rawDataSize'='-1')
Time taken: 0.7 seconds, Fetched: 21 row(s)
Mysql Table:
==========
show create table Test_Employee;
+---------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+---------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Test_Employee | CREATE TABLE `Test_Employee` (
`id` int(11) DEFAULT NULL,
`name` varchar(20) DEFAULT NULL,
`salary` int(11) DEFAULT NULL,
`country` varchar(20) DEFAULT NULL,
`LAST_UPDATED` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
) ENGINE=MyISAM DEFAULT CHARSET=latin1 |
+---------------+————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————+
Any help will be highly appreciated.
Thanks and Regards,
Rakesh.
From: Abraham Elmahrek <ab...@cloudera.com>>
Date: Friday, September 26, 2014 at 5:25 AM
To: Rakesh Sharma <ra...@expedia.com>>
Cc: Atul Gupta <at...@expedia.com>>, "dev@sqoop.apache.org<ma...@sqoop.apache.org>" <de...@sqoop.apache.org>>, Shashank Tandon <st...@expedia.com>>
Subject: Re: Stable release for sqoop
I'd still recommend Sqoop 1.4.5 or use a version of Sqoop that's provided by one of the vendors.
For reference though, could you provide the Schema of this table?
On Thu, Sep 25, 2014 at 3:41 AM, Rakesh Sharma <ra...@expedia.com>> wrote:
Hi Abe,
Thanks for the prompt reply and support. We are using the following sqoop command for code from 1.4.5 branch:
sqoop import -Dmapred.job.queue.name<http://Dmapred.job.queue.name>=default --connect jdbc:mysql://localhost/test --username root --password '' --table Temp_Employee --incremental lastmodified --check-column last_updated --last-value "2014-001-01 00:00:00" --hive-import --hive-table="Temp_Employee" -m 1
And we are getting the following exception :
14/09/25 03:35:11 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `Temp_Employee` AS t LIMIT 1
14/09/25 03:35:11 ERROR manager.SqlManager: Error reading from database: java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@69222fc3 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@69222fc3 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:934)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:931)
at com.mysql.jdbc.MysqlIO.checkForOutstandingStreamingData(MysqlIO.java:2735)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1899)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2619)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2569)
at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1524)
at com.mysql.jdbc.ConnectionImpl.getMaxBytesPerChar(ConnectionImpl.java:3003)
at com.mysql.jdbc.Field.getMaxBytesPerCharacter(Field.java:602)
at com.mysql.jdbc.ResultSetMetaData.getPrecision(ResultSetMetaData.java:445)
at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:285)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:240)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:226)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1773)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1578)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:601)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
14/09/25 03:35:11 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: No columns to generate for ClassWriter
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1584)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:601)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
The same command works on 1.4.6(trunk) and even on 1.4.4 version of sqoop. Could you please suggest what code is stable and we can make that as base of our development.
And Regarding the feature list, I will send you a separate mail with list of features. A couple of them are very generic and we are planning to contribute them back to community. May be we will need your help for that.
Thanks and Regards,
Rakesh.
From: Atul Gupta <at...@expedia.com>>
Date: Tuesday, September 23, 2014 at 12:42 PM
To: Abraham Elmahrek <ab...@cloudera.com>>
Cc: "dev@sqoop.apache.org<ma...@sqoop.apache.org>" <de...@sqoop.apache.org>>, Shashank Tandon <st...@expedia.com>>, Rakesh Sharma <ra...@expedia.com>>
Subject: Re: Stable release for sqoop
++rakesh
Regards,
Atul
On 23-Sep-2014, at 12:16 pm, Abraham Elmahrek <ab...@cloudera.com>> wrote:
Hey there,
Could you provide your incremental import command? Also, what security enhancements and customizations are you looking into? It might be something that fits generically into Sqoop. Sqoop 1.4.5 is the latest community release of Sqoop.
-Abe
On Mon, Sep 22, 2014 at 9:59 PM, Rakesh Sharma <ra...@expedia.com>> wrote:
Hi,
We are planning to use Sqoop at Expedia but we might have to do some enhancements and customizations into Sqoop. We were planning to base our work with 1.4.5 branch but we found a couple of issues with code from that branch.
eg., We were not able to import data from mysql db using incremental import with 1.4.5 branch but it is running properly with latest code and even in 1.4.4 branch as well.
Could you please suggest the stable version after 1.4.5 which we can pick to use as base for our development.
Thanks and Regards,
Rakesh.
Re: Stable release for sqoop
Posted by Abraham Elmahrek <ab...@cloudera.com>.
I'd still recommend Sqoop 1.4.5 or use a version of Sqoop that's provided
by one of the vendors.
For reference though, could you provide the Schema of this table?
On Thu, Sep 25, 2014 at 3:41 AM, Rakesh Sharma <ra...@expedia.com>
wrote:
> Hi Abe,
>
> Thanks for the prompt reply and support. We are using the following
> sqoop command for code from 1.4.5 branch:
>
> sqoop import -Dmapred.job.queue.name=default --connect
> jdbc:mysql://localhost/test --username root --password '' --table
> Temp_Employee --incremental lastmodified --check-column last_updated
> --last-value "2014-001-01 00:00:00" --hive-import
> --hive-table="Temp_Employee" -m 1
>
> And we are getting the following exception :
> 14/09/25 03:35:11 INFO manager.SqlManager: Executing SQL statement:
> SELECT t.* FROM `Temp_Employee` AS t LIMIT 1
> 14/09/25 03:35:11 ERROR manager.SqlManager: Error reading from database:
> java.sql.SQLException: Streaming result set
> com.mysql.jdbc.RowDataDynamic@69222fc3 is still active. No statements may
> be issued when any streaming result sets are open and in use on a given
> connection. Ensure that you have called .close() on any active streaming
> result sets before attempting more queries.
> java.sql.SQLException: Streaming result set
> com.mysql.jdbc.RowDataDynamic@69222fc3 is still active. No statements may
> be issued when any streaming result sets are open and in use on a given
> connection. Ensure that you have called .close() on any active streaming
> result sets before attempting more queries.
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:934)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:931)
> at
> com.mysql.jdbc.MysqlIO.checkForOutstandingStreamingData(MysqlIO.java:2735)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1899)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2619)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2569)
> at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1524)
> at
> com.mysql.jdbc.ConnectionImpl.getMaxBytesPerChar(ConnectionImpl.java:3003)
> at com.mysql.jdbc.Field.getMaxBytesPerCharacter(Field.java:602)
> at
> com.mysql.jdbc.ResultSetMetaData.getPrecision(ResultSetMetaData.java:445)
> at
> org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:285)
> at
> org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:240)
> at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:226)
> at
> org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
> at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1773)
> at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1578)
> at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:601)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
> 14/09/25 03:35:11 ERROR tool.ImportTool: Encountered IOException running
> import job: java.io.IOException: No columns to generate for ClassWriter
> at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1584)
> at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:601)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
>
> The same command works on 1.4.6(trunk) and even on 1.4.4 version of
> sqoop. Could you please suggest what code is stable and we can make that as
> base of our development.
>
> And Regarding the feature list, I will send you a separate mail with
> list of features. A couple of them are very generic and we are planning to
> contribute them back to community. May be we will need your help for that.
>
> Thanks and Regards,
> Rakesh.
>
>
>
> From: Atul Gupta <at...@expedia.com>
> Date: Tuesday, September 23, 2014 at 12:42 PM
> To: Abraham Elmahrek <ab...@cloudera.com>
> Cc: "dev@sqoop.apache.org" <de...@sqoop.apache.org>, Shashank Tandon <
> standon@expedia.com>, Rakesh Sharma <ra...@expedia.com>
> Subject: Re: Stable release for sqoop
>
> ++rakesh
>
> Regards,
> Atul
>
> On 23-Sep-2014, at 12:16 pm, Abraham Elmahrek <ab...@cloudera.com> wrote:
>
> Hey there,
>
> Could you provide your incremental import command? Also, what security
> enhancements and customizations are you looking into? It might be something
> that fits generically into Sqoop. Sqoop 1.4.5 is the latest community
> release of Sqoop.
>
> -Abe
>
> On Mon, Sep 22, 2014 at 9:59 PM, Rakesh Sharma <ra...@expedia.com>
> wrote:
>
>> Hi,
>>
>> We are planning to use Sqoop at Expedia but we might have to do some
>> enhancements and customizations into Sqoop. We were planning to base our
>> work with 1.4.5 branch but we found a couple of issues with code from that
>> branch.
>> eg., We were not able to import data from mysql db using incremental
>> import with 1.4.5 branch but it is running properly with latest code and
>> even in 1.4.4 branch as well.
>>
>> Could you please suggest the stable version after 1.4.5 which we can pick
>> to use as base for our development.
>>
>> Thanks and Regards,
>> Rakesh.
>>
>>
>>
>>
>
Re: Stable release for sqoop
Posted by Rakesh Sharma <ra...@expedia.com>.
Hi Abe,
Thanks for the prompt reply and support. We are using the following sqoop command for code from 1.4.5 branch:
sqoop import -Dmapred.job.queue.name=default --connect jdbc:mysql://localhost/test --username root --password '' --table Temp_Employee --incremental lastmodified --check-column last_updated --last-value "2014-001-01 00:00:00" --hive-import --hive-table="Temp_Employee" -m 1
And we are getting the following exception :
14/09/25 03:35:11 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `Temp_Employee` AS t LIMIT 1
14/09/25 03:35:11 ERROR manager.SqlManager: Error reading from database: java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@69222fc3 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@69222fc3 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:934)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:931)
at com.mysql.jdbc.MysqlIO.checkForOutstandingStreamingData(MysqlIO.java:2735)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1899)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2619)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2569)
at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1524)
at com.mysql.jdbc.ConnectionImpl.getMaxBytesPerChar(ConnectionImpl.java:3003)
at com.mysql.jdbc.Field.getMaxBytesPerCharacter(Field.java:602)
at com.mysql.jdbc.ResultSetMetaData.getPrecision(ResultSetMetaData.java:445)
at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:285)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:240)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:226)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1773)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1578)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:601)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
14/09/25 03:35:11 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: No columns to generate for ClassWriter
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1584)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:601)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
The same command works on 1.4.6(trunk) and even on 1.4.4 version of sqoop. Could you please suggest what code is stable and we can make that as base of our development.
And Regarding the feature list, I will send you a separate mail with list of features. A couple of them are very generic and we are planning to contribute them back to community. May be we will need your help for that.
Thanks and Regards,
Rakesh.
From: Atul Gupta <at...@expedia.com>>
Date: Tuesday, September 23, 2014 at 12:42 PM
To: Abraham Elmahrek <ab...@cloudera.com>>
Cc: "dev@sqoop.apache.org<ma...@sqoop.apache.org>" <de...@sqoop.apache.org>>, Shashank Tandon <st...@expedia.com>>, Rakesh Sharma <ra...@expedia.com>>
Subject: Re: Stable release for sqoop
++rakesh
Regards,
Atul
On 23-Sep-2014, at 12:16 pm, Abraham Elmahrek <ab...@cloudera.com>> wrote:
Hey there,
Could you provide your incremental import command? Also, what security enhancements and customizations are you looking into? It might be something that fits generically into Sqoop. Sqoop 1.4.5 is the latest community release of Sqoop.
-Abe
On Mon, Sep 22, 2014 at 9:59 PM, Rakesh Sharma <ra...@expedia.com>> wrote:
Hi,
We are planning to use Sqoop at Expedia but we might have to do some enhancements and customizations into Sqoop. We were planning to base our work with 1.4.5 branch but we found a couple of issues with code from that branch.
eg., We were not able to import data from mysql db using incremental import with 1.4.5 branch but it is running properly with latest code and even in 1.4.4 branch as well.
Could you please suggest the stable version after 1.4.5 which we can pick to use as base for our development.
Thanks and Regards,
Rakesh.
Re: Stable release for sqoop
Posted by Atul Gupta <at...@expedia.com>.
++rakesh
Regards,
Atul
On 23-Sep-2014, at 12:16 pm, Abraham Elmahrek <ab...@cloudera.com>> wrote:
Hey there,
Could you provide your incremental import command? Also, what security enhancements and customizations are you looking into? It might be something that fits generically into Sqoop. Sqoop 1.4.5 is the latest community release of Sqoop.
-Abe
On Mon, Sep 22, 2014 at 9:59 PM, Rakesh Sharma <ra...@expedia.com>> wrote:
Hi,
We are planning to use Sqoop at Expedia but we might have to do some enhancements and customizations into Sqoop. We were planning to base our work with 1.4.5 branch but we found a couple of issues with code from that branch.
eg., We were not able to import data from mysql db using incremental import with 1.4.5 branch but it is running properly with latest code and even in 1.4.4 branch as well.
Could you please suggest the stable version after 1.4.5 which we can pick to use as base for our development.
Thanks and Regards,
Rakesh.
Re: Stable release for sqoop
Posted by Abraham Elmahrek <ab...@cloudera.com>.
Hey there,
Could you provide your incremental import command? Also, what security
enhancements and customizations are you looking into? It might be something
that fits generically into Sqoop. Sqoop 1.4.5 is the latest community
release of Sqoop.
-Abe
On Mon, Sep 22, 2014 at 9:59 PM, Rakesh Sharma <ra...@expedia.com>
wrote:
> Hi,
>
> We are planning to use Sqoop at Expedia but we might have to do some
> enhancements and customizations into Sqoop. We were planning to base our
> work with 1.4.5 branch but we found a couple of issues with code from that
> branch.
> eg., We were not able to import data from mysql db using incremental
> import with 1.4.5 branch but it is running properly with latest code and
> even in 1.4.4 branch as well.
>
> Could you please suggest the stable version after 1.4.5 which we can pick
> to use as base for our development.
>
> Thanks and Regards,
> Rakesh.
>
>
>
>