Re: Stable release for sqoop

[Rakesh Sharma <ra...@expedia.com>]

Hi,

We are trying to generate an rpm for sqoop but it is taking time for us for the same.
If anyone has already created an rpm for sqoop, it will be of great help. Could you please provide us the spec file and send us step by step process to create rpm.

We are working on a  tight schedule and any help will be really appreciated.

Thanks and Regards,
Rakesh.


From: Abraham Elmahrek <ab...@cloudera.com>>
Date: Friday, September 26, 2014 at 5:25 AM
To: Rakesh Sharma <ra...@expedia.com>>
Cc: Atul Gupta <at...@expedia.com>>, "dev@sqoop.apache.org<ma...@sqoop.apache.org>" <de...@sqoop.apache.org>>, Shashank Tandon <st...@expedia.com>>
Subject: Re: Stable release for sqoop

I'd still recommend Sqoop 1.4.5 or use a version of Sqoop that's provided by one of the vendors.

For reference though, could you provide the Schema of this table?

On Thu, Sep 25, 2014 at 3:41 AM, Rakesh Sharma <ra...@expedia.com>> wrote:
Hi Abe,

Thanks for the prompt reply and support. We are using the following sqoop command  for code from 1.4.5 branch:

sqoop import -Dmapred.job.queue.name<http://Dmapred.job.queue.name>=default --connect jdbc:mysql://localhost/test --username root --password '' --table Temp_Employee  --incremental lastmodified --check-column last_updated  --last-value "2014-001-01 00:00:00"  --hive-import  --hive-table="Temp_Employee" -m 1

And we are getting the following exception :
14/09/25 03:35:11 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `Temp_Employee` AS t LIMIT 1
14/09/25 03:35:11 ERROR manager.SqlManager: Error reading from database: java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@69222fc3 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@69222fc3 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:934)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:931)
at com.mysql.jdbc.MysqlIO.checkForOutstandingStreamingData(MysqlIO.java:2735)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1899)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2619)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2569)
at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1524)
at com.mysql.jdbc.ConnectionImpl.getMaxBytesPerChar(ConnectionImpl.java:3003)
at com.mysql.jdbc.Field.getMaxBytesPerCharacter(Field.java:602)
at com.mysql.jdbc.ResultSetMetaData.getPrecision(ResultSetMetaData.java:445)
at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:285)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:240)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:226)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1773)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1578)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:601)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
14/09/25 03:35:11 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: No columns to generate for ClassWriter
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1584)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:601)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)

The same command works on 1.4.6(trunk) and even on 1.4.4 version of sqoop. Could you please suggest what code is stable and we can make that as base of our development.

And Regarding the feature list, I will send you a separate mail with list of features. A couple of them are very generic and we are planning to contribute them back to community. May be we will need your help for that.

Thanks and Regards,
Rakesh.



From: Atul Gupta <at...@expedia.com>>
Date: Tuesday, September 23, 2014 at 12:42 PM
To: Abraham Elmahrek <ab...@cloudera.com>>
Cc: "dev@sqoop.apache.org<ma...@sqoop.apache.org>" <de...@sqoop.apache.org>>, Shashank Tandon <st...@expedia.com>>, Rakesh Sharma <ra...@expedia.com>>
Subject: Re: Stable release for sqoop

++rakesh

Regards,
Atul

On 23-Sep-2014, at 12:16 pm, Abraham Elmahrek <ab...@cloudera.com>> wrote:

Hey there,

Could you provide your incremental import command? Also, what security enhancements and customizations are you looking into? It might be something that fits generically into Sqoop. Sqoop 1.4.5 is the latest community release of Sqoop.

-Abe

On Mon, Sep 22, 2014 at 9:59 PM, Rakesh Sharma <ra...@expedia.com>> wrote:
Hi,

We are planning to use Sqoop at Expedia but we might have to do some enhancements and customizations into Sqoop. We were planning to base our work with 1.4.5 branch but we found a couple of issues with code from that branch.
eg., We were not able to import data from mysql db using incremental import with 1.4.5 branch but it is running properly with latest code and even in 1.4.4 branch as well.

Could you please suggest the stable version after 1.4.5 which we can pick to use as base for our development.

Thanks and Regards,
Rakesh.