You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by kopal niranjan <ni...@gmail.com> on 2015/10/29 10:25:27 UTC
Re: Review Request 39643: Review for SQOOP-2620:Sqoop fails with
Incremental Imports and Upserts in Teradata using Teradata JDBC Driver
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39643/
-----------------------------------------------------------
(Updated Oct. 29, 2015, 9:25 a.m.)
Review request for Sqoop and Venkat Ranganathan.
Summary (updated)
-----------------
Review for SQOOP-2620:Sqoop fails with Incremental Imports and Upserts in Teradata using Teradata JDBC Driver
Repository: sqoop-trunk
Description (updated)
-------
When sqoop incremental import is run over Teradata DB,
sqoop import --driver com.teradata.jdbc.TeraDriver --connect jdbc:teradata://10.126.62.46/DATABASE=SampleDb,TMODE=ANSI,LOGMECH=LDAP --username abc -P --table test1 --incremental lastmodified --check-column DAT --last-value '2014-10-22 07:00:00' --target-dir /user/$USER/hive_data_dynapart/test1dir
then the following error occurs:
15/09/24 03:57:10 ERROR manager.SqlManager: SQL exception accessing current timestamp: com.teradata.jdbc.jdbc_4.util.JDBCException: [Teradata Database] [TeraJDBC 14.10.00.26] [Error 3706] [SQLState 42000] Syntax error: expected something between '(' and ')'.
com.teradata.jdbc.jdbc_4.util.JDBCException: [Teradata Database] [TeraJDBC 14.10.00.26] [Error 3706] [SQLState 42000] Syntax error: expected something between '(' and ')'.
at com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDatabaseSQLException(ErrorFactory.java:307)
at com.teradata.jdbc.jdbc_4.statemachine.ReceiveInitSubState.action(ReceiveInitSubState.java:109)
at com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.subStateMachine(StatementReceiveState.java:314)
at com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.action(StatementReceiveState.java:202)
at com.teradata.jdbc.jdbc_4.statemachine.StatementController.runBody(StatementController.java:123)
at com.teradata.jdbc.jdbc_4.statemachine.StatementController.run(StatementController.java:114)
at com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:384)
at com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:326)
at com.teradata.jdbc.jdbc_4.TDStatement.doNonPrepExecuteQuery(TDStatement.java:314)
at com.teradata.jdbc.jdbc_4.TDStatement.executeQuery(TDStatement.java:1091)
at org.apache.sqoop.manager.SqlManager.getCurrentDbTimestamp(SqlManager.java:987)
at org.apache.sqoop.tool.ImportTool.initIncrementalConstraints(ImportTool.java:328)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:488)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
15/09/24 03:57:10 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: Could not get current time from database
at org.apache.sqoop.tool.ImportTool.initIncrementalConstraints(ImportTool.java:330)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:488)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
-----------------------------------------------------------------------------------------------------------------------------------------
Sqoop export fails when we perform upserts from HDFS to Teradta
sqoop export --driver com.teradata.jdbc.TeraDriver --connect jdbc:teradata://10.127.54.34/DATABASE=user_db,TMODE=ANSI,LOGMECH=LDAP --username abc -P --table upsert_tbl --export-dir /user/root/exportRec/ --update-key id --update-mode allowinsert
Above command fails with the following error:
15/10/16 02:51:19 ERROR tool.ExportTool: Error during export: Mixed update/insert is not supported against the target database yet
Diffs (updated)
-----
src/docs/user/connectors.txt c5ce4d6
src/java/org/apache/sqoop/manager/TeradataJdbcManager.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/TeradataUpsertOutputFormat.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBRecordReader.java b734e05
src/java/org/apache/sqoop/mapreduce/db/TeradataDataDrivenDBInputFormat.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/db/TeradataDataDrivenDBRecordReader.java PRE-CREATION
src/test/com/cloudera/sqoop/manager/TeradataJdbcManagerIncrementalImportTest.java PRE-CREATION
src/test/com/cloudera/sqoop/manager/TeradataJdbcManagerUpsertTest.java PRE-CREATION
src/test/org/apache/sqoop/mapreduce/TestTeradataUpsertOutputFormat.java PRE-CREATION
Diff: https://reviews.apache.org/r/39643/diff/
Testing
-------
yes
Thanks,
kopal niranjan
Re: Review Request 39643: Review for SQOOP-2620:Sqoop fails with
Incremental Imports and Upserts in Teradata using Teradata JDBC Driver
Posted by Venkat Ranganathan <n....@live.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39643/#review104649
-----------------------------------------------------------
All new files need to have apache license. Please make sure you run through ant checkstyle target and resolve any new violations
src/test/org/apache/sqoop/mapreduce/TestTeradataUpsertOutputFormat.java (line 30)
<https://reviews.apache.org/r/39643/#comment162897>
Not sure why are we using jdbc.hsqldb here
- Venkat Ranganathan
On Oct. 29, 2015, 2:25 a.m., kopal niranjan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39643/
> -----------------------------------------------------------
>
> (Updated Oct. 29, 2015, 2:25 a.m.)
>
>
> Review request for Sqoop and Venkat Ranganathan.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> When sqoop incremental import is run over Teradata DB,
> sqoop import --driver com.teradata.jdbc.TeraDriver --connect jdbc:teradata://10.126.62.46/DATABASE=SampleDb,TMODE=ANSI,LOGMECH=LDAP --username abc -P --table test1 --incremental lastmodified --check-column DAT --last-value '2014-10-22 07:00:00' --target-dir /user/$USER/hive_data_dynapart/test1dir
> then the following error occurs:
> 15/09/24 03:57:10 ERROR manager.SqlManager: SQL exception accessing current timestamp: com.teradata.jdbc.jdbc_4.util.JDBCException: [Teradata Database] [TeraJDBC 14.10.00.26] [Error 3706] [SQLState 42000] Syntax error: expected something between '(' and ')'.
> com.teradata.jdbc.jdbc_4.util.JDBCException: [Teradata Database] [TeraJDBC 14.10.00.26] [Error 3706] [SQLState 42000] Syntax error: expected something between '(' and ')'.
> at com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDatabaseSQLException(ErrorFactory.java:307)
> at com.teradata.jdbc.jdbc_4.statemachine.ReceiveInitSubState.action(ReceiveInitSubState.java:109)
> at com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.subStateMachine(StatementReceiveState.java:314)
> at com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.action(StatementReceiveState.java:202)
> at com.teradata.jdbc.jdbc_4.statemachine.StatementController.runBody(StatementController.java:123)
> at com.teradata.jdbc.jdbc_4.statemachine.StatementController.run(StatementController.java:114)
> at com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:384)
> at com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:326)
> at com.teradata.jdbc.jdbc_4.TDStatement.doNonPrepExecuteQuery(TDStatement.java:314)
> at com.teradata.jdbc.jdbc_4.TDStatement.executeQuery(TDStatement.java:1091)
> at org.apache.sqoop.manager.SqlManager.getCurrentDbTimestamp(SqlManager.java:987)
> at org.apache.sqoop.tool.ImportTool.initIncrementalConstraints(ImportTool.java:328)
> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:488)
> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
> 15/09/24 03:57:10 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: Could not get current time from database
> at org.apache.sqoop.tool.ImportTool.initIncrementalConstraints(ImportTool.java:330)
> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:488)
> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
> -----------------------------------------------------------------------------------------------------------------------------------------
>
>
> Sqoop export fails when we perform upserts from HDFS to Teradta
>
> sqoop export --driver com.teradata.jdbc.TeraDriver --connect jdbc:teradata://10.127.54.34/DATABASE=user_db,TMODE=ANSI,LOGMECH=LDAP --username abc -P --table upsert_tbl --export-dir /user/root/exportRec/ --update-key id --update-mode allowinsert
>
> Above command fails with the following error:
>
> 15/10/16 02:51:19 ERROR tool.ExportTool: Error during export: Mixed update/insert is not supported against the target database yet
>
>
> Diffs
> -----
>
> src/docs/user/connectors.txt c5ce4d6
> src/java/org/apache/sqoop/manager/TeradataJdbcManager.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/TeradataUpsertOutputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBRecordReader.java b734e05
> src/java/org/apache/sqoop/mapreduce/db/TeradataDataDrivenDBInputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/db/TeradataDataDrivenDBRecordReader.java PRE-CREATION
> src/test/com/cloudera/sqoop/manager/TeradataJdbcManagerIncrementalImportTest.java PRE-CREATION
> src/test/com/cloudera/sqoop/manager/TeradataJdbcManagerUpsertTest.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestTeradataUpsertOutputFormat.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/39643/diff/
>
>
> Testing
> -------
>
> yes
>
>
> Thanks,
>
> kopal niranjan
>
>
Re: Review Request 39643: Review for SQOOP-2620:Sqoop fails with
Incremental Imports and Upserts in Teradata using Teradata JDBC Driver
Posted by kopal niranjan <ni...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39643/
-----------------------------------------------------------
(Updated Nov. 5, 2015, 11:21 a.m.)
Review request for Sqoop and Venkat Ranganathan.
Changes
-------
added license and made changes according to previous review comments.
Repository: sqoop-trunk
Description
-------
When sqoop incremental import is run over Teradata DB,
sqoop import --driver com.teradata.jdbc.TeraDriver --connect jdbc:teradata://10.126.62.46/DATABASE=SampleDb,TMODE=ANSI,LOGMECH=LDAP --username abc -P --table test1 --incremental lastmodified --check-column DAT --last-value '2014-10-22 07:00:00' --target-dir /user/$USER/hive_data_dynapart/test1dir
then the following error occurs:
15/09/24 03:57:10 ERROR manager.SqlManager: SQL exception accessing current timestamp: com.teradata.jdbc.jdbc_4.util.JDBCException: [Teradata Database] [TeraJDBC 14.10.00.26] [Error 3706] [SQLState 42000] Syntax error: expected something between '(' and ')'.
com.teradata.jdbc.jdbc_4.util.JDBCException: [Teradata Database] [TeraJDBC 14.10.00.26] [Error 3706] [SQLState 42000] Syntax error: expected something between '(' and ')'.
at com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDatabaseSQLException(ErrorFactory.java:307)
at com.teradata.jdbc.jdbc_4.statemachine.ReceiveInitSubState.action(ReceiveInitSubState.java:109)
at com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.subStateMachine(StatementReceiveState.java:314)
at com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.action(StatementReceiveState.java:202)
at com.teradata.jdbc.jdbc_4.statemachine.StatementController.runBody(StatementController.java:123)
at com.teradata.jdbc.jdbc_4.statemachine.StatementController.run(StatementController.java:114)
at com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:384)
at com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:326)
at com.teradata.jdbc.jdbc_4.TDStatement.doNonPrepExecuteQuery(TDStatement.java:314)
at com.teradata.jdbc.jdbc_4.TDStatement.executeQuery(TDStatement.java:1091)
at org.apache.sqoop.manager.SqlManager.getCurrentDbTimestamp(SqlManager.java:987)
at org.apache.sqoop.tool.ImportTool.initIncrementalConstraints(ImportTool.java:328)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:488)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
15/09/24 03:57:10 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: Could not get current time from database
at org.apache.sqoop.tool.ImportTool.initIncrementalConstraints(ImportTool.java:330)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:488)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
-----------------------------------------------------------------------------------------------------------------------------------------
Sqoop export fails when we perform upserts from HDFS to Teradta
sqoop export --driver com.teradata.jdbc.TeraDriver --connect jdbc:teradata://10.127.54.34/DATABASE=user_db,TMODE=ANSI,LOGMECH=LDAP --username abc -P --table upsert_tbl --export-dir /user/root/exportRec/ --update-key id --update-mode allowinsert
Above command fails with the following error:
15/10/16 02:51:19 ERROR tool.ExportTool: Error during export: Mixed update/insert is not supported against the target database yet
Diffs (updated)
-----
src/docs/user/connectors.txt c5ce4d6
src/java/org/apache/sqoop/manager/TeradataJdbcManager.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/TeradataUpsertOutputFormat.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBRecordReader.java b734e05
src/java/org/apache/sqoop/mapreduce/db/TeradataDataDrivenDBInputFormat.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/db/TeradataDataDrivenDBRecordReader.java PRE-CREATION
src/test/com/cloudera/sqoop/manager/TeradataJdbcManagerIncrementalImportTest.java PRE-CREATION
src/test/com/cloudera/sqoop/manager/TeradataJdbcManagerUpsertTest.java PRE-CREATION
src/test/org/apache/sqoop/mapreduce/TestTeradataUpsertOutputFormat.java PRE-CREATION
Diff: https://reviews.apache.org/r/39643/diff/
Testing
-------
yes
Thanks,
kopal niranjan