You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@sqoop.apache.org by John Ho <jo...@gmail.com> on 2014/03/14 04:56:41 UTC
Does sqoop2 support Teradata?
Hi Sqoop Team,
Does sqoop2 support importing data from Teradata?
I'm using CDH4.5 and not sure where to put Teradata's JDBC driver files.
I upload tdgssconfig.jar and terajdbc4.jar to /var/lib/sqoop2 and add
"-classpath
/var/lib/sqoop2/tdgssconfig.jar" to Java Configuration Options for
TaskTracker for TaskTracker but it does not work as expected.
I also upload the jar files to
/opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/sqoop2/webapps/sqoop/WEB-INF/lib/
but it does not work.
There's no issues if I use sqoop 1.
Thanks and regards,
John Ho
===
Task Log (CDH4.5.0 and Teradata Express 14.10 on SLES11):
org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs
during extractor run at
org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:101) at
org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) at
org.apache.hadoop.mapred.Child$4.run(Child.java:268) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:415) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by:
org.apache.sqoop.common.SqoopException: GENERIC_JDBC_CONNECTOR_0002:Unable
to execute the SQL statement at
org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:59)
at
org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:50)
at
org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.ja
org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs
during extractor run at
org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:101) at
org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) at
org.apache.hadoop.mapred.Child$4.run(Child.java:268) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:415) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by:
org.apache.sqoop.common.SqoopException: GENERIC_JDBC_CONNECTOR_0002:Unable
to execute the SQL statement at
org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:59)
at
org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:50)
at
org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.ja
org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs
during extractor run at
org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:101) at
org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) at
org.apache.hadoop.mapred.Child$4.run(Child.java:268) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:415) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by:
org.apache.sqoop.common.SqoopException: GENERIC_JDBC_CONNECTOR_0002:Unable
to execute the SQL statement at
org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:59)
at
org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:50)
at
org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.ja
org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs
during extractor run at
org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:101) at
org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) at
org.apache.hadoop.mapred.Child$4.run(Child.java:268) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:415) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by:
org.apache.sqoop.common.SqoopException: GENERIC_JDBC_CONNECTOR_0002:Unable
to execute the SQL statement at
org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:59)
at
org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:50)
at
org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.ja
Re: Does sqoop2 support Teradata?
Posted by John Ho <jo...@gmail.com>.
Thanks so much Jarcec for the reply.
It's working now with INT as a partition column instead of VARCHAR.
Thanks and regards,
John
On Tue, Mar 18, 2014 at 11:46 PM, Jarek Jarcec Cecho <ja...@apache.org>wrote:
> You seems to be using VARCHAR column for creating splits, which is
> generally very bad idea. I would strongly advise you to use a number column
> instead.
>
> There is many things that can go wrong with using VARCHAR as a partition
> column. Looking at the exception my wild guess would be charset - are you
> explicitly stating that the connection should be in UTF? If not than
> Teradata default charset is ASCII that clearly won't work.
>
>
> http://developer.teradata.com/doc/connectivity/jdbc/reference/current/jdbcug_chapter_2.html#URL_CHARSET
>
> Jarcec
>
> On Tue, Mar 18, 2014 at 01:26:39PM +0800, John Ho wrote:
> > Many thanks Jarcec for the reply.
> >
> > I merged 2 jar files as you suggested and sqoop2 able to connect to
> > Teradata.
> >
> > This time it got issue when extracting data from Teradata using small
> data
> > set (20 rows)
> > Below are details and logs collected from Hue GUI for the failed task.
> > Sqoop1 has no issues so far.
> >
> >
> > Thanks and regards,
> > John
> >
> > =======
> >
> > Schema:
> > CREATE TABLE dailyprices (
> > stockname varchar(256) NOT NULL default '',
> > tradedate varchar(256) NOT NULL default '',
> > openprice float default NULL,
> > highprice float default NULL,
> > lowprice float default NULL,
> > closeprice float default NULL,
> > tradevolume bigint default NULL,
> > adjcloseprice float default NULL,
> > PRIMARY KEY (stockname,tradedate)
> > );
> >
> > ====
> > Data for testing:
> >
> > HPQ|2014-02-07|28.7|29.16|28.69|29.07|7537300|29.07
> > HPQ|2014-02-06|28.23|28.66|28.2|28.49|6693700|28.49
> > HPQ|2014-02-05|28.17|28.44|27.9|28.01|10278000|28.01
> > HPQ|2014-02-04|28.14|28.41|27.89|28.33|9581300|28.33
> > HPQ|2014-02-03|29.06|29.29|27.96|28.04|14655400|28.04
> > HPQ|2014-01-31|28.94|29.19|28.74|29|12934300|29
> > HPQ|2014-01-30|29.15|29.42|29.08|29.25|9095700|29.25
> > HPQ|2014-01-29|28.92|29.15|28.75|29.02|13915700|29.02
> > HPQ|2014-01-28|28.57|29.08|28.49|29|12407000|29
> > HPQ|2014-01-27|28.53|29.09|28.38|28.6|15924800|28.6
> > IBM|2014-02-07|175.64|177.56|175.07|177.25|4692900|177.25
> > IBM|2014-02-06|173.97|174.85|173.79|174.67|4292200|174.67
> > IBM|2014-02-05|172.19|174.97|172.19|174.24|4712300|173.29
> > IBM|2014-02-04|173.53|173.75|172.36|172.84|4349800|171.9
> > IBM|2014-02-03|176.02|176.02|172.72|172.9|7186800|171.96
> > IBM|2014-01-31|176.11|177.84|175.34|176.68|5193400|175.72
> > IBM|2014-01-30|177.17|177.86|176.36|177.36|4853700|176.39
> > IBM|2014-01-29|175.98|178.53|175.89|176.4|4970900|175.44
> > IBM|2014-01-28|178.05|178.45|176.16|176.85|5333300|175.89
> > IBM|2014-01-27|179.61|179.65|177.66|177.9|5208600|176.93
> >
> > ====
> > Sqoop2's task diagnostic log (from hue UI):
> > org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs
> > during extractor run
> > at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:101)
> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
> > at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:415)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
> > at org.apache.hadoop.mapred.Child.main(Child.java:262)
> > Caused by: org.apache.sqoop.common.SqoopException:
> > GENERIC_JDBC_CONNECTOR_0002:Unable to execute the SQL statement
> > at
> >
> org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:59)
> > at
> >
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:50)
> > at
> >
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.ja
> >
> > ====
> > stderr (from hue UI):
> > 2014-03-18 13:10:33,969 [main] INFO org.apache.sqoop.job.mr.SqoopMapper
> -
> > Starting progress service
> > 2014-03-18 13:10:33,970 [main] INFO org.apache.sqoop.job.mr.SqoopMapper
> -
> > Running extractor class
> > org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor
> > 2014-03-18 13:10:33,978 [pool-2-thread-1] DEBUG
> > org.apache.sqoop.job.mr.ProgressRunnable - Auto-progress thread
> reporting
> > progress
> > 2014-03-18 13:10:35,454 [main] INFO
> > org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor - Using
> query:
> > SELECT stockname,tradedate,closeprice,tradevolume FROM dailyprices WHERE
> > 'HPQ' <= stockname AND stockname < 'Hó EURO '
> > 2014-03-18 13:10:35,492 [main] INFO org.apache.sqoop.job.mr.SqoopMapper
> -
> > Stopping progress service
> >
> >
> > ====
> > syslog (from hue UI):
> > 2014-03-18 13:10:31,622 WARN mapreduce.Counters: Group
> > org.apache.hadoop.mapred.Task$Counter is deprecated. Use
> > org.apache.hadoop.mapreduce.TaskCounter instead
> > 2014-03-18 13:10:33,076 WARN org.apache.hadoop.conf.Configuration:
> > session.id is deprecated. Instead, use dfs.metrics.session-id
> > 2014-03-18 13:10:33,077 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> > Initializing JVM Metrics with processName=MAP, sessionId=
> > 2014-03-18 13:10:33,458 INFO org.apache.hadoop.util.ProcessTree: setsid
> > exited with exit code 0
> > 2014-03-18 13:10:33,482 INFO org.apache.hadoop.mapred.Task: Using
> > ResourceCalculatorPlugin :
> > org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1f28706
> > 2014-03-18 13:10:33,774 INFO org.apache.hadoop.mapred.MapTask: Processing
> > split: org.apache.sqoop.job.mr.SqoopSplit@72c1d428
> > 2014-03-18 13:10:33,781 INFO org.apache.hadoop.mapred.MapTask: Map output
> > collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
> > 2014-03-18 13:10:33,785 INFO org.apache.hadoop.mapred.MapTask:
> io.sort.mb =
> > 136
> > 2014-03-18 13:10:33,925 INFO org.apache.hadoop.mapred.MapTask: data
> buffer
> > = 108380824/135476032
> > 2014-03-18 13:10:33,925 INFO org.apache.hadoop.mapred.MapTask: record
> > buffer = 356515/445644
> > 2014-03-18 13:10:33,969 INFO org.apache.sqoop.job.mr.SqoopMapper:
> Starting
> > progress service
> > 2014-03-18 13:10:33,970 INFO org.apache.sqoop.job.mr.SqoopMapper: Running
> > extractor class
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor
> > 2014-03-18 13:10:33,978 DEBUG org.apache.sqoop.job.mr.ProgressRunnable:
> > Auto-progress thread reporting progress
> > 2014-03-18 13:10:35,454 INFO
> > org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor: Using query:
> > SELECT stockname,tradedate,closeprice,tradevolume FROM dailyprices WHERE
> > 'HPQ' <= stockname AND stockname < 'Hó EURO '
> > 2014-03-18 13:10:35,492 INFO org.apache.sqoop.job.mr.SqoopMapper:
> Stopping
> > progress service
> > 2014-03-18 13:10:35,495 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
> > Initializing logs' truncater with mapRetainSize=-1 and
> reduceRetainSize=-1
> > 2014-03-18 13:10:35,501 WARN org.apache.hadoop.mapred.Child: Error
> running
> > child
> > org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs
> > during extractor run
> > at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:101)
> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
> > at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:415)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
> > at org.apache.hadoop.mapred.Child.main(Child.java:262)
> > Caused by: org.apache.sqoop.common.SqoopException:
> > GENERIC_JDBC_CONNECTOR_0002:Unable to execute the SQL statement
> > at
> >
> org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:59)
> > at
> >
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:50)
> > at
> >
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:31)
> > at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:96)
> > ... 7 more
> > Caused by: com.teradata.jdbc.jdbc_4.util.JDBCException: [Teradata
> Database]
> > [TeraJDBC 14.10.00.26] [Error 6705] [SQLState HY000] An illegally formed
> > character string was encountered during translation.
> > at
> >
> com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDatabaseSQLException(ErrorFactory.java:307)
> > at
> >
> com.teradata.jdbc.jdbc_4.statemachine.ReceiveInitSubState.action(ReceiveInitSubState.java:109)
> > at
> >
> com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.subStateMachine(StatementReceiveState.java:314)
> > at
> >
> com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.action(StatementReceiveState.java:202)
> > at
> >
> com.teradata.jdbc.jdbc_4.statemachine.StatementController.runBody(StatementController.java:123)
> > at
> >
> com.teradata.jdbc.jdbc_4.statemachine.StatementController.run(StatementController.java:114)
> > at
> >
> com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:384)
> > at
> >
> com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:326)
> > at
> >
> com.teradata.jdbc.jdbc_4.TDStatement.doNonPrepExecuteQuery(TDStatement.java:314)
> > at
> com.teradata.jdbc.jdbc_4.TDStatement.executeQuery(TDStatement.java:1091)
> > at
> >
> org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:56)
> > ... 10 more
> > 2014-03-18 13:10:35,506 INFO org.apache.hadoop.mapred.Task: Runnning
> > cleanup for the task
> >
> >
> > On Sat, Mar 15, 2014 at 1:17 AM, Jarek Jarcec Cecho <jarcec@apache.org
> >wrote:
> >
> > > Hi John,
> > > Sqoop2 is not polluting the DistributedCache like Sqoop 1 with putting
> > > entire classpath to the MapReduce job. Instead we are propagating only
> jars
> > > that are required. Sadly this mechanism is not working correctly in
> > > Teradata case as the JDBC driver consist of two jars that are not
> > > referencing each other - the dependency is something that user have to
> > > know, not something that jar file exposes. As a result Sqoop will
> propagate
> > > only one of them to the mapreduce job and that will in turn fail. This
> will
> > > be fixed later when we introduce special Teradata connector. For now,
> you
> > > could merge both Teradata JDBC driver jars together and put the
> resulting
> > > jar into /var/log/sqoop2/.
> > >
> > > Jarcec
> > >
> > > On Fri, Mar 14, 2014 at 11:56:41AM +0800, John Ho wrote:
> > > > Hi Sqoop Team,
> > > >
> > > > Does sqoop2 support importing data from Teradata?
> > > >
> > > > I'm using CDH4.5 and not sure where to put Teradata's JDBC driver
> files.
> > > >
> > > > I upload tdgssconfig.jar and terajdbc4.jar to /var/lib/sqoop2 and add
> > > > "-classpath
> > > > /var/lib/sqoop2/tdgssconfig.jar" to Java Configuration Options for
> > > > TaskTracker for TaskTracker but it does not work as expected.
> > > >
> > > > I also upload the jar files to
> > > >
> > >
> /opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/sqoop2/webapps/sqoop/WEB-INF/lib/
> > > > but it does not work.
> > > >
> > > > There's no issues if I use sqoop 1.
> > > >
> > > >
> > > > Thanks and regards,
> > > > John Ho
> > > >
> > > > ===
> > > > Task Log (CDH4.5.0 and Teradata Express 14.10 on SLES11):
> > > >
> > > > org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs
> > > > during extractor run at
> > > > org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:101) at
> > > > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) at
> > > > org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) at
> > > > org.apache.hadoop.mapred.Child$4.run(Child.java:268) at
> > > > java.security.AccessController.doPrivileged(Native Method) at
> > > > javax.security.auth.Subject.doAs(Subject.java:415) at
> > > >
> > >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> > > > at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by:
> > > > org.apache.sqoop.common.SqoopException:
> > > GENERIC_JDBC_CONNECTOR_0002:Unable
> > > > to execute the SQL statement at
> > > >
> > >
> org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:59)
> > > > at
> > > >
> > >
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:50)
> > > > at
> > > >
> > >
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.ja
> > > > org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs
> > > > during extractor run at
> > > > org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:101) at
> > > > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) at
> > > > org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) at
> > > > org.apache.hadoop.mapred.Child$4.run(Child.java:268) at
> > > > java.security.AccessController.doPrivileged(Native Method) at
> > > > javax.security.auth.Subject.doAs(Subject.java:415) at
> > > >
> > >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> > > > at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by:
> > > > org.apache.sqoop.common.SqoopException:
> > > GENERIC_JDBC_CONNECTOR_0002:Unable
> > > > to execute the SQL statement at
> > > >
> > >
> org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:59)
> > > > at
> > > >
> > >
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:50)
> > > > at
> > > >
> > >
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.ja
> > > > org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs
> > > > during extractor run at
> > > > org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:101) at
> > > > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) at
> > > > org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) at
> > > > org.apache.hadoop.mapred.Child$4.run(Child.java:268) at
> > > > java.security.AccessController.doPrivileged(Native Method) at
> > > > javax.security.auth.Subject.doAs(Subject.java:415) at
> > > >
> > >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> > > > at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by:
> > > > org.apache.sqoop.common.SqoopException:
> > > GENERIC_JDBC_CONNECTOR_0002:Unable
> > > > to execute the SQL statement at
> > > >
> > >
> org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:59)
> > > > at
> > > >
> > >
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:50)
> > > > at
> > > >
> > >
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.ja
> > > > org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs
> > > > during extractor run at
> > > > org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:101) at
> > > > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) at
> > > > org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) at
> > > > org.apache.hadoop.mapred.Child$4.run(Child.java:268) at
> > > > java.security.AccessController.doPrivileged(Native Method) at
> > > > javax.security.auth.Subject.doAs(Subject.java:415) at
> > > >
> > >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> > > > at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by:
> > > > org.apache.sqoop.common.SqoopException:
> > > GENERIC_JDBC_CONNECTOR_0002:Unable
> > > > to execute the SQL statement at
> > > >
> > >
> org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:59)
> > > > at
> > > >
> > >
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:50)
> > > > at
> > > >
> > >
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.ja
> > >
>
Re: Does sqoop2 support Teradata?
Posted by Jarek Jarcec Cecho <ja...@apache.org>.
You seems to be using VARCHAR column for creating splits, which is generally very bad idea. I would strongly advise you to use a number column instead.
There is many things that can go wrong with using VARCHAR as a partition column. Looking at the exception my wild guess would be charset - are you explicitly stating that the connection should be in UTF? If not than Teradata default charset is ASCII that clearly won't work.
http://developer.teradata.com/doc/connectivity/jdbc/reference/current/jdbcug_chapter_2.html#URL_CHARSET
Jarcec
On Tue, Mar 18, 2014 at 01:26:39PM +0800, John Ho wrote:
> Many thanks Jarcec for the reply.
>
> I merged 2 jar files as you suggested and sqoop2 able to connect to
> Teradata.
>
> This time it got issue when extracting data from Teradata using small data
> set (20 rows)
> Below are details and logs collected from Hue GUI for the failed task.
> Sqoop1 has no issues so far.
>
>
> Thanks and regards,
> John
>
> =======
>
> Schema:
> CREATE TABLE dailyprices (
> stockname varchar(256) NOT NULL default '',
> tradedate varchar(256) NOT NULL default '',
> openprice float default NULL,
> highprice float default NULL,
> lowprice float default NULL,
> closeprice float default NULL,
> tradevolume bigint default NULL,
> adjcloseprice float default NULL,
> PRIMARY KEY (stockname,tradedate)
> );
>
> ====
> Data for testing:
>
> HPQ|2014-02-07|28.7|29.16|28.69|29.07|7537300|29.07
> HPQ|2014-02-06|28.23|28.66|28.2|28.49|6693700|28.49
> HPQ|2014-02-05|28.17|28.44|27.9|28.01|10278000|28.01
> HPQ|2014-02-04|28.14|28.41|27.89|28.33|9581300|28.33
> HPQ|2014-02-03|29.06|29.29|27.96|28.04|14655400|28.04
> HPQ|2014-01-31|28.94|29.19|28.74|29|12934300|29
> HPQ|2014-01-30|29.15|29.42|29.08|29.25|9095700|29.25
> HPQ|2014-01-29|28.92|29.15|28.75|29.02|13915700|29.02
> HPQ|2014-01-28|28.57|29.08|28.49|29|12407000|29
> HPQ|2014-01-27|28.53|29.09|28.38|28.6|15924800|28.6
> IBM|2014-02-07|175.64|177.56|175.07|177.25|4692900|177.25
> IBM|2014-02-06|173.97|174.85|173.79|174.67|4292200|174.67
> IBM|2014-02-05|172.19|174.97|172.19|174.24|4712300|173.29
> IBM|2014-02-04|173.53|173.75|172.36|172.84|4349800|171.9
> IBM|2014-02-03|176.02|176.02|172.72|172.9|7186800|171.96
> IBM|2014-01-31|176.11|177.84|175.34|176.68|5193400|175.72
> IBM|2014-01-30|177.17|177.86|176.36|177.36|4853700|176.39
> IBM|2014-01-29|175.98|178.53|175.89|176.4|4970900|175.44
> IBM|2014-01-28|178.05|178.45|176.16|176.85|5333300|175.89
> IBM|2014-01-27|179.61|179.65|177.66|177.9|5208600|176.93
>
> ====
> Sqoop2's task diagnostic log (from hue UI):
> org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs
> during extractor run
> at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:101)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
> at org.apache.hadoop.mapred.Child.main(Child.java:262)
> Caused by: org.apache.sqoop.common.SqoopException:
> GENERIC_JDBC_CONNECTOR_0002:Unable to execute the SQL statement
> at
> org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:59)
> at
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:50)
> at
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.ja
>
> ====
> stderr (from hue UI):
> 2014-03-18 13:10:33,969 [main] INFO org.apache.sqoop.job.mr.SqoopMapper -
> Starting progress service
> 2014-03-18 13:10:33,970 [main] INFO org.apache.sqoop.job.mr.SqoopMapper -
> Running extractor class
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor
> 2014-03-18 13:10:33,978 [pool-2-thread-1] DEBUG
> org.apache.sqoop.job.mr.ProgressRunnable - Auto-progress thread reporting
> progress
> 2014-03-18 13:10:35,454 [main] INFO
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor - Using query:
> SELECT stockname,tradedate,closeprice,tradevolume FROM dailyprices WHERE
> 'HPQ' <= stockname AND stockname < 'Hó '
> 2014-03-18 13:10:35,492 [main] INFO org.apache.sqoop.job.mr.SqoopMapper -
> Stopping progress service
>
>
> ====
> syslog (from hue UI):
> 2014-03-18 13:10:31,622 WARN mapreduce.Counters: Group
> org.apache.hadoop.mapred.Task$Counter is deprecated. Use
> org.apache.hadoop.mapreduce.TaskCounter instead
> 2014-03-18 13:10:33,076 WARN org.apache.hadoop.conf.Configuration:
> session.id is deprecated. Instead, use dfs.metrics.session-id
> 2014-03-18 13:10:33,077 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=MAP, sessionId=
> 2014-03-18 13:10:33,458 INFO org.apache.hadoop.util.ProcessTree: setsid
> exited with exit code 0
> 2014-03-18 13:10:33,482 INFO org.apache.hadoop.mapred.Task: Using
> ResourceCalculatorPlugin :
> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1f28706
> 2014-03-18 13:10:33,774 INFO org.apache.hadoop.mapred.MapTask: Processing
> split: org.apache.sqoop.job.mr.SqoopSplit@72c1d428
> 2014-03-18 13:10:33,781 INFO org.apache.hadoop.mapred.MapTask: Map output
> collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
> 2014-03-18 13:10:33,785 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb =
> 136
> 2014-03-18 13:10:33,925 INFO org.apache.hadoop.mapred.MapTask: data buffer
> = 108380824/135476032
> 2014-03-18 13:10:33,925 INFO org.apache.hadoop.mapred.MapTask: record
> buffer = 356515/445644
> 2014-03-18 13:10:33,969 INFO org.apache.sqoop.job.mr.SqoopMapper: Starting
> progress service
> 2014-03-18 13:10:33,970 INFO org.apache.sqoop.job.mr.SqoopMapper: Running
> extractor class org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor
> 2014-03-18 13:10:33,978 DEBUG org.apache.sqoop.job.mr.ProgressRunnable:
> Auto-progress thread reporting progress
> 2014-03-18 13:10:35,454 INFO
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor: Using query:
> SELECT stockname,tradedate,closeprice,tradevolume FROM dailyprices WHERE
> 'HPQ' <= stockname AND stockname < 'Hó '
> 2014-03-18 13:10:35,492 INFO org.apache.sqoop.job.mr.SqoopMapper: Stopping
> progress service
> 2014-03-18 13:10:35,495 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
> 2014-03-18 13:10:35,501 WARN org.apache.hadoop.mapred.Child: Error running
> child
> org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs
> during extractor run
> at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:101)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
> at org.apache.hadoop.mapred.Child.main(Child.java:262)
> Caused by: org.apache.sqoop.common.SqoopException:
> GENERIC_JDBC_CONNECTOR_0002:Unable to execute the SQL statement
> at
> org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:59)
> at
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:50)
> at
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:31)
> at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:96)
> ... 7 more
> Caused by: com.teradata.jdbc.jdbc_4.util.JDBCException: [Teradata Database]
> [TeraJDBC 14.10.00.26] [Error 6705] [SQLState HY000] An illegally formed
> character string was encountered during translation.
> at
> com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDatabaseSQLException(ErrorFactory.java:307)
> at
> com.teradata.jdbc.jdbc_4.statemachine.ReceiveInitSubState.action(ReceiveInitSubState.java:109)
> at
> com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.subStateMachine(StatementReceiveState.java:314)
> at
> com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.action(StatementReceiveState.java:202)
> at
> com.teradata.jdbc.jdbc_4.statemachine.StatementController.runBody(StatementController.java:123)
> at
> com.teradata.jdbc.jdbc_4.statemachine.StatementController.run(StatementController.java:114)
> at
> com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:384)
> at
> com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:326)
> at
> com.teradata.jdbc.jdbc_4.TDStatement.doNonPrepExecuteQuery(TDStatement.java:314)
> at com.teradata.jdbc.jdbc_4.TDStatement.executeQuery(TDStatement.java:1091)
> at
> org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:56)
> ... 10 more
> 2014-03-18 13:10:35,506 INFO org.apache.hadoop.mapred.Task: Runnning
> cleanup for the task
>
>
> On Sat, Mar 15, 2014 at 1:17 AM, Jarek Jarcec Cecho <ja...@apache.org>wrote:
>
> > Hi John,
> > Sqoop2 is not polluting the DistributedCache like Sqoop 1 with putting
> > entire classpath to the MapReduce job. Instead we are propagating only jars
> > that are required. Sadly this mechanism is not working correctly in
> > Teradata case as the JDBC driver consist of two jars that are not
> > referencing each other - the dependency is something that user have to
> > know, not something that jar file exposes. As a result Sqoop will propagate
> > only one of them to the mapreduce job and that will in turn fail. This will
> > be fixed later when we introduce special Teradata connector. For now, you
> > could merge both Teradata JDBC driver jars together and put the resulting
> > jar into /var/log/sqoop2/.
> >
> > Jarcec
> >
> > On Fri, Mar 14, 2014 at 11:56:41AM +0800, John Ho wrote:
> > > Hi Sqoop Team,
> > >
> > > Does sqoop2 support importing data from Teradata?
> > >
> > > I'm using CDH4.5 and not sure where to put Teradata's JDBC driver files.
> > >
> > > I upload tdgssconfig.jar and terajdbc4.jar to /var/lib/sqoop2 and add
> > > "-classpath
> > > /var/lib/sqoop2/tdgssconfig.jar" to Java Configuration Options for
> > > TaskTracker for TaskTracker but it does not work as expected.
> > >
> > > I also upload the jar files to
> > >
> > /opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/sqoop2/webapps/sqoop/WEB-INF/lib/
> > > but it does not work.
> > >
> > > There's no issues if I use sqoop 1.
> > >
> > >
> > > Thanks and regards,
> > > John Ho
> > >
> > > ===
> > > Task Log (CDH4.5.0 and Teradata Express 14.10 on SLES11):
> > >
> > > org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs
> > > during extractor run at
> > > org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:101) at
> > > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) at
> > > org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) at
> > > org.apache.hadoop.mapred.Child$4.run(Child.java:268) at
> > > java.security.AccessController.doPrivileged(Native Method) at
> > > javax.security.auth.Subject.doAs(Subject.java:415) at
> > >
> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> > > at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by:
> > > org.apache.sqoop.common.SqoopException:
> > GENERIC_JDBC_CONNECTOR_0002:Unable
> > > to execute the SQL statement at
> > >
> > org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:59)
> > > at
> > >
> > org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:50)
> > > at
> > >
> > org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.ja
> > > org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs
> > > during extractor run at
> > > org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:101) at
> > > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) at
> > > org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) at
> > > org.apache.hadoop.mapred.Child$4.run(Child.java:268) at
> > > java.security.AccessController.doPrivileged(Native Method) at
> > > javax.security.auth.Subject.doAs(Subject.java:415) at
> > >
> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> > > at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by:
> > > org.apache.sqoop.common.SqoopException:
> > GENERIC_JDBC_CONNECTOR_0002:Unable
> > > to execute the SQL statement at
> > >
> > org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:59)
> > > at
> > >
> > org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:50)
> > > at
> > >
> > org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.ja
> > > org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs
> > > during extractor run at
> > > org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:101) at
> > > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) at
> > > org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) at
> > > org.apache.hadoop.mapred.Child$4.run(Child.java:268) at
> > > java.security.AccessController.doPrivileged(Native Method) at
> > > javax.security.auth.Subject.doAs(Subject.java:415) at
> > >
> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> > > at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by:
> > > org.apache.sqoop.common.SqoopException:
> > GENERIC_JDBC_CONNECTOR_0002:Unable
> > > to execute the SQL statement at
> > >
> > org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:59)
> > > at
> > >
> > org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:50)
> > > at
> > >
> > org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.ja
> > > org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs
> > > during extractor run at
> > > org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:101) at
> > > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) at
> > > org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) at
> > > org.apache.hadoop.mapred.Child$4.run(Child.java:268) at
> > > java.security.AccessController.doPrivileged(Native Method) at
> > > javax.security.auth.Subject.doAs(Subject.java:415) at
> > >
> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> > > at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by:
> > > org.apache.sqoop.common.SqoopException:
> > GENERIC_JDBC_CONNECTOR_0002:Unable
> > > to execute the SQL statement at
> > >
> > org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:59)
> > > at
> > >
> > org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:50)
> > > at
> > >
> > org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.ja
> >
Re: Does sqoop2 support Teradata?
Posted by John Ho <jo...@gmail.com>.
Many thanks Jarcec for the reply.
I merged 2 jar files as you suggested and sqoop2 able to connect to
Teradata.
This time it got issue when extracting data from Teradata using small data
set (20 rows)
Below are details and logs collected from Hue GUI for the failed task.
Sqoop1 has no issues so far.
Thanks and regards,
John
=======
Schema:
CREATE TABLE dailyprices (
stockname varchar(256) NOT NULL default '',
tradedate varchar(256) NOT NULL default '',
openprice float default NULL,
highprice float default NULL,
lowprice float default NULL,
closeprice float default NULL,
tradevolume bigint default NULL,
adjcloseprice float default NULL,
PRIMARY KEY (stockname,tradedate)
);
====
Data for testing:
HPQ|2014-02-07|28.7|29.16|28.69|29.07|7537300|29.07
HPQ|2014-02-06|28.23|28.66|28.2|28.49|6693700|28.49
HPQ|2014-02-05|28.17|28.44|27.9|28.01|10278000|28.01
HPQ|2014-02-04|28.14|28.41|27.89|28.33|9581300|28.33
HPQ|2014-02-03|29.06|29.29|27.96|28.04|14655400|28.04
HPQ|2014-01-31|28.94|29.19|28.74|29|12934300|29
HPQ|2014-01-30|29.15|29.42|29.08|29.25|9095700|29.25
HPQ|2014-01-29|28.92|29.15|28.75|29.02|13915700|29.02
HPQ|2014-01-28|28.57|29.08|28.49|29|12407000|29
HPQ|2014-01-27|28.53|29.09|28.38|28.6|15924800|28.6
IBM|2014-02-07|175.64|177.56|175.07|177.25|4692900|177.25
IBM|2014-02-06|173.97|174.85|173.79|174.67|4292200|174.67
IBM|2014-02-05|172.19|174.97|172.19|174.24|4712300|173.29
IBM|2014-02-04|173.53|173.75|172.36|172.84|4349800|171.9
IBM|2014-02-03|176.02|176.02|172.72|172.9|7186800|171.96
IBM|2014-01-31|176.11|177.84|175.34|176.68|5193400|175.72
IBM|2014-01-30|177.17|177.86|176.36|177.36|4853700|176.39
IBM|2014-01-29|175.98|178.53|175.89|176.4|4970900|175.44
IBM|2014-01-28|178.05|178.45|176.16|176.85|5333300|175.89
IBM|2014-01-27|179.61|179.65|177.66|177.9|5208600|176.93
====
Sqoop2's task diagnostic log (from hue UI):
org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs
during extractor run
at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:101)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: org.apache.sqoop.common.SqoopException:
GENERIC_JDBC_CONNECTOR_0002:Unable to execute the SQL statement
at
org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:59)
at
org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:50)
at
org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.ja
====
stderr (from hue UI):
2014-03-18 13:10:33,969 [main] INFO org.apache.sqoop.job.mr.SqoopMapper -
Starting progress service
2014-03-18 13:10:33,970 [main] INFO org.apache.sqoop.job.mr.SqoopMapper -
Running extractor class
org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor
2014-03-18 13:10:33,978 [pool-2-thread-1] DEBUG
org.apache.sqoop.job.mr.ProgressRunnable - Auto-progress thread reporting
progress
2014-03-18 13:10:35,454 [main] INFO
org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor - Using query:
SELECT stockname,tradedate,closeprice,tradevolume FROM dailyprices WHERE
'HPQ' <= stockname AND stockname < 'Hó '
2014-03-18 13:10:35,492 [main] INFO org.apache.sqoop.job.mr.SqoopMapper -
Stopping progress service
====
syslog (from hue UI):
2014-03-18 13:10:31,622 WARN mapreduce.Counters: Group
org.apache.hadoop.mapred.Task$Counter is deprecated. Use
org.apache.hadoop.mapreduce.TaskCounter instead
2014-03-18 13:10:33,076 WARN org.apache.hadoop.conf.Configuration:
session.id is deprecated. Instead, use dfs.metrics.session-id
2014-03-18 13:10:33,077 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=MAP, sessionId=
2014-03-18 13:10:33,458 INFO org.apache.hadoop.util.ProcessTree: setsid
exited with exit code 0
2014-03-18 13:10:33,482 INFO org.apache.hadoop.mapred.Task: Using
ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1f28706
2014-03-18 13:10:33,774 INFO org.apache.hadoop.mapred.MapTask: Processing
split: org.apache.sqoop.job.mr.SqoopSplit@72c1d428
2014-03-18 13:10:33,781 INFO org.apache.hadoop.mapred.MapTask: Map output
collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2014-03-18 13:10:33,785 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb =
136
2014-03-18 13:10:33,925 INFO org.apache.hadoop.mapred.MapTask: data buffer
= 108380824/135476032
2014-03-18 13:10:33,925 INFO org.apache.hadoop.mapred.MapTask: record
buffer = 356515/445644
2014-03-18 13:10:33,969 INFO org.apache.sqoop.job.mr.SqoopMapper: Starting
progress service
2014-03-18 13:10:33,970 INFO org.apache.sqoop.job.mr.SqoopMapper: Running
extractor class org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor
2014-03-18 13:10:33,978 DEBUG org.apache.sqoop.job.mr.ProgressRunnable:
Auto-progress thread reporting progress
2014-03-18 13:10:35,454 INFO
org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor: Using query:
SELECT stockname,tradedate,closeprice,tradevolume FROM dailyprices WHERE
'HPQ' <= stockname AND stockname < 'Hó '
2014-03-18 13:10:35,492 INFO org.apache.sqoop.job.mr.SqoopMapper: Stopping
progress service
2014-03-18 13:10:35,495 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2014-03-18 13:10:35,501 WARN org.apache.hadoop.mapred.Child: Error running
child
org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs
during extractor run
at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:101)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: org.apache.sqoop.common.SqoopException:
GENERIC_JDBC_CONNECTOR_0002:Unable to execute the SQL statement
at
org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:59)
at
org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:50)
at
org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:31)
at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:96)
... 7 more
Caused by: com.teradata.jdbc.jdbc_4.util.JDBCException: [Teradata Database]
[TeraJDBC 14.10.00.26] [Error 6705] [SQLState HY000] An illegally formed
character string was encountered during translation.
at
com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDatabaseSQLException(ErrorFactory.java:307)
at
com.teradata.jdbc.jdbc_4.statemachine.ReceiveInitSubState.action(ReceiveInitSubState.java:109)
at
com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.subStateMachine(StatementReceiveState.java:314)
at
com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.action(StatementReceiveState.java:202)
at
com.teradata.jdbc.jdbc_4.statemachine.StatementController.runBody(StatementController.java:123)
at
com.teradata.jdbc.jdbc_4.statemachine.StatementController.run(StatementController.java:114)
at
com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:384)
at
com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:326)
at
com.teradata.jdbc.jdbc_4.TDStatement.doNonPrepExecuteQuery(TDStatement.java:314)
at com.teradata.jdbc.jdbc_4.TDStatement.executeQuery(TDStatement.java:1091)
at
org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:56)
... 10 more
2014-03-18 13:10:35,506 INFO org.apache.hadoop.mapred.Task: Runnning
cleanup for the task
On Sat, Mar 15, 2014 at 1:17 AM, Jarek Jarcec Cecho <ja...@apache.org>wrote:
> Hi John,
> Sqoop2 is not polluting the DistributedCache like Sqoop 1 with putting
> entire classpath to the MapReduce job. Instead we are propagating only jars
> that are required. Sadly this mechanism is not working correctly in
> Teradata case as the JDBC driver consist of two jars that are not
> referencing each other - the dependency is something that user have to
> know, not something that jar file exposes. As a result Sqoop will propagate
> only one of them to the mapreduce job and that will in turn fail. This will
> be fixed later when we introduce special Teradata connector. For now, you
> could merge both Teradata JDBC driver jars together and put the resulting
> jar into /var/log/sqoop2/.
>
> Jarcec
>
> On Fri, Mar 14, 2014 at 11:56:41AM +0800, John Ho wrote:
> > Hi Sqoop Team,
> >
> > Does sqoop2 support importing data from Teradata?
> >
> > I'm using CDH4.5 and not sure where to put Teradata's JDBC driver files.
> >
> > I upload tdgssconfig.jar and terajdbc4.jar to /var/lib/sqoop2 and add
> > "-classpath
> > /var/lib/sqoop2/tdgssconfig.jar" to Java Configuration Options for
> > TaskTracker for TaskTracker but it does not work as expected.
> >
> > I also upload the jar files to
> >
> /opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/sqoop2/webapps/sqoop/WEB-INF/lib/
> > but it does not work.
> >
> > There's no issues if I use sqoop 1.
> >
> >
> > Thanks and regards,
> > John Ho
> >
> > ===
> > Task Log (CDH4.5.0 and Teradata Express 14.10 on SLES11):
> >
> > org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs
> > during extractor run at
> > org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:101) at
> > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) at
> > org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) at
> > org.apache.hadoop.mapred.Child$4.run(Child.java:268) at
> > java.security.AccessController.doPrivileged(Native Method) at
> > javax.security.auth.Subject.doAs(Subject.java:415) at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> > at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by:
> > org.apache.sqoop.common.SqoopException:
> GENERIC_JDBC_CONNECTOR_0002:Unable
> > to execute the SQL statement at
> >
> org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:59)
> > at
> >
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:50)
> > at
> >
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.ja
> > org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs
> > during extractor run at
> > org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:101) at
> > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) at
> > org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) at
> > org.apache.hadoop.mapred.Child$4.run(Child.java:268) at
> > java.security.AccessController.doPrivileged(Native Method) at
> > javax.security.auth.Subject.doAs(Subject.java:415) at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> > at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by:
> > org.apache.sqoop.common.SqoopException:
> GENERIC_JDBC_CONNECTOR_0002:Unable
> > to execute the SQL statement at
> >
> org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:59)
> > at
> >
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:50)
> > at
> >
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.ja
> > org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs
> > during extractor run at
> > org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:101) at
> > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) at
> > org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) at
> > org.apache.hadoop.mapred.Child$4.run(Child.java:268) at
> > java.security.AccessController.doPrivileged(Native Method) at
> > javax.security.auth.Subject.doAs(Subject.java:415) at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> > at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by:
> > org.apache.sqoop.common.SqoopException:
> GENERIC_JDBC_CONNECTOR_0002:Unable
> > to execute the SQL statement at
> >
> org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:59)
> > at
> >
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:50)
> > at
> >
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.ja
> > org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs
> > during extractor run at
> > org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:101) at
> > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) at
> > org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) at
> > org.apache.hadoop.mapred.Child$4.run(Child.java:268) at
> > java.security.AccessController.doPrivileged(Native Method) at
> > javax.security.auth.Subject.doAs(Subject.java:415) at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> > at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by:
> > org.apache.sqoop.common.SqoopException:
> GENERIC_JDBC_CONNECTOR_0002:Unable
> > to execute the SQL statement at
> >
> org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:59)
> > at
> >
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:50)
> > at
> >
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.ja
>
Re: Does sqoop2 support Teradata?
Posted by Jarek Jarcec Cecho <ja...@apache.org>.
Hi John,
Sqoop2 is not polluting the DistributedCache like Sqoop 1 with putting entire classpath to the MapReduce job. Instead we are propagating only jars that are required. Sadly this mechanism is not working correctly in Teradata case as the JDBC driver consist of two jars that are not referencing each other - the dependency is something that user have to know, not something that jar file exposes. As a result Sqoop will propagate only one of them to the mapreduce job and that will in turn fail. This will be fixed later when we introduce special Teradata connector. For now, you could merge both Teradata JDBC driver jars together and put the resulting jar into /var/log/sqoop2/.
Jarcec
On Fri, Mar 14, 2014 at 11:56:41AM +0800, John Ho wrote:
> Hi Sqoop Team,
>
> Does sqoop2 support importing data from Teradata?
>
> I'm using CDH4.5 and not sure where to put Teradata's JDBC driver files.
>
> I upload tdgssconfig.jar and terajdbc4.jar to /var/lib/sqoop2 and add
> "-classpath
> /var/lib/sqoop2/tdgssconfig.jar" to Java Configuration Options for
> TaskTracker for TaskTracker but it does not work as expected.
>
> I also upload the jar files to
> /opt/cloudera/parcels/CDH-4.5.0-1.cdh4.5.0.p0.30/lib/sqoop2/webapps/sqoop/WEB-INF/lib/
> but it does not work.
>
> There's no issues if I use sqoop 1.
>
>
> Thanks and regards,
> John Ho
>
> ===
> Task Log (CDH4.5.0 and Teradata Express 14.10 on SLES11):
>
> org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs
> during extractor run at
> org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:101) at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) at
> org.apache.hadoop.mapred.Child$4.run(Child.java:268) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:415) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by:
> org.apache.sqoop.common.SqoopException: GENERIC_JDBC_CONNECTOR_0002:Unable
> to execute the SQL statement at
> org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:59)
> at
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:50)
> at
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.ja
> org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs
> during extractor run at
> org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:101) at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) at
> org.apache.hadoop.mapred.Child$4.run(Child.java:268) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:415) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by:
> org.apache.sqoop.common.SqoopException: GENERIC_JDBC_CONNECTOR_0002:Unable
> to execute the SQL statement at
> org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:59)
> at
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:50)
> at
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.ja
> org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs
> during extractor run at
> org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:101) at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) at
> org.apache.hadoop.mapred.Child$4.run(Child.java:268) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:415) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by:
> org.apache.sqoop.common.SqoopException: GENERIC_JDBC_CONNECTOR_0002:Unable
> to execute the SQL statement at
> org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:59)
> at
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:50)
> at
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.ja
> org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs
> during extractor run at
> org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:101) at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) at
> org.apache.hadoop.mapred.Child$4.run(Child.java:268) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:415) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by:
> org.apache.sqoop.common.SqoopException: GENERIC_JDBC_CONNECTOR_0002:Unable
> to execute the SQL statement at
> org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeQuery(GenericJdbcExecutor.java:59)
> at
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.java:50)
> at
> org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor.extract(GenericJdbcImportExtractor.ja