You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by "Ruslan Dautkhanov (JIRA)" <ji...@apache.org> on 2016/10/15 16:25:20 UTC
[jira] [Created] (SQOOP-3025) Oracle Direct export from HCatalog
parquet table gives NPE in kerberized cluster
Ruslan Dautkhanov created SQOOP-3025:
----------------------------------------
Summary: Oracle Direct export from HCatalog parquet table gives NPE in kerberized cluster
Key: SQOOP-3025
URL: https://issues.apache.org/jira/browse/SQOOP-3025
Project: Sqoop
Issue Type: Bug
Components: build, codegen, connectors/oracle, hive-integration, metastore
Affects Versions: 1.4.6
Environment: Hadoop 2.6; Sqoop 1.4.6; Oracle 12c
Reporter: Ruslan Dautkhanov
Attachments: sqoop_OracleDirect_mapper_nullPointer_exception.txt
Getting following exception in one of the mappers stack when try to export to Oracle from Hive Parquet
{noformat}
2016-10-14 23:14:01,027 ERROR [main] org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: The following error occurred during configurePreparedStatement()
java.lang.NullPointerException
at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.setBindValueAtName(OraOopOutputFormatBase.java:432)
at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.configurePreparedStatementColumns(OraOopOutputFormatBase.java:547)
at org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.configurePreparedStatement(OraOopOutputFormatInsert.java:246)
at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.getPreparedStatement(OraOopOutputFormatBase.java:303)
at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:153)
at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:194)
at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.close(OraOopOutputFormatBase.java:578)
at org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.close(OraOopOutputFormatInsert.java:107)
at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670)
at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
2016-10-14 23:14:01,074 INFO [main] org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: Time spent performing an "exchange subpartition with table": 0.046365 sec.
2016-10-14 23:14:01,074 DEBUG [main] org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: Dropping temporary mapper table "ORAOOP_20161014_231320_3"
2016-10-14 23:14:01,131 INFO [main] org.apache.hadoop.mapred.MapTask: Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector@54848551
java.io.IOException: java.sql.SQLException: java.lang.NullPointerException
at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:197)
at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.close(OraOopOutputFormatBase.java:578)
at org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.close(OraOopOutputFormatInsert.java:107)
at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670)
at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.sql.SQLException: java.lang.NullPointerException
at org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.configurePreparedStatement(OraOopOutputFormatInsert.java:255)
at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.getPreparedStatement(OraOopOutputFormatBase.java:303)
at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:153)
at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:194)
{noformat}
Will attach full log file as an attachment.
Sqoop export options used:
{noformat}
sqoop_params = ['sqoop', 'export'
# marreduce options:
, '-Dmapreduce.map.memory.mb=5000'
, '-Dmapreduce.map.java.opts=-Xmx4000m'
, '-Dmapreduce.job.name=sqooping %s@hadoop -> %s@oracle' % (inp_table, out_table)
, '-Dmapreduce.map.maxattempts=1'
# Oracle (OraOOP) specific Sqoop options:
, '-Dsqoop.connection.factories=org.apache.sqoop.manager.oracle.OraOopManagerFactory'
, '-Doraoop.oracle.append.values.hint.usage=ON'
, '-Doraoop.partitioned=true'
, '-Doraoop.nologging=true'
, '-Doraoop.drop.table=true'
, '-Doraoop.template.table=%s_template' % out_table
, '-Doraoop.temporary.table.storage.clause=' + storage_clause
, '-Doraoop.table.storage.clause=' + storage_clause
, '-Doraoop.oracle.rac.service.name=adash_prod_batch'
# generic sqoop options:
, '-Dsqoop.export.records.per.statement=5000'
, '-Dsqoop.export.statements.per.transaction=2000'
, '--direct' #activate Oracle direct mode = 'Data Connector for Oracle and Hadoop'
, '--connect', "jdbc:oracle:thin:@" + jdbc_tns
, '--username', out_schema, '--password', jdbc_password.decode('base64')
, '--table', out_table
, '--hcatalog-database', inp_schema
, '--hcatalog-table', inp_table
, '--batch'
, '--num-mappers', str(num_mappers)
]
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)