You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Jesus Camacho Rodriguez (JIRA)" <ji...@apache.org> on 2016/05/31 19:46:12 UTC
[jira] [Commented] (HIVE-13856) Fetching transaction batches during
ACID streaming against Hive Metastore using Oracle DB fails
[ https://issues.apache.org/jira/browse/HIVE-13856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15308455#comment-15308455 ]
Jesus Camacho Rodriguez commented on HIVE-13856:
------------------------------------------------
[~ekoifman], what is the status on this one? I plan to create the first 2.1.0 RC tomorrow and this is marked as Blocker. Should it go in and can it be deferred as it seems to only affect Metastore using OracleDB? Thanks
> Fetching transaction batches during ACID streaming against Hive Metastore using Oracle DB fails
> -----------------------------------------------------------------------------------------------
>
> Key: HIVE-13856
> URL: https://issues.apache.org/jira/browse/HIVE-13856
> Project: Hive
> Issue Type: Bug
> Components: Transactions
> Affects Versions: 1.3.0, 2.1.0, 2.2.0
> Reporter: Deepesh Khandelwal
> Assignee: Eugene Koifman
> Priority: Blocker
> Attachments: HIVE-13856.patch
>
>
> {noformat}
> 2016-05-25 00:43:49,682 INFO [pool-4-thread-5]: txn.TxnHandler (TxnHandler.java:checkRetryable(1585)) - Non-retryable error: ORA-00933: SQL command not properly ended
> (SQLState=42000, ErrorCode=933)
> 2016-05-25 00:43:49,685 ERROR [pool-4-thread-5]: metastore.RetryingHMSHandler (RetryingHMSHandler.java:invoke(159)) - MetaException(message:Unable to select from transaction database java.sql.SQLSyntaxErrorException: ORA-00933: SQL command not properly ended
> at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:440)
> at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
> at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:837)
> at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:445)
> at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)
> at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:523)
> at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:193)
> at oracle.jdbc.driver.T4CStatement.executeForRows(T4CStatement.java:999)
> at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1315)
> at oracle.jdbc.driver.OracleStatement.executeInternal(OracleStatement.java:1890)
> at oracle.jdbc.driver.OracleStatement.execute(OracleStatement.java:1855)
> at oracle.jdbc.driver.OracleStatementWrapper.execute(OracleStatementWrapper.java:304)
> at com.jolbox.bonecp.StatementHandle.execute(StatementHandle.java:254)
> at org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(TxnHandler.java:429)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.open_txns(HiveMetaStore.java:5647)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> at com.sun.proxy.$Proxy15.open_txns(Unknown Source)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$open_txns.getResult(ThriftHiveMetastore.java:11604)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$open_txns.getResult(ThriftHiveMetastore.java:11589)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> )
> at org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(TxnHandler.java:438)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.open_txns(HiveMetaStore.java:5647)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> at com.sun.proxy.$Proxy15.open_txns(Unknown Source)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$open_txns.getResult(ThriftHiveMetastore.java:11604)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$open_txns.getResult(ThriftHiveMetastore.java:11589)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2016-05-25 00:43:49,685 ERROR [pool-4-thread-5]: thrift.ProcessFunction (ProcessFunction.java:process(41)) - Internal error processing open_txns
> MetaException(message:Unable to select from transaction database java.sql.SQLSyntaxErrorException: ORA-00933: SQL command not properly ended
> at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:440)
> at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
> at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:837)
> at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:445)
> at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)
> at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:523)
> at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:193)
> at oracle.jdbc.driver.T4CStatement.executeForRows(T4CStatement.java:999)
> at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1315)
> at oracle.jdbc.driver.OracleStatement.executeInternal(OracleStatement.java:1890)
> at oracle.jdbc.driver.OracleStatement.execute(OracleStatement.java:1855)
> at oracle.jdbc.driver.OracleStatementWrapper.execute(OracleStatementWrapper.java:304)
> at com.jolbox.bonecp.StatementHandle.execute(StatementHandle.java:254)
> at org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(TxnHandler.java:429)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.open_txns(HiveMetaStore.java:5647)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> at com.sun.proxy.$Proxy15.open_txns(Unknown Source)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$open_txns.getResult(ThriftHiveMetastore.java:11604)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$open_txns.getResult(ThriftHiveMetastore.java:11589)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> )
> at org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(TxnHandler.java:438)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.open_txns(HiveMetaStore.java:5647)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> at com.sun.proxy.$Proxy15.open_txns(Unknown Source)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$open_txns.getResult(ThriftHiveMetastore.java:11604)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$open_txns.getResult(ThriftHiveMetastore.java:11589)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> I think the reason here is that
> {code:title=metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java|borderStyle=solid}
> public OpenTxnsResponse openTxns(OpenTxnRequest rqst) throws MetaException {
> ...
> String query;
> String insertClause = "insert into TXNS (txn_id, txn_state, txn_started, txn_last_heartbeat, txn_user, txn_host) values ";
> StringBuilder valuesClause = new StringBuilder();
> for (long i = first; i < first + numTxns; i++) {
> txnIds.add(i);
> if (i > first &&
> (i - first) % conf.getIntVar(HiveConf.ConfVars.METASTORE_DIRECT_SQL_MAX_ELEMENTS_VALUES_CLAUSE) == 0) {
> // wrap up the current query, and start a new one
> query = insertClause + valuesClause.toString();
> queries.add(query);
> valuesClause.setLength(0);
> valuesClause.append("(").append(i).append(", 'o', ").append(now).append(", ").append(now)
> .append(", '").append(rqst.getUser()).append("', '").append(rqst.getHostname())
> .append("')");
> continue;
> }
> if (i > first) {
> valuesClause.append(", ");
> }
> valuesClause.append("(").append(i).append(", 'o', ").append(now).append(", ").append(now)
> .append(", '").append(rqst.getUser()).append("', '").append(rqst.getHostname())
> .append("')");
> }
> query = insertClause + valuesClause.toString();
> ...
> }
> {code}
> ends up building a query of the form
> {code:sql}
> INSERT INTO TXNS (...) VALUES (...), (...)
> {code}
> Oracle doesn't like this way of inserting multiple rows of data. Couple of ways the following [post|http://www.oratable.com/oracle-insert-all/] describe is either inserting each row individually or use the {{INSERT ALL}} semantics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)