You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Aaron Baff (JIRA)" <ji...@apache.org> on 2012/10/12 20:27:03 UTC
[jira] [Created] (FLUME-1636) JDBC Channel receiving deadlock
transaction rollback
Aaron Baff created FLUME-1636:
---------------------------------
Summary: JDBC Channel receiving deadlock transaction rollback
Key: FLUME-1636
URL: https://issues.apache.org/jira/browse/FLUME-1636
Project: Flume
Issue Type: Bug
Components: Channel
Affects Versions: v1.1.0
Environment: Running on Linux with Java 1.6.0u21 (Oracle Hotspot JVM), CDH 4.0.1 version of Flume v1.1.0.
Reporter: Aaron Baff
Seeing the following error which causes Avro clients to disconnect (I'm guessing due to transaction failure), failover to another agent, then eventually reconnect to the original agent.
#JDBC Channel Configuration
collector_agent.channels.defaultChannel.type = jdbc
collector_agent.channels.defaultChannel.db.type = DERBY
collector_agent.channels.defaultChannel.sysprop.user.home = /f/flume-channels/default/
collector_agent.channels.defaultChannel.capacity = 0
collector_agent.channels.defaultChannel.maximum.capacity = 0
#Log File error
2012-10-01 10:34:09,267 WARN impl.JdbcTransactionImpl: Marking transaction for rollback
2012-10-01 10:34:09,267 WARN impl.JdbcTransactionImpl: Marking transaction for rollback
2012-10-01 10:34:09,267 INFO impl.JdbcTransactionImpl: Attempting transaction roll-back
2012-10-01 10:34:09,268 ERROR source.AvroSource: Unable to process event batch
org.apache.flume.channel.jdbc.JdbcChannelException: Failed to persist event
at org.apache.flume.channel.jdbc.impl.JdbcChannelProviderImpl.persistEvent(JdbcChannelProviderImpl.java:315)
at org.apache.flume.channel.jdbc.JdbcChannel.put(JdbcChannel.java:44)
at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:187)
at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:229)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
at org.apache.avro.ipc.Responder.respond(Responder.java:149)
at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:140)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:125)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302)
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:214)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.flume.channel.jdbc.JdbcChannelException: Unable to persist event: org.apache.flume.channel.jdbc.impl.PersistableEvent@1b6ca9f0
at org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.storeEvent(DerbySchemaHandler.java:733)
at org.apache.flume.channel.jdbc.impl.JdbcChannelProviderImpl.persistEvent(JdbcChannelProviderImpl.java:308)
... 30 more
Caused by: java.sql.SQLTransactionRollbackException: A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
Lock : ROW, SYSCOLUMNS, (4,43)
Waiting XID : {242883779, X} , SA, INSERT INTO FLUME.FL_HEADER (FLH_EVENT, FLH_NAME, FLH_VALUE, FLH_NMSPILL, FLH_VLSPILL) VALUES ( ?, ?, ?, ?, ?)
Granted XID : {242883747, X}
Lock : ROW, SYSCOLUMNS, (4,37)
Waiting XID : {242883747, X} , SA, INSERT INTO FLUME.FL_EVENT (FLE_PAYLOAD, FLE_CHANNEL, FLE_SPILL) VALUES ( ?, ?, ?)
Granted XID : {242883779, X}
. The selected victim is XID : 242883779.
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Unknown Source)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
at org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.storeEvent(DerbySchemaHandler.java:646)
... 31 more
Caused by: java.sql.SQLException: A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
Lock : ROW, SYSCOLUMNS, (4,43)
Waiting XID : {242883779, X} , SA, INSERT INTO FLUME.FL_HEADER (FLH_EVENT, FLH_NAME, FLH_VALUE, FLH_NMSPILL, FLH_VLSPILL) VALUES ( ?, ?, ?, ?, ?)
Granted XID : {242883747, X}
Lock : ROW, SYSCOLUMNS, (4,37)
Waiting XID : {242883747, X} , SA, INSERT INTO FLUME.FL_EVENT (FLE_PAYLOAD, FLE_CHANNEL, FLE_SPILL) VALUES ( ?, ?, ?)
Granted XID : {242883779, X}
. The selected victim is XID : 242883779.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
... 44 more
Caused by: ERROR 40001: A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
Lock : ROW, SYSCOLUMNS, (4,43)
Waiting XID : {242883779, X} , SA, INSERT INTO FLUME.FL_HEADER (FLH_EVENT, FLH_NAME, FLH_VALUE, FLH_NMSPILL, FLH_VLSPILL) VALUES ( ?, ?, ?, ?, ?)
Granted XID : {242883747, X}
Lock : ROW, SYSCOLUMNS, (4,37)
Waiting XID : {242883747, X} , SA, INSERT INTO FLUME.FL_EVENT (FLE_PAYLOAD, FLE_CHANNEL, FLE_SPILL) VALUES ( ?, ?, ?)
Granted XID : {242883779, X}
. The selected victim is XID : 242883779.
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.impl.services.locks.Deadlock.buildException(Unknown Source)
at org.apache.derby.impl.services.locks.ConcurrentLockSet.lockObject(Unknown Source)
at org.apache.derby.impl.services.locks.AbstractPool.lockObject(Unknown Source)
at org.apache.derby.impl.services.locks.ConcurrentPool.lockObject(Unknown Source)
at org.apache.derby.impl.store.raw.xact.RowLocking3.lockRecordForWrite(Unknown Source)
at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.lockPositionForWrite(Unknown Source)
at org.apache.derby.impl.store.access.conglomerate.GenericConglomerateController.fetch(Unknown Source)
at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSetAutoincrementValue(Unknown Source)
at org.apache.derby.impl.sql.execute.InsertResultSet.getSetAutoincrementValue(Unknown Source)
at org.apache.derby.impl.sql.execute.BaseActivation.getSetAutoincrementValue(Unknown Source)
at org.apache.derby.exe.ace8b858bfx013ax1d43x6465x000004f30e0821d.e0(Unknown Source)
at org.apache.derby.impl.services.reflect.DirectCall.invoke(Unknown Source)
at org.apache.derby.impl.sql.execute.RowResultSet.getNextRowCore(Unknown Source)
at org.apache.derby.impl.sql.execute.NormalizeResultSet.getNextRowCore(Unknown Source)
at org.apache.derby.impl.sql.execute.DMLWriteResultSet.getNextRowCore(Unknown Source)
at org.apache.derby.impl.sql.execute.InsertResultSet.open(Unknown Source)
at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown Source)
at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown Source)
... 38 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1636) JDBC Channel receiving deadlock
transaction rollback
Posted by "Aaron Baff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Baff updated FLUME-1636:
------------------------------
Description:
Seeing the following error which causes Avro clients to disconnect (I'm guessing due to transaction failure), failover to another agent, then eventually reconnect to the original agent. There were 3 nodes which were writing (over Avro) to this single node.
In the log file there are approximately 183,390 log records over a 2 to 2 1/2 weeks of run time. Additionally, after doing a check of the data being written to HDFS, it appears only approximately 10% of the data is there.
#JDBC Channel Configuration
collector_agent.channels.defaultChannel.type = jdbc
collector_agent.channels.defaultChannel.db.type = DERBY
collector_agent.channels.defaultChannel.sysprop.user.home = /f/flume-channels/default/
collector_agent.channels.defaultChannel.capacity = 0
collector_agent.channels.defaultChannel.maximum.capacity = 0
#Log File error
2012-10-01 10:34:09,267 WARN impl.JdbcTransactionImpl: Marking transaction for rollback
2012-10-01 10:34:09,267 WARN impl.JdbcTransactionImpl: Marking transaction for rollback
2012-10-01 10:34:09,267 INFO impl.JdbcTransactionImpl: Attempting transaction roll-back
2012-10-01 10:34:09,268 ERROR source.AvroSource: Unable to process event batch
org.apache.flume.channel.jdbc.JdbcChannelException: Failed to persist event
at org.apache.flume.channel.jdbc.impl.JdbcChannelProviderImpl.persistEvent(JdbcChannelProviderImpl.java:315)
at org.apache.flume.channel.jdbc.JdbcChannel.put(JdbcChannel.java:44)
at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:187)
at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:229)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
at org.apache.avro.ipc.Responder.respond(Responder.java:149)
at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:140)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:125)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302)
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:214)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.flume.channel.jdbc.JdbcChannelException: Unable to persist event: org.apache.flume.channel.jdbc.impl.PersistableEvent@1b6ca9f0
at org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.storeEvent(DerbySchemaHandler.java:733)
at org.apache.flume.channel.jdbc.impl.JdbcChannelProviderImpl.persistEvent(JdbcChannelProviderImpl.java:308)
... 30 more
Caused by: java.sql.SQLTransactionRollbackException: A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
Lock : ROW, SYSCOLUMNS, (4,43)
Waiting XID : {242883779, X} , SA, INSERT INTO FLUME.FL_HEADER (FLH_EVENT, FLH_NAME, FLH_VALUE, FLH_NMSPILL, FLH_VLSPILL) VALUES ( ?, ?, ?, ?, ?)
Granted XID : {242883747, X}
Lock : ROW, SYSCOLUMNS, (4,37)
Waiting XID : {242883747, X} , SA, INSERT INTO FLUME.FL_EVENT (FLE_PAYLOAD, FLE_CHANNEL, FLE_SPILL) VALUES ( ?, ?, ?)
Granted XID : {242883779, X}
. The selected victim is XID : 242883779.
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Unknown Source)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
at org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.storeEvent(DerbySchemaHandler.java:646)
... 31 more
Caused by: java.sql.SQLException: A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
Lock : ROW, SYSCOLUMNS, (4,43)
Waiting XID : {242883779, X} , SA, INSERT INTO FLUME.FL_HEADER (FLH_EVENT, FLH_NAME, FLH_VALUE, FLH_NMSPILL, FLH_VLSPILL) VALUES ( ?, ?, ?, ?, ?)
Granted XID : {242883747, X}
Lock : ROW, SYSCOLUMNS, (4,37)
Waiting XID : {242883747, X} , SA, INSERT INTO FLUME.FL_EVENT (FLE_PAYLOAD, FLE_CHANNEL, FLE_SPILL) VALUES ( ?, ?, ?)
Granted XID : {242883779, X}
. The selected victim is XID : 242883779.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
... 44 more
Caused by: ERROR 40001: A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
Lock : ROW, SYSCOLUMNS, (4,43)
Waiting XID : {242883779, X} , SA, INSERT INTO FLUME.FL_HEADER (FLH_EVENT, FLH_NAME, FLH_VALUE, FLH_NMSPILL, FLH_VLSPILL) VALUES ( ?, ?, ?, ?, ?)
Granted XID : {242883747, X}
Lock : ROW, SYSCOLUMNS, (4,37)
Waiting XID : {242883747, X} , SA, INSERT INTO FLUME.FL_EVENT (FLE_PAYLOAD, FLE_CHANNEL, FLE_SPILL) VALUES ( ?, ?, ?)
Granted XID : {242883779, X}
. The selected victim is XID : 242883779.
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.impl.services.locks.Deadlock.buildException(Unknown Source)
at org.apache.derby.impl.services.locks.ConcurrentLockSet.lockObject(Unknown Source)
at org.apache.derby.impl.services.locks.AbstractPool.lockObject(Unknown Source)
at org.apache.derby.impl.services.locks.ConcurrentPool.lockObject(Unknown Source)
at org.apache.derby.impl.store.raw.xact.RowLocking3.lockRecordForWrite(Unknown Source)
at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.lockPositionForWrite(Unknown Source)
at org.apache.derby.impl.store.access.conglomerate.GenericConglomerateController.fetch(Unknown Source)
at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSetAutoincrementValue(Unknown Source)
at org.apache.derby.impl.sql.execute.InsertResultSet.getSetAutoincrementValue(Unknown Source)
at org.apache.derby.impl.sql.execute.BaseActivation.getSetAutoincrementValue(Unknown Source)
at org.apache.derby.exe.ace8b858bfx013ax1d43x6465x000004f30e0821d.e0(Unknown Source)
at org.apache.derby.impl.services.reflect.DirectCall.invoke(Unknown Source)
at org.apache.derby.impl.sql.execute.RowResultSet.getNextRowCore(Unknown Source)
at org.apache.derby.impl.sql.execute.NormalizeResultSet.getNextRowCore(Unknown Source)
at org.apache.derby.impl.sql.execute.DMLWriteResultSet.getNextRowCore(Unknown Source)
at org.apache.derby.impl.sql.execute.InsertResultSet.open(Unknown Source)
at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown Source)
at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown Source)
... 38 more
was:
Seeing the following error which causes Avro clients to disconnect (I'm guessing due to transaction failure), failover to another agent, then eventually reconnect to the original agent.
#JDBC Channel Configuration
collector_agent.channels.defaultChannel.type = jdbc
collector_agent.channels.defaultChannel.db.type = DERBY
collector_agent.channels.defaultChannel.sysprop.user.home = /f/flume-channels/default/
collector_agent.channels.defaultChannel.capacity = 0
collector_agent.channels.defaultChannel.maximum.capacity = 0
#Log File error
2012-10-01 10:34:09,267 WARN impl.JdbcTransactionImpl: Marking transaction for rollback
2012-10-01 10:34:09,267 WARN impl.JdbcTransactionImpl: Marking transaction for rollback
2012-10-01 10:34:09,267 INFO impl.JdbcTransactionImpl: Attempting transaction roll-back
2012-10-01 10:34:09,268 ERROR source.AvroSource: Unable to process event batch
org.apache.flume.channel.jdbc.JdbcChannelException: Failed to persist event
at org.apache.flume.channel.jdbc.impl.JdbcChannelProviderImpl.persistEvent(JdbcChannelProviderImpl.java:315)
at org.apache.flume.channel.jdbc.JdbcChannel.put(JdbcChannel.java:44)
at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:187)
at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:229)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
at org.apache.avro.ipc.Responder.respond(Responder.java:149)
at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:140)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:125)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302)
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:214)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.flume.channel.jdbc.JdbcChannelException: Unable to persist event: org.apache.flume.channel.jdbc.impl.PersistableEvent@1b6ca9f0
at org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.storeEvent(DerbySchemaHandler.java:733)
at org.apache.flume.channel.jdbc.impl.JdbcChannelProviderImpl.persistEvent(JdbcChannelProviderImpl.java:308)
... 30 more
Caused by: java.sql.SQLTransactionRollbackException: A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
Lock : ROW, SYSCOLUMNS, (4,43)
Waiting XID : {242883779, X} , SA, INSERT INTO FLUME.FL_HEADER (FLH_EVENT, FLH_NAME, FLH_VALUE, FLH_NMSPILL, FLH_VLSPILL) VALUES ( ?, ?, ?, ?, ?)
Granted XID : {242883747, X}
Lock : ROW, SYSCOLUMNS, (4,37)
Waiting XID : {242883747, X} , SA, INSERT INTO FLUME.FL_EVENT (FLE_PAYLOAD, FLE_CHANNEL, FLE_SPILL) VALUES ( ?, ?, ?)
Granted XID : {242883779, X}
. The selected victim is XID : 242883779.
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Unknown Source)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
at org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.storeEvent(DerbySchemaHandler.java:646)
... 31 more
Caused by: java.sql.SQLException: A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
Lock : ROW, SYSCOLUMNS, (4,43)
Waiting XID : {242883779, X} , SA, INSERT INTO FLUME.FL_HEADER (FLH_EVENT, FLH_NAME, FLH_VALUE, FLH_NMSPILL, FLH_VLSPILL) VALUES ( ?, ?, ?, ?, ?)
Granted XID : {242883747, X}
Lock : ROW, SYSCOLUMNS, (4,37)
Waiting XID : {242883747, X} , SA, INSERT INTO FLUME.FL_EVENT (FLE_PAYLOAD, FLE_CHANNEL, FLE_SPILL) VALUES ( ?, ?, ?)
Granted XID : {242883779, X}
. The selected victim is XID : 242883779.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
... 44 more
Caused by: ERROR 40001: A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
Lock : ROW, SYSCOLUMNS, (4,43)
Waiting XID : {242883779, X} , SA, INSERT INTO FLUME.FL_HEADER (FLH_EVENT, FLH_NAME, FLH_VALUE, FLH_NMSPILL, FLH_VLSPILL) VALUES ( ?, ?, ?, ?, ?)
Granted XID : {242883747, X}
Lock : ROW, SYSCOLUMNS, (4,37)
Waiting XID : {242883747, X} , SA, INSERT INTO FLUME.FL_EVENT (FLE_PAYLOAD, FLE_CHANNEL, FLE_SPILL) VALUES ( ?, ?, ?)
Granted XID : {242883779, X}
. The selected victim is XID : 242883779.
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.impl.services.locks.Deadlock.buildException(Unknown Source)
at org.apache.derby.impl.services.locks.ConcurrentLockSet.lockObject(Unknown Source)
at org.apache.derby.impl.services.locks.AbstractPool.lockObject(Unknown Source)
at org.apache.derby.impl.services.locks.ConcurrentPool.lockObject(Unknown Source)
at org.apache.derby.impl.store.raw.xact.RowLocking3.lockRecordForWrite(Unknown Source)
at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.lockPositionForWrite(Unknown Source)
at org.apache.derby.impl.store.access.conglomerate.GenericConglomerateController.fetch(Unknown Source)
at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSetAutoincrementValue(Unknown Source)
at org.apache.derby.impl.sql.execute.InsertResultSet.getSetAutoincrementValue(Unknown Source)
at org.apache.derby.impl.sql.execute.BaseActivation.getSetAutoincrementValue(Unknown Source)
at org.apache.derby.exe.ace8b858bfx013ax1d43x6465x000004f30e0821d.e0(Unknown Source)
at org.apache.derby.impl.services.reflect.DirectCall.invoke(Unknown Source)
at org.apache.derby.impl.sql.execute.RowResultSet.getNextRowCore(Unknown Source)
at org.apache.derby.impl.sql.execute.NormalizeResultSet.getNextRowCore(Unknown Source)
at org.apache.derby.impl.sql.execute.DMLWriteResultSet.getNextRowCore(Unknown Source)
at org.apache.derby.impl.sql.execute.InsertResultSet.open(Unknown Source)
at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown Source)
at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown Source)
... 38 more
> JDBC Channel receiving deadlock transaction rollback
> ----------------------------------------------------
>
> Key: FLUME-1636
> URL: https://issues.apache.org/jira/browse/FLUME-1636
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.1.0
> Environment: Running on Linux with Java 1.6.0u21 (Oracle Hotspot JVM), CDH 4.0.1 version of Flume v1.1.0.
> Reporter: Aaron Baff
>
> Seeing the following error which causes Avro clients to disconnect (I'm guessing due to transaction failure), failover to another agent, then eventually reconnect to the original agent. There were 3 nodes which were writing (over Avro) to this single node.
> In the log file there are approximately 183,390 log records over a 2 to 2 1/2 weeks of run time. Additionally, after doing a check of the data being written to HDFS, it appears only approximately 10% of the data is there.
> #JDBC Channel Configuration
> collector_agent.channels.defaultChannel.type = jdbc
> collector_agent.channels.defaultChannel.db.type = DERBY
> collector_agent.channels.defaultChannel.sysprop.user.home = /f/flume-channels/default/
> collector_agent.channels.defaultChannel.capacity = 0
> collector_agent.channels.defaultChannel.maximum.capacity = 0
> #Log File error
> 2012-10-01 10:34:09,267 WARN impl.JdbcTransactionImpl: Marking transaction for rollback
> 2012-10-01 10:34:09,267 WARN impl.JdbcTransactionImpl: Marking transaction for rollback
> 2012-10-01 10:34:09,267 INFO impl.JdbcTransactionImpl: Attempting transaction roll-back
> 2012-10-01 10:34:09,268 ERROR source.AvroSource: Unable to process event batch
> org.apache.flume.channel.jdbc.JdbcChannelException: Failed to persist event
> at org.apache.flume.channel.jdbc.impl.JdbcChannelProviderImpl.persistEvent(JdbcChannelProviderImpl.java:315)
> at org.apache.flume.channel.jdbc.JdbcChannel.put(JdbcChannel.java:44)
> at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:187)
> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:229)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:140)
> at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
> at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:125)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:214)
> at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
> at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
> at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.flume.channel.jdbc.JdbcChannelException: Unable to persist event: org.apache.flume.channel.jdbc.impl.PersistableEvent@1b6ca9f0
> at org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.storeEvent(DerbySchemaHandler.java:733)
> at org.apache.flume.channel.jdbc.impl.JdbcChannelProviderImpl.persistEvent(JdbcChannelProviderImpl.java:308)
> ... 30 more
> Caused by: java.sql.SQLTransactionRollbackException: A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
> Lock : ROW, SYSCOLUMNS, (4,43)
> Waiting XID : {242883779, X} , SA, INSERT INTO FLUME.FL_HEADER (FLH_EVENT, FLH_NAME, FLH_VALUE, FLH_NMSPILL, FLH_VLSPILL) VALUES ( ?, ?, ?, ?, ?)
> Granted XID : {242883747, X}
> Lock : ROW, SYSCOLUMNS, (4,37)
> Waiting XID : {242883747, X} , SA, INSERT INTO FLUME.FL_EVENT (FLE_PAYLOAD, FLE_CHANNEL, FLE_SPILL) VALUES ( ?, ?, ?)
> Granted XID : {242883779, X}
> . The selected victim is XID : 242883779.
> at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Unknown Source)
> at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
> at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
> at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
> at org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.storeEvent(DerbySchemaHandler.java:646)
> ... 31 more
> Caused by: java.sql.SQLException: A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
> Lock : ROW, SYSCOLUMNS, (4,43)
> Waiting XID : {242883779, X} , SA, INSERT INTO FLUME.FL_HEADER (FLH_EVENT, FLH_NAME, FLH_VALUE, FLH_NMSPILL, FLH_VLSPILL) VALUES ( ?, ?, ?, ?, ?)
> Granted XID : {242883747, X}
> Lock : ROW, SYSCOLUMNS, (4,37)
> Waiting XID : {242883747, X} , SA, INSERT INTO FLUME.FL_EVENT (FLE_PAYLOAD, FLE_CHANNEL, FLE_SPILL) VALUES ( ?, ?, ?)
> Granted XID : {242883779, X}
> . The selected victim is XID : 242883779.
> at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
> ... 44 more
> Caused by: ERROR 40001: A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
> Lock : ROW, SYSCOLUMNS, (4,43)
> Waiting XID : {242883779, X} , SA, INSERT INTO FLUME.FL_HEADER (FLH_EVENT, FLH_NAME, FLH_VALUE, FLH_NMSPILL, FLH_VLSPILL) VALUES ( ?, ?, ?, ?, ?)
> Granted XID : {242883747, X}
> Lock : ROW, SYSCOLUMNS, (4,37)
> Waiting XID : {242883747, X} , SA, INSERT INTO FLUME.FL_EVENT (FLE_PAYLOAD, FLE_CHANNEL, FLE_SPILL) VALUES ( ?, ?, ?)
> Granted XID : {242883779, X}
> . The selected victim is XID : 242883779.
> at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
> at org.apache.derby.impl.services.locks.Deadlock.buildException(Unknown Source)
> at org.apache.derby.impl.services.locks.ConcurrentLockSet.lockObject(Unknown Source)
> at org.apache.derby.impl.services.locks.AbstractPool.lockObject(Unknown Source)
> at org.apache.derby.impl.services.locks.ConcurrentPool.lockObject(Unknown Source)
> at org.apache.derby.impl.store.raw.xact.RowLocking3.lockRecordForWrite(Unknown Source)
> at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.lockPositionForWrite(Unknown Source)
> at org.apache.derby.impl.store.access.conglomerate.GenericConglomerateController.fetch(Unknown Source)
> at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSetAutoincrementValue(Unknown Source)
> at org.apache.derby.impl.sql.execute.InsertResultSet.getSetAutoincrementValue(Unknown Source)
> at org.apache.derby.impl.sql.execute.BaseActivation.getSetAutoincrementValue(Unknown Source)
> at org.apache.derby.exe.ace8b858bfx013ax1d43x6465x000004f30e0821d.e0(Unknown Source)
> at org.apache.derby.impl.services.reflect.DirectCall.invoke(Unknown Source)
> at org.apache.derby.impl.sql.execute.RowResultSet.getNextRowCore(Unknown Source)
> at org.apache.derby.impl.sql.execute.NormalizeResultSet.getNextRowCore(Unknown Source)
> at org.apache.derby.impl.sql.execute.DMLWriteResultSet.getNextRowCore(Unknown Source)
> at org.apache.derby.impl.sql.execute.InsertResultSet.open(Unknown Source)
> at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown Source)
> at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown Source)
> ... 38 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1636) JDBC Channel receiving deadlock
transaction rollback
Posted by "Aaron Baff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Baff updated FLUME-1636:
------------------------------
Description:
Seeing the following error which causes Avro clients to disconnect (I'm guessing due to transaction failure), failover to another agent, then eventually reconnect to the original agent. There were 3 nodes which were writing (over Avro) to this single node.
In the log file there are approximately 900+ log records over a 2 to 2 1/2 weeks of run time. Additionally, after doing a check of the data being written to HDFS, it appears only approximately 10% of the data is there.
#JDBC Channel Configuration
collector_agent.channels.defaultChannel.type = jdbc
collector_agent.channels.defaultChannel.db.type = DERBY
collector_agent.channels.defaultChannel.sysprop.user.home = /f/flume-channels/default/
collector_agent.channels.defaultChannel.capacity = 0
collector_agent.channels.defaultChannel.maximum.capacity = 0
#Log File error
2012-10-01 10:34:09,267 WARN impl.JdbcTransactionImpl: Marking transaction for rollback
2012-10-01 10:34:09,267 WARN impl.JdbcTransactionImpl: Marking transaction for rollback
2012-10-01 10:34:09,267 INFO impl.JdbcTransactionImpl: Attempting transaction roll-back
2012-10-01 10:34:09,268 ERROR source.AvroSource: Unable to process event batch
org.apache.flume.channel.jdbc.JdbcChannelException: Failed to persist event
at org.apache.flume.channel.jdbc.impl.JdbcChannelProviderImpl.persistEvent(JdbcChannelProviderImpl.java:315)
at org.apache.flume.channel.jdbc.JdbcChannel.put(JdbcChannel.java:44)
at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:187)
at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:229)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
at org.apache.avro.ipc.Responder.respond(Responder.java:149)
at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:140)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:125)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302)
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:214)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.flume.channel.jdbc.JdbcChannelException: Unable to persist event: org.apache.flume.channel.jdbc.impl.PersistableEvent@1b6ca9f0
at org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.storeEvent(DerbySchemaHandler.java:733)
at org.apache.flume.channel.jdbc.impl.JdbcChannelProviderImpl.persistEvent(JdbcChannelProviderImpl.java:308)
... 30 more
Caused by: java.sql.SQLTransactionRollbackException: A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
Lock : ROW, SYSCOLUMNS, (4,43)
Waiting XID : {242883779, X} , SA, INSERT INTO FLUME.FL_HEADER (FLH_EVENT, FLH_NAME, FLH_VALUE, FLH_NMSPILL, FLH_VLSPILL) VALUES ( ?, ?, ?, ?, ?)
Granted XID : {242883747, X}
Lock : ROW, SYSCOLUMNS, (4,37)
Waiting XID : {242883747, X} , SA, INSERT INTO FLUME.FL_EVENT (FLE_PAYLOAD, FLE_CHANNEL, FLE_SPILL) VALUES ( ?, ?, ?)
Granted XID : {242883779, X}
. The selected victim is XID : 242883779.
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Unknown Source)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
at org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.storeEvent(DerbySchemaHandler.java:646)
... 31 more
Caused by: java.sql.SQLException: A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
Lock : ROW, SYSCOLUMNS, (4,43)
Waiting XID : {242883779, X} , SA, INSERT INTO FLUME.FL_HEADER (FLH_EVENT, FLH_NAME, FLH_VALUE, FLH_NMSPILL, FLH_VLSPILL) VALUES ( ?, ?, ?, ?, ?)
Granted XID : {242883747, X}
Lock : ROW, SYSCOLUMNS, (4,37)
Waiting XID : {242883747, X} , SA, INSERT INTO FLUME.FL_EVENT (FLE_PAYLOAD, FLE_CHANNEL, FLE_SPILL) VALUES ( ?, ?, ?)
Granted XID : {242883779, X}
. The selected victim is XID : 242883779.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
... 44 more
Caused by: ERROR 40001: A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
Lock : ROW, SYSCOLUMNS, (4,43)
Waiting XID : {242883779, X} , SA, INSERT INTO FLUME.FL_HEADER (FLH_EVENT, FLH_NAME, FLH_VALUE, FLH_NMSPILL, FLH_VLSPILL) VALUES ( ?, ?, ?, ?, ?)
Granted XID : {242883747, X}
Lock : ROW, SYSCOLUMNS, (4,37)
Waiting XID : {242883747, X} , SA, INSERT INTO FLUME.FL_EVENT (FLE_PAYLOAD, FLE_CHANNEL, FLE_SPILL) VALUES ( ?, ?, ?)
Granted XID : {242883779, X}
. The selected victim is XID : 242883779.
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.impl.services.locks.Deadlock.buildException(Unknown Source)
at org.apache.derby.impl.services.locks.ConcurrentLockSet.lockObject(Unknown Source)
at org.apache.derby.impl.services.locks.AbstractPool.lockObject(Unknown Source)
at org.apache.derby.impl.services.locks.ConcurrentPool.lockObject(Unknown Source)
at org.apache.derby.impl.store.raw.xact.RowLocking3.lockRecordForWrite(Unknown Source)
at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.lockPositionForWrite(Unknown Source)
at org.apache.derby.impl.store.access.conglomerate.GenericConglomerateController.fetch(Unknown Source)
at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSetAutoincrementValue(Unknown Source)
at org.apache.derby.impl.sql.execute.InsertResultSet.getSetAutoincrementValue(Unknown Source)
at org.apache.derby.impl.sql.execute.BaseActivation.getSetAutoincrementValue(Unknown Source)
at org.apache.derby.exe.ace8b858bfx013ax1d43x6465x000004f30e0821d.e0(Unknown Source)
at org.apache.derby.impl.services.reflect.DirectCall.invoke(Unknown Source)
at org.apache.derby.impl.sql.execute.RowResultSet.getNextRowCore(Unknown Source)
at org.apache.derby.impl.sql.execute.NormalizeResultSet.getNextRowCore(Unknown Source)
at org.apache.derby.impl.sql.execute.DMLWriteResultSet.getNextRowCore(Unknown Source)
at org.apache.derby.impl.sql.execute.InsertResultSet.open(Unknown Source)
at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown Source)
at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown Source)
... 38 more
was:
Seeing the following error which causes Avro clients to disconnect (I'm guessing due to transaction failure), failover to another agent, then eventually reconnect to the original agent. There were 3 nodes which were writing (over Avro) to this single node.
In the log file there are approximately 183,390 log records over a 2 to 2 1/2 weeks of run time. Additionally, after doing a check of the data being written to HDFS, it appears only approximately 10% of the data is there.
#JDBC Channel Configuration
collector_agent.channels.defaultChannel.type = jdbc
collector_agent.channels.defaultChannel.db.type = DERBY
collector_agent.channels.defaultChannel.sysprop.user.home = /f/flume-channels/default/
collector_agent.channels.defaultChannel.capacity = 0
collector_agent.channels.defaultChannel.maximum.capacity = 0
#Log File error
2012-10-01 10:34:09,267 WARN impl.JdbcTransactionImpl: Marking transaction for rollback
2012-10-01 10:34:09,267 WARN impl.JdbcTransactionImpl: Marking transaction for rollback
2012-10-01 10:34:09,267 INFO impl.JdbcTransactionImpl: Attempting transaction roll-back
2012-10-01 10:34:09,268 ERROR source.AvroSource: Unable to process event batch
org.apache.flume.channel.jdbc.JdbcChannelException: Failed to persist event
at org.apache.flume.channel.jdbc.impl.JdbcChannelProviderImpl.persistEvent(JdbcChannelProviderImpl.java:315)
at org.apache.flume.channel.jdbc.JdbcChannel.put(JdbcChannel.java:44)
at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:187)
at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:229)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
at org.apache.avro.ipc.Responder.respond(Responder.java:149)
at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:140)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:125)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302)
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:214)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.flume.channel.jdbc.JdbcChannelException: Unable to persist event: org.apache.flume.channel.jdbc.impl.PersistableEvent@1b6ca9f0
at org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.storeEvent(DerbySchemaHandler.java:733)
at org.apache.flume.channel.jdbc.impl.JdbcChannelProviderImpl.persistEvent(JdbcChannelProviderImpl.java:308)
... 30 more
Caused by: java.sql.SQLTransactionRollbackException: A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
Lock : ROW, SYSCOLUMNS, (4,43)
Waiting XID : {242883779, X} , SA, INSERT INTO FLUME.FL_HEADER (FLH_EVENT, FLH_NAME, FLH_VALUE, FLH_NMSPILL, FLH_VLSPILL) VALUES ( ?, ?, ?, ?, ?)
Granted XID : {242883747, X}
Lock : ROW, SYSCOLUMNS, (4,37)
Waiting XID : {242883747, X} , SA, INSERT INTO FLUME.FL_EVENT (FLE_PAYLOAD, FLE_CHANNEL, FLE_SPILL) VALUES ( ?, ?, ?)
Granted XID : {242883779, X}
. The selected victim is XID : 242883779.
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Unknown Source)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
at org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.storeEvent(DerbySchemaHandler.java:646)
... 31 more
Caused by: java.sql.SQLException: A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
Lock : ROW, SYSCOLUMNS, (4,43)
Waiting XID : {242883779, X} , SA, INSERT INTO FLUME.FL_HEADER (FLH_EVENT, FLH_NAME, FLH_VALUE, FLH_NMSPILL, FLH_VLSPILL) VALUES ( ?, ?, ?, ?, ?)
Granted XID : {242883747, X}
Lock : ROW, SYSCOLUMNS, (4,37)
Waiting XID : {242883747, X} , SA, INSERT INTO FLUME.FL_EVENT (FLE_PAYLOAD, FLE_CHANNEL, FLE_SPILL) VALUES ( ?, ?, ?)
Granted XID : {242883779, X}
. The selected victim is XID : 242883779.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
... 44 more
Caused by: ERROR 40001: A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
Lock : ROW, SYSCOLUMNS, (4,43)
Waiting XID : {242883779, X} , SA, INSERT INTO FLUME.FL_HEADER (FLH_EVENT, FLH_NAME, FLH_VALUE, FLH_NMSPILL, FLH_VLSPILL) VALUES ( ?, ?, ?, ?, ?)
Granted XID : {242883747, X}
Lock : ROW, SYSCOLUMNS, (4,37)
Waiting XID : {242883747, X} , SA, INSERT INTO FLUME.FL_EVENT (FLE_PAYLOAD, FLE_CHANNEL, FLE_SPILL) VALUES ( ?, ?, ?)
Granted XID : {242883779, X}
. The selected victim is XID : 242883779.
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.impl.services.locks.Deadlock.buildException(Unknown Source)
at org.apache.derby.impl.services.locks.ConcurrentLockSet.lockObject(Unknown Source)
at org.apache.derby.impl.services.locks.AbstractPool.lockObject(Unknown Source)
at org.apache.derby.impl.services.locks.ConcurrentPool.lockObject(Unknown Source)
at org.apache.derby.impl.store.raw.xact.RowLocking3.lockRecordForWrite(Unknown Source)
at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.lockPositionForWrite(Unknown Source)
at org.apache.derby.impl.store.access.conglomerate.GenericConglomerateController.fetch(Unknown Source)
at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSetAutoincrementValue(Unknown Source)
at org.apache.derby.impl.sql.execute.InsertResultSet.getSetAutoincrementValue(Unknown Source)
at org.apache.derby.impl.sql.execute.BaseActivation.getSetAutoincrementValue(Unknown Source)
at org.apache.derby.exe.ace8b858bfx013ax1d43x6465x000004f30e0821d.e0(Unknown Source)
at org.apache.derby.impl.services.reflect.DirectCall.invoke(Unknown Source)
at org.apache.derby.impl.sql.execute.RowResultSet.getNextRowCore(Unknown Source)
at org.apache.derby.impl.sql.execute.NormalizeResultSet.getNextRowCore(Unknown Source)
at org.apache.derby.impl.sql.execute.DMLWriteResultSet.getNextRowCore(Unknown Source)
at org.apache.derby.impl.sql.execute.InsertResultSet.open(Unknown Source)
at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown Source)
at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown Source)
... 38 more
> JDBC Channel receiving deadlock transaction rollback
> ----------------------------------------------------
>
> Key: FLUME-1636
> URL: https://issues.apache.org/jira/browse/FLUME-1636
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.1.0
> Environment: Running on Linux with Java 1.6.0u21 (Oracle Hotspot JVM), CDH 4.0.1 version of Flume v1.1.0.
> Reporter: Aaron Baff
>
> Seeing the following error which causes Avro clients to disconnect (I'm guessing due to transaction failure), failover to another agent, then eventually reconnect to the original agent. There were 3 nodes which were writing (over Avro) to this single node.
> In the log file there are approximately 900+ log records over a 2 to 2 1/2 weeks of run time. Additionally, after doing a check of the data being written to HDFS, it appears only approximately 10% of the data is there.
> #JDBC Channel Configuration
> collector_agent.channels.defaultChannel.type = jdbc
> collector_agent.channels.defaultChannel.db.type = DERBY
> collector_agent.channels.defaultChannel.sysprop.user.home = /f/flume-channels/default/
> collector_agent.channels.defaultChannel.capacity = 0
> collector_agent.channels.defaultChannel.maximum.capacity = 0
> #Log File error
> 2012-10-01 10:34:09,267 WARN impl.JdbcTransactionImpl: Marking transaction for rollback
> 2012-10-01 10:34:09,267 WARN impl.JdbcTransactionImpl: Marking transaction for rollback
> 2012-10-01 10:34:09,267 INFO impl.JdbcTransactionImpl: Attempting transaction roll-back
> 2012-10-01 10:34:09,268 ERROR source.AvroSource: Unable to process event batch
> org.apache.flume.channel.jdbc.JdbcChannelException: Failed to persist event
> at org.apache.flume.channel.jdbc.impl.JdbcChannelProviderImpl.persistEvent(JdbcChannelProviderImpl.java:315)
> at org.apache.flume.channel.jdbc.JdbcChannel.put(JdbcChannel.java:44)
> at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:187)
> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:229)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:140)
> at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
> at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:125)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:214)
> at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
> at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
> at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.flume.channel.jdbc.JdbcChannelException: Unable to persist event: org.apache.flume.channel.jdbc.impl.PersistableEvent@1b6ca9f0
> at org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.storeEvent(DerbySchemaHandler.java:733)
> at org.apache.flume.channel.jdbc.impl.JdbcChannelProviderImpl.persistEvent(JdbcChannelProviderImpl.java:308)
> ... 30 more
> Caused by: java.sql.SQLTransactionRollbackException: A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
> Lock : ROW, SYSCOLUMNS, (4,43)
> Waiting XID : {242883779, X} , SA, INSERT INTO FLUME.FL_HEADER (FLH_EVENT, FLH_NAME, FLH_VALUE, FLH_NMSPILL, FLH_VLSPILL) VALUES ( ?, ?, ?, ?, ?)
> Granted XID : {242883747, X}
> Lock : ROW, SYSCOLUMNS, (4,37)
> Waiting XID : {242883747, X} , SA, INSERT INTO FLUME.FL_EVENT (FLE_PAYLOAD, FLE_CHANNEL, FLE_SPILL) VALUES ( ?, ?, ?)
> Granted XID : {242883779, X}
> . The selected victim is XID : 242883779.
> at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Unknown Source)
> at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
> at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
> at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
> at org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.storeEvent(DerbySchemaHandler.java:646)
> ... 31 more
> Caused by: java.sql.SQLException: A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
> Lock : ROW, SYSCOLUMNS, (4,43)
> Waiting XID : {242883779, X} , SA, INSERT INTO FLUME.FL_HEADER (FLH_EVENT, FLH_NAME, FLH_VALUE, FLH_NMSPILL, FLH_VLSPILL) VALUES ( ?, ?, ?, ?, ?)
> Granted XID : {242883747, X}
> Lock : ROW, SYSCOLUMNS, (4,37)
> Waiting XID : {242883747, X} , SA, INSERT INTO FLUME.FL_EVENT (FLE_PAYLOAD, FLE_CHANNEL, FLE_SPILL) VALUES ( ?, ?, ?)
> Granted XID : {242883779, X}
> . The selected victim is XID : 242883779.
> at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
> ... 44 more
> Caused by: ERROR 40001: A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
> Lock : ROW, SYSCOLUMNS, (4,43)
> Waiting XID : {242883779, X} , SA, INSERT INTO FLUME.FL_HEADER (FLH_EVENT, FLH_NAME, FLH_VALUE, FLH_NMSPILL, FLH_VLSPILL) VALUES ( ?, ?, ?, ?, ?)
> Granted XID : {242883747, X}
> Lock : ROW, SYSCOLUMNS, (4,37)
> Waiting XID : {242883747, X} , SA, INSERT INTO FLUME.FL_EVENT (FLE_PAYLOAD, FLE_CHANNEL, FLE_SPILL) VALUES ( ?, ?, ?)
> Granted XID : {242883779, X}
> . The selected victim is XID : 242883779.
> at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
> at org.apache.derby.impl.services.locks.Deadlock.buildException(Unknown Source)
> at org.apache.derby.impl.services.locks.ConcurrentLockSet.lockObject(Unknown Source)
> at org.apache.derby.impl.services.locks.AbstractPool.lockObject(Unknown Source)
> at org.apache.derby.impl.services.locks.ConcurrentPool.lockObject(Unknown Source)
> at org.apache.derby.impl.store.raw.xact.RowLocking3.lockRecordForWrite(Unknown Source)
> at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.lockPositionForWrite(Unknown Source)
> at org.apache.derby.impl.store.access.conglomerate.GenericConglomerateController.fetch(Unknown Source)
> at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSetAutoincrementValue(Unknown Source)
> at org.apache.derby.impl.sql.execute.InsertResultSet.getSetAutoincrementValue(Unknown Source)
> at org.apache.derby.impl.sql.execute.BaseActivation.getSetAutoincrementValue(Unknown Source)
> at org.apache.derby.exe.ace8b858bfx013ax1d43x6465x000004f30e0821d.e0(Unknown Source)
> at org.apache.derby.impl.services.reflect.DirectCall.invoke(Unknown Source)
> at org.apache.derby.impl.sql.execute.RowResultSet.getNextRowCore(Unknown Source)
> at org.apache.derby.impl.sql.execute.NormalizeResultSet.getNextRowCore(Unknown Source)
> at org.apache.derby.impl.sql.execute.DMLWriteResultSet.getNextRowCore(Unknown Source)
> at org.apache.derby.impl.sql.execute.InsertResultSet.open(Unknown Source)
> at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown Source)
> at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown Source)
> ... 38 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira