You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Dian Fu <di...@gmail.com> on 2015/08/20 10:10:44 UTC
Review Request 37637: Sqoop2: Deadlock occurs between getJobStatus and
JobManager.UpdateThread#updateSubmission
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37637/
-----------------------------------------------------------
Review request for Sqoop.
Bugs: SQOOP-2513
https://issues.apache.org/jira/browse/SQOOP-2513
Repository: sqoop-sqoop2
Description
-------
I encountered the following test failure log in the test of SQOOP-2439:
```Java
java.sql.SQLTransactionRollbackException: A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
Lock : ROW, SQ_JOB_INPUT, (1,187)
Waiting XID : {15543, S} , SA, SELECT "SQI_ID", "SQI_NAME", "SQI_CONFIG", "SQI_INDEX", "SQI_TYPE", "SQI_STRMASK", "SQI_STRLENGTH", "SQI_EDITABLE", "SQI_ENUMVALS", "SQBI_VALUE" FROM "SQOOP"."SQ_INPUT" LEFT OUTER JOIN "SQOOP"."SQ_JOB_INPUT" ON "SQBI_INPUT" = "SQI_ID" AND "SQBI_JOB" = ? WHERE "SQI_CONFIG" = ? ORDER BY "SQI_INDEX" Granted XID : {15545, X}
Lock : ROW, SQ_JOB, (1,17)
Waiting XID : {15545, X} , SA, UPDATE "SQOOP"."SQ_JOB" SET "SQB_NAME" = ?, "SQB_UPDATE_USER" = ?, "SQB_UPDATE_DATE" = ? WHERE "SQB_ID" = ? Granted XID : {15543, S} . The selected victim is XID : 15543. at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source) at
org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source) at
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source) at
org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source) at
org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source) at
org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source) at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown Source) at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeQuery(Unknown Source) at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) at
org.apache.sqoop.repository.common.CommonRepositoryHandler.loadDriverConfigs(CommonRepositoryHandler.java:2265) at
org.apache.sqoop.repository.common.CommonRepositoryHandler.loadJobs(CommonRepositoryHandler.java:1876) at
org.apache.sqoop.repository.common.CommonRepositoryHandler.findJob(CommonRepositoryHandler.java:885) at
org.apache.sqoop.repository.JdbcRepository$20.doIt(JdbcRepository.java:500) at
org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:92) at
org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:63) at
org.apache.sqoop.repository.JdbcRepository.findJob(JdbcRepository.java:497) at
org.apache.sqoop.driver.JobManager.getJob(JobManager.java:438) at
org.apache.sqoop.driver.JobManager.invokeDestroyerOnJobSuccess(JobManager.java:517) at
org.apache.sqoop.driver.JobManager.updateSubmission(JobManager.java:655) at
org.apache.sqoop.driver.JobManager.status(JobManager.java:636) at
org.apache.sqoop.handler.JobRequestHandler.getJobStatus(JobRequestHandler.java:410) at
org.apache.sqoop.handler.JobRequestHandler.handleEvent(JobRequestHandler.java:102)
```
>From the above log, we can see that transaction 15543 has the shared lock of table SQ_JOB and is requesting the shared lock of table SQ_JOB_INPUT, while transaction 15545 has the exclusive lock of table SQ_JOB_INPUT and is requesting the shared lock of table SQ_JOB. Deadlock occurs between these two transactions.
PS: transaction 15543 is a thread calling JobManager#getJobStatus and transaction 15545 is the updatethread in JobManager which is calling updateSubmission.
Diffs
-----
repository/repository-common/src/main/java/org/apache/sqoop/repository/common/CommonRepositoryHandler.java c090d8e
Diff: https://reviews.apache.org/r/37637/diff/
Testing
-------
Thanks,
Dian Fu
Re: Review Request 37637: Sqoop2: Deadlock occurs between
getJobStatus and JobManager.UpdateThread#updateSubmission
Posted by Dian Fu <di...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37637/
-----------------------------------------------------------
(Updated Aug. 25, 2015, 1:26 a.m.)
Review request for Sqoop.
Bugs: SQOOP-2513
https://issues.apache.org/jira/browse/SQOOP-2513
Repository: sqoop-sqoop2
Description
-------
I encountered the following test failure log in the test of SQOOP-2439:
```Java
java.sql.SQLTransactionRollbackException: A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
Lock : ROW, SQ_JOB_INPUT, (1,187)
Waiting XID : {15543, S} , SA, SELECT "SQI_ID", "SQI_NAME", "SQI_CONFIG", "SQI_INDEX", "SQI_TYPE", "SQI_STRMASK", "SQI_STRLENGTH", "SQI_EDITABLE", "SQI_ENUMVALS", "SQBI_VALUE" FROM "SQOOP"."SQ_INPUT" LEFT OUTER JOIN "SQOOP"."SQ_JOB_INPUT" ON "SQBI_INPUT" = "SQI_ID" AND "SQBI_JOB" = ? WHERE "SQI_CONFIG" = ? ORDER BY "SQI_INDEX" Granted XID : {15545, X}
Lock : ROW, SQ_JOB, (1,17)
Waiting XID : {15545, X} , SA, UPDATE "SQOOP"."SQ_JOB" SET "SQB_NAME" = ?, "SQB_UPDATE_USER" = ?, "SQB_UPDATE_DATE" = ? WHERE "SQB_ID" = ? Granted XID : {15543, S} . The selected victim is XID : 15543. at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source) at
org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source) at
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source) at
org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source) at
org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source) at
org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source) at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown Source) at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeQuery(Unknown Source) at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) at
org.apache.sqoop.repository.common.CommonRepositoryHandler.loadDriverConfigs(CommonRepositoryHandler.java:2265) at
org.apache.sqoop.repository.common.CommonRepositoryHandler.loadJobs(CommonRepositoryHandler.java:1876) at
org.apache.sqoop.repository.common.CommonRepositoryHandler.findJob(CommonRepositoryHandler.java:885) at
org.apache.sqoop.repository.JdbcRepository$20.doIt(JdbcRepository.java:500) at
org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:92) at
org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:63) at
org.apache.sqoop.repository.JdbcRepository.findJob(JdbcRepository.java:497) at
org.apache.sqoop.driver.JobManager.getJob(JobManager.java:438) at
org.apache.sqoop.driver.JobManager.invokeDestroyerOnJobSuccess(JobManager.java:517) at
org.apache.sqoop.driver.JobManager.updateSubmission(JobManager.java:655) at
org.apache.sqoop.driver.JobManager.status(JobManager.java:636) at
org.apache.sqoop.handler.JobRequestHandler.getJobStatus(JobRequestHandler.java:410) at
org.apache.sqoop.handler.JobRequestHandler.handleEvent(JobRequestHandler.java:102)
```
>From the above log, we can see that transaction 15543 has the shared lock of table SQ_JOB and is requesting the shared lock of table SQ_JOB_INPUT, while transaction 15545 has the exclusive lock of table SQ_JOB_INPUT and is requesting the shared lock of table SQ_JOB. Deadlock occurs between these two transactions.
PS: transaction 15543 is a thread calling JobManager#getJobStatus and transaction 15545 is the updatethread in JobManager which is calling updateSubmission.
Diffs (updated)
-----
repository/repository-common/src/main/java/org/apache/sqoop/repository/common/CommonRepositoryHandler.java b36e685
Diff: https://reviews.apache.org/r/37637/diff/
Testing
-------
Thanks,
Dian Fu
Re: Review Request 37637: Sqoop2: Deadlock occurs between
getJobStatus and JobManager.UpdateThread#updateSubmission
Posted by Dian Fu <di...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37637/
-----------------------------------------------------------
(Updated Aug. 21, 2015, 4:39 a.m.)
Review request for Sqoop.
Bugs: SQOOP-2513
https://issues.apache.org/jira/browse/SQOOP-2513
Repository: sqoop-sqoop2
Description
-------
I encountered the following test failure log in the test of SQOOP-2439:
```Java
java.sql.SQLTransactionRollbackException: A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
Lock : ROW, SQ_JOB_INPUT, (1,187)
Waiting XID : {15543, S} , SA, SELECT "SQI_ID", "SQI_NAME", "SQI_CONFIG", "SQI_INDEX", "SQI_TYPE", "SQI_STRMASK", "SQI_STRLENGTH", "SQI_EDITABLE", "SQI_ENUMVALS", "SQBI_VALUE" FROM "SQOOP"."SQ_INPUT" LEFT OUTER JOIN "SQOOP"."SQ_JOB_INPUT" ON "SQBI_INPUT" = "SQI_ID" AND "SQBI_JOB" = ? WHERE "SQI_CONFIG" = ? ORDER BY "SQI_INDEX" Granted XID : {15545, X}
Lock : ROW, SQ_JOB, (1,17)
Waiting XID : {15545, X} , SA, UPDATE "SQOOP"."SQ_JOB" SET "SQB_NAME" = ?, "SQB_UPDATE_USER" = ?, "SQB_UPDATE_DATE" = ? WHERE "SQB_ID" = ? Granted XID : {15543, S} . The selected victim is XID : 15543. at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source) at
org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source) at
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source) at
org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source) at
org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source) at
org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source) at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown Source) at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeQuery(Unknown Source) at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) at
org.apache.sqoop.repository.common.CommonRepositoryHandler.loadDriverConfigs(CommonRepositoryHandler.java:2265) at
org.apache.sqoop.repository.common.CommonRepositoryHandler.loadJobs(CommonRepositoryHandler.java:1876) at
org.apache.sqoop.repository.common.CommonRepositoryHandler.findJob(CommonRepositoryHandler.java:885) at
org.apache.sqoop.repository.JdbcRepository$20.doIt(JdbcRepository.java:500) at
org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:92) at
org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:63) at
org.apache.sqoop.repository.JdbcRepository.findJob(JdbcRepository.java:497) at
org.apache.sqoop.driver.JobManager.getJob(JobManager.java:438) at
org.apache.sqoop.driver.JobManager.invokeDestroyerOnJobSuccess(JobManager.java:517) at
org.apache.sqoop.driver.JobManager.updateSubmission(JobManager.java:655) at
org.apache.sqoop.driver.JobManager.status(JobManager.java:636) at
org.apache.sqoop.handler.JobRequestHandler.getJobStatus(JobRequestHandler.java:410) at
org.apache.sqoop.handler.JobRequestHandler.handleEvent(JobRequestHandler.java:102)
```
>From the above log, we can see that transaction 15543 has the shared lock of table SQ_JOB and is requesting the shared lock of table SQ_JOB_INPUT, while transaction 15545 has the exclusive lock of table SQ_JOB_INPUT and is requesting the shared lock of table SQ_JOB. Deadlock occurs between these two transactions.
PS: transaction 15543 is a thread calling JobManager#getJobStatus and transaction 15545 is the updatethread in JobManager which is calling updateSubmission.
Diffs (updated)
-----
repository/repository-common/src/main/java/org/apache/sqoop/repository/common/CommonRepositoryHandler.java c090d8e
Diff: https://reviews.apache.org/r/37637/diff/
Testing
-------
Thanks,
Dian Fu
Re: Review Request 37637: Sqoop2: Deadlock occurs between
getJobStatus and JobManager.UpdateThread#updateSubmission
Posted by Dian Fu <di...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37637/
-----------------------------------------------------------
(Updated Aug. 20, 2015, 8:13 a.m.)
Review request for Sqoop.
Bugs: SQOOP-2513
https://issues.apache.org/jira/browse/SQOOP-2513
Repository: sqoop-sqoop2
Description
-------
I encountered the following test failure log in the test of SQOOP-2439:
```Java
java.sql.SQLTransactionRollbackException: A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
Lock : ROW, SQ_JOB_INPUT, (1,187)
Waiting XID : {15543, S} , SA, SELECT "SQI_ID", "SQI_NAME", "SQI_CONFIG", "SQI_INDEX", "SQI_TYPE", "SQI_STRMASK", "SQI_STRLENGTH", "SQI_EDITABLE", "SQI_ENUMVALS", "SQBI_VALUE" FROM "SQOOP"."SQ_INPUT" LEFT OUTER JOIN "SQOOP"."SQ_JOB_INPUT" ON "SQBI_INPUT" = "SQI_ID" AND "SQBI_JOB" = ? WHERE "SQI_CONFIG" = ? ORDER BY "SQI_INDEX" Granted XID : {15545, X}
Lock : ROW, SQ_JOB, (1,17)
Waiting XID : {15545, X} , SA, UPDATE "SQOOP"."SQ_JOB" SET "SQB_NAME" = ?, "SQB_UPDATE_USER" = ?, "SQB_UPDATE_DATE" = ? WHERE "SQB_ID" = ? Granted XID : {15543, S} . The selected victim is XID : 15543. at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source) at
org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source) at
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source) at
org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source) at
org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source) at
org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source) at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown Source) at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeQuery(Unknown Source) at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) at
org.apache.sqoop.repository.common.CommonRepositoryHandler.loadDriverConfigs(CommonRepositoryHandler.java:2265) at
org.apache.sqoop.repository.common.CommonRepositoryHandler.loadJobs(CommonRepositoryHandler.java:1876) at
org.apache.sqoop.repository.common.CommonRepositoryHandler.findJob(CommonRepositoryHandler.java:885) at
org.apache.sqoop.repository.JdbcRepository$20.doIt(JdbcRepository.java:500) at
org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:92) at
org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:63) at
org.apache.sqoop.repository.JdbcRepository.findJob(JdbcRepository.java:497) at
org.apache.sqoop.driver.JobManager.getJob(JobManager.java:438) at
org.apache.sqoop.driver.JobManager.invokeDestroyerOnJobSuccess(JobManager.java:517) at
org.apache.sqoop.driver.JobManager.updateSubmission(JobManager.java:655) at
org.apache.sqoop.driver.JobManager.status(JobManager.java:636) at
org.apache.sqoop.handler.JobRequestHandler.getJobStatus(JobRequestHandler.java:410) at
org.apache.sqoop.handler.JobRequestHandler.handleEvent(JobRequestHandler.java:102)
```
>From the above log, we can see that transaction 15543 has the shared lock of table SQ_JOB and is requesting the shared lock of table SQ_JOB_INPUT, while transaction 15545 has the exclusive lock of table SQ_JOB_INPUT and is requesting the shared lock of table SQ_JOB. Deadlock occurs between these two transactions.
PS: transaction 15543 is a thread calling JobManager#getJobStatus and transaction 15545 is the updatethread in JobManager which is calling updateSubmission.
Diffs (updated)
-----
repository/repository-common/src/main/java/org/apache/sqoop/repository/common/CommonRepositoryHandler.java c090d8e
Diff: https://reviews.apache.org/r/37637/diff/
Testing
-------
Thanks,
Dian Fu