You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@manifoldcf.apache.org by "Karl Wright (JIRA)" <ji...@apache.org> on 2017/01/16 08:32:26 UTC
[jira] [Commented] (CONNECTORS-1367) Alfresco Connector deadlock
with postgresql
[ https://issues.apache.org/jira/browse/CONNECTORS-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15823613#comment-15823613 ]
Karl Wright commented on CONNECTORS-1367:
-----------------------------------------
This is a pretty odd looking stack frame; none of the thread ID's are there and thus it looks like it was snapped when the agents process was shutting down, rather than actually running and stuck. It does look, however, like an HTTP transaction with Alfresco is hung for some reason:
{code}
Thread 19436: (state = IN_NATIVE)
- java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) @bci=0 (Compiled frame; information may be imprecise)
- java.net.SocketInputStream.socketRead(java.io.FileDescriptor, byte[], int, int, int) @bci=8, line=116 (Compiled frame)
- java.net.SocketInputStream.read(byte[], int, int, int) @bci=79, line=170 (Compiled frame)
- java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=141 (Compiled frame)
- org.apache.http.impl.io.SessionInputBufferImpl.streamRead(byte[], int, int) @bci=16, line=139 (Compiled frame)
- org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer() @bci=68, line=155 (Compiled frame)
- org.apache.http.impl.io.SessionInputBufferImpl.readLine(org.apache.http.util.CharArrayBuffer) @bci=227, line=284 (Compiled frame)
- org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer) @bci=16, line=140 (Compiled frame)
- org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer) @bci=2, line=57 (Compiled frame)
- org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38, line=261 (Compiled frame)
- org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader() @bci=8, line=165 (Compiled frame)
- org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader() @bci=4, line=167 (Compiled frame)
- org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest, org.apache.http.HttpClientConnection, org.apache.http.protocol.HttpContext) @bci=41, line=272 (Compiled frame)
- org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest, org.apache.http.HttpClientConnection, org.apache.http.protocol.HttpContext) @bci=39, line=124 (Compiled frame)
- org.apache.http.impl.execchain.MainClientExec.execute(org.apache.http.conn.routing.HttpRoute, org.apache.http.client.methods.HttpRequestWrapper, org.apache.http.client.protocol.HttpClientContext, org.apache.http.client.methods.HttpExecutionAware) @bci=714, line=271 (Compiled frame)
- org.apache.http.impl.execchain.ProtocolExec.execute(org.apache.http.conn.routing.HttpRoute, org.apache.http.client.methods.HttpRequestWrapper, org.apache.http.client.protocol.HttpClientContext, org.apache.http.client.methods.HttpExecutionAware) @bci=447, line=184 (Compiled frame)
- org.apache.http.impl.execchain.RetryExec.execute(org.apache.http.conn.routing.HttpRoute, org.apache.http.client.methods.HttpRequestWrapper, org.apache.http.client.protocol.HttpClientContext, org.apache.http.client.methods.HttpExecutionAware) @bci=39, line=88 (Compiled frame)
- org.apache.http.impl.execchain.RedirectExec.execute(org.apache.http.conn.routing.HttpRoute, org.apache.http.client.methods.HttpRequestWrapper, org.apache.http.client.protocol.HttpClientContext, org.apache.http.client.methods.HttpExecutionAware) @bci=85, line=110 (Compiled frame)
- org.apache.http.impl.client.InternalHttpClient.doExecute(org.apache.http.HttpHost, org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext) @bci=168, line=184 (Compiled frame)
- org.apache.http.impl.client.CloseableHttpClient.execute(org.apache.http.client.methods.HttpUriRequest, org.apache.http.protocol.HttpContext) @bci=14, line=82 (Compiled frame)
- org.apache.http.impl.client.CloseableHttpClient.execute(org.apache.http.client.methods.HttpUriRequest) @bci=6, line=107 (Compiled frame)
- com.github.maoo.indexer.client.WebScriptsAlfrescoClient.getDocumentsActions(java.lang.String) @bci=24, line=118 (Compiled frame)
- com.github.maoo.indexer.client.WebScriptsAlfrescoClient.fetchNodes(long, long, com.github.maoo.indexer.client.AlfrescoFilters) @bci=32, line=103 (Compiled frame)
- org.apache.manifoldcf.crawler.connectors.alfrescowebscript.AlfrescoConnector.addSeedDocuments(org.apache.manifoldcf.crawler.interfaces.ISeedingActivity, org.apache.manifoldcf.core.interfaces.Specification, java.lang.String, long, int) @bci=140, line=188 (Compiled frame)
- org.apache.manifoldcf.crawler.system.StartupThread.run() @bci=614, line=154 (Interpreted frame)
{code}
If any one thread that throws locks gets stuck in this kind of way, then it can cause a "train wreck" that causes transactions to eventually time out -- which seems to be what's happening -- so this could be the smoking gun.
This is occurring while the job is being "seeded". In jobs that are continuous, that happens periodically. But if the job is *not* continuous, that occurs once on every job run -- before any documents are crawled. [~chalitha.perera], is this a continuous job, or not? Very important to know the answer to that question.
One possibility is that the connector's seeding request is causing Alfresco to fall apart in trying to respond to it, probably because Alfresco is running out of memory. The thread that is answering the HTTP request is still alive but the thread that is computing the result dies, so the connection stays up but it will never be fulfilled. I don't know if this is possible or not but if it is you would see evidence for that in the Alfresco logs. [~maoo], what do you think?
> Alfresco Connector deadlock with postgresql
> -------------------------------------------
>
> Key: CONNECTORS-1367
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1367
> Project: ManifoldCF
> Issue Type: Bug
> Components: Alfresco webscript connector
> Affects Versions: ManifoldCF 2.5
> Reporter: Chalitha Perera
> Assignee: Karl Wright
> Attachments: jstack5.out
>
>
> Alfresco connector gets stuck when processing documents around 35,000 and postgres manifold database in idle state and after days gets deadlock message
> "
> 2016-12-21 10:54:17.946 UTC >ERROR: deadlock detected
> < 2016-12-21 10:54:17.946 UTC >DETAIL: Process 28072 waits for ShareLock on transaction 432231; blocked by process 28038.
> Process 28038 waits for ShareLock on transaction 432227; blocked by process 28072.
> Process 28072: UPDATE jobqueue SET failtime=NULL,checktime=$1,failcount=NULL,checkaction=$2 WHERE jobid=$3 AND (status=$4 OR status=$5)
> Process 28038: SELECT needpriority FROM jobqueue WHERE id=$1 FOR UPDATE
> < 2016-12-21 10:54:17.946 UTC >HINT: See server log for query details."
> On separate note, ran ant load-pg and it was completed successfully.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)