You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oodt.apache.org by "Mallder, Valerie" <Va...@jhuapl.edu> on 2016/05/31 21:27:34 UTC
Unable to index workflow instance error
Hi All,
I am getting a new error today. This is has never happened before, and nothing has changed except I think the machine was rebooted and oodt was not shut down properly. Has anyone seen this before? (Note, the left-most numbers on each line are just the line numbers to my oodt.out file.)
Thanks,
Val
119069 INFO: WorkflowManager: Workflow processLevel0FilesWorkflow retrieved for event processLevel0Files
119070 May 31, 2016 4:23:27 PM org.apache.oodt.cas.workflow.instrepo.LuceneWorkflowInstanceRepository addWorkflowInstanceToCatalog
119071 WARNING: Unable to index workflow instance: [e974c6a8-2775-11e6-8e5b-af388491908f]: Message: Lock obtain timed out: Lock@/tmp/lucene-27b0592844527a5c6b7179d062354965-write.lock
119072 org.apache.oodt.cas.workflow.structs.exceptions.InstanceRepositoryException: Unable to index workflow instance: [e974c6a8-2775-11e6-8e5b-af388491908f]: Message: Lock obtain timed out: Lock@/tmp/lucene-27b0592844527a5c6b717 9d062354965-write.lock
119073 at org.apache.oodt.cas.workflow.instrepo.LuceneWorkflowInstanceRepository.addWorkflowInstanceToCatalog(LuceneWorkflowInstanceRepository.java:464)
119074 at org.apache.oodt.cas.workflow.instrepo.LuceneWorkflowInstanceRepository.addWorkflowInstance(LuceneWorkflowInstanceRepository.java:177)
119075 at org.apache.oodt.cas.workflow.engine.ThreadPoolWorkflowEngine.persistWorkflowInstance(ThreadPoolWorkflowEngine.java:444)
119076 at org.apache.oodt.cas.workflow.engine.ThreadPoolWorkflowEngine.startWorkflow(ThreadPoolWorkflowEngine.java:203)
119077 at org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager.handleEvent(XmlRpcWorkflowManager.java:311)
119078 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
119079 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
119080 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
119081 at java.lang.reflect.Method.invoke(Method.java:606)
119082 at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
119083 at org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
119084 at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
119085 at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
119086 at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
119087 at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
119088 at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
Re: Unable to index workflow instance error
Posted by Tom Barber <to...@meteorite.bi>.
Don't quote me on it but it's probably just stale lock files:
https://wiki.apache.org/lucene-java/LuceneFAQ#My_program_crashed_and_now_I_get_a_.22Lock_obtain_timed_out..22_error._Where_is_the_lock_and_how_can_i_delete_it.3F
Tom
On Tue, May 31, 2016 at 10:27 PM, Mallder, Valerie <
Valerie.Mallder@jhuapl.edu> wrote:
> Hi All,
>
> I am getting a new error today. This is has never happened before, and
> nothing has changed except I think the machine was rebooted and oodt was
> not shut down properly. Has anyone seen this before? (Note, the left-most
> numbers on each line are just the line numbers to my oodt.out file.)
>
> Thanks,
> Val
>
>
> 119069 INFO: WorkflowManager: Workflow processLevel0FilesWorkflow
> retrieved for event processLevel0Files
> 119070 May 31, 2016 4:23:27 PM
> org.apache.oodt.cas.workflow.instrepo.LuceneWorkflowInstanceRepository
> addWorkflowInstanceToCatalog
> 119071 WARNING: Unable to index workflow instance:
> [e974c6a8-2775-11e6-8e5b-af388491908f]: Message: Lock obtain timed out:
> Lock@/tmp/lucene-27b0592844527a5c6b7179d062354965-write.lock
> 119072
> org.apache.oodt.cas.workflow.structs.exceptions.InstanceRepositoryException:
> Unable to index workflow instance: [e974c6a8-2775-11e6-8e5b-af388491908f]:
> Message: Lock obtain timed out: Lock@/tmp/lucene-27b0592844527a5c6b717
> 9d062354965-write.lock
> 119073 at
> org.apache.oodt.cas.workflow.instrepo.LuceneWorkflowInstanceRepository.addWorkflowInstanceToCatalog(LuceneWorkflowInstanceRepository.java:464)
> 119074 at
> org.apache.oodt.cas.workflow.instrepo.LuceneWorkflowInstanceRepository.addWorkflowInstance(LuceneWorkflowInstanceRepository.java:177)
> 119075 at
> org.apache.oodt.cas.workflow.engine.ThreadPoolWorkflowEngine.persistWorkflowInstance(ThreadPoolWorkflowEngine.java:444)
> 119076 at
> org.apache.oodt.cas.workflow.engine.ThreadPoolWorkflowEngine.startWorkflow(ThreadPoolWorkflowEngine.java:203)
> 119077 at
> org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager.handleEvent(XmlRpcWorkflowManager.java:311)
> 119078 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 119079 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 119080 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 119081 at java.lang.reflect.Method.invoke(Method.java:606)
> 119082 at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
> 119083 at
> org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
> 119084 at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
> 119085 at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
> 119086 at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
> 119087 at
> org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
> 119088 at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
>
>
RE: Unable to index workflow instance error
Posted by "Mallder, Valerie" <Va...@jhuapl.edu>.
Awesome! Thank you!
-----Original Message-----
From: Mattmann, Chris A (3980) [mailto:chris.a.mattmann@jpl.nasa.gov]
Sent: Tuesday, May 31, 2016 5:38 PM
To: dev@oodt.apache.org
Subject: Re: Unable to index workflow instance error
Yep has happened to me before. Race condition due to the old Lucene lib version (we really need to upgrade this) :/
Fix is:
1. stop workflow mgr
2. rm -rf /tmp/lucene-27b0592844527a5c6b7179d062354965-write.lock
3. restart workflow mgr
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Director, Information Retrieval and Data Science Group (IRDS) Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA
WWW: http://irds.usc.edu/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 5/31/16, 2:27 PM, "Mallder, Valerie" <Va...@jhuapl.edu> wrote:
>Hi All,
>
>I am getting a new error today. This is has never happened before, and
>nothing has changed except I think the machine was rebooted and oodt
>was not shut down properly. Has anyone seen this before? (Note, the
>left-most numbers on each line are just the line numbers to my oodt.out
>file.)
>
>Thanks,
>Val
>
>
>119069 INFO: WorkflowManager: Workflow processLevel0FilesWorkflow
>retrieved for event processLevel0Files
>119070 May 31, 2016 4:23:27 PM
>org.apache.oodt.cas.workflow.instrepo.LuceneWorkflowInstanceRepository
>addWorkflowInstanceToCatalog
>119071 WARNING: Unable to index workflow instance: [e974c6a8-2775-11e6-8e5b-af388491908f]: Message: Lock obtain timed out: Lock@/tmp/lucene-27b0592844527a5c6b7179d062354965-write.lock
>119072 org.apache.oodt.cas.workflow.structs.exceptions.InstanceRepositoryException: Unable to index workflow instance: [e974c6a8-2775-11e6-8e5b-af388491908f]: Message: Lock obtain timed out: Lock@/tmp/lucene-27b0592844527a5c6b717 9d062354965-write.lock
>119073 at org.apache.oodt.cas.workflow.instrepo.LuceneWorkflowInstanceRepository.addWorkflowInstanceToCatalog(LuceneWorkflowInstanceRepository.java:464)
>119074 at org.apache.oodt.cas.workflow.instrepo.LuceneWorkflowInstanceRepository.addWorkflowInstance(LuceneWorkflowInstanceRepository.java:177)
>119075 at org.apache.oodt.cas.workflow.engine.ThreadPoolWorkflowEngine.persistWorkflowInstance(ThreadPoolWorkflowEngine.java:444)
>119076 at org.apache.oodt.cas.workflow.engine.ThreadPoolWorkflowEngine.startWorkflow(ThreadPoolWorkflowEngine.java:203)
>119077 at org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager.handleEvent(XmlRpcWorkflowManager.java:311)
>119078 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>119079 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>119080 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>119081 at java.lang.reflect.Method.invoke(Method.java:606)
>119082 at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
>119083 at org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
>119084 at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
>119085 at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
>119086 at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
>119087 at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
>119088 at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
>
Re: Unable to index workflow instance error
Posted by "Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov>.
Yep has happened to me before. Race condition due to the old Lucene
lib version (we really need to upgrade this) :/
Fix is:
1. stop workflow mgr
2. rm -rf /tmp/lucene-27b0592844527a5c6b7179d062354965-write.lock
3. restart workflow mgr
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Director, Information Retrieval and Data Science Group (IRDS)
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
WWW: http://irds.usc.edu/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On 5/31/16, 2:27 PM, "Mallder, Valerie" <Va...@jhuapl.edu> wrote:
>Hi All,
>
>I am getting a new error today. This is has never happened before, and nothing has changed except I think the machine was rebooted and oodt was not shut down properly. Has anyone seen this before? (Note, the left-most numbers on each line are just the line numbers to my oodt.out file.)
>
>Thanks,
>Val
>
>
>119069 INFO: WorkflowManager: Workflow processLevel0FilesWorkflow retrieved for event processLevel0Files
>119070 May 31, 2016 4:23:27 PM org.apache.oodt.cas.workflow.instrepo.LuceneWorkflowInstanceRepository addWorkflowInstanceToCatalog
>119071 WARNING: Unable to index workflow instance: [e974c6a8-2775-11e6-8e5b-af388491908f]: Message: Lock obtain timed out: Lock@/tmp/lucene-27b0592844527a5c6b7179d062354965-write.lock
>119072 org.apache.oodt.cas.workflow.structs.exceptions.InstanceRepositoryException: Unable to index workflow instance: [e974c6a8-2775-11e6-8e5b-af388491908f]: Message: Lock obtain timed out: Lock@/tmp/lucene-27b0592844527a5c6b717 9d062354965-write.lock
>119073 at org.apache.oodt.cas.workflow.instrepo.LuceneWorkflowInstanceRepository.addWorkflowInstanceToCatalog(LuceneWorkflowInstanceRepository.java:464)
>119074 at org.apache.oodt.cas.workflow.instrepo.LuceneWorkflowInstanceRepository.addWorkflowInstance(LuceneWorkflowInstanceRepository.java:177)
>119075 at org.apache.oodt.cas.workflow.engine.ThreadPoolWorkflowEngine.persistWorkflowInstance(ThreadPoolWorkflowEngine.java:444)
>119076 at org.apache.oodt.cas.workflow.engine.ThreadPoolWorkflowEngine.startWorkflow(ThreadPoolWorkflowEngine.java:203)
>119077 at org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager.handleEvent(XmlRpcWorkflowManager.java:311)
>119078 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>119079 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>119080 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>119081 at java.lang.reflect.Method.invoke(Method.java:606)
>119082 at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
>119083 at org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
>119084 at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
>119085 at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
>119086 at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
>119087 at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
>119088 at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
>