You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Andy Seaborne <an...@apache.org> on 2015/10/05 14:16:14 UTC
Re: Fuseki 2+- Lucene doesn't play well with Tomcat Manager

Hi Todd,

Sorry for the delay.  Minor tidal wave of email.

On 25/09/15 19:41, Todd Detwiler wrote:
> Thanks again Andy. I upgraded to Fuseki 2.3.0 / Jena 3.0.0, no
> difference in how it behaves with Tomcat start/stop service.

Useful information.  I created JENA-1043 to record this overall.

There is now some code in the development (post 2.3.0) version that 
resets the internal cache of TDB datasets.

That is only a partial fix - the text datasets / Lucene integration is 
not going to see that and I don't know if it will work or not.

It would be helpful for someone to try out TDB in Fuseki in Tomcat and 
see if the stop/start via the Tomcat manager is better in the current 
development build.


> You asked
> what I'd expect to happen with active transactions. If they are
> transactions, I'd expect them to complete or be undone. Really, I just
> want the same behavior that occurs with starting and stopping the
> complete running Tomcat instance (which works, but it forces me to bring
> down other services).

undoing them (aborting) is easy to provide :-)

TDB is safe to just blow the process away - the TDB transactions will 
sort it out.  (To show I believe that - I stop Fuseki with "kill" 
including "kill -9")

>
> By large files (with regard to SPARQL upload vs. bulkloader) I mean
> large for an ontology, not large for data. I'm talking about 2 files,
> each with about 2.x million triples.
>
> So, I'm now trying to just use sparql update, but I can't get it to
> work. I don't get any error, it just hangs indefinitely on a command like:
>
> s-update --service http://localhost:8080/fuseki/sig/update 'CLEAR DEFAULT'

How long did you wait?

The only thing I can think of is that there is another write transaction 
somewhere.

>
> I'm thinking that it may be in read-only mode somehow? I've seen docs
> mentioning passing the argument --update to the fuseki server. But I'm
> not sure how this works if launched via Tomcat. Any suggestions that you
> might have here would be much appreciated.

--update is just a way when there isn't a full configuration -- it 
causes the default update services to be part of the configuration built 
from the command line.  A configuration file is a better way to take 
detailed control in Tomcat.

	Andy

>
> Thanks,
> Todd
>
> Landon Todd Detwiler
> Structural Informatics Group (SIG)
> University of Washington
>
> phone: 206-351-7721
>
> On 9/16/15 12:44 PM, Andy Seaborne wrote:
>> On 15/09/15 21:00, Todd Detwiler wrote:
>>> Hi Andy,
>>> In this case I am running the version of Jena that came with a recent
>>> (within the last couple of months) Fuseki2 war file distribution. The
>>> core libs look to be 2.13.*.
>>
>> Fuseki 2.3.0 / Jena 3.0.0 was July.
>>
>>>
>>> I can't say what Tomcat does under the hood when you start and stop
>>> apps, but the behavior I saw wasn't just through the text-based http
>>> tools, the same behavior occurs via the manager gui. I know that there
>>> are start-up and shutdown listener hooks in Tomcat. So I could insert
>>> some clean-up code there if I knew what needed to happen.
>>
>> I don't know about the text index.  For TDB, something needs to
>> release the database (StoreConnection.reset() - caution - what about
>> active transactions?)
>>
>>> Yes, I could do as you say and limit the update endpoint to localhost.
>>> Will rebuilding the whole store (I want to drop everything in it,
>>> rebuild from two large files, and regenerate the Lucene indexes) be as
>>> efficient as using the bulk loader? Is that your recommended method?
>>
>> It depends on "large".  The TDB bulkloader is faster than just loading
>> via the usual API.  Maybe the same for the text index.
>>
>>     Andy
>>
>>>
>>> Your last sentence said,
>>>
>>>> SPARQL Update (in the latest version where text changes and database
>>>> changes are coordinated) is
>>>
>>> What did you intend to say after "is"?
>>>
>>> Thanks,
>>> Todd
>>>
>>> Landon Todd Detwiler
>>> Structural Informatics Group (SIG)
>>> University of Washington
>>>
>>> phone: 206-351-7721
>>>
>>> On 9/15/15 12:46 PM, Andy Seaborne wrote:
>>>> On 15/09/15 17:39, Todd Detwiler wrote:
>>>>> Let me start with what I'm trying to do. Basically, I'm writing a
>>>>> script, that will run off of a cron job, that will rebuild my TDB
>>>>> store
>>>>> (and Lucene indexes) on a regular basis. I looked into using the TDB
>>>>> command line scripts, but they fail because another JVM (the tomcat
>>>>> app)
>>>>> is already using the TDB store. I looked into using Sparql update
>>>>> through Fuseki, but I like the idea of the store being read only (and
>>>>> therefore update is not allowed). So, I am experimenting with
>>>>> having my
>>>>> script interact with the Tomcat Manager, stopping the running
>>>>> Fuseki app
>>>>> (but not all of tomcat), making changes, and then restarting. I am
>>>>> able
>>>>> to stop it fine, using a url like:
>>>>>
>>>>> (domain):(port)/manager/text/stop?path=/fuseki
>>>>>
>>>>> However, the corresponding call to start fails:
>>>>>
>>>>> (domain):(port)/manager/text/start?path=/fuseki
>>>>>
>>>>> I am getting a stack trace in the tomcat log file that indicates an
>>>>> issue with file locks related to Lucene indexing. Now, if I stop
>>>>> tomcat
>>>>> entirely, and start it again, everything works fine. But I don't
>>>>> want to
>>>>> shut down the other services. Do I need to add a listener that
>>>>> performs
>>>>> some sort of operation when Tomcat stops? Have others encountered the
>>>>> same issue. I've included the stack trace at the bottom of this
>>>>> message.
>>>>> I'd appreciate any insight anyone could offer. Or let me know if you
>>>>> think using Sparql update would be a preferred way of rebuilding the
>>>>> store?
>>>>>
>>>>> Thanks,
>>>>> Todd
>>>>
>>>> Hi Todd,
>>>>
>>>> Firstly, which version of jena are you using? There were some changes
>>>> to the Lucene indexing around updates that changed the way it works
>>>> which may play into this.  I see "com.hp.hpl.jena" so it's not the
>>>> latest.
>>>>
>>>> Also, I don't know what stopping a webapp in Tomcat actually does. The
>>>> standard ServletContextListener only has an operation called
>>>> synchronously so it is tricky to clear up in-flight transactions (TDB
>>>> does not have forced system aborts by design). Fuseki2, and maybe
>>>> wider, needs a proper service lifecycle.
>>>>
>>>> Restarting is clearly calling assembler processing.
>>>>
>>>> Could you protect the update endpoints with the tomcat configuration
>>>> to, say, only respond to requests for localhost? While forcing
>>>> listening to a specific host isn't very safe, localhost is different
>>>> because it means the request was on the loop back interface so can't
>>>> be done from outside.
>>>>
>>>> SPARQL Update (in the latest version where text changes and database
>>>> changes are coordinated) is
>>>>
>>>>     Andy
>>>>
>>>>>
>>>>> 15-Sep-2015 09:02:55.968 SEVERE [http-nio-8080-exec-10]
>>>>> org.apache.catalina.core.StandardContext.listenerStart Exception
>>>>> sending
>>>>> context initialized event to listener instance of class
>>>>> org.apache.jena.fuseki.server.FusekiServerListener
>>>>>   com.hp.hpl.jena.assembler.exceptions.AssemblerException: caught:
>>>>> null
>>>>>    doing:
>>>>>      root: http://localhost/jena_example/#indexLucene with type:
>>>>> http://jena.apache.org/text#TextIndexLucene assembler class: class
>>>>> org.apache.jena.query.text.assembler.TextIndexLuceneAssembler
>>>>>      root: http://localhost/jena_example/#text_dataset with type:
>>>>> http://jena.apache.org/text#TextDataset assembler class: class
>>>>> org.apache.jena.query.text.assembler.TextDatasetAssembler
>>>>>
>>>>>      at
>>>>> com.hp.hpl.jena.assembler.assemblers.AssemblerGroup$PlainAssemblerGroup.openBySpecificType(AssemblerGroup.java:138)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> com.hp.hpl.jena.assembler.assemblers.AssemblerGroup$PlainAssemblerGroup.open(AssemblerGroup.java:117)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> com.hp.hpl.jena.assembler.assemblers.AssemblerGroup$ExpandingAssemblerGroup.open(AssemblerGroup.java:81)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> com.hp.hpl.jena.assembler.assemblers.AssemblerBase.open(AssemblerBase.java:52)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> com.hp.hpl.jena.assembler.assemblers.AssemblerBase.open(AssemblerBase.java:48)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.jena.query.text.assembler.TextDatasetAssembler.open(TextDatasetAssembler.java:63)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.jena.query.text.assembler.TextDatasetAssembler.open(TextDatasetAssembler.java:41)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> com.hp.hpl.jena.assembler.assemblers.AssemblerGroup$PlainAssemblerGroup.openBySpecificType(AssemblerGroup.java:130)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> com.hp.hpl.jena.assembler.assemblers.AssemblerGroup$PlainAssemblerGroup.open(AssemblerGroup.java:117)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> com.hp.hpl.jena.assembler.assemblers.AssemblerGroup$ExpandingAssemblerGroup.open(AssemblerGroup.java:81)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> com.hp.hpl.jena.assembler.assemblers.AssemblerBase.open(AssemblerBase.java:52)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> com.hp.hpl.jena.assembler.assemblers.AssemblerBase.open(AssemblerBase.java:48)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.jena.fuseki.build.Builder.buildDataService(Builder.java:76)
>>>>>      at
>>>>> org.apache.jena.fuseki.build.Builder.buildDataAccessPoint(Builder.java:61)
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.jena.fuseki.build.FusekiConfig.servicesAndDatasets(FusekiConfig.java:133)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.jena.fuseki.build.FusekiConfig.readConfigFile(FusekiConfig.java:75)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.jena.fuseki.server.FusekiServer.initServerConfiguration(FusekiServer.java:248)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.jena.fuseki.server.FusekiServer.initializeDataAccessPoints(FusekiServer.java:216)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.jena.fuseki.server.FusekiServerListener.init(FusekiServerListener.java:70)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.jena.fuseki.server.FusekiServerListener.contextInitialized(FusekiServerListener.java:45)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4729)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5167)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
>>>>>      at
>>>>> org.apache.catalina.manager.ManagerServlet.start(ManagerServlet.java:1293)
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.catalina.manager.HTMLManagerServlet.start(HTMLManagerServlet.java:673)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.catalina.manager.HTMLManagerServlet.doPost(HTMLManagerServlet.java:221)
>>>>>
>>>>>
>>>>>
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>>>>>      at
>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.catalina.filters.CsrfPreventionFilter.doFilter(CsrfPreventionFilter.java:213)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
>>>>>      at
>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:108)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:614)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:617)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518)
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:668)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1521)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1478)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>>>>>
>>>>>
>>>>>
>>>>>      at java.lang.Thread.run(Thread.java:745)
>>>>> Caused by: java.nio.channels.OverlappingFileLockException
>>>>>      at
>>>>> sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
>>>>>      at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
>>>>>      at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1075)
>>>>>      at java.nio.channels.FileChannel.tryLock(FileChannel.java:1155)
>>>>>      at
>>>>> org.apache.lucene.store.NativeFSLock.obtain(NativeFSLockFactory.java:217)
>>>>>
>>>>>
>>>>>      at org.apache.lucene.store.Lock.obtain(Lock.java:72)
>>>>>      at
>>>>> org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:702)
>>>>>      at
>>>>> org.apache.jena.query.text.TextIndexLucene.openIndexWriter(TextIndexLucene.java:121)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.jena.query.text.TextIndexLucene.<init>(TextIndexLucene.java:114)
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.jena.query.text.TextDatasetFactory.createLuceneIndex(TextDatasetFactory.java:97)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.jena.query.text.assembler.TextIndexLuceneAssembler.open(TextIndexLuceneAssembler.java:96)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> org.apache.jena.query.text.assembler.TextIndexLuceneAssembler.open(TextIndexLuceneAssembler.java:47)
>>>>>
>>>>>
>>>>>
>>>>>      at
>>>>> com.hp.hpl.jena.assembler.assemblers.AssemblerGroup$PlainAssemblerGroup.openBySpecificType(AssemblerGroup.java:130)
>>>>>
>>>>>
>>>>>
>>>>>      ... 54 more
>>>>>
>>>>
>>>
>>
>