You are viewing a plain text version of this content. The canonical link for it is here.
Posted to slide-dev@jakarta.apache.org by Eirikur Hrafnsson <ei...@idega.is> on 2006/07/26 02:28:41 UTC
Better Slide performance on a database, PUT hangs and Lucene creating out of memory troubles
Hey guys and gals.
First I just wanted to share my experience with Slide for the last week.
One of our apps started to hang on every PUT into the Slide
repository that is running a custom JDBC store on top of Oracle.
I really though the problem was the sequential mode and the isolation
level in the database but then I discovered that the PROPERTIES table
for Slide in the Oracle DB
suddenly had 850000 records in it which seems like ALOT considering
the repository only contains about 200 files at most (but a lot of
user folders though).
The effect of so many records was that the upload (OR rather the
preparation for the upload) that Slide does (selecting data etc.)
would take forever!
And effectively locked some tables while it was at it. So I tried to
figure out some indexes to speed things up and these seem to work
very well and seem to be necessary actually!
Has someone else figured out some tuning for a database driven Slide
store? Anything would help!
CREATE INDEX properties_idx1 ON properties(version_id,property_name);
CREATE INDEX binding_idx1 ON binding(name);
CREATE INDEX binding_idx2 ON binding(uri_id,child_uuri_id);
CREATE INDEX locks_idx1 ON locks(object_id,lock_id,subject_id,type_id);
CREATE INDEX object_idx1 ON object(class_name,uri_id);
CREATE INDEX parent_bind_idx1 ON parent_binding(name);
CREATE INDEX parent_bind_idx2 ON parent_binding(uri_id,parent_uuri_id);
CREATE INDEX permissions_idx1 ON permissions
(object_id,subject_id,action_id);
CREATE INDEX permissions_idx2 ON permissions(succession);
CREATE INDEX permissions_idx3 ON permissions(object_id);
CREATE INDEX uri_idx1 ON uri(uri_string);
CREATE INDEX uri_idx2 ON uri(uri_id,uri_string);
Secondly I have a problem with the RE-indexing of a Slide store. I
deleted the index and restarted tomcat and the RE-indexing starts.
Then the memory usage skyrockets, not by some tens of megabytes but
by hundreds of precious MB's at once and then gives me a OUT OF
MEMORY error and a failed transaction when it stops at 1500MB of RAM!
Has anyone had this problem? First of all, why does the Slide-Lucene
index use transactions? Its writing to files in the file system right?
And what's with the memory feast?
Best regards
Eiki
¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
Eirikur S. Hrafnsson
Chief Software Engineer
eiki@idega.is
¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
Idega hf. ı Engjavegi 6 ı 104 Reykjavík ı Iceland
Tel. +354 554 7557 ı Fax +354 885 7557
http://www.idega.com ı idega@idega.com
---------------------------------------------------------------------
To unsubscribe, e-mail: slide-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: slide-dev-help@jakarta.apache.org
Re: Better Slide performance on a database, PUT hangs and Lucene creating out of memory troubles
Posted by Eirikur Hrafnsson <ei...@idega.is>.
Hi
thanks for the tip. I wasn't using any "asynchron" property but
setting it to false changed nothing. I still get the out of memory.
The stacktrace goes like this:
26 Jul 2006 11:19:19 - org.apache.slide.transaction.SlideTransaction
- WARNING - Prepare failure: Resource manager
com.idega.slide.store.IWSlideStore@1386751 Error code XA_RBBASE in
Transaction 2 xid [B@158539f-[B@1da1a93 in thread TP-Processor1
javax.transaction.xa.XAException
at
org.apache.commons.transaction.util.xa.AbstractXAResource.prepare
(AbstractXAResource.java:122)
at com.idega.slide.store.AbstractSlideStore.prepare
(AbstractSlideStore.java:544)
at org.apache.slide.transaction.SlideTransaction.commit
(SlideTransaction.java:251)
at
org.apache.slide.transaction.SlideTransactionManager.commit
(SlideTransactionManager.java:186)
at org.apache.slide.common.NamespaceAccessTokenImpl.commit
(NamespaceAccessTokenImpl.java:390)
at
org.apache.slide.index.lucene.IndexInitializer.namespaceInitialized
(IndexInitializer.java:152)
at org.apache.slide.event.DomainEvent
$NamespaceInitialized.fireEvent(DomainEvent.java:74)
at org.apache.slide.event.EventDispatcher.fireEvent
(EventDispatcher.java:72)
at org.apache.slide.common.Domain.initNamespace(Domain.java:
929)
at org.apache.slide.common.Domain.init(Domain.java:462)
at org.apache.slide.common.Domain.init(Domain.java:375)
at org.apache.slide.common.Domain.init(Domain.java:338)
at org.apache.slide.webdav.WebdavServlet.init
(WebdavServlet.java:251)
at javax.servlet.GenericServlet.init(GenericServlet.java:211)
at org.apache.slide.webdav.WebdavServlet.init
(WebdavServlet.java:191)
at com.idega.servlet.ServletWrapper.init(ServletWrapper.java:
125)
at com.idega.slide.webdavservlet.WebdavExtendedServlet.init
(WebdavExtendedServlet.java:67)
at org.apache.catalina.core.StandardWrapper.loadServlet
(StandardWrapper.java:1091)
at org.apache.catalina.core.StandardWrapper.allocate
(StandardWrapper.java:750)
at org.apache.catalina.core.StandardWrapperValve.invoke
(StandardWrapperValve.java:130)
at org.apache.catalina.core.StandardContextValve.invoke
(StandardContextValve.java:178)
at org.apache.catalina.core.StandardHostValve.invoke
(StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke
(ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invoke
(StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service
(CoyoteAdapter.java:148)
at org.apache.jk.server.JkCoyoteHandler.invoke
(JkCoyoteHandler.java:307)
at org.apache.jk.common.HandlerRequest.invoke
(HandlerRequest.java:385)
at org.apache.jk.common.ChannelSocket.invoke
(ChannelSocket.java:748)
at org.apache.jk.common.ChannelSocket.processConnection
(ChannelSocket.java:678)
at org.apache.jk.common.SocketConnection.runIt
(ChannelSocket.java:871)
at org.apache.tomcat.util.threads.ThreadPool
$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:534)
26 Jul 2006 11:19:19 - org.apache.slide.common.Domain - ERROR - Error
while commiting initialized properties index for scope / - null
26 Jul 2006 11:19:19 - org.apache.slide.common.Domain - ERROR -
javax.transaction.RollbackException
javax.transaction.RollbackException
at org.apache.slide.transaction.SlideTransaction.commit
(SlideTransaction.java:328)
at
org.apache.slide.transaction.SlideTransactionManager.commit
(SlideTransactionManager.java:186)
at org.apache.slide.common.NamespaceAccessTokenImpl.commit
(NamespaceAccessTokenImpl.java:390)
at
org.apache.slide.index.lucene.IndexInitializer.namespaceInitialized
(IndexInitializer.java:152)
at org.apache.slide.event.DomainEvent
$NamespaceInitialized.fireEvent(DomainEvent.java:74)
at org.apache.slide.event.EventDispatcher.fireEvent
(EventDispatcher.java:72)
at org.apache.slide.common.Domain.initNamespace(Domain.java:
929)
at org.apache.slide.common.Domain.init(Domain.java:462)
at org.apache.slide.common.Domain.init(Domain.java:375)
at org.apache.slide.common.Domain.init(Domain.java:338)
at org.apache.slide.webdav.WebdavServlet.init
(WebdavServlet.java:251)
at javax.servlet.GenericServlet.init(GenericServlet.java:211)
at org.apache.slide.webdav.WebdavServlet.init
(WebdavServlet.java:191)
at com.idega.servlet.ServletWrapper.init(ServletWrapper.java:
125)
at com.idega.slide.webdavservlet.WebdavExtendedServlet.init
(WebdavExtendedServlet.java:67)
at org.apache.catalina.core.StandardWrapper.loadServlet
(StandardWrapper.java:1091)
at org.apache.catalina.core.StandardWrapper.allocate
(StandardWrapper.java:750)
at org.apache.catalina.core.StandardWrapperValve.invoke
(StandardWrapperValve.java:130)
at org.apache.catalina.core.StandardContextValve.invoke
(StandardContextValve.java:178)
at org.apache.catalina.core.StandardHostValve.invoke
(StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke
(ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invoke
(StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service
(CoyoteAdapter.java:148)
at org.apache.jk.server.JkCoyoteHandler.invoke
(JkCoyoteHandler.java:307)
at org.apache.jk.common.HandlerRequest.invoke
(HandlerRequest.java:385)
at org.apache.jk.common.ChannelSocket.invoke
(ChannelSocket.java:748)
at org.apache.jk.common.ChannelSocket.processConnection
(ChannelSocket.java:678)
at org.apache.jk.common.SocketConnection.runIt
(ChannelSocket.java:871)
at org.apache.tomcat.util.threads.ThreadPool
$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:534)
26 Jul 2006 11:19:19 - org.apache.slide.common.Domain - ERROR -
java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
Any further ideas?
-Eiki
On 26.7.2006, at 09:37, Honoré David N. wrote:
>
>
> Eirikur Hrafnsson wrote:
>>
>> Secondly I have a problem with the RE-indexing of a Slide store. I
>> deleted the index and restarted tomcat and the RE-indexing starts.
>> Then the memory usage skyrockets, not by some tens of megabytes
>> but by
>> hundreds of precious MB's at once and then gives me a OUT OF MEMORY
>> error and a failed transaction when it stops at 1500MB of RAM!
>>
>> Has anyone had this problem? First of all, why does the Slide-Lucene
>> index use transactions? Its writing to files in the file system
>> right?
>> And what's with the memory feast?
>>
> Try to desactivate the async mode. Because while indexing (and content
> extraction), Slide keep the binary content of document into memory.
> In synchronized mode slide wait the end of the indexing of one
> document
> before loading the content of another for indexing.
> In Asynchone mode, all "IndexJob" (which reference a
> NodeRevisionContent
> with the binary content in memory) are put in a queue at the same
> time.
> Loading the content of all documents into stream in memory at the same
> time cause your OutOfMemoryError.
>
>> Best regards
>> Eiki
>>
>> ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
>> ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
>>
>> Eirikur S. Hrafnsson
>> Chief Software Engineer
>>
>> eiki@idega.is
>>
>> ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
>> ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
>>
>> Idega hf. ı Engjavegi 6 ı 104 Reykjavík ı Iceland
>> Tel. +354 554 7557 ı Fax +354 885 7557
>>
>> http://www.idega.com ı idega@idega.com
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: slide-dev-unsubscribe@jakarta.apache.org
>> For additional commands, e-mail: slide-dev-help@jakarta.apache.org
>>
>>
> --
>
> ----------------------------------------------------------------------
> --
> Honoré David N.
> Projet DISSCO (http://www.meteo.be/DISSCO/)
>
> email: dho@oma.be
> tel: 02 37 30 530
> ----------------------------------------------------------------------
> --
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: slide-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: slide-dev-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: slide-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: slide-dev-help@jakarta.apache.org
Re: Better Slide performance on a database, PUT hangs and Lucene
creating out of memory troubles
Posted by "Honoré David N." <dh...@oma.be>.
Eirikur Hrafnsson wrote:
>
> Secondly I have a problem with the RE-indexing of a Slide store. I
> deleted the index and restarted tomcat and the RE-indexing starts.
> Then the memory usage skyrockets, not by some tens of megabytes but by
> hundreds of precious MB's at once and then gives me a OUT OF MEMORY
> error and a failed transaction when it stops at 1500MB of RAM!
>
> Has anyone had this problem? First of all, why does the Slide-Lucene
> index use transactions? Its writing to files in the file system right?
> And what's with the memory feast?
>
Try to desactivate the async mode. Because while indexing (and content
extraction), Slide keep the binary content of document into memory.
In synchronized mode slide wait the end of the indexing of one document
before loading the content of another for indexing.
In Asynchone mode, all "IndexJob" (which reference a NodeRevisionContent
with the binary content in memory) are put in a queue at the same time.
Loading the content of all documents into stream in memory at the same
time cause your OutOfMemoryError.
> Best regards
> Eiki
>
> ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
>
> Eirikur S. Hrafnsson
> Chief Software Engineer
>
> eiki@idega.is
>
> ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
>
> Idega hf. ı Engjavegi 6 ı 104 Reykjavík ı Iceland
> Tel. +354 554 7557 ı Fax +354 885 7557
>
> http://www.idega.com ı idega@idega.com
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: slide-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: slide-dev-help@jakarta.apache.org
>
>
--
------------------------------------------------------------------------
Honoré David N.
Projet DISSCO (http://www.meteo.be/DISSCO/)
email: dho@oma.be
tel: 02 37 30 530
------------------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: slide-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: slide-dev-help@jakarta.apache.org