You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jackrabbit.apache.org by Morrell Jacobs <mj...@maned.com> on 2012/12/17 18:04:28 UTC
Intermittent Slow Down
Hello all,
We're currently experience a problem where JackRabbit function will occasionally slow down; according to our QA, the slow down comes in waves: good performance for a while (hours) then bad for a while (minutes / hours). The slow down is on the order of minutes to perform create or modify an object (creation involves creating a 2-3 levels of nodes, and setting properties). Normally these operations take 100 - 300 ms, but during the slow down they can take as much 5 minutes.
The slow down does not seem to be connected to the load on the server: we've seen it perform well, when many people are connecting, then slow down with only one user. I suspect the slow down is related to indexing, but that's just a guess on my part - I'm looking for something that would happen at some interval.
Environment:
* JackRabbit is built into war running in Tomcat (was 7.0.25, recently upgraded to 7.0.34)
* Tomcat was running with default configs but recently up'd memory - -Xms128M -Xmx512M -XX:MaxPermSize=256M
* Repository XML is at end of email
* JackRabbit was originally configured to be clustered, but is now running as just a single instance.
Data:
* Our data is structured similar to a file system, except any object can be both a file and contain other objects (files); each object has various metadata (some in properties, some in child nodes), a file and child objects
* We don't use full text searching and don't want file (binary) contents indexed; the only way I was able to prevent the search index from examining the files (some are large 100+ M) was to create my own node types for files. My node types are identical nt:file and nt:content, except that they are named prd:file and prd:content; the unrecognized name seems to prevent indexing from touch them.
When I've attempted to use profiling tools, it appears that during the slow down the CPU is pegged; memory usage is also at or close to the limit. In depth profiling tools (NetBeans) get overwhelmed during the slow down.
We're going thru a variety of tests, but any advice the community can provide will be greatly appreciated.
Here's the repository.xml:
<Repository>
<FileSystem class="org.apache.jackrabbit.core.fs.db.DbFileSystem">
<param name="driver" value="com.mysql.jdbc.Driver"/>
<param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}global_repository?autoReconnect=true&createDatabaseIfNotExist=true"/>
<param name="schema" value="mysql"/>
<param name="schemaObjectPrefix" value="rep_"/>
<param name="user" value="${pm.database.user}"/>
<param name="password" value="${pm.database.pwd}"/>
</FileSystem>
<!--
security configuration
-->
<Security appName="Jackrabbit">
<!--
security manager:
class: FQN of class implementing the JackrabbitSecurityManager interface
-->
<SecurityManager class="org.apache.jackrabbit.core.DefaultSecurityManager" workspaceName="default">
<!--
workspace access:
class: FQN of class implementing the WorkspaceAccessManager interface
-->
<!-- <WorkspaceAccessManager class="..."/> -->
<!-- <param name="config" value="${rep.home}/security.xml"/> -->
</SecurityManager>
<!--
access manager:
class: FQN of class implementing the AccessManager interface
-->
<AccessManager class="org.apache.jackrabbit.core.security.DefaultAccessManager">
<!-- <param name="config" value="${rep.home}/access.xml"/> -->
</AccessManager>
<LoginModule class="org.apache.jackrabbit.core.security.authentication.DefaultLoginModule">
<!--
anonymous user name ('anonymous' is the default value)
-->
<param name="anonymousId" value="anonymous"/>
<!--
administrator user id (default value if param is missing is 'admin')
-->
<param name="adminId" value="admin"/>
</LoginModule>
</Security>
<!--
location of workspaces root directory and name of default workspace
-->
<Workspaces rootPath="${rep.home}/workspaces" defaultWorkspace="default"/>
<!--
workspace configuration template:
used to create the initial workspace if there's no workspace yet
-->
<!-- <Workspace name="default">-->
<Workspace name="default">
<!--
virtual file system of the workspace:
class: FQN of class implementing the FileSystem interface
-->
<FileSystem class="org.apache.jackrabbit.core.fs.local.LocalFileSystem">
<param name="path" value="${wsp.home}"/>
</FileSystem>
<!--
persistence manager of the workspace:
class: FQN of class implementing the PersistenceManager interface
-->
<PersistenceManager class="org.apache.jackrabbit.core.persistence.pool.MySqlPersistenceManager">
<param name="driver" value="com.mysql.jdbc.Driver"/>
<param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}product?createDatabaseIfNotExist=true"/>
<param name="user" value="${pm.database.user}"/>
<param name="password" value="${pm.database.pwd}"/>
<param name="schemaObjectPrefix" value="Product_"/>
<param name="schema" value="mysql"/>
</PersistenceManager>
<!--
Search index and the file system it uses.
class: FQN of class implementing the QueryHandler interface
-->
<SearchIndex class="org.apache.jackrabbit.core.query.lucene.SearchIndex">
<param name="path" value="${wsp.home}/index"/>
<!--<param name="textFilterClasses"
value="org.apache.jackrabbit.extractor.PlainTextExtractor,org.apache.jackrabbit.extractor.MsWordTextExtractor,org.apache.jackrabbit.extractor.MsExcelTextExtractor,org.apache.jackrabbit.extractor.MsPowerPointTextExtractor,org.apache.jackrabbit.extractor.PdfTextExtractor,org.apache.jackrabbit.extractor.OpenOfficeTextExtractor,org.apache.jackrabbit.extractor.RTFTextExtractor,org.apache.jackrabbit.extractor.HTMLTextExtractor,org.apache.jackrabbit.extractor.XMLTextExtractor"/>
<param name="extractorPoolSize" value="2"/>
<param name="supportHighlighting" value="true"/> -->
</SearchIndex>
</Workspace>
<!--
Configures the versioning
-->
<Versioning rootPath="${rep.home}/version">
<!--
Configures the filesystem to use for versioning for the respective
persistence manager
-->
<FileSystem class="org.apache.jackrabbit.core.fs.local.LocalFileSystem">
<param name="path" value="${rep.home}/version"/>
</FileSystem>
<!--
Configures the persistence manager to be used for persisting version state.
Please note that the current versioning implementation is based on
a 'normal' persistence manager, but this could change in future
implementations.
-->
<PersistenceManager class="org.apache.jackrabbit.core.persistence.pool.MySqlPersistenceManager">
<param name="driver" value="com.mysql.jdbc.Driver"/>
<param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}truedit_versions?createDatabaseIfNotExist=true"/>
<param name="user" value="${pm.database.user}"/>
<param name="password" value="${pm.database.pwd}"/>
<param name="schemaObjectPrefix" value="version_"/>
<param name="schema" value="mysql"/>
</PersistenceManager>
</Versioning>
<!--
Configures the Data Store for large binary objects.
-->
<DataStore class="org.apache.jackrabbit.core.data.FileDataStore">
<param name="path" value="${datastore.location.path}"/>
<param name="minRecordLength" value="100"/>
</DataStore>
<!--
<Cluster id="${cluster.id}" syncDelay="2000">
<Journal class="org.apache.jackrabbit.core.journal.DatabaseJournal">
<param name="driver" value="com.mysql.jdbc.Driver" />
<param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}journal?autoReconnect=true&createDatabaseIfNotExist=true"/>
<param name="schemaObjectPrefix" value="journal_"/>
<param name="databaseType" value="mysql"/>
<param name="user" value="${pm.database.user}"/>
<param name="password" value="${pm.database.pwd}"/>
</Journal>
</Cluster>
-->
</Repository>
--
Morrell Jacobs
Chief Software Architect
MEI
610 Old York Road, Suite 250
Jenkintown, PA 19046
Phone: 215-886-5662, ext. 252
Fax: 215-886-5681
http://www.maned.com
E-mail: mjacobs@maned.com
AOL IM: MorrellMEI
Have you seen Nervous Pixel, MEI's creative services division?
www.nervouspixel.com
Re: Intermittent Slow Down
Posted by Ron Wheeler <rw...@artifact-software.com>.
On 18/12/2012 12:32 PM, Morrell Jacobs wrote:
> Ron, I think your point of pushing the virtual memory handling to the OS layer is very sage.
>
> As for the current memory results, the changes are to the JVM heap size. Using tools to review the JVM heap usage, it's generally pretty low (usually less than 300 Mb, occasional spikes to around 400 Mb), but we are not putting much load on the server during the current test.
>
> The GC tuning article posted by Ian is great; the notes about default GC explain much of the behavior we've been seeing. I still need to review the effects of the -server, to see if any additional changes would make sense.
>
> Once I have a good configuration, I'm going to do some tests to see the impact of adjusting memory allocations & processing power, to get some insight as to what is appropriate and how changes impact server response.
Good plan.
It probably will be interesting to know the sensitivity to VM parameters
and real memory availability.
Post -"everything is running fine", it is sometimes hard to get these
kinds of tests to the top of the queue.
Ron
>
> On Dec 18, 2012, at 9:17 AM, Ron Wheeler wrote:
>
>> The memory is easy to play with if you want to lower it to see where the bottom lies.
>> I find it hard to believe that your memory footprint is that low given our experience with Tomcat and the size of the blobs that you are throwing around in MySQL unless you are talking about real memory not virtual memory in the JVM.
>> We had to set the JVM to 1 GB with very small records and fairly simple transactions just get our app to perform as we expected it, so I am not shocked or surprised that 2Gb is reasonable for your application.
>>
>> Since JVM memory is virtual at the OS level, it is hard to see much advantage in going under 2GB.
>>
>> I think that there is an important distinction between real and JVM memory.
>> You can probably make a virtual machine thrash while lots of real memory is free.
>> The Java garbage collection lives inside the JVM container and has to manage the VM defined at the JVM level even if the OS has lots of real memory available.
>>
>> Ron
>>
>> On 18/12/2012 8:44 AM, Morrell Jacobs wrote:
>>> The combo of extra memory and changes seems to have eliminated the problem or at least hidden / delayed it. Tomcat settings include -sever and I've allocated 2 GB of ram: after about 13 hours of running (with light load), the problem has not occurred.
>>>
>>> Memory usage is ranging between 150 MB and 175 MB. Forcing garbage collection to run dropped memory under 100 MB, which supports the theory that GC is / was at the heart of the problem.
>>>
>>> The assistance and advice is much appreciated. I follow up with additional details as I find them.
>>>
>>> Thanks
>>> Morrell
>>>
>>>
>>>
>>> On Dec 17, 2012, at 6:34 PM, Ian Boston wrote:
>>>
>>>> On 18 December 2012 09:57, Morrell Jacobs <mj...@maned.com> wrote:
>>>>> Thanks Ian, will try.
>>>>>
>>>>> I added the -server flag earlier today so I'll see if it helps. I've used the JMX tools (VisualVM) and running GC manually seems to drop back down to a base level (or close).
>>>>>
>>>>> When tomcat / web app is idle, the heap usage seems to be a saw tooth pattern, which isn't a surprise. The test takes a while to run so have to wait and see if -server helps.
>>>>>
>>>>> I'm using JVM 6.
>>>> I think -server will switch to the ParallelGC. If you still have
>>>> problems read the section on Ergonomics [1], and if the adjustments
>>>> there make no difference think about switching to the Concurrent GC,
>>>> however if you have no free cores for long periods the Concurrent GC
>>>> may not get enough time to free tenured space which will lead to a JVM
>>>> pause. [2]
>>>>
>>>> VisualVM should show you which pool is being exhausted and give you a
>>>> clue to the root cause of a GC related pause to the application
>>>> threads.
>>>>
>>>> (if you wanted to get forensic you could turn on GC logging leave it
>>>> for 24h and then graph the output in a spreadsheet.).
>>>>
>>>> HTH
>>>> Ian
>>>>
>>>>
>>>> 1 http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#par_gc.ergonomics
>>>>
>>>> 2 http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#cms.concurrent_mode_failure
>>>>
>>>>> Thanks!
>>>>>
>>>>>
>>>>> On Dec 17, 2012, at 5:16 PM, Ian Boston wrote:
>>>>>
>>>>>> Hi Morrell,
>>>>>> Sounds suspiciously like garbage collection activity. Rather than
>>>>>> attaching a profiler which may impact heap usage, try attaching a JMX
>>>>>> console which should be less impact (eg JConsole) and looking at the
>>>>>> graph of Garbage Collection activity during the slow down.
>>>>>>
>>>>>> If you see it taking up a significant amount of time and CPU, try
>>>>>> changing the type of garbage collector[1]. IIRC the default without
>>>>>> the -server flag is a serial GC. Increasing heap will lengthen the
>>>>>> time between these slowdowns but won't eliminate them.
>>>>>>
>>>>>> Also, you may be able to recreate the slowdown by forcing a full GC
>>>>>> operation from JConsole. Obviously if a full GC operation doesn't
>>>>>> leave the JVM with significantly more free heap, then you need to up
>>>>>> the limit to prevent the JVM spending all its time trying to free
>>>>>> more.
>>>>>>
>>>>>> You didn't say which JVM you are using, the link below is for 6.
>>>>>> Adjust appropriately.
>>>>>>
>>>>>> HTH
>>>>>> Ian
>>>>>>
>>>>>>
>>>>>> 1 http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#available_collectors
>>>>>>
>>>>>> On 18 December 2012 05:14, Morrell Jacobs <mj...@maned.com> wrote:
>>>>>>> We're running MySQL 5.5.19 for Linux
>>>>>>>
>>>>>>> We had the blobs in the database because of clustering. Since we've moved to single stack, I suppose we could pull them out - not a long term solution, but we could try and see if it helps.
>>>>>>>
>>>>>>> I'll have our MySQL experts review the settings, the restructuring may be accounting for some of the slow down. If we were seeing delays from MySQL, I wouldn't expect Tomcat's CPU usage to be pegged though - I would think all the threads would be idle, waiting for a response. Does that sound right to you?
>>>>>>>
>>>>>>> Thanks
>>>>>>> Morrell
>>>>>>>
>>>>>>>
>>>>>>> On Dec 17, 2012, at 12:48 PM, Ron Wheeler wrote:
>>>>>>>
>>>>>>>> I would play with the JVM memory configuration.
>>>>>>>> It still looks very small.
>>>>>>>> Try doubling it and see if it changes anything.
>>>>>>>> If not, you can look elsewhere and if it does, try making it larger until it stops improving things.
>>>>>>>>
>>>>>>>> What version of MySQL are you running?
>>>>>>>> Have you tuned your MySQL?
>>>>>>>> You might want to look at how MySQL is going to treat these objects and what configuration parameters are recommended when you have large blobs.
>>>>>>>> Have you thought about getting your binary content out of the database. Databases are not very good at 100Mb blobs.
>>>>>>>> Links to 100Mb blobs/files work much better.
>>>>>>>>
>>>>>>>> Is there a compelling reason to put the binary data inside a database?
>>>>>>>> Anytime MySQL has to restructure its indexes or physical storage, you are going to see a pause in the action.
>>>>>>>>
>>>>>>>> Ron
>>>>>>>>
>>>>>>>> On 17/12/2012 12:04 PM, Morrell Jacobs wrote:
>>>>>>>>> Hello all,
>>>>>>>>>
>>>>>>>>> We're currently experience a problem where JackRabbit function will occasionally slow down; according to our QA, the slow down comes in waves: good performance for a while (hours) then bad for a while (minutes / hours). The slow down is on the order of minutes to perform create or modify an object (creation involves creating a 2-3 levels of nodes, and setting properties). Normally these operations take 100 - 300 ms, but during the slow down they can take as much 5 minutes.
>>>>>>>>>
>>>>>>>>> The slow down does not seem to be connected to the load on the server: we've seen it perform well, when many people are connecting, then slow down with only one user. I suspect the slow down is related to indexing, but that's just a guess on my part - I'm looking for something that would happen at some interval.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Environment:
>>>>>>>>> * JackRabbit is built into war running in Tomcat (was 7.0.25, recently upgraded to 7.0.34)
>>>>>>>>> * Tomcat was running with default configs but recently up'd memory - -Xms128M -Xmx512M -XX:MaxPermSize=256M
>>>>>>>>> * Repository XML is at end of email
>>>>>>>>> * JackRabbit was originally configured to be clustered, but is now running as just a single instance.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Data:
>>>>>>>>> * Our data is structured similar to a file system, except any object can be both a file and contain other objects (files); each object has various metadata (some in properties, some in child nodes), a file and child objects
>>>>>>>>> * We don't use full text searching and don't want file (binary) contents indexed; the only way I was able to prevent the search index from examining the files (some are large 100+ M) was to create my own node types for files. My node types are identical nt:file and nt:content, except that they are named prd:file and prd:content; the unrecognized name seems to prevent indexing from touch them.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> When I've attempted to use profiling tools, it appears that during the slow down the CPU is pegged; memory usage is also at or close to the limit. In depth profiling tools (NetBeans) get overwhelmed during the slow down.
>>>>>>>>>
>>>>>>>>> We're going thru a variety of tests, but any advice the community can provide will be greatly appreciated.
>>>>>>>>>
>>>>>>>>> Here's the repository.xml:
>>>>>>>>> <Repository>
>>>>>>>>> <FileSystem class="org.apache.jackrabbit.core.fs.db.DbFileSystem">
>>>>>>>>> <param name="driver" value="com.mysql.jdbc.Driver"/>
>>>>>>>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}global_repository?autoReconnect=true&createDatabaseIfNotExist=true"/>
>>>>>>>>> <param name="schema" value="mysql"/>
>>>>>>>>> <param name="schemaObjectPrefix" value="rep_"/>
>>>>>>>>> <param name="user" value="${pm.database.user}"/>
>>>>>>>>> <param name="password" value="${pm.database.pwd}"/>
>>>>>>>>> </FileSystem>
>>>>>>>>>
>>>>>>>>> <!--
>>>>>>>>> security configuration
>>>>>>>>> -->
>>>>>>>>> <Security appName="Jackrabbit">
>>>>>>>>> <!--
>>>>>>>>> security manager:
>>>>>>>>> class: FQN of class implementing the JackrabbitSecurityManager interface
>>>>>>>>> -->
>>>>>>>>> <SecurityManager class="org.apache.jackrabbit.core.DefaultSecurityManager" workspaceName="default">
>>>>>>>>> <!--
>>>>>>>>> workspace access:
>>>>>>>>> class: FQN of class implementing the WorkspaceAccessManager interface
>>>>>>>>> -->
>>>>>>>>> <!-- <WorkspaceAccessManager class="..."/> -->
>>>>>>>>> <!-- <param name="config" value="${rep.home}/security.xml"/> -->
>>>>>>>>> </SecurityManager>
>>>>>>>>> <!--
>>>>>>>>> access manager:
>>>>>>>>> class: FQN of class implementing the AccessManager interface
>>>>>>>>> -->
>>>>>>>>> <AccessManager class="org.apache.jackrabbit.core.security.DefaultAccessManager">
>>>>>>>>> <!-- <param name="config" value="${rep.home}/access.xml"/> -->
>>>>>>>>> </AccessManager>
>>>>>>>>> <LoginModule class="org.apache.jackrabbit.core.security.authentication.DefaultLoginModule">
>>>>>>>>> <!--
>>>>>>>>> anonymous user name ('anonymous' is the default value)
>>>>>>>>> -->
>>>>>>>>> <param name="anonymousId" value="anonymous"/>
>>>>>>>>> <!--
>>>>>>>>> administrator user id (default value if param is missing is 'admin')
>>>>>>>>> -->
>>>>>>>>> <param name="adminId" value="admin"/>
>>>>>>>>> </LoginModule>
>>>>>>>>> </Security>
>>>>>>>>> <!--
>>>>>>>>> location of workspaces root directory and name of default workspace
>>>>>>>>> -->
>>>>>>>>> <Workspaces rootPath="${rep.home}/workspaces" defaultWorkspace="default"/>
>>>>>>>>> <!--
>>>>>>>>> workspace configuration template:
>>>>>>>>> used to create the initial workspace if there's no workspace yet
>>>>>>>>> -->
>>>>>>>>> <!-- <Workspace name="default">-->
>>>>>>>>> <Workspace name="default">
>>>>>>>>> <!--
>>>>>>>>> virtual file system of the workspace:
>>>>>>>>> class: FQN of class implementing the FileSystem interface
>>>>>>>>> -->
>>>>>>>>> <FileSystem class="org.apache.jackrabbit.core.fs.local.LocalFileSystem">
>>>>>>>>> <param name="path" value="${wsp.home}"/>
>>>>>>>>> </FileSystem>
>>>>>>>>> <!--
>>>>>>>>> persistence manager of the workspace:
>>>>>>>>> class: FQN of class implementing the PersistenceManager interface
>>>>>>>>> -->
>>>>>>>>> <PersistenceManager class="org.apache.jackrabbit.core.persistence.pool.MySqlPersistenceManager">
>>>>>>>>> <param name="driver" value="com.mysql.jdbc.Driver"/>
>>>>>>>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}product?createDatabaseIfNotExist=true"/>
>>>>>>>>> <param name="user" value="${pm.database.user}"/>
>>>>>>>>> <param name="password" value="${pm.database.pwd}"/>
>>>>>>>>> <param name="schemaObjectPrefix" value="Product_"/>
>>>>>>>>> <param name="schema" value="mysql"/>
>>>>>>>>> </PersistenceManager>
>>>>>>>>> <!--
>>>>>>>>> Search index and the file system it uses.
>>>>>>>>> class: FQN of class implementing the QueryHandler interface
>>>>>>>>> -->
>>>>>>>>> <SearchIndex class="org.apache.jackrabbit.core.query.lucene.SearchIndex">
>>>>>>>>> <param name="path" value="${wsp.home}/index"/>
>>>>>>>>> <!--<param name="textFilterClasses"
>>>>>>>>> value="org.apache.jackrabbit.extractor.PlainTextExtractor,org.apache.jackrabbit.extractor.MsWordTextExtractor,org.apache.jackrabbit.extractor.MsExcelTextExtractor,org.apache.jackrabbit.extractor.MsPowerPointTextExtractor,org.apache.jackrabbit.extractor.PdfTextExtractor,org.apache.jackrabbit.extractor.OpenOfficeTextExtractor,org.apache.jackrabbit.extractor.RTFTextExtractor,org.apache.jackrabbit.extractor.HTMLTextExtractor,org.apache.jackrabbit.extractor.XMLTextExtractor"/>
>>>>>>>>> <param name="extractorPoolSize" value="2"/>
>>>>>>>>> <param name="supportHighlighting" value="true"/> -->
>>>>>>>>> </SearchIndex>
>>>>>>>>> </Workspace>
>>>>>>>>> <!--
>>>>>>>>> Configures the versioning
>>>>>>>>> -->
>>>>>>>>> <Versioning rootPath="${rep.home}/version">
>>>>>>>>> <!--
>>>>>>>>> Configures the filesystem to use for versioning for the respective
>>>>>>>>> persistence manager
>>>>>>>>> -->
>>>>>>>>> <FileSystem class="org.apache.jackrabbit.core.fs.local.LocalFileSystem">
>>>>>>>>> <param name="path" value="${rep.home}/version"/>
>>>>>>>>> </FileSystem>
>>>>>>>>> <!--
>>>>>>>>> Configures the persistence manager to be used for persisting version state.
>>>>>>>>> Please note that the current versioning implementation is based on
>>>>>>>>> a 'normal' persistence manager, but this could change in future
>>>>>>>>> implementations.
>>>>>>>>> -->
>>>>>>>>> <PersistenceManager class="org.apache.jackrabbit.core.persistence.pool.MySqlPersistenceManager">
>>>>>>>>> <param name="driver" value="com.mysql.jdbc.Driver"/>
>>>>>>>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}truedit_versions?createDatabaseIfNotExist=true"/>
>>>>>>>>> <param name="user" value="${pm.database.user}"/>
>>>>>>>>> <param name="password" value="${pm.database.pwd}"/>
>>>>>>>>> <param name="schemaObjectPrefix" value="version_"/>
>>>>>>>>> <param name="schema" value="mysql"/>
>>>>>>>>> </PersistenceManager>
>>>>>>>>> </Versioning>
>>>>>>>>> <!--
>>>>>>>>> Configures the Data Store for large binary objects.
>>>>>>>>> -->
>>>>>>>>> <DataStore class="org.apache.jackrabbit.core.data.FileDataStore">
>>>>>>>>> <param name="path" value="${datastore.location.path}"/>
>>>>>>>>> <param name="minRecordLength" value="100"/>
>>>>>>>>> </DataStore>
>>>>>>>>> <!--
>>>>>>>>> <Cluster id="${cluster.id}" syncDelay="2000">
>>>>>>>>> <Journal class="org.apache.jackrabbit.core.journal.DatabaseJournal">
>>>>>>>>> <param name="driver" value="com.mysql.jdbc.Driver" />
>>>>>>>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}journal?autoReconnect=true&createDatabaseIfNotExist=true"/>
>>>>>>>>> <param name="schemaObjectPrefix" value="journal_"/>
>>>>>>>>> <param name="databaseType" value="mysql"/>
>>>>>>>>> <param name="user" value="${pm.database.user}"/>
>>>>>>>>> <param name="password" value="${pm.database.pwd}"/>
>>>>>>>>> </Journal>
>>>>>>>>> </Cluster>
>>>>>>>>> -->
>>>>>>>>>
>>>>>>>>> </Repository>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Morrell Jacobs
>>>>>>>>> Chief Software Architect
>>>>>>>>> MEI
>>>>>>>>> 610 Old York Road, Suite 250
>>>>>>>>> Jenkintown, PA 19046
>>>>>>>>> Phone: 215-886-5662, ext. 252
>>>>>>>>> Fax: 215-886-5681
>>>>>>>>> http://www.maned.com
>>>>>>>>> E-mail: mjacobs@maned.com
>>>>>>>>> AOL IM: MorrellMEI
>>>>>>>>>
>>>>>>>>> Have you seen Nervous Pixel, MEI's creative services division?
>>>>>>>>> www.nervouspixel.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> --
>>>>>>>> Ron Wheeler
>>>>>>>> President
>>>>>>>> Artifact Software Inc
>>>>>>>> email: rwheeler@artifact-software.com
>>>>>>>> skype: ronaldmwheeler
>>>>>>>> phone: 866-970-2435, ext 102
>>>>>>>>
>>
>> --
>> Ron Wheeler
>> President
>> Artifact Software Inc
>> email: rwheeler@artifact-software.com
>> skype: ronaldmwheeler
>> phone: 866-970-2435, ext 102
>>
>
--
Ron Wheeler
President
Artifact Software Inc
email: rwheeler@artifact-software.com
skype: ronaldmwheeler
phone: 866-970-2435, ext 102
Re: Intermittent Slow Down
Posted by Morrell Jacobs <mj...@maned.com>.
Ron, I think your point of pushing the virtual memory handling to the OS layer is very sage.
As for the current memory results, the changes are to the JVM heap size. Using tools to review the JVM heap usage, it's generally pretty low (usually less than 300 Mb, occasional spikes to around 400 Mb), but we are not putting much load on the server during the current test.
The GC tuning article posted by Ian is great; the notes about default GC explain much of the behavior we've been seeing. I still need to review the effects of the -server, to see if any additional changes would make sense.
Once I have a good configuration, I'm going to do some tests to see the impact of adjusting memory allocations & processing power, to get some insight as to what is appropriate and how changes impact server response.
On Dec 18, 2012, at 9:17 AM, Ron Wheeler wrote:
> The memory is easy to play with if you want to lower it to see where the bottom lies.
> I find it hard to believe that your memory footprint is that low given our experience with Tomcat and the size of the blobs that you are throwing around in MySQL unless you are talking about real memory not virtual memory in the JVM.
> We had to set the JVM to 1 GB with very small records and fairly simple transactions just get our app to perform as we expected it, so I am not shocked or surprised that 2Gb is reasonable for your application.
>
> Since JVM memory is virtual at the OS level, it is hard to see much advantage in going under 2GB.
>
> I think that there is an important distinction between real and JVM memory.
> You can probably make a virtual machine thrash while lots of real memory is free.
> The Java garbage collection lives inside the JVM container and has to manage the VM defined at the JVM level even if the OS has lots of real memory available.
>
> Ron
>
> On 18/12/2012 8:44 AM, Morrell Jacobs wrote:
>> The combo of extra memory and changes seems to have eliminated the problem or at least hidden / delayed it. Tomcat settings include -sever and I've allocated 2 GB of ram: after about 13 hours of running (with light load), the problem has not occurred.
>>
>> Memory usage is ranging between 150 MB and 175 MB. Forcing garbage collection to run dropped memory under 100 MB, which supports the theory that GC is / was at the heart of the problem.
>>
>> The assistance and advice is much appreciated. I follow up with additional details as I find them.
>>
>> Thanks
>> Morrell
>>
>>
>>
>> On Dec 17, 2012, at 6:34 PM, Ian Boston wrote:
>>
>>> On 18 December 2012 09:57, Morrell Jacobs <mj...@maned.com> wrote:
>>>> Thanks Ian, will try.
>>>>
>>>> I added the -server flag earlier today so I'll see if it helps. I've used the JMX tools (VisualVM) and running GC manually seems to drop back down to a base level (or close).
>>>>
>>>> When tomcat / web app is idle, the heap usage seems to be a saw tooth pattern, which isn't a surprise. The test takes a while to run so have to wait and see if -server helps.
>>>>
>>>> I'm using JVM 6.
>>> I think -server will switch to the ParallelGC. If you still have
>>> problems read the section on Ergonomics [1], and if the adjustments
>>> there make no difference think about switching to the Concurrent GC,
>>> however if you have no free cores for long periods the Concurrent GC
>>> may not get enough time to free tenured space which will lead to a JVM
>>> pause. [2]
>>>
>>> VisualVM should show you which pool is being exhausted and give you a
>>> clue to the root cause of a GC related pause to the application
>>> threads.
>>>
>>> (if you wanted to get forensic you could turn on GC logging leave it
>>> for 24h and then graph the output in a spreadsheet.).
>>>
>>> HTH
>>> Ian
>>>
>>>
>>> 1 http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#par_gc.ergonomics
>>>
>>> 2 http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#cms.concurrent_mode_failure
>>>
>>>> Thanks!
>>>>
>>>>
>>>> On Dec 17, 2012, at 5:16 PM, Ian Boston wrote:
>>>>
>>>>> Hi Morrell,
>>>>> Sounds suspiciously like garbage collection activity. Rather than
>>>>> attaching a profiler which may impact heap usage, try attaching a JMX
>>>>> console which should be less impact (eg JConsole) and looking at the
>>>>> graph of Garbage Collection activity during the slow down.
>>>>>
>>>>> If you see it taking up a significant amount of time and CPU, try
>>>>> changing the type of garbage collector[1]. IIRC the default without
>>>>> the -server flag is a serial GC. Increasing heap will lengthen the
>>>>> time between these slowdowns but won't eliminate them.
>>>>>
>>>>> Also, you may be able to recreate the slowdown by forcing a full GC
>>>>> operation from JConsole. Obviously if a full GC operation doesn't
>>>>> leave the JVM with significantly more free heap, then you need to up
>>>>> the limit to prevent the JVM spending all its time trying to free
>>>>> more.
>>>>>
>>>>> You didn't say which JVM you are using, the link below is for 6.
>>>>> Adjust appropriately.
>>>>>
>>>>> HTH
>>>>> Ian
>>>>>
>>>>>
>>>>> 1 http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#available_collectors
>>>>>
>>>>> On 18 December 2012 05:14, Morrell Jacobs <mj...@maned.com> wrote:
>>>>>> We're running MySQL 5.5.19 for Linux
>>>>>>
>>>>>> We had the blobs in the database because of clustering. Since we've moved to single stack, I suppose we could pull them out - not a long term solution, but we could try and see if it helps.
>>>>>>
>>>>>> I'll have our MySQL experts review the settings, the restructuring may be accounting for some of the slow down. If we were seeing delays from MySQL, I wouldn't expect Tomcat's CPU usage to be pegged though - I would think all the threads would be idle, waiting for a response. Does that sound right to you?
>>>>>>
>>>>>> Thanks
>>>>>> Morrell
>>>>>>
>>>>>>
>>>>>> On Dec 17, 2012, at 12:48 PM, Ron Wheeler wrote:
>>>>>>
>>>>>>> I would play with the JVM memory configuration.
>>>>>>> It still looks very small.
>>>>>>> Try doubling it and see if it changes anything.
>>>>>>> If not, you can look elsewhere and if it does, try making it larger until it stops improving things.
>>>>>>>
>>>>>>> What version of MySQL are you running?
>>>>>>> Have you tuned your MySQL?
>>>>>>> You might want to look at how MySQL is going to treat these objects and what configuration parameters are recommended when you have large blobs.
>>>>>>> Have you thought about getting your binary content out of the database. Databases are not very good at 100Mb blobs.
>>>>>>> Links to 100Mb blobs/files work much better.
>>>>>>>
>>>>>>> Is there a compelling reason to put the binary data inside a database?
>>>>>>> Anytime MySQL has to restructure its indexes or physical storage, you are going to see a pause in the action.
>>>>>>>
>>>>>>> Ron
>>>>>>>
>>>>>>> On 17/12/2012 12:04 PM, Morrell Jacobs wrote:
>>>>>>>> Hello all,
>>>>>>>>
>>>>>>>> We're currently experience a problem where JackRabbit function will occasionally slow down; according to our QA, the slow down comes in waves: good performance for a while (hours) then bad for a while (minutes / hours). The slow down is on the order of minutes to perform create or modify an object (creation involves creating a 2-3 levels of nodes, and setting properties). Normally these operations take 100 - 300 ms, but during the slow down they can take as much 5 minutes.
>>>>>>>>
>>>>>>>> The slow down does not seem to be connected to the load on the server: we've seen it perform well, when many people are connecting, then slow down with only one user. I suspect the slow down is related to indexing, but that's just a guess on my part - I'm looking for something that would happen at some interval.
>>>>>>>>
>>>>>>>>
>>>>>>>> Environment:
>>>>>>>> * JackRabbit is built into war running in Tomcat (was 7.0.25, recently upgraded to 7.0.34)
>>>>>>>> * Tomcat was running with default configs but recently up'd memory - -Xms128M -Xmx512M -XX:MaxPermSize=256M
>>>>>>>> * Repository XML is at end of email
>>>>>>>> * JackRabbit was originally configured to be clustered, but is now running as just a single instance.
>>>>>>>>
>>>>>>>>
>>>>>>>> Data:
>>>>>>>> * Our data is structured similar to a file system, except any object can be both a file and contain other objects (files); each object has various metadata (some in properties, some in child nodes), a file and child objects
>>>>>>>> * We don't use full text searching and don't want file (binary) contents indexed; the only way I was able to prevent the search index from examining the files (some are large 100+ M) was to create my own node types for files. My node types are identical nt:file and nt:content, except that they are named prd:file and prd:content; the unrecognized name seems to prevent indexing from touch them.
>>>>>>>>
>>>>>>>>
>>>>>>>> When I've attempted to use profiling tools, it appears that during the slow down the CPU is pegged; memory usage is also at or close to the limit. In depth profiling tools (NetBeans) get overwhelmed during the slow down.
>>>>>>>>
>>>>>>>> We're going thru a variety of tests, but any advice the community can provide will be greatly appreciated.
>>>>>>>>
>>>>>>>> Here's the repository.xml:
>>>>>>>> <Repository>
>>>>>>>> <FileSystem class="org.apache.jackrabbit.core.fs.db.DbFileSystem">
>>>>>>>> <param name="driver" value="com.mysql.jdbc.Driver"/>
>>>>>>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}global_repository?autoReconnect=true&createDatabaseIfNotExist=true"/>
>>>>>>>> <param name="schema" value="mysql"/>
>>>>>>>> <param name="schemaObjectPrefix" value="rep_"/>
>>>>>>>> <param name="user" value="${pm.database.user}"/>
>>>>>>>> <param name="password" value="${pm.database.pwd}"/>
>>>>>>>> </FileSystem>
>>>>>>>>
>>>>>>>> <!--
>>>>>>>> security configuration
>>>>>>>> -->
>>>>>>>> <Security appName="Jackrabbit">
>>>>>>>> <!--
>>>>>>>> security manager:
>>>>>>>> class: FQN of class implementing the JackrabbitSecurityManager interface
>>>>>>>> -->
>>>>>>>> <SecurityManager class="org.apache.jackrabbit.core.DefaultSecurityManager" workspaceName="default">
>>>>>>>> <!--
>>>>>>>> workspace access:
>>>>>>>> class: FQN of class implementing the WorkspaceAccessManager interface
>>>>>>>> -->
>>>>>>>> <!-- <WorkspaceAccessManager class="..."/> -->
>>>>>>>> <!-- <param name="config" value="${rep.home}/security.xml"/> -->
>>>>>>>> </SecurityManager>
>>>>>>>> <!--
>>>>>>>> access manager:
>>>>>>>> class: FQN of class implementing the AccessManager interface
>>>>>>>> -->
>>>>>>>> <AccessManager class="org.apache.jackrabbit.core.security.DefaultAccessManager">
>>>>>>>> <!-- <param name="config" value="${rep.home}/access.xml"/> -->
>>>>>>>> </AccessManager>
>>>>>>>> <LoginModule class="org.apache.jackrabbit.core.security.authentication.DefaultLoginModule">
>>>>>>>> <!--
>>>>>>>> anonymous user name ('anonymous' is the default value)
>>>>>>>> -->
>>>>>>>> <param name="anonymousId" value="anonymous"/>
>>>>>>>> <!--
>>>>>>>> administrator user id (default value if param is missing is 'admin')
>>>>>>>> -->
>>>>>>>> <param name="adminId" value="admin"/>
>>>>>>>> </LoginModule>
>>>>>>>> </Security>
>>>>>>>> <!--
>>>>>>>> location of workspaces root directory and name of default workspace
>>>>>>>> -->
>>>>>>>> <Workspaces rootPath="${rep.home}/workspaces" defaultWorkspace="default"/>
>>>>>>>> <!--
>>>>>>>> workspace configuration template:
>>>>>>>> used to create the initial workspace if there's no workspace yet
>>>>>>>> -->
>>>>>>>> <!-- <Workspace name="default">-->
>>>>>>>> <Workspace name="default">
>>>>>>>> <!--
>>>>>>>> virtual file system of the workspace:
>>>>>>>> class: FQN of class implementing the FileSystem interface
>>>>>>>> -->
>>>>>>>> <FileSystem class="org.apache.jackrabbit.core.fs.local.LocalFileSystem">
>>>>>>>> <param name="path" value="${wsp.home}"/>
>>>>>>>> </FileSystem>
>>>>>>>> <!--
>>>>>>>> persistence manager of the workspace:
>>>>>>>> class: FQN of class implementing the PersistenceManager interface
>>>>>>>> -->
>>>>>>>> <PersistenceManager class="org.apache.jackrabbit.core.persistence.pool.MySqlPersistenceManager">
>>>>>>>> <param name="driver" value="com.mysql.jdbc.Driver"/>
>>>>>>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}product?createDatabaseIfNotExist=true"/>
>>>>>>>> <param name="user" value="${pm.database.user}"/>
>>>>>>>> <param name="password" value="${pm.database.pwd}"/>
>>>>>>>> <param name="schemaObjectPrefix" value="Product_"/>
>>>>>>>> <param name="schema" value="mysql"/>
>>>>>>>> </PersistenceManager>
>>>>>>>> <!--
>>>>>>>> Search index and the file system it uses.
>>>>>>>> class: FQN of class implementing the QueryHandler interface
>>>>>>>> -->
>>>>>>>> <SearchIndex class="org.apache.jackrabbit.core.query.lucene.SearchIndex">
>>>>>>>> <param name="path" value="${wsp.home}/index"/>
>>>>>>>> <!--<param name="textFilterClasses"
>>>>>>>> value="org.apache.jackrabbit.extractor.PlainTextExtractor,org.apache.jackrabbit.extractor.MsWordTextExtractor,org.apache.jackrabbit.extractor.MsExcelTextExtractor,org.apache.jackrabbit.extractor.MsPowerPointTextExtractor,org.apache.jackrabbit.extractor.PdfTextExtractor,org.apache.jackrabbit.extractor.OpenOfficeTextExtractor,org.apache.jackrabbit.extractor.RTFTextExtractor,org.apache.jackrabbit.extractor.HTMLTextExtractor,org.apache.jackrabbit.extractor.XMLTextExtractor"/>
>>>>>>>> <param name="extractorPoolSize" value="2"/>
>>>>>>>> <param name="supportHighlighting" value="true"/> -->
>>>>>>>> </SearchIndex>
>>>>>>>> </Workspace>
>>>>>>>> <!--
>>>>>>>> Configures the versioning
>>>>>>>> -->
>>>>>>>> <Versioning rootPath="${rep.home}/version">
>>>>>>>> <!--
>>>>>>>> Configures the filesystem to use for versioning for the respective
>>>>>>>> persistence manager
>>>>>>>> -->
>>>>>>>> <FileSystem class="org.apache.jackrabbit.core.fs.local.LocalFileSystem">
>>>>>>>> <param name="path" value="${rep.home}/version"/>
>>>>>>>> </FileSystem>
>>>>>>>> <!--
>>>>>>>> Configures the persistence manager to be used for persisting version state.
>>>>>>>> Please note that the current versioning implementation is based on
>>>>>>>> a 'normal' persistence manager, but this could change in future
>>>>>>>> implementations.
>>>>>>>> -->
>>>>>>>> <PersistenceManager class="org.apache.jackrabbit.core.persistence.pool.MySqlPersistenceManager">
>>>>>>>> <param name="driver" value="com.mysql.jdbc.Driver"/>
>>>>>>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}truedit_versions?createDatabaseIfNotExist=true"/>
>>>>>>>> <param name="user" value="${pm.database.user}"/>
>>>>>>>> <param name="password" value="${pm.database.pwd}"/>
>>>>>>>> <param name="schemaObjectPrefix" value="version_"/>
>>>>>>>> <param name="schema" value="mysql"/>
>>>>>>>> </PersistenceManager>
>>>>>>>> </Versioning>
>>>>>>>> <!--
>>>>>>>> Configures the Data Store for large binary objects.
>>>>>>>> -->
>>>>>>>> <DataStore class="org.apache.jackrabbit.core.data.FileDataStore">
>>>>>>>> <param name="path" value="${datastore.location.path}"/>
>>>>>>>> <param name="minRecordLength" value="100"/>
>>>>>>>> </DataStore>
>>>>>>>> <!--
>>>>>>>> <Cluster id="${cluster.id}" syncDelay="2000">
>>>>>>>> <Journal class="org.apache.jackrabbit.core.journal.DatabaseJournal">
>>>>>>>> <param name="driver" value="com.mysql.jdbc.Driver" />
>>>>>>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}journal?autoReconnect=true&createDatabaseIfNotExist=true"/>
>>>>>>>> <param name="schemaObjectPrefix" value="journal_"/>
>>>>>>>> <param name="databaseType" value="mysql"/>
>>>>>>>> <param name="user" value="${pm.database.user}"/>
>>>>>>>> <param name="password" value="${pm.database.pwd}"/>
>>>>>>>> </Journal>
>>>>>>>> </Cluster>
>>>>>>>> -->
>>>>>>>>
>>>>>>>> </Repository>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Morrell Jacobs
>>>>>>>> Chief Software Architect
>>>>>>>> MEI
>>>>>>>> 610 Old York Road, Suite 250
>>>>>>>> Jenkintown, PA 19046
>>>>>>>> Phone: 215-886-5662, ext. 252
>>>>>>>> Fax: 215-886-5681
>>>>>>>> http://www.maned.com
>>>>>>>> E-mail: mjacobs@maned.com
>>>>>>>> AOL IM: MorrellMEI
>>>>>>>>
>>>>>>>> Have you seen Nervous Pixel, MEI's creative services division?
>>>>>>>> www.nervouspixel.com
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Ron Wheeler
>>>>>>> President
>>>>>>> Artifact Software Inc
>>>>>>> email: rwheeler@artifact-software.com
>>>>>>> skype: ronaldmwheeler
>>>>>>> phone: 866-970-2435, ext 102
>>>>>>>
>>
>
>
> --
> Ron Wheeler
> President
> Artifact Software Inc
> email: rwheeler@artifact-software.com
> skype: ronaldmwheeler
> phone: 866-970-2435, ext 102
>
Re: Intermittent Slow Down
Posted by Ron Wheeler <rw...@artifact-software.com>.
The memory is easy to play with if you want to lower it to see where the
bottom lies.
I find it hard to believe that your memory footprint is that low given
our experience with Tomcat and the size of the blobs that you are
throwing around in MySQL unless you are talking about real memory not
virtual memory in the JVM.
We had to set the JVM to 1 GB with very small records and fairly simple
transactions just get our app to perform as we expected it, so I am not
shocked or surprised that 2Gb is reasonable for your application.
Since JVM memory is virtual at the OS level, it is hard to see much
advantage in going under 2GB.
I think that there is an important distinction between real and JVM memory.
You can probably make a virtual machine thrash while lots of real memory
is free.
The Java garbage collection lives inside the JVM container and has to
manage the VM defined at the JVM level even if the OS has lots of real
memory available.
Ron
On 18/12/2012 8:44 AM, Morrell Jacobs wrote:
> The combo of extra memory and changes seems to have eliminated the problem or at least hidden / delayed it. Tomcat settings include -sever and I've allocated 2 GB of ram: after about 13 hours of running (with light load), the problem has not occurred.
>
> Memory usage is ranging between 150 MB and 175 MB. Forcing garbage collection to run dropped memory under 100 MB, which supports the theory that GC is / was at the heart of the problem.
>
> The assistance and advice is much appreciated. I follow up with additional details as I find them.
>
> Thanks
> Morrell
>
>
>
> On Dec 17, 2012, at 6:34 PM, Ian Boston wrote:
>
>> On 18 December 2012 09:57, Morrell Jacobs <mj...@maned.com> wrote:
>>> Thanks Ian, will try.
>>>
>>> I added the -server flag earlier today so I'll see if it helps. I've used the JMX tools (VisualVM) and running GC manually seems to drop back down to a base level (or close).
>>>
>>> When tomcat / web app is idle, the heap usage seems to be a saw tooth pattern, which isn't a surprise. The test takes a while to run so have to wait and see if -server helps.
>>>
>>> I'm using JVM 6.
>> I think -server will switch to the ParallelGC. If you still have
>> problems read the section on Ergonomics [1], and if the adjustments
>> there make no difference think about switching to the Concurrent GC,
>> however if you have no free cores for long periods the Concurrent GC
>> may not get enough time to free tenured space which will lead to a JVM
>> pause. [2]
>>
>> VisualVM should show you which pool is being exhausted and give you a
>> clue to the root cause of a GC related pause to the application
>> threads.
>>
>> (if you wanted to get forensic you could turn on GC logging leave it
>> for 24h and then graph the output in a spreadsheet.).
>>
>> HTH
>> Ian
>>
>>
>> 1 http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#par_gc.ergonomics
>>
>> 2 http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#cms.concurrent_mode_failure
>>
>>> Thanks!
>>>
>>>
>>> On Dec 17, 2012, at 5:16 PM, Ian Boston wrote:
>>>
>>>> Hi Morrell,
>>>> Sounds suspiciously like garbage collection activity. Rather than
>>>> attaching a profiler which may impact heap usage, try attaching a JMX
>>>> console which should be less impact (eg JConsole) and looking at the
>>>> graph of Garbage Collection activity during the slow down.
>>>>
>>>> If you see it taking up a significant amount of time and CPU, try
>>>> changing the type of garbage collector[1]. IIRC the default without
>>>> the -server flag is a serial GC. Increasing heap will lengthen the
>>>> time between these slowdowns but won't eliminate them.
>>>>
>>>> Also, you may be able to recreate the slowdown by forcing a full GC
>>>> operation from JConsole. Obviously if a full GC operation doesn't
>>>> leave the JVM with significantly more free heap, then you need to up
>>>> the limit to prevent the JVM spending all its time trying to free
>>>> more.
>>>>
>>>> You didn't say which JVM you are using, the link below is for 6.
>>>> Adjust appropriately.
>>>>
>>>> HTH
>>>> Ian
>>>>
>>>>
>>>> 1 http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#available_collectors
>>>>
>>>> On 18 December 2012 05:14, Morrell Jacobs <mj...@maned.com> wrote:
>>>>> We're running MySQL 5.5.19 for Linux
>>>>>
>>>>> We had the blobs in the database because of clustering. Since we've moved to single stack, I suppose we could pull them out - not a long term solution, but we could try and see if it helps.
>>>>>
>>>>> I'll have our MySQL experts review the settings, the restructuring may be accounting for some of the slow down. If we were seeing delays from MySQL, I wouldn't expect Tomcat's CPU usage to be pegged though - I would think all the threads would be idle, waiting for a response. Does that sound right to you?
>>>>>
>>>>> Thanks
>>>>> Morrell
>>>>>
>>>>>
>>>>> On Dec 17, 2012, at 12:48 PM, Ron Wheeler wrote:
>>>>>
>>>>>> I would play with the JVM memory configuration.
>>>>>> It still looks very small.
>>>>>> Try doubling it and see if it changes anything.
>>>>>> If not, you can look elsewhere and if it does, try making it larger until it stops improving things.
>>>>>>
>>>>>> What version of MySQL are you running?
>>>>>> Have you tuned your MySQL?
>>>>>> You might want to look at how MySQL is going to treat these objects and what configuration parameters are recommended when you have large blobs.
>>>>>> Have you thought about getting your binary content out of the database. Databases are not very good at 100Mb blobs.
>>>>>> Links to 100Mb blobs/files work much better.
>>>>>>
>>>>>> Is there a compelling reason to put the binary data inside a database?
>>>>>> Anytime MySQL has to restructure its indexes or physical storage, you are going to see a pause in the action.
>>>>>>
>>>>>> Ron
>>>>>>
>>>>>> On 17/12/2012 12:04 PM, Morrell Jacobs wrote:
>>>>>>> Hello all,
>>>>>>>
>>>>>>> We're currently experience a problem where JackRabbit function will occasionally slow down; according to our QA, the slow down comes in waves: good performance for a while (hours) then bad for a while (minutes / hours). The slow down is on the order of minutes to perform create or modify an object (creation involves creating a 2-3 levels of nodes, and setting properties). Normally these operations take 100 - 300 ms, but during the slow down they can take as much 5 minutes.
>>>>>>>
>>>>>>> The slow down does not seem to be connected to the load on the server: we've seen it perform well, when many people are connecting, then slow down with only one user. I suspect the slow down is related to indexing, but that's just a guess on my part - I'm looking for something that would happen at some interval.
>>>>>>>
>>>>>>>
>>>>>>> Environment:
>>>>>>> * JackRabbit is built into war running in Tomcat (was 7.0.25, recently upgraded to 7.0.34)
>>>>>>> * Tomcat was running with default configs but recently up'd memory - -Xms128M -Xmx512M -XX:MaxPermSize=256M
>>>>>>> * Repository XML is at end of email
>>>>>>> * JackRabbit was originally configured to be clustered, but is now running as just a single instance.
>>>>>>>
>>>>>>>
>>>>>>> Data:
>>>>>>> * Our data is structured similar to a file system, except any object can be both a file and contain other objects (files); each object has various metadata (some in properties, some in child nodes), a file and child objects
>>>>>>> * We don't use full text searching and don't want file (binary) contents indexed; the only way I was able to prevent the search index from examining the files (some are large 100+ M) was to create my own node types for files. My node types are identical nt:file and nt:content, except that they are named prd:file and prd:content; the unrecognized name seems to prevent indexing from touch them.
>>>>>>>
>>>>>>>
>>>>>>> When I've attempted to use profiling tools, it appears that during the slow down the CPU is pegged; memory usage is also at or close to the limit. In depth profiling tools (NetBeans) get overwhelmed during the slow down.
>>>>>>>
>>>>>>> We're going thru a variety of tests, but any advice the community can provide will be greatly appreciated.
>>>>>>>
>>>>>>> Here's the repository.xml:
>>>>>>> <Repository>
>>>>>>> <FileSystem class="org.apache.jackrabbit.core.fs.db.DbFileSystem">
>>>>>>> <param name="driver" value="com.mysql.jdbc.Driver"/>
>>>>>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}global_repository?autoReconnect=true&createDatabaseIfNotExist=true"/>
>>>>>>> <param name="schema" value="mysql"/>
>>>>>>> <param name="schemaObjectPrefix" value="rep_"/>
>>>>>>> <param name="user" value="${pm.database.user}"/>
>>>>>>> <param name="password" value="${pm.database.pwd}"/>
>>>>>>> </FileSystem>
>>>>>>>
>>>>>>> <!--
>>>>>>> security configuration
>>>>>>> -->
>>>>>>> <Security appName="Jackrabbit">
>>>>>>> <!--
>>>>>>> security manager:
>>>>>>> class: FQN of class implementing the JackrabbitSecurityManager interface
>>>>>>> -->
>>>>>>> <SecurityManager class="org.apache.jackrabbit.core.DefaultSecurityManager" workspaceName="default">
>>>>>>> <!--
>>>>>>> workspace access:
>>>>>>> class: FQN of class implementing the WorkspaceAccessManager interface
>>>>>>> -->
>>>>>>> <!-- <WorkspaceAccessManager class="..."/> -->
>>>>>>> <!-- <param name="config" value="${rep.home}/security.xml"/> -->
>>>>>>> </SecurityManager>
>>>>>>> <!--
>>>>>>> access manager:
>>>>>>> class: FQN of class implementing the AccessManager interface
>>>>>>> -->
>>>>>>> <AccessManager class="org.apache.jackrabbit.core.security.DefaultAccessManager">
>>>>>>> <!-- <param name="config" value="${rep.home}/access.xml"/> -->
>>>>>>> </AccessManager>
>>>>>>> <LoginModule class="org.apache.jackrabbit.core.security.authentication.DefaultLoginModule">
>>>>>>> <!--
>>>>>>> anonymous user name ('anonymous' is the default value)
>>>>>>> -->
>>>>>>> <param name="anonymousId" value="anonymous"/>
>>>>>>> <!--
>>>>>>> administrator user id (default value if param is missing is 'admin')
>>>>>>> -->
>>>>>>> <param name="adminId" value="admin"/>
>>>>>>> </LoginModule>
>>>>>>> </Security>
>>>>>>> <!--
>>>>>>> location of workspaces root directory and name of default workspace
>>>>>>> -->
>>>>>>> <Workspaces rootPath="${rep.home}/workspaces" defaultWorkspace="default"/>
>>>>>>> <!--
>>>>>>> workspace configuration template:
>>>>>>> used to create the initial workspace if there's no workspace yet
>>>>>>> -->
>>>>>>> <!-- <Workspace name="default">-->
>>>>>>> <Workspace name="default">
>>>>>>> <!--
>>>>>>> virtual file system of the workspace:
>>>>>>> class: FQN of class implementing the FileSystem interface
>>>>>>> -->
>>>>>>> <FileSystem class="org.apache.jackrabbit.core.fs.local.LocalFileSystem">
>>>>>>> <param name="path" value="${wsp.home}"/>
>>>>>>> </FileSystem>
>>>>>>> <!--
>>>>>>> persistence manager of the workspace:
>>>>>>> class: FQN of class implementing the PersistenceManager interface
>>>>>>> -->
>>>>>>> <PersistenceManager class="org.apache.jackrabbit.core.persistence.pool.MySqlPersistenceManager">
>>>>>>> <param name="driver" value="com.mysql.jdbc.Driver"/>
>>>>>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}product?createDatabaseIfNotExist=true"/>
>>>>>>> <param name="user" value="${pm.database.user}"/>
>>>>>>> <param name="password" value="${pm.database.pwd}"/>
>>>>>>> <param name="schemaObjectPrefix" value="Product_"/>
>>>>>>> <param name="schema" value="mysql"/>
>>>>>>> </PersistenceManager>
>>>>>>> <!--
>>>>>>> Search index and the file system it uses.
>>>>>>> class: FQN of class implementing the QueryHandler interface
>>>>>>> -->
>>>>>>> <SearchIndex class="org.apache.jackrabbit.core.query.lucene.SearchIndex">
>>>>>>> <param name="path" value="${wsp.home}/index"/>
>>>>>>> <!--<param name="textFilterClasses"
>>>>>>> value="org.apache.jackrabbit.extractor.PlainTextExtractor,org.apache.jackrabbit.extractor.MsWordTextExtractor,org.apache.jackrabbit.extractor.MsExcelTextExtractor,org.apache.jackrabbit.extractor.MsPowerPointTextExtractor,org.apache.jackrabbit.extractor.PdfTextExtractor,org.apache.jackrabbit.extractor.OpenOfficeTextExtractor,org.apache.jackrabbit.extractor.RTFTextExtractor,org.apache.jackrabbit.extractor.HTMLTextExtractor,org.apache.jackrabbit.extractor.XMLTextExtractor"/>
>>>>>>> <param name="extractorPoolSize" value="2"/>
>>>>>>> <param name="supportHighlighting" value="true"/> -->
>>>>>>> </SearchIndex>
>>>>>>> </Workspace>
>>>>>>> <!--
>>>>>>> Configures the versioning
>>>>>>> -->
>>>>>>> <Versioning rootPath="${rep.home}/version">
>>>>>>> <!--
>>>>>>> Configures the filesystem to use for versioning for the respective
>>>>>>> persistence manager
>>>>>>> -->
>>>>>>> <FileSystem class="org.apache.jackrabbit.core.fs.local.LocalFileSystem">
>>>>>>> <param name="path" value="${rep.home}/version"/>
>>>>>>> </FileSystem>
>>>>>>> <!--
>>>>>>> Configures the persistence manager to be used for persisting version state.
>>>>>>> Please note that the current versioning implementation is based on
>>>>>>> a 'normal' persistence manager, but this could change in future
>>>>>>> implementations.
>>>>>>> -->
>>>>>>> <PersistenceManager class="org.apache.jackrabbit.core.persistence.pool.MySqlPersistenceManager">
>>>>>>> <param name="driver" value="com.mysql.jdbc.Driver"/>
>>>>>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}truedit_versions?createDatabaseIfNotExist=true"/>
>>>>>>> <param name="user" value="${pm.database.user}"/>
>>>>>>> <param name="password" value="${pm.database.pwd}"/>
>>>>>>> <param name="schemaObjectPrefix" value="version_"/>
>>>>>>> <param name="schema" value="mysql"/>
>>>>>>> </PersistenceManager>
>>>>>>> </Versioning>
>>>>>>> <!--
>>>>>>> Configures the Data Store for large binary objects.
>>>>>>> -->
>>>>>>> <DataStore class="org.apache.jackrabbit.core.data.FileDataStore">
>>>>>>> <param name="path" value="${datastore.location.path}"/>
>>>>>>> <param name="minRecordLength" value="100"/>
>>>>>>> </DataStore>
>>>>>>> <!--
>>>>>>> <Cluster id="${cluster.id}" syncDelay="2000">
>>>>>>> <Journal class="org.apache.jackrabbit.core.journal.DatabaseJournal">
>>>>>>> <param name="driver" value="com.mysql.jdbc.Driver" />
>>>>>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}journal?autoReconnect=true&createDatabaseIfNotExist=true"/>
>>>>>>> <param name="schemaObjectPrefix" value="journal_"/>
>>>>>>> <param name="databaseType" value="mysql"/>
>>>>>>> <param name="user" value="${pm.database.user}"/>
>>>>>>> <param name="password" value="${pm.database.pwd}"/>
>>>>>>> </Journal>
>>>>>>> </Cluster>
>>>>>>> -->
>>>>>>>
>>>>>>> </Repository>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Morrell Jacobs
>>>>>>> Chief Software Architect
>>>>>>> MEI
>>>>>>> 610 Old York Road, Suite 250
>>>>>>> Jenkintown, PA 19046
>>>>>>> Phone: 215-886-5662, ext. 252
>>>>>>> Fax: 215-886-5681
>>>>>>> http://www.maned.com
>>>>>>> E-mail: mjacobs@maned.com
>>>>>>> AOL IM: MorrellMEI
>>>>>>>
>>>>>>> Have you seen Nervous Pixel, MEI's creative services division?
>>>>>>> www.nervouspixel.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Ron Wheeler
>>>>>> President
>>>>>> Artifact Software Inc
>>>>>> email: rwheeler@artifact-software.com
>>>>>> skype: ronaldmwheeler
>>>>>> phone: 866-970-2435, ext 102
>>>>>>
>
--
Ron Wheeler
President
Artifact Software Inc
email: rwheeler@artifact-software.com
skype: ronaldmwheeler
phone: 866-970-2435, ext 102
Re: Intermittent Slow Down
Posted by Morrell Jacobs <mj...@maned.com>.
The combo of extra memory and changes seems to have eliminated the problem or at least hidden / delayed it. Tomcat settings include -sever and I've allocated 2 GB of ram: after about 13 hours of running (with light load), the problem has not occurred.
Memory usage is ranging between 150 MB and 175 MB. Forcing garbage collection to run dropped memory under 100 MB, which supports the theory that GC is / was at the heart of the problem.
The assistance and advice is much appreciated. I follow up with additional details as I find them.
Thanks
Morrell
On Dec 17, 2012, at 6:34 PM, Ian Boston wrote:
> On 18 December 2012 09:57, Morrell Jacobs <mj...@maned.com> wrote:
>> Thanks Ian, will try.
>>
>> I added the -server flag earlier today so I'll see if it helps. I've used the JMX tools (VisualVM) and running GC manually seems to drop back down to a base level (or close).
>>
>> When tomcat / web app is idle, the heap usage seems to be a saw tooth pattern, which isn't a surprise. The test takes a while to run so have to wait and see if -server helps.
>>
>> I'm using JVM 6.
>
> I think -server will switch to the ParallelGC. If you still have
> problems read the section on Ergonomics [1], and if the adjustments
> there make no difference think about switching to the Concurrent GC,
> however if you have no free cores for long periods the Concurrent GC
> may not get enough time to free tenured space which will lead to a JVM
> pause. [2]
>
> VisualVM should show you which pool is being exhausted and give you a
> clue to the root cause of a GC related pause to the application
> threads.
>
> (if you wanted to get forensic you could turn on GC logging leave it
> for 24h and then graph the output in a spreadsheet.).
>
> HTH
> Ian
>
>
> 1 http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#par_gc.ergonomics
>
> 2 http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#cms.concurrent_mode_failure
>
>>
>> Thanks!
>>
>>
>> On Dec 17, 2012, at 5:16 PM, Ian Boston wrote:
>>
>>> Hi Morrell,
>>> Sounds suspiciously like garbage collection activity. Rather than
>>> attaching a profiler which may impact heap usage, try attaching a JMX
>>> console which should be less impact (eg JConsole) and looking at the
>>> graph of Garbage Collection activity during the slow down.
>>>
>>> If you see it taking up a significant amount of time and CPU, try
>>> changing the type of garbage collector[1]. IIRC the default without
>>> the -server flag is a serial GC. Increasing heap will lengthen the
>>> time between these slowdowns but won't eliminate them.
>>>
>>> Also, you may be able to recreate the slowdown by forcing a full GC
>>> operation from JConsole. Obviously if a full GC operation doesn't
>>> leave the JVM with significantly more free heap, then you need to up
>>> the limit to prevent the JVM spending all its time trying to free
>>> more.
>>>
>>> You didn't say which JVM you are using, the link below is for 6.
>>> Adjust appropriately.
>>>
>>> HTH
>>> Ian
>>>
>>>
>>> 1 http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#available_collectors
>>>
>>> On 18 December 2012 05:14, Morrell Jacobs <mj...@maned.com> wrote:
>>>> We're running MySQL 5.5.19 for Linux
>>>>
>>>> We had the blobs in the database because of clustering. Since we've moved to single stack, I suppose we could pull them out - not a long term solution, but we could try and see if it helps.
>>>>
>>>> I'll have our MySQL experts review the settings, the restructuring may be accounting for some of the slow down. If we were seeing delays from MySQL, I wouldn't expect Tomcat's CPU usage to be pegged though - I would think all the threads would be idle, waiting for a response. Does that sound right to you?
>>>>
>>>> Thanks
>>>> Morrell
>>>>
>>>>
>>>> On Dec 17, 2012, at 12:48 PM, Ron Wheeler wrote:
>>>>
>>>>> I would play with the JVM memory configuration.
>>>>> It still looks very small.
>>>>> Try doubling it and see if it changes anything.
>>>>> If not, you can look elsewhere and if it does, try making it larger until it stops improving things.
>>>>>
>>>>> What version of MySQL are you running?
>>>>> Have you tuned your MySQL?
>>>>> You might want to look at how MySQL is going to treat these objects and what configuration parameters are recommended when you have large blobs.
>>>>> Have you thought about getting your binary content out of the database. Databases are not very good at 100Mb blobs.
>>>>> Links to 100Mb blobs/files work much better.
>>>>>
>>>>> Is there a compelling reason to put the binary data inside a database?
>>>>> Anytime MySQL has to restructure its indexes or physical storage, you are going to see a pause in the action.
>>>>>
>>>>> Ron
>>>>>
>>>>> On 17/12/2012 12:04 PM, Morrell Jacobs wrote:
>>>>>> Hello all,
>>>>>>
>>>>>> We're currently experience a problem where JackRabbit function will occasionally slow down; according to our QA, the slow down comes in waves: good performance for a while (hours) then bad for a while (minutes / hours). The slow down is on the order of minutes to perform create or modify an object (creation involves creating a 2-3 levels of nodes, and setting properties). Normally these operations take 100 - 300 ms, but during the slow down they can take as much 5 minutes.
>>>>>>
>>>>>> The slow down does not seem to be connected to the load on the server: we've seen it perform well, when many people are connecting, then slow down with only one user. I suspect the slow down is related to indexing, but that's just a guess on my part - I'm looking for something that would happen at some interval.
>>>>>>
>>>>>>
>>>>>> Environment:
>>>>>> * JackRabbit is built into war running in Tomcat (was 7.0.25, recently upgraded to 7.0.34)
>>>>>> * Tomcat was running with default configs but recently up'd memory - -Xms128M -Xmx512M -XX:MaxPermSize=256M
>>>>>> * Repository XML is at end of email
>>>>>> * JackRabbit was originally configured to be clustered, but is now running as just a single instance.
>>>>>>
>>>>>>
>>>>>> Data:
>>>>>> * Our data is structured similar to a file system, except any object can be both a file and contain other objects (files); each object has various metadata (some in properties, some in child nodes), a file and child objects
>>>>>> * We don't use full text searching and don't want file (binary) contents indexed; the only way I was able to prevent the search index from examining the files (some are large 100+ M) was to create my own node types for files. My node types are identical nt:file and nt:content, except that they are named prd:file and prd:content; the unrecognized name seems to prevent indexing from touch them.
>>>>>>
>>>>>>
>>>>>> When I've attempted to use profiling tools, it appears that during the slow down the CPU is pegged; memory usage is also at or close to the limit. In depth profiling tools (NetBeans) get overwhelmed during the slow down.
>>>>>>
>>>>>> We're going thru a variety of tests, but any advice the community can provide will be greatly appreciated.
>>>>>>
>>>>>> Here's the repository.xml:
>>>>>> <Repository>
>>>>>> <FileSystem class="org.apache.jackrabbit.core.fs.db.DbFileSystem">
>>>>>> <param name="driver" value="com.mysql.jdbc.Driver"/>
>>>>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}global_repository?autoReconnect=true&createDatabaseIfNotExist=true"/>
>>>>>> <param name="schema" value="mysql"/>
>>>>>> <param name="schemaObjectPrefix" value="rep_"/>
>>>>>> <param name="user" value="${pm.database.user}"/>
>>>>>> <param name="password" value="${pm.database.pwd}"/>
>>>>>> </FileSystem>
>>>>>>
>>>>>> <!--
>>>>>> security configuration
>>>>>> -->
>>>>>> <Security appName="Jackrabbit">
>>>>>> <!--
>>>>>> security manager:
>>>>>> class: FQN of class implementing the JackrabbitSecurityManager interface
>>>>>> -->
>>>>>> <SecurityManager class="org.apache.jackrabbit.core.DefaultSecurityManager" workspaceName="default">
>>>>>> <!--
>>>>>> workspace access:
>>>>>> class: FQN of class implementing the WorkspaceAccessManager interface
>>>>>> -->
>>>>>> <!-- <WorkspaceAccessManager class="..."/> -->
>>>>>> <!-- <param name="config" value="${rep.home}/security.xml"/> -->
>>>>>> </SecurityManager>
>>>>>> <!--
>>>>>> access manager:
>>>>>> class: FQN of class implementing the AccessManager interface
>>>>>> -->
>>>>>> <AccessManager class="org.apache.jackrabbit.core.security.DefaultAccessManager">
>>>>>> <!-- <param name="config" value="${rep.home}/access.xml"/> -->
>>>>>> </AccessManager>
>>>>>> <LoginModule class="org.apache.jackrabbit.core.security.authentication.DefaultLoginModule">
>>>>>> <!--
>>>>>> anonymous user name ('anonymous' is the default value)
>>>>>> -->
>>>>>> <param name="anonymousId" value="anonymous"/>
>>>>>> <!--
>>>>>> administrator user id (default value if param is missing is 'admin')
>>>>>> -->
>>>>>> <param name="adminId" value="admin"/>
>>>>>> </LoginModule>
>>>>>> </Security>
>>>>>> <!--
>>>>>> location of workspaces root directory and name of default workspace
>>>>>> -->
>>>>>> <Workspaces rootPath="${rep.home}/workspaces" defaultWorkspace="default"/>
>>>>>> <!--
>>>>>> workspace configuration template:
>>>>>> used to create the initial workspace if there's no workspace yet
>>>>>> -->
>>>>>> <!-- <Workspace name="default">-->
>>>>>> <Workspace name="default">
>>>>>> <!--
>>>>>> virtual file system of the workspace:
>>>>>> class: FQN of class implementing the FileSystem interface
>>>>>> -->
>>>>>> <FileSystem class="org.apache.jackrabbit.core.fs.local.LocalFileSystem">
>>>>>> <param name="path" value="${wsp.home}"/>
>>>>>> </FileSystem>
>>>>>> <!--
>>>>>> persistence manager of the workspace:
>>>>>> class: FQN of class implementing the PersistenceManager interface
>>>>>> -->
>>>>>> <PersistenceManager class="org.apache.jackrabbit.core.persistence.pool.MySqlPersistenceManager">
>>>>>> <param name="driver" value="com.mysql.jdbc.Driver"/>
>>>>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}product?createDatabaseIfNotExist=true"/>
>>>>>> <param name="user" value="${pm.database.user}"/>
>>>>>> <param name="password" value="${pm.database.pwd}"/>
>>>>>> <param name="schemaObjectPrefix" value="Product_"/>
>>>>>> <param name="schema" value="mysql"/>
>>>>>> </PersistenceManager>
>>>>>> <!--
>>>>>> Search index and the file system it uses.
>>>>>> class: FQN of class implementing the QueryHandler interface
>>>>>> -->
>>>>>> <SearchIndex class="org.apache.jackrabbit.core.query.lucene.SearchIndex">
>>>>>> <param name="path" value="${wsp.home}/index"/>
>>>>>> <!--<param name="textFilterClasses"
>>>>>> value="org.apache.jackrabbit.extractor.PlainTextExtractor,org.apache.jackrabbit.extractor.MsWordTextExtractor,org.apache.jackrabbit.extractor.MsExcelTextExtractor,org.apache.jackrabbit.extractor.MsPowerPointTextExtractor,org.apache.jackrabbit.extractor.PdfTextExtractor,org.apache.jackrabbit.extractor.OpenOfficeTextExtractor,org.apache.jackrabbit.extractor.RTFTextExtractor,org.apache.jackrabbit.extractor.HTMLTextExtractor,org.apache.jackrabbit.extractor.XMLTextExtractor"/>
>>>>>> <param name="extractorPoolSize" value="2"/>
>>>>>> <param name="supportHighlighting" value="true"/> -->
>>>>>> </SearchIndex>
>>>>>> </Workspace>
>>>>>> <!--
>>>>>> Configures the versioning
>>>>>> -->
>>>>>> <Versioning rootPath="${rep.home}/version">
>>>>>> <!--
>>>>>> Configures the filesystem to use for versioning for the respective
>>>>>> persistence manager
>>>>>> -->
>>>>>> <FileSystem class="org.apache.jackrabbit.core.fs.local.LocalFileSystem">
>>>>>> <param name="path" value="${rep.home}/version"/>
>>>>>> </FileSystem>
>>>>>> <!--
>>>>>> Configures the persistence manager to be used for persisting version state.
>>>>>> Please note that the current versioning implementation is based on
>>>>>> a 'normal' persistence manager, but this could change in future
>>>>>> implementations.
>>>>>> -->
>>>>>> <PersistenceManager class="org.apache.jackrabbit.core.persistence.pool.MySqlPersistenceManager">
>>>>>> <param name="driver" value="com.mysql.jdbc.Driver"/>
>>>>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}truedit_versions?createDatabaseIfNotExist=true"/>
>>>>>> <param name="user" value="${pm.database.user}"/>
>>>>>> <param name="password" value="${pm.database.pwd}"/>
>>>>>> <param name="schemaObjectPrefix" value="version_"/>
>>>>>> <param name="schema" value="mysql"/>
>>>>>> </PersistenceManager>
>>>>>> </Versioning>
>>>>>> <!--
>>>>>> Configures the Data Store for large binary objects.
>>>>>> -->
>>>>>> <DataStore class="org.apache.jackrabbit.core.data.FileDataStore">
>>>>>> <param name="path" value="${datastore.location.path}"/>
>>>>>> <param name="minRecordLength" value="100"/>
>>>>>> </DataStore>
>>>>>> <!--
>>>>>> <Cluster id="${cluster.id}" syncDelay="2000">
>>>>>> <Journal class="org.apache.jackrabbit.core.journal.DatabaseJournal">
>>>>>> <param name="driver" value="com.mysql.jdbc.Driver" />
>>>>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}journal?autoReconnect=true&createDatabaseIfNotExist=true"/>
>>>>>> <param name="schemaObjectPrefix" value="journal_"/>
>>>>>> <param name="databaseType" value="mysql"/>
>>>>>> <param name="user" value="${pm.database.user}"/>
>>>>>> <param name="password" value="${pm.database.pwd}"/>
>>>>>> </Journal>
>>>>>> </Cluster>
>>>>>> -->
>>>>>>
>>>>>> </Repository>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Morrell Jacobs
>>>>>> Chief Software Architect
>>>>>> MEI
>>>>>> 610 Old York Road, Suite 250
>>>>>> Jenkintown, PA 19046
>>>>>> Phone: 215-886-5662, ext. 252
>>>>>> Fax: 215-886-5681
>>>>>> http://www.maned.com
>>>>>> E-mail: mjacobs@maned.com
>>>>>> AOL IM: MorrellMEI
>>>>>>
>>>>>> Have you seen Nervous Pixel, MEI's creative services division?
>>>>>> www.nervouspixel.com
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Ron Wheeler
>>>>> President
>>>>> Artifact Software Inc
>>>>> email: rwheeler@artifact-software.com
>>>>> skype: ronaldmwheeler
>>>>> phone: 866-970-2435, ext 102
>>>>>
>>>>
>>
Re: Intermittent Slow Down
Posted by Ron Wheeler <rw...@artifact-software.com>.
Did increasing the memory affect the situation?
That is probably the easiest stick to poke the beast with.
If is has no effect, it is less likely to be a garbage collection issue.
Ron
On 17/12/2012 6:34 PM, Ian Boston wrote:
> On 18 December 2012 09:57, Morrell Jacobs <mj...@maned.com> wrote:
>> Thanks Ian, will try.
>>
>> I added the -server flag earlier today so I'll see if it helps. I've used the JMX tools (VisualVM) and running GC manually seems to drop back down to a base level (or close).
>>
>> When tomcat / web app is idle, the heap usage seems to be a saw tooth pattern, which isn't a surprise. The test takes a while to run so have to wait and see if -server helps.
>>
>> I'm using JVM 6.
> I think -server will switch to the ParallelGC. If you still have
> problems read the section on Ergonomics [1], and if the adjustments
> there make no difference think about switching to the Concurrent GC,
> however if you have no free cores for long periods the Concurrent GC
> may not get enough time to free tenured space which will lead to a JVM
> pause. [2]
>
> VisualVM should show you which pool is being exhausted and give you a
> clue to the root cause of a GC related pause to the application
> threads.
>
> (if you wanted to get forensic you could turn on GC logging leave it
> for 24h and then graph the output in a spreadsheet.).
>
> HTH
> Ian
>
>
> 1 http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#par_gc.ergonomics
>
> 2 http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#cms.concurrent_mode_failure
>
>> Thanks!
>>
>>
>> On Dec 17, 2012, at 5:16 PM, Ian Boston wrote:
>>
>>> Hi Morrell,
>>> Sounds suspiciously like garbage collection activity. Rather than
>>> attaching a profiler which may impact heap usage, try attaching a JMX
>>> console which should be less impact (eg JConsole) and looking at the
>>> graph of Garbage Collection activity during the slow down.
>>>
>>> If you see it taking up a significant amount of time and CPU, try
>>> changing the type of garbage collector[1]. IIRC the default without
>>> the -server flag is a serial GC. Increasing heap will lengthen the
>>> time between these slowdowns but won't eliminate them.
>>>
>>> Also, you may be able to recreate the slowdown by forcing a full GC
>>> operation from JConsole. Obviously if a full GC operation doesn't
>>> leave the JVM with significantly more free heap, then you need to up
>>> the limit to prevent the JVM spending all its time trying to free
>>> more.
>>>
>>> You didn't say which JVM you are using, the link below is for 6.
>>> Adjust appropriately.
>>>
>>> HTH
>>> Ian
>>>
>>>
>>> 1 http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#available_collectors
>>>
>>> On 18 December 2012 05:14, Morrell Jacobs <mj...@maned.com> wrote:
>>>> We're running MySQL 5.5.19 for Linux
>>>>
>>>> We had the blobs in the database because of clustering. Since we've moved to single stack, I suppose we could pull them out - not a long term solution, but we could try and see if it helps.
>>>>
>>>> I'll have our MySQL experts review the settings, the restructuring may be accounting for some of the slow down. If we were seeing delays from MySQL, I wouldn't expect Tomcat's CPU usage to be pegged though - I would think all the threads would be idle, waiting for a response. Does that sound right to you?
>>>>
>>>> Thanks
>>>> Morrell
>>>>
>>>>
>>>> On Dec 17, 2012, at 12:48 PM, Ron Wheeler wrote:
>>>>
>>>>> I would play with the JVM memory configuration.
>>>>> It still looks very small.
>>>>> Try doubling it and see if it changes anything.
>>>>> If not, you can look elsewhere and if it does, try making it larger until it stops improving things.
>>>>>
>>>>> What version of MySQL are you running?
>>>>> Have you tuned your MySQL?
>>>>> You might want to look at how MySQL is going to treat these objects and what configuration parameters are recommended when you have large blobs.
>>>>> Have you thought about getting your binary content out of the database. Databases are not very good at 100Mb blobs.
>>>>> Links to 100Mb blobs/files work much better.
>>>>>
>>>>> Is there a compelling reason to put the binary data inside a database?
>>>>> Anytime MySQL has to restructure its indexes or physical storage, you are going to see a pause in the action.
>>>>>
>>>>> Ron
>>>>>
>>>>> On 17/12/2012 12:04 PM, Morrell Jacobs wrote:
>>>>>> Hello all,
>>>>>>
>>>>>> We're currently experience a problem where JackRabbit function will occasionally slow down; according to our QA, the slow down comes in waves: good performance for a while (hours) then bad for a while (minutes / hours). The slow down is on the order of minutes to perform create or modify an object (creation involves creating a 2-3 levels of nodes, and setting properties). Normally these operations take 100 - 300 ms, but during the slow down they can take as much 5 minutes.
>>>>>>
>>>>>> The slow down does not seem to be connected to the load on the server: we've seen it perform well, when many people are connecting, then slow down with only one user. I suspect the slow down is related to indexing, but that's just a guess on my part - I'm looking for something that would happen at some interval.
>>>>>>
>>>>>>
>>>>>> Environment:
>>>>>> * JackRabbit is built into war running in Tomcat (was 7.0.25, recently upgraded to 7.0.34)
>>>>>> * Tomcat was running with default configs but recently up'd memory - -Xms128M -Xmx512M -XX:MaxPermSize=256M
>>>>>> * Repository XML is at end of email
>>>>>> * JackRabbit was originally configured to be clustered, but is now running as just a single instance.
>>>>>>
>>>>>>
>>>>>> Data:
>>>>>> * Our data is structured similar to a file system, except any object can be both a file and contain other objects (files); each object has various metadata (some in properties, some in child nodes), a file and child objects
>>>>>> * We don't use full text searching and don't want file (binary) contents indexed; the only way I was able to prevent the search index from examining the files (some are large 100+ M) was to create my own node types for files. My node types are identical nt:file and nt:content, except that they are named prd:file and prd:content; the unrecognized name seems to prevent indexing from touch them.
>>>>>>
>>>>>>
>>>>>> When I've attempted to use profiling tools, it appears that during the slow down the CPU is pegged; memory usage is also at or close to the limit. In depth profiling tools (NetBeans) get overwhelmed during the slow down.
>>>>>>
>>>>>> We're going thru a variety of tests, but any advice the community can provide will be greatly appreciated.
>>>>>>
>>>>>> Here's the repository.xml:
>>>>>> <Repository>
>>>>>> <FileSystem class="org.apache.jackrabbit.core.fs.db.DbFileSystem">
>>>>>> <param name="driver" value="com.mysql.jdbc.Driver"/>
>>>>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}global_repository?autoReconnect=true&createDatabaseIfNotExist=true"/>
>>>>>> <param name="schema" value="mysql"/>
>>>>>> <param name="schemaObjectPrefix" value="rep_"/>
>>>>>> <param name="user" value="${pm.database.user}"/>
>>>>>> <param name="password" value="${pm.database.pwd}"/>
>>>>>> </FileSystem>
>>>>>>
>>>>>> <!--
>>>>>> security configuration
>>>>>> -->
>>>>>> <Security appName="Jackrabbit">
>>>>>> <!--
>>>>>> security manager:
>>>>>> class: FQN of class implementing the JackrabbitSecurityManager interface
>>>>>> -->
>>>>>> <SecurityManager class="org.apache.jackrabbit.core.DefaultSecurityManager" workspaceName="default">
>>>>>> <!--
>>>>>> workspace access:
>>>>>> class: FQN of class implementing the WorkspaceAccessManager interface
>>>>>> -->
>>>>>> <!-- <WorkspaceAccessManager class="..."/> -->
>>>>>> <!-- <param name="config" value="${rep.home}/security.xml"/> -->
>>>>>> </SecurityManager>
>>>>>> <!--
>>>>>> access manager:
>>>>>> class: FQN of class implementing the AccessManager interface
>>>>>> -->
>>>>>> <AccessManager class="org.apache.jackrabbit.core.security.DefaultAccessManager">
>>>>>> <!-- <param name="config" value="${rep.home}/access.xml"/> -->
>>>>>> </AccessManager>
>>>>>> <LoginModule class="org.apache.jackrabbit.core.security.authentication.DefaultLoginModule">
>>>>>> <!--
>>>>>> anonymous user name ('anonymous' is the default value)
>>>>>> -->
>>>>>> <param name="anonymousId" value="anonymous"/>
>>>>>> <!--
>>>>>> administrator user id (default value if param is missing is 'admin')
>>>>>> -->
>>>>>> <param name="adminId" value="admin"/>
>>>>>> </LoginModule>
>>>>>> </Security>
>>>>>> <!--
>>>>>> location of workspaces root directory and name of default workspace
>>>>>> -->
>>>>>> <Workspaces rootPath="${rep.home}/workspaces" defaultWorkspace="default"/>
>>>>>> <!--
>>>>>> workspace configuration template:
>>>>>> used to create the initial workspace if there's no workspace yet
>>>>>> -->
>>>>>> <!-- <Workspace name="default">-->
>>>>>> <Workspace name="default">
>>>>>> <!--
>>>>>> virtual file system of the workspace:
>>>>>> class: FQN of class implementing the FileSystem interface
>>>>>> -->
>>>>>> <FileSystem class="org.apache.jackrabbit.core.fs.local.LocalFileSystem">
>>>>>> <param name="path" value="${wsp.home}"/>
>>>>>> </FileSystem>
>>>>>> <!--
>>>>>> persistence manager of the workspace:
>>>>>> class: FQN of class implementing the PersistenceManager interface
>>>>>> -->
>>>>>> <PersistenceManager class="org.apache.jackrabbit.core.persistence.pool.MySqlPersistenceManager">
>>>>>> <param name="driver" value="com.mysql.jdbc.Driver"/>
>>>>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}product?createDatabaseIfNotExist=true"/>
>>>>>> <param name="user" value="${pm.database.user}"/>
>>>>>> <param name="password" value="${pm.database.pwd}"/>
>>>>>> <param name="schemaObjectPrefix" value="Product_"/>
>>>>>> <param name="schema" value="mysql"/>
>>>>>> </PersistenceManager>
>>>>>> <!--
>>>>>> Search index and the file system it uses.
>>>>>> class: FQN of class implementing the QueryHandler interface
>>>>>> -->
>>>>>> <SearchIndex class="org.apache.jackrabbit.core.query.lucene.SearchIndex">
>>>>>> <param name="path" value="${wsp.home}/index"/>
>>>>>> <!--<param name="textFilterClasses"
>>>>>> value="org.apache.jackrabbit.extractor.PlainTextExtractor,org.apache.jackrabbit.extractor.MsWordTextExtractor,org.apache.jackrabbit.extractor.MsExcelTextExtractor,org.apache.jackrabbit.extractor.MsPowerPointTextExtractor,org.apache.jackrabbit.extractor.PdfTextExtractor,org.apache.jackrabbit.extractor.OpenOfficeTextExtractor,org.apache.jackrabbit.extractor.RTFTextExtractor,org.apache.jackrabbit.extractor.HTMLTextExtractor,org.apache.jackrabbit.extractor.XMLTextExtractor"/>
>>>>>> <param name="extractorPoolSize" value="2"/>
>>>>>> <param name="supportHighlighting" value="true"/> -->
>>>>>> </SearchIndex>
>>>>>> </Workspace>
>>>>>> <!--
>>>>>> Configures the versioning
>>>>>> -->
>>>>>> <Versioning rootPath="${rep.home}/version">
>>>>>> <!--
>>>>>> Configures the filesystem to use for versioning for the respective
>>>>>> persistence manager
>>>>>> -->
>>>>>> <FileSystem class="org.apache.jackrabbit.core.fs.local.LocalFileSystem">
>>>>>> <param name="path" value="${rep.home}/version"/>
>>>>>> </FileSystem>
>>>>>> <!--
>>>>>> Configures the persistence manager to be used for persisting version state.
>>>>>> Please note that the current versioning implementation is based on
>>>>>> a 'normal' persistence manager, but this could change in future
>>>>>> implementations.
>>>>>> -->
>>>>>> <PersistenceManager class="org.apache.jackrabbit.core.persistence.pool.MySqlPersistenceManager">
>>>>>> <param name="driver" value="com.mysql.jdbc.Driver"/>
>>>>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}truedit_versions?createDatabaseIfNotExist=true"/>
>>>>>> <param name="user" value="${pm.database.user}"/>
>>>>>> <param name="password" value="${pm.database.pwd}"/>
>>>>>> <param name="schemaObjectPrefix" value="version_"/>
>>>>>> <param name="schema" value="mysql"/>
>>>>>> </PersistenceManager>
>>>>>> </Versioning>
>>>>>> <!--
>>>>>> Configures the Data Store for large binary objects.
>>>>>> -->
>>>>>> <DataStore class="org.apache.jackrabbit.core.data.FileDataStore">
>>>>>> <param name="path" value="${datastore.location.path}"/>
>>>>>> <param name="minRecordLength" value="100"/>
>>>>>> </DataStore>
>>>>>> <!--
>>>>>> <Cluster id="${cluster.id}" syncDelay="2000">
>>>>>> <Journal class="org.apache.jackrabbit.core.journal.DatabaseJournal">
>>>>>> <param name="driver" value="com.mysql.jdbc.Driver" />
>>>>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}journal?autoReconnect=true&createDatabaseIfNotExist=true"/>
>>>>>> <param name="schemaObjectPrefix" value="journal_"/>
>>>>>> <param name="databaseType" value="mysql"/>
>>>>>> <param name="user" value="${pm.database.user}"/>
>>>>>> <param name="password" value="${pm.database.pwd}"/>
>>>>>> </Journal>
>>>>>> </Cluster>
>>>>>> -->
>>>>>>
>>>>>> </Repository>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Morrell Jacobs
>>>>>> Chief Software Architect
>>>>>> MEI
>>>>>> 610 Old York Road, Suite 250
>>>>>> Jenkintown, PA 19046
>>>>>> Phone: 215-886-5662, ext. 252
>>>>>> Fax: 215-886-5681
>>>>>> http://www.maned.com
>>>>>> E-mail: mjacobs@maned.com
>>>>>> AOL IM: MorrellMEI
>>>>>>
>>>>>> Have you seen Nervous Pixel, MEI's creative services division?
>>>>>> www.nervouspixel.com
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Ron Wheeler
>>>>> President
>>>>> Artifact Software Inc
>>>>> email: rwheeler@artifact-software.com
>>>>> skype: ronaldmwheeler
>>>>> phone: 866-970-2435, ext 102
>>>>>
--
Ron Wheeler
President
Artifact Software Inc
email: rwheeler@artifact-software.com
skype: ronaldmwheeler
phone: 866-970-2435, ext 102
Re: Intermittent Slow Down
Posted by Ian Boston <ie...@tfd.co.uk>.
On 18 December 2012 09:57, Morrell Jacobs <mj...@maned.com> wrote:
> Thanks Ian, will try.
>
> I added the -server flag earlier today so I'll see if it helps. I've used the JMX tools (VisualVM) and running GC manually seems to drop back down to a base level (or close).
>
> When tomcat / web app is idle, the heap usage seems to be a saw tooth pattern, which isn't a surprise. The test takes a while to run so have to wait and see if -server helps.
>
> I'm using JVM 6.
I think -server will switch to the ParallelGC. If you still have
problems read the section on Ergonomics [1], and if the adjustments
there make no difference think about switching to the Concurrent GC,
however if you have no free cores for long periods the Concurrent GC
may not get enough time to free tenured space which will lead to a JVM
pause. [2]
VisualVM should show you which pool is being exhausted and give you a
clue to the root cause of a GC related pause to the application
threads.
(if you wanted to get forensic you could turn on GC logging leave it
for 24h and then graph the output in a spreadsheet.).
HTH
Ian
1 http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#par_gc.ergonomics
2 http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#cms.concurrent_mode_failure
>
> Thanks!
>
>
> On Dec 17, 2012, at 5:16 PM, Ian Boston wrote:
>
>> Hi Morrell,
>> Sounds suspiciously like garbage collection activity. Rather than
>> attaching a profiler which may impact heap usage, try attaching a JMX
>> console which should be less impact (eg JConsole) and looking at the
>> graph of Garbage Collection activity during the slow down.
>>
>> If you see it taking up a significant amount of time and CPU, try
>> changing the type of garbage collector[1]. IIRC the default without
>> the -server flag is a serial GC. Increasing heap will lengthen the
>> time between these slowdowns but won't eliminate them.
>>
>> Also, you may be able to recreate the slowdown by forcing a full GC
>> operation from JConsole. Obviously if a full GC operation doesn't
>> leave the JVM with significantly more free heap, then you need to up
>> the limit to prevent the JVM spending all its time trying to free
>> more.
>>
>> You didn't say which JVM you are using, the link below is for 6.
>> Adjust appropriately.
>>
>> HTH
>> Ian
>>
>>
>> 1 http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#available_collectors
>>
>> On 18 December 2012 05:14, Morrell Jacobs <mj...@maned.com> wrote:
>>> We're running MySQL 5.5.19 for Linux
>>>
>>> We had the blobs in the database because of clustering. Since we've moved to single stack, I suppose we could pull them out - not a long term solution, but we could try and see if it helps.
>>>
>>> I'll have our MySQL experts review the settings, the restructuring may be accounting for some of the slow down. If we were seeing delays from MySQL, I wouldn't expect Tomcat's CPU usage to be pegged though - I would think all the threads would be idle, waiting for a response. Does that sound right to you?
>>>
>>> Thanks
>>> Morrell
>>>
>>>
>>> On Dec 17, 2012, at 12:48 PM, Ron Wheeler wrote:
>>>
>>>> I would play with the JVM memory configuration.
>>>> It still looks very small.
>>>> Try doubling it and see if it changes anything.
>>>> If not, you can look elsewhere and if it does, try making it larger until it stops improving things.
>>>>
>>>> What version of MySQL are you running?
>>>> Have you tuned your MySQL?
>>>> You might want to look at how MySQL is going to treat these objects and what configuration parameters are recommended when you have large blobs.
>>>> Have you thought about getting your binary content out of the database. Databases are not very good at 100Mb blobs.
>>>> Links to 100Mb blobs/files work much better.
>>>>
>>>> Is there a compelling reason to put the binary data inside a database?
>>>> Anytime MySQL has to restructure its indexes or physical storage, you are going to see a pause in the action.
>>>>
>>>> Ron
>>>>
>>>> On 17/12/2012 12:04 PM, Morrell Jacobs wrote:
>>>>> Hello all,
>>>>>
>>>>> We're currently experience a problem where JackRabbit function will occasionally slow down; according to our QA, the slow down comes in waves: good performance for a while (hours) then bad for a while (minutes / hours). The slow down is on the order of minutes to perform create or modify an object (creation involves creating a 2-3 levels of nodes, and setting properties). Normally these operations take 100 - 300 ms, but during the slow down they can take as much 5 minutes.
>>>>>
>>>>> The slow down does not seem to be connected to the load on the server: we've seen it perform well, when many people are connecting, then slow down with only one user. I suspect the slow down is related to indexing, but that's just a guess on my part - I'm looking for something that would happen at some interval.
>>>>>
>>>>>
>>>>> Environment:
>>>>> * JackRabbit is built into war running in Tomcat (was 7.0.25, recently upgraded to 7.0.34)
>>>>> * Tomcat was running with default configs but recently up'd memory - -Xms128M -Xmx512M -XX:MaxPermSize=256M
>>>>> * Repository XML is at end of email
>>>>> * JackRabbit was originally configured to be clustered, but is now running as just a single instance.
>>>>>
>>>>>
>>>>> Data:
>>>>> * Our data is structured similar to a file system, except any object can be both a file and contain other objects (files); each object has various metadata (some in properties, some in child nodes), a file and child objects
>>>>> * We don't use full text searching and don't want file (binary) contents indexed; the only way I was able to prevent the search index from examining the files (some are large 100+ M) was to create my own node types for files. My node types are identical nt:file and nt:content, except that they are named prd:file and prd:content; the unrecognized name seems to prevent indexing from touch them.
>>>>>
>>>>>
>>>>> When I've attempted to use profiling tools, it appears that during the slow down the CPU is pegged; memory usage is also at or close to the limit. In depth profiling tools (NetBeans) get overwhelmed during the slow down.
>>>>>
>>>>> We're going thru a variety of tests, but any advice the community can provide will be greatly appreciated.
>>>>>
>>>>> Here's the repository.xml:
>>>>> <Repository>
>>>>> <FileSystem class="org.apache.jackrabbit.core.fs.db.DbFileSystem">
>>>>> <param name="driver" value="com.mysql.jdbc.Driver"/>
>>>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}global_repository?autoReconnect=true&createDatabaseIfNotExist=true"/>
>>>>> <param name="schema" value="mysql"/>
>>>>> <param name="schemaObjectPrefix" value="rep_"/>
>>>>> <param name="user" value="${pm.database.user}"/>
>>>>> <param name="password" value="${pm.database.pwd}"/>
>>>>> </FileSystem>
>>>>>
>>>>> <!--
>>>>> security configuration
>>>>> -->
>>>>> <Security appName="Jackrabbit">
>>>>> <!--
>>>>> security manager:
>>>>> class: FQN of class implementing the JackrabbitSecurityManager interface
>>>>> -->
>>>>> <SecurityManager class="org.apache.jackrabbit.core.DefaultSecurityManager" workspaceName="default">
>>>>> <!--
>>>>> workspace access:
>>>>> class: FQN of class implementing the WorkspaceAccessManager interface
>>>>> -->
>>>>> <!-- <WorkspaceAccessManager class="..."/> -->
>>>>> <!-- <param name="config" value="${rep.home}/security.xml"/> -->
>>>>> </SecurityManager>
>>>>> <!--
>>>>> access manager:
>>>>> class: FQN of class implementing the AccessManager interface
>>>>> -->
>>>>> <AccessManager class="org.apache.jackrabbit.core.security.DefaultAccessManager">
>>>>> <!-- <param name="config" value="${rep.home}/access.xml"/> -->
>>>>> </AccessManager>
>>>>> <LoginModule class="org.apache.jackrabbit.core.security.authentication.DefaultLoginModule">
>>>>> <!--
>>>>> anonymous user name ('anonymous' is the default value)
>>>>> -->
>>>>> <param name="anonymousId" value="anonymous"/>
>>>>> <!--
>>>>> administrator user id (default value if param is missing is 'admin')
>>>>> -->
>>>>> <param name="adminId" value="admin"/>
>>>>> </LoginModule>
>>>>> </Security>
>>>>> <!--
>>>>> location of workspaces root directory and name of default workspace
>>>>> -->
>>>>> <Workspaces rootPath="${rep.home}/workspaces" defaultWorkspace="default"/>
>>>>> <!--
>>>>> workspace configuration template:
>>>>> used to create the initial workspace if there's no workspace yet
>>>>> -->
>>>>> <!-- <Workspace name="default">-->
>>>>> <Workspace name="default">
>>>>> <!--
>>>>> virtual file system of the workspace:
>>>>> class: FQN of class implementing the FileSystem interface
>>>>> -->
>>>>> <FileSystem class="org.apache.jackrabbit.core.fs.local.LocalFileSystem">
>>>>> <param name="path" value="${wsp.home}"/>
>>>>> </FileSystem>
>>>>> <!--
>>>>> persistence manager of the workspace:
>>>>> class: FQN of class implementing the PersistenceManager interface
>>>>> -->
>>>>> <PersistenceManager class="org.apache.jackrabbit.core.persistence.pool.MySqlPersistenceManager">
>>>>> <param name="driver" value="com.mysql.jdbc.Driver"/>
>>>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}product?createDatabaseIfNotExist=true"/>
>>>>> <param name="user" value="${pm.database.user}"/>
>>>>> <param name="password" value="${pm.database.pwd}"/>
>>>>> <param name="schemaObjectPrefix" value="Product_"/>
>>>>> <param name="schema" value="mysql"/>
>>>>> </PersistenceManager>
>>>>> <!--
>>>>> Search index and the file system it uses.
>>>>> class: FQN of class implementing the QueryHandler interface
>>>>> -->
>>>>> <SearchIndex class="org.apache.jackrabbit.core.query.lucene.SearchIndex">
>>>>> <param name="path" value="${wsp.home}/index"/>
>>>>> <!--<param name="textFilterClasses"
>>>>> value="org.apache.jackrabbit.extractor.PlainTextExtractor,org.apache.jackrabbit.extractor.MsWordTextExtractor,org.apache.jackrabbit.extractor.MsExcelTextExtractor,org.apache.jackrabbit.extractor.MsPowerPointTextExtractor,org.apache.jackrabbit.extractor.PdfTextExtractor,org.apache.jackrabbit.extractor.OpenOfficeTextExtractor,org.apache.jackrabbit.extractor.RTFTextExtractor,org.apache.jackrabbit.extractor.HTMLTextExtractor,org.apache.jackrabbit.extractor.XMLTextExtractor"/>
>>>>> <param name="extractorPoolSize" value="2"/>
>>>>> <param name="supportHighlighting" value="true"/> -->
>>>>> </SearchIndex>
>>>>> </Workspace>
>>>>> <!--
>>>>> Configures the versioning
>>>>> -->
>>>>> <Versioning rootPath="${rep.home}/version">
>>>>> <!--
>>>>> Configures the filesystem to use for versioning for the respective
>>>>> persistence manager
>>>>> -->
>>>>> <FileSystem class="org.apache.jackrabbit.core.fs.local.LocalFileSystem">
>>>>> <param name="path" value="${rep.home}/version"/>
>>>>> </FileSystem>
>>>>> <!--
>>>>> Configures the persistence manager to be used for persisting version state.
>>>>> Please note that the current versioning implementation is based on
>>>>> a 'normal' persistence manager, but this could change in future
>>>>> implementations.
>>>>> -->
>>>>> <PersistenceManager class="org.apache.jackrabbit.core.persistence.pool.MySqlPersistenceManager">
>>>>> <param name="driver" value="com.mysql.jdbc.Driver"/>
>>>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}truedit_versions?createDatabaseIfNotExist=true"/>
>>>>> <param name="user" value="${pm.database.user}"/>
>>>>> <param name="password" value="${pm.database.pwd}"/>
>>>>> <param name="schemaObjectPrefix" value="version_"/>
>>>>> <param name="schema" value="mysql"/>
>>>>> </PersistenceManager>
>>>>> </Versioning>
>>>>> <!--
>>>>> Configures the Data Store for large binary objects.
>>>>> -->
>>>>> <DataStore class="org.apache.jackrabbit.core.data.FileDataStore">
>>>>> <param name="path" value="${datastore.location.path}"/>
>>>>> <param name="minRecordLength" value="100"/>
>>>>> </DataStore>
>>>>> <!--
>>>>> <Cluster id="${cluster.id}" syncDelay="2000">
>>>>> <Journal class="org.apache.jackrabbit.core.journal.DatabaseJournal">
>>>>> <param name="driver" value="com.mysql.jdbc.Driver" />
>>>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}journal?autoReconnect=true&createDatabaseIfNotExist=true"/>
>>>>> <param name="schemaObjectPrefix" value="journal_"/>
>>>>> <param name="databaseType" value="mysql"/>
>>>>> <param name="user" value="${pm.database.user}"/>
>>>>> <param name="password" value="${pm.database.pwd}"/>
>>>>> </Journal>
>>>>> </Cluster>
>>>>> -->
>>>>>
>>>>> </Repository>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Morrell Jacobs
>>>>> Chief Software Architect
>>>>> MEI
>>>>> 610 Old York Road, Suite 250
>>>>> Jenkintown, PA 19046
>>>>> Phone: 215-886-5662, ext. 252
>>>>> Fax: 215-886-5681
>>>>> http://www.maned.com
>>>>> E-mail: mjacobs@maned.com
>>>>> AOL IM: MorrellMEI
>>>>>
>>>>> Have you seen Nervous Pixel, MEI's creative services division?
>>>>> www.nervouspixel.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Ron Wheeler
>>>> President
>>>> Artifact Software Inc
>>>> email: rwheeler@artifact-software.com
>>>> skype: ronaldmwheeler
>>>> phone: 866-970-2435, ext 102
>>>>
>>>
>
Re: Intermittent Slow Down
Posted by Morrell Jacobs <mj...@maned.com>.
Thanks Ian, will try.
I added the -server flag earlier today so I'll see if it helps. I've used the JMX tools (VisualVM) and running GC manually seems to drop back down to a base level (or close).
When tomcat / web app is idle, the heap usage seems to be a saw tooth pattern, which isn't a surprise. The test takes a while to run so have to wait and see if -server helps.
I'm using JVM 6.
Thanks!
On Dec 17, 2012, at 5:16 PM, Ian Boston wrote:
> Hi Morrell,
> Sounds suspiciously like garbage collection activity. Rather than
> attaching a profiler which may impact heap usage, try attaching a JMX
> console which should be less impact (eg JConsole) and looking at the
> graph of Garbage Collection activity during the slow down.
>
> If you see it taking up a significant amount of time and CPU, try
> changing the type of garbage collector[1]. IIRC the default without
> the -server flag is a serial GC. Increasing heap will lengthen the
> time between these slowdowns but won't eliminate them.
>
> Also, you may be able to recreate the slowdown by forcing a full GC
> operation from JConsole. Obviously if a full GC operation doesn't
> leave the JVM with significantly more free heap, then you need to up
> the limit to prevent the JVM spending all its time trying to free
> more.
>
> You didn't say which JVM you are using, the link below is for 6.
> Adjust appropriately.
>
> HTH
> Ian
>
>
> 1 http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#available_collectors
>
> On 18 December 2012 05:14, Morrell Jacobs <mj...@maned.com> wrote:
>> We're running MySQL 5.5.19 for Linux
>>
>> We had the blobs in the database because of clustering. Since we've moved to single stack, I suppose we could pull them out - not a long term solution, but we could try and see if it helps.
>>
>> I'll have our MySQL experts review the settings, the restructuring may be accounting for some of the slow down. If we were seeing delays from MySQL, I wouldn't expect Tomcat's CPU usage to be pegged though - I would think all the threads would be idle, waiting for a response. Does that sound right to you?
>>
>> Thanks
>> Morrell
>>
>>
>> On Dec 17, 2012, at 12:48 PM, Ron Wheeler wrote:
>>
>>> I would play with the JVM memory configuration.
>>> It still looks very small.
>>> Try doubling it and see if it changes anything.
>>> If not, you can look elsewhere and if it does, try making it larger until it stops improving things.
>>>
>>> What version of MySQL are you running?
>>> Have you tuned your MySQL?
>>> You might want to look at how MySQL is going to treat these objects and what configuration parameters are recommended when you have large blobs.
>>> Have you thought about getting your binary content out of the database. Databases are not very good at 100Mb blobs.
>>> Links to 100Mb blobs/files work much better.
>>>
>>> Is there a compelling reason to put the binary data inside a database?
>>> Anytime MySQL has to restructure its indexes or physical storage, you are going to see a pause in the action.
>>>
>>> Ron
>>>
>>> On 17/12/2012 12:04 PM, Morrell Jacobs wrote:
>>>> Hello all,
>>>>
>>>> We're currently experience a problem where JackRabbit function will occasionally slow down; according to our QA, the slow down comes in waves: good performance for a while (hours) then bad for a while (minutes / hours). The slow down is on the order of minutes to perform create or modify an object (creation involves creating a 2-3 levels of nodes, and setting properties). Normally these operations take 100 - 300 ms, but during the slow down they can take as much 5 minutes.
>>>>
>>>> The slow down does not seem to be connected to the load on the server: we've seen it perform well, when many people are connecting, then slow down with only one user. I suspect the slow down is related to indexing, but that's just a guess on my part - I'm looking for something that would happen at some interval.
>>>>
>>>>
>>>> Environment:
>>>> * JackRabbit is built into war running in Tomcat (was 7.0.25, recently upgraded to 7.0.34)
>>>> * Tomcat was running with default configs but recently up'd memory - -Xms128M -Xmx512M -XX:MaxPermSize=256M
>>>> * Repository XML is at end of email
>>>> * JackRabbit was originally configured to be clustered, but is now running as just a single instance.
>>>>
>>>>
>>>> Data:
>>>> * Our data is structured similar to a file system, except any object can be both a file and contain other objects (files); each object has various metadata (some in properties, some in child nodes), a file and child objects
>>>> * We don't use full text searching and don't want file (binary) contents indexed; the only way I was able to prevent the search index from examining the files (some are large 100+ M) was to create my own node types for files. My node types are identical nt:file and nt:content, except that they are named prd:file and prd:content; the unrecognized name seems to prevent indexing from touch them.
>>>>
>>>>
>>>> When I've attempted to use profiling tools, it appears that during the slow down the CPU is pegged; memory usage is also at or close to the limit. In depth profiling tools (NetBeans) get overwhelmed during the slow down.
>>>>
>>>> We're going thru a variety of tests, but any advice the community can provide will be greatly appreciated.
>>>>
>>>> Here's the repository.xml:
>>>> <Repository>
>>>> <FileSystem class="org.apache.jackrabbit.core.fs.db.DbFileSystem">
>>>> <param name="driver" value="com.mysql.jdbc.Driver"/>
>>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}global_repository?autoReconnect=true&createDatabaseIfNotExist=true"/>
>>>> <param name="schema" value="mysql"/>
>>>> <param name="schemaObjectPrefix" value="rep_"/>
>>>> <param name="user" value="${pm.database.user}"/>
>>>> <param name="password" value="${pm.database.pwd}"/>
>>>> </FileSystem>
>>>>
>>>> <!--
>>>> security configuration
>>>> -->
>>>> <Security appName="Jackrabbit">
>>>> <!--
>>>> security manager:
>>>> class: FQN of class implementing the JackrabbitSecurityManager interface
>>>> -->
>>>> <SecurityManager class="org.apache.jackrabbit.core.DefaultSecurityManager" workspaceName="default">
>>>> <!--
>>>> workspace access:
>>>> class: FQN of class implementing the WorkspaceAccessManager interface
>>>> -->
>>>> <!-- <WorkspaceAccessManager class="..."/> -->
>>>> <!-- <param name="config" value="${rep.home}/security.xml"/> -->
>>>> </SecurityManager>
>>>> <!--
>>>> access manager:
>>>> class: FQN of class implementing the AccessManager interface
>>>> -->
>>>> <AccessManager class="org.apache.jackrabbit.core.security.DefaultAccessManager">
>>>> <!-- <param name="config" value="${rep.home}/access.xml"/> -->
>>>> </AccessManager>
>>>> <LoginModule class="org.apache.jackrabbit.core.security.authentication.DefaultLoginModule">
>>>> <!--
>>>> anonymous user name ('anonymous' is the default value)
>>>> -->
>>>> <param name="anonymousId" value="anonymous"/>
>>>> <!--
>>>> administrator user id (default value if param is missing is 'admin')
>>>> -->
>>>> <param name="adminId" value="admin"/>
>>>> </LoginModule>
>>>> </Security>
>>>> <!--
>>>> location of workspaces root directory and name of default workspace
>>>> -->
>>>> <Workspaces rootPath="${rep.home}/workspaces" defaultWorkspace="default"/>
>>>> <!--
>>>> workspace configuration template:
>>>> used to create the initial workspace if there's no workspace yet
>>>> -->
>>>> <!-- <Workspace name="default">-->
>>>> <Workspace name="default">
>>>> <!--
>>>> virtual file system of the workspace:
>>>> class: FQN of class implementing the FileSystem interface
>>>> -->
>>>> <FileSystem class="org.apache.jackrabbit.core.fs.local.LocalFileSystem">
>>>> <param name="path" value="${wsp.home}"/>
>>>> </FileSystem>
>>>> <!--
>>>> persistence manager of the workspace:
>>>> class: FQN of class implementing the PersistenceManager interface
>>>> -->
>>>> <PersistenceManager class="org.apache.jackrabbit.core.persistence.pool.MySqlPersistenceManager">
>>>> <param name="driver" value="com.mysql.jdbc.Driver"/>
>>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}product?createDatabaseIfNotExist=true"/>
>>>> <param name="user" value="${pm.database.user}"/>
>>>> <param name="password" value="${pm.database.pwd}"/>
>>>> <param name="schemaObjectPrefix" value="Product_"/>
>>>> <param name="schema" value="mysql"/>
>>>> </PersistenceManager>
>>>> <!--
>>>> Search index and the file system it uses.
>>>> class: FQN of class implementing the QueryHandler interface
>>>> -->
>>>> <SearchIndex class="org.apache.jackrabbit.core.query.lucene.SearchIndex">
>>>> <param name="path" value="${wsp.home}/index"/>
>>>> <!--<param name="textFilterClasses"
>>>> value="org.apache.jackrabbit.extractor.PlainTextExtractor,org.apache.jackrabbit.extractor.MsWordTextExtractor,org.apache.jackrabbit.extractor.MsExcelTextExtractor,org.apache.jackrabbit.extractor.MsPowerPointTextExtractor,org.apache.jackrabbit.extractor.PdfTextExtractor,org.apache.jackrabbit.extractor.OpenOfficeTextExtractor,org.apache.jackrabbit.extractor.RTFTextExtractor,org.apache.jackrabbit.extractor.HTMLTextExtractor,org.apache.jackrabbit.extractor.XMLTextExtractor"/>
>>>> <param name="extractorPoolSize" value="2"/>
>>>> <param name="supportHighlighting" value="true"/> -->
>>>> </SearchIndex>
>>>> </Workspace>
>>>> <!--
>>>> Configures the versioning
>>>> -->
>>>> <Versioning rootPath="${rep.home}/version">
>>>> <!--
>>>> Configures the filesystem to use for versioning for the respective
>>>> persistence manager
>>>> -->
>>>> <FileSystem class="org.apache.jackrabbit.core.fs.local.LocalFileSystem">
>>>> <param name="path" value="${rep.home}/version"/>
>>>> </FileSystem>
>>>> <!--
>>>> Configures the persistence manager to be used for persisting version state.
>>>> Please note that the current versioning implementation is based on
>>>> a 'normal' persistence manager, but this could change in future
>>>> implementations.
>>>> -->
>>>> <PersistenceManager class="org.apache.jackrabbit.core.persistence.pool.MySqlPersistenceManager">
>>>> <param name="driver" value="com.mysql.jdbc.Driver"/>
>>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}truedit_versions?createDatabaseIfNotExist=true"/>
>>>> <param name="user" value="${pm.database.user}"/>
>>>> <param name="password" value="${pm.database.pwd}"/>
>>>> <param name="schemaObjectPrefix" value="version_"/>
>>>> <param name="schema" value="mysql"/>
>>>> </PersistenceManager>
>>>> </Versioning>
>>>> <!--
>>>> Configures the Data Store for large binary objects.
>>>> -->
>>>> <DataStore class="org.apache.jackrabbit.core.data.FileDataStore">
>>>> <param name="path" value="${datastore.location.path}"/>
>>>> <param name="minRecordLength" value="100"/>
>>>> </DataStore>
>>>> <!--
>>>> <Cluster id="${cluster.id}" syncDelay="2000">
>>>> <Journal class="org.apache.jackrabbit.core.journal.DatabaseJournal">
>>>> <param name="driver" value="com.mysql.jdbc.Driver" />
>>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}journal?autoReconnect=true&createDatabaseIfNotExist=true"/>
>>>> <param name="schemaObjectPrefix" value="journal_"/>
>>>> <param name="databaseType" value="mysql"/>
>>>> <param name="user" value="${pm.database.user}"/>
>>>> <param name="password" value="${pm.database.pwd}"/>
>>>> </Journal>
>>>> </Cluster>
>>>> -->
>>>>
>>>> </Repository>
>>>>
>>>>
>>>>
>>>> --
>>>> Morrell Jacobs
>>>> Chief Software Architect
>>>> MEI
>>>> 610 Old York Road, Suite 250
>>>> Jenkintown, PA 19046
>>>> Phone: 215-886-5662, ext. 252
>>>> Fax: 215-886-5681
>>>> http://www.maned.com
>>>> E-mail: mjacobs@maned.com
>>>> AOL IM: MorrellMEI
>>>>
>>>> Have you seen Nervous Pixel, MEI's creative services division?
>>>> www.nervouspixel.com
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Ron Wheeler
>>> President
>>> Artifact Software Inc
>>> email: rwheeler@artifact-software.com
>>> skype: ronaldmwheeler
>>> phone: 866-970-2435, ext 102
>>>
>>
Re: Intermittent Slow Down
Posted by Ian Boston <ie...@tfd.co.uk>.
Hi Morrell,
Sounds suspiciously like garbage collection activity. Rather than
attaching a profiler which may impact heap usage, try attaching a JMX
console which should be less impact (eg JConsole) and looking at the
graph of Garbage Collection activity during the slow down.
If you see it taking up a significant amount of time and CPU, try
changing the type of garbage collector[1]. IIRC the default without
the -server flag is a serial GC. Increasing heap will lengthen the
time between these slowdowns but won't eliminate them.
Also, you may be able to recreate the slowdown by forcing a full GC
operation from JConsole. Obviously if a full GC operation doesn't
leave the JVM with significantly more free heap, then you need to up
the limit to prevent the JVM spending all its time trying to free
more.
You didn't say which JVM you are using, the link below is for 6.
Adjust appropriately.
HTH
Ian
1 http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#available_collectors
On 18 December 2012 05:14, Morrell Jacobs <mj...@maned.com> wrote:
> We're running MySQL 5.5.19 for Linux
>
> We had the blobs in the database because of clustering. Since we've moved to single stack, I suppose we could pull them out - not a long term solution, but we could try and see if it helps.
>
> I'll have our MySQL experts review the settings, the restructuring may be accounting for some of the slow down. If we were seeing delays from MySQL, I wouldn't expect Tomcat's CPU usage to be pegged though - I would think all the threads would be idle, waiting for a response. Does that sound right to you?
>
> Thanks
> Morrell
>
>
> On Dec 17, 2012, at 12:48 PM, Ron Wheeler wrote:
>
>> I would play with the JVM memory configuration.
>> It still looks very small.
>> Try doubling it and see if it changes anything.
>> If not, you can look elsewhere and if it does, try making it larger until it stops improving things.
>>
>> What version of MySQL are you running?
>> Have you tuned your MySQL?
>> You might want to look at how MySQL is going to treat these objects and what configuration parameters are recommended when you have large blobs.
>> Have you thought about getting your binary content out of the database. Databases are not very good at 100Mb blobs.
>> Links to 100Mb blobs/files work much better.
>>
>> Is there a compelling reason to put the binary data inside a database?
>> Anytime MySQL has to restructure its indexes or physical storage, you are going to see a pause in the action.
>>
>> Ron
>>
>> On 17/12/2012 12:04 PM, Morrell Jacobs wrote:
>>> Hello all,
>>>
>>> We're currently experience a problem where JackRabbit function will occasionally slow down; according to our QA, the slow down comes in waves: good performance for a while (hours) then bad for a while (minutes / hours). The slow down is on the order of minutes to perform create or modify an object (creation involves creating a 2-3 levels of nodes, and setting properties). Normally these operations take 100 - 300 ms, but during the slow down they can take as much 5 minutes.
>>>
>>> The slow down does not seem to be connected to the load on the server: we've seen it perform well, when many people are connecting, then slow down with only one user. I suspect the slow down is related to indexing, but that's just a guess on my part - I'm looking for something that would happen at some interval.
>>>
>>>
>>> Environment:
>>> * JackRabbit is built into war running in Tomcat (was 7.0.25, recently upgraded to 7.0.34)
>>> * Tomcat was running with default configs but recently up'd memory - -Xms128M -Xmx512M -XX:MaxPermSize=256M
>>> * Repository XML is at end of email
>>> * JackRabbit was originally configured to be clustered, but is now running as just a single instance.
>>>
>>>
>>> Data:
>>> * Our data is structured similar to a file system, except any object can be both a file and contain other objects (files); each object has various metadata (some in properties, some in child nodes), a file and child objects
>>> * We don't use full text searching and don't want file (binary) contents indexed; the only way I was able to prevent the search index from examining the files (some are large 100+ M) was to create my own node types for files. My node types are identical nt:file and nt:content, except that they are named prd:file and prd:content; the unrecognized name seems to prevent indexing from touch them.
>>>
>>>
>>> When I've attempted to use profiling tools, it appears that during the slow down the CPU is pegged; memory usage is also at or close to the limit. In depth profiling tools (NetBeans) get overwhelmed during the slow down.
>>>
>>> We're going thru a variety of tests, but any advice the community can provide will be greatly appreciated.
>>>
>>> Here's the repository.xml:
>>> <Repository>
>>> <FileSystem class="org.apache.jackrabbit.core.fs.db.DbFileSystem">
>>> <param name="driver" value="com.mysql.jdbc.Driver"/>
>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}global_repository?autoReconnect=true&createDatabaseIfNotExist=true"/>
>>> <param name="schema" value="mysql"/>
>>> <param name="schemaObjectPrefix" value="rep_"/>
>>> <param name="user" value="${pm.database.user}"/>
>>> <param name="password" value="${pm.database.pwd}"/>
>>> </FileSystem>
>>>
>>> <!--
>>> security configuration
>>> -->
>>> <Security appName="Jackrabbit">
>>> <!--
>>> security manager:
>>> class: FQN of class implementing the JackrabbitSecurityManager interface
>>> -->
>>> <SecurityManager class="org.apache.jackrabbit.core.DefaultSecurityManager" workspaceName="default">
>>> <!--
>>> workspace access:
>>> class: FQN of class implementing the WorkspaceAccessManager interface
>>> -->
>>> <!-- <WorkspaceAccessManager class="..."/> -->
>>> <!-- <param name="config" value="${rep.home}/security.xml"/> -->
>>> </SecurityManager>
>>> <!--
>>> access manager:
>>> class: FQN of class implementing the AccessManager interface
>>> -->
>>> <AccessManager class="org.apache.jackrabbit.core.security.DefaultAccessManager">
>>> <!-- <param name="config" value="${rep.home}/access.xml"/> -->
>>> </AccessManager>
>>> <LoginModule class="org.apache.jackrabbit.core.security.authentication.DefaultLoginModule">
>>> <!--
>>> anonymous user name ('anonymous' is the default value)
>>> -->
>>> <param name="anonymousId" value="anonymous"/>
>>> <!--
>>> administrator user id (default value if param is missing is 'admin')
>>> -->
>>> <param name="adminId" value="admin"/>
>>> </LoginModule>
>>> </Security>
>>> <!--
>>> location of workspaces root directory and name of default workspace
>>> -->
>>> <Workspaces rootPath="${rep.home}/workspaces" defaultWorkspace="default"/>
>>> <!--
>>> workspace configuration template:
>>> used to create the initial workspace if there's no workspace yet
>>> -->
>>> <!-- <Workspace name="default">-->
>>> <Workspace name="default">
>>> <!--
>>> virtual file system of the workspace:
>>> class: FQN of class implementing the FileSystem interface
>>> -->
>>> <FileSystem class="org.apache.jackrabbit.core.fs.local.LocalFileSystem">
>>> <param name="path" value="${wsp.home}"/>
>>> </FileSystem>
>>> <!--
>>> persistence manager of the workspace:
>>> class: FQN of class implementing the PersistenceManager interface
>>> -->
>>> <PersistenceManager class="org.apache.jackrabbit.core.persistence.pool.MySqlPersistenceManager">
>>> <param name="driver" value="com.mysql.jdbc.Driver"/>
>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}product?createDatabaseIfNotExist=true"/>
>>> <param name="user" value="${pm.database.user}"/>
>>> <param name="password" value="${pm.database.pwd}"/>
>>> <param name="schemaObjectPrefix" value="Product_"/>
>>> <param name="schema" value="mysql"/>
>>> </PersistenceManager>
>>> <!--
>>> Search index and the file system it uses.
>>> class: FQN of class implementing the QueryHandler interface
>>> -->
>>> <SearchIndex class="org.apache.jackrabbit.core.query.lucene.SearchIndex">
>>> <param name="path" value="${wsp.home}/index"/>
>>> <!--<param name="textFilterClasses"
>>> value="org.apache.jackrabbit.extractor.PlainTextExtractor,org.apache.jackrabbit.extractor.MsWordTextExtractor,org.apache.jackrabbit.extractor.MsExcelTextExtractor,org.apache.jackrabbit.extractor.MsPowerPointTextExtractor,org.apache.jackrabbit.extractor.PdfTextExtractor,org.apache.jackrabbit.extractor.OpenOfficeTextExtractor,org.apache.jackrabbit.extractor.RTFTextExtractor,org.apache.jackrabbit.extractor.HTMLTextExtractor,org.apache.jackrabbit.extractor.XMLTextExtractor"/>
>>> <param name="extractorPoolSize" value="2"/>
>>> <param name="supportHighlighting" value="true"/> -->
>>> </SearchIndex>
>>> </Workspace>
>>> <!--
>>> Configures the versioning
>>> -->
>>> <Versioning rootPath="${rep.home}/version">
>>> <!--
>>> Configures the filesystem to use for versioning for the respective
>>> persistence manager
>>> -->
>>> <FileSystem class="org.apache.jackrabbit.core.fs.local.LocalFileSystem">
>>> <param name="path" value="${rep.home}/version"/>
>>> </FileSystem>
>>> <!--
>>> Configures the persistence manager to be used for persisting version state.
>>> Please note that the current versioning implementation is based on
>>> a 'normal' persistence manager, but this could change in future
>>> implementations.
>>> -->
>>> <PersistenceManager class="org.apache.jackrabbit.core.persistence.pool.MySqlPersistenceManager">
>>> <param name="driver" value="com.mysql.jdbc.Driver"/>
>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}truedit_versions?createDatabaseIfNotExist=true"/>
>>> <param name="user" value="${pm.database.user}"/>
>>> <param name="password" value="${pm.database.pwd}"/>
>>> <param name="schemaObjectPrefix" value="version_"/>
>>> <param name="schema" value="mysql"/>
>>> </PersistenceManager>
>>> </Versioning>
>>> <!--
>>> Configures the Data Store for large binary objects.
>>> -->
>>> <DataStore class="org.apache.jackrabbit.core.data.FileDataStore">
>>> <param name="path" value="${datastore.location.path}"/>
>>> <param name="minRecordLength" value="100"/>
>>> </DataStore>
>>> <!--
>>> <Cluster id="${cluster.id}" syncDelay="2000">
>>> <Journal class="org.apache.jackrabbit.core.journal.DatabaseJournal">
>>> <param name="driver" value="com.mysql.jdbc.Driver" />
>>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}journal?autoReconnect=true&createDatabaseIfNotExist=true"/>
>>> <param name="schemaObjectPrefix" value="journal_"/>
>>> <param name="databaseType" value="mysql"/>
>>> <param name="user" value="${pm.database.user}"/>
>>> <param name="password" value="${pm.database.pwd}"/>
>>> </Journal>
>>> </Cluster>
>>> -->
>>>
>>> </Repository>
>>>
>>>
>>>
>>> --
>>> Morrell Jacobs
>>> Chief Software Architect
>>> MEI
>>> 610 Old York Road, Suite 250
>>> Jenkintown, PA 19046
>>> Phone: 215-886-5662, ext. 252
>>> Fax: 215-886-5681
>>> http://www.maned.com
>>> E-mail: mjacobs@maned.com
>>> AOL IM: MorrellMEI
>>>
>>> Have you seen Nervous Pixel, MEI's creative services division?
>>> www.nervouspixel.com
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Ron Wheeler
>> President
>> Artifact Software Inc
>> email: rwheeler@artifact-software.com
>> skype: ronaldmwheeler
>> phone: 866-970-2435, ext 102
>>
>
Re: Intermittent Slow Down
Posted by Morrell Jacobs <mj...@maned.com>.
We're running MySQL 5.5.19 for Linux
We had the blobs in the database because of clustering. Since we've moved to single stack, I suppose we could pull them out - not a long term solution, but we could try and see if it helps.
I'll have our MySQL experts review the settings, the restructuring may be accounting for some of the slow down. If we were seeing delays from MySQL, I wouldn't expect Tomcat's CPU usage to be pegged though - I would think all the threads would be idle, waiting for a response. Does that sound right to you?
Thanks
Morrell
On Dec 17, 2012, at 12:48 PM, Ron Wheeler wrote:
> I would play with the JVM memory configuration.
> It still looks very small.
> Try doubling it and see if it changes anything.
> If not, you can look elsewhere and if it does, try making it larger until it stops improving things.
>
> What version of MySQL are you running?
> Have you tuned your MySQL?
> You might want to look at how MySQL is going to treat these objects and what configuration parameters are recommended when you have large blobs.
> Have you thought about getting your binary content out of the database. Databases are not very good at 100Mb blobs.
> Links to 100Mb blobs/files work much better.
>
> Is there a compelling reason to put the binary data inside a database?
> Anytime MySQL has to restructure its indexes or physical storage, you are going to see a pause in the action.
>
> Ron
>
> On 17/12/2012 12:04 PM, Morrell Jacobs wrote:
>> Hello all,
>>
>> We're currently experience a problem where JackRabbit function will occasionally slow down; according to our QA, the slow down comes in waves: good performance for a while (hours) then bad for a while (minutes / hours). The slow down is on the order of minutes to perform create or modify an object (creation involves creating a 2-3 levels of nodes, and setting properties). Normally these operations take 100 - 300 ms, but during the slow down they can take as much 5 minutes.
>>
>> The slow down does not seem to be connected to the load on the server: we've seen it perform well, when many people are connecting, then slow down with only one user. I suspect the slow down is related to indexing, but that's just a guess on my part - I'm looking for something that would happen at some interval.
>>
>>
>> Environment:
>> * JackRabbit is built into war running in Tomcat (was 7.0.25, recently upgraded to 7.0.34)
>> * Tomcat was running with default configs but recently up'd memory - -Xms128M -Xmx512M -XX:MaxPermSize=256M
>> * Repository XML is at end of email
>> * JackRabbit was originally configured to be clustered, but is now running as just a single instance.
>>
>>
>> Data:
>> * Our data is structured similar to a file system, except any object can be both a file and contain other objects (files); each object has various metadata (some in properties, some in child nodes), a file and child objects
>> * We don't use full text searching and don't want file (binary) contents indexed; the only way I was able to prevent the search index from examining the files (some are large 100+ M) was to create my own node types for files. My node types are identical nt:file and nt:content, except that they are named prd:file and prd:content; the unrecognized name seems to prevent indexing from touch them.
>>
>>
>> When I've attempted to use profiling tools, it appears that during the slow down the CPU is pegged; memory usage is also at or close to the limit. In depth profiling tools (NetBeans) get overwhelmed during the slow down.
>>
>> We're going thru a variety of tests, but any advice the community can provide will be greatly appreciated.
>>
>> Here's the repository.xml:
>> <Repository>
>> <FileSystem class="org.apache.jackrabbit.core.fs.db.DbFileSystem">
>> <param name="driver" value="com.mysql.jdbc.Driver"/>
>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}global_repository?autoReconnect=true&createDatabaseIfNotExist=true"/>
>> <param name="schema" value="mysql"/>
>> <param name="schemaObjectPrefix" value="rep_"/>
>> <param name="user" value="${pm.database.user}"/>
>> <param name="password" value="${pm.database.pwd}"/>
>> </FileSystem>
>>
>> <!--
>> security configuration
>> -->
>> <Security appName="Jackrabbit">
>> <!--
>> security manager:
>> class: FQN of class implementing the JackrabbitSecurityManager interface
>> -->
>> <SecurityManager class="org.apache.jackrabbit.core.DefaultSecurityManager" workspaceName="default">
>> <!--
>> workspace access:
>> class: FQN of class implementing the WorkspaceAccessManager interface
>> -->
>> <!-- <WorkspaceAccessManager class="..."/> -->
>> <!-- <param name="config" value="${rep.home}/security.xml"/> -->
>> </SecurityManager>
>> <!--
>> access manager:
>> class: FQN of class implementing the AccessManager interface
>> -->
>> <AccessManager class="org.apache.jackrabbit.core.security.DefaultAccessManager">
>> <!-- <param name="config" value="${rep.home}/access.xml"/> -->
>> </AccessManager>
>> <LoginModule class="org.apache.jackrabbit.core.security.authentication.DefaultLoginModule">
>> <!--
>> anonymous user name ('anonymous' is the default value)
>> -->
>> <param name="anonymousId" value="anonymous"/>
>> <!--
>> administrator user id (default value if param is missing is 'admin')
>> -->
>> <param name="adminId" value="admin"/>
>> </LoginModule>
>> </Security>
>> <!--
>> location of workspaces root directory and name of default workspace
>> -->
>> <Workspaces rootPath="${rep.home}/workspaces" defaultWorkspace="default"/>
>> <!--
>> workspace configuration template:
>> used to create the initial workspace if there's no workspace yet
>> -->
>> <!-- <Workspace name="default">-->
>> <Workspace name="default">
>> <!--
>> virtual file system of the workspace:
>> class: FQN of class implementing the FileSystem interface
>> -->
>> <FileSystem class="org.apache.jackrabbit.core.fs.local.LocalFileSystem">
>> <param name="path" value="${wsp.home}"/>
>> </FileSystem>
>> <!--
>> persistence manager of the workspace:
>> class: FQN of class implementing the PersistenceManager interface
>> -->
>> <PersistenceManager class="org.apache.jackrabbit.core.persistence.pool.MySqlPersistenceManager">
>> <param name="driver" value="com.mysql.jdbc.Driver"/>
>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}product?createDatabaseIfNotExist=true"/>
>> <param name="user" value="${pm.database.user}"/>
>> <param name="password" value="${pm.database.pwd}"/>
>> <param name="schemaObjectPrefix" value="Product_"/>
>> <param name="schema" value="mysql"/>
>> </PersistenceManager>
>> <!--
>> Search index and the file system it uses.
>> class: FQN of class implementing the QueryHandler interface
>> -->
>> <SearchIndex class="org.apache.jackrabbit.core.query.lucene.SearchIndex">
>> <param name="path" value="${wsp.home}/index"/>
>> <!--<param name="textFilterClasses"
>> value="org.apache.jackrabbit.extractor.PlainTextExtractor,org.apache.jackrabbit.extractor.MsWordTextExtractor,org.apache.jackrabbit.extractor.MsExcelTextExtractor,org.apache.jackrabbit.extractor.MsPowerPointTextExtractor,org.apache.jackrabbit.extractor.PdfTextExtractor,org.apache.jackrabbit.extractor.OpenOfficeTextExtractor,org.apache.jackrabbit.extractor.RTFTextExtractor,org.apache.jackrabbit.extractor.HTMLTextExtractor,org.apache.jackrabbit.extractor.XMLTextExtractor"/>
>> <param name="extractorPoolSize" value="2"/>
>> <param name="supportHighlighting" value="true"/> -->
>> </SearchIndex>
>> </Workspace>
>> <!--
>> Configures the versioning
>> -->
>> <Versioning rootPath="${rep.home}/version">
>> <!--
>> Configures the filesystem to use for versioning for the respective
>> persistence manager
>> -->
>> <FileSystem class="org.apache.jackrabbit.core.fs.local.LocalFileSystem">
>> <param name="path" value="${rep.home}/version"/>
>> </FileSystem>
>> <!--
>> Configures the persistence manager to be used for persisting version state.
>> Please note that the current versioning implementation is based on
>> a 'normal' persistence manager, but this could change in future
>> implementations.
>> -->
>> <PersistenceManager class="org.apache.jackrabbit.core.persistence.pool.MySqlPersistenceManager">
>> <param name="driver" value="com.mysql.jdbc.Driver"/>
>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}truedit_versions?createDatabaseIfNotExist=true"/>
>> <param name="user" value="${pm.database.user}"/>
>> <param name="password" value="${pm.database.pwd}"/>
>> <param name="schemaObjectPrefix" value="version_"/>
>> <param name="schema" value="mysql"/>
>> </PersistenceManager>
>> </Versioning>
>> <!--
>> Configures the Data Store for large binary objects.
>> -->
>> <DataStore class="org.apache.jackrabbit.core.data.FileDataStore">
>> <param name="path" value="${datastore.location.path}"/>
>> <param name="minRecordLength" value="100"/>
>> </DataStore>
>> <!--
>> <Cluster id="${cluster.id}" syncDelay="2000">
>> <Journal class="org.apache.jackrabbit.core.journal.DatabaseJournal">
>> <param name="driver" value="com.mysql.jdbc.Driver" />
>> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}journal?autoReconnect=true&createDatabaseIfNotExist=true"/>
>> <param name="schemaObjectPrefix" value="journal_"/>
>> <param name="databaseType" value="mysql"/>
>> <param name="user" value="${pm.database.user}"/>
>> <param name="password" value="${pm.database.pwd}"/>
>> </Journal>
>> </Cluster>
>> -->
>>
>> </Repository>
>>
>>
>>
>> --
>> Morrell Jacobs
>> Chief Software Architect
>> MEI
>> 610 Old York Road, Suite 250
>> Jenkintown, PA 19046
>> Phone: 215-886-5662, ext. 252
>> Fax: 215-886-5681
>> http://www.maned.com
>> E-mail: mjacobs@maned.com
>> AOL IM: MorrellMEI
>>
>> Have you seen Nervous Pixel, MEI's creative services division?
>> www.nervouspixel.com
>>
>>
>>
>>
>
>
> --
> Ron Wheeler
> President
> Artifact Software Inc
> email: rwheeler@artifact-software.com
> skype: ronaldmwheeler
> phone: 866-970-2435, ext 102
>
Re: Intermittent Slow Down
Posted by Ron Wheeler <rw...@artifact-software.com>.
I would play with the JVM memory configuration.
It still looks very small.
Try doubling it and see if it changes anything.
If not, you can look elsewhere and if it does, try making it larger
until it stops improving things.
What version of MySQL are you running?
Have you tuned your MySQL?
You might want to look at how MySQL is going to treat these objects and
what configuration parameters are recommended when you have large blobs.
Have you thought about getting your binary content out of the database.
Databases are not very good at 100Mb blobs.
Links to 100Mb blobs/files work much better.
Is there a compelling reason to put the binary data inside a database?
Anytime MySQL has to restructure its indexes or physical storage, you
are going to see a pause in the action.
Ron
On 17/12/2012 12:04 PM, Morrell Jacobs wrote:
> Hello all,
>
> We're currently experience a problem where JackRabbit function will occasionally slow down; according to our QA, the slow down comes in waves: good performance for a while (hours) then bad for a while (minutes / hours). The slow down is on the order of minutes to perform create or modify an object (creation involves creating a 2-3 levels of nodes, and setting properties). Normally these operations take 100 - 300 ms, but during the slow down they can take as much 5 minutes.
>
> The slow down does not seem to be connected to the load on the server: we've seen it perform well, when many people are connecting, then slow down with only one user. I suspect the slow down is related to indexing, but that's just a guess on my part - I'm looking for something that would happen at some interval.
>
>
> Environment:
> * JackRabbit is built into war running in Tomcat (was 7.0.25, recently upgraded to 7.0.34)
> * Tomcat was running with default configs but recently up'd memory - -Xms128M -Xmx512M -XX:MaxPermSize=256M
> * Repository XML is at end of email
> * JackRabbit was originally configured to be clustered, but is now running as just a single instance.
>
>
> Data:
> * Our data is structured similar to a file system, except any object can be both a file and contain other objects (files); each object has various metadata (some in properties, some in child nodes), a file and child objects
> * We don't use full text searching and don't want file (binary) contents indexed; the only way I was able to prevent the search index from examining the files (some are large 100+ M) was to create my own node types for files. My node types are identical nt:file and nt:content, except that they are named prd:file and prd:content; the unrecognized name seems to prevent indexing from touch them.
>
>
> When I've attempted to use profiling tools, it appears that during the slow down the CPU is pegged; memory usage is also at or close to the limit. In depth profiling tools (NetBeans) get overwhelmed during the slow down.
>
> We're going thru a variety of tests, but any advice the community can provide will be greatly appreciated.
>
> Here's the repository.xml:
> <Repository>
> <FileSystem class="org.apache.jackrabbit.core.fs.db.DbFileSystem">
> <param name="driver" value="com.mysql.jdbc.Driver"/>
> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}global_repository?autoReconnect=true&createDatabaseIfNotExist=true"/>
> <param name="schema" value="mysql"/>
> <param name="schemaObjectPrefix" value="rep_"/>
> <param name="user" value="${pm.database.user}"/>
> <param name="password" value="${pm.database.pwd}"/>
> </FileSystem>
>
> <!--
> security configuration
> -->
> <Security appName="Jackrabbit">
> <!--
> security manager:
> class: FQN of class implementing the JackrabbitSecurityManager interface
> -->
> <SecurityManager class="org.apache.jackrabbit.core.DefaultSecurityManager" workspaceName="default">
> <!--
> workspace access:
> class: FQN of class implementing the WorkspaceAccessManager interface
> -->
> <!-- <WorkspaceAccessManager class="..."/> -->
> <!-- <param name="config" value="${rep.home}/security.xml"/> -->
> </SecurityManager>
> <!--
> access manager:
> class: FQN of class implementing the AccessManager interface
> -->
> <AccessManager class="org.apache.jackrabbit.core.security.DefaultAccessManager">
> <!-- <param name="config" value="${rep.home}/access.xml"/> -->
> </AccessManager>
>
> <LoginModule class="org.apache.jackrabbit.core.security.authentication.DefaultLoginModule">
> <!--
> anonymous user name ('anonymous' is the default value)
> -->
> <param name="anonymousId" value="anonymous"/>
> <!--
> administrator user id (default value if param is missing is 'admin')
> -->
> <param name="adminId" value="admin"/>
> </LoginModule>
> </Security>
>
> <!--
> location of workspaces root directory and name of default workspace
> -->
> <Workspaces rootPath="${rep.home}/workspaces" defaultWorkspace="default"/>
>
> <!--
> workspace configuration template:
> used to create the initial workspace if there's no workspace yet
> -->
> <!-- <Workspace name="default">-->
> <Workspace name="default">
> <!--
> virtual file system of the workspace:
> class: FQN of class implementing the FileSystem interface
> -->
> <FileSystem class="org.apache.jackrabbit.core.fs.local.LocalFileSystem">
> <param name="path" value="${wsp.home}"/>
> </FileSystem>
> <!--
> persistence manager of the workspace:
> class: FQN of class implementing the PersistenceManager interface
> -->
> <PersistenceManager class="org.apache.jackrabbit.core.persistence.pool.MySqlPersistenceManager">
> <param name="driver" value="com.mysql.jdbc.Driver"/>
> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}product?createDatabaseIfNotExist=true"/>
> <param name="user" value="${pm.database.user}"/>
> <param name="password" value="${pm.database.pwd}"/>
> <param name="schemaObjectPrefix" value="Product_"/>
> <param name="schema" value="mysql"/>
> </PersistenceManager>
> <!--
> Search index and the file system it uses.
> class: FQN of class implementing the QueryHandler interface
> -->
> <SearchIndex class="org.apache.jackrabbit.core.query.lucene.SearchIndex">
> <param name="path" value="${wsp.home}/index"/>
> <!--<param name="textFilterClasses"
> value="org.apache.jackrabbit.extractor.PlainTextExtractor,org.apache.jackrabbit.extractor.MsWordTextExtractor,org.apache.jackrabbit.extractor.MsExcelTextExtractor,org.apache.jackrabbit.extractor.MsPowerPointTextExtractor,org.apache.jackrabbit.extractor.PdfTextExtractor,org.apache.jackrabbit.extractor.OpenOfficeTextExtractor,org.apache.jackrabbit.extractor.RTFTextExtractor,org.apache.jackrabbit.extractor.HTMLTextExtractor,org.apache.jackrabbit.extractor.XMLTextExtractor"/>
> <param name="extractorPoolSize" value="2"/>
> <param name="supportHighlighting" value="true"/> -->
> </SearchIndex>
> </Workspace>
>
> <!--
> Configures the versioning
> -->
> <Versioning rootPath="${rep.home}/version">
> <!--
> Configures the filesystem to use for versioning for the respective
> persistence manager
> -->
> <FileSystem class="org.apache.jackrabbit.core.fs.local.LocalFileSystem">
> <param name="path" value="${rep.home}/version"/>
> </FileSystem>
> <!--
> Configures the persistence manager to be used for persisting version state.
> Please note that the current versioning implementation is based on
> a 'normal' persistence manager, but this could change in future
> implementations.
> -->
> <PersistenceManager class="org.apache.jackrabbit.core.persistence.pool.MySqlPersistenceManager">
> <param name="driver" value="com.mysql.jdbc.Driver"/>
> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}truedit_versions?createDatabaseIfNotExist=true"/>
> <param name="user" value="${pm.database.user}"/>
> <param name="password" value="${pm.database.pwd}"/>
> <param name="schemaObjectPrefix" value="version_"/>
> <param name="schema" value="mysql"/>
> </PersistenceManager>
> </Versioning>
>
> <!--
> Configures the Data Store for large binary objects.
> -->
> <DataStore class="org.apache.jackrabbit.core.data.FileDataStore">
> <param name="path" value="${datastore.location.path}"/>
> <param name="minRecordLength" value="100"/>
> </DataStore>
> <!--
> <Cluster id="${cluster.id}" syncDelay="2000">
> <Journal class="org.apache.jackrabbit.core.journal.DatabaseJournal">
> <param name="driver" value="com.mysql.jdbc.Driver" />
> <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}journal?autoReconnect=true&createDatabaseIfNotExist=true"/>
> <param name="schemaObjectPrefix" value="journal_"/>
> <param name="databaseType" value="mysql"/>
> <param name="user" value="${pm.database.user}"/>
> <param name="password" value="${pm.database.pwd}"/>
> </Journal>
> </Cluster>
> -->
>
> </Repository>
>
>
>
> --
> Morrell Jacobs
> Chief Software Architect
> MEI
> 610 Old York Road, Suite 250
> Jenkintown, PA 19046
> Phone: 215-886-5662, ext. 252
> Fax: 215-886-5681
> http://www.maned.com
> E-mail: mjacobs@maned.com
> AOL IM: MorrellMEI
>
> Have you seen Nervous Pixel, MEI's creative services division?
> www.nervouspixel.com
>
>
>
>
--
Ron Wheeler
President
Artifact Software Inc
email: rwheeler@artifact-software.com
skype: ronaldmwheeler
phone: 866-970-2435, ext 102