[jira] [Commented] (OFBIZ-10592) OutOfMemory and stucked JobPoller issue

["Giulio Speri (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OFBIZ-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16859466#comment-16859466 ] 

Giulio Speri commented on OFBIZ-10592:
--------------------------------------

Hi everyone,

sorry for the latency, but I can upload the patch files that make the service avoid the OOM problems.

Just few notes on the solution I have thought:
 * shopping list are looped and processed at 1000 at a time; potentially, on very large amount of lists the service could run for quite a lot. To avoid this, I used an internal timer, with the same value of the service transaction timeout, that is checked for expiration at the end of each block of SL processed.Doing this way, if the timer expires in the middle of a processing block, the service won't stop immediately, but it will finish to loop the block.
 * I noticed that if a small timer is configured in the service definition (I tested 10 secs), when the timer expires and the parent transaction is resumed, the transaction timeout will light up.Also in this case, the first block of 1000 SL is processed anyway;
 * The number of SL retrieved at a time is configurable in the order.properties file.

I already installed this solution on our customer's servers and it is doing it's work; I just reduced the days of the validity of the SL down to 5.

Let me know if you encounter problems on applying the patch files.

[^ShoppingListServices.patch]

[^order_properties.patch]

 

 

 

> OutOfMemory and stucked JobPoller issue
> ---------------------------------------
>
>                 Key: OFBIZ-10592
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-10592
>             Project: OFBiz
>          Issue Type: Bug
>          Components: ALL APPLICATIONS
>    Affects Versions: Release Branch 13.07
>            Reporter: Giulio Speri
>            Assignee: Giulio Speri
>            Priority: Critical
>         Attachments: Screenshot from 2019-04-20 02-32-37.png, ShoppingListServices.patch, ShoppingListServices.patch, alloc_tree_600k_12102018.png, jvm_ofbiz1_profi_telem.png, jvm_prof_ofbiz1_telem2.png, ofbiz1_jvm_profil_nojobpoller.png, order_properties.patch, recorder_object_600k_12102018.png, telemetry_ovrl_600k_12102018.png
>
>
>  
> This installation is composed by two instances of OFBiz (v13.07.03), served via an Apache Tomcat webserver, along with a load balancer.
> The database server is MariaDB.
>  
> We had the first problems, about 3 weeks ago, when suddenly, the front1 (ofbiz instance 1), stopped serving web requests; front2, instead, was still working correctly.
>  
> Obviously we checked the log files, and we saw that async services were failing; the failure was accompanied by this error line:
>  
> *_Thread "AsyncAppender-async" java.lang.OutOfMemoryError: GC overhead limit exceeded_*
>  
> We analyzed the situation with our system specialists, and they told us that the application was highly stressing machine resources (cpu always at or near 100%, RAM usage rapidly increasing), until the jvm run out of memory.
> This "resource-high-consumption situation", occurred only when ofbiz1 instance was started with the JobPoller enabled; if the JobPoller was not enabled, ofbiz run with low resource usage. 
>  
> We then focused on the db, to check first of all the dimensions; the result was disconcerting; 45GB, mainly divided on four tables: SERVER_HIT (about 18 GB), VISIT (about 15 GB), ENTITY_SYNC_REMOVE (about 8 GB), VISITOR (about 2 GB).
> All the other tables had a size in the order of few MB each.
>  
> The first thing we did, was to clear all those tables, reducing considerably the db size.
> After the cleaning, we tried to start ofbiz1 again, with the JobPoller component enabled; this caused a lot of old scheduled/queued jobs, to execute.
> Except than for the start-up time, the resource usage of the machine, stabilized around normal to low values (cpu 1-10%).
> Ofbiz seemed to work (web request was served), but we noticed that the JobPoller did not schedule or run jobs, anymore. 
> The number of job in "Pending" state in the JobSandbox entity was small (about 20); no Queued, no Failed, no jobs in other states.
> In addition to this, unfortunately, after few hours, jvm run out of memory again.
>  
> Our jvm has an heap maximum size of 20GB ( we have 32GB on the  machine), so it's not so small, I think.
> The next step we're going to do is set-up locally the application over the same production db to see what happens.
>  
> Now that I explained the situation, I am going to ask if, in your opinion/experience:
>  
> Could the JobPoller component be the root (and only) cause of the OutOfMemory of the jvm?
>  
> Could this issue be related to OFBIZ-5710?
>  
> Dumping and analyzing the heap of the jvm could help in some way to understand what or who fills the memory or is this operation a waste of time?
>  
> Is there something that we did not considered or missed during the whole process of problem analysis?
>  
>  
> I really thank you all for your attention and your help; any suggestion or advice would really be greatly appreciated.
>  
> Kind regards,
> Giulio



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)