You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@hop.apache.org by "00114100 (via GitHub)" <gi...@apache.org> on 2023/03/31 09:37:44 UTC

[GitHub] [hop] 00114100 opened a new issue, #2796: [Bug]: Hop Server's Memory usage keeps increasing inspite of setting object_timeout_minutes in sever config

00114100 opened a new issue, #2796:
URL: https://github.com/apache/hop/issues/2796

   ### Apache Hop version?
   
   2.3.0
   
   ### Java version?
   
   openjdk version "11.0.18" 2023-01-17 LTS
   
   ### Operating system
   
   Windows
   
   ### What happened?
   
   ## Procedure
   - Define the pipleline as following: (longRun.hpl)
   ![image](https://user-images.githubusercontent.com/82439164/229048797-fe881e36-cf42-403b-84b1-f7bfd5db4710.png)
   DataGrid hast 1000 lines. This sample is writting them to log.
   
   - Define the webservice as following (Test01.json)
   ![image](https://user-images.githubusercontent.com/82439164/229050717-1161b807-2a30-4df3-8688-63d8976d2386.png)
   
   - Define the hop server config.
   ` <object_timeout_minutes>1</object_timeout_minutes>`
   
   - Run the hop server. Set the max memory with  HOP_OPTIONS=-Xmx300m for the test.
   `hopserver.bat hopsever.xml`
   ![image](https://user-images.githubusercontent.com/82439164/229083141-a3ccc82d-6097-4c98-aa31-7a0b9ebafe45.png)
   
   - Call the rest api of the web service every 5 seconds with the following bat file.
   ```
   :top
   timeout 5
   curl --anyauth --user cluster:cluster -X GET "http://localhost:8080/hop/webService/?service=Test01
   goto top
   ```
   - Hop Server's Memory usage keeps increasing. Heap: Out of Memory occured.
   ![image](https://user-images.githubusercontent.com/82439164/229062098-809d06ba-e183-48f4-b0d0-f4cc85f1ff83.png)
   
   ![image](https://user-images.githubusercontent.com/82439164/229067372-0fda24a5-baa3-4af8-847b-880f0416f8ff.png)
   Hop Server
   ![image](https://user-images.githubusercontent.com/82439164/229066641-eda35ce7-bd22-499b-9e60-828207f55001.png)
   Curl Command
   
   ## Expected results
   Our use case would like to repeat calling web service APIs under the certain amount of memory usage. Because this system calls rest API of web service frequently in the short period (For example: every 1 minutes)
   
   ## Workaround
   
   - After calling the rest api of web service, call the remove pipeline.
   ```
   :top
   timeout 5
   curl --anyauth --user cluster:cluster -X GET "http://localhost:8080/hop/webService/?service=Test01
   curl --anyauth --user cluster:cluster -X GET http://localhost:8080/hop/removePipeline/?name=longRun
   goto top
   ```
   removePipeline should be specified with the id of pipeline. But it works for this case to just delete the object of the pipeline.
   
   ![image](https://user-images.githubusercontent.com/82439164/229081790-81b5018f-a4e7-4572-beb6-ddbd05782b8e.png)
   
   ## Hypothesis
   There is the difference between removePipeline and  setting of object_timeout_minutes for discarding the log objects.
   
   https://github.com/apache/hop/blob/7aa99530306a8bf3318288e2704bf907214d46d8/engine/src/main/java/org/apache/hop/www/RemovePipelineServlet.java
   ![image](https://user-images.githubusercontent.com/82439164/229077657-5af43ff4-fa3a-453b-9d31-2d09547ba5c6.png)
   
   https://github.com/apache/hop/blob/f76ac0eb75fd0570444d69766cab07e58bb00247/engine/src/main/java/org/apache/hop/www/HopServerSingleton.java
   
   ![image](https://user-images.githubusercontent.com/82439164/229080107-e846817f-1a46-44ac-8b99-2019ff0108ed.png)
   
   The second argument of HopLogStore.discardLines funtion is different. If the behaviour of  removePipeline is criterion, should be true? Or my setting is not enough?
   
   ## Attached files
   [sample.zip](https://github.com/apache/hop/files/11120401/sample.zip)
   
   
   ### Issue Priority
   
   Priority: 1
   
   ### Issue Component
   
   Component: Hop Server


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hop] hansva commented on issue #2796: [Bug]: Hop Server's Memory usage keeps increasing inspite of setting object_timeout_minutes in sever config

Posted by "hansva (via GitHub)" <gi...@apache.org>.

hansva commented on issue #2796:
URL: https://github.com/apache/hop/issues/2796#issuecomment-1494502997

   .take-issue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hop] hansva commented on issue #2796: [Bug]: Hop Server's Memory usage keeps increasing inspite of setting object_timeout_minutes in sever config

Posted by "hansva (via GitHub)" <gi...@apache.org>.

hansva commented on issue #2796:
URL: https://github.com/apache/hop/issues/2796#issuecomment-1494502797

   thanks for the update @00114100 ! I'll take a look ASAP


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hop] 00114100 commented on issue #2796: [Bug]: Hop Server's Memory usage keeps increasing inspite of setting object_timeout_minutes in sever config

Posted by "00114100 (via GitHub)" <gi...@apache.org>.

00114100 commented on issue #2796:
URL: https://github.com/apache/hop/issues/2796#issuecomment-1494158696

   I would like to add the other result as following. The maximum age of a log line is also 1 minutes.
   ![image](https://user-images.githubusercontent.com/82439164/229497080-9849d547-be2c-4511-8885-3540c619e6ff.png)
   
   ![image](https://user-images.githubusercontent.com/82439164/229497284-9eddb68d-b436-44d5-b39f-b435d4767dd0.png)
   
   ![image](https://user-images.githubusercontent.com/82439164/229497597-4de8674e-d7e6-467e-b335-74dadce6fb7e.png)
   
   Hop Server's Memory usage keeps increasing. Heap: Out of Memory occured.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug]: Hop Server's Memory usage keeps increasing inspite of setting object_timeout_minutes in sever config (hop)

Posted by "SzymonZy (via GitHub)" <gi...@apache.org>.

SzymonZy commented on issue #2796:
URL: https://github.com/apache/hop/issues/2796#issuecomment-3085308189

   Is there any fix for that? I see momory increasing by running any new workflow or pipeline. We set removal of completed workflows via object_timeout_minutes and max_log_timeout_minutes to 120 minutes but hop 2.13 and 2.14 seems to ignore it under linux. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug]: Hop Server's Memory usage keeps increasing inspite of setting object_timeout_minutes in sever config (hop)

Posted by "bamaer (via GitHub)" <gi...@apache.org>.

bamaer commented on issue #2796:
URL: https://github.com/apache/hop/issues/2796#issuecomment-1802590653

   @00114100 tried to reproduce this (after waaaay too long) with your sample in 2.7.0-SNAPSHOT, but couldn't. 
   Nothing in the code seems to have changed, I used your sample and memory settings (300MB), but seems to behave just fine. 
   
   Would you mind checking if this issue still occurs in a recent release and let us know? 
   
   ![image](https://github.com/apache/hop/assets/948257/e44670f7-8928-437e-a309-0dd2c93f6a78)
   
   ![image](https://github.com/apache/hop/assets/948257/e5b1c4c1-29c3-4a64-bcca-dd0e39f66e91)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug]: Hop Server's Memory usage keeps increasing inspite of setting object_timeout_minutes in sever config (hop)

Posted by "bamaer (via GitHub)" <gi...@apache.org>.

bamaer commented on issue #2796:
URL: https://github.com/apache/hop/issues/2796#issuecomment-1802721961

   update: reproduced with a loop for 1000 runs


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug]: Hop Server's Memory usage keeps increasing inspite of setting object_timeout_minutes in sever config (hop)

Posted by "hansva (via GitHub)" <gi...@apache.org>.

hansva closed issue #2796: [Bug]: Hop Server's Memory usage keeps increasing inspite of setting object_timeout_minutes in sever config
URL: https://github.com/apache/hop/issues/2796


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org