[jira] [Created] (UNOMI-731) jCustomer - Geonames DB loading generates OOM on 1Go heap size JVM

["David Griffon (Jira)" <ji...@apache.org>]

David Griffon created UNOMI-731:
-----------------------------------

             Summary: jCustomer - Geonames DB loading generates OOM on 1Go heap size JVM
                 Key: UNOMI-731
                 URL: https://issues.apache.org/jira/browse/UNOMI-731
             Project: Apache Unomi
          Issue Type: Bug
            Reporter: David Griffon


On a docker container of 4Go, with no jvm configuration regarding memory, loading of geonames generates an out of memory error.

The main issue with this is that the base is not fully loaded after a restart

h2. Quick investigation
The allCountries.zip file contains one single entry of 1.5 Go data, meaning that the entire file will be loaded to ram before being processed. 

Also, at time update, the index is removed then recreated, this could be improved by updating the documents only.

It could be we move that data outside of ES (depending of the cost to store it in ES Cloud) in some file system database (such as ip address database) 

h2. The fix
Instead of processing the zip file, we will unzip the file on the filesystem (temp folder) using a system command line
Then process the file without loading it entirely in the JVM.
We'll check that there is enough space on the file system before uncompress the zip file. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)