[jira] [Assigned] (UNOMI-731) jCustomer - Geonames DB loading generates OOM on 1Go heap size JVM

["David Griffon (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/UNOMI-731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

David Griffon reassigned UNOMI-731:
-----------------------------------

    Assignee: David Griffon

> jCustomer - Geonames DB loading generates OOM on 1Go heap size JVM
> ------------------------------------------------------------------
>
>                 Key: UNOMI-731
>                 URL: https://issues.apache.org/jira/browse/UNOMI-731
>             Project: Apache Unomi
>          Issue Type: Bug
>    Affects Versions: unomi-1.6.0
>            Reporter: David Griffon
>            Assignee: David Griffon
>            Priority: Major
>             Fix For: unomi-1.8.0
>
>
> On a docker container of 4Go, with no jvm configuration regarding memory, loading of geonames generates an out of memory error.
> The main issue with this is that the base is not fully loaded after a restart
> h2. Quick investigation
> The allCountries.zip file contains one single entry of 1.5 Go data, meaning that the entire file will be loaded to ram before being processed. 
> Also, at time update, the index is removed then recreated, this could be improved by updating the documents only.
> It could be we move that data outside of ES (depending of the cost to store it in ES Cloud) in some file system database (such as ip address database) 
> h2. The fix
> Instead of processing the zip file, we will unzip the file on the filesystem (temp folder) using a system command line
> Then process the file without loading it entirely in the JVM.
> We'll check that there is enough space on the file system before uncompress the zip file. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)