You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@joshua.apache.org by thammegowda <gi...@git.apache.org> on 2018/03/06 01:16:28 UTC
[GitHub] incubator-joshua pull request #81: Fix: memory overflow in tuning with large...
GitHub user thammegowda opened a pull request:
https://github.com/apache/incubator-joshua/pull/81
Fix: memory overflow in tuning with larger language models
tuner needs more memory when larger language models are used.
Even though we allocate more memory in the training pipeline script
the value is ignored at one place in pipeline.
This PR fixes it by exporting env var, and passing it to the JVM at the right place
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/thammegowda/incubator-joshua master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-joshua/pull/81.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #81
----
commit cad86aa7512f2829c8a964f2ccc3f69c7cd2f1c7
Author: Thamme Gowda <tg...@...>
Date: 2018-03-06T01:13:27Z
Fix: memory overflow in tuning
when larger language models are used, the tuner needs more memory.
Even though we allocate more memory in the training pipeline script
the value is ignored at one place in pipeline.
This patch fixes it by exporting env var, and passing it to the JVM
----
---
[GitHub] incubator-joshua issue #81: Fix: memory overflow in tuning with larger langu...
Posted by thammegowda <gi...@git.apache.org>.
Github user thammegowda commented on the issue:
https://github.com/apache/incubator-joshua/pull/81
Thanks for the tips @KellenSunderland . There is plenty of RAM on server so I should let KenLM use more RAM for indexing instead of IO. I will use Java Flight recorder to get more details.
---
[GitHub] incubator-joshua issue #81: Fix: memory overflow in tuning with larger langu...
Posted by tteofili <gi...@git.apache.org>.
Github user tteofili commented on the issue:
https://github.com/apache/incubator-joshua/pull/81
+1, thanks TG!
---
[GitHub] incubator-joshua issue #81: Fix: memory overflow in tuning with larger langu...
Posted by KellenSunderland <gi...@git.apache.org>.
Github user KellenSunderland commented on the issue:
https://github.com/apache/incubator-joshua/pull/81
Cool thanks @thammegowda. One thing to be aware of is that as you increase JVM memory you effectively take away memory from KenLM. What I've done in the past is just run a quick Java Flight Recorder session to verify the issue is excess garbage collection. If it it then increasing jvm args is the way to go. Otherwise you may want to take a look at how much IO is being caused by KenLM, and what mode of memory allocation it's using.
---
[GitHub] incubator-joshua pull request #81: Fix: memory overflow in tuning with large...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-joshua/pull/81
---
[GitHub] incubator-joshua issue #81: Fix: memory overflow in tuning with larger langu...
Posted by chrismattmann <gi...@git.apache.org>.
Github user chrismattmann commented on the issue:
https://github.com/apache/incubator-joshua/pull/81
+1 from me
---
[GitHub] incubator-joshua issue #81: Fix: memory overflow in tuning with larger langu...
Posted by KellenSunderland <gi...@git.apache.org>.
Github user KellenSunderland commented on the issue:
https://github.com/apache/incubator-joshua/pull/81
@thammegowda Are you using a kenlm language model?
---
[GitHub] incubator-joshua issue #81: Fix: memory overflow in tuning with larger langu...
Posted by thammegowda <gi...@git.apache.org>.
Github user thammegowda commented on the issue:
https://github.com/apache/incubator-joshua/pull/81
I am trying to run an experiment with a bunch of big language models, but the tuner is taking forever!
In the code base, I found a few more (possible) bottlenecks:
1. https://github.com/apache/incubator-joshua/blob/000298e555fbc71315b1d8719f5c3918a2102e5b/scripts/training/run_tuner.py#L421
2. https://github.com/apache/incubator-joshua/blob/a30e95563a20e2ccad574f4065654b955fe8fa25/src/main/java/org/apache/joshua/zmert/ZMERT.java#L61
So, 4000 MB of heap is hardcoded to the Mert.
Possible explanation: My language models are huge (a big one is ~90GB), they definitely don't fit into 4GB, so JVM is spending all the time in garbage collection.
---
[GitHub] incubator-joshua issue #81: Fix: memory overflow in tuning with larger langu...
Posted by thammegowda <gi...@git.apache.org>.
Github user thammegowda commented on the issue:
https://github.com/apache/incubator-joshua/pull/81
@KellenSunderland Yes Kenlm is used for generating a language model from the parallel corpus.
In addition, I am including two more precooked models in ARPA format for the target language(copied from other MT system).
[This wiki page is helpful](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65871630#TheJoshuaPipeline(6.1)-Languagemodel)
---
[GitHub] incubator-joshua issue #81: Fix: memory overflow in tuning with larger langu...
Posted by chrismattmann <gi...@git.apache.org>.
Github user chrismattmann commented on the issue:
https://github.com/apache/incubator-joshua/pull/81
I will commit this if I don't see any comments in the next 24 hours
---