[jira] [Created] (MAHOUT-1766) Increase default PermGen size for spark-shell

["Sergey Tryuber (JIRA)" <ji...@apache.org>]

Sergey Tryuber created MAHOUT-1766:
--------------------------------------

             Summary: Increase default PermGen size for spark-shell
                 Key: MAHOUT-1766
                 URL: https://issues.apache.org/jira/browse/MAHOUT-1766
             Project: Mahout
          Issue Type: Improvement
          Components: Mahout spark shell
            Reporter: Sergey Tryuber


Mahout spark-shell is run with default perm gen size (64MB). Taking into account that it depends on lots of external jars and the whole count of used Java classes is very large, we constantly observe spontaneous corresponding OOM exceptions.

A hot fix from our side is to modify envelope bash script (added -XX:PermSize=512m):
{code}
"$JAVA" $JAVA_HEAP_MAX -XX:PermSize=512m $MAHOUT_OPTS -classpath "$CLASSPATH" "org.apache.mahout.sparkbindings.shell.Main" $@
{code}
Of course, more elegant solution is needed. After the applied fix, the errors had gone.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [jira] [Created] (MAHOUT-1766) Increase default PermGen size for spark-shell

[Dmitriy Lyubimov <dl...@gmail.com>]

yes i observed that too. however, it is a parameter of a specific job (and
spark version, perhaps). toy-ish problems do not cause that happen. so the
idea was that MAHOUT-OPTS should be one place to adjust it (along with
other gen sizes) to fit a particular purpose.


On Thu, Aug 13, 2015 at 8:42 AM, Sergey Tryuber (JIRA) <ji...@apache.org>
wrote:

> Sergey Tryuber created MAHOUT-1766:
> --------------------------------------
>
>              Summary: Increase default PermGen size for spark-shell
>                  Key: MAHOUT-1766
>                  URL: https://issues.apache.org/jira/browse/MAHOUT-1766
>              Project: Mahout
>           Issue Type: Improvement
>           Components: Mahout spark shell
>             Reporter: Sergey Tryuber
>
>
> Mahout spark-shell is run with default perm gen size (64MB). Taking into
> account that it depends on lots of external jars and the whole count of
> used Java classes is very large, we constantly observe spontaneous
> corresponding OOM exceptions.
>
> A hot fix from our side is to modify envelope bash script (added
> -XX:PermSize=512m):
> {code}
> "$JAVA" $JAVA_HEAP_MAX -XX:PermSize=512m $MAHOUT_OPTS -classpath
> "$CLASSPATH" "org.apache.mahout.sparkbindings.shell.Main" $@
> {code}
> Of course, more elegant solution is needed. After the applied fix, the
> errors had gone.
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>