OutOfMemoryError - Java heap space

[Sami Serbey <sa...@designer-24.com>]

Hello,

I am running 3 predicition io applications on a single machine (4vCPU & 16GB RAM) where I feed the engines with data on a daily basis and then train them and deploy them.

I have 2 engines that work on the universal recommender template and the third one use the similarProduct template.

The first two applications running on the universal recommender were deployed 5 months ago, and everything was running smoothly. There were no errors and no bugs. 5 days ago, we deployed the third application running the similarProduct template. Right after that deployment, we started noticing a malfunction in the first two applications. In fact, when sending a request with some input to the recommender, the recommender responds with correct data, but generates a timeout error for requests with different input with the following message (The server was not able to produce a timely response to your request.)

This message only shows up for particular users input. In more details, for specific input, the recommender always replies with data, while for other input, the recommender always generates a timeout error. When the error occurs, I get the following error log on the machine:

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOfRange(Arrays.java:3664)
        at java.lang.String.<init>(String.java:207)
        at java.lang.StringBuilder.toString(StringBuilder.java:407)
        at scala.collection.mutable.StringBuilder.toString(StringBuilder.scala:430)
        at scala.collection.TraversableOnce$class.mkString(TraversableOnce.scala:323)
        at scala.collection.immutable.Stream.toString(Stream.scala:823)
        at java.lang.String.valueOf(String.java:2994)
        at java.lang.StringBuilder.append(StringBuilder.java:131)
        at scala.StringContext.standardInterpolator(StringContext.scala:125)
        at scala.StringContext.s(StringContext.scala:95)
        at com.actionml.URAlgorithm$$anonfun$buildQuery$3.apply(URAlgorithm.scala:576)
        at com.actionml.URAlgorithm$$anonfun$buildQuery$3.apply(URAlgorithm.scala:576)
        at grizzled.slf4j.Logger.info(slf4j.scala:128)
        at com.actionml.URAlgorithm.buildQuery(URAlgorithm.scala:576)
        at com.actionml.URAlgorithm.predict(URAlgorithm.scala:488)
        at com.actionml.URAlgorithm.predict(URAlgorithm.scala:180)
        at org.apache.predictionio.controller.P2LAlgorithm.predictBase(P2LAlgorithm.scala:73)

Note that we have monitored the CPU, memory, and disk metrics of the machine. Everything seems working well. We have a lot of free memory and disk space, and the CPU utilization is in normal levels.

Is there anyway I can fix or debug what is happening? Any help would be greatly appreciated.

Regards,
Sami Serbey