Re: Issue while training Recommendation Engine

[Donald Szeto <do...@apache.org>]

Hi Amado,

PredictionIO does not officially support Spark 2.0 yet, and I have not
personally tried the latest version of MLlib, so I cannot comment on
whether it has gotten better with the same number of iterations. My guess
is that it would be similar if you don't specify a non-default JVM heap
memory (default is half a GB).

If you are testing locally, you can use spark-submit bypass arguments
(explained in "pio help"), e.g.

pio train -- --driver-memory 2g

Regards,
Donald

On Mon, Oct 31, 2016 at 1:58 PM, Amado Espinosa <am...@odegiecommerce.com>
wrote:

> Hi, thanks you very much for your help. We recently increase our server
> memory until 12GB but I still doing the predictionio setup. I realize that
> your manual install reference recommends spark-1.5.1 and Hadoop 2.6 ..
> there are newer versions of Spark and also a new 2.0 branch ... Can I
> expect better results with this implementation?? I really appreciate any
> comments about that.
>
> Do you have any reference or documentation to try Increase Spark driver
> memory locally??
>
> Best Regards,
>
> On Mon, Oct 31, 2016 at 12:51 PM, Donald Szeto <do...@apache.org> wrote:
>
>> Hi Amado,
>>
>> You have several options:
>>
>> 1. Train on a real Spark cluster with more executor memory.
>> 2. Increase Spark driver memory if you want to test locally.
>> 3. Reduce number of iterations in engine.json.
>>
>> Regards,
>> Donald
>>
>> On Sun, Oct 30, 2016 at 7:42 AM, Amado Espinosa <amado@odegiecommerce.com
>> > wrote:
>>
>>>
>>> Hi, I stuck trying to train the Recommendation Engine. The issue seems
>>> to be related with
>>>
>>> [WARN] [BLAS] Failed to load implementation from:
>>> com.github.fommil.netlib.NativeSystemBLAS
>>> [WARN] [BLAS] Failed to load implementation from:
>>> com.github.fommil.netlib.NativeRefBLAS
>>> [WARN] [LAPACK] Failed to load implementation from:
>>> com.github.fommil.netlib.NativeSystemLAPACK
>>> [WARN] [LAPACK] Failed to load implementation from:
>>> com.github.fommil.netlib.NativeRefLAPACK
>>>
>>> Warnings, I googled and I find some tips like
>>>
>>> To fix this, you need to download the source code, and build it yourself
>>> passing the “-Pnetlib-lgpl” option to maven.
>>>
>>> I've did that using this command
>>>
>>> ./make-distribution.sh --name hadoop-2.6 --tgz -Phadoop-2.6 -Pyarn
>>> -Pnetlib-lgpl -DskipTests
>>>
>>> but still I;m getting the same issue ..
>>> I attached the log, do you have any guideline to avoid this issue? I'm
>>> miss some steps during the installation/setup?
>>>
>>> Thanks for your help,
>>>   Best Regards.
>>>
>>>
>>
>