You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@predictionio.apache.org by Amado Espinosa <am...@odegiecommerce.com> on 2016/10/30 14:42:06 UTC
Fwd: Issue while training Recommendation Engine
Hi, I stuck trying to train the Recommendation Engine. The issue seems to
be related with
[WARN] [BLAS] Failed to load implementation from: com.github.fommil.netlib.
NativeSystemBLAS
[WARN] [BLAS] Failed to load implementation from: com.github.fommil.netlib.
NativeRefBLAS
[WARN] [LAPACK] Failed to load implementation from:
com.github.fommil.netlib.NativeSystemLAPACK
[WARN] [LAPACK] Failed to load implementation from:
com.github.fommil.netlib.NativeRefLAPACK
Warnings, I googled and I find some tips like
To fix this, you need to download the source code, and build it yourself
passing the “-Pnetlib-lgpl” option to maven.
I've did that using this command
./make-distribution.sh --name hadoop-2.6 --tgz -Phadoop-2.6 -Pyarn
-Pnetlib-lgpl -DskipTests
but still I;m getting the same issue ..
I attached the log, do you have any guideline to avoid this issue? I'm miss
some steps during the installation/setup?
Thanks for your help,
Best Regards.
Re: Issue while training Recommendation Engine
Posted by Donald Szeto <do...@apache.org>.
Hi Amado,
PredictionIO does not officially support Spark 2.0 yet, and I have not
personally tried the latest version of MLlib, so I cannot comment on
whether it has gotten better with the same number of iterations. My guess
is that it would be similar if you don't specify a non-default JVM heap
memory (default is half a GB).
If you are testing locally, you can use spark-submit bypass arguments
(explained in "pio help"), e.g.
pio train -- --driver-memory 2g
Regards,
Donald
On Mon, Oct 31, 2016 at 1:58 PM, Amado Espinosa <am...@odegiecommerce.com>
wrote:
> Hi, thanks you very much for your help. We recently increase our server
> memory until 12GB but I still doing the predictionio setup. I realize that
> your manual install reference recommends spark-1.5.1 and Hadoop 2.6 ..
> there are newer versions of Spark and also a new 2.0 branch ... Can I
> expect better results with this implementation?? I really appreciate any
> comments about that.
>
> Do you have any reference or documentation to try Increase Spark driver
> memory locally??
>
> Best Regards,
>
> On Mon, Oct 31, 2016 at 12:51 PM, Donald Szeto <do...@apache.org> wrote:
>
>> Hi Amado,
>>
>> You have several options:
>>
>> 1. Train on a real Spark cluster with more executor memory.
>> 2. Increase Spark driver memory if you want to test locally.
>> 3. Reduce number of iterations in engine.json.
>>
>> Regards,
>> Donald
>>
>> On Sun, Oct 30, 2016 at 7:42 AM, Amado Espinosa <amado@odegiecommerce.com
>> > wrote:
>>
>>>
>>> Hi, I stuck trying to train the Recommendation Engine. The issue seems
>>> to be related with
>>>
>>> [WARN] [BLAS] Failed to load implementation from:
>>> com.github.fommil.netlib.NativeSystemBLAS
>>> [WARN] [BLAS] Failed to load implementation from:
>>> com.github.fommil.netlib.NativeRefBLAS
>>> [WARN] [LAPACK] Failed to load implementation from:
>>> com.github.fommil.netlib.NativeSystemLAPACK
>>> [WARN] [LAPACK] Failed to load implementation from:
>>> com.github.fommil.netlib.NativeRefLAPACK
>>>
>>> Warnings, I googled and I find some tips like
>>>
>>> To fix this, you need to download the source code, and build it yourself
>>> passing the “-Pnetlib-lgpl” option to maven.
>>>
>>> I've did that using this command
>>>
>>> ./make-distribution.sh --name hadoop-2.6 --tgz -Phadoop-2.6 -Pyarn
>>> -Pnetlib-lgpl -DskipTests
>>>
>>> but still I;m getting the same issue ..
>>> I attached the log, do you have any guideline to avoid this issue? I'm
>>> miss some steps during the installation/setup?
>>>
>>> Thanks for your help,
>>> Best Regards.
>>>
>>>
>>
>
Re: Issue while training Recommendation Engine
Posted by Amado Espinosa <am...@odegiecommerce.com>.
Hi, thanks you very much for your help. We recently increase our server
memory until 12GB but I still doing the predictionio setup. I realize that
your manual install reference recommends spark-1.5.1 and Hadoop 2.6 ..
there are newer versions of Spark and also a new 2.0 branch ... Can I
expect better results with this implementation?? I really appreciate any
comments about that.
Do you have any reference or documentation to try Increase Spark driver
memory locally??
Best Regards,
On Mon, Oct 31, 2016 at 12:51 PM, Donald Szeto <do...@apache.org> wrote:
> Hi Amado,
>
> You have several options:
>
> 1. Train on a real Spark cluster with more executor memory.
> 2. Increase Spark driver memory if you want to test locally.
> 3. Reduce number of iterations in engine.json.
>
> Regards,
> Donald
>
> On Sun, Oct 30, 2016 at 7:42 AM, Amado Espinosa <am...@odegiecommerce.com>
> wrote:
>
>>
>> Hi, I stuck trying to train the Recommendation Engine. The issue seems to
>> be related with
>>
>> [WARN] [BLAS] Failed to load implementation from:
>> com.github.fommil.netlib.NativeSystemBLAS
>> [WARN] [BLAS] Failed to load implementation from:
>> com.github.fommil.netlib.NativeRefBLAS
>> [WARN] [LAPACK] Failed to load implementation from:
>> com.github.fommil.netlib.NativeSystemLAPACK
>> [WARN] [LAPACK] Failed to load implementation from:
>> com.github.fommil.netlib.NativeRefLAPACK
>>
>> Warnings, I googled and I find some tips like
>>
>> To fix this, you need to download the source code, and build it yourself
>> passing the “-Pnetlib-lgpl” option to maven.
>>
>> I've did that using this command
>>
>> ./make-distribution.sh --name hadoop-2.6 --tgz -Phadoop-2.6 -Pyarn
>> -Pnetlib-lgpl -DskipTests
>>
>> but still I;m getting the same issue ..
>> I attached the log, do you have any guideline to avoid this issue? I'm
>> miss some steps during the installation/setup?
>>
>> Thanks for your help,
>> Best Regards.
>>
>>
>
Re: Issue while training Recommendation Engine
Posted by Donald Szeto <do...@apache.org>.
Hi Amado,
You have several options:
1. Train on a real Spark cluster with more executor memory.
2. Increase Spark driver memory if you want to test locally.
3. Reduce number of iterations in engine.json.
Regards,
Donald
On Sun, Oct 30, 2016 at 7:42 AM, Amado Espinosa <am...@odegiecommerce.com>
wrote:
>
> Hi, I stuck trying to train the Recommendation Engine. The issue seems to
> be related with
>
> [WARN] [BLAS] Failed to load implementation from:
> com.github.fommil.netlib.NativeSystemBLAS
> [WARN] [BLAS] Failed to load implementation from:
> com.github.fommil.netlib.NativeRefBLAS
> [WARN] [LAPACK] Failed to load implementation from:
> com.github.fommil.netlib.NativeSystemLAPACK
> [WARN] [LAPACK] Failed to load implementation from:
> com.github.fommil.netlib.NativeRefLAPACK
>
> Warnings, I googled and I find some tips like
>
> To fix this, you need to download the source code, and build it yourself
> passing the “-Pnetlib-lgpl” option to maven.
>
> I've did that using this command
>
> ./make-distribution.sh --name hadoop-2.6 --tgz -Phadoop-2.6 -Pyarn
> -Pnetlib-lgpl -DskipTests
>
> but still I;m getting the same issue ..
> I attached the log, do you have any guideline to avoid this issue? I'm
> miss some steps during the installation/setup?
>
> Thanks for your help,
> Best Regards.
>
>