You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@madlib.apache.org by LUYAO CHEN <lu...@hotmail.com> on 2018/07/27 14:41:38 UTC
Out of memory for neural network
Dear user community,
I would report a problem regarding in neural network. I am using a 16G RAM machine.
After some iterations (~100 ). I got the below error
ERROR: spiexceptions.OutOfMemory: out of memory
DETAIL: Failed on request of size 32800.
CONTEXT: Traceback (most recent call last):
PL/Python function "mlp_classification", line 36, in <module>
grouping_col
PL/Python function "mlp_classification", line 45, in wrapper
PL/Python function "mlp_classification", line 325, in mlp
PL/Python function "mlp_classification", line 580, in update
PL/Python function "mlp_classification"
Below is the command ,
SELECT madlib.mlp_classification(
'train_data_sub', -- Source table
'mlp_model', -- Destination table
'features', -- Input features
'positive', -- Label
ARRAY[5], -- Number of units per layer
'learning_rate_init=0.003,
n_iterations=500,
tolerance=0', -- Optimizer params
'tanh', -- Activation function
NULL, -- Default weight (1)
FALSE, -- No warm start
true, -- verbose
'case_icd' -- Grouping
);
Is that a problem or just caused by the data size?
Regards,
Luyao Chen
Re: Out of memory for neural network
Posted by Nandish Jayaram <nj...@pivotal.io>.
Thank you Luyao. I have created
https://issues.apache.org/jira/browse/MADLIB-1259 to track this issue.
NJ
On Fri, Jul 27, 2018 at 8:51 AM LUYAO CHEN <lu...@hotmail.com> wrote:
> Hi NJ,
>
>
> I use the same date set for the issue
> https://issues.apache.org/jira/browse/MADLIB-1257.
>
>
> It has 23K instances / 300 features in 263 groups.
>
> SELECT madlib.mlp_classification(
> 'train_data', -- Source table
> 'mlp_model', -- Destination table
> 'features', -- Input features
> 'positive', -- Label
> ARRAY[5], -- Number of units per layer
> 'learning_rate_init=0.003,
> n_iterations=500,
> tolerance=0', -- Optimizer params
> 'tanh', -- Activation function
> NULL, -- Default weight (1)
> FALSE, -- No warm start
> true, -- verbose
> 'caseid' -- Grouping
> );
>
> PostgreSQL memory setup.
>
> checkpoint_completion_target = '0.9';
> default_statistics_target = '500';
> effective_cache_size = '12GB';
> effective_io_concurrency = '200';
> maintenance_work_mem = '2GB';
> max_connections = '20';
> max_parallel_workers = '4';
> max_parallel_workers_per_gather = '2';
> max_wal_size = '8GB';
> max_worker_processes = '4';
> min_wal_size = '4GB';
> random_page_cost = '1.1';
> shared_buffers = '4GB';
> wal_buffers = '16MB';
> work_mem = '52428kB';
>
> also i setup sysctl -w vm.overcommit_memory=2 to avoide the crash of
> postmaster
>
> Regards,
> Luyao
>
>
> ------------------------------
> *From:* Nandish Jayaram <nj...@pivotal.io>
> *Sent:* Friday, July 27, 2018 10:05 AM
> *To:* user@madlib.apache.org
> *Subject:* Re: Out of memory for neural network
>
> Hi Luyao,
>
> Can you kindly include details about the database settings (btw, are you
> using Postgres or Greenplum?) and the training dataset size used?
> If your dataset is publicly available, please do share the details of
> where we can access it too.
>
> NJ
>
> Sent from my iPhone
>
> On Jul 27, 2018, at 7:41 AM, LUYAO CHEN <lu...@hotmail.com> wrote:
>
> Dear user community,
>
>
> I would report a problem regarding in neural network. I am using a 16G
> RAM machine.
>
>
> After some iterations (~100 ). I got the below error
>
>
> ERROR: spiexceptions.OutOfMemory: out of memory
> DETAIL: Failed on request of size 32800.
> CONTEXT: Traceback (most recent call last):
> PL/Python function "mlp_classification", line 36, in <module>
> grouping_col
> PL/Python function "mlp_classification", line 45, in wrapper
> PL/Python function "mlp_classification", line 325, in mlp
> PL/Python function "mlp_classification", line 580, in update
> PL/Python function "mlp_classification"
>
> Below is the command ,
>
> SELECT madlib.mlp_classification(
> 'train_data_sub', -- Source table
> 'mlp_model', -- Destination table
> 'features', -- Input features
> 'positive', -- Label
> ARRAY[5], -- Number of units per layer
> 'learning_rate_init=0.003,
> n_iterations=500,
> tolerance=0', -- Optimizer params
> 'tanh', -- Activation function
> NULL, -- Default weight (1)
> FALSE, -- No warm start
> true, -- verbose
> 'case_icd' -- Grouping
> );
>
> Is that a problem or just caused by the data size?
>
> Regards,
> Luyao Chen
>
>
Re: Out of memory for neural network
Posted by LUYAO CHEN <lu...@hotmail.com>.
Hi NJ,
I use the same date set for the issue https://issues.apache.org/jira/browse/MADLIB-1257.
It has 23K instances / 300 features in 263 groups.
SELECT madlib.mlp_classification(
'train_data', -- Source table
'mlp_model', -- Destination table
'features', -- Input features
'positive', -- Label
ARRAY[5], -- Number of units per layer
'learning_rate_init=0.003,
n_iterations=500,
tolerance=0', -- Optimizer params
'tanh', -- Activation function
NULL, -- Default weight (1)
FALSE, -- No warm start
true, -- verbose
'caseid' -- Grouping
);
PostgreSQL memory setup.
checkpoint_completion_target = '0.9';
default_statistics_target = '500';
effective_cache_size = '12GB';
effective_io_concurrency = '200';
maintenance_work_mem = '2GB';
max_connections = '20';
max_parallel_workers = '4';
max_parallel_workers_per_gather = '2';
max_wal_size = '8GB';
max_worker_processes = '4';
min_wal_size = '4GB';
random_page_cost = '1.1';
shared_buffers = '4GB';
wal_buffers = '16MB';
work_mem = '52428kB';
also i setup sysctl -w vm.overcommit_memory=2 to avoide the crash of postmaster
Regards,
Luyao
________________________________
From: Nandish Jayaram <nj...@pivotal.io>
Sent: Friday, July 27, 2018 10:05 AM
To: user@madlib.apache.org
Subject: Re: Out of memory for neural network
Hi Luyao,
Can you kindly include details about the database settings (btw, are you using Postgres or Greenplum?) and the training dataset size used?
If your dataset is publicly available, please do share the details of where we can access it too.
NJ
Sent from my iPhone
On Jul 27, 2018, at 7:41 AM, LUYAO CHEN <lu...@hotmail.com>> wrote:
Dear user community,
I would report a problem regarding in neural network. I am using a 16G RAM machine.
After some iterations (~100 ). I got the below error
ERROR: spiexceptions.OutOfMemory: out of memory
DETAIL: Failed on request of size 32800.
CONTEXT: Traceback (most recent call last):
PL/Python function "mlp_classification", line 36, in <module>
grouping_col
PL/Python function "mlp_classification", line 45, in wrapper
PL/Python function "mlp_classification", line 325, in mlp
PL/Python function "mlp_classification", line 580, in update
PL/Python function "mlp_classification"
Below is the command ,
SELECT madlib.mlp_classification(
'train_data_sub', -- Source table
'mlp_model', -- Destination table
'features', -- Input features
'positive', -- Label
ARRAY[5], -- Number of units per layer
'learning_rate_init=0.003,
n_iterations=500,
tolerance=0', -- Optimizer params
'tanh', -- Activation function
NULL, -- Default weight (1)
FALSE, -- No warm start
true, -- verbose
'case_icd' -- Grouping
);
Is that a problem or just caused by the data size?
Regards,
Luyao Chen
Re: Out of memory for neural network
Posted by Nandish Jayaram <nj...@pivotal.io>.
Hi Luyao,
Can you kindly include details about the database settings (btw, are you using Postgres or Greenplum?) and the training dataset size used?
If your dataset is publicly available, please do share the details of where we can access it too.
NJ
Sent from my iPhone
> On Jul 27, 2018, at 7:41 AM, LUYAO CHEN <lu...@hotmail.com> wrote:
>
> Dear user community,
>
> I would report a problem regarding in neural network. I am using a 16G RAM machine.
>
> After some iterations (~100 ). I got the below error
>
> ERROR: spiexceptions.OutOfMemory: out of memory
> DETAIL: Failed on request of size 32800.
> CONTEXT: Traceback (most recent call last):
> PL/Python function "mlp_classification", line 36, in <module>
> grouping_col
> PL/Python function "mlp_classification", line 45, in wrapper
> PL/Python function "mlp_classification", line 325, in mlp
> PL/Python function "mlp_classification", line 580, in update
> PL/Python function "mlp_classification"
>
> Below is the command ,
>
> SELECT madlib.mlp_classification(
> 'train_data_sub', -- Source table
> 'mlp_model', -- Destination table
> 'features', -- Input features
> 'positive', -- Label
> ARRAY[5], -- Number of units per layer
> 'learning_rate_init=0.003,
> n_iterations=500,
> tolerance=0', -- Optimizer params
> 'tanh', -- Activation function
> NULL, -- Default weight (1)
> FALSE, -- No warm start
> true, -- verbose
> 'case_icd' -- Grouping
> );
>
> Is that a problem or just caused by the data size?
>
> Regards,
> Luyao Chen
>