You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@madlib.apache.org by Satoshi Nagayasu <sn...@uptime.jp> on 2016/08/07 08:50:31 UTC
create_indicator_variables() with svec?
Hi,
I'm trying create_indicator_variables() to encode categorical variables.
https://madlib.incubator.apache.org/docs/latest/group__grp__data__prep.html
And I found that PostgreSQL had a limitation of maximum number of variables
in SELECT list (called target list in PostgreSQL), up to 1664.
You may see this error when you have more than 1664 categories in your variable.
spiexceptions.ProgramLimitExceeded: target lists can have at most 1664 entries
Now, I'm considering using PostgreSQL arrays to contain indicators instead of
allocating single column per category.
If create_indicator_variables() supports arrays as its output, it
allows us to deal
with categorical variables which have more than 1664 categories.
And of course, I would like to use the sparse vector for it to compress them.
https://madlib.incubator.apache.org/docs/latest/group__grp__svec.html
Seems good to you? Any comments?
Regards,
--
Satoshi Nagayasu <sn...@uptime.jp>
Re: create_indicator_variables() with svec?
Posted by Rahul Iyer <ri...@pivotal.io>.
Thanks, Satoshi.
Created feature request (MADLIB-1013
<https://issues.apache.org/jira/browse/MADLIB-1013>), this will probably go
into 1.9.2 since we've already started the release process for 1.9.1.
On Mon, Aug 8, 2016 at 7:21 PM, Satoshi Nagayasu <sn...@uptime.jp> wrote:
> Hi Rahul,
>
> 2016-08-09 2:05 GMT+09:00 Rahul Iyer <ri...@pivotal.io>:
> > Array output for *create_indicator_variables* would be quite helpful when
> > number of categories is large and the svec representation would be ideal
> > for it. There might be similar implications for *pivoting*, but we can
> keep
> > that as future discussion.
>
> Sounds great.
>
> > I'm curious about how you're using the indicator variables - svec is not
> > widely supported in MADlib (yet) and might not give much benefit after
> the
> > encoding is complete.
>
> I'm trying to implement some recommendation or similarity search stuff
> for several media items (movies, books, documents, else) with its metadata.
> It has several categorical variables, such as authors, publishers,
> actors/actresses, genres, else. Some of them have many categories.
>
> BTW, I'm a starter of data-mining and machine-learning, not having much
> experience.
>
> Of course, I can reduce number of those categories, but playing with raw
> data would be more fun. :)
>
> Regards,
> --
> Satoshi Nagayasu <sn...@uptime.jp>
>
--
---------------------------------------------------------
Rahul Iyer
Principal software engineer | Predictive Analytics
*Pivotal**A new platform for a new era*
Re: create_indicator_variables() with svec?
Posted by Satoshi Nagayasu <sn...@uptime.jp>.
Hi Rahul,
2016-08-09 2:05 GMT+09:00 Rahul Iyer <ri...@pivotal.io>:
> Array output for *create_indicator_variables* would be quite helpful when
> number of categories is large and the svec representation would be ideal
> for it. There might be similar implications for *pivoting*, but we can keep
> that as future discussion.
Sounds great.
> I'm curious about how you're using the indicator variables - svec is not
> widely supported in MADlib (yet) and might not give much benefit after the
> encoding is complete.
I'm trying to implement some recommendation or similarity search stuff
for several media items (movies, books, documents, else) with its metadata.
It has several categorical variables, such as authors, publishers,
actors/actresses, genres, else. Some of them have many categories.
BTW, I'm a starter of data-mining and machine-learning, not having much
experience.
Of course, I can reduce number of those categories, but playing with raw
data would be more fun. :)
Regards,
--
Satoshi Nagayasu <sn...@uptime.jp>
Re: create_indicator_variables() with svec?
Posted by Rahul Iyer <ri...@pivotal.io>.
Hi Satoshi,
Array output for *create_indicator_variables* would be quite helpful when
number of categories is large and the svec representation would be ideal
for it. There might be similar implications for *pivoting*, but we can keep
that as future discussion.
I'm curious about how you're using the indicator variables - svec is not
widely supported in MADlib (yet) and might not give much benefit after the
encoding is complete.
Best,
Rahul
On Sun, Aug 7, 2016 at 1:50 AM, Satoshi Nagayasu <sn...@uptime.jp> wrote:
> Hi,
>
> I'm trying create_indicator_variables() to encode categorical variables.
>
> https://madlib.incubator.apache.org/docs/latest/group__
> grp__data__prep.html
>
> And I found that PostgreSQL had a limitation of maximum number of variables
> in SELECT list (called target list in PostgreSQL), up to 1664.
>
> You may see this error when you have more than 1664 categories in your
> variable.
>
> spiexceptions.ProgramLimitExceeded: target lists can have at most 1664
> entries
>
> Now, I'm considering using PostgreSQL arrays to contain indicators instead
> of
> allocating single column per category.
>
> If create_indicator_variables() supports arrays as its output, it
> allows us to deal
> with categorical variables which have more than 1664 categories.
> And of course, I would like to use the sparse vector for it to compress
> them.
>
> https://madlib.incubator.apache.org/docs/latest/group__grp__svec.html
>
> Seems good to you? Any comments?
>
> Regards,
> --
> Satoshi Nagayasu <sn...@uptime.jp>
>
--
---------------------------------------------------------
Rahul Iyer
Principal software engineer | Predictive Analytics
*Pivotal**A new platform for a new era*