You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemml.apache.org by Nikolay Manchev <NM...@uk.ibm.com> on 2016/07/07 16:43:44 UTC
Restricted Boltzmann Machine scripts
Hi all,
I am dropping a note as suggested by Fred below.
I wrote a couple of DML scripts that implement Restricted Boltzmann
Machines, which are available on GitHub. This one uses CD-1 mini-batch
training to fit the model (weights and biases):
https://github.com/nmanchev/incubator-systemml/blob/neuralnets/scripts/algorithms/rbm_minibatch.dml
This one runs a data sets through the trained RBM and outputs sample of
P(h=1|v) for each observation:
https://github.com/nmanchev/incubator-systemml/blob/neuralnets/scripts/algorithms/rbm_run.dml
Can you please add them to scripts/algorithms? I also created a JIRA issue
for adding them here: https://issues.apache.org/jira/browse/SYSTEMML-777
Many thanks
Nikolay
----- Original message -----
From: Frederick R Reiss/Almaden/IBM
To: Nikolay Manchev/UK/IBM@IBMGB
Cc: dev@systemml.incubator.apache.org
Subject: Re: Question on SystemML - RBMs and repmat()
Date: Thu, Jul 7, 2016 3:07 PM
Hi Nikolay,
I appreciate your interest in the project. To answer your question: You
should be able to write "X%*%W + B" and get the semantics you want. The
SystemML compiler automatically pads vectors with copies of themselves
when it sees a cellwise operation between a matrix and a vector. So if you
run the DML code:
A = matrix (1.0, rows=3, cols=3)
v = matrix (2.0, rows=1, cols=3)
sum = A + v
print(toString(sum))
the output will be:
3.000 3.000 3.000
3.000 3.000 3.000
3.000 3.000 3.000
Exposing cellwise matrix-vector operations to the SystemML optimizer in
this way should result in more efficient parallel plans, since it's easier
for the optimizer to detect that it can broadcast the vector and stream
the matrix.
The PNMF script on the SystemML home page (http://systemml.apache.org) has
a more in-depth example of the same pattern:
while (iter < max_iterations) {
iter = iter + 1;
H = (H * (t(W) %*% (V/(W%*%H)))) / t(colSums(W));
W = (W * ((V/(W%*%H)) %*% t(H))) / t(rowSums(H));
obj = as.scalar(colSums(W) %*% rowSums(H)) - sum(V * log(W%*%H));
print("iter=" + iter + " obj=" + obj);
}
The part in red divides the matrix (H * (t(W) %*% (V/(W%*%H)))) by the
vector t(colSums(W)). In R, the divisor in this expression would need to
be (t(matrix(colSums(W),nrow=1))%*%matrix(rep(1,m),nrow=1)) or something
equivalent.
I think that an example script for training Boltzmann machines would be a
useful addition to the SystemML distribution. Would you mind opening a
JIRA issue for adding this script and posting a link to the JIRA on the
SystemML mailing list? Our JIRA instance is at
https://issues.apache.org/jira/browse/SYSTEMML, and our mailing list is at
http://systemml.apache.org/community. By the way, it's good to post
questions like your question below the mailing list so that others who run
into the same issue will have an easier time finding the solution; I'm
CCing the list with my response here.
Fred
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: Restricted Boltzmann Machine scripts
Posted by Nikolay Manchev <NM...@uk.ibm.com>.
Hi Matthias,
Thank you for your feedback. I've just created a PR as requested.
Regards
Nikolay
"Matthias Boehm" <mb...@us.ibm.com> wrote on 08/07/2016 08:43:55:
> From: "Matthias Boehm" <mb...@us.ibm.com>
> To: dev@systemml.incubator.apache.org
> Date: 08/07/2016 08:44
> Subject: Re: Restricted Boltzmann Machine scripts
>
> thanks for reaching out Nikolay,
>
> 1) Scripts: Could you please create a PR to add them to /scripts/
> staging? This is the place we typically use to share new scripts.
> Once they are tested for accuracy and runtime, we would migrate them
> into scripts/algorithms along with some basic documentation. Thanks.
>
> 2) Matrix/Vector element-wise binary operations: As Fred already
> mentioned, SystemML supports both Matrix-Row Vector and Matrix-
> Column Vector operations, where the right-hand side input is
> logically replicated which often allows us to broadcast the vector
> and hence avoid shuffle operations. However, if you prefer to write
> it explicitly as repmat like X %*% (v %*% matrix(1, 1, ncols(X))),
> that's fine too - the SystemML compiler would then anyway
> automatically rewrite this to matrix-vector operations if the repmat
> is in the same basic block of statements.
>
> Regards,
> Matthias
>
>
> [image removed] Nikolay Manchev ---07/07/2016 09:44:28 AM---Hi all,
> I am dropping a note as suggested by Fred below.
>
> From: Nikolay Manchev <NM...@uk.ibm.com>
> To: dev@systemml.incubator.apache.org
> Date: 07/07/2016 09:44 AM
> Subject: Restricted Boltzmann Machine scripts
>
>
>
> Hi all,
>
> I am dropping a note as suggested by Fred below.
>
> I wrote a couple of DML scripts that implement Restricted Boltzmann
> Machines, which are available on GitHub. This one uses CD-1 mini-batch
> training to fit the model (weights and biases):
> https://github.com/nmanchev/incubator-systemml/blob/neuralnets/
> scripts/algorithms/rbm_minibatch.dml
>
> This one runs a data sets through the trained RBM and outputs sample of
> P(h=1|v) for each observation:
> https://github.com/nmanchev/incubator-systemml/blob/neuralnets/
> scripts/algorithms/rbm_run.dml
>
> Can you please add them to scripts/algorithms? I also created a JIRA
issue
> for adding them here: https://issues.apache.org/jira/browse/SYSTEMML-777
>
> Many thanks
> Nikolay
>
>
> ----- Original message -----
> From: Frederick R Reiss/Almaden/IBM
> To: Nikolay Manchev/UK/IBM@IBMGB
> Cc: dev@systemml.incubator.apache.org
> Subject: Re: Question on SystemML - RBMs and repmat()
> Date: Thu, Jul 7, 2016 3:07 PM
>
> Hi Nikolay,
>
> I appreciate your interest in the project. To answer your question: You
> should be able to write "X%*%W + B" and get the semantics you want. The
> SystemML compiler automatically pads vectors with copies of themselves
> when it sees a cellwise operation between a matrix and a vector. So if
you
> run the DML code:
>
>
> A = matrix (1.0, rows=3, cols=3)
> v = matrix (2.0, rows=1, cols=3)
> sum = A + v
> print(toString(sum))
>
> the output will be:
>
> 3.000 3.000 3.000
> 3.000 3.000 3.000
> 3.000 3.000 3.000
>
>
> Exposing cellwise matrix-vector operations to the SystemML optimizer in
> this way should result in more efficient parallel plans, since it's
easier
> for the optimizer to detect that it can broadcast the vector and stream
> the matrix.
>
> The PNMF script on the SystemML home page (http://systemml.apache.org)
has
> a more in-depth example of the same pattern:
>
> while (iter < max_iterations) {
> iter = iter + 1;
> H = (H * (t(W) %*% (V/(W%*%H)))) / t(colSums(W));
> W = (W * ((V/(W%*%H)) %*% t(H))) / t(rowSums(H));
> obj = as.scalar(colSums(W) %*% rowSums(H)) - sum(V * log(W%*%H));
> print("iter=" + iter + " obj=" + obj);
> }
>
> The part in red divides the matrix (H * (t(W) %*% (V/(W%*%H)))) by the
> vector t(colSums(W)). In R, the divisor in this expression would need to
> be (t(matrix(colSums(W),nrow=1))%*%matrix(rep(1,m),nrow=1)) or something
> equivalent.
>
>
>
> I think that an example script for training Boltzmann machines would be
a
> useful addition to the SystemML distribution. Would you mind opening a
> JIRA issue for adding this script and posting a link to the JIRA on the
> SystemML mailing list? Our JIRA instance is at
> https://issues.apache.org/jira/browse/SYSTEMML, and our mailing list is
at
> http://systemml.apache.org/community. By the way, it's good to post
> questions like your question below the mailing list so that others who
run
> into the same issue will have an easier time finding the solution; I'm
> CCing the list with my response here.
>
> Fred
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU
>
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: Restricted Boltzmann Machine scripts
Posted by Matthias Boehm <mb...@us.ibm.com>.
thanks for reaching out Nikolay,
1) Scripts: Could you please create a PR to add them to /scripts/staging?
This is the place we typically use to share new scripts. Once they are
tested for accuracy and runtime, we would migrate them into
scripts/algorithms along with some basic documentation. Thanks.
2) Matrix/Vector element-wise binary operations: As Fred already mentioned,
SystemML supports both Matrix-Row Vector and Matrix-Column Vector
operations, where the right-hand side input is logically replicated which
often allows us to broadcast the vector and hence avoid shuffle operations.
However, if you prefer to write it explicitly as repmat like X %*% (v %*%
matrix(1, 1, ncols(X))), that's fine too - the SystemML compiler would then
anyway automatically rewrite this to matrix-vector operations if the repmat
is in the same basic block of statements.
Regards,
Matthias
From: Nikolay Manchev <NM...@uk.ibm.com>
To: dev@systemml.incubator.apache.org
Date: 07/07/2016 09:44 AM
Subject: Restricted Boltzmann Machine scripts
Hi all,
I am dropping a note as suggested by Fred below.
I wrote a couple of DML scripts that implement Restricted Boltzmann
Machines, which are available on GitHub. This one uses CD-1 mini-batch
training to fit the model (weights and biases):
https://github.com/nmanchev/incubator-systemml/blob/neuralnets/scripts/algorithms/rbm_minibatch.dml
This one runs a data sets through the trained RBM and outputs sample of
P(h=1|v) for each observation:
https://github.com/nmanchev/incubator-systemml/blob/neuralnets/scripts/algorithms/rbm_run.dml
Can you please add them to scripts/algorithms? I also created a JIRA issue
for adding them here: https://issues.apache.org/jira/browse/SYSTEMML-777
Many thanks
Nikolay
----- Original message -----
From: Frederick R Reiss/Almaden/IBM
To: Nikolay Manchev/UK/IBM@IBMGB
Cc: dev@systemml.incubator.apache.org
Subject: Re: Question on SystemML - RBMs and repmat()
Date: Thu, Jul 7, 2016 3:07 PM
Hi Nikolay,
I appreciate your interest in the project. To answer your question: You
should be able to write "X%*%W + B" and get the semantics you want. The
SystemML compiler automatically pads vectors with copies of themselves
when it sees a cellwise operation between a matrix and a vector. So if you
run the DML code:
A = matrix (1.0, rows=3, cols=3)
v = matrix (2.0, rows=1, cols=3)
sum = A + v
print(toString(sum))
the output will be:
3.000 3.000 3.000
3.000 3.000 3.000
3.000 3.000 3.000
Exposing cellwise matrix-vector operations to the SystemML optimizer in
this way should result in more efficient parallel plans, since it's easier
for the optimizer to detect that it can broadcast the vector and stream
the matrix.
The PNMF script on the SystemML home page (http://systemml.apache.org) has
a more in-depth example of the same pattern:
while (iter < max_iterations) {
iter = iter + 1;
H = (H * (t(W) %*% (V/(W%*%H)))) / t(colSums(W));
W = (W * ((V/(W%*%H)) %*% t(H))) / t(rowSums(H));
obj = as.scalar(colSums(W) %*% rowSums(H)) - sum(V * log(W%*%H));
print("iter=" + iter + " obj=" + obj);
}
The part in red divides the matrix (H * (t(W) %*% (V/(W%*%H)))) by the
vector t(colSums(W)). In R, the divisor in this expression would need to
be (t(matrix(colSums(W),nrow=1))%*%matrix(rep(1,m),nrow=1)) or something
equivalent.
I think that an example script for training Boltzmann machines would be a
useful addition to the SystemML distribution. Would you mind opening a
JIRA issue for adding this script and posting a link to the JIRA on the
SystemML mailing list? Our JIRA instance is at
https://issues.apache.org/jira/browse/SYSTEMML, and our mailing list is at
http://systemml.apache.org/community. By the way, it's good to post
questions like your question below the mailing list so that others who run
into the same issue will have an easier time finding the solution; I'm
CCing the list with my response here.
Fred
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU