You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemml.apache.org by Deron Eriksson <de...@gmail.com> on 2015/12/17 01:02:14 UTC
DML example on main SystemML website
Hi,
I think the main SystemML website at http://systemml.incubator.apache.org/
needs to be updated so that the DML example is an actual algorithm or at
least a fragment of an algorithm.
Does anyone have a recommendation for a short, concise example that shows
the power of DML?
Thanks!
Deron
Re: DML example on main SystemML website
Posted by Deron Eriksson <de...@gmail.com>.
I updated the DML example on the main site to be the PNMF example from Fred
and Matthias.
I think the idea of a DML cookbook is a fantastic idea. I can begin a
cookbook based on the examples from Shirish. Additionally, I believe Alok
Singh has several functions as a result of his algorithm development work
that could be useful in a cookbook. I like Mike's suggestion of keeping
things simple. Such a notebook would probably not have a lot of structure
at first but would gain structure over time as code snippets can be grouped
and sorted by common functionality as their numbers increase.
Deron
On Thu, Dec 17, 2015 at 12:55 AM, Berthold Reinwald <re...@us.ibm.com>
wrote:
> Creating a DML cookbook sounds very useful ... especially for new data
> scientists starting to pick up DML. It shows vectorization avoiding loops,
> and people are familiar with the semantics. There may be more nuggets like
> that through-out the DML algorithms. And the list will grow in the course
> of time.
>
> Regards,
> Berthold Reinwald
> IBM Almaden Research Center
> office: (408) 927 2208; T/L: 457 2208
> e-mail: reinwald@us.ibm.com
>
>
>
> From: Shirish Tatikonda <sh...@gmail.com>
> To: dev@systemml.incubator.apache.org
> Date: 12/16/2015 10:03 PM
> Subject: Re: DML example on main SystemML website
>
>
>
> Deron,
>
> Along with such a complete algorithm, we could also include one/two common
> and useful DML snippets. We could also create a "DML Cookbook" with such
> snippets and keep adding more over time.
> Some example snippets are below -- note that I created them quite a while
> back, and they may need revision and testing.
>
> *Classifier Performance*
>
> # Confusion matrix
> cm = table(truthLabels, predictedLabels)
> TP = as.scalar(cm[1,1])
> TN = as.scalar(cm[2,2])
> FP = as.scalar(cm[1,2])
> FN = as.scalar(cm[2,1])
>
> accuracy = (TP+TN)/nrow(truthLabels)
> precision = TP / (TP+FP)
> recall = TP / (TP+FN)
>
> print("Accuracy = " + accuracy + ", Precision = " + precision + ", Recall
> =
> " + recall);
>
> *Covariance matrix*
>
> A = read("input.mtx");
> N = nrow(A);
> # column means
> mu = colSums(A)/N;
> # Covarianace matrix
> C = (t(A) %*% A)/(N-1) - (N/(N-1))*t(mu) %*% mu;
>
> *Select rows satisfying a predicate*
>
> ind = diag(ppred(A[,1], thresh, ">"));
> ind = removeEmpty(target=ind, margin="rows");
> result = ind %*% A;
>
>
> *Center and Scale columns*
>
> A = read("input.mtx");
> cm = colMeans(A);
> cvars = (colSums (A^2));
> cvars = (cvars - N*(cm^2))/(N-1);
> Ascaled = (A-cm)/sqrt(cvars);
>
>
> *Random shuffling of rows*
>
> N = nrow(A);
> s = sample(N, N, replace=FALSE);
> tab = table(seq(1:N), s);
> result = tab %*% A;
>
> On Wed, Dec 16, 2015 at 5:59 PM, Deron Eriksson <de...@gmail.com>
> wrote:
>
> > That example is perfect. Concise and powerful. Thank you Fred.
> >
> > Deron
> >
> >
> > On Wed, Dec 16, 2015 at 4:16 PM, Frederick R Reiss <fr...@us.ibm.com>
> > wrote:
> >
> > > We can use the Poisson nonnegative matrix factorization example from
> last
> > > week's webcast:
> > >
> > > i = 0
> > > while(i < max_iterations) {
> > > H = (H * (t(W) %*% (V/(W%*%H + epsilon)))) / t(colSums(W))
> > > W = (W * ((V/(W%*%H) + epsilon) %*% t(H))) / t(rowSums(H))
> > > i = i + 1;
> > > }
> > >
> > >
> > > Sound ok to everyone?
> > >
> > > Fred
> > >
> > >
> > > [image: Inactive hide details for Deron Eriksson ---12/16/2015
> 04:02:26
> > > PM---Hi, I think the main SystemML website at http://systemml.i]Deron
> > > Eriksson ---12/16/2015 04:02:26 PM---Hi, I think the main SystemML
> > website
> > > at http://systemml.incubator.apache.org/
> > >
> > > From: Deron Eriksson <de...@gmail.com>
> > > To: dev@systemml.incubator.apache.org
> > > Date: 12/16/2015 04:02 PM
> > > Subject: DML example on main SystemML website
> > > ------------------------------
> > >
> > >
> > >
> > > Hi,
> > >
> > > I think the main SystemML website at
> > http://systemml.incubator.apache.org/
> > > needs to be updated so that the DML example is an actual algorithm or
> at
> > > least a fragment of an algorithm.
> > >
> > > Does anyone have a recommendation for a short, concise example that
> shows
> > > the power of DML?
> > >
> > > Thanks!
> > > Deron
> > >
> > >
> > >
> >
>
>
Re: DML example on main SystemML website
Posted by Berthold Reinwald <re...@us.ibm.com>.
Creating a DML cookbook sounds very useful ... especially for new data
scientists starting to pick up DML. It shows vectorization avoiding loops,
and people are familiar with the semantics. There may be more nuggets like
that through-out the DML algorithms. And the list will grow in the course
of time.
Regards,
Berthold Reinwald
IBM Almaden Research Center
office: (408) 927 2208; T/L: 457 2208
e-mail: reinwald@us.ibm.com
From: Shirish Tatikonda <sh...@gmail.com>
To: dev@systemml.incubator.apache.org
Date: 12/16/2015 10:03 PM
Subject: Re: DML example on main SystemML website
Deron,
Along with such a complete algorithm, we could also include one/two common
and useful DML snippets. We could also create a "DML Cookbook" with such
snippets and keep adding more over time.
Some example snippets are below -- note that I created them quite a while
back, and they may need revision and testing.
*Classifier Performance*
# Confusion matrix
cm = table(truthLabels, predictedLabels)
TP = as.scalar(cm[1,1])
TN = as.scalar(cm[2,2])
FP = as.scalar(cm[1,2])
FN = as.scalar(cm[2,1])
accuracy = (TP+TN)/nrow(truthLabels)
precision = TP / (TP+FP)
recall = TP / (TP+FN)
print("Accuracy = " + accuracy + ", Precision = " + precision + ", Recall
=
" + recall);
*Covariance matrix*
A = read("input.mtx");
N = nrow(A);
# column means
mu = colSums(A)/N;
# Covarianace matrix
C = (t(A) %*% A)/(N-1) - (N/(N-1))*t(mu) %*% mu;
*Select rows satisfying a predicate*
ind = diag(ppred(A[,1], thresh, ">"));
ind = removeEmpty(target=ind, margin="rows");
result = ind %*% A;
*Center and Scale columns*
A = read("input.mtx");
cm = colMeans(A);
cvars = (colSums (A^2));
cvars = (cvars - N*(cm^2))/(N-1);
Ascaled = (A-cm)/sqrt(cvars);
*Random shuffling of rows*
N = nrow(A);
s = sample(N, N, replace=FALSE);
tab = table(seq(1:N), s);
result = tab %*% A;
On Wed, Dec 16, 2015 at 5:59 PM, Deron Eriksson <de...@gmail.com>
wrote:
> That example is perfect. Concise and powerful. Thank you Fred.
>
> Deron
>
>
> On Wed, Dec 16, 2015 at 4:16 PM, Frederick R Reiss <fr...@us.ibm.com>
> wrote:
>
> > We can use the Poisson nonnegative matrix factorization example from
last
> > week's webcast:
> >
> > i = 0
> > while(i < max_iterations) {
> > H = (H * (t(W) %*% (V/(W%*%H + epsilon)))) / t(colSums(W))
> > W = (W * ((V/(W%*%H) + epsilon) %*% t(H))) / t(rowSums(H))
> > i = i + 1;
> > }
> >
> >
> > Sound ok to everyone?
> >
> > Fred
> >
> >
> > [image: Inactive hide details for Deron Eriksson ---12/16/2015
04:02:26
> > PM---Hi, I think the main SystemML website at http://systemml.i]Deron
> > Eriksson ---12/16/2015 04:02:26 PM---Hi, I think the main SystemML
> website
> > at http://systemml.incubator.apache.org/
> >
> > From: Deron Eriksson <de...@gmail.com>
> > To: dev@systemml.incubator.apache.org
> > Date: 12/16/2015 04:02 PM
> > Subject: DML example on main SystemML website
> > ------------------------------
> >
> >
> >
> > Hi,
> >
> > I think the main SystemML website at
> http://systemml.incubator.apache.org/
> > needs to be updated so that the DML example is an actual algorithm or
at
> > least a fragment of an algorithm.
> >
> > Does anyone have a recommendation for a short, concise example that
shows
> > the power of DML?
> >
> > Thanks!
> > Deron
> >
> >
> >
>
Re: DML example on main SystemML website
Posted by Shirish Tatikonda <sh...@gmail.com>.
Deron,
Along with such a complete algorithm, we could also include one/two common
and useful DML snippets. We could also create a "DML Cookbook" with such
snippets and keep adding more over time.
Some example snippets are below -- note that I created them quite a while
back, and they may need revision and testing.
*Classifier Performance*
# Confusion matrix
cm = table(truthLabels, predictedLabels)
TP = as.scalar(cm[1,1])
TN = as.scalar(cm[2,2])
FP = as.scalar(cm[1,2])
FN = as.scalar(cm[2,1])
accuracy = (TP+TN)/nrow(truthLabels)
precision = TP / (TP+FP)
recall = TP / (TP+FN)
print("Accuracy = " + accuracy + ", Precision = " + precision + ", Recall =
" + recall);
*Covariance matrix*
A = read("input.mtx");
N = nrow(A);
# column means
mu = colSums(A)/N;
# Covarianace matrix
C = (t(A) %*% A)/(N-1) - (N/(N-1))*t(mu) %*% mu;
*Select rows satisfying a predicate*
ind = diag(ppred(A[,1], thresh, ">"));
ind = removeEmpty(target=ind, margin="rows");
result = ind %*% A;
*Center and Scale columns*
A = read("input.mtx");
cm = colMeans(A);
cvars = (colSums (A^2));
cvars = (cvars - N*(cm^2))/(N-1);
Ascaled = (A-cm)/sqrt(cvars);
*Random shuffling of rows*
N = nrow(A);
s = sample(N, N, replace=FALSE);
tab = table(seq(1:N), s);
result = tab %*% A;
On Wed, Dec 16, 2015 at 5:59 PM, Deron Eriksson <de...@gmail.com>
wrote:
> That example is perfect. Concise and powerful. Thank you Fred.
>
> Deron
>
>
> On Wed, Dec 16, 2015 at 4:16 PM, Frederick R Reiss <fr...@us.ibm.com>
> wrote:
>
> > We can use the Poisson nonnegative matrix factorization example from last
> > week's webcast:
> >
> > i = 0
> > while(i < max_iterations) {
> > H = (H * (t(W) %*% (V/(W%*%H + epsilon)))) / t(colSums(W))
> > W = (W * ((V/(W%*%H) + epsilon) %*% t(H))) / t(rowSums(H))
> > i = i + 1;
> > }
> >
> >
> > Sound ok to everyone?
> >
> > Fred
> >
> >
> > [image: Inactive hide details for Deron Eriksson ---12/16/2015 04:02:26
> > PM---Hi, I think the main SystemML website at http://systemml.i]Deron
> > Eriksson ---12/16/2015 04:02:26 PM---Hi, I think the main SystemML
> website
> > at http://systemml.incubator.apache.org/
> >
> > From: Deron Eriksson <de...@gmail.com>
> > To: dev@systemml.incubator.apache.org
> > Date: 12/16/2015 04:02 PM
> > Subject: DML example on main SystemML website
> > ------------------------------
> >
> >
> >
> > Hi,
> >
> > I think the main SystemML website at
> http://systemml.incubator.apache.org/
> > needs to be updated so that the DML example is an actual algorithm or at
> > least a fragment of an algorithm.
> >
> > Does anyone have a recommendation for a short, concise example that shows
> > the power of DML?
> >
> > Thanks!
> > Deron
> >
> >
> >
>
Re: DML example on main SystemML website
Posted by Deron Eriksson <de...@gmail.com>.
That example is perfect. Concise and powerful. Thank you Fred.
Deron
On Wed, Dec 16, 2015 at 4:16 PM, Frederick R Reiss <fr...@us.ibm.com>
wrote:
> We can use the Poisson nonnegative matrix factorization example from last
> week's webcast:
>
> i = 0
> while(i < max_iterations) {
> H = (H * (t(W) %*% (V/(W%*%H + epsilon)))) / t(colSums(W))
> W = (W * ((V/(W%*%H) + epsilon) %*% t(H))) / t(rowSums(H))
> i = i + 1;
> }
>
>
> Sound ok to everyone?
>
> Fred
>
>
> [image: Inactive hide details for Deron Eriksson ---12/16/2015 04:02:26
> PM---Hi, I think the main SystemML website at http://systemml.i]Deron
> Eriksson ---12/16/2015 04:02:26 PM---Hi, I think the main SystemML website
> at http://systemml.incubator.apache.org/
>
> From: Deron Eriksson <de...@gmail.com>
> To: dev@systemml.incubator.apache.org
> Date: 12/16/2015 04:02 PM
> Subject: DML example on main SystemML website
> ------------------------------
>
>
>
> Hi,
>
> I think the main SystemML website at http://systemml.incubator.apache.org/
> needs to be updated so that the DML example is an actual algorithm or at
> least a fragment of an algorithm.
>
> Does anyone have a recommendation for a short, concise example that shows
> the power of DML?
>
> Thanks!
> Deron
>
>
>
Re: DML example on main SystemML website
Posted by Mike Dusenberry <du...@gmail.com>.
I think the NMF algorithm will be a great example to display to users. It's
concise, still displays several aspects of the DML language, and is a
useful ML algorithm. I also think Shirish's idea of a DML "cookbook" would
be great. For the later, I would suggest that we keep the documentation
simple by focusing mostly on just the code snippets, with corresponding
labels for each.
- Mike
On Wed, Dec 16, 2015 at 11:20 PM Matthias Boehm <mb...@us.ibm.com> wrote:
> please include the computation of the objective function (for convergence
> checks or at least print outs). Thanks.
>
> while( iter < max_iterations ){
> iter = iter + 1;
> H = (H * (t(W) %*% (V/(W%*%H))))/t(colSums(W));
> W = (W * ((V/(W%*%H)) %*% t(H)))/t(rowSums(H));
> obj = as.scalar(colSums(W)%*%rowSums(H)) -sum(V * log(W%*%H));
> print("ITER=" + iter + " obj=" + obj);
> }
>
> Regards,
> Matthias
>
> [image: Inactive hide details for Frederick R Reiss---12/17/2015 01:41:00
> AM---We can use the Poisson nonnegative matrix factorization]Frederick R
> Reiss---12/17/2015 01:41:00 AM---We can use the Poisson nonnegative matrix
> factorization example from last week's webcast:
>
> From: Frederick R Reiss/Almaden/IBM@IBMUS
> To: dev@systemml.incubator.apache.org
> Date: 12/17/2015 01:41 AM
> Subject: Re: DML example on main SystemML website
> ------------------------------
>
>
>
> We can use the Poisson nonnegative matrix factorization example from last
> week's webcast:
>
> i = 0
> while(i < max_iterations) {
> H = (H * (t(W) %*% (V/(W%*%H + epsilon)))) / t(colSums(W))
> W = (W * ((V/(W%*%H) + epsilon) %*% t(H))) / t(rowSums(H))
> i = i + 1;
> }
>
>
> Sound ok to everyone?
>
> Fred
>
>
> Deron Eriksson ---12/16/2015 04:02:26 PM---Hi, I think the main SystemML
> website at *http://systemml.incubator.apache.org/*
> <http://systemml.incubator.apache.org/>
>
> From: Deron Eriksson <de...@gmail.com>
> To: dev@systemml.incubator.apache.org
> Date: 12/16/2015 04:02 PM
> Subject: DML example on main SystemML website
> ------------------------------
>
>
>
> Hi,
>
> I think the main SystemML website at
> *http://systemml.incubator.apache.org/*
> <http://systemml.incubator.apache.org/>
> needs to be updated so that the DML example is an actual algorithm or at
> least a fragment of an algorithm.
>
> Does anyone have a recommendation for a short, concise example that shows
> the power of DML?
>
> Thanks!
> Deron
>
>
> --
Mike Dusenberry
GitHub: github.com/dusenberrymw
LinkedIn: linkedin.com/in/mikedusenberry
Sent from my iPhone
Re: DML example on main SystemML website
Posted by Matthias Boehm <mb...@us.ibm.com>.
please include the computation of the objective function (for convergence
checks or at least print outs). Thanks.
while( iter < max_iterations ){
iter = iter + 1;
H = (H * (t(W) %*% (V/(W%*%H))))/t(colSums(W));
W = (W * ((V/(W%*%H)) %*% t(H)))/t(rowSums(H));
obj = as.scalar(colSums(W)%*%rowSums(H)) -sum(V * log(W%*%H));
print("ITER=" + iter + " obj=" + obj);
}
Regards,
Matthias
From: Frederick R Reiss/Almaden/IBM@IBMUS
To: dev@systemml.incubator.apache.org
Date: 12/17/2015 01:41 AM
Subject: Re: DML example on main SystemML website
We can use the Poisson nonnegative matrix factorization example from last
week's webcast:
i = 0
while(i < max_iterations) {
H = (H * (t(W) %*% (V/(W%*%H + epsilon)))) / t(colSums(W))
W = (W * ((V/(W%*%H) + epsilon) %*% t(H))) / t(rowSums(H))
i = i + 1;
}
Sound ok to everyone?
Fred
Deron Eriksson ---12/16/2015 04:02:26 PM---Hi, I think the main SystemML
website at http://systemml.incubator.apache.org/
From: Deron Eriksson <de...@gmail.com>
To: dev@systemml.incubator.apache.org
Date: 12/16/2015 04:02 PM
Subject: DML example on main SystemML website
Hi,
I think the main SystemML website at http://systemml.incubator.apache.org/
needs to be updated so that the DML example is an actual algorithm or at
least a fragment of an algorithm.
Does anyone have a recommendation for a short, concise example that shows
the power of DML?
Thanks!
Deron
Re: DML example on main SystemML website
Posted by Frederick R Reiss <fr...@us.ibm.com>.
We can use the Poisson nonnegative matrix factorization example from last
week's webcast:
i = 0
while(i < max_iterations) {
H = (H * (t(W) %*% (V/(W%*%H + epsilon)))) / t(colSums(W))
W = (W * ((V/(W%*%H) + epsilon) %*% t(H))) / t(rowSums(H))
i = i + 1;
}
Sound ok to everyone?
Fred
From: Deron Eriksson <de...@gmail.com>
To: dev@systemml.incubator.apache.org
Date: 12/16/2015 04:02 PM
Subject: DML example on main SystemML website
Hi,
I think the main SystemML website at http://systemml.incubator.apache.org/
needs to be updated so that the DML example is an actual algorithm or at
least a fragment of an algorithm.
Does anyone have a recommendation for a short, concise example that shows
the power of DML?
Thanks!
Deron