You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by Pat Ferrel <pa...@occamsmachete.com> on 2014/07/08 19:29:10 UTC
Problem of dimensions
In cooccurrence for the case of B'A the real-world dimensionality of the matrices can be compatible even though the data read in from tuples would leave some rows or columns blank—no non-zero elements. At least this is what I suspect trying to run cooccurrence on the epinions data (ratings_data.txt, trust_data.txt)
I get:
Exception in thread "main" java.lang.AssertionError: assertion failed: Incompatible operand geometry
at scala.Predef$.assert(Predef.scala:179)
at org.apache.mahout.math.drm.logical.OpAB.<init>(OpAB.scala:29)
...
from
val drmBtA = drmB.t %*% drmA
Regardless of what is causing this problem there _will_ be cases where the auto-calculated dimensions of A and B (calculated from the non-blank rows when the DRM is read in from a text file) are not compatible but the data actually is. This is the case where the union of all userIDs is greater than the number of user IDs in one or both of the DRMs.
To do this correctly for all cases the row IDs for all unique row keys would have to be created across all drms for cooccurrence. This implies using a single Map for the row space of all drms read in with a single incrementing integer for the DRM row key. The length of this Map would be the row dimension for all DRMs. After the row dimension is calculated the Map could be thrown away since only columns (input items) need to have application specific IDs applied at output.
Does this sound like the right way to handle this case? Will the drmB.t %*% drmA do the right thing for non-existent rows/columns, which I think means to treat a non-existent vector as if it were all 0s. I believe this worked in the hadoop version.
Re: Problem of dimensions
Posted by Pat Ferrel <pa...@occamsmachete.com>.
>
> On Jul 16, 2014, at 8:34 AM, Anand Avati <av...@gluster.org> wrote:
>
> On Wed, Jul 16, 2014 at 7:53 AM, Pat Ferrel <pa...@occamsmachete.com> wrote:
> There IS no issue with nrow being a lazy val. I never touch it read below.
>
> The value itself is may not be immutable. But it sounds like the same matrix would return different values for nrow() depending on when you called it. That sounds very much like a problem if the same matrix is part of two separate spark graph hierarchies where each is making a different assumption about its cardinality.
My understanding of a CheckpointeDrmSpark is to assure that the rdd is checkpointed so no optimizer graph at that point. Yes it would produce different results but so would the matrix pre and post rbind.
There are serveral matrix reads, that create one dictionary for all rows in all matrices. The length of this dictionary is the row cardinality for all matrices. Each CheckpointedDrmSpark has the rows with Int IDs/keys for the data it knows about and so on. The cardinality is changed before any operation is performed.
>
> creating a new matrix val is fine if it doesn’t cause a new rdd to be created I’ll look into that.
>
> rbind as I read it requires me to construct the rows to be added. I don’t know what their keys are and don’t want to calculate them. If I’m right about how the math works the actual rows are not needed.
>
> See the example code to add empty rows. Numerical keys are auto computed based on the sizes.
>
I’m pretty sure that’s impossible for this case. The only way to calculate the true cardinality is read in all data for all matrices.The only way to know what IDs are _missing_ from a single DRM is to read them all in. The internal Int IDs are different for each matrix and there is no telling which is in which. I’d have to keep the unified row dictionary and some collection for each matrix then subtract the smaller from the unified to get the missing keys. Only then could I create empty rows with rbind using the missing keys. The users you see for a specific interaction recorded in the several matrices may bear no relationship any of the others. In fact that’s what cooccurrence may tell you.
Any operation performed on the drms before these hypothetical rbinds are performed will be wonky anyway so I miss the point about that.
If my solution works (forcing cardinality without actually adding empty rows) it is far far simpler than keeping these extra collections and calculating the IDs for rbind. If I’m wrong, what’s described above should work.
This is for one rather unique pipeline so my shortcut may not be safe for any use that is my worry. We’ll find out.
Re: Problem of dimensions
Posted by Anand Avati <av...@gluster.org>.
On Wed, Jul 16, 2014 at 7:53 AM, Pat Ferrel <pa...@occamsmachete.com> wrote:
> There IS no issue with nrow being a lazy val. I never touch it read below.
>
The value itself is may not be immutable. But it sounds like the same
matrix would return different values for nrow() depending on when you
called it. That sounds very much like a problem if the same matrix is part
of two separate spark graph hierarchies where each is making a different
assumption about its cardinality.
> creating a new matrix val is fine if it doesn’t cause a new rdd to be
> created I’ll look into that.
>
> rbind as I read it requires me to construct the rows to be added. I don’t
> know what their keys are and don’t want to calculate them. If I’m right
> about how the math works the actual rows are not needed.
See the example code to add empty rows. Numerical keys are auto computed
based on the sizes.
> This looks like a much heavier weight operation than just changing the row
> cardinality and works for other cases where you are adding real vectors.
>
I'm not sure if there is a lighter _and_ safe operation. Happy to take
suggestions.
Thanks
On Tue, Jul 15, 2014 at 1:04 PM, Anand Avati <av...@gluster.org> wrote:
>
>
>
> On Tue, Jul 15, 2014 at 12:45 PM, Pat Ferrel <pa...@gmail.com> wrote:
>
>> I appreciate the thoughts.
>>
>> I don’t change nrow it is still a lazy val. I change _nrow, which is a
>> var and is used to calculate nrow when it is needed. The only thing run
on
>> them is the CheckpointedDrmSpark constructor. The class exists to
guarantee
>> the drm is pinned down and _nrow is changed after construction but before
>> any math is done on it. Changing _nrow may be safe on a
>> CheckpointedDrmSpark but the question is why I’ll put it up on a PR.
>>
>> btw I was thinking of calling the method
>> CheckpointedDrmSpark#addEmptyRows, which since it’s sparse will just
change
>> _nrow and will flag the purpose of the method not to mention it avoids
the
>> question about reducing the number of rows.
>
>
>
> I would prefer a new rbind() operator instead of addEmptyRows() method.
> Just feels more consistent.
>
> Thanks
>
Re: Problem of dimensions
Posted by Pat Ferrel <pa...@occamsmachete.com>.
There IS no issue with nrow being a lazy val. I never touch it read below.
creating a new matrix val is fine if it doesn’t cause a new rdd to be created I’ll look into that.
rbind as I read it requires me to construct the rows to be added. I don’t know what their keys are and don’t want to calculate them. If I’m right about how the math works the actual rows are not needed. This looks like a much heavier weight operation than just changing the row cardinality and works for other cases where you are adding real vectors.
I’ll look deeper now that cross-cooccurrence seems to be fixed.
On Jul 15, 2014, at 7:40 PM, Ted Dunning <te...@gmail.com> wrote:
The rbind approach also gives a new object and avoids all questions of lazy
evaluation.
On Tue, Jul 15, 2014 at 1:04 PM, Anand Avati <av...@gluster.org> wrote:
>
>
>
> On Tue, Jul 15, 2014 at 12:45 PM, Pat Ferrel <pa...@gmail.com> wrote:
>
>> I appreciate the thoughts.
>>
>> I don’t change nrow it is still a lazy val. I change _nrow, which is a
>> var and is used to calculate nrow when it is needed. The only thing run on
>> them is the CheckpointedDrmSpark constructor. The class exists to guarantee
>> the drm is pinned down and _nrow is changed after construction but before
>> any math is done on it. Changing _nrow may be safe on a
>> CheckpointedDrmSpark but the question is why I’ll put it up on a PR.
>>
>> btw I was thinking of calling the method
>> CheckpointedDrmSpark#addEmptyRows, which since it’s sparse will just change
>> _nrow and will flag the purpose of the method not to mention it avoids the
>> question about reducing the number of rows.
>
>
>
> I would prefer a new rbind() operator instead of addEmptyRows() method.
> Just feels more consistent.
>
> Thanks
>
Re: Problem of dimensions
Posted by Ted Dunning <te...@gmail.com>.
The rbind approach also gives a new object and avoids all questions of lazy
evaluation.
On Tue, Jul 15, 2014 at 1:04 PM, Anand Avati <av...@gluster.org> wrote:
>
>
>
> On Tue, Jul 15, 2014 at 12:45 PM, Pat Ferrel <pa...@gmail.com> wrote:
>
>> I appreciate the thoughts.
>>
>> I don’t change nrow it is still a lazy val. I change _nrow, which is a
>> var and is used to calculate nrow when it is needed. The only thing run on
>> them is the CheckpointedDrmSpark constructor. The class exists to guarantee
>> the drm is pinned down and _nrow is changed after construction but before
>> any math is done on it. Changing _nrow may be safe on a
>> CheckpointedDrmSpark but the question is why I’ll put it up on a PR.
>>
>> btw I was thinking of calling the method
>> CheckpointedDrmSpark#addEmptyRows, which since it’s sparse will just change
>> _nrow and will flag the purpose of the method not to mention it avoids the
>> question about reducing the number of rows.
>
>
>
> I would prefer a new rbind() operator instead of addEmptyRows() method.
> Just feels more consistent.
>
> Thanks
>
Re: Problem of dimensions
Posted by Pat Ferrel <pa...@gmail.com>.
BTW I’m now wondering about a non-existant row too. The question was put to Dmitriy a couple replies down. It worked on one test but now I’m bogged down fixing cross-coocurrence tests and won’t get back to itemsimilarity to try it again until later.
On Jul 15, 2014, at 12:45 PM, Pat Ferrel <pa...@gmail.com> wrote:
I appreciate the thoughts.
I don’t change nrow it is still a lazy val. I change _nrow, which is a var and is used to calculate nrow when it is needed. The only thing run on them is the CheckpointedDrmSpark constructor. The class exists to guarantee the drm is pinned down and _nrow is changed after construction but before any math is done on it. Changing _nrow may be safe on a CheckpointedDrmSpark but the question is why I’ll put it up on a PR.
btw I was thinking of calling the method CheckpointedDrmSpark#addEmptyRows, which since it’s sparse will just change _nrow and will flag the purpose of the method not to mention it avoids the question about reducing the number of rows.
On Jul 15, 2014, at 12:11 PM, Anand Avati <av...@gluster.org> wrote:
On Tue, Jul 15, 2014 at 11:10 AM, Ted Dunning <te...@gmail.com> wrote:
>
> My worry about this is that I thought that DSL objects needed to remain
> immutable due to the lazy evaluation.
>
> For instance, suppose that A and B can be multiplied because they are the
> right shapes:
>
> C = A.t %*% B
>
> Now I change number of rows in A
>
> A.nrow = 2 * A.nrow
>
> and A now has the shape compatible with D
>
> E = A.t %*% D
>
> And now I ask for some materialization:
>
> x = C.sum + E.sum
>
> And this fails. The reason is that the change in A's shape happened
> before computation was started on multiplying A and B.
>
> Now, to my mind, lazy evaluation is pretty crucial because it allows the
> optimizer to work. That seems to me to say that mutation should not be
> allowed.
>
> (and yes, I know that these aren't the correct notation ... that isn't the
> point)
>
This is a good enough reason to avoid mutating nrow field. This apart, I'm
still not convinced the spark engine can "just work" without adding new
(key, vector) tuples. It seems it is bound to fail in some future operation
if the (key, vector) tuples (even if sparse/empty) are not physically added.
Thanks
Re: Problem of dimensions
Posted by Anand Avati <av...@gluster.org>.
On Tue, Jul 15, 2014 at 1:04 PM, Anand Avati <av...@gluster.org> wrote:
>
>
>
> On Tue, Jul 15, 2014 at 12:45 PM, Pat Ferrel <pa...@gmail.com> wrote:
>
>> I appreciate the thoughts.
>>
>> I don’t change nrow it is still a lazy val. I change _nrow, which is a
>> var and is used to calculate nrow when it is needed. The only thing run on
>> them is the CheckpointedDrmSpark constructor. The class exists to guarantee
>> the drm is pinned down and _nrow is changed after construction but before
>> any math is done on it. Changing _nrow may be safe on a
>> CheckpointedDrmSpark but the question is why I’ll put it up on a PR.
>>
>> btw I was thinking of calling the method
>> CheckpointedDrmSpark#addEmptyRows, which since it’s sparse will just change
>> _nrow and will flag the purpose of the method not to mention it avoids the
>> question about reducing the number of rows.
>
>
>
> I would prefer a new rbind() operator instead of addEmptyRows() method.
> Just feels more consistent.
>
Pat, please have a look at https://github.com/apache/mahout/pull/30. It
would be cool if you could adjust your matrix cardinality the way it is
done in the test example.
Thanks
Re: Problem of dimensions
Posted by Anand Avati <av...@gluster.org>.
On Tue, Jul 15, 2014 at 12:45 PM, Pat Ferrel <pa...@gmail.com> wrote:
> I appreciate the thoughts.
>
> I don’t change nrow it is still a lazy val. I change _nrow, which is a var
> and is used to calculate nrow when it is needed. The only thing run on them
> is the CheckpointedDrmSpark constructor. The class exists to guarantee the
> drm is pinned down and _nrow is changed after construction but before any
> math is done on it. Changing _nrow may be safe on a CheckpointedDrmSpark
> but the question is why I’ll put it up on a PR.
>
> btw I was thinking of calling the method
> CheckpointedDrmSpark#addEmptyRows, which since it’s sparse will just change
> _nrow and will flag the purpose of the method not to mention it avoids the
> question about reducing the number of rows.
I would prefer a new rbind() operator instead of addEmptyRows() method.
Just feels more consistent.
Thanks
Re: Problem of dimensions
Posted by Pat Ferrel <pa...@gmail.com>.
I appreciate the thoughts.
I don’t change nrow it is still a lazy val. I change _nrow, which is a var and is used to calculate nrow when it is needed. The only thing run on them is the CheckpointedDrmSpark constructor. The class exists to guarantee the drm is pinned down and _nrow is changed after construction but before any math is done on it. Changing _nrow may be safe on a CheckpointedDrmSpark but the question is why I’ll put it up on a PR.
btw I was thinking of calling the method CheckpointedDrmSpark#addEmptyRows, which since it’s sparse will just change _nrow and will flag the purpose of the method not to mention it avoids the question about reducing the number of rows.
On Jul 15, 2014, at 12:11 PM, Anand Avati <av...@gluster.org> wrote:
On Tue, Jul 15, 2014 at 11:10 AM, Ted Dunning <te...@gmail.com> wrote:
>
> My worry about this is that I thought that DSL objects needed to remain
> immutable due to the lazy evaluation.
>
> For instance, suppose that A and B can be multiplied because they are the
> right shapes:
>
> C = A.t %*% B
>
> Now I change number of rows in A
>
> A.nrow = 2 * A.nrow
>
> and A now has the shape compatible with D
>
> E = A.t %*% D
>
> And now I ask for some materialization:
>
> x = C.sum + E.sum
>
> And this fails. The reason is that the change in A's shape happened
> before computation was started on multiplying A and B.
>
> Now, to my mind, lazy evaluation is pretty crucial because it allows the
> optimizer to work. That seems to me to say that mutation should not be
> allowed.
>
> (and yes, I know that these aren't the correct notation ... that isn't the
> point)
>
This is a good enough reason to avoid mutating nrow field. This apart, I'm
still not convinced the spark engine can "just work" without adding new
(key, vector) tuples. It seems it is bound to fail in some future operation
if the (key, vector) tuples (even if sparse/empty) are not physically added.
Thanks
Re: Problem of dimensions
Posted by Anand Avati <av...@gluster.org>.
On Tue, Jul 15, 2014 at 11:10 AM, Ted Dunning <te...@gmail.com> wrote:
>
> My worry about this is that I thought that DSL objects needed to remain
> immutable due to the lazy evaluation.
>
> For instance, suppose that A and B can be multiplied because they are the
> right shapes:
>
> C = A.t %*% B
>
> Now I change number of rows in A
>
> A.nrow = 2 * A.nrow
>
> and A now has the shape compatible with D
>
> E = A.t %*% D
>
> And now I ask for some materialization:
>
> x = C.sum + E.sum
>
> And this fails. The reason is that the change in A's shape happened
> before computation was started on multiplying A and B.
>
> Now, to my mind, lazy evaluation is pretty crucial because it allows the
> optimizer to work. That seems to me to say that mutation should not be
> allowed.
>
> (and yes, I know that these aren't the correct notation ... that isn't the
> point)
>
This is a good enough reason to avoid mutating nrow field. This apart, I'm
still not convinced the spark engine can "just work" without adding new
(key, vector) tuples. It seems it is bound to fail in some future operation
if the (key, vector) tuples (even if sparse/empty) are not physically added.
Thanks
Re: Problem of dimensions
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
also, note that parallelizeEmpty() does not create anything but a standard
int-keyed matrix with all rows indexed accordingly. That means it cannot be
r-bound with something that is not int-keyed (but perhaps it could be bound
after intermediate map-block for keys).
On Mon, Jul 21, 2014 at 1:42 PM, Pat Ferrel <pa...@gmail.com> wrote:
> Thank you! This is what I understood and I’m doing a little dance for joy
> (in my mind). This makes sparseness all encompassing, at least for
> sequential Int keys.
>
> However Anand has found several math ops that don’t work.
>
> I’ll write up a few tests for transpose and multiply at least since these
> are used in cooccurrence. And I’ll be happy to implement something that
> changes nrow in an immutable R-like way. Anand and Ted suggested rbind
> of drmParallelizeEmpty with added row cardinality. This would really only
> change nrow of the resulting CheckPointedDrm, it would not alter the rdd.
>
>
>
>
> On Jul 21, 2014, at 1:12 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
>
> "missing" rows are only valid in context of int-keyed matrices and
> physical transposition operations. These are the only that may depend on
> it, since obviously one can't define "missing-ness" for something that is
> String-keyed.
>
> So the only thing that may fail because of "missing-ness" effect is
> probably physical transposition operator (we don't have test for such case,
> so maybe there's a bug in that case). Everything else should work.
>
> And no, i suppose it is ok to have "missing" rows even in case of
> int-keyed matrices.
>
> there's one thing that you probably should be aware in this context
> though: many algorithms don't survive empty (row-less) partitions, in
> whatever way they may come to be. Other than that, I don't feel every row
> must be present -- even if there's implied order of the rows.
>
>
> On Mon, Jul 21, 2014 at 12:22 PM, Pat Ferrel <pa...@gmail.com> wrote:
>
>> I appreciate that you can’t read all the back and forth Dmitriy hence the
>> private email. Please disregard all other code or talk in the thread for
>> the moment.
>>
>> Does a DRM need to have a row for every sequential row key from 0 to
>> nrow-1 ? Can there be missing row keys in the sequence and will they be
>> treated as {}, an all zero row? In terms of the rdd in the CheckpointedDrm
>> these “missing” rows will not have a corresponding n => {}, they will just
>> not exist in the rdd. This will happen when a row is “missing” from the DRM
>> but the true cardinality is known and passed in to the CheckpointedDrm
>> constructor.
>>
>> Will R-like operations on these matrices work correctly. Will A.t %*% A
>> and A + 1 work correctly?
>>
>> The answer is no, but _should_ they work correctly?
>>
>>
>
>
Re: Problem of dimensions
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
at the very least, we should check for it since obviously guarantees are
broken with A+1.
as for where to check, it is a problem of production operators vs. lazy
consumption operators. The answer to that is really whether consumption
requiring consistent naming is more frequent thing than not.
At this point it seems that relatively few (1, at this point) operators
require consitent int-keying. which means it is easier to prepend them with
test/fix operations.
Production does not always guarantee (drmWrap doesn't guarantee consistent
keying, neither does drmFromHdfs()). Like i said, there are probably some
additional computational cases that may produce this as well that is not
readily obvious.
On Mon, Jul 21, 2014 at 4:15 PM, Anand Avati <av...@gluster.org> wrote:
> No, I don't think multiplication is affected by missing rows (though I'm
> not sure). I'm saying prevention is better than cure. Pat's original
> problem was A and B were having incompatible dimensions, because missing
> rows had resulted in decremented nrow value. So, by solving an AewScalar
> problem with missing rows, we would have added new code to a problem which
> nobody has yet encountered in real world, and the original problem is still
> "unfixed". Instead it might be safer to say missing rows are not supported
> and avoid the new code which nobody use in the end.
>
> Int key DRM validations are best done (optionally) at output of mapBlock()
> (compare old int array with new int array) and drmHDFS if on-disk drm had
> missing rows to begin with. Otherwise there is no chance for rows to go
> missing (drmParallelize() guarantees, other operators guarantee).
>
> On Mon, Jul 21, 2014 at 4:04 PM, Dmitriy Lyubimov <dl...@gmail.com>
> wrote:
>
> > you are saying multiplication crowd is affected by this too?
> >
> >
> >
> > On Mon, Jul 21, 2014 at 4:02 PM, Anand Avati <av...@gluster.org> wrote:
> >
> > > On Mon, Jul 21, 2014 at 3:54 PM, Dmitriy Lyubimov <dl...@gmail.com>
> > > wrote:
> > >
> > > > ok so it looks like ew scalar needs to do the all-rows test in case
> of
> > > > DrmLike[Int], i.e. lazy props nrow == count (and nrow in this case
> > gives
> > > > max(key)+1). if test doesn't hold, then it needs to pre-process by
> > > > cogrouping with parallelizing 0 until nrow and insert stuff where is
> > > > missing. Some care needs to be taken with co-grouping shuffle, as
> > usual,
> > > > based on expected number of added rows.
> > > >
> > > > ok this is probably an easy fix.
> > >
> > >
> > > Well, we could add this code (though I still don't see a practical use
> > for
> > > it). But I dont see how this will help Pat's cause. His problem is
> fixing
> > > up cardinality for matrix multiplication. Should we add such
> cardinality
> > > fixup for all operators? If we have to keep fixing up cardinality for
> > every
> > > operator which has the dimension check (like matrix multiplication),
> why
> > > even "support" missing rows to begin with? It only seems to be adding
> > extra
> > > ifs and elses. Instead specify the requirement to have full rows for
> Int
> > > keyed DRMs and avoid code cruft to fixup things.
> > >
> > >
> > >
> > >
> > >
> > > >
> > > > On Mon, Jul 21, 2014 at 3:50 PM, Anand Avati <av...@gluster.org>
> > wrote:
> > > >
> > > > > On Mon, Jul 21, 2014 at 3:46 PM, Pat Ferrel <pa...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > And the conversion to Matrix instantiates the new rows, why not
> the
> > > > > > conversion to Dense?
> > > > >
> > > > >
> > > > > I addressed this point too in my previous emails. The problem with
> > > > plus(n)
> > > > > is not about sparse vs dense. It is in-core Matrix vs DRM.
> > > > >
> > > > > Thanks
> > > > >
> > > > >
> > > > >
> > > > > > On Jul 21, 2014, at 3:41 PM, Anand Avati <av...@gluster.org>
> > wrote:
> > > > > >
> > > > > > On Mon, Jul 21, 2014 at 3:35 PM, Pat Ferrel <
> pat@occamsmachete.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > If you do drm.plus(1) this converts to a dense matrix, which is
> > > what
> > > > > the
> > > > > > > result must be anyway, and does add the scalar to all rows,
> even
> > > > > missing
> > > > > > > ones.
> > > > > > >
> > > > > > >
> > > > > > Pat, I mentioned this in my previous email already. drm.plus(1)
> > > > > completely
> > > > > > misses the point. It converts DRM into an in-core matrix and
> > applies
> > > > > plus()
> > > > > > method on Matrix. The result is a Matrix, not DRM.
> > > > > >
> > > > > > drm.plus(1) is EXACTLY the same as:
> > > > > >
> > > > > > Matrix m = drm.collect()
> > > > > > m.plus(1)
> > > > > >
> > > > > > The implicit def drm2InCore() syntactic sugar is probably turning
> > out
> > > > to
> > > > > be
> > > > > > dangerous in this case, in terms of hinting the wrong meaning.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > > On Jul 21, 2014, at 3:23 PM, Dmitriy Lyubimov <
> dlieu.7@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > perhaps just compare row count with max(key)? that's exactly
> what
> > > > lazy
> > > > > > > nrow() currently does in this case.
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jul 21, 2014 at 3:21 PM, Dmitriy Lyubimov <
> > > dlieu.7@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >>
> > > > > > >> ok. so it should be easy to fix at least everything but
> > > elementwise
> > > > > > > scalar
> > > > > > >> i guess.
> > > > > > >>
> > > > > > >> Since the notion of "missing rows" is only defined for
> int-keyed
> > > > > > > datasets,
> > > > > > >> then ew scalar technically should work for non-int keyed
> > datasets
> > > > > > > already.
> > > > > > >>
> > > > > > >> as for int-keyed datasets, i am not sure what is the best
> > > strategy.
> > > > > > >> Obviously, one can define sort of normalization/validation of
> > > > > int-keyed
> > > > > > >> dataset routine, but it would be fairly expensive to run "just
> > > > > because".
> > > > > > >> Perhaps there's a cheap test (as cheap as row count job) to
> run
> > to
> > > > > test
> > > > > > > for
> > > > > > >> int keys consistency when matrix is first created.
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> On Mon, Jul 21, 2014 at 3:12 PM, Anand Avati <
> avati@gluster.org
> > >
> > > > > wrote:
> > > > > > >>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> On Mon, Jul 21, 2014 at 3:08 PM, Dmitriy Lyubimov <
> > > > dlieu.7@gmail.com
> > > > > >
> > > > > > >>> wrote:
> > > > > > >>>
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> On Mon, Jul 21, 2014 at 3:06 PM, Anand Avati <
> > avati@gluster.org
> > > >
> > > > > > > wrote:
> > > > > > >>>>
> > > > > > >>>>> Dmitriy, comments inline -
> > > > > > >>>>>
> > > > > > >>>>> On Jul 21, 2014, at 1:12 PM, Dmitriy Lyubimov <
> > > dlieu.7@gmail.com
> > > > >
> > > > > > >>>>> wrote:
> > > > > > >>>>>
> > > > > > >>>>>> And no, i suppose it is ok to have "missing" rows even in
> > case
> > > > of
> > > > > > >>>>>> int-keyed matrices.
> > > > > > >>>>>>
> > > > > > >>>>>> there's one thing that you probably should be aware in
> this
> > > > > context
> > > > > > >>>>>> though: many algorithms don't survive empty (row-less)
> > > > partitions,
> > > > > > in
> > > > > > >>>>>> whatever way they may come to be. Other than that, I don't
> > > feel
> > > > > > > every row
> > > > > > >>>>>> must be present -- even if there's implied order of the
> > rows.
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>>> I'm not sure if that is necessarily true. There are three
> > > > operators
> > > > > > >>>>> which break pretty badly with with missing rows.
> > > > > > >>>>>
> > > > > > >>>>> AewScalar - operation like A + 1 is just not applied on the
> > > > missing
> > > > > > >>>>> row, so the final matrix will have 0's in place of 1s.
> > > > > > >>>>>
> > > > > > >>>>
> > > > > > >>>> Indeed. i have no recourse at this point.
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>>>
> > > > > > >>>>> AewB, CbindAB - function after cogroup() throws exception
> if
> > a
> > > > row
> > > > > > was
> > > > > > >>>>> present on only one matrix. So I guess it is OK to have
> > missing
> > > > > rows
> > > > > > > as
> > > > > > >>>>> long as both A and B have the exact same missing row set.
> > > > Somewhat
> > > > > > >>>>> quirky/nuanced requirement.
> > > > > > >>>>>
> > > > > > >>>>
> > > > > > >>>> Agree. i actually was not aware that's a cogroup() semantics
> > in
> > > > > spark.
> > > > > > > I
> > > > > > >>>> though it would have an outer join semantics (as in Pig, i
> > > > believe).
> > > > > > > Alas,
> > > > > > >>>> no recourse at this point either.
> > > > > > >>>>
> > > > > > >>>
> > > > > > >>> The exception is actually during reduceLeft after cogroup().
> > > > > Cogroup()
> > > > > > >>> itself is probably an outer-join.
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Problem of dimensions
Posted by Anand Avati <av...@gluster.org>.
No, I don't think multiplication is affected by missing rows (though I'm
not sure). I'm saying prevention is better than cure. Pat's original
problem was A and B were having incompatible dimensions, because missing
rows had resulted in decremented nrow value. So, by solving an AewScalar
problem with missing rows, we would have added new code to a problem which
nobody has yet encountered in real world, and the original problem is still
"unfixed". Instead it might be safer to say missing rows are not supported
and avoid the new code which nobody use in the end.
Int key DRM validations are best done (optionally) at output of mapBlock()
(compare old int array with new int array) and drmHDFS if on-disk drm had
missing rows to begin with. Otherwise there is no chance for rows to go
missing (drmParallelize() guarantees, other operators guarantee).
On Mon, Jul 21, 2014 at 4:04 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> you are saying multiplication crowd is affected by this too?
>
>
>
> On Mon, Jul 21, 2014 at 4:02 PM, Anand Avati <av...@gluster.org> wrote:
>
> > On Mon, Jul 21, 2014 at 3:54 PM, Dmitriy Lyubimov <dl...@gmail.com>
> > wrote:
> >
> > > ok so it looks like ew scalar needs to do the all-rows test in case of
> > > DrmLike[Int], i.e. lazy props nrow == count (and nrow in this case
> gives
> > > max(key)+1). if test doesn't hold, then it needs to pre-process by
> > > cogrouping with parallelizing 0 until nrow and insert stuff where is
> > > missing. Some care needs to be taken with co-grouping shuffle, as
> usual,
> > > based on expected number of added rows.
> > >
> > > ok this is probably an easy fix.
> >
> >
> > Well, we could add this code (though I still don't see a practical use
> for
> > it). But I dont see how this will help Pat's cause. His problem is fixing
> > up cardinality for matrix multiplication. Should we add such cardinality
> > fixup for all operators? If we have to keep fixing up cardinality for
> every
> > operator which has the dimension check (like matrix multiplication), why
> > even "support" missing rows to begin with? It only seems to be adding
> extra
> > ifs and elses. Instead specify the requirement to have full rows for Int
> > keyed DRMs and avoid code cruft to fixup things.
> >
> >
> >
> >
> >
> > >
> > > On Mon, Jul 21, 2014 at 3:50 PM, Anand Avati <av...@gluster.org>
> wrote:
> > >
> > > > On Mon, Jul 21, 2014 at 3:46 PM, Pat Ferrel <pa...@gmail.com>
> > > wrote:
> > > >
> > > > > And the conversion to Matrix instantiates the new rows, why not the
> > > > > conversion to Dense?
> > > >
> > > >
> > > > I addressed this point too in my previous emails. The problem with
> > > plus(n)
> > > > is not about sparse vs dense. It is in-core Matrix vs DRM.
> > > >
> > > > Thanks
> > > >
> > > >
> > > >
> > > > > On Jul 21, 2014, at 3:41 PM, Anand Avati <av...@gluster.org>
> wrote:
> > > > >
> > > > > On Mon, Jul 21, 2014 at 3:35 PM, Pat Ferrel <pat@occamsmachete.com
> >
> > > > wrote:
> > > > >
> > > > > > If you do drm.plus(1) this converts to a dense matrix, which is
> > what
> > > > the
> > > > > > result must be anyway, and does add the scalar to all rows, even
> > > > missing
> > > > > > ones.
> > > > > >
> > > > > >
> > > > > Pat, I mentioned this in my previous email already. drm.plus(1)
> > > > completely
> > > > > misses the point. It converts DRM into an in-core matrix and
> applies
> > > > plus()
> > > > > method on Matrix. The result is a Matrix, not DRM.
> > > > >
> > > > > drm.plus(1) is EXACTLY the same as:
> > > > >
> > > > > Matrix m = drm.collect()
> > > > > m.plus(1)
> > > > >
> > > > > The implicit def drm2InCore() syntactic sugar is probably turning
> out
> > > to
> > > > be
> > > > > dangerous in this case, in terms of hinting the wrong meaning.
> > > > >
> > > > > Thanks
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > > On Jul 21, 2014, at 3:23 PM, Dmitriy Lyubimov <dlieu.7@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > perhaps just compare row count with max(key)? that's exactly what
> > > lazy
> > > > > > nrow() currently does in this case.
> > > > > >
> > > > > >
> > > > > > On Mon, Jul 21, 2014 at 3:21 PM, Dmitriy Lyubimov <
> > dlieu.7@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > >>
> > > > > >> ok. so it should be easy to fix at least everything but
> > elementwise
> > > > > > scalar
> > > > > >> i guess.
> > > > > >>
> > > > > >> Since the notion of "missing rows" is only defined for int-keyed
> > > > > > datasets,
> > > > > >> then ew scalar technically should work for non-int keyed
> datasets
> > > > > > already.
> > > > > >>
> > > > > >> as for int-keyed datasets, i am not sure what is the best
> > strategy.
> > > > > >> Obviously, one can define sort of normalization/validation of
> > > > int-keyed
> > > > > >> dataset routine, but it would be fairly expensive to run "just
> > > > because".
> > > > > >> Perhaps there's a cheap test (as cheap as row count job) to run
> to
> > > > test
> > > > > > for
> > > > > >> int keys consistency when matrix is first created.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Mon, Jul 21, 2014 at 3:12 PM, Anand Avati <avati@gluster.org
> >
> > > > wrote:
> > > > > >>
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>> On Mon, Jul 21, 2014 at 3:08 PM, Dmitriy Lyubimov <
> > > dlieu.7@gmail.com
> > > > >
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>>>
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> On Mon, Jul 21, 2014 at 3:06 PM, Anand Avati <
> avati@gluster.org
> > >
> > > > > > wrote:
> > > > > >>>>
> > > > > >>>>> Dmitriy, comments inline -
> > > > > >>>>>
> > > > > >>>>> On Jul 21, 2014, at 1:12 PM, Dmitriy Lyubimov <
> > dlieu.7@gmail.com
> > > >
> > > > > >>>>> wrote:
> > > > > >>>>>
> > > > > >>>>>> And no, i suppose it is ok to have "missing" rows even in
> case
> > > of
> > > > > >>>>>> int-keyed matrices.
> > > > > >>>>>>
> > > > > >>>>>> there's one thing that you probably should be aware in this
> > > > context
> > > > > >>>>>> though: many algorithms don't survive empty (row-less)
> > > partitions,
> > > > > in
> > > > > >>>>>> whatever way they may come to be. Other than that, I don't
> > feel
> > > > > > every row
> > > > > >>>>>> must be present -- even if there's implied order of the
> rows.
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>> I'm not sure if that is necessarily true. There are three
> > > operators
> > > > > >>>>> which break pretty badly with with missing rows.
> > > > > >>>>>
> > > > > >>>>> AewScalar - operation like A + 1 is just not applied on the
> > > missing
> > > > > >>>>> row, so the final matrix will have 0's in place of 1s.
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>> Indeed. i have no recourse at this point.
> > > > > >>>>
> > > > > >>>>
> > > > > >>>>>
> > > > > >>>>> AewB, CbindAB - function after cogroup() throws exception if
> a
> > > row
> > > > > was
> > > > > >>>>> present on only one matrix. So I guess it is OK to have
> missing
> > > > rows
> > > > > > as
> > > > > >>>>> long as both A and B have the exact same missing row set.
> > > Somewhat
> > > > > >>>>> quirky/nuanced requirement.
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>> Agree. i actually was not aware that's a cogroup() semantics
> in
> > > > spark.
> > > > > > I
> > > > > >>>> though it would have an outer join semantics (as in Pig, i
> > > believe).
> > > > > > Alas,
> > > > > >>>> no recourse at this point either.
> > > > > >>>>
> > > > > >>>
> > > > > >>> The exception is actually during reduceLeft after cogroup().
> > > > Cogroup()
> > > > > >>> itself is probably an outer-join.
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: Problem of dimensions
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
you are saying multiplication crowd is affected by this too?
On Mon, Jul 21, 2014 at 4:02 PM, Anand Avati <av...@gluster.org> wrote:
> On Mon, Jul 21, 2014 at 3:54 PM, Dmitriy Lyubimov <dl...@gmail.com>
> wrote:
>
> > ok so it looks like ew scalar needs to do the all-rows test in case of
> > DrmLike[Int], i.e. lazy props nrow == count (and nrow in this case gives
> > max(key)+1). if test doesn't hold, then it needs to pre-process by
> > cogrouping with parallelizing 0 until nrow and insert stuff where is
> > missing. Some care needs to be taken with co-grouping shuffle, as usual,
> > based on expected number of added rows.
> >
> > ok this is probably an easy fix.
>
>
> Well, we could add this code (though I still don't see a practical use for
> it). But I dont see how this will help Pat's cause. His problem is fixing
> up cardinality for matrix multiplication. Should we add such cardinality
> fixup for all operators? If we have to keep fixing up cardinality for every
> operator which has the dimension check (like matrix multiplication), why
> even "support" missing rows to begin with? It only seems to be adding extra
> ifs and elses. Instead specify the requirement to have full rows for Int
> keyed DRMs and avoid code cruft to fixup things.
>
>
>
>
>
> >
> > On Mon, Jul 21, 2014 at 3:50 PM, Anand Avati <av...@gluster.org> wrote:
> >
> > > On Mon, Jul 21, 2014 at 3:46 PM, Pat Ferrel <pa...@gmail.com>
> > wrote:
> > >
> > > > And the conversion to Matrix instantiates the new rows, why not the
> > > > conversion to Dense?
> > >
> > >
> > > I addressed this point too in my previous emails. The problem with
> > plus(n)
> > > is not about sparse vs dense. It is in-core Matrix vs DRM.
> > >
> > > Thanks
> > >
> > >
> > >
> > > > On Jul 21, 2014, at 3:41 PM, Anand Avati <av...@gluster.org> wrote:
> > > >
> > > > On Mon, Jul 21, 2014 at 3:35 PM, Pat Ferrel <pa...@occamsmachete.com>
> > > wrote:
> > > >
> > > > > If you do drm.plus(1) this converts to a dense matrix, which is
> what
> > > the
> > > > > result must be anyway, and does add the scalar to all rows, even
> > > missing
> > > > > ones.
> > > > >
> > > > >
> > > > Pat, I mentioned this in my previous email already. drm.plus(1)
> > > completely
> > > > misses the point. It converts DRM into an in-core matrix and applies
> > > plus()
> > > > method on Matrix. The result is a Matrix, not DRM.
> > > >
> > > > drm.plus(1) is EXACTLY the same as:
> > > >
> > > > Matrix m = drm.collect()
> > > > m.plus(1)
> > > >
> > > > The implicit def drm2InCore() syntactic sugar is probably turning out
> > to
> > > be
> > > > dangerous in this case, in terms of hinting the wrong meaning.
> > > >
> > > > Thanks
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > > On Jul 21, 2014, at 3:23 PM, Dmitriy Lyubimov <dl...@gmail.com>
> > > wrote:
> > > > >
> > > > > perhaps just compare row count with max(key)? that's exactly what
> > lazy
> > > > > nrow() currently does in this case.
> > > > >
> > > > >
> > > > > On Mon, Jul 21, 2014 at 3:21 PM, Dmitriy Lyubimov <
> dlieu.7@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > >>
> > > > >> ok. so it should be easy to fix at least everything but
> elementwise
> > > > > scalar
> > > > >> i guess.
> > > > >>
> > > > >> Since the notion of "missing rows" is only defined for int-keyed
> > > > > datasets,
> > > > >> then ew scalar technically should work for non-int keyed datasets
> > > > > already.
> > > > >>
> > > > >> as for int-keyed datasets, i am not sure what is the best
> strategy.
> > > > >> Obviously, one can define sort of normalization/validation of
> > > int-keyed
> > > > >> dataset routine, but it would be fairly expensive to run "just
> > > because".
> > > > >> Perhaps there's a cheap test (as cheap as row count job) to run to
> > > test
> > > > > for
> > > > >> int keys consistency when matrix is first created.
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Mon, Jul 21, 2014 at 3:12 PM, Anand Avati <av...@gluster.org>
> > > wrote:
> > > > >>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> On Mon, Jul 21, 2014 at 3:08 PM, Dmitriy Lyubimov <
> > dlieu.7@gmail.com
> > > >
> > > > >>> wrote:
> > > > >>>
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> On Mon, Jul 21, 2014 at 3:06 PM, Anand Avati <avati@gluster.org
> >
> > > > > wrote:
> > > > >>>>
> > > > >>>>> Dmitriy, comments inline -
> > > > >>>>>
> > > > >>>>> On Jul 21, 2014, at 1:12 PM, Dmitriy Lyubimov <
> dlieu.7@gmail.com
> > >
> > > > >>>>> wrote:
> > > > >>>>>
> > > > >>>>>> And no, i suppose it is ok to have "missing" rows even in case
> > of
> > > > >>>>>> int-keyed matrices.
> > > > >>>>>>
> > > > >>>>>> there's one thing that you probably should be aware in this
> > > context
> > > > >>>>>> though: many algorithms don't survive empty (row-less)
> > partitions,
> > > > in
> > > > >>>>>> whatever way they may come to be. Other than that, I don't
> feel
> > > > > every row
> > > > >>>>>> must be present -- even if there's implied order of the rows.
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>> I'm not sure if that is necessarily true. There are three
> > operators
> > > > >>>>> which break pretty badly with with missing rows.
> > > > >>>>>
> > > > >>>>> AewScalar - operation like A + 1 is just not applied on the
> > missing
> > > > >>>>> row, so the final matrix will have 0's in place of 1s.
> > > > >>>>>
> > > > >>>>
> > > > >>>> Indeed. i have no recourse at this point.
> > > > >>>>
> > > > >>>>
> > > > >>>>>
> > > > >>>>> AewB, CbindAB - function after cogroup() throws exception if a
> > row
> > > > was
> > > > >>>>> present on only one matrix. So I guess it is OK to have missing
> > > rows
> > > > > as
> > > > >>>>> long as both A and B have the exact same missing row set.
> > Somewhat
> > > > >>>>> quirky/nuanced requirement.
> > > > >>>>>
> > > > >>>>
> > > > >>>> Agree. i actually was not aware that's a cogroup() semantics in
> > > spark.
> > > > > I
> > > > >>>> though it would have an outer join semantics (as in Pig, i
> > believe).
> > > > > Alas,
> > > > >>>> no recourse at this point either.
> > > > >>>>
> > > > >>>
> > > > >>> The exception is actually during reduceLeft after cogroup().
> > > Cogroup()
> > > > >>> itself is probably an outer-join.
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>
> > > > >
> > > > >
> > > >
> > > >
> > >
> >
>
Re: Problem of dimensions
Posted by Anand Avati <av...@gluster.org>.
On Mon, Jul 21, 2014 at 3:54 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> ok so it looks like ew scalar needs to do the all-rows test in case of
> DrmLike[Int], i.e. lazy props nrow == count (and nrow in this case gives
> max(key)+1). if test doesn't hold, then it needs to pre-process by
> cogrouping with parallelizing 0 until nrow and insert stuff where is
> missing. Some care needs to be taken with co-grouping shuffle, as usual,
> based on expected number of added rows.
>
> ok this is probably an easy fix.
Well, we could add this code (though I still don't see a practical use for
it). But I dont see how this will help Pat's cause. His problem is fixing
up cardinality for matrix multiplication. Should we add such cardinality
fixup for all operators? If we have to keep fixing up cardinality for every
operator which has the dimension check (like matrix multiplication), why
even "support" missing rows to begin with? It only seems to be adding extra
ifs and elses. Instead specify the requirement to have full rows for Int
keyed DRMs and avoid code cruft to fixup things.
>
> On Mon, Jul 21, 2014 at 3:50 PM, Anand Avati <av...@gluster.org> wrote:
>
> > On Mon, Jul 21, 2014 at 3:46 PM, Pat Ferrel <pa...@gmail.com>
> wrote:
> >
> > > And the conversion to Matrix instantiates the new rows, why not the
> > > conversion to Dense?
> >
> >
> > I addressed this point too in my previous emails. The problem with
> plus(n)
> > is not about sparse vs dense. It is in-core Matrix vs DRM.
> >
> > Thanks
> >
> >
> >
> > > On Jul 21, 2014, at 3:41 PM, Anand Avati <av...@gluster.org> wrote:
> > >
> > > On Mon, Jul 21, 2014 at 3:35 PM, Pat Ferrel <pa...@occamsmachete.com>
> > wrote:
> > >
> > > > If you do drm.plus(1) this converts to a dense matrix, which is what
> > the
> > > > result must be anyway, and does add the scalar to all rows, even
> > missing
> > > > ones.
> > > >
> > > >
> > > Pat, I mentioned this in my previous email already. drm.plus(1)
> > completely
> > > misses the point. It converts DRM into an in-core matrix and applies
> > plus()
> > > method on Matrix. The result is a Matrix, not DRM.
> > >
> > > drm.plus(1) is EXACTLY the same as:
> > >
> > > Matrix m = drm.collect()
> > > m.plus(1)
> > >
> > > The implicit def drm2InCore() syntactic sugar is probably turning out
> to
> > be
> > > dangerous in this case, in terms of hinting the wrong meaning.
> > >
> > > Thanks
> > >
> > >
> > >
> > >
> > >
> > > > On Jul 21, 2014, at 3:23 PM, Dmitriy Lyubimov <dl...@gmail.com>
> > wrote:
> > > >
> > > > perhaps just compare row count with max(key)? that's exactly what
> lazy
> > > > nrow() currently does in this case.
> > > >
> > > >
> > > > On Mon, Jul 21, 2014 at 3:21 PM, Dmitriy Lyubimov <dlieu.7@gmail.com
> >
> > > > wrote:
> > > >
> > > >>
> > > >> ok. so it should be easy to fix at least everything but elementwise
> > > > scalar
> > > >> i guess.
> > > >>
> > > >> Since the notion of "missing rows" is only defined for int-keyed
> > > > datasets,
> > > >> then ew scalar technically should work for non-int keyed datasets
> > > > already.
> > > >>
> > > >> as for int-keyed datasets, i am not sure what is the best strategy.
> > > >> Obviously, one can define sort of normalization/validation of
> > int-keyed
> > > >> dataset routine, but it would be fairly expensive to run "just
> > because".
> > > >> Perhaps there's a cheap test (as cheap as row count job) to run to
> > test
> > > > for
> > > >> int keys consistency when matrix is first created.
> > > >>
> > > >>
> > > >>
> > > >> On Mon, Jul 21, 2014 at 3:12 PM, Anand Avati <av...@gluster.org>
> > wrote:
> > > >>
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Mon, Jul 21, 2014 at 3:08 PM, Dmitriy Lyubimov <
> dlieu.7@gmail.com
> > >
> > > >>> wrote:
> > > >>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> On Mon, Jul 21, 2014 at 3:06 PM, Anand Avati <av...@gluster.org>
> > > > wrote:
> > > >>>>
> > > >>>>> Dmitriy, comments inline -
> > > >>>>>
> > > >>>>> On Jul 21, 2014, at 1:12 PM, Dmitriy Lyubimov <dlieu.7@gmail.com
> >
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>> And no, i suppose it is ok to have "missing" rows even in case
> of
> > > >>>>>> int-keyed matrices.
> > > >>>>>>
> > > >>>>>> there's one thing that you probably should be aware in this
> > context
> > > >>>>>> though: many algorithms don't survive empty (row-less)
> partitions,
> > > in
> > > >>>>>> whatever way they may come to be. Other than that, I don't feel
> > > > every row
> > > >>>>>> must be present -- even if there's implied order of the rows.
> > > >>>>>>
> > > >>>>>
> > > >>>>> I'm not sure if that is necessarily true. There are three
> operators
> > > >>>>> which break pretty badly with with missing rows.
> > > >>>>>
> > > >>>>> AewScalar - operation like A + 1 is just not applied on the
> missing
> > > >>>>> row, so the final matrix will have 0's in place of 1s.
> > > >>>>>
> > > >>>>
> > > >>>> Indeed. i have no recourse at this point.
> > > >>>>
> > > >>>>
> > > >>>>>
> > > >>>>> AewB, CbindAB - function after cogroup() throws exception if a
> row
> > > was
> > > >>>>> present on only one matrix. So I guess it is OK to have missing
> > rows
> > > > as
> > > >>>>> long as both A and B have the exact same missing row set.
> Somewhat
> > > >>>>> quirky/nuanced requirement.
> > > >>>>>
> > > >>>>
> > > >>>> Agree. i actually was not aware that's a cogroup() semantics in
> > spark.
> > > > I
> > > >>>> though it would have an outer join semantics (as in Pig, i
> believe).
> > > > Alas,
> > > >>>> no recourse at this point either.
> > > >>>>
> > > >>>
> > > >>> The exception is actually during reduceLeft after cogroup().
> > Cogroup()
> > > >>> itself is probably an outer-join.
> > > >>>
> > > >>>
> > > >>>
> > > >>
> > > >
> > > >
> > >
> > >
> >
>
Re: Problem of dimensions
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
ok so it looks like ew scalar needs to do the all-rows test in case of
DrmLike[Int], i.e. lazy props nrow == count (and nrow in this case gives
max(key)+1). if test doesn't hold, then it needs to pre-process by
cogrouping with parallelizing 0 until nrow and insert stuff where is
missing. Some care needs to be taken with co-grouping shuffle, as usual,
based on expected number of added rows.
ok this is probably an easy fix.
On Mon, Jul 21, 2014 at 3:50 PM, Anand Avati <av...@gluster.org> wrote:
> On Mon, Jul 21, 2014 at 3:46 PM, Pat Ferrel <pa...@gmail.com> wrote:
>
> > And the conversion to Matrix instantiates the new rows, why not the
> > conversion to Dense?
>
>
> I addressed this point too in my previous emails. The problem with plus(n)
> is not about sparse vs dense. It is in-core Matrix vs DRM.
>
> Thanks
>
>
>
> > On Jul 21, 2014, at 3:41 PM, Anand Avati <av...@gluster.org> wrote:
> >
> > On Mon, Jul 21, 2014 at 3:35 PM, Pat Ferrel <pa...@occamsmachete.com>
> wrote:
> >
> > > If you do drm.plus(1) this converts to a dense matrix, which is what
> the
> > > result must be anyway, and does add the scalar to all rows, even
> missing
> > > ones.
> > >
> > >
> > Pat, I mentioned this in my previous email already. drm.plus(1)
> completely
> > misses the point. It converts DRM into an in-core matrix and applies
> plus()
> > method on Matrix. The result is a Matrix, not DRM.
> >
> > drm.plus(1) is EXACTLY the same as:
> >
> > Matrix m = drm.collect()
> > m.plus(1)
> >
> > The implicit def drm2InCore() syntactic sugar is probably turning out to
> be
> > dangerous in this case, in terms of hinting the wrong meaning.
> >
> > Thanks
> >
> >
> >
> >
> >
> > > On Jul 21, 2014, at 3:23 PM, Dmitriy Lyubimov <dl...@gmail.com>
> wrote:
> > >
> > > perhaps just compare row count with max(key)? that's exactly what lazy
> > > nrow() currently does in this case.
> > >
> > >
> > > On Mon, Jul 21, 2014 at 3:21 PM, Dmitriy Lyubimov <dl...@gmail.com>
> > > wrote:
> > >
> > >>
> > >> ok. so it should be easy to fix at least everything but elementwise
> > > scalar
> > >> i guess.
> > >>
> > >> Since the notion of "missing rows" is only defined for int-keyed
> > > datasets,
> > >> then ew scalar technically should work for non-int keyed datasets
> > > already.
> > >>
> > >> as for int-keyed datasets, i am not sure what is the best strategy.
> > >> Obviously, one can define sort of normalization/validation of
> int-keyed
> > >> dataset routine, but it would be fairly expensive to run "just
> because".
> > >> Perhaps there's a cheap test (as cheap as row count job) to run to
> test
> > > for
> > >> int keys consistency when matrix is first created.
> > >>
> > >>
> > >>
> > >> On Mon, Jul 21, 2014 at 3:12 PM, Anand Avati <av...@gluster.org>
> wrote:
> > >>
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Jul 21, 2014 at 3:08 PM, Dmitriy Lyubimov <dlieu.7@gmail.com
> >
> > >>> wrote:
> > >>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Mon, Jul 21, 2014 at 3:06 PM, Anand Avati <av...@gluster.org>
> > > wrote:
> > >>>>
> > >>>>> Dmitriy, comments inline -
> > >>>>>
> > >>>>> On Jul 21, 2014, at 1:12 PM, Dmitriy Lyubimov <dl...@gmail.com>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> And no, i suppose it is ok to have "missing" rows even in case of
> > >>>>>> int-keyed matrices.
> > >>>>>>
> > >>>>>> there's one thing that you probably should be aware in this
> context
> > >>>>>> though: many algorithms don't survive empty (row-less) partitions,
> > in
> > >>>>>> whatever way they may come to be. Other than that, I don't feel
> > > every row
> > >>>>>> must be present -- even if there's implied order of the rows.
> > >>>>>>
> > >>>>>
> > >>>>> I'm not sure if that is necessarily true. There are three operators
> > >>>>> which break pretty badly with with missing rows.
> > >>>>>
> > >>>>> AewScalar - operation like A + 1 is just not applied on the missing
> > >>>>> row, so the final matrix will have 0's in place of 1s.
> > >>>>>
> > >>>>
> > >>>> Indeed. i have no recourse at this point.
> > >>>>
> > >>>>
> > >>>>>
> > >>>>> AewB, CbindAB - function after cogroup() throws exception if a row
> > was
> > >>>>> present on only one matrix. So I guess it is OK to have missing
> rows
> > > as
> > >>>>> long as both A and B have the exact same missing row set. Somewhat
> > >>>>> quirky/nuanced requirement.
> > >>>>>
> > >>>>
> > >>>> Agree. i actually was not aware that's a cogroup() semantics in
> spark.
> > > I
> > >>>> though it would have an outer join semantics (as in Pig, i believe).
> > > Alas,
> > >>>> no recourse at this point either.
> > >>>>
> > >>>
> > >>> The exception is actually during reduceLeft after cogroup().
> Cogroup()
> > >>> itself is probably an outer-join.
> > >>>
> > >>>
> > >>>
> > >>
> > >
> > >
> >
> >
>
Re: Problem of dimensions
Posted by Anand Avati <av...@gluster.org>.
On Mon, Jul 21, 2014 at 3:46 PM, Pat Ferrel <pa...@gmail.com> wrote:
> And the conversion to Matrix instantiates the new rows, why not the
> conversion to Dense?
I addressed this point too in my previous emails. The problem with plus(n)
is not about sparse vs dense. It is in-core Matrix vs DRM.
Thanks
> On Jul 21, 2014, at 3:41 PM, Anand Avati <av...@gluster.org> wrote:
>
> On Mon, Jul 21, 2014 at 3:35 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:
>
> > If you do drm.plus(1) this converts to a dense matrix, which is what the
> > result must be anyway, and does add the scalar to all rows, even missing
> > ones.
> >
> >
> Pat, I mentioned this in my previous email already. drm.plus(1) completely
> misses the point. It converts DRM into an in-core matrix and applies plus()
> method on Matrix. The result is a Matrix, not DRM.
>
> drm.plus(1) is EXACTLY the same as:
>
> Matrix m = drm.collect()
> m.plus(1)
>
> The implicit def drm2InCore() syntactic sugar is probably turning out to be
> dangerous in this case, in terms of hinting the wrong meaning.
>
> Thanks
>
>
>
>
>
> > On Jul 21, 2014, at 3:23 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> >
> > perhaps just compare row count with max(key)? that's exactly what lazy
> > nrow() currently does in this case.
> >
> >
> > On Mon, Jul 21, 2014 at 3:21 PM, Dmitriy Lyubimov <dl...@gmail.com>
> > wrote:
> >
> >>
> >> ok. so it should be easy to fix at least everything but elementwise
> > scalar
> >> i guess.
> >>
> >> Since the notion of "missing rows" is only defined for int-keyed
> > datasets,
> >> then ew scalar technically should work for non-int keyed datasets
> > already.
> >>
> >> as for int-keyed datasets, i am not sure what is the best strategy.
> >> Obviously, one can define sort of normalization/validation of int-keyed
> >> dataset routine, but it would be fairly expensive to run "just because".
> >> Perhaps there's a cheap test (as cheap as row count job) to run to test
> > for
> >> int keys consistency when matrix is first created.
> >>
> >>
> >>
> >> On Mon, Jul 21, 2014 at 3:12 PM, Anand Avati <av...@gluster.org> wrote:
> >>
> >>>
> >>>
> >>>
> >>> On Mon, Jul 21, 2014 at 3:08 PM, Dmitriy Lyubimov <dl...@gmail.com>
> >>> wrote:
> >>>
> >>>>
> >>>>
> >>>>
> >>>> On Mon, Jul 21, 2014 at 3:06 PM, Anand Avati <av...@gluster.org>
> > wrote:
> >>>>
> >>>>> Dmitriy, comments inline -
> >>>>>
> >>>>> On Jul 21, 2014, at 1:12 PM, Dmitriy Lyubimov <dl...@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> And no, i suppose it is ok to have "missing" rows even in case of
> >>>>>> int-keyed matrices.
> >>>>>>
> >>>>>> there's one thing that you probably should be aware in this context
> >>>>>> though: many algorithms don't survive empty (row-less) partitions,
> in
> >>>>>> whatever way they may come to be. Other than that, I don't feel
> > every row
> >>>>>> must be present -- even if there's implied order of the rows.
> >>>>>>
> >>>>>
> >>>>> I'm not sure if that is necessarily true. There are three operators
> >>>>> which break pretty badly with with missing rows.
> >>>>>
> >>>>> AewScalar - operation like A + 1 is just not applied on the missing
> >>>>> row, so the final matrix will have 0's in place of 1s.
> >>>>>
> >>>>
> >>>> Indeed. i have no recourse at this point.
> >>>>
> >>>>
> >>>>>
> >>>>> AewB, CbindAB - function after cogroup() throws exception if a row
> was
> >>>>> present on only one matrix. So I guess it is OK to have missing rows
> > as
> >>>>> long as both A and B have the exact same missing row set. Somewhat
> >>>>> quirky/nuanced requirement.
> >>>>>
> >>>>
> >>>> Agree. i actually was not aware that's a cogroup() semantics in spark.
> > I
> >>>> though it would have an outer join semantics (as in Pig, i believe).
> > Alas,
> >>>> no recourse at this point either.
> >>>>
> >>>
> >>> The exception is actually during reduceLeft after cogroup(). Cogroup()
> >>> itself is probably an outer-join.
> >>>
> >>>
> >>>
> >>
> >
> >
>
>
Re: Problem of dimensions
Posted by Pat Ferrel <pa...@gmail.com>.
And the conversion to Matrix instantiates the new rows, why not the conversion to Dense?
On Jul 21, 2014, at 3:41 PM, Anand Avati <av...@gluster.org> wrote:
On Mon, Jul 21, 2014 at 3:35 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:
> If you do drm.plus(1) this converts to a dense matrix, which is what the
> result must be anyway, and does add the scalar to all rows, even missing
> ones.
>
>
Pat, I mentioned this in my previous email already. drm.plus(1) completely
misses the point. It converts DRM into an in-core matrix and applies plus()
method on Matrix. The result is a Matrix, not DRM.
drm.plus(1) is EXACTLY the same as:
Matrix m = drm.collect()
m.plus(1)
The implicit def drm2InCore() syntactic sugar is probably turning out to be
dangerous in this case, in terms of hinting the wrong meaning.
Thanks
> On Jul 21, 2014, at 3:23 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
>
> perhaps just compare row count with max(key)? that's exactly what lazy
> nrow() currently does in this case.
>
>
> On Mon, Jul 21, 2014 at 3:21 PM, Dmitriy Lyubimov <dl...@gmail.com>
> wrote:
>
>>
>> ok. so it should be easy to fix at least everything but elementwise
> scalar
>> i guess.
>>
>> Since the notion of "missing rows" is only defined for int-keyed
> datasets,
>> then ew scalar technically should work for non-int keyed datasets
> already.
>>
>> as for int-keyed datasets, i am not sure what is the best strategy.
>> Obviously, one can define sort of normalization/validation of int-keyed
>> dataset routine, but it would be fairly expensive to run "just because".
>> Perhaps there's a cheap test (as cheap as row count job) to run to test
> for
>> int keys consistency when matrix is first created.
>>
>>
>>
>> On Mon, Jul 21, 2014 at 3:12 PM, Anand Avati <av...@gluster.org> wrote:
>>
>>>
>>>
>>>
>>> On Mon, Jul 21, 2014 at 3:08 PM, Dmitriy Lyubimov <dl...@gmail.com>
>>> wrote:
>>>
>>>>
>>>>
>>>>
>>>> On Mon, Jul 21, 2014 at 3:06 PM, Anand Avati <av...@gluster.org>
> wrote:
>>>>
>>>>> Dmitriy, comments inline -
>>>>>
>>>>> On Jul 21, 2014, at 1:12 PM, Dmitriy Lyubimov <dl...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> And no, i suppose it is ok to have "missing" rows even in case of
>>>>>> int-keyed matrices.
>>>>>>
>>>>>> there's one thing that you probably should be aware in this context
>>>>>> though: many algorithms don't survive empty (row-less) partitions, in
>>>>>> whatever way they may come to be. Other than that, I don't feel
> every row
>>>>>> must be present -- even if there's implied order of the rows.
>>>>>>
>>>>>
>>>>> I'm not sure if that is necessarily true. There are three operators
>>>>> which break pretty badly with with missing rows.
>>>>>
>>>>> AewScalar - operation like A + 1 is just not applied on the missing
>>>>> row, so the final matrix will have 0's in place of 1s.
>>>>>
>>>>
>>>> Indeed. i have no recourse at this point.
>>>>
>>>>
>>>>>
>>>>> AewB, CbindAB - function after cogroup() throws exception if a row was
>>>>> present on only one matrix. So I guess it is OK to have missing rows
> as
>>>>> long as both A and B have the exact same missing row set. Somewhat
>>>>> quirky/nuanced requirement.
>>>>>
>>>>
>>>> Agree. i actually was not aware that's a cogroup() semantics in spark.
> I
>>>> though it would have an outer join semantics (as in Pig, i believe).
> Alas,
>>>> no recourse at this point either.
>>>>
>>>
>>> The exception is actually during reduceLeft after cogroup(). Cogroup()
>>> itself is probably an outer-join.
>>>
>>>
>>>
>>
>
>
Re: Problem of dimensions
Posted by Anand Avati <av...@gluster.org>.
src/main/java/org/apache/mahout/math/AbstractMatrix.java:444: public
Matrix plus(double x)
which ends up seeming to "work" on a DRM because of
math-scala/src/main/scala/org/apache/mahout/math/drm/package.scala:
implicit def drm2InCore[K: ClassTag](drm: DrmLike[K]): Matrix = drm.collect
On Mon, Jul 21, 2014 at 3:44 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> not sure what is this plus() thing. Is it something that is not yet
> committed?
>
>
> On Mon, Jul 21, 2014 at 3:41 PM, Anand Avati <av...@gluster.org> wrote:
>
> > On Mon, Jul 21, 2014 at 3:35 PM, Pat Ferrel <pa...@occamsmachete.com>
> wrote:
> >
> > > If you do drm.plus(1) this converts to a dense matrix, which is what
> the
> > > result must be anyway, and does add the scalar to all rows, even
> missing
> > > ones.
> > >
> > >
> > Pat, I mentioned this in my previous email already. drm.plus(1)
> completely
> > misses the point. It converts DRM into an in-core matrix and applies
> plus()
> > method on Matrix. The result is a Matrix, not DRM.
> >
> > drm.plus(1) is EXACTLY the same as:
> >
> > Matrix m = drm.collect()
> > m.plus(1)
> >
> > The implicit def drm2InCore() syntactic sugar is probably turning out to
> be
> > dangerous in this case, in terms of hinting the wrong meaning.
> >
> > Thanks
> >
> >
> >
> >
> >
> > > On Jul 21, 2014, at 3:23 PM, Dmitriy Lyubimov <dl...@gmail.com>
> > wrote:
> > >
> > > perhaps just compare row count with max(key)? that's exactly what lazy
> > > nrow() currently does in this case.
> > >
> > >
> > > On Mon, Jul 21, 2014 at 3:21 PM, Dmitriy Lyubimov <dl...@gmail.com>
> > > wrote:
> > >
> > > >
> > > > ok. so it should be easy to fix at least everything but elementwise
> > > scalar
> > > > i guess.
> > > >
> > > > Since the notion of "missing rows" is only defined for int-keyed
> > > datasets,
> > > > then ew scalar technically should work for non-int keyed datasets
> > > already.
> > > >
> > > > as for int-keyed datasets, i am not sure what is the best strategy.
> > > > Obviously, one can define sort of normalization/validation of
> int-keyed
> > > > dataset routine, but it would be fairly expensive to run "just
> > because".
> > > > Perhaps there's a cheap test (as cheap as row count job) to run to
> test
> > > for
> > > > int keys consistency when matrix is first created.
> > > >
> > > >
> > > >
> > > > On Mon, Jul 21, 2014 at 3:12 PM, Anand Avati <av...@gluster.org>
> > wrote:
> > > >
> > > >>
> > > >>
> > > >>
> > > >> On Mon, Jul 21, 2014 at 3:08 PM, Dmitriy Lyubimov <
> dlieu.7@gmail.com>
> > > >> wrote:
> > > >>
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Mon, Jul 21, 2014 at 3:06 PM, Anand Avati <av...@gluster.org>
> > > wrote:
> > > >>>
> > > >>>> Dmitriy, comments inline -
> > > >>>>
> > > >>>> On Jul 21, 2014, at 1:12 PM, Dmitriy Lyubimov <dl...@gmail.com>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> And no, i suppose it is ok to have "missing" rows even in case of
> > > >>>>> int-keyed matrices.
> > > >>>>>
> > > >>>>> there's one thing that you probably should be aware in this
> context
> > > >>>>> though: many algorithms don't survive empty (row-less)
> partitions,
> > in
> > > >>>>> whatever way they may come to be. Other than that, I don't feel
> > > every row
> > > >>>>> must be present -- even if there's implied order of the rows.
> > > >>>>>
> > > >>>>
> > > >>>> I'm not sure if that is necessarily true. There are three
> operators
> > > >>>> which break pretty badly with with missing rows.
> > > >>>>
> > > >>>> AewScalar - operation like A + 1 is just not applied on the
> missing
> > > >>>> row, so the final matrix will have 0's in place of 1s.
> > > >>>>
> > > >>>
> > > >>> Indeed. i have no recourse at this point.
> > > >>>
> > > >>>
> > > >>>>
> > > >>>> AewB, CbindAB - function after cogroup() throws exception if a row
> > was
> > > >>>> present on only one matrix. So I guess it is OK to have missing
> rows
> > > as
> > > >>>> long as both A and B have the exact same missing row set. Somewhat
> > > >>>> quirky/nuanced requirement.
> > > >>>>
> > > >>>
> > > >>> Agree. i actually was not aware that's a cogroup() semantics in
> > spark.
> > > I
> > > >>> though it would have an outer join semantics (as in Pig, i
> believe).
> > > Alas,
> > > >>> no recourse at this point either.
> > > >>>
> > > >>
> > > >> The exception is actually during reduceLeft after cogroup().
> Cogroup()
> > > >> itself is probably an outer-join.
> > > >>
> > > >>
> > > >>
> > > >
> > >
> > >
> >
>
Re: Problem of dimensions
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
ah. got it. it is java api via implicit conversion. gosh. no good.
On Mon, Jul 21, 2014 at 3:44 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> not sure what is this plus() thing. Is it something that is not yet
> committed?
>
>
> On Mon, Jul 21, 2014 at 3:41 PM, Anand Avati <av...@gluster.org> wrote:
>
>> On Mon, Jul 21, 2014 at 3:35 PM, Pat Ferrel <pa...@occamsmachete.com>
>> wrote:
>>
>> > If you do drm.plus(1) this converts to a dense matrix, which is what the
>> > result must be anyway, and does add the scalar to all rows, even missing
>> > ones.
>> >
>> >
>> Pat, I mentioned this in my previous email already. drm.plus(1) completely
>> misses the point. It converts DRM into an in-core matrix and applies
>> plus()
>> method on Matrix. The result is a Matrix, not DRM.
>>
>> drm.plus(1) is EXACTLY the same as:
>>
>> Matrix m = drm.collect()
>> m.plus(1)
>>
>> The implicit def drm2InCore() syntactic sugar is probably turning out to
>> be
>> dangerous in this case, in terms of hinting the wrong meaning.
>>
>> Thanks
>>
>>
>>
>>
>>
>> > On Jul 21, 2014, at 3:23 PM, Dmitriy Lyubimov <dl...@gmail.com>
>> wrote:
>> >
>> > perhaps just compare row count with max(key)? that's exactly what lazy
>> > nrow() currently does in this case.
>> >
>> >
>> > On Mon, Jul 21, 2014 at 3:21 PM, Dmitriy Lyubimov <dl...@gmail.com>
>> > wrote:
>> >
>> > >
>> > > ok. so it should be easy to fix at least everything but elementwise
>> > scalar
>> > > i guess.
>> > >
>> > > Since the notion of "missing rows" is only defined for int-keyed
>> > datasets,
>> > > then ew scalar technically should work for non-int keyed datasets
>> > already.
>> > >
>> > > as for int-keyed datasets, i am not sure what is the best strategy.
>> > > Obviously, one can define sort of normalization/validation of
>> int-keyed
>> > > dataset routine, but it would be fairly expensive to run "just
>> because".
>> > > Perhaps there's a cheap test (as cheap as row count job) to run to
>> test
>> > for
>> > > int keys consistency when matrix is first created.
>> > >
>> > >
>> > >
>> > > On Mon, Jul 21, 2014 at 3:12 PM, Anand Avati <av...@gluster.org>
>> wrote:
>> > >
>> > >>
>> > >>
>> > >>
>> > >> On Mon, Jul 21, 2014 at 3:08 PM, Dmitriy Lyubimov <dlieu.7@gmail.com
>> >
>> > >> wrote:
>> > >>
>> > >>>
>> > >>>
>> > >>>
>> > >>> On Mon, Jul 21, 2014 at 3:06 PM, Anand Avati <av...@gluster.org>
>> > wrote:
>> > >>>
>> > >>>> Dmitriy, comments inline -
>> > >>>>
>> > >>>> On Jul 21, 2014, at 1:12 PM, Dmitriy Lyubimov <dl...@gmail.com>
>> > >>>> wrote:
>> > >>>>
>> > >>>>> And no, i suppose it is ok to have "missing" rows even in case of
>> > >>>>> int-keyed matrices.
>> > >>>>>
>> > >>>>> there's one thing that you probably should be aware in this
>> context
>> > >>>>> though: many algorithms don't survive empty (row-less)
>> partitions, in
>> > >>>>> whatever way they may come to be. Other than that, I don't feel
>> > every row
>> > >>>>> must be present -- even if there's implied order of the rows.
>> > >>>>>
>> > >>>>
>> > >>>> I'm not sure if that is necessarily true. There are three operators
>> > >>>> which break pretty badly with with missing rows.
>> > >>>>
>> > >>>> AewScalar - operation like A + 1 is just not applied on the missing
>> > >>>> row, so the final matrix will have 0's in place of 1s.
>> > >>>>
>> > >>>
>> > >>> Indeed. i have no recourse at this point.
>> > >>>
>> > >>>
>> > >>>>
>> > >>>> AewB, CbindAB - function after cogroup() throws exception if a row
>> was
>> > >>>> present on only one matrix. So I guess it is OK to have missing
>> rows
>> > as
>> > >>>> long as both A and B have the exact same missing row set. Somewhat
>> > >>>> quirky/nuanced requirement.
>> > >>>>
>> > >>>
>> > >>> Agree. i actually was not aware that's a cogroup() semantics in
>> spark.
>> > I
>> > >>> though it would have an outer join semantics (as in Pig, i believe).
>> > Alas,
>> > >>> no recourse at this point either.
>> > >>>
>> > >>
>> > >> The exception is actually during reduceLeft after cogroup().
>> Cogroup()
>> > >> itself is probably an outer-join.
>> > >>
>> > >>
>> > >>
>> > >
>> >
>> >
>>
>
>
Re: Problem of dimensions
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
not sure what is this plus() thing. Is it something that is not yet
committed?
On Mon, Jul 21, 2014 at 3:41 PM, Anand Avati <av...@gluster.org> wrote:
> On Mon, Jul 21, 2014 at 3:35 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:
>
> > If you do drm.plus(1) this converts to a dense matrix, which is what the
> > result must be anyway, and does add the scalar to all rows, even missing
> > ones.
> >
> >
> Pat, I mentioned this in my previous email already. drm.plus(1) completely
> misses the point. It converts DRM into an in-core matrix and applies plus()
> method on Matrix. The result is a Matrix, not DRM.
>
> drm.plus(1) is EXACTLY the same as:
>
> Matrix m = drm.collect()
> m.plus(1)
>
> The implicit def drm2InCore() syntactic sugar is probably turning out to be
> dangerous in this case, in terms of hinting the wrong meaning.
>
> Thanks
>
>
>
>
>
> > On Jul 21, 2014, at 3:23 PM, Dmitriy Lyubimov <dl...@gmail.com>
> wrote:
> >
> > perhaps just compare row count with max(key)? that's exactly what lazy
> > nrow() currently does in this case.
> >
> >
> > On Mon, Jul 21, 2014 at 3:21 PM, Dmitriy Lyubimov <dl...@gmail.com>
> > wrote:
> >
> > >
> > > ok. so it should be easy to fix at least everything but elementwise
> > scalar
> > > i guess.
> > >
> > > Since the notion of "missing rows" is only defined for int-keyed
> > datasets,
> > > then ew scalar technically should work for non-int keyed datasets
> > already.
> > >
> > > as for int-keyed datasets, i am not sure what is the best strategy.
> > > Obviously, one can define sort of normalization/validation of int-keyed
> > > dataset routine, but it would be fairly expensive to run "just
> because".
> > > Perhaps there's a cheap test (as cheap as row count job) to run to test
> > for
> > > int keys consistency when matrix is first created.
> > >
> > >
> > >
> > > On Mon, Jul 21, 2014 at 3:12 PM, Anand Avati <av...@gluster.org>
> wrote:
> > >
> > >>
> > >>
> > >>
> > >> On Mon, Jul 21, 2014 at 3:08 PM, Dmitriy Lyubimov <dl...@gmail.com>
> > >> wrote:
> > >>
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Jul 21, 2014 at 3:06 PM, Anand Avati <av...@gluster.org>
> > wrote:
> > >>>
> > >>>> Dmitriy, comments inline -
> > >>>>
> > >>>> On Jul 21, 2014, at 1:12 PM, Dmitriy Lyubimov <dl...@gmail.com>
> > >>>> wrote:
> > >>>>
> > >>>>> And no, i suppose it is ok to have "missing" rows even in case of
> > >>>>> int-keyed matrices.
> > >>>>>
> > >>>>> there's one thing that you probably should be aware in this context
> > >>>>> though: many algorithms don't survive empty (row-less) partitions,
> in
> > >>>>> whatever way they may come to be. Other than that, I don't feel
> > every row
> > >>>>> must be present -- even if there's implied order of the rows.
> > >>>>>
> > >>>>
> > >>>> I'm not sure if that is necessarily true. There are three operators
> > >>>> which break pretty badly with with missing rows.
> > >>>>
> > >>>> AewScalar - operation like A + 1 is just not applied on the missing
> > >>>> row, so the final matrix will have 0's in place of 1s.
> > >>>>
> > >>>
> > >>> Indeed. i have no recourse at this point.
> > >>>
> > >>>
> > >>>>
> > >>>> AewB, CbindAB - function after cogroup() throws exception if a row
> was
> > >>>> present on only one matrix. So I guess it is OK to have missing rows
> > as
> > >>>> long as both A and B have the exact same missing row set. Somewhat
> > >>>> quirky/nuanced requirement.
> > >>>>
> > >>>
> > >>> Agree. i actually was not aware that's a cogroup() semantics in
> spark.
> > I
> > >>> though it would have an outer join semantics (as in Pig, i believe).
> > Alas,
> > >>> no recourse at this point either.
> > >>>
> > >>
> > >> The exception is actually during reduceLeft after cogroup(). Cogroup()
> > >> itself is probably an outer-join.
> > >>
> > >>
> > >>
> > >
> >
> >
>
Re: Problem of dimensions
Posted by Anand Avati <av...@gluster.org>.
On Mon, Jul 21, 2014 at 3:35 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:
> If you do drm.plus(1) this converts to a dense matrix, which is what the
> result must be anyway, and does add the scalar to all rows, even missing
> ones.
>
>
Pat, I mentioned this in my previous email already. drm.plus(1) completely
misses the point. It converts DRM into an in-core matrix and applies plus()
method on Matrix. The result is a Matrix, not DRM.
drm.plus(1) is EXACTLY the same as:
Matrix m = drm.collect()
m.plus(1)
The implicit def drm2InCore() syntactic sugar is probably turning out to be
dangerous in this case, in terms of hinting the wrong meaning.
Thanks
> On Jul 21, 2014, at 3:23 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
>
> perhaps just compare row count with max(key)? that's exactly what lazy
> nrow() currently does in this case.
>
>
> On Mon, Jul 21, 2014 at 3:21 PM, Dmitriy Lyubimov <dl...@gmail.com>
> wrote:
>
> >
> > ok. so it should be easy to fix at least everything but elementwise
> scalar
> > i guess.
> >
> > Since the notion of "missing rows" is only defined for int-keyed
> datasets,
> > then ew scalar technically should work for non-int keyed datasets
> already.
> >
> > as for int-keyed datasets, i am not sure what is the best strategy.
> > Obviously, one can define sort of normalization/validation of int-keyed
> > dataset routine, but it would be fairly expensive to run "just because".
> > Perhaps there's a cheap test (as cheap as row count job) to run to test
> for
> > int keys consistency when matrix is first created.
> >
> >
> >
> > On Mon, Jul 21, 2014 at 3:12 PM, Anand Avati <av...@gluster.org> wrote:
> >
> >>
> >>
> >>
> >> On Mon, Jul 21, 2014 at 3:08 PM, Dmitriy Lyubimov <dl...@gmail.com>
> >> wrote:
> >>
> >>>
> >>>
> >>>
> >>> On Mon, Jul 21, 2014 at 3:06 PM, Anand Avati <av...@gluster.org>
> wrote:
> >>>
> >>>> Dmitriy, comments inline -
> >>>>
> >>>> On Jul 21, 2014, at 1:12 PM, Dmitriy Lyubimov <dl...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> And no, i suppose it is ok to have "missing" rows even in case of
> >>>>> int-keyed matrices.
> >>>>>
> >>>>> there's one thing that you probably should be aware in this context
> >>>>> though: many algorithms don't survive empty (row-less) partitions, in
> >>>>> whatever way they may come to be. Other than that, I don't feel
> every row
> >>>>> must be present -- even if there's implied order of the rows.
> >>>>>
> >>>>
> >>>> I'm not sure if that is necessarily true. There are three operators
> >>>> which break pretty badly with with missing rows.
> >>>>
> >>>> AewScalar - operation like A + 1 is just not applied on the missing
> >>>> row, so the final matrix will have 0's in place of 1s.
> >>>>
> >>>
> >>> Indeed. i have no recourse at this point.
> >>>
> >>>
> >>>>
> >>>> AewB, CbindAB - function after cogroup() throws exception if a row was
> >>>> present on only one matrix. So I guess it is OK to have missing rows
> as
> >>>> long as both A and B have the exact same missing row set. Somewhat
> >>>> quirky/nuanced requirement.
> >>>>
> >>>
> >>> Agree. i actually was not aware that's a cogroup() semantics in spark.
> I
> >>> though it would have an outer join semantics (as in Pig, i believe).
> Alas,
> >>> no recourse at this point either.
> >>>
> >>
> >> The exception is actually during reduceLeft after cogroup(). Cogroup()
> >> itself is probably an outer-join.
> >>
> >>
> >>
> >
>
>
Re: Problem of dimensions
Posted by Pat Ferrel <pa...@occamsmachete.com>.
If you do drm.plus(1) this converts to a dense matrix, which is what the result must be anyway, and does add the scalar to all rows, even missing ones.
On Jul 21, 2014, at 3:23 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
perhaps just compare row count with max(key)? that's exactly what lazy
nrow() currently does in this case.
On Mon, Jul 21, 2014 at 3:21 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
>
> ok. so it should be easy to fix at least everything but elementwise scalar
> i guess.
>
> Since the notion of "missing rows" is only defined for int-keyed datasets,
> then ew scalar technically should work for non-int keyed datasets already.
>
> as for int-keyed datasets, i am not sure what is the best strategy.
> Obviously, one can define sort of normalization/validation of int-keyed
> dataset routine, but it would be fairly expensive to run "just because".
> Perhaps there's a cheap test (as cheap as row count job) to run to test for
> int keys consistency when matrix is first created.
>
>
>
> On Mon, Jul 21, 2014 at 3:12 PM, Anand Avati <av...@gluster.org> wrote:
>
>>
>>
>>
>> On Mon, Jul 21, 2014 at 3:08 PM, Dmitriy Lyubimov <dl...@gmail.com>
>> wrote:
>>
>>>
>>>
>>>
>>> On Mon, Jul 21, 2014 at 3:06 PM, Anand Avati <av...@gluster.org> wrote:
>>>
>>>> Dmitriy, comments inline -
>>>>
>>>> On Jul 21, 2014, at 1:12 PM, Dmitriy Lyubimov <dl...@gmail.com>
>>>> wrote:
>>>>
>>>>> And no, i suppose it is ok to have "missing" rows even in case of
>>>>> int-keyed matrices.
>>>>>
>>>>> there's one thing that you probably should be aware in this context
>>>>> though: many algorithms don't survive empty (row-less) partitions, in
>>>>> whatever way they may come to be. Other than that, I don't feel every row
>>>>> must be present -- even if there's implied order of the rows.
>>>>>
>>>>
>>>> I'm not sure if that is necessarily true. There are three operators
>>>> which break pretty badly with with missing rows.
>>>>
>>>> AewScalar - operation like A + 1 is just not applied on the missing
>>>> row, so the final matrix will have 0's in place of 1s.
>>>>
>>>
>>> Indeed. i have no recourse at this point.
>>>
>>>
>>>>
>>>> AewB, CbindAB - function after cogroup() throws exception if a row was
>>>> present on only one matrix. So I guess it is OK to have missing rows as
>>>> long as both A and B have the exact same missing row set. Somewhat
>>>> quirky/nuanced requirement.
>>>>
>>>
>>> Agree. i actually was not aware that's a cogroup() semantics in spark. I
>>> though it would have an outer join semantics (as in Pig, i believe). Alas,
>>> no recourse at this point either.
>>>
>>
>> The exception is actually during reduceLeft after cogroup(). Cogroup()
>> itself is probably an outer-join.
>>
>>
>>
>
Re: Problem of dimensions
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
lazy nrow actually computes max + 1 in case of DrmLike[Int] and count in
every other case. so it would seem to me it should compute both max and
count and assert max == count-1 iff count >0 whenever int-based matrix
comes to be. (which is drmWrap, loaders and perhaps there are some corner
cases in something like big gram matrix computation). how/if we want to
repair this, i am not sure. We should be able to parallelize a 0 until nrow
rdd and cogroup rows with it, i guess, if fix is really needed.
On Mon, Jul 21, 2014 at 3:37 PM, Anand Avati <av...@gluster.org> wrote:
> easy to fix except elementwise scalar and the problems associated with
> missing partitions (if the missing rows amounts to that extent).
>
> As far as consistency check, maybe a combination of max, count, sum
> computed in a single reduce sweep should be reasonably sufficient?
>
> max == nrow -1
> count == nrow
> sum == nrow * (nrow -1) / 2 /* since keys are 0 based */
>
> It might be expensive to run after all int-DRM creation.
>
> On Mon, Jul 21, 2014 at 3:23 PM, Dmitriy Lyubimov <dl...@gmail.com>
> wrote:
>
>> perhaps just compare row count with max(key)? that's exactly what lazy
>> nrow() currently does in this case.
>>
>> On Mon, Jul 21, 2014 at 3:21 PM, Dmitriy Lyubimov <dl...@gmail.com>
>> wrote:
>>
>>>
>>> ok. so it should be easy to fix at least everything but elementwise
>>> scalar i guess.
>>>
>>> Since the notion of "missing rows" is only defined for int-keyed
>>> datasets, then ew scalar technically should work for non-int keyed datasets
>>> already.
>>>
>>> as for int-keyed datasets, i am not sure what is the best strategy.
>>> Obviously, one can define sort of normalization/validation of int-keyed
>>> dataset routine, but it would be fairly expensive to run "just because".
>>> Perhaps there's a cheap test (as cheap as row count job) to run to test for
>>> int keys consistency when matrix is first created.
>>>
>>>
>>>
>>> On Mon, Jul 21, 2014 at 3:12 PM, Anand Avati <av...@gluster.org> wrote:
>>>
>>>>
>>>>
>>>>
>>>> On Mon, Jul 21, 2014 at 3:08 PM, Dmitriy Lyubimov <dl...@gmail.com>
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Jul 21, 2014 at 3:06 PM, Anand Avati <av...@gluster.org>
>>>>> wrote:
>>>>>
>>>>>> Dmitriy, comments inline -
>>>>>>
>>>>>> On Jul 21, 2014, at 1:12 PM, Dmitriy Lyubimov <dl...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> And no, i suppose it is ok to have "missing" rows even in case of
>>>>>>> int-keyed matrices.
>>>>>>>
>>>>>>> there's one thing that you probably should be aware in this context
>>>>>>> though: many algorithms don't survive empty (row-less) partitions, in
>>>>>>> whatever way they may come to be. Other than that, I don't feel every row
>>>>>>> must be present -- even if there's implied order of the rows.
>>>>>>>
>>>>>>
>>>>>> I'm not sure if that is necessarily true. There are three operators
>>>>>> which break pretty badly with with missing rows.
>>>>>>
>>>>>> AewScalar - operation like A + 1 is just not applied on the missing
>>>>>> row, so the final matrix will have 0's in place of 1s.
>>>>>>
>>>>>
>>>>> Indeed. i have no recourse at this point.
>>>>>
>>>>>
>>>>>>
>>>>>> AewB, CbindAB - function after cogroup() throws exception if a row
>>>>>> was present on only one matrix. So I guess it is OK to have missing rows as
>>>>>> long as both A and B have the exact same missing row set. Somewhat
>>>>>> quirky/nuanced requirement.
>>>>>>
>>>>>
>>>>> Agree. i actually was not aware that's a cogroup() semantics in spark.
>>>>> I though it would have an outer join semantics (as in Pig, i believe).
>>>>> Alas, no recourse at this point either.
>>>>>
>>>>
>>>> The exception is actually during reduceLeft after cogroup(). Cogroup()
>>>> itself is probably an outer-join.
>>>>
>>>>
>>>>
>>>
>>
>
Re: Problem of dimensions
Posted by Pat Ferrel <pa...@gmail.com>.
Since the rdd is not being changed why are there missing partitions?
On Jul 21, 2014, at 3:37 PM, Anand Avati <av...@gluster.org> wrote:
easy to fix except elementwise scalar and the problems associated with
missing partitions (if the missing rows amounts to that extent).
As far as consistency check, maybe a combination of max, count, sum
computed in a single reduce sweep should be reasonably sufficient?
max == nrow -1
count == nrow
sum == nrow * (nrow -1) / 2 /* since keys are 0 based */
It might be expensive to run after all int-DRM creation.
On Mon, Jul 21, 2014 at 3:23 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> perhaps just compare row count with max(key)? that's exactly what lazy
> nrow() currently does in this case.
>
> On Mon, Jul 21, 2014 at 3:21 PM, Dmitriy Lyubimov <dl...@gmail.com>
> wrote:
>
>>
>> ok. so it should be easy to fix at least everything but elementwise
>> scalar i guess.
>>
>> Since the notion of "missing rows" is only defined for int-keyed
>> datasets, then ew scalar technically should work for non-int keyed datasets
>> already.
>>
>> as for int-keyed datasets, i am not sure what is the best strategy.
>> Obviously, one can define sort of normalization/validation of int-keyed
>> dataset routine, but it would be fairly expensive to run "just because".
>> Perhaps there's a cheap test (as cheap as row count job) to run to test for
>> int keys consistency when matrix is first created.
>>
>>
>>
>> On Mon, Jul 21, 2014 at 3:12 PM, Anand Avati <av...@gluster.org> wrote:
>>
>>>
>>>
>>>
>>> On Mon, Jul 21, 2014 at 3:08 PM, Dmitriy Lyubimov <dl...@gmail.com>
>>> wrote:
>>>
>>>>
>>>>
>>>>
>>>> On Mon, Jul 21, 2014 at 3:06 PM, Anand Avati <av...@gluster.org> wrote:
>>>>
>>>>> Dmitriy, comments inline -
>>>>>
>>>>> On Jul 21, 2014, at 1:12 PM, Dmitriy Lyubimov <dl...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> And no, i suppose it is ok to have "missing" rows even in case of
>>>>>> int-keyed matrices.
>>>>>>
>>>>>> there's one thing that you probably should be aware in this context
>>>>>> though: many algorithms don't survive empty (row-less) partitions, in
>>>>>> whatever way they may come to be. Other than that, I don't feel every row
>>>>>> must be present -- even if there's implied order of the rows.
>>>>>>
>>>>>
>>>>> I'm not sure if that is necessarily true. There are three operators
>>>>> which break pretty badly with with missing rows.
>>>>>
>>>>> AewScalar - operation like A + 1 is just not applied on the missing
>>>>> row, so the final matrix will have 0's in place of 1s.
>>>>>
>>>>
>>>> Indeed. i have no recourse at this point.
>>>>
>>>>
>>>>>
>>>>> AewB, CbindAB - function after cogroup() throws exception if a row was
>>>>> present on only one matrix. So I guess it is OK to have missing rows as
>>>>> long as both A and B have the exact same missing row set. Somewhat
>>>>> quirky/nuanced requirement.
>>>>>
>>>>
>>>> Agree. i actually was not aware that's a cogroup() semantics in spark.
>>>> I though it would have an outer join semantics (as in Pig, i believe).
>>>> Alas, no recourse at this point either.
>>>>
>>>
>>> The exception is actually during reduceLeft after cogroup(). Cogroup()
>>> itself is probably an outer-join.
>>>
>>>
>>>
>>
>
Re: Problem of dimensions
Posted by Anand Avati <av...@gluster.org>.
easy to fix except elementwise scalar and the problems associated with
missing partitions (if the missing rows amounts to that extent).
As far as consistency check, maybe a combination of max, count, sum
computed in a single reduce sweep should be reasonably sufficient?
max == nrow -1
count == nrow
sum == nrow * (nrow -1) / 2 /* since keys are 0 based */
It might be expensive to run after all int-DRM creation.
On Mon, Jul 21, 2014 at 3:23 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> perhaps just compare row count with max(key)? that's exactly what lazy
> nrow() currently does in this case.
>
> On Mon, Jul 21, 2014 at 3:21 PM, Dmitriy Lyubimov <dl...@gmail.com>
> wrote:
>
>>
>> ok. so it should be easy to fix at least everything but elementwise
>> scalar i guess.
>>
>> Since the notion of "missing rows" is only defined for int-keyed
>> datasets, then ew scalar technically should work for non-int keyed datasets
>> already.
>>
>> as for int-keyed datasets, i am not sure what is the best strategy.
>> Obviously, one can define sort of normalization/validation of int-keyed
>> dataset routine, but it would be fairly expensive to run "just because".
>> Perhaps there's a cheap test (as cheap as row count job) to run to test for
>> int keys consistency when matrix is first created.
>>
>>
>>
>> On Mon, Jul 21, 2014 at 3:12 PM, Anand Avati <av...@gluster.org> wrote:
>>
>>>
>>>
>>>
>>> On Mon, Jul 21, 2014 at 3:08 PM, Dmitriy Lyubimov <dl...@gmail.com>
>>> wrote:
>>>
>>>>
>>>>
>>>>
>>>> On Mon, Jul 21, 2014 at 3:06 PM, Anand Avati <av...@gluster.org> wrote:
>>>>
>>>>> Dmitriy, comments inline -
>>>>>
>>>>> On Jul 21, 2014, at 1:12 PM, Dmitriy Lyubimov <dl...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> And no, i suppose it is ok to have "missing" rows even in case of
>>>>>> int-keyed matrices.
>>>>>>
>>>>>> there's one thing that you probably should be aware in this context
>>>>>> though: many algorithms don't survive empty (row-less) partitions, in
>>>>>> whatever way they may come to be. Other than that, I don't feel every row
>>>>>> must be present -- even if there's implied order of the rows.
>>>>>>
>>>>>
>>>>> I'm not sure if that is necessarily true. There are three operators
>>>>> which break pretty badly with with missing rows.
>>>>>
>>>>> AewScalar - operation like A + 1 is just not applied on the missing
>>>>> row, so the final matrix will have 0's in place of 1s.
>>>>>
>>>>
>>>> Indeed. i have no recourse at this point.
>>>>
>>>>
>>>>>
>>>>> AewB, CbindAB - function after cogroup() throws exception if a row was
>>>>> present on only one matrix. So I guess it is OK to have missing rows as
>>>>> long as both A and B have the exact same missing row set. Somewhat
>>>>> quirky/nuanced requirement.
>>>>>
>>>>
>>>> Agree. i actually was not aware that's a cogroup() semantics in spark.
>>>> I though it would have an outer join semantics (as in Pig, i believe).
>>>> Alas, no recourse at this point either.
>>>>
>>>
>>> The exception is actually during reduceLeft after cogroup(). Cogroup()
>>> itself is probably an outer-join.
>>>
>>>
>>>
>>
>
Re: Problem of dimensions
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
perhaps just compare row count with max(key)? that's exactly what lazy
nrow() currently does in this case.
On Mon, Jul 21, 2014 at 3:21 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
>
> ok. so it should be easy to fix at least everything but elementwise scalar
> i guess.
>
> Since the notion of "missing rows" is only defined for int-keyed datasets,
> then ew scalar technically should work for non-int keyed datasets already.
>
> as for int-keyed datasets, i am not sure what is the best strategy.
> Obviously, one can define sort of normalization/validation of int-keyed
> dataset routine, but it would be fairly expensive to run "just because".
> Perhaps there's a cheap test (as cheap as row count job) to run to test for
> int keys consistency when matrix is first created.
>
>
>
> On Mon, Jul 21, 2014 at 3:12 PM, Anand Avati <av...@gluster.org> wrote:
>
>>
>>
>>
>> On Mon, Jul 21, 2014 at 3:08 PM, Dmitriy Lyubimov <dl...@gmail.com>
>> wrote:
>>
>>>
>>>
>>>
>>> On Mon, Jul 21, 2014 at 3:06 PM, Anand Avati <av...@gluster.org> wrote:
>>>
>>>> Dmitriy, comments inline -
>>>>
>>>> On Jul 21, 2014, at 1:12 PM, Dmitriy Lyubimov <dl...@gmail.com>
>>>> wrote:
>>>>
>>>>> And no, i suppose it is ok to have "missing" rows even in case of
>>>>> int-keyed matrices.
>>>>>
>>>>> there's one thing that you probably should be aware in this context
>>>>> though: many algorithms don't survive empty (row-less) partitions, in
>>>>> whatever way they may come to be. Other than that, I don't feel every row
>>>>> must be present -- even if there's implied order of the rows.
>>>>>
>>>>
>>>> I'm not sure if that is necessarily true. There are three operators
>>>> which break pretty badly with with missing rows.
>>>>
>>>> AewScalar - operation like A + 1 is just not applied on the missing
>>>> row, so the final matrix will have 0's in place of 1s.
>>>>
>>>
>>> Indeed. i have no recourse at this point.
>>>
>>>
>>>>
>>>> AewB, CbindAB - function after cogroup() throws exception if a row was
>>>> present on only one matrix. So I guess it is OK to have missing rows as
>>>> long as both A and B have the exact same missing row set. Somewhat
>>>> quirky/nuanced requirement.
>>>>
>>>
>>> Agree. i actually was not aware that's a cogroup() semantics in spark. I
>>> though it would have an outer join semantics (as in Pig, i believe). Alas,
>>> no recourse at this point either.
>>>
>>
>> The exception is actually during reduceLeft after cogroup(). Cogroup()
>> itself is probably an outer-join.
>>
>>
>>
>
Re: Problem of dimensions
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
ok. so it should be easy to fix at least everything but elementwise scalar
i guess.
Since the notion of "missing rows" is only defined for int-keyed datasets,
then ew scalar technically should work for non-int keyed datasets already.
as for int-keyed datasets, i am not sure what is the best strategy.
Obviously, one can define sort of normalization/validation of int-keyed
dataset routine, but it would be fairly expensive to run "just because".
Perhaps there's a cheap test (as cheap as row count job) to run to test for
int keys consistency when matrix is first created.
On Mon, Jul 21, 2014 at 3:12 PM, Anand Avati <av...@gluster.org> wrote:
>
>
>
> On Mon, Jul 21, 2014 at 3:08 PM, Dmitriy Lyubimov <dl...@gmail.com>
> wrote:
>
>>
>>
>>
>> On Mon, Jul 21, 2014 at 3:06 PM, Anand Avati <av...@gluster.org> wrote:
>>
>>> Dmitriy, comments inline -
>>>
>>> On Jul 21, 2014, at 1:12 PM, Dmitriy Lyubimov <dl...@gmail.com>
>>> wrote:
>>>
>>>> And no, i suppose it is ok to have "missing" rows even in case of
>>>> int-keyed matrices.
>>>>
>>>> there's one thing that you probably should be aware in this context
>>>> though: many algorithms don't survive empty (row-less) partitions, in
>>>> whatever way they may come to be. Other than that, I don't feel every row
>>>> must be present -- even if there's implied order of the rows.
>>>>
>>>
>>> I'm not sure if that is necessarily true. There are three operators
>>> which break pretty badly with with missing rows.
>>>
>>> AewScalar - operation like A + 1 is just not applied on the missing row,
>>> so the final matrix will have 0's in place of 1s.
>>>
>>
>> Indeed. i have no recourse at this point.
>>
>>
>>>
>>> AewB, CbindAB - function after cogroup() throws exception if a row was
>>> present on only one matrix. So I guess it is OK to have missing rows as
>>> long as both A and B have the exact same missing row set. Somewhat
>>> quirky/nuanced requirement.
>>>
>>
>> Agree. i actually was not aware that's a cogroup() semantics in spark. I
>> though it would have an outer join semantics (as in Pig, i believe). Alas,
>> no recourse at this point either.
>>
>
> The exception is actually during reduceLeft after cogroup(). Cogroup()
> itself is probably an outer-join.
>
>
>
Re: Problem of dimensions
Posted by Anand Avati <av...@gluster.org>.
On Mon, Jul 21, 2014 at 3:08 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
>
>
>
> On Mon, Jul 21, 2014 at 3:06 PM, Anand Avati <av...@gluster.org> wrote:
>
>> Dmitriy, comments inline -
>>
>> On Jul 21, 2014, at 1:12 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
>>
>>> And no, i suppose it is ok to have "missing" rows even in case of
>>> int-keyed matrices.
>>>
>>> there's one thing that you probably should be aware in this context
>>> though: many algorithms don't survive empty (row-less) partitions, in
>>> whatever way they may come to be. Other than that, I don't feel every row
>>> must be present -- even if there's implied order of the rows.
>>>
>>
>> I'm not sure if that is necessarily true. There are three operators which
>> break pretty badly with with missing rows.
>>
>> AewScalar - operation like A + 1 is just not applied on the missing row,
>> so the final matrix will have 0's in place of 1s.
>>
>
> Indeed. i have no recourse at this point.
>
>
>>
>> AewB, CbindAB - function after cogroup() throws exception if a row was
>> present on only one matrix. So I guess it is OK to have missing rows as
>> long as both A and B have the exact same missing row set. Somewhat
>> quirky/nuanced requirement.
>>
>
> Agree. i actually was not aware that's a cogroup() semantics in spark. I
> though it would have an outer join semantics (as in Pig, i believe). Alas,
> no recourse at this point either.
>
The exception is actually during reduceLeft after cogroup(). Cogroup()
itself is probably an outer-join.
Re: Problem of dimensions
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
On Mon, Jul 21, 2014 at 3:06 PM, Anand Avati <av...@gluster.org> wrote:
> Dmitriy, comments inline -
>
> On Jul 21, 2014, at 1:12 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
>
>> And no, i suppose it is ok to have "missing" rows even in case of
>> int-keyed matrices.
>>
>> there's one thing that you probably should be aware in this context
>> though: many algorithms don't survive empty (row-less) partitions, in
>> whatever way they may come to be. Other than that, I don't feel every row
>> must be present -- even if there's implied order of the rows.
>>
>
> I'm not sure if that is necessarily true. There are three operators which
> break pretty badly with with missing rows.
>
> AewScalar - operation like A + 1 is just not applied on the missing row,
> so the final matrix will have 0's in place of 1s.
>
Indeed. i have no recourse at this point.
>
> AewB, CbindAB - function after cogroup() throws exception if a row was
> present on only one matrix. So I guess it is OK to have missing rows as
> long as both A and B have the exact same missing row set. Somewhat
> quirky/nuanced requirement.
>
Agree. i actually was not aware that's a cogroup() semantics in spark. I
though it would have an outer join semantics (as in Pig, i believe). Alas,
no recourse at this point either.
>
> These issues are other than the empty partition problem. So, if we were to
> fix the above issues (I don't see a simple way to), I guess we could say
> "it is ok to have missing rows even in case of int-keyed matrices." Given
> the state of things, I think it is safer to change the stance. Besides,
> what is the benefit/advantage of "supporting" missing rows, it is a
> physical implementation detail after all.
>
> Thanks
>
Re: Problem of dimensions
Posted by Anand Avati <av...@gluster.org>.
Dmitriy, comments inline -
On Jul 21, 2014, at 1:12 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> And no, i suppose it is ok to have "missing" rows even in case of
> int-keyed matrices.
>
> there's one thing that you probably should be aware in this context
> though: many algorithms don't survive empty (row-less) partitions, in
> whatever way they may come to be. Other than that, I don't feel every row
> must be present -- even if there's implied order of the rows.
>
I'm not sure if that is necessarily true. There are three operators which
break pretty badly with with missing rows.
AewScalar - operation like A + 1 is just not applied on the missing row, so
the final matrix will have 0's in place of 1s.
AewB, CbindAB - function after cogroup() throws exception if a row was
present on only one matrix. So I guess it is OK to have missing rows as
long as both A and B have the exact same missing row set. Somewhat
quirky/nuanced requirement.
These issues are other than the empty partition problem. So, if we were to
fix the above issues (I don't see a simple way to), I guess we could say
"it is ok to have missing rows even in case of int-keyed matrices." Given
the state of things, I think it is safer to change the stance. Besides,
what is the benefit/advantage of "supporting" missing rows, it is a
physical implementation detail after all.
Thanks
Re: Problem of dimensions
Posted by Pat Ferrel <pa...@gmail.com>.
Thank you! This is what I understood and I’m doing a little dance for joy (in my mind). This makes sparseness all encompassing, at least for sequential Int keys.
However Anand has found several math ops that don’t work.
I’ll write up a few tests for transpose and multiply at least since these are used in cooccurrence. And I’ll be happy to implement something that changes nrow in an immutable R-like way. Anand and Ted suggested rbind of drmParallelizeEmpty with added row cardinality. This would really only change nrow of the resulting CheckPointedDrm, it would not alter the rdd.
On Jul 21, 2014, at 1:12 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
"missing" rows are only valid in context of int-keyed matrices and physical transposition operations. These are the only that may depend on it, since obviously one can't define "missing-ness" for something that is String-keyed.
So the only thing that may fail because of "missing-ness" effect is probably physical transposition operator (we don't have test for such case, so maybe there's a bug in that case). Everything else should work.
And no, i suppose it is ok to have "missing" rows even in case of int-keyed matrices.
there's one thing that you probably should be aware in this context though: many algorithms don't survive empty (row-less) partitions, in whatever way they may come to be. Other than that, I don't feel every row must be present -- even if there's implied order of the rows.
On Mon, Jul 21, 2014 at 12:22 PM, Pat Ferrel <pa...@gmail.com> wrote:
I appreciate that you can’t read all the back and forth Dmitriy hence the private email. Please disregard all other code or talk in the thread for the moment.
Does a DRM need to have a row for every sequential row key from 0 to nrow-1 ? Can there be missing row keys in the sequence and will they be treated as {}, an all zero row? In terms of the rdd in the CheckpointedDrm these “missing” rows will not have a corresponding n => {}, they will just not exist in the rdd. This will happen when a row is “missing” from the DRM but the true cardinality is known and passed in to the CheckpointedDrm constructor.
Will R-like operations on these matrices work correctly. Will A.t %*% A and A + 1 work correctly?
The answer is no, but _should_ they work correctly?
Re: A + 1
Posted by Anand Avati <av...@gluster.org>.
On Sun, Jul 20, 2014 at 8:16 AM, Pat Ferrel <pa...@gmail.com> wrote:
> Oh, it’s working. The conversion to dense is saving the day in #1 I think.
> Where it is using nrow to create enough rows rather than looking in the rdd
> for row keys—just a guess.
>
The conversion to in-core Matrix (dense or sparse does not matter) is
saving the day.
>
> I think it was you that said dense or sparse the math should produce the
> same result. One or the other is a bug, right?
>
>
Yes, sparse or dense the math should produce the same result. And
similarly, with in-core or DRM. #1 is using the syntactic sugar to convert
DRM to in-core and is testing on in-core. The variable drmABigger1 is not
even a drm. Note that there isn't even a plus() method defined on DRM.
> I am going to add some matrix multiply tests that work on rdd backed
> objects. The current tests could use some additions. I suspect that
> multiply and transpose will work correctly with non-existant rows/columns
> but still need the designers to come out on one side or the other.
>
A failing test case trumps any number of working test cases. If you want
nrow fudging to work, you will have to first make sure it does not have any
surprise side effects for future users of that operator/method (especially
correctness, not even performance), no matter what the intention of the
designers was.
> If row keys need to be sequential and unbroken, this is s big deal and new
> to me.
Int row keys are sequential and unbroken. There is code which depends on
this:
- AewScalar operator does not even get hold of rows if key/vec is non
existent, resulting in bugs like I have shown
- AewB operator (element-wise +,-,*,/ with B) will fail for the same
reason, but with "java.lang.UnsupportedOperationException: empty.reduceLeft"
and some which hint at the intention:
- rowRange() operator rewrites/modifies int keys to be sequential/unbroken
in the resulting drm.
- Even in-core sparse matrices have all rows (even if the given row is
empty).
There are probably more issues if you dig deeper. Even if the intention of
the designers was other-wise (which I doubt), reality is you have plenty of
code and operators to fix (in non trivial ways) to make nrow fudging a safe
operation.
Thanks
> On Jul 19, 2014, at 10:09 PM, Anand Avati <av...@gluster.org> wrote:
>
> On Sat, Jul 19, 2014 at 9:06 PM, Pat Ferrel <pa...@gmail.com> wrote:
>
> > Using methods instead of symbolic ops returns different types so methods
> > work, ops don’t. If math is math, they should do the smae thing so I’d
> like
> > to know what it is supposed to do, can you please allow me to ask
> specific
> > people the question?
> >
>
>
> I don't see how I'm coming in the way of anybody else from replying. By
> posting to a public mailing list, you by definition allow anybody to review
> and comment.
>
>
> test("plus one"){
> > val a = dense(
> > (1, 1),
> > (0, 0))
> >
> > val drmA1 = drmParallelize(m = a, numPartitions = 2)
> >
> > // modified to return a new CheckpointedDrm so maintains immutability
> > but still only increases the row cardinality
> > // by returning new CheckpointedDrmSpark[K](rdd, nrow + n, ncol,
> > _cacheStorageLevel ) Hack for now.
> > val drmABigger1 = drmA1.addToRowCardinality(1)
> >
> > val drmABiggerPlusOne1 = drmABigger1.plus(1.0) // drmABigger has no
> > row 2 in the rdd but an empty row 1
> > // drmABiggerPlusOne1 is a dense matrix
> > println(drmABiggerPlusOne1)
> >
> > val drmA2 = drmParallelize(m = a, numPartitions = 2)
> > val drmABigger2 = drmA2.addToRowCardinality(1)
> > val drmABiggerPlusOne2 = drmABigger2 + 1.0
> > drmABiggerPlusOne2.writeDRM("tmp/plus-one/drma-bigger-plus-one-ops/")
> >
> >
> > val bp = 0
> > }
> >
> > method #1 works, #2 doesn’t. Even when I create a new CheckpintedDrmSpark
> > with larger _nrow than is in the data—even without the
> addToRowCardinality.
> >
>
> In Method #1, plus() is not even operating on the DRM. The plus() is
> operating on in core Matrix which is implicitly collect()ed because of the
> drm2InCore implicit type converter. So drmABiggerPlusOne1 is neither a
> DrmLike nor CheckpointedDrm, but is actually just an incore Matrix. You
> will have to drmParallelize() it again in order to do any distributed
> operations.
>
> I agree that in some cases it doesn’t can you please allow me to ask if it
> > _should_?
> >
>
> I don't think it is working in any case. The implicit converter is making
> it feel like it is working.
>
> Thanks
>
>
>
>
> > On Jul 19, 2014, at 8:56 PM, Anand Avati <av...@gluster.org> wrote:
> >
> >
> >
> >
> > On Sat, Jul 19, 2014 at 6:50 PM, Pat Ferrel <pa...@gmail.com>
> wrote:
> >
> >>
> >> On another thread I’ll send you code that shows A + 1 works with blank
> >> rows in A.
> >>
> >
> > I don't see how that worked for you. See this:
> >
> > test("DRM addToRowCardinality - will fail") {
> > val inCoreA = sparse(
> > 0 -> 1 :: 1 -> 2 :: Nil,
> > 0 -> 3 :: 1 -> 4 :: Nil,
> > 0 -> 2 :: 1 -> 0.0 :: Nil
> > )
> >
> > val inCoreBControl = sparse(
> > 0 -> 2 :: 1 -> 3 :: Nil,
> > 0 -> 4 :: 1 -> 5 :: Nil,
> > 0 -> 3 :: 1 -> 1 :: Nil,
> > 0 -> 1 :: 1 -> 1 :: Nil,
> > 0 -> 1 :: 1 -> 1 :: Nil
> > )
> >
> > val drmA = drmParallelize(inCoreA)
> > drmA.addToRowCardinality(2)
> > val drmB = (drmA + 1.0).checkpoint()
> >
> > (drmB.collect - inCoreBControl).norm should be < 1e-3
> >
> > }
> >
> > test("DRM addToRowCardinality - wont fail") {
> > val inCoreA = sparse(
> > 0 -> 1 :: 1 -> 2 :: Nil,
> > 0 -> 3 :: 1 -> 4 :: Nil,
> > 0 -> 2 :: 1 -> 0.0 :: Nil
> > )
> >
> > val inCoreBWrong = sparse(
> > 0 -> 2 :: 1 -> 3 :: Nil,
> > 0 -> 4 :: 1 -> 5 :: Nil,
> > 0 -> 3 :: 1 -> 1 :: Nil,
> > 0 -> 0 :: 1 -> 0 :: Nil,
> > 0 -> 0 :: 1 -> 0 :: Nil
> > )
> >
> > val drmA = drmParallelize(inCoreA)
> > drmA.addToRowCardinality(2)
> > val drmB = (drmA + 1.0).checkpoint()
> >
> > (drmB.collect - inCoreBWrong).norm should be < 1e-3
> > }
> >
> >
> > And sure enough, inCoreBControl fails, and inCoreBWrong succeeds:
> >
> > - DRM addToRowCardinality - will fail *** FAILED ***
> >
> > 2.0 was not less than 0.001 (DrmLikeSuiteBase.scala:116)
> > - DRM addToRowCardinality - wont fail
> >
> >
> > BTW this implies rbind will not solve the problem, it is firmly in data
> >> prep. But until I know the rules I won’t know how to do the right thing.
> >>
> >
> > Rbind expects both A and B to have their Int row keys filled from 0 to
> > nrow-1, which is how they should be ideally.
> >
> >
> >
>
>
Re: A + 1
Posted by Pat Ferrel <pa...@gmail.com>.
Sorry but my question keep getting lost in the back and forth, no one reads that much email.
Oh, it’s working. The conversion to dense is saving the day in #1 I think. Where it is using nrow to create enough rows rather than looking in the rdd for row keys—just a guess.
I think it was you that said dense or sparse the math should produce the same result. One or the other is a bug, right?
I am going to add some matrix multiply tests that work on rdd backed objects. The current tests could use some additions. I suspect that multiply and transpose will work correctly with non-existant rows/columns but still need the designers to come out on one side or the other.
If row keys need to be sequential and unbroken, this is s big deal and new to me.
On Jul 19, 2014, at 10:09 PM, Anand Avati <av...@gluster.org> wrote:
On Sat, Jul 19, 2014 at 9:06 PM, Pat Ferrel <pa...@gmail.com> wrote:
> Using methods instead of symbolic ops returns different types so methods
> work, ops don’t. If math is math, they should do the smae thing so I’d like
> to know what it is supposed to do, can you please allow me to ask specific
> people the question?
>
I don't see how I'm coming in the way of anybody else from replying. By
posting to a public mailing list, you by definition allow anybody to review
and comment.
test("plus one"){
> val a = dense(
> (1, 1),
> (0, 0))
>
> val drmA1 = drmParallelize(m = a, numPartitions = 2)
>
> // modified to return a new CheckpointedDrm so maintains immutability
> but still only increases the row cardinality
> // by returning new CheckpointedDrmSpark[K](rdd, nrow + n, ncol,
> _cacheStorageLevel ) Hack for now.
> val drmABigger1 = drmA1.addToRowCardinality(1)
>
> val drmABiggerPlusOne1 = drmABigger1.plus(1.0) // drmABigger has no
> row 2 in the rdd but an empty row 1
> // drmABiggerPlusOne1 is a dense matrix
> println(drmABiggerPlusOne1)
>
> val drmA2 = drmParallelize(m = a, numPartitions = 2)
> val drmABigger2 = drmA2.addToRowCardinality(1)
> val drmABiggerPlusOne2 = drmABigger2 + 1.0
> drmABiggerPlusOne2.writeDRM("tmp/plus-one/drma-bigger-plus-one-ops/")
>
>
> val bp = 0
> }
>
> method #1 works, #2 doesn’t. Even when I create a new CheckpintedDrmSpark
> with larger _nrow than is in the data—even without the addToRowCardinality.
>
In Method #1, plus() is not even operating on the DRM. The plus() is
operating on in core Matrix which is implicitly collect()ed because of the
drm2InCore implicit type converter. So drmABiggerPlusOne1 is neither a
DrmLike nor CheckpointedDrm, but is actually just an incore Matrix. You
will have to drmParallelize() it again in order to do any distributed
operations.
I agree that in some cases it doesn’t can you please allow me to ask if it
> _should_?
>
I don't think it is working in any case. The implicit converter is making
it feel like it is working.
Thanks
> On Jul 19, 2014, at 8:56 PM, Anand Avati <av...@gluster.org> wrote:
>
>
>
>
> On Sat, Jul 19, 2014 at 6:50 PM, Pat Ferrel <pa...@gmail.com> wrote:
>
>>
>> On another thread I’ll send you code that shows A + 1 works with blank
>> rows in A.
>>
>
> I don't see how that worked for you. See this:
>
> test("DRM addToRowCardinality - will fail") {
> val inCoreA = sparse(
> 0 -> 1 :: 1 -> 2 :: Nil,
> 0 -> 3 :: 1 -> 4 :: Nil,
> 0 -> 2 :: 1 -> 0.0 :: Nil
> )
>
> val inCoreBControl = sparse(
> 0 -> 2 :: 1 -> 3 :: Nil,
> 0 -> 4 :: 1 -> 5 :: Nil,
> 0 -> 3 :: 1 -> 1 :: Nil,
> 0 -> 1 :: 1 -> 1 :: Nil,
> 0 -> 1 :: 1 -> 1 :: Nil
> )
>
> val drmA = drmParallelize(inCoreA)
> drmA.addToRowCardinality(2)
> val drmB = (drmA + 1.0).checkpoint()
>
> (drmB.collect - inCoreBControl).norm should be < 1e-3
>
> }
>
> test("DRM addToRowCardinality - wont fail") {
> val inCoreA = sparse(
> 0 -> 1 :: 1 -> 2 :: Nil,
> 0 -> 3 :: 1 -> 4 :: Nil,
> 0 -> 2 :: 1 -> 0.0 :: Nil
> )
>
> val inCoreBWrong = sparse(
> 0 -> 2 :: 1 -> 3 :: Nil,
> 0 -> 4 :: 1 -> 5 :: Nil,
> 0 -> 3 :: 1 -> 1 :: Nil,
> 0 -> 0 :: 1 -> 0 :: Nil,
> 0 -> 0 :: 1 -> 0 :: Nil
> )
>
> val drmA = drmParallelize(inCoreA)
> drmA.addToRowCardinality(2)
> val drmB = (drmA + 1.0).checkpoint()
>
> (drmB.collect - inCoreBWrong).norm should be < 1e-3
> }
>
>
> And sure enough, inCoreBControl fails, and inCoreBWrong succeeds:
>
> - DRM addToRowCardinality - will fail *** FAILED ***
>
> 2.0 was not less than 0.001 (DrmLikeSuiteBase.scala:116)
> - DRM addToRowCardinality - wont fail
>
>
> BTW this implies rbind will not solve the problem, it is firmly in data
>> prep. But until I know the rules I won’t know how to do the right thing.
>>
>
> Rbind expects both A and B to have their Int row keys filled from 0 to
> nrow-1, which is how they should be ideally.
>
>
>
Re: A + 1
Posted by Anand Avati <av...@gluster.org>.
On Sat, Jul 19, 2014 at 9:06 PM, Pat Ferrel <pa...@gmail.com> wrote:
> Using methods instead of symbolic ops returns different types so methods
> work, ops don’t. If math is math, they should do the smae thing so I’d like
> to know what it is supposed to do, can you please allow me to ask specific
> people the question?
>
I don't see how I'm coming in the way of anybody else from replying. By
posting to a public mailing list, you by definition allow anybody to review
and comment.
test("plus one"){
> val a = dense(
> (1, 1),
> (0, 0))
>
> val drmA1 = drmParallelize(m = a, numPartitions = 2)
>
> // modified to return a new CheckpointedDrm so maintains immutability
> but still only increases the row cardinality
> // by returning new CheckpointedDrmSpark[K](rdd, nrow + n, ncol,
> _cacheStorageLevel ) Hack for now.
> val drmABigger1 = drmA1.addToRowCardinality(1)
>
> val drmABiggerPlusOne1 = drmABigger1.plus(1.0) // drmABigger has no
> row 2 in the rdd but an empty row 1
> // drmABiggerPlusOne1 is a dense matrix
> println(drmABiggerPlusOne1)
>
> val drmA2 = drmParallelize(m = a, numPartitions = 2)
> val drmABigger2 = drmA2.addToRowCardinality(1)
> val drmABiggerPlusOne2 = drmABigger2 + 1.0
> drmABiggerPlusOne2.writeDRM("tmp/plus-one/drma-bigger-plus-one-ops/")
>
>
> val bp = 0
> }
>
> method #1 works, #2 doesn’t. Even when I create a new CheckpintedDrmSpark
> with larger _nrow than is in the data—even without the addToRowCardinality.
>
In Method #1, plus() is not even operating on the DRM. The plus() is
operating on in core Matrix which is implicitly collect()ed because of the
drm2InCore implicit type converter. So drmABiggerPlusOne1 is neither a
DrmLike nor CheckpointedDrm, but is actually just an incore Matrix. You
will have to drmParallelize() it again in order to do any distributed
operations.
I agree that in some cases it doesn’t can you please allow me to ask if it
> _should_?
>
I don't think it is working in any case. The implicit converter is making
it feel like it is working.
Thanks
> On Jul 19, 2014, at 8:56 PM, Anand Avati <av...@gluster.org> wrote:
>
>
>
>
> On Sat, Jul 19, 2014 at 6:50 PM, Pat Ferrel <pa...@gmail.com> wrote:
>
>>
>> On another thread I’ll send you code that shows A + 1 works with blank
>> rows in A.
>>
>
> I don't see how that worked for you. See this:
>
> test("DRM addToRowCardinality - will fail") {
> val inCoreA = sparse(
> 0 -> 1 :: 1 -> 2 :: Nil,
> 0 -> 3 :: 1 -> 4 :: Nil,
> 0 -> 2 :: 1 -> 0.0 :: Nil
> )
>
> val inCoreBControl = sparse(
> 0 -> 2 :: 1 -> 3 :: Nil,
> 0 -> 4 :: 1 -> 5 :: Nil,
> 0 -> 3 :: 1 -> 1 :: Nil,
> 0 -> 1 :: 1 -> 1 :: Nil,
> 0 -> 1 :: 1 -> 1 :: Nil
> )
>
> val drmA = drmParallelize(inCoreA)
> drmA.addToRowCardinality(2)
> val drmB = (drmA + 1.0).checkpoint()
>
> (drmB.collect - inCoreBControl).norm should be < 1e-3
>
> }
>
> test("DRM addToRowCardinality - wont fail") {
> val inCoreA = sparse(
> 0 -> 1 :: 1 -> 2 :: Nil,
> 0 -> 3 :: 1 -> 4 :: Nil,
> 0 -> 2 :: 1 -> 0.0 :: Nil
> )
>
> val inCoreBWrong = sparse(
> 0 -> 2 :: 1 -> 3 :: Nil,
> 0 -> 4 :: 1 -> 5 :: Nil,
> 0 -> 3 :: 1 -> 1 :: Nil,
> 0 -> 0 :: 1 -> 0 :: Nil,
> 0 -> 0 :: 1 -> 0 :: Nil
> )
>
> val drmA = drmParallelize(inCoreA)
> drmA.addToRowCardinality(2)
> val drmB = (drmA + 1.0).checkpoint()
>
> (drmB.collect - inCoreBWrong).norm should be < 1e-3
> }
>
>
> And sure enough, inCoreBControl fails, and inCoreBWrong succeeds:
>
> - DRM addToRowCardinality - will fail *** FAILED ***
>
> 2.0 was not less than 0.001 (DrmLikeSuiteBase.scala:116)
> - DRM addToRowCardinality - wont fail
>
>
> BTW this implies rbind will not solve the problem, it is firmly in data
>> prep. But until I know the rules I won’t know how to do the right thing.
>>
>
> Rbind expects both A and B to have their Int row keys filled from 0 to
> nrow-1, which is how they should be ideally.
>
>
>
A + 1
Posted by Pat Ferrel <pa...@gmail.com>.
Using methods instead of symbolic ops returns different types so methods work, ops don’t. If math is math, they should do the smae thing so I’d like to know what it is supposed to do, can you please allow me to ask specific people the question?
test("plus one"){
val a = dense(
(1, 1),
(0, 0))
val drmA1 = drmParallelize(m = a, numPartitions = 2)
// modified to return a new CheckpointedDrm so maintains immutability but still only increases the row cardinality
// by returning new CheckpointedDrmSpark[K](rdd, nrow + n, ncol, _cacheStorageLevel ) Hack for now.
val drmABigger1 = drmA1.addToRowCardinality(1)
val drmABiggerPlusOne1 = drmABigger1.plus(1.0) // drmABigger has no row 2 in the rdd but an empty row 1
// drmABiggerPlusOne1 is a dense matrix
println(drmABiggerPlusOne1)
val drmA2 = drmParallelize(m = a, numPartitions = 2)
val drmABigger2 = drmA2.addToRowCardinality(1)
val drmABiggerPlusOne2 = drmABigger2 + 1.0
drmABiggerPlusOne2.writeDRM("tmp/plus-one/drma-bigger-plus-one-ops/")
val bp = 0
}
method #1 works, #2 doesn’t. Even when I create a new CheckpintedDrmSpark with larger _nrow than is in the data—even without the addToRowCardinality.
I agree that in some cases it doesn’t can you please allow me to ask if it _should_?
On Jul 19, 2014, at 8:56 PM, Anand Avati <av...@gluster.org> wrote:
On Sat, Jul 19, 2014 at 6:50 PM, Pat Ferrel <pa...@gmail.com> wrote:
On another thread I’ll send you code that shows A + 1 works with blank rows in A.
I don't see how that worked for you. See this:
test("DRM addToRowCardinality - will fail") {
val inCoreA = sparse(
0 -> 1 :: 1 -> 2 :: Nil,
0 -> 3 :: 1 -> 4 :: Nil,
0 -> 2 :: 1 -> 0.0 :: Nil
)
val inCoreBControl = sparse(
0 -> 2 :: 1 -> 3 :: Nil,
0 -> 4 :: 1 -> 5 :: Nil,
0 -> 3 :: 1 -> 1 :: Nil,
0 -> 1 :: 1 -> 1 :: Nil,
0 -> 1 :: 1 -> 1 :: Nil
)
val drmA = drmParallelize(inCoreA)
drmA.addToRowCardinality(2)
val drmB = (drmA + 1.0).checkpoint()
(drmB.collect - inCoreBControl).norm should be < 1e-3
}
test("DRM addToRowCardinality - wont fail") {
val inCoreA = sparse(
0 -> 1 :: 1 -> 2 :: Nil,
0 -> 3 :: 1 -> 4 :: Nil,
0 -> 2 :: 1 -> 0.0 :: Nil
)
val inCoreBWrong = sparse(
0 -> 2 :: 1 -> 3 :: Nil,
0 -> 4 :: 1 -> 5 :: Nil,
0 -> 3 :: 1 -> 1 :: Nil,
0 -> 0 :: 1 -> 0 :: Nil,
0 -> 0 :: 1 -> 0 :: Nil
)
val drmA = drmParallelize(inCoreA)
drmA.addToRowCardinality(2)
val drmB = (drmA + 1.0).checkpoint()
(drmB.collect - inCoreBWrong).norm should be < 1e-3
}
And sure enough, inCoreBControl fails, and inCoreBWrong succeeds:
- DRM addToRowCardinality - will fail *** FAILED ***
2.0 was not less than 0.001 (DrmLikeSuiteBase.scala:116)
- DRM addToRowCardinality - wont fail
BTW this implies rbind will not solve the problem, it is firmly in data prep. But until I know the rules I won’t know how to do the right thing.
Rbind expects both A and B to have their Int row keys filled from 0 to nrow-1, which is how they should be ideally.
Re: Problem of dimensions
Posted by Anand Avati <av...@gluster.org>.
On Sat, Jul 19, 2014 at 6:50 PM, Pat Ferrel <pa...@gmail.com> wrote:
>
> On another thread I’ll send you code that shows A + 1 works with blank
> rows in A.
>
I don't see how that worked for you. See this:
test("DRM addToRowCardinality - will fail") {
val inCoreA = sparse(
0 -> 1 :: 1 -> 2 :: Nil,
0 -> 3 :: 1 -> 4 :: Nil,
0 -> 2 :: 1 -> 0.0 :: Nil
)
val inCoreBControl = sparse(
0 -> 2 :: 1 -> 3 :: Nil,
0 -> 4 :: 1 -> 5 :: Nil,
0 -> 3 :: 1 -> 1 :: Nil,
0 -> 1 :: 1 -> 1 :: Nil,
0 -> 1 :: 1 -> 1 :: Nil
)
val drmA = drmParallelize(inCoreA)
drmA.addToRowCardinality(2)
val drmB = (drmA + 1.0).checkpoint()
(drmB.collect - inCoreBControl).norm should be < 1e-3
}
test("DRM addToRowCardinality - wont fail") {
val inCoreA = sparse(
0 -> 1 :: 1 -> 2 :: Nil,
0 -> 3 :: 1 -> 4 :: Nil,
0 -> 2 :: 1 -> 0.0 :: Nil
)
val inCoreBWrong = sparse(
0 -> 2 :: 1 -> 3 :: Nil,
0 -> 4 :: 1 -> 5 :: Nil,
0 -> 3 :: 1 -> 1 :: Nil,
0 -> 0 :: 1 -> 0 :: Nil,
0 -> 0 :: 1 -> 0 :: Nil
)
val drmA = drmParallelize(inCoreA)
drmA.addToRowCardinality(2)
val drmB = (drmA + 1.0).checkpoint()
(drmB.collect - inCoreBWrong).norm should be < 1e-3
}
And sure enough, inCoreBControl fails, and inCoreBWrong succeeds:
- DRM addToRowCardinality - will fail *** FAILED ***
2.0 was not less than 0.001 (DrmLikeSuiteBase.scala:116)
- DRM addToRowCardinality - wont fail
BTW this implies rbind will not solve the problem, it is firmly in data
> prep. But until I know the rules I won’t know how to do the right thing.
>
Rbind expects both A and B to have their Int row keys filled from 0 to
nrow-1, which is how they should be ideally.
Re: Problem of dimensions
Posted by Pat Ferrel <pa...@gmail.com>.
I appreciate your help Anand. You may know the area better than most but my question is about _should_ not _does_. It is one of intent, what is our covenant with the gods (to: line).
On another thread I’ll send you code that shows A + 1 works with blank rows in A. BTW this implies rbind will not solve the problem, it is firmly in data prep. But until I know the rules I won’t know how to do the right thing.
On Jul 19, 2014, at 5:25 PM, Anand Avati <av...@gluster.org> wrote:
On Sat, Jul 19, 2014 at 5:04 PM, Pat Ferrel <pa...@gmail.com> wrote:
We need to back up a bit here. This involves two questions, one for core math one for data prep:
1) The math question: does a CheckpointedDrm need to have a row for every sequential row key from 0 to nrow? Can there be missing row keys in the sequence and still get correct results for B %*% C where C and/or B have rows that have no representation in the underlying rdd, not even n => {} but have the same _nrow passed in during creation.
2) The data prep issue depends on the answer to #1: potentially there are matrices A, B, C, … All come from data whose rows are IDed by external User IDs. The total of these IDs define a row cardinality for all matrices. The total number of Mahout row keys will come from the collected number of unique User IDs.
If the answer to #1 is “yes you must have at least n => {} for every sequential row key 0 through nrow”. Then A, B, C, and so on will need to have the Int row Keys inserted at all points in the matrices where no data for the external ID was seen. This implies reading them in as a unit. Rbind cannot do this after each matrix has bee read in since the row key gaps may not all be at the end of a matrix.
If the answer to #1 is that a non-existant row key (a gap in the sequence) is exactly the same as having in rdd n => {} then changing only the row cardinality of all matrices to match the total number of IDs seen will create the correct result. If rbind with drmParallelizeEmpty can be used to only change the cardinality then it may work.
I’ll keep poking at #1 but would love a definitive answer.
The answer _has_ to be "yes". There cannot be missing row keys for Int keyed DRMs. The proof for my claim is in my previous mail, that
val drmB = drmA + 1
will give incorrect result (at least on spark backend) if there are such missing rows.
Thanks
Re: Problem of dimensions
Posted by Anand Avati <av...@gluster.org>.
On Sat, Jul 19, 2014 at 5:04 PM, Pat Ferrel <pa...@gmail.com> wrote:
> We need to back up a bit here. This involves two questions, one for core
> math one for data prep:
>
> 1) The math question: does a CheckpointedDrm need to have a row for every
> sequential row key from 0 to nrow? Can there be missing row keys in the
> sequence and still get correct results for B %*% C where C and/or B have
> rows that have no representation in the underlying rdd, not even n => {}
> but have the same _nrow passed in during creation.
>
> 2) The data prep issue depends on the answer to #1: potentially there are
> matrices A, B, C, … All come from data whose rows are IDed by external User
> IDs. The total of these IDs define a row cardinality for all matrices. The
> total number of Mahout row keys will come from the collected number of
> unique User IDs.
>
> If the answer to #1 is “yes you must have at least n => {} for every
> sequential row key 0 through nrow”. Then A, B, C, and so on will need to
> have the Int row Keys inserted at all points in the matrices where no data
> for the external ID was seen. This implies reading them in as a unit. Rbind
> cannot do this after each matrix has bee read in since the row key gaps may
> not all be at the end of a matrix.
>
> If the answer to #1 is that a non-existant row key (a gap in the sequence)
> is exactly the same as having in rdd n => {} then changing only the row
> cardinality of all matrices to match the total number of IDs seen will
> create the correct result. If rbind with drmParallelizeEmpty can be used to
> only change the cardinality then it may work.
>
> I’ll keep poking at #1 but would love a definitive answer.
The answer _has_ to be "yes". There cannot be missing row keys for Int
keyed DRMs. The proof for my claim is in my previous mail, that
val drmB = drmA + 1
will give incorrect result (at least on spark backend) if there are such
missing rows.
Thanks
Re: Problem of dimensions
Posted by Pat Ferrel <pa...@gmail.com>.
We need to back up a bit here. This involves two questions, one for core math one for data prep:
1) The math question: does a CheckpointedDrm need to have a row for every sequential row key from 0 to nrow? Can there be missing row keys in the sequence and still get correct results for B %*% C where C and/or B have rows that have no representation in the underlying rdd, not even n => {} but have the same _nrow passed in during creation.
2) The data prep issue depends on the answer to #1: potentially there are matrices A, B, C, … All come from data whose rows are IDed by external User IDs. The total of these IDs define a row cardinality for all matrices. The total number of Mahout row keys will come from the collected number of unique User IDs.
If the answer to #1 is “yes you must have at least n => {} for every sequential row key 0 through nrow”. Then A, B, C, and so on will need to have the Int row Keys inserted at all points in the matrices where no data for the external ID was seen. This implies reading them in as a unit. Rbind cannot do this after each matrix has bee read in since the row key gaps may not all be at the end of a matrix.
If the answer to #1 is that a non-existant row key (a gap in the sequence) is exactly the same as having in rdd n => {} then changing only the row cardinality of all matrices to match the total number of IDs seen will create the correct result. If rbind with drmParallelizeEmpty can be used to only change the cardinality then it may work.
I’ll keep poking at #1 but would love a definitive answer.
Re: Problem of dimensions
Posted by Anand Avati <av...@gluster.org>.
Replies inline.
On Fri, Jul 18, 2014 at 12:42 PM, Pat Ferrel <pa...@gmail.com> wrote:
> If you could comment on the PR that would be great. In this case I know
> where the code is you are talking about.
>
> 1) OK, this is a good catch. I didn’t know CheckpointedDrmSpark or really
> all Drms are to be immutable, which is actually documented in 2.4 of the
> DSL PDF. I think this is what Ted was saying too, assuming I knew it was
> supposed to be immutable. Scala puts “immutable’ in the fully qualified
> class name to flag the fact. Wonder if that’s a good idea here?
>
> 2) I’m talking about the R semantics for rbind. Out of the box R is only
> dense so the semantics by definition are dense. Putting in all zero rows
> adds a bunch of 0.0 doubles to a matrix. I’m saying you don’t even need or
> want the empty row keys. This is certainly not what we want in a sparse
> vector or matrix unless needed. Please rely on Dmitriy, Sebastian, or Ted
> about this and maybe they can contradict me.
>
> 3) If I did an rbind do you want me to overload it to take an Int and only
> touch _nrow (not even sure this is possible—haven’t looked)? Is this really
> what you want?
>
>
> > On Jul 17, 2014, at 4:58 PM, Anand Avati <av...@gluster.org> wrote:
> >
> > And I still really doubt if just fudging nrow is a "complete". For e.g,
> if
> > after fixing up nrow (either mutating or by creating a new
> CheckpointedDrm
> > as I described in my previous mail), if you were to do:
> >
> > drmA = ... // somehow nrow is fudged
> >
> > drmB = drmA + 1 // invoke OpAewScalar operator
> >
> > I don't see how this would return the correct answer in drmB. mapBlock()
> on
> > drmA is just not performed on those "invisible" rows for the "+ 1" to be
> > applied on the cells.
>
> Seems like a good test. I certainly can be done correctly given my
> understanding below—not sure if it is.
>
> First you are creating a dense matrix from a sparse one—drmB is really a
> non-sparse matrix that is distributed. This requires that all non-existent
> rows and columns be created in the new matrix. The map would be over over
> all IDs from 0 to nrow, also each Vector elements needs to have 1 added,
> even the non-existent ones so you need to use the right vector iterator.
But this is not how things would happen internally. mapBlock() does not
even refer to nrow (at least in the spark implementation). It just marches
on to execute the operator on drmrdd which is unaware of the nrow fudge at
CheckpointedDrmSpark. What this would result in is, a matrix drmB where the
first orig_nrow rows have all elements incremented by +1, and the remaining
rows (orig_nrow .. nrow) will just continue to be "sparse" (0s) -- clearly
wrong.
> There are several cases where dense matrices are created from sparse ones
> like factorization. Assumptions about ordinality and the row or columns IDs
> allow this to happen. So new dense rows and elements may be created
> assuming that key ordinality and nrow can be used to determine missing rows
> (or columns). The point would be not to force a dense anything unless
> needed, as in your case above.
>
> The question is good and i admit that my knowledge of this is not the best
> so please refer to the experts.
>
> >
> > I think rbind() is the safest approach here. Also, I'm not sure why you
> > feel rbind() is only for "dense" matrices. If the B matrix for rbind
> > operator was created from drmParallelizeEmpty() (as shown in the example
> in
> > the commit), the DrmRdd will be holding only empty
> RandomAccessSparseVectors
> > and will be significantly less expensive than a dense operation.
>
> I didn’t say that rbind is only for dense matrices, at least that isn’t
> what I meant. If it requires me to calculate missing rows IDs for no
> reason, it’s wrong.
Can you please explain? All you need to do is literally replace
selfSimilarityDataset.addToRowCardinality(newRowIDsFound)
with
selfSimilarityDataset = selfSimilarityDataset rbind
drmParallelizeEmpty(newRowIDsFound)
(I would actually look at re-organizing the whole code and avoid such "C
like".. if (condition) { statement; } and instead make it more functional
style)
> It violates my understanding of sparse semantics—don’t mess with
> non-exsistant data in vectors of matrices unless needed (as in your example
> of adding 1 to all elements, even non-existant ones).
Sparse vs Dense are just implementation styles for efficiency (of space and
time). Logical operators must always yield the right answer whether
operands are sparse or dense. Non-existence is only in the internal
representation, not in the logical matrix. Sparseness typically is for
optimizing off lots of 0 values. You are probably referring to "missing
elements"?
> Also I meant that we shouldn’t be inflexibly bound to R since there are a
> few sparse cases that don’t fit and this seems like one.
>
rbind is an R "like" operation, just like all other R "like" operations in
the DSL. It is (I think) similar enough to indicate that we are splicing
two compatible matrices one on top of the other. I don't think it deserves
a sparse vs dense argument - just like cbind.
Thanks
Re: Problem of dimensions
Posted by Ted Dunning <te...@gmail.com>.
On Fri, Jul 18, 2014 at 12:42 PM, Pat Ferrel <pa...@gmail.com> wrote:
> 2) I’m talking about the R semantics for rbind. Out of the box R is only
> dense so the semantics by definition are dense. Putting in all zero rows
> adds a bunch of 0.0 doubles to a matrix. I’m saying you don’t even need or
> want the empty row keys. This is certainly not what we want in a sparse
> vector or matrix unless needed. Please rely on Dmitriy, Sebastian, or Ted
> about this and maybe they can contradict me.
>
I think that slavish adherence to core R semantics here is not intended or
desirable. Rbind is a function that returns a new object which has the
same effect as the dense copying, but could quite reasonably cheat for
sparse objects.
>
> 3) If I did an rbind do you want me to overload it to take an Int and only
> touch _nrow (not even sure this is possible—haven’t looked)? Is this really
> what you want?
>
rbind should construct a new object. Since RDD's are immutable, it is OK
to share structure (I think).
Re: Problem of dimensions
Posted by Pat Ferrel <pa...@gmail.com>.
If you could comment on the PR that would be great. In this case I know where the code is you are talking about.
1) OK, this is a good catch. I didn’t know CheckpointedDrmSpark or really all Drms are to be immutable, which is actually documented in 2.4 of the DSL PDF. I think this is what Ted was saying too, assuming I knew it was supposed to be immutable. Scala puts “immutable’ in the fully qualified class name to flag the fact. Wonder if that’s a good idea here?
2) I’m talking about the R semantics for rbind. Out of the box R is only dense so the semantics by definition are dense. Putting in all zero rows adds a bunch of 0.0 doubles to a matrix. I’m saying you don’t even need or want the empty row keys. This is certainly not what we want in a sparse vector or matrix unless needed. Please rely on Dmitriy, Sebastian, or Ted about this and maybe they can contradict me.
3) If I did an rbind do you want me to overload it to take an Int and only touch _nrow (not even sure this is possible—haven’t looked)? Is this really what you want?
> On Jul 17, 2014, at 4:58 PM, Anand Avati <av...@gluster.org> wrote:
>
> And I still really doubt if just fudging nrow is a "complete". For e.g, if
> after fixing up nrow (either mutating or by creating a new CheckpointedDrm
> as I described in my previous mail), if you were to do:
>
> drmA = ... // somehow nrow is fudged
>
> drmB = drmA + 1 // invoke OpAewScalar operator
>
> I don't see how this would return the correct answer in drmB. mapBlock() on
> drmA is just not performed on those "invisible" rows for the "+ 1" to be
> applied on the cells.
Seems like a good test. I certainly can be done correctly given my understanding below—not sure if it is.
First you are creating a dense matrix from a sparse one—drmB is really a non-sparse matrix that is distributed. This requires that all non-existent rows and columns be created in the new matrix. The map would be over over all IDs from 0 to nrow, also each Vector elements needs to have 1 added, even the non-existent ones so you need to use the right vector iterator. There are several cases where dense matrices are created from sparse ones like factorization. Assumptions about ordinality and the row or columns IDs allow this to happen. So new dense rows and elements may be created assuming that key ordinality and nrow can be used to determine missing rows (or columns). The point would be not to force a dense anything unless needed, as in your case above.
The question is good and i admit that my knowledge of this is not the best so please refer to the experts.
>
> I think rbind() is the safest approach here. Also, I'm not sure why you
> feel rbind() is only for "dense" matrices. If the B matrix for rbind
> operator was created from drmParallelizeEmpty() (as shown in the example in
> the commit), the DrmRdd will be holding only empty RandomAccessSparseVectors
> and will be significantly less expensive than a dense operation.
I didn’t say that rbind is only for dense matrices, at least that isn’t what I meant. If it requires me to calculate missing rows IDs for no reason, it’s wrong. It violates my understanding of sparse semantics—don’t mess with non-exsistant data in vectors of matrices unless needed (as in your example of adding 1 to all elements, even non-existant ones). Also I meant that we shouldn’t be inflexibly bound to R since there are a few sparse cases that don’t fit and this seems like one.
>
Re: Problem of dimensions
Posted by Anand Avati <av...@gluster.org>.
And I still really doubt if just fudging nrow is a "complete". For e.g, if
after fixing up nrow (either mutating or by creating a new CheckpointedDrm
as I described in my previous mail), if you were to do:
drmA = ... // somehow nrow is fudged
drmB = drmA + 1 // invoke OpAewScalar operator
I don't see how this would return the correct answer in drmB. mapBlock() on
drmA is just not performed on those "invisible" rows for the "+ 1" to be
applied on the cells.
I think rbind() is the safest approach here. Also, I'm not sure why you
feel rbind() is only for "dense" matrices. If the B matrix for rbind
operator was created from drmParallelizeEmpty() (as shown in the example in
the commit), the DrmRdd will be holding only empty RandomAccessSparseVectors
and will be significantly less expensive than a dense operation.
Thanks
On Thu, Jul 17, 2014 at 4:20 PM, Anand Avati <av...@gluster.org> wrote:
> Pat,
> I had a look at the PR. The addToRowCardinality() method is clearly
> mutating the object. A CheckpointedDrm is a "computed" object itself, but
> it very much can be a leaf node of many *other* lazy evaluated graphs for
> computing other DRMs. Besides, mutating the object goes right against the
> principles of functional programming.
>
> Whether CheckpointedDrmSpark is tolerant to arbitrary nrow values (or
> higher nrow) or not, I am not very certain yet. Assuming it is tolerant,
> the way to exploit that tolerance by mutating the object is definitely not
> ideal. Instead please create a new operator which will end up creating a
> new CheckpointedDrmSpark object with the same DrmRdd of the original
> CheckpointedDrmSpark, but newer nrow/ncol values. This way it is
> inexpensive, compatible with functional programming principles, and we can
> be a LOT more confident about having no unforeseen side effects.
>
> Thanks
>
>
> On Thu, Jul 17, 2014 at 3:11 PM, Pat Ferrel <pa...@gmail.com> wrote:
>
>> comments on PR #31
>> https://github.com/apache/mahout/pull/31
>>
>>
>> On Jul 16, 2014, at 3:46 PM, Pat Ferrel <pa...@gmail.com> wrote:
>>
>> I didn’t do anything with nrow, it is still lazy. I added a method that
>> changes _nrow, which is a var anyway. This change is made after all
>> matrices are read in and before any math/optimizer stuff is done. When nrow
>> is calculated _nrow should already be the right value unless I’ve missed
>> something. Also this is a CheckpointedDrmSpark and I thought that
>> guaranteed that it was not mid optimization.
>>
>> Let me put a PR up so you can have a look.
>>
>> I’m now wondering is changing _nrow is
>>
>>
>> On Jul 15, 2014, at 11:10 AM, Ted Dunning <te...@gmail.com> wrote:
>>
>> My worry about this is that I thought that DSL objects needed to remain
>> immutable due to the lazy evaluation.
>>
>> For instance, suppose that A and B can be multiplied because they are the
>> right shapes:
>>
>> C = A.t %*% B
>>
>> Now I change number of rows in A
>>
>> A.nrow = 2 * A.nrow
>>
>> and A now has the shape compatible with D
>>
>> E = A.t %*% D
>>
>> And now I ask for some materialization:
>>
>> x = C.sum + E.sum
>>
>> And this fails. The reason is that the change in A's shape happened
>> before
>> computation was started on multiplying A and B.
>>
>> Now, to my mind, lazy evaluation is pretty crucial because it allows the
>> optimizer to work. That seems to me to say that mutation should not be
>> allowed.
>>
>> (and yes, I know that these aren't the correct notation ... that isn't the
>> point)
>>
>>
>>
>>
>>
>> On Tue, Jul 15, 2014 at 8:26 AM, Pat Ferrel <pa...@occamsmachete.com>
>> wrote:
>>
>> > Now you have me doubting. At least for cases I haven’t tested.
>> >
>> > Dmitriy: on a sparse drm (CheckpointedDrmSpark) will increasing _nrow do
>> > the equivalent of adding empty row vectors, with keys but no vector
>> > elements?
>> >
>> >
>> > On Jul 14, 2014, at 7:30 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:
>> >
>> > It’ll be in a PR for review before it goes in and if you have already
>> > merged it will be stubbed out in the h2o implementation if I can figure
>> out
>> > where. The tests are all in the Spark specific module. All I did was
>> change
>> > _nrow in CheckpointedDrmSpark, which is a private var. I added a method
>> to
>> > do that but not in the core-math only on Spark specific code.
>> >
>> > cooccurrence works at a lower level and does not need to worry about
>> this
>> > because drm multiply and other ops just work. The place I change nrow
>> is in
>> > ItemSimilarity, which is non-core.
>> >
>> > I would indeed make sure your sparse matrix code handles this but not
>> sure
>> > where you’d test that—bug Dmitriy.
>> >
>> > On Jul 14, 2014, at 5:09 PM, Anand Avati <av...@gluster.org> wrote:
>> >
>> >
>> > On Mon, Jul 14, 2014 at 3:02 PM, Pat Ferrel <pa...@occamsmachete.com>
>> wrote:
>> >>
>> >> On Jul 14, 2014, at 12:10 PM, Anand Avati <av...@gluster.org> wrote:
>> >>
>> >> On Mon, Jul 14, 2014 at 11:56 AM, Pat Ferrel <pa...@occamsmachete.com>
>> > wrote:
>> >> In the application, the number of rows will always be increased, adding
>> > blank rows. I don’t think shuffle is necessary in this case because
>> there
>> > is no actual row, no data in the drm it’s just needed to make the
>> > cardinality match, the IDs will take care of data matching . Maybe
>> calling
>> > it something else is a good idea to emphasize the special case for it’s
>> > use. I went over this with Dmitriy and, though I haven’t checked actual
>> > values on large datasets, it works.
>> >>
>> >>
>> >> Does that mean the cardinality is faked at the logical layer with no
>> > changes at the engine level? Does that means the physical operators
>> need to
>> > be prepared to handle non-matching matrix multiplication by assuming the
>> > missing rows or columns are 0's? Does that really work with no changes?
>> >
>> > yes, Dmitriy recently confirmed this. But not faked, it is just not
>> > possible to calculate it from data in some cases since “does not exist”
>> may
>> > mean “= 0”.
>> >
>> > I’m no R expert but base R seems to assume a 0 value actually exists in
>> > the Matrix encoded in as much space as it’s type dictates, like Dense
>> > things in Mahout. I think there are R packages that add support for
>> sparse
>> > things (slam?) and so assume this is one place where some rethinking is
>> > required:
>> >
>> http://www.johnmyleswhite.com/notebook/2011/10/31/using-sparse-matrices-in-r/
>> >
>> > Sparse linear algebra with the added complication of foreign IDs makes
>> for
>> > some odd cases. The number of extrenal/foreign IDs for rows and columns
>> > defines the true cardinality even though in a sparse matrix the empty
>> row
>> > or column is just absent. In Mahout the IDs are row and column numbers
>> so
>> > there are cases where the real-world cardinality does not match the
>> number
>> > of Mahout IDs or DRM cardinality and the calculations should be fine
>> with
>> > that as long as the real-world dimensions are supplied for cardinality
>> > checking and various values calculated from cardinality.
>> >
>> > I’m maintaining a mapping of external ID to/from Mahout ID. For instance
>> > in item similarity it’s use is where the rows have a key = user ID. For
>> a
>> > single application the row space is defined by all user IDs. In the
>> cross
>> > similarity A’B reading in A may not ID every user so if B find more or
>> > different ones then the union of the two is our best guess at the total.
>> > And in fact more matrices could be added and the user ID space is all
>> users
>> > seen in all of the data. If we knew how many users are defined in the
>> > application we could also use that but it’s not needed of there is no
>> data
>> > at all for some users.
>> >
>> > BTW this mapping seems to be one of the biggest generator of questions
>> on
>> > the lists. The above issue is one that would likeli further trip up
>> users
>> > generating their own ID mapping, which is why we are finally doing it
>> for
>> > them.
>> >
>> >>
>> >> This sounds like a need to introduce a new R-like rbind() operator.
>> This
>> > way you could fix up row cardinality like:
>> >>
>> >> drmAnew = drmA rbind drmParallelizeEmpty(extra_rows, drmA.ncol)
>> >>
>> >
>> > true, add an empty slice.
>> >
>> >> You could already do this, though twisted::
>> >>
>> >> drmAnew = (drmA.t cbind drmParallelizeEmpty(drmA.ncol, extra_rows).t
>> >>
>> >>
>> >
>> > yes and I can dig deeper and do another drmWrap constructing a new
>> larger
>> > matrix.
>> >
>> > Still changing the number of rows on Sparse matrices is so much simpler
>> > but I think "drm.nrow = “ may hide the special nature of what we are
>> doing.
>> >
>> >
>> > So finally how are you changing the cardinality? I just want to make h2o
>> > engine "works" with that technique.
>> >
>> >
>> >
>> >
>> >
>>
>>
>>
>
Re: Problem of dimensions
Posted by Anand Avati <av...@gluster.org>.
Pat,
I had a look at the PR. The addToRowCardinality() method is clearly
mutating the object. A CheckpointedDrm is a "computed" object itself, but
it very much can be a leaf node of many *other* lazy evaluated graphs for
computing other DRMs. Besides, mutating the object goes right against the
principles of functional programming.
Whether CheckpointedDrmSpark is tolerant to arbitrary nrow values (or
higher nrow) or not, I am not very certain yet. Assuming it is tolerant,
the way to exploit that tolerance by mutating the object is definitely not
ideal. Instead please create a new operator which will end up creating a
new CheckpointedDrmSpark object with the same DrmRdd of the original
CheckpointedDrmSpark, but newer nrow/ncol values. This way it is
inexpensive, compatible with functional programming principles, and we can
be a LOT more confident about having no unforeseen side effects.
Thanks
On Thu, Jul 17, 2014 at 3:11 PM, Pat Ferrel <pa...@gmail.com> wrote:
> comments on PR #31
> https://github.com/apache/mahout/pull/31
>
>
> On Jul 16, 2014, at 3:46 PM, Pat Ferrel <pa...@gmail.com> wrote:
>
> I didn’t do anything with nrow, it is still lazy. I added a method that
> changes _nrow, which is a var anyway. This change is made after all
> matrices are read in and before any math/optimizer stuff is done. When nrow
> is calculated _nrow should already be the right value unless I’ve missed
> something. Also this is a CheckpointedDrmSpark and I thought that
> guaranteed that it was not mid optimization.
>
> Let me put a PR up so you can have a look.
>
> I’m now wondering is changing _nrow is
>
>
> On Jul 15, 2014, at 11:10 AM, Ted Dunning <te...@gmail.com> wrote:
>
> My worry about this is that I thought that DSL objects needed to remain
> immutable due to the lazy evaluation.
>
> For instance, suppose that A and B can be multiplied because they are the
> right shapes:
>
> C = A.t %*% B
>
> Now I change number of rows in A
>
> A.nrow = 2 * A.nrow
>
> and A now has the shape compatible with D
>
> E = A.t %*% D
>
> And now I ask for some materialization:
>
> x = C.sum + E.sum
>
> And this fails. The reason is that the change in A's shape happened before
> computation was started on multiplying A and B.
>
> Now, to my mind, lazy evaluation is pretty crucial because it allows the
> optimizer to work. That seems to me to say that mutation should not be
> allowed.
>
> (and yes, I know that these aren't the correct notation ... that isn't the
> point)
>
>
>
>
>
> On Tue, Jul 15, 2014 at 8:26 AM, Pat Ferrel <pa...@occamsmachete.com> wrote:
>
> > Now you have me doubting. At least for cases I haven’t tested.
> >
> > Dmitriy: on a sparse drm (CheckpointedDrmSpark) will increasing _nrow do
> > the equivalent of adding empty row vectors, with keys but no vector
> > elements?
> >
> >
> > On Jul 14, 2014, at 7:30 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:
> >
> > It’ll be in a PR for review before it goes in and if you have already
> > merged it will be stubbed out in the h2o implementation if I can figure
> out
> > where. The tests are all in the Spark specific module. All I did was
> change
> > _nrow in CheckpointedDrmSpark, which is a private var. I added a method
> to
> > do that but not in the core-math only on Spark specific code.
> >
> > cooccurrence works at a lower level and does not need to worry about this
> > because drm multiply and other ops just work. The place I change nrow is
> in
> > ItemSimilarity, which is non-core.
> >
> > I would indeed make sure your sparse matrix code handles this but not
> sure
> > where you’d test that—bug Dmitriy.
> >
> > On Jul 14, 2014, at 5:09 PM, Anand Avati <av...@gluster.org> wrote:
> >
> >
> > On Mon, Jul 14, 2014 at 3:02 PM, Pat Ferrel <pa...@occamsmachete.com>
> wrote:
> >>
> >> On Jul 14, 2014, at 12:10 PM, Anand Avati <av...@gluster.org> wrote:
> >>
> >> On Mon, Jul 14, 2014 at 11:56 AM, Pat Ferrel <pa...@occamsmachete.com>
> > wrote:
> >> In the application, the number of rows will always be increased, adding
> > blank rows. I don’t think shuffle is necessary in this case because there
> > is no actual row, no data in the drm it’s just needed to make the
> > cardinality match, the IDs will take care of data matching . Maybe
> calling
> > it something else is a good idea to emphasize the special case for it’s
> > use. I went over this with Dmitriy and, though I haven’t checked actual
> > values on large datasets, it works.
> >>
> >>
> >> Does that mean the cardinality is faked at the logical layer with no
> > changes at the engine level? Does that means the physical operators need
> to
> > be prepared to handle non-matching matrix multiplication by assuming the
> > missing rows or columns are 0's? Does that really work with no changes?
> >
> > yes, Dmitriy recently confirmed this. But not faked, it is just not
> > possible to calculate it from data in some cases since “does not exist”
> may
> > mean “= 0”.
> >
> > I’m no R expert but base R seems to assume a 0 value actually exists in
> > the Matrix encoded in as much space as it’s type dictates, like Dense
> > things in Mahout. I think there are R packages that add support for
> sparse
> > things (slam?) and so assume this is one place where some rethinking is
> > required:
> >
> http://www.johnmyleswhite.com/notebook/2011/10/31/using-sparse-matrices-in-r/
> >
> > Sparse linear algebra with the added complication of foreign IDs makes
> for
> > some odd cases. The number of extrenal/foreign IDs for rows and columns
> > defines the true cardinality even though in a sparse matrix the empty row
> > or column is just absent. In Mahout the IDs are row and column numbers so
> > there are cases where the real-world cardinality does not match the
> number
> > of Mahout IDs or DRM cardinality and the calculations should be fine with
> > that as long as the real-world dimensions are supplied for cardinality
> > checking and various values calculated from cardinality.
> >
> > I’m maintaining a mapping of external ID to/from Mahout ID. For instance
> > in item similarity it’s use is where the rows have a key = user ID. For a
> > single application the row space is defined by all user IDs. In the cross
> > similarity A’B reading in A may not ID every user so if B find more or
> > different ones then the union of the two is our best guess at the total.
> > And in fact more matrices could be added and the user ID space is all
> users
> > seen in all of the data. If we knew how many users are defined in the
> > application we could also use that but it’s not needed of there is no
> data
> > at all for some users.
> >
> > BTW this mapping seems to be one of the biggest generator of questions on
> > the lists. The above issue is one that would likeli further trip up users
> > generating their own ID mapping, which is why we are finally doing it for
> > them.
> >
> >>
> >> This sounds like a need to introduce a new R-like rbind() operator. This
> > way you could fix up row cardinality like:
> >>
> >> drmAnew = drmA rbind drmParallelizeEmpty(extra_rows, drmA.ncol)
> >>
> >
> > true, add an empty slice.
> >
> >> You could already do this, though twisted::
> >>
> >> drmAnew = (drmA.t cbind drmParallelizeEmpty(drmA.ncol, extra_rows).t
> >>
> >>
> >
> > yes and I can dig deeper and do another drmWrap constructing a new larger
> > matrix.
> >
> > Still changing the number of rows on Sparse matrices is so much simpler
> > but I think "drm.nrow = “ may hide the special nature of what we are
> doing.
> >
> >
> > So finally how are you changing the cardinality? I just want to make h2o
> > engine "works" with that technique.
> >
> >
> >
> >
> >
>
>
>
Re: Problem of dimensions
Posted by Pat Ferrel <pa...@gmail.com>.
comments on PR #31
https://github.com/apache/mahout/pull/31
On Jul 16, 2014, at 3:46 PM, Pat Ferrel <pa...@gmail.com> wrote:
I didn’t do anything with nrow, it is still lazy. I added a method that changes _nrow, which is a var anyway. This change is made after all matrices are read in and before any math/optimizer stuff is done. When nrow is calculated _nrow should already be the right value unless I’ve missed something. Also this is a CheckpointedDrmSpark and I thought that guaranteed that it was not mid optimization.
Let me put a PR up so you can have a look.
I’m now wondering is changing _nrow is
On Jul 15, 2014, at 11:10 AM, Ted Dunning <te...@gmail.com> wrote:
My worry about this is that I thought that DSL objects needed to remain
immutable due to the lazy evaluation.
For instance, suppose that A and B can be multiplied because they are the
right shapes:
C = A.t %*% B
Now I change number of rows in A
A.nrow = 2 * A.nrow
and A now has the shape compatible with D
E = A.t %*% D
And now I ask for some materialization:
x = C.sum + E.sum
And this fails. The reason is that the change in A's shape happened before
computation was started on multiplying A and B.
Now, to my mind, lazy evaluation is pretty crucial because it allows the
optimizer to work. That seems to me to say that mutation should not be
allowed.
(and yes, I know that these aren't the correct notation ... that isn't the
point)
On Tue, Jul 15, 2014 at 8:26 AM, Pat Ferrel <pa...@occamsmachete.com> wrote:
> Now you have me doubting. At least for cases I haven’t tested.
>
> Dmitriy: on a sparse drm (CheckpointedDrmSpark) will increasing _nrow do
> the equivalent of adding empty row vectors, with keys but no vector
> elements?
>
>
> On Jul 14, 2014, at 7:30 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:
>
> It’ll be in a PR for review before it goes in and if you have already
> merged it will be stubbed out in the h2o implementation if I can figure out
> where. The tests are all in the Spark specific module. All I did was change
> _nrow in CheckpointedDrmSpark, which is a private var. I added a method to
> do that but not in the core-math only on Spark specific code.
>
> cooccurrence works at a lower level and does not need to worry about this
> because drm multiply and other ops just work. The place I change nrow is in
> ItemSimilarity, which is non-core.
>
> I would indeed make sure your sparse matrix code handles this but not sure
> where you’d test that—bug Dmitriy.
>
> On Jul 14, 2014, at 5:09 PM, Anand Avati <av...@gluster.org> wrote:
>
>
> On Mon, Jul 14, 2014 at 3:02 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:
>>
>> On Jul 14, 2014, at 12:10 PM, Anand Avati <av...@gluster.org> wrote:
>>
>> On Mon, Jul 14, 2014 at 11:56 AM, Pat Ferrel <pa...@occamsmachete.com>
> wrote:
>> In the application, the number of rows will always be increased, adding
> blank rows. I don’t think shuffle is necessary in this case because there
> is no actual row, no data in the drm it’s just needed to make the
> cardinality match, the IDs will take care of data matching . Maybe calling
> it something else is a good idea to emphasize the special case for it’s
> use. I went over this with Dmitriy and, though I haven’t checked actual
> values on large datasets, it works.
>>
>>
>> Does that mean the cardinality is faked at the logical layer with no
> changes at the engine level? Does that means the physical operators need to
> be prepared to handle non-matching matrix multiplication by assuming the
> missing rows or columns are 0's? Does that really work with no changes?
>
> yes, Dmitriy recently confirmed this. But not faked, it is just not
> possible to calculate it from data in some cases since “does not exist” may
> mean “= 0”.
>
> I’m no R expert but base R seems to assume a 0 value actually exists in
> the Matrix encoded in as much space as it’s type dictates, like Dense
> things in Mahout. I think there are R packages that add support for sparse
> things (slam?) and so assume this is one place where some rethinking is
> required:
> http://www.johnmyleswhite.com/notebook/2011/10/31/using-sparse-matrices-in-r/
>
> Sparse linear algebra with the added complication of foreign IDs makes for
> some odd cases. The number of extrenal/foreign IDs for rows and columns
> defines the true cardinality even though in a sparse matrix the empty row
> or column is just absent. In Mahout the IDs are row and column numbers so
> there are cases where the real-world cardinality does not match the number
> of Mahout IDs or DRM cardinality and the calculations should be fine with
> that as long as the real-world dimensions are supplied for cardinality
> checking and various values calculated from cardinality.
>
> I’m maintaining a mapping of external ID to/from Mahout ID. For instance
> in item similarity it’s use is where the rows have a key = user ID. For a
> single application the row space is defined by all user IDs. In the cross
> similarity A’B reading in A may not ID every user so if B find more or
> different ones then the union of the two is our best guess at the total.
> And in fact more matrices could be added and the user ID space is all users
> seen in all of the data. If we knew how many users are defined in the
> application we could also use that but it’s not needed of there is no data
> at all for some users.
>
> BTW this mapping seems to be one of the biggest generator of questions on
> the lists. The above issue is one that would likeli further trip up users
> generating their own ID mapping, which is why we are finally doing it for
> them.
>
>>
>> This sounds like a need to introduce a new R-like rbind() operator. This
> way you could fix up row cardinality like:
>>
>> drmAnew = drmA rbind drmParallelizeEmpty(extra_rows, drmA.ncol)
>>
>
> true, add an empty slice.
>
>> You could already do this, though twisted::
>>
>> drmAnew = (drmA.t cbind drmParallelizeEmpty(drmA.ncol, extra_rows).t
>>
>>
>
> yes and I can dig deeper and do another drmWrap constructing a new larger
> matrix.
>
> Still changing the number of rows on Sparse matrices is so much simpler
> but I think "drm.nrow = “ may hide the special nature of what we are doing.
>
>
> So finally how are you changing the cardinality? I just want to make h2o
> engine "works" with that technique.
>
>
>
>
>
Re: Problem of dimensions
Posted by Pat Ferrel <pa...@gmail.com>.
I didn’t do anything with nrow, it is still lazy. I added a method that changes _nrow, which is a var anyway. This change is made after all matrices are read in and before any math/optimizer stuff is done. When nrow is calculated _nrow should already be the right value unless I’ve missed something. Also this is a CheckpointedDrmSpark and I thought that guaranteed that it was not mid optimization.
Let me put a PR up so you can have a look.
I’m now wondering is changing _nrow is
On Jul 15, 2014, at 11:10 AM, Ted Dunning <te...@gmail.com> wrote:
My worry about this is that I thought that DSL objects needed to remain
immutable due to the lazy evaluation.
For instance, suppose that A and B can be multiplied because they are the
right shapes:
C = A.t %*% B
Now I change number of rows in A
A.nrow = 2 * A.nrow
and A now has the shape compatible with D
E = A.t %*% D
And now I ask for some materialization:
x = C.sum + E.sum
And this fails. The reason is that the change in A's shape happened before
computation was started on multiplying A and B.
Now, to my mind, lazy evaluation is pretty crucial because it allows the
optimizer to work. That seems to me to say that mutation should not be
allowed.
(and yes, I know that these aren't the correct notation ... that isn't the
point)
On Tue, Jul 15, 2014 at 8:26 AM, Pat Ferrel <pa...@occamsmachete.com> wrote:
> Now you have me doubting. At least for cases I haven’t tested.
>
> Dmitriy: on a sparse drm (CheckpointedDrmSpark) will increasing _nrow do
> the equivalent of adding empty row vectors, with keys but no vector
> elements?
>
>
> On Jul 14, 2014, at 7:30 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:
>
> It’ll be in a PR for review before it goes in and if you have already
> merged it will be stubbed out in the h2o implementation if I can figure out
> where. The tests are all in the Spark specific module. All I did was change
> _nrow in CheckpointedDrmSpark, which is a private var. I added a method to
> do that but not in the core-math only on Spark specific code.
>
> cooccurrence works at a lower level and does not need to worry about this
> because drm multiply and other ops just work. The place I change nrow is in
> ItemSimilarity, which is non-core.
>
> I would indeed make sure your sparse matrix code handles this but not sure
> where you’d test that—bug Dmitriy.
>
> On Jul 14, 2014, at 5:09 PM, Anand Avati <av...@gluster.org> wrote:
>
>
> On Mon, Jul 14, 2014 at 3:02 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:
>>
>> On Jul 14, 2014, at 12:10 PM, Anand Avati <av...@gluster.org> wrote:
>>
>> On Mon, Jul 14, 2014 at 11:56 AM, Pat Ferrel <pa...@occamsmachete.com>
> wrote:
>> In the application, the number of rows will always be increased, adding
> blank rows. I don’t think shuffle is necessary in this case because there
> is no actual row, no data in the drm it’s just needed to make the
> cardinality match, the IDs will take care of data matching . Maybe calling
> it something else is a good idea to emphasize the special case for it’s
> use. I went over this with Dmitriy and, though I haven’t checked actual
> values on large datasets, it works.
>>
>>
>> Does that mean the cardinality is faked at the logical layer with no
> changes at the engine level? Does that means the physical operators need to
> be prepared to handle non-matching matrix multiplication by assuming the
> missing rows or columns are 0's? Does that really work with no changes?
>
> yes, Dmitriy recently confirmed this. But not faked, it is just not
> possible to calculate it from data in some cases since “does not exist” may
> mean “= 0”.
>
> I’m no R expert but base R seems to assume a 0 value actually exists in
> the Matrix encoded in as much space as it’s type dictates, like Dense
> things in Mahout. I think there are R packages that add support for sparse
> things (slam?) and so assume this is one place where some rethinking is
> required:
> http://www.johnmyleswhite.com/notebook/2011/10/31/using-sparse-matrices-in-r/
>
> Sparse linear algebra with the added complication of foreign IDs makes for
> some odd cases. The number of extrenal/foreign IDs for rows and columns
> defines the true cardinality even though in a sparse matrix the empty row
> or column is just absent. In Mahout the IDs are row and column numbers so
> there are cases where the real-world cardinality does not match the number
> of Mahout IDs or DRM cardinality and the calculations should be fine with
> that as long as the real-world dimensions are supplied for cardinality
> checking and various values calculated from cardinality.
>
> I’m maintaining a mapping of external ID to/from Mahout ID. For instance
> in item similarity it’s use is where the rows have a key = user ID. For a
> single application the row space is defined by all user IDs. In the cross
> similarity A’B reading in A may not ID every user so if B find more or
> different ones then the union of the two is our best guess at the total.
> And in fact more matrices could be added and the user ID space is all users
> seen in all of the data. If we knew how many users are defined in the
> application we could also use that but it’s not needed of there is no data
> at all for some users.
>
> BTW this mapping seems to be one of the biggest generator of questions on
> the lists. The above issue is one that would likeli further trip up users
> generating their own ID mapping, which is why we are finally doing it for
> them.
>
>>
>> This sounds like a need to introduce a new R-like rbind() operator. This
> way you could fix up row cardinality like:
>>
>> drmAnew = drmA rbind drmParallelizeEmpty(extra_rows, drmA.ncol)
>>
>
> true, add an empty slice.
>
>> You could already do this, though twisted::
>>
>> drmAnew = (drmA.t cbind drmParallelizeEmpty(drmA.ncol, extra_rows).t
>>
>>
>
> yes and I can dig deeper and do another drmWrap constructing a new larger
> matrix.
>
> Still changing the number of rows on Sparse matrices is so much simpler
> but I think "drm.nrow = “ may hide the special nature of what we are doing.
>
>
> So finally how are you changing the cardinality? I just want to make h2o
> engine "works" with that technique.
>
>
>
>
>
Re: Problem of dimensions
Posted by Ted Dunning <te...@gmail.com>.
My worry about this is that I thought that DSL objects needed to remain
immutable due to the lazy evaluation.
For instance, suppose that A and B can be multiplied because they are the
right shapes:
C = A.t %*% B
Now I change number of rows in A
A.nrow = 2 * A.nrow
and A now has the shape compatible with D
E = A.t %*% D
And now I ask for some materialization:
x = C.sum + E.sum
And this fails. The reason is that the change in A's shape happened before
computation was started on multiplying A and B.
Now, to my mind, lazy evaluation is pretty crucial because it allows the
optimizer to work. That seems to me to say that mutation should not be
allowed.
(and yes, I know that these aren't the correct notation ... that isn't the
point)
On Tue, Jul 15, 2014 at 8:26 AM, Pat Ferrel <pa...@occamsmachete.com> wrote:
> Now you have me doubting. At least for cases I haven’t tested.
>
> Dmitriy: on a sparse drm (CheckpointedDrmSpark) will increasing _nrow do
> the equivalent of adding empty row vectors, with keys but no vector
> elements?
>
>
> On Jul 14, 2014, at 7:30 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:
>
> It’ll be in a PR for review before it goes in and if you have already
> merged it will be stubbed out in the h2o implementation if I can figure out
> where. The tests are all in the Spark specific module. All I did was change
> _nrow in CheckpointedDrmSpark, which is a private var. I added a method to
> do that but not in the core-math only on Spark specific code.
>
> cooccurrence works at a lower level and does not need to worry about this
> because drm multiply and other ops just work. The place I change nrow is in
> ItemSimilarity, which is non-core.
>
> I would indeed make sure your sparse matrix code handles this but not sure
> where you’d test that—bug Dmitriy.
>
> On Jul 14, 2014, at 5:09 PM, Anand Avati <av...@gluster.org> wrote:
>
>
> On Mon, Jul 14, 2014 at 3:02 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:
> >
> > On Jul 14, 2014, at 12:10 PM, Anand Avati <av...@gluster.org> wrote:
> >
> > On Mon, Jul 14, 2014 at 11:56 AM, Pat Ferrel <pa...@occamsmachete.com>
> wrote:
> > In the application, the number of rows will always be increased, adding
> blank rows. I don’t think shuffle is necessary in this case because there
> is no actual row, no data in the drm it’s just needed to make the
> cardinality match, the IDs will take care of data matching . Maybe calling
> it something else is a good idea to emphasize the special case for it’s
> use. I went over this with Dmitriy and, though I haven’t checked actual
> values on large datasets, it works.
> >
> >
> > Does that mean the cardinality is faked at the logical layer with no
> changes at the engine level? Does that means the physical operators need to
> be prepared to handle non-matching matrix multiplication by assuming the
> missing rows or columns are 0's? Does that really work with no changes?
>
> yes, Dmitriy recently confirmed this. But not faked, it is just not
> possible to calculate it from data in some cases since “does not exist” may
> mean “= 0”.
>
> I’m no R expert but base R seems to assume a 0 value actually exists in
> the Matrix encoded in as much space as it’s type dictates, like Dense
> things in Mahout. I think there are R packages that add support for sparse
> things (slam?) and so assume this is one place where some rethinking is
> required:
> http://www.johnmyleswhite.com/notebook/2011/10/31/using-sparse-matrices-in-r/
>
> Sparse linear algebra with the added complication of foreign IDs makes for
> some odd cases. The number of extrenal/foreign IDs for rows and columns
> defines the true cardinality even though in a sparse matrix the empty row
> or column is just absent. In Mahout the IDs are row and column numbers so
> there are cases where the real-world cardinality does not match the number
> of Mahout IDs or DRM cardinality and the calculations should be fine with
> that as long as the real-world dimensions are supplied for cardinality
> checking and various values calculated from cardinality.
>
> I’m maintaining a mapping of external ID to/from Mahout ID. For instance
> in item similarity it’s use is where the rows have a key = user ID. For a
> single application the row space is defined by all user IDs. In the cross
> similarity A’B reading in A may not ID every user so if B find more or
> different ones then the union of the two is our best guess at the total.
> And in fact more matrices could be added and the user ID space is all users
> seen in all of the data. If we knew how many users are defined in the
> application we could also use that but it’s not needed of there is no data
> at all for some users.
>
> BTW this mapping seems to be one of the biggest generator of questions on
> the lists. The above issue is one that would likeli further trip up users
> generating their own ID mapping, which is why we are finally doing it for
> them.
>
> >
> > This sounds like a need to introduce a new R-like rbind() operator. This
> way you could fix up row cardinality like:
> >
> > drmAnew = drmA rbind drmParallelizeEmpty(extra_rows, drmA.ncol)
> >
>
> true, add an empty slice.
>
> > You could already do this, though twisted::
> >
> > drmAnew = (drmA.t cbind drmParallelizeEmpty(drmA.ncol, extra_rows).t
> >
> >
>
> yes and I can dig deeper and do another drmWrap constructing a new larger
> matrix.
>
> Still changing the number of rows on Sparse matrices is so much simpler
> but I think "drm.nrow = “ may hide the special nature of what we are doing.
>
>
> So finally how are you changing the cardinality? I just want to make h2o
> engine "works" with that technique.
>
>
>
>
>
Re: Problem of dimensions
Posted by Pat Ferrel <pa...@occamsmachete.com>.
Now you have me doubting. At least for cases I haven’t tested.
Dmitriy: on a sparse drm (CheckpointedDrmSpark) will increasing _nrow do the equivalent of adding empty row vectors, with keys but no vector elements?
On Jul 14, 2014, at 7:30 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:
It’ll be in a PR for review before it goes in and if you have already merged it will be stubbed out in the h2o implementation if I can figure out where. The tests are all in the Spark specific module. All I did was change _nrow in CheckpointedDrmSpark, which is a private var. I added a method to do that but not in the core-math only on Spark specific code.
cooccurrence works at a lower level and does not need to worry about this because drm multiply and other ops just work. The place I change nrow is in ItemSimilarity, which is non-core.
I would indeed make sure your sparse matrix code handles this but not sure where you’d test that—bug Dmitriy.
On Jul 14, 2014, at 5:09 PM, Anand Avati <av...@gluster.org> wrote:
On Mon, Jul 14, 2014 at 3:02 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:
>
> On Jul 14, 2014, at 12:10 PM, Anand Avati <av...@gluster.org> wrote:
>
> On Mon, Jul 14, 2014 at 11:56 AM, Pat Ferrel <pa...@occamsmachete.com> wrote:
> In the application, the number of rows will always be increased, adding blank rows. I don’t think shuffle is necessary in this case because there is no actual row, no data in the drm it’s just needed to make the cardinality match, the IDs will take care of data matching . Maybe calling it something else is a good idea to emphasize the special case for it’s use. I went over this with Dmitriy and, though I haven’t checked actual values on large datasets, it works.
>
>
> Does that mean the cardinality is faked at the logical layer with no changes at the engine level? Does that means the physical operators need to be prepared to handle non-matching matrix multiplication by assuming the missing rows or columns are 0's? Does that really work with no changes?
yes, Dmitriy recently confirmed this. But not faked, it is just not possible to calculate it from data in some cases since “does not exist” may mean “= 0”.
I’m no R expert but base R seems to assume a 0 value actually exists in the Matrix encoded in as much space as it’s type dictates, like Dense things in Mahout. I think there are R packages that add support for sparse things (slam?) and so assume this is one place where some rethinking is required: http://www.johnmyleswhite.com/notebook/2011/10/31/using-sparse-matrices-in-r/
Sparse linear algebra with the added complication of foreign IDs makes for some odd cases. The number of extrenal/foreign IDs for rows and columns defines the true cardinality even though in a sparse matrix the empty row or column is just absent. In Mahout the IDs are row and column numbers so there are cases where the real-world cardinality does not match the number of Mahout IDs or DRM cardinality and the calculations should be fine with that as long as the real-world dimensions are supplied for cardinality checking and various values calculated from cardinality.
I’m maintaining a mapping of external ID to/from Mahout ID. For instance in item similarity it’s use is where the rows have a key = user ID. For a single application the row space is defined by all user IDs. In the cross similarity A’B reading in A may not ID every user so if B find more or different ones then the union of the two is our best guess at the total. And in fact more matrices could be added and the user ID space is all users seen in all of the data. If we knew how many users are defined in the application we could also use that but it’s not needed of there is no data at all for some users.
BTW this mapping seems to be one of the biggest generator of questions on the lists. The above issue is one that would likeli further trip up users generating their own ID mapping, which is why we are finally doing it for them.
>
> This sounds like a need to introduce a new R-like rbind() operator. This way you could fix up row cardinality like:
>
> drmAnew = drmA rbind drmParallelizeEmpty(extra_rows, drmA.ncol)
>
true, add an empty slice.
> You could already do this, though twisted::
>
> drmAnew = (drmA.t cbind drmParallelizeEmpty(drmA.ncol, extra_rows).t
>
>
yes and I can dig deeper and do another drmWrap constructing a new larger matrix.
Still changing the number of rows on Sparse matrices is so much simpler but I think "drm.nrow = “ may hide the special nature of what we are doing.
So finally how are you changing the cardinality? I just want to make h2o engine "works" with that technique.
Re: Problem of dimensions
Posted by Pat Ferrel <pa...@occamsmachete.com>.
It’ll be in a PR for review before it goes in and if you have already merged it will be stubbed out in the h2o implementation if I can figure out where. The tests are all in the Spark specific module. All I did was change _nrow in CheckpointedDrmSpark, which is a private var. I added a method to do that but not in the core-math only on Spark specific code.
cooccurrence works at a lower level and does not need to worry about this because drm multiply and other ops just work. The place I change nrow is in ItemSimilarity, which is non-core.
I would indeed make sure your sparse matrix code handles this but not sure where you’d test that—bug Dmitriy.
On Jul 14, 2014, at 5:09 PM, Anand Avati <av...@gluster.org> wrote:
On Mon, Jul 14, 2014 at 3:02 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:
>
> On Jul 14, 2014, at 12:10 PM, Anand Avati <av...@gluster.org> wrote:
>
> On Mon, Jul 14, 2014 at 11:56 AM, Pat Ferrel <pa...@occamsmachete.com> wrote:
> In the application, the number of rows will always be increased, adding blank rows. I don’t think shuffle is necessary in this case because there is no actual row, no data in the drm it’s just needed to make the cardinality match, the IDs will take care of data matching . Maybe calling it something else is a good idea to emphasize the special case for it’s use. I went over this with Dmitriy and, though I haven’t checked actual values on large datasets, it works.
>
>
> Does that mean the cardinality is faked at the logical layer with no changes at the engine level? Does that means the physical operators need to be prepared to handle non-matching matrix multiplication by assuming the missing rows or columns are 0's? Does that really work with no changes?
yes, Dmitriy recently confirmed this. But not faked, it is just not possible to calculate it from data in some cases since “does not exist” may mean “= 0”.
I’m no R expert but base R seems to assume a 0 value actually exists in the Matrix encoded in as much space as it’s type dictates, like Dense things in Mahout. I think there are R packages that add support for sparse things (slam?) and so assume this is one place where some rethinking is required: http://www.johnmyleswhite.com/notebook/2011/10/31/using-sparse-matrices-in-r/
Sparse linear algebra with the added complication of foreign IDs makes for some odd cases. The number of extrenal/foreign IDs for rows and columns defines the true cardinality even though in a sparse matrix the empty row or column is just absent. In Mahout the IDs are row and column numbers so there are cases where the real-world cardinality does not match the number of Mahout IDs or DRM cardinality and the calculations should be fine with that as long as the real-world dimensions are supplied for cardinality checking and various values calculated from cardinality.
I’m maintaining a mapping of external ID to/from Mahout ID. For instance in item similarity it’s use is where the rows have a key = user ID. For a single application the row space is defined by all user IDs. In the cross similarity A’B reading in A may not ID every user so if B find more or different ones then the union of the two is our best guess at the total. And in fact more matrices could be added and the user ID space is all users seen in all of the data. If we knew how many users are defined in the application we could also use that but it’s not needed of there is no data at all for some users.
BTW this mapping seems to be one of the biggest generator of questions on the lists. The above issue is one that would likeli further trip up users generating their own ID mapping, which is why we are finally doing it for them.
>
> This sounds like a need to introduce a new R-like rbind() operator. This way you could fix up row cardinality like:
>
> drmAnew = drmA rbind drmParallelizeEmpty(extra_rows, drmA.ncol)
>
true, add an empty slice.
> You could already do this, though twisted::
>
> drmAnew = (drmA.t cbind drmParallelizeEmpty(drmA.ncol, extra_rows).t
>
>
yes and I can dig deeper and do another drmWrap constructing a new larger matrix.
Still changing the number of rows on Sparse matrices is so much simpler but I think "drm.nrow = “ may hide the special nature of what we are doing.
So finally how are you changing the cardinality? I just want to make h2o engine "works" with that technique.
Re: Problem of dimensions
Posted by Anand Avati <av...@gluster.org>.
On Mon, Jul 14, 2014 at 3:02 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:
>
> On Jul 14, 2014, at 12:10 PM, Anand Avati <av...@gluster.org> wrote:
>
> On Mon, Jul 14, 2014 at 11:56 AM, Pat Ferrel <pa...@occamsmachete.com>
> wrote:
>
>> In the application, the number of rows will always be increased, adding
>> blank rows. I don’t think shuffle is necessary in this case because there
>> is no actual row, no data in the drm it’s just needed to make the
>> cardinality match, the IDs will take care of data matching . Maybe calling
>> it something else is a good idea to emphasize the special case for it’s
>> use. I went over this with Dmitriy and, though I haven’t checked actual
>> values on large datasets, it works.
>>
>
>
> Does that mean the cardinality is faked at the logical layer with no
> changes at the engine level? Does that means the physical operators need to
> be prepared to handle non-matching matrix multiplication by assuming the
> missing rows or columns are 0's? Does that really work with no changes?
>
>
> yes, Dmitriy recently confirmed this. But not faked, it is just not
> possible to calculate it from data in some cases since “does not exist” may
> mean “= 0”.
>
> I’m no R expert but base R seems to assume a 0 value actually exists in
> the Matrix encoded in as much space as it’s type dictates, like Dense
> things in Mahout. I think there are R packages that add support for sparse
> things (slam?) and so assume this is one place where some rethinking is
> required:
> http://www.johnmyleswhite.com/notebook/2011/10/31/using-sparse-matrices-in-r/
>
> Sparse linear algebra with the added complication of foreign IDs makes for
> some odd cases. The number of extrenal/foreign IDs for rows and columns
> defines the true cardinality even though in a sparse matrix the empty row
> or column is just absent. In Mahout the IDs are row and column numbers so
> there are cases where the real-world cardinality does not match the number
> of Mahout IDs or DRM cardinality and the calculations should be fine with
> that as long as the real-world dimensions are supplied for cardinality
> checking and various values calculated from cardinality.
>
> I’m maintaining a mapping of external ID to/from Mahout ID. For instance
> in item similarity it’s use is where the rows have a key = user ID. For a
> single application the row space is defined by all user IDs. In the cross
> similarity A’B reading in A may not ID every user so if B find more or
> different ones then the union of the two is our best guess at the total.
> And in fact more matrices could be added and the user ID space is all users
> seen in all of the data. If we knew how many users are defined in the
> application we could also use that but it’s not needed of there is no data
> at all for some users.
>
> BTW this mapping seems to be one of the biggest generator of questions on
> the lists. The above issue is one that would likeli further trip up users
> generating their own ID mapping, which is why we are finally doing it for
> them.
>
>
> This sounds like a need to introduce a new R-like rbind() operator. This
> way you could fix up row cardinality like:
>
> drmAnew = drmA rbind drmParallelizeEmpty(extra_rows, drmA.ncol)
>
>
> true, add an empty slice.
>
> You could already do this, though twisted::
>
> drmAnew = (drmA.t cbind drmParallelizeEmpty(drmA.ncol, extra_rows).t
>
>
>
> yes and I can dig deeper and do another drmWrap constructing a new larger
> matrix.
>
> Still changing the number of rows on Sparse matrices is so much simpler
> but I think "drm.nrow = “ may hide the special nature of what we are doing.
>
>
So finally how are you changing the cardinality? I just want to make h2o
engine "works" with that technique.
Re: Problem of dimensions
Posted by Pat Ferrel <pa...@occamsmachete.com>.
>
> On Jul 14, 2014, at 12:10 PM, Anand Avati <av...@gluster.org> wrote:
>
> On Mon, Jul 14, 2014 at 11:56 AM, Pat Ferrel <pa...@occamsmachete.com> wrote:
> In the application, the number of rows will always be increased, adding blank rows. I don’t think shuffle is necessary in this case because there is no actual row, no data in the drm it’s just needed to make the cardinality match, the IDs will take care of data matching . Maybe calling it something else is a good idea to emphasize the special case for it’s use. I went over this with Dmitriy and, though I haven’t checked actual values on large datasets, it works.
>
>
> Does that mean the cardinality is faked at the logical layer with no changes at the engine level? Does that means the physical operators need to be prepared to handle non-matching matrix multiplication by assuming the missing rows or columns are 0's? Does that really work with no changes?
yes, Dmitriy recently confirmed this. But not faked, it is just not possible to calculate it from data in some cases since “does not exist” may mean “= 0”.
I’m no R expert but base R seems to assume a 0 value actually exists in the Matrix encoded in as much space as it’s type dictates, like Dense things in Mahout. I think there are R packages that add support for sparse things (slam?) and so assume this is one place where some rethinking is required: http://www.johnmyleswhite.com/notebook/2011/10/31/using-sparse-matrices-in-r/
Sparse linear algebra with the added complication of foreign IDs makes for some odd cases. The number of extrenal/foreign IDs for rows and columns defines the true cardinality even though in a sparse matrix the empty row or column is just absent. In Mahout the IDs are row and column numbers so there are cases where the real-world cardinality does not match the number of Mahout IDs or DRM cardinality and the calculations should be fine with that as long as the real-world dimensions are supplied for cardinality checking and various values calculated from cardinality.
I’m maintaining a mapping of external ID to/from Mahout ID. For instance in item similarity it’s use is where the rows have a key = user ID. For a single application the row space is defined by all user IDs. In the cross similarity A’B reading in A may not ID every user so if B find more or different ones then the union of the two is our best guess at the total. And in fact more matrices could be added and the user ID space is all users seen in all of the data. If we knew how many users are defined in the application we could also use that but it’s not needed of there is no data at all for some users.
BTW this mapping seems to be one of the biggest generator of questions on the lists. The above issue is one that would likeli further trip up users generating their own ID mapping, which is why we are finally doing it for them.
>
> This sounds like a need to introduce a new R-like rbind() operator. This way you could fix up row cardinality like:
>
> drmAnew = drmA rbind drmParallelizeEmpty(extra_rows, drmA.ncol)
>
true, add an empty slice.
> You could already do this, though twisted::
>
> drmAnew = (drmA.t cbind drmParallelizeEmpty(drmA.ncol, extra_rows).t
>
>
yes and I can dig deeper and do another drmWrap constructing a new larger matrix.
Still changing the number of rows on Sparse matrices is so much simpler but I think "drm.nrow = “ may hide the special nature of what we are doing.
Re: Problem of dimensions
Posted by Anand Avati <av...@gluster.org>.
On Mon, Jul 14, 2014 at 11:56 AM, Pat Ferrel <pa...@occamsmachete.com> wrote:
> In the application, the number of rows will always be increased, adding
> blank rows. I don’t think shuffle is necessary in this case because there
> is no actual row, no data in the drm it’s just needed to make the
> cardinality match, the IDs will take care of data matching . Maybe calling
> it something else is a good idea to emphasize the special case for it’s
> use. I went over this with Dmitriy and, though I haven’t checked actual
> values on large datasets, it works.
>
Does that mean the cardinality is faked at the logical layer with no
changes at the engine level? Does that means the physical operators need to
be prepared to handle non-matching matrix multiplication by assuming the
missing rows or columns are 0's? Does that really work with no changes?
This sounds like a need to introduce a new R-like rbind() operator. This
way you could fix up row cardinality like:
drmAnew = drmA rbind drmParallelizeEmpty(extra_rows, drmA.ncol)
You could already do this, though twisted::
drmAnew = (drmA.t cbind drmParallelizeEmpty(drmA.ncol, extra_rows).t
Re: Problem of dimensions
Posted by Pat Ferrel <pa...@occamsmachete.com>.
In the application, the number of rows will always be increased, adding blank rows. I don’t think shuffle is necessary in this case because there is no actual row, no data in the drm it’s just needed to make the cardinality match, the IDs will take care of data matching . Maybe calling it something else is a good idea to emphasize the special case for it’s use. I went over this with Dmitriy and, though I haven’t checked actual values on large datasets, it works.
On Jul 14, 2014, at 11:04 AM, Anand Avati <av...@gluster.org> wrote:
On Mon, Jul 14, 2014 at 10:58 AM, Ted Dunning <te...@gmail.com> wrote:
On Mon, Jul 14, 2014 at 9:47 AM, Pat Ferrel <pa...@occamsmachete.com> wrote:
> BTW that requires that drm.nrow be mutable. That is defined as immutable
> in the DSL and so will require a change to several traits. I’ve done this
> but am still trying to decide the cleanest.
Hmmm.... immutability has lots of virtues. And changing nrows is just the
tip of the iceberg. You also have to shuffle the rows to match the row
partitioning between the two matrices.
Or it requires more than one pass through the data. Since you have to read
both matrices before you can deal with either, and since one matrix is
likely to be shuffled relative to the other, might it just be better to
either do two read passes or pay the cost to shuffle the matrices after
getting a consensus view. Note that the second read pass will have to do a
shuffle any way so the only savings to doing two passes is to decrease
memory usage.
*Anand,*
I think I remember you were addressing a shuffle problem in some of your
earlier work. What did you conclude?
I think the larger question is, what does it mean to make drm.nrow mutable. If changed to a smaller value, which rows do you "sacrifice". Why not just do a RowRange operation to get a new DRM with fewer rows (instead of mutating the given drm)? After that, if you care specifically about partitioning the Par operator can shuffle data for you.
Re: Problem of dimensions
Posted by Anand Avati <av...@gluster.org>.
On Mon, Jul 14, 2014 at 10:58 AM, Ted Dunning <te...@gmail.com> wrote:
> On Mon, Jul 14, 2014 at 9:47 AM, Pat Ferrel <pa...@occamsmachete.com> wrote:
>
> > BTW that requires that drm.nrow be mutable. That is defined as immutable
> > in the DSL and so will require a change to several traits. I’ve done this
> > but am still trying to decide the cleanest.
>
>
> Hmmm.... immutability has lots of virtues. And changing nrows is just the
> tip of the iceberg. You also have to shuffle the rows to match the row
> partitioning between the two matrices.
>
> Or it requires more than one pass through the data. Since you have to read
> both matrices before you can deal with either, and since one matrix is
> likely to be shuffled relative to the other, might it just be better to
> either do two read passes or pay the cost to shuffle the matrices after
> getting a consensus view. Note that the second read pass will have to do a
> shuffle any way so the only savings to doing two passes is to decrease
> memory usage.
>
> *Anand,*
>
> I think I remember you were addressing a shuffle problem in some of your
> earlier work. What did you conclude?
>
I think the larger question is, what does it mean to make drm.nrow mutable.
If changed to a smaller value, which rows do you "sacrifice". Why not just
do a RowRange operation to get a new DRM with fewer rows (instead of
mutating the given drm)? After that, if you care specifically about
partitioning the Par operator can shuffle data for you.
Re: Problem of dimensions
Posted by Ted Dunning <te...@gmail.com>.
On Mon, Jul 14, 2014 at 9:47 AM, Pat Ferrel <pa...@occamsmachete.com> wrote:
> BTW that requires that drm.nrow be mutable. That is defined as immutable
> in the DSL and so will require a change to several traits. I’ve done this
> but am still trying to decide the cleanest.
Hmmm.... immutability has lots of virtues. And changing nrows is just the
tip of the iceberg. You also have to shuffle the rows to match the row
partitioning between the two matrices.
Or it requires more than one pass through the data. Since you have to read
both matrices before you can deal with either, and since one matrix is
likely to be shuffled relative to the other, might it just be better to
either do two read passes or pay the cost to shuffle the matrices after
getting a consensus view. Note that the second read pass will have to do a
shuffle any way so the only savings to doing two passes is to decrease
memory usage.
*Anand,*
I think I remember you were addressing a shuffle problem in some of your
earlier work. What did you conclude?
Re: Problem of dimensions
Posted by Pat Ferrel <pa...@occamsmachete.com>.
right, I’m past that and did what you describe. BTW that requires that drm.nrow be mutable. That is defined as immutable in the DSL and so will require a change to several traits. I’ve done this but am still trying to decide the cleanest.
The current issue is that the cooc takes a primary matrix (drmA) and list of secondary ones (drmBs). It was doing
drmB.t %*% drmA
but it should be
drmA.t %*% drmB
This seems obvious but wasn’t caught until I created a drmB with completely different column ID space. I’ve also fixed this but need to rewrite a bunch of tests. This is just FYI in case I missed something.
On Jul 14, 2014, at 9:28 AM, Ted Dunning <te...@gmail.com> wrote:
On Tue, Jul 8, 2014 at 10:29 AM, Pat Ferrel <pa...@occamsmachete.com> wrote:
To do this correctly for all cases the row IDs for all unique row keys would have to be created across all drms for cooccurrence. This implies using a single Map for the row space of all drms read in with a single incrementing integer for the DRM row key. The length of this Map would be the row dimension for all DRMs. After the row dimension is calculated the Map could be thrown away since only columns (input items) need to have application specific IDs applied at output.
The row id's have to be identical for both matrices in any case. You really do need to convert to an identical set of id's and have the same size.
(Stated more simply ... this exception is actually correct. This won't work unless the matrices have the same rows)
Re: Problem of dimensions
Posted by Ted Dunning <te...@gmail.com>.
On Tue, Jul 8, 2014 at 10:29 AM, Pat Ferrel <pa...@occamsmachete.com> wrote:
> To do this correctly for all cases the row IDs for all unique row keys
> would have to be created across all drms for cooccurrence. This implies
> using a single Map for the row space of all drms read in with a single
> incrementing integer for the DRM row key. The length of this Map would be
> the row dimension for all DRMs. After the row dimension is calculated the
> Map could be thrown away since only columns (input items) need to have
> application specific IDs applied at output.
The row id's have to be identical for both matrices in any case. You
really do need to convert to an identical set of id's and have the same
size.
(Stated more simply ... this exception is actually correct. This won't
work unless the matrices have the same rows)
Re: Problem of dimensions
Posted by Pat Ferrel <pa...@occamsmachete.com>.
Got it, that is what I hoped was happening. I agree it’s the prep logic that is the only place that dimensionality can be known. So I think I know how to handle this case unless mr. Schelter has further thoughts.
On Jul 8, 2014, at 10:49 AM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
Empty row still counts as one row.
_all_ rows (or, depending on current backing orientation, columns) are expected to be of A.ncol (A.nrow) size. Algebraic algorithms don't additionally validate that, if it is not true, they may fail. These are algebraic guarantees imposed on to matrices.
Optimizer validates, kind of slack-y, these guarantees, which is what you encounter here. It may even miss them, but it doesn't mean that guarantees are not necessary. it only means that some algorithms may fail with more wacky, meaningless messages such as OOB. It is, generally, responsibility of feature prep logic to ensure algebraic guarantees. Optimizer can catch something that can be done inexpensively, or it can even miss it, but it can never repair it.
for further details you need to bother mr. Schelter.
-d
On Tue, Jul 8, 2014 at 10:29 AM, Pat Ferrel <pa...@occamsmachete.com> wrote:
In cooccurrence for the case of B'A the real-world dimensionality of the matrices can be compatible even though the data read in from tuples would leave some rows or columns blank—no non-zero elements. At least this is what I suspect trying to run cooccurrence on the epinions data (ratings_data.txt, trust_data.txt)
I get:
Exception in thread "main" java.lang.AssertionError: assertion failed: Incompatible operand geometry
at scala.Predef$.assert(Predef.scala:179)
at org.apache.mahout.math.drm.logical.OpAB.<init>(OpAB.scala:29)
...
from
val drmBtA = drmB.t %*% drmA
Regardless of what is causing this problem there _will_ be cases where the auto-calculated dimensions of A and B (calculated from the non-blank rows when the DRM is read in from a text file) are not compatible but the data actually is. This is the case where the union of all userIDs is greater than the number of user IDs in one or both of the DRMs.
To do this correctly for all cases the row IDs for all unique row keys would have to be created across all drms for cooccurrence. This implies using a single Map for the row space of all drms read in with a single incrementing integer for the DRM row key. The length of this Map would be the row dimension for all DRMs. After the row dimension is calculated the Map could be thrown away since only columns (input items) need to have application specific IDs applied at output.
Does this sound like the right way to handle this case? Will the drmB.t %*% drmA do the right thing for non-existent rows/columns, which I think means to treat a non-existent vector as if it were all 0s. I believe this worked in the hadoop version.