You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by "Jeff Eastman (JIRA)" <ji...@apache.org> on 2008/07/06 20:54:40 UTC
[jira] Created: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Add Element Labels to Vectors and Matrices
------------------------------------------
Key: MAHOUT-65
URL: https://issues.apache.org/jira/browse/MAHOUT-65
Project: Mahout
Issue Type: New Feature
Components: Matrix
Reporter: Jeff Eastman
Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and Matrices
Posted by Ted Dunning <te...@gmail.com>.
I would love to hear from Karl. Whatever we choose to do has implications
on what kinds of attributes are reasonable to serialize. He was the driving
force behind having something fancier than simple strings as attribute
values.
On Mon, Oct 20, 2008 at 10:03 PM, Jeff Eastman
<jd...@windwardsolutions.com>wrote:
> Ted Dunning wrote:
>
>> I see what you mean.
>>
>> To repeat in other words, the problems that need to be solved are:
>>
>> a) there are many uses already so adding attributes should be transparent
>> to
>> those who don't use them
>>
>> b) the encoding should not be ad hoc because this would be our second ad
>> hoc
>> encoding and only one should ever be allowed before using a standard
>>
>>
> +1
>
>> So here is a (kind of) concrete proposal:
>>
>> a) use JSON or Thrift for concrete syntax
>>
>>
> Any preferences here? This might also impact other Mahout packages in the
> future, so everybody please weigh in. In general, it seems that having a
> common, public encoding for matrix and vector data would help users mix and
> match the Mahout services. What are the requirements of these other
> services? From inspection, it looks like only the clustering packages use
> them currently.
>
> Jeff
>
--
ted
[math] Fwd: [jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and Matrices
Posted by Ted Dunning <te...@gmail.com>.
Luc and other commons math folk:
Do you guys have opinions about serialization formats for matrices (both
dense and sparse, both with and without row, column and cell attributes)?
---------- Forwarded message ----------
From: Jeff Eastman <jd...@windwardsolutions.com>
Date: Mon, Oct 20, 2008 at 10:03 PM
Subject: Re: [jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
To: mahout-dev@lucene.apache.org
Ted Dunning wrote:
> I see what you mean.
>
> To repeat in other words, the problems that need to be solved are:
>
> a) there are many uses already so adding attributes should be transparent
> to
> those who don't use them
>
> b) the encoding should not be ad hoc because this would be our second ad
> hoc
> encoding and only one should ever be allowed before using a standard
>
>
+1
> So here is a (kind of) concrete proposal:
>
> a) use JSON or Thrift for concrete syntax
>
>
Any preferences here? This might also impact other Mahout packages in the
future, so everybody please weigh in. In general, it seems that having a
common, public encoding for matrix and vector data would help users mix and
match the Mahout services. What are the requirements of these other
services? From inspection, it looks like only the clustering packages use
them currently.
Jeff
--
ted
Re: [jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and Matrices
Posted by Grant Ingersoll <gs...@apache.org>.
On Oct 21, 2008, at 8:18 AM, Sean Owen wrote:
> I don't think JSON is quite appropriate, er, optimal. We're not
> encoding data to be sent to an AJAX app. XML is a thought, but pretty
> verbose.
-1 on XML, although it is possibly reasonable to think about being
able to import/export whatever we choose to various other things.
>
>
> Thrift is good. Protocol Buffers from Google (obviously, this is
> something I know about and prefer) are also an option -- about the
> same thing really. I would use one of the two with a pref for PBs
> myself.
>
Either is fine by me, but I can't say I know enough either way. PBs
is ASL, so it can be incorporated, and Thrift is a incubator project
at Apache.
>
> On Tue, Oct 21, 2008 at 6:03 AM, Jeff Eastman
> <jd...@windwardsolutions.com> wrote:
>> +1
>>>
>>> So here is a (kind of) concrete proposal:
>>>
>>> a) use JSON or Thrift for concrete syntax
>>>
>>
>> Any preferences here? This might also impact other Mahout packages
>> in the
>> future, so everybody please weigh in. In general, it seems that
>> having a
>> common, public encoding for matrix and vector data would help users
>> mix and
>> match the Mahout services. What are the requirements of these other
>> services? From inspection, it looks like only the clustering
>> packages use
>> them currently.
Re: [jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and Matrices
Posted by Sean Owen <sr...@gmail.com>.
I don't think JSON is quite appropriate, er, optimal. We're not
encoding data to be sent to an AJAX app. XML is a thought, but pretty
verbose.
Thrift is good. Protocol Buffers from Google (obviously, this is
something I know about and prefer) are also an option -- about the
same thing really. I would use one of the two with a pref for PBs
myself.
On Tue, Oct 21, 2008 at 6:03 AM, Jeff Eastman
<jd...@windwardsolutions.com> wrote:
> +1
>>
>> So here is a (kind of) concrete proposal:
>>
>> a) use JSON or Thrift for concrete syntax
>>
>
> Any preferences here? This might also impact other Mahout packages in the
> future, so everybody please weigh in. In general, it seems that having a
> common, public encoding for matrix and vector data would help users mix and
> match the Mahout services. What are the requirements of these other
> services? From inspection, it looks like only the clustering packages use
> them currently.
Re: [jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by Jeff Eastman <jd...@windwardsolutions.com>.
Ted Dunning wrote:
> I see what you mean.
>
> To repeat in other words, the problems that need to be solved are:
>
> a) there are many uses already so adding attributes should be transparent to
> those who don't use them
>
> b) the encoding should not be ad hoc because this would be our second ad hoc
> encoding and only one should ever be allowed before using a standard
>
+1
> So here is a (kind of) concrete proposal:
>
> a) use JSON or Thrift for concrete syntax
>
Any preferences here? This might also impact other Mahout packages in
the future, so everybody please weigh in. In general, it seems that
having a common, public encoding for matrix and vector data would help
users mix and match the Mahout services. What are the requirements of
these other services? From inspection, it looks like only the clustering
packages use them currently.
Jeff
Re: [jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and Matrices
Posted by Ted Dunning <te...@gmail.com>.
I see what you mean.
To repeat in other words, the problems that need to be solved are:
a) there are many uses already so adding attributes should be transparent to
those who don't use them
b) the encoding should not be ad hoc because this would be our second ad hoc
encoding and only one should ever be allowed before using a standard
So here is a (kind of) concrete proposal:
a) use JSON or Thrift for concrete syntax
b) design the encoding so that it can carry matrix/vector type information
(dense/sparse, float/double) as a matrix level attribute
c) have the main array be serialized as a map of row number to (list of cell
data) for dense formats or as map of row number to (list of column numbers,
list of corresponding cell data) for sparse matrices
d) serialize vectors as a list of cell data for dense formats or list of
column numbers, list of cell data for sparse matrices
e) optionally include attribute maps at the matrix, row, column and cell
level (for matrices) or at the vector or cell level (for vectors).
The thing I don't like about this is that it seems nicer if the matrix is a
row map of vectors. Doing that cleanly and still maintaining column level
attributes seems hard to do economically.
Thoughts?
On Sun, Oct 19, 2008 at 8:05 AM, Jeff Eastman <jd...@windwardsolutions.com>wrote:
> .. move them all to a uniform serialization/deserialization approach.
> ... about 70 references to the asFormatString and asWritableComparable that
> need to be checked so this is not a trivial change.
>
> We should discuss the alternative techniques (Java Serializable, XML, JSON,
> ...) and make this change before implementing element labels. That was where
> I ran out of steam, though I seem to have more now :).
>
>
Re: [jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by Jeff Eastman <jd...@windwardsolutions.com>.
Hi Ted,
Sure, consider the following, somewhat simplified implementation. What
the various TODOs indicate is the places where encoding/decoding
decisions need to be made. Some of these are non-trivial, such as in
Attributes where the nameIndex and attributes need to refer to common
values. I got bogged down trying to invent more ad-hoc encodings for
these things; what would be better is to move them all to a uniform
serialization/deserialization approach. There are about 70 references to
the asFormatString and asWritableComparable that need to be checked so
this is not a trivial change.
We should discuss the alternative techniques (Java Serializable, XML,
JSON, ...) and make this change before implementing element labels. That
was where I ran out of steam, though I seem to have more now :).
Jeff
=====
class Attributes {
Map<String, Attribute> nameIndex;
List<Attribute> attributes;
void setName(Attribute, String);
String getName(Attribute);
public String asFormatString() {
// TODO: how to encode?
}
public static Attributes decodeFormat(String format) {
// TODO: reverse the encoding
}
}
class Attribute {
enum Type {ordinal, numeric};
Type type;
List<String> ordinalValues;
public String asFormatString() {
// TODO: how to encode?
}
public static Attribute decodeFormat(String format) {
// TODO: reverse the encoding
}
}
class DenseVector {
private double[] values;
private Attributes attributes;
public Attributes getAttributes() {
if (attributes == null)
attributes = new Attributes();
return attributes;
}
public String asFormatString() {
StringBuilder out = new StringBuilder();
out.append("[, ");
// TODO: how to delimit the Attributes?
out.append(attributes.asFormatString());
for (int i = 0; i < values.length; i++)
out.append(values[i]).append(", ");
out.append("] ");
return out.toString();
}
public static Vector decodeFormat(String formattedString) {
// TODO: how to isolate the attribute substring?
String[] pts = formattedString.split(",");
double[] point = new double[pts.length - 2];
for (int i = 1; i < pts.length - 1; i++)
point[i - 1] = Double.parseDouble(pts[i]);
return new DenseVector(point);
}
Ted Dunning wrote:
> Jeff,
>
> Can you say just a bit more about what richness and what awkwardness that
> you ran into?
>
> On Fri, Oct 17, 2008 at 11:33 AM, Jeff Eastman (JIRA) <ji...@apache.org>wrote:
>
>
>> [
>> https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640616#action_12640616]
>>
>> Jeff Eastman commented on MAHOUT-65:
>> ------------------------------------
>>
>> I did a test implementation of element labeling based upon Karl's
>> suggestion and Ted's use case. It used a lazy instantiation of a label map
>> rather than a wrapper that was similar to my earlier patch. The rub comes
>> when trying to serialize the label map for asFormatString(). Here, the
>> richness of the labeling semantics made serialization/deserialization of its
>> state most awkward.
>>
>>
>>
>
>
Re: [jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and Matrices
Posted by Ted Dunning <te...@gmail.com>.
Jeff,
Can you say just a bit more about what richness and what awkwardness that
you ran into?
On Fri, Oct 17, 2008 at 11:33 AM, Jeff Eastman (JIRA) <ji...@apache.org>wrote:
>
> [
> https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640616#action_12640616]
>
> Jeff Eastman commented on MAHOUT-65:
> ------------------------------------
>
> I did a test implementation of element labeling based upon Karl's
> suggestion and Ted's use case. It used a lazy instantiation of a label map
> rather than a wrapper that was similar to my earlier patch. The rub comes
> when trying to serialize the label map for asFormatString(). Here, the
> richness of the labeling semantics made serialization/deserialization of its
> state most awkward.
>
>
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720885#action_12720885 ]
Grant Ingersoll commented on MAHOUT-65:
---------------------------------------
JSON is fine for In and Out of Mahout, IMO, but for internal use (i.e. between mappers/reducers, etc.) a binary protocol makes more sense. For now, we can likely live with JSON, but as others have stated, Thrift, PB, Avro, et. al. are really designed for this kind of thing
At any rate, we probably should take this discussion to a separate thread, as it has nothing to do with the thread subject.
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Jeff Eastman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720128#action_12720128 ]
Jeff Eastman commented on MAHOUT-65:
------------------------------------
I committed MAHOUT-65c and another patch to make bindings transient. Sorry about the formatting, I use the Lucene formatter in Eclipse and reformat as a reflex. I will also add the set(label, index, value) function you requested and will implement labels in Matrix next. In Matrix, I expect the bindings should not be transient, but be amortized as part of the matrix's state as Ted suggested above.
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Karl Wettin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615292#action_12615292 ]
Karl Wettin commented on MAHOUT-65:
-----------------------------------
bq. That seems like shoe-horning two functions into one, but I see your point that these are both characteristics of the element of the vector.
What about something like this:
{code:java}
class Matrix {
Attributes attributes;
}
class Vector {
transient Matrix matrix;
Attributes attributes;
Attributes getAttributes() {
return attributes != null ? attributes : matrix != null ? matrix.attributes : null;
}
}
class Attributes {
Map<String, Attribute> nameIndex;
List<Attribute> attributes;
void setName(Attribute, String);
String getName(Attribute);
}
class Attribute {
Type type {ordinal, numeric};
List<String> ordinalValues;
}
{code}
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Reporter: Jeff Eastman
> Attachments: MAHOUT-65.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Jeff Eastman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720768#action_12720768 ]
Jeff Eastman commented on MAHOUT-65:
------------------------------------
Perhaps, though the Json representation of Sean's new SparseVector looks pretty dense. Realistic element values would be larger than these test values, of course. A thousand double terms will still add up:
{noformat}
Vec: {"class":"org.apache.mahout.matrix.SparseVector","vector":"{\"values\":{\"indices\":[0,1,2],\"values\":[1.0,2.0,3.0],\"numMappings\":3},\"cardinality\":3}"}
{noformat}
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Jeff Eastman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720732#action_12720732 ]
Jeff Eastman commented on MAHOUT-65:
------------------------------------
This particular operation is just extracting the Vector's serialized String value from the JsonPrimitive. The real heavy lifting is done on line 69:
{noformat}
return (Vector) gson.fromJson(vector, cl);
{noformat}
For really big vectors I'd expect serialization and deserialization to be String-intensive.
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Robert Burrell Donkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720817#action_12720817 ]
Robert Burrell Donkin commented on MAHOUT-65:
---------------------------------------------
The canopy clustering example has been broken by the changes to the internal representation of Vector. The problem is that Canopy, Cluster and the example OutputMapper all rely on string concatenation. Judging by the code, the string parsing work looks inefficient and has proved fragile. (Personally speaking, I also find it hard to understand the code when the wire format and object designs are quite different.)
IMHO adopting a binary serialization system which could be used for both Vector and other types would make the code more robust in this area
I agree with Ted that either thrift or avro would be a good choice
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Jeff Eastman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615869#action_12615869 ]
Jeff Eastman commented on MAHOUT-65:
------------------------------------
Whether or not we decide to split this issue or just rename it, the semantics of "associate arbitrary metadata with" are a significant generalization of the original "use labels to access". As a minimalist, my only concern is that we have real, testable user stories that support such a generalization else we risk premature abstraction complications down the road. In particular, do we have a concrete ML algorithm under development for Mahout that requires this capability or are we just imagining that we might encounter one down the road? If we do have such user stories then it would be instructive to describe them here. I'm not sure that the "use labels to access" feature even passes this test, though I think the bar is lower. Finally, I think a wrapper implementation would be just fine.
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Reporter: Jeff Eastman
> Attachments: MAHOUT-65.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Jeff Eastman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Eastman updated MAHOUT-65:
-------------------------------
Assignee: Jeff Eastman
Affects Version/s: 0.1
Status: Patch Available (was: Open)
Here's a patch that re-implements the previous simple scheme for supporting Vector accessing via a user-settable Map of index bindings. In this patch, the map itself is not serialized with the vector. In preparation for changing the serialization to use Json I moved the JsonVectorAdapter from Dirichlet (where it was not actually used).
This implementation could be generalized by making the map hold a richer record containing other properties of the index.
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65.patch, MAHOUT-65b.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Benson Margulies (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720881#action_12720881 ]
Benson Margulies commented on MAHOUT-65:
----------------------------------------
XML FastInfoSet?
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Ted Dunning (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12613262#action_12613262 ]
Ted Dunning commented on MAHOUT-65:
-----------------------------------
It is a fair preference to prefer that the labels be loosely bound to vectors, but I would prefer otherwise. Obviously, either of us could live with the opposing view.
The idea that the label should be more than a string is interesting. I am not sure if it should have such an elaborate structure, as opposed to something based on inheritance more like this:
{code:java}
interface Label {
String getName()
}
class StringLabel extends Label {
String name;
String getName() {return name}
}
class OrdinalLabel extends Label { ... }
{code}
and so on. Note that I only use interface here for clarity of exposition. I know your antipathy to interfaces and preference for abstract classes.
Also, what do you mean by a label with type "real"?
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Reporter: Jeff Eastman
> Attachments: MAHOUT-65.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720218#action_12720218 ]
Grant Ingersoll commented on MAHOUT-65:
---------------------------------------
Yep, I have a patch for that, will upload shortly, just writing the tests.
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Grant Ingersoll updated MAHOUT-65:
----------------------------------
Attachment: MAHOUT-65-name.patch
implement hashCode better, require equals and hashcode as part of the interface, same as java.util.List
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Jeff Eastman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720871#action_12720871 ]
Jeff Eastman commented on MAHOUT-65:
------------------------------------
I implemented a little test using a TestVector that inherits from AbstractVector and has Sean's new SparseVector instvars. It also implements Serializable. The results are a bit surprising:
{noformat}
public void testThree() {
int[] indexes = { 1, 2, 3, 4, 5, 6, 7, 8, 9 };
double[] values = { 1.111, 2.222, 3.333, 4.444, 5.555, 6.666, 7.777, 8.888,
9.999 };
Vector obj = new TestVector(indexes, values);
Type vectorType = new TypeToken<Vector>() {
}.getType();
GsonBuilder builder = new GsonBuilder();
builder.registerTypeAdapter(vectorType, new JsonVectorAdapter());
Gson gson = builder.create();
String json = gson.toJson(obj, vectorType);
System.out.println("3 " + json.length() + " = " + json);
}
public void testFour() throws IOException {
int[] indexes = { 1, 2, 3, 4, 5, 6, 7, 8, 9 };
double[] values = { 1.111, 2.222, 3.333, 4.444, 5.555, 6.666, 7.777, 8.888,
9.999 };
TestVector obj = new TestVector(indexes, values);
OutputStream stm = new ByteArrayOutputStream();
ObjectOutputStream out = new ObjectOutputStream(stm);
out.writeObject(obj);
String json = stm.toString();
System.out.println("4 " + json.length() + " = " + json);
}
{noformat}
It produced the following results:
{noformat}
3 159 = {"class":"org.apache.mahout.matrix.TestVector","vector":"{\"indexes\":[1,2,3,4,5,6,7,8,9],\"values\":[1.111,2.222,3.333,4.444,5.555,6.666,7.777,8.888,9.999]}"}
4 239 = ¨Ì
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Jeff Eastman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720886#action_12720886 ]
Jeff Eastman commented on MAHOUT-65:
------------------------------------
I changed the code to substitute 10 random double values and it tipped the scales in favor of #4, but not by much:
{noformat}
3 300 = {"class":"org.apache.mahout.matrix.TestVector","vector":"{\"indexes\":[1,2,3,4,5,6,7,8,9],\"values <snip>
4 247 = <snip>
{noformat}
Then I upped the number to 1000 entries to correspond with our text term vectors. Almost a 2:1 advantage but no readability:
{noformat}
3 23283
4 12131
{noformat}
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720302#action_12720302 ]
Grant Ingersoll commented on MAHOUT-65:
---------------------------------------
Jeff,
One comment on the GSON serialization stuff. It can get pretty verbose storing the class name repeatedly, although I do realize it's a drop in the bucket compared to the vector itself. Perhaps we could do like Solr does and, if some abbreviated form is present where a class name is required (maybe 'DV' or 'SV') it could know to use those forms, otherwise it can do the full class lookup. Might just save a little bit on size of a serialized file, which I imagine can add up.
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65-name.patch, MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Jeff Eastman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640616#action_12640616 ]
Jeff Eastman commented on MAHOUT-65:
------------------------------------
I did a test implementation of element labeling based upon Karl's suggestion and Ted's use case. It used a lazy instantiation of a label map rather than a wrapper that was similar to my earlier patch. The rub comes when trying to serialize the label map for asFormatString(). Here, the richness of the labeling semantics made serialization/deserialization of its state most awkward.
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Reporter: Jeff Eastman
> Attachments: MAHOUT-65.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720671#action_12720671 ]
Grant Ingersoll commented on MAHOUT-65:
---------------------------------------
So, with Sean's patch to MAHOUT-121, it seems the bottleneck is now in deserialization. In looking at it, it seems like there is a whole lot of String stuff going on. In particular, line 61:
{code}
String vector = obj.get(VECTOR).getAsString();
{code}
seems a bit worrisome. I haven't looked into GSON yet, but is there any advice on making it faster?
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720037#action_12720037 ]
Grant Ingersoll commented on MAHOUT-65:
---------------------------------------
Hey Jeff,
Minor request, it seems like you have some sort of reformatting going on that causes the patch to contain all kinds of formatting changes that make it a lot harder to see the actual changes.
In thinking about this a little bit more, is there a way to just name a vector and a row in a Matrix. All I really want right now is to be able to track which Vector is associated with which document, and I could do this by setting a unique name on the Vector and having that serialized. The name itself could be stored in the first entry (for SparseVector, it would have to coincide with the sCardinality stuff.
I'm fine with all the other label stuff, too.
Also, the patch doesn't apply because of the JSONVectorAdapter
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720850#action_12720850 ]
Sean Owen commented on MAHOUT-65:
---------------------------------
I am neutral on JSON, if only because its headline feature is not one we care about, and that is being especially nice to parse in Javascript -- since it represents the data as actual javascript code. (Right?) As such, it happens to be a bit more compact that XML, as these self-describing text formats go.
Is the goal to take as input, and emit as output, human-readable output? if so, JSON seems OK -- I am left wishing there were a format more readily designed for this purpose. XML is of course an option but it is even more verbose... but standard...
Custom binary formats seem unfriendly, as does even Java serialized objects (for *external* interaction, that is). So we have stuff like PBs and Thrift. I have a bias towards PBs only because I have used them; I imagine it's about the same as Thrift. You can relatively easily crank out a human readable version of a PB with a tool -- at least, it worked just fine for us at Google in practice.
Anyway, all that said, not really against JSON. There isn't an ideal solution I know of, and it's not obviously worse than alternatives. Kinda depends on our priorities.
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720838#action_12720838 ]
Grant Ingersoll commented on MAHOUT-65:
---------------------------------------
bq. The canopy clustering example has been broken by the changes to the internal representation of Vector. The problem is that Canopy, Cluster and the example OutputMapper all rely on string concatenation. Judging by the code, the string parsing work looks inefficient and has proved fragile. (Personally speaking, I also find it hard to understand the code when the wire format and object designs are quite different.)
Sounds like we need some better tests that prevent this from happening, too. FWIW,, I like the JSON for external representation, but I admit you guys know more about this than me.
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Ted Dunning (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12613352#action_12613352 ]
Ted Dunning commented on MAHOUT-65:
-----------------------------------
Ahhh...
So this would be the type of the element of the vector. That seems like shoe-horning two functions into one, but I see your point that these are both characteristics of the element of the vector.
And I do think that labels on matrix rows should be inherited from the column labels of the matrix.
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Reporter: Jeff Eastman
> Attachments: MAHOUT-65.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (MAHOUT-65) Add Element Labels to
Vectors and Matrices
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720199#action_12720199 ]
Grant Ingersoll edited comment on MAHOUT-65 at 6/16/09 9:24 AM:
----------------------------------------------------------------
That works for Matrix. For Vector, I was thinking, probably naively, we simply need to be able to add a name attribute. For MAHOUT-126, I just dumped the column/cell labels out separately. Like you said, I'm not sure we want all of that serialized.
was (Author: gsingers):
That works for Matrix. For Vector, I was thinking, probably naively, we simply need to be able to add a name attribute. For MAHOUT-130, I just dumped the column/cell labels out separately. Like you said, I'm not sure we want all of that serialized.
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Jeff Eastman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Eastman updated MAHOUT-65:
-------------------------------
Attachment: MAHOUT-65.patch
This patch introduces four new operations on Vectors:
1. setLabelBindings(Map<String, Integer> bindings) - sets a map of bindings between string labels and vector index values
2. getLabelBindings() - returns the map, creating an empty one if needed
3. set(String label, double value) throws UnboundLabelException, IndexException - sets the labeled element of the vector to the given value
4. get(String label) throws UnboundLabelException, IndexException - returns the value indexed by the given label
This patch required reworking of the serializing/deserializing code so that the maps are included in the formatted value if present. Added some new unit tests to verify it all works. All tests still run.
This is about as minimal a change as I could imagine, and I chose to just expose the map rather than try to encapsulate all the operations that clients might want to perform on it. Is this the kind of functionality that people will find useful?
Jeff
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Reporter: Jeff Eastman
> Attachments: MAHOUT-65.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Karl Wettin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615636#action_12615636 ]
Karl Wettin commented on MAHOUT-65:
-----------------------------------
bq. could you please state the problem you are trying to solve
The way I look at it attribute names (labels), types and meta data is the part of the same thing. I predict the future will contain more meta data in class Attribute and Attributes, but right now I'm only concerned about attribute names and ordinal value codec. Typically this would be interesting at pre processing time and not when invoking some algorithm on the matrix.
bq. and how closely binding these Attributes to every Vector/Matrix instance helps us to solve it?
I honestly don't see how my dry code is more closly bound to Vector/Matrix than the Map in the patch?
bq. But it really should be a separate JIRA, possibly as a follow-on to this one.
I really don't agree. If anything I think we should rename this issue to something like "Attribute meta data". You see, I'm doing my best here to come up with an alternative to the patch as I don't like it. It is to static and has no room for more meta data. I mean, it's just a map! Should we also add a map in the vector for ordinal values, one for types, et c? I really don't think that is a good solution. I want meta data be gathered together in some aggregated layer because I think such data belongs together. (I still think it should be completely separated from the Vector and Matrix.)
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Reporter: Jeff Eastman
> Attachments: MAHOUT-65.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Grant Ingersoll updated MAHOUT-65:
----------------------------------
Attachment: MAHOUT-65-name.patch
How about a version where the tests actually pass? Will commit shortly.
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Jeff Eastman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719181#action_12719181 ]
Jeff Eastman commented on MAHOUT-65:
------------------------------------
Will do,
Jeff
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Reporter: Jeff Eastman
> Attachments: MAHOUT-65.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "M. Arshad Khan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727378#action_12727378 ]
M. Arshad Khan commented on MAHOUT-65:
--------------------------------------
Hi All
Looking at the implementation of clone method in SparseVector, it seems that the "name" field is not being copied to the new instance. As a result, for example, when the the "idField" parameter is passed to org.apache.mahout.util.vectors.Driver, the output does not contain any id field. A fix as follows solves the problem:
Index: core/src/main/java/org/apache/mahout/matrix/SparseVector.java
===================================================================
--- core/src/main/java/org/apache/mahout/matrix/SparseVector.java (revision 791354)
+++ core/src/main/java/org/apache/mahout/matrix/SparseVector.java (working copy)
@@ -73,6 +73,7 @@
public SparseVector clone() {
SparseVector result = like();
result.values = (OrderedIntDoubleMapping) values.clone();
+ result.setName(this.name);
return result;
}
Not sure if other classes like SparseMatrix also require a similar fix. Comments?
Regards
Arshad
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720199#action_12720199 ]
Grant Ingersoll commented on MAHOUT-65:
---------------------------------------
That works for Matrix. For Vector, I was thinking, probably naively, we simply need to be able to add a name attribute. For MAHOUT-130, I just dumped the column/cell labels out separately. Like you said, I'm not sure we want all of that serialized.
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Ted Dunning (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720883#action_12720883 ]
Ted Dunning commented on MAHOUT-65:
-----------------------------------
{noformat}
public void testFour() throws IOException {
int[] indexes = { 1, 2, 3, 4, 5, 6, 7, 8, 9 };
double[] values = { 1.111, 2.222, 3.333, 4.444, 5.555, 6.666, 7.777, 8.888,
9.999 };
TestVector obj = new TestVector(indexes, values);
OutputStream stm = new ByteArrayOutputStream();
ObjectOutputStream out = new ObjectOutputStream(stm);
out.writeObject(obj);
String json = stm.toString();
System.out.println("4 " + json.length() + " = " + json);
}
{noformat}
This looks like you are serializing this object to bytes and then converting those bytes to characters and printing them. The name of the string is json, but I don't think the contents are anything like that.
What did you find surprising here?
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Jeff Eastman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Eastman updated MAHOUT-65:
-------------------------------
Attachment: MAHOUT-65d.patch
Naming a Vector and having that be stateful - as opposed to bindings which are not - sounds useful. It could be used to implement the row and column Matrix label bindings, a SWAG of which is attached.
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Ted Dunning (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615612#action_12615612 ]
Ted Dunning commented on MAHOUT-65:
-----------------------------------
Adding attributes to elements of vectors or columns of a matrix is a reasonable thing to do, especially where you need to provide a decoding rule for ordinal values.
But it really should be a separate JIRA, possibly as a follow-on to this one.
The topic here is naming rows and columns for access by name and (hopefully) joining during multiplication.
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Reporter: Jeff Eastman
> Attachments: MAHOUT-65.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Jeff Eastman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Eastman updated MAHOUT-65:
-------------------------------
Attachment: MAHOUT-65b.patch
Here's the patch.
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65.patch, MAHOUT-65b.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720345#action_12720345 ]
Grant Ingersoll commented on MAHOUT-65:
---------------------------------------
Committed the name stuff:
Committed revision 785386.
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by Jeff Eastman <jd...@windwardsolutions.com>.
QED.
+1
Thanks, Ted, that rocks.
Jeff
Ted Dunning (JIRA) wrote:
> [ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615897#action_12615897 ]
>
> Ted Dunning commented on MAHOUT-65:
> -----------------------------------
>
>
> The user story I have in mind is one I encounter at least once a week.
>
> I have data in a matrix that represents test conditions, clicks and impressions.
>
> I build a model using some or all of the test conditions including or excluding interactions and some simple functions of the test conditions. To do this, the modeling software has to convert my data matrix into a so-called design matrix. Each column that contains categorial data has to be converted to a 1 of n-1 binary encoding, but other values need to be carried along intact. The columns containing the binary encoding need to be named according to the ordinal values that they represent.
>
> The result is a set of rows that represent possible coefficient values. These are, of course, labelled with the design matrix columns.
>
> Sometimes I will get new data. It won't have the result columns and design variables may be shifted around somewhat, but the column labels will be the same. I need to compute predictions for the new data so I have to create a new design matrix and multiply it by the coefficients. This has to be done by matching labels.
>
> This user story requires
>
> - labels
>
> - data type (categorial or continuous)
>
> - the names of the values of the ordinal types
>
> - matrix multiply by label instead of just column.
>
> i.e. all of the capabilities that Karl mentioned and that I included in my list.
>
>
>> Add Element Labels to Vectors and Matrices
>> ------------------------------------------
>>
>> Key: MAHOUT-65
>> URL: https://issues.apache.org/jira/browse/MAHOUT-65
>> Project: Mahout
>> Issue Type: New Feature
>> Components: Matrix
>> Reporter: Jeff Eastman
>> Attachments: MAHOUT-65.patch
>>
>>
>> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
>>
>
>
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Ted Dunning (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615897#action_12615897 ]
Ted Dunning commented on MAHOUT-65:
-----------------------------------
The user story I have in mind is one I encounter at least once a week.
I have data in a matrix that represents test conditions, clicks and impressions.
I build a model using some or all of the test conditions including or excluding interactions and some simple functions of the test conditions. To do this, the modeling software has to convert my data matrix into a so-called design matrix. Each column that contains categorial data has to be converted to a 1 of n-1 binary encoding, but other values need to be carried along intact. The columns containing the binary encoding need to be named according to the ordinal values that they represent.
The result is a set of rows that represent possible coefficient values. These are, of course, labelled with the design matrix columns.
Sometimes I will get new data. It won't have the result columns and design variables may be shifted around somewhat, but the column labels will be the same. I need to compute predictions for the new data so I have to create a new design matrix and multiply it by the coefficients. This has to be done by matching labels.
This user story requires
- labels
- data type (categorial or continuous)
- the names of the values of the ordinal types
- matrix multiply by label instead of just column.
i.e. all of the capabilities that Karl mentioned and that I included in my list.
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Reporter: Jeff Eastman
> Attachments: MAHOUT-65.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (MAHOUT-65) Add Element Labels to
Vectors and Matrices
Posted by "Jeff Eastman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720871#action_12720871 ]
Jeff Eastman edited comment on MAHOUT-65 at 6/17/09 1:38 PM:
-------------------------------------------------------------
I implemented a little test using a TestVector that inherits from AbstractVector and has Sean's new SparseVector instvars. It also implements Serializable. The results are a bit surprising:
{noformat}
public void testThree() {
int[] indexes = { 1, 2, 3, 4, 5, 6, 7, 8, 9 };
double[] values = { 1.111, 2.222, 3.333, 4.444, 5.555, 6.666, 7.777, 8.888,
9.999 };
Vector obj = new TestVector(indexes, values);
Type vectorType = new TypeToken<Vector>() {
}.getType();
GsonBuilder builder = new GsonBuilder();
builder.registerTypeAdapter(vectorType, new JsonVectorAdapter());
Gson gson = builder.create();
String json = gson.toJson(obj, vectorType);
System.out.println("3 " + json.length() + " = " + json);
}
public void testFour() throws IOException {
int[] indexes = { 1, 2, 3, 4, 5, 6, 7, 8, 9 };
double[] values = { 1.111, 2.222, 3.333, 4.444, 5.555, 6.666, 7.777, 8.888,
9.999 };
TestVector obj = new TestVector(indexes, values);
OutputStream stm = new ByteArrayOutputStream();
ObjectOutputStream out = new ObjectOutputStream(stm);
out.writeObject(obj);
String json = stm.toString();
System.out.println("4 " + json.length() + " = " + json);
}
{noformat}
It produced the following results:
{noformat}
3 159 = {"class":"org.apache.mahout.matrix.TestVector","vector":"{\"indexes\":[1,2,3,4,5,6,7,8,9],\"values\":[1.111,2.222,3.333,4.444,5.555,6.666,7.777,8.888,9.999]}"}
4 239 = <a long string of gibberish that the wiki cannot render correctly so I've replaced it with this long string of gibberish>
{noformat}
was (Author: jeastman):
I implemented a little test using a TestVector that inherits from AbstractVector and has Sean's new SparseVector instvars. It also implements Serializable. The results are a bit surprising:
{noformat}
public void testThree() {
int[] indexes = { 1, 2, 3, 4, 5, 6, 7, 8, 9 };
double[] values = { 1.111, 2.222, 3.333, 4.444, 5.555, 6.666, 7.777, 8.888,
9.999 };
Vector obj = new TestVector(indexes, values);
Type vectorType = new TypeToken<Vector>() {
}.getType();
GsonBuilder builder = new GsonBuilder();
builder.registerTypeAdapter(vectorType, new JsonVectorAdapter());
Gson gson = builder.create();
String json = gson.toJson(obj, vectorType);
System.out.println("3 " + json.length() + " = " + json);
}
public void testFour() throws IOException {
int[] indexes = { 1, 2, 3, 4, 5, 6, 7, 8, 9 };
double[] values = { 1.111, 2.222, 3.333, 4.444, 5.555, 6.666, 7.777, 8.888,
9.999 };
TestVector obj = new TestVector(indexes, values);
OutputStream stm = new ByteArrayOutputStream();
ObjectOutputStream out = new ObjectOutputStream(stm);
out.writeObject(obj);
String json = stm.toString();
System.out.println("4 " + json.length() + " = " + json);
}
{noformat}
It produced the following results:
{noformat}
3 159 = {"class":"org.apache.mahout.matrix.TestVector","vector":"{\"indexes\":[1,2,3,4,5,6,7,8,9],\"values\":[1.111,2.222,3.333,4.444,5.555,6.666,7.777,8.888,9.999]}"}
4 239 = ¨Ì
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720161#action_12720161 ]
Grant Ingersoll commented on MAHOUT-65:
---------------------------------------
OK, I will work up a patch for the name thing on a vector, unless you think that can be handled through the bindings thing. Basically, I think we need a way to name a vector and have it carried through.
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Ted Dunning (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719884#action_12719884 ]
Ted Dunning commented on MAHOUT-65:
-----------------------------------
bq. For text analysis, we are seeing sparse term vectors with about 50k term cardinality, and only about 1k term size. If such a sparse vector is serialized with Gson, each instance will currently (in 65c anyway) include the 50k element bindings map that names each of the vector indices. This is not, IMHO, the kind of semantics we want. Nor do I think we want the input vectors to contain this redundant overhead.
Generally, these vectors are part of a matrix which should have this mapping available at the matrix level rather than at the vector (row) level. If you do that, the cost of storing the label map is amortized over many rows and it becomes irrelevant.
The reason that it is very important to store this map at the matrix level is so that matrix multiplications can be made efficient. If I am multiplying a matrix with column labels by a vector with element labels, I want the iteration to proceed by multiplying elements with the same label. This can be done by putting a permutation between the two operands or by remapping one or them to use the other's label map, or by sharing a label map across all elements of interest, or by iteratoring over labels instead of indexes.
My general preference is to have the code magically notice if the label map is shared (so that iteration over index is safe) and to iterate over labels if not.
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Jeff Eastman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719863#action_12719863 ]
Jeff Eastman commented on MAHOUT-65:
------------------------------------
Here's an issue that needs some further discussion:
For text analysis, we are seeing sparse term vectors with about 50k term cardinality, and only about 1k term size. If such a sparse vector is serialized with Gson, each instance will currently (in 65c anyway) include the 50k element bindings map that names each of the vector indices. This is not, IMHO, the kind of semantics we want. Nor do I think we want the input vectors to contain this redundant overhead.
I would propose to make the bindings map be transient, so that Gson will not output the binding maps when it serializes Vectors. Since the map is sharable - the API allows it to be set in one method, presumably processors of such term vectors would be ok with managing a shared instance independently from the vector points themselves. I also can imagine allowing a binding map to be named as an optional argument to e.g. Canopy clustering which, if supplied, would then be associated with each input point by the code as it is read in. But, Canopy does not need the map and its values would not be output either. So why bother?
Associating a binding map - or, in general, any similar meta-information - with each vector instance only makes sense to me if we have a system-wide policy for managing it efficiently.
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Updated: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by Jeff Eastman <jd...@windwardsolutions.com>.
Transient fields are used by the Gson serialization as well as Java
serialization. If you remove the transient label bindings indicator in
AbstractVector you will get rather surprising growth in your KMeans
output, especially if you do not name your input points. I suggest
keeping the transient in this case.
Jeff
Sean Owen (JIRA) wrote:
> [ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
>
> Sean Owen updated MAHOUT-65:
> ----------------------------
>
> Attachment: Clone_changes.patch
>
> OK yeah, the clone() methods needed some work. It is somewhat tricky to get right -- think I got it here. It takes even a little more work to do it efficiently. Worth taking a look at the patch. I am testing now to double-check before submitting.
>
> (And while I'm at it... can we place member fields before constructors, and not use 'transient' where Java serialization is not involved)
>
>
>> Add Element Labels to Vectors and Matrices
>> ------------------------------------------
>>
>> Key: MAHOUT-65
>> URL: https://issues.apache.org/jira/browse/MAHOUT-65
>> Project: Mahout
>> Issue Type: New Feature
>> Components: Matrix
>> Affects Versions: 0.1
>> Reporter: Jeff Eastman
>> Assignee: Jeff Eastman
>> Attachments: Clone_changes.patch, MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>>
>>
>> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
>>
>
>
[jira] Updated: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen updated MAHOUT-65:
----------------------------
Attachment: Clone_changes.patch
OK yeah, the clone() methods needed some work. It is somewhat tricky to get right -- think I got it here. It takes even a little more work to do it efficiently. Worth taking a look at the patch. I am testing now to double-check before submitting.
(And while I'm at it... can we place member fields before constructors, and not use 'transient' where Java serialization is not involved)
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: Clone_changes.patch, MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by Sean Owen <sr...@gmail.com>.
OK makes sense, I see. Will put it back -- and that I am sure will
make the test pass.
On Mon, Jul 6, 2009 at 4:49 PM, Jeff Eastman<jd...@windwardsolutions.com> wrote:
> See previous note. For space considerations we agreed Vector label bindings
> should not be serialized but that Matrix bindings should. Gson serialization
> uses the transient indicator.
Re: [jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by Jeff Eastman <jd...@windwardsolutions.com>.
See previous note. For space considerations we agreed Vector label
bindings should not be serialized but that Matrix bindings should. Gson
serialization uses the transient indicator.
Sean Owen (JIRA) wrote:
> [ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727440#action_12727440 ]
>
> Sean Owen commented on MAHOUT-65:
> ---------------------------------
>
> OK ready to submit, after I fixed a bone-headed NPE in the patch.
>
> Some tests in VectorTest fail, but I think they are actually wrong -- and passed before due to this bug. The test creates a labelled Vector, serializes/deserializes, but then expects it to be unbound?
>
>
>> Add Element Labels to Vectors and Matrices
>> ------------------------------------------
>>
>> Key: MAHOUT-65
>> URL: https://issues.apache.org/jira/browse/MAHOUT-65
>> Project: Mahout
>> Issue Type: New Feature
>> Components: Matrix
>> Affects Versions: 0.1
>> Reporter: Jeff Eastman
>> Assignee: Jeff Eastman
>> Attachments: Clone_changes.patch, MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>>
>>
>> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
>>
>
>
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727440#action_12727440 ]
Sean Owen commented on MAHOUT-65:
---------------------------------
OK ready to submit, after I fixed a bone-headed NPE in the patch.
Some tests in VectorTest fail, but I think they are actually wrong -- and passed before due to this bug. The test creates a labelled Vector, serializes/deserializes, but then expects it to be unbound?
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: Clone_changes.patch, MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Ted Dunning (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720784#action_12720784 ]
Ted Dunning commented on MAHOUT-65:
-----------------------------------
bq. I've seen this happen a lot with Solr and other XML based apps in that if you control both ends of the pipe, String based approaches, while nice from a readability standpoint, are showstoppers for performance. Having a converter from binary to String-based can then be employed when readability/debugging is required.
The natural binary alternatives would include:
1) thrift (or not ... could be complicated due to lots of compatibility concerns)
2) protocol buffers (not really open source)
3) ad hoc hadoop writable format (but writables may be going away in favor of (4))
4) avro (but it is really new)
My tendency would be either (1) or (4).
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Jeff Eastman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720216#action_12720216 ]
Jeff Eastman commented on MAHOUT-65:
------------------------------------
How about we just add a name attribute to AbstractVector and a getter and setter thereof? It would automatically be serialized with the vector state and can be further integrated with the Matrix stuff next. I can commit that now if you want?
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720797#action_12720797 ]
Sean Owen commented on MAHOUT-65:
---------------------------------
For strictly internal representations, the default Java serialization mechanism isn't bad at all. It is a pretty straightforward binary serialization (and can be overridden) and has the advantage of being pretty seamless to use in Java.
I would have also suggested PBs myself, what is the open-sourceness concern?
I agree that any string-based representation is going to be a lot slower, I'm almost sure, so this is a worthy inquiry. For external input and output however a more readable format could be desirable indeed.
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719119#action_12719119 ]
Grant Ingersoll commented on MAHOUT-65:
---------------------------------------
Jeff,
Could you bring this up to date with trunk?
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Reporter: Jeff Eastman
> Attachments: MAHOUT-65.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727430#action_12727430 ]
Sean Owen commented on MAHOUT-65:
---------------------------------
You are right. I think these two changes kind of collided as they were submitted about the same time. I will review all the existing clone() methods and make fixes.
NB that the clone() method must be updated when you add fields! think of it as another pseudo-constructor (which is one reason Josh Bloch kind of argues against clone()). May seem bad, but this is just the old copy() method, and it does have its uses.
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Karl Wettin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12613255#action_12613255 ]
Karl Wettin commented on MAHOUT-65:
-----------------------------------
Late submission, but I think I'd prefer if the labels was a bit less hard bound to vectors. And I want to associate them more than a name, at least class values too. Perhaps something like this:
{code:java}
abstract class Labels {
double get(Vector v, String l) {
return (v.get(indexOf(l));
}
void set(Vector v, String l, double w) {
v.set(indexOf(l), w);
}
int indexOf(String);
Label getLabel(int);
Iterator<Label> iterate();
}
class Label {
String name;
enum Type { real, ordinal }
Type type;
List<String> ordinalValues;
}
{code}
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Reporter: Jeff Eastman
> Attachments: MAHOUT-65.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720045#action_12720045 ]
Grant Ingersoll commented on MAHOUT-65:
---------------------------------------
Is the only way to add bindings by setting the map? Seems like
{code}
set(String label, int index, double value)
{code}
would be useful. And, that if bindings is null, it would create a new map.
Also, I'll see if I can work up the name thing.
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "M. Arshad Khan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728466#action_12728466 ]
M. Arshad Khan commented on MAHOUT-65:
--------------------------------------
The patch seems to work for me. However, I get some strange results when I run KMeans (through KMeansDriver) on the list of points. Previously, when I used k=4, the "points" would contain name of the point and the cluster name (0, 1, 2, 3). After this patch, the point name is there but cluster name is changed to the name of the centroid point id.
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: Clone_changes.patch, MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Jeff Eastman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615560#action_12615560 ]
Jeff Eastman commented on MAHOUT-65:
------------------------------------
This thread is beginning to diverge significantly from its original problem statement. While I can understand the trade-offs between binding optional Label maps to every Vector/Matrix instance vs. creating a new wrapper type to add labelled accessing when needed (Wettin 14-Jul) the above proposal seems to combine the worst of both worlds. I'd sure like to understand the problem we are trying to solve before going further. In particular, I do not understand the generality or universal utility of the specific attributes that are being discussed.
Karl, could you please state the problem you are trying to solve and how closely binding these Attributes to every Vector/Matrix instance helps us to solve it?
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Reporter: Jeff Eastman
> Attachments: MAHOUT-65.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Karl Wettin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12613330#action_12613330 ]
Karl Wettin commented on MAHOUT-65:
-----------------------------------
With "real" I just mean "plain old double weight feature value". With "ordinal" I mean associating vector values with feature string class values.
How do you propose one get to know if the feature is of ordinal type if we use subclasses? Wouldn't that mean lots of instanceof or a visitor? To me it seems much easier to bake it all up in one and the same class.
If the labels are hard bound to vectors, then I think it also makes sense to allow vectors to be hard bound to a matrix in which we keep a default list of labels to fall back on. I'm not going to push that hard for hard or loosly bound labels, but I'm going to take the oppertunity to say that many people have spent lots of time to define flexible data definition models, such as the one in JSR 247. We don't need all that stuff they got in there but the stuff we talk about here is very related to what their logical/physical data model is about.
Labels are not really needed, it's just meta data that makes life simpler when you are constructing a matrix. Let's not make such a tool difficult to use.
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Reporter: Jeff Eastman
> Attachments: MAHOUT-65.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and Matrices
Posted by Grant Ingersoll <gs...@apache.org>.
On Jul 22, 2008, at 1:09 PM, Ted Dunning (JIRA) wrote:
>
> [ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615675
> #action_12615675 ]
>
> Ted Dunning commented on MAHOUT-65:
> -----------------------------------
>
> Karl,
>
> That makes sense. Of course, you tend to make sense so that isn't a
> surprise.
>
> To summarize what I think people are after here:
>
> a) a wrapping structure over Matrix and Vector to allow annotation.
>
> b) the wrapper for Matrix should provide row and column labels where
> are just strings.
>
> c) the wrapper for Vector should allow a single label as well as per
> element labels (i.e. row and column labels) that may or may not be
> inherited
> from a matrix if the vector is a row or column view.
>
> d) attributes should be applicable to rows or column just as labels
> are.
>
> e) attributes should be able to specify, at least, the general type
> of data (continuous, ordinal, categorial) as well as provide some
> information about the encoding of ordinal and categorial values.
>
> f) the underlying matrix or vector should not know about labels and
> attributes
>
> g) the wrapped structure should admit matrix like operations (such
> as element-wise operations, aggregates and matrix multiplication)
> that work label-wise rather than row or column index-wise.
>
> Is that a fair summary?
+1
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Ted Dunning (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615675#action_12615675 ]
Ted Dunning commented on MAHOUT-65:
-----------------------------------
Karl,
That makes sense. Of course, you tend to make sense so that isn't a surprise.
To summarize what I think people are after here:
a) a wrapping structure over Matrix and Vector to allow annotation.
b) the wrapper for Matrix should provide row and column labels where are just strings.
c) the wrapper for Vector should allow a single label as well as per element labels (i.e. row and column labels) that may or may not be inherited
from a matrix if the vector is a row or column view.
d) attributes should be applicable to rows or column just as labels are.
e) attributes should be able to specify, at least, the general type of data (continuous, ordinal, categorial) as well as provide some information about the encoding of ordinal and categorial values.
f) the underlying matrix or vector should not know about labels and attributes
g) the wrapped structure should admit matrix like operations (such as element-wise operations, aggregates and matrix multiplication) that work label-wise rather than row or column index-wise.
Is that a fair summary?
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Reporter: Jeff Eastman
> Attachments: MAHOUT-65.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Jeff Eastman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Eastman updated MAHOUT-65:
-------------------------------
Attachment: MAHOUT-65c.patch
This patch goes a step further than 65b and changes Vector and Matrix serializations to use Gson. Most of the clustering packages are impacted as they had dependencies upon the encoded format, but the rest of Mahout seems to be unaffected.
mvn install succeeds.
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720752#action_12720752 ]
Grant Ingersoll commented on MAHOUT-65:
---------------------------------------
bq. For really big vectors I'd expect serialization and deserialization to be String-intensive.
Yeah, it is. It's the majority of the work, in fact, in the profiling (85%+ of time spent) I've done so far on this. Not sure if it is premature optimization, but it seems likely that a compact binary format may be a nice option to have for internal pieces of the puzzle. In other words, the input can still be String based as we have now, but the internal mappers and reducers can use something more compact.
I've seen this happen a lot with Solr and other XML based apps in that if you control both ends of the pipe, String based approaches, while nice from a readability standpoint, are showstoppers for performance. Having a converter from binary to String-based can then be employed when readability/debugging is required.
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Grant Ingersoll updated MAHOUT-65:
----------------------------------
Attachment: MAHOUT-65-name.patch
Add name attribute. Also added some docs on equals and added a strict equivalence notion that can be useful if one cares about the implementation.
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-65-name.patch, MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAHOUT-65) Add Element Labels to Vectors and
Matrices
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-65?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen updated MAHOUT-65:
----------------------------
Resolution: Fixed
Fix Version/s: 0.2
Status: Resolved (was: Patch Available)
Am i correct that the original topic of this issue is basically done and resolved? we have labels, and the subsequent discussions here were committed and reflected in code.
> Add Element Labels to Vectors and Matrices
> ------------------------------------------
>
> Key: MAHOUT-65
> URL: https://issues.apache.org/jira/browse/MAHOUT-65
> Project: Mahout
> Issue Type: New Feature
> Components: Matrix
> Affects Versions: 0.1
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Fix For: 0.2
>
> Attachments: Clone_changes.patch, MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65-name.patch, MAHOUT-65.patch, MAHOUT-65b.patch, MAHOUT-65c.patch, MAHOUT-65d.patch
>
>
> Many applications can benefit by accessing elements in vectors and matrices using String labels in addition to numeric indices. Investigate adding such a capability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.