You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by Karl Wettin <ka...@gmail.com> on 2008/07/11 17:05:05 UTC
Normalized job input
Almost anything in the trunk and almost all patches have a unique way
to read input data. I think it is time to at least start talking about
a way to normalize this: how to serialize a matrix. It could be
something silly simple for a start: I'm thinking a directory that
contains a sequence file of vectors, a map file for feature labels
column index and one map file for numeric feature class value index.
That should be enough.
There is also a bit of reoccuring pre-processing code in
classification patches (and probably elsewhere too) that does
calculations such as idf and what not. It would be great to factor
these things out to some common API that can be pre-calculated or lazy
evaluated, whatever fits the algorithm and the corpus best.
But I also feel that what I've explained above should be an
implementation of something abstract. I can't help thinking that the
coolest thing one can do with lazy algorithms is to run them on a 100M
document Lucene index as matrix.
karl
Re: Normalized job input
Posted by Michael Marth <mm...@day.com>.
hi,
if you consider protocol buffers you might want to have a look at thrift as
well (comparison to pb[1]). it is a project originating from facebook, but
now proposed for apache incubation[2].
cheers
michael
[1] http://stuartsierra.com/2008/07/10/thrift-vs-protocol-buffers
[2] http://wiki.apache.org/incubator/ThriftProposal
On Sun, Jul 13, 2008 at 6:19 PM, Sean Owen <sr...@gmail.com> wrote:
> I vote for using existing serializations where possible. I don't
> necessarily find it valuable to make the serialization mechanism
> pluggable or selectable. Then you have to support n different formats,
> when I imagine most possible encodings are not even very desirable to
> use, so it's a bit of complexity that may not be worth it.
>
> There is Java's own serialization mechanism, which is customizable.
> The only downside is that it's a binary format and so probably hard to
> generate from a non-Java platform.
>
> XML is too verbose.
>
> Some simple text-based format wouldn't be unreasonable, like what Karl
> is suggesting, though any text-based format is going to be a bit
> verbose by nature.
>
> There is protocol buffers! Obviously I know a bit about this from
> Google, and now it's open source. It's a pretty lean simple binary
> format framework. http://code.google.com/p/protobuf/ I can't say it's
> very standard yet as it was only released this week.
>
>
> I would imagine that making up a sensible text format is the best
> solution for interoperability, followed by using Java serialization.
>
>
> On Sun, Jul 13, 2008 at 8:36 AM, Isabel Drost
> <ap...@isabel-drost.de> wrote:
> > On Friday 11 July 2008, Karl Wettin wrote:
> >> Almost anything in the trunk and almost all patches have a unique way
> >> to read input data. I think it is time to at least start talking about
> >> a way to normalize this: how to serialize a matrix.
> >
> > +1
> >
> >
> >> It could be something silly simple for a start: I'm thinking a directory
> >> that contains a sequence file of vectors, a map file for feature labels
> >> column index and one map file for numeric feature class value index.
> >> That should be enough.
> >
> > I think it should be possible to come up with some simple to use API that
> > makes it possible to use different serialization strategies underneath.
> The
> > first strategy could be the simple one you proposed. It would be nice if
> it
> > were simply a matter of configuration or initialization to change the
> > physical organization of the data.
> >
> >
> >> There is also a bit of reoccuring pre-processing code in
> >> classification patches (and probably elsewhere too) that does
> >> calculations such as idf and what not. It would be great to factor
> >> these things out to some common API that can be pre-calculated or lazy
> >> evaluated, whatever fits the algorithm and the corpus best.
> >
> > +1 There should be one package for pre processing text and converting it
> into
> > feature vectors. We might look into interfacing with UIMA for this task
> as
> > well.
> >
> >
> >> But I also feel that what I've explained above should be an
> >> implementation of something abstract. I can't help thinking that the
> >> coolest thing one can do with lazy algorithms is to run them on a 100M
> >> document Lucene index as matrix.
> >
> > I think for processing texts that could be a viable option for a start.
> The
> > algorithms should not directly depend on Lucene, but there could be a
> data
> > serialization strategy that does.
> >
> > What do others think about data (de-)serialization?
> >
> >
> > Isabel
> >
> > --
> > Faith goes out through the window when beauty comes in at the door.
> > |\ _,,,---,,_ Web: <http://www.isabel-drost.de>
> > /,`.-'`' -. ;-;;,_
> > |,4- ) )-,_..;\ ( `'-'
> > '---''(_/--' `-'\_) (fL) IM: <xm...@spaceboyz.net>
> >
>
--
Michael Marth | Day JCR Cup 08 | Win a MacBook Pro: http://dev.day.com/
Re: Normalized job input
Posted by Isabel Drost <ap...@isabel-drost.de>.
On Monday 28 July 2008, Isabel Drost wrote:
> | Loading pre processed datasets
One additional note to that one: I guess this use case fits very well with
the "problems" reading arff formatted data Deneche ran into with MAHOUT-56.
Isabel
--
If things don't improve soon, you'd better ask them to stop helping you.
|\ _,,,---,,_ Web: <http://www.isabel-drost.de>
/,`.-'`' -. ;-;;,_
|,4- ) )-,_..;\ ( `'-'
'---''(_/--' `-'\_) (fL) IM: <xm...@spaceboyz.net>
Re: Normalized job input
Posted by Ted Dunning <te...@gmail.com>.
Thrift is also a likely candidate for binary serialization.
Neither of these address the question of interface with external tools.
On Mon, Jul 28, 2008 at 5:11 AM, Isabel Drost <apache_mahout@isabel-drost.de
> wrote:
>
> > There is protocol buffers!
>
> I am neither intimately familiar with protocol buffers nor with the Hessian
> protocol, but on first sight both seem to be rather similar. Could you
> please
> elaborate a little on the advantages of protocol buffers?
>
>
>
Re: Normalized job input
Posted by Isabel Drost <ap...@isabel-drost.de>.
On Sunday 13 July 2008, Sean Owen wrote:
> I vote for using existing serializations where possible. I don't
> necessarily find it valuable to make the serialization mechanism
> pluggable or selectable. Then you have to support n different formats,
> when I imagine most possible encodings are not even very desirable to
> use, so it's a bit of complexity that may not be worth it.
I think it could help to clarify the use cases I had in mind for "pluggable"
serialization implementations:
In my oppinion there are at least three stages where reading/writing data is
beneficial for Mahout users.
| Loading raw data for processing (either during training or at application
| time):
The input would be some raw datapoints (texts, images, data records) that need
to be processed. I think we should provide an interface for retrieving data
from various sources that is to be implemented by the user of Mahout to fit
his needs.
I think it would be a good idea to look into UIMA before designing the
interface. The reason is that UIMA in my humble opinion is suitable for pre
processing data points, be it text, images, sound files or what ever you can
imagine.
The resulting pipeline would look something like the following:
raw data -> UIMA -> user specific implementation of Mahout interface -> Mahout
or in a more simple setup:
raw data -> user specific implementation of Mahout interface -> Mahout
| Loading pre processed datasets
There may be cases where you are evaluating algorithms but want to keep the
pre processing steps stable. Say, you want to find out whether Naive Bayes,
CNB, SVM or some other fancy algorithm fits your problem best. Imagine
further that in your setting pre processing takes a huge amount of time. One
way to optimise this setting would be to do pre processing once and repeat
training over and over again. Than you want to store the result of your pre
processing as an intermediate result.
Now imagine you have done your evaluation, and your colleague comes along and
tells you about this cool new collection of learning algorithm
implementations. The collection contains algorithms not yet supported by
Mahout. But lucky you, it supports reading pre processed data sets that are
stored in some standard file format. So new you want to export your pre
processed data to said format. I do not argue for us to implement each and
every import filter, but we should make it easy for users of Mahout to do so,
where possible.
The resulting pipelines would look something like the following:
... -> Mahout pre processing -> user specific export filter for processed data
user specific import filter for pre processed data -> Mahout
| Reading/Writing model files
The third use case I see for serialization is storing model files. Imagine you
have trained a model with Mahout on a lot of data. But you want to use the
model in a setting in a less scalable environment, say in your personal mail
client on your sub notebook. Alternatively you could imagine to load your
model into another data mining environment to reuse it.
There are standards for storing and loading model files. At least for some
algorihtms. Again I do not argue for implementing these standards ourselves,
but we should make it easy for users to add these implementations.
I guess, arguing in favour of configurable or pluggable implementations was
the wrong term to use. I should have said, that we should make it easy to
provide user specific implementations for the common (de-)serialization
tasks. I would also argue to make it easy for people to use Mahout in a
dependency injection framework like Spring or Google Guice.
> There is protocol buffers!
I am neither intimately familiar with protocol buffers nor with the Hessian
protocol, but on first sight both seem to be rather similar. Could you please
elaborate a little on the advantages of protocol buffers?
Sorry, for the delay: Been on vacation in Prague.
Isabel
--
You know it's going to be a bad day when you want to put on the clothesyou
wore home from the party and there aren't any.
|\ _,,,---,,_ Web: <http://www.isabel-drost.de>
/,`.-'`' -. ;-;;,_
|,4- ) )-,_..;\ ( `'-'
'---''(_/--' `-'\_) (fL) IM: <xm...@spaceboyz.net>
Re: Normalized job input
Posted by Sean Owen <sr...@gmail.com>.
I vote for using existing serializations where possible. I don't
necessarily find it valuable to make the serialization mechanism
pluggable or selectable. Then you have to support n different formats,
when I imagine most possible encodings are not even very desirable to
use, so it's a bit of complexity that may not be worth it.
There is Java's own serialization mechanism, which is customizable.
The only downside is that it's a binary format and so probably hard to
generate from a non-Java platform.
XML is too verbose.
Some simple text-based format wouldn't be unreasonable, like what Karl
is suggesting, though any text-based format is going to be a bit
verbose by nature.
There is protocol buffers! Obviously I know a bit about this from
Google, and now it's open source. It's a pretty lean simple binary
format framework. http://code.google.com/p/protobuf/ I can't say it's
very standard yet as it was only released this week.
I would imagine that making up a sensible text format is the best
solution for interoperability, followed by using Java serialization.
On Sun, Jul 13, 2008 at 8:36 AM, Isabel Drost
<ap...@isabel-drost.de> wrote:
> On Friday 11 July 2008, Karl Wettin wrote:
>> Almost anything in the trunk and almost all patches have a unique way
>> to read input data. I think it is time to at least start talking about
>> a way to normalize this: how to serialize a matrix.
>
> +1
>
>
>> It could be something silly simple for a start: I'm thinking a directory
>> that contains a sequence file of vectors, a map file for feature labels
>> column index and one map file for numeric feature class value index.
>> That should be enough.
>
> I think it should be possible to come up with some simple to use API that
> makes it possible to use different serialization strategies underneath. The
> first strategy could be the simple one you proposed. It would be nice if it
> were simply a matter of configuration or initialization to change the
> physical organization of the data.
>
>
>> There is also a bit of reoccuring pre-processing code in
>> classification patches (and probably elsewhere too) that does
>> calculations such as idf and what not. It would be great to factor
>> these things out to some common API that can be pre-calculated or lazy
>> evaluated, whatever fits the algorithm and the corpus best.
>
> +1 There should be one package for pre processing text and converting it into
> feature vectors. We might look into interfacing with UIMA for this task as
> well.
>
>
>> But I also feel that what I've explained above should be an
>> implementation of something abstract. I can't help thinking that the
>> coolest thing one can do with lazy algorithms is to run them on a 100M
>> document Lucene index as matrix.
>
> I think for processing texts that could be a viable option for a start. The
> algorithms should not directly depend on Lucene, but there could be a data
> serialization strategy that does.
>
> What do others think about data (de-)serialization?
>
>
> Isabel
>
> --
> Faith goes out through the window when beauty comes in at the door.
> |\ _,,,---,,_ Web: <http://www.isabel-drost.de>
> /,`.-'`' -. ;-;;,_
> |,4- ) )-,_..;\ ( `'-'
> '---''(_/--' `-'\_) (fL) IM: <xm...@spaceboyz.net>
>
Re: Normalized job input
Posted by Isabel Drost <ap...@isabel-drost.de>.
On Friday 11 July 2008, Karl Wettin wrote:
> Almost anything in the trunk and almost all patches have a unique way
> to read input data. I think it is time to at least start talking about
> a way to normalize this: how to serialize a matrix.
+1
> It could be something silly simple for a start: I'm thinking a directory
> that contains a sequence file of vectors, a map file for feature labels
> column index and one map file for numeric feature class value index.
> That should be enough.
I think it should be possible to come up with some simple to use API that
makes it possible to use different serialization strategies underneath. The
first strategy could be the simple one you proposed. It would be nice if it
were simply a matter of configuration or initialization to change the
physical organization of the data.
> There is also a bit of reoccuring pre-processing code in
> classification patches (and probably elsewhere too) that does
> calculations such as idf and what not. It would be great to factor
> these things out to some common API that can be pre-calculated or lazy
> evaluated, whatever fits the algorithm and the corpus best.
+1 There should be one package for pre processing text and converting it into
feature vectors. We might look into interfacing with UIMA for this task as
well.
> But I also feel that what I've explained above should be an
> implementation of something abstract. I can't help thinking that the
> coolest thing one can do with lazy algorithms is to run them on a 100M
> document Lucene index as matrix.
I think for processing texts that could be a viable option for a start. The
algorithms should not directly depend on Lucene, but there could be a data
serialization strategy that does.
What do others think about data (de-)serialization?
Isabel
--
Faith goes out through the window when beauty comes in at the door.
|\ _,,,---,,_ Web: <http://www.isabel-drost.de>
/,`.-'`' -. ;-;;,_
|,4- ) )-,_..;\ ( `'-'
'---''(_/--' `-'\_) (fL) IM: <xm...@spaceboyz.net>