You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by Paul Elschot <pa...@xs4all.nl> on 2008/02/05 09:12:27 UTC
Jython
Dear readers,
I had a quick look at Elefant. The GUI has a prominent Python
menu entry, and Python has a nice place in its architecture.
I've been using Jython, the java implementation of Python, only occasionally,
but I'd like to use it again. The biggest advantage of Jython is its almost
transparent java interface, i.e. there is no need at all to write wrappers around
java libraries, and it provides the ease of Python for scripting.
Is there more interest in using Jython here?
Regards,
Paul Elschot
Re: Linear algebra in a JVM [was Re: Jython]
Posted by edward yoon <ed...@udanax.org>.
> It mean that BigTable is used for analysis processing with arbitrary
> set of elements by query, not a relational data processing.
Sorry, I think it could easily be understood wrongly.
It's a end of expression of my thoughts.
Let me see about it. :)
On 2/7/08, edward yoon <ed...@udanax.org> wrote:
> Actually, My most hadoop applications are made for numeric analysis.
> Therefore, I was tried to make a generalized matrix in/out format.
> https://issues.apache.org/jira/browse/HADOOP-2515
> as a Map<row, Map<column, cell>> structure after review the code and
> discuss with gary bradski.
>
> But, If i make a new matrix file structure on Hadoop HDFS, i think it
> could be some resemblancing going on Hbase. So, I think Hadoop + Hbase
> is good fit with matrix management & operation.
>
> "It (BigTable) presents the abstraction of a 2-dimensional
> table of data cells, with different versions over time making
> up a third dimension." -- Failure Trends in a Large Disk Drive Population, 2007
>
> It mean that BigTable is used for analysis processing with arbitrary
> set of elements by query, not a relational data processing.
>
> > I see http://wiki.apache.org/hadoop/Matrix
>
> Thanks for your review.
> I hope we talk together soon.
>
> On 2/7/08, Grant Ingersoll <gs...@apache.org> wrote:
> > How do you think these various libraries fit into Hadoop? Does it
> > make sense to just build what we need using HBase? I see http://wiki.apache.org/hadoop/Matrix
> > does some matrix things, but then it has a Groovy overlay, so it
> > isn't quite what we want, I don't think.
> >
> > Perhaps, we should just think about, and push up to Hadoop if we can,
> > our own set of Hadoop based matrix libraries. Starting off, we need a
> > decent way to create a matrix and populate it, then also basic matrix
> > things like addition, multiplication, etc. Then we can add other
> > things as we need them? For instance, I am interested in TextRank
> > (search for Mihalcea and TextRank) and it essentially comes down to
> > doing an iterative algorithm over a matrix. I was thinking I might,
> > as a way to get deeper into the latest Hadoop, use it as a sample,
> > useful algorithm. It's not specifically ML, but it does have
> > interesting results and it is fairly easy to implement.
> >
> > Should we just lay out a page on the Wiki where we can start thinking
> > about matrix needs? Using other libraries is definitely an option,
> > but I am not sure if they will be optimal in the Hadoop environment.
> >
> > -Grant
> >
> > On Feb 6, 2008, at 12:18 PM, Ted Dunning wrote:
> >
> > >
> > > There are unfortunately many choices for linear algebra in JVM's, none
> > > particularly satisfactory.
> > >
> > > Colt is the one I use. It has a very odd syntax, but gives good
> > > performance. The structure is such that it is very hard to extend
> > > to, say,
> > > sparse matrices. The licensing on Colt isn't particularly easy,
> > > either and
> > > I have been unable to contact the author to see about liberalizing it.
> > >
> > > Jama is now essentially defunct, but it had a very simple API and
> > > not very
> > > high performance. Extending to additional matrix types is also not
> > > feasible
> > > due to the design exposing matrix internal structure as a double
> > > indexed
> > > matrix. The licensing on Jama is very open.
> > >
> > > MTJ is high performance and has a less strange API than Colt, but I
> > > haven't
> > > used it so I can't say much about performance. I get the impression
> > > it
> > > would be difficult to extend, but I could well be wrong about that.
> > >
> > > Commons math uses an extension of Jama, I think. I haven't used
> > > it. The
> > > last time I looked seriously at commons math, the committers had
> > > some very
> > > odd agendas going on so I dropped it from consideration. It looks
> > > like it
> > > has come quite a ways since then, but I haven't dug into it deeply
> > > since my
> > > first evaluation.
> > >
> > >
> > > On 2/6/08 12:45 AM, "Paul Elschot" <pa...@xs4all.nl> wrote:
> > >
> > >> Op Wednesday 06 February 2008 05:23:31 schreef Markus Weimer:
> > >>> Hi,
> > >>> One of my contributions to Elefant is an adapter to the Java
> > >>> Version of UIMA
> > >>> which allows you to pipe Python strings through a UIMA annotation
> > >>> engine and
> > >>> get feature vectors to work with back. This was done using JPype: <
> > >>> http://jpype.sourceforge.net/>, a tool which links the JVM to the
> > >>> CPython
> > >>> VM.
> > >>>
> > >>> I choose this non-obvious approach because we use native code Python
> > >>> extensions for the matrix operations, an area where Java
> > >>> regrettably lacks
> > >>> behind big time compared to native code. So, Jython was out of the
> > >>> question
> > >>> as I don't know any way to access a CPython extension from Jython.
> > >>> I found
> > >>> JPype to do the job and to do it well (the overhead per Cross-VM
> > >>> call was
> > >>> around 1ms on my laptop). So for those craving for a state-of-the-
> > >>> art Python
> > >>> with decent extensions and access to Java code, this might be an
> > >>> option.
> > >>
> > >> Well, one of my favourite Java libraries made it into the email
> > >> address of
> > >> this
> > >> list, and I must say, I was hoping to get some good solutions to
> > >> the problem
> > >> of
> > >> linear algebra in a JVM here. Has this problem been discussed
> > >> beforehand?
> > >>
> > >> I have only used linear algebra packages well before there was Java,
> > >> so I wonder how to go about it now.
> > >>
> > >> Regards,
> > >> Paul Elschot
> > >>
> > >
> >
> > --------------------------
> > Grant Ingersoll
> > http://lucene.grantingersoll.com
> > http://www.lucenebootcamp.com
> >
> > Lucene Helpful Hints:
> > http://wiki.apache.org/lucene-java/BasicsOfPerformance
> > http://wiki.apache.org/lucene-java/LuceneFAQ
> >
> >
> >
> >
> >
>
>
> --
> B. Regards,
> Edward yoon @ NHN, corp.
>
--
B. Regards,
Edward yoon @ NHN, corp.
Re: Linear algebra in a JVM [was Re: Jython]
Posted by edward yoon <ed...@udanax.org>.
> It mean that BigTable is used for analysis processing with arbitrary
> set of elements by query, not a relational data processing.
Sorry, I think it could easily be understood wrongly.
It's a end of expression of my thoughts.
Let me see about it. :)
On 2/7/08, edward yoon <ed...@udanax.org> wrote:
> Actually, My most hadoop applications are made for numeric analysis.
> Therefore, I was tried to make a generalized matrix in/out format.
> https://issues.apache.org/jira/browse/HADOOP-2515
> as a Map<row, Map<column, cell>> structure after review the code and
> discuss with gary bradski.
>
> But, If i make a new matrix file structure on Hadoop HDFS, i think it
> could be some resemblancing going on Hbase. So, I think Hadoop + Hbase
> is good fit with matrix management & operation.
>
> "It (BigTable) presents the abstraction of a 2-dimensional
> table of data cells, with different versions over time making
> up a third dimension." -- Failure Trends in a Large Disk Drive Population, 2007
>
> It mean that BigTable is used for analysis processing with arbitrary
> set of elements by query, not a relational data processing.
>
> > I see http://wiki.apache.org/hadoop/Matrix
>
> Thanks for your review.
> I hope we talk together soon.
>
> On 2/7/08, Grant Ingersoll <gs...@apache.org> wrote:
> > How do you think these various libraries fit into Hadoop? Does it
> > make sense to just build what we need using HBase? I see http://wiki.apache.org/hadoop/Matrix
> > does some matrix things, but then it has a Groovy overlay, so it
> > isn't quite what we want, I don't think.
> >
> > Perhaps, we should just think about, and push up to Hadoop if we can,
> > our own set of Hadoop based matrix libraries. Starting off, we need a
> > decent way to create a matrix and populate it, then also basic matrix
> > things like addition, multiplication, etc. Then we can add other
> > things as we need them? For instance, I am interested in TextRank
> > (search for Mihalcea and TextRank) and it essentially comes down to
> > doing an iterative algorithm over a matrix. I was thinking I might,
> > as a way to get deeper into the latest Hadoop, use it as a sample,
> > useful algorithm. It's not specifically ML, but it does have
> > interesting results and it is fairly easy to implement.
> >
> > Should we just lay out a page on the Wiki where we can start thinking
> > about matrix needs? Using other libraries is definitely an option,
> > but I am not sure if they will be optimal in the Hadoop environment.
> >
> > -Grant
> >
> > On Feb 6, 2008, at 12:18 PM, Ted Dunning wrote:
> >
> > >
> > > There are unfortunately many choices for linear algebra in JVM's, none
> > > particularly satisfactory.
> > >
> > > Colt is the one I use. It has a very odd syntax, but gives good
> > > performance. The structure is such that it is very hard to extend
> > > to, say,
> > > sparse matrices. The licensing on Colt isn't particularly easy,
> > > either and
> > > I have been unable to contact the author to see about liberalizing it.
> > >
> > > Jama is now essentially defunct, but it had a very simple API and
> > > not very
> > > high performance. Extending to additional matrix types is also not
> > > feasible
> > > due to the design exposing matrix internal structure as a double
> > > indexed
> > > matrix. The licensing on Jama is very open.
> > >
> > > MTJ is high performance and has a less strange API than Colt, but I
> > > haven't
> > > used it so I can't say much about performance. I get the impression
> > > it
> > > would be difficult to extend, but I could well be wrong about that.
> > >
> > > Commons math uses an extension of Jama, I think. I haven't used
> > > it. The
> > > last time I looked seriously at commons math, the committers had
> > > some very
> > > odd agendas going on so I dropped it from consideration. It looks
> > > like it
> > > has come quite a ways since then, but I haven't dug into it deeply
> > > since my
> > > first evaluation.
> > >
> > >
> > > On 2/6/08 12:45 AM, "Paul Elschot" <pa...@xs4all.nl> wrote:
> > >
> > >> Op Wednesday 06 February 2008 05:23:31 schreef Markus Weimer:
> > >>> Hi,
> > >>> One of my contributions to Elefant is an adapter to the Java
> > >>> Version of UIMA
> > >>> which allows you to pipe Python strings through a UIMA annotation
> > >>> engine and
> > >>> get feature vectors to work with back. This was done using JPype: <
> > >>> http://jpype.sourceforge.net/>, a tool which links the JVM to the
> > >>> CPython
> > >>> VM.
> > >>>
> > >>> I choose this non-obvious approach because we use native code Python
> > >>> extensions for the matrix operations, an area where Java
> > >>> regrettably lacks
> > >>> behind big time compared to native code. So, Jython was out of the
> > >>> question
> > >>> as I don't know any way to access a CPython extension from Jython.
> > >>> I found
> > >>> JPype to do the job and to do it well (the overhead per Cross-VM
> > >>> call was
> > >>> around 1ms on my laptop). So for those craving for a state-of-the-
> > >>> art Python
> > >>> with decent extensions and access to Java code, this might be an
> > >>> option.
> > >>
> > >> Well, one of my favourite Java libraries made it into the email
> > >> address of
> > >> this
> > >> list, and I must say, I was hoping to get some good solutions to
> > >> the problem
> > >> of
> > >> linear algebra in a JVM here. Has this problem been discussed
> > >> beforehand?
> > >>
> > >> I have only used linear algebra packages well before there was Java,
> > >> so I wonder how to go about it now.
> > >>
> > >> Regards,
> > >> Paul Elschot
> > >>
> > >
> >
> > --------------------------
> > Grant Ingersoll
> > http://lucene.grantingersoll.com
> > http://www.lucenebootcamp.com
> >
> > Lucene Helpful Hints:
> > http://wiki.apache.org/lucene-java/BasicsOfPerformance
> > http://wiki.apache.org/lucene-java/LuceneFAQ
> >
> >
> >
> >
> >
>
>
> --
> B. Regards,
> Edward yoon @ NHN, corp.
>
--
B. Regards,
Edward yoon @ NHN, corp.
Re: Linear algebra in a JVM [was Re: Jython]
Posted by edward yoon <ed...@udanax.org>.
> Wouldn't that mean a huge overhead in storage space requirements?
Not exactly, I'm willing to make the sacrifice in order to get a super
computer even if overhead. And I considered various methods for
storing the vectors and matrices which contain dimensioned elements
and are generic computable.
On 2/7/08, Markus Weimer <ma...@gmail.com> wrote:
> Hi,
> On Feb 7, 2008 6:06 PM, edward yoon <ed...@udanax.org> wrote:
>
> > Actually, My most hadoop applications are made for numeric analysis.
> > Therefore, I was tried to make a generalized matrix in/out format.
> > https://issues.apache.org/jira/browse/HADOOP-2515
> > as a Map<row, Map<column, cell>> structure after review the code and
> > discuss with gary bradski.
> >
>
> Wouldn't that mean a huge overhead in storage space requirements? Even if it
> was just a simple Collection, the whole boxing/unboxing business can easily
> increase the memory requirements 5 fold: Take the netflix dataset as an
> example. It as 100M entries on a scale of 1...5. Those values fit in 100MB
> of memory if they are stored as chars. Boxing would add an overhead of at
> least 4 Bytes per entry to hold a reference to the object, which leaves us
> with at least 500MB for the same amount of actual data stored. Probably the
> overhead is even more, as the object itself will probably need some
> information like a pointer to its class, too.
>
> Or did I grossly misunderstand you?
>
> Thanks,
>
> Markus
>
--
B. Regards,
Edward yoon @ NHN, corp.
Re: Linear algebra in a JVM [was Re: Jython]
Posted by Markus Weimer <ma...@gmail.com>.
Hi,
On Feb 7, 2008 6:06 PM, edward yoon <ed...@udanax.org> wrote:
> Actually, My most hadoop applications are made for numeric analysis.
> Therefore, I was tried to make a generalized matrix in/out format.
> https://issues.apache.org/jira/browse/HADOOP-2515
> as a Map<row, Map<column, cell>> structure after review the code and
> discuss with gary bradski.
>
Wouldn't that mean a huge overhead in storage space requirements? Even if it
was just a simple Collection, the whole boxing/unboxing business can easily
increase the memory requirements 5 fold: Take the netflix dataset as an
example. It as 100M entries on a scale of 1...5. Those values fit in 100MB
of memory if they are stored as chars. Boxing would add an overhead of at
least 4 Bytes per entry to hold a reference to the object, which leaves us
with at least 500MB for the same amount of actual data stored. Probably the
overhead is even more, as the object itself will probably need some
information like a pointer to its class, too.
Or did I grossly misunderstand you?
Thanks,
Markus
Re: Linear algebra in a JVM [was Re: Jython]
Posted by edward yoon <ed...@udanax.org>.
Actually, My most hadoop applications are made for numeric analysis.
Therefore, I was tried to make a generalized matrix in/out format.
https://issues.apache.org/jira/browse/HADOOP-2515
as a Map<row, Map<column, cell>> structure after review the code and
discuss with gary bradski.
But, If i make a new matrix file structure on Hadoop HDFS, i think it
could be some resemblancing going on Hbase. So, I think Hadoop + Hbase
is good fit with matrix management & operation.
"It (BigTable) presents the abstraction of a 2-dimensional
table of data cells, with different versions over time making
up a third dimension." -- Failure Trends in a Large Disk Drive Population, 2007
It mean that BigTable is used for analysis processing with arbitrary
set of elements by query, not a relational data processing.
> I see http://wiki.apache.org/hadoop/Matrix
Thanks for your review.
I hope we talk together soon.
On 2/7/08, Grant Ingersoll <gs...@apache.org> wrote:
> How do you think these various libraries fit into Hadoop? Does it
> make sense to just build what we need using HBase? I see http://wiki.apache.org/hadoop/Matrix
> does some matrix things, but then it has a Groovy overlay, so it
> isn't quite what we want, I don't think.
>
> Perhaps, we should just think about, and push up to Hadoop if we can,
> our own set of Hadoop based matrix libraries. Starting off, we need a
> decent way to create a matrix and populate it, then also basic matrix
> things like addition, multiplication, etc. Then we can add other
> things as we need them? For instance, I am interested in TextRank
> (search for Mihalcea and TextRank) and it essentially comes down to
> doing an iterative algorithm over a matrix. I was thinking I might,
> as a way to get deeper into the latest Hadoop, use it as a sample,
> useful algorithm. It's not specifically ML, but it does have
> interesting results and it is fairly easy to implement.
>
> Should we just lay out a page on the Wiki where we can start thinking
> about matrix needs? Using other libraries is definitely an option,
> but I am not sure if they will be optimal in the Hadoop environment.
>
> -Grant
>
> On Feb 6, 2008, at 12:18 PM, Ted Dunning wrote:
>
> >
> > There are unfortunately many choices for linear algebra in JVM's, none
> > particularly satisfactory.
> >
> > Colt is the one I use. It has a very odd syntax, but gives good
> > performance. The structure is such that it is very hard to extend
> > to, say,
> > sparse matrices. The licensing on Colt isn't particularly easy,
> > either and
> > I have been unable to contact the author to see about liberalizing it.
> >
> > Jama is now essentially defunct, but it had a very simple API and
> > not very
> > high performance. Extending to additional matrix types is also not
> > feasible
> > due to the design exposing matrix internal structure as a double
> > indexed
> > matrix. The licensing on Jama is very open.
> >
> > MTJ is high performance and has a less strange API than Colt, but I
> > haven't
> > used it so I can't say much about performance. I get the impression
> > it
> > would be difficult to extend, but I could well be wrong about that.
> >
> > Commons math uses an extension of Jama, I think. I haven't used
> > it. The
> > last time I looked seriously at commons math, the committers had
> > some very
> > odd agendas going on so I dropped it from consideration. It looks
> > like it
> > has come quite a ways since then, but I haven't dug into it deeply
> > since my
> > first evaluation.
> >
> >
> > On 2/6/08 12:45 AM, "Paul Elschot" <pa...@xs4all.nl> wrote:
> >
> >> Op Wednesday 06 February 2008 05:23:31 schreef Markus Weimer:
> >>> Hi,
> >>> One of my contributions to Elefant is an adapter to the Java
> >>> Version of UIMA
> >>> which allows you to pipe Python strings through a UIMA annotation
> >>> engine and
> >>> get feature vectors to work with back. This was done using JPype: <
> >>> http://jpype.sourceforge.net/>, a tool which links the JVM to the
> >>> CPython
> >>> VM.
> >>>
> >>> I choose this non-obvious approach because we use native code Python
> >>> extensions for the matrix operations, an area where Java
> >>> regrettably lacks
> >>> behind big time compared to native code. So, Jython was out of the
> >>> question
> >>> as I don't know any way to access a CPython extension from Jython.
> >>> I found
> >>> JPype to do the job and to do it well (the overhead per Cross-VM
> >>> call was
> >>> around 1ms on my laptop). So for those craving for a state-of-the-
> >>> art Python
> >>> with decent extensions and access to Java code, this might be an
> >>> option.
> >>
> >> Well, one of my favourite Java libraries made it into the email
> >> address of
> >> this
> >> list, and I must say, I was hoping to get some good solutions to
> >> the problem
> >> of
> >> linear algebra in a JVM here. Has this problem been discussed
> >> beforehand?
> >>
> >> I have only used linear algebra packages well before there was Java,
> >> so I wonder how to go about it now.
> >>
> >> Regards,
> >> Paul Elschot
> >>
> >
>
> --------------------------
> Grant Ingersoll
> http://lucene.grantingersoll.com
> http://www.lucenebootcamp.com
>
> Lucene Helpful Hints:
> http://wiki.apache.org/lucene-java/BasicsOfPerformance
> http://wiki.apache.org/lucene-java/LuceneFAQ
>
>
>
>
>
--
B. Regards,
Edward yoon @ NHN, corp.
Re: Linear algebra in a JVM [was Re: Jython]
Posted by edward yoon <ed...@udanax.org>.
Actually, My most hadoop applications are made for numeric analysis.
Therefore, I was tried to make a generalized matrix in/out format.
https://issues.apache.org/jira/browse/HADOOP-2515
as a Map<row, Map<column, cell>> structure after review the code and
discuss with gary bradski.
But, If i make a new matrix file structure on Hadoop HDFS, i think it
could be some resemblancing going on Hbase. So, I think Hadoop + Hbase
is good fit with matrix management & operation.
"It (BigTable) presents the abstraction of a 2-dimensional
table of data cells, with different versions over time making
up a third dimension." -- Failure Trends in a Large Disk Drive Population, 2007
It mean that BigTable is used for analysis processing with arbitrary
set of elements by query, not a relational data processing.
> I see http://wiki.apache.org/hadoop/Matrix
Thanks for your review.
I hope we talk together soon.
On 2/7/08, Grant Ingersoll <gs...@apache.org> wrote:
> How do you think these various libraries fit into Hadoop? Does it
> make sense to just build what we need using HBase? I see http://wiki.apache.org/hadoop/Matrix
> does some matrix things, but then it has a Groovy overlay, so it
> isn't quite what we want, I don't think.
>
> Perhaps, we should just think about, and push up to Hadoop if we can,
> our own set of Hadoop based matrix libraries. Starting off, we need a
> decent way to create a matrix and populate it, then also basic matrix
> things like addition, multiplication, etc. Then we can add other
> things as we need them? For instance, I am interested in TextRank
> (search for Mihalcea and TextRank) and it essentially comes down to
> doing an iterative algorithm over a matrix. I was thinking I might,
> as a way to get deeper into the latest Hadoop, use it as a sample,
> useful algorithm. It's not specifically ML, but it does have
> interesting results and it is fairly easy to implement.
>
> Should we just lay out a page on the Wiki where we can start thinking
> about matrix needs? Using other libraries is definitely an option,
> but I am not sure if they will be optimal in the Hadoop environment.
>
> -Grant
>
> On Feb 6, 2008, at 12:18 PM, Ted Dunning wrote:
>
> >
> > There are unfortunately many choices for linear algebra in JVM's, none
> > particularly satisfactory.
> >
> > Colt is the one I use. It has a very odd syntax, but gives good
> > performance. The structure is such that it is very hard to extend
> > to, say,
> > sparse matrices. The licensing on Colt isn't particularly easy,
> > either and
> > I have been unable to contact the author to see about liberalizing it.
> >
> > Jama is now essentially defunct, but it had a very simple API and
> > not very
> > high performance. Extending to additional matrix types is also not
> > feasible
> > due to the design exposing matrix internal structure as a double
> > indexed
> > matrix. The licensing on Jama is very open.
> >
> > MTJ is high performance and has a less strange API than Colt, but I
> > haven't
> > used it so I can't say much about performance. I get the impression
> > it
> > would be difficult to extend, but I could well be wrong about that.
> >
> > Commons math uses an extension of Jama, I think. I haven't used
> > it. The
> > last time I looked seriously at commons math, the committers had
> > some very
> > odd agendas going on so I dropped it from consideration. It looks
> > like it
> > has come quite a ways since then, but I haven't dug into it deeply
> > since my
> > first evaluation.
> >
> >
> > On 2/6/08 12:45 AM, "Paul Elschot" <pa...@xs4all.nl> wrote:
> >
> >> Op Wednesday 06 February 2008 05:23:31 schreef Markus Weimer:
> >>> Hi,
> >>> One of my contributions to Elefant is an adapter to the Java
> >>> Version of UIMA
> >>> which allows you to pipe Python strings through a UIMA annotation
> >>> engine and
> >>> get feature vectors to work with back. This was done using JPype: <
> >>> http://jpype.sourceforge.net/>, a tool which links the JVM to the
> >>> CPython
> >>> VM.
> >>>
> >>> I choose this non-obvious approach because we use native code Python
> >>> extensions for the matrix operations, an area where Java
> >>> regrettably lacks
> >>> behind big time compared to native code. So, Jython was out of the
> >>> question
> >>> as I don't know any way to access a CPython extension from Jython.
> >>> I found
> >>> JPype to do the job and to do it well (the overhead per Cross-VM
> >>> call was
> >>> around 1ms on my laptop). So for those craving for a state-of-the-
> >>> art Python
> >>> with decent extensions and access to Java code, this might be an
> >>> option.
> >>
> >> Well, one of my favourite Java libraries made it into the email
> >> address of
> >> this
> >> list, and I must say, I was hoping to get some good solutions to
> >> the problem
> >> of
> >> linear algebra in a JVM here. Has this problem been discussed
> >> beforehand?
> >>
> >> I have only used linear algebra packages well before there was Java,
> >> so I wonder how to go about it now.
> >>
> >> Regards,
> >> Paul Elschot
> >>
> >
>
> --------------------------
> Grant Ingersoll
> http://lucene.grantingersoll.com
> http://www.lucenebootcamp.com
>
> Lucene Helpful Hints:
> http://wiki.apache.org/lucene-java/BasicsOfPerformance
> http://wiki.apache.org/lucene-java/LuceneFAQ
>
>
>
>
>
--
B. Regards,
Edward yoon @ NHN, corp.
Re: Linear algebra in a JVM [was Re: Jython]
Posted by Ted Dunning <td...@veoh.com>.
Please do. I will be very interested in your thoughts, but would like to
add mine as well. You + me + others = wiki.
On 2/6/08 12:54 PM, "Grant Ingersoll" <gs...@apache.org> wrote:
> Should we just lay out a page on the Wiki where we can start thinking
> about matrix needs?
Re: Linear algebra in a JVM [was Re: Jython]
Posted by Grant Ingersoll <gs...@apache.org>.
How do you think these various libraries fit into Hadoop? Does it
make sense to just build what we need using HBase? I see http://wiki.apache.org/hadoop/Matrix
does some matrix things, but then it has a Groovy overlay, so it
isn't quite what we want, I don't think.
Perhaps, we should just think about, and push up to Hadoop if we can,
our own set of Hadoop based matrix libraries. Starting off, we need a
decent way to create a matrix and populate it, then also basic matrix
things like addition, multiplication, etc. Then we can add other
things as we need them? For instance, I am interested in TextRank
(search for Mihalcea and TextRank) and it essentially comes down to
doing an iterative algorithm over a matrix. I was thinking I might,
as a way to get deeper into the latest Hadoop, use it as a sample,
useful algorithm. It's not specifically ML, but it does have
interesting results and it is fairly easy to implement.
Should we just lay out a page on the Wiki where we can start thinking
about matrix needs? Using other libraries is definitely an option,
but I am not sure if they will be optimal in the Hadoop environment.
-Grant
On Feb 6, 2008, at 12:18 PM, Ted Dunning wrote:
>
> There are unfortunately many choices for linear algebra in JVM's, none
> particularly satisfactory.
>
> Colt is the one I use. It has a very odd syntax, but gives good
> performance. The structure is such that it is very hard to extend
> to, say,
> sparse matrices. The licensing on Colt isn't particularly easy,
> either and
> I have been unable to contact the author to see about liberalizing it.
>
> Jama is now essentially defunct, but it had a very simple API and
> not very
> high performance. Extending to additional matrix types is also not
> feasible
> due to the design exposing matrix internal structure as a double
> indexed
> matrix. The licensing on Jama is very open.
>
> MTJ is high performance and has a less strange API than Colt, but I
> haven't
> used it so I can't say much about performance. I get the impression
> it
> would be difficult to extend, but I could well be wrong about that.
>
> Commons math uses an extension of Jama, I think. I haven't used
> it. The
> last time I looked seriously at commons math, the committers had
> some very
> odd agendas going on so I dropped it from consideration. It looks
> like it
> has come quite a ways since then, but I haven't dug into it deeply
> since my
> first evaluation.
>
>
> On 2/6/08 12:45 AM, "Paul Elschot" <pa...@xs4all.nl> wrote:
>
>> Op Wednesday 06 February 2008 05:23:31 schreef Markus Weimer:
>>> Hi,
>>> One of my contributions to Elefant is an adapter to the Java
>>> Version of UIMA
>>> which allows you to pipe Python strings through a UIMA annotation
>>> engine and
>>> get feature vectors to work with back. This was done using JPype: <
>>> http://jpype.sourceforge.net/>, a tool which links the JVM to the
>>> CPython
>>> VM.
>>>
>>> I choose this non-obvious approach because we use native code Python
>>> extensions for the matrix operations, an area where Java
>>> regrettably lacks
>>> behind big time compared to native code. So, Jython was out of the
>>> question
>>> as I don't know any way to access a CPython extension from Jython.
>>> I found
>>> JPype to do the job and to do it well (the overhead per Cross-VM
>>> call was
>>> around 1ms on my laptop). So for those craving for a state-of-the-
>>> art Python
>>> with decent extensions and access to Java code, this might be an
>>> option.
>>
>> Well, one of my favourite Java libraries made it into the email
>> address of
>> this
>> list, and I must say, I was hoping to get some good solutions to
>> the problem
>> of
>> linear algebra in a JVM here. Has this problem been discussed
>> beforehand?
>>
>> I have only used linear algebra packages well before there was Java,
>> so I wonder how to go about it now.
>>
>> Regards,
>> Paul Elschot
>>
>
--------------------------
Grant Ingersoll
http://lucene.grantingersoll.com
http://www.lucenebootcamp.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ
Re: Linear algebra in a JVM [was Re: Jython]
Posted by Ted Dunning <td...@veoh.com>.
There are unfortunately many choices for linear algebra in JVM's, none
particularly satisfactory.
Colt is the one I use. It has a very odd syntax, but gives good
performance. The structure is such that it is very hard to extend to, say,
sparse matrices. The licensing on Colt isn't particularly easy, either and
I have been unable to contact the author to see about liberalizing it.
Jama is now essentially defunct, but it had a very simple API and not very
high performance. Extending to additional matrix types is also not feasible
due to the design exposing matrix internal structure as a double indexed
matrix. The licensing on Jama is very open.
MTJ is high performance and has a less strange API than Colt, but I haven't
used it so I can't say much about performance. I get the impression it
would be difficult to extend, but I could well be wrong about that.
Commons math uses an extension of Jama, I think. I haven't used it. The
last time I looked seriously at commons math, the committers had some very
odd agendas going on so I dropped it from consideration. It looks like it
has come quite a ways since then, but I haven't dug into it deeply since my
first evaluation.
On 2/6/08 12:45 AM, "Paul Elschot" <pa...@xs4all.nl> wrote:
> Op Wednesday 06 February 2008 05:23:31 schreef Markus Weimer:
>> Hi,
>> One of my contributions to Elefant is an adapter to the Java Version of UIMA
>> which allows you to pipe Python strings through a UIMA annotation engine and
>> get feature vectors to work with back. This was done using JPype: <
>> http://jpype.sourceforge.net/>, a tool which links the JVM to the CPython
>> VM.
>>
>> I choose this non-obvious approach because we use native code Python
>> extensions for the matrix operations, an area where Java regrettably lacks
>> behind big time compared to native code. So, Jython was out of the question
>> as I don't know any way to access a CPython extension from Jython. I found
>> JPype to do the job and to do it well (the overhead per Cross-VM call was
>> around 1ms on my laptop). So for those craving for a state-of-the-art Python
>> with decent extensions and access to Java code, this might be an option.
>
> Well, one of my favourite Java libraries made it into the email address of
> this
> list, and I must say, I was hoping to get some good solutions to the problem
> of
> linear algebra in a JVM here. Has this problem been discussed beforehand?
>
> I have only used linear algebra packages well before there was Java,
> so I wonder how to go about it now.
>
> Regards,
> Paul Elschot
>
Linear algebra in a JVM [was Re: Jython]
Posted by Paul Elschot <pa...@xs4all.nl>.
Op Wednesday 06 February 2008 05:23:31 schreef Markus Weimer:
> Hi,
> One of my contributions to Elefant is an adapter to the Java Version of UIMA
> which allows you to pipe Python strings through a UIMA annotation engine and
> get feature vectors to work with back. This was done using JPype: <
> http://jpype.sourceforge.net/>, a tool which links the JVM to the CPython
> VM.
>
> I choose this non-obvious approach because we use native code Python
> extensions for the matrix operations, an area where Java regrettably lacks
> behind big time compared to native code. So, Jython was out of the question
> as I don't know any way to access a CPython extension from Jython. I found
> JPype to do the job and to do it well (the overhead per Cross-VM call was
> around 1ms on my laptop). So for those craving for a state-of-the-art Python
> with decent extensions and access to Java code, this might be an option.
Well, one of my favourite Java libraries made it into the email address of this
list, and I must say, I was hoping to get some good solutions to the problem of
linear algebra in a JVM here. Has this problem been discussed beforehand?
I have only used linear algebra packages well before there was Java,
so I wonder how to go about it now.
Regards,
Paul Elschot
Re: Jython
Posted by Ted Dunning <td...@veoh.com>.
Cpython would be an option for integration, but not from the Java side.
Each map call would involve a cross call so processing a typical GB of data
would likely involve millions of calls to the mapper. At 1ms per call, that
is just infeasible.
The other option would be to build on the C ports of hadoop, but I have lots
of Java code that I can't leave behind for these apps.
On 2/5/08 8:23 PM, "Markus Weimer" <ma...@gmail.com> wrote:
> So, Jython was out of the question
> as I don't know any way to access a CPython extension from Jython. I found
> JPype to do the job and to do it well (the overhead per Cross-VM call was
> around 1ms on my laptop). So for those craving for a state-of-the-art Python
> with decent extensions and access to Java code, this might be an option.
Re: Jython
Posted by Markus Weimer <ma...@gmail.com>.
Hi,
One of my contributions to Elefant is an adapter to the Java Version of UIMA
which allows you to pipe Python strings through a UIMA annotation engine and
get feature vectors to work with back. This was done using JPype: <
http://jpype.sourceforge.net/>, a tool which links the JVM to the CPython
VM.
I choose this non-obvious approach because we use native code Python
extensions for the matrix operations, an area where Java regrettably lacks
behind big time compared to native code. So, Jython was out of the question
as I don't know any way to access a CPython extension from Jython. I found
JPype to do the job and to do it well (the overhead per Cross-VM call was
around 1ms on my laptop). So for those craving for a state-of-the-art Python
with decent extensions and access to Java code, this might be an option.
Greetings from Canberra,
Markus
Re: Jython
Posted by Paul Elschot <pa...@xs4all.nl>.
...
>
> >> b) related to (a) is the fact that jython is like a python time capsule.
> >> Many of the features that make python attractive are missing from jython.
> >
> > For me, that is limited to the with statement and conditional statement
> > in Python 2.5 that are not in Jython 2.2.
>
> Comprehensions, dates, regex differences, yield and a bunch of other things
> for me. I found that I had to worry about portability from current python
> scripts quite a bit.
>
List comprehensions work well for me in Jython 2.2b1.
I don't remember in which CPython version they were introduced, but this
may be one of the features of Jython for which it is a bit ahead of its official
version number.
I never used Python regexes, preferring the Java ones, which are just as
easily usable from Jython.
But then I don't have to worry about portability at all, at least so far.
Regards,
Paul Elschot
Re: Jython
Posted by Ted Dunning <td...@veoh.com>.
On 2/5/08 11:39 AM, "Paul Elschot" <pa...@xs4all.nl> wrote:
> ... I don't want to spend time learning
> yet another programming language.
I was very impressed at how easy Groovy is to learn for a Java programmer.
Certainly much easier than Python (which you already know, of course).
>> b) related to (a) is the fact that jython is like a python time capsule.
>> Many of the features that make python attractive are missing from jython.
>
> For me, that is limited to the with statement and conditional statement
> in Python 2.5 that are not in Jython 2.2.
Comprehensions, dates, regex differences, yield and a bunch of other things
for me. I found that I had to worry about portability from current python
scripts quite a bit.
>> c) Groovy's integration with java is far superior to Jython's. Jython can
>> call java, but java has more trouble calling Jython. With groovy, the
>> interface is completely transparent in both directions.
>
> That's an advantage and a disadvantage: more distance from java means
> easier programming, allthough probably not much.
I didn't find that to be true at all. Groovy definitely has gotten this
very right.
> Then calling back into Jython is not an issue, also because inheritance
> from Java works well in Jython, despite the distance.
But in writing map-reduce frameworks, I care a LOT about how easy it is to
call into the scripting language. I also want to be able to very easily
avoid crossing the boundary. That is relatively trivial in Groovy, hard in
Jython.
Re: Jython
Posted by Paul Elschot <pa...@xs4all.nl>.
I had a look at Groovy. For the moment I'm not going to switch,
but the main reason is that I don't want to spend time learning
yet another programming language. Some more comments below.
Op Tuesday 05 February 2008 17:38:01 schreef Ted Dunning:
>
> I spent quite a bit of time recently evaluating alternatives for scripting
> in Java. Jython was my presumptive choice at the beginning of the
> evaluation.
>
> I eventually chose Groovy.
>
> The major issues that made Jython unacceptable for my project were:
>
> a) no significant development progress over a period of years and still
> only a tiny developer community.
>
> b) related to (a) is the fact that jython is like a python time capsule.
> Many of the features that make python attractive are missing from jython.
For me, that is limited to the with statement and conditional statement
in Python 2.5 that are not in Jython 2.2.
> The small developer community is both a cause of this and a reason that it
> won't be repaired any time soon.
I think the main reason for the small development community some time agon
on Jython is that 2.1 was simply too good. Jython development towards 2.5
seems to be nicely on its way again now.
> c) Groovy's integration with java is far superior to Jython's. Jython can
> call java, but java has more trouble calling Jython. With groovy, the
> interface is completely transparent in both directions.
That's an advantage and a disadvantage: more distance from java means
easier programming, allthough probably not much.
Jython has more distance from Java than Groovy, and indeed calling Jython
from Java is not nice. I have mainly used Jython by simply adding Java
libraries to the class path, and writing jython glue code around these.
Then calling back into Jython is not an issue, also because inheritance
from Java works well in Jython, despite the distance.
> d) development tools for Jython are poor while IntelliJ Idea supports
> debugging and incremental compilation of Groovy as a first class language.
For development I only need the editor to be able to follow a traceback,
from a test run in whatever language. With a bit of setting up, jEdit does
that nicely, both in Java and Jython (and probably in Groovy, too).
And Jython needs a bit of help to run all unit tests in a directory structure,
on the Java side that is directly usable from ant an junit.
> Frankly, I am not at all unhappy with the decision and would advocate use of
> Groovy in mahout over Jython if there is a choice that has to be made. Of
> course, if some people want one and some the other then this is open source
> we are talking about.
Groovy is quite readable, so I wouldn't mind to give it a try to provide patches.
Regards,
Paul Elschot
> On 2/5/08 12:12 AM, "Paul Elschot" <pa...@xs4all.nl> wrote:
>
> > Dear readers,
> >
> > I had a quick look at Elefant. The GUI has a prominent Python
> > menu entry, and Python has a nice place in its architecture.
> >
> > I've been using Jython, the java implementation of Python, only occasionally,
> > but I'd like to use it again. The biggest advantage of Jython is its almost
> > transparent java interface, i.e. there is no need at all to write wrappers
> > around
> > java libraries, and it provides the ease of Python for scripting.
> >
> > Is there more interest in using Jython here?
> >
Re: Jython
Posted by Ted Dunning <td...@veoh.com>.
I spent quite a bit of time recently evaluating alternatives for scripting
in Java. Jython was my presumptive choice at the beginning of the
evaluation.
I eventually chose Groovy.
The major issues that made Jython unacceptable for my project were:
a) no significant development progress over a period of years and still
only a tiny developer community.
b) related to (a) is the fact that jython is like a python time capsule.
Many of the features that make python attractive are missing from jython.
The small developer community is both a cause of this and a reason that it
won't be repaired any time soon.
c) Groovy's integration with java is far superior to Jython's. Jython can
call java, but java has more trouble calling Jython. With groovy, the
interface is completely transparent in both directions.
d) development tools for Jython are poor while IntelliJ Idea supports
debugging and incremental compilation of Groovy as a first class language.
Frankly, I am not at all unhappy with the decision and would advocate use of
Groovy in mahout over Jython if there is a choice that has to be made. Of
course, if some people want one and some the other then this is open source
we are talking about.
On 2/5/08 12:12 AM, "Paul Elschot" <pa...@xs4all.nl> wrote:
> Dear readers,
>
> I had a quick look at Elefant. The GUI has a prominent Python
> menu entry, and Python has a nice place in its architecture.
>
> I've been using Jython, the java implementation of Python, only occasionally,
> but I'd like to use it again. The biggest advantage of Jython is its almost
> transparent java interface, i.e. there is no need at all to write wrappers
> around
> java libraries, and it provides the ease of Python for scripting.
>
> Is there more interest in using Jython here?
>
> Regards,
> Paul Elschot