You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov> on 2016/01/19 06:58:26 UTC
Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine
Translation Toolkit
Great Hen, we’d love to have you on board as a mentor! Please
add yourself to the proposal on the wiki.
Anyone else have interest in Machine Translation? Any OpenNLP folks,
Hadoop folks, Tika, or Lucene folks? CC’ing the dev lists for visibility
please feel free to reply to general@i.a.o.
I’ll leave the DISCUSS thread open for a few more days.
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Henri Yandell <ba...@apache.org>
Reply-To: "general@incubator.apache.org" <ge...@incubator.apache.org>
Date: Monday, January 18, 2016 at 7:57 PM
To: jpluser <ch...@jpl.nasa.gov>,
"general@incubator.apache.org" <ge...@incubator.apache.org>
Subject: Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine
Translation Toolkit
>Non-binding +1 to Joshua joining the Incubator. I'd be interested in
>mentoring.
>
>
>> -----Original Message-----
>> From: jpluser <ch...@jpl.nasa.gov>
>> Reply-To: "general@incubator.apache.org" <ge...@incubator.apache.org>
>> Date: Tuesday, January 12, 2016 at 10:56 PM
>> To: "general@incubator.apache.org" <ge...@incubator.apache.org>
>> Cc: "post@cs.jhu.edu" <po...@cs.jhu.edu>
>> Subject: [DISCUSS] Apache Joshua Incubator Proposal - Machine
>>Translation
>> Toolkit
>>
>> >Hi Everyone,
>> >
>> >Please find attached for your viewing pleasure a proposed new project,
>> >Apache Joshua, a statistical machine translation toolkit. The proposal
>> >is in wiki draft form at:
>> https://wiki.apache.org/incubator/JoshuaProposal
>> >
>> >Proposal text is copied below. I’ll leave the discussion open for a
>>week
>> >and we are interested in folks who would like to be initial committers
>> >and mentors. Please discuss here on the thread.
>> >
>> >Thanks!
>> >
>> >Cheers,
>> >Chris (Champion)
>> >
>> >———
>> >
>> >= Joshua Proposal =
>> >
>> >== Abstract ==
>> >[[joshua-decoder.org|Joshua]] is an open-source statistical machine
>> >translation toolkit. It includes a Java-based decoder for translating
>>with
>> >phrase-based, hierarchical, and syntax-based translation models, a
>> >Hadoop-based grammar extractor (Thrax), and an extensive set of tools
>>and
>> >scripts for training and evaluating new models from parallel text.
>> >
>> >== Proposal ==
>> >Joshua is a state of the art statistical machine translation system
>>that
>> >provides a number of features:
>> >
>> > * Support for the two main paradigms in statistical machine
>>translation:
>> >phrase-based and hierarchical / syntactic.
>> > * A sparse feature API that makes it easy to add new feature templates
>> >supporting millions of features
>> > * Native implementations of many tuners (MERT, MIRA, PRO, and AdaGrad)
>> > * Support for lattice decoding, allowing upstream NLP tools to expose
>> >their hypothesis space to the MT system
>> > * An efficient representation for models, allowing for quick loading
>>of
>> >multi-gigabyte model files
>> > * Fast decoding speed (on par with Moses and mtplz)
>> > * Language packs — precompiled models that allow the decoder to be
>>run as
>> >a black box
>> > * Thrax, a Hadoop-based tool for learning translation models from
>> >parallel text
>> > * A suite of tools for constructing new models for any language pair
>>for
>> >which sufficient training data exists
>> >
>> >== Background and Rationale ==
>> >A number of factors make this a good time for an Apache project
>>focused on
>> >machine translation (MT): the quality of MT output (for many language
>> >pairs); the average computing resources available on computers,
>>relative
>> >to the needs of MT systems; and the availability of a number of
>> >high-quality toolkits, together with a large base of researchers
>>working
>> >on them.
>> >
>> >Over the past decade, machine translation (MT; the automatic
>>translation
>> >of one human language to another) has become a reality. The research
>>into
>> >statistical approaches to translation that began in the early nineties,
>> >together with the availability of large amounts of training data, and
>> >better computing infrastructure, have all come together to produce
>> >translations results that are “good enough” for a large set of language
>> >pairs and use cases. Free services like
>> >[[https://www.bing.com/translator|Bing Translator]] and
>> >[[https://translate.google.com|Google Translate]] have made these
>> services
>> >available to the average person through direct interfaces and through
>> >tools like browser plugins, and sites across the world with higher
>> >translation needs use them to translate their pages through
>>automatically.
>> >
>> >MT does not require the infrastructure of large corporations in order
>>to
>> >produce feasible output. Machine translation can be resource-intensive,
>> >but need not be prohibitively so. Disk and memory usage are mostly a
>> >matter of model size, which for most language pairs is a few gigabytes
>>at
>> >most, at which size models can provide coverage on the order of tens or
>> >even hundreds of thousands of words in the input and output languages.
>>The
>> >computational complexity of the algorithms used to search for
>>translations
>> >of new sentences are typically linear in the number of words in the
>>input
>> >sentence, making it possible to run a translation engine on a personal
>> >computer.
>> >
>> >The research community has produced many different open source
>>translation
>> >projects for a range of programming languages and under a variety of
>> >licenses. These projects include the core “decoder”, which takes a
>>model
>> >and uses it to translate new sentences between the language pair the
>>model
>> >was defined for. They also typically include a large set of tools that
>> >enable new models to be built from large sets of example translations
>> >(“parallel data”) and monolingual texts. These toolkits are usually
>>built
>> >to support the agendas of the (largely) academic researchers that build
>> >them: the repeated cycle of building new models, tuning model
>>parameters
>> >against development data, and evaluating them against held-out test
>>data,
>> >using standard metrics for testing the quality of MT output.
>> >
>> >Together, these three factors—the quality of machine translation
>>output,
>> >the feasibility of translating on standard computers, and the
>>availability
>> >of tools to build models—make it reasonable for the end users to use
>>MT as
>> >a black-box service, and to run it on their personal machine.
>> >
>> >These factors make it a good time for an organization with the status
>>of
>> >the Apache Foundation to host a machine translation project.
>> >
>> >== Current Status ==
>> >Joshua was originally ported from David Chiang’s Python implementation
>>of
>> >Hiero by Zhifei Li, while he was a Ph.D. student at Johns Hopkins
>> >University. The current version is maintained by Matt Post at Johns
>> >Hopkins’ Human Language Technology Center of Excellence. Joshua has
>>made
>> >many releases with a list of over 20 source code tags. The last
>>release of
>> >Joshua was 6.0.5 on November 5th, 2015.
>> >
>> >== Meritocracy ==
>> >The current developers are familiar with meritocratic open source
>> >development at Apache. Apache was chosen specifically because we want
>>to
>> >encourage this style of development for the project.
>> >
>> >== Community ==
>> >Joshua is used widely across the world. Perhaps its biggest (known)
>> >research / industrial user is the Amazon research group in Berlin.
>>Another
>> >user is the US Army Research Lab. No formal census has been undertaken,
>> >but posts to the Joshua technical support mailing list, along with the
>> >occasional contributions, suggest small research and academic
>>communities
>> >spread across the world, many of them in India.
>> >
>> >During incubation, we will explicitly seek to increase our usage across
>> >the board, including academic research, industry, and other end users
>> >interested in statistical machine translation.
>> >
>> >== Core Developers ==
>> >The current set of core developers is fairly small, having fallen with
>>the
>> >graduation from Johns Hopkins of some core student participants.
>>However,
>> >Joshua is used fairly widely, as mentioned above, and there remains a
>> >commitment from the principal researcher at Johns Hopkins to continue
>>to
>> >use and develop it. Joshua has seen a number of new community members
>> >become interested recently due to a potential for its projected use in
>>a
>> >number of ongoing DARPA projects such as XDATA and Memex.
>> >
>> >== Alignment ==
>> >Joshua is currently Copyright (c) 2015, Johns Hopkins University All
>> >rights reserved and licensed under BSD 2-clause license. It would of
>> >course be the intention to relicense this code under AL2.0 which would
>> >permit expanded and increased use of the software within Apache
>>projects.
>> >There is currently an ongoing effort within the Apache Tika community
>>to
>> >utilize Joshua within Tika’s Translate API, see
>> >[[https://issues.apache.org/jira/browse/TIKA-1343|TIKA-1343]].
>> >
>> >== Known Risks ==
>> >
>> >=== Orphaned products ===
>> >At the moment, regular contributions are made by a single contributor,
>>the
>> >lead maintainer. He (Matt Post) plans to continue development for the
>>next
>> >few years, but it is still a single point of failure, since the
>>graduate
>> >students who worked on the project have moved on to jobs, mostly in
>> >industry. However, our goal is to help that process by growing the
>> >community in Apache, and at least in growing the community with users
>>and
>> >participants from NASA JPL.
>> >
>> >=== Inexperience with Open Source ===
>> >The team both at Johns Hopkins and NASA JPL have experience with many
>>OSS
>> >software projects at Apache and elsewhere. We understand "how it works"
>> >here at the foundation.
>> >
>> >
>> >== Relationships with Other Apache Products ==
>> >Joshua includes dependences on Hadoop, and also is included as a
>>plugin in
>> >Apache Tika. We are also interested in coordinating with other projects
>> >including Spark, and other projects needing MT services for language
>> >translation.
>> >
>> >== Developers ==
>> >Joshua only has one regular developer who is employed by Johns Hopkins
>> >University. NASA JPL (Mattmann and McGibbney) have been contributing
>> >lately including a Brew formula and other contributions to the project
>> >through the DARPA XDATA and Memex programs.
>> >
>> >== Documentation ==
>> >Documentation and publications related to Joshua can be found at
>> >joshua-decoder.org. The source for the Joshua documentation is
>>currently
>> >hosted on Github at
>> >https://github.com/joshua-decoder/joshua-decoder.github.com
>> >
>> >== Initial Source ==
>> >Current source resides at Github: github.com/joshua-decoder/joshua (the
>> >main decoder and toolkit) and github.com/joshua-decoder/thrax (the
>> grammar
>> >extraction tool).
>> >
>> >== External Dependencies ==
>> >Joshua has a number of external dependencies. Only BerkeleyLM (Apache
>>2.0)
>> >and KenLM (LGPG 2.1) are run-time decoder dependencies (one of which is
>> >needed for translating sentences with pre-built models). The rest are
>> >dependencies for the build system and pipeline, used for constructing
>>and
>> >training new models from parallel text.
>> >
>> >Apache projects:
>> > * Ant
>> > * Hadoop
>> > * Commons
>> > * Maven
>> > * Ivy
>> >
>> >There are also a number of other open-source projects with various
>> >licenses that the project depends on both dynamically (runtime), and
>> >statically.
>> >
>> >=== GNU GPL 2 ===
>> > * Berkeley Aligner: https://code.google.com/p/berkeleyaligner/
>> >
>> >=== LGPG 2.1 ===
>> > * KenLM: github.com/kpu/kenlm
>> >
>> >=== Apache 2.0 ===
>> > * BerkeleyLM: https://code.google.com/p/berkeleylm/
>> >
>> >=== GNU GPL ===
>> > * GIZA++: http://www.statmt.org/moses/giza/GIZA++.html
>> >
>> >== Required Resources ==
>> > * Mailing Lists
>> > * private@joshua.incubator.apache.org
>> > * dev@joshua.incubator.apache.org
>> > * commits@joshua.incubator.apache.org
>> >
>> > * Git Repos
>> > * https://git-wip-us.apache.org/repos/asf/joshua.git
>> >
>> > * Issue Tracking
>> > * JIRA Joshua (JOSHUA)
>> >
>> > * Continuous Integration
>> > * Jenkins builds on https://builds.apache.org/
>> >
>> > * Web
>> > * http://joshua.incubator.apache.org/
>> > * wiki at http://cwiki.apache.org
>> >
>> >== Initial Committers ==
>> >The following is a list of the planned initial Apache committers (the
>> >active subset of the committers for the current repository on Github).
>> >
>> > * Matt Post (post@cs.jhu.edu)
>> > * Lewis John McGibbney (lewismc@apache.org)
>> > * Chris Mattmann (mattmann@apache.org)
>> >
>> >== Affiliations ==
>> >
>> > * Johns Hopkins University
>> > * Matt Post
>> >
>> > * NASA JPL
>> > * Chris Mattmann
>> > * Lewis John McGibbney
>> >
>> >
>> >== Sponsors ==
>> >=== Champion ===
>> > * Chris Mattmann (NASA/JPL)
>> >
>> >=== Nominated Mentors ===
>> > * Paul Ramirez
>> > * Lewis John McGibbney
>> > * Chris Mattmann
>> >
>> >== Sponsoring Entity ==
>> >The Apache Incubator
>> >
>> >
>> >
>> >
>> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >Chris Mattmann, Ph.D.
>> >Chief Architect
>> >Instrument Software and Science Data Systems Section (398)
>> >NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> >Office: 168-519, Mailstop: 168-527
>> >Email: chris.a.mattmann@nasa.gov
>> >WWW: http://sunset.usc.edu/~mattmann/
>> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >Adjunct Associate Professor, Computer Science Department
>> >University of Southern California, Los Angeles, CA 90089 USA
>> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >
>> >
>> >
>>
>>>?B�KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKC
>>>B�
>>
>>>?�?[��X��ܚX�K??K[XZ[?�?�[�\�[?][��X��ܚX�P?[��X�]?܋�\?X�?K�ܙ�B��܈?Y??]?[ۘ
>>>[?
>> >?��[X[�?�??K[XZ[?�?�[�\�[?Z?[???[��X�]?܋�\?X�?K�ܙ�B
Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine
Translation Toolkit
Posted by "Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov>.
Awesome glad to have you onboard please add your info to the proposal on the wiki
Sent from my iPhone
> On Jan 19, 2016, at 9:02 AM, ben gao <ba...@gmail.com> wrote:
>
> Yes, I am.
>
> On Tue, Jan 19, 2016, 11:56 AM Mattmann, Chris A (3980) <
> chris.a.mattmann@jpl.nasa.gov> wrote:
>
>> Thanks Martin, filed:
>>
>> https://github.com/joshua-decoder/joshua/issues/239
>>
>>
>> Are you interested in joining in the Incubation efforts?
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Chief Architect
>> Instrument Software and Science Data Systems Section (398)
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 168-519, Mailstop: 168-527
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Associate Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Martin Gainty <mg...@hotmail.com>
>> Reply-To: "dev@opennlp.apache.org" <de...@opennlp.apache.org>
>> Date: Tuesday, January 19, 2016 at 7:57 AM
>> To: "dev@opennlp.apache.org" <de...@opennlp.apache.org>, Lewis McGibbney
>> <le...@gmail.com>
>> Subject: RE: [DISCUSS] Apache Joshua Incubator Proposal - Machine
>> Translation Toolkit
>>
>>> dependency addition to Joshua-Decoder/Joshua/pom.xml
>>> <!-- MCG added args4j for
>>> /Joshua-Decoder/Joshua/src/joshua/decoder/ff/tm/CreateGlueGrammar.java:[17
>>> ,26] package org.kohsuke.args4j does not exist error --> <dependency>
>>> <groupId>args4j</groupId> <artifactId>args4j</artifactId>
>>> <version>2.32</version> </dependency>
>>> Joshua-Decoder/Joshua committer please add this dependency to pom.xml
>>> thank you/
>>> Martin
>>> ______________________________________________
>>>
>>>
>>>
>>>> From: chris.a.mattmann@jpl.nasa.gov
>>>> To: general@incubator.apache.org
>>>> CC: dev@opennlp.apache.org; dev@lucene.apache.org; dev@tika.apache.org;
>>>> common-dev@hadoop.apache.org
>>>> Subject: Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine
>>>> Translation Toolkit
>>>> Date: Tue, 19 Jan 2016 05:58:26 +0000
>>>>
>>>> Great Hen, we’d love to have you on board as a mentor! Please
>>>> add yourself to the proposal on the wiki.
>>>>
>>>> Anyone else have interest in Machine Translation? Any OpenNLP folks,
>>>> Hadoop folks, Tika, or Lucene folks? CC’ing the dev lists for visibility
>>>> please feel free to reply to general@i.a.o.
>>>>
>>>> I’ll leave the DISCUSS thread open for a few more days.
>>>>
>>>> Cheers,
>>>> Chris
>>>>
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> Chris Mattmann, Ph.D.
>>>> Chief Architect
>>>> Instrument Software and Science Data Systems Section (398)
>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>> Office: 168-519, Mailstop: 168-527
>>>> Email: chris.a.mattmann@nasa.gov
>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> Adjunct Associate Professor, Computer Science Department
>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Henri Yandell <ba...@apache.org>
>>>> Reply-To: "general@incubator.apache.org" <ge...@incubator.apache.org>
>>>> Date: Monday, January 18, 2016 at 7:57 PM
>>>> To: jpluser <ch...@jpl.nasa.gov>,
>>>> "general@incubator.apache.org" <ge...@incubator.apache.org>
>>>> Subject: Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine
>>>> Translation Toolkit
>>>>
>>>>> Non-binding +1 to Joshua joining the Incubator. I'd be interested in
>>>>> mentoring.
>>>>>
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: jpluser <ch...@jpl.nasa.gov>
>>>>>> Reply-To: "general@incubator.apache.org"
>>>> <ge...@incubator.apache.org>
>>>>>> Date: Tuesday, January 12, 2016 at 10:56 PM
>>>>>> To: "general@incubator.apache.org" <ge...@incubator.apache.org>
>>>>>> Cc: "post@cs.jhu.edu" <po...@cs.jhu.edu>
>>>>>> Subject: [DISCUSS] Apache Joshua Incubator Proposal - Machine
>>>>>> Translation
>>>>>> Toolkit
>>>>>>
>>>>>>> Hi Everyone,
>>>>>>>
>>>>>>> Please find attached for your viewing pleasure a proposed new
>>>> project,
>>>>>>> Apache Joshua, a statistical machine translation toolkit. The
>>>> proposal
>>>>>>> is in wiki draft form at:
>>>>>> https://wiki.apache.org/incubator/JoshuaProposal
>>>>>>>
>>>>>>> Proposal text is copied below. I’ll leave the discussion open for a
>>>>>> week
>>>>>>> and we are interested in folks who would like to be initial
>>>> committers
>>>>>>> and mentors. Please discuss here on the thread.
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Chris (Champion)
>>>>>>>
>>>>>>> ———
>>>>>>>
>>>>>>> = Joshua Proposal =
>>>>>>>
>>>>>>> == Abstract ==
>>>>>>> [[joshua-decoder.org|Joshua]] is an open-source statistical machine
>>>>>>> translation toolkit. It includes a Java-based decoder for
>>>> translating
>>>>>> with
>>>>>>> phrase-based, hierarchical, and syntax-based translation models, a
>>>>>>> Hadoop-based grammar extractor (Thrax), and an extensive set of
>>>> tools
>>>>>> and
>>>>>>> scripts for training and evaluating new models from parallel text.
>>>>>>>
>>>>>>> == Proposal ==
>>>>>>> Joshua is a state of the art statistical machine translation system
>>>>>> that
>>>>>>> provides a number of features:
>>>>>>>
>>>>>>> * Support for the two main paradigms in statistical machine
>>>>>> translation:
>>>>>>> phrase-based and hierarchical / syntactic.
>>>>>>> * A sparse feature API that makes it easy to add new feature
>>>> templates
>>>>>>> supporting millions of features
>>>>>>> * Native implementations of many tuners (MERT, MIRA, PRO, and
>>>> AdaGrad)
>>>>>>> * Support for lattice decoding, allowing upstream NLP tools to
>>>> expose
>>>>>>> their hypothesis space to the MT system
>>>>>>> * An efficient representation for models, allowing for quick
>>>> loading
>>>>>> of
>>>>>>> multi-gigabyte model files
>>>>>>> * Fast decoding speed (on par with Moses and mtplz)
>>>>>>> * Language packs — precompiled models that allow the decoder to be
>>>>>> run as
>>>>>>> a black box
>>>>>>> * Thrax, a Hadoop-based tool for learning translation models from
>>>>>>> parallel text
>>>>>>> * A suite of tools for constructing new models for any language
>>>> pair
>>>>>> for
>>>>>>> which sufficient training data exists
>>>>>>>
>>>>>>> == Background and Rationale ==
>>>>>>> A number of factors make this a good time for an Apache project
>>>>>> focused on
>>>>>>> machine translation (MT): the quality of MT output (for many
>>>> language
>>>>>>> pairs); the average computing resources available on computers,
>>>>>> relative
>>>>>>> to the needs of MT systems; and the availability of a number of
>>>>>>> high-quality toolkits, together with a large base of researchers
>>>>>> working
>>>>>>> on them.
>>>>>>>
>>>>>>> Over the past decade, machine translation (MT; the automatic
>>>>>> translation
>>>>>>> of one human language to another) has become a reality. The research
>>>>>> into
>>>>>>> statistical approaches to translation that began in the early
>>>> nineties,
>>>>>>> together with the availability of large amounts of training data,
>>>> and
>>>>>>> better computing infrastructure, have all come together to produce
>>>>>>> translations results that are “good enough” for a large set of
>>>> language
>>>>>>> pairs and use cases. Free services like
>>>>>>> [[https://www.bing.com/translator|Bing Translator]] and
>>>>>>> [[https://translate.google.com|Google Translate]] have made these
>>>>>> services
>>>>>>> available to the average person through direct interfaces and
>>>> through
>>>>>>> tools like browser plugins, and sites across the world with higher
>>>>>>> translation needs use them to translate their pages through
>>>>>> automatically.
>>>>>>>
>>>>>>> MT does not require the infrastructure of large corporations in
>>>> order
>>>>>> to
>>>>>>> produce feasible output. Machine translation can be
>>>> resource-intensive,
>>>>>>> but need not be prohibitively so. Disk and memory usage are mostly a
>>>>>>> matter of model size, which for most language pairs is a few
>>>> gigabytes
>>>>>> at
>>>>>>> most, at which size models can provide coverage on the order of
>>>> tens or
>>>>>>> even hundreds of thousands of words in the input and output
>>>> languages.
>>>>>> The
>>>>>>> computational complexity of the algorithms used to search for
>>>>>> translations
>>>>>>> of new sentences are typically linear in the number of words in the
>>>>>> input
>>>>>>> sentence, making it possible to run a translation engine on a
>>>> personal
>>>>>>> computer.
>>>>>>>
>>>>>>> The research community has produced many different open source
>>>>>> translation
>>>>>>> projects for a range of programming languages and under a variety of
>>>>>>> licenses. These projects include the core “decoder”, which takes a
>>>>>> model
>>>>>>> and uses it to translate new sentences between the language pair the
>>>>>> model
>>>>>>> was defined for. They also typically include a large set of tools
>>>> that
>>>>>>> enable new models to be built from large sets of example
>>>> translations
>>>>>>> (“parallel data”) and monolingual texts. These toolkits are usually
>>>>>> built
>>>>>>> to support the agendas of the (largely) academic researchers that
>>>> build
>>>>>>> them: the repeated cycle of building new models, tuning model
>>>>>> parameters
>>>>>>> against development data, and evaluating them against held-out test
>>>>>> data,
>>>>>>> using standard metrics for testing the quality of MT output.
>>>>>>>
>>>>>>> Together, these three factors—the quality of machine translation
>>>>>> output,
>>>>>>> the feasibility of translating on standard computers, and the
>>>>>> availability
>>>>>>> of tools to build models—make it reasonable for the end users to use
>>>>>> MT as
>>>>>>> a black-box service, and to run it on their personal machine.
>>>>>>>
>>>>>>> These factors make it a good time for an organization with the
>>>> status
>>>>>> of
>>>>>>> the Apache Foundation to host a machine translation project.
>>>>>>>
>>>>>>> == Current Status ==
>>>>>>> Joshua was originally ported from David Chiang’s Python
>>>> implementation
>>>>>> of
>>>>>>> Hiero by Zhifei Li, while he was a Ph.D. student at Johns Hopkins
>>>>>>> University. The current version is maintained by Matt Post at Johns
>>>>>>> Hopkins’ Human Language Technology Center of Excellence. Joshua has
>>>>>> made
>>>>>>> many releases with a list of over 20 source code tags. The last
>>>>>> release of
>>>>>>> Joshua was 6.0.5 on November 5th, 2015.
>>>>>>>
>>>>>>> == Meritocracy ==
>>>>>>> The current developers are familiar with meritocratic open source
>>>>>>> development at Apache. Apache was chosen specifically because we
>>>> want
>>>>>> to
>>>>>>> encourage this style of development for the project.
>>>>>>>
>>>>>>> == Community ==
>>>>>>> Joshua is used widely across the world. Perhaps its biggest (known)
>>>>>>> research / industrial user is the Amazon research group in Berlin.
>>>>>> Another
>>>>>>> user is the US Army Research Lab. No formal census has been
>>>> undertaken,
>>>>>>> but posts to the Joshua technical support mailing list, along with
>>>> the
>>>>>>> occasional contributions, suggest small research and academic
>>>>>> communities
>>>>>>> spread across the world, many of them in India.
>>>>>>>
>>>>>>> During incubation, we will explicitly seek to increase our usage
>>>> across
>>>>>>> the board, including academic research, industry, and other end
>>>> users
>>>>>>> interested in statistical machine translation.
>>>>>>>
>>>>>>> == Core Developers ==
>>>>>>> The current set of core developers is fairly small, having fallen
>>>> with
>>>>>> the
>>>>>>> graduation from Johns Hopkins of some core student participants.
>>>>>> However,
>>>>>>> Joshua is used fairly widely, as mentioned above, and there remains
>>>> a
>>>>>>> commitment from the principal researcher at Johns Hopkins to
>>>> continue
>>>>>> to
>>>>>>> use and develop it. Joshua has seen a number of new community
>>>> members
>>>>>>> become interested recently due to a potential for its projected use
>>>> in
>>>>>> a
>>>>>>> number of ongoing DARPA projects such as XDATA and Memex.
>>>>>>>
>>>>>>> == Alignment ==
>>>>>>> Joshua is currently Copyright (c) 2015, Johns Hopkins University All
>>>>>>> rights reserved and licensed under BSD 2-clause license. It would of
>>>>>>> course be the intention to relicense this code under AL2.0 which
>>>> would
>>>>>>> permit expanded and increased use of the software within Apache
>>>>>> projects.
>>>>>>> There is currently an ongoing effort within the Apache Tika
>>>> community
>>>>>> to
>>>>>>> utilize Joshua within Tika’s Translate API, see
>>>>>>> [[https://issues.apache.org/jira/browse/TIKA-1343|TIKA-1343]].
>>>>>>>
>>>>>>> == Known Risks ==
>>>>>>>
>>>>>>> === Orphaned products ===
>>>>>>> At the moment, regular contributions are made by a single
>>>> contributor,
>>>>>> the
>>>>>>> lead maintainer. He (Matt Post) plans to continue development for
>>>> the
>>>>>> next
>>>>>>> few years, but it is still a single point of failure, since the
>>>>>> graduate
>>>>>>> students who worked on the project have moved on to jobs, mostly in
>>>>>>> industry. However, our goal is to help that process by growing the
>>>>>>> community in Apache, and at least in growing the community with
>>>> users
>>>>>> and
>>>>>>> participants from NASA JPL.
>>>>>>>
>>>>>>> === Inexperience with Open Source ===
>>>>>>> The team both at Johns Hopkins and NASA JPL have experience with
>>>> many
>>>>>> OSS
>>>>>>> software projects at Apache and elsewhere. We understand "how it
>>>> works"
>>>>>>> here at the foundation.
>>>>>>>
>>>>>>>
>>>>>>> == Relationships with Other Apache Products ==
>>>>>>> Joshua includes dependences on Hadoop, and also is included as a
>>>>>> plugin in
>>>>>>> Apache Tika. We are also interested in coordinating with other
>>>> projects
>>>>>>> including Spark, and other projects needing MT services for language
>>>>>>> translation.
>>>>>>>
>>>>>>> == Developers ==
>>>>>>> Joshua only has one regular developer who is employed by Johns
>>>> Hopkins
>>>>>>> University. NASA JPL (Mattmann and McGibbney) have been contributing
>>>>>>> lately including a Brew formula and other contributions to the
>>>> project
>>>>>>> through the DARPA XDATA and Memex programs.
>>>>>>>
>>>>>>> == Documentation ==
>>>>>>> Documentation and publications related to Joshua can be found at
>>>>>>> joshua-decoder.org. The source for the Joshua documentation is
>>>>>> currently
>>>>>>> hosted on Github at
>>>>>>> https://github.com/joshua-decoder/joshua-decoder.github.com
>>>>>>>
>>>>>>> == Initial Source ==
>>>>>>> Current source resides at Github: github.com/joshua-decoder/joshua
>>>> (the
>>>>>>> main decoder and toolkit) and github.com/joshua-decoder/thrax (the
>>>>>> grammar
>>>>>>> extraction tool).
>>>>>>>
>>>>>>> == External Dependencies ==
>>>>>>> Joshua has a number of external dependencies. Only BerkeleyLM
>>>> (Apache
>>>>>> 2.0)
>>>>>>> and KenLM (LGPG 2.1) are run-time decoder dependencies (one of
>>>> which is
>>>>>>> needed for translating sentences with pre-built models). The rest
>>>> are
>>>>>>> dependencies for the build system and pipeline, used for
>>>> constructing
>>>>>> and
>>>>>>> training new models from parallel text.
>>>>>>>
>>>>>>> Apache projects:
>>>>>>> * Ant
>>>>>>> * Hadoop
>>>>>>> * Commons
>>>>>>> * Maven
>>>>>>> * Ivy
>>>>>>>
>>>>>>> There are also a number of other open-source projects with various
>>>>>>> licenses that the project depends on both dynamically (runtime), and
>>>>>>> statically.
>>>>>>>
>>>>>>> === GNU GPL 2 ===
>>>>>>> * Berkeley Aligner: https://code.google.com/p/berkeleyaligner/
>>>>>>>
>>>>>>> === LGPG 2.1 ===
>>>>>>> * KenLM: github.com/kpu/kenlm
>>>>>>>
>>>>>>> === Apache 2.0 ===
>>>>>>> * BerkeleyLM: https://code.google.com/p/berkeleylm/
>>>>>>>
>>>>>>> === GNU GPL ===
>>>>>>> * GIZA++: http://www.statmt.org/moses/giza/GIZA++.html
>>>>>>>
>>>>>>> == Required Resources ==
>>>>>>> * Mailing Lists
>>>>>>> * private@joshua.incubator.apache.org
>>>>>>> * dev@joshua.incubator.apache.org
>>>>>>> * commits@joshua.incubator.apache.org
>>>>>>>
>>>>>>> * Git Repos
>>>>>>> * https://git-wip-us.apache.org/repos/asf/joshua.git
>>>>>>>
>>>>>>> * Issue Tracking
>>>>>>> * JIRA Joshua (JOSHUA)
>>>>>>>
>>>>>>> * Continuous Integration
>>>>>>> * Jenkins builds on https://builds.apache.org/
>>>>>>>
>>>>>>> * Web
>>>>>>> * http://joshua.incubator.apache.org/
>>>>>>> * wiki at http://cwiki.apache.org
>>>>>>>
>>>>>>> == Initial Committers ==
>>>>>>> The following is a list of the planned initial Apache committers
>>>> (the
>>>>>>> active subset of the committers for the current repository on
>>>> Github).
>>>>>>>
>>>>>>> * Matt Post (post@cs.jhu.edu)
>>>>>>> * Lewis John McGibbney (lewismc@apache.org)
>>>>>>> * Chris Mattmann (mattmann@apache.org)
>>>>>>>
>>>>>>> == Affiliations ==
>>>>>>>
>>>>>>> * Johns Hopkins University
>>>>>>> * Matt Post
>>>>>>>
>>>>>>> * NASA JPL
>>>>>>> * Chris Mattmann
>>>>>>> * Lewis John McGibbney
>>>>>>>
>>>>>>>
>>>>>>> == Sponsors ==
>>>>>>> === Champion ===
>>>>>>> * Chris Mattmann (NASA/JPL)
>>>>>>>
>>>>>>> === Nominated Mentors ===
>>>>>>> * Paul Ramirez
>>>>>>> * Lewis John McGibbney
>>>>>>> * Chris Mattmann
>>>>>>>
>>>>>>> == Sponsoring Entity ==
>>>>>>> The Apache Incubator
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>> Chris Mattmann, Ph.D.
>>>>>>> Chief Architect
>>>>>>> Instrument Software and Science Data Systems Section (398)
>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>> Office: 168-519, Mailstop: 168-527
>>>>>>> Email: chris.a.mattmann@nasa.gov
>>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>> Adjunct Associate Professor, Computer Science Department
>>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>
>>>>>>> ?B?KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK
>>>>>>> KC
>>>>>>> B?
>>>>
>>>>>>> ???[??X???X?K??K[XZ[????[?\?[?][??X???X?P?[??X?]???\?X??K???B????Y??]?
>>>>>>> [?
>>>>>>> [?
>>>>>>> ???[X[?????K[XZ[????[?\?[?Z?[???[??X?]???\?X??K???B
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine Translation Toolkit
Posted by ben gao <ba...@gmail.com>.
Yes, I am.
On Tue, Jan 19, 2016, 11:56 AM Mattmann, Chris A (3980) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Thanks Martin, filed:
>
> https://github.com/joshua-decoder/joshua/issues/239
>
>
> Are you interested in joining in the Incubation efforts?
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
> -----Original Message-----
> From: Martin Gainty <mg...@hotmail.com>
> Reply-To: "dev@opennlp.apache.org" <de...@opennlp.apache.org>
> Date: Tuesday, January 19, 2016 at 7:57 AM
> To: "dev@opennlp.apache.org" <de...@opennlp.apache.org>, Lewis McGibbney
> <le...@gmail.com>
> Subject: RE: [DISCUSS] Apache Joshua Incubator Proposal - Machine
> Translation Toolkit
>
> >dependency addition to Joshua-Decoder/Joshua/pom.xml
> ><!-- MCG added args4j for
> >/Joshua-Decoder/Joshua/src/joshua/decoder/ff/tm/CreateGlueGrammar.java:[17
> >,26] package org.kohsuke.args4j does not exist error --> <dependency>
> > <groupId>args4j</groupId> <artifactId>args4j</artifactId>
> ><version>2.32</version> </dependency>
> >Joshua-Decoder/Joshua committer please add this dependency to pom.xml
> >thank you/
> >Martin
> >______________________________________________
> >
> >
> >
> >> From: chris.a.mattmann@jpl.nasa.gov
> >> To: general@incubator.apache.org
> >> CC: dev@opennlp.apache.org; dev@lucene.apache.org; dev@tika.apache.org;
> >>common-dev@hadoop.apache.org
> >> Subject: Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine
> >>Translation Toolkit
> >> Date: Tue, 19 Jan 2016 05:58:26 +0000
> >>
> >> Great Hen, we’d love to have you on board as a mentor! Please
> >> add yourself to the proposal on the wiki.
> >>
> >> Anyone else have interest in Machine Translation? Any OpenNLP folks,
> >> Hadoop folks, Tika, or Lucene folks? CC’ing the dev lists for visibility
> >> please feel free to reply to general@i.a.o.
> >>
> >> I’ll leave the DISCUSS thread open for a few more days.
> >>
> >> Cheers,
> >> Chris
> >>
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> Chris Mattmann, Ph.D.
> >> Chief Architect
> >> Instrument Software and Science Data Systems Section (398)
> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >> Office: 168-519, Mailstop: 168-527
> >> Email: chris.a.mattmann@nasa.gov
> >> WWW: http://sunset.usc.edu/~mattmann/
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> Adjunct Associate Professor, Computer Science Department
> >> University of Southern California, Los Angeles, CA 90089 USA
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>
> >>
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: Henri Yandell <ba...@apache.org>
> >> Reply-To: "general@incubator.apache.org" <ge...@incubator.apache.org>
> >> Date: Monday, January 18, 2016 at 7:57 PM
> >> To: jpluser <ch...@jpl.nasa.gov>,
> >> "general@incubator.apache.org" <ge...@incubator.apache.org>
> >> Subject: Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine
> >> Translation Toolkit
> >>
> >> >Non-binding +1 to Joshua joining the Incubator. I'd be interested in
> >> >mentoring.
> >> >
> >> >
> >> >> -----Original Message-----
> >> >> From: jpluser <ch...@jpl.nasa.gov>
> >> >> Reply-To: "general@incubator.apache.org"
> >><ge...@incubator.apache.org>
> >> >> Date: Tuesday, January 12, 2016 at 10:56 PM
> >> >> To: "general@incubator.apache.org" <ge...@incubator.apache.org>
> >> >> Cc: "post@cs.jhu.edu" <po...@cs.jhu.edu>
> >> >> Subject: [DISCUSS] Apache Joshua Incubator Proposal - Machine
> >> >>Translation
> >> >> Toolkit
> >> >>
> >> >> >Hi Everyone,
> >> >> >
> >> >> >Please find attached for your viewing pleasure a proposed new
> >>project,
> >> >> >Apache Joshua, a statistical machine translation toolkit. The
> >>proposal
> >> >> >is in wiki draft form at:
> >> >> https://wiki.apache.org/incubator/JoshuaProposal
> >> >> >
> >> >> >Proposal text is copied below. I’ll leave the discussion open for a
> >> >>week
> >> >> >and we are interested in folks who would like to be initial
> >>committers
> >> >> >and mentors. Please discuss here on the thread.
> >> >> >
> >> >> >Thanks!
> >> >> >
> >> >> >Cheers,
> >> >> >Chris (Champion)
> >> >> >
> >> >> >———
> >> >> >
> >> >> >= Joshua Proposal =
> >> >> >
> >> >> >== Abstract ==
> >> >> >[[joshua-decoder.org|Joshua]] is an open-source statistical machine
> >> >> >translation toolkit. It includes a Java-based decoder for
> >>translating
> >> >>with
> >> >> >phrase-based, hierarchical, and syntax-based translation models, a
> >> >> >Hadoop-based grammar extractor (Thrax), and an extensive set of
> >>tools
> >> >>and
> >> >> >scripts for training and evaluating new models from parallel text.
> >> >> >
> >> >> >== Proposal ==
> >> >> >Joshua is a state of the art statistical machine translation system
> >> >>that
> >> >> >provides a number of features:
> >> >> >
> >> >> > * Support for the two main paradigms in statistical machine
> >> >>translation:
> >> >> >phrase-based and hierarchical / syntactic.
> >> >> > * A sparse feature API that makes it easy to add new feature
> >>templates
> >> >> >supporting millions of features
> >> >> > * Native implementations of many tuners (MERT, MIRA, PRO, and
> >>AdaGrad)
> >> >> > * Support for lattice decoding, allowing upstream NLP tools to
> >>expose
> >> >> >their hypothesis space to the MT system
> >> >> > * An efficient representation for models, allowing for quick
> >>loading
> >> >>of
> >> >> >multi-gigabyte model files
> >> >> > * Fast decoding speed (on par with Moses and mtplz)
> >> >> > * Language packs — precompiled models that allow the decoder to be
> >> >>run as
> >> >> >a black box
> >> >> > * Thrax, a Hadoop-based tool for learning translation models from
> >> >> >parallel text
> >> >> > * A suite of tools for constructing new models for any language
> >>pair
> >> >>for
> >> >> >which sufficient training data exists
> >> >> >
> >> >> >== Background and Rationale ==
> >> >> >A number of factors make this a good time for an Apache project
> >> >>focused on
> >> >> >machine translation (MT): the quality of MT output (for many
> >>language
> >> >> >pairs); the average computing resources available on computers,
> >> >>relative
> >> >> >to the needs of MT systems; and the availability of a number of
> >> >> >high-quality toolkits, together with a large base of researchers
> >> >>working
> >> >> >on them.
> >> >> >
> >> >> >Over the past decade, machine translation (MT; the automatic
> >> >>translation
> >> >> >of one human language to another) has become a reality. The research
> >> >>into
> >> >> >statistical approaches to translation that began in the early
> >>nineties,
> >> >> >together with the availability of large amounts of training data,
> >>and
> >> >> >better computing infrastructure, have all come together to produce
> >> >> >translations results that are “good enough” for a large set of
> >>language
> >> >> >pairs and use cases. Free services like
> >> >> >[[https://www.bing.com/translator|Bing Translator]] and
> >> >> >[[https://translate.google.com|Google Translate]] have made these
> >> >> services
> >> >> >available to the average person through direct interfaces and
> >>through
> >> >> >tools like browser plugins, and sites across the world with higher
> >> >> >translation needs use them to translate their pages through
> >> >>automatically.
> >> >> >
> >> >> >MT does not require the infrastructure of large corporations in
> >>order
> >> >>to
> >> >> >produce feasible output. Machine translation can be
> >>resource-intensive,
> >> >> >but need not be prohibitively so. Disk and memory usage are mostly a
> >> >> >matter of model size, which for most language pairs is a few
> >>gigabytes
> >> >>at
> >> >> >most, at which size models can provide coverage on the order of
> >>tens or
> >> >> >even hundreds of thousands of words in the input and output
> >>languages.
> >> >>The
> >> >> >computational complexity of the algorithms used to search for
> >> >>translations
> >> >> >of new sentences are typically linear in the number of words in the
> >> >>input
> >> >> >sentence, making it possible to run a translation engine on a
> >>personal
> >> >> >computer.
> >> >> >
> >> >> >The research community has produced many different open source
> >> >>translation
> >> >> >projects for a range of programming languages and under a variety of
> >> >> >licenses. These projects include the core “decoder”, which takes a
> >> >>model
> >> >> >and uses it to translate new sentences between the language pair the
> >> >>model
> >> >> >was defined for. They also typically include a large set of tools
> >>that
> >> >> >enable new models to be built from large sets of example
> >>translations
> >> >> >(“parallel data”) and monolingual texts. These toolkits are usually
> >> >>built
> >> >> >to support the agendas of the (largely) academic researchers that
> >>build
> >> >> >them: the repeated cycle of building new models, tuning model
> >> >>parameters
> >> >> >against development data, and evaluating them against held-out test
> >> >>data,
> >> >> >using standard metrics for testing the quality of MT output.
> >> >> >
> >> >> >Together, these three factors—the quality of machine translation
> >> >>output,
> >> >> >the feasibility of translating on standard computers, and the
> >> >>availability
> >> >> >of tools to build models—make it reasonable for the end users to use
> >> >>MT as
> >> >> >a black-box service, and to run it on their personal machine.
> >> >> >
> >> >> >These factors make it a good time for an organization with the
> >>status
> >> >>of
> >> >> >the Apache Foundation to host a machine translation project.
> >> >> >
> >> >> >== Current Status ==
> >> >> >Joshua was originally ported from David Chiang’s Python
> >>implementation
> >> >>of
> >> >> >Hiero by Zhifei Li, while he was a Ph.D. student at Johns Hopkins
> >> >> >University. The current version is maintained by Matt Post at Johns
> >> >> >Hopkins’ Human Language Technology Center of Excellence. Joshua has
> >> >>made
> >> >> >many releases with a list of over 20 source code tags. The last
> >> >>release of
> >> >> >Joshua was 6.0.5 on November 5th, 2015.
> >> >> >
> >> >> >== Meritocracy ==
> >> >> >The current developers are familiar with meritocratic open source
> >> >> >development at Apache. Apache was chosen specifically because we
> >>want
> >> >>to
> >> >> >encourage this style of development for the project.
> >> >> >
> >> >> >== Community ==
> >> >> >Joshua is used widely across the world. Perhaps its biggest (known)
> >> >> >research / industrial user is the Amazon research group in Berlin.
> >> >>Another
> >> >> >user is the US Army Research Lab. No formal census has been
> >>undertaken,
> >> >> >but posts to the Joshua technical support mailing list, along with
> >>the
> >> >> >occasional contributions, suggest small research and academic
> >> >>communities
> >> >> >spread across the world, many of them in India.
> >> >> >
> >> >> >During incubation, we will explicitly seek to increase our usage
> >>across
> >> >> >the board, including academic research, industry, and other end
> >>users
> >> >> >interested in statistical machine translation.
> >> >> >
> >> >> >== Core Developers ==
> >> >> >The current set of core developers is fairly small, having fallen
> >>with
> >> >>the
> >> >> >graduation from Johns Hopkins of some core student participants.
> >> >>However,
> >> >> >Joshua is used fairly widely, as mentioned above, and there remains
> >>a
> >> >> >commitment from the principal researcher at Johns Hopkins to
> >>continue
> >> >>to
> >> >> >use and develop it. Joshua has seen a number of new community
> >>members
> >> >> >become interested recently due to a potential for its projected use
> >>in
> >> >>a
> >> >> >number of ongoing DARPA projects such as XDATA and Memex.
> >> >> >
> >> >> >== Alignment ==
> >> >> >Joshua is currently Copyright (c) 2015, Johns Hopkins University All
> >> >> >rights reserved and licensed under BSD 2-clause license. It would of
> >> >> >course be the intention to relicense this code under AL2.0 which
> >>would
> >> >> >permit expanded and increased use of the software within Apache
> >> >>projects.
> >> >> >There is currently an ongoing effort within the Apache Tika
> >>community
> >> >>to
> >> >> >utilize Joshua within Tika’s Translate API, see
> >> >> >[[https://issues.apache.org/jira/browse/TIKA-1343|TIKA-1343]].
> >> >> >
> >> >> >== Known Risks ==
> >> >> >
> >> >> >=== Orphaned products ===
> >> >> >At the moment, regular contributions are made by a single
> >>contributor,
> >> >>the
> >> >> >lead maintainer. He (Matt Post) plans to continue development for
> >>the
> >> >>next
> >> >> >few years, but it is still a single point of failure, since the
> >> >>graduate
> >> >> >students who worked on the project have moved on to jobs, mostly in
> >> >> >industry. However, our goal is to help that process by growing the
> >> >> >community in Apache, and at least in growing the community with
> >>users
> >> >>and
> >> >> >participants from NASA JPL.
> >> >> >
> >> >> >=== Inexperience with Open Source ===
> >> >> >The team both at Johns Hopkins and NASA JPL have experience with
> >>many
> >> >>OSS
> >> >> >software projects at Apache and elsewhere. We understand "how it
> >>works"
> >> >> >here at the foundation.
> >> >> >
> >> >> >
> >> >> >== Relationships with Other Apache Products ==
> >> >> >Joshua includes dependences on Hadoop, and also is included as a
> >> >>plugin in
> >> >> >Apache Tika. We are also interested in coordinating with other
> >>projects
> >> >> >including Spark, and other projects needing MT services for language
> >> >> >translation.
> >> >> >
> >> >> >== Developers ==
> >> >> >Joshua only has one regular developer who is employed by Johns
> >>Hopkins
> >> >> >University. NASA JPL (Mattmann and McGibbney) have been contributing
> >> >> >lately including a Brew formula and other contributions to the
> >>project
> >> >> >through the DARPA XDATA and Memex programs.
> >> >> >
> >> >> >== Documentation ==
> >> >> >Documentation and publications related to Joshua can be found at
> >> >> >joshua-decoder.org. The source for the Joshua documentation is
> >> >>currently
> >> >> >hosted on Github at
> >> >> >https://github.com/joshua-decoder/joshua-decoder.github.com
> >> >> >
> >> >> >== Initial Source ==
> >> >> >Current source resides at Github: github.com/joshua-decoder/joshua
> >>(the
> >> >> >main decoder and toolkit) and github.com/joshua-decoder/thrax (the
> >> >> grammar
> >> >> >extraction tool).
> >> >> >
> >> >> >== External Dependencies ==
> >> >> >Joshua has a number of external dependencies. Only BerkeleyLM
> >>(Apache
> >> >>2.0)
> >> >> >and KenLM (LGPG 2.1) are run-time decoder dependencies (one of
> >>which is
> >> >> >needed for translating sentences with pre-built models). The rest
> >>are
> >> >> >dependencies for the build system and pipeline, used for
> >>constructing
> >> >>and
> >> >> >training new models from parallel text.
> >> >> >
> >> >> >Apache projects:
> >> >> > * Ant
> >> >> > * Hadoop
> >> >> > * Commons
> >> >> > * Maven
> >> >> > * Ivy
> >> >> >
> >> >> >There are also a number of other open-source projects with various
> >> >> >licenses that the project depends on both dynamically (runtime), and
> >> >> >statically.
> >> >> >
> >> >> >=== GNU GPL 2 ===
> >> >> > * Berkeley Aligner: https://code.google.com/p/berkeleyaligner/
> >> >> >
> >> >> >=== LGPG 2.1 ===
> >> >> > * KenLM: github.com/kpu/kenlm
> >> >> >
> >> >> >=== Apache 2.0 ===
> >> >> > * BerkeleyLM: https://code.google.com/p/berkeleylm/
> >> >> >
> >> >> >=== GNU GPL ===
> >> >> > * GIZA++: http://www.statmt.org/moses/giza/GIZA++.html
> >> >> >
> >> >> >== Required Resources ==
> >> >> > * Mailing Lists
> >> >> > * private@joshua.incubator.apache.org
> >> >> > * dev@joshua.incubator.apache.org
> >> >> > * commits@joshua.incubator.apache.org
> >> >> >
> >> >> > * Git Repos
> >> >> > * https://git-wip-us.apache.org/repos/asf/joshua.git
> >> >> >
> >> >> > * Issue Tracking
> >> >> > * JIRA Joshua (JOSHUA)
> >> >> >
> >> >> > * Continuous Integration
> >> >> > * Jenkins builds on https://builds.apache.org/
> >> >> >
> >> >> > * Web
> >> >> > * http://joshua.incubator.apache.org/
> >> >> > * wiki at http://cwiki.apache.org
> >> >> >
> >> >> >== Initial Committers ==
> >> >> >The following is a list of the planned initial Apache committers
> >>(the
> >> >> >active subset of the committers for the current repository on
> >>Github).
> >> >> >
> >> >> > * Matt Post (post@cs.jhu.edu)
> >> >> > * Lewis John McGibbney (lewismc@apache.org)
> >> >> > * Chris Mattmann (mattmann@apache.org)
> >> >> >
> >> >> >== Affiliations ==
> >> >> >
> >> >> > * Johns Hopkins University
> >> >> > * Matt Post
> >> >> >
> >> >> > * NASA JPL
> >> >> > * Chris Mattmann
> >> >> > * Lewis John McGibbney
> >> >> >
> >> >> >
> >> >> >== Sponsors ==
> >> >> >=== Champion ===
> >> >> > * Chris Mattmann (NASA/JPL)
> >> >> >
> >> >> >=== Nominated Mentors ===
> >> >> > * Paul Ramirez
> >> >> > * Lewis John McGibbney
> >> >> > * Chris Mattmann
> >> >> >
> >> >> >== Sponsoring Entity ==
> >> >> >The Apache Incubator
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> >> >Chris Mattmann, Ph.D.
> >> >> >Chief Architect
> >> >> >Instrument Software and Science Data Systems Section (398)
> >> >> >NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >> >> >Office: 168-519, Mailstop: 168-527
> >> >> >Email: chris.a.mattmann@nasa.gov
> >> >> >WWW: http://sunset.usc.edu/~mattmann/
> >> >> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> >> >Adjunct Associate Professor, Computer Science Department
> >> >> >University of Southern California, Los Angeles, CA 90089 USA
> >> >> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> >> >
> >> >> >
> >> >> >
> >> >>
> >>
> >>>>>?B�KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK
> >>>>>KC
> >> >>>B�
> >> >>
> >>
> >>>>>?�?[��X��ܚX�K??K[XZ[?�?�[�\�[?][��X��ܚX�P?[��X�]?܋�\?X�?K�ܙ�B��܈?Y??]?
> >>>>>[ۘ
> >> >>>[?
> >> >> >?��[X[�?�??K[XZ[?�?�[�\�[?Z?[???[��X�]?܋�\?X�?K�ܙ�B
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: dev-help@lucene.apache.org
> >>
> >
>
>
Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine
Translation Toolkit
Posted by "Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov>.
Thanks Martin, filed:
https://github.com/joshua-decoder/joshua/issues/239
Are you interested in joining in the Incubation efforts?
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Martin Gainty <mg...@hotmail.com>
Reply-To: "dev@opennlp.apache.org" <de...@opennlp.apache.org>
Date: Tuesday, January 19, 2016 at 7:57 AM
To: "dev@opennlp.apache.org" <de...@opennlp.apache.org>, Lewis McGibbney
<le...@gmail.com>
Subject: RE: [DISCUSS] Apache Joshua Incubator Proposal - Machine
Translation Toolkit
>dependency addition to Joshua-Decoder/Joshua/pom.xml
><!-- MCG added args4j for
>/Joshua-Decoder/Joshua/src/joshua/decoder/ff/tm/CreateGlueGrammar.java:[17
>,26] package org.kohsuke.args4j does not exist error --> <dependency>
> <groupId>args4j</groupId> <artifactId>args4j</artifactId>
><version>2.32</version> </dependency>
>Joshua-Decoder/Joshua committer please add this dependency to pom.xml
>thank you/
>Martin
>______________________________________________
>
>
>
>> From: chris.a.mattmann@jpl.nasa.gov
>> To: general@incubator.apache.org
>> CC: dev@opennlp.apache.org; dev@lucene.apache.org; dev@tika.apache.org;
>>common-dev@hadoop.apache.org
>> Subject: Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine
>>Translation Toolkit
>> Date: Tue, 19 Jan 2016 05:58:26 +0000
>>
>> Great Hen, we’d love to have you on board as a mentor! Please
>> add yourself to the proposal on the wiki.
>>
>> Anyone else have interest in Machine Translation? Any OpenNLP folks,
>> Hadoop folks, Tika, or Lucene folks? CC’ing the dev lists for visibility
>> please feel free to reply to general@i.a.o.
>>
>> I’ll leave the DISCUSS thread open for a few more days.
>>
>> Cheers,
>> Chris
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Chief Architect
>> Instrument Software and Science Data Systems Section (398)
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 168-519, Mailstop: 168-527
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Associate Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Henri Yandell <ba...@apache.org>
>> Reply-To: "general@incubator.apache.org" <ge...@incubator.apache.org>
>> Date: Monday, January 18, 2016 at 7:57 PM
>> To: jpluser <ch...@jpl.nasa.gov>,
>> "general@incubator.apache.org" <ge...@incubator.apache.org>
>> Subject: Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine
>> Translation Toolkit
>>
>> >Non-binding +1 to Joshua joining the Incubator. I'd be interested in
>> >mentoring.
>> >
>> >
>> >> -----Original Message-----
>> >> From: jpluser <ch...@jpl.nasa.gov>
>> >> Reply-To: "general@incubator.apache.org"
>><ge...@incubator.apache.org>
>> >> Date: Tuesday, January 12, 2016 at 10:56 PM
>> >> To: "general@incubator.apache.org" <ge...@incubator.apache.org>
>> >> Cc: "post@cs.jhu.edu" <po...@cs.jhu.edu>
>> >> Subject: [DISCUSS] Apache Joshua Incubator Proposal - Machine
>> >>Translation
>> >> Toolkit
>> >>
>> >> >Hi Everyone,
>> >> >
>> >> >Please find attached for your viewing pleasure a proposed new
>>project,
>> >> >Apache Joshua, a statistical machine translation toolkit. The
>>proposal
>> >> >is in wiki draft form at:
>> >> https://wiki.apache.org/incubator/JoshuaProposal
>> >> >
>> >> >Proposal text is copied below. I’ll leave the discussion open for a
>> >>week
>> >> >and we are interested in folks who would like to be initial
>>committers
>> >> >and mentors. Please discuss here on the thread.
>> >> >
>> >> >Thanks!
>> >> >
>> >> >Cheers,
>> >> >Chris (Champion)
>> >> >
>> >> >———
>> >> >
>> >> >= Joshua Proposal =
>> >> >
>> >> >== Abstract ==
>> >> >[[joshua-decoder.org|Joshua]] is an open-source statistical machine
>> >> >translation toolkit. It includes a Java-based decoder for
>>translating
>> >>with
>> >> >phrase-based, hierarchical, and syntax-based translation models, a
>> >> >Hadoop-based grammar extractor (Thrax), and an extensive set of
>>tools
>> >>and
>> >> >scripts for training and evaluating new models from parallel text.
>> >> >
>> >> >== Proposal ==
>> >> >Joshua is a state of the art statistical machine translation system
>> >>that
>> >> >provides a number of features:
>> >> >
>> >> > * Support for the two main paradigms in statistical machine
>> >>translation:
>> >> >phrase-based and hierarchical / syntactic.
>> >> > * A sparse feature API that makes it easy to add new feature
>>templates
>> >> >supporting millions of features
>> >> > * Native implementations of many tuners (MERT, MIRA, PRO, and
>>AdaGrad)
>> >> > * Support for lattice decoding, allowing upstream NLP tools to
>>expose
>> >> >their hypothesis space to the MT system
>> >> > * An efficient representation for models, allowing for quick
>>loading
>> >>of
>> >> >multi-gigabyte model files
>> >> > * Fast decoding speed (on par with Moses and mtplz)
>> >> > * Language packs — precompiled models that allow the decoder to be
>> >>run as
>> >> >a black box
>> >> > * Thrax, a Hadoop-based tool for learning translation models from
>> >> >parallel text
>> >> > * A suite of tools for constructing new models for any language
>>pair
>> >>for
>> >> >which sufficient training data exists
>> >> >
>> >> >== Background and Rationale ==
>> >> >A number of factors make this a good time for an Apache project
>> >>focused on
>> >> >machine translation (MT): the quality of MT output (for many
>>language
>> >> >pairs); the average computing resources available on computers,
>> >>relative
>> >> >to the needs of MT systems; and the availability of a number of
>> >> >high-quality toolkits, together with a large base of researchers
>> >>working
>> >> >on them.
>> >> >
>> >> >Over the past decade, machine translation (MT; the automatic
>> >>translation
>> >> >of one human language to another) has become a reality. The research
>> >>into
>> >> >statistical approaches to translation that began in the early
>>nineties,
>> >> >together with the availability of large amounts of training data,
>>and
>> >> >better computing infrastructure, have all come together to produce
>> >> >translations results that are “good enough” for a large set of
>>language
>> >> >pairs and use cases. Free services like
>> >> >[[https://www.bing.com/translator|Bing Translator]] and
>> >> >[[https://translate.google.com|Google Translate]] have made these
>> >> services
>> >> >available to the average person through direct interfaces and
>>through
>> >> >tools like browser plugins, and sites across the world with higher
>> >> >translation needs use them to translate their pages through
>> >>automatically.
>> >> >
>> >> >MT does not require the infrastructure of large corporations in
>>order
>> >>to
>> >> >produce feasible output. Machine translation can be
>>resource-intensive,
>> >> >but need not be prohibitively so. Disk and memory usage are mostly a
>> >> >matter of model size, which for most language pairs is a few
>>gigabytes
>> >>at
>> >> >most, at which size models can provide coverage on the order of
>>tens or
>> >> >even hundreds of thousands of words in the input and output
>>languages.
>> >>The
>> >> >computational complexity of the algorithms used to search for
>> >>translations
>> >> >of new sentences are typically linear in the number of words in the
>> >>input
>> >> >sentence, making it possible to run a translation engine on a
>>personal
>> >> >computer.
>> >> >
>> >> >The research community has produced many different open source
>> >>translation
>> >> >projects for a range of programming languages and under a variety of
>> >> >licenses. These projects include the core “decoder”, which takes a
>> >>model
>> >> >and uses it to translate new sentences between the language pair the
>> >>model
>> >> >was defined for. They also typically include a large set of tools
>>that
>> >> >enable new models to be built from large sets of example
>>translations
>> >> >(“parallel data”) and monolingual texts. These toolkits are usually
>> >>built
>> >> >to support the agendas of the (largely) academic researchers that
>>build
>> >> >them: the repeated cycle of building new models, tuning model
>> >>parameters
>> >> >against development data, and evaluating them against held-out test
>> >>data,
>> >> >using standard metrics for testing the quality of MT output.
>> >> >
>> >> >Together, these three factors—the quality of machine translation
>> >>output,
>> >> >the feasibility of translating on standard computers, and the
>> >>availability
>> >> >of tools to build models—make it reasonable for the end users to use
>> >>MT as
>> >> >a black-box service, and to run it on their personal machine.
>> >> >
>> >> >These factors make it a good time for an organization with the
>>status
>> >>of
>> >> >the Apache Foundation to host a machine translation project.
>> >> >
>> >> >== Current Status ==
>> >> >Joshua was originally ported from David Chiang’s Python
>>implementation
>> >>of
>> >> >Hiero by Zhifei Li, while he was a Ph.D. student at Johns Hopkins
>> >> >University. The current version is maintained by Matt Post at Johns
>> >> >Hopkins’ Human Language Technology Center of Excellence. Joshua has
>> >>made
>> >> >many releases with a list of over 20 source code tags. The last
>> >>release of
>> >> >Joshua was 6.0.5 on November 5th, 2015.
>> >> >
>> >> >== Meritocracy ==
>> >> >The current developers are familiar with meritocratic open source
>> >> >development at Apache. Apache was chosen specifically because we
>>want
>> >>to
>> >> >encourage this style of development for the project.
>> >> >
>> >> >== Community ==
>> >> >Joshua is used widely across the world. Perhaps its biggest (known)
>> >> >research / industrial user is the Amazon research group in Berlin.
>> >>Another
>> >> >user is the US Army Research Lab. No formal census has been
>>undertaken,
>> >> >but posts to the Joshua technical support mailing list, along with
>>the
>> >> >occasional contributions, suggest small research and academic
>> >>communities
>> >> >spread across the world, many of them in India.
>> >> >
>> >> >During incubation, we will explicitly seek to increase our usage
>>across
>> >> >the board, including academic research, industry, and other end
>>users
>> >> >interested in statistical machine translation.
>> >> >
>> >> >== Core Developers ==
>> >> >The current set of core developers is fairly small, having fallen
>>with
>> >>the
>> >> >graduation from Johns Hopkins of some core student participants.
>> >>However,
>> >> >Joshua is used fairly widely, as mentioned above, and there remains
>>a
>> >> >commitment from the principal researcher at Johns Hopkins to
>>continue
>> >>to
>> >> >use and develop it. Joshua has seen a number of new community
>>members
>> >> >become interested recently due to a potential for its projected use
>>in
>> >>a
>> >> >number of ongoing DARPA projects such as XDATA and Memex.
>> >> >
>> >> >== Alignment ==
>> >> >Joshua is currently Copyright (c) 2015, Johns Hopkins University All
>> >> >rights reserved and licensed under BSD 2-clause license. It would of
>> >> >course be the intention to relicense this code under AL2.0 which
>>would
>> >> >permit expanded and increased use of the software within Apache
>> >>projects.
>> >> >There is currently an ongoing effort within the Apache Tika
>>community
>> >>to
>> >> >utilize Joshua within Tika’s Translate API, see
>> >> >[[https://issues.apache.org/jira/browse/TIKA-1343|TIKA-1343]].
>> >> >
>> >> >== Known Risks ==
>> >> >
>> >> >=== Orphaned products ===
>> >> >At the moment, regular contributions are made by a single
>>contributor,
>> >>the
>> >> >lead maintainer. He (Matt Post) plans to continue development for
>>the
>> >>next
>> >> >few years, but it is still a single point of failure, since the
>> >>graduate
>> >> >students who worked on the project have moved on to jobs, mostly in
>> >> >industry. However, our goal is to help that process by growing the
>> >> >community in Apache, and at least in growing the community with
>>users
>> >>and
>> >> >participants from NASA JPL.
>> >> >
>> >> >=== Inexperience with Open Source ===
>> >> >The team both at Johns Hopkins and NASA JPL have experience with
>>many
>> >>OSS
>> >> >software projects at Apache and elsewhere. We understand "how it
>>works"
>> >> >here at the foundation.
>> >> >
>> >> >
>> >> >== Relationships with Other Apache Products ==
>> >> >Joshua includes dependences on Hadoop, and also is included as a
>> >>plugin in
>> >> >Apache Tika. We are also interested in coordinating with other
>>projects
>> >> >including Spark, and other projects needing MT services for language
>> >> >translation.
>> >> >
>> >> >== Developers ==
>> >> >Joshua only has one regular developer who is employed by Johns
>>Hopkins
>> >> >University. NASA JPL (Mattmann and McGibbney) have been contributing
>> >> >lately including a Brew formula and other contributions to the
>>project
>> >> >through the DARPA XDATA and Memex programs.
>> >> >
>> >> >== Documentation ==
>> >> >Documentation and publications related to Joshua can be found at
>> >> >joshua-decoder.org. The source for the Joshua documentation is
>> >>currently
>> >> >hosted on Github at
>> >> >https://github.com/joshua-decoder/joshua-decoder.github.com
>> >> >
>> >> >== Initial Source ==
>> >> >Current source resides at Github: github.com/joshua-decoder/joshua
>>(the
>> >> >main decoder and toolkit) and github.com/joshua-decoder/thrax (the
>> >> grammar
>> >> >extraction tool).
>> >> >
>> >> >== External Dependencies ==
>> >> >Joshua has a number of external dependencies. Only BerkeleyLM
>>(Apache
>> >>2.0)
>> >> >and KenLM (LGPG 2.1) are run-time decoder dependencies (one of
>>which is
>> >> >needed for translating sentences with pre-built models). The rest
>>are
>> >> >dependencies for the build system and pipeline, used for
>>constructing
>> >>and
>> >> >training new models from parallel text.
>> >> >
>> >> >Apache projects:
>> >> > * Ant
>> >> > * Hadoop
>> >> > * Commons
>> >> > * Maven
>> >> > * Ivy
>> >> >
>> >> >There are also a number of other open-source projects with various
>> >> >licenses that the project depends on both dynamically (runtime), and
>> >> >statically.
>> >> >
>> >> >=== GNU GPL 2 ===
>> >> > * Berkeley Aligner: https://code.google.com/p/berkeleyaligner/
>> >> >
>> >> >=== LGPG 2.1 ===
>> >> > * KenLM: github.com/kpu/kenlm
>> >> >
>> >> >=== Apache 2.0 ===
>> >> > * BerkeleyLM: https://code.google.com/p/berkeleylm/
>> >> >
>> >> >=== GNU GPL ===
>> >> > * GIZA++: http://www.statmt.org/moses/giza/GIZA++.html
>> >> >
>> >> >== Required Resources ==
>> >> > * Mailing Lists
>> >> > * private@joshua.incubator.apache.org
>> >> > * dev@joshua.incubator.apache.org
>> >> > * commits@joshua.incubator.apache.org
>> >> >
>> >> > * Git Repos
>> >> > * https://git-wip-us.apache.org/repos/asf/joshua.git
>> >> >
>> >> > * Issue Tracking
>> >> > * JIRA Joshua (JOSHUA)
>> >> >
>> >> > * Continuous Integration
>> >> > * Jenkins builds on https://builds.apache.org/
>> >> >
>> >> > * Web
>> >> > * http://joshua.incubator.apache.org/
>> >> > * wiki at http://cwiki.apache.org
>> >> >
>> >> >== Initial Committers ==
>> >> >The following is a list of the planned initial Apache committers
>>(the
>> >> >active subset of the committers for the current repository on
>>Github).
>> >> >
>> >> > * Matt Post (post@cs.jhu.edu)
>> >> > * Lewis John McGibbney (lewismc@apache.org)
>> >> > * Chris Mattmann (mattmann@apache.org)
>> >> >
>> >> >== Affiliations ==
>> >> >
>> >> > * Johns Hopkins University
>> >> > * Matt Post
>> >> >
>> >> > * NASA JPL
>> >> > * Chris Mattmann
>> >> > * Lewis John McGibbney
>> >> >
>> >> >
>> >> >== Sponsors ==
>> >> >=== Champion ===
>> >> > * Chris Mattmann (NASA/JPL)
>> >> >
>> >> >=== Nominated Mentors ===
>> >> > * Paul Ramirez
>> >> > * Lewis John McGibbney
>> >> > * Chris Mattmann
>> >> >
>> >> >== Sponsoring Entity ==
>> >> >The Apache Incubator
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> >Chris Mattmann, Ph.D.
>> >> >Chief Architect
>> >> >Instrument Software and Science Data Systems Section (398)
>> >> >NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> >> >Office: 168-519, Mailstop: 168-527
>> >> >Email: chris.a.mattmann@nasa.gov
>> >> >WWW: http://sunset.usc.edu/~mattmann/
>> >> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> >Adjunct Associate Professor, Computer Science Department
>> >> >University of Southern California, Los Angeles, CA 90089 USA
>> >> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> >
>> >> >
>> >> >
>> >>
>>
>>>>>?B�KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK
>>>>>KC
>> >>>B�
>> >>
>>
>>>>>?�?[��X��ܚX�K??K[XZ[?�?�[�\�[?][��X��ܚX�P?[��X�]?܋�\?X�?K�ܙ�B��܈?Y??]?
>>>>>[ۘ
>> >>>[?
>> >> >?��[X[�?�??K[XZ[?�?�[�\�[?Z?[???[��X�]?܋�\?X�?K�ܙ�B
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
RE: [DISCUSS] Apache Joshua Incubator Proposal - Machine
Translation Toolkit
Posted by Martin Gainty <mg...@hotmail.com>.
dependency addition to Joshua-Decoder/Joshua/pom.xml
<!-- MCG added args4j for /Joshua-Decoder/Joshua/src/joshua/decoder/ff/tm/CreateGlueGrammar.java:[17,26] package org.kohsuke.args4j does not exist error --> <dependency> <groupId>args4j</groupId> <artifactId>args4j</artifactId> <version>2.32</version> </dependency>
Joshua-Decoder/Joshua committer please add this dependency to pom.xml
thank you/
Martin
______________________________________________
> From: chris.a.mattmann@jpl.nasa.gov
> To: general@incubator.apache.org
> CC: dev@opennlp.apache.org; dev@lucene.apache.org; dev@tika.apache.org; common-dev@hadoop.apache.org
> Subject: Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine Translation Toolkit
> Date: Tue, 19 Jan 2016 05:58:26 +0000
>
> Great Hen, we’d love to have you on board as a mentor! Please
> add yourself to the proposal on the wiki.
>
> Anyone else have interest in Machine Translation? Any OpenNLP folks,
> Hadoop folks, Tika, or Lucene folks? CC’ing the dev lists for visibility
> please feel free to reply to general@i.a.o.
>
> I’ll leave the DISCUSS thread open for a few more days.
>
> Cheers,
> Chris
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
> -----Original Message-----
> From: Henri Yandell <ba...@apache.org>
> Reply-To: "general@incubator.apache.org" <ge...@incubator.apache.org>
> Date: Monday, January 18, 2016 at 7:57 PM
> To: jpluser <ch...@jpl.nasa.gov>,
> "general@incubator.apache.org" <ge...@incubator.apache.org>
> Subject: Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine
> Translation Toolkit
>
> >Non-binding +1 to Joshua joining the Incubator. I'd be interested in
> >mentoring.
> >
> >
> >> -----Original Message-----
> >> From: jpluser <ch...@jpl.nasa.gov>
> >> Reply-To: "general@incubator.apache.org" <ge...@incubator.apache.org>
> >> Date: Tuesday, January 12, 2016 at 10:56 PM
> >> To: "general@incubator.apache.org" <ge...@incubator.apache.org>
> >> Cc: "post@cs.jhu.edu" <po...@cs.jhu.edu>
> >> Subject: [DISCUSS] Apache Joshua Incubator Proposal - Machine
> >>Translation
> >> Toolkit
> >>
> >> >Hi Everyone,
> >> >
> >> >Please find attached for your viewing pleasure a proposed new project,
> >> >Apache Joshua, a statistical machine translation toolkit. The proposal
> >> >is in wiki draft form at:
> >> https://wiki.apache.org/incubator/JoshuaProposal
> >> >
> >> >Proposal text is copied below. I’ll leave the discussion open for a
> >>week
> >> >and we are interested in folks who would like to be initial committers
> >> >and mentors. Please discuss here on the thread.
> >> >
> >> >Thanks!
> >> >
> >> >Cheers,
> >> >Chris (Champion)
> >> >
> >> >———
> >> >
> >> >= Joshua Proposal =
> >> >
> >> >== Abstract ==
> >> >[[joshua-decoder.org|Joshua]] is an open-source statistical machine
> >> >translation toolkit. It includes a Java-based decoder for translating
> >>with
> >> >phrase-based, hierarchical, and syntax-based translation models, a
> >> >Hadoop-based grammar extractor (Thrax), and an extensive set of tools
> >>and
> >> >scripts for training and evaluating new models from parallel text.
> >> >
> >> >== Proposal ==
> >> >Joshua is a state of the art statistical machine translation system
> >>that
> >> >provides a number of features:
> >> >
> >> > * Support for the two main paradigms in statistical machine
> >>translation:
> >> >phrase-based and hierarchical / syntactic.
> >> > * A sparse feature API that makes it easy to add new feature templates
> >> >supporting millions of features
> >> > * Native implementations of many tuners (MERT, MIRA, PRO, and AdaGrad)
> >> > * Support for lattice decoding, allowing upstream NLP tools to expose
> >> >their hypothesis space to the MT system
> >> > * An efficient representation for models, allowing for quick loading
> >>of
> >> >multi-gigabyte model files
> >> > * Fast decoding speed (on par with Moses and mtplz)
> >> > * Language packs — precompiled models that allow the decoder to be
> >>run as
> >> >a black box
> >> > * Thrax, a Hadoop-based tool for learning translation models from
> >> >parallel text
> >> > * A suite of tools for constructing new models for any language pair
> >>for
> >> >which sufficient training data exists
> >> >
> >> >== Background and Rationale ==
> >> >A number of factors make this a good time for an Apache project
> >>focused on
> >> >machine translation (MT): the quality of MT output (for many language
> >> >pairs); the average computing resources available on computers,
> >>relative
> >> >to the needs of MT systems; and the availability of a number of
> >> >high-quality toolkits, together with a large base of researchers
> >>working
> >> >on them.
> >> >
> >> >Over the past decade, machine translation (MT; the automatic
> >>translation
> >> >of one human language to another) has become a reality. The research
> >>into
> >> >statistical approaches to translation that began in the early nineties,
> >> >together with the availability of large amounts of training data, and
> >> >better computing infrastructure, have all come together to produce
> >> >translations results that are “good enough” for a large set of language
> >> >pairs and use cases. Free services like
> >> >[[https://www.bing.com/translator|Bing Translator]] and
> >> >[[https://translate.google.com|Google Translate]] have made these
> >> services
> >> >available to the average person through direct interfaces and through
> >> >tools like browser plugins, and sites across the world with higher
> >> >translation needs use them to translate their pages through
> >>automatically.
> >> >
> >> >MT does not require the infrastructure of large corporations in order
> >>to
> >> >produce feasible output. Machine translation can be resource-intensive,
> >> >but need not be prohibitively so. Disk and memory usage are mostly a
> >> >matter of model size, which for most language pairs is a few gigabytes
> >>at
> >> >most, at which size models can provide coverage on the order of tens or
> >> >even hundreds of thousands of words in the input and output languages.
> >>The
> >> >computational complexity of the algorithms used to search for
> >>translations
> >> >of new sentences are typically linear in the number of words in the
> >>input
> >> >sentence, making it possible to run a translation engine on a personal
> >> >computer.
> >> >
> >> >The research community has produced many different open source
> >>translation
> >> >projects for a range of programming languages and under a variety of
> >> >licenses. These projects include the core “decoder”, which takes a
> >>model
> >> >and uses it to translate new sentences between the language pair the
> >>model
> >> >was defined for. They also typically include a large set of tools that
> >> >enable new models to be built from large sets of example translations
> >> >(“parallel data”) and monolingual texts. These toolkits are usually
> >>built
> >> >to support the agendas of the (largely) academic researchers that build
> >> >them: the repeated cycle of building new models, tuning model
> >>parameters
> >> >against development data, and evaluating them against held-out test
> >>data,
> >> >using standard metrics for testing the quality of MT output.
> >> >
> >> >Together, these three factors—the quality of machine translation
> >>output,
> >> >the feasibility of translating on standard computers, and the
> >>availability
> >> >of tools to build models—make it reasonable for the end users to use
> >>MT as
> >> >a black-box service, and to run it on their personal machine.
> >> >
> >> >These factors make it a good time for an organization with the status
> >>of
> >> >the Apache Foundation to host a machine translation project.
> >> >
> >> >== Current Status ==
> >> >Joshua was originally ported from David Chiang’s Python implementation
> >>of
> >> >Hiero by Zhifei Li, while he was a Ph.D. student at Johns Hopkins
> >> >University. The current version is maintained by Matt Post at Johns
> >> >Hopkins’ Human Language Technology Center of Excellence. Joshua has
> >>made
> >> >many releases with a list of over 20 source code tags. The last
> >>release of
> >> >Joshua was 6.0.5 on November 5th, 2015.
> >> >
> >> >== Meritocracy ==
> >> >The current developers are familiar with meritocratic open source
> >> >development at Apache. Apache was chosen specifically because we want
> >>to
> >> >encourage this style of development for the project.
> >> >
> >> >== Community ==
> >> >Joshua is used widely across the world. Perhaps its biggest (known)
> >> >research / industrial user is the Amazon research group in Berlin.
> >>Another
> >> >user is the US Army Research Lab. No formal census has been undertaken,
> >> >but posts to the Joshua technical support mailing list, along with the
> >> >occasional contributions, suggest small research and academic
> >>communities
> >> >spread across the world, many of them in India.
> >> >
> >> >During incubation, we will explicitly seek to increase our usage across
> >> >the board, including academic research, industry, and other end users
> >> >interested in statistical machine translation.
> >> >
> >> >== Core Developers ==
> >> >The current set of core developers is fairly small, having fallen with
> >>the
> >> >graduation from Johns Hopkins of some core student participants.
> >>However,
> >> >Joshua is used fairly widely, as mentioned above, and there remains a
> >> >commitment from the principal researcher at Johns Hopkins to continue
> >>to
> >> >use and develop it. Joshua has seen a number of new community members
> >> >become interested recently due to a potential for its projected use in
> >>a
> >> >number of ongoing DARPA projects such as XDATA and Memex.
> >> >
> >> >== Alignment ==
> >> >Joshua is currently Copyright (c) 2015, Johns Hopkins University All
> >> >rights reserved and licensed under BSD 2-clause license. It would of
> >> >course be the intention to relicense this code under AL2.0 which would
> >> >permit expanded and increased use of the software within Apache
> >>projects.
> >> >There is currently an ongoing effort within the Apache Tika community
> >>to
> >> >utilize Joshua within Tika’s Translate API, see
> >> >[[https://issues.apache.org/jira/browse/TIKA-1343|TIKA-1343]].
> >> >
> >> >== Known Risks ==
> >> >
> >> >=== Orphaned products ===
> >> >At the moment, regular contributions are made by a single contributor,
> >>the
> >> >lead maintainer. He (Matt Post) plans to continue development for the
> >>next
> >> >few years, but it is still a single point of failure, since the
> >>graduate
> >> >students who worked on the project have moved on to jobs, mostly in
> >> >industry. However, our goal is to help that process by growing the
> >> >community in Apache, and at least in growing the community with users
> >>and
> >> >participants from NASA JPL.
> >> >
> >> >=== Inexperience with Open Source ===
> >> >The team both at Johns Hopkins and NASA JPL have experience with many
> >>OSS
> >> >software projects at Apache and elsewhere. We understand "how it works"
> >> >here at the foundation.
> >> >
> >> >
> >> >== Relationships with Other Apache Products ==
> >> >Joshua includes dependences on Hadoop, and also is included as a
> >>plugin in
> >> >Apache Tika. We are also interested in coordinating with other projects
> >> >including Spark, and other projects needing MT services for language
> >> >translation.
> >> >
> >> >== Developers ==
> >> >Joshua only has one regular developer who is employed by Johns Hopkins
> >> >University. NASA JPL (Mattmann and McGibbney) have been contributing
> >> >lately including a Brew formula and other contributions to the project
> >> >through the DARPA XDATA and Memex programs.
> >> >
> >> >== Documentation ==
> >> >Documentation and publications related to Joshua can be found at
> >> >joshua-decoder.org. The source for the Joshua documentation is
> >>currently
> >> >hosted on Github at
> >> >https://github.com/joshua-decoder/joshua-decoder.github.com
> >> >
> >> >== Initial Source ==
> >> >Current source resides at Github: github.com/joshua-decoder/joshua (the
> >> >main decoder and toolkit) and github.com/joshua-decoder/thrax (the
> >> grammar
> >> >extraction tool).
> >> >
> >> >== External Dependencies ==
> >> >Joshua has a number of external dependencies. Only BerkeleyLM (Apache
> >>2.0)
> >> >and KenLM (LGPG 2.1) are run-time decoder dependencies (one of which is
> >> >needed for translating sentences with pre-built models). The rest are
> >> >dependencies for the build system and pipeline, used for constructing
> >>and
> >> >training new models from parallel text.
> >> >
> >> >Apache projects:
> >> > * Ant
> >> > * Hadoop
> >> > * Commons
> >> > * Maven
> >> > * Ivy
> >> >
> >> >There are also a number of other open-source projects with various
> >> >licenses that the project depends on both dynamically (runtime), and
> >> >statically.
> >> >
> >> >=== GNU GPL 2 ===
> >> > * Berkeley Aligner: https://code.google.com/p/berkeleyaligner/
> >> >
> >> >=== LGPG 2.1 ===
> >> > * KenLM: github.com/kpu/kenlm
> >> >
> >> >=== Apache 2.0 ===
> >> > * BerkeleyLM: https://code.google.com/p/berkeleylm/
> >> >
> >> >=== GNU GPL ===
> >> > * GIZA++: http://www.statmt.org/moses/giza/GIZA++.html
> >> >
> >> >== Required Resources ==
> >> > * Mailing Lists
> >> > * private@joshua.incubator.apache.org
> >> > * dev@joshua.incubator.apache.org
> >> > * commits@joshua.incubator.apache.org
> >> >
> >> > * Git Repos
> >> > * https://git-wip-us.apache.org/repos/asf/joshua.git
> >> >
> >> > * Issue Tracking
> >> > * JIRA Joshua (JOSHUA)
> >> >
> >> > * Continuous Integration
> >> > * Jenkins builds on https://builds.apache.org/
> >> >
> >> > * Web
> >> > * http://joshua.incubator.apache.org/
> >> > * wiki at http://cwiki.apache.org
> >> >
> >> >== Initial Committers ==
> >> >The following is a list of the planned initial Apache committers (the
> >> >active subset of the committers for the current repository on Github).
> >> >
> >> > * Matt Post (post@cs.jhu.edu)
> >> > * Lewis John McGibbney (lewismc@apache.org)
> >> > * Chris Mattmann (mattmann@apache.org)
> >> >
> >> >== Affiliations ==
> >> >
> >> > * Johns Hopkins University
> >> > * Matt Post
> >> >
> >> > * NASA JPL
> >> > * Chris Mattmann
> >> > * Lewis John McGibbney
> >> >
> >> >
> >> >== Sponsors ==
> >> >=== Champion ===
> >> > * Chris Mattmann (NASA/JPL)
> >> >
> >> >=== Nominated Mentors ===
> >> > * Paul Ramirez
> >> > * Lewis John McGibbney
> >> > * Chris Mattmann
> >> >
> >> >== Sponsoring Entity ==
> >> >The Apache Incubator
> >> >
> >> >
> >> >
> >> >
> >> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> >Chris Mattmann, Ph.D.
> >> >Chief Architect
> >> >Instrument Software and Science Data Systems Section (398)
> >> >NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >> >Office: 168-519, Mailstop: 168-527
> >> >Email: chris.a.mattmann@nasa.gov
> >> >WWW: http://sunset.usc.edu/~mattmann/
> >> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> >Adjunct Associate Professor, Computer Science Department
> >> >University of Southern California, Los Angeles, CA 90089 USA
> >> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> >
> >> >
> >> >
> >>
> >>>?B�KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKC
> >>>B�
> >>
> >>>?�?[��X��ܚX�K??K[XZ[?�?�[�\�[?][��X��ܚX�P?[��X�]?܋�\?X�?K�ܙ�B��܈?Y??]?[ۘ
> >>>[?
> >> >?��[X[�?�??K[XZ[?�?�[�\�[?Z?[???[��X�]?܋�\?X�?K�ܙ�B
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine Translation
Toolkit
Posted by toki <to...@gmail.com>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 20/01/2016 11:18, Henri Yandell wrote:
> 'TM'?
Translation Memory.
jonathon
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
iQIcBAEBAgAGBQJWn8SIAAoJEKG7hs8nSMR7h8EP/01XxxF896gbhRJBhlv5ng0J
GxWcFpUYD3yyXp3c1CjmSI5rmLaoDja+ezRwQ/4iuZ+3T39FxSuijBD6oqZi0+gM
3b9E3OXYqpToi2FS2uW2LajEGLhU+H3JSPqmABH3qFye/MOjeiSqoAD/yg150KLp
oHxUH3Sd69BGAVvGgobOpZt/5xm09rKZcVowGH6HMlRtUOHzzLeVr5rG/6KvGEEZ
hkbdDOdpz9/njVhDZWrxEGsvItSCAN29LFM8sdpusCx4X0ewYp7P5M4bgvJVfttX
WWY+Arn5dGHj7kjBtE5tZXUZVV0tgavTrhuzLCdMMh16wUygfHwypNX0rsP2dM59
+tTB0SxxXdSEDGgIaCw+p3EvO8jNeFPr2Yx0hUTWPsbmipP/CNoo7WtG63426x7W
ehUQECBThrRajC7ifRl3ZWtOKTwjpNdwzzhOLI3K346/9VcreNsQrT5HcRP5y3/n
/SEyV5FvrPQOErgA+0K+heSs0RFDRiCfMzdwFfBmgzGjzwWzMeFwqvCYnSCR6VKp
UvbEZJxDaCz3yIukR25RN3H9JdjfyqpOc1rxD9anOJw2cnbDzNwv8pyn/RWRfCct
BM5RAvKqnVNoQqwCjvWdpfVxF4+BoLyrcAE+t8yKsY1j1tGc9F6b/kzkXai60omf
KrGAFX69koM/BCfirdQv
=Sjo6
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine Translation Toolkit
Posted by Henri Yandell <ba...@apache.org>.
'TM'?
On Tue, Jan 19, 2016 at 9:42 PM, toki <to...@gmail.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 20/01/2016 03:38, Henri Yandell wrote:
>
> > License-wise, any expectation of problems from the GPL and LGPL depend
> encies?
>
> I'd also recommend closely examining the license for any TM libraries
> that ship with the program, with a focus on the providence of the libra
> ry.
>
> Whilst it has been several years since I've used a TM library, when I
> did, the licenses were, under Debian Guidelines, non-free. Back then, I
> didn't look at licenses, to see if they complied with Apache Guidelines.
>
> About fifteen years ago, I was on the fringes of a lawsuit involving a
> TM library. All intellectual property rights of the TM Library belonged
> to the developers of the software program, even though the words in the
> TM Library were either deliberately selected by the licensed user of the
> software program, or came from parsing content that was created, and to
> whom all intellectual property rights apparently were owned, by the user
> of the software program.
>
> jonathon
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.22 (GNU/Linux)
>
> iQIcBAEBAgAGBQJWnx5QAAoJEKG7hs8nSMR7YM8QAIVT3FdUncmBryIsd4fFUwDV
> nUwinxGfFON6PGdyfOPmZsl2CRz8c+sJxrCVxJePj0PbwCLTfIy+HHNro3lpAefC
> iHLBHdWyFOBtlkxac+QigRG7tuKw9xRd5lo7Y5A3cNx/C1nL0c4GULDZCaZtDnQi
> yf5bUdmyU57KYeMbg4hTePeYkiOYt1Bez37V00hj57AAuo3JZWCzGv/rFcgFZ0r5
> nGaQrSll7N8UJ68HurBPReRdxhnVYv5jTTgI0sCoJfBaIH7DZga6Mur87LGbvjUW
> 3xomBzpHuMGpZBnuNBMyyMOFpZlu41zmLHeAWL/L27fOkKcfuVcDeZsXlg/jOULg
> P/iqxuVQqpmMThpSUVDBcsBF/VjgHTGCFz7buv/4GKhNtbTohldMX6oTBaIfgPOg
> QIEM9k4LYLbl4jLBP/I3fESAte+pW1bFglMEj+YfoUTBMYQsvclMqSwi/oYA5xS2
> IbmQtqxXG26Dm/a8fb5eRJ3fp4lGkXWHmq/vdrkNLtYgQ5kwGTilQpiWjowJnLVm
> ODlTvgS44JMFeMw2cAzVUjdpH9IZT89Glo6iL3J7ncfnaIZSwozQRCbLStf1jY/u
> CznX492Lywdw2dm+QcpEpq7dm5ewm0AORFumodvn+P+cYqLigsjNOz0yO/RCaBOg
> pkYvV6RHni2vFqn44uRq
> =88z1
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine Translation
Toolkit
Posted by toki <to...@gmail.com>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 20/01/2016 03:38, Henri Yandell wrote:
> License-wise, any expectation of problems from the GPL and LGPL depend
encies?
I'd also recommend closely examining the license for any TM libraries
that ship with the program, with a focus on the providence of the libra
ry.
Whilst it has been several years since I've used a TM library, when I
did, the licenses were, under Debian Guidelines, non-free. Back then, I
didn't look at licenses, to see if they complied with Apache Guidelines.
About fifteen years ago, I was on the fringes of a lawsuit involving a
TM library. All intellectual property rights of the TM Library belonged
to the developers of the software program, even though the words in the
TM Library were either deliberately selected by the licensed user of the
software program, or came from parsing content that was created, and to
whom all intellectual property rights apparently were owned, by the user
of the software program.
jonathon
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
iQIcBAEBAgAGBQJWnx5QAAoJEKG7hs8nSMR7YM8QAIVT3FdUncmBryIsd4fFUwDV
nUwinxGfFON6PGdyfOPmZsl2CRz8c+sJxrCVxJePj0PbwCLTfIy+HHNro3lpAefC
iHLBHdWyFOBtlkxac+QigRG7tuKw9xRd5lo7Y5A3cNx/C1nL0c4GULDZCaZtDnQi
yf5bUdmyU57KYeMbg4hTePeYkiOYt1Bez37V00hj57AAuo3JZWCzGv/rFcgFZ0r5
nGaQrSll7N8UJ68HurBPReRdxhnVYv5jTTgI0sCoJfBaIH7DZga6Mur87LGbvjUW
3xomBzpHuMGpZBnuNBMyyMOFpZlu41zmLHeAWL/L27fOkKcfuVcDeZsXlg/jOULg
P/iqxuVQqpmMThpSUVDBcsBF/VjgHTGCFz7buv/4GKhNtbTohldMX6oTBaIfgPOg
QIEM9k4LYLbl4jLBP/I3fESAte+pW1bFglMEj+YfoUTBMYQsvclMqSwi/oYA5xS2
IbmQtqxXG26Dm/a8fb5eRJ3fp4lGkXWHmq/vdrkNLtYgQ5kwGTilQpiWjowJnLVm
ODlTvgS44JMFeMw2cAzVUjdpH9IZT89Glo6iL3J7ncfnaIZSwozQRCbLStf1jY/u
CznX492Lywdw2dm+QcpEpq7dm5ewm0AORFumodvn+P+cYqLigsjNOz0yO/RCaBOg
pkYvV6RHni2vFqn44uRq
=88z1
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine Translation Toolkit
Posted by Tommaso Teofili <to...@gmail.com>.
2016-01-30 2:31 GMT+01:00 Mattmann, Chris A (3980) <
chris.a.mattmann@jpl.nasa.gov>:
> Yep I was giving it a few more days to see if there was any interest.
>
> I intend to call the VOTE in a few hours :-)
>
cool, thanks!
>
> Thanks Tommaso! Have you added your interest to the wiki?
>
sure, I've added myself as a committer :)
Cheers,
Tommaso
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
> -----Original Message-----
> From: Tommaso Teofili <to...@gmail.com>
> Reply-To: "general@incubator.apache.org" <ge...@incubator.apache.org>
> Date: Friday, January 29, 2016 at 8:29 AM
> To: "general@incubator.apache.org" <ge...@incubator.apache.org>
> Subject: Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine
> Translation Toolkit
>
> >what about moving forward to [VOTE] ?
> >
> >Regards,
> >Tommaso
> >
> >2016-01-22 4:31 GMT+01:00 Henri Yandell <ba...@apache.org>:
> >
> >> On Wed, Jan 20, 2016 at 10:37 AM, Mattmann, Chris A (3980) <
> >> chris.a.mattmann@jpl.nasa.gov> wrote:
> >>
> >> > All of this can be worked during Incubation and I think we have
> >> > the right folks here who can help to get it set up.
> >> >
> >>
> >> Yup - my question was well answered; thanks all :)
> >>
> >> Hen
> >>
>
>
Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine
Translation Toolkit
Posted by "Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov>.
Yep I was giving it a few more days to see if there was any interest.
I intend to call the VOTE in a few hours :-)
Thanks Tommaso! Have you added your interest to the wiki?
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Tommaso Teofili <to...@gmail.com>
Reply-To: "general@incubator.apache.org" <ge...@incubator.apache.org>
Date: Friday, January 29, 2016 at 8:29 AM
To: "general@incubator.apache.org" <ge...@incubator.apache.org>
Subject: Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine
Translation Toolkit
>what about moving forward to [VOTE] ?
>
>Regards,
>Tommaso
>
>2016-01-22 4:31 GMT+01:00 Henri Yandell <ba...@apache.org>:
>
>> On Wed, Jan 20, 2016 at 10:37 AM, Mattmann, Chris A (3980) <
>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>
>> > All of this can be worked during Incubation and I think we have
>> > the right folks here who can help to get it set up.
>> >
>>
>> Yup - my question was well answered; thanks all :)
>>
>> Hen
>>
Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine Translation Toolkit
Posted by Tommaso Teofili <to...@gmail.com>.
what about moving forward to [VOTE] ?
Regards,
Tommaso
2016-01-22 4:31 GMT+01:00 Henri Yandell <ba...@apache.org>:
> On Wed, Jan 20, 2016 at 10:37 AM, Mattmann, Chris A (3980) <
> chris.a.mattmann@jpl.nasa.gov> wrote:
>
> > All of this can be worked during Incubation and I think we have
> > the right folks here who can help to get it set up.
> >
>
> Yup - my question was well answered; thanks all :)
>
> Hen
>
Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine Translation Toolkit
Posted by Henri Yandell <ba...@apache.org>.
On Wed, Jan 20, 2016 at 10:37 AM, Mattmann, Chris A (3980) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> All of this can be worked during Incubation and I think we have
> the right folks here who can help to get it set up.
>
Yup - my question was well answered; thanks all :)
Hen
Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine
Translation Toolkit
Posted by "Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov>.
All of this can be worked during Incubation and I think we have
the right folks here who can help to get it set up.
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Henry Saputra <he...@gmail.com>
Reply-To: "general@incubator.apache.org" <ge...@incubator.apache.org>
Date: Wednesday, January 20, 2016 at 10:33 AM
To: "general@incubator.apache.org" <ge...@incubator.apache.org>
Subject: Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine
Translation Toolkit
>This is a bit tricky and I suppose we could leave behind the GPL/LGPL
>dependencies that are used for model building when generating releases
>under ASF license.
>I hope that will work.
>
>- Henry
>
>On Wed, Jan 20, 2016 at 7:51 AM, Matt Post <po...@cs.jhu.edu> wrote:
>
>> I imagine so. Model building is very technical and resource intensive
>>and
>> something only a few people will want or need to do. Working on and
>>running
>> the decoder (#2) should be the much more common use case, and with the
>> (included, Apache-licensed) Berkeley LM, that can be done without the
>>need
>> for any external dependencies.
>>
>>
>> > On Jan 20, 2016, at 10:46 AM, Alex Harui <ah...@adobe.com> wrote:
>> >
>> > External is good news. I'm not sure how much leeway there is in the
>> > following quote from [1], but what percentage of your users are
>>currently
>> > using an all-ASF-compatible set of projects?
>> >
>> > The question to ask yourself in this situation is:
>> > * "Will the majority of users want to use my
>> > product without adding the optional components?"
>> >
>> > -Alex
>> >
>> > [1] http://www.apache.org/legal/resolved.html
>> >
>> >
>> > On 1/20/16, 7:17 AM, "Matt Post" <po...@cs.jhu.edu> wrote:
>> >
>> >> The dependencies can be split into two kinds: ones required for
>>building
>> >> new models, and ones needed by the decoder to translate new sentences
>> >> with a pre-built model (i.e., black-box translation with the language
>> >> packs).
>> >>
>> >> 1. For building new models, you need a way to align the words between
>> >> sentences in parallel text. Both the aligners used by Joshua (GIZA++
>>and
>> >> the Berkeley aligner) are GPL of some form. These can be implemented
>>as
>> >> external dependencies, or can be replaced with another aligner, like
>> >> fast_align (https://github.com/clab/fast_align), which is
>> >> Apache-licensed. There are many other options, in fact. So this
>>should
>> >> not be a worry.
>> >>
>> >> 2. For doing black-box translation, one needs to represent the
>>language
>> >> model, which is very large. The best tool for this is KenLM
>> >> (github.com/kpu/kenlm), which is LGPL 2.1. There is also BerkeleyLM,
>> >> which is just as good for practical purposes and is Apache-licensed.
>> >> KenLM is C++ and is loaded via the JNI, whereas BerkeleyLM is
>>written in
>> >> Java. I have moved to including BerkeleyLM in language packs,
>>because I
>> >> can then include the Joshua-runtime, and people can translate without
>> >> even having to compile anything.
>> >>
>> >> So in short, there are no hard dependencies on unfavorably-licensed
>> >> external projects.
>> >>
>> >> matt
>> >>
>> >>
>> >>
>> >>
>> >>> On Jan 20, 2016, at 10:08 AM, Mattmann, Chris A (3980)
>> >>> <ch...@jpl.nasa.gov> wrote:
>> >>>
>> >>> Hey Hen,
>> >>>
>> >>> Matt Post who I believe is monitoring this list and who has
>> >>> been one of the key Joshua developers and I have discussed this
>> >>> and we believe that potentially GPL/LGPL dependencies can:
>> >>>
>> >>> 1. be replaced with category-A or category-B alternatives. Matt
>> >>> mentioned one already to me which has slipped my mind.
>> >>> 2. be made in such a way that they are external tools and the
>> >>> bindings exist in Joshua to call those external tools (aka runtime
>> >>> deps akin to depending on a C compiler, etc.)
>> >>>
>> >>> Cheers,
>> >>> Chris
>> >>>
>> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>> Chris Mattmann, Ph.D.
>> >>> Chief Architect
>> >>> Instrument Software and Science Data Systems Section (398)
>> >>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> >>> Office: 168-519, Mailstop: 168-527
>> >>> Email: chris.a.mattmann@nasa.gov
>> >>> WWW: http://sunset.usc.edu/~mattmann/
>> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>> Adjunct Associate Professor, Computer Science Department
>> >>> University of Southern California, Los Angeles, CA 90089 USA
>> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine Translation Toolkit
Posted by Henry Saputra <he...@gmail.com>.
This is a bit tricky and I suppose we could leave behind the GPL/LGPL
dependencies that are used for model building when generating releases
under ASF license.
I hope that will work.
- Henry
On Wed, Jan 20, 2016 at 7:51 AM, Matt Post <po...@cs.jhu.edu> wrote:
> I imagine so. Model building is very technical and resource intensive and
> something only a few people will want or need to do. Working on and running
> the decoder (#2) should be the much more common use case, and with the
> (included, Apache-licensed) Berkeley LM, that can be done without the need
> for any external dependencies.
>
>
> > On Jan 20, 2016, at 10:46 AM, Alex Harui <ah...@adobe.com> wrote:
> >
> > External is good news. I'm not sure how much leeway there is in the
> > following quote from [1], but what percentage of your users are currently
> > using an all-ASF-compatible set of projects?
> >
> > The question to ask yourself in this situation is:
> > * "Will the majority of users want to use my
> > product without adding the optional components?"
> >
> > -Alex
> >
> > [1] http://www.apache.org/legal/resolved.html
> >
> >
> > On 1/20/16, 7:17 AM, "Matt Post" <po...@cs.jhu.edu> wrote:
> >
> >> The dependencies can be split into two kinds: ones required for building
> >> new models, and ones needed by the decoder to translate new sentences
> >> with a pre-built model (i.e., black-box translation with the language
> >> packs).
> >>
> >> 1. For building new models, you need a way to align the words between
> >> sentences in parallel text. Both the aligners used by Joshua (GIZA++ and
> >> the Berkeley aligner) are GPL of some form. These can be implemented as
> >> external dependencies, or can be replaced with another aligner, like
> >> fast_align (https://github.com/clab/fast_align), which is
> >> Apache-licensed. There are many other options, in fact. So this should
> >> not be a worry.
> >>
> >> 2. For doing black-box translation, one needs to represent the language
> >> model, which is very large. The best tool for this is KenLM
> >> (github.com/kpu/kenlm), which is LGPL 2.1. There is also BerkeleyLM,
> >> which is just as good for practical purposes and is Apache-licensed.
> >> KenLM is C++ and is loaded via the JNI, whereas BerkeleyLM is written in
> >> Java. I have moved to including BerkeleyLM in language packs, because I
> >> can then include the Joshua-runtime, and people can translate without
> >> even having to compile anything.
> >>
> >> So in short, there are no hard dependencies on unfavorably-licensed
> >> external projects.
> >>
> >> matt
> >>
> >>
> >>
> >>
> >>> On Jan 20, 2016, at 10:08 AM, Mattmann, Chris A (3980)
> >>> <ch...@jpl.nasa.gov> wrote:
> >>>
> >>> Hey Hen,
> >>>
> >>> Matt Post who I believe is monitoring this list and who has
> >>> been one of the key Joshua developers and I have discussed this
> >>> and we believe that potentially GPL/LGPL dependencies can:
> >>>
> >>> 1. be replaced with category-A or category-B alternatives. Matt
> >>> mentioned one already to me which has slipped my mind.
> >>> 2. be made in such a way that they are external tools and the
> >>> bindings exist in Joshua to call those external tools (aka runtime
> >>> deps akin to depending on a C compiler, etc.)
> >>>
> >>> Cheers,
> >>> Chris
> >>>
> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> Chris Mattmann, Ph.D.
> >>> Chief Architect
> >>> Instrument Software and Science Data Systems Section (398)
> >>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>> Office: 168-519, Mailstop: 168-527
> >>> Email: chris.a.mattmann@nasa.gov
> >>> WWW: http://sunset.usc.edu/~mattmann/
> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> Adjunct Associate Professor, Computer Science Department
> >>> University of Southern California, Los Angeles, CA 90089 USA
> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine Translation Toolkit
Posted by Matt Post <po...@cs.jhu.edu>.
I imagine so. Model building is very technical and resource intensive and something only a few people will want or need to do. Working on and running the decoder (#2) should be the much more common use case, and with the (included, Apache-licensed) Berkeley LM, that can be done without the need for any external dependencies.
> On Jan 20, 2016, at 10:46 AM, Alex Harui <ah...@adobe.com> wrote:
>
> External is good news. I'm not sure how much leeway there is in the
> following quote from [1], but what percentage of your users are currently
> using an all-ASF-compatible set of projects?
>
> The question to ask yourself in this situation is:
> * "Will the majority of users want to use my
> product without adding the optional components?"
>
> -Alex
>
> [1] http://www.apache.org/legal/resolved.html
>
>
> On 1/20/16, 7:17 AM, "Matt Post" <po...@cs.jhu.edu> wrote:
>
>> The dependencies can be split into two kinds: ones required for building
>> new models, and ones needed by the decoder to translate new sentences
>> with a pre-built model (i.e., black-box translation with the language
>> packs).
>>
>> 1. For building new models, you need a way to align the words between
>> sentences in parallel text. Both the aligners used by Joshua (GIZA++ and
>> the Berkeley aligner) are GPL of some form. These can be implemented as
>> external dependencies, or can be replaced with another aligner, like
>> fast_align (https://github.com/clab/fast_align), which is
>> Apache-licensed. There are many other options, in fact. So this should
>> not be a worry.
>>
>> 2. For doing black-box translation, one needs to represent the language
>> model, which is very large. The best tool for this is KenLM
>> (github.com/kpu/kenlm), which is LGPL 2.1. There is also BerkeleyLM,
>> which is just as good for practical purposes and is Apache-licensed.
>> KenLM is C++ and is loaded via the JNI, whereas BerkeleyLM is written in
>> Java. I have moved to including BerkeleyLM in language packs, because I
>> can then include the Joshua-runtime, and people can translate without
>> even having to compile anything.
>>
>> So in short, there are no hard dependencies on unfavorably-licensed
>> external projects.
>>
>> matt
>>
>>
>>
>>
>>> On Jan 20, 2016, at 10:08 AM, Mattmann, Chris A (3980)
>>> <ch...@jpl.nasa.gov> wrote:
>>>
>>> Hey Hen,
>>>
>>> Matt Post who I believe is monitoring this list and who has
>>> been one of the key Joshua developers and I have discussed this
>>> and we believe that potentially GPL/LGPL dependencies can:
>>>
>>> 1. be replaced with category-A or category-B alternatives. Matt
>>> mentioned one already to me which has slipped my mind.
>>> 2. be made in such a way that they are external tools and the
>>> bindings exist in Joshua to call those external tools (aka runtime
>>> deps akin to depending on a C compiler, etc.)
>>>
>>> Cheers,
>>> Chris
>>>
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Chris Mattmann, Ph.D.
>>> Chief Architect
>>> Instrument Software and Science Data Systems Section (398)
>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> Office: 168-519, Mailstop: 168-527
>>> Email: chris.a.mattmann@nasa.gov
>>> WWW: http://sunset.usc.edu/~mattmann/
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Adjunct Associate Professor, Computer Science Department
>>> University of Southern California, Los Angeles, CA 90089 USA
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine Translation
Toolkit
Posted by Alex Harui <ah...@adobe.com>.
External is good news. I'm not sure how much leeway there is in the
following quote from [1], but what percentage of your users are currently
using an all-ASF-compatible set of projects?
The question to ask yourself in this situation is:
* "Will the majority of users want to use my
product without adding the optional components?"
-Alex
[1] http://www.apache.org/legal/resolved.html
On 1/20/16, 7:17 AM, "Matt Post" <po...@cs.jhu.edu> wrote:
>The dependencies can be split into two kinds: ones required for building
>new models, and ones needed by the decoder to translate new sentences
>with a pre-built model (i.e., black-box translation with the language
>packs).
>
>1. For building new models, you need a way to align the words between
>sentences in parallel text. Both the aligners used by Joshua (GIZA++ and
>the Berkeley aligner) are GPL of some form. These can be implemented as
>external dependencies, or can be replaced with another aligner, like
>fast_align (https://github.com/clab/fast_align), which is
>Apache-licensed. There are many other options, in fact. So this should
>not be a worry.
>
>2. For doing black-box translation, one needs to represent the language
>model, which is very large. The best tool for this is KenLM
>(github.com/kpu/kenlm), which is LGPL 2.1. There is also BerkeleyLM,
>which is just as good for practical purposes and is Apache-licensed.
>KenLM is C++ and is loaded via the JNI, whereas BerkeleyLM is written in
>Java. I have moved to including BerkeleyLM in language packs, because I
>can then include the Joshua-runtime, and people can translate without
>even having to compile anything.
>
>So in short, there are no hard dependencies on unfavorably-licensed
>external projects.
>
>matt
>
>
>
>
>> On Jan 20, 2016, at 10:08 AM, Mattmann, Chris A (3980)
>><ch...@jpl.nasa.gov> wrote:
>>
>> Hey Hen,
>>
>> Matt Post who I believe is monitoring this list and who has
>> been one of the key Joshua developers and I have discussed this
>> and we believe that potentially GPL/LGPL dependencies can:
>>
>> 1. be replaced with category-A or category-B alternatives. Matt
>> mentioned one already to me which has slipped my mind.
>> 2. be made in such a way that they are external tools and the
>> bindings exist in Joshua to call those external tools (aka runtime
>> deps akin to depending on a C compiler, etc.)
>>
>> Cheers,
>> Chris
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Chief Architect
>> Instrument Software and Science Data Systems Section (398)
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 168-519, Mailstop: 168-527
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Associate Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Henri Yandell <ba...@apache.org>
>> Reply-To: "general@incubator.apache.org" <ge...@incubator.apache.org>
>> Date: Tuesday, January 19, 2016 at 7:38 PM
>> To: "general@incubator.apache.org" <ge...@incubator.apache.org>
>> Subject: Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine
>> Translation Toolkit
>>
>>> License-wise, any expectation of problems from the GPL and LGPL
>>> dependencies?
>>>
>>> On Mon, Jan 18, 2016 at 9:58 PM, Mattmann, Chris A (3980) <
>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>
>>>> Great Hen, we’d love to have you on board as a mentor! Please
>>>> add yourself to the proposal on the wiki.
>>>>
>>>> Anyone else have interest in Machine Translation? Any OpenNLP folks,
>>>> Hadoop folks, Tika, or Lucene folks? CC’ing the dev lists for
>>>>visibility
>>>> please feel free to reply to general@i.a.o.
>>>>
>>>> I’ll leave the DISCUSS thread open for a few more days.
>>>>
>>>> Cheers,
>>>> Chris
>>>>
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> Chris Mattmann, Ph.D.
>>>> Chief Architect
>>>> Instrument Software and Science Data Systems Section (398)
>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>> Office: 168-519, Mailstop: 168-527
>>>> Email: chris.a.mattmann@nasa.gov
>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> Adjunct Associate Professor, Computer Science Department
>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Henri Yandell <ba...@apache.org>
>>>> Reply-To: "general@incubator.apache.org"
>>>><ge...@incubator.apache.org>
>>>> Date: Monday, January 18, 2016 at 7:57 PM
>>>> To: jpluser <ch...@jpl.nasa.gov>,
>>>> "general@incubator.apache.org" <ge...@incubator.apache.org>
>>>> Subject: Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine
>>>> Translation Toolkit
>>>>
>>>>> Non-binding +1 to Joshua joining the Incubator. I'd be interested in
>>>>> mentoring.
>>>>>
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: jpluser <ch...@jpl.nasa.gov>
>>>>>> Reply-To: "general@incubator.apache.org"
>>>> <ge...@incubator.apache.org>
>>>>>> Date: Tuesday, January 12, 2016 at 10:56 PM
>>>>>> To: "general@incubator.apache.org" <ge...@incubator.apache.org>
>>>>>> Cc: "post@cs.jhu.edu" <po...@cs.jhu.edu>
>>>>>> Subject: [DISCUSS] Apache Joshua Incubator Proposal - Machine
>>>>>> Translation
>>>>>> Toolkit
>>>>>>
>>>>>>> Hi Everyone,
>>>>>>>
>>>>>>> Please find attached for your viewing pleasure a proposed new
>>>> project,
>>>>>>> Apache Joshua, a statistical machine translation toolkit. The
>>>> proposal
>>>>>>> is in wiki draft form at:
>>>>>> https://wiki.apache.org/incubator/JoshuaProposal
>>>>>>>
>>>>>>> Proposal text is copied below. I’ll leave the discussion open for a
>>>>>> week
>>>>>>> and we are interested in folks who would like to be initial
>>>> committers
>>>>>>> and mentors. Please discuss here on the thread.
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Chris (Champion)
>>>>>>>
>>>>>>> ———
>>>>>>>
>>>>>>> = Joshua Proposal =
>>>>>>>
>>>>>>> == Abstract ==
>>>>>>> [[joshua-decoder.org|Joshua]] is an open-source statistical machine
>>>>>>> translation toolkit. It includes a Java-based decoder for
>>>> translating
>>>>>> with
>>>>>>> phrase-based, hierarchical, and syntax-based translation models, a
>>>>>>> Hadoop-based grammar extractor (Thrax), and an extensive set of
>>>> tools
>>>>>> and
>>>>>>> scripts for training and evaluating new models from parallel text.
>>>>>>>
>>>>>>> == Proposal ==
>>>>>>> Joshua is a state of the art statistical machine translation system
>>>>>> that
>>>>>>> provides a number of features:
>>>>>>>
>>>>>>> * Support for the two main paradigms in statistical machine
>>>>>> translation:
>>>>>>> phrase-based and hierarchical / syntactic.
>>>>>>> * A sparse feature API that makes it easy to add new feature
>>>> templates
>>>>>>> supporting millions of features
>>>>>>> * Native implementations of many tuners (MERT, MIRA, PRO, and
>>>> AdaGrad)
>>>>>>> * Support for lattice decoding, allowing upstream NLP tools to
>>>> expose
>>>>>>> their hypothesis space to the MT system
>>>>>>> * An efficient representation for models, allowing for quick
>>>> loading
>>>>>> of
>>>>>>> multi-gigabyte model files
>>>>>>> * Fast decoding speed (on par with Moses and mtplz)
>>>>>>> * Language packs — precompiled models that allow the decoder to be
>>>>>> run as
>>>>>>> a black box
>>>>>>> * Thrax, a Hadoop-based tool for learning translation models from
>>>>>>> parallel text
>>>>>>> * A suite of tools for constructing new models for any language
>>>> pair
>>>>>> for
>>>>>>> which sufficient training data exists
>>>>>>>
>>>>>>> == Background and Rationale ==
>>>>>>> A number of factors make this a good time for an Apache project
>>>>>> focused on
>>>>>>> machine translation (MT): the quality of MT output (for many
>>>> language
>>>>>>> pairs); the average computing resources available on computers,
>>>>>> relative
>>>>>>> to the needs of MT systems; and the availability of a number of
>>>>>>> high-quality toolkits, together with a large base of researchers
>>>>>> working
>>>>>>> on them.
>>>>>>>
>>>>>>> Over the past decade, machine translation (MT; the automatic
>>>>>> translation
>>>>>>> of one human language to another) has become a reality. The
>>>>>>>research
>>>>>> into
>>>>>>> statistical approaches to translation that began in the early
>>>> nineties,
>>>>>>> together with the availability of large amounts of training data,
>>>> and
>>>>>>> better computing infrastructure, have all come together to produce
>>>>>>> translations results that are “good enough” for a large set of
>>>> language
>>>>>>> pairs and use cases. Free services like
>>>>>>> [[https://www.bing.com/translator|Bing Translator]] and
>>>>>>> [[https://translate.google.com|Google Translate]] have made these
>>>>>> services
>>>>>>> available to the average person through direct interfaces and
>>>> through
>>>>>>> tools like browser plugins, and sites across the world with higher
>>>>>>> translation needs use them to translate their pages through
>>>>>> automatically.
>>>>>>>
>>>>>>> MT does not require the infrastructure of large corporations in
>>>> order
>>>>>> to
>>>>>>> produce feasible output. Machine translation can be
>>>> resource-intensive,
>>>>>>> but need not be prohibitively so. Disk and memory usage are mostly
>>>>>>>a
>>>>>>> matter of model size, which for most language pairs is a few
>>>> gigabytes
>>>>>> at
>>>>>>> most, at which size models can provide coverage on the order of
>>>> tens or
>>>>>>> even hundreds of thousands of words in the input and output
>>>> languages.
>>>>>> The
>>>>>>> computational complexity of the algorithms used to search for
>>>>>> translations
>>>>>>> of new sentences are typically linear in the number of words in the
>>>>>> input
>>>>>>> sentence, making it possible to run a translation engine on a
>>>> personal
>>>>>>> computer.
>>>>>>>
>>>>>>> The research community has produced many different open source
>>>>>> translation
>>>>>>> projects for a range of programming languages and under a variety
>>>>>>>of
>>>>>>> licenses. These projects include the core “decoder”, which takes a
>>>>>> model
>>>>>>> and uses it to translate new sentences between the language pair
>>>>>>>the
>>>>>> model
>>>>>>> was defined for. They also typically include a large set of tools
>>>> that
>>>>>>> enable new models to be built from large sets of example
>>>> translations
>>>>>>> (“parallel data”) and monolingual texts. These toolkits are usually
>>>>>> built
>>>>>>> to support the agendas of the (largely) academic researchers that
>>>> build
>>>>>>> them: the repeated cycle of building new models, tuning model
>>>>>> parameters
>>>>>>> against development data, and evaluating them against held-out test
>>>>>> data,
>>>>>>> using standard metrics for testing the quality of MT output.
>>>>>>>
>>>>>>> Together, these three factors—the quality of machine translation
>>>>>> output,
>>>>>>> the feasibility of translating on standard computers, and the
>>>>>> availability
>>>>>>> of tools to build models—make it reasonable for the end users to
>>>>>>>use
>>>>>> MT as
>>>>>>> a black-box service, and to run it on their personal machine.
>>>>>>>
>>>>>>> These factors make it a good time for an organization with the
>>>> status
>>>>>> of
>>>>>>> the Apache Foundation to host a machine translation project.
>>>>>>>
>>>>>>> == Current Status ==
>>>>>>> Joshua was originally ported from David Chiang’s Python
>>>> implementation
>>>>>> of
>>>>>>> Hiero by Zhifei Li, while he was a Ph.D. student at Johns Hopkins
>>>>>>> University. The current version is maintained by Matt Post at Johns
>>>>>>> Hopkins’ Human Language Technology Center of Excellence. Joshua has
>>>>>> made
>>>>>>> many releases with a list of over 20 source code tags. The last
>>>>>> release of
>>>>>>> Joshua was 6.0.5 on November 5th, 2015.
>>>>>>>
>>>>>>> == Meritocracy ==
>>>>>>> The current developers are familiar with meritocratic open source
>>>>>>> development at Apache. Apache was chosen specifically because we
>>>> want
>>>>>> to
>>>>>>> encourage this style of development for the project.
>>>>>>>
>>>>>>> == Community ==
>>>>>>> Joshua is used widely across the world. Perhaps its biggest (known)
>>>>>>> research / industrial user is the Amazon research group in Berlin.
>>>>>> Another
>>>>>>> user is the US Army Research Lab. No formal census has been
>>>> undertaken,
>>>>>>> but posts to the Joshua technical support mailing list, along with
>>>> the
>>>>>>> occasional contributions, suggest small research and academic
>>>>>> communities
>>>>>>> spread across the world, many of them in India.
>>>>>>>
>>>>>>> During incubation, we will explicitly seek to increase our usage
>>>> across
>>>>>>> the board, including academic research, industry, and other end
>>>> users
>>>>>>> interested in statistical machine translation.
>>>>>>>
>>>>>>> == Core Developers ==
>>>>>>> The current set of core developers is fairly small, having fallen
>>>> with
>>>>>> the
>>>>>>> graduation from Johns Hopkins of some core student participants.
>>>>>> However,
>>>>>>> Joshua is used fairly widely, as mentioned above, and there remains
>>>> a
>>>>>>> commitment from the principal researcher at Johns Hopkins to
>>>> continue
>>>>>> to
>>>>>>> use and develop it. Joshua has seen a number of new community
>>>> members
>>>>>>> become interested recently due to a potential for its projected use
>>>> in
>>>>>> a
>>>>>>> number of ongoing DARPA projects such as XDATA and Memex.
>>>>>>>
>>>>>>> == Alignment ==
>>>>>>> Joshua is currently Copyright (c) 2015, Johns Hopkins University
>>>>>>>All
>>>>>>> rights reserved and licensed under BSD 2-clause license. It would
>>>>>>>of
>>>>>>> course be the intention to relicense this code under AL2.0 which
>>>> would
>>>>>>> permit expanded and increased use of the software within Apache
>>>>>> projects.
>>>>>>> There is currently an ongoing effort within the Apache Tika
>>>> community
>>>>>> to
>>>>>>> utilize Joshua within Tika’s Translate API, see
>>>>>>> [[https://issues.apache.org/jira/browse/TIKA-1343|TIKA-1343]].
>>>>>>>
>>>>>>> == Known Risks ==
>>>>>>>
>>>>>>> === Orphaned products ===
>>>>>>> At the moment, regular contributions are made by a single
>>>> contributor,
>>>>>> the
>>>>>>> lead maintainer. He (Matt Post) plans to continue development for
>>>> the
>>>>>> next
>>>>>>> few years, but it is still a single point of failure, since the
>>>>>> graduate
>>>>>>> students who worked on the project have moved on to jobs, mostly in
>>>>>>> industry. However, our goal is to help that process by growing the
>>>>>>> community in Apache, and at least in growing the community with
>>>> users
>>>>>> and
>>>>>>> participants from NASA JPL.
>>>>>>>
>>>>>>> === Inexperience with Open Source ===
>>>>>>> The team both at Johns Hopkins and NASA JPL have experience with
>>>> many
>>>>>> OSS
>>>>>>> software projects at Apache and elsewhere. We understand "how it
>>>> works"
>>>>>>> here at the foundation.
>>>>>>>
>>>>>>>
>>>>>>> == Relationships with Other Apache Products ==
>>>>>>> Joshua includes dependences on Hadoop, and also is included as a
>>>>>> plugin in
>>>>>>> Apache Tika. We are also interested in coordinating with other
>>>> projects
>>>>>>> including Spark, and other projects needing MT services for
>>>>>>>language
>>>>>>> translation.
>>>>>>>
>>>>>>> == Developers ==
>>>>>>> Joshua only has one regular developer who is employed by Johns
>>>> Hopkins
>>>>>>> University. NASA JPL (Mattmann and McGibbney) have been
>>>>>>>contributing
>>>>>>> lately including a Brew formula and other contributions to the
>>>> project
>>>>>>> through the DARPA XDATA and Memex programs.
>>>>>>>
>>>>>>> == Documentation ==
>>>>>>> Documentation and publications related to Joshua can be found at
>>>>>>> joshua-decoder.org. The source for the Joshua documentation is
>>>>>> currently
>>>>>>> hosted on Github at
>>>>>>> https://github.com/joshua-decoder/joshua-decoder.github.com
>>>>>>>
>>>>>>> == Initial Source ==
>>>>>>> Current source resides at Github: github.com/joshua-decoder/joshua
>>>> (the
>>>>>>> main decoder and toolkit) and github.com/joshua-decoder/thrax (the
>>>>>> grammar
>>>>>>> extraction tool).
>>>>>>>
>>>>>>> == External Dependencies ==
>>>>>>> Joshua has a number of external dependencies. Only BerkeleyLM
>>>> (Apache
>>>>>> 2.0)
>>>>>>> and KenLM (LGPG 2.1) are run-time decoder dependencies (one of
>>>> which is
>>>>>>> needed for translating sentences with pre-built models). The rest
>>>> are
>>>>>>> dependencies for the build system and pipeline, used for
>>>> constructing
>>>>>> and
>>>>>>> training new models from parallel text.
>>>>>>>
>>>>>>> Apache projects:
>>>>>>> * Ant
>>>>>>> * Hadoop
>>>>>>> * Commons
>>>>>>> * Maven
>>>>>>> * Ivy
>>>>>>>
>>>>>>> There are also a number of other open-source projects with various
>>>>>>> licenses that the project depends on both dynamically (runtime),
>>>>>>>and
>>>>>>> statically.
>>>>>>>
>>>>>>> === GNU GPL 2 ===
>>>>>>> * Berkeley Aligner: https://code.google.com/p/berkeleyaligner/
>>>>>>>
>>>>>>> === LGPG 2.1 ===
>>>>>>> * KenLM: github.com/kpu/kenlm
>>>>>>>
>>>>>>> === Apache 2.0 ===
>>>>>>> * BerkeleyLM: https://code.google.com/p/berkeleylm/
>>>>>>>
>>>>>>> === GNU GPL ===
>>>>>>> * GIZA++: http://www.statmt.org/moses/giza/GIZA++.html
>>>>>>>
>>>>>>> == Required Resources ==
>>>>>>> * Mailing Lists
>>>>>>> * private@joshua.incubator.apache.org
>>>>>>> * dev@joshua.incubator.apache.org
>>>>>>> * commits@joshua.incubator.apache.org
>>>>>>>
>>>>>>> * Git Repos
>>>>>>> * https://git-wip-us.apache.org/repos/asf/joshua.git
>>>>>>>
>>>>>>> * Issue Tracking
>>>>>>> * JIRA Joshua (JOSHUA)
>>>>>>>
>>>>>>> * Continuous Integration
>>>>>>> * Jenkins builds on https://builds.apache.org/
>>>>>>>
>>>>>>> * Web
>>>>>>> * http://joshua.incubator.apache.org/
>>>>>>> * wiki at http://cwiki.apache.org
>>>>>>>
>>>>>>> == Initial Committers ==
>>>>>>> The following is a list of the planned initial Apache committers
>>>> (the
>>>>>>> active subset of the committers for the current repository on
>>>> Github).
>>>>>>>
>>>>>>> * Matt Post (post@cs.jhu.edu)
>>>>>>> * Lewis John McGibbney (lewismc@apache.org)
>>>>>>> * Chris Mattmann (mattmann@apache.org)
>>>>>>>
>>>>>>> == Affiliations ==
>>>>>>>
>>>>>>> * Johns Hopkins University
>>>>>>> * Matt Post
>>>>>>>
>>>>>>> * NASA JPL
>>>>>>> * Chris Mattmann
>>>>>>> * Lewis John McGibbney
>>>>>>>
>>>>>>>
>>>>>>> == Sponsors ==
>>>>>>> === Champion ===
>>>>>>> * Chris Mattmann (NASA/JPL)
>>>>>>>
>>>>>>> === Nominated Mentors ===
>>>>>>> * Paul Ramirez
>>>>>>> * Lewis John McGibbney
>>>>>>> * Chris Mattmann
>>>>>>>
>>>>>>> == Sponsoring Entity ==
>>>>>>> The Apache Incubator
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>> Chris Mattmann, Ph.D.
>>>>>>> Chief Architect
>>>>>>> Instrument Software and Science Data Systems Section (398)
>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>> Office: 168-519, Mailstop: 168-527
>>>>>>> Email: chris.a.mattmann@nasa.gov
>>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>> Adjunct Associate Professor, Computer Science Department
>>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>>>>>
>>>>>>>?B�KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK
>>>>>>>KK
>>>>>>> KC
>>>>>>> B�
>>>>>>
>>>>
>>>>>>>
>>>>>>>?�?[��X��ܚX�K??K[XZ[?�?�[�\�[?][��X��ܚX�P?[��X�]?܋�\?X�?K�ܙ�B��܈?Y??
>>>>>>>]?
>>>>>>> [ۘ
>>>>>>> [?
>>>>>>> ?��[X[�?�??K[XZ[?�?�[�\�[?Z?[???[��X�]?܋�\?X�?K�ܙ�B
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>For additional commands, e-mail: general-help@incubator.apache.org
>
Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine Translation Toolkit
Posted by Matt Post <po...@cs.jhu.edu>.
The dependencies can be split into two kinds: ones required for building new models, and ones needed by the decoder to translate new sentences with a pre-built model (i.e., black-box translation with the language packs).
1. For building new models, you need a way to align the words between sentences in parallel text. Both the aligners used by Joshua (GIZA++ and the Berkeley aligner) are GPL of some form. These can be implemented as external dependencies, or can be replaced with another aligner, like fast_align (https://github.com/clab/fast_align), which is Apache-licensed. There are many other options, in fact. So this should not be a worry.
2. For doing black-box translation, one needs to represent the language model, which is very large. The best tool for this is KenLM (github.com/kpu/kenlm), which is LGPL 2.1. There is also BerkeleyLM, which is just as good for practical purposes and is Apache-licensed. KenLM is C++ and is loaded via the JNI, whereas BerkeleyLM is written in Java. I have moved to including BerkeleyLM in language packs, because I can then include the Joshua-runtime, and people can translate without even having to compile anything.
So in short, there are no hard dependencies on unfavorably-licensed external projects.
matt
> On Jan 20, 2016, at 10:08 AM, Mattmann, Chris A (3980) <ch...@jpl.nasa.gov> wrote:
>
> Hey Hen,
>
> Matt Post who I believe is monitoring this list and who has
> been one of the key Joshua developers and I have discussed this
> and we believe that potentially GPL/LGPL dependencies can:
>
> 1. be replaced with category-A or category-B alternatives. Matt
> mentioned one already to me which has slipped my mind.
> 2. be made in such a way that they are external tools and the
> bindings exist in Joshua to call those external tools (aka runtime
> deps akin to depending on a C compiler, etc.)
>
> Cheers,
> Chris
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
> -----Original Message-----
> From: Henri Yandell <ba...@apache.org>
> Reply-To: "general@incubator.apache.org" <ge...@incubator.apache.org>
> Date: Tuesday, January 19, 2016 at 7:38 PM
> To: "general@incubator.apache.org" <ge...@incubator.apache.org>
> Subject: Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine
> Translation Toolkit
>
>> License-wise, any expectation of problems from the GPL and LGPL
>> dependencies?
>>
>> On Mon, Jan 18, 2016 at 9:58 PM, Mattmann, Chris A (3980) <
>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>
>>> Great Hen, we’d love to have you on board as a mentor! Please
>>> add yourself to the proposal on the wiki.
>>>
>>> Anyone else have interest in Machine Translation? Any OpenNLP folks,
>>> Hadoop folks, Tika, or Lucene folks? CC’ing the dev lists for visibility
>>> please feel free to reply to general@i.a.o.
>>>
>>> I’ll leave the DISCUSS thread open for a few more days.
>>>
>>> Cheers,
>>> Chris
>>>
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Chris Mattmann, Ph.D.
>>> Chief Architect
>>> Instrument Software and Science Data Systems Section (398)
>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> Office: 168-519, Mailstop: 168-527
>>> Email: chris.a.mattmann@nasa.gov
>>> WWW: http://sunset.usc.edu/~mattmann/
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Adjunct Associate Professor, Computer Science Department
>>> University of Southern California, Los Angeles, CA 90089 USA
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Henri Yandell <ba...@apache.org>
>>> Reply-To: "general@incubator.apache.org" <ge...@incubator.apache.org>
>>> Date: Monday, January 18, 2016 at 7:57 PM
>>> To: jpluser <ch...@jpl.nasa.gov>,
>>> "general@incubator.apache.org" <ge...@incubator.apache.org>
>>> Subject: Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine
>>> Translation Toolkit
>>>
>>>> Non-binding +1 to Joshua joining the Incubator. I'd be interested in
>>>> mentoring.
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: jpluser <ch...@jpl.nasa.gov>
>>>>> Reply-To: "general@incubator.apache.org"
>>> <ge...@incubator.apache.org>
>>>>> Date: Tuesday, January 12, 2016 at 10:56 PM
>>>>> To: "general@incubator.apache.org" <ge...@incubator.apache.org>
>>>>> Cc: "post@cs.jhu.edu" <po...@cs.jhu.edu>
>>>>> Subject: [DISCUSS] Apache Joshua Incubator Proposal - Machine
>>>>> Translation
>>>>> Toolkit
>>>>>
>>>>>> Hi Everyone,
>>>>>>
>>>>>> Please find attached for your viewing pleasure a proposed new
>>> project,
>>>>>> Apache Joshua, a statistical machine translation toolkit. The
>>> proposal
>>>>>> is in wiki draft form at:
>>>>> https://wiki.apache.org/incubator/JoshuaProposal
>>>>>>
>>>>>> Proposal text is copied below. I’ll leave the discussion open for a
>>>>> week
>>>>>> and we are interested in folks who would like to be initial
>>> committers
>>>>>> and mentors. Please discuss here on the thread.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> Cheers,
>>>>>> Chris (Champion)
>>>>>>
>>>>>> ———
>>>>>>
>>>>>> = Joshua Proposal =
>>>>>>
>>>>>> == Abstract ==
>>>>>> [[joshua-decoder.org|Joshua]] is an open-source statistical machine
>>>>>> translation toolkit. It includes a Java-based decoder for
>>> translating
>>>>> with
>>>>>> phrase-based, hierarchical, and syntax-based translation models, a
>>>>>> Hadoop-based grammar extractor (Thrax), and an extensive set of
>>> tools
>>>>> and
>>>>>> scripts for training and evaluating new models from parallel text.
>>>>>>
>>>>>> == Proposal ==
>>>>>> Joshua is a state of the art statistical machine translation system
>>>>> that
>>>>>> provides a number of features:
>>>>>>
>>>>>> * Support for the two main paradigms in statistical machine
>>>>> translation:
>>>>>> phrase-based and hierarchical / syntactic.
>>>>>> * A sparse feature API that makes it easy to add new feature
>>> templates
>>>>>> supporting millions of features
>>>>>> * Native implementations of many tuners (MERT, MIRA, PRO, and
>>> AdaGrad)
>>>>>> * Support for lattice decoding, allowing upstream NLP tools to
>>> expose
>>>>>> their hypothesis space to the MT system
>>>>>> * An efficient representation for models, allowing for quick
>>> loading
>>>>> of
>>>>>> multi-gigabyte model files
>>>>>> * Fast decoding speed (on par with Moses and mtplz)
>>>>>> * Language packs — precompiled models that allow the decoder to be
>>>>> run as
>>>>>> a black box
>>>>>> * Thrax, a Hadoop-based tool for learning translation models from
>>>>>> parallel text
>>>>>> * A suite of tools for constructing new models for any language
>>> pair
>>>>> for
>>>>>> which sufficient training data exists
>>>>>>
>>>>>> == Background and Rationale ==
>>>>>> A number of factors make this a good time for an Apache project
>>>>> focused on
>>>>>> machine translation (MT): the quality of MT output (for many
>>> language
>>>>>> pairs); the average computing resources available on computers,
>>>>> relative
>>>>>> to the needs of MT systems; and the availability of a number of
>>>>>> high-quality toolkits, together with a large base of researchers
>>>>> working
>>>>>> on them.
>>>>>>
>>>>>> Over the past decade, machine translation (MT; the automatic
>>>>> translation
>>>>>> of one human language to another) has become a reality. The research
>>>>> into
>>>>>> statistical approaches to translation that began in the early
>>> nineties,
>>>>>> together with the availability of large amounts of training data,
>>> and
>>>>>> better computing infrastructure, have all come together to produce
>>>>>> translations results that are “good enough” for a large set of
>>> language
>>>>>> pairs and use cases. Free services like
>>>>>> [[https://www.bing.com/translator|Bing Translator]] and
>>>>>> [[https://translate.google.com|Google Translate]] have made these
>>>>> services
>>>>>> available to the average person through direct interfaces and
>>> through
>>>>>> tools like browser plugins, and sites across the world with higher
>>>>>> translation needs use them to translate their pages through
>>>>> automatically.
>>>>>>
>>>>>> MT does not require the infrastructure of large corporations in
>>> order
>>>>> to
>>>>>> produce feasible output. Machine translation can be
>>> resource-intensive,
>>>>>> but need not be prohibitively so. Disk and memory usage are mostly a
>>>>>> matter of model size, which for most language pairs is a few
>>> gigabytes
>>>>> at
>>>>>> most, at which size models can provide coverage on the order of
>>> tens or
>>>>>> even hundreds of thousands of words in the input and output
>>> languages.
>>>>> The
>>>>>> computational complexity of the algorithms used to search for
>>>>> translations
>>>>>> of new sentences are typically linear in the number of words in the
>>>>> input
>>>>>> sentence, making it possible to run a translation engine on a
>>> personal
>>>>>> computer.
>>>>>>
>>>>>> The research community has produced many different open source
>>>>> translation
>>>>>> projects for a range of programming languages and under a variety of
>>>>>> licenses. These projects include the core “decoder”, which takes a
>>>>> model
>>>>>> and uses it to translate new sentences between the language pair the
>>>>> model
>>>>>> was defined for. They also typically include a large set of tools
>>> that
>>>>>> enable new models to be built from large sets of example
>>> translations
>>>>>> (“parallel data”) and monolingual texts. These toolkits are usually
>>>>> built
>>>>>> to support the agendas of the (largely) academic researchers that
>>> build
>>>>>> them: the repeated cycle of building new models, tuning model
>>>>> parameters
>>>>>> against development data, and evaluating them against held-out test
>>>>> data,
>>>>>> using standard metrics for testing the quality of MT output.
>>>>>>
>>>>>> Together, these three factors—the quality of machine translation
>>>>> output,
>>>>>> the feasibility of translating on standard computers, and the
>>>>> availability
>>>>>> of tools to build models—make it reasonable for the end users to use
>>>>> MT as
>>>>>> a black-box service, and to run it on their personal machine.
>>>>>>
>>>>>> These factors make it a good time for an organization with the
>>> status
>>>>> of
>>>>>> the Apache Foundation to host a machine translation project.
>>>>>>
>>>>>> == Current Status ==
>>>>>> Joshua was originally ported from David Chiang’s Python
>>> implementation
>>>>> of
>>>>>> Hiero by Zhifei Li, while he was a Ph.D. student at Johns Hopkins
>>>>>> University. The current version is maintained by Matt Post at Johns
>>>>>> Hopkins’ Human Language Technology Center of Excellence. Joshua has
>>>>> made
>>>>>> many releases with a list of over 20 source code tags. The last
>>>>> release of
>>>>>> Joshua was 6.0.5 on November 5th, 2015.
>>>>>>
>>>>>> == Meritocracy ==
>>>>>> The current developers are familiar with meritocratic open source
>>>>>> development at Apache. Apache was chosen specifically because we
>>> want
>>>>> to
>>>>>> encourage this style of development for the project.
>>>>>>
>>>>>> == Community ==
>>>>>> Joshua is used widely across the world. Perhaps its biggest (known)
>>>>>> research / industrial user is the Amazon research group in Berlin.
>>>>> Another
>>>>>> user is the US Army Research Lab. No formal census has been
>>> undertaken,
>>>>>> but posts to the Joshua technical support mailing list, along with
>>> the
>>>>>> occasional contributions, suggest small research and academic
>>>>> communities
>>>>>> spread across the world, many of them in India.
>>>>>>
>>>>>> During incubation, we will explicitly seek to increase our usage
>>> across
>>>>>> the board, including academic research, industry, and other end
>>> users
>>>>>> interested in statistical machine translation.
>>>>>>
>>>>>> == Core Developers ==
>>>>>> The current set of core developers is fairly small, having fallen
>>> with
>>>>> the
>>>>>> graduation from Johns Hopkins of some core student participants.
>>>>> However,
>>>>>> Joshua is used fairly widely, as mentioned above, and there remains
>>> a
>>>>>> commitment from the principal researcher at Johns Hopkins to
>>> continue
>>>>> to
>>>>>> use and develop it. Joshua has seen a number of new community
>>> members
>>>>>> become interested recently due to a potential for its projected use
>>> in
>>>>> a
>>>>>> number of ongoing DARPA projects such as XDATA and Memex.
>>>>>>
>>>>>> == Alignment ==
>>>>>> Joshua is currently Copyright (c) 2015, Johns Hopkins University All
>>>>>> rights reserved and licensed under BSD 2-clause license. It would of
>>>>>> course be the intention to relicense this code under AL2.0 which
>>> would
>>>>>> permit expanded and increased use of the software within Apache
>>>>> projects.
>>>>>> There is currently an ongoing effort within the Apache Tika
>>> community
>>>>> to
>>>>>> utilize Joshua within Tika’s Translate API, see
>>>>>> [[https://issues.apache.org/jira/browse/TIKA-1343|TIKA-1343]].
>>>>>>
>>>>>> == Known Risks ==
>>>>>>
>>>>>> === Orphaned products ===
>>>>>> At the moment, regular contributions are made by a single
>>> contributor,
>>>>> the
>>>>>> lead maintainer. He (Matt Post) plans to continue development for
>>> the
>>>>> next
>>>>>> few years, but it is still a single point of failure, since the
>>>>> graduate
>>>>>> students who worked on the project have moved on to jobs, mostly in
>>>>>> industry. However, our goal is to help that process by growing the
>>>>>> community in Apache, and at least in growing the community with
>>> users
>>>>> and
>>>>>> participants from NASA JPL.
>>>>>>
>>>>>> === Inexperience with Open Source ===
>>>>>> The team both at Johns Hopkins and NASA JPL have experience with
>>> many
>>>>> OSS
>>>>>> software projects at Apache and elsewhere. We understand "how it
>>> works"
>>>>>> here at the foundation.
>>>>>>
>>>>>>
>>>>>> == Relationships with Other Apache Products ==
>>>>>> Joshua includes dependences on Hadoop, and also is included as a
>>>>> plugin in
>>>>>> Apache Tika. We are also interested in coordinating with other
>>> projects
>>>>>> including Spark, and other projects needing MT services for language
>>>>>> translation.
>>>>>>
>>>>>> == Developers ==
>>>>>> Joshua only has one regular developer who is employed by Johns
>>> Hopkins
>>>>>> University. NASA JPL (Mattmann and McGibbney) have been contributing
>>>>>> lately including a Brew formula and other contributions to the
>>> project
>>>>>> through the DARPA XDATA and Memex programs.
>>>>>>
>>>>>> == Documentation ==
>>>>>> Documentation and publications related to Joshua can be found at
>>>>>> joshua-decoder.org. The source for the Joshua documentation is
>>>>> currently
>>>>>> hosted on Github at
>>>>>> https://github.com/joshua-decoder/joshua-decoder.github.com
>>>>>>
>>>>>> == Initial Source ==
>>>>>> Current source resides at Github: github.com/joshua-decoder/joshua
>>> (the
>>>>>> main decoder and toolkit) and github.com/joshua-decoder/thrax (the
>>>>> grammar
>>>>>> extraction tool).
>>>>>>
>>>>>> == External Dependencies ==
>>>>>> Joshua has a number of external dependencies. Only BerkeleyLM
>>> (Apache
>>>>> 2.0)
>>>>>> and KenLM (LGPG 2.1) are run-time decoder dependencies (one of
>>> which is
>>>>>> needed for translating sentences with pre-built models). The rest
>>> are
>>>>>> dependencies for the build system and pipeline, used for
>>> constructing
>>>>> and
>>>>>> training new models from parallel text.
>>>>>>
>>>>>> Apache projects:
>>>>>> * Ant
>>>>>> * Hadoop
>>>>>> * Commons
>>>>>> * Maven
>>>>>> * Ivy
>>>>>>
>>>>>> There are also a number of other open-source projects with various
>>>>>> licenses that the project depends on both dynamically (runtime), and
>>>>>> statically.
>>>>>>
>>>>>> === GNU GPL 2 ===
>>>>>> * Berkeley Aligner: https://code.google.com/p/berkeleyaligner/
>>>>>>
>>>>>> === LGPG 2.1 ===
>>>>>> * KenLM: github.com/kpu/kenlm
>>>>>>
>>>>>> === Apache 2.0 ===
>>>>>> * BerkeleyLM: https://code.google.com/p/berkeleylm/
>>>>>>
>>>>>> === GNU GPL ===
>>>>>> * GIZA++: http://www.statmt.org/moses/giza/GIZA++.html
>>>>>>
>>>>>> == Required Resources ==
>>>>>> * Mailing Lists
>>>>>> * private@joshua.incubator.apache.org
>>>>>> * dev@joshua.incubator.apache.org
>>>>>> * commits@joshua.incubator.apache.org
>>>>>>
>>>>>> * Git Repos
>>>>>> * https://git-wip-us.apache.org/repos/asf/joshua.git
>>>>>>
>>>>>> * Issue Tracking
>>>>>> * JIRA Joshua (JOSHUA)
>>>>>>
>>>>>> * Continuous Integration
>>>>>> * Jenkins builds on https://builds.apache.org/
>>>>>>
>>>>>> * Web
>>>>>> * http://joshua.incubator.apache.org/
>>>>>> * wiki at http://cwiki.apache.org
>>>>>>
>>>>>> == Initial Committers ==
>>>>>> The following is a list of the planned initial Apache committers
>>> (the
>>>>>> active subset of the committers for the current repository on
>>> Github).
>>>>>>
>>>>>> * Matt Post (post@cs.jhu.edu)
>>>>>> * Lewis John McGibbney (lewismc@apache.org)
>>>>>> * Chris Mattmann (mattmann@apache.org)
>>>>>>
>>>>>> == Affiliations ==
>>>>>>
>>>>>> * Johns Hopkins University
>>>>>> * Matt Post
>>>>>>
>>>>>> * NASA JPL
>>>>>> * Chris Mattmann
>>>>>> * Lewis John McGibbney
>>>>>>
>>>>>>
>>>>>> == Sponsors ==
>>>>>> === Champion ===
>>>>>> * Chris Mattmann (NASA/JPL)
>>>>>>
>>>>>> === Nominated Mentors ===
>>>>>> * Paul Ramirez
>>>>>> * Lewis John McGibbney
>>>>>> * Chris Mattmann
>>>>>>
>>>>>> == Sponsoring Entity ==
>>>>>> The Apache Incubator
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>> Chris Mattmann, Ph.D.
>>>>>> Chief Architect
>>>>>> Instrument Software and Science Data Systems Section (398)
>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>> Office: 168-519, Mailstop: 168-527
>>>>>> Email: chris.a.mattmann@nasa.gov
>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>> Adjunct Associate Professor, Computer Science Department
>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
>>>>>> ?B�KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK
>>>>>> KC
>>>>>> B�
>>>>>
>>>
>>>>>> ?�?[��X��ܚX�K??K[XZ[?�?�[�\�[?][��X��ܚX�P?[��X�]?܋�\?X�?K�ܙ�B��܈?Y??]?
>>>>>> [ۘ
>>>>>> [?
>>>>>> ?��[X[�?�??K[XZ[?�?�[�\�[?Z?[???[��X�]?܋�\?X�?K�ܙ�B
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine
Translation Toolkit
Posted by "Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov>.
Hey Hen,
Matt Post who I believe is monitoring this list and who has
been one of the key Joshua developers and I have discussed this
and we believe that potentially GPL/LGPL dependencies can:
1. be replaced with category-A or category-B alternatives. Matt
mentioned one already to me which has slipped my mind.
2. be made in such a way that they are external tools and the
bindings exist in Joshua to call those external tools (aka runtime
deps akin to depending on a C compiler, etc.)
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Henri Yandell <ba...@apache.org>
Reply-To: "general@incubator.apache.org" <ge...@incubator.apache.org>
Date: Tuesday, January 19, 2016 at 7:38 PM
To: "general@incubator.apache.org" <ge...@incubator.apache.org>
Subject: Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine
Translation Toolkit
>License-wise, any expectation of problems from the GPL and LGPL
>dependencies?
>
>On Mon, Jan 18, 2016 at 9:58 PM, Mattmann, Chris A (3980) <
>chris.a.mattmann@jpl.nasa.gov> wrote:
>
>> Great Hen, we’d love to have you on board as a mentor! Please
>> add yourself to the proposal on the wiki.
>>
>> Anyone else have interest in Machine Translation? Any OpenNLP folks,
>> Hadoop folks, Tika, or Lucene folks? CC’ing the dev lists for visibility
>> please feel free to reply to general@i.a.o.
>>
>> I’ll leave the DISCUSS thread open for a few more days.
>>
>> Cheers,
>> Chris
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Chief Architect
>> Instrument Software and Science Data Systems Section (398)
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 168-519, Mailstop: 168-527
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Associate Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Henri Yandell <ba...@apache.org>
>> Reply-To: "general@incubator.apache.org" <ge...@incubator.apache.org>
>> Date: Monday, January 18, 2016 at 7:57 PM
>> To: jpluser <ch...@jpl.nasa.gov>,
>> "general@incubator.apache.org" <ge...@incubator.apache.org>
>> Subject: Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine
>> Translation Toolkit
>>
>> >Non-binding +1 to Joshua joining the Incubator. I'd be interested in
>> >mentoring.
>> >
>> >
>> >> -----Original Message-----
>> >> From: jpluser <ch...@jpl.nasa.gov>
>> >> Reply-To: "general@incubator.apache.org"
>><ge...@incubator.apache.org>
>> >> Date: Tuesday, January 12, 2016 at 10:56 PM
>> >> To: "general@incubator.apache.org" <ge...@incubator.apache.org>
>> >> Cc: "post@cs.jhu.edu" <po...@cs.jhu.edu>
>> >> Subject: [DISCUSS] Apache Joshua Incubator Proposal - Machine
>> >>Translation
>> >> Toolkit
>> >>
>> >> >Hi Everyone,
>> >> >
>> >> >Please find attached for your viewing pleasure a proposed new
>>project,
>> >> >Apache Joshua, a statistical machine translation toolkit. The
>>proposal
>> >> >is in wiki draft form at:
>> >> https://wiki.apache.org/incubator/JoshuaProposal
>> >> >
>> >> >Proposal text is copied below. I’ll leave the discussion open for a
>> >>week
>> >> >and we are interested in folks who would like to be initial
>>committers
>> >> >and mentors. Please discuss here on the thread.
>> >> >
>> >> >Thanks!
>> >> >
>> >> >Cheers,
>> >> >Chris (Champion)
>> >> >
>> >> >———
>> >> >
>> >> >= Joshua Proposal =
>> >> >
>> >> >== Abstract ==
>> >> >[[joshua-decoder.org|Joshua]] is an open-source statistical machine
>> >> >translation toolkit. It includes a Java-based decoder for
>>translating
>> >>with
>> >> >phrase-based, hierarchical, and syntax-based translation models, a
>> >> >Hadoop-based grammar extractor (Thrax), and an extensive set of
>>tools
>> >>and
>> >> >scripts for training and evaluating new models from parallel text.
>> >> >
>> >> >== Proposal ==
>> >> >Joshua is a state of the art statistical machine translation system
>> >>that
>> >> >provides a number of features:
>> >> >
>> >> > * Support for the two main paradigms in statistical machine
>> >>translation:
>> >> >phrase-based and hierarchical / syntactic.
>> >> > * A sparse feature API that makes it easy to add new feature
>>templates
>> >> >supporting millions of features
>> >> > * Native implementations of many tuners (MERT, MIRA, PRO, and
>>AdaGrad)
>> >> > * Support for lattice decoding, allowing upstream NLP tools to
>>expose
>> >> >their hypothesis space to the MT system
>> >> > * An efficient representation for models, allowing for quick
>>loading
>> >>of
>> >> >multi-gigabyte model files
>> >> > * Fast decoding speed (on par with Moses and mtplz)
>> >> > * Language packs — precompiled models that allow the decoder to be
>> >>run as
>> >> >a black box
>> >> > * Thrax, a Hadoop-based tool for learning translation models from
>> >> >parallel text
>> >> > * A suite of tools for constructing new models for any language
>>pair
>> >>for
>> >> >which sufficient training data exists
>> >> >
>> >> >== Background and Rationale ==
>> >> >A number of factors make this a good time for an Apache project
>> >>focused on
>> >> >machine translation (MT): the quality of MT output (for many
>>language
>> >> >pairs); the average computing resources available on computers,
>> >>relative
>> >> >to the needs of MT systems; and the availability of a number of
>> >> >high-quality toolkits, together with a large base of researchers
>> >>working
>> >> >on them.
>> >> >
>> >> >Over the past decade, machine translation (MT; the automatic
>> >>translation
>> >> >of one human language to another) has become a reality. The research
>> >>into
>> >> >statistical approaches to translation that began in the early
>>nineties,
>> >> >together with the availability of large amounts of training data,
>>and
>> >> >better computing infrastructure, have all come together to produce
>> >> >translations results that are “good enough” for a large set of
>>language
>> >> >pairs and use cases. Free services like
>> >> >[[https://www.bing.com/translator|Bing Translator]] and
>> >> >[[https://translate.google.com|Google Translate]] have made these
>> >> services
>> >> >available to the average person through direct interfaces and
>>through
>> >> >tools like browser plugins, and sites across the world with higher
>> >> >translation needs use them to translate their pages through
>> >>automatically.
>> >> >
>> >> >MT does not require the infrastructure of large corporations in
>>order
>> >>to
>> >> >produce feasible output. Machine translation can be
>>resource-intensive,
>> >> >but need not be prohibitively so. Disk and memory usage are mostly a
>> >> >matter of model size, which for most language pairs is a few
>>gigabytes
>> >>at
>> >> >most, at which size models can provide coverage on the order of
>>tens or
>> >> >even hundreds of thousands of words in the input and output
>>languages.
>> >>The
>> >> >computational complexity of the algorithms used to search for
>> >>translations
>> >> >of new sentences are typically linear in the number of words in the
>> >>input
>> >> >sentence, making it possible to run a translation engine on a
>>personal
>> >> >computer.
>> >> >
>> >> >The research community has produced many different open source
>> >>translation
>> >> >projects for a range of programming languages and under a variety of
>> >> >licenses. These projects include the core “decoder”, which takes a
>> >>model
>> >> >and uses it to translate new sentences between the language pair the
>> >>model
>> >> >was defined for. They also typically include a large set of tools
>>that
>> >> >enable new models to be built from large sets of example
>>translations
>> >> >(“parallel data”) and monolingual texts. These toolkits are usually
>> >>built
>> >> >to support the agendas of the (largely) academic researchers that
>>build
>> >> >them: the repeated cycle of building new models, tuning model
>> >>parameters
>> >> >against development data, and evaluating them against held-out test
>> >>data,
>> >> >using standard metrics for testing the quality of MT output.
>> >> >
>> >> >Together, these three factors—the quality of machine translation
>> >>output,
>> >> >the feasibility of translating on standard computers, and the
>> >>availability
>> >> >of tools to build models—make it reasonable for the end users to use
>> >>MT as
>> >> >a black-box service, and to run it on their personal machine.
>> >> >
>> >> >These factors make it a good time for an organization with the
>>status
>> >>of
>> >> >the Apache Foundation to host a machine translation project.
>> >> >
>> >> >== Current Status ==
>> >> >Joshua was originally ported from David Chiang’s Python
>>implementation
>> >>of
>> >> >Hiero by Zhifei Li, while he was a Ph.D. student at Johns Hopkins
>> >> >University. The current version is maintained by Matt Post at Johns
>> >> >Hopkins’ Human Language Technology Center of Excellence. Joshua has
>> >>made
>> >> >many releases with a list of over 20 source code tags. The last
>> >>release of
>> >> >Joshua was 6.0.5 on November 5th, 2015.
>> >> >
>> >> >== Meritocracy ==
>> >> >The current developers are familiar with meritocratic open source
>> >> >development at Apache. Apache was chosen specifically because we
>>want
>> >>to
>> >> >encourage this style of development for the project.
>> >> >
>> >> >== Community ==
>> >> >Joshua is used widely across the world. Perhaps its biggest (known)
>> >> >research / industrial user is the Amazon research group in Berlin.
>> >>Another
>> >> >user is the US Army Research Lab. No formal census has been
>>undertaken,
>> >> >but posts to the Joshua technical support mailing list, along with
>>the
>> >> >occasional contributions, suggest small research and academic
>> >>communities
>> >> >spread across the world, many of them in India.
>> >> >
>> >> >During incubation, we will explicitly seek to increase our usage
>>across
>> >> >the board, including academic research, industry, and other end
>>users
>> >> >interested in statistical machine translation.
>> >> >
>> >> >== Core Developers ==
>> >> >The current set of core developers is fairly small, having fallen
>>with
>> >>the
>> >> >graduation from Johns Hopkins of some core student participants.
>> >>However,
>> >> >Joshua is used fairly widely, as mentioned above, and there remains
>>a
>> >> >commitment from the principal researcher at Johns Hopkins to
>>continue
>> >>to
>> >> >use and develop it. Joshua has seen a number of new community
>>members
>> >> >become interested recently due to a potential for its projected use
>>in
>> >>a
>> >> >number of ongoing DARPA projects such as XDATA and Memex.
>> >> >
>> >> >== Alignment ==
>> >> >Joshua is currently Copyright (c) 2015, Johns Hopkins University All
>> >> >rights reserved and licensed under BSD 2-clause license. It would of
>> >> >course be the intention to relicense this code under AL2.0 which
>>would
>> >> >permit expanded and increased use of the software within Apache
>> >>projects.
>> >> >There is currently an ongoing effort within the Apache Tika
>>community
>> >>to
>> >> >utilize Joshua within Tika’s Translate API, see
>> >> >[[https://issues.apache.org/jira/browse/TIKA-1343|TIKA-1343]].
>> >> >
>> >> >== Known Risks ==
>> >> >
>> >> >=== Orphaned products ===
>> >> >At the moment, regular contributions are made by a single
>>contributor,
>> >>the
>> >> >lead maintainer. He (Matt Post) plans to continue development for
>>the
>> >>next
>> >> >few years, but it is still a single point of failure, since the
>> >>graduate
>> >> >students who worked on the project have moved on to jobs, mostly in
>> >> >industry. However, our goal is to help that process by growing the
>> >> >community in Apache, and at least in growing the community with
>>users
>> >>and
>> >> >participants from NASA JPL.
>> >> >
>> >> >=== Inexperience with Open Source ===
>> >> >The team both at Johns Hopkins and NASA JPL have experience with
>>many
>> >>OSS
>> >> >software projects at Apache and elsewhere. We understand "how it
>>works"
>> >> >here at the foundation.
>> >> >
>> >> >
>> >> >== Relationships with Other Apache Products ==
>> >> >Joshua includes dependences on Hadoop, and also is included as a
>> >>plugin in
>> >> >Apache Tika. We are also interested in coordinating with other
>>projects
>> >> >including Spark, and other projects needing MT services for language
>> >> >translation.
>> >> >
>> >> >== Developers ==
>> >> >Joshua only has one regular developer who is employed by Johns
>>Hopkins
>> >> >University. NASA JPL (Mattmann and McGibbney) have been contributing
>> >> >lately including a Brew formula and other contributions to the
>>project
>> >> >through the DARPA XDATA and Memex programs.
>> >> >
>> >> >== Documentation ==
>> >> >Documentation and publications related to Joshua can be found at
>> >> >joshua-decoder.org. The source for the Joshua documentation is
>> >>currently
>> >> >hosted on Github at
>> >> >https://github.com/joshua-decoder/joshua-decoder.github.com
>> >> >
>> >> >== Initial Source ==
>> >> >Current source resides at Github: github.com/joshua-decoder/joshua
>> (the
>> >> >main decoder and toolkit) and github.com/joshua-decoder/thrax (the
>> >> grammar
>> >> >extraction tool).
>> >> >
>> >> >== External Dependencies ==
>> >> >Joshua has a number of external dependencies. Only BerkeleyLM
>>(Apache
>> >>2.0)
>> >> >and KenLM (LGPG 2.1) are run-time decoder dependencies (one of
>>which is
>> >> >needed for translating sentences with pre-built models). The rest
>>are
>> >> >dependencies for the build system and pipeline, used for
>>constructing
>> >>and
>> >> >training new models from parallel text.
>> >> >
>> >> >Apache projects:
>> >> > * Ant
>> >> > * Hadoop
>> >> > * Commons
>> >> > * Maven
>> >> > * Ivy
>> >> >
>> >> >There are also a number of other open-source projects with various
>> >> >licenses that the project depends on both dynamically (runtime), and
>> >> >statically.
>> >> >
>> >> >=== GNU GPL 2 ===
>> >> > * Berkeley Aligner: https://code.google.com/p/berkeleyaligner/
>> >> >
>> >> >=== LGPG 2.1 ===
>> >> > * KenLM: github.com/kpu/kenlm
>> >> >
>> >> >=== Apache 2.0 ===
>> >> > * BerkeleyLM: https://code.google.com/p/berkeleylm/
>> >> >
>> >> >=== GNU GPL ===
>> >> > * GIZA++: http://www.statmt.org/moses/giza/GIZA++.html
>> >> >
>> >> >== Required Resources ==
>> >> > * Mailing Lists
>> >> > * private@joshua.incubator.apache.org
>> >> > * dev@joshua.incubator.apache.org
>> >> > * commits@joshua.incubator.apache.org
>> >> >
>> >> > * Git Repos
>> >> > * https://git-wip-us.apache.org/repos/asf/joshua.git
>> >> >
>> >> > * Issue Tracking
>> >> > * JIRA Joshua (JOSHUA)
>> >> >
>> >> > * Continuous Integration
>> >> > * Jenkins builds on https://builds.apache.org/
>> >> >
>> >> > * Web
>> >> > * http://joshua.incubator.apache.org/
>> >> > * wiki at http://cwiki.apache.org
>> >> >
>> >> >== Initial Committers ==
>> >> >The following is a list of the planned initial Apache committers
>>(the
>> >> >active subset of the committers for the current repository on
>>Github).
>> >> >
>> >> > * Matt Post (post@cs.jhu.edu)
>> >> > * Lewis John McGibbney (lewismc@apache.org)
>> >> > * Chris Mattmann (mattmann@apache.org)
>> >> >
>> >> >== Affiliations ==
>> >> >
>> >> > * Johns Hopkins University
>> >> > * Matt Post
>> >> >
>> >> > * NASA JPL
>> >> > * Chris Mattmann
>> >> > * Lewis John McGibbney
>> >> >
>> >> >
>> >> >== Sponsors ==
>> >> >=== Champion ===
>> >> > * Chris Mattmann (NASA/JPL)
>> >> >
>> >> >=== Nominated Mentors ===
>> >> > * Paul Ramirez
>> >> > * Lewis John McGibbney
>> >> > * Chris Mattmann
>> >> >
>> >> >== Sponsoring Entity ==
>> >> >The Apache Incubator
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> >Chris Mattmann, Ph.D.
>> >> >Chief Architect
>> >> >Instrument Software and Science Data Systems Section (398)
>> >> >NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> >> >Office: 168-519, Mailstop: 168-527
>> >> >Email: chris.a.mattmann@nasa.gov
>> >> >WWW: http://sunset.usc.edu/~mattmann/
>> >> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> >Adjunct Associate Professor, Computer Science Department
>> >> >University of Southern California, Los Angeles, CA 90089 USA
>> >> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> >
>> >> >
>> >> >
>> >>
>>
>>>>>?B�KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK
>>>>>KC
>> >>>B�
>> >>
>>
>>>>>?�?[��X��ܚX�K??K[XZ[?�?�[�\�[?][��X��ܚX�P?[��X�]?܋�\?X�?K�ܙ�B��܈?Y??]?
>>>>>[ۘ
>> >>>[?
>> >> >?��[X[�?�??K[XZ[?�?�[�\�[?Z?[???[��X�]?܋�\?X�?K�ܙ�B
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>>
Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine Translation Toolkit
Posted by Henri Yandell <ba...@apache.org>.
License-wise, any expectation of problems from the GPL and LGPL
dependencies?
On Mon, Jan 18, 2016 at 9:58 PM, Mattmann, Chris A (3980) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Great Hen, we’d love to have you on board as a mentor! Please
> add yourself to the proposal on the wiki.
>
> Anyone else have interest in Machine Translation? Any OpenNLP folks,
> Hadoop folks, Tika, or Lucene folks? CC’ing the dev lists for visibility
> please feel free to reply to general@i.a.o.
>
> I’ll leave the DISCUSS thread open for a few more days.
>
> Cheers,
> Chris
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
> -----Original Message-----
> From: Henri Yandell <ba...@apache.org>
> Reply-To: "general@incubator.apache.org" <ge...@incubator.apache.org>
> Date: Monday, January 18, 2016 at 7:57 PM
> To: jpluser <ch...@jpl.nasa.gov>,
> "general@incubator.apache.org" <ge...@incubator.apache.org>
> Subject: Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine
> Translation Toolkit
>
> >Non-binding +1 to Joshua joining the Incubator. I'd be interested in
> >mentoring.
> >
> >
> >> -----Original Message-----
> >> From: jpluser <ch...@jpl.nasa.gov>
> >> Reply-To: "general@incubator.apache.org" <ge...@incubator.apache.org>
> >> Date: Tuesday, January 12, 2016 at 10:56 PM
> >> To: "general@incubator.apache.org" <ge...@incubator.apache.org>
> >> Cc: "post@cs.jhu.edu" <po...@cs.jhu.edu>
> >> Subject: [DISCUSS] Apache Joshua Incubator Proposal - Machine
> >>Translation
> >> Toolkit
> >>
> >> >Hi Everyone,
> >> >
> >> >Please find attached for your viewing pleasure a proposed new project,
> >> >Apache Joshua, a statistical machine translation toolkit. The proposal
> >> >is in wiki draft form at:
> >> https://wiki.apache.org/incubator/JoshuaProposal
> >> >
> >> >Proposal text is copied below. I’ll leave the discussion open for a
> >>week
> >> >and we are interested in folks who would like to be initial committers
> >> >and mentors. Please discuss here on the thread.
> >> >
> >> >Thanks!
> >> >
> >> >Cheers,
> >> >Chris (Champion)
> >> >
> >> >———
> >> >
> >> >= Joshua Proposal =
> >> >
> >> >== Abstract ==
> >> >[[joshua-decoder.org|Joshua]] is an open-source statistical machine
> >> >translation toolkit. It includes a Java-based decoder for translating
> >>with
> >> >phrase-based, hierarchical, and syntax-based translation models, a
> >> >Hadoop-based grammar extractor (Thrax), and an extensive set of tools
> >>and
> >> >scripts for training and evaluating new models from parallel text.
> >> >
> >> >== Proposal ==
> >> >Joshua is a state of the art statistical machine translation system
> >>that
> >> >provides a number of features:
> >> >
> >> > * Support for the two main paradigms in statistical machine
> >>translation:
> >> >phrase-based and hierarchical / syntactic.
> >> > * A sparse feature API that makes it easy to add new feature templates
> >> >supporting millions of features
> >> > * Native implementations of many tuners (MERT, MIRA, PRO, and AdaGrad)
> >> > * Support for lattice decoding, allowing upstream NLP tools to expose
> >> >their hypothesis space to the MT system
> >> > * An efficient representation for models, allowing for quick loading
> >>of
> >> >multi-gigabyte model files
> >> > * Fast decoding speed (on par with Moses and mtplz)
> >> > * Language packs — precompiled models that allow the decoder to be
> >>run as
> >> >a black box
> >> > * Thrax, a Hadoop-based tool for learning translation models from
> >> >parallel text
> >> > * A suite of tools for constructing new models for any language pair
> >>for
> >> >which sufficient training data exists
> >> >
> >> >== Background and Rationale ==
> >> >A number of factors make this a good time for an Apache project
> >>focused on
> >> >machine translation (MT): the quality of MT output (for many language
> >> >pairs); the average computing resources available on computers,
> >>relative
> >> >to the needs of MT systems; and the availability of a number of
> >> >high-quality toolkits, together with a large base of researchers
> >>working
> >> >on them.
> >> >
> >> >Over the past decade, machine translation (MT; the automatic
> >>translation
> >> >of one human language to another) has become a reality. The research
> >>into
> >> >statistical approaches to translation that began in the early nineties,
> >> >together with the availability of large amounts of training data, and
> >> >better computing infrastructure, have all come together to produce
> >> >translations results that are “good enough” for a large set of language
> >> >pairs and use cases. Free services like
> >> >[[https://www.bing.com/translator|Bing Translator]] and
> >> >[[https://translate.google.com|Google Translate]] have made these
> >> services
> >> >available to the average person through direct interfaces and through
> >> >tools like browser plugins, and sites across the world with higher
> >> >translation needs use them to translate their pages through
> >>automatically.
> >> >
> >> >MT does not require the infrastructure of large corporations in order
> >>to
> >> >produce feasible output. Machine translation can be resource-intensive,
> >> >but need not be prohibitively so. Disk and memory usage are mostly a
> >> >matter of model size, which for most language pairs is a few gigabytes
> >>at
> >> >most, at which size models can provide coverage on the order of tens or
> >> >even hundreds of thousands of words in the input and output languages.
> >>The
> >> >computational complexity of the algorithms used to search for
> >>translations
> >> >of new sentences are typically linear in the number of words in the
> >>input
> >> >sentence, making it possible to run a translation engine on a personal
> >> >computer.
> >> >
> >> >The research community has produced many different open source
> >>translation
> >> >projects for a range of programming languages and under a variety of
> >> >licenses. These projects include the core “decoder”, which takes a
> >>model
> >> >and uses it to translate new sentences between the language pair the
> >>model
> >> >was defined for. They also typically include a large set of tools that
> >> >enable new models to be built from large sets of example translations
> >> >(“parallel data”) and monolingual texts. These toolkits are usually
> >>built
> >> >to support the agendas of the (largely) academic researchers that build
> >> >them: the repeated cycle of building new models, tuning model
> >>parameters
> >> >against development data, and evaluating them against held-out test
> >>data,
> >> >using standard metrics for testing the quality of MT output.
> >> >
> >> >Together, these three factors—the quality of machine translation
> >>output,
> >> >the feasibility of translating on standard computers, and the
> >>availability
> >> >of tools to build models—make it reasonable for the end users to use
> >>MT as
> >> >a black-box service, and to run it on their personal machine.
> >> >
> >> >These factors make it a good time for an organization with the status
> >>of
> >> >the Apache Foundation to host a machine translation project.
> >> >
> >> >== Current Status ==
> >> >Joshua was originally ported from David Chiang’s Python implementation
> >>of
> >> >Hiero by Zhifei Li, while he was a Ph.D. student at Johns Hopkins
> >> >University. The current version is maintained by Matt Post at Johns
> >> >Hopkins’ Human Language Technology Center of Excellence. Joshua has
> >>made
> >> >many releases with a list of over 20 source code tags. The last
> >>release of
> >> >Joshua was 6.0.5 on November 5th, 2015.
> >> >
> >> >== Meritocracy ==
> >> >The current developers are familiar with meritocratic open source
> >> >development at Apache. Apache was chosen specifically because we want
> >>to
> >> >encourage this style of development for the project.
> >> >
> >> >== Community ==
> >> >Joshua is used widely across the world. Perhaps its biggest (known)
> >> >research / industrial user is the Amazon research group in Berlin.
> >>Another
> >> >user is the US Army Research Lab. No formal census has been undertaken,
> >> >but posts to the Joshua technical support mailing list, along with the
> >> >occasional contributions, suggest small research and academic
> >>communities
> >> >spread across the world, many of them in India.
> >> >
> >> >During incubation, we will explicitly seek to increase our usage across
> >> >the board, including academic research, industry, and other end users
> >> >interested in statistical machine translation.
> >> >
> >> >== Core Developers ==
> >> >The current set of core developers is fairly small, having fallen with
> >>the
> >> >graduation from Johns Hopkins of some core student participants.
> >>However,
> >> >Joshua is used fairly widely, as mentioned above, and there remains a
> >> >commitment from the principal researcher at Johns Hopkins to continue
> >>to
> >> >use and develop it. Joshua has seen a number of new community members
> >> >become interested recently due to a potential for its projected use in
> >>a
> >> >number of ongoing DARPA projects such as XDATA and Memex.
> >> >
> >> >== Alignment ==
> >> >Joshua is currently Copyright (c) 2015, Johns Hopkins University All
> >> >rights reserved and licensed under BSD 2-clause license. It would of
> >> >course be the intention to relicense this code under AL2.0 which would
> >> >permit expanded and increased use of the software within Apache
> >>projects.
> >> >There is currently an ongoing effort within the Apache Tika community
> >>to
> >> >utilize Joshua within Tika’s Translate API, see
> >> >[[https://issues.apache.org/jira/browse/TIKA-1343|TIKA-1343]].
> >> >
> >> >== Known Risks ==
> >> >
> >> >=== Orphaned products ===
> >> >At the moment, regular contributions are made by a single contributor,
> >>the
> >> >lead maintainer. He (Matt Post) plans to continue development for the
> >>next
> >> >few years, but it is still a single point of failure, since the
> >>graduate
> >> >students who worked on the project have moved on to jobs, mostly in
> >> >industry. However, our goal is to help that process by growing the
> >> >community in Apache, and at least in growing the community with users
> >>and
> >> >participants from NASA JPL.
> >> >
> >> >=== Inexperience with Open Source ===
> >> >The team both at Johns Hopkins and NASA JPL have experience with many
> >>OSS
> >> >software projects at Apache and elsewhere. We understand "how it works"
> >> >here at the foundation.
> >> >
> >> >
> >> >== Relationships with Other Apache Products ==
> >> >Joshua includes dependences on Hadoop, and also is included as a
> >>plugin in
> >> >Apache Tika. We are also interested in coordinating with other projects
> >> >including Spark, and other projects needing MT services for language
> >> >translation.
> >> >
> >> >== Developers ==
> >> >Joshua only has one regular developer who is employed by Johns Hopkins
> >> >University. NASA JPL (Mattmann and McGibbney) have been contributing
> >> >lately including a Brew formula and other contributions to the project
> >> >through the DARPA XDATA and Memex programs.
> >> >
> >> >== Documentation ==
> >> >Documentation and publications related to Joshua can be found at
> >> >joshua-decoder.org. The source for the Joshua documentation is
> >>currently
> >> >hosted on Github at
> >> >https://github.com/joshua-decoder/joshua-decoder.github.com
> >> >
> >> >== Initial Source ==
> >> >Current source resides at Github: github.com/joshua-decoder/joshua
> (the
> >> >main decoder and toolkit) and github.com/joshua-decoder/thrax (the
> >> grammar
> >> >extraction tool).
> >> >
> >> >== External Dependencies ==
> >> >Joshua has a number of external dependencies. Only BerkeleyLM (Apache
> >>2.0)
> >> >and KenLM (LGPG 2.1) are run-time decoder dependencies (one of which is
> >> >needed for translating sentences with pre-built models). The rest are
> >> >dependencies for the build system and pipeline, used for constructing
> >>and
> >> >training new models from parallel text.
> >> >
> >> >Apache projects:
> >> > * Ant
> >> > * Hadoop
> >> > * Commons
> >> > * Maven
> >> > * Ivy
> >> >
> >> >There are also a number of other open-source projects with various
> >> >licenses that the project depends on both dynamically (runtime), and
> >> >statically.
> >> >
> >> >=== GNU GPL 2 ===
> >> > * Berkeley Aligner: https://code.google.com/p/berkeleyaligner/
> >> >
> >> >=== LGPG 2.1 ===
> >> > * KenLM: github.com/kpu/kenlm
> >> >
> >> >=== Apache 2.0 ===
> >> > * BerkeleyLM: https://code.google.com/p/berkeleylm/
> >> >
> >> >=== GNU GPL ===
> >> > * GIZA++: http://www.statmt.org/moses/giza/GIZA++.html
> >> >
> >> >== Required Resources ==
> >> > * Mailing Lists
> >> > * private@joshua.incubator.apache.org
> >> > * dev@joshua.incubator.apache.org
> >> > * commits@joshua.incubator.apache.org
> >> >
> >> > * Git Repos
> >> > * https://git-wip-us.apache.org/repos/asf/joshua.git
> >> >
> >> > * Issue Tracking
> >> > * JIRA Joshua (JOSHUA)
> >> >
> >> > * Continuous Integration
> >> > * Jenkins builds on https://builds.apache.org/
> >> >
> >> > * Web
> >> > * http://joshua.incubator.apache.org/
> >> > * wiki at http://cwiki.apache.org
> >> >
> >> >== Initial Committers ==
> >> >The following is a list of the planned initial Apache committers (the
> >> >active subset of the committers for the current repository on Github).
> >> >
> >> > * Matt Post (post@cs.jhu.edu)
> >> > * Lewis John McGibbney (lewismc@apache.org)
> >> > * Chris Mattmann (mattmann@apache.org)
> >> >
> >> >== Affiliations ==
> >> >
> >> > * Johns Hopkins University
> >> > * Matt Post
> >> >
> >> > * NASA JPL
> >> > * Chris Mattmann
> >> > * Lewis John McGibbney
> >> >
> >> >
> >> >== Sponsors ==
> >> >=== Champion ===
> >> > * Chris Mattmann (NASA/JPL)
> >> >
> >> >=== Nominated Mentors ===
> >> > * Paul Ramirez
> >> > * Lewis John McGibbney
> >> > * Chris Mattmann
> >> >
> >> >== Sponsoring Entity ==
> >> >The Apache Incubator
> >> >
> >> >
> >> >
> >> >
> >> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> >Chris Mattmann, Ph.D.
> >> >Chief Architect
> >> >Instrument Software and Science Data Systems Section (398)
> >> >NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >> >Office: 168-519, Mailstop: 168-527
> >> >Email: chris.a.mattmann@nasa.gov
> >> >WWW: http://sunset.usc.edu/~mattmann/
> >> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> >Adjunct Associate Professor, Computer Science Department
> >> >University of Southern California, Los Angeles, CA 90089 USA
> >> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> >
> >> >
> >> >
> >>
> >>>?B�KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKC
> >>>B�
> >>
> >>>?�?[��X��ܚX�K??K[XZ[?�?�[�\�[?][��X��ܚX�P?[��X�]?܋�\?X�?K�ܙ�B��܈?Y??]?[ۘ
> >>>[?
> >> >?��[X[�?�??K[XZ[?�?�[�\�[?Z?[???[��X�]?܋�\?X�?K�ܙ�B
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine
Translation Toolkit
Posted by "Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov>.
Dear Ben,
Awesome! We would love to have you as a member of the project.
Please add your name to the wiki as a committer/PPMC member and
happy to have ya!
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: ben gao <ba...@gmail.com>
Reply-To: "general@incubator.apache.org" <ge...@incubator.apache.org>
Date: Tuesday, January 19, 2016 at 8:11 AM
To: "general@incubator.apache.org" <ge...@incubator.apache.org>
Subject: Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine
Translation Toolkit
>Hi Chris,
>
>I am very interested in this project. I am a senior java architect, I have
>been worked on java for about 15 years, and various project related to
>Lucene and NLP. Please advise how can I participate it.
>
>Thanks,
>-Ben
>
>On Tue, Jan 19, 2016 at 12:58 AM, Mattmann, Chris A (3980) <
>chris.a.mattmann@jpl.nasa.gov> wrote:
>
>> Great Hen, we’d love to have you on board as a mentor! Please
>> add yourself to the proposal on the wiki.
>>
>> Anyone else have interest in Machine Translation? Any OpenNLP folks,
>> Hadoop folks, Tika, or Lucene folks? CC’ing the dev lists for visibility
>> please feel free to reply to general@i.a.o.
>>
>> I’ll leave the DISCUSS thread open for a few more days.
>>
>> Cheers,
>> Chris
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Chief Architect
>> Instrument Software and Science Data Systems Section (398)
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 168-519, Mailstop: 168-527
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Associate Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Henri Yandell <ba...@apache.org>
>> Reply-To: "general@incubator.apache.org" <ge...@incubator.apache.org>
>> Date: Monday, January 18, 2016 at 7:57 PM
>> To: jpluser <ch...@jpl.nasa.gov>,
>> "general@incubator.apache.org" <ge...@incubator.apache.org>
>> Subject: Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine
>> Translation Toolkit
>>
>> >Non-binding +1 to Joshua joining the Incubator. I'd be interested in
>> >mentoring.
>> >
>> >
>> >> -----Original Message-----
>> >> From: jpluser <ch...@jpl.nasa.gov>
>> >> Reply-To: "general@incubator.apache.org"
>><ge...@incubator.apache.org>
>> >> Date: Tuesday, January 12, 2016 at 10:56 PM
>> >> To: "general@incubator.apache.org" <ge...@incubator.apache.org>
>> >> Cc: "post@cs.jhu.edu" <po...@cs.jhu.edu>
>> >> Subject: [DISCUSS] Apache Joshua Incubator Proposal - Machine
>> >>Translation
>> >> Toolkit
>> >>
>> >> >Hi Everyone,
>> >> >
>> >> >Please find attached for your viewing pleasure a proposed new
>>project,
>> >> >Apache Joshua, a statistical machine translation toolkit. The
>>proposal
>> >> >is in wiki draft form at:
>> >> https://wiki.apache.org/incubator/JoshuaProposal
>> >> >
>> >> >Proposal text is copied below. I’ll leave the discussion open for a
>> >>week
>> >> >and we are interested in folks who would like to be initial
>>committers
>> >> >and mentors. Please discuss here on the thread.
>> >> >
>> >> >Thanks!
>> >> >
>> >> >Cheers,
>> >> >Chris (Champion)
>> >> >
>> >> >———
>> >> >
>> >> >= Joshua Proposal =
>> >> >
>> >> >== Abstract ==
>> >> >[[joshua-decoder.org|Joshua]] is an open-source statistical machine
>> >> >translation toolkit. It includes a Java-based decoder for
>>translating
>> >>with
>> >> >phrase-based, hierarchical, and syntax-based translation models, a
>> >> >Hadoop-based grammar extractor (Thrax), and an extensive set of
>>tools
>> >>and
>> >> >scripts for training and evaluating new models from parallel text.
>> >> >
>> >> >== Proposal ==
>> >> >Joshua is a state of the art statistical machine translation system
>> >>that
>> >> >provides a number of features:
>> >> >
>> >> > * Support for the two main paradigms in statistical machine
>> >>translation:
>> >> >phrase-based and hierarchical / syntactic.
>> >> > * A sparse feature API that makes it easy to add new feature
>>templates
>> >> >supporting millions of features
>> >> > * Native implementations of many tuners (MERT, MIRA, PRO, and
>>AdaGrad)
>> >> > * Support for lattice decoding, allowing upstream NLP tools to
>>expose
>> >> >their hypothesis space to the MT system
>> >> > * An efficient representation for models, allowing for quick
>>loading
>> >>of
>> >> >multi-gigabyte model files
>> >> > * Fast decoding speed (on par with Moses and mtplz)
>> >> > * Language packs — precompiled models that allow the decoder to be
>> >>run as
>> >> >a black box
>> >> > * Thrax, a Hadoop-based tool for learning translation models from
>> >> >parallel text
>> >> > * A suite of tools for constructing new models for any language
>>pair
>> >>for
>> >> >which sufficient training data exists
>> >> >
>> >> >== Background and Rationale ==
>> >> >A number of factors make this a good time for an Apache project
>> >>focused on
>> >> >machine translation (MT): the quality of MT output (for many
>>language
>> >> >pairs); the average computing resources available on computers,
>> >>relative
>> >> >to the needs of MT systems; and the availability of a number of
>> >> >high-quality toolkits, together with a large base of researchers
>> >>working
>> >> >on them.
>> >> >
>> >> >Over the past decade, machine translation (MT; the automatic
>> >>translation
>> >> >of one human language to another) has become a reality. The research
>> >>into
>> >> >statistical approaches to translation that began in the early
>>nineties,
>> >> >together with the availability of large amounts of training data,
>>and
>> >> >better computing infrastructure, have all come together to produce
>> >> >translations results that are “good enough” for a large set of
>>language
>> >> >pairs and use cases. Free services like
>> >> >[[https://www.bing.com/translator|Bing Translator]] and
>> >> >[[https://translate.google.com|Google Translate]] have made these
>> >> services
>> >> >available to the average person through direct interfaces and
>>through
>> >> >tools like browser plugins, and sites across the world with higher
>> >> >translation needs use them to translate their pages through
>> >>automatically.
>> >> >
>> >> >MT does not require the infrastructure of large corporations in
>>order
>> >>to
>> >> >produce feasible output. Machine translation can be
>>resource-intensive,
>> >> >but need not be prohibitively so. Disk and memory usage are mostly a
>> >> >matter of model size, which for most language pairs is a few
>>gigabytes
>> >>at
>> >> >most, at which size models can provide coverage on the order of
>>tens or
>> >> >even hundreds of thousands of words in the input and output
>>languages.
>> >>The
>> >> >computational complexity of the algorithms used to search for
>> >>translations
>> >> >of new sentences are typically linear in the number of words in the
>> >>input
>> >> >sentence, making it possible to run a translation engine on a
>>personal
>> >> >computer.
>> >> >
>> >> >The research community has produced many different open source
>> >>translation
>> >> >projects for a range of programming languages and under a variety of
>> >> >licenses. These projects include the core “decoder”, which takes a
>> >>model
>> >> >and uses it to translate new sentences between the language pair the
>> >>model
>> >> >was defined for. They also typically include a large set of tools
>>that
>> >> >enable new models to be built from large sets of example
>>translations
>> >> >(“parallel data”) and monolingual texts. These toolkits are usually
>> >>built
>> >> >to support the agendas of the (largely) academic researchers that
>>build
>> >> >them: the repeated cycle of building new models, tuning model
>> >>parameters
>> >> >against development data, and evaluating them against held-out test
>> >>data,
>> >> >using standard metrics for testing the quality of MT output.
>> >> >
>> >> >Together, these three factors—the quality of machine translation
>> >>output,
>> >> >the feasibility of translating on standard computers, and the
>> >>availability
>> >> >of tools to build models—make it reasonable for the end users to use
>> >>MT as
>> >> >a black-box service, and to run it on their personal machine.
>> >> >
>> >> >These factors make it a good time for an organization with the
>>status
>> >>of
>> >> >the Apache Foundation to host a machine translation project.
>> >> >
>> >> >== Current Status ==
>> >> >Joshua was originally ported from David Chiang’s Python
>>implementation
>> >>of
>> >> >Hiero by Zhifei Li, while he was a Ph.D. student at Johns Hopkins
>> >> >University. The current version is maintained by Matt Post at Johns
>> >> >Hopkins’ Human Language Technology Center of Excellence. Joshua has
>> >>made
>> >> >many releases with a list of over 20 source code tags. The last
>> >>release of
>> >> >Joshua was 6.0.5 on November 5th, 2015.
>> >> >
>> >> >== Meritocracy ==
>> >> >The current developers are familiar with meritocratic open source
>> >> >development at Apache. Apache was chosen specifically because we
>>want
>> >>to
>> >> >encourage this style of development for the project.
>> >> >
>> >> >== Community ==
>> >> >Joshua is used widely across the world. Perhaps its biggest (known)
>> >> >research / industrial user is the Amazon research group in Berlin.
>> >>Another
>> >> >user is the US Army Research Lab. No formal census has been
>>undertaken,
>> >> >but posts to the Joshua technical support mailing list, along with
>>the
>> >> >occasional contributions, suggest small research and academic
>> >>communities
>> >> >spread across the world, many of them in India.
>> >> >
>> >> >During incubation, we will explicitly seek to increase our usage
>>across
>> >> >the board, including academic research, industry, and other end
>>users
>> >> >interested in statistical machine translation.
>> >> >
>> >> >== Core Developers ==
>> >> >The current set of core developers is fairly small, having fallen
>>with
>> >>the
>> >> >graduation from Johns Hopkins of some core student participants.
>> >>However,
>> >> >Joshua is used fairly widely, as mentioned above, and there remains
>>a
>> >> >commitment from the principal researcher at Johns Hopkins to
>>continue
>> >>to
>> >> >use and develop it. Joshua has seen a number of new community
>>members
>> >> >become interested recently due to a potential for its projected use
>>in
>> >>a
>> >> >number of ongoing DARPA projects such as XDATA and Memex.
>> >> >
>> >> >== Alignment ==
>> >> >Joshua is currently Copyright (c) 2015, Johns Hopkins University All
>> >> >rights reserved and licensed under BSD 2-clause license. It would of
>> >> >course be the intention to relicense this code under AL2.0 which
>>would
>> >> >permit expanded and increased use of the software within Apache
>> >>projects.
>> >> >There is currently an ongoing effort within the Apache Tika
>>community
>> >>to
>> >> >utilize Joshua within Tika’s Translate API, see
>> >> >[[https://issues.apache.org/jira/browse/TIKA-1343|TIKA-1343]].
>> >> >
>> >> >== Known Risks ==
>> >> >
>> >> >=== Orphaned products ===
>> >> >At the moment, regular contributions are made by a single
>>contributor,
>> >>the
>> >> >lead maintainer. He (Matt Post) plans to continue development for
>>the
>> >>next
>> >> >few years, but it is still a single point of failure, since the
>> >>graduate
>> >> >students who worked on the project have moved on to jobs, mostly in
>> >> >industry. However, our goal is to help that process by growing the
>> >> >community in Apache, and at least in growing the community with
>>users
>> >>and
>> >> >participants from NASA JPL.
>> >> >
>> >> >=== Inexperience with Open Source ===
>> >> >The team both at Johns Hopkins and NASA JPL have experience with
>>many
>> >>OSS
>> >> >software projects at Apache and elsewhere. We understand "how it
>>works"
>> >> >here at the foundation.
>> >> >
>> >> >
>> >> >== Relationships with Other Apache Products ==
>> >> >Joshua includes dependences on Hadoop, and also is included as a
>> >>plugin in
>> >> >Apache Tika. We are also interested in coordinating with other
>>projects
>> >> >including Spark, and other projects needing MT services for language
>> >> >translation.
>> >> >
>> >> >== Developers ==
>> >> >Joshua only has one regular developer who is employed by Johns
>>Hopkins
>> >> >University. NASA JPL (Mattmann and McGibbney) have been contributing
>> >> >lately including a Brew formula and other contributions to the
>>project
>> >> >through the DARPA XDATA and Memex programs.
>> >> >
>> >> >== Documentation ==
>> >> >Documentation and publications related to Joshua can be found at
>> >> >joshua-decoder.org. The source for the Joshua documentation is
>> >>currently
>> >> >hosted on Github at
>> >> >https://github.com/joshua-decoder/joshua-decoder.github.com
>> >> >
>> >> >== Initial Source ==
>> >> >Current source resides at Github: github.com/joshua-decoder/joshua
>> (the
>> >> >main decoder and toolkit) and github.com/joshua-decoder/thrax (the
>> >> grammar
>> >> >extraction tool).
>> >> >
>> >> >== External Dependencies ==
>> >> >Joshua has a number of external dependencies. Only BerkeleyLM
>>(Apache
>> >>2.0)
>> >> >and KenLM (LGPG 2.1) are run-time decoder dependencies (one of
>>which is
>> >> >needed for translating sentences with pre-built models). The rest
>>are
>> >> >dependencies for the build system and pipeline, used for
>>constructing
>> >>and
>> >> >training new models from parallel text.
>> >> >
>> >> >Apache projects:
>> >> > * Ant
>> >> > * Hadoop
>> >> > * Commons
>> >> > * Maven
>> >> > * Ivy
>> >> >
>> >> >There are also a number of other open-source projects with various
>> >> >licenses that the project depends on both dynamically (runtime), and
>> >> >statically.
>> >> >
>> >> >=== GNU GPL 2 ===
>> >> > * Berkeley Aligner: https://code.google.com/p/berkeleyaligner/
>> >> >
>> >> >=== LGPG 2.1 ===
>> >> > * KenLM: github.com/kpu/kenlm
>> >> >
>> >> >=== Apache 2.0 ===
>> >> > * BerkeleyLM: https://code.google.com/p/berkeleylm/
>> >> >
>> >> >=== GNU GPL ===
>> >> > * GIZA++: http://www.statmt.org/moses/giza/GIZA++.html
>> >> >
>> >> >== Required Resources ==
>> >> > * Mailing Lists
>> >> > * private@joshua.incubator.apache.org
>> >> > * dev@joshua.incubator.apache.org
>> >> > * commits@joshua.incubator.apache.org
>> >> >
>> >> > * Git Repos
>> >> > * https://git-wip-us.apache.org/repos/asf/joshua.git
>> >> >
>> >> > * Issue Tracking
>> >> > * JIRA Joshua (JOSHUA)
>> >> >
>> >> > * Continuous Integration
>> >> > * Jenkins builds on https://builds.apache.org/
>> >> >
>> >> > * Web
>> >> > * http://joshua.incubator.apache.org/
>> >> > * wiki at http://cwiki.apache.org
>> >> >
>> >> >== Initial Committers ==
>> >> >The following is a list of the planned initial Apache committers
>>(the
>> >> >active subset of the committers for the current repository on
>>Github).
>> >> >
>> >> > * Matt Post (post@cs.jhu.edu)
>> >> > * Lewis John McGibbney (lewismc@apache.org)
>> >> > * Chris Mattmann (mattmann@apache.org)
>> >> >
>> >> >== Affiliations ==
>> >> >
>> >> > * Johns Hopkins University
>> >> > * Matt Post
>> >> >
>> >> > * NASA JPL
>> >> > * Chris Mattmann
>> >> > * Lewis John McGibbney
>> >> >
>> >> >
>> >> >== Sponsors ==
>> >> >=== Champion ===
>> >> > * Chris Mattmann (NASA/JPL)
>> >> >
>> >> >=== Nominated Mentors ===
>> >> > * Paul Ramirez
>> >> > * Lewis John McGibbney
>> >> > * Chris Mattmann
>> >> >
>> >> >== Sponsoring Entity ==
>> >> >The Apache Incubator
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> >Chris Mattmann, Ph.D.
>> >> >Chief Architect
>> >> >Instrument Software and Science Data Systems Section (398)
>> >> >NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> >> >Office: 168-519, Mailstop: 168-527
>> >> >Email: chris.a.mattmann@nasa.gov
>> >> >WWW: http://sunset.usc.edu/~mattmann/
>> >> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> >Adjunct Associate Professor, Computer Science Department
>> >> >University of Southern California, Los Angeles, CA 90089 USA
>> >> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> >
>> >> >
>> >> >
>> >>
>>
>>>>>?B�KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK
>>>>>KC
>> >>>B�
>> >>
>>
>>>>>?�?[��X��ܚX�K??K[XZ[?�?�[�\�[?][��X��ܚX�P?[��X�]?܋�\?X�?K�ܙ�B��܈?Y??]?
>>>>>[ۘ
>> >>>[?
>> >> >?��[X[�?�??K[XZ[?�?�[�\�[?Z?[???[��X�]?܋�\?X�?K�ܙ�B
>>
>>
Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine Translation Toolkit
Posted by ben gao <ba...@gmail.com>.
Hi Chris,
I am very interested in this project. I am a senior java architect, I have
been worked on java for about 15 years, and various project related to
Lucene and NLP. Please advise how can I participate it.
Thanks,
-Ben
On Tue, Jan 19, 2016 at 12:58 AM, Mattmann, Chris A (3980) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Great Hen, we’d love to have you on board as a mentor! Please
> add yourself to the proposal on the wiki.
>
> Anyone else have interest in Machine Translation? Any OpenNLP folks,
> Hadoop folks, Tika, or Lucene folks? CC’ing the dev lists for visibility
> please feel free to reply to general@i.a.o.
>
> I’ll leave the DISCUSS thread open for a few more days.
>
> Cheers,
> Chris
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
> -----Original Message-----
> From: Henri Yandell <ba...@apache.org>
> Reply-To: "general@incubator.apache.org" <ge...@incubator.apache.org>
> Date: Monday, January 18, 2016 at 7:57 PM
> To: jpluser <ch...@jpl.nasa.gov>,
> "general@incubator.apache.org" <ge...@incubator.apache.org>
> Subject: Re: [DISCUSS] Apache Joshua Incubator Proposal - Machine
> Translation Toolkit
>
> >Non-binding +1 to Joshua joining the Incubator. I'd be interested in
> >mentoring.
> >
> >
> >> -----Original Message-----
> >> From: jpluser <ch...@jpl.nasa.gov>
> >> Reply-To: "general@incubator.apache.org" <ge...@incubator.apache.org>
> >> Date: Tuesday, January 12, 2016 at 10:56 PM
> >> To: "general@incubator.apache.org" <ge...@incubator.apache.org>
> >> Cc: "post@cs.jhu.edu" <po...@cs.jhu.edu>
> >> Subject: [DISCUSS] Apache Joshua Incubator Proposal - Machine
> >>Translation
> >> Toolkit
> >>
> >> >Hi Everyone,
> >> >
> >> >Please find attached for your viewing pleasure a proposed new project,
> >> >Apache Joshua, a statistical machine translation toolkit. The proposal
> >> >is in wiki draft form at:
> >> https://wiki.apache.org/incubator/JoshuaProposal
> >> >
> >> >Proposal text is copied below. I’ll leave the discussion open for a
> >>week
> >> >and we are interested in folks who would like to be initial committers
> >> >and mentors. Please discuss here on the thread.
> >> >
> >> >Thanks!
> >> >
> >> >Cheers,
> >> >Chris (Champion)
> >> >
> >> >———
> >> >
> >> >= Joshua Proposal =
> >> >
> >> >== Abstract ==
> >> >[[joshua-decoder.org|Joshua]] is an open-source statistical machine
> >> >translation toolkit. It includes a Java-based decoder for translating
> >>with
> >> >phrase-based, hierarchical, and syntax-based translation models, a
> >> >Hadoop-based grammar extractor (Thrax), and an extensive set of tools
> >>and
> >> >scripts for training and evaluating new models from parallel text.
> >> >
> >> >== Proposal ==
> >> >Joshua is a state of the art statistical machine translation system
> >>that
> >> >provides a number of features:
> >> >
> >> > * Support for the two main paradigms in statistical machine
> >>translation:
> >> >phrase-based and hierarchical / syntactic.
> >> > * A sparse feature API that makes it easy to add new feature templates
> >> >supporting millions of features
> >> > * Native implementations of many tuners (MERT, MIRA, PRO, and AdaGrad)
> >> > * Support for lattice decoding, allowing upstream NLP tools to expose
> >> >their hypothesis space to the MT system
> >> > * An efficient representation for models, allowing for quick loading
> >>of
> >> >multi-gigabyte model files
> >> > * Fast decoding speed (on par with Moses and mtplz)
> >> > * Language packs — precompiled models that allow the decoder to be
> >>run as
> >> >a black box
> >> > * Thrax, a Hadoop-based tool for learning translation models from
> >> >parallel text
> >> > * A suite of tools for constructing new models for any language pair
> >>for
> >> >which sufficient training data exists
> >> >
> >> >== Background and Rationale ==
> >> >A number of factors make this a good time for an Apache project
> >>focused on
> >> >machine translation (MT): the quality of MT output (for many language
> >> >pairs); the average computing resources available on computers,
> >>relative
> >> >to the needs of MT systems; and the availability of a number of
> >> >high-quality toolkits, together with a large base of researchers
> >>working
> >> >on them.
> >> >
> >> >Over the past decade, machine translation (MT; the automatic
> >>translation
> >> >of one human language to another) has become a reality. The research
> >>into
> >> >statistical approaches to translation that began in the early nineties,
> >> >together with the availability of large amounts of training data, and
> >> >better computing infrastructure, have all come together to produce
> >> >translations results that are “good enough” for a large set of language
> >> >pairs and use cases. Free services like
> >> >[[https://www.bing.com/translator|Bing Translator]] and
> >> >[[https://translate.google.com|Google Translate]] have made these
> >> services
> >> >available to the average person through direct interfaces and through
> >> >tools like browser plugins, and sites across the world with higher
> >> >translation needs use them to translate their pages through
> >>automatically.
> >> >
> >> >MT does not require the infrastructure of large corporations in order
> >>to
> >> >produce feasible output. Machine translation can be resource-intensive,
> >> >but need not be prohibitively so. Disk and memory usage are mostly a
> >> >matter of model size, which for most language pairs is a few gigabytes
> >>at
> >> >most, at which size models can provide coverage on the order of tens or
> >> >even hundreds of thousands of words in the input and output languages.
> >>The
> >> >computational complexity of the algorithms used to search for
> >>translations
> >> >of new sentences are typically linear in the number of words in the
> >>input
> >> >sentence, making it possible to run a translation engine on a personal
> >> >computer.
> >> >
> >> >The research community has produced many different open source
> >>translation
> >> >projects for a range of programming languages and under a variety of
> >> >licenses. These projects include the core “decoder”, which takes a
> >>model
> >> >and uses it to translate new sentences between the language pair the
> >>model
> >> >was defined for. They also typically include a large set of tools that
> >> >enable new models to be built from large sets of example translations
> >> >(“parallel data”) and monolingual texts. These toolkits are usually
> >>built
> >> >to support the agendas of the (largely) academic researchers that build
> >> >them: the repeated cycle of building new models, tuning model
> >>parameters
> >> >against development data, and evaluating them against held-out test
> >>data,
> >> >using standard metrics for testing the quality of MT output.
> >> >
> >> >Together, these three factors—the quality of machine translation
> >>output,
> >> >the feasibility of translating on standard computers, and the
> >>availability
> >> >of tools to build models—make it reasonable for the end users to use
> >>MT as
> >> >a black-box service, and to run it on their personal machine.
> >> >
> >> >These factors make it a good time for an organization with the status
> >>of
> >> >the Apache Foundation to host a machine translation project.
> >> >
> >> >== Current Status ==
> >> >Joshua was originally ported from David Chiang’s Python implementation
> >>of
> >> >Hiero by Zhifei Li, while he was a Ph.D. student at Johns Hopkins
> >> >University. The current version is maintained by Matt Post at Johns
> >> >Hopkins’ Human Language Technology Center of Excellence. Joshua has
> >>made
> >> >many releases with a list of over 20 source code tags. The last
> >>release of
> >> >Joshua was 6.0.5 on November 5th, 2015.
> >> >
> >> >== Meritocracy ==
> >> >The current developers are familiar with meritocratic open source
> >> >development at Apache. Apache was chosen specifically because we want
> >>to
> >> >encourage this style of development for the project.
> >> >
> >> >== Community ==
> >> >Joshua is used widely across the world. Perhaps its biggest (known)
> >> >research / industrial user is the Amazon research group in Berlin.
> >>Another
> >> >user is the US Army Research Lab. No formal census has been undertaken,
> >> >but posts to the Joshua technical support mailing list, along with the
> >> >occasional contributions, suggest small research and academic
> >>communities
> >> >spread across the world, many of them in India.
> >> >
> >> >During incubation, we will explicitly seek to increase our usage across
> >> >the board, including academic research, industry, and other end users
> >> >interested in statistical machine translation.
> >> >
> >> >== Core Developers ==
> >> >The current set of core developers is fairly small, having fallen with
> >>the
> >> >graduation from Johns Hopkins of some core student participants.
> >>However,
> >> >Joshua is used fairly widely, as mentioned above, and there remains a
> >> >commitment from the principal researcher at Johns Hopkins to continue
> >>to
> >> >use and develop it. Joshua has seen a number of new community members
> >> >become interested recently due to a potential for its projected use in
> >>a
> >> >number of ongoing DARPA projects such as XDATA and Memex.
> >> >
> >> >== Alignment ==
> >> >Joshua is currently Copyright (c) 2015, Johns Hopkins University All
> >> >rights reserved and licensed under BSD 2-clause license. It would of
> >> >course be the intention to relicense this code under AL2.0 which would
> >> >permit expanded and increased use of the software within Apache
> >>projects.
> >> >There is currently an ongoing effort within the Apache Tika community
> >>to
> >> >utilize Joshua within Tika’s Translate API, see
> >> >[[https://issues.apache.org/jira/browse/TIKA-1343|TIKA-1343]].
> >> >
> >> >== Known Risks ==
> >> >
> >> >=== Orphaned products ===
> >> >At the moment, regular contributions are made by a single contributor,
> >>the
> >> >lead maintainer. He (Matt Post) plans to continue development for the
> >>next
> >> >few years, but it is still a single point of failure, since the
> >>graduate
> >> >students who worked on the project have moved on to jobs, mostly in
> >> >industry. However, our goal is to help that process by growing the
> >> >community in Apache, and at least in growing the community with users
> >>and
> >> >participants from NASA JPL.
> >> >
> >> >=== Inexperience with Open Source ===
> >> >The team both at Johns Hopkins and NASA JPL have experience with many
> >>OSS
> >> >software projects at Apache and elsewhere. We understand "how it works"
> >> >here at the foundation.
> >> >
> >> >
> >> >== Relationships with Other Apache Products ==
> >> >Joshua includes dependences on Hadoop, and also is included as a
> >>plugin in
> >> >Apache Tika. We are also interested in coordinating with other projects
> >> >including Spark, and other projects needing MT services for language
> >> >translation.
> >> >
> >> >== Developers ==
> >> >Joshua only has one regular developer who is employed by Johns Hopkins
> >> >University. NASA JPL (Mattmann and McGibbney) have been contributing
> >> >lately including a Brew formula and other contributions to the project
> >> >through the DARPA XDATA and Memex programs.
> >> >
> >> >== Documentation ==
> >> >Documentation and publications related to Joshua can be found at
> >> >joshua-decoder.org. The source for the Joshua documentation is
> >>currently
> >> >hosted on Github at
> >> >https://github.com/joshua-decoder/joshua-decoder.github.com
> >> >
> >> >== Initial Source ==
> >> >Current source resides at Github: github.com/joshua-decoder/joshua
> (the
> >> >main decoder and toolkit) and github.com/joshua-decoder/thrax (the
> >> grammar
> >> >extraction tool).
> >> >
> >> >== External Dependencies ==
> >> >Joshua has a number of external dependencies. Only BerkeleyLM (Apache
> >>2.0)
> >> >and KenLM (LGPG 2.1) are run-time decoder dependencies (one of which is
> >> >needed for translating sentences with pre-built models). The rest are
> >> >dependencies for the build system and pipeline, used for constructing
> >>and
> >> >training new models from parallel text.
> >> >
> >> >Apache projects:
> >> > * Ant
> >> > * Hadoop
> >> > * Commons
> >> > * Maven
> >> > * Ivy
> >> >
> >> >There are also a number of other open-source projects with various
> >> >licenses that the project depends on both dynamically (runtime), and
> >> >statically.
> >> >
> >> >=== GNU GPL 2 ===
> >> > * Berkeley Aligner: https://code.google.com/p/berkeleyaligner/
> >> >
> >> >=== LGPG 2.1 ===
> >> > * KenLM: github.com/kpu/kenlm
> >> >
> >> >=== Apache 2.0 ===
> >> > * BerkeleyLM: https://code.google.com/p/berkeleylm/
> >> >
> >> >=== GNU GPL ===
> >> > * GIZA++: http://www.statmt.org/moses/giza/GIZA++.html
> >> >
> >> >== Required Resources ==
> >> > * Mailing Lists
> >> > * private@joshua.incubator.apache.org
> >> > * dev@joshua.incubator.apache.org
> >> > * commits@joshua.incubator.apache.org
> >> >
> >> > * Git Repos
> >> > * https://git-wip-us.apache.org/repos/asf/joshua.git
> >> >
> >> > * Issue Tracking
> >> > * JIRA Joshua (JOSHUA)
> >> >
> >> > * Continuous Integration
> >> > * Jenkins builds on https://builds.apache.org/
> >> >
> >> > * Web
> >> > * http://joshua.incubator.apache.org/
> >> > * wiki at http://cwiki.apache.org
> >> >
> >> >== Initial Committers ==
> >> >The following is a list of the planned initial Apache committers (the
> >> >active subset of the committers for the current repository on Github).
> >> >
> >> > * Matt Post (post@cs.jhu.edu)
> >> > * Lewis John McGibbney (lewismc@apache.org)
> >> > * Chris Mattmann (mattmann@apache.org)
> >> >
> >> >== Affiliations ==
> >> >
> >> > * Johns Hopkins University
> >> > * Matt Post
> >> >
> >> > * NASA JPL
> >> > * Chris Mattmann
> >> > * Lewis John McGibbney
> >> >
> >> >
> >> >== Sponsors ==
> >> >=== Champion ===
> >> > * Chris Mattmann (NASA/JPL)
> >> >
> >> >=== Nominated Mentors ===
> >> > * Paul Ramirez
> >> > * Lewis John McGibbney
> >> > * Chris Mattmann
> >> >
> >> >== Sponsoring Entity ==
> >> >The Apache Incubator
> >> >
> >> >
> >> >
> >> >
> >> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> >Chris Mattmann, Ph.D.
> >> >Chief Architect
> >> >Instrument Software and Science Data Systems Section (398)
> >> >NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >> >Office: 168-519, Mailstop: 168-527
> >> >Email: chris.a.mattmann@nasa.gov
> >> >WWW: http://sunset.usc.edu/~mattmann/
> >> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> >Adjunct Associate Professor, Computer Science Department
> >> >University of Southern California, Los Angeles, CA 90089 USA
> >> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> >
> >> >
> >> >
> >>
> >>>?B�KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKC
> >>>B�
> >>
> >>>?�?[��X��ܚX�K??K[XZ[?�?�[�\�[?][��X��ܚX�P?[��X�]?܋�\?X�?K�ܙ�B��܈?Y??]?[ۘ
> >>>[?
> >> >?��[X[�?�??K[XZ[?�?�[�\�[?Z?[???[��X�]?܋�\?X�?K�ܙ�B
>
>