You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@opennlp.apache.org by Jörn Kottmann <ko...@gmail.com> on 2011/02/01 12:51:31 UTC
Getting our first release out
Hi all,
I would like to go ahead and get our first release out. The release is
backward compatible with the models we had over at SourceForge.
Which means we do not need to release new models right now.
The logic to train most of the models is already included in OpenNLP
and enables our users just to train the models them self or even
mix with their own data.
To release the models at Apache we have to go trough a series of legal
issues which I believe should not postpone our first release for
weeks or months.
What do others think ?
Jörn
Re: Getting our first release out
Posted by Jason Baldridge <jb...@mail.utexas.edu>.
I agree that free annotations on free corpora are the way forward. There are
some ways to get this going, beyond waiting for volunteering:
- People like me apply for grant funds (e.g. NSF) to set up infrastructure
and initial annotations in two languages. I'm currently fully loaded on
cycles I have for being a principal investigator on grants, but would
support an application by others, or wait until fall or so. (A big problem
with this is the waiting time between applying for and receiving grant
funds, plus extra cycles if one gets turned down.)
- Companies who are using OpenNLP could contribute to a fund that would pay
developers to build an annotation infrastructure and annotators to label
things. This could be done somewhat in the Google summer of code mode, with
students getting experience and some pay, working under the supervision of
experienced developers.
- Individual companies fund one-off small projects to develop particular
corpora or annotation capabilities.
The key here for companies is that the cost of creating the resources would
be much less than it would be to develop in house and they have the
potential to realize benefits from others contributing in like manner, or
even just spurring the volunteer community by getting the ball rolling.
Oh, and definite +1 to getting things released and compatible with old
models.
Jason
On Tue, Feb 1, 2011 at 4:05 PM, Jörn Kottmann <ko...@gmail.com> wrote:
> On 2/1/11 10:45 PM, Grant Ingersoll wrote:
>
>> Yes, we should start assembling a list of corpora, even so we at least
>> have it for others that come later and want to reproduce them. In the
>> meantime, I would agree that we can just keep the models elsewhere. We
>> don't have to provide models. They are a convenience for all involved, but
>> not a requirement in order to run. I wonder how many people actually train
>> there own. (BTW, we should update our website to point to older models,
>> too. They are really hard to find unless you do some URL rewriting.)
>>
>
> OK, then lets get out the release as quickly as possible without depending
> on the legal issues for the models
> And lets do as much as possible to resolve these issues, just next to the
> release work. I might have a
> few spare cycles here and there to work on that.
>
> To get started with the legal stuff we need to compile a list with all the
> necessary information,
> that list will make a nice corpora page in our wiki.
> Our documentation already contains instructions on how to train on some
> freely available data.
>
> In the end I believe we are all best served with a wikinews corpus which
> can be labeled by our community.
>
> Jörn
>
--
Jason Baldridge
Assistant Professor, Department of Linguistics
The University of Texas at Austin
http://www.jasonbaldridge.com
Re: Getting our first release out
Posted by Jörn Kottmann <ko...@gmail.com>.
On 2/1/11 10:45 PM, Grant Ingersoll wrote:
> Yes, we should start assembling a list of corpora, even so we at least have it for others that come later and want to reproduce them. In the meantime, I would agree that we can just keep the models elsewhere. We don't have to provide models. They are a convenience for all involved, but not a requirement in order to run. I wonder how many people actually train there own. (BTW, we should update our website to point to older models, too. They are really hard to find unless you do some URL rewriting.)
OK, then lets get out the release as quickly as possible without
depending on the legal issues for the models
And lets do as much as possible to resolve these issues, just next to
the release work. I might have a
few spare cycles here and there to work on that.
To get started with the legal stuff we need to compile a list with all
the necessary information,
that list will make a nice corpora page in our wiki.
Our documentation already contains instructions on how to train on some
freely available data.
In the end I believe we are all best served with a wikinews corpus which
can be labeled by our community.
Jörn
Re: Getting our first release out
Posted by Benson Margulies <bi...@gmail.com>.
http://mail-archives.apache.org/mod_mbox/www-legal-discuss/201011.mbox/%3CAANLkTimSCa_ErcRdhK6rvyuyBF1eJ_U=nag3jY+x0Axj@mail.gmail.com%3E
On Tue, Feb 1, 2011 at 4:45 PM, Grant Ingersoll <gs...@apache.org> wrote:
>
> On Feb 1, 2011, at 11:20 AM, Benson Margulies wrote:
>
>> With somewhat mixed feelings, I've been following this discussion. In
>> the interests of full disclosure, I'll explain the mixed feelings in a
>> moment. I warmed up legal-discuss for you during the incubator
>> discussion and learned some things.
>
> What's the thread for this one?
>
>>
>> Based on my legal understanding, I feel fairly confident that models
>> derived from textual corpora are not 'derived works' subject to the
>> copyrights and licenses of the corpora. However, IANAL, and this needs
>> to be explored. Some remarks on legal-discuss suggest that, in Europe,
>> I may be completely wrong. Still, this is probably the *good* news.
>>
>> The less-good news is that, as a general principle, the ASF would not
>> want a release to contain a binary artifact derived from sources hat
>> cannot be released under the Apache license, or even obtained under
>> the Apache license or something remotely like it. An even stronger
>> principle is that the source materials must be available, period
>> (e.g. not available only to LDC members or something).
>
> This is the single most frustrating issue facing open source text tools to date. It's why I started the Open Relevance Project, but until we have enough of us willing to band together and work on it, we will be stuck.
>
>>
>> The less bad news is that there is a precedent here: SpamAssassin. To
>> train spam models, SpamAssassin has to collect and maintain large
>> collections of materials that have restrictive licenses. The
>> Foundation has decided that this is tolerable if these materials are
>> kept on a Foundation server, and access to that granted to legitimate
>> members of the development community, one by one. This avoids the
>> spectre of 'publication' but permits open participation.
>
> This is OK, but it discourages newbies from participating.
>
>>
>> The bottom line of the legal-discuss discussion was that this path
>> was, broadly, available to OpenNLP. However, legal-discuss hates to
>> discuss hypotheticals, so you won't get a definitive ruling until you
>> ask a specific question. I recommend opening a JIRA on legal-discuss
>> as a way to clarify that you need a clear and definitive ruling and
>> not just an email food-fight.
>
> Yes, we should start assembling a list of corpora, even so we at least have it for others that come later and want to reproduce them. In the meantime, I would agree that we can just keep the models elsewhere. We don't have to provide models. They are a convenience for all involved, but not a requirement in order to run. I wonder how many people actually train there own. (BTW, we should update our website to point to older models, too. They are really hard to find unless you do some URL rewriting.)
>
>
> -Grant
Re: Getting our first release out
Posted by Grant Ingersoll <gs...@apache.org>.
On Feb 1, 2011, at 11:20 AM, Benson Margulies wrote:
> With somewhat mixed feelings, I've been following this discussion. In
> the interests of full disclosure, I'll explain the mixed feelings in a
> moment. I warmed up legal-discuss for you during the incubator
> discussion and learned some things.
What's the thread for this one?
>
> Based on my legal understanding, I feel fairly confident that models
> derived from textual corpora are not 'derived works' subject to the
> copyrights and licenses of the corpora. However, IANAL, and this needs
> to be explored. Some remarks on legal-discuss suggest that, in Europe,
> I may be completely wrong. Still, this is probably the *good* news.
>
> The less-good news is that, as a general principle, the ASF would not
> want a release to contain a binary artifact derived from sources hat
> cannot be released under the Apache license, or even obtained under
> the Apache license or something remotely like it. An even stronger
> principle is that the source materials must be available, period
> (e.g. not available only to LDC members or something).
This is the single most frustrating issue facing open source text tools to date. It's why I started the Open Relevance Project, but until we have enough of us willing to band together and work on it, we will be stuck.
>
> The less bad news is that there is a precedent here: SpamAssassin. To
> train spam models, SpamAssassin has to collect and maintain large
> collections of materials that have restrictive licenses. The
> Foundation has decided that this is tolerable if these materials are
> kept on a Foundation server, and access to that granted to legitimate
> members of the development community, one by one. This avoids the
> spectre of 'publication' but permits open participation.
This is OK, but it discourages newbies from participating.
>
> The bottom line of the legal-discuss discussion was that this path
> was, broadly, available to OpenNLP. However, legal-discuss hates to
> discuss hypotheticals, so you won't get a definitive ruling until you
> ask a specific question. I recommend opening a JIRA on legal-discuss
> as a way to clarify that you need a clear and definitive ruling and
> not just an email food-fight.
Yes, we should start assembling a list of corpora, even so we at least have it for others that come later and want to reproduce them. In the meantime, I would agree that we can just keep the models elsewhere. We don't have to provide models. They are a convenience for all involved, but not a requirement in order to run. I wonder how many people actually train there own. (BTW, we should update our website to point to older models, too. They are really hard to find unless you do some URL rewriting.)
-Grant
Re: Getting our first release out
Posted by Benson Margulies <bi...@gmail.com>.
With somewhat mixed feelings, I've been following this discussion. In
the interests of full disclosure, I'll explain the mixed feelings in a
moment. I warmed up legal-discuss for you during the incubator
discussion and learned some things.
Based on my legal understanding, I feel fairly confident that models
derived from textual corpora are not 'derived works' subject to the
copyrights and licenses of the corpora. However, IANAL, and this needs
to be explored. Some remarks on legal-discuss suggest that, in Europe,
I may be completely wrong. Still, this is probably the *good* news.
The less-good news is that, as a general principle, the ASF would not
want a release to contain a binary artifact derived from sources hat
cannot be released under the Apache license, or even obtained under
the Apache license or something remotely like it. An even stronger
principle is that the source materials must be available, period
(e.g. not available only to LDC members or something).
The less bad news is that there is a precedent here: SpamAssassin. To
train spam models, SpamAssassin has to collect and maintain large
collections of materials that have restrictive licenses. The
Foundation has decided that this is tolerable if these materials are
kept on a Foundation server, and access to that granted to legitimate
members of the development community, one by one. This avoids the
spectre of 'publication' but permits open participation.
The bottom line of the legal-discuss discussion was that this path
was, broadly, available to OpenNLP. However, legal-discuss hates to
discuss hypotheticals, so you won't get a definitive ruling until you
ask a specific question. I recommend opening a JIRA on legal-discuss
as a way to clarify that you need a clear and definitive ruling and
not just an email food-fight.
Now, the full disclosure dept. My paycheck depends, in part, on the
success of a closed source set of NLP modules that have some overlap
with the contents of OpenNLP. It is my belief that, over the long run,
our willingness to spend large sums of money to acquire, clean,
organize, and annotate volumes of data that are directly relevant to
our customers genres-of-interest will preserve that paycheck,
regardless of the success of OpenNLP. In fact, if I could get 5 spare
days, I'd compare your perceptron model to my perceptron model (mine
includes, for example, "Passive Aggressive" training and is tuned to
run like a bat out of hell) and contemplate mushing them together at
OpenNLP. Still, I thought it only fair to disclose that I could be
viewed as a sort of fox in the henhouse here.
--benson
On Tue, Feb 1, 2011 at 11:07 AM, Jörn Kottmann <ko...@gmail.com> wrote:
> On 2/1/11 4:57 PM, Grant Ingersoll wrote:
>>
>> Your timing is great, as I was just about to suggest the same thing.
>>
>> On Feb 1, 2011, at 6:51 AM, Jörn Kottmann wrote:
>>
>>> Hi all,
>>>
>>> I would like to go ahead and get our first release out. The release is
>>> backward compatible with the models we had over at SourceForge.
>>> Which means we do not need to release new models right now.
>>>
>>> The logic to train most of the models is already included in OpenNLP
>>> and enables our users just to train the models them self or even
>>> mix with their own data.
>>>
>>> To release the models at Apache we have to go trough a series of legal
>>> issues which I believe should not postpone our first release for
>>> weeks or months.
>>
>> Can you summarize here the issues? The last thread is mountainous. To
>> some extent, there is no time like the present to address the legal issues.
>> The ASF has legal counsel, if you can summarize what we do to make the
>> models and what the concerns are, we can take it over to legal-discuss@ and
>> start working on it. It may not be as big a deal as one might think.
>
> The concerns are, that our models are trained on various closed or free
> corpora which almost all have different licenses.
> We would have to discuss if the trained model from each corpora is allowed
> to be distributed under AL 2.0.
>
> I believe in most cases we do not validate any copyright, because statistics
> about text is not protected
> by its copyright. We would for example generate bigram or trigram features
> over the whole corpus.
>
> In my opinion we at least need to provide a list of corpora and licenses to
> start a discussion over at
> the legal list, which alone will take some time to dig out, at least for the
> english training data.
> We also have training data where we are unsure about the license.
>
> On the other side we have no advantage of doing it now as part of the
> release.
>
> In my opinion we should try to get the process started, maybe put together a
> wiki page and as soon
> as we have all the information we need for the legal people we start talking
> to them.
>
> Jörn
>
>
Re: Getting our first release out
Posted by Jörn Kottmann <ko...@gmail.com>.
On 2/1/11 4:57 PM, Grant Ingersoll wrote:
> Your timing is great, as I was just about to suggest the same thing.
>
> On Feb 1, 2011, at 6:51 AM, Jörn Kottmann wrote:
>
>> Hi all,
>>
>> I would like to go ahead and get our first release out. The release is
>> backward compatible with the models we had over at SourceForge.
>> Which means we do not need to release new models right now.
>>
>> The logic to train most of the models is already included in OpenNLP
>> and enables our users just to train the models them self or even
>> mix with their own data.
>>
>> To release the models at Apache we have to go trough a series of legal
>> issues which I believe should not postpone our first release for
>> weeks or months.
> Can you summarize here the issues? The last thread is mountainous. To some extent, there is no time like the present to address the legal issues. The ASF has legal counsel, if you can summarize what we do to make the models and what the concerns are, we can take it over to legal-discuss@ and start working on it. It may not be as big a deal as one might think.
The concerns are, that our models are trained on various closed or free
corpora which almost all have different licenses.
We would have to discuss if the trained model from each corpora is
allowed to be distributed under AL 2.0.
I believe in most cases we do not validate any copyright, because
statistics about text is not protected
by its copyright. We would for example generate bigram or trigram
features over the whole corpus.
In my opinion we at least need to provide a list of corpora and licenses
to start a discussion over at
the legal list, which alone will take some time to dig out, at least for
the english training data.
We also have training data where we are unsure about the license.
On the other side we have no advantage of doing it now as part of the
release.
In my opinion we should try to get the process started, maybe put
together a wiki page and as soon
as we have all the information we need for the legal people we start
talking to them.
Jörn
Re: Getting our first release out
Posted by Grant Ingersoll <gs...@apache.org>.
Your timing is great, as I was just about to suggest the same thing.
On Feb 1, 2011, at 6:51 AM, Jörn Kottmann wrote:
> Hi all,
>
> I would like to go ahead and get our first release out. The release is
> backward compatible with the models we had over at SourceForge.
> Which means we do not need to release new models right now.
>
> The logic to train most of the models is already included in OpenNLP
> and enables our users just to train the models them self or even
> mix with their own data.
>
> To release the models at Apache we have to go trough a series of legal
> issues which I believe should not postpone our first release for
> weeks or months.
Can you summarize here the issues? The last thread is mountainous. To some extent, there is no time like the present to address the legal issues. The ASF has legal counsel, if you can summarize what we do to make the models and what the concerns are, we can take it over to legal-discuss@ and start working on it. It may not be as big a deal as one might think.
>
> What do others think ?
>
> Jörn
--------------------------
Grant Ingersoll
http://www.lucidimagination.com