You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@opennlp.apache.org by Mark G <ma...@apache.org> on 2014/05/12 21:03:47 UTC
MarkableFileInputStreamFactory
Does MarkableFileInputStreamFactory need to be package private? I am using
it in an addon (modelbuilder-addon), I would like to either move it or make
it a public class. Perhaps I should be using a different class altogether?
I am using it like this
ObjectStream<String> lineStream =
new PlainTextByLineStream(new
MarkableFileInputStreamFactory(params.getAnnotatedTrainingDataFile()),
charset);
ObjectStream<NameSample> sampleStream = new
NameSampleDataStream(lineStream);
where getAnnotatedTrainingDataFile returns a java File object.
thanks
Re: MarkableFileInputStreamFactory
Posted by Mark G <gi...@gmail.com>.
That is correct , sentence file does not need annotations, and the other files Are one name per line.
It uses the names file to annotate the sentences, and won't annotate anything that's in the blacklist file.
Let me know how it goes!
> On May 20, 2014, at 4:16 AM, Carlos Scheidecker <na...@gmail.com> wrote:
>
> I have not move forward on it, but yes Mark, want to use it.
>
> I have seen one of your examples.
>
> But have not figured out the proper format of the files. Here' s what I
> think from what I have been reading. Tell me if I am write.
>
> From class DefaultModelBuilderUtil method generateModel
>
> @param sentences a file that contains one sentence per line.
> * There should be at least 15K sentences
> * consisting of a representative sample
> from
> * user data
>
> This seems to be a text file where each sentence is on one line.
> I wonder if it has to be annotated, for instance:
>
> <START:person> Archimedes <END> used the method of exhaustion to
> approximate the value of π.Archimedes ( 287&ndash ;212 BC ) was the first
> to estimate π rigorously .
>
> Or just:
>
> Archimedes used the method of exhaustion to approximate the value of
> π.Archimedes ( 287&ndash ;212 BC ) was the first to estimate π rigorously .
>
>
> @param knownEntities a file consisting of a simple list of
> * unambiguous entities, one entry per
> line.
> * For instance, if one was trying to
> build a
> * person NER model then this file would
> be a
> * list of person names that are
> unambiguous
> * and are known to exist in the sentences
>
> This would be a text file list?
>
> Something like one name per line?
>
> Archimedes
> Socrates
> ....
>
>
> * @param knownEntitiesBlacklist This file contains a list of known bad
> hits
> * that the NER phase of this processing
> might
> * catch early one before the model
> iterates
> * to maturity
>
> Same as the knownEntities but a list of what NOT to mark as an entity?
>
>
> The rest seemed quite straight forward.
>
> Thanks,
>
> Carlos.
>
>
>
>
>> On Mon, May 19, 2014 at 5:34 PM, Mark G <gi...@gmail.com> wrote:
>>
>> No problem, Carlos are you using the model builder add on ?
>>
>>
>> Mg
>>
>>>> On May 19, 2014, at 6:29 PM, Carlos Scheidecker <na...@gmail.com>
>>> wrote:
>>>
>>> Thanks mate! Saw you updated the code. Cheers.
>>>
>>>
>>>> On Mon, May 19, 2014 at 3:24 PM, Mark G <gi...@gmail.com> wrote:
>>>>
>>>> OK, thanks Carlos, I think I will commit the change, seems like it
>> wouldn't
>>>> hurt. Anybody else?
>>>>
>>>>
>>>> On Mon, May 19, 2014 at 5:07 PM, Carlos Scheidecker <
>> nando.nlp@gmail.com
>>>>> wrote:
>>>>
>>>>> I am having the same issue Mark.
>>>>>
>>>>> The class is not public so it has no visibility
>>>>> inside opennlp.addons.modelbuilder.impls.GenericModelableImpl therefore
>>>> it
>>>>> cannot be built with Maven or resolved inside Eclipse.
>>>>>
>>>>> I have also been looking at new commits to fix that and there were
>> none.
>>>>>
>>>>>
>>>>>> On Mon, May 12, 2014 at 1:03 PM, Mark G <ma...@apache.org> wrote:
>>>>>>
>>>>>> Does MarkableFileInputStreamFactory need to be package private? I am
>>>>> using
>>>>>> it in an addon (modelbuilder-addon), I would like to either move it or
>>>>> make
>>>>>> it a public class. Perhaps I should be using a different class
>>>>> altogether?
>>>>>>
>>>>>> I am using it like this
>>>>>>
>>>>>> ObjectStream<String> lineStream =
>>>>>> new PlainTextByLineStream(new
>>>>>> MarkableFileInputStreamFactory(params.getAnnotatedTrainingDataFile()),
>>>>>> charset);
>>>>>> ObjectStream<NameSample> sampleStream = new
>>>>>> NameSampleDataStream(lineStream);
>>>>>>
>>>>>> where getAnnotatedTrainingDataFile returns a java File object.
>>>>>>
>>>>>> thanks
>>
Re: MarkableFileInputStreamFactory
Posted by Carlos Scheidecker <na...@gmail.com>.
I have not move forward on it, but yes Mark, want to use it.
I have seen one of your examples.
But have not figured out the proper format of the files. Here' s what I
think from what I have been reading. Tell me if I am write.
>From class DefaultModelBuilderUtil method generateModel
@param sentences a file that contains one sentence per line.
* There should be at least 15K sentences
* consisting of a representative sample
from
* user data
This seems to be a text file where each sentence is on one line.
I wonder if it has to be annotated, for instance:
<START:person> Archimedes <END> used the method of exhaustion to
approximate the value of π.Archimedes ( 287&ndash ;212 BC ) was the first
to estimate π rigorously .
Or just:
Archimedes used the method of exhaustion to approximate the value of
π.Archimedes ( 287&ndash ;212 BC ) was the first to estimate π rigorously .
@param knownEntities a file consisting of a simple list of
* unambiguous entities, one entry per
line.
* For instance, if one was trying to
build a
* person NER model then this file would
be a
* list of person names that are
unambiguous
* and are known to exist in the sentences
This would be a text file list?
Something like one name per line?
Archimedes
Socrates
....
* @param knownEntitiesBlacklist This file contains a list of known bad
hits
* that the NER phase of this processing
might
* catch early one before the model
iterates
* to maturity
Same as the knownEntities but a list of what NOT to mark as an entity?
The rest seemed quite straight forward.
Thanks,
Carlos.
On Mon, May 19, 2014 at 5:34 PM, Mark G <gi...@gmail.com> wrote:
> No problem, Carlos are you using the model builder add on ?
>
>
> Mg
>
> > On May 19, 2014, at 6:29 PM, Carlos Scheidecker <na...@gmail.com>
> wrote:
> >
> > Thanks mate! Saw you updated the code. Cheers.
> >
> >
> >> On Mon, May 19, 2014 at 3:24 PM, Mark G <gi...@gmail.com> wrote:
> >>
> >> OK, thanks Carlos, I think I will commit the change, seems like it
> wouldn't
> >> hurt. Anybody else?
> >>
> >>
> >> On Mon, May 19, 2014 at 5:07 PM, Carlos Scheidecker <
> nando.nlp@gmail.com
> >>> wrote:
> >>
> >>> I am having the same issue Mark.
> >>>
> >>> The class is not public so it has no visibility
> >>> inside opennlp.addons.modelbuilder.impls.GenericModelableImpl therefore
> >> it
> >>> cannot be built with Maven or resolved inside Eclipse.
> >>>
> >>> I have also been looking at new commits to fix that and there were
> none.
> >>>
> >>>
> >>>> On Mon, May 12, 2014 at 1:03 PM, Mark G <ma...@apache.org> wrote:
> >>>>
> >>>> Does MarkableFileInputStreamFactory need to be package private? I am
> >>> using
> >>>> it in an addon (modelbuilder-addon), I would like to either move it or
> >>> make
> >>>> it a public class. Perhaps I should be using a different class
> >>> altogether?
> >>>>
> >>>> I am using it like this
> >>>>
> >>>> ObjectStream<String> lineStream =
> >>>> new PlainTextByLineStream(new
> >>>> MarkableFileInputStreamFactory(params.getAnnotatedTrainingDataFile()),
> >>>> charset);
> >>>> ObjectStream<NameSample> sampleStream = new
> >>>> NameSampleDataStream(lineStream);
> >>>>
> >>>> where getAnnotatedTrainingDataFile returns a java File object.
> >>>>
> >>>> thanks
> >>
>
Re: MarkableFileInputStreamFactory
Posted by Mark G <gi...@gmail.com>.
No problem, Carlos are you using the model builder add on ?
Mg
> On May 19, 2014, at 6:29 PM, Carlos Scheidecker <na...@gmail.com> wrote:
>
> Thanks mate! Saw you updated the code. Cheers.
>
>
>> On Mon, May 19, 2014 at 3:24 PM, Mark G <gi...@gmail.com> wrote:
>>
>> OK, thanks Carlos, I think I will commit the change, seems like it wouldn't
>> hurt. Anybody else?
>>
>>
>> On Mon, May 19, 2014 at 5:07 PM, Carlos Scheidecker <nando.nlp@gmail.com
>>> wrote:
>>
>>> I am having the same issue Mark.
>>>
>>> The class is not public so it has no visibility
>>> inside opennlp.addons.modelbuilder.impls.GenericModelableImpl therefore
>> it
>>> cannot be built with Maven or resolved inside Eclipse.
>>>
>>> I have also been looking at new commits to fix that and there were none.
>>>
>>>
>>>> On Mon, May 12, 2014 at 1:03 PM, Mark G <ma...@apache.org> wrote:
>>>>
>>>> Does MarkableFileInputStreamFactory need to be package private? I am
>>> using
>>>> it in an addon (modelbuilder-addon), I would like to either move it or
>>> make
>>>> it a public class. Perhaps I should be using a different class
>>> altogether?
>>>>
>>>> I am using it like this
>>>>
>>>> ObjectStream<String> lineStream =
>>>> new PlainTextByLineStream(new
>>>> MarkableFileInputStreamFactory(params.getAnnotatedTrainingDataFile()),
>>>> charset);
>>>> ObjectStream<NameSample> sampleStream = new
>>>> NameSampleDataStream(lineStream);
>>>>
>>>> where getAnnotatedTrainingDataFile returns a java File object.
>>>>
>>>> thanks
>>
Re: MarkableFileInputStreamFactory
Posted by Carlos Scheidecker <na...@gmail.com>.
Thanks mate! Saw you updated the code. Cheers.
On Mon, May 19, 2014 at 3:24 PM, Mark G <gi...@gmail.com> wrote:
> OK, thanks Carlos, I think I will commit the change, seems like it wouldn't
> hurt. Anybody else?
>
>
> On Mon, May 19, 2014 at 5:07 PM, Carlos Scheidecker <nando.nlp@gmail.com
> >wrote:
>
> > I am having the same issue Mark.
> >
> > The class is not public so it has no visibility
> > inside opennlp.addons.modelbuilder.impls.GenericModelableImpl therefore
> it
> > cannot be built with Maven or resolved inside Eclipse.
> >
> > I have also been looking at new commits to fix that and there were none.
> >
> >
> > On Mon, May 12, 2014 at 1:03 PM, Mark G <ma...@apache.org> wrote:
> >
> > > Does MarkableFileInputStreamFactory need to be package private? I am
> > using
> > > it in an addon (modelbuilder-addon), I would like to either move it or
> > make
> > > it a public class. Perhaps I should be using a different class
> > altogether?
> > >
> > > I am using it like this
> > >
> > > ObjectStream<String> lineStream =
> > > new PlainTextByLineStream(new
> > > MarkableFileInputStreamFactory(params.getAnnotatedTrainingDataFile()),
> > > charset);
> > > ObjectStream<NameSample> sampleStream = new
> > > NameSampleDataStream(lineStream);
> > >
> > > where getAnnotatedTrainingDataFile returns a java File object.
> > >
> > > thanks
> > >
> >
>
Re: MarkableFileInputStreamFactory
Posted by Mark G <gi...@gmail.com>.
+1 to move it
> On May 20, 2014, at 4:11 AM, Jörn Kottmann <ko...@gmail.com> wrote:
>
>> On 05/19/2014 11:24 PM, Mark G wrote:
>> OK, thanks Carlos, I think I will commit the change, seems like it wouldn't
>> hurt. Anybody else?
>
> We can't do it like that, the cmdline package is marked as internal usage only. Therefore the classes in there should not be
> used outside of it. Inside cmdline pacakge the code is calling this factory method:
> CmdLineUtil.createInputStreamFactory
>
> Anyway, since user will need to create InputStreamFactory instances here and there it might be a good
> idea to move this class to the util package, or introduce an InputStreamFactoryUtil class which offers a couple
> of factory methods for various inputs (e.g. File, URL, Classpath, etc.).
>
> Any thoughts?
>
> Jörn
>
Re: MarkableFileInputStreamFactory
Posted by Rodrigo Agerri <ag...@gmail.com>.
+1 to move it to the util package.
Rodrigo
On Tue, May 20, 2014 at 10:11 AM, Jörn Kottmann <ko...@gmail.com> wrote:
> On 05/19/2014 11:24 PM, Mark G wrote:
>>
>> OK, thanks Carlos, I think I will commit the change, seems like it
>> wouldn't
>> hurt. Anybody else?
>
>
> We can't do it like that, the cmdline package is marked as internal usage
> only. Therefore the classes in there should not be
> used outside of it. Inside cmdline pacakge the code is calling this factory
> method:
> CmdLineUtil.createInputStreamFactory
>
> Anyway, since user will need to create InputStreamFactory instances here and
> there it might be a good
> idea to move this class to the util package, or introduce an
> InputStreamFactoryUtil class which offers a couple
> of factory methods for various inputs (e.g. File, URL, Classpath, etc.).
>
> Any thoughts?
>
> Jörn
>
Re: MarkableFileInputStreamFactory
Posted by Carlos Scheidecker <na...@gmail.com>.
Jorn,
I think it has to be fixed because the add-ons package does not compile
without it.
So, regardless the type of fix, as long as the add-ons package compiles and
works then I would say I am OK with anything.
Cheers
On Tue, May 20, 2014 at 2:11 AM, Jörn Kottmann <ko...@gmail.com> wrote:
> On 05/19/2014 11:24 PM, Mark G wrote:
>
>> OK, thanks Carlos, I think I will commit the change, seems like it
>> wouldn't
>> hurt. Anybody else?
>>
>
> We can't do it like that, the cmdline package is marked as internal usage
> only. Therefore the classes in there should not be
> used outside of it. Inside cmdline pacakge the code is calling this
> factory method:
> CmdLineUtil.createInputStreamFactory
>
> Anyway, since user will need to create InputStreamFactory instances here
> and there it might be a good
> idea to move this class to the util package, or introduce an
> InputStreamFactoryUtil class which offers a couple
> of factory methods for various inputs (e.g. File, URL, Classpath, etc.).
>
> Any thoughts?
>
> Jörn
>
>
Re: MarkableFileInputStreamFactory
Posted by Jörn Kottmann <ko...@gmail.com>.
On 05/19/2014 11:24 PM, Mark G wrote:
> OK, thanks Carlos, I think I will commit the change, seems like it wouldn't
> hurt. Anybody else?
We can't do it like that, the cmdline package is marked as internal
usage only. Therefore the classes in there should not be
used outside of it. Inside cmdline pacakge the code is calling this
factory method:
CmdLineUtil.createInputStreamFactory
Anyway, since user will need to create InputStreamFactory instances here
and there it might be a good
idea to move this class to the util package, or introduce an
InputStreamFactoryUtil class which offers a couple
of factory methods for various inputs (e.g. File, URL, Classpath, etc.).
Any thoughts?
Jörn
Re: MarkableFileInputStreamFactory
Posted by Carlos Scheidecker <na...@gmail.com>.
Mark,
If you do that , would save me a bunch because was fixing it myself and I
know it would be bad once I get an updated version. Never likes to mess
with code out of the brank.
Thank you so much!
Btw, trying to add more names to the person entity model.
cheers,
Carlos.
On Mon, May 19, 2014 at 3:24 PM, Mark G <gi...@gmail.com> wrote:
> OK, thanks Carlos, I think I will commit the change, seems like it wouldn't
> hurt. Anybody else?
>
>
> On Mon, May 19, 2014 at 5:07 PM, Carlos Scheidecker <nando.nlp@gmail.com
> >wrote:
>
> > I am having the same issue Mark.
> >
> > The class is not public so it has no visibility
> > inside opennlp.addons.modelbuilder.impls.GenericModelableImpl therefore
> it
> > cannot be built with Maven or resolved inside Eclipse.
> >
> > I have also been looking at new commits to fix that and there were none.
> >
> >
> > On Mon, May 12, 2014 at 1:03 PM, Mark G <ma...@apache.org> wrote:
> >
> > > Does MarkableFileInputStreamFactory need to be package private? I am
> > using
> > > it in an addon (modelbuilder-addon), I would like to either move it or
> > make
> > > it a public class. Perhaps I should be using a different class
> > altogether?
> > >
> > > I am using it like this
> > >
> > > ObjectStream<String> lineStream =
> > > new PlainTextByLineStream(new
> > > MarkableFileInputStreamFactory(params.getAnnotatedTrainingDataFile()),
> > > charset);
> > > ObjectStream<NameSample> sampleStream = new
> > > NameSampleDataStream(lineStream);
> > >
> > > where getAnnotatedTrainingDataFile returns a java File object.
> > >
> > > thanks
> > >
> >
>
Re: MarkableFileInputStreamFactory
Posted by Mark G <gi...@gmail.com>.
OK, thanks Carlos, I think I will commit the change, seems like it wouldn't
hurt. Anybody else?
On Mon, May 19, 2014 at 5:07 PM, Carlos Scheidecker <na...@gmail.com>wrote:
> I am having the same issue Mark.
>
> The class is not public so it has no visibility
> inside opennlp.addons.modelbuilder.impls.GenericModelableImpl therefore it
> cannot be built with Maven or resolved inside Eclipse.
>
> I have also been looking at new commits to fix that and there were none.
>
>
> On Mon, May 12, 2014 at 1:03 PM, Mark G <ma...@apache.org> wrote:
>
> > Does MarkableFileInputStreamFactory need to be package private? I am
> using
> > it in an addon (modelbuilder-addon), I would like to either move it or
> make
> > it a public class. Perhaps I should be using a different class
> altogether?
> >
> > I am using it like this
> >
> > ObjectStream<String> lineStream =
> > new PlainTextByLineStream(new
> > MarkableFileInputStreamFactory(params.getAnnotatedTrainingDataFile()),
> > charset);
> > ObjectStream<NameSample> sampleStream = new
> > NameSampleDataStream(lineStream);
> >
> > where getAnnotatedTrainingDataFile returns a java File object.
> >
> > thanks
> >
>
Re: MarkableFileInputStreamFactory
Posted by Carlos Scheidecker <na...@gmail.com>.
I am having the same issue Mark.
The class is not public so it has no visibility
inside opennlp.addons.modelbuilder.impls.GenericModelableImpl therefore it
cannot be built with Maven or resolved inside Eclipse.
I have also been looking at new commits to fix that and there were none.
On Mon, May 12, 2014 at 1:03 PM, Mark G <ma...@apache.org> wrote:
> Does MarkableFileInputStreamFactory need to be package private? I am using
> it in an addon (modelbuilder-addon), I would like to either move it or make
> it a public class. Perhaps I should be using a different class altogether?
>
> I am using it like this
>
> ObjectStream<String> lineStream =
> new PlainTextByLineStream(new
> MarkableFileInputStreamFactory(params.getAnnotatedTrainingDataFile()),
> charset);
> ObjectStream<NameSample> sampleStream = new
> NameSampleDataStream(lineStream);
>
> where getAnnotatedTrainingDataFile returns a java File object.
>
> thanks
>