Adding a script generate test data table for CRF

["Nguyen,Giang H" <gi...@ufl.edu>]

Hello everyone,

I'm Giang Nguyen, a student at the University of Florida. I have been trying to get familiar with Madlib as a user to find out what I can potentially contribute to Madlib. I have run CRF as a user, and one of the thing I noticed that could cause users (especially users that aren't familiar with CRF and Postgres) some trouble is that they have to manually create the testing segment table test_segmenttbl(doc_id integer, start_pos integer, seg_text text) to feed into the crf_test_fgen(). This could be a tedious task for users especially when they have a big corpus of text. I think It could be very helpful if we write a python script in Madlib to tokenize words and assign the doc_id and start_pos correspondingly and store it into the database. Hence, users can save a lot more time when using CRF and also enable them to conveniently run crf model on big testing data.


Best,

Giang Nguyen

University of Florida

Re: Adding a script generate test data table for CRF

["Nguyen,Giang H" <gi...@ufl.edu>]

Yes, I will create a JIRA for it and start working on it also.

-Giang

> On Feb 23, 2016, at 7:59 PM, Frank McQuillan <fm...@pivotal.io> wrote:
> 
> Jim's approach seems like a reasonable way to go.
> 
> Giang, can you create a JIRA for this request?  You are welcome to start
> working on it if you would like to contribute this to improve CRF usability.
> 
> Frank
> 
>> On Tue, Feb 23, 2016 at 3:24 PM, Jim Nasby <Ji...@bluetreble.com> wrote:
>> 
>>> On 2/23/16 11:07 AM, Nguyen,Giang H wrote:
>>> 
>>> I think It could be very helpful if we write a python script in Madlib to
>>> tokenize words and assign the doc_id and start_pos correspondingly and
>>> store it into the database. Hence, users can save a lot more time when
>>> using CRF and also enable them to conveniently run crf model on big testing
>>> data.
>> 
>> Perhaps the Postgres text search stuff could be used for this (maybe
>> to_tsvector())?
>> --
>> Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
>> Experts in Analytics, Data Architecture and PostgreSQL
>> Data in Trouble? Get it in Treble! http://BlueTreble.com
>> 

Re: Adding a script generate test data table for CRF

[Frank McQuillan <fm...@pivotal.io>]

Jim's approach seems like a reasonable way to go.

Giang, can you create a JIRA for this request?  You are welcome to start
working on it if you would like to contribute this to improve CRF usability.

Frank

On Tue, Feb 23, 2016 at 3:24 PM, Jim Nasby <Ji...@bluetreble.com> wrote:

> On 2/23/16 11:07 AM, Nguyen,Giang H wrote:
>
>> I think It could be very helpful if we write a python script in Madlib to
>> tokenize words and assign the doc_id and start_pos correspondingly and
>> store it into the database. Hence, users can save a lot more time when
>> using CRF and also enable them to conveniently run crf model on big testing
>> data.
>>
>
> Perhaps the Postgres text search stuff could be used for this (maybe
> to_tsvector())?
> --
> Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
> Experts in Analytics, Data Architecture and PostgreSQL
> Data in Trouble? Get it in Treble! http://BlueTreble.com
>

Re: Adding a script generate test data table for CRF

[Jim Nasby <Ji...@BlueTreble.com>]

On 2/23/16 11:07 AM, Nguyen,Giang H wrote:
> I think It could be very helpful if we write a python script in Madlib to tokenize words and assign the doc_id and start_pos correspondingly and store it into the database. Hence, users can save a lot more time when using CRF and also enable them to conveniently run crf model on big testing data.

Perhaps the Postgres text search stuff could be used for this (maybe 
to_tsvector())?
-- 
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com