You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@pig.apache.org by Russell Jurney <ru...@gmail.com> on 2013/02/19 00:16:50 UTC

Review Request: Review for PIG-3190, add LuceneTokenize and SnowballTokenize

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9511/
-----------------------------------------------------------

Review request for pig, Alan Gates, Prashant Sharma, Jonathan Coveney, and Gunther Hagleitner.


Description
-------

PIG-3190 adds two 'sane' tokenizers to Pig


This addresses bug PIG-3190.
    https://issues.apache.org/jira/browse/PIG-3190


Diffs
-----

  ivy.xml 70e8d50 
  src/org/apache/pig/builtin/LuceneTokenize.java PRE-CREATION 
  src/org/apache/pig/builtin/SnowballTokenize.java PRE-CREATION 
  test/org/apache/pig/test/TestLuceneTokenize.java PRE-CREATION 
  test/org/apache/pig/test/TestSnowballTokenize.java PRE-CREATION 
  test/org/apache/pig/test/data/ExpectedLuceneTokens.txt PRE-CREATION 
  test/org/apache/pig/test/data/ExpectedSnowballTokens.txt PRE-CREATION 
  test/org/apache/pig/test/data/InputFiles/ten_enron_emails.txt PRE-CREATION 

Diff: https://reviews.apache.org/r/9511/diff/


Testing
-------

Runs locally for me, two unit tests pass


Thanks,

Russell Jurney