[GitHub] [incubator-hivemall] myui opened a new pull request #237: Implement Korean text tokenizer

[GitBox <gi...@apache.org>]

myui opened a new pull request #237:
URL: https://github.com/apache/incubator-hivemall/pull/237


   ## What changes were proposed in this pull request?
   
   Implement Korean text tokenizer
   
   ## What type of PR is it?
   
   Feature
   
   ## What is the Jira issue?
   
   https://issues.apache.org/jira/browse/HIVEMALL-307
   
   ## How was this patch tested?
   
   unit tests and manual tests on EMR
   
   ## How to use this feature?
   
   ```sql
   -- show version of lucene-analyzers-nori
   select tokenize_ko();
   > 8.8.2
   
   select tokenize_ko("소설 무궁화꽃이 피었습니다.");
   > ["소설","무궁","화","꽃","피"]
   
   select tokenize_ko("소설 무궁화꽃이 피었습니다.", null, "mixed");
   > ["소설","무궁화","무궁","화","꽃","피"]
   
   select tokenize_ko("소설 무궁화꽃이 피었습니다.", null, "discard", array("E", "VV"));
   > ["소설","무궁","화","꽃","이"]
   
   select tokenize_ko("Hello, world.", null, "none", array(), true);
   > ["h","e","l","l","o","w","o","r","l","d"]
   
   select tokenize_ko("Hello, world.", null, "none", array(), false);
   > ["hello","world"]
   
   select tokenize_ko("나는 C++ 언어를 프로그래밍 언어로 사랑한다.", null, "discard", array());
   > ["나","는","c","언어","를","프로그래밍","언어","로","사랑","하","ᆫ다"]
   
   select tokenize_ko("나는 C++ 언어를 프로그래밍 언어로 사랑한다.", array("C++"), "discard", array());
   > ["나","는","c++","언어","를","프로그래밍","언어","로","사랑","하","ᆫ다"]
   ```
   
   ## Checklist
   
   - [x] Did you apply source code formatter, i.e., `./bin/format_code.sh`, for your commit?
   - [x] Did you run system tests on Hive (or Spark)?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-hivemall] asfgit closed pull request #237: Implement Korean text tokenizer

[GitBox <gi...@apache.org>]

asfgit closed pull request #237:
URL: https://github.com/apache/incubator-hivemall/pull/237


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org