[I] WordpieceTokenizer ignores changes to the shared WhitespaceTokenizer and shares punctuation isolation with WordpieceEncoder (opennlp)

["rzo1 (via GitHub)" <gi...@apache.org>]

rzo1 opened a new issue, #1440:
URL: https://github.com/apache/opennlp/issues/1440

   WordpieceTokenizer split words with the shared WhitespaceTokenizer.INSTANCE, so calling setKeepNewLines(true) on that instance elsewhere in an application changed the wordpiece output and added line-separator tokens. BertNormalization also kept two separate punctuation-isolation loops for WordpieceTokenizer and WordpieceEncoder that could drift apart. WordpieceTokenizer should use its own whitespace splitter, and both BERT paths should share one punctuation-isolation routine.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] WordpieceTokenizer ignores changes to the shared WhitespaceTokenizer and shares punctuation isolation with WordpieceEncoder (opennlp)

["rzo1 (via GitHub)" <gi...@apache.org>]

rzo1 closed issue #1440: WordpieceTokenizer ignores changes to the shared WhitespaceTokenizer and shares punctuation isolation with WordpieceEncoder
URL: https://github.com/apache/opennlp/issues/1440


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org