[GitHub] Neutron3529 opened a new issue #11526: Bug in mxnet.contrib.text.utils.count_tokens_from_str

[GitBox <gi...@apache.org>]

Neutron3529 opened a new issue #11526: Bug in mxnet.contrib.text.utils.count_tokens_from_str
URL: https://github.com/apache/incubator-mxnet/issues/11526
 
 
    mxnet.contrib.text.utils.count_tokens_from_str(source_str, token_delim=' ', seq_delim='\n', to_lower=False, counter_to_update=None)[source]
   
   the description is
   Counts tokens in the specified string.
   
   For token_delim=’’ and seq_delim=’’, a specified string of two sequences of tokens may look like:
   ```
   <td>token1<td>token2<td>token3<td><sd><td>token4<td>token5<td><sd>
   ```
   But if one use "*" as a token_delim,executing`text.utils.count_tokens_from_str("*".join(s),"*")`, Error occurs:
   ```
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "d:\program files\python37\lib\site-packages\mxnet\contrib\text\utils.py", line 69, in count_tokens_from_str
       re.split(token_delim + '|' + seq_delim, source_str))
     File "d:\program files\python37\lib\re.py", line 213, in split
       return _compile(pattern, flags).split(string, maxsplit)
     File "d:\program files\python37\lib\re.py", line 286, in _compile
       p = sre_compile.compile(pattern, flags)
     File "d:\program files\python37\lib\sre_compile.py", line 764, in compile
       p = sre_parse.parse(p, flags)
     File "d:\program files\python37\lib\sre_parse.py", line 930, in parse
       p = _parse_sub(source, pattern, flags & SRE_FLAG_VERBOSE, 0)
     File "d:\program files\python37\lib\sre_parse.py", line 426, in _parse_sub
       not nested and not items))
     File "d:\program files\python37\lib\sre_parse.py", line 651, in _parse
       source.tell() - here + len(this))
   re.error: nothing to repeat at position 1
   ```
   if one really wanna regard "*" as a delim they should use `"\\*"` rather than `"*"`
   more, using "\\" as a token delim will get a unexpect response.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services