You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "singhpratech (via GitHub)" <gi...@apache.org> on 2026/09/08 03:44:35 UTC
[I] [Python][C++] utf8_normalize ignores its `form` option: NFC and NFKC output is decomposed [arrow]
singhpratech opened a new issue, #51225:
URL: https://github.com/apache/arrow/issues/51225
**Describe the bug**
`pyarrow.compute.utf8_normalize` returns decomposed text for every `form`. With `form="NFC"` the
output is the NFD form (base letter plus combining mark), whether the input was precomposed or
decomposed. `unicodedata.normalize` from the standard library gives the expected result.
pyarrow 25.0.1, Python 3.13.9, macOS 26.6 (arm64), wheel from PyPI.
```python
import pyarrow as pa, pyarrow.compute as pc, unicodedata
s = "héllo" # precomposed e-acute, U+00E9
a = pa.array([s], pa.string())
for form in ("NFC", "NFD", "NFKC", "NFKD"):
out = pc.utf8_normalize(a, form=form).to_pylist()[0]
exp = unicodedata.normalize(form, s)
print(form, [hex(ord(c)) for c in out], "expected", [hex(ord(c)) for c in exp])
```
Output:
```
NFC ['0x68', '0x65', '0x301', '0x6c', '0x6c', '0x6f'] expected ['0x68', '0xe9', '0x6c', '0x6c', '0x6f']
NFD ['0x68', '0x65', '0x301', '0x6c', '0x6c', '0x6f'] expected ['0x68', '0x65', '0x301', '0x6c', '0x6c', '0x6f']
NFKC ['0x68', '0x65', '0x301', '0x6c', '0x6c', '0x6f'] expected ['0x68', '0xe9', '0x6c', '0x6c', '0x6f']
NFKD ['0x68', '0x65', '0x301', '0x6c', '0x6c', '0x6f'] expected ['0x68', '0x65', '0x301', '0x6c', '0x6c', '0x6f']
```
The same happens when the input is already decomposed (`"héllo"`): NFC and NFKC return it
unchanged instead of composing to U+00E9.
**Expected behavior**
`form="NFC"` and `form="NFKC"` compose, as `unicodedata.normalize` does; NFD and NFKD are correct
today.
**Component(s)**
Python, C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] utf8_normalize ignores its `form` option: NFC and NFKC output is decomposed [arrow]
Posted by "singhpratech (via GitHub)" <gi...@apache.org>.
singhpratech commented on issue #51225:
URL: https://github.com/apache/arrow/issues/51225#issuecomment-5585680692
Thank you for the fixture analysis, that is exactly the trap. I have a branch with the
`utf8proc_normalize_utf32()` call, the fixture fix and a pyarrow case ready and will open the PR
now; happy to have your review on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] utf8_normalize ignores its `form` option: NFC and NFKC output is decomposed [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #51225:
URL: https://github.com/apache/arrow/issues/51225#issuecomment-5621660020
Issue resolved by pull request 51237
https://github.com/apache/arrow/pull/51237
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] utf8_normalize ignores its `form` option: NFC and NFKC output is decomposed [arrow]
Posted by "singhpratech (via GitHub)" <gi...@apache.org>.
singhpratech commented on issue #51225:
URL: https://github.com/apache/arrow/issues/51225#issuecomment-5579337380
Reading the kernel on main (cpp/src/arrow/compute/kernels/scalar_string_utf8.cc, `Utf8NormalizeBase`):
`MakeDecomposeOptions` returns `UTF8PROC_COMPOSE` for NFC and NFKC, but the only utf8proc call the
kernel makes is `utf8proc_decompose`, which decomposes regardless of that flag; composition happens
in `utf8proc_reencode`, which is never called. So NFC and NFKC come out as NFD and NFKD. Calling
`utf8proc_reencode` on the scratch buffer with the same options after the decompose (as
`utf8proc_map` does internally) would give the composed forms.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] utf8_normalize ignores its `form` option: NFC and NFKC output is decomposed [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou closed issue #51225: [Python][C++] utf8_normalize ignores its `form` option: NFC and NFKC output is decomposed
URL: https://github.com/apache/arrow/issues/51225
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org