You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@commons.apache.org by "Gary D. Gregory (Jira)" <ji...@apache.org> on 2025/04/13 23:45:00 UTC
[jira] [Created] (LANG-1770) StringUtils.abbreviate is not emoji aware, breaks surrogate pairs
Gary D. Gregory created LANG-1770:
-------------------------------------
Summary: StringUtils.abbreviate is not emoji aware, breaks surrogate pairs
Key: LANG-1770
URL: https://issues.apache.org/jira/browse/LANG-1770
Project: Commons Lang
Issue Type: Bug
Components: lang.*
Affects Versions: 3.17.0
Reporter: Gary D. Gregory
---------- Forwarded message ---------
From: Carsten Kirschner <ca...@corussoft.de.invalid>
Date: Fri, Apr 11, 2025 at 10:15 AM
Subject: [lang] StringUtils.abbreviate is not emoji aware, breaks surrogate pairs
To: dev@commons.apache.org <de...@commons.apache.org>
Hello,
The current commons lang3 StringUtils.abbreviate (3.17.0) implementation will destroy 4 byte emoji characters and larger grapheme clusters. I know that handling grapheme correctly before java 20 is not possible, but at least a codepoint aware solution with String.offsetByCodPoints could be build. I wrote a small test to show the problem.
The zero width joiners in the family emoji are questionable for the abbreviate, but there should never be a question mark for an invalid char in the result as there is now.
The problem is not so much the „doesn’t look nice“ aspect of the broken emoji, but if that abbreviated string is passed to an XML Writer (com.ctc.wstx.io.UTF8Writer in my case) it throws an exception on this broken byte sequence. Like this: Caused by: java.io.IOException: Broken surrogate pair: first char 0xd83c, second 0x2e; illegal combination
at com.ctc.wstx.io.UTF8Writer._convertSurrogate(UTF8Writer.java:402) ~[woodstox-core-7.0.0.jar:7.0.0]
Thanks,
Carsten
import org.apache.commons.lang3.StringUtils;
import org.junit.Test;
import static org.junit.Assert.*;
public class AbbreviateTest {
String[] expectedResultsFox = {
"🦊...", // 4
"🦊🦊...",
"🦊🦊🦊...",
"🦊🦊🦊🦊...",
"🦊🦊🦊🦊🦊...",
"🦊🦊🦊🦊🦊🦊...",
"🦊🦊🦊🦊🦊🦊🦊...", // 10
};
String[] expectedResultsFamilyWithCodepoints = {
"👩...",
"👩🏻...",
"👩🏻...", // zero width joiner
"👩🏻👨...",
"👩🏻👨🏻...",
"👩🏻👨🏻...",
"👩🏻👨🏻👦..."
};
String[] expectedResultsFamilyWithGrapheme = {
"👩🏻👨🏻👦🏻👦🏻...", // 4
"👩🏻👨🏻👦🏻👦🏻👩🏼👨🏼👦🏼👦🏼...",
"👩🏻👨🏻👦🏻👦🏻👩🏼👨🏼👦🏼👦🏼👩🏽👨🏽👦🏽👦🏽...",
"👩🏻👨🏻👦🏻👦🏻👩🏼👨🏼👦🏼👦🏼👩🏽👨🏽👦🏽👦🏽👩🏾👨🏾👦🏾👦🏾...",
"👩🏻👨🏻👦🏻👦🏻👩🏼👨🏼👦🏼👦🏼👩🏽👨🏽👦🏽👦🏽👩🏾👨🏾👦🏾👦🏾👩🏿👨🏿👦🏿👦🏿...",
"👩🏻👨🏻👦🏻👦🏻👩🏼👨🏼👦🏼👦🏼👩🏽👨🏽👦🏽👦🏽👩🏾👨🏾👦🏾👦🏾👩🏿👨🏿👦🏿👦🏿👩🏻👨🏻👦🏻👦🏻...",
"👩🏻👨🏻👦🏻👦🏻👩🏼👨🏼👦🏼👦🏼👩🏽👨🏽👦🏽👦🏽👩🏾👨🏾👦🏾👦🏾👩🏿👨🏿👦🏿👦🏿👩🏻👨🏻👦🏻👦🏻👩🏼👨🏼👦🏼👦🏼..." // 10
};
@Test
public void abberviateTest() {
String abbreviateResult;
for(var i = 4; i <= 10; i++) {
abbreviateResult = StringUtils.abbreviate("🦊🦊🦊🦊🦊🦊🦊🦊🦊🦊🦊🦊🦊🦊", i);
System.out.println(abbreviateResult);
//assertEquals(expectedResultsFox[i - 4], abbreviateResult);
}
for(var i = 4; i <= 10; i++) {
abbreviateResult = StringUtils.abbreviate("👩🏻👨🏻👦🏻👦🏻👩🏼👨🏼👦🏼👦🏼👩🏽👨🏽👦🏽👦🏽👩🏾👨🏾👦🏾👦🏾👩🏿👨🏿👦🏿👦🏿👩🏻👨🏻👦🏻👦🏻👩🏼👨🏼👦🏼👦🏼👩🏽👨🏽👦🏽👦🏽👩🏾👨🏾👦🏾👦🏾👩🏿👨🏿👦🏿👦🏿", i);
System.out.println(abbreviateResult);
//assertEquals(expectedResultsFamilyWithCodepoints[i - 4], abbreviateResult);
}
}
}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)