You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "nikolamand-db (via GitHub)" <gi...@apache.org> on 2024/04/23 08:28:58 UTC
[PR] [SPARK-46841] Add collation support for ICU locales and collation specifiers [spark]
nikolamand-db opened a new pull request, #46180:
URL: https://github.com/apache/spark/pull/46180
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606742620
##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -30,31 +33,107 @@ import org.apache.spark.unsafe.types.UTF8String.{fromString => toUTF8}
class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ignore funsuite
test("collationId stability") {
- val utf8Binary = fetchCollation(0)
+ assert(UTF8_BINARY_COLLATION_ID == 0)
+ assert(INDETERMINATE_COLLATION_ID == 1 << 30)
+
+ val utf8Binary = fetchCollation(UTF8_BINARY_COLLATION_ID)
assert(utf8Binary.collationName == "UTF8_BINARY")
assert(utf8Binary.supportsBinaryEquality)
- val utf8BinaryLcase = fetchCollation(1)
+ val utf8BinaryLcase = fetchCollation(UTF8_BINARY_LCASE_COLLATION_ID)
assert(utf8BinaryLcase.collationName == "UTF8_BINARY_LCASE")
assert(!utf8BinaryLcase.supportsBinaryEquality)
- val unicode = fetchCollation(2)
+ val unicode = fetchCollation(UNICODE_COLLATION_ID)
assert(unicode.collationName == "UNICODE")
- assert(unicode.supportsBinaryEquality);
+ assert(unicode.supportsBinaryEquality)
- val unicodeCi = fetchCollation(3)
+ val unicodeCi = fetchCollation(UNICODE_CI_COLLATION_ID)
assert(unicodeCi.collationName == "UNICODE_CI")
assert(!unicodeCi.supportsBinaryEquality)
}
- test("fetch invalid collation name") {
- val error = intercept[SparkException] {
- fetchCollation("UTF8_BS")
+ test("UTF8_BINARY and ICU root locale collation names") {
+ // collation name already normalized
+ Seq(
+ "UTF8_BINARY",
+ "UTF8_BINARY_LCASE",
+ "UTF8_BINARY_UCASE",
+ "UNICODE",
+ "UNICODE_CI",
+ "UNICODE_AI",
+ "UNICODE_CI_AI",
+ "UNICODE_LCASE",
+ "UNICODE_UCASE"
+ ).foreach(collationName => {
+ val col = fetchCollation(collationName)
+ assert(col.collationName == collationName)
+ })
+ // collation name normalization
+ Seq(
+ // UTF8_BINARY
+ ("UTF8_BINARY_CS", "UTF8_BINARY"),
+ ("UTF8_BINARY_AS", "UTF8_BINARY"),
+ ("UTF8_BINARY_CS_AS", "UTF8_BINARY"),
+ ("UTF8_BINARY_AS_CS", "UTF8_BINARY"),
Review Comment:
This is still under debate, we need to resolve whether we allow relaxed ordering for specifiers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606730469
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected static int parseSpecifiers(
+ String originalName, String collationName, int splitStart) throws SparkException {
+ int specifiers = 0;
+ String[] parts = collationName.substring(splitStart).split("_");
+ for (String part : parts) {
+ if (!part.isEmpty()) {
+ if (part.equals("UNSPECIFIED") || part.equals("INDETERMINATE")) {
+ throw collationInvalidNameException(originalName);
+ } else if (Arrays.stream(CaseSensitivity.values()).anyMatch(
+ (s) -> s.toString().equals(part))) {
+ specifiers |=
+ CaseSensitivity.valueOf(part).ordinal() << CASE_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(AccentSensitivity.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ AccentSensitivity.valueOf(part).ordinal() << ACCENT_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(CaseConversion.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ CaseConversion.valueOf(part).ordinal() << CASE_CONVERSION_OFFSET;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+ }
+ return specifiers;
+ }
+
+ protected abstract int getCollationId();
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ public static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.LCASE).getCollationId();
+ public static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).buildCollation();
+
+ private CollationSpecUTF8Binary(CaseConversion caseConversion) {
+ super(null, CaseSensitivity.CS, AccentSensitivity.AS, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int collationId = 0;
+ int specifiers = CollationSpec.parseSpecifiers(
+ originalName, collationName, "UTF8_BINARY".length());
+ if ((specifiers & ~(CASE_CONVERSION_MASK << CASE_CONVERSION_OFFSET)) != 0) {
+ throw collationInvalidNameException(originalName);
+ }
+ collationId |= specifiers;
+ return collationId;
+ }
+
+ @Override
+ protected int getCollationId() {
+ int collationId = 0;
+ collationId |= caseConversion.ordinal() << CASE_CONVERSION_OFFSET;
+ return collationId;
+ }
+
+ public static CollationSpecUTF8Binary fromCollationId(int collationId)
+ throws SparkException {
+ int originalCollationId = collationId;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
+ if (collationId != 0 || caseConversionOrdinal >= CaseConversion.values().length) {
+ throw SparkException.internalError("Invalid UTF8_BINARY collation id " +
+ originalCollationId);
+ } else {
+ CaseConversion caseConversion = CaseConversion.values()[caseConversionOrdinal];
+ return new CollationSpecUTF8Binary(caseConversion);
+ }
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ Comparator<UTF8String> comparator;
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ comparator = UTF8String::binaryCompare;
+ } else if (collationId == UTF8_BINARY_LCASE_COLLATION_ID) {
+ comparator = UTF8String::compareLowerCase;
+ } else {
+ comparator = (s1, s2) -> {
+ UTF8String convertedS1 = caseConversion(s1);
+ UTF8String convertedS2 = caseConversion(s2);
+ return convertedS1.binaryCompare(convertedS2);
+ };
+ }
+ return new Collation(
+ collationName(),
+ null,
+ comparator,
+ "1.0",
+ s -> (long) caseConversion(s).hashCode(),
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_LCASE_COLLATION_ID
+ );
+ }
+
+ private UTF8String caseConversion(UTF8String s) {
+ if (caseConversion == CaseConversion.LCASE) {
+ return s.toLowerCase();
+ } else if (caseConversion == CaseConversion.UCASE) {
+ return s.toUpperCase();
+ } else {
+ return s;
+ }
+ }
+
+ private String collationName() {
+ StringBuilder builder = new StringBuilder();
+ builder.append("UTF8_BINARY");
+ if (caseConversion != CaseConversion.UNSPECIFIED) {
+ builder.append('_');
+ builder.append(caseConversion.toString());
+ }
+ return builder.toString();
+ }
+ }
+
+ private static class CollationSpecICU extends CollationSpec {
+
+ private static final String[] ICULocaleNames;
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+ private static final String ICUCollatorVersion = "153.120.0.0";
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
+ for (ULocale locale : locales) {
+ if (locale.getVariant().isEmpty()) {
+ String language = locale.getLanguage();
+ assert (!language.isEmpty());
+ StringBuilder builder = new StringBuilder(language);
+ String script = locale.getScript();
+ if (!script.isEmpty()) {
+ builder.append('_');
+ builder.append(script);
+ }
+ String country = locale.getISO3Country();
+ if (!country.isEmpty()) {
+ builder.append('_');
+ builder.append(country);
+ }
+ String localeName = builder.toString();
+ assert (!ICULocaleMap.containsKey(localeName));
+ ICULocaleMap.put(localeName, locale);
+ }
+ }
+ for (String localeName : ICULocaleMap.keySet()) {
+ String localeUppercase = localeName.toUpperCase();
+ assert (!ICULocaleMapUppercase.containsKey(localeUppercase));
+ ICULocaleMapUppercase.put(localeUppercase, localeName);
+ }
+ ICULocaleNames = ICULocaleMap.keySet().toArray(new String[0]);
+ Arrays.sort(ICULocaleNames);
+ assert (ICULocaleNames.length <= (1 << 16));
+ for (int i = 0; i < ICULocaleNames.length; i++) {
+ ICULocaleToId.put(ICULocaleNames[i], i);
+ }
+ }
+
+ public static final int UNICODE_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CS, AccentSensitivity.AS,
+ CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UNICODE_CI_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CI, AccentSensitivity.AS,
+ CaseConversion.UNSPECIFIED).getCollationId();
+
+ private CollationSpecICU(String locale, CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity, CaseConversion caseConversion) {
+ super(locale, caseSensitivity, accentSensitivity, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int lastPos = -1;
+ for (int i = 1; i <= collationName.length(); i++) {
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606706322
##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -152,4 +219,147 @@ class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ig
}
})
}
+
+ test("test collation caching") {
+ Seq(
+ "UTF8_BINARY",
+ "UTF8_BINARY_LCASE",
+ "UTF8_BINARY_UCASE",
+ "UNICODE",
+ "UNICODE_LCASE",
+ "UNICODE_UCASE",
+ "UNICODE_CI",
+ "UNICODE_AI_CI",
+ "UNICODE_AI_CI_LCASE",
+ "UNICODE_AI_CI_UCASE"
+ ).foreach(collationId => {
+ val col1 = fetchCollation(collationId)
+ val col2 = fetchCollation(collationId)
+ assert(col1 eq col2) // reference equality
+ })
+ }
+
+ test("collations with ICU non-root localization") {
+ Seq(
+ // language only
+ "en",
+ "en_CS",
+ "en_CI",
+ "en_AS",
+ "en_AI",
+ "en_LCASE",
+ "en_UCASE",
+ // language + 3-letter country code
+ "en_USA",
+ "en_USA_CS",
+ "en_USA_CI",
+ "en_USA_AS",
+ "en_USA_AI",
+ "en_USA_LCASE",
+ "en_USA_UCASE",
+ // language + script code
+ "sr_Cyrl",
+ "sr_Cyrl_CS",
+ "sr_Cyrl_CI",
+ "sr_Cyrl_AS",
+ "sr_Cyrl_AI",
+ "sr_Cyrl_LCASE",
+ "sr_Cyrl_UCASE",
+ // language + script code + 3-letter country code
+ "sr_Cyrl_SRB",
+ "sr_Cyrl_SRB_CS",
+ "sr_Cyrl_SRB_CI",
+ "sr_Cyrl_SRB_AS",
+ "sr_Cyrl_SRB_AI",
+ "sr_Cyrl_SRB_LCASE",
+ "sr_Cyrl_SRB_UCASE"
+ ).foreach(collationICU => {
+ val col = fetchCollation(collationICU)
+ assert(col.collator.getLocale(ULocale.VALID_LOCALE) != ULocale.ROOT)
+ })
+ }
+
+ test("invalid names of collations with ICU non-root localization") {
+ Seq(
+ "en_US", // must use 3-letter country code
+ "enn",
+ "en_AAA",
+ "en_Something",
+ "en_Something_USA",
+ "en_Latn_USA", // use en_USA instead
+ "en_Cyrl_USA",
+ "en_USA_AAA",
+ "sr_Cyrl_SRB_AAA"
+ ).foreach(collationName => {
+ val error = intercept[SparkException] {
+ fetchCollation(collationName)
+ }
+
+ assert(error.getErrorClass === "COLLATION_INVALID_NAME")
+ assert(error.getMessageParameters.asScala === Map("collationName" -> collationName))
+ })
+ }
+
+ test("collations name normalization for ICU non-root localization") {
+ Seq(
+ ("en_USA", "en_USA"),
+ ("en_CS", "en"),
+ ("en_AS", "en"),
+ ("en_CS_AS", "en"),
+ ("en_AI_CI", "en_CI_AI"),
+ ("en_USA_AI_CI", "en_USA_CI_AI"),
+ // randomized case
+ ("EN_USA", "en_USA"),
+ ("eN_usA_ci_uCASe_aI", "en_USA_CI_AI_UCASE"),
+ ("SR_CYRL", "sr_Cyrl"),
+ ("sr_cyrl_srb", "sr_Cyrl_SRB"),
+ ("sR_cYRl_sRb", "sr_Cyrl_SRB")
+ ).foreach {
+ case (name, normalized) =>
+ val col = fetchCollation(name)
+ assert(col.collationName == normalized)
+ }
+ }
+
+ test("invalid collationId") {
+ val badCollationIds = Seq(
+ -1, // user-defined collation range
Review Comment:
Outdated, new approach is https://github.com/apache/spark/pull/46180/files#diff-9c12d32db9d55dd6ecb5b10f2fc57c7ba7de7275cab57bf157fa42cbc09f3876R331.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "dbatomic (via GitHub)" <gi...@apache.org>.
dbatomic commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1592441389
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,438 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ public static int collationNameToId(String originalCollationName) throws SparkException {
+ String collationName = originalCollationName.toUpperCase();
+ try {
+ if (collationName.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName);
+ }
+ } catch (SparkException e) {
+ throw new SparkException(
+ "COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", originalCollationName)),
+ e);
+ }
+ }
+
+ protected static int parseSpecifiers(String specString) throws SparkException {
+ int specifiers = 0;
+ String[] parts = specString.split("_");
+ for (String part : parts) {
+ if (!part.isEmpty()) {
+ if (part.equals("UNSPECIFIED") || part.equals("INDETERMINATE")) {
+ throw new SparkException(part + " collation specifier reserved for internal use");
+ } else if (Arrays.stream(CaseSensitivity.values()).anyMatch(
+ (s) -> s.toString().equals(part))) {
+ specifiers |=
+ CaseSensitivity.valueOf(part).ordinal() << CASE_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(AccentSensitivity.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ AccentSensitivity.valueOf(part).ordinal() << ACCENT_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(CaseConversion.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ CaseConversion.valueOf(part).ordinal() << CASE_CONVERSION_OFFSET;
+ } else {
+ throw new SparkException("Invalid collation specifier value " + part);
+ }
+ }
+ }
+ return specifiers;
+ }
+
+ protected abstract int getCollationId();
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ public static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.LCASE).getCollationId();
+ public static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).buildCollation();
+
+ private CollationSpecUTF8Binary(CaseConversion caseConversion) {
+ super(null, CaseSensitivity.CS, AccentSensitivity.AS, caseConversion);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ int collationId = 0;
+ int specifiers = CollationSpec.parseSpecifiers(
+ collationName.substring("UTF8_BINARY".length()));
+ collationId |= specifiers & (CASE_CONVERSION_MASK << CASE_CONVERSION_OFFSET);
+ return collationId;
+ }
+
+ @Override
+ protected int getCollationId() {
+ int collationId = 0;
+ collationId |= caseConversion.ordinal() << CASE_CONVERSION_OFFSET;
+ return collationId;
+ }
+
+ public static CollationSpecUTF8Binary fromCollationId(int collationId)
+ throws SparkException {
+ int originalCollationId = collationId;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
+ if (collationId != 0 || caseConversionOrdinal >= CaseConversion.values().length) {
+ throw SparkException.internalError("Invalid UTF8_BINARY collation id " +
+ originalCollationId);
+ } else {
+ CaseConversion caseConversion = CaseConversion.values()[caseConversionOrdinal];
+ return new CollationSpecUTF8Binary(caseConversion);
+ }
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ Comparator<UTF8String> comparator;
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ comparator = UTF8String::binaryCompare;
+ } else if (collationId == UTF8_BINARY_LCASE_COLLATION_ID) {
+ comparator = UTF8String::compareLowerCase;
+ } else {
+ comparator = (s1, s2) -> {
+ UTF8String convertedS1 = caseConversion(s1);
+ UTF8String convertedS2 = caseConversion(s2);
+ return convertedS1.binaryCompare(convertedS2);
+ };
+ }
+ return new Collation(
+ collationName(),
+ null,
+ comparator,
+ "1.0",
+ s -> (long) caseConversion(s).hashCode(),
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_LCASE_COLLATION_ID
+ );
+ }
+
+ private UTF8String caseConversion(UTF8String s) {
+ if (caseConversion == CaseConversion.LCASE) {
+ return s.toLowerCase();
+ } else if (caseConversion == CaseConversion.UCASE) {
+ return s.toUpperCase();
+ } else {
+ return s;
+ }
+ }
+
+ private String collationName() {
+ StringBuilder builder = new StringBuilder();
+ builder.append("UTF8_BINARY");
+ if (caseConversion != CaseConversion.UNSPECIFIED) {
+ builder.append('_');
+ builder.append(caseConversion.toString());
+ }
+ return builder.toString();
+ }
+ }
+
+ private static class CollationSpecICU extends CollationSpec {
+
+ private static final String[] ICULocaleNames;
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
+ for (ULocale locale : locales) {
+ if (locale.getVariant().isEmpty()) {
+ String language = locale.getLanguage();
+ assert (!language.isEmpty());
+ StringBuilder builder = new StringBuilder(language);
+ String script = locale.getScript();
+ if (!script.isEmpty()) {
+ builder.append('_');
+ builder.append(script);
+ }
+ String country = locale.getISO3Country();
+ if (!country.isEmpty()) {
+ builder.append('_');
+ builder.append(country);
+ }
+ String localeName = builder.toString();
+ assert (!ICULocaleMap.containsKey(localeName));
+ ICULocaleMap.put(localeName, locale);
+ }
+ }
+ for (String localeName : ICULocaleMap.keySet()) {
+ String localeUppercase = localeName.toUpperCase();
+ assert (!ICULocaleMapUppercase.containsKey(localeUppercase));
+ ICULocaleMapUppercase.put(localeUppercase, localeName);
+ }
+ ICULocaleNames = ICULocaleMap.keySet().toArray(new String[0]);
+ Arrays.sort(ICULocaleNames);
+ assert (ICULocaleNames.length <= (1 << 16));
+ for (int i = 0; i < ICULocaleNames.length; i++) {
+ ICULocaleToId.put(ICULocaleNames[i], i);
+ }
+ }
+
+ public static final int UNICODE_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CS, AccentSensitivity.AS,
+ CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UNICODE_CI_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CI, AccentSensitivity.AS,
+ CaseConversion.UNSPECIFIED).getCollationId();
+
+ private CollationSpecICU(String locale, CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity, CaseConversion caseConversion) {
+ super(locale, caseSensitivity, accentSensitivity, caseConversion);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ int lastPos = -1;
+ for (int i = 1; i <= collationName.length(); i++) {
+ String localeName = collationName.substring(0, i);
+ if (ICULocaleMapUppercase.containsKey(localeName)) {
+ lastPos = i;
+ }
+ }
+ if (lastPos == -1) {
+ throw new SparkException("Invalid locale in collation name value " + collationName);
+ } else {
+ int collationId = 0;
+ collationId |= ImplementationProvider.ICU.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+ collationId |= CollationSpec.parseSpecifiers(collationName.substring(lastPos));
+ String normalizedLocaleName = ICULocaleMapUppercase.get(
+ collationName.substring(0, lastPos));
+ collationId |= ICULocaleToId.get(normalizedLocaleName) << LOCALE_OFFSET;
+ return collationId;
+ }
+ }
+
+ public static CollationSpecICU fromCollationId(int collationId) throws SparkException {
+ int originalCollationId = collationId;
+ int caseSensitivityOrdinal =
+ (collationId >> CASE_SENSITIVITY_OFFSET) & CASE_SENSITIVITY_MASK;
+ int accentSensitivityOrdinal =
+ (collationId >> ACCENT_SENSITIVITY_OFFSET) & ACCENT_SENSITIVITY_MASK;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ int localeOrdinal = (collationId >> LOCALE_OFFSET) & LOCALE_MASK;
+ collationId ^= ImplementationProvider.ICU.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+ collationId ^= caseSensitivityOrdinal << CASE_SENSITIVITY_OFFSET;
+ collationId ^= accentSensitivityOrdinal << ACCENT_SENSITIVITY_OFFSET;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
+ collationId ^= localeOrdinal << LOCALE_OFFSET;
+ if (collationId != 0 || caseConversionOrdinal >= CaseConversion.values().length ||
+ localeOrdinal >= ICULocaleNames.length) {
+ throw SparkException.internalError("Invalid ICU collation id " + originalCollationId);
+ } else {
+ CaseSensitivity caseSensitivity = CaseSensitivity.values()[caseSensitivityOrdinal];
+ AccentSensitivity accentSensitivity =
+ AccentSensitivity.values()[accentSensitivityOrdinal];
+ CaseConversion caseConversion = CaseConversion.values()[caseConversionOrdinal];
+ String locale = ICULocaleNames[localeOrdinal];
+ return new CollationSpecICU(locale, caseSensitivity, accentSensitivity, caseConversion);
+ }
+ }
+
+ @Override
+ protected int getCollationId() {
+ int collationId = 0;
+ collationId |= ImplementationProvider.ICU.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+ collationId |= caseSensitivity.ordinal() << CASE_SENSITIVITY_OFFSET;
+ collationId |= accentSensitivity.ordinal() << ACCENT_SENSITIVITY_OFFSET;
+ collationId |= caseConversion.ordinal() << CASE_CONVERSION_OFFSET;
+ collationId |= ICULocaleToId.get(locale);
+ return collationId;
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ ULocale.Builder builder = new ULocale.Builder();
+ builder.setLocale(ICULocaleMap.get(locale));
+ if (caseSensitivity == CaseSensitivity.CS &&
+ accentSensitivity == AccentSensitivity.AS) {
+ builder.setUnicodeLocaleKeyword("ks", "level3");
+ } else if (caseSensitivity == CaseSensitivity.CS &&
+ accentSensitivity == AccentSensitivity.AI) {
+ builder
+ .setUnicodeLocaleKeyword("ks", "level1")
+ .setUnicodeLocaleKeyword("kc", "true");
+ } else if (caseSensitivity == CaseSensitivity.CI &&
+ accentSensitivity == AccentSensitivity.AS) {
+ builder.setUnicodeLocaleKeyword("ks", "level2");
+ } else if (caseSensitivity == CaseSensitivity.CI &&
+ accentSensitivity == AccentSensitivity.AI) {
+ builder.setUnicodeLocaleKeyword("ks", "level1");
+ }
+ ULocale resultLocale = builder.build();
+ Collator collator = Collator.getInstance(resultLocale);
+ Comparator<UTF8String> comparator = (s1, s2) -> collator.compare(
+ caseConversion(resultLocale, s1), caseConversion(resultLocale, s2));
+ collator.freeze();
+ return new Collation(
+ collationName(),
+ collator,
+ comparator,
+ "153.120.0.0",
Review Comment:
Can you move this to const?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "dbatomic (via GitHub)" <gi...@apache.org>.
dbatomic commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1592445524
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,438 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
Review Comment:
Would this be hit if user says `collate(StringType(INDETERMINATE))`? If so than this can't be internal error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1593910153
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
Review Comment:
My opinion is that it would be better to stick with this naming with following reasoning.
If we used bit 31 for indeterminate collation, we would shrink user collation space because we use additional bit to distinguish between predefined and user collations. It's more convenient to distinguish this at first bit since indeterminate collation falls into predefined space.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606723151
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected static int parseSpecifiers(
+ String originalName, String collationName, int splitStart) throws SparkException {
+ int specifiers = 0;
+ String[] parts = collationName.substring(splitStart).split("_");
+ for (String part : parts) {
+ if (!part.isEmpty()) {
+ if (part.equals("UNSPECIFIED") || part.equals("INDETERMINATE")) {
+ throw collationInvalidNameException(originalName);
+ } else if (Arrays.stream(CaseSensitivity.values()).anyMatch(
+ (s) -> s.toString().equals(part))) {
+ specifiers |=
+ CaseSensitivity.valueOf(part).ordinal() << CASE_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(AccentSensitivity.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ AccentSensitivity.valueOf(part).ordinal() << ACCENT_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(CaseConversion.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ CaseConversion.valueOf(part).ordinal() << CASE_CONVERSION_OFFSET;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+ }
+ return specifiers;
+ }
+
+ protected abstract int getCollationId();
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ public static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.LCASE).getCollationId();
+ public static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).buildCollation();
+
+ private CollationSpecUTF8Binary(CaseConversion caseConversion) {
+ super(null, CaseSensitivity.CS, AccentSensitivity.AS, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int collationId = 0;
+ int specifiers = CollationSpec.parseSpecifiers(
+ originalName, collationName, "UTF8_BINARY".length());
+ if ((specifiers & ~(CASE_CONVERSION_MASK << CASE_CONVERSION_OFFSET)) != 0) {
+ throw collationInvalidNameException(originalName);
+ }
+ collationId |= specifiers;
+ return collationId;
+ }
+
+ @Override
+ protected int getCollationId() {
+ int collationId = 0;
+ collationId |= caseConversion.ordinal() << CASE_CONVERSION_OFFSET;
+ return collationId;
+ }
+
+ public static CollationSpecUTF8Binary fromCollationId(int collationId)
+ throws SparkException {
+ int originalCollationId = collationId;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
+ if (collationId != 0 || caseConversionOrdinal >= CaseConversion.values().length) {
Review Comment:
Outdated.
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected static int parseSpecifiers(
+ String originalName, String collationName, int splitStart) throws SparkException {
+ int specifiers = 0;
+ String[] parts = collationName.substring(splitStart).split("_");
+ for (String part : parts) {
+ if (!part.isEmpty()) {
+ if (part.equals("UNSPECIFIED") || part.equals("INDETERMINATE")) {
+ throw collationInvalidNameException(originalName);
+ } else if (Arrays.stream(CaseSensitivity.values()).anyMatch(
+ (s) -> s.toString().equals(part))) {
+ specifiers |=
+ CaseSensitivity.valueOf(part).ordinal() << CASE_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(AccentSensitivity.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ AccentSensitivity.valueOf(part).ordinal() << ACCENT_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(CaseConversion.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ CaseConversion.valueOf(part).ordinal() << CASE_CONVERSION_OFFSET;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+ }
+ return specifiers;
+ }
+
+ protected abstract int getCollationId();
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ public static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.LCASE).getCollationId();
+ public static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).buildCollation();
+
+ private CollationSpecUTF8Binary(CaseConversion caseConversion) {
+ super(null, CaseSensitivity.CS, AccentSensitivity.AS, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int collationId = 0;
+ int specifiers = CollationSpec.parseSpecifiers(
+ originalName, collationName, "UTF8_BINARY".length());
+ if ((specifiers & ~(CASE_CONVERSION_MASK << CASE_CONVERSION_OFFSET)) != 0) {
+ throw collationInvalidNameException(originalName);
+ }
+ collationId |= specifiers;
+ return collationId;
+ }
+
+ @Override
+ protected int getCollationId() {
+ int collationId = 0;
+ collationId |= caseConversion.ordinal() << CASE_CONVERSION_OFFSET;
+ return collationId;
+ }
+
+ public static CollationSpecUTF8Binary fromCollationId(int collationId)
+ throws SparkException {
+ int originalCollationId = collationId;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
+ if (collationId != 0 || caseConversionOrdinal >= CaseConversion.values().length) {
+ throw SparkException.internalError("Invalid UTF8_BINARY collation id " +
+ originalCollationId);
+ } else {
+ CaseConversion caseConversion = CaseConversion.values()[caseConversionOrdinal];
+ return new CollationSpecUTF8Binary(caseConversion);
+ }
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ Comparator<UTF8String> comparator;
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ comparator = UTF8String::binaryCompare;
+ } else if (collationId == UTF8_BINARY_LCASE_COLLATION_ID) {
+ comparator = UTF8String::compareLowerCase;
+ } else {
+ comparator = (s1, s2) -> {
+ UTF8String convertedS1 = caseConversion(s1);
+ UTF8String convertedS2 = caseConversion(s2);
+ return convertedS1.binaryCompare(convertedS2);
+ };
+ }
+ return new Collation(
+ collationName(),
+ null,
+ comparator,
+ "1.0",
+ s -> (long) caseConversion(s).hashCode(),
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_LCASE_COLLATION_ID
+ );
+ }
+
+ private UTF8String caseConversion(UTF8String s) {
+ if (caseConversion == CaseConversion.LCASE) {
+ return s.toLowerCase();
+ } else if (caseConversion == CaseConversion.UCASE) {
+ return s.toUpperCase();
+ } else {
+ return s;
+ }
+ }
+
+ private String collationName() {
+ StringBuilder builder = new StringBuilder();
+ builder.append("UTF8_BINARY");
+ if (caseConversion != CaseConversion.UNSPECIFIED) {
+ builder.append('_');
+ builder.append(caseConversion.toString());
+ }
+ return builder.toString();
+ }
+ }
+
+ private static class CollationSpecICU extends CollationSpec {
+
+ private static final String[] ICULocaleNames;
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+ private static final String ICUCollatorVersion = "153.120.0.0";
Review Comment:
Added comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606746445
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -772,12 +772,17 @@ object SQLConf {
" produced by a builtin function such as to_char or CAST")
.version("4.0.0")
.stringConf
- .checkValue(CollationFactory.isValidCollation,
+ .checkValue(
+ collationName => {
+ try {
+ CollationFactory.fetchCollation(collationName)
+ true
+ } catch {
+ case e: SparkException if e.getErrorClass == "COLLATION_INVALID_NAME" => false
+ }
+ },
Review Comment:
Resolving the discussion, we removed `isValidCollation` from `CollationFactory` and chose to scope it to only here as it's logic of catching exception as part of control flow is not a good approach for general-purpose utility function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #46180:
URL: https://github.com/apache/spark/pull/46180#issuecomment-2124611128
do we have end-to-end tests for this new feature?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1609915373
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -173,26 +174,546 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * Collation ID is defined as 32-bit integer. We specify binary layouts for different classes of
+ * collations. Classes of collations are differentiated by most significant 3 bits (bit 31, 30
+ * and 29), bit 31 being most significant and bit 0 being least significant.
+ * ---
+ * General collation ID binary layout:
+ * bit 31: 1 for INDETERMINATE (requires all other bits to be 1 as well), 0 otherwise.
+ * bit 30: 0 for predefined, 1 for user-defined.
+ * Following bits are specified for predefined collations:
+ * bit 29: 0 for UTF8_BINARY, 1 for ICU collations.
+ * bit 28-24: Reserved.
+ * bit 23-22: Reserved for version.
+ * bit 21-18: Reserved for space trimming.
+ * bit 17-0: Depend on collation family.
+ * ---
+ * INDETERMINATE collation ID binary layout:
+ * bit 31-0: 1
+ * INDETERMINATE collation ID is equal to -1.
+ * ---
+ * User-defined collation ID binary layout:
+ * bit 31: 0
+ * bit 30: 1
+ * bit 29-0: Undefined, reserved for future use.
+ * ---
+ * UTF8_BINARY collation ID binary layout:
+ * bit 31-24: Zeroes.
+ * bit 23-22: Zeroes, reserved for version.
+ * bit 21-18: Zeroes, reserved for space trimming.
+ * bit 17-3: Zeroes.
+ * bit 2: 0, reserved for accent sensitivity.
+ * bit 1: 0, reserved for uppercase and case-insensitive.
+ * bit 0: 0 = case-sensitive, 1 = lowercase.
+ * ---
+ * ICU collation ID binary layout:
Review Comment:
We did design binary layout. There are several reasons for having accent and case sensitivity bits in different positions:
- case-sensitivity doesn't have same meaning in `UTF8_BINARY` and ICU collations; for ICU we only support `CS`/`CI` while in `UTF8_BINARY` we only have unspecified/`LCASE` for now, while `UCASE` and `CI` are reserved for future use
- in order to preserve collation ids we're using before these changes for `UTF8_BINARY` (0) and `UTF8_BINARY_LCASE` (1) we needed to set aside the least significant bit for case-sensitivity while for ICU collations it makes more sense to use least significant bits for locale id
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1609808967
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -173,26 +174,546 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * Collation ID is defined as 32-bit integer. We specify binary layouts for different classes of
+ * collations. Classes of collations are differentiated by most significant 3 bits (bit 31, 30
+ * and 29), bit 31 being most significant and bit 0 being least significant.
+ * ---
+ * General collation ID binary layout:
+ * bit 31: 1 for INDETERMINATE (requires all other bits to be 1 as well), 0 otherwise.
+ * bit 30: 0 for predefined, 1 for user-defined.
+ * Following bits are specified for predefined collations:
+ * bit 29: 0 for UTF8_BINARY, 1 for ICU collations.
+ * bit 28-24: Reserved.
+ * bit 23-22: Reserved for version.
+ * bit 21-18: Reserved for space trimming.
+ * bit 17-0: Depend on collation family.
+ * ---
+ * INDETERMINATE collation ID binary layout:
+ * bit 31-0: 1
+ * INDETERMINATE collation ID is equal to -1.
+ * ---
+ * User-defined collation ID binary layout:
+ * bit 31: 0
+ * bit 30: 1
+ * bit 29-0: Undefined, reserved for future use.
+ * ---
+ * UTF8_BINARY collation ID binary layout:
+ * bit 31-24: Zeroes.
+ * bit 23-22: Zeroes, reserved for version.
+ * bit 21-18: Zeroes, reserved for space trimming.
+ * bit 17-3: Zeroes.
+ * bit 2: 0, reserved for accent sensitivity.
+ * bit 1: 0, reserved for uppercase and case-insensitive.
+ * bit 0: 0 = case-sensitive, 1 = lowercase.
+ * ---
+ * ICU collation ID binary layout:
Review Comment:
Do we design the binary layout by ourselves? It looks a bit weird that we put case-sensitivity and accent-sensitivity bits in different places between `UTF8_BINARY` and other collations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1592489108
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,438 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
Review Comment:
I don't think user can specify `INDETERMINATE` collation name explicitly. This function can only be called internally with `collationId` integer parameter which is not exposed to user; clients of this function are other functions which parse string name first to integer collation ID and then pass it around.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1593127364
##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -30,31 +33,101 @@ import org.apache.spark.unsafe.types.UTF8String.{fromString => toUTF8}
class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ignore funsuite
test("collationId stability") {
- val utf8Binary = fetchCollation(0)
+ assert(UTF8_BINARY_COLLATION_ID == 0)
+ assert(INDETERMINATE_COLLATION_ID == 1 << 30)
+
+ val utf8Binary = fetchCollation(UTF8_BINARY_COLLATION_ID)
assert(utf8Binary.collationName == "UTF8_BINARY")
assert(utf8Binary.supportsBinaryEquality)
- val utf8BinaryLcase = fetchCollation(1)
+ val utf8BinaryLcase = fetchCollation(UTF8_BINARY_LCASE_COLLATION_ID)
assert(utf8BinaryLcase.collationName == "UTF8_BINARY_LCASE")
assert(!utf8BinaryLcase.supportsBinaryEquality)
- val unicode = fetchCollation(2)
+ val unicode = fetchCollation(UNICODE_COLLATION_ID)
assert(unicode.collationName == "UNICODE")
- assert(unicode.supportsBinaryEquality);
+ assert(unicode.supportsBinaryEquality)
- val unicodeCi = fetchCollation(3)
+ val unicodeCi = fetchCollation(UNICODE_CI_COLLATION_ID)
assert(unicodeCi.collationName == "UNICODE_CI")
assert(!unicodeCi.supportsBinaryEquality)
}
- test("fetch invalid collation name") {
- val error = intercept[SparkException] {
- fetchCollation("UTF8_BS")
+ test("UTF8_BINARY and ICU root locale collation names") {
+ // collation name already normalized
+ Seq(
+ "UTF8_BINARY",
+ "UTF8_BINARY_LCASE",
+ "UTF8_BINARY_UCASE",
+ "UNICODE",
+ "UNICODE_CI",
+ "UNICODE_AI",
+ "UNICODE_CI_AI",
+ "UNICODE_LCASE",
+ "UNICODE_UCASE"
+ ).foreach(collationName => {
+ val col = fetchCollation(collationName)
+ assert(col.collationName == collationName)
+ })
+ // collation name normalization
+ Seq(
+ // UTF8_BINARY
+ ("UTF8_BINARY_CS", "UTF8_BINARY"),
Review Comment:
Reworked the approach to reflect this. `_AI` and `_CI` will be disabled but `_CS` and `_AS` will be treated as default case and accent sensitive collations with no effect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "mkaravel (via GitHub)" <gi...@apache.org>.
mkaravel commented on PR #46180:
URL: https://github.com/apache/spark/pull/46180#issuecomment-2101717141
How do we name a trailing-space-insensitive collation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "mkaravel (via GitHub)" <gi...@apache.org>.
mkaravel commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1594835020
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,438 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ public static int collationNameToId(String originalCollationName) throws SparkException {
+ String collationName = originalCollationName.toUpperCase();
+ try {
+ if (collationName.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName);
+ }
+ } catch (SparkException e) {
+ throw new SparkException(
+ "COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", originalCollationName)),
+ e);
+ }
+ }
+
+ protected static int parseSpecifiers(String specString) throws SparkException {
+ int specifiers = 0;
+ String[] parts = specString.split("_");
+ for (String part : parts) {
+ if (!part.isEmpty()) {
+ if (part.equals("UNSPECIFIED") || part.equals("INDETERMINATE")) {
+ throw new SparkException(part + " collation specifier reserved for internal use");
Review Comment:
+1 on this. Please create an error class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1592505295
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,438 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
Review Comment:
Actually I'm not sure about this, indeterminate collation id may be propagate to places like https://github.com/apache/spark/blob/b20356ef55a230fe00d0463757fb327aefe5ccff/sql/api/src/main/scala/org/apache/spark/sql/types/StringType.scala#L38
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1593900492
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected static int parseSpecifiers(
+ String originalName, String collationName, int splitStart) throws SparkException {
+ int specifiers = 0;
+ String[] parts = collationName.substring(splitStart).split("_");
+ for (String part : parts) {
+ if (!part.isEmpty()) {
+ if (part.equals("UNSPECIFIED") || part.equals("INDETERMINATE")) {
+ throw collationInvalidNameException(originalName);
+ } else if (Arrays.stream(CaseSensitivity.values()).anyMatch(
+ (s) -> s.toString().equals(part))) {
+ specifiers |=
+ CaseSensitivity.valueOf(part).ordinal() << CASE_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(AccentSensitivity.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ AccentSensitivity.valueOf(part).ordinal() << ACCENT_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(CaseConversion.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ CaseConversion.valueOf(part).ordinal() << CASE_CONVERSION_OFFSET;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+ }
+ return specifiers;
+ }
+
+ protected abstract int getCollationId();
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ public static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.LCASE).getCollationId();
+ public static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).buildCollation();
+
+ private CollationSpecUTF8Binary(CaseConversion caseConversion) {
+ super(null, CaseSensitivity.CS, AccentSensitivity.AS, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int collationId = 0;
+ int specifiers = CollationSpec.parseSpecifiers(
+ originalName, collationName, "UTF8_BINARY".length());
+ if ((specifiers & ~(CASE_CONVERSION_MASK << CASE_CONVERSION_OFFSET)) != 0) {
+ throw collationInvalidNameException(originalName);
+ }
+ collationId |= specifiers;
+ return collationId;
+ }
+
+ @Override
+ protected int getCollationId() {
+ int collationId = 0;
+ collationId |= caseConversion.ordinal() << CASE_CONVERSION_OFFSET;
+ return collationId;
+ }
+
+ public static CollationSpecUTF8Binary fromCollationId(int collationId)
+ throws SparkException {
+ int originalCollationId = collationId;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
+ if (collationId != 0 || caseConversionOrdinal >= CaseConversion.values().length) {
+ throw SparkException.internalError("Invalid UTF8_BINARY collation id " +
+ originalCollationId);
+ } else {
+ CaseConversion caseConversion = CaseConversion.values()[caseConversionOrdinal];
+ return new CollationSpecUTF8Binary(caseConversion);
+ }
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ Comparator<UTF8String> comparator;
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ comparator = UTF8String::binaryCompare;
+ } else if (collationId == UTF8_BINARY_LCASE_COLLATION_ID) {
+ comparator = UTF8String::compareLowerCase;
+ } else {
+ comparator = (s1, s2) -> {
+ UTF8String convertedS1 = caseConversion(s1);
+ UTF8String convertedS2 = caseConversion(s2);
+ return convertedS1.binaryCompare(convertedS2);
+ };
+ }
+ return new Collation(
+ collationName(),
+ null,
+ comparator,
+ "1.0",
+ s -> (long) caseConversion(s).hashCode(),
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_LCASE_COLLATION_ID
+ );
+ }
+
+ private UTF8String caseConversion(UTF8String s) {
+ if (caseConversion == CaseConversion.LCASE) {
+ return s.toLowerCase();
+ } else if (caseConversion == CaseConversion.UCASE) {
+ return s.toUpperCase();
+ } else {
+ return s;
+ }
+ }
+
+ private String collationName() {
+ StringBuilder builder = new StringBuilder();
+ builder.append("UTF8_BINARY");
+ if (caseConversion != CaseConversion.UNSPECIFIED) {
+ builder.append('_');
+ builder.append(caseConversion.toString());
+ }
+ return builder.toString();
+ }
+ }
+
+ private static class CollationSpecICU extends CollationSpec {
+
+ private static final String[] ICULocaleNames;
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+ private static final String ICUCollatorVersion = "153.120.0.0";
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
+ for (ULocale locale : locales) {
+ if (locale.getVariant().isEmpty()) {
+ String language = locale.getLanguage();
+ assert (!language.isEmpty());
+ StringBuilder builder = new StringBuilder(language);
+ String script = locale.getScript();
+ if (!script.isEmpty()) {
+ builder.append('_');
+ builder.append(script);
+ }
+ String country = locale.getISO3Country();
+ if (!country.isEmpty()) {
+ builder.append('_');
+ builder.append(country);
+ }
+ String localeName = builder.toString();
+ assert (!ICULocaleMap.containsKey(localeName));
+ ICULocaleMap.put(localeName, locale);
+ }
+ }
+ for (String localeName : ICULocaleMap.keySet()) {
+ String localeUppercase = localeName.toUpperCase();
+ assert (!ICULocaleMapUppercase.containsKey(localeUppercase));
+ ICULocaleMapUppercase.put(localeUppercase, localeName);
+ }
+ ICULocaleNames = ICULocaleMap.keySet().toArray(new String[0]);
+ Arrays.sort(ICULocaleNames);
+ assert (ICULocaleNames.length <= (1 << 16));
+ for (int i = 0; i < ICULocaleNames.length; i++) {
+ ICULocaleToId.put(ICULocaleNames[i], i);
+ }
+ }
+
+ public static final int UNICODE_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CS, AccentSensitivity.AS,
+ CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UNICODE_CI_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CI, AccentSensitivity.AS,
+ CaseConversion.UNSPECIFIED).getCollationId();
+
+ private CollationSpecICU(String locale, CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity, CaseConversion caseConversion) {
+ super(locale, caseSensitivity, accentSensitivity, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int lastPos = -1;
+ for (int i = 1; i <= collationName.length(); i++) {
+ String localeName = collationName.substring(0, i);
+ if (ICULocaleMapUppercase.containsKey(localeName)) {
+ lastPos = i;
+ }
+ }
+ if (lastPos == -1) {
+ throw collationInvalidNameException(originalName);
+ } else {
+ int collationId = 0;
+ collationId |= ImplementationProvider.ICU.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+ collationId |= CollationSpec.parseSpecifiers(originalName, collationName, lastPos);
+ String normalizedLocaleName = ICULocaleMapUppercase.get(
+ collationName.substring(0, lastPos));
+ collationId |= ICULocaleToId.get(normalizedLocaleName) << LOCALE_OFFSET;
+ return collationId;
+ }
+ }
+
+ public static CollationSpecICU fromCollationId(int collationId) throws SparkException {
+ int originalCollationId = collationId;
+ int caseSensitivityOrdinal =
+ (collationId >> CASE_SENSITIVITY_OFFSET) & CASE_SENSITIVITY_MASK;
+ int accentSensitivityOrdinal =
+ (collationId >> ACCENT_SENSITIVITY_OFFSET) & ACCENT_SENSITIVITY_MASK;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ int localeOrdinal = (collationId >> LOCALE_OFFSET) & LOCALE_MASK;
+ collationId ^= ImplementationProvider.ICU.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+ collationId ^= caseSensitivityOrdinal << CASE_SENSITIVITY_OFFSET;
+ collationId ^= accentSensitivityOrdinal << ACCENT_SENSITIVITY_OFFSET;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
+ collationId ^= localeOrdinal << LOCALE_OFFSET;
+ if (collationId != 0 || caseConversionOrdinal >= CaseConversion.values().length ||
+ localeOrdinal >= ICULocaleNames.length) {
+ throw SparkException.internalError("Invalid ICU collation id " + originalCollationId);
+ } else {
+ CaseSensitivity caseSensitivity = CaseSensitivity.values()[caseSensitivityOrdinal];
+ AccentSensitivity accentSensitivity =
+ AccentSensitivity.values()[accentSensitivityOrdinal];
+ CaseConversion caseConversion = CaseConversion.values()[caseConversionOrdinal];
+ String locale = ICULocaleNames[localeOrdinal];
+ return new CollationSpecICU(locale, caseSensitivity, accentSensitivity, caseConversion);
+ }
+ }
+
+ @Override
+ protected int getCollationId() {
+ int collationId = 0;
+ collationId |= ImplementationProvider.ICU.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+ collationId |= caseSensitivity.ordinal() << CASE_SENSITIVITY_OFFSET;
+ collationId |= accentSensitivity.ordinal() << ACCENT_SENSITIVITY_OFFSET;
+ collationId |= caseConversion.ordinal() << CASE_CONVERSION_OFFSET;
+ collationId |= ICULocaleToId.get(locale);
+ return collationId;
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ ULocale.Builder builder = new ULocale.Builder();
+ builder.setLocale(ICULocaleMap.get(locale));
+ if (caseSensitivity == CaseSensitivity.CS &&
+ accentSensitivity == AccentSensitivity.AS) {
+ builder.setUnicodeLocaleKeyword("ks", "level3");
+ } else if (caseSensitivity == CaseSensitivity.CS &&
+ accentSensitivity == AccentSensitivity.AI) {
+ builder
+ .setUnicodeLocaleKeyword("ks", "level1")
+ .setUnicodeLocaleKeyword("kc", "true");
+ } else if (caseSensitivity == CaseSensitivity.CI &&
+ accentSensitivity == AccentSensitivity.AS) {
+ builder.setUnicodeLocaleKeyword("ks", "level2");
+ } else if (caseSensitivity == CaseSensitivity.CI &&
+ accentSensitivity == AccentSensitivity.AI) {
+ builder.setUnicodeLocaleKeyword("ks", "level1");
+ }
+ ULocale resultLocale = builder.build();
+ Collator collator = Collator.getInstance(resultLocale);
+ Comparator<UTF8String> comparator = (s1, s2) -> collator.compare(
+ caseConversion(resultLocale, s1), caseConversion(resultLocale, s2));
+ collator.freeze();
+ return new Collation(
+ collationName(),
+ collator,
+ comparator,
+ ICUCollatorVersion,
+ s -> (long) collator.getCollationKey(s.toString()).hashCode(),
+ collationId == UNICODE_COLLATION_ID,
Review Comment:
I'm not sure it's safe to go with this assumption in general. `CS` and `AS` combination maps to tertiary level, but there is an additional "identity" level, for which ICU [docs](https://unicode-org.github.io/icu/userguide/collation/concepts.html) specifies:
> For example, Hebrew cantillation marks are only distinguished at this level.
This is an example where strings could compare equal at tertiary level (`CS` and `AS`) but would differ in their binary representation. We shouldn't make any assumptions for a particular language.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "dbatomic (via GitHub)" <gi...@apache.org>.
dbatomic commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1604672125
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected static int parseSpecifiers(
+ String originalName, String collationName, int splitStart) throws SparkException {
+ int specifiers = 0;
+ String[] parts = collationName.substring(splitStart).split("_");
+ for (String part : parts) {
+ if (!part.isEmpty()) {
+ if (part.equals("UNSPECIFIED") || part.equals("INDETERMINATE")) {
+ throw collationInvalidNameException(originalName);
+ } else if (Arrays.stream(CaseSensitivity.values()).anyMatch(
+ (s) -> s.toString().equals(part))) {
+ specifiers |=
+ CaseSensitivity.valueOf(part).ordinal() << CASE_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(AccentSensitivity.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ AccentSensitivity.valueOf(part).ordinal() << ACCENT_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(CaseConversion.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ CaseConversion.valueOf(part).ordinal() << CASE_CONVERSION_OFFSET;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+ }
+ return specifiers;
+ }
+
+ protected abstract int getCollationId();
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ public static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.LCASE).getCollationId();
+ public static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).buildCollation();
+
+ private CollationSpecUTF8Binary(CaseConversion caseConversion) {
+ super(null, CaseSensitivity.CS, AccentSensitivity.AS, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int collationId = 0;
+ int specifiers = CollationSpec.parseSpecifiers(
+ originalName, collationName, "UTF8_BINARY".length());
+ if ((specifiers & ~(CASE_CONVERSION_MASK << CASE_CONVERSION_OFFSET)) != 0) {
+ throw collationInvalidNameException(originalName);
+ }
+ collationId |= specifiers;
+ return collationId;
+ }
+
+ @Override
+ protected int getCollationId() {
+ int collationId = 0;
+ collationId |= caseConversion.ordinal() << CASE_CONVERSION_OFFSET;
+ return collationId;
+ }
+
+ public static CollationSpecUTF8Binary fromCollationId(int collationId)
+ throws SparkException {
+ int originalCollationId = collationId;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
+ if (collationId != 0 || caseConversionOrdinal >= CaseConversion.values().length) {
+ throw SparkException.internalError("Invalid UTF8_BINARY collation id " +
+ originalCollationId);
+ } else {
+ CaseConversion caseConversion = CaseConversion.values()[caseConversionOrdinal];
+ return new CollationSpecUTF8Binary(caseConversion);
+ }
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ Comparator<UTF8String> comparator;
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ comparator = UTF8String::binaryCompare;
+ } else if (collationId == UTF8_BINARY_LCASE_COLLATION_ID) {
+ comparator = UTF8String::compareLowerCase;
+ } else {
+ comparator = (s1, s2) -> {
+ UTF8String convertedS1 = caseConversion(s1);
+ UTF8String convertedS2 = caseConversion(s2);
+ return convertedS1.binaryCompare(convertedS2);
+ };
+ }
+ return new Collation(
+ collationName(),
+ null,
+ comparator,
+ "1.0",
+ s -> (long) caseConversion(s).hashCode(),
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_LCASE_COLLATION_ID
+ );
+ }
+
+ private UTF8String caseConversion(UTF8String s) {
+ if (caseConversion == CaseConversion.LCASE) {
+ return s.toLowerCase();
+ } else if (caseConversion == CaseConversion.UCASE) {
+ return s.toUpperCase();
+ } else {
+ return s;
+ }
+ }
+
+ private String collationName() {
+ StringBuilder builder = new StringBuilder();
+ builder.append("UTF8_BINARY");
+ if (caseConversion != CaseConversion.UNSPECIFIED) {
+ builder.append('_');
+ builder.append(caseConversion.toString());
+ }
+ return builder.toString();
+ }
+ }
+
+ private static class CollationSpecICU extends CollationSpec {
+
+ private static final String[] ICULocaleNames;
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+ private static final String ICUCollatorVersion = "153.120.0.0";
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
+ for (ULocale locale : locales) {
+ if (locale.getVariant().isEmpty()) {
+ String language = locale.getLanguage();
+ assert (!language.isEmpty());
+ StringBuilder builder = new StringBuilder(language);
+ String script = locale.getScript();
+ if (!script.isEmpty()) {
+ builder.append('_');
+ builder.append(script);
+ }
+ String country = locale.getISO3Country();
+ if (!country.isEmpty()) {
+ builder.append('_');
+ builder.append(country);
+ }
+ String localeName = builder.toString();
+ assert (!ICULocaleMap.containsKey(localeName));
+ ICULocaleMap.put(localeName, locale);
+ }
+ }
+ for (String localeName : ICULocaleMap.keySet()) {
+ String localeUppercase = localeName.toUpperCase();
+ assert (!ICULocaleMapUppercase.containsKey(localeUppercase));
+ ICULocaleMapUppercase.put(localeUppercase, localeName);
+ }
+ ICULocaleNames = ICULocaleMap.keySet().toArray(new String[0]);
+ Arrays.sort(ICULocaleNames);
+ assert (ICULocaleNames.length <= (1 << 16));
+ for (int i = 0; i < ICULocaleNames.length; i++) {
+ ICULocaleToId.put(ICULocaleNames[i], i);
+ }
+ }
+
+ public static final int UNICODE_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CS, AccentSensitivity.AS,
+ CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UNICODE_CI_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CI, AccentSensitivity.AS,
+ CaseConversion.UNSPECIFIED).getCollationId();
+
+ private CollationSpecICU(String locale, CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity, CaseConversion caseConversion) {
+ super(locale, caseSensitivity, accentSensitivity, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int lastPos = -1;
+ for (int i = 1; i <= collationName.length(); i++) {
+ String localeName = collationName.substring(0, i);
+ if (ICULocaleMapUppercase.containsKey(localeName)) {
+ lastPos = i;
+ }
+ }
+ if (lastPos == -1) {
+ throw collationInvalidNameException(originalName);
+ } else {
+ int collationId = 0;
+ collationId |= ImplementationProvider.ICU.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+ collationId |= CollationSpec.parseSpecifiers(originalName, collationName, lastPos);
+ String normalizedLocaleName = ICULocaleMapUppercase.get(
+ collationName.substring(0, lastPos));
+ collationId |= ICULocaleToId.get(normalizedLocaleName) << LOCALE_OFFSET;
+ return collationId;
+ }
+ }
+
+ public static CollationSpecICU fromCollationId(int collationId) throws SparkException {
+ int originalCollationId = collationId;
+ int caseSensitivityOrdinal =
+ (collationId >> CASE_SENSITIVITY_OFFSET) & CASE_SENSITIVITY_MASK;
+ int accentSensitivityOrdinal =
+ (collationId >> ACCENT_SENSITIVITY_OFFSET) & ACCENT_SENSITIVITY_MASK;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ int localeOrdinal = (collationId >> LOCALE_OFFSET) & LOCALE_MASK;
+ collationId ^= ImplementationProvider.ICU.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+ collationId ^= caseSensitivityOrdinal << CASE_SENSITIVITY_OFFSET;
+ collationId ^= accentSensitivityOrdinal << ACCENT_SENSITIVITY_OFFSET;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
+ collationId ^= localeOrdinal << LOCALE_OFFSET;
+ if (collationId != 0 || caseConversionOrdinal >= CaseConversion.values().length ||
+ localeOrdinal >= ICULocaleNames.length) {
+ throw SparkException.internalError("Invalid ICU collation id " + originalCollationId);
+ } else {
+ CaseSensitivity caseSensitivity = CaseSensitivity.values()[caseSensitivityOrdinal];
+ AccentSensitivity accentSensitivity =
+ AccentSensitivity.values()[accentSensitivityOrdinal];
+ CaseConversion caseConversion = CaseConversion.values()[caseConversionOrdinal];
+ String locale = ICULocaleNames[localeOrdinal];
+ return new CollationSpecICU(locale, caseSensitivity, accentSensitivity, caseConversion);
+ }
+ }
+
+ @Override
+ protected int getCollationId() {
+ int collationId = 0;
+ collationId |= ImplementationProvider.ICU.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+ collationId |= caseSensitivity.ordinal() << CASE_SENSITIVITY_OFFSET;
+ collationId |= accentSensitivity.ordinal() << ACCENT_SENSITIVITY_OFFSET;
+ collationId |= caseConversion.ordinal() << CASE_CONVERSION_OFFSET;
+ collationId |= ICULocaleToId.get(locale);
+ return collationId;
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ ULocale.Builder builder = new ULocale.Builder();
+ builder.setLocale(ICULocaleMap.get(locale));
+ if (caseSensitivity == CaseSensitivity.CS &&
+ accentSensitivity == AccentSensitivity.AS) {
+ builder.setUnicodeLocaleKeyword("ks", "level3");
+ } else if (caseSensitivity == CaseSensitivity.CS &&
+ accentSensitivity == AccentSensitivity.AI) {
+ builder
+ .setUnicodeLocaleKeyword("ks", "level1")
+ .setUnicodeLocaleKeyword("kc", "true");
+ } else if (caseSensitivity == CaseSensitivity.CI &&
+ accentSensitivity == AccentSensitivity.AS) {
+ builder.setUnicodeLocaleKeyword("ks", "level2");
+ } else if (caseSensitivity == CaseSensitivity.CI &&
+ accentSensitivity == AccentSensitivity.AI) {
+ builder.setUnicodeLocaleKeyword("ks", "level1");
+ }
+ ULocale resultLocale = builder.build();
+ Collator collator = Collator.getInstance(resultLocale);
+ Comparator<UTF8String> comparator = (s1, s2) -> collator.compare(
+ caseConversion(resultLocale, s1), caseConversion(resultLocale, s2));
+ collator.freeze();
+ return new Collation(
+ collationName(),
+ collator,
+ comparator,
+ ICUCollatorVersion,
+ s -> (long) collator.getCollationKey(s.toString()).hashCode(),
+ collationId == UNICODE_COLLATION_ID,
Review Comment:
Ok, let's go them being non-binary. In future, we may want to add _BIN suffix that will indicate that we are doing binary comparisons (which is going to be faster).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606732106
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected static int parseSpecifiers(
+ String originalName, String collationName, int splitStart) throws SparkException {
+ int specifiers = 0;
+ String[] parts = collationName.substring(splitStart).split("_");
+ for (String part : parts) {
+ if (!part.isEmpty()) {
+ if (part.equals("UNSPECIFIED") || part.equals("INDETERMINATE")) {
+ throw collationInvalidNameException(originalName);
+ } else if (Arrays.stream(CaseSensitivity.values()).anyMatch(
+ (s) -> s.toString().equals(part))) {
+ specifiers |=
+ CaseSensitivity.valueOf(part).ordinal() << CASE_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(AccentSensitivity.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ AccentSensitivity.valueOf(part).ordinal() << ACCENT_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(CaseConversion.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ CaseConversion.valueOf(part).ordinal() << CASE_CONVERSION_OFFSET;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+ }
+ return specifiers;
+ }
+
+ protected abstract int getCollationId();
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ public static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.LCASE).getCollationId();
+ public static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).buildCollation();
+
+ private CollationSpecUTF8Binary(CaseConversion caseConversion) {
+ super(null, CaseSensitivity.CS, AccentSensitivity.AS, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int collationId = 0;
+ int specifiers = CollationSpec.parseSpecifiers(
+ originalName, collationName, "UTF8_BINARY".length());
+ if ((specifiers & ~(CASE_CONVERSION_MASK << CASE_CONVERSION_OFFSET)) != 0) {
+ throw collationInvalidNameException(originalName);
+ }
+ collationId |= specifiers;
+ return collationId;
+ }
+
+ @Override
+ protected int getCollationId() {
+ int collationId = 0;
+ collationId |= caseConversion.ordinal() << CASE_CONVERSION_OFFSET;
+ return collationId;
+ }
+
+ public static CollationSpecUTF8Binary fromCollationId(int collationId)
+ throws SparkException {
+ int originalCollationId = collationId;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
+ if (collationId != 0 || caseConversionOrdinal >= CaseConversion.values().length) {
+ throw SparkException.internalError("Invalid UTF8_BINARY collation id " +
+ originalCollationId);
+ } else {
+ CaseConversion caseConversion = CaseConversion.values()[caseConversionOrdinal];
+ return new CollationSpecUTF8Binary(caseConversion);
+ }
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ Comparator<UTF8String> comparator;
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ comparator = UTF8String::binaryCompare;
+ } else if (collationId == UTF8_BINARY_LCASE_COLLATION_ID) {
+ comparator = UTF8String::compareLowerCase;
+ } else {
+ comparator = (s1, s2) -> {
+ UTF8String convertedS1 = caseConversion(s1);
+ UTF8String convertedS2 = caseConversion(s2);
+ return convertedS1.binaryCompare(convertedS2);
+ };
+ }
+ return new Collation(
+ collationName(),
+ null,
+ comparator,
+ "1.0",
+ s -> (long) caseConversion(s).hashCode(),
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_LCASE_COLLATION_ID
+ );
+ }
+
+ private UTF8String caseConversion(UTF8String s) {
+ if (caseConversion == CaseConversion.LCASE) {
+ return s.toLowerCase();
+ } else if (caseConversion == CaseConversion.UCASE) {
+ return s.toUpperCase();
+ } else {
+ return s;
+ }
+ }
+
+ private String collationName() {
+ StringBuilder builder = new StringBuilder();
+ builder.append("UTF8_BINARY");
+ if (caseConversion != CaseConversion.UNSPECIFIED) {
+ builder.append('_');
+ builder.append(caseConversion.toString());
+ }
+ return builder.toString();
+ }
+ }
+
+ private static class CollationSpecICU extends CollationSpec {
+
+ private static final String[] ICULocaleNames;
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+ private static final String ICUCollatorVersion = "153.120.0.0";
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
+ for (ULocale locale : locales) {
+ if (locale.getVariant().isEmpty()) {
+ String language = locale.getLanguage();
+ assert (!language.isEmpty());
+ StringBuilder builder = new StringBuilder(language);
+ String script = locale.getScript();
+ if (!script.isEmpty()) {
+ builder.append('_');
+ builder.append(script);
+ }
+ String country = locale.getISO3Country();
+ if (!country.isEmpty()) {
+ builder.append('_');
+ builder.append(country);
+ }
+ String localeName = builder.toString();
+ assert (!ICULocaleMap.containsKey(localeName));
+ ICULocaleMap.put(localeName, locale);
+ }
+ }
+ for (String localeName : ICULocaleMap.keySet()) {
+ String localeUppercase = localeName.toUpperCase();
+ assert (!ICULocaleMapUppercase.containsKey(localeUppercase));
+ ICULocaleMapUppercase.put(localeUppercase, localeName);
+ }
+ ICULocaleNames = ICULocaleMap.keySet().toArray(new String[0]);
+ Arrays.sort(ICULocaleNames);
+ assert (ICULocaleNames.length <= (1 << 16));
+ for (int i = 0; i < ICULocaleNames.length; i++) {
+ ICULocaleToId.put(ICULocaleNames[i], i);
+ }
+ }
+
+ public static final int UNICODE_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CS, AccentSensitivity.AS,
+ CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UNICODE_CI_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CI, AccentSensitivity.AS,
+ CaseConversion.UNSPECIFIED).getCollationId();
+
+ private CollationSpecICU(String locale, CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity, CaseConversion caseConversion) {
+ super(locale, caseSensitivity, accentSensitivity, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int lastPos = -1;
+ for (int i = 1; i <= collationName.length(); i++) {
+ String localeName = collationName.substring(0, i);
+ if (ICULocaleMapUppercase.containsKey(localeName)) {
+ lastPos = i;
+ }
Review Comment:
Resolving discussion, added clarification comment which explains how specifiers are not allowed to be equal to country code, resulting in property that only valid match is the longest one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606743213
##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -30,31 +33,107 @@ import org.apache.spark.unsafe.types.UTF8String.{fromString => toUTF8}
class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ignore funsuite
test("collationId stability") {
- val utf8Binary = fetchCollation(0)
+ assert(UTF8_BINARY_COLLATION_ID == 0)
+ assert(INDETERMINATE_COLLATION_ID == 1 << 30)
+
+ val utf8Binary = fetchCollation(UTF8_BINARY_COLLATION_ID)
assert(utf8Binary.collationName == "UTF8_BINARY")
assert(utf8Binary.supportsBinaryEquality)
- val utf8BinaryLcase = fetchCollation(1)
+ val utf8BinaryLcase = fetchCollation(UTF8_BINARY_LCASE_COLLATION_ID)
assert(utf8BinaryLcase.collationName == "UTF8_BINARY_LCASE")
assert(!utf8BinaryLcase.supportsBinaryEquality)
- val unicode = fetchCollation(2)
+ val unicode = fetchCollation(UNICODE_COLLATION_ID)
assert(unicode.collationName == "UNICODE")
- assert(unicode.supportsBinaryEquality);
+ assert(unicode.supportsBinaryEquality)
- val unicodeCi = fetchCollation(3)
+ val unicodeCi = fetchCollation(UNICODE_CI_COLLATION_ID)
assert(unicodeCi.collationName == "UNICODE_CI")
assert(!unicodeCi.supportsBinaryEquality)
}
- test("fetch invalid collation name") {
- val error = intercept[SparkException] {
- fetchCollation("UTF8_BS")
+ test("UTF8_BINARY and ICU root locale collation names") {
+ // collation name already normalized
+ Seq(
+ "UTF8_BINARY",
+ "UTF8_BINARY_LCASE",
+ "UTF8_BINARY_UCASE",
+ "UNICODE",
+ "UNICODE_CI",
+ "UNICODE_AI",
+ "UNICODE_CI_AI",
+ "UNICODE_LCASE",
+ "UNICODE_UCASE"
+ ).foreach(collationName => {
+ val col = fetchCollation(collationName)
+ assert(col.collationName == collationName)
+ })
+ // collation name normalization
+ Seq(
+ // UTF8_BINARY
+ ("UTF8_BINARY_CS", "UTF8_BINARY"),
+ ("UTF8_BINARY_AS", "UTF8_BINARY"),
+ ("UTF8_BINARY_CS_AS", "UTF8_BINARY"),
+ ("UTF8_BINARY_AS_CS", "UTF8_BINARY"),
+ ("UTF8_BINARY_LCASE_CS", "UTF8_BINARY_LCASE"),
+ ("UTF8_BINARY_CS_LCASE", "UTF8_BINARY_LCASE"),
+ ("UTF8_BINARY_LCASE_AS", "UTF8_BINARY_LCASE"),
+ ("UTF8_BINARY_AS_LCASE", "UTF8_BINARY_LCASE"),
+ ("UTF8_BINARY_CS_LCASE_AS", "UTF8_BINARY_LCASE"),
+ ("UTF8_BINARY_AS_CS_LCASE", "UTF8_BINARY_LCASE"),
+ // ICU root locale
+ ("UNICODE_CS", "UNICODE"),
+ ("UNICODE_CS_AS", "UNICODE"),
+ ("UNICODE_CI_AS", "UNICODE_CI"),
+ ("UNICODE_AI_CS", "UNICODE_AI"),
+ ("UNICODE_AI_CI", "UNICODE_CI_AI"),
+ ("UNICODE_LCASE_CS", "UNICODE_LCASE"),
+ ("UNICODE_LCASE_AS", "UNICODE_LCASE"),
+ ("UNICODE_LCASE_CI", "UNICODE_CI_LCASE"),
+ ("UNICODE_UCASE_CI", "UNICODE_CI_UCASE"),
+ ("UNICODE_AI_UCASE_CI", "UNICODE_CI_AI_UCASE"),
Review Comment:
Outdated, these are invalid collations now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606739234
##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -30,31 +33,107 @@ import org.apache.spark.unsafe.types.UTF8String.{fromString => toUTF8}
class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ignore funsuite
test("collationId stability") {
- val utf8Binary = fetchCollation(0)
+ assert(UTF8_BINARY_COLLATION_ID == 0)
+ assert(INDETERMINATE_COLLATION_ID == 1 << 30)
Review Comment:
Added parentheses.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1601306785
##########
connector/connect/common/src/main/protobuf/spark/connect/types.proto:
##########
@@ -101,7 +101,7 @@ message DataType {
message String {
uint32 type_variation_reference = 1;
- uint32 collation_id = 2;
+ string collation = 2;
Review Comment:
This is protobuf collation field used by Spark Connect. We wanted to decouple knowledge about collation ids in Spark Connect client and server in order to make collation ids as internal as possible, thus we're not serializing collation id anymore but collation name instead.
This enables us to forget about collation ids in PySpark and use only collation name which can be an arbitrary string and all logic of parsing/validating will be run on Scala side, so we don't need to have duplicate code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #46180:
URL: https://github.com/apache/spark/pull/46180#issuecomment-2135717554
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1612118108
##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/common/DataTypeProtoConverter.scala:
##########
@@ -177,7 +178,11 @@ object DataTypeProtoConverter {
case s: StringType =>
proto.DataType
.newBuilder()
- .setString(proto.DataType.String.newBuilder().setCollationId(s.collationId).build())
+ .setString(
+ proto.DataType.String
+ .newBuilder()
+ .setCollation(CollationFactory.fetchCollation(s.collationId).collationName)
Review Comment:
This file is in `common` directory, so it's used by both client and server, right? If I got it correctly, the utility function `toConnectProtoType` would be used by server to serialize `StringType` with collation and send it to the PySpark client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1595124256
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected static int parseSpecifiers(
+ String originalName, String collationName, int splitStart) throws SparkException {
+ int specifiers = 0;
+ String[] parts = collationName.substring(splitStart).split("_");
+ for (String part : parts) {
+ if (!part.isEmpty()) {
+ if (part.equals("UNSPECIFIED") || part.equals("INDETERMINATE")) {
+ throw collationInvalidNameException(originalName);
+ } else if (Arrays.stream(CaseSensitivity.values()).anyMatch(
+ (s) -> s.toString().equals(part))) {
+ specifiers |=
+ CaseSensitivity.valueOf(part).ordinal() << CASE_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(AccentSensitivity.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ AccentSensitivity.valueOf(part).ordinal() << ACCENT_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(CaseConversion.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ CaseConversion.valueOf(part).ordinal() << CASE_CONVERSION_OFFSET;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+ }
+ return specifiers;
+ }
+
+ protected abstract int getCollationId();
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ public static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.LCASE).getCollationId();
+ public static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).buildCollation();
+
+ private CollationSpecUTF8Binary(CaseConversion caseConversion) {
+ super(null, CaseSensitivity.CS, AccentSensitivity.AS, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int collationId = 0;
+ int specifiers = CollationSpec.parseSpecifiers(
+ originalName, collationName, "UTF8_BINARY".length());
+ if ((specifiers & ~(CASE_CONVERSION_MASK << CASE_CONVERSION_OFFSET)) != 0) {
+ throw collationInvalidNameException(originalName);
+ }
+ collationId |= specifiers;
+ return collationId;
+ }
+
+ @Override
+ protected int getCollationId() {
+ int collationId = 0;
+ collationId |= caseConversion.ordinal() << CASE_CONVERSION_OFFSET;
+ return collationId;
+ }
+
+ public static CollationSpecUTF8Binary fromCollationId(int collationId)
+ throws SparkException {
+ int originalCollationId = collationId;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
+ if (collationId != 0 || caseConversionOrdinal >= CaseConversion.values().length) {
+ throw SparkException.internalError("Invalid UTF8_BINARY collation id " +
+ originalCollationId);
+ } else {
+ CaseConversion caseConversion = CaseConversion.values()[caseConversionOrdinal];
+ return new CollationSpecUTF8Binary(caseConversion);
+ }
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ Comparator<UTF8String> comparator;
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ comparator = UTF8String::binaryCompare;
+ } else if (collationId == UTF8_BINARY_LCASE_COLLATION_ID) {
+ comparator = UTF8String::compareLowerCase;
+ } else {
+ comparator = (s1, s2) -> {
+ UTF8String convertedS1 = caseConversion(s1);
+ UTF8String convertedS2 = caseConversion(s2);
+ return convertedS1.binaryCompare(convertedS2);
+ };
+ }
+ return new Collation(
+ collationName(),
+ null,
+ comparator,
+ "1.0",
+ s -> (long) caseConversion(s).hashCode(),
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_LCASE_COLLATION_ID
+ );
+ }
+
+ private UTF8String caseConversion(UTF8String s) {
+ if (caseConversion == CaseConversion.LCASE) {
+ return s.toLowerCase();
+ } else if (caseConversion == CaseConversion.UCASE) {
+ return s.toUpperCase();
+ } else {
+ return s;
+ }
+ }
+
+ private String collationName() {
+ StringBuilder builder = new StringBuilder();
+ builder.append("UTF8_BINARY");
+ if (caseConversion != CaseConversion.UNSPECIFIED) {
+ builder.append('_');
+ builder.append(caseConversion.toString());
+ }
+ return builder.toString();
+ }
+ }
+
+ private static class CollationSpecICU extends CollationSpec {
+
+ private static final String[] ICULocaleNames;
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+ private static final String ICUCollatorVersion = "153.120.0.0";
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
+ for (ULocale locale : locales) {
+ if (locale.getVariant().isEmpty()) {
+ String language = locale.getLanguage();
+ assert (!language.isEmpty());
+ StringBuilder builder = new StringBuilder(language);
+ String script = locale.getScript();
+ if (!script.isEmpty()) {
+ builder.append('_');
+ builder.append(script);
+ }
+ String country = locale.getISO3Country();
+ if (!country.isEmpty()) {
+ builder.append('_');
+ builder.append(country);
+ }
+ String localeName = builder.toString();
+ assert (!ICULocaleMap.containsKey(localeName));
+ ICULocaleMap.put(localeName, locale);
+ }
+ }
+ for (String localeName : ICULocaleMap.keySet()) {
+ String localeUppercase = localeName.toUpperCase();
+ assert (!ICULocaleMapUppercase.containsKey(localeUppercase));
+ ICULocaleMapUppercase.put(localeUppercase, localeName);
+ }
+ ICULocaleNames = ICULocaleMap.keySet().toArray(new String[0]);
+ Arrays.sort(ICULocaleNames);
+ assert (ICULocaleNames.length <= (1 << 16));
+ for (int i = 0; i < ICULocaleNames.length; i++) {
+ ICULocaleToId.put(ICULocaleNames[i], i);
+ }
+ }
+
+ public static final int UNICODE_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CS, AccentSensitivity.AS,
+ CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UNICODE_CI_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CI, AccentSensitivity.AS,
+ CaseConversion.UNSPECIFIED).getCollationId();
+
+ private CollationSpecICU(String locale, CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity, CaseConversion caseConversion) {
+ super(locale, caseSensitivity, accentSensitivity, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int lastPos = -1;
+ for (int i = 1; i <= collationName.length(); i++) {
+ String localeName = collationName.substring(0, i);
+ if (ICULocaleMapUppercase.containsKey(localeName)) {
+ lastPos = i;
+ }
Review Comment:
Actually that's not true, we can have `en` and `en_USA` locales. The idea is that if we have match for multiple locales, only the longest one will work because specifiers are designed to be different from any locale part. For example, collation `en_USA_CI` can match `en` locale but `USA` is invalid specifier, thus we go with `en_USA`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1601363774
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected static int parseSpecifiers(
+ String originalName, String collationName, int splitStart) throws SparkException {
+ int specifiers = 0;
+ String[] parts = collationName.substring(splitStart).split("_");
+ for (String part : parts) {
+ if (!part.isEmpty()) {
+ if (part.equals("UNSPECIFIED") || part.equals("INDETERMINATE")) {
+ throw collationInvalidNameException(originalName);
+ } else if (Arrays.stream(CaseSensitivity.values()).anyMatch(
+ (s) -> s.toString().equals(part))) {
+ specifiers |=
+ CaseSensitivity.valueOf(part).ordinal() << CASE_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(AccentSensitivity.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ AccentSensitivity.valueOf(part).ordinal() << ACCENT_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(CaseConversion.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ CaseConversion.valueOf(part).ordinal() << CASE_CONVERSION_OFFSET;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+ }
+ return specifiers;
+ }
+
+ protected abstract int getCollationId();
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ public static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.LCASE).getCollationId();
+ public static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).buildCollation();
+
+ private CollationSpecUTF8Binary(CaseConversion caseConversion) {
+ super(null, CaseSensitivity.CS, AccentSensitivity.AS, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int collationId = 0;
+ int specifiers = CollationSpec.parseSpecifiers(
+ originalName, collationName, "UTF8_BINARY".length());
+ if ((specifiers & ~(CASE_CONVERSION_MASK << CASE_CONVERSION_OFFSET)) != 0) {
+ throw collationInvalidNameException(originalName);
+ }
+ collationId |= specifiers;
+ return collationId;
+ }
+
+ @Override
+ protected int getCollationId() {
+ int collationId = 0;
+ collationId |= caseConversion.ordinal() << CASE_CONVERSION_OFFSET;
+ return collationId;
+ }
+
+ public static CollationSpecUTF8Binary fromCollationId(int collationId)
+ throws SparkException {
+ int originalCollationId = collationId;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
+ if (collationId != 0 || caseConversionOrdinal >= CaseConversion.values().length) {
+ throw SparkException.internalError("Invalid UTF8_BINARY collation id " +
+ originalCollationId);
+ } else {
+ CaseConversion caseConversion = CaseConversion.values()[caseConversionOrdinal];
+ return new CollationSpecUTF8Binary(caseConversion);
+ }
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ Comparator<UTF8String> comparator;
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ comparator = UTF8String::binaryCompare;
+ } else if (collationId == UTF8_BINARY_LCASE_COLLATION_ID) {
+ comparator = UTF8String::compareLowerCase;
+ } else {
+ comparator = (s1, s2) -> {
+ UTF8String convertedS1 = caseConversion(s1);
+ UTF8String convertedS2 = caseConversion(s2);
+ return convertedS1.binaryCompare(convertedS2);
+ };
+ }
+ return new Collation(
+ collationName(),
+ null,
+ comparator,
+ "1.0",
+ s -> (long) caseConversion(s).hashCode(),
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_LCASE_COLLATION_ID
+ );
+ }
+
+ private UTF8String caseConversion(UTF8String s) {
+ if (caseConversion == CaseConversion.LCASE) {
+ return s.toLowerCase();
+ } else if (caseConversion == CaseConversion.UCASE) {
+ return s.toUpperCase();
+ } else {
+ return s;
+ }
+ }
+
+ private String collationName() {
+ StringBuilder builder = new StringBuilder();
+ builder.append("UTF8_BINARY");
+ if (caseConversion != CaseConversion.UNSPECIFIED) {
+ builder.append('_');
+ builder.append(caseConversion.toString());
+ }
+ return builder.toString();
+ }
+ }
+
+ private static class CollationSpecICU extends CollationSpec {
+
+ private static final String[] ICULocaleNames;
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+ private static final String ICUCollatorVersion = "153.120.0.0";
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
+ for (ULocale locale : locales) {
+ if (locale.getVariant().isEmpty()) {
+ String language = locale.getLanguage();
+ assert (!language.isEmpty());
+ StringBuilder builder = new StringBuilder(language);
+ String script = locale.getScript();
+ if (!script.isEmpty()) {
+ builder.append('_');
+ builder.append(script);
+ }
+ String country = locale.getISO3Country();
+ if (!country.isEmpty()) {
+ builder.append('_');
+ builder.append(country);
+ }
+ String localeName = builder.toString();
+ assert (!ICULocaleMap.containsKey(localeName));
+ ICULocaleMap.put(localeName, locale);
+ }
+ }
+ for (String localeName : ICULocaleMap.keySet()) {
+ String localeUppercase = localeName.toUpperCase();
+ assert (!ICULocaleMapUppercase.containsKey(localeUppercase));
+ ICULocaleMapUppercase.put(localeUppercase, localeName);
+ }
+ ICULocaleNames = ICULocaleMap.keySet().toArray(new String[0]);
+ Arrays.sort(ICULocaleNames);
+ assert (ICULocaleNames.length <= (1 << 16));
+ for (int i = 0; i < ICULocaleNames.length; i++) {
+ ICULocaleToId.put(ICULocaleNames[i], i);
+ }
Review Comment:
The goal is to have stable mapping between integer ids and locale names. Please check generated golden [file](https://github.com/apache/spark/pull/46180/files#diff-23fa9a48d97f6a09df9ffbaffce69c040bbd1be2a6cf430242adf9597839b3e3) which outlines this mapping. It's also described as part of collation id [spec](https://github.com/apache/spark/pull/46180/files#diff-640c14aa5d7473df79b2435ce5a327dffcc16ca29354b153956b4f8d19fdb16cR133). Any changes to the mapping locale name - id mapping will require golden file update to add more visibility.
However, since the collation ids and therefore locale ids (as part of collation id) are now internal-only and never exposed to user (please confirm @dbatomic @stefankandic), they don't have to strictly remain the same across different Spark versions. That said, it could be problematic to have executors with different Spark versions with different locale name - id mapping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1601309336
##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/common/DataTypeProtoConverter.scala:
##########
@@ -80,7 +81,7 @@ object DataTypeProtoConverter {
}
private def toCatalystStringType(t: proto.DataType.String): StringType =
- StringType(t.getCollationId)
+ StringType(if (t.getCollation.nonEmpty) t.getCollation else "UTF8_BINARY")
Review Comment:
Please check protobuf [comment](https://github.com/apache/spark/pull/46180#discussion_r1601306785). The goal here is to support backwards compatibility in case of old PySpark client version for example. From protobuf [docs](https://protobuf.dev/programming-guides/proto3/#default), the default value for string is empty string so if client fails to set this field (in case of old client), we interpret collation as `UTF8_BINARY`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1609814129
##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/common/DataTypeProtoConverter.scala:
##########
@@ -177,7 +178,11 @@ object DataTypeProtoConverter {
case s: StringType =>
proto.DataType
.newBuilder()
- .setString(proto.DataType.String.newBuilder().setCollationId(s.collationId).build())
+ .setString(
+ proto.DataType.String
+ .newBuilder()
+ .setCollation(CollationFactory.fetchCollation(s.collationId).collationName)
Review Comment:
does it mean that the Spark Connect client already knows the collation id, but we turn it back to name when sending proto messages to the server?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1609933009
##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/common/DataTypeProtoConverter.scala:
##########
@@ -177,7 +178,11 @@ object DataTypeProtoConverter {
case s: StringType =>
proto.DataType
.newBuilder()
- .setString(proto.DataType.String.newBuilder().setCollationId(s.collationId).build())
+ .setString(
+ proto.DataType.String
+ .newBuilder()
+ .setCollation(CollationFactory.fetchCollation(s.collationId).collationName)
Review Comment:
It's true that we use collation name in protobuf messages, this is also part of the changes (before we sent collation id).
We decided to go with this approach in order to remove collation id logic from PySpark side because if collation id is part of the protobuf message, we would need to have collation name parser logic in Python as well besides Scala (`CollationFactory`) which would be unnecessary overhead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606716828
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
Review Comment:
Outdated, we have new schema and explanation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606718203
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
Review Comment:
Added examples to binary layout explanation comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606750756
##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -30,31 +33,95 @@ import org.apache.spark.unsafe.types.UTF8String.{fromString => toUTF8}
class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ignore funsuite
test("collationId stability") {
- val utf8Binary = fetchCollation(0)
+ assert(INDETERMINATE_COLLATION_ID == -1)
+
+ assert(UTF8_BINARY_COLLATION_ID == 0)
+ val utf8Binary = fetchCollation(UTF8_BINARY_COLLATION_ID)
assert(utf8Binary.collationName == "UTF8_BINARY")
assert(utf8Binary.supportsBinaryEquality)
- val utf8BinaryLcase = fetchCollation(1)
+ assert(UTF8_BINARY_LCASE_COLLATION_ID == 1)
+ val utf8BinaryLcase = fetchCollation(UTF8_BINARY_LCASE_COLLATION_ID)
assert(utf8BinaryLcase.collationName == "UTF8_BINARY_LCASE")
assert(!utf8BinaryLcase.supportsBinaryEquality)
- val unicode = fetchCollation(2)
+ assert(UNICODE_COLLATION_ID == (1 << 29))
+ val unicode = fetchCollation(UNICODE_COLLATION_ID)
assert(unicode.collationName == "UNICODE")
- assert(unicode.supportsBinaryEquality);
+ assert(unicode.supportsBinaryEquality)
- val unicodeCi = fetchCollation(3)
+ assert(UNICODE_CI_COLLATION_ID == ((1 << 29) | (1 << 17)))
+ val unicodeCi = fetchCollation(UNICODE_CI_COLLATION_ID)
assert(unicodeCi.collationName == "UNICODE_CI")
assert(!unicodeCi.supportsBinaryEquality)
}
- test("fetch invalid collation name") {
- val error = intercept[SparkException] {
- fetchCollation("UTF8_BS")
+ test("UTF8_BINARY and ICU root locale collation names") {
+ // collation name already normalized
+ Seq(
+ "UTF8_BINARY",
+ "UTF8_BINARY_LCASE",
+ "UNICODE",
+ "UNICODE_CI",
+ "UNICODE_AI",
+ "UNICODE_CI_AI"
+ ).foreach(collationName => {
+ val col = fetchCollation(collationName)
+ assert(col.collationName == collationName)
+ })
+ // collation name normalization
+ Seq(
+ // ICU root locale
+ ("UNICODE_CS", "UNICODE"),
+ ("UNICODE_CS_AS", "UNICODE"),
+ ("UNICODE_CI_AS", "UNICODE_CI"),
+ ("UNICODE_AI_CS", "UNICODE_AI"),
+ ("UNICODE_AI_CI", "UNICODE_CI_AI"),
+ // randomized case collation names
+ ("utf8_binary", "UTF8_BINARY"),
+ ("UtF8_binARy_LcasE", "UTF8_BINARY_LCASE"),
+ ("unicode", "UNICODE"),
+ ("UnICoDe_cs_aI", "UNICODE_AI")
+ ).foreach{
+ case (name, normalized) =>
+ val col = fetchCollation(name)
+ assert(col.collationName == normalized)
}
+ }
+
+ test("fetch invalid UTF8_BINARY and ICU root locale collation names") {
Review Comment:
We already have these tests, please check https://github.com/apache/spark/pull/46180/files#diff-9c12d32db9d55dd6ecb5b10f2fc57c7ba7de7275cab57bf157fa42cbc09f3876R386-R397.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606707381
##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -30,31 +33,101 @@ import org.apache.spark.unsafe.types.UTF8String.{fromString => toUTF8}
class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ignore funsuite
test("collationId stability") {
- val utf8Binary = fetchCollation(0)
+ assert(UTF8_BINARY_COLLATION_ID == 0)
+ assert(INDETERMINATE_COLLATION_ID == 1 << 30)
+
+ val utf8Binary = fetchCollation(UTF8_BINARY_COLLATION_ID)
assert(utf8Binary.collationName == "UTF8_BINARY")
assert(utf8Binary.supportsBinaryEquality)
- val utf8BinaryLcase = fetchCollation(1)
+ val utf8BinaryLcase = fetchCollation(UTF8_BINARY_LCASE_COLLATION_ID)
assert(utf8BinaryLcase.collationName == "UTF8_BINARY_LCASE")
assert(!utf8BinaryLcase.supportsBinaryEquality)
- val unicode = fetchCollation(2)
+ val unicode = fetchCollation(UNICODE_COLLATION_ID)
assert(unicode.collationName == "UNICODE")
- assert(unicode.supportsBinaryEquality);
+ assert(unicode.supportsBinaryEquality)
- val unicodeCi = fetchCollation(3)
+ val unicodeCi = fetchCollation(UNICODE_CI_COLLATION_ID)
assert(unicodeCi.collationName == "UNICODE_CI")
assert(!unicodeCi.supportsBinaryEquality)
}
- test("fetch invalid collation name") {
- val error = intercept[SparkException] {
- fetchCollation("UTF8_BS")
+ test("UTF8_BINARY and ICU root locale collation names") {
+ // collation name already normalized
+ Seq(
+ "UTF8_BINARY",
+ "UTF8_BINARY_LCASE",
+ "UTF8_BINARY_UCASE",
+ "UNICODE",
+ "UNICODE_CI",
+ "UNICODE_AI",
+ "UNICODE_CI_AI",
+ "UNICODE_LCASE",
+ "UNICODE_UCASE"
+ ).foreach(collationName => {
+ val col = fetchCollation(collationName)
+ assert(col.collationName == collationName)
+ })
+ // collation name normalization
+ Seq(
+ // UTF8_BINARY
+ ("UTF8_BINARY_CS", "UTF8_BINARY"),
Review Comment:
Approach is different now. We scope this PR to `UTF8_BINARY` and `UTF8_BINARY_LCASE`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606956472
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -118,76 +119,433 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * Collation id is defined as 32-bit integer.
+ * We specify binary layouts for different classes of collations.
+ * Classes of collations are differentiated by most significant 3 bits (bit 31, 30 and 29),
+ * bit 31 being most significant and bit 0 being least significant.
+ * ---
+ * INDETERMINATE collation id binary layout:
+ * bit 31-0: 1
+ * INDETERMINATE collation id is equal to -1
+ * ---
+ * user-defined collation id binary layout:
+ * bit 31: 0
+ * bit 30: 1
+ * bit 29-0: undefined, reserved for future use
+ * ---
+ * UTF8_BINARY collation id binary layout:
+ * bit 31-22: zeroes
+ * bit 21-18: zeroes, reserved for space trimming
+ * bit 17-16: zeroes, reserved for version
+ * bit 15-3: zeroes
+ * bit 2: 0, reserved for accent sensitivity
+ * bit 1: 0, reserved for uppercase and case-insensitive
+ * bit 0: 0 = case-sensitive, 1 = lowercase
+ * ---
+ * ICU collation id binary layout:
+ * bit 31-30: zeroes
+ * bit 29: 1
+ * bit 28-24: zeroes
+ * bit 23-22: zeroes, reserved for version
Review Comment:
Thanks for catching this. I refactored the comment to unify bits used for versioning and space trimming in both `UTF8_BINARY` and ICU collations. Added bit-centric view as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1579098122
##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -152,4 +219,147 @@ class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ig
}
})
}
+
+ test("test collation caching") {
+ Seq(
+ "UTF8_BINARY",
+ "UTF8_BINARY_LCASE",
+ "UTF8_BINARY_UCASE",
+ "UNICODE",
+ "UNICODE_LCASE",
+ "UNICODE_UCASE",
+ "UNICODE_CI",
+ "UNICODE_AI_CI",
+ "UNICODE_AI_CI_LCASE",
+ "UNICODE_AI_CI_UCASE"
+ ).foreach(collationId => {
+ val col1 = fetchCollation(collationId)
+ val col2 = fetchCollation(collationId)
+ assert(col1 eq col2) // reference equality
+ })
+ }
+
+ test("collations with ICU non-root localization") {
+ Seq(
+ // language only
+ "en",
+ "en_CS",
+ "en_CI",
+ "en_AS",
+ "en_AI",
+ "en_LCASE",
+ "en_UCASE",
+ // language + 3-letter country code
+ "en_USA",
+ "en_USA_CS",
+ "en_USA_CI",
+ "en_USA_AS",
+ "en_USA_AI",
+ "en_USA_LCASE",
+ "en_USA_UCASE",
+ // language + script code
+ "sr_Cyrl",
+ "sr_Cyrl_CS",
+ "sr_Cyrl_CI",
+ "sr_Cyrl_AS",
+ "sr_Cyrl_AI",
+ "sr_Cyrl_LCASE",
+ "sr_Cyrl_UCASE",
+ // language + script code + 3-letter country code
+ "sr_Cyrl_SRB",
+ "sr_Cyrl_SRB_CS",
+ "sr_Cyrl_SRB_CI",
+ "sr_Cyrl_SRB_AS",
+ "sr_Cyrl_SRB_AI",
+ "sr_Cyrl_SRB_LCASE",
+ "sr_Cyrl_SRB_UCASE"
+ ).foreach(collationICU => {
+ val col = fetchCollation(collationICU)
+ assert(col.collator.getLocale(ULocale.VALID_LOCALE) != ULocale.ROOT)
+ })
+ }
+
+ test("invalid names of collations with ICU non-root localization") {
+ Seq(
+ "en_US", // must use 3-letter country code
+ "enn",
+ "en_AAA",
+ "en_Something",
+ "en_Something_USA",
+ "en_Latn_USA", // use en_USA instead
+ "en_Cyrl_USA",
+ "en_USA_AAA",
+ "sr_Cyrl_SRB_AAA"
+ ).foreach(collationName => {
+ val error = intercept[SparkException] {
+ fetchCollation(collationName)
+ }
+
+ assert(error.getErrorClass === "COLLATION_INVALID_NAME")
+ assert(error.getMessageParameters.asScala === Map("collationName" -> collationName))
+ })
+ }
+
+ test("collations name normalization for ICU non-root localization") {
+ Seq(
+ ("en_USA", "en_USA"),
+ ("en_CS", "en"),
+ ("en_AS", "en"),
+ ("en_CS_AS", "en"),
+ ("en_AI_CI", "en_CI_AI"),
+ ("en_USA_AI_CI", "en_USA_CI_AI"),
+ // randomized case
+ ("EN_USA", "en_USA"),
+ ("eN_usA_ci_uCASe_aI", "en_USA_CI_AI_UCASE"),
+ ("SR_CYRL", "sr_Cyrl"),
+ ("sr_cyrl_srb", "sr_Cyrl_SRB"),
+ ("sR_cYRl_sRb", "sr_Cyrl_SRB")
+ ).foreach {
+ case (name, normalized) =>
+ val col = fetchCollation(name)
+ assert(col.collationName == normalized)
+ }
+ }
+
+ test("invalid collationId") {
+ val badCollationIds = Seq(
+ -1, // user-defined collation range
Review Comment:
From my understanding, Spark runs in Java virtual machine in which ints are always 32 bits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "mihailom-db (via GitHub)" <gi...@apache.org>.
mihailom-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1579089752
##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -152,4 +219,147 @@ class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ig
}
})
}
+
+ test("test collation caching") {
+ Seq(
+ "UTF8_BINARY",
+ "UTF8_BINARY_LCASE",
+ "UTF8_BINARY_UCASE",
+ "UNICODE",
+ "UNICODE_LCASE",
+ "UNICODE_UCASE",
+ "UNICODE_CI",
+ "UNICODE_AI_CI",
+ "UNICODE_AI_CI_LCASE",
+ "UNICODE_AI_CI_UCASE"
+ ).foreach(collationId => {
+ val col1 = fetchCollation(collationId)
+ val col2 = fetchCollation(collationId)
+ assert(col1 eq col2) // reference equality
+ })
+ }
+
+ test("collations with ICU non-root localization") {
+ Seq(
+ // language only
+ "en",
+ "en_CS",
+ "en_CI",
+ "en_AS",
+ "en_AI",
+ "en_LCASE",
+ "en_UCASE",
+ // language + 3-letter country code
+ "en_USA",
+ "en_USA_CS",
+ "en_USA_CI",
+ "en_USA_AS",
+ "en_USA_AI",
+ "en_USA_LCASE",
+ "en_USA_UCASE",
+ // language + script code
+ "sr_Cyrl",
+ "sr_Cyrl_CS",
+ "sr_Cyrl_CI",
+ "sr_Cyrl_AS",
+ "sr_Cyrl_AI",
+ "sr_Cyrl_LCASE",
+ "sr_Cyrl_UCASE",
+ // language + script code + 3-letter country code
+ "sr_Cyrl_SRB",
+ "sr_Cyrl_SRB_CS",
+ "sr_Cyrl_SRB_CI",
+ "sr_Cyrl_SRB_AS",
+ "sr_Cyrl_SRB_AI",
+ "sr_Cyrl_SRB_LCASE",
+ "sr_Cyrl_SRB_UCASE"
+ ).foreach(collationICU => {
+ val col = fetchCollation(collationICU)
+ assert(col.collator.getLocale(ULocale.VALID_LOCALE) != ULocale.ROOT)
+ })
+ }
+
+ test("invalid names of collations with ICU non-root localization") {
+ Seq(
+ "en_US", // must use 3-letter country code
+ "enn",
+ "en_AAA",
+ "en_Something",
+ "en_Something_USA",
+ "en_Latn_USA", // use en_USA instead
+ "en_Cyrl_USA",
+ "en_USA_AAA",
+ "sr_Cyrl_SRB_AAA"
+ ).foreach(collationName => {
+ val error = intercept[SparkException] {
+ fetchCollation(collationName)
+ }
+
+ assert(error.getErrorClass === "COLLATION_INVALID_NAME")
+ assert(error.getMessageParameters.asScala === Map("collationName" -> collationName))
+ })
+ }
+
+ test("collations name normalization for ICU non-root localization") {
+ Seq(
+ ("en_USA", "en_USA"),
+ ("en_CS", "en"),
+ ("en_AS", "en"),
+ ("en_CS_AS", "en"),
+ ("en_AI_CI", "en_CI_AI"),
+ ("en_USA_AI_CI", "en_USA_CI_AI"),
+ // randomized case
+ ("EN_USA", "en_USA"),
+ ("eN_usA_ci_uCASe_aI", "en_USA_CI_AI_UCASE"),
+ ("SR_CYRL", "sr_Cyrl"),
+ ("sr_cyrl_srb", "sr_Cyrl_SRB"),
+ ("sR_cYRl_sRb", "sr_Cyrl_SRB")
+ ).foreach {
+ case (name, normalized) =>
+ val col = fetchCollation(name)
+ assert(col.collationName == normalized)
+ }
+ }
+
+ test("invalid collationId") {
+ val badCollationIds = Seq(
+ -1, // user-defined collation range
Review Comment:
Are we sure that scala int will always be interpreted as 32 bit value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841] Add collation support for ICU locales and collation specifiers [spark]
Posted by "dbatomic (via GitHub)" <gi...@apache.org>.
dbatomic commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1576145404
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +118,490 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30: 0 = utf8-binary, 1 = ICU
+ * bit 29: 0 = case-sensitive, 1 = case-insensitive
+ * bit 28: 0 = accent-sensitive, 1 = accent-insensitive
+ * bit 27-26: 00 = unspecified, 01 = punctuation-sensitive, 10 = punctuation-insensitive
+ * bit 25-24: 00 = unspecified, 01 = first-lower, 10 = first-upper
+ * bit 23-22: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 21-20: 00 = unspecified, 01 = trim-left, 10 = trim-right, 11 = trim-both
+ * bit 19-18: zeroes, reserved for version
+ * bit 17-16: zeroes
+ * bit 15-0: locale id for ICU collations / zeroes for utf8-binary
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
- }
- }
+ private static class CollationSpec {
+ private enum ImplementationProvider {
+ UTF8_BINARY, ICU
+ }
+
+ private enum CaseSensitivity {
+ CS, CI
+ }
+
+ private enum AccentSensitivity {
+ AS, AI
+ }
+
+ private enum PunctuationSensitivity {
+ UNSPECIFIED, PS, PI
+ }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private enum FirstLetterPreference {
+ UNSPECIFIED, FU, FL
+ }
+
+ private enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ private enum SpaceTrimming {
+ UNSPECIFIED, LTRIM, RTRIM, TRIM
+ }
+
+ private static final int implementationProviderOffset = 30;
+ private static final int implementationProviderLen = 1;
+ private static final int caseSensitivityOffset = 29;
+ private static final int caseSensitivityLen = 1;
+ private static final int accentSensitivityOffset = 28;
+ private static final int accentSensitivityLen = 1;
+ private static final int punctuationSensitivityOffset = 26;
+ private static final int punctuationSensitivityLen = 2;
+ private static final int firstLetterPreferenceOffset = 24;
+ private static final int firstLetterPreferenceLen = 2;
+ private static final int caseConversionOffset = 22;
+ private static final int caseConversionLen = 2;
+ private static final int spaceTrimmingOffset = 20;
+ private static final int spaceTrimmingLen = 2;
+ private static final int localeOffset = 0;
+ private static final int localeLen = 16;
+
+ private static final String[] ICULocaleNames;
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
+ for (ULocale locale : locales) {
+ if (locale.getVariant().isEmpty()) {
+ String language = locale.getLanguage();
+ assert (!language.isEmpty());
+ StringBuilder builder = new StringBuilder(language);
+ String script = locale.getScript();
+ if (!script.isEmpty()) {
+ builder.append('_');
+ builder.append(script);
+ }
+ String country = locale.getISO3Country();
+ if (!country.isEmpty()) {
+ builder.append('_');
+ builder.append(country);
+ }
+ String localeName = builder.toString();
+ assert (!ICULocaleMap.containsKey(localeName));
+ ICULocaleMap.put(localeName, locale);
+ }
+ }
+ for (String localeName : ICULocaleMap.keySet()) {
+ String localeUppercase = localeName.toUpperCase();
+ assert (!ICULocaleMapUppercase.containsKey(localeUppercase));
+ ICULocaleMapUppercase.put(localeUppercase, localeName);
+ }
+ ICULocaleNames = ICULocaleMap.keySet().toArray(new String[0]);
+ Arrays.sort(ICULocaleNames);
+ assert (ICULocaleNames.length <= (1 << 16));
+ for (int i = 0; i < ICULocaleNames.length; i++) {
+ ICULocaleToId.put(ICULocaleNames[i], i);
+ }
+ }
+
+ public static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpec(
+ ImplementationProvider.UTF8_BINARY,
+ null,
+ CaseSensitivity.CS,
+ AccentSensitivity.AS,
+ PunctuationSensitivity.UNSPECIFIED,
+ FirstLetterPreference.UNSPECIFIED,
+ CaseConversion.UNSPECIFIED,
+ SpaceTrimming.UNSPECIFIED
+ ).getCollationId();
+
+ public static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpec(
+ ImplementationProvider.UTF8_BINARY,
+ null,
+ CaseSensitivity.CS,
+ AccentSensitivity.AS,
+ PunctuationSensitivity.UNSPECIFIED,
+ FirstLetterPreference.UNSPECIFIED,
+ CaseConversion.LCASE,
+ SpaceTrimming.UNSPECIFIED
+ ).getCollationId();
+
+ public static final int UNICODE_COLLATION_ID =
+ new CollationSpec(
+ ImplementationProvider.ICU,
+ "UNICODE",
+ CaseSensitivity.CS,
+ AccentSensitivity.AS,
+ PunctuationSensitivity.UNSPECIFIED,
+ FirstLetterPreference.UNSPECIFIED,
+ CaseConversion.UNSPECIFIED,
+ SpaceTrimming.UNSPECIFIED
+ ).getCollationId();
+
+ public static final int UNICODE_CI_COLLATION_ID =
+ new CollationSpec(
+ ImplementationProvider.ICU,
+ "UNICODE",
+ CaseSensitivity.CI,
+ AccentSensitivity.AS,
+ PunctuationSensitivity.UNSPECIFIED,
+ FirstLetterPreference.UNSPECIFIED,
+ CaseConversion.UNSPECIFIED,
+ SpaceTrimming.UNSPECIFIED
+ ).getCollationId();
+
+ private final ImplementationProvider implementationProvider;
+ private final CaseSensitivity caseSensitivity;
+ private final AccentSensitivity accentSensitivity;
+ private final PunctuationSensitivity punctuationSensitivity;
+ private final FirstLetterPreference firstLetterPreference;
+ private final CaseConversion caseConversion;
+ private final SpaceTrimming spaceTrimming;
+ private final String locale;
+ private final int collationId;
+
+ private CollationSpec(
+ ImplementationProvider implementationProvider,
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ PunctuationSensitivity punctuationSensitivity,
+ FirstLetterPreference firstLetterPreference,
+ CaseConversion caseConversion,
+ SpaceTrimming spaceTrimming) {
+ this.implementationProvider = implementationProvider;
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.punctuationSensitivity = punctuationSensitivity;
+ this.firstLetterPreference = firstLetterPreference;
+ this.caseConversion = caseConversion;
+ this.spaceTrimming = spaceTrimming;
+ this.collationId = getCollationId();
+ }
+
+ public int getCollationId() {
+ int collationId = 0;
+ collationId |= implementationProvider.ordinal() << implementationProviderOffset;
+ collationId |= caseSensitivity.ordinal() << caseSensitivityOffset;
+ collationId |= accentSensitivity.ordinal() << accentSensitivityOffset;
+ collationId |= punctuationSensitivity.ordinal() << punctuationSensitivityOffset;
+ collationId |= firstLetterPreference.ordinal() << firstLetterPreferenceOffset;
+ collationId |= caseConversion.ordinal() << caseConversionOffset;
+ collationId |= spaceTrimming.ordinal() << spaceTrimmingOffset;
+ if (implementationProvider == ImplementationProvider.ICU) {
+ collationId |= ICULocaleToId.get(locale);
+ }
+ return collationId;
+ }
+
+ public static CollationSpec fromCollationId(int collationId) {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ (collationId >> implementationProviderOffset) & ((1 << implementationProviderLen) - 1)];
+ CaseSensitivity caseSensitivity = CaseSensitivity.values()[
+ (collationId >> caseSensitivityOffset) & ((1 << caseSensitivityLen) - 1)];
+ AccentSensitivity accentSensitivity = AccentSensitivity.values()[
+ (collationId >> accentSensitivityOffset) & ((1 << accentSensitivityLen) - 1)];
+ PunctuationSensitivity punctuationSensitivity = PunctuationSensitivity.values()[
+ (collationId >> punctuationSensitivityOffset) & ((1 << punctuationSensitivityLen) - 1)];
+ FirstLetterPreference firstLetterPreference = FirstLetterPreference.values()[
+ (collationId >> firstLetterPreferenceOffset) & ((1 << firstLetterPreferenceLen) - 1)];
+ CaseConversion caseConversion = CaseConversion.values()[
+ (collationId >> caseConversionOffset) & ((1 << caseConversionLen) - 1)];
+ SpaceTrimming spaceTrimming = SpaceTrimming.values()[
+ (collationId >> spaceTrimmingOffset) & ((1 << spaceTrimmingLen) - 1)];
+ String locale;
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ locale = "UTF8_BINARY";
+ } else {
+ locale = ICULocaleNames[(collationId >> localeOffset) & ((1 << localeLen) - 1)];
+ }
+ return new CollationSpec(
+ implementationProvider,
+ locale,
+ caseSensitivity,
+ accentSensitivity,
+ punctuationSensitivity,
+ firstLetterPreference,
+ caseConversion,
+ spaceTrimming
+ );
+ }
+
+ public static int collationNameToId(String originalCollationName) throws SparkException {
+ String collationName = originalCollationName.toUpperCase();
+ try {
+ if (collationName.startsWith("UTF8_BINARY")) {
+ return collationUTF8BinaryNameToId(collationName);
+ } else {
+ return collationICUNameToId(collationName);
+ }
+ } catch (SparkException e) {
+ throw new SparkException(
+ "COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", originalCollationName)),
+ e);
+ }
+ }
+
+ private static int collationUTF8BinaryNameToId(String collationName) throws SparkException {
+ int collationId = 0;
+ collationId |= ImplementationProvider.UTF8_BINARY.ordinal() << implementationProviderOffset;
+ collationId |= parseSpecifiers(collationName.substring("UTF8_BINARY".length()));
+ return collationId;
+ }
+
+ private static int collationICUNameToId(String collationName) throws SparkException {
+ int lastPos = -1;
+ for (int i = 1; i <= collationName.length(); i++) {
+ String localeName = collationName.substring(0, i);
+ if (ICULocaleMapUppercase.containsKey(localeName)) {
+ lastPos = i;
+ }
+ }
+ if (lastPos == -1) {
+ throw new SparkException("Invalid locale in collation name value " + collationName);
+ } else {
+ int collationId = 0;
+ collationId |= ImplementationProvider.ICU.ordinal() << implementationProviderOffset;
+ collationId |= parseSpecifiers(collationName.substring(lastPos));
+ String normalizedLocaleName = ICULocaleMapUppercase.get(
+ collationName.substring(0, lastPos));
+ collationId |= ICULocaleToId.get(normalizedLocaleName) << localeOffset;
+ return collationId;
+ }
+ }
+
+ private static int parseSpecifiers(String specString) throws SparkException {
+ int specifiers = 0;
+ String[] parts = specString.split("_");
+ for (String part : parts) {
+ if (!part.isEmpty()) {
+ if (part.equals("UNSPECIFIED")) {
+ throw new SparkException("UNSPECIFIED collation specifier reserved for internal use");
+ } else if (Arrays.stream(CaseSensitivity.values()).anyMatch(
+ (s) -> s.toString().equals(part))) {
+ specifiers |=
+ CaseSensitivity.valueOf(part).ordinal() << caseSensitivityOffset;
+ } else if (Arrays.stream(AccentSensitivity.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ AccentSensitivity.valueOf(part).ordinal() << accentSensitivityOffset;
+ } else if (Arrays.stream(PunctuationSensitivity.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ PunctuationSensitivity.valueOf(part).ordinal() << punctuationSensitivityOffset;
+ } else if (Arrays.stream(FirstLetterPreference.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ FirstLetterPreference.valueOf(part).ordinal() << firstLetterPreferenceOffset;
+ } else if (Arrays.stream(CaseConversion.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ CaseConversion.valueOf(part).ordinal() << caseConversionOffset;
+ } else if (Arrays.stream(SpaceTrimming.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ SpaceTrimming.valueOf(part).ordinal() << spaceTrimmingOffset;
+ } else {
+ throw new SparkException("Invalid collation specifier value " + part);
+ }
+ }
+ }
+ return specifiers;
+ }
+
+ public Collation buildCollation() {
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ return buildUTF8BinaryCollation();
+ } else {
+ return buildICUCollation();
+ }
+ }
+
+ public Collation buildUTF8BinaryCollation() {
+ Comparator<UTF8String> comparator;
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ comparator = UTF8String::binaryCompare;
+ } else {
+ comparator = (s1, s2) -> {
+ UTF8String convertedS1 = caseAndTrimmingConversionUTF8Binary(s1);
+ UTF8String convertedS2 = caseAndTrimmingConversionUTF8Binary(s2);
+ return convertedS1.binaryCompare(convertedS2);
+ };
+ }
+ return new Collation(
+ collationName(),
+ null,
+ comparator,
+ "1.0",
+ s -> (long) caseAndTrimmingConversionUTF8Binary(s).hashCode(),
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_LCASE_COLLATION_ID
+ );
+ }
+
+ private UTF8String caseAndTrimmingConversionUTF8Binary(UTF8String s) {
Review Comment:
My suggestion would be to only cover collation id format, as you did above. No need to implement support/parsing for anything but case and accent at this point.
We can follow up with trimming and co support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "stefankandic (via GitHub)" <gi...@apache.org>.
stefankandic commented on PR #46180:
URL: https://github.com/apache/spark/pull/46180#issuecomment-2098821075
will we have to do the same for pyspark - as `StringType` there only supports 4 initial collations?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "dbatomic (via GitHub)" <gi...@apache.org>.
dbatomic commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1592463454
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,438 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ public static int collationNameToId(String originalCollationName) throws SparkException {
+ String collationName = originalCollationName.toUpperCase();
+ try {
+ if (collationName.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName);
+ }
+ } catch (SparkException e) {
+ throw new SparkException(
+ "COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", originalCollationName)),
+ e);
+ }
+ }
+
+ protected static int parseSpecifiers(String specString) throws SparkException {
+ int specifiers = 0;
+ String[] parts = specString.split("_");
+ for (String part : parts) {
+ if (!part.isEmpty()) {
+ if (part.equals("UNSPECIFIED") || part.equals("INDETERMINATE")) {
+ throw new SparkException(part + " collation specifier reserved for internal use");
Review Comment:
User should never get generic `SparkException`. We should create new exception for invalid collation name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on PR #46180:
URL: https://github.com/apache/spark/pull/46180#issuecomment-2116855540
@mkaravel @dbatomic please review again, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1601334716
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -245,29 +599,26 @@ public static StringSearch getStringSearch(
* Returns the collation id for the given collation name.
*/
public static int collationNameToId(String collationName) throws SparkException {
- String normalizedName = collationName.toUpperCase();
- if (collationNameToIdMap.containsKey(normalizedName)) {
- return collationNameToIdMap.get(normalizedName);
- } else {
- Collation suggestion = Collections.min(List.of(collationTable), Comparator.comparingInt(
- c -> UTF8String.fromString(c.collationName).levenshteinDistance(
- UTF8String.fromString(normalizedName))));
-
- Map<String, String> params = new HashMap<>();
- params.put("collationName", collationName);
- params.put("proposal", suggestion.collationName);
-
- throw new SparkException(
- "COLLATION_INVALID_NAME", SparkException.constructMessageParams(params), null);
- }
+ return Collation.CollationSpec.collationNameToId(collationName);
+ }
+
+ public static Collation fetchCollationUnsafe(int collationId) throws SparkException {
+ return Collation.CollationSpec.fetchCollation(collationId);
}
public static Collation fetchCollation(int collationId) {
- return collationTable[collationId];
+ try {
+ return fetchCollationUnsafe(collationId);
+ } catch (SparkException e) {
+ return Collation.CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ }
Review Comment:
The idea for this function is that it is free of exceptions because we assume internal implementation will always call the function with valid collation id parameter obtained earlier by parsing collation name string. We forbid the user to explicitly pass collation id to `StringType` by marking this constructor as private.
However, internal fetch with collation id does potentially throw an exception. So by returning `UTF8_BINARY` if the error does occur (which would indicate code logic problems - internal error) we don't need to change the signature of this function to throw an exception and propagate the change to numerous places where function is called (mainly in `CollationSupport`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606718885
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
Review Comment:
Scoped bitwise operations (shifting, or, and, xor) to separate utility class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1609900650
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -173,26 +174,546 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * Collation ID is defined as 32-bit integer. We specify binary layouts for different classes of
+ * collations. Classes of collations are differentiated by most significant 3 bits (bit 31, 30
+ * and 29), bit 31 being most significant and bit 0 being least significant.
+ * ---
+ * General collation ID binary layout:
+ * bit 31: 1 for INDETERMINATE (requires all other bits to be 1 as well), 0 otherwise.
+ * bit 30: 0 for predefined, 1 for user-defined.
Review Comment:
We don't support user-defined collations for now. This is reserved for future use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "uros-db (via GitHub)" <gi...@apache.org>.
uros-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1579147729
##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -152,4 +219,147 @@ class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ig
}
})
}
+
+ test("test collation caching") {
+ Seq(
+ "UTF8_BINARY",
+ "UTF8_BINARY_LCASE",
+ "UTF8_BINARY_UCASE",
+ "UNICODE",
+ "UNICODE_LCASE",
+ "UNICODE_UCASE",
+ "UNICODE_CI",
+ "UNICODE_AI_CI",
+ "UNICODE_AI_CI_LCASE",
+ "UNICODE_AI_CI_UCASE"
+ ).foreach(collationId => {
+ val col1 = fetchCollation(collationId)
+ val col2 = fetchCollation(collationId)
+ assert(col1 eq col2) // reference equality
+ })
+ }
+
+ test("collations with ICU non-root localization") {
+ Seq(
+ // language only
+ "en",
+ "en_CS",
+ "en_CI",
+ "en_AS",
+ "en_AI",
+ "en_LCASE",
+ "en_UCASE",
+ // language + 3-letter country code
+ "en_USA",
+ "en_USA_CS",
+ "en_USA_CI",
+ "en_USA_AS",
+ "en_USA_AI",
+ "en_USA_LCASE",
+ "en_USA_UCASE",
+ // language + script code
+ "sr_Cyrl",
+ "sr_Cyrl_CS",
+ "sr_Cyrl_CI",
+ "sr_Cyrl_AS",
+ "sr_Cyrl_AI",
+ "sr_Cyrl_LCASE",
+ "sr_Cyrl_UCASE",
+ // language + script code + 3-letter country code
+ "sr_Cyrl_SRB",
+ "sr_Cyrl_SRB_CS",
+ "sr_Cyrl_SRB_CI",
+ "sr_Cyrl_SRB_AS",
+ "sr_Cyrl_SRB_AI",
+ "sr_Cyrl_SRB_LCASE",
+ "sr_Cyrl_SRB_UCASE"
+ ).foreach(collationICU => {
+ val col = fetchCollation(collationICU)
+ assert(col.collator.getLocale(ULocale.VALID_LOCALE) != ULocale.ROOT)
+ })
+ }
+
+ test("invalid names of collations with ICU non-root localization") {
+ Seq(
+ "en_US", // must use 3-letter country code
+ "enn",
+ "en_AAA",
+ "en_Something",
+ "en_Something_USA",
+ "en_Latn_USA", // use en_USA instead
+ "en_Cyrl_USA",
+ "en_USA_AAA",
+ "sr_Cyrl_SRB_AAA"
+ ).foreach(collationName => {
+ val error = intercept[SparkException] {
+ fetchCollation(collationName)
+ }
+
+ assert(error.getErrorClass === "COLLATION_INVALID_NAME")
+ assert(error.getMessageParameters.asScala === Map("collationName" -> collationName))
+ })
+ }
+
+ test("collations name normalization for ICU non-root localization") {
+ Seq(
+ ("en_USA", "en_USA"),
+ ("en_CS", "en"),
+ ("en_AS", "en"),
+ ("en_CS_AS", "en"),
+ ("en_AI_CI", "en_CI_AI"),
+ ("en_USA_AI_CI", "en_USA_CI_AI"),
+ // randomized case
+ ("EN_USA", "en_USA"),
+ ("eN_usA_ci_uCASe_aI", "en_USA_CI_AI_UCASE"),
+ ("SR_CYRL", "sr_Cyrl"),
+ ("sr_cyrl_srb", "sr_Cyrl_SRB"),
+ ("sR_cYRl_sRb", "sr_Cyrl_SRB")
+ ).foreach {
+ case (name, normalized) =>
+ val col = fetchCollation(name)
+ assert(col.collationName == normalized)
+ }
+ }
+
+ test("invalid collationId") {
+ val badCollationIds = Seq(
+ -1, // user-defined collation range
Review Comment:
If it does end up doing that for 64b shouldn't be a problem (even with hypothetical indeterminate collation id = -1), but potentially 16b would be dangerous - although I don't know of a JVM that would interpret this as anything other than 32b, someone should probably confirm this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "dbatomic (via GitHub)" <gi...@apache.org>.
dbatomic commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1592428205
##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -30,31 +33,101 @@ import org.apache.spark.unsafe.types.UTF8String.{fromString => toUTF8}
class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ignore funsuite
test("collationId stability") {
- val utf8Binary = fetchCollation(0)
+ assert(UTF8_BINARY_COLLATION_ID == 0)
+ assert(INDETERMINATE_COLLATION_ID == 1 << 30)
+
+ val utf8Binary = fetchCollation(UTF8_BINARY_COLLATION_ID)
assert(utf8Binary.collationName == "UTF8_BINARY")
assert(utf8Binary.supportsBinaryEquality)
- val utf8BinaryLcase = fetchCollation(1)
+ val utf8BinaryLcase = fetchCollation(UTF8_BINARY_LCASE_COLLATION_ID)
assert(utf8BinaryLcase.collationName == "UTF8_BINARY_LCASE")
assert(!utf8BinaryLcase.supportsBinaryEquality)
- val unicode = fetchCollation(2)
+ val unicode = fetchCollation(UNICODE_COLLATION_ID)
assert(unicode.collationName == "UNICODE")
- assert(unicode.supportsBinaryEquality);
+ assert(unicode.supportsBinaryEquality)
- val unicodeCi = fetchCollation(3)
+ val unicodeCi = fetchCollation(UNICODE_CI_COLLATION_ID)
assert(unicodeCi.collationName == "UNICODE_CI")
assert(!unicodeCi.supportsBinaryEquality)
}
- test("fetch invalid collation name") {
- val error = intercept[SparkException] {
- fetchCollation("UTF8_BS")
+ test("UTF8_BINARY and ICU root locale collation names") {
+ // collation name already normalized
+ Seq(
+ "UTF8_BINARY",
+ "UTF8_BINARY_LCASE",
+ "UTF8_BINARY_UCASE",
+ "UNICODE",
+ "UNICODE_CI",
+ "UNICODE_AI",
+ "UNICODE_CI_AI",
+ "UNICODE_LCASE",
+ "UNICODE_UCASE"
+ ).foreach(collationName => {
+ val col = fetchCollation(collationName)
+ assert(col.collationName == collationName)
+ })
+ // collation name normalization
+ Seq(
+ // UTF8_BINARY
+ ("UTF8_BINARY_CS", "UTF8_BINARY"),
Review Comment:
I don't think that we should allow UTF8_BINARY_CS, UTF8_BINARY_CI and UTF8_BINARY_AS and silently fallback to UTF8_BINARY. IMO, passing UTF8_BINARY_CS should throw an error as invalid collation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "dbatomic (via GitHub)" <gi...@apache.org>.
dbatomic commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1592461113
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,438 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
Review Comment:
it's hard to gauge perf impact of having `ConcurrentHashMap` here. I am looking at some benchmarks and it seems like it is close to HashMap.
Btw, is saying `fetchCollation` should never be called for collationId=0 a valid statement? AFAIK, we almost always have a fast path for UTF8_BINARY? If we still need fetchCollation for UTF8_BINARY, can we add special case `if collationId == 0` and avoid touching the map cache?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "uros-db (via GitHub)" <gi...@apache.org>.
uros-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1593716451
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
Review Comment:
should we consider using bit 31 for indeterminate collation?
then, indeterminate collation could conveniently be -1 / all 1s in binary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1609803490
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -173,26 +174,546 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * Collation ID is defined as 32-bit integer. We specify binary layouts for different classes of
+ * collations. Classes of collations are differentiated by most significant 3 bits (bit 31, 30
+ * and 29), bit 31 being most significant and bit 0 being least significant.
+ * ---
+ * General collation ID binary layout:
+ * bit 31: 1 for INDETERMINATE (requires all other bits to be 1 as well), 0 otherwise.
+ * bit 30: 0 for predefined, 1 for user-defined.
Review Comment:
do we support it now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "mkaravel (via GitHub)" <gi...@apache.org>.
mkaravel commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1594834488
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected static int parseSpecifiers(
+ String originalName, String collationName, int splitStart) throws SparkException {
+ int specifiers = 0;
+ String[] parts = collationName.substring(splitStart).split("_");
+ for (String part : parts) {
+ if (!part.isEmpty()) {
+ if (part.equals("UNSPECIFIED") || part.equals("INDETERMINATE")) {
+ throw collationInvalidNameException(originalName);
+ } else if (Arrays.stream(CaseSensitivity.values()).anyMatch(
+ (s) -> s.toString().equals(part))) {
+ specifiers |=
+ CaseSensitivity.valueOf(part).ordinal() << CASE_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(AccentSensitivity.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ AccentSensitivity.valueOf(part).ordinal() << ACCENT_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(CaseConversion.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ CaseConversion.valueOf(part).ordinal() << CASE_CONVERSION_OFFSET;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+ }
+ return specifiers;
+ }
+
+ protected abstract int getCollationId();
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ public static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.LCASE).getCollationId();
+ public static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).buildCollation();
+
+ private CollationSpecUTF8Binary(CaseConversion caseConversion) {
+ super(null, CaseSensitivity.CS, AccentSensitivity.AS, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int collationId = 0;
+ int specifiers = CollationSpec.parseSpecifiers(
+ originalName, collationName, "UTF8_BINARY".length());
+ if ((specifiers & ~(CASE_CONVERSION_MASK << CASE_CONVERSION_OFFSET)) != 0) {
+ throw collationInvalidNameException(originalName);
+ }
+ collationId |= specifiers;
+ return collationId;
+ }
+
+ @Override
+ protected int getCollationId() {
+ int collationId = 0;
+ collationId |= caseConversion.ordinal() << CASE_CONVERSION_OFFSET;
+ return collationId;
+ }
+
+ public static CollationSpecUTF8Binary fromCollationId(int collationId)
+ throws SparkException {
+ int originalCollationId = collationId;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
+ if (collationId != 0 || caseConversionOrdinal >= CaseConversion.values().length) {
+ throw SparkException.internalError("Invalid UTF8_BINARY collation id " +
+ originalCollationId);
+ } else {
+ CaseConversion caseConversion = CaseConversion.values()[caseConversionOrdinal];
+ return new CollationSpecUTF8Binary(caseConversion);
+ }
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ Comparator<UTF8String> comparator;
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ comparator = UTF8String::binaryCompare;
+ } else if (collationId == UTF8_BINARY_LCASE_COLLATION_ID) {
+ comparator = UTF8String::compareLowerCase;
+ } else {
+ comparator = (s1, s2) -> {
+ UTF8String convertedS1 = caseConversion(s1);
+ UTF8String convertedS2 = caseConversion(s2);
+ return convertedS1.binaryCompare(convertedS2);
+ };
+ }
+ return new Collation(
+ collationName(),
+ null,
+ comparator,
+ "1.0",
+ s -> (long) caseConversion(s).hashCode(),
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_LCASE_COLLATION_ID
+ );
+ }
+
+ private UTF8String caseConversion(UTF8String s) {
+ if (caseConversion == CaseConversion.LCASE) {
+ return s.toLowerCase();
+ } else if (caseConversion == CaseConversion.UCASE) {
+ return s.toUpperCase();
+ } else {
+ return s;
+ }
+ }
+
+ private String collationName() {
+ StringBuilder builder = new StringBuilder();
+ builder.append("UTF8_BINARY");
+ if (caseConversion != CaseConversion.UNSPECIFIED) {
+ builder.append('_');
+ builder.append(caseConversion.toString());
+ }
+ return builder.toString();
+ }
+ }
+
+ private static class CollationSpecICU extends CollationSpec {
+
+ private static final String[] ICULocaleNames;
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+ private static final String ICUCollatorVersion = "153.120.0.0";
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
+ for (ULocale locale : locales) {
+ if (locale.getVariant().isEmpty()) {
+ String language = locale.getLanguage();
+ assert (!language.isEmpty());
+ StringBuilder builder = new StringBuilder(language);
+ String script = locale.getScript();
+ if (!script.isEmpty()) {
+ builder.append('_');
+ builder.append(script);
+ }
+ String country = locale.getISO3Country();
+ if (!country.isEmpty()) {
+ builder.append('_');
+ builder.append(country);
+ }
+ String localeName = builder.toString();
+ assert (!ICULocaleMap.containsKey(localeName));
+ ICULocaleMap.put(localeName, locale);
+ }
+ }
+ for (String localeName : ICULocaleMap.keySet()) {
+ String localeUppercase = localeName.toUpperCase();
+ assert (!ICULocaleMapUppercase.containsKey(localeUppercase));
+ ICULocaleMapUppercase.put(localeUppercase, localeName);
+ }
+ ICULocaleNames = ICULocaleMap.keySet().toArray(new String[0]);
+ Arrays.sort(ICULocaleNames);
+ assert (ICULocaleNames.length <= (1 << 16));
+ for (int i = 0; i < ICULocaleNames.length; i++) {
+ ICULocaleToId.put(ICULocaleNames[i], i);
+ }
+ }
+
+ public static final int UNICODE_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CS, AccentSensitivity.AS,
+ CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UNICODE_CI_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CI, AccentSensitivity.AS,
+ CaseConversion.UNSPECIFIED).getCollationId();
+
+ private CollationSpecICU(String locale, CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity, CaseConversion caseConversion) {
+ super(locale, caseSensitivity, accentSensitivity, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int lastPos = -1;
+ for (int i = 1; i <= collationName.length(); i++) {
+ String localeName = collationName.substring(0, i);
+ if (ICULocaleMapUppercase.containsKey(localeName)) {
+ lastPos = i;
+ }
+ }
+ if (lastPos == -1) {
+ throw collationInvalidNameException(originalName);
+ } else {
+ int collationId = 0;
+ collationId |= ImplementationProvider.ICU.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+ collationId |= CollationSpec.parseSpecifiers(originalName, collationName, lastPos);
+ String normalizedLocaleName = ICULocaleMapUppercase.get(
+ collationName.substring(0, lastPos));
+ collationId |= ICULocaleToId.get(normalizedLocaleName) << LOCALE_OFFSET;
+ return collationId;
+ }
+ }
+
+ public static CollationSpecICU fromCollationId(int collationId) throws SparkException {
+ int originalCollationId = collationId;
+ int caseSensitivityOrdinal =
+ (collationId >> CASE_SENSITIVITY_OFFSET) & CASE_SENSITIVITY_MASK;
+ int accentSensitivityOrdinal =
+ (collationId >> ACCENT_SENSITIVITY_OFFSET) & ACCENT_SENSITIVITY_MASK;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ int localeOrdinal = (collationId >> LOCALE_OFFSET) & LOCALE_MASK;
+ collationId ^= ImplementationProvider.ICU.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+ collationId ^= caseSensitivityOrdinal << CASE_SENSITIVITY_OFFSET;
+ collationId ^= accentSensitivityOrdinal << ACCENT_SENSITIVITY_OFFSET;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
+ collationId ^= localeOrdinal << LOCALE_OFFSET;
+ if (collationId != 0 || caseConversionOrdinal >= CaseConversion.values().length ||
+ localeOrdinal >= ICULocaleNames.length) {
+ throw SparkException.internalError("Invalid ICU collation id " + originalCollationId);
+ } else {
+ CaseSensitivity caseSensitivity = CaseSensitivity.values()[caseSensitivityOrdinal];
+ AccentSensitivity accentSensitivity =
+ AccentSensitivity.values()[accentSensitivityOrdinal];
+ CaseConversion caseConversion = CaseConversion.values()[caseConversionOrdinal];
+ String locale = ICULocaleNames[localeOrdinal];
+ return new CollationSpecICU(locale, caseSensitivity, accentSensitivity, caseConversion);
+ }
+ }
+
+ @Override
+ protected int getCollationId() {
+ int collationId = 0;
+ collationId |= ImplementationProvider.ICU.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+ collationId |= caseSensitivity.ordinal() << CASE_SENSITIVITY_OFFSET;
+ collationId |= accentSensitivity.ordinal() << ACCENT_SENSITIVITY_OFFSET;
+ collationId |= caseConversion.ordinal() << CASE_CONVERSION_OFFSET;
+ collationId |= ICULocaleToId.get(locale);
+ return collationId;
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ ULocale.Builder builder = new ULocale.Builder();
+ builder.setLocale(ICULocaleMap.get(locale));
+ if (caseSensitivity == CaseSensitivity.CS &&
+ accentSensitivity == AccentSensitivity.AS) {
+ builder.setUnicodeLocaleKeyword("ks", "level3");
+ } else if (caseSensitivity == CaseSensitivity.CS &&
+ accentSensitivity == AccentSensitivity.AI) {
+ builder
+ .setUnicodeLocaleKeyword("ks", "level1")
+ .setUnicodeLocaleKeyword("kc", "true");
+ } else if (caseSensitivity == CaseSensitivity.CI &&
+ accentSensitivity == AccentSensitivity.AS) {
+ builder.setUnicodeLocaleKeyword("ks", "level2");
+ } else if (caseSensitivity == CaseSensitivity.CI &&
+ accentSensitivity == AccentSensitivity.AI) {
+ builder.setUnicodeLocaleKeyword("ks", "level1");
+ }
+ ULocale resultLocale = builder.build();
+ Collator collator = Collator.getInstance(resultLocale);
+ Comparator<UTF8String> comparator = (s1, s2) -> collator.compare(
+ caseConversion(resultLocale, s1), caseConversion(resultLocale, s2));
+ collator.freeze();
+ return new Collation(
+ collationName(),
+ collator,
+ comparator,
+ ICUCollatorVersion,
+ s -> (long) collator.getCollationKey(s.toString()).hashCode(),
+ collationId == UNICODE_COLLATION_ID,
Review Comment:
+1 On @nikolamand-db thinking. We should not make any assumptions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "dbatomic (via GitHub)" <gi...@apache.org>.
dbatomic commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1604916810
##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -152,4 +231,168 @@ class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ig
}
})
}
+
+ test("test collation caching") {
+ Seq(
+ "UTF8_BINARY",
+ "UTF8_BINARY_LCASE",
+ "UTF8_BINARY_UCASE",
+ "UNICODE",
+ "UNICODE_LCASE",
+ "UNICODE_UCASE",
+ "UNICODE_CI",
+ "UNICODE_AI_CI",
+ "UNICODE_AI_CI_LCASE",
+ "UNICODE_AI_CI_UCASE"
+ ).foreach(collationId => {
+ val col1 = fetchCollation(collationId)
+ val col2 = fetchCollation(collationId)
+ assert(col1 eq col2) // reference equality
+ })
+ }
+
+ test("collations with ICU non-root localization") {
+ Seq(
+ // language only
+ "en",
+ "en_CS",
+ "en_CI",
+ "en_AS",
+ "en_AI",
+ "en_LCASE",
+ "en_UCASE",
+ // language + 3-letter country code
+ "en_USA",
+ "en_USA_CS",
+ "en_USA_CI",
+ "en_USA_AS",
+ "en_USA_AI",
+ "en_USA_LCASE",
+ "en_USA_UCASE",
+ // language + script code
+ "sr_Cyrl",
+ "sr_Cyrl_CS",
+ "sr_Cyrl_CI",
+ "sr_Cyrl_AS",
+ "sr_Cyrl_AI",
+ "sr_Cyrl_LCASE",
+ "sr_Cyrl_UCASE",
+ // language + script code + 3-letter country code
+ "sr_Cyrl_SRB",
+ "sr_Cyrl_SRB_CS",
+ "sr_Cyrl_SRB_CI",
+ "sr_Cyrl_SRB_AS",
+ "sr_Cyrl_SRB_AI",
+ "sr_Cyrl_SRB_LCASE",
+ "sr_Cyrl_SRB_UCASE"
+ ).foreach(collationICU => {
+ val col = fetchCollation(collationICU)
+ assert(col.collator.getLocale(ULocale.VALID_LOCALE) != ULocale.ROOT)
+ })
+ }
+
+ test("invalid names of collations with ICU non-root localization") {
+ Seq(
+ "en_US", // must use 3-letter country code
+ "enn",
+ "en_AAA",
+ "en_Something",
+ "en_Something_USA",
+ "en_CI_UNSPECIFIED",
+ "en_USA_UNSPECIFIED",
+ "en_USA_UNSPECIFIED_CI",
+ "en_INDETERMINATE",
+ "en_USA_INDETERMINATE",
+ "en_Latn_USA", // use en_USA instead
+ "en_Cyrl_USA",
+ "en_USA_AAA",
+ "sr_Cyrl_SRB_AAA"
+ ).foreach(collationName => {
+ val error = intercept[SparkException] {
+ fetchCollation(collationName)
+ }
+
+ assert(error.getErrorClass === "COLLATION_INVALID_NAME")
+ assert(error.getMessageParameters.asScala === Map("collationName" -> collationName))
+ })
+ }
+
+ test("collations name normalization for ICU non-root localization") {
+ Seq(
+ ("en_USA", "en_USA"),
+ ("en_CS", "en"),
+ ("en_AS", "en"),
+ ("en_CS_AS", "en"),
+ ("en_AS_CS", "en"),
+ ("en_CI", "en_CI"),
+ ("en_AI", "en_AI"),
+ ("en_AI_CI", "en_CI_AI"),
+ ("en_USA_AI_CI", "en_USA_CI_AI"),
+ ("en_USA_LCASE_AI_CI", "en_USA_CI_AI_LCASE"),
+ ("en_USA_LCASE_CI_AI", "en_USA_CI_AI_LCASE"),
+ ("en_USA_AI_LCASE_CI", "en_USA_CI_AI_LCASE"),
+ ("en_USA_CI_LCASE_AI", "en_USA_CI_AI_LCASE"),
+ // randomized case
+ ("EN_USA", "en_USA"),
+ ("eN_usA_ci_uCASe_aI", "en_USA_CI_AI_UCASE"),
+ ("SR_CYRL", "sr_Cyrl"),
+ ("sr_cyrl_srb", "sr_Cyrl_SRB"),
+ ("sR_cYRl_sRb", "sr_Cyrl_SRB")
+ ).foreach {
+ case (name, normalized) =>
+ val col = fetchCollation(name)
+ assert(col.collationName == normalized)
+ }
+ }
+
+ test("invalid collationId") {
+ val badCollationIds = Seq(
+ -1, // user-defined collation range
+ 1 << 31, // user-defined collation range
+ 1 << 12, // utf8-binary mandatory zero bit 12 breach
+ 1 << 13, // utf8-binary mandatory zero bit 13 breach
+ 1 << 14, // utf8-binary mandatory zero bit 14 breach
+ 1 << 15, // utf8-binary mandatory zero bit 15 breach
+ 1 << 16, // utf8-binary mandatory zero bit 16 breach
+ 1 << 17, // utf8-binary mandatory zero bit 17 breach
+ 1 << 18, // utf8-binary mandatory zero bit 18 breach
+ 1 << 19, // utf8-binary mandatory zero bit 19 breach
+ 1 << 20, // utf8-binary mandatory zero bit 20 breach
+ 1 << 23, // utf8-binary mandatory zero bit 23 breach
+ 1 << 24, // utf8-binary mandatory zero bit 24 breach
+ 1 << 25, // utf8-binary mandatory zero bit 25 breach
+ 1 << 26, // utf8-binary mandatory zero bit 26 breach
+ (1 << 29) | (1 << 12), // ICU mandatory zero bit 12 breach
+ (1 << 29) | (1 << 13), // ICU mandatory zero bit 13 breach
+ (1 << 29) | (1 << 14), // ICU mandatory zero bit 14 breach
+ (1 << 29) | (1 << 15), // ICU mandatory zero bit 15 breach
+ (1 << 29) | (1 << 16), // ICU mandatory zero bit 16 breach
+ (1 << 29) | (1 << 17), // ICU mandatory zero bit 17 breach
+ (1 << 29) | (1 << 18), // ICU mandatory zero bit 18 breach
+ (1 << 29) | (1 << 19), // ICU mandatory zero bit 19 breach
+ (1 << 29) | (1 << 20), // ICU mandatory zero bit 20 breach
+ (1 << 29) | (1 << 23), // ICU mandatory zero bit 23 breach
+ (1 << 29) | (1 << 24), // ICU mandatory zero bit 24 breach
+ (1 << 29) | (1 << 25), // ICU mandatory zero bit 25 breach
+ (1 << 29) | (1 << 26), // ICU mandatory zero bit 26 breach
+ 123, // utf8-binary with non-zero locale id
+ (1 << 29) | (1 << 12), // ICU with invalid locale id
+ (1 << 29) | 0xFFFF, // ICU with invalid locale id
+ (1 << 23) | (1 << 22), // utf8-binary with invalid case conversion
+ (1 << 29) | (1 << 23) | (1 << 22), // ICU with invalid case conversion
+ 1 << 27, // utf8-binary accent-insensitive
+ 1 << 28, // utf8-binary case-insensitive
+ INDETERMINATE_COLLATION_ID, // indeterminate collation
+ 1 << 30, // indeterminate collation
+ (1 << 30) | (1 << 29) // invalid implementation provider
+ )
+ badCollationIds.foreach(collationId => {
+ val e = intercept[SparkException](fetchCollationUnsafe(collationId))
+ // user cannot specify collation ids directly so this is an internal error
+ assert(e.getErrorClass === "INTERNAL_ERROR")
+ })
+ badCollationIds.foreach(collationId => {
+ assert(fetchCollation(collationId).collationName == "UTF8_BINARY")
+ })
Review Comment:
+1. Let's not do this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606732800
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected static int parseSpecifiers(
+ String originalName, String collationName, int splitStart) throws SparkException {
+ int specifiers = 0;
+ String[] parts = collationName.substring(splitStart).split("_");
+ for (String part : parts) {
+ if (!part.isEmpty()) {
+ if (part.equals("UNSPECIFIED") || part.equals("INDETERMINATE")) {
+ throw collationInvalidNameException(originalName);
+ } else if (Arrays.stream(CaseSensitivity.values()).anyMatch(
+ (s) -> s.toString().equals(part))) {
+ specifiers |=
+ CaseSensitivity.valueOf(part).ordinal() << CASE_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(AccentSensitivity.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ AccentSensitivity.valueOf(part).ordinal() << ACCENT_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(CaseConversion.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ CaseConversion.valueOf(part).ordinal() << CASE_CONVERSION_OFFSET;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+ }
+ return specifiers;
+ }
+
+ protected abstract int getCollationId();
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ public static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.LCASE).getCollationId();
+ public static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).buildCollation();
+
+ private CollationSpecUTF8Binary(CaseConversion caseConversion) {
+ super(null, CaseSensitivity.CS, AccentSensitivity.AS, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int collationId = 0;
+ int specifiers = CollationSpec.parseSpecifiers(
+ originalName, collationName, "UTF8_BINARY".length());
+ if ((specifiers & ~(CASE_CONVERSION_MASK << CASE_CONVERSION_OFFSET)) != 0) {
+ throw collationInvalidNameException(originalName);
+ }
+ collationId |= specifiers;
+ return collationId;
+ }
+
+ @Override
+ protected int getCollationId() {
+ int collationId = 0;
+ collationId |= caseConversion.ordinal() << CASE_CONVERSION_OFFSET;
+ return collationId;
+ }
+
+ public static CollationSpecUTF8Binary fromCollationId(int collationId)
+ throws SparkException {
+ int originalCollationId = collationId;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
+ if (collationId != 0 || caseConversionOrdinal >= CaseConversion.values().length) {
+ throw SparkException.internalError("Invalid UTF8_BINARY collation id " +
+ originalCollationId);
+ } else {
+ CaseConversion caseConversion = CaseConversion.values()[caseConversionOrdinal];
+ return new CollationSpecUTF8Binary(caseConversion);
+ }
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ Comparator<UTF8String> comparator;
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ comparator = UTF8String::binaryCompare;
+ } else if (collationId == UTF8_BINARY_LCASE_COLLATION_ID) {
+ comparator = UTF8String::compareLowerCase;
+ } else {
+ comparator = (s1, s2) -> {
+ UTF8String convertedS1 = caseConversion(s1);
+ UTF8String convertedS2 = caseConversion(s2);
+ return convertedS1.binaryCompare(convertedS2);
+ };
+ }
+ return new Collation(
+ collationName(),
+ null,
+ comparator,
+ "1.0",
+ s -> (long) caseConversion(s).hashCode(),
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_LCASE_COLLATION_ID
+ );
+ }
+
+ private UTF8String caseConversion(UTF8String s) {
+ if (caseConversion == CaseConversion.LCASE) {
+ return s.toLowerCase();
+ } else if (caseConversion == CaseConversion.UCASE) {
+ return s.toUpperCase();
+ } else {
+ return s;
+ }
+ }
+
+ private String collationName() {
+ StringBuilder builder = new StringBuilder();
+ builder.append("UTF8_BINARY");
+ if (caseConversion != CaseConversion.UNSPECIFIED) {
+ builder.append('_');
+ builder.append(caseConversion.toString());
+ }
+ return builder.toString();
+ }
+ }
+
+ private static class CollationSpecICU extends CollationSpec {
+
+ private static final String[] ICULocaleNames;
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+ private static final String ICUCollatorVersion = "153.120.0.0";
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
+ for (ULocale locale : locales) {
+ if (locale.getVariant().isEmpty()) {
+ String language = locale.getLanguage();
+ assert (!language.isEmpty());
+ StringBuilder builder = new StringBuilder(language);
+ String script = locale.getScript();
+ if (!script.isEmpty()) {
+ builder.append('_');
+ builder.append(script);
+ }
+ String country = locale.getISO3Country();
+ if (!country.isEmpty()) {
+ builder.append('_');
+ builder.append(country);
+ }
+ String localeName = builder.toString();
+ assert (!ICULocaleMap.containsKey(localeName));
+ ICULocaleMap.put(localeName, locale);
+ }
+ }
+ for (String localeName : ICULocaleMap.keySet()) {
+ String localeUppercase = localeName.toUpperCase();
+ assert (!ICULocaleMapUppercase.containsKey(localeUppercase));
+ ICULocaleMapUppercase.put(localeUppercase, localeName);
+ }
+ ICULocaleNames = ICULocaleMap.keySet().toArray(new String[0]);
+ Arrays.sort(ICULocaleNames);
+ assert (ICULocaleNames.length <= (1 << 16));
+ for (int i = 0; i < ICULocaleNames.length; i++) {
+ ICULocaleToId.put(ICULocaleNames[i], i);
+ }
+ }
+
+ public static final int UNICODE_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CS, AccentSensitivity.AS,
+ CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UNICODE_CI_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CI, AccentSensitivity.AS,
+ CaseConversion.UNSPECIFIED).getCollationId();
+
+ private CollationSpecICU(String locale, CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity, CaseConversion caseConversion) {
+ super(locale, caseSensitivity, accentSensitivity, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int lastPos = -1;
+ for (int i = 1; i <= collationName.length(); i++) {
+ String localeName = collationName.substring(0, i);
+ if (ICULocaleMapUppercase.containsKey(localeName)) {
+ lastPos = i;
+ }
+ }
+ if (lastPos == -1) {
+ throw collationInvalidNameException(originalName);
+ } else {
+ int collationId = 0;
+ collationId |= ImplementationProvider.ICU.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+ collationId |= CollationSpec.parseSpecifiers(originalName, collationName, lastPos);
+ String normalizedLocaleName = ICULocaleMapUppercase.get(
+ collationName.substring(0, lastPos));
+ collationId |= ICULocaleToId.get(normalizedLocaleName) << LOCALE_OFFSET;
+ return collationId;
+ }
+ }
+
+ public static CollationSpecICU fromCollationId(int collationId) throws SparkException {
+ int originalCollationId = collationId;
+ int caseSensitivityOrdinal =
+ (collationId >> CASE_SENSITIVITY_OFFSET) & CASE_SENSITIVITY_MASK;
+ int accentSensitivityOrdinal =
+ (collationId >> ACCENT_SENSITIVITY_OFFSET) & ACCENT_SENSITIVITY_MASK;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ int localeOrdinal = (collationId >> LOCALE_OFFSET) & LOCALE_MASK;
+ collationId ^= ImplementationProvider.ICU.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+ collationId ^= caseSensitivityOrdinal << CASE_SENSITIVITY_OFFSET;
+ collationId ^= accentSensitivityOrdinal << ACCENT_SENSITIVITY_OFFSET;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
+ collationId ^= localeOrdinal << LOCALE_OFFSET;
+ if (collationId != 0 || caseConversionOrdinal >= CaseConversion.values().length ||
Review Comment:
Outdated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606746841
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,438 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ public static int collationNameToId(String originalCollationName) throws SparkException {
+ String collationName = originalCollationName.toUpperCase();
+ try {
+ if (collationName.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName);
+ }
+ } catch (SparkException e) {
+ throw new SparkException(
+ "COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", originalCollationName)),
+ e);
+ }
+ }
+
+ protected static int parseSpecifiers(String specString) throws SparkException {
+ int specifiers = 0;
+ String[] parts = specString.split("_");
+ for (String part : parts) {
+ if (!part.isEmpty()) {
+ if (part.equals("UNSPECIFIED") || part.equals("INDETERMINATE")) {
+ throw new SparkException(part + " collation specifier reserved for internal use");
Review Comment:
Added tests to cover such problematic edge cases https://github.com/apache/spark/pull/46180/files#diff-9c12d32db9d55dd6ecb5b10f2fc57c7ba7de7275cab57bf157fa42cbc09f3876R386-R397.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "mkaravel (via GitHub)" <gi...@apache.org>.
mkaravel commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1607446081
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -288,13 +338,24 @@ private static int collationNameToId(String collationName) throws SparkException
private static class CollationSpecUTF8Binary extends CollationSpec {
- private static final int CASE_SENSITIVITY_OFFSET = 0;
- private static final int CASE_SENSITIVITY_MASK = 0b1;
-
+ /**
+ * Bit 0 in collation id having value
+ * 0 for plain UTF8_BINARY and 1 for UTF8_BINARY_LCASE collation.
Review Comment:
Please use the full extent of the 100 characters per line that you are allowed for comments.
I am not suggesting to pack everything together all the time, but in cases like the one above, I would have expected to have the first line "filled" before going to the second line.
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -221,41 +231,80 @@ public Collation(
*/
private abstract static class CollationSpec {
+ /**
+ * Bit 30 in collation id having value 0 for predefined and 1 for user-defined collation.
+ */
private enum DefinitionOrigin {
PREDEFINED, USER_DEFINED
}
+ /**
+ * Bit 29 in collation id having value
+ * 0 for UTF8_BINARY family and 1 for ICU family of collations.
+ */
protected enum ImplementationProvider {
UTF8_BINARY, ICU
}
+ /**
+ * Offset in binary collation id layout.
+ */
private static final int DEFINITION_ORIGIN_OFFSET = 30;
+
+ /**
+ * Bitmask corresponding to width in bits in binary collation id layout.
+ */
private static final int DEFINITION_ORIGIN_MASK = 0b1;
+
+ /**
+ * Offset in binary collation id layout.
+ */
protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+
+ /**
+ * Bitmask corresponding to width in bits in binary collation id layout.
+ */
protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b1;
private static final int INDETERMINATE_COLLATION_ID = -1;
+ /**
+ * Thread-safe cache mapping collation ids to corresponding `Collation` instances.
+ * We add entries to this cache lazily as new `Collation` instances are requested.
+ */
private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+ /**
+ * Utility function to retrieve `ImplementationProvider` enum instance from collation id.
+ */
private static ImplementationProvider getImplementationProvider(int collationId) {
return ImplementationProvider.values()[SpecifierUtils.getSpecValue(collationId,
IMPLEMENTATION_PROVIDER_OFFSET, IMPLEMENTATION_PROVIDER_MASK)];
}
+ /**
+ * Utility function to retrieve `DefinitionOrigin` enum instance from collation id.
+ */
private static DefinitionOrigin getDefinitionOrigin(int collationId) {
return DefinitionOrigin.values()[SpecifierUtils.getSpecValue(collationId,
DEFINITION_ORIGIN_OFFSET, DEFINITION_ORIGIN_MASK)];
}
+ /**
+ * Main entry point for retrieving `Collation` instance from collation id.
+ */
private static Collation fetchCollation(int collationId) {
+ // User-defined collations and INDETERMINATE collations cannot produce `Collation` instance
Review Comment:
```suggestion
// User-defined collations and INDETERMINATE collations cannot produce a `Collation` instance.
```
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -333,77 +397,136 @@ private static CollationSpecUTF8Binary fromCollationId(int collationId) {
@Override
protected Collation buildCollation() {
if (collationId == UTF8_BINARY_COLLATION_ID) {
- return new Collation("UTF8_BINARY", null, UTF8String::binaryCompare, "1.0",
- s -> (long) s.hashCode(), true, true, false);
+ return new Collation(
+ "UTF8_BINARY",
+ PROVIDER_SPARK,
+ null,
+ UTF8String::binaryCompare,
+ "1.0",
+ s -> (long) s.hashCode(),
+ true,
+ true,
+ false);
Review Comment:
Way better! Thank you!
Since we are in Java and not Scala, I suggest using comments for the boolean constants. Something like:
```java
/*isBinaryCollation=*/ true,
```
##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -152,4 +219,238 @@ class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ig
}
})
}
+
+ test("test collation caching") {
+ Seq(
+ "UTF8_BINARY",
+ "UTF8_BINARY_LCASE",
+ "UNICODE",
+ "UNICODE_CI",
+ "UNICODE_AI",
+ "UNICODE_CI_AI",
+ "UNICODE_AI_CI"
+ ).foreach(collationId => {
+ val col1 = fetchCollation(collationId)
+ val col2 = fetchCollation(collationId)
+ assert(col1 eq col2) // Check for reference equality.
+ })
+ }
+
+ test("collations with ICU non-root localization") {
+ Seq(
+ // Language only.
+ "en",
+ "en_CS",
+ "en_CI",
+ "en_AS",
+ "en_AI",
+ // Language + 3-letter country code.
+ "en_USA",
+ "en_USA_CS",
+ "en_USA_CI",
+ "en_USA_AS",
+ "en_USA_AI",
+ // Language + script code.
+ "sr_Cyrl",
+ "sr_Cyrl_CS",
+ "sr_Cyrl_CI",
+ "sr_Cyrl_AS",
+ "sr_Cyrl_AI",
+ // Language + script code + 3-letter country code.
+ "sr_Cyrl_SRB",
+ "sr_Cyrl_SRB_CS",
+ "sr_Cyrl_SRB_CI",
+ "sr_Cyrl_SRB_AS",
+ "sr_Cyrl_SRB_AI"
+ ).foreach(collationICU => {
+ val col = fetchCollation(collationICU)
+ assert(col.collator.getLocale(ULocale.VALID_LOCALE) != ULocale.ROOT)
+ })
+ }
+
+ test("invalid names of collations with ICU non-root localization") {
+ Seq(
+ "en_US", // Must use 3-letter country code
+ "enn",
+ "en_AAA",
+ "en_Something",
+ "en_Something_USA",
+ "en_LCASE",
+ "en_UCASE",
+ "en_CI_LCASE",
+ "en_CI_UCASE",
+ "en_CI_UNSPECIFIED",
+ "en_USA_UNSPECIFIED",
+ "en_USA_UNSPECIFIED_CI",
+ "en_INDETERMINATE",
+ "en_USA_INDETERMINATE",
+ "en_Latn_USA", // Use en_USA instead.
+ "en_Cyrl_USA",
+ "en_USA_AAA",
+ "sr_Cyrl_SRB_AAA",
+ // Invalid ordering of language, script and country code.
+ "USA_en",
+ "sr_SRB_Cyrl",
+ "SRB_sr",
+ "SRB_sr_Cyrl",
+ "SRB_Cyrl_sr",
+ "Cyrl_sr",
+ "Cyrl_sr_SRB",
+ "Cyrl_SRB_sr",
+ // Collation specifiers in the middle of locale.
+ "CI_en",
+ "USA_CI_en",
+ "en_CI_USA",
+ "CI_sr_Cyrl_SRB",
+ "sr_CI_Cyrl_SRB",
+ "sr_Cyrl_CI_SRB",
+ "CI_Cyrl_sr",
+ "Cyrl_CI_sr",
+ "Cyrl_CI_sr_SRB",
+ "Cyrl_sr_CI_SRB"
+ ).foreach(collationName => {
+ val error = intercept[SparkException] {
+ fetchCollation(collationName)
+ }
+
+ assert(error.getErrorClass === "COLLATION_INVALID_NAME")
+ assert(error.getMessageParameters.asScala === Map("collationName" -> collationName))
+ })
+ }
+
+ test("collations name normalization for ICU non-root localization") {
+ Seq(
+ ("en_USA", "en_USA"),
+ ("en_CS", "en"),
+ ("en_AS", "en"),
+ ("en_CS_AS", "en"),
+ ("en_AS_CS", "en"),
+ ("en_CI", "en_CI"),
+ ("en_AI", "en_AI"),
+ ("en_AI_CI", "en_CI_AI"),
+ ("en_CI_AI", "en_CI_AI"),
+ ("en_CS_AI", "en_AI"),
+ ("en_AI_CS", "en_AI"),
+ ("en_CI_AS", "en_CI"),
+ ("en_AS_CI", "en_CI"),
+ ("en_USA_AI_CI", "en_USA_CI_AI"),
+ // Randomized case.
+ ("EN_USA", "en_USA"),
+ ("SR_CYRL", "sr_Cyrl"),
+ ("sr_cyrl_srb", "sr_Cyrl_SRB"),
+ ("sR_cYRl_sRb", "sr_Cyrl_SRB")
+ ).foreach {
+ case (name, normalized) =>
+ val col = fetchCollation(name)
+ assert(col.collationName == normalized)
+ }
+ }
+
+ test("invalid collationId") {
+ val badCollationIds = Seq(
+ INDETERMINATE_COLLATION_ID, // Indeterminate collation.
+ 1 << 30, // User-defined collation range.
+ (1 << 30) | 1, // User-defined collation range.
+ (1 << 30) | (1 << 29), // User-defined collation range.
+ 1 << 1, // UTF8_BINARY mandatory zero bit 1 breach.
+ 1 << 2, // UTF8_BINARY mandatory zero bit 2 breach.
+ 1 << 3, // UTF8_BINARY mandatory zero bit 3 breach.
+ 1 << 4, // UTF8_BINARY mandatory zero bit 4 breach.
+ 1 << 5, // UTF8_BINARY mandatory zero bit 5 breach.
+ 1 << 6, // UTF8_BINARY mandatory zero bit 6 breach.
+ 1 << 7, // UTF8_BINARY mandatory zero bit 7 breach.
+ 1 << 8, // UTF8_BINARY mandatory zero bit 8 breach.
+ 1 << 9, // UTF8_BINARY mandatory zero bit 9 breach.
+ 1 << 10, // UTF8_BINARY mandatory zero bit 10 breach.
+ 1 << 11, // UTF8_BINARY mandatory zero bit 11 breach.
+ 1 << 12, // UTF8_BINARY mandatory zero bit 12 breach.
+ 1 << 13, // UTF8_BINARY mandatory zero bit 13 breach.
+ 1 << 14, // UTF8_BINARY mandatory zero bit 14 breach.
+ 1 << 15, // UTF8_BINARY mandatory zero bit 15 breach.
+ 1 << 16, // UTF8_BINARY mandatory zero bit 16 breach.
+ 1 << 17, // UTF8_BINARY mandatory zero bit 17 breach.
+ 1 << 18, // UTF8_BINARY mandatory zero bit 18 breach.
+ 1 << 19, // UTF8_BINARY mandatory zero bit 19 breach.
+ 1 << 20, // UTF8_BINARY mandatory zero bit 20 breach.
+ 1 << 23, // UTF8_BINARY mandatory zero bit 23 breach.
+ 1 << 24, // UTF8_BINARY mandatory zero bit 24 breach.
+ 1 << 25, // UTF8_BINARY mandatory zero bit 25 breach.
+ 1 << 26, // UTF8_BINARY mandatory zero bit 26 breach.
+ 1 << 27, // UTF8_BINARY mandatory zero bit 27 breach.
+ 1 << 28, // UTF8_BINARY mandatory zero bit 28 breach.
+ (1 << 29) | (1 << 12), // ICU mandatory zero bit 12 breach.
+ (1 << 29) | (1 << 13), // ICU mandatory zero bit 13 breach.
+ (1 << 29) | (1 << 14), // ICU mandatory zero bit 14 breach.
+ (1 << 29) | (1 << 15), // ICU mandatory zero bit 15 breach.
+ (1 << 29) | (1 << 18), // ICU mandatory zero bit 18 breach.
+ (1 << 29) | (1 << 19), // ICU mandatory zero bit 19 breach.
+ (1 << 29) | (1 << 20), // ICU mandatory zero bit 20 breach.
+ (1 << 29) | (1 << 21), // ICU mandatory zero bit 21 breach.
+ (1 << 29) | (1 << 22), // ICU mandatory zero bit 22 breach.
+ (1 << 29) | (1 << 23), // ICU mandatory zero bit 23 breach.
+ (1 << 29) | (1 << 24), // ICU mandatory zero bit 24 breach.
+ (1 << 29) | (1 << 25), // ICU mandatory zero bit 25 breach.
+ (1 << 29) | (1 << 26), // ICU mandatory zero bit 26 breach.
+ (1 << 29) | (1 << 27), // ICU mandatory zero bit 27 breach.
+ (1 << 29) | (1 << 28), // ICU mandatory zero bit 28 breach.
+ (1 << 29) | 0xFFFF // ICU with invalid locale id.
+ )
+ badCollationIds.foreach(collationId => {
+ // Assumptions about collation id will break and assert statement will fail.
+ intercept[AssertionError](fetchCollation(collationId))
+ })
+ }
+
+ test("repeated specifiers in collation name") {
Review Comment:
```suggestion
test("repeated and/or incompatible specifiers in collation name") {
```
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -179,35 +179,45 @@ public Collation(
* Classes of collations are differentiated by most significant 3 bits (bit 31, 30 and 29),
* bit 31 being most significant and bit 0 being least significant.
* ---
+ * General collation id binary layout:
Review Comment:
```suggestion
* General collation ID binary layout:
```
This is a nit. Would be good though to consistently use "ID" in the comments (code is different).
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
Review Comment:
SG. Let's have this in mind for user-defined collations.
##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -152,4 +219,218 @@ class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ig
}
})
}
+
+ test("test collation caching") {
+ Seq(
+ "UTF8_BINARY",
+ "UTF8_BINARY_LCASE",
+ "UNICODE",
+ "UNICODE_CI",
+ "UNICODE_AI",
+ "UNICODE_CI_AI",
+ "UNICODE_AI_CI"
+ ).foreach(collationId => {
+ val col1 = fetchCollation(collationId)
+ val col2 = fetchCollation(collationId)
+ assert(col1 eq col2) // reference equality
+ })
+ }
+
+ test("collations with ICU non-root localization") {
+ Seq(
+ // language only
+ "en",
+ "en_CS",
+ "en_CI",
+ "en_AS",
+ "en_AI",
+ // language + 3-letter country code
+ "en_USA",
+ "en_USA_CS",
+ "en_USA_CI",
+ "en_USA_AS",
+ "en_USA_AI",
+ // language + script code
+ "sr_Cyrl",
+ "sr_Cyrl_CS",
+ "sr_Cyrl_CI",
+ "sr_Cyrl_AS",
+ "sr_Cyrl_AI",
+ // language + script code + 3-letter country code
+ "sr_Cyrl_SRB",
+ "sr_Cyrl_SRB_CS",
+ "sr_Cyrl_SRB_CI",
+ "sr_Cyrl_SRB_AS",
+ "sr_Cyrl_SRB_AI"
+ ).foreach(collationICU => {
+ val col = fetchCollation(collationICU)
+ assert(col.collator.getLocale(ULocale.VALID_LOCALE) != ULocale.ROOT)
+ })
+ }
+
+ test("invalid names of collations with ICU non-root localization") {
Review Comment:
I missed these test cases. I made a suggestion for the test name that makes it more clear that we do not only test repeated specifiers, but also incompatible ones.
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -179,35 +179,45 @@ public Collation(
* Classes of collations are differentiated by most significant 3 bits (bit 31, 30 and 29),
* bit 31 being most significant and bit 0 being least significant.
* ---
+ * General collation id binary layout:
Review Comment:
Also below of course.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "bart-samwel (via GitHub)" <gi...@apache.org>.
bart-samwel commented on PR #46180:
URL: https://github.com/apache/spark/pull/46180#issuecomment-2079429882
> User can use collation specifiers in any order except of locale which is mandatory and must go first. There is a one-to-one mapping between collation ids and collation names defined in CollationFactory.
Does that mean that there are multiple collation names that have identical meaning but different names (with the specifiers in different order)? That seems confusing and error prone. Are we at least normalizing them internally somehow?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #46180: [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers
URL: https://github.com/apache/spark/pull/46180
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "dbatomic (via GitHub)" <gi...@apache.org>.
dbatomic commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1592440194
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,438 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ public static int collationNameToId(String originalCollationName) throws SparkException {
+ String collationName = originalCollationName.toUpperCase();
+ try {
+ if (collationName.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName);
+ }
+ } catch (SparkException e) {
Review Comment:
Why do we need to catch `SparkException` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "dbatomic (via GitHub)" <gi...@apache.org>.
dbatomic commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1592442858
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -217,17 +581,12 @@ public static StringSearch getStringSearch(
* Returns if the given collationName is valid one.
*/
public static boolean isValidCollation(String collationName) {
- return collationNameToIdMap.containsKey(collationName.toUpperCase());
- }
-
- /**
- * Returns closest valid name to collationName
- */
- public static String getClosestCollation(String collationName) {
- Collation suggestion = Collections.min(List.of(collationTable), Comparator.comparingInt(
- c -> UTF8String.fromString(c.collationName).levenshteinDistance(
- UTF8String.fromString(collationName.toUpperCase()))));
- return suggestion.collationName;
+ try {
Review Comment:
Again, catching (generic) exception as control flow isn't a good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "uros-db (via GitHub)" <gi...@apache.org>.
uros-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1593724507
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected static int parseSpecifiers(
+ String originalName, String collationName, int splitStart) throws SparkException {
+ int specifiers = 0;
+ String[] parts = collationName.substring(splitStart).split("_");
+ for (String part : parts) {
+ if (!part.isEmpty()) {
+ if (part.equals("UNSPECIFIED") || part.equals("INDETERMINATE")) {
+ throw collationInvalidNameException(originalName);
+ } else if (Arrays.stream(CaseSensitivity.values()).anyMatch(
+ (s) -> s.toString().equals(part))) {
+ specifiers |=
+ CaseSensitivity.valueOf(part).ordinal() << CASE_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(AccentSensitivity.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ AccentSensitivity.valueOf(part).ordinal() << ACCENT_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(CaseConversion.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ CaseConversion.valueOf(part).ordinal() << CASE_CONVERSION_OFFSET;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+ }
+ return specifiers;
+ }
+
+ protected abstract int getCollationId();
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ public static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.LCASE).getCollationId();
+ public static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).buildCollation();
+
+ private CollationSpecUTF8Binary(CaseConversion caseConversion) {
+ super(null, CaseSensitivity.CS, AccentSensitivity.AS, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int collationId = 0;
+ int specifiers = CollationSpec.parseSpecifiers(
+ originalName, collationName, "UTF8_BINARY".length());
+ if ((specifiers & ~(CASE_CONVERSION_MASK << CASE_CONVERSION_OFFSET)) != 0) {
+ throw collationInvalidNameException(originalName);
+ }
+ collationId |= specifiers;
+ return collationId;
+ }
+
+ @Override
+ protected int getCollationId() {
+ int collationId = 0;
+ collationId |= caseConversion.ordinal() << CASE_CONVERSION_OFFSET;
+ return collationId;
+ }
+
+ public static CollationSpecUTF8Binary fromCollationId(int collationId)
+ throws SparkException {
+ int originalCollationId = collationId;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
+ if (collationId != 0 || caseConversionOrdinal >= CaseConversion.values().length) {
+ throw SparkException.internalError("Invalid UTF8_BINARY collation id " +
+ originalCollationId);
+ } else {
+ CaseConversion caseConversion = CaseConversion.values()[caseConversionOrdinal];
+ return new CollationSpecUTF8Binary(caseConversion);
+ }
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ Comparator<UTF8String> comparator;
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ comparator = UTF8String::binaryCompare;
+ } else if (collationId == UTF8_BINARY_LCASE_COLLATION_ID) {
+ comparator = UTF8String::compareLowerCase;
+ } else {
+ comparator = (s1, s2) -> {
+ UTF8String convertedS1 = caseConversion(s1);
+ UTF8String convertedS2 = caseConversion(s2);
+ return convertedS1.binaryCompare(convertedS2);
+ };
+ }
+ return new Collation(
+ collationName(),
+ null,
+ comparator,
+ "1.0",
+ s -> (long) caseConversion(s).hashCode(),
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_LCASE_COLLATION_ID
+ );
+ }
+
+ private UTF8String caseConversion(UTF8String s) {
+ if (caseConversion == CaseConversion.LCASE) {
+ return s.toLowerCase();
+ } else if (caseConversion == CaseConversion.UCASE) {
+ return s.toUpperCase();
+ } else {
+ return s;
+ }
+ }
+
+ private String collationName() {
+ StringBuilder builder = new StringBuilder();
+ builder.append("UTF8_BINARY");
+ if (caseConversion != CaseConversion.UNSPECIFIED) {
+ builder.append('_');
+ builder.append(caseConversion.toString());
+ }
+ return builder.toString();
+ }
+ }
+
+ private static class CollationSpecICU extends CollationSpec {
+
+ private static final String[] ICULocaleNames;
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+ private static final String ICUCollatorVersion = "153.120.0.0";
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
+ for (ULocale locale : locales) {
+ if (locale.getVariant().isEmpty()) {
+ String language = locale.getLanguage();
+ assert (!language.isEmpty());
+ StringBuilder builder = new StringBuilder(language);
+ String script = locale.getScript();
+ if (!script.isEmpty()) {
+ builder.append('_');
+ builder.append(script);
+ }
+ String country = locale.getISO3Country();
+ if (!country.isEmpty()) {
+ builder.append('_');
+ builder.append(country);
+ }
+ String localeName = builder.toString();
+ assert (!ICULocaleMap.containsKey(localeName));
+ ICULocaleMap.put(localeName, locale);
+ }
+ }
+ for (String localeName : ICULocaleMap.keySet()) {
+ String localeUppercase = localeName.toUpperCase();
+ assert (!ICULocaleMapUppercase.containsKey(localeUppercase));
+ ICULocaleMapUppercase.put(localeUppercase, localeName);
+ }
+ ICULocaleNames = ICULocaleMap.keySet().toArray(new String[0]);
+ Arrays.sort(ICULocaleNames);
+ assert (ICULocaleNames.length <= (1 << 16));
+ for (int i = 0; i < ICULocaleNames.length; i++) {
+ ICULocaleToId.put(ICULocaleNames[i], i);
+ }
+ }
+
+ public static final int UNICODE_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CS, AccentSensitivity.AS,
+ CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UNICODE_CI_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CI, AccentSensitivity.AS,
+ CaseConversion.UNSPECIFIED).getCollationId();
+
+ private CollationSpecICU(String locale, CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity, CaseConversion caseConversion) {
+ super(locale, caseSensitivity, accentSensitivity, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int lastPos = -1;
+ for (int i = 1; i <= collationName.length(); i++) {
+ String localeName = collationName.substring(0, i);
+ if (ICULocaleMapUppercase.containsKey(localeName)) {
+ lastPos = i;
+ }
+ }
+ if (lastPos == -1) {
+ throw collationInvalidNameException(originalName);
+ } else {
+ int collationId = 0;
+ collationId |= ImplementationProvider.ICU.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+ collationId |= CollationSpec.parseSpecifiers(originalName, collationName, lastPos);
+ String normalizedLocaleName = ICULocaleMapUppercase.get(
+ collationName.substring(0, lastPos));
+ collationId |= ICULocaleToId.get(normalizedLocaleName) << LOCALE_OFFSET;
+ return collationId;
+ }
+ }
+
+ public static CollationSpecICU fromCollationId(int collationId) throws SparkException {
+ int originalCollationId = collationId;
+ int caseSensitivityOrdinal =
+ (collationId >> CASE_SENSITIVITY_OFFSET) & CASE_SENSITIVITY_MASK;
+ int accentSensitivityOrdinal =
+ (collationId >> ACCENT_SENSITIVITY_OFFSET) & ACCENT_SENSITIVITY_MASK;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ int localeOrdinal = (collationId >> LOCALE_OFFSET) & LOCALE_MASK;
+ collationId ^= ImplementationProvider.ICU.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+ collationId ^= caseSensitivityOrdinal << CASE_SENSITIVITY_OFFSET;
+ collationId ^= accentSensitivityOrdinal << ACCENT_SENSITIVITY_OFFSET;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
+ collationId ^= localeOrdinal << LOCALE_OFFSET;
+ if (collationId != 0 || caseConversionOrdinal >= CaseConversion.values().length ||
+ localeOrdinal >= ICULocaleNames.length) {
+ throw SparkException.internalError("Invalid ICU collation id " + originalCollationId);
+ } else {
+ CaseSensitivity caseSensitivity = CaseSensitivity.values()[caseSensitivityOrdinal];
+ AccentSensitivity accentSensitivity =
+ AccentSensitivity.values()[accentSensitivityOrdinal];
+ CaseConversion caseConversion = CaseConversion.values()[caseConversionOrdinal];
+ String locale = ICULocaleNames[localeOrdinal];
+ return new CollationSpecICU(locale, caseSensitivity, accentSensitivity, caseConversion);
+ }
+ }
+
+ @Override
+ protected int getCollationId() {
+ int collationId = 0;
+ collationId |= ImplementationProvider.ICU.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+ collationId |= caseSensitivity.ordinal() << CASE_SENSITIVITY_OFFSET;
+ collationId |= accentSensitivity.ordinal() << ACCENT_SENSITIVITY_OFFSET;
+ collationId |= caseConversion.ordinal() << CASE_CONVERSION_OFFSET;
+ collationId |= ICULocaleToId.get(locale);
+ return collationId;
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ ULocale.Builder builder = new ULocale.Builder();
+ builder.setLocale(ICULocaleMap.get(locale));
+ if (caseSensitivity == CaseSensitivity.CS &&
+ accentSensitivity == AccentSensitivity.AS) {
+ builder.setUnicodeLocaleKeyword("ks", "level3");
+ } else if (caseSensitivity == CaseSensitivity.CS &&
+ accentSensitivity == AccentSensitivity.AI) {
+ builder
+ .setUnicodeLocaleKeyword("ks", "level1")
+ .setUnicodeLocaleKeyword("kc", "true");
+ } else if (caseSensitivity == CaseSensitivity.CI &&
+ accentSensitivity == AccentSensitivity.AS) {
+ builder.setUnicodeLocaleKeyword("ks", "level2");
+ } else if (caseSensitivity == CaseSensitivity.CI &&
+ accentSensitivity == AccentSensitivity.AI) {
+ builder.setUnicodeLocaleKeyword("ks", "level1");
+ }
+ ULocale resultLocale = builder.build();
+ Collator collator = Collator.getInstance(resultLocale);
+ Comparator<UTF8String> comparator = (s1, s2) -> collator.compare(
+ caseConversion(resultLocale, s1), caseConversion(resultLocale, s2));
+ collator.freeze();
+ return new Collation(
+ collationName(),
+ collator,
+ comparator,
+ ICUCollatorVersion,
+ s -> (long) collator.getCollationKey(s.toString()).hashCode(),
+ collationId == UNICODE_COLLATION_ID,
Review Comment:
I think other collations can `supportBinaryEquality` too?
for example: `SRB_CS_AS` is essentially binary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "uros-db (via GitHub)" <gi...@apache.org>.
uros-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1595090780
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
Review Comment:
is quiet map eviction a reasonable solution? i.e. just limit map size using some LRU policy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1604679971
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
Review Comment:
Also user can go with `collate('something', 'UTF8_BINARY_LCASE')` and here it's definitely allowed to pass anything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "dbatomic (via GitHub)" <gi...@apache.org>.
dbatomic commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1604917892
##########
connector/connect/common/src/main/protobuf/spark/connect/types.proto:
##########
@@ -101,7 +101,7 @@ message DataType {
message String {
uint32 type_variation_reference = 1;
- uint32 collation_id = 2;
+ string collation = 2;
Review Comment:
And it makes sense. Collation id should be internal spark concept. Outside of spark you should always use collation name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606762581
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -118,76 +119,433 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * Collation id is defined as 32-bit integer.
+ * We specify binary layouts for different classes of collations.
+ * Classes of collations are differentiated by most significant 3 bits (bit 31, 30 and 29),
+ * bit 31 being most significant and bit 0 being least significant.
+ * ---
+ * INDETERMINATE collation id binary layout:
+ * bit 31-0: 1
+ * INDETERMINATE collation id is equal to -1
+ * ---
+ * user-defined collation id binary layout:
+ * bit 31: 0
+ * bit 30: 1
+ * bit 29-0: undefined, reserved for future use
+ * ---
+ * UTF8_BINARY collation id binary layout:
+ * bit 31-22: zeroes
+ * bit 21-18: zeroes, reserved for space trimming
+ * bit 17-16: zeroes, reserved for version
+ * bit 15-3: zeroes
+ * bit 2: 0, reserved for accent sensitivity
+ * bit 1: 0, reserved for uppercase and case-insensitive
+ * bit 0: 0 = case-sensitive, 1 = lowercase
+ * ---
+ * ICU collation id binary layout:
+ * bit 31-30: zeroes
+ * bit 29: 1
+ * bit 28-24: zeroes
+ * bit 23-22: zeroes, reserved for version
+ * bit 21-18: zeroes, reserved for space trimming
+ * bit 17: 0 = case-sensitive, 1 = case-insensitive
+ * bit 16: 0 = accent-sensitive, 1 = accent-insensitive
+ * bit 15-14: zeroes, reserved for punctuation sensitivity
+ * bit 13-12: zeroes, reserved for first letter preference
+ * bit 11-0: locale id as specified in `ICULocaleToId` mapping
+ * ---
+ * Some illustrative examples of collation name to id mapping:
+ * - UTF8_BINARY -> 0
+ * - UTF8_BINARY_LCASE -> 1
+ * - UNICODE -> 0x20000000
+ * - UNICODE_AI -> 0x20010000
+ * - UNICODE_CI -> 0x20020000
+ * - UNICODE_CI_AI -> 0x20030000
+ * - af -> 0x20000001
+ * - af_CI_AI -> 0x20030001
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+
+ private enum DefinitionOrigin {
+ PREDEFINED, USER_DEFINED
+ }
+
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU
+ }
+
+ private static final int DEFINITION_ORIGIN_OFFSET = 30;
+ private static final int DEFINITION_ORIGIN_MASK = 0b1;
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b1;
+
+ private static final int INDETERMINATE_COLLATION_ID = -1;
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ private static ImplementationProvider getImplementationProvider(int collationId) {
+ return ImplementationProvider.values()[SpecifierUtils.getSpecValue(collationId,
+ IMPLEMENTATION_PROVIDER_OFFSET, IMPLEMENTATION_PROVIDER_MASK)];
+ }
+
+ private static DefinitionOrigin getDefinitionOrigin(int collationId) {
+ return DefinitionOrigin.values()[SpecifierUtils.getSpecValue(collationId,
+ DEFINITION_ORIGIN_OFFSET, DEFINITION_ORIGIN_MASK)];
+ }
+
+ private static Collation fetchCollation(int collationId) {
+ assert (collationId >= 0 && getDefinitionOrigin(collationId)
+ == DefinitionOrigin.PREDEFINED);
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ ImplementationProvider implementationProvider = getImplementationProvider(collationId);
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ private static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ private static final int CASE_SENSITIVITY_OFFSET = 0;
+ private static final int CASE_SENSITIVITY_MASK = 0b1;
+
+ private enum CaseSensitivity {
+ UNSPECIFIED, LCASE
+ }
+
+ private static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseSensitivity.UNSPECIFIED).collationId;
+ private static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseSensitivity.LCASE).collationId;
+ protected static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseSensitivity.UNSPECIFIED).buildCollation();
+ protected static Collation UTF8_BINARY_LCASE_COLLATION =
+ new CollationSpecUTF8Binary(CaseSensitivity.LCASE).buildCollation();
+
+ private final int collationId;
+
+ private CollationSpecUTF8Binary(CaseSensitivity caseSensitivity) {
+ this.collationId =
+ SpecifierUtils.setSpecValue(0, CASE_SENSITIVITY_OFFSET, caseSensitivity);
+ }
+
+ private static int collationNameToId(String originalName, String collationName)
+ throws SparkException {
+ if (UTF8_BINARY_COLLATION.collationName.equals(collationName)) {
+ return UTF8_BINARY_COLLATION_ID;
+ } else if (UTF8_BINARY_LCASE_COLLATION.collationName.equals(collationName)) {
+ return UTF8_BINARY_LCASE_COLLATION_ID;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+
+ private static CollationSpecUTF8Binary fromCollationId(int collationId) {
+ int caseConversionOrdinal = SpecifierUtils.getSpecValue(collationId,
+ CASE_SENSITIVITY_OFFSET, CASE_SENSITIVITY_MASK);
+ assert (SpecifierUtils.removeSpec(collationId,
+ CASE_SENSITIVITY_OFFSET, CASE_SENSITIVITY_MASK) == 0);
+ return new CollationSpecUTF8Binary(CaseSensitivity.values()[caseConversionOrdinal]);
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return new Collation("UTF8_BINARY", null, UTF8String::binaryCompare, "1.0",
+ s -> (long) s.hashCode(), true, true, false);
+ } else {
+ return new Collation("UTF8_BINARY_LCASE", null, UTF8String::compareLowerCase, "1.0",
+ s -> (long) s.toLowerCase().hashCode(), false, false, true);
+ }
+ }
+ }
+
+ private static class CollationSpecICU extends CollationSpec {
+
+ private enum CaseSensitivity {
+ CS, CI
+ }
+
+ private enum AccentSensitivity {
+ AS, AI
+ }
+
+ private static final int CASE_SENSITIVITY_OFFSET = 17;
+ private static final int CASE_SENSITIVITY_MASK = 0b1;
+ private static final int ACCENT_SENSITIVITY_OFFSET = 16;
+ private static final int ACCENT_SENSITIVITY_MASK = 0b1;
+
+ // Array of locale names, each locale id corresponds to the index in this array
+ private static final String[] ICULocaleNames;
+
+ // Mapping of locale names to corresponding `ULocale` instance
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+
+ // Used to parse user input collation names which are converted to uppercase
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+
+ // Reverse mapping of `ICULocaleNames`
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+
+ private static final String ICU_COLLATOR_VERSION = "153.120.0.0";
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
+ for (ULocale locale : locales) {
+ if (locale.getVariant().isEmpty()) {
+ String language = locale.getLanguage();
+ assert (!language.isEmpty());
+ StringBuilder builder = new StringBuilder(language);
+ String script = locale.getScript();
+ if (!script.isEmpty()) {
+ builder.append('_');
+ builder.append(script);
+ }
+ String country = locale.getISO3Country();
+ if (!country.isEmpty()) {
+ builder.append('_');
+ builder.append(country);
+ }
+ String localeName = builder.toString();
+ // locale names are unique
+ assert (!ICULocaleMap.containsKey(localeName));
+ ICULocaleMap.put(localeName, locale);
+ }
+ }
+ for (String localeName : ICULocaleMap.keySet()) {
+ String localeUppercase = localeName.toUpperCase();
+ // locale names are unique case-insensitively
+ assert (!ICULocaleMapUppercase.containsKey(localeUppercase));
+ ICULocaleMapUppercase.put(localeUppercase, localeName);
+ }
+ ICULocaleNames = ICULocaleMap.keySet().toArray(new String[0]);
+ Arrays.sort(ICULocaleNames);
+ // maximum number of locale ids as defined by binary layout
+ assert (ICULocaleNames.length <= (1 << 12));
+ for (int i = 0; i < ICULocaleNames.length; ++i) {
+ ICULocaleToId.put(ICULocaleNames[i], i);
+ }
+ }
+
+ private static final int UNICODE_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CS, AccentSensitivity.AS).collationId;
+ private static final int UNICODE_CI_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CI, AccentSensitivity.AS).collationId;
+
+ private final CaseSensitivity caseSensitivity;
+ private final AccentSensitivity accentSensitivity;
+ private final String locale;
+ private final int collationId;
+
+ private CollationSpecICU(String locale, CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ int collationId = ICULocaleToId.get(locale);
+ collationId = SpecifierUtils.setSpecValue(collationId, IMPLEMENTATION_PROVIDER_OFFSET,
+ ImplementationProvider.ICU);
+ collationId = SpecifierUtils.setSpecValue(collationId, CASE_SENSITIVITY_OFFSET,
+ caseSensitivity);
+ collationId = SpecifierUtils.setSpecValue(collationId, ACCENT_SENSITIVITY_OFFSET,
+ accentSensitivity);
+ this.collationId = collationId;
+ }
+
+ private static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ // search for the longest locale match because specifiers are designed to be different from
+ // script tag and country code, meaning the only valid locale name match can be
+ // the longest one
+ int lastPos = -1;
+ for (int i = 1; i <= collationName.length(); i++) {
+ String localeName = collationName.substring(0, i);
+ if (ICULocaleMapUppercase.containsKey(localeName)) {
+ lastPos = i;
+ }
+ }
+ if (lastPos == -1) {
+ throw collationInvalidNameException(originalName);
+ } else {
+ String locale = collationName.substring(0, lastPos);
+ int collationId = ICULocaleToId.get(ICULocaleMapUppercase.get(locale));
+
+ // try all combinations of AS/AI and CS/CI
+ CaseSensitivity caseSensitivity;
+ AccentSensitivity accentSensitivity;
+ if (collationName.equals(locale) ||
+ collationName.equals(locale + "_AS") ||
Review Comment:
The `collationName` is not expected to be of a particular case, we convert it to uppercase [here](https://github.com/apache/spark/pull/46180/files#diff-640c14aa5d7473df79b2435ce5a327dffcc16ca29354b153956b4f8d19fdb16cR278) before performing comparisons.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606952858
##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -152,4 +219,218 @@ class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ig
}
})
}
+
+ test("test collation caching") {
+ Seq(
+ "UTF8_BINARY",
+ "UTF8_BINARY_LCASE",
+ "UNICODE",
+ "UNICODE_CI",
+ "UNICODE_AI",
+ "UNICODE_CI_AI",
+ "UNICODE_AI_CI"
+ ).foreach(collationId => {
+ val col1 = fetchCollation(collationId)
+ val col2 = fetchCollation(collationId)
+ assert(col1 eq col2) // reference equality
+ })
+ }
+
+ test("collations with ICU non-root localization") {
+ Seq(
+ // language only
+ "en",
+ "en_CS",
+ "en_CI",
+ "en_AS",
+ "en_AI",
+ // language + 3-letter country code
+ "en_USA",
+ "en_USA_CS",
+ "en_USA_CI",
+ "en_USA_AS",
+ "en_USA_AI",
+ // language + script code
+ "sr_Cyrl",
+ "sr_Cyrl_CS",
+ "sr_Cyrl_CI",
+ "sr_Cyrl_AS",
+ "sr_Cyrl_AI",
+ // language + script code + 3-letter country code
+ "sr_Cyrl_SRB",
+ "sr_Cyrl_SRB_CS",
+ "sr_Cyrl_SRB_CI",
+ "sr_Cyrl_SRB_AS",
+ "sr_Cyrl_SRB_AI"
+ ).foreach(collationICU => {
+ val col = fetchCollation(collationICU)
+ assert(col.collator.getLocale(ULocale.VALID_LOCALE) != ULocale.ROOT)
+ })
+ }
+
+ test("invalid names of collations with ICU non-root localization") {
Review Comment:
Added invalid ordering [checks](https://github.com/apache/spark/pull/46180/files#diff-9c12d32db9d55dd6ecb5b10f2fc57c7ba7de7275cab57bf157fa42cbc09f3876R291-R310) as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606712998
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
Review Comment:
Resolving this discussion since maximum size of the map is quite small with current scope of changes (<200 locales x 2 case sensitivity x 2 accent sensitivity ~ 800 entries total). Let's revisit this when we expand the scope to space trimming, punctuation sensitivity and first letter preference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606709431
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
Review Comment:
Schema is outdated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606736620
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -245,29 +599,26 @@ public static StringSearch getStringSearch(
* Returns the collation id for the given collation name.
*/
public static int collationNameToId(String collationName) throws SparkException {
- String normalizedName = collationName.toUpperCase();
- if (collationNameToIdMap.containsKey(normalizedName)) {
- return collationNameToIdMap.get(normalizedName);
- } else {
- Collation suggestion = Collections.min(List.of(collationTable), Comparator.comparingInt(
- c -> UTF8String.fromString(c.collationName).levenshteinDistance(
- UTF8String.fromString(normalizedName))));
-
- Map<String, String> params = new HashMap<>();
- params.put("collationName", collationName);
- params.put("proposal", suggestion.collationName);
-
- throw new SparkException(
- "COLLATION_INVALID_NAME", SparkException.constructMessageParams(params), null);
- }
+ return Collation.CollationSpec.collationNameToId(collationName);
+ }
+
+ public static Collation fetchCollationUnsafe(int collationId) throws SparkException {
+ return Collation.CollationSpec.fetchCollation(collationId);
}
public static Collation fetchCollation(int collationId) {
- return collationTable[collationId];
+ try {
+ return fetchCollationUnsafe(collationId);
+ } catch (SparkException e) {
+ return Collation.CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ }
Review Comment:
Fixed this. We don't throw or swallow errors anymore in `fetchCollation`, but rather perform assertions around assumptions which are validated during testing. Note that `fetchCollation` is only used internally and assumes that valid collation id is passed as parameter, this enables us to remove error handling logic; user-facing collation name to id parsing logic does throw error and has regular control flow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606774756
##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -152,4 +219,218 @@ class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ig
}
})
}
+
+ test("test collation caching") {
+ Seq(
+ "UTF8_BINARY",
+ "UTF8_BINARY_LCASE",
+ "UNICODE",
+ "UNICODE_CI",
+ "UNICODE_AI",
+ "UNICODE_CI_AI",
+ "UNICODE_AI_CI"
+ ).foreach(collationId => {
+ val col1 = fetchCollation(collationId)
+ val col2 = fetchCollation(collationId)
+ assert(col1 eq col2) // reference equality
+ })
+ }
+
+ test("collations with ICU non-root localization") {
+ Seq(
+ // language only
+ "en",
+ "en_CS",
+ "en_CI",
+ "en_AS",
+ "en_AI",
+ // language + 3-letter country code
+ "en_USA",
+ "en_USA_CS",
+ "en_USA_CI",
+ "en_USA_AS",
+ "en_USA_AI",
+ // language + script code
+ "sr_Cyrl",
+ "sr_Cyrl_CS",
+ "sr_Cyrl_CI",
+ "sr_Cyrl_AS",
+ "sr_Cyrl_AI",
+ // language + script code + 3-letter country code
+ "sr_Cyrl_SRB",
+ "sr_Cyrl_SRB_CS",
+ "sr_Cyrl_SRB_CI",
+ "sr_Cyrl_SRB_AS",
+ "sr_Cyrl_SRB_AI"
+ ).foreach(collationICU => {
+ val col = fetchCollation(collationICU)
+ assert(col.collator.getLocale(ULocale.VALID_LOCALE) != ULocale.ROOT)
+ })
+ }
+
+ test("invalid names of collations with ICU non-root localization") {
Review Comment:
Please check [comment](https://github.com/apache/spark/pull/46180#discussion_r1606750756) as for conflicting specifiers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606738032
##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -30,31 +33,107 @@ import org.apache.spark.unsafe.types.UTF8String.{fromString => toUTF8}
class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ignore funsuite
test("collationId stability") {
- val utf8Binary = fetchCollation(0)
+ assert(UTF8_BINARY_COLLATION_ID == 0)
Review Comment:
Added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606743718
##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -152,4 +231,168 @@ class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ig
}
})
}
+
+ test("test collation caching") {
+ Seq(
+ "UTF8_BINARY",
+ "UTF8_BINARY_LCASE",
+ "UTF8_BINARY_UCASE",
+ "UNICODE",
+ "UNICODE_LCASE",
+ "UNICODE_UCASE",
+ "UNICODE_CI",
+ "UNICODE_AI_CI",
+ "UNICODE_AI_CI_LCASE",
+ "UNICODE_AI_CI_UCASE"
+ ).foreach(collationId => {
+ val col1 = fetchCollation(collationId)
+ val col2 = fetchCollation(collationId)
+ assert(col1 eq col2) // reference equality
+ })
+ }
+
+ test("collations with ICU non-root localization") {
+ Seq(
+ // language only
+ "en",
+ "en_CS",
+ "en_CI",
+ "en_AS",
+ "en_AI",
+ "en_LCASE",
+ "en_UCASE",
+ // language + 3-letter country code
+ "en_USA",
+ "en_USA_CS",
+ "en_USA_CI",
+ "en_USA_AS",
+ "en_USA_AI",
+ "en_USA_LCASE",
+ "en_USA_UCASE",
+ // language + script code
+ "sr_Cyrl",
+ "sr_Cyrl_CS",
+ "sr_Cyrl_CI",
+ "sr_Cyrl_AS",
+ "sr_Cyrl_AI",
+ "sr_Cyrl_LCASE",
+ "sr_Cyrl_UCASE",
+ // language + script code + 3-letter country code
+ "sr_Cyrl_SRB",
+ "sr_Cyrl_SRB_CS",
+ "sr_Cyrl_SRB_CI",
+ "sr_Cyrl_SRB_AS",
+ "sr_Cyrl_SRB_AI",
+ "sr_Cyrl_SRB_LCASE",
+ "sr_Cyrl_SRB_UCASE"
+ ).foreach(collationICU => {
+ val col = fetchCollation(collationICU)
+ assert(col.collator.getLocale(ULocale.VALID_LOCALE) != ULocale.ROOT)
+ })
+ }
+
+ test("invalid names of collations with ICU non-root localization") {
+ Seq(
+ "en_US", // must use 3-letter country code
+ "enn",
+ "en_AAA",
+ "en_Something",
+ "en_Something_USA",
+ "en_CI_UNSPECIFIED",
+ "en_USA_UNSPECIFIED",
+ "en_USA_UNSPECIFIED_CI",
+ "en_INDETERMINATE",
+ "en_USA_INDETERMINATE",
+ "en_Latn_USA", // use en_USA instead
+ "en_Cyrl_USA",
+ "en_USA_AAA",
+ "sr_Cyrl_SRB_AAA"
+ ).foreach(collationName => {
+ val error = intercept[SparkException] {
+ fetchCollation(collationName)
+ }
+
+ assert(error.getErrorClass === "COLLATION_INVALID_NAME")
+ assert(error.getMessageParameters.asScala === Map("collationName" -> collationName))
+ })
+ }
+
+ test("collations name normalization for ICU non-root localization") {
+ Seq(
+ ("en_USA", "en_USA"),
+ ("en_CS", "en"),
+ ("en_AS", "en"),
+ ("en_CS_AS", "en"),
+ ("en_AS_CS", "en"),
+ ("en_CI", "en_CI"),
+ ("en_AI", "en_AI"),
+ ("en_AI_CI", "en_CI_AI"),
+ ("en_USA_AI_CI", "en_USA_CI_AI"),
+ ("en_USA_LCASE_AI_CI", "en_USA_CI_AI_LCASE"),
+ ("en_USA_LCASE_CI_AI", "en_USA_CI_AI_LCASE"),
+ ("en_USA_AI_LCASE_CI", "en_USA_CI_AI_LCASE"),
+ ("en_USA_CI_LCASE_AI", "en_USA_CI_AI_LCASE"),
+ // randomized case
+ ("EN_USA", "en_USA"),
+ ("eN_usA_ci_uCASe_aI", "en_USA_CI_AI_UCASE"),
+ ("SR_CYRL", "sr_Cyrl"),
+ ("sr_cyrl_srb", "sr_Cyrl_SRB"),
+ ("sR_cYRl_sRb", "sr_Cyrl_SRB")
+ ).foreach {
+ case (name, normalized) =>
+ val col = fetchCollation(name)
+ assert(col.collationName == normalized)
+ }
+ }
+
+ test("invalid collationId") {
+ val badCollationIds = Seq(
+ -1, // user-defined collation range
+ 1 << 31, // user-defined collation range
+ 1 << 12, // utf8-binary mandatory zero bit 12 breach
+ 1 << 13, // utf8-binary mandatory zero bit 13 breach
+ 1 << 14, // utf8-binary mandatory zero bit 14 breach
+ 1 << 15, // utf8-binary mandatory zero bit 15 breach
+ 1 << 16, // utf8-binary mandatory zero bit 16 breach
+ 1 << 17, // utf8-binary mandatory zero bit 17 breach
+ 1 << 18, // utf8-binary mandatory zero bit 18 breach
+ 1 << 19, // utf8-binary mandatory zero bit 19 breach
+ 1 << 20, // utf8-binary mandatory zero bit 20 breach
+ 1 << 23, // utf8-binary mandatory zero bit 23 breach
+ 1 << 24, // utf8-binary mandatory zero bit 24 breach
+ 1 << 25, // utf8-binary mandatory zero bit 25 breach
+ 1 << 26, // utf8-binary mandatory zero bit 26 breach
+ (1 << 29) | (1 << 12), // ICU mandatory zero bit 12 breach
+ (1 << 29) | (1 << 13), // ICU mandatory zero bit 13 breach
+ (1 << 29) | (1 << 14), // ICU mandatory zero bit 14 breach
+ (1 << 29) | (1 << 15), // ICU mandatory zero bit 15 breach
+ (1 << 29) | (1 << 16), // ICU mandatory zero bit 16 breach
+ (1 << 29) | (1 << 17), // ICU mandatory zero bit 17 breach
+ (1 << 29) | (1 << 18), // ICU mandatory zero bit 18 breach
+ (1 << 29) | (1 << 19), // ICU mandatory zero bit 19 breach
+ (1 << 29) | (1 << 20), // ICU mandatory zero bit 20 breach
+ (1 << 29) | (1 << 23), // ICU mandatory zero bit 23 breach
+ (1 << 29) | (1 << 24), // ICU mandatory zero bit 24 breach
+ (1 << 29) | (1 << 25), // ICU mandatory zero bit 25 breach
+ (1 << 29) | (1 << 26), // ICU mandatory zero bit 26 breach
+ 123, // utf8-binary with non-zero locale id
+ (1 << 29) | (1 << 12), // ICU with invalid locale id
+ (1 << 29) | 0xFFFF, // ICU with invalid locale id
+ (1 << 23) | (1 << 22), // utf8-binary with invalid case conversion
+ (1 << 29) | (1 << 23) | (1 << 22), // ICU with invalid case conversion
+ 1 << 27, // utf8-binary accent-insensitive
+ 1 << 28, // utf8-binary case-insensitive
+ INDETERMINATE_COLLATION_ID, // indeterminate collation
+ 1 << 30, // indeterminate collation
+ (1 << 30) | (1 << 29) // invalid implementation provider
+ )
+ badCollationIds.foreach(collationId => {
+ val e = intercept[SparkException](fetchCollationUnsafe(collationId))
+ // user cannot specify collation ids directly so this is an internal error
+ assert(e.getErrorClass === "INTERNAL_ERROR")
+ })
+ badCollationIds.foreach(collationId => {
+ assert(fetchCollation(collationId).collationName == "UTF8_BINARY")
+ })
Review Comment:
Fixed, we don't do this anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "dbatomic (via GitHub)" <gi...@apache.org>.
dbatomic commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1594153009
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
Review Comment:
Just a note - we are never removing items from this hashmap. This means that if someone goes through all collations there will be non-trivial memory impact on the system.
This is probably not a concern since I can't imagine why customer would use 10k+ different collations but may be a way to DOS the system. i.e. with given query pattern you may get system to OOM.
@cloud-fan - does Spark in general worries about such things?
I am fine with this going as is btw.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1595078750
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
Review Comment:
For now (looking at changes of this PR) it's not that big deal since we're supporting ~200 locales x 2 case sensitivity x 2 accent sensitivity x 3 case conversions. So if user wants to use all available collations (highly unlikely, user will probably use several different collations on average) it will be < 2000 entries in the map.
The idea with map (cache) approach was that we don't want to hardcode all the collations into the table which was the previous approach. This will make a considerable difference when we add support for punctuation sensitivity, first letter preference and space trimming; this will grow number of available collations to ~40k. I think it would make sense to worry about caching strategies after these additions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606723662
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected static int parseSpecifiers(
+ String originalName, String collationName, int splitStart) throws SparkException {
+ int specifiers = 0;
+ String[] parts = collationName.substring(splitStart).split("_");
+ for (String part : parts) {
+ if (!part.isEmpty()) {
+ if (part.equals("UNSPECIFIED") || part.equals("INDETERMINATE")) {
+ throw collationInvalidNameException(originalName);
+ } else if (Arrays.stream(CaseSensitivity.values()).anyMatch(
+ (s) -> s.toString().equals(part))) {
+ specifiers |=
+ CaseSensitivity.valueOf(part).ordinal() << CASE_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(AccentSensitivity.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ AccentSensitivity.valueOf(part).ordinal() << ACCENT_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(CaseConversion.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ CaseConversion.valueOf(part).ordinal() << CASE_CONVERSION_OFFSET;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+ }
+ return specifiers;
+ }
+
+ protected abstract int getCollationId();
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ public static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.LCASE).getCollationId();
+ public static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).buildCollation();
+
+ private CollationSpecUTF8Binary(CaseConversion caseConversion) {
+ super(null, CaseSensitivity.CS, AccentSensitivity.AS, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int collationId = 0;
+ int specifiers = CollationSpec.parseSpecifiers(
+ originalName, collationName, "UTF8_BINARY".length());
+ if ((specifiers & ~(CASE_CONVERSION_MASK << CASE_CONVERSION_OFFSET)) != 0) {
+ throw collationInvalidNameException(originalName);
+ }
+ collationId |= specifiers;
+ return collationId;
+ }
+
+ @Override
+ protected int getCollationId() {
+ int collationId = 0;
+ collationId |= caseConversion.ordinal() << CASE_CONVERSION_OFFSET;
+ return collationId;
+ }
+
+ public static CollationSpecUTF8Binary fromCollationId(int collationId)
+ throws SparkException {
+ int originalCollationId = collationId;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
+ if (collationId != 0 || caseConversionOrdinal >= CaseConversion.values().length) {
+ throw SparkException.internalError("Invalid UTF8_BINARY collation id " +
+ originalCollationId);
+ } else {
+ CaseConversion caseConversion = CaseConversion.values()[caseConversionOrdinal];
+ return new CollationSpecUTF8Binary(caseConversion);
+ }
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ Comparator<UTF8String> comparator;
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ comparator = UTF8String::binaryCompare;
+ } else if (collationId == UTF8_BINARY_LCASE_COLLATION_ID) {
+ comparator = UTF8String::compareLowerCase;
+ } else {
+ comparator = (s1, s2) -> {
+ UTF8String convertedS1 = caseConversion(s1);
+ UTF8String convertedS2 = caseConversion(s2);
+ return convertedS1.binaryCompare(convertedS2);
+ };
+ }
+ return new Collation(
+ collationName(),
+ null,
+ comparator,
+ "1.0",
+ s -> (long) caseConversion(s).hashCode(),
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_LCASE_COLLATION_ID
+ );
+ }
+
+ private UTF8String caseConversion(UTF8String s) {
+ if (caseConversion == CaseConversion.LCASE) {
+ return s.toLowerCase();
+ } else if (caseConversion == CaseConversion.UCASE) {
+ return s.toUpperCase();
+ } else {
+ return s;
+ }
+ }
+
+ private String collationName() {
+ StringBuilder builder = new StringBuilder();
+ builder.append("UTF8_BINARY");
+ if (caseConversion != CaseConversion.UNSPECIFIED) {
+ builder.append('_');
+ builder.append(caseConversion.toString());
+ }
+ return builder.toString();
+ }
+ }
+
+ private static class CollationSpecICU extends CollationSpec {
+
+ private static final String[] ICULocaleNames;
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+ private static final String ICUCollatorVersion = "153.120.0.0";
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
+ for (ULocale locale : locales) {
+ if (locale.getVariant().isEmpty()) {
+ String language = locale.getLanguage();
+ assert (!language.isEmpty());
+ StringBuilder builder = new StringBuilder(language);
+ String script = locale.getScript();
+ if (!script.isEmpty()) {
+ builder.append('_');
+ builder.append(script);
+ }
+ String country = locale.getISO3Country();
+ if (!country.isEmpty()) {
+ builder.append('_');
+ builder.append(country);
+ }
+ String localeName = builder.toString();
+ assert (!ICULocaleMap.containsKey(localeName));
+ ICULocaleMap.put(localeName, locale);
+ }
+ }
+ for (String localeName : ICULocaleMap.keySet()) {
+ String localeUppercase = localeName.toUpperCase();
+ assert (!ICULocaleMapUppercase.containsKey(localeUppercase));
+ ICULocaleMapUppercase.put(localeUppercase, localeName);
+ }
Review Comment:
Added comments to static members.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606722004
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
Review Comment:
Everything for predefined collations is in ASCII space (uppercase/lowercase ASCII letters + underscores) and `CollationFactory` is for the time being only scoped to predefined collations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1612086900
##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/common/DataTypeProtoConverter.scala:
##########
@@ -177,7 +178,11 @@ object DataTypeProtoConverter {
case s: StringType =>
proto.DataType
.newBuilder()
- .setString(proto.DataType.String.newBuilder().setCollationId(s.collationId).build())
+ .setString(
+ proto.DataType.String
+ .newBuilder()
+ .setCollation(CollationFactory.fetchCollation(s.collationId).collationName)
Review Comment:
I'm not sure how is this possible. `StringType` can be accessed in the client side (Scala client) and it needs to parse the collation name to ID.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1601366046
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected static int parseSpecifiers(
+ String originalName, String collationName, int splitStart) throws SparkException {
+ int specifiers = 0;
+ String[] parts = collationName.substring(splitStart).split("_");
+ for (String part : parts) {
+ if (!part.isEmpty()) {
+ if (part.equals("UNSPECIFIED") || part.equals("INDETERMINATE")) {
+ throw collationInvalidNameException(originalName);
+ } else if (Arrays.stream(CaseSensitivity.values()).anyMatch(
+ (s) -> s.toString().equals(part))) {
+ specifiers |=
+ CaseSensitivity.valueOf(part).ordinal() << CASE_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(AccentSensitivity.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ AccentSensitivity.valueOf(part).ordinal() << ACCENT_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(CaseConversion.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ CaseConversion.valueOf(part).ordinal() << CASE_CONVERSION_OFFSET;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+ }
+ return specifiers;
+ }
+
+ protected abstract int getCollationId();
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ public static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.LCASE).getCollationId();
+ public static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).buildCollation();
+
+ private CollationSpecUTF8Binary(CaseConversion caseConversion) {
+ super(null, CaseSensitivity.CS, AccentSensitivity.AS, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int collationId = 0;
+ int specifiers = CollationSpec.parseSpecifiers(
+ originalName, collationName, "UTF8_BINARY".length());
+ if ((specifiers & ~(CASE_CONVERSION_MASK << CASE_CONVERSION_OFFSET)) != 0) {
+ throw collationInvalidNameException(originalName);
+ }
+ collationId |= specifiers;
+ return collationId;
+ }
+
+ @Override
+ protected int getCollationId() {
+ int collationId = 0;
+ collationId |= caseConversion.ordinal() << CASE_CONVERSION_OFFSET;
+ return collationId;
+ }
+
+ public static CollationSpecUTF8Binary fromCollationId(int collationId)
+ throws SparkException {
+ int originalCollationId = collationId;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
+ if (collationId != 0 || caseConversionOrdinal >= CaseConversion.values().length) {
+ throw SparkException.internalError("Invalid UTF8_BINARY collation id " +
+ originalCollationId);
+ } else {
+ CaseConversion caseConversion = CaseConversion.values()[caseConversionOrdinal];
+ return new CollationSpecUTF8Binary(caseConversion);
+ }
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ Comparator<UTF8String> comparator;
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ comparator = UTF8String::binaryCompare;
+ } else if (collationId == UTF8_BINARY_LCASE_COLLATION_ID) {
+ comparator = UTF8String::compareLowerCase;
+ } else {
+ comparator = (s1, s2) -> {
+ UTF8String convertedS1 = caseConversion(s1);
+ UTF8String convertedS2 = caseConversion(s2);
+ return convertedS1.binaryCompare(convertedS2);
+ };
+ }
+ return new Collation(
+ collationName(),
+ null,
+ comparator,
+ "1.0",
+ s -> (long) caseConversion(s).hashCode(),
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_LCASE_COLLATION_ID
+ );
+ }
+
+ private UTF8String caseConversion(UTF8String s) {
+ if (caseConversion == CaseConversion.LCASE) {
+ return s.toLowerCase();
+ } else if (caseConversion == CaseConversion.UCASE) {
+ return s.toUpperCase();
+ } else {
+ return s;
+ }
+ }
+
+ private String collationName() {
+ StringBuilder builder = new StringBuilder();
+ builder.append("UTF8_BINARY");
+ if (caseConversion != CaseConversion.UNSPECIFIED) {
+ builder.append('_');
+ builder.append(caseConversion.toString());
+ }
+ return builder.toString();
+ }
+ }
+
+ private static class CollationSpecICU extends CollationSpec {
+
+ private static final String[] ICULocaleNames;
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+ private static final String ICUCollatorVersion = "153.120.0.0";
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
+ for (ULocale locale : locales) {
+ if (locale.getVariant().isEmpty()) {
+ String language = locale.getLanguage();
+ assert (!language.isEmpty());
+ StringBuilder builder = new StringBuilder(language);
+ String script = locale.getScript();
+ if (!script.isEmpty()) {
+ builder.append('_');
+ builder.append(script);
+ }
+ String country = locale.getISO3Country();
+ if (!country.isEmpty()) {
+ builder.append('_');
+ builder.append(country);
+ }
+ String localeName = builder.toString();
+ assert (!ICULocaleMap.containsKey(localeName));
+ ICULocaleMap.put(localeName, locale);
+ }
+ }
+ for (String localeName : ICULocaleMap.keySet()) {
+ String localeUppercase = localeName.toUpperCase();
+ assert (!ICULocaleMapUppercase.containsKey(localeUppercase));
+ ICULocaleMapUppercase.put(localeUppercase, localeName);
+ }
+ ICULocaleNames = ICULocaleMap.keySet().toArray(new String[0]);
+ Arrays.sort(ICULocaleNames);
+ assert (ICULocaleNames.length <= (1 << 16));
+ for (int i = 0; i < ICULocaleNames.length; i++) {
+ ICULocaleToId.put(ICULocaleNames[i], i);
+ }
Review Comment:
Overall design of collation ids in the PR is that `UTF8_BINARY` always has collation id 0. `UTF8_BINARY_LCASE` doesn't have collation id 1 anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1601383798
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +118,490 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30: 0 = utf8-binary, 1 = ICU
+ * bit 29: 0 = case-sensitive, 1 = case-insensitive
+ * bit 28: 0 = accent-sensitive, 1 = accent-insensitive
+ * bit 27-26: 00 = unspecified, 01 = punctuation-sensitive, 10 = punctuation-insensitive
+ * bit 25-24: 00 = unspecified, 01 = first-lower, 10 = first-upper
+ * bit 23-22: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 21-20: 00 = unspecified, 01 = trim-left, 10 = trim-right, 11 = trim-both
+ * bit 19-18: zeroes, reserved for version
+ * bit 17-16: zeroes
+ * bit 15-0: locale id for ICU collations / zeroes for utf8-binary
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
- }
- }
+ private static class CollationSpec {
+ private enum ImplementationProvider {
+ UTF8_BINARY, ICU
+ }
+
+ private enum CaseSensitivity {
+ CS, CI
+ }
+
+ private enum AccentSensitivity {
+ AS, AI
+ }
+
+ private enum PunctuationSensitivity {
+ UNSPECIFIED, PS, PI
+ }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private enum FirstLetterPreference {
+ UNSPECIFIED, FU, FL
+ }
+
+ private enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ private enum SpaceTrimming {
+ UNSPECIFIED, LTRIM, RTRIM, TRIM
+ }
+
+ private static final int implementationProviderOffset = 30;
+ private static final int implementationProviderLen = 1;
+ private static final int caseSensitivityOffset = 29;
+ private static final int caseSensitivityLen = 1;
+ private static final int accentSensitivityOffset = 28;
+ private static final int accentSensitivityLen = 1;
+ private static final int punctuationSensitivityOffset = 26;
+ private static final int punctuationSensitivityLen = 2;
+ private static final int firstLetterPreferenceOffset = 24;
+ private static final int firstLetterPreferenceLen = 2;
+ private static final int caseConversionOffset = 22;
+ private static final int caseConversionLen = 2;
+ private static final int spaceTrimmingOffset = 20;
+ private static final int spaceTrimmingLen = 2;
+ private static final int localeOffset = 0;
+ private static final int localeLen = 16;
+
+ private static final String[] ICULocaleNames;
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
Review Comment:
Please check [comment](https://github.com/apache/spark/pull/46180/files#r1601363774) below. We assume is that list of available locales is going to be pretty stable and added golden file to verify this. However, if we bump ICU version and any changes do occur (for example, ICU adds support for new locales and the sorted list of locale names changes), users shouldn't still be affected because collation ids are internal to Spark. Still, problems could arise if there are multiple executors where one is running a Spark version prior to this breaking change and the other is running Spark version after the change though I'm not aware though of particular scenario where this could happen.
But even if this does happen and golden file reports unwanted changes in the diff, we can add more logic to enforce no breaking changes to the existing locale ids are made.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "dbatomic (via GitHub)" <gi...@apache.org>.
dbatomic commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1604667645
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
Review Comment:
It's an `identifier` from parser perspective. Which pretty much means that anything is allowed, especially if name is backticked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "dbatomic (via GitHub)" <gi...@apache.org>.
dbatomic commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1604678110
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -245,29 +599,26 @@ public static StringSearch getStringSearch(
* Returns the collation id for the given collation name.
*/
public static int collationNameToId(String collationName) throws SparkException {
- String normalizedName = collationName.toUpperCase();
- if (collationNameToIdMap.containsKey(normalizedName)) {
- return collationNameToIdMap.get(normalizedName);
- } else {
- Collation suggestion = Collections.min(List.of(collationTable), Comparator.comparingInt(
- c -> UTF8String.fromString(c.collationName).levenshteinDistance(
- UTF8String.fromString(normalizedName))));
-
- Map<String, String> params = new HashMap<>();
- params.put("collationName", collationName);
- params.put("proposal", suggestion.collationName);
-
- throw new SparkException(
- "COLLATION_INVALID_NAME", SparkException.constructMessageParams(params), null);
- }
+ return Collation.CollationSpec.collationNameToId(collationName);
+ }
+
+ public static Collation fetchCollationUnsafe(int collationId) throws SparkException {
+ return Collation.CollationSpec.fetchCollation(collationId);
}
public static Collation fetchCollation(int collationId) {
- return collationTable[collationId];
+ try {
+ return fetchCollationUnsafe(collationId);
+ } catch (SparkException e) {
+ return Collation.CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ }
Review Comment:
TBH, I don't like this either. Swallowing generic exception and returning default seems like a big red flag to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "mihailom-db (via GitHub)" <gi...@apache.org>.
mihailom-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1595382023
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,438 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ public static int collationNameToId(String originalCollationName) throws SparkException {
+ String collationName = originalCollationName.toUpperCase();
+ try {
+ if (collationName.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName);
+ }
+ } catch (SparkException e) {
+ throw new SparkException(
+ "COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", originalCollationName)),
+ e);
+ }
+ }
+
+ protected static int parseSpecifiers(String specString) throws SparkException {
+ int specifiers = 0;
+ String[] parts = specString.split("_");
+ for (String part : parts) {
+ if (!part.isEmpty()) {
+ if (part.equals("UNSPECIFIED") || part.equals("INDETERMINATE")) {
+ throw new SparkException(part + " collation specifier reserved for internal use");
Review Comment:
Why is this case not combined with the default branch were we throw an error? Also one more question I have what happens if a user defines for example UNICODE_CI_CI_CI, do we allow this redundancy?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "mihailom-db (via GitHub)" <gi...@apache.org>.
mihailom-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1595376570
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,438 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
Review Comment:
I believe for now it is not possible to create `INDETERMINATE` collation. What we do need is to be able to alias the values with this collation in select queries (this is a print of result of for example concat function). This is something tricky which I would say we could deal with after this PR, as we need to allow printing, but not storing of the INDETERMINATE collation, but for now I believe it is sufficient to just block users from creating INDETERMINATE collation explicitly with COLLATE in SQL. Same blocking should apply to pyspark and dataframe API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1576144441
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +118,490 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30: 0 = utf8-binary, 1 = ICU
+ * bit 29: 0 = case-sensitive, 1 = case-insensitive
+ * bit 28: 0 = accent-sensitive, 1 = accent-insensitive
+ * bit 27-26: 00 = unspecified, 01 = punctuation-sensitive, 10 = punctuation-insensitive
+ * bit 25-24: 00 = unspecified, 01 = first-lower, 10 = first-upper
+ * bit 23-22: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 21-20: 00 = unspecified, 01 = trim-left, 10 = trim-right, 11 = trim-both
+ * bit 19-18: zeroes, reserved for version
+ * bit 17-16: zeroes
+ * bit 15-0: locale id for ICU collations / zeroes for utf8-binary
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
- }
- }
+ private static class CollationSpec {
+ private enum ImplementationProvider {
+ UTF8_BINARY, ICU
+ }
+
+ private enum CaseSensitivity {
+ CS, CI
+ }
+
+ private enum AccentSensitivity {
+ AS, AI
+ }
+
+ private enum PunctuationSensitivity {
+ UNSPECIFIED, PS, PI
+ }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private enum FirstLetterPreference {
+ UNSPECIFIED, FU, FL
+ }
+
+ private enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ private enum SpaceTrimming {
+ UNSPECIFIED, LTRIM, RTRIM, TRIM
+ }
+
+ private static final int implementationProviderOffset = 30;
+ private static final int implementationProviderLen = 1;
+ private static final int caseSensitivityOffset = 29;
+ private static final int caseSensitivityLen = 1;
+ private static final int accentSensitivityOffset = 28;
+ private static final int accentSensitivityLen = 1;
+ private static final int punctuationSensitivityOffset = 26;
+ private static final int punctuationSensitivityLen = 2;
+ private static final int firstLetterPreferenceOffset = 24;
+ private static final int firstLetterPreferenceLen = 2;
+ private static final int caseConversionOffset = 22;
+ private static final int caseConversionLen = 2;
+ private static final int spaceTrimmingOffset = 20;
+ private static final int spaceTrimmingLen = 2;
+ private static final int localeOffset = 0;
+ private static final int localeLen = 16;
+
+ private static final String[] ICULocaleNames;
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
Review Comment:
That's right, I'll be adding checks for this including golden file with all supported ICU locales.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "mihailom-db (via GitHub)" <gi...@apache.org>.
mihailom-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1579005770
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,422 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30: 0 = utf8-binary, 1 = ICU
+ * bit 29: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 28: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 27-26: zeroes, reserved for punctuation sensitivity
+ * bit 25-24: zeroes, reserved for first letter preference
+ * bit 23-22: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 21-20: zeroes, reserved for space trimming
+ * bit 19-18: zeroes, reserved for version
+ * bit 17-16: zeroes
+ * bit 15-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int implementationProviderOffset = 30;
+ protected static final int implementationProviderLen = 1;
+ protected static final int caseSensitivityOffset = 29;
+ protected static final int caseSensitivityLen = 1;
+ protected static final int accentSensitivityOffset = 28;
+ protected static final int accentSensitivityLen = 1;
+ protected static final int caseConversionOffset = 22;
+ protected static final int caseConversionLen = 2;
+ protected static final int localeOffset = 0;
+ protected static final int localeLen = 16;
Review Comment:
Why don't we set these set some of these values to hexadecimal (0x0..). I would expect to see some bit masking if you have bit representation of collationId. What do others thing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606714530
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
Review Comment:
Outdated, we have new binary layout schema with expanded explanation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606762581
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -118,76 +119,433 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * Collation id is defined as 32-bit integer.
+ * We specify binary layouts for different classes of collations.
+ * Classes of collations are differentiated by most significant 3 bits (bit 31, 30 and 29),
+ * bit 31 being most significant and bit 0 being least significant.
+ * ---
+ * INDETERMINATE collation id binary layout:
+ * bit 31-0: 1
+ * INDETERMINATE collation id is equal to -1
+ * ---
+ * user-defined collation id binary layout:
+ * bit 31: 0
+ * bit 30: 1
+ * bit 29-0: undefined, reserved for future use
+ * ---
+ * UTF8_BINARY collation id binary layout:
+ * bit 31-22: zeroes
+ * bit 21-18: zeroes, reserved for space trimming
+ * bit 17-16: zeroes, reserved for version
+ * bit 15-3: zeroes
+ * bit 2: 0, reserved for accent sensitivity
+ * bit 1: 0, reserved for uppercase and case-insensitive
+ * bit 0: 0 = case-sensitive, 1 = lowercase
+ * ---
+ * ICU collation id binary layout:
+ * bit 31-30: zeroes
+ * bit 29: 1
+ * bit 28-24: zeroes
+ * bit 23-22: zeroes, reserved for version
+ * bit 21-18: zeroes, reserved for space trimming
+ * bit 17: 0 = case-sensitive, 1 = case-insensitive
+ * bit 16: 0 = accent-sensitive, 1 = accent-insensitive
+ * bit 15-14: zeroes, reserved for punctuation sensitivity
+ * bit 13-12: zeroes, reserved for first letter preference
+ * bit 11-0: locale id as specified in `ICULocaleToId` mapping
+ * ---
+ * Some illustrative examples of collation name to id mapping:
+ * - UTF8_BINARY -> 0
+ * - UTF8_BINARY_LCASE -> 1
+ * - UNICODE -> 0x20000000
+ * - UNICODE_AI -> 0x20010000
+ * - UNICODE_CI -> 0x20020000
+ * - UNICODE_CI_AI -> 0x20030000
+ * - af -> 0x20000001
+ * - af_CI_AI -> 0x20030001
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+
+ private enum DefinitionOrigin {
+ PREDEFINED, USER_DEFINED
+ }
+
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU
+ }
+
+ private static final int DEFINITION_ORIGIN_OFFSET = 30;
+ private static final int DEFINITION_ORIGIN_MASK = 0b1;
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b1;
+
+ private static final int INDETERMINATE_COLLATION_ID = -1;
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ private static ImplementationProvider getImplementationProvider(int collationId) {
+ return ImplementationProvider.values()[SpecifierUtils.getSpecValue(collationId,
+ IMPLEMENTATION_PROVIDER_OFFSET, IMPLEMENTATION_PROVIDER_MASK)];
+ }
+
+ private static DefinitionOrigin getDefinitionOrigin(int collationId) {
+ return DefinitionOrigin.values()[SpecifierUtils.getSpecValue(collationId,
+ DEFINITION_ORIGIN_OFFSET, DEFINITION_ORIGIN_MASK)];
+ }
+
+ private static Collation fetchCollation(int collationId) {
+ assert (collationId >= 0 && getDefinitionOrigin(collationId)
+ == DefinitionOrigin.PREDEFINED);
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ ImplementationProvider implementationProvider = getImplementationProvider(collationId);
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ private static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ private static final int CASE_SENSITIVITY_OFFSET = 0;
+ private static final int CASE_SENSITIVITY_MASK = 0b1;
+
+ private enum CaseSensitivity {
+ UNSPECIFIED, LCASE
+ }
+
+ private static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseSensitivity.UNSPECIFIED).collationId;
+ private static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseSensitivity.LCASE).collationId;
+ protected static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseSensitivity.UNSPECIFIED).buildCollation();
+ protected static Collation UTF8_BINARY_LCASE_COLLATION =
+ new CollationSpecUTF8Binary(CaseSensitivity.LCASE).buildCollation();
+
+ private final int collationId;
+
+ private CollationSpecUTF8Binary(CaseSensitivity caseSensitivity) {
+ this.collationId =
+ SpecifierUtils.setSpecValue(0, CASE_SENSITIVITY_OFFSET, caseSensitivity);
+ }
+
+ private static int collationNameToId(String originalName, String collationName)
+ throws SparkException {
+ if (UTF8_BINARY_COLLATION.collationName.equals(collationName)) {
+ return UTF8_BINARY_COLLATION_ID;
+ } else if (UTF8_BINARY_LCASE_COLLATION.collationName.equals(collationName)) {
+ return UTF8_BINARY_LCASE_COLLATION_ID;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+
+ private static CollationSpecUTF8Binary fromCollationId(int collationId) {
+ int caseConversionOrdinal = SpecifierUtils.getSpecValue(collationId,
+ CASE_SENSITIVITY_OFFSET, CASE_SENSITIVITY_MASK);
+ assert (SpecifierUtils.removeSpec(collationId,
+ CASE_SENSITIVITY_OFFSET, CASE_SENSITIVITY_MASK) == 0);
+ return new CollationSpecUTF8Binary(CaseSensitivity.values()[caseConversionOrdinal]);
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return new Collation("UTF8_BINARY", null, UTF8String::binaryCompare, "1.0",
+ s -> (long) s.hashCode(), true, true, false);
+ } else {
+ return new Collation("UTF8_BINARY_LCASE", null, UTF8String::compareLowerCase, "1.0",
+ s -> (long) s.toLowerCase().hashCode(), false, false, true);
+ }
+ }
+ }
+
+ private static class CollationSpecICU extends CollationSpec {
+
+ private enum CaseSensitivity {
+ CS, CI
+ }
+
+ private enum AccentSensitivity {
+ AS, AI
+ }
+
+ private static final int CASE_SENSITIVITY_OFFSET = 17;
+ private static final int CASE_SENSITIVITY_MASK = 0b1;
+ private static final int ACCENT_SENSITIVITY_OFFSET = 16;
+ private static final int ACCENT_SENSITIVITY_MASK = 0b1;
+
+ // Array of locale names, each locale id corresponds to the index in this array
+ private static final String[] ICULocaleNames;
+
+ // Mapping of locale names to corresponding `ULocale` instance
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+
+ // Used to parse user input collation names which are converted to uppercase
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+
+ // Reverse mapping of `ICULocaleNames`
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+
+ private static final String ICU_COLLATOR_VERSION = "153.120.0.0";
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
+ for (ULocale locale : locales) {
+ if (locale.getVariant().isEmpty()) {
+ String language = locale.getLanguage();
+ assert (!language.isEmpty());
+ StringBuilder builder = new StringBuilder(language);
+ String script = locale.getScript();
+ if (!script.isEmpty()) {
+ builder.append('_');
+ builder.append(script);
+ }
+ String country = locale.getISO3Country();
+ if (!country.isEmpty()) {
+ builder.append('_');
+ builder.append(country);
+ }
+ String localeName = builder.toString();
+ // locale names are unique
+ assert (!ICULocaleMap.containsKey(localeName));
+ ICULocaleMap.put(localeName, locale);
+ }
+ }
+ for (String localeName : ICULocaleMap.keySet()) {
+ String localeUppercase = localeName.toUpperCase();
+ // locale names are unique case-insensitively
+ assert (!ICULocaleMapUppercase.containsKey(localeUppercase));
+ ICULocaleMapUppercase.put(localeUppercase, localeName);
+ }
+ ICULocaleNames = ICULocaleMap.keySet().toArray(new String[0]);
+ Arrays.sort(ICULocaleNames);
+ // maximum number of locale ids as defined by binary layout
+ assert (ICULocaleNames.length <= (1 << 12));
+ for (int i = 0; i < ICULocaleNames.length; ++i) {
+ ICULocaleToId.put(ICULocaleNames[i], i);
+ }
+ }
+
+ private static final int UNICODE_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CS, AccentSensitivity.AS).collationId;
+ private static final int UNICODE_CI_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CI, AccentSensitivity.AS).collationId;
+
+ private final CaseSensitivity caseSensitivity;
+ private final AccentSensitivity accentSensitivity;
+ private final String locale;
+ private final int collationId;
+
+ private CollationSpecICU(String locale, CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ int collationId = ICULocaleToId.get(locale);
+ collationId = SpecifierUtils.setSpecValue(collationId, IMPLEMENTATION_PROVIDER_OFFSET,
+ ImplementationProvider.ICU);
+ collationId = SpecifierUtils.setSpecValue(collationId, CASE_SENSITIVITY_OFFSET,
+ caseSensitivity);
+ collationId = SpecifierUtils.setSpecValue(collationId, ACCENT_SENSITIVITY_OFFSET,
+ accentSensitivity);
+ this.collationId = collationId;
+ }
+
+ private static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ // search for the longest locale match because specifiers are designed to be different from
+ // script tag and country code, meaning the only valid locale name match can be
+ // the longest one
+ int lastPos = -1;
+ for (int i = 1; i <= collationName.length(); i++) {
+ String localeName = collationName.substring(0, i);
+ if (ICULocaleMapUppercase.containsKey(localeName)) {
+ lastPos = i;
+ }
+ }
+ if (lastPos == -1) {
+ throw collationInvalidNameException(originalName);
+ } else {
+ String locale = collationName.substring(0, lastPos);
+ int collationId = ICULocaleToId.get(ICULocaleMapUppercase.get(locale));
+
+ // try all combinations of AS/AI and CS/CI
+ CaseSensitivity caseSensitivity;
+ AccentSensitivity accentSensitivity;
+ if (collationName.equals(locale) ||
+ collationName.equals(locale + "_AS") ||
Review Comment:
The name is not expected to be of a particular case, we convert it to uppercase [here](https://github.com/apache/spark/pull/46180/files#diff-640c14aa5d7473df79b2435ce5a327dffcc16ca29354b153956b4f8d19fdb16cR278) before performing comparisons.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606797872
##########
sql/core/src/test/scala/org/apache/spark/sql/ICUCollationsMap.scala:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.util.{fileToString, stringToFile, CollationFactory}
+
+// scalastyle:off line.size.limit
+/**
+ * Guard against breaking changes in ICU locale names and codes supported by Collator class and provider by CollationFactory.
+ * Map is in form of rows of pairs (locale name, locale id); locale name consists of three parts:
+ * - 2-letter lowercase language code
+ * - 4-letter script code (optional)
+ * - 3-letter uppercase country code
+ *
+ * To re-generate collations map golden file, run:
+ * {{{
+ * SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly org.apache.spark.sql.ICUCollationsMap"
+ * }}}
+ */
+// scalastyle:on line.size.limit
+class ICUCollationsMap extends SparkFunSuite {
Review Comment:
Good point, refactored this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1612123955
##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/common/DataTypeProtoConverter.scala:
##########
@@ -177,7 +178,11 @@ object DataTypeProtoConverter {
case s: StringType =>
proto.DataType
.newBuilder()
- .setString(proto.DataType.String.newBuilder().setCollationId(s.collationId).build())
+ .setString(
+ proto.DataType.String
+ .newBuilder()
+ .setCollation(CollationFactory.fetchCollation(s.collationId).collationName)
Review Comment:
Here's usage [example](https://github.com/apache/spark/blob/21548a8cc5c527d4416a276a852f967b4410bd4b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/execution/SparkConnectPlanExecution.scala#L252) in Spark connect server.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1601298545
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -772,12 +772,17 @@ object SQLConf {
" produced by a builtin function such as to_char or CAST")
.version("4.0.0")
.stringConf
- .checkValue(CollationFactory.isValidCollation,
+ .checkValue(
+ collationName => {
+ try {
+ CollationFactory.fetchCollation(collationName)
+ true
+ } catch {
+ case e: SparkException if e.getErrorClass == "COLLATION_INVALID_NAME" => false
+ }
+ },
Review Comment:
`isValidCollation` would contain exception catching as part of its control flow, just as this approach does. Our reasoning was that it's better to scope this approach only here and not leave it open for use in other places.
cc @dbatomic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606716285
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
Review Comment:
That's right, we reserve space for maximum of 4 versions which is enough for phase-in and phase-out ICU version bumping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606736620
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -245,29 +599,26 @@ public static StringSearch getStringSearch(
* Returns the collation id for the given collation name.
*/
public static int collationNameToId(String collationName) throws SparkException {
- String normalizedName = collationName.toUpperCase();
- if (collationNameToIdMap.containsKey(normalizedName)) {
- return collationNameToIdMap.get(normalizedName);
- } else {
- Collation suggestion = Collections.min(List.of(collationTable), Comparator.comparingInt(
- c -> UTF8String.fromString(c.collationName).levenshteinDistance(
- UTF8String.fromString(normalizedName))));
-
- Map<String, String> params = new HashMap<>();
- params.put("collationName", collationName);
- params.put("proposal", suggestion.collationName);
-
- throw new SparkException(
- "COLLATION_INVALID_NAME", SparkException.constructMessageParams(params), null);
- }
+ return Collation.CollationSpec.collationNameToId(collationName);
+ }
+
+ public static Collation fetchCollationUnsafe(int collationId) throws SparkException {
+ return Collation.CollationSpec.fetchCollation(collationId);
}
public static Collation fetchCollation(int collationId) {
- return collationTable[collationId];
+ try {
+ return fetchCollationUnsafe(collationId);
+ } catch (SparkException e) {
+ return Collation.CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ }
Review Comment:
Fixed this. We don't throw or swallow errors anymore in `fetchCollation`, but rather perform assertions around assumptions which are validated during testing. Note that `fetchCollation` is only used internally and assumes that valid collation id is passed as parameter, this enables us to remove error handling logic; user-facing name-to-id parsing logic does throw error and has regular control flow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606770749
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -118,76 +119,433 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * Collation id is defined as 32-bit integer.
+ * We specify binary layouts for different classes of collations.
+ * Classes of collations are differentiated by most significant 3 bits (bit 31, 30 and 29),
+ * bit 31 being most significant and bit 0 being least significant.
+ * ---
+ * INDETERMINATE collation id binary layout:
+ * bit 31-0: 1
+ * INDETERMINATE collation id is equal to -1
+ * ---
+ * user-defined collation id binary layout:
+ * bit 31: 0
+ * bit 30: 1
+ * bit 29-0: undefined, reserved for future use
+ * ---
+ * UTF8_BINARY collation id binary layout:
+ * bit 31-22: zeroes
+ * bit 21-18: zeroes, reserved for space trimming
+ * bit 17-16: zeroes, reserved for version
+ * bit 15-3: zeroes
+ * bit 2: 0, reserved for accent sensitivity
+ * bit 1: 0, reserved for uppercase and case-insensitive
+ * bit 0: 0 = case-sensitive, 1 = lowercase
+ * ---
+ * ICU collation id binary layout:
+ * bit 31-30: zeroes
+ * bit 29: 1
+ * bit 28-24: zeroes
+ * bit 23-22: zeroes, reserved for version
+ * bit 21-18: zeroes, reserved for space trimming
+ * bit 17: 0 = case-sensitive, 1 = case-insensitive
+ * bit 16: 0 = accent-sensitive, 1 = accent-insensitive
+ * bit 15-14: zeroes, reserved for punctuation sensitivity
+ * bit 13-12: zeroes, reserved for first letter preference
+ * bit 11-0: locale id as specified in `ICULocaleToId` mapping
+ * ---
+ * Some illustrative examples of collation name to id mapping:
+ * - UTF8_BINARY -> 0
+ * - UTF8_BINARY_LCASE -> 1
+ * - UNICODE -> 0x20000000
+ * - UNICODE_AI -> 0x20010000
+ * - UNICODE_CI -> 0x20020000
+ * - UNICODE_CI_AI -> 0x20030000
+ * - af -> 0x20000001
+ * - af_CI_AI -> 0x20030001
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+
+ private enum DefinitionOrigin {
+ PREDEFINED, USER_DEFINED
+ }
+
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU
+ }
+
+ private static final int DEFINITION_ORIGIN_OFFSET = 30;
+ private static final int DEFINITION_ORIGIN_MASK = 0b1;
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b1;
+
+ private static final int INDETERMINATE_COLLATION_ID = -1;
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ private static ImplementationProvider getImplementationProvider(int collationId) {
+ return ImplementationProvider.values()[SpecifierUtils.getSpecValue(collationId,
+ IMPLEMENTATION_PROVIDER_OFFSET, IMPLEMENTATION_PROVIDER_MASK)];
+ }
+
+ private static DefinitionOrigin getDefinitionOrigin(int collationId) {
+ return DefinitionOrigin.values()[SpecifierUtils.getSpecValue(collationId,
+ DEFINITION_ORIGIN_OFFSET, DEFINITION_ORIGIN_MASK)];
+ }
+
+ private static Collation fetchCollation(int collationId) {
+ assert (collationId >= 0 && getDefinitionOrigin(collationId)
+ == DefinitionOrigin.PREDEFINED);
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ ImplementationProvider implementationProvider = getImplementationProvider(collationId);
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ private static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ private static final int CASE_SENSITIVITY_OFFSET = 0;
+ private static final int CASE_SENSITIVITY_MASK = 0b1;
+
+ private enum CaseSensitivity {
+ UNSPECIFIED, LCASE
+ }
+
+ private static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseSensitivity.UNSPECIFIED).collationId;
+ private static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseSensitivity.LCASE).collationId;
+ protected static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseSensitivity.UNSPECIFIED).buildCollation();
+ protected static Collation UTF8_BINARY_LCASE_COLLATION =
+ new CollationSpecUTF8Binary(CaseSensitivity.LCASE).buildCollation();
+
+ private final int collationId;
+
+ private CollationSpecUTF8Binary(CaseSensitivity caseSensitivity) {
+ this.collationId =
+ SpecifierUtils.setSpecValue(0, CASE_SENSITIVITY_OFFSET, caseSensitivity);
+ }
+
+ private static int collationNameToId(String originalName, String collationName)
+ throws SparkException {
+ if (UTF8_BINARY_COLLATION.collationName.equals(collationName)) {
+ return UTF8_BINARY_COLLATION_ID;
+ } else if (UTF8_BINARY_LCASE_COLLATION.collationName.equals(collationName)) {
+ return UTF8_BINARY_LCASE_COLLATION_ID;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+
+ private static CollationSpecUTF8Binary fromCollationId(int collationId) {
+ int caseConversionOrdinal = SpecifierUtils.getSpecValue(collationId,
+ CASE_SENSITIVITY_OFFSET, CASE_SENSITIVITY_MASK);
+ assert (SpecifierUtils.removeSpec(collationId,
+ CASE_SENSITIVITY_OFFSET, CASE_SENSITIVITY_MASK) == 0);
+ return new CollationSpecUTF8Binary(CaseSensitivity.values()[caseConversionOrdinal]);
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return new Collation("UTF8_BINARY", null, UTF8String::binaryCompare, "1.0",
+ s -> (long) s.hashCode(), true, true, false);
+ } else {
+ return new Collation("UTF8_BINARY_LCASE", null, UTF8String::compareLowerCase, "1.0",
+ s -> (long) s.toLowerCase().hashCode(), false, false, true);
+ }
+ }
+ }
+
+ private static class CollationSpecICU extends CollationSpec {
+
+ private enum CaseSensitivity {
+ CS, CI
+ }
+
+ private enum AccentSensitivity {
+ AS, AI
+ }
+
+ private static final int CASE_SENSITIVITY_OFFSET = 17;
+ private static final int CASE_SENSITIVITY_MASK = 0b1;
+ private static final int ACCENT_SENSITIVITY_OFFSET = 16;
+ private static final int ACCENT_SENSITIVITY_MASK = 0b1;
+
+ // Array of locale names, each locale id corresponds to the index in this array
+ private static final String[] ICULocaleNames;
+
+ // Mapping of locale names to corresponding `ULocale` instance
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+
+ // Used to parse user input collation names which are converted to uppercase
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+
+ // Reverse mapping of `ICULocaleNames`
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+
+ private static final String ICU_COLLATOR_VERSION = "153.120.0.0";
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
+ for (ULocale locale : locales) {
+ if (locale.getVariant().isEmpty()) {
+ String language = locale.getLanguage();
+ assert (!language.isEmpty());
+ StringBuilder builder = new StringBuilder(language);
+ String script = locale.getScript();
+ if (!script.isEmpty()) {
+ builder.append('_');
+ builder.append(script);
+ }
+ String country = locale.getISO3Country();
+ if (!country.isEmpty()) {
+ builder.append('_');
+ builder.append(country);
+ }
+ String localeName = builder.toString();
+ // locale names are unique
+ assert (!ICULocaleMap.containsKey(localeName));
+ ICULocaleMap.put(localeName, locale);
+ }
+ }
+ for (String localeName : ICULocaleMap.keySet()) {
+ String localeUppercase = localeName.toUpperCase();
+ // locale names are unique case-insensitively
+ assert (!ICULocaleMapUppercase.containsKey(localeUppercase));
+ ICULocaleMapUppercase.put(localeUppercase, localeName);
+ }
+ ICULocaleNames = ICULocaleMap.keySet().toArray(new String[0]);
+ Arrays.sort(ICULocaleNames);
+ // maximum number of locale ids as defined by binary layout
+ assert (ICULocaleNames.length <= (1 << 12));
+ for (int i = 0; i < ICULocaleNames.length; ++i) {
+ ICULocaleToId.put(ICULocaleNames[i], i);
+ }
+ }
+
+ private static final int UNICODE_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CS, AccentSensitivity.AS).collationId;
+ private static final int UNICODE_CI_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CI, AccentSensitivity.AS).collationId;
+
+ private final CaseSensitivity caseSensitivity;
+ private final AccentSensitivity accentSensitivity;
+ private final String locale;
+ private final int collationId;
+
+ private CollationSpecICU(String locale, CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ int collationId = ICULocaleToId.get(locale);
+ collationId = SpecifierUtils.setSpecValue(collationId, IMPLEMENTATION_PROVIDER_OFFSET,
+ ImplementationProvider.ICU);
+ collationId = SpecifierUtils.setSpecValue(collationId, CASE_SENSITIVITY_OFFSET,
+ caseSensitivity);
+ collationId = SpecifierUtils.setSpecValue(collationId, ACCENT_SENSITIVITY_OFFSET,
+ accentSensitivity);
+ this.collationId = collationId;
+ }
+
+ private static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ // search for the longest locale match because specifiers are designed to be different from
+ // script tag and country code, meaning the only valid locale name match can be
+ // the longest one
+ int lastPos = -1;
+ for (int i = 1; i <= collationName.length(); i++) {
+ String localeName = collationName.substring(0, i);
+ if (ICULocaleMapUppercase.containsKey(localeName)) {
+ lastPos = i;
+ }
+ }
+ if (lastPos == -1) {
+ throw collationInvalidNameException(originalName);
+ } else {
+ String locale = collationName.substring(0, lastPos);
+ int collationId = ICULocaleToId.get(ICULocaleMapUppercase.get(locale));
+
+ // try all combinations of AS/AI and CS/CI
+ CaseSensitivity caseSensitivity;
+ AccentSensitivity accentSensitivity;
+ if (collationName.equals(locale) ||
+ collationName.equals(locale + "_AS") ||
+ collationName.equals(locale + "_CS") ||
+ collationName.equals(locale + "_AS_CS") ||
+ collationName.equals(locale + "_CS_AS")
+ ) {
+ caseSensitivity = CaseSensitivity.CS;
+ accentSensitivity = AccentSensitivity.AS;
+ } else if (collationName.equals(locale + "_CI") ||
+ collationName.equals(locale + "_AS_CI") ||
+ collationName.equals(locale + "_CI_AS")) {
+ caseSensitivity = CaseSensitivity.CI;
+ accentSensitivity = AccentSensitivity.AS;
+ } else if (collationName.equals(locale + "_AI") ||
+ collationName.equals(locale + "_CS_AI") ||
+ collationName.equals(locale + "_AI_CS")) {
+ caseSensitivity = CaseSensitivity.CS;
+ accentSensitivity = AccentSensitivity.AI;
+ } else if (collationName.equals(locale + "_AI_CI") ||
+ collationName.equals(locale + "_CI_AI")) {
+ caseSensitivity = CaseSensitivity.CI;
+ accentSensitivity = AccentSensitivity.AI;
Review Comment:
It's true that the number of combinations will explode as we add new specifiers for space trimming, punctuation sensitivity, first letter preference etc., and we would need to rework this logic a bit then.
Since the agreement was to scope down this PR only to case/accent sensitivity, I found it simpler to manually specify all combinations and skip adding general-purpose specifier parser logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "mkaravel (via GitHub)" <gi...@apache.org>.
mkaravel commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1605449462
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -118,76 +119,433 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * Collation id is defined as 32-bit integer.
+ * We specify binary layouts for different classes of collations.
+ * Classes of collations are differentiated by most significant 3 bits (bit 31, 30 and 29),
+ * bit 31 being most significant and bit 0 being least significant.
+ * ---
+ * INDETERMINATE collation id binary layout:
+ * bit 31-0: 1
+ * INDETERMINATE collation id is equal to -1
+ * ---
+ * user-defined collation id binary layout:
+ * bit 31: 0
+ * bit 30: 1
+ * bit 29-0: undefined, reserved for future use
+ * ---
+ * UTF8_BINARY collation id binary layout:
+ * bit 31-22: zeroes
+ * bit 21-18: zeroes, reserved for space trimming
+ * bit 17-16: zeroes, reserved for version
+ * bit 15-3: zeroes
+ * bit 2: 0, reserved for accent sensitivity
+ * bit 1: 0, reserved for uppercase and case-insensitive
+ * bit 0: 0 = case-sensitive, 1 = lowercase
+ * ---
+ * ICU collation id binary layout:
+ * bit 31-30: zeroes
+ * bit 29: 1
+ * bit 28-24: zeroes
+ * bit 23-22: zeroes, reserved for version
Review Comment:
Is there a good reason why version appears at different positions for the UTF8_BINARY family and ICU collations?
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -118,76 +119,433 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * Collation id is defined as 32-bit integer.
+ * We specify binary layouts for different classes of collations.
+ * Classes of collations are differentiated by most significant 3 bits (bit 31, 30 and 29),
+ * bit 31 being most significant and bit 0 being least significant.
+ * ---
+ * INDETERMINATE collation id binary layout:
+ * bit 31-0: 1
+ * INDETERMINATE collation id is equal to -1
+ * ---
+ * user-defined collation id binary layout:
+ * bit 31: 0
+ * bit 30: 1
+ * bit 29-0: undefined, reserved for future use
+ * ---
+ * UTF8_BINARY collation id binary layout:
+ * bit 31-22: zeroes
+ * bit 21-18: zeroes, reserved for space trimming
+ * bit 17-16: zeroes, reserved for version
+ * bit 15-3: zeroes
+ * bit 2: 0, reserved for accent sensitivity
+ * bit 1: 0, reserved for uppercase and case-insensitive
+ * bit 0: 0 = case-sensitive, 1 = lowercase
+ * ---
+ * ICU collation id binary layout:
+ * bit 31-30: zeroes
+ * bit 29: 1
+ * bit 28-24: zeroes
+ * bit 23-22: zeroes, reserved for version
+ * bit 21-18: zeroes, reserved for space trimming
+ * bit 17: 0 = case-sensitive, 1 = case-insensitive
+ * bit 16: 0 = accent-sensitive, 1 = accent-insensitive
+ * bit 15-14: zeroes, reserved for punctuation sensitivity
+ * bit 13-12: zeroes, reserved for first letter preference
+ * bit 11-0: locale id as specified in `ICULocaleToId` mapping
+ * ---
+ * Some illustrative examples of collation name to id mapping:
+ * - UTF8_BINARY -> 0
+ * - UTF8_BINARY_LCASE -> 1
+ * - UNICODE -> 0x20000000
+ * - UNICODE_AI -> 0x20010000
+ * - UNICODE_CI -> 0x20020000
+ * - UNICODE_CI_AI -> 0x20030000
+ * - af -> 0x20000001
+ * - af_CI_AI -> 0x20030001
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+
+ private enum DefinitionOrigin {
Review Comment:
I do not want to block this PR for this, but I find the lack of comments for variables, inner classes, methods, etc. quite problematic. The only way to understand what is going on is by looking at the code, which could be okay in some cases, but it definitely does not help in understanding how the different pieces fit together.
##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -152,4 +219,218 @@ class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ig
}
})
}
+
+ test("test collation caching") {
+ Seq(
+ "UTF8_BINARY",
+ "UTF8_BINARY_LCASE",
+ "UNICODE",
+ "UNICODE_CI",
+ "UNICODE_AI",
+ "UNICODE_CI_AI",
+ "UNICODE_AI_CI"
+ ).foreach(collationId => {
+ val col1 = fetchCollation(collationId)
+ val col2 = fetchCollation(collationId)
+ assert(col1 eq col2) // reference equality
+ })
+ }
+
+ test("collations with ICU non-root localization") {
+ Seq(
+ // language only
+ "en",
+ "en_CS",
+ "en_CI",
+ "en_AS",
+ "en_AI",
+ // language + 3-letter country code
+ "en_USA",
+ "en_USA_CS",
+ "en_USA_CI",
+ "en_USA_AS",
+ "en_USA_AI",
+ // language + script code
+ "sr_Cyrl",
+ "sr_Cyrl_CS",
+ "sr_Cyrl_CI",
+ "sr_Cyrl_AS",
+ "sr_Cyrl_AI",
+ // language + script code + 3-letter country code
+ "sr_Cyrl_SRB",
+ "sr_Cyrl_SRB_CS",
+ "sr_Cyrl_SRB_CI",
+ "sr_Cyrl_SRB_AS",
+ "sr_Cyrl_SRB_AI"
+ ).foreach(collationICU => {
+ val col = fetchCollation(collationICU)
+ assert(col.collator.getLocale(ULocale.VALID_LOCALE) != ULocale.ROOT)
+ })
+ }
+
+ test("invalid names of collations with ICU non-root localization") {
Review Comment:
Same ask here for conflicting specifiers.
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -118,76 +119,433 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * Collation id is defined as 32-bit integer.
+ * We specify binary layouts for different classes of collations.
+ * Classes of collations are differentiated by most significant 3 bits (bit 31, 30 and 29),
+ * bit 31 being most significant and bit 0 being least significant.
+ * ---
+ * INDETERMINATE collation id binary layout:
+ * bit 31-0: 1
+ * INDETERMINATE collation id is equal to -1
+ * ---
+ * user-defined collation id binary layout:
+ * bit 31: 0
+ * bit 30: 1
+ * bit 29-0: undefined, reserved for future use
+ * ---
+ * UTF8_BINARY collation id binary layout:
+ * bit 31-22: zeroes
+ * bit 21-18: zeroes, reserved for space trimming
+ * bit 17-16: zeroes, reserved for version
+ * bit 15-3: zeroes
+ * bit 2: 0, reserved for accent sensitivity
+ * bit 1: 0, reserved for uppercase and case-insensitive
+ * bit 0: 0 = case-sensitive, 1 = lowercase
+ * ---
+ * ICU collation id binary layout:
+ * bit 31-30: zeroes
+ * bit 29: 1
+ * bit 28-24: zeroes
+ * bit 23-22: zeroes, reserved for version
+ * bit 21-18: zeroes, reserved for space trimming
+ * bit 17: 0 = case-sensitive, 1 = case-insensitive
+ * bit 16: 0 = accent-sensitive, 1 = accent-insensitive
+ * bit 15-14: zeroes, reserved for punctuation sensitivity
+ * bit 13-12: zeroes, reserved for first letter preference
+ * bit 11-0: locale id as specified in `ICULocaleToId` mapping
+ * ---
+ * Some illustrative examples of collation name to id mapping:
+ * - UTF8_BINARY -> 0
+ * - UTF8_BINARY_LCASE -> 1
+ * - UNICODE -> 0x20000000
+ * - UNICODE_AI -> 0x20010000
+ * - UNICODE_CI -> 0x20020000
+ * - UNICODE_CI_AI -> 0x20030000
+ * - af -> 0x20000001
+ * - af_CI_AI -> 0x20030001
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+
+ private enum DefinitionOrigin {
+ PREDEFINED, USER_DEFINED
+ }
+
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU
+ }
+
+ private static final int DEFINITION_ORIGIN_OFFSET = 30;
+ private static final int DEFINITION_ORIGIN_MASK = 0b1;
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b1;
+
+ private static final int INDETERMINATE_COLLATION_ID = -1;
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ private static ImplementationProvider getImplementationProvider(int collationId) {
+ return ImplementationProvider.values()[SpecifierUtils.getSpecValue(collationId,
+ IMPLEMENTATION_PROVIDER_OFFSET, IMPLEMENTATION_PROVIDER_MASK)];
+ }
+
+ private static DefinitionOrigin getDefinitionOrigin(int collationId) {
+ return DefinitionOrigin.values()[SpecifierUtils.getSpecValue(collationId,
+ DEFINITION_ORIGIN_OFFSET, DEFINITION_ORIGIN_MASK)];
+ }
+
+ private static Collation fetchCollation(int collationId) {
+ assert (collationId >= 0 && getDefinitionOrigin(collationId)
+ == DefinitionOrigin.PREDEFINED);
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ ImplementationProvider implementationProvider = getImplementationProvider(collationId);
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ private static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ private static final int CASE_SENSITIVITY_OFFSET = 0;
+ private static final int CASE_SENSITIVITY_MASK = 0b1;
+
+ private enum CaseSensitivity {
+ UNSPECIFIED, LCASE
+ }
+
+ private static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseSensitivity.UNSPECIFIED).collationId;
+ private static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseSensitivity.LCASE).collationId;
+ protected static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseSensitivity.UNSPECIFIED).buildCollation();
+ protected static Collation UTF8_BINARY_LCASE_COLLATION =
+ new CollationSpecUTF8Binary(CaseSensitivity.LCASE).buildCollation();
+
+ private final int collationId;
+
+ private CollationSpecUTF8Binary(CaseSensitivity caseSensitivity) {
+ this.collationId =
+ SpecifierUtils.setSpecValue(0, CASE_SENSITIVITY_OFFSET, caseSensitivity);
+ }
+
+ private static int collationNameToId(String originalName, String collationName)
+ throws SparkException {
+ if (UTF8_BINARY_COLLATION.collationName.equals(collationName)) {
+ return UTF8_BINARY_COLLATION_ID;
+ } else if (UTF8_BINARY_LCASE_COLLATION.collationName.equals(collationName)) {
+ return UTF8_BINARY_LCASE_COLLATION_ID;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+
+ private static CollationSpecUTF8Binary fromCollationId(int collationId) {
+ int caseConversionOrdinal = SpecifierUtils.getSpecValue(collationId,
+ CASE_SENSITIVITY_OFFSET, CASE_SENSITIVITY_MASK);
+ assert (SpecifierUtils.removeSpec(collationId,
+ CASE_SENSITIVITY_OFFSET, CASE_SENSITIVITY_MASK) == 0);
+ return new CollationSpecUTF8Binary(CaseSensitivity.values()[caseConversionOrdinal]);
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return new Collation("UTF8_BINARY", null, UTF8String::binaryCompare, "1.0",
+ s -> (long) s.hashCode(), true, true, false);
+ } else {
+ return new Collation("UTF8_BINARY_LCASE", null, UTF8String::compareLowerCase, "1.0",
+ s -> (long) s.toLowerCase().hashCode(), false, false, true);
+ }
+ }
+ }
+
+ private static class CollationSpecICU extends CollationSpec {
+
+ private enum CaseSensitivity {
+ CS, CI
+ }
+
+ private enum AccentSensitivity {
+ AS, AI
+ }
+
+ private static final int CASE_SENSITIVITY_OFFSET = 17;
+ private static final int CASE_SENSITIVITY_MASK = 0b1;
+ private static final int ACCENT_SENSITIVITY_OFFSET = 16;
+ private static final int ACCENT_SENSITIVITY_MASK = 0b1;
+
+ // Array of locale names, each locale id corresponds to the index in this array
+ private static final String[] ICULocaleNames;
+
+ // Mapping of locale names to corresponding `ULocale` instance
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+
+ // Used to parse user input collation names which are converted to uppercase
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+
+ // Reverse mapping of `ICULocaleNames`
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+
+ private static final String ICU_COLLATOR_VERSION = "153.120.0.0";
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
+ for (ULocale locale : locales) {
+ if (locale.getVariant().isEmpty()) {
+ String language = locale.getLanguage();
+ assert (!language.isEmpty());
+ StringBuilder builder = new StringBuilder(language);
+ String script = locale.getScript();
+ if (!script.isEmpty()) {
+ builder.append('_');
+ builder.append(script);
+ }
+ String country = locale.getISO3Country();
+ if (!country.isEmpty()) {
+ builder.append('_');
+ builder.append(country);
+ }
+ String localeName = builder.toString();
+ // locale names are unique
+ assert (!ICULocaleMap.containsKey(localeName));
+ ICULocaleMap.put(localeName, locale);
+ }
+ }
+ for (String localeName : ICULocaleMap.keySet()) {
+ String localeUppercase = localeName.toUpperCase();
+ // locale names are unique case-insensitively
+ assert (!ICULocaleMapUppercase.containsKey(localeUppercase));
+ ICULocaleMapUppercase.put(localeUppercase, localeName);
+ }
+ ICULocaleNames = ICULocaleMap.keySet().toArray(new String[0]);
+ Arrays.sort(ICULocaleNames);
+ // maximum number of locale ids as defined by binary layout
Review Comment:
```suggestion
// Maximum number of locale IDs as defined by binary layout.
```
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -118,76 +119,433 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * Collation id is defined as 32-bit integer.
+ * We specify binary layouts for different classes of collations.
+ * Classes of collations are differentiated by most significant 3 bits (bit 31, 30 and 29),
+ * bit 31 being most significant and bit 0 being least significant.
+ * ---
+ * INDETERMINATE collation id binary layout:
+ * bit 31-0: 1
+ * INDETERMINATE collation id is equal to -1
+ * ---
+ * user-defined collation id binary layout:
+ * bit 31: 0
+ * bit 30: 1
+ * bit 29-0: undefined, reserved for future use
+ * ---
+ * UTF8_BINARY collation id binary layout:
+ * bit 31-22: zeroes
+ * bit 21-18: zeroes, reserved for space trimming
+ * bit 17-16: zeroes, reserved for version
+ * bit 15-3: zeroes
+ * bit 2: 0, reserved for accent sensitivity
+ * bit 1: 0, reserved for uppercase and case-insensitive
+ * bit 0: 0 = case-sensitive, 1 = lowercase
+ * ---
+ * ICU collation id binary layout:
+ * bit 31-30: zeroes
+ * bit 29: 1
+ * bit 28-24: zeroes
+ * bit 23-22: zeroes, reserved for version
+ * bit 21-18: zeroes, reserved for space trimming
+ * bit 17: 0 = case-sensitive, 1 = case-insensitive
+ * bit 16: 0 = accent-sensitive, 1 = accent-insensitive
+ * bit 15-14: zeroes, reserved for punctuation sensitivity
+ * bit 13-12: zeroes, reserved for first letter preference
+ * bit 11-0: locale id as specified in `ICULocaleToId` mapping
+ * ---
+ * Some illustrative examples of collation name to id mapping:
+ * - UTF8_BINARY -> 0
+ * - UTF8_BINARY_LCASE -> 1
+ * - UNICODE -> 0x20000000
+ * - UNICODE_AI -> 0x20010000
+ * - UNICODE_CI -> 0x20020000
+ * - UNICODE_CI_AI -> 0x20030000
+ * - af -> 0x20000001
+ * - af_CI_AI -> 0x20030001
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+
+ private enum DefinitionOrigin {
+ PREDEFINED, USER_DEFINED
+ }
+
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU
+ }
+
+ private static final int DEFINITION_ORIGIN_OFFSET = 30;
+ private static final int DEFINITION_ORIGIN_MASK = 0b1;
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b1;
+
+ private static final int INDETERMINATE_COLLATION_ID = -1;
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ private static ImplementationProvider getImplementationProvider(int collationId) {
+ return ImplementationProvider.values()[SpecifierUtils.getSpecValue(collationId,
+ IMPLEMENTATION_PROVIDER_OFFSET, IMPLEMENTATION_PROVIDER_MASK)];
+ }
+
+ private static DefinitionOrigin getDefinitionOrigin(int collationId) {
+ return DefinitionOrigin.values()[SpecifierUtils.getSpecValue(collationId,
+ DEFINITION_ORIGIN_OFFSET, DEFINITION_ORIGIN_MASK)];
+ }
+
+ private static Collation fetchCollation(int collationId) {
+ assert (collationId >= 0 && getDefinitionOrigin(collationId)
+ == DefinitionOrigin.PREDEFINED);
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ ImplementationProvider implementationProvider = getImplementationProvider(collationId);
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ private static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ private static final int CASE_SENSITIVITY_OFFSET = 0;
+ private static final int CASE_SENSITIVITY_MASK = 0b1;
+
+ private enum CaseSensitivity {
+ UNSPECIFIED, LCASE
+ }
+
+ private static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseSensitivity.UNSPECIFIED).collationId;
+ private static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseSensitivity.LCASE).collationId;
+ protected static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseSensitivity.UNSPECIFIED).buildCollation();
+ protected static Collation UTF8_BINARY_LCASE_COLLATION =
+ new CollationSpecUTF8Binary(CaseSensitivity.LCASE).buildCollation();
+
+ private final int collationId;
+
+ private CollationSpecUTF8Binary(CaseSensitivity caseSensitivity) {
+ this.collationId =
+ SpecifierUtils.setSpecValue(0, CASE_SENSITIVITY_OFFSET, caseSensitivity);
+ }
+
+ private static int collationNameToId(String originalName, String collationName)
+ throws SparkException {
+ if (UTF8_BINARY_COLLATION.collationName.equals(collationName)) {
+ return UTF8_BINARY_COLLATION_ID;
+ } else if (UTF8_BINARY_LCASE_COLLATION.collationName.equals(collationName)) {
+ return UTF8_BINARY_LCASE_COLLATION_ID;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+
+ private static CollationSpecUTF8Binary fromCollationId(int collationId) {
+ int caseConversionOrdinal = SpecifierUtils.getSpecValue(collationId,
+ CASE_SENSITIVITY_OFFSET, CASE_SENSITIVITY_MASK);
+ assert (SpecifierUtils.removeSpec(collationId,
+ CASE_SENSITIVITY_OFFSET, CASE_SENSITIVITY_MASK) == 0);
+ return new CollationSpecUTF8Binary(CaseSensitivity.values()[caseConversionOrdinal]);
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return new Collation("UTF8_BINARY", null, UTF8String::binaryCompare, "1.0",
+ s -> (long) s.hashCode(), true, true, false);
+ } else {
+ return new Collation("UTF8_BINARY_LCASE", null, UTF8String::compareLowerCase, "1.0",
+ s -> (long) s.toLowerCase().hashCode(), false, false, true);
+ }
+ }
+ }
+
+ private static class CollationSpecICU extends CollationSpec {
+
+ private enum CaseSensitivity {
+ CS, CI
+ }
+
+ private enum AccentSensitivity {
+ AS, AI
+ }
+
+ private static final int CASE_SENSITIVITY_OFFSET = 17;
+ private static final int CASE_SENSITIVITY_MASK = 0b1;
+ private static final int ACCENT_SENSITIVITY_OFFSET = 16;
+ private static final int ACCENT_SENSITIVITY_MASK = 0b1;
+
+ // Array of locale names, each locale id corresponds to the index in this array
+ private static final String[] ICULocaleNames;
+
+ // Mapping of locale names to corresponding `ULocale` instance
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+
+ // Used to parse user input collation names which are converted to uppercase
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+
+ // Reverse mapping of `ICULocaleNames`
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+
+ private static final String ICU_COLLATOR_VERSION = "153.120.0.0";
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
+ for (ULocale locale : locales) {
+ if (locale.getVariant().isEmpty()) {
+ String language = locale.getLanguage();
+ assert (!language.isEmpty());
+ StringBuilder builder = new StringBuilder(language);
+ String script = locale.getScript();
+ if (!script.isEmpty()) {
+ builder.append('_');
+ builder.append(script);
+ }
+ String country = locale.getISO3Country();
+ if (!country.isEmpty()) {
+ builder.append('_');
+ builder.append(country);
+ }
+ String localeName = builder.toString();
+ // locale names are unique
+ assert (!ICULocaleMap.containsKey(localeName));
+ ICULocaleMap.put(localeName, locale);
+ }
+ }
+ for (String localeName : ICULocaleMap.keySet()) {
+ String localeUppercase = localeName.toUpperCase();
+ // locale names are unique case-insensitively
+ assert (!ICULocaleMapUppercase.containsKey(localeUppercase));
+ ICULocaleMapUppercase.put(localeUppercase, localeName);
+ }
+ ICULocaleNames = ICULocaleMap.keySet().toArray(new String[0]);
+ Arrays.sort(ICULocaleNames);
+ // maximum number of locale ids as defined by binary layout
+ assert (ICULocaleNames.length <= (1 << 12));
+ for (int i = 0; i < ICULocaleNames.length; ++i) {
+ ICULocaleToId.put(ICULocaleNames[i], i);
+ }
+ }
+
+ private static final int UNICODE_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CS, AccentSensitivity.AS).collationId;
+ private static final int UNICODE_CI_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CI, AccentSensitivity.AS).collationId;
+
+ private final CaseSensitivity caseSensitivity;
+ private final AccentSensitivity accentSensitivity;
+ private final String locale;
+ private final int collationId;
+
+ private CollationSpecICU(String locale, CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ int collationId = ICULocaleToId.get(locale);
+ collationId = SpecifierUtils.setSpecValue(collationId, IMPLEMENTATION_PROVIDER_OFFSET,
+ ImplementationProvider.ICU);
+ collationId = SpecifierUtils.setSpecValue(collationId, CASE_SENSITIVITY_OFFSET,
+ caseSensitivity);
+ collationId = SpecifierUtils.setSpecValue(collationId, ACCENT_SENSITIVITY_OFFSET,
+ accentSensitivity);
+ this.collationId = collationId;
+ }
+
+ private static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ // search for the longest locale match because specifiers are designed to be different from
+ // script tag and country code, meaning the only valid locale name match can be
+ // the longest one
+ int lastPos = -1;
+ for (int i = 1; i <= collationName.length(); i++) {
+ String localeName = collationName.substring(0, i);
+ if (ICULocaleMapUppercase.containsKey(localeName)) {
+ lastPos = i;
+ }
+ }
+ if (lastPos == -1) {
+ throw collationInvalidNameException(originalName);
+ } else {
+ String locale = collationName.substring(0, lastPos);
+ int collationId = ICULocaleToId.get(ICULocaleMapUppercase.get(locale));
+
+ // try all combinations of AS/AI and CS/CI
+ CaseSensitivity caseSensitivity;
+ AccentSensitivity accentSensitivity;
+ if (collationName.equals(locale) ||
+ collationName.equals(locale + "_AS") ||
+ collationName.equals(locale + "_CS") ||
+ collationName.equals(locale + "_AS_CS") ||
+ collationName.equals(locale + "_CS_AS")
+ ) {
+ caseSensitivity = CaseSensitivity.CS;
+ accentSensitivity = AccentSensitivity.AS;
+ } else if (collationName.equals(locale + "_CI") ||
+ collationName.equals(locale + "_AS_CI") ||
+ collationName.equals(locale + "_CI_AS")) {
+ caseSensitivity = CaseSensitivity.CI;
+ accentSensitivity = AccentSensitivity.AS;
+ } else if (collationName.equals(locale + "_AI") ||
+ collationName.equals(locale + "_CS_AI") ||
+ collationName.equals(locale + "_AI_CS")) {
+ caseSensitivity = CaseSensitivity.CS;
+ accentSensitivity = AccentSensitivity.AI;
+ } else if (collationName.equals(locale + "_AI_CI") ||
+ collationName.equals(locale + "_CI_AI")) {
+ caseSensitivity = CaseSensitivity.CI;
+ accentSensitivity = AccentSensitivity.AI;
Review Comment:
When we have trimming, do we expect a combinatorial blow up on that we have to test for equality, to do we plan to follow a string substring search approach?
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -118,76 +119,433 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * Collation id is defined as 32-bit integer.
+ * We specify binary layouts for different classes of collations.
+ * Classes of collations are differentiated by most significant 3 bits (bit 31, 30 and 29),
+ * bit 31 being most significant and bit 0 being least significant.
+ * ---
+ * INDETERMINATE collation id binary layout:
+ * bit 31-0: 1
+ * INDETERMINATE collation id is equal to -1
+ * ---
+ * user-defined collation id binary layout:
+ * bit 31: 0
+ * bit 30: 1
+ * bit 29-0: undefined, reserved for future use
+ * ---
+ * UTF8_BINARY collation id binary layout:
+ * bit 31-22: zeroes
+ * bit 21-18: zeroes, reserved for space trimming
+ * bit 17-16: zeroes, reserved for version
+ * bit 15-3: zeroes
+ * bit 2: 0, reserved for accent sensitivity
+ * bit 1: 0, reserved for uppercase and case-insensitive
+ * bit 0: 0 = case-sensitive, 1 = lowercase
+ * ---
+ * ICU collation id binary layout:
+ * bit 31-30: zeroes
+ * bit 29: 1
+ * bit 28-24: zeroes
+ * bit 23-22: zeroes, reserved for version
+ * bit 21-18: zeroes, reserved for space trimming
+ * bit 17: 0 = case-sensitive, 1 = case-insensitive
+ * bit 16: 0 = accent-sensitive, 1 = accent-insensitive
+ * bit 15-14: zeroes, reserved for punctuation sensitivity
+ * bit 13-12: zeroes, reserved for first letter preference
+ * bit 11-0: locale id as specified in `ICULocaleToId` mapping
+ * ---
+ * Some illustrative examples of collation name to id mapping:
+ * - UTF8_BINARY -> 0
+ * - UTF8_BINARY_LCASE -> 1
+ * - UNICODE -> 0x20000000
+ * - UNICODE_AI -> 0x20010000
+ * - UNICODE_CI -> 0x20020000
+ * - UNICODE_CI_AI -> 0x20030000
+ * - af -> 0x20000001
+ * - af_CI_AI -> 0x20030001
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+
+ private enum DefinitionOrigin {
+ PREDEFINED, USER_DEFINED
+ }
+
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU
+ }
+
+ private static final int DEFINITION_ORIGIN_OFFSET = 30;
+ private static final int DEFINITION_ORIGIN_MASK = 0b1;
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b1;
+
+ private static final int INDETERMINATE_COLLATION_ID = -1;
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ private static ImplementationProvider getImplementationProvider(int collationId) {
+ return ImplementationProvider.values()[SpecifierUtils.getSpecValue(collationId,
+ IMPLEMENTATION_PROVIDER_OFFSET, IMPLEMENTATION_PROVIDER_MASK)];
+ }
+
+ private static DefinitionOrigin getDefinitionOrigin(int collationId) {
+ return DefinitionOrigin.values()[SpecifierUtils.getSpecValue(collationId,
+ DEFINITION_ORIGIN_OFFSET, DEFINITION_ORIGIN_MASK)];
+ }
+
+ private static Collation fetchCollation(int collationId) {
+ assert (collationId >= 0 && getDefinitionOrigin(collationId)
+ == DefinitionOrigin.PREDEFINED);
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ ImplementationProvider implementationProvider = getImplementationProvider(collationId);
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ private static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ private static final int CASE_SENSITIVITY_OFFSET = 0;
+ private static final int CASE_SENSITIVITY_MASK = 0b1;
+
+ private enum CaseSensitivity {
+ UNSPECIFIED, LCASE
+ }
+
+ private static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseSensitivity.UNSPECIFIED).collationId;
+ private static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseSensitivity.LCASE).collationId;
+ protected static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseSensitivity.UNSPECIFIED).buildCollation();
+ protected static Collation UTF8_BINARY_LCASE_COLLATION =
+ new CollationSpecUTF8Binary(CaseSensitivity.LCASE).buildCollation();
+
+ private final int collationId;
+
+ private CollationSpecUTF8Binary(CaseSensitivity caseSensitivity) {
+ this.collationId =
+ SpecifierUtils.setSpecValue(0, CASE_SENSITIVITY_OFFSET, caseSensitivity);
+ }
+
+ private static int collationNameToId(String originalName, String collationName)
+ throws SparkException {
+ if (UTF8_BINARY_COLLATION.collationName.equals(collationName)) {
+ return UTF8_BINARY_COLLATION_ID;
+ } else if (UTF8_BINARY_LCASE_COLLATION.collationName.equals(collationName)) {
+ return UTF8_BINARY_LCASE_COLLATION_ID;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+
+ private static CollationSpecUTF8Binary fromCollationId(int collationId) {
+ int caseConversionOrdinal = SpecifierUtils.getSpecValue(collationId,
+ CASE_SENSITIVITY_OFFSET, CASE_SENSITIVITY_MASK);
+ assert (SpecifierUtils.removeSpec(collationId,
+ CASE_SENSITIVITY_OFFSET, CASE_SENSITIVITY_MASK) == 0);
+ return new CollationSpecUTF8Binary(CaseSensitivity.values()[caseConversionOrdinal]);
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return new Collation("UTF8_BINARY", null, UTF8String::binaryCompare, "1.0",
+ s -> (long) s.hashCode(), true, true, false);
+ } else {
+ return new Collation("UTF8_BINARY_LCASE", null, UTF8String::compareLowerCase, "1.0",
+ s -> (long) s.toLowerCase().hashCode(), false, false, true);
+ }
+ }
+ }
+
+ private static class CollationSpecICU extends CollationSpec {
+
+ private enum CaseSensitivity {
+ CS, CI
+ }
+
+ private enum AccentSensitivity {
+ AS, AI
+ }
+
+ private static final int CASE_SENSITIVITY_OFFSET = 17;
+ private static final int CASE_SENSITIVITY_MASK = 0b1;
+ private static final int ACCENT_SENSITIVITY_OFFSET = 16;
+ private static final int ACCENT_SENSITIVITY_MASK = 0b1;
+
+ // Array of locale names, each locale id corresponds to the index in this array
+ private static final String[] ICULocaleNames;
+
+ // Mapping of locale names to corresponding `ULocale` instance
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+
+ // Used to parse user input collation names which are converted to uppercase
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+
+ // Reverse mapping of `ICULocaleNames`
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+
+ private static final String ICU_COLLATOR_VERSION = "153.120.0.0";
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
+ for (ULocale locale : locales) {
+ if (locale.getVariant().isEmpty()) {
+ String language = locale.getLanguage();
+ assert (!language.isEmpty());
+ StringBuilder builder = new StringBuilder(language);
+ String script = locale.getScript();
+ if (!script.isEmpty()) {
+ builder.append('_');
+ builder.append(script);
+ }
+ String country = locale.getISO3Country();
+ if (!country.isEmpty()) {
+ builder.append('_');
+ builder.append(country);
+ }
+ String localeName = builder.toString();
+ // locale names are unique
+ assert (!ICULocaleMap.containsKey(localeName));
+ ICULocaleMap.put(localeName, locale);
+ }
+ }
+ for (String localeName : ICULocaleMap.keySet()) {
+ String localeUppercase = localeName.toUpperCase();
+ // locale names are unique case-insensitively
Review Comment:
```suggestion
// Locale names are unique case-insensitively.
```
##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -30,31 +33,95 @@ import org.apache.spark.unsafe.types.UTF8String.{fromString => toUTF8}
class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ignore funsuite
test("collationId stability") {
- val utf8Binary = fetchCollation(0)
+ assert(INDETERMINATE_COLLATION_ID == -1)
+
+ assert(UTF8_BINARY_COLLATION_ID == 0)
+ val utf8Binary = fetchCollation(UTF8_BINARY_COLLATION_ID)
assert(utf8Binary.collationName == "UTF8_BINARY")
assert(utf8Binary.supportsBinaryEquality)
- val utf8BinaryLcase = fetchCollation(1)
+ assert(UTF8_BINARY_LCASE_COLLATION_ID == 1)
+ val utf8BinaryLcase = fetchCollation(UTF8_BINARY_LCASE_COLLATION_ID)
assert(utf8BinaryLcase.collationName == "UTF8_BINARY_LCASE")
assert(!utf8BinaryLcase.supportsBinaryEquality)
- val unicode = fetchCollation(2)
+ assert(UNICODE_COLLATION_ID == (1 << 29))
+ val unicode = fetchCollation(UNICODE_COLLATION_ID)
assert(unicode.collationName == "UNICODE")
- assert(unicode.supportsBinaryEquality);
+ assert(unicode.supportsBinaryEquality)
- val unicodeCi = fetchCollation(3)
+ assert(UNICODE_CI_COLLATION_ID == ((1 << 29) | (1 << 17)))
+ val unicodeCi = fetchCollation(UNICODE_CI_COLLATION_ID)
assert(unicodeCi.collationName == "UNICODE_CI")
assert(!unicodeCi.supportsBinaryEquality)
}
- test("fetch invalid collation name") {
- val error = intercept[SparkException] {
- fetchCollation("UTF8_BS")
+ test("UTF8_BINARY and ICU root locale collation names") {
+ // collation name already normalized
+ Seq(
+ "UTF8_BINARY",
+ "UTF8_BINARY_LCASE",
+ "UNICODE",
+ "UNICODE_CI",
+ "UNICODE_AI",
+ "UNICODE_CI_AI"
+ ).foreach(collationName => {
+ val col = fetchCollation(collationName)
+ assert(col.collationName == collationName)
+ })
+ // collation name normalization
+ Seq(
+ // ICU root locale
+ ("UNICODE_CS", "UNICODE"),
+ ("UNICODE_CS_AS", "UNICODE"),
+ ("UNICODE_CI_AS", "UNICODE_CI"),
+ ("UNICODE_AI_CS", "UNICODE_AI"),
+ ("UNICODE_AI_CI", "UNICODE_CI_AI"),
+ // randomized case collation names
+ ("utf8_binary", "UTF8_BINARY"),
+ ("UtF8_binARy_LcasE", "UTF8_BINARY_LCASE"),
+ ("unicode", "UNICODE"),
+ ("UnICoDe_cs_aI", "UNICODE_AI")
+ ).foreach{
+ case (name, normalized) =>
+ val col = fetchCollation(name)
+ assert(col.collationName == normalized)
}
+ }
+
+ test("fetch invalid UTF8_BINARY and ICU root locale collation names") {
Review Comment:
Can we add test cases where we specify conflicting specifiers? For example "UNICODE_CI_CS" or "UNICODE_AS_AI".
##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -152,4 +219,218 @@ class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ig
}
})
}
+
+ test("test collation caching") {
+ Seq(
+ "UTF8_BINARY",
+ "UTF8_BINARY_LCASE",
+ "UNICODE",
+ "UNICODE_CI",
+ "UNICODE_AI",
+ "UNICODE_CI_AI",
+ "UNICODE_AI_CI"
+ ).foreach(collationId => {
+ val col1 = fetchCollation(collationId)
+ val col2 = fetchCollation(collationId)
+ assert(col1 eq col2) // reference equality
+ })
+ }
+
+ test("collations with ICU non-root localization") {
+ Seq(
+ // language only
+ "en",
+ "en_CS",
+ "en_CI",
+ "en_AS",
+ "en_AI",
+ // language + 3-letter country code
+ "en_USA",
+ "en_USA_CS",
+ "en_USA_CI",
+ "en_USA_AS",
+ "en_USA_AI",
+ // language + script code
+ "sr_Cyrl",
+ "sr_Cyrl_CS",
+ "sr_Cyrl_CI",
+ "sr_Cyrl_AS",
+ "sr_Cyrl_AI",
+ // language + script code + 3-letter country code
+ "sr_Cyrl_SRB",
+ "sr_Cyrl_SRB_CS",
+ "sr_Cyrl_SRB_CI",
+ "sr_Cyrl_SRB_AS",
+ "sr_Cyrl_SRB_AI"
+ ).foreach(collationICU => {
+ val col = fetchCollation(collationICU)
+ assert(col.collator.getLocale(ULocale.VALID_LOCALE) != ULocale.ROOT)
+ })
+ }
+
+ test("invalid names of collations with ICU non-root localization") {
Review Comment:
Also test cases where the script or the country code are not at the right position?
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -118,76 +119,433 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * Collation id is defined as 32-bit integer.
+ * We specify binary layouts for different classes of collations.
+ * Classes of collations are differentiated by most significant 3 bits (bit 31, 30 and 29),
+ * bit 31 being most significant and bit 0 being least significant.
+ * ---
+ * INDETERMINATE collation id binary layout:
+ * bit 31-0: 1
+ * INDETERMINATE collation id is equal to -1
+ * ---
+ * user-defined collation id binary layout:
+ * bit 31: 0
+ * bit 30: 1
+ * bit 29-0: undefined, reserved for future use
+ * ---
+ * UTF8_BINARY collation id binary layout:
+ * bit 31-22: zeroes
+ * bit 21-18: zeroes, reserved for space trimming
+ * bit 17-16: zeroes, reserved for version
+ * bit 15-3: zeroes
+ * bit 2: 0, reserved for accent sensitivity
+ * bit 1: 0, reserved for uppercase and case-insensitive
+ * bit 0: 0 = case-sensitive, 1 = lowercase
+ * ---
+ * ICU collation id binary layout:
+ * bit 31-30: zeroes
+ * bit 29: 1
+ * bit 28-24: zeroes
+ * bit 23-22: zeroes, reserved for version
+ * bit 21-18: zeroes, reserved for space trimming
+ * bit 17: 0 = case-sensitive, 1 = case-insensitive
+ * bit 16: 0 = accent-sensitive, 1 = accent-insensitive
+ * bit 15-14: zeroes, reserved for punctuation sensitivity
+ * bit 13-12: zeroes, reserved for first letter preference
+ * bit 11-0: locale id as specified in `ICULocaleToId` mapping
+ * ---
+ * Some illustrative examples of collation name to id mapping:
+ * - UTF8_BINARY -> 0
+ * - UTF8_BINARY_LCASE -> 1
+ * - UNICODE -> 0x20000000
+ * - UNICODE_AI -> 0x20010000
+ * - UNICODE_CI -> 0x20020000
+ * - UNICODE_CI_AI -> 0x20030000
+ * - af -> 0x20000001
+ * - af_CI_AI -> 0x20030001
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+
+ private enum DefinitionOrigin {
+ PREDEFINED, USER_DEFINED
+ }
+
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU
+ }
+
+ private static final int DEFINITION_ORIGIN_OFFSET = 30;
+ private static final int DEFINITION_ORIGIN_MASK = 0b1;
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b1;
+
+ private static final int INDETERMINATE_COLLATION_ID = -1;
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ private static ImplementationProvider getImplementationProvider(int collationId) {
+ return ImplementationProvider.values()[SpecifierUtils.getSpecValue(collationId,
+ IMPLEMENTATION_PROVIDER_OFFSET, IMPLEMENTATION_PROVIDER_MASK)];
+ }
+
+ private static DefinitionOrigin getDefinitionOrigin(int collationId) {
+ return DefinitionOrigin.values()[SpecifierUtils.getSpecValue(collationId,
+ DEFINITION_ORIGIN_OFFSET, DEFINITION_ORIGIN_MASK)];
+ }
+
+ private static Collation fetchCollation(int collationId) {
+ assert (collationId >= 0 && getDefinitionOrigin(collationId)
+ == DefinitionOrigin.PREDEFINED);
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ ImplementationProvider implementationProvider = getImplementationProvider(collationId);
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ private static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ private static final int CASE_SENSITIVITY_OFFSET = 0;
+ private static final int CASE_SENSITIVITY_MASK = 0b1;
+
+ private enum CaseSensitivity {
+ UNSPECIFIED, LCASE
+ }
+
+ private static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseSensitivity.UNSPECIFIED).collationId;
+ private static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseSensitivity.LCASE).collationId;
+ protected static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseSensitivity.UNSPECIFIED).buildCollation();
+ protected static Collation UTF8_BINARY_LCASE_COLLATION =
+ new CollationSpecUTF8Binary(CaseSensitivity.LCASE).buildCollation();
+
+ private final int collationId;
+
+ private CollationSpecUTF8Binary(CaseSensitivity caseSensitivity) {
+ this.collationId =
+ SpecifierUtils.setSpecValue(0, CASE_SENSITIVITY_OFFSET, caseSensitivity);
+ }
+
+ private static int collationNameToId(String originalName, String collationName)
+ throws SparkException {
+ if (UTF8_BINARY_COLLATION.collationName.equals(collationName)) {
+ return UTF8_BINARY_COLLATION_ID;
+ } else if (UTF8_BINARY_LCASE_COLLATION.collationName.equals(collationName)) {
+ return UTF8_BINARY_LCASE_COLLATION_ID;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+
+ private static CollationSpecUTF8Binary fromCollationId(int collationId) {
+ int caseConversionOrdinal = SpecifierUtils.getSpecValue(collationId,
+ CASE_SENSITIVITY_OFFSET, CASE_SENSITIVITY_MASK);
+ assert (SpecifierUtils.removeSpec(collationId,
+ CASE_SENSITIVITY_OFFSET, CASE_SENSITIVITY_MASK) == 0);
+ return new CollationSpecUTF8Binary(CaseSensitivity.values()[caseConversionOrdinal]);
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return new Collation("UTF8_BINARY", null, UTF8String::binaryCompare, "1.0",
+ s -> (long) s.hashCode(), true, true, false);
+ } else {
+ return new Collation("UTF8_BINARY_LCASE", null, UTF8String::compareLowerCase, "1.0",
+ s -> (long) s.toLowerCase().hashCode(), false, false, true);
+ }
+ }
+ }
+
+ private static class CollationSpecICU extends CollationSpec {
+
+ private enum CaseSensitivity {
+ CS, CI
+ }
+
+ private enum AccentSensitivity {
+ AS, AI
+ }
+
+ private static final int CASE_SENSITIVITY_OFFSET = 17;
+ private static final int CASE_SENSITIVITY_MASK = 0b1;
+ private static final int ACCENT_SENSITIVITY_OFFSET = 16;
+ private static final int ACCENT_SENSITIVITY_MASK = 0b1;
+
+ // Array of locale names, each locale id corresponds to the index in this array
+ private static final String[] ICULocaleNames;
+
+ // Mapping of locale names to corresponding `ULocale` instance
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+
+ // Used to parse user input collation names which are converted to uppercase
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+
+ // Reverse mapping of `ICULocaleNames`
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+
+ private static final String ICU_COLLATOR_VERSION = "153.120.0.0";
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
+ for (ULocale locale : locales) {
+ if (locale.getVariant().isEmpty()) {
+ String language = locale.getLanguage();
+ assert (!language.isEmpty());
+ StringBuilder builder = new StringBuilder(language);
+ String script = locale.getScript();
+ if (!script.isEmpty()) {
+ builder.append('_');
+ builder.append(script);
+ }
+ String country = locale.getISO3Country();
+ if (!country.isEmpty()) {
+ builder.append('_');
+ builder.append(country);
+ }
+ String localeName = builder.toString();
+ // locale names are unique
+ assert (!ICULocaleMap.containsKey(localeName));
+ ICULocaleMap.put(localeName, locale);
+ }
+ }
+ for (String localeName : ICULocaleMap.keySet()) {
+ String localeUppercase = localeName.toUpperCase();
+ // locale names are unique case-insensitively
+ assert (!ICULocaleMapUppercase.containsKey(localeUppercase));
+ ICULocaleMapUppercase.put(localeUppercase, localeName);
+ }
+ ICULocaleNames = ICULocaleMap.keySet().toArray(new String[0]);
+ Arrays.sort(ICULocaleNames);
+ // maximum number of locale ids as defined by binary layout
+ assert (ICULocaleNames.length <= (1 << 12));
+ for (int i = 0; i < ICULocaleNames.length; ++i) {
+ ICULocaleToId.put(ICULocaleNames[i], i);
+ }
+ }
+
+ private static final int UNICODE_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CS, AccentSensitivity.AS).collationId;
+ private static final int UNICODE_CI_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CI, AccentSensitivity.AS).collationId;
+
+ private final CaseSensitivity caseSensitivity;
+ private final AccentSensitivity accentSensitivity;
+ private final String locale;
+ private final int collationId;
+
+ private CollationSpecICU(String locale, CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ int collationId = ICULocaleToId.get(locale);
+ collationId = SpecifierUtils.setSpecValue(collationId, IMPLEMENTATION_PROVIDER_OFFSET,
+ ImplementationProvider.ICU);
+ collationId = SpecifierUtils.setSpecValue(collationId, CASE_SENSITIVITY_OFFSET,
+ caseSensitivity);
+ collationId = SpecifierUtils.setSpecValue(collationId, ACCENT_SENSITIVITY_OFFSET,
+ accentSensitivity);
+ this.collationId = collationId;
+ }
+
+ private static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ // search for the longest locale match because specifiers are designed to be different from
+ // script tag and country code, meaning the only valid locale name match can be
+ // the longest one
+ int lastPos = -1;
+ for (int i = 1; i <= collationName.length(); i++) {
+ String localeName = collationName.substring(0, i);
+ if (ICULocaleMapUppercase.containsKey(localeName)) {
+ lastPos = i;
+ }
+ }
+ if (lastPos == -1) {
+ throw collationInvalidNameException(originalName);
+ } else {
+ String locale = collationName.substring(0, lastPos);
+ int collationId = ICULocaleToId.get(ICULocaleMapUppercase.get(locale));
+
+ // try all combinations of AS/AI and CS/CI
Review Comment:
```suggestion
// Try all combinations of AS/AI and CS/CI.
```
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -118,76 +119,433 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * Collation id is defined as 32-bit integer.
+ * We specify binary layouts for different classes of collations.
+ * Classes of collations are differentiated by most significant 3 bits (bit 31, 30 and 29),
+ * bit 31 being most significant and bit 0 being least significant.
+ * ---
+ * INDETERMINATE collation id binary layout:
+ * bit 31-0: 1
+ * INDETERMINATE collation id is equal to -1
+ * ---
+ * user-defined collation id binary layout:
+ * bit 31: 0
+ * bit 30: 1
+ * bit 29-0: undefined, reserved for future use
+ * ---
+ * UTF8_BINARY collation id binary layout:
+ * bit 31-22: zeroes
+ * bit 21-18: zeroes, reserved for space trimming
+ * bit 17-16: zeroes, reserved for version
+ * bit 15-3: zeroes
+ * bit 2: 0, reserved for accent sensitivity
+ * bit 1: 0, reserved for uppercase and case-insensitive
+ * bit 0: 0 = case-sensitive, 1 = lowercase
+ * ---
+ * ICU collation id binary layout:
+ * bit 31-30: zeroes
+ * bit 29: 1
+ * bit 28-24: zeroes
+ * bit 23-22: zeroes, reserved for version
+ * bit 21-18: zeroes, reserved for space trimming
+ * bit 17: 0 = case-sensitive, 1 = case-insensitive
+ * bit 16: 0 = accent-sensitive, 1 = accent-insensitive
+ * bit 15-14: zeroes, reserved for punctuation sensitivity
+ * bit 13-12: zeroes, reserved for first letter preference
+ * bit 11-0: locale id as specified in `ICULocaleToId` mapping
+ * ---
+ * Some illustrative examples of collation name to id mapping:
+ * - UTF8_BINARY -> 0
+ * - UTF8_BINARY_LCASE -> 1
+ * - UNICODE -> 0x20000000
+ * - UNICODE_AI -> 0x20010000
+ * - UNICODE_CI -> 0x20020000
+ * - UNICODE_CI_AI -> 0x20030000
+ * - af -> 0x20000001
+ * - af_CI_AI -> 0x20030001
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+
+ private enum DefinitionOrigin {
+ PREDEFINED, USER_DEFINED
+ }
+
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU
+ }
+
+ private static final int DEFINITION_ORIGIN_OFFSET = 30;
+ private static final int DEFINITION_ORIGIN_MASK = 0b1;
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b1;
+
+ private static final int INDETERMINATE_COLLATION_ID = -1;
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ private static ImplementationProvider getImplementationProvider(int collationId) {
+ return ImplementationProvider.values()[SpecifierUtils.getSpecValue(collationId,
+ IMPLEMENTATION_PROVIDER_OFFSET, IMPLEMENTATION_PROVIDER_MASK)];
+ }
+
+ private static DefinitionOrigin getDefinitionOrigin(int collationId) {
+ return DefinitionOrigin.values()[SpecifierUtils.getSpecValue(collationId,
+ DEFINITION_ORIGIN_OFFSET, DEFINITION_ORIGIN_MASK)];
+ }
+
+ private static Collation fetchCollation(int collationId) {
+ assert (collationId >= 0 && getDefinitionOrigin(collationId)
+ == DefinitionOrigin.PREDEFINED);
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ ImplementationProvider implementationProvider = getImplementationProvider(collationId);
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ private static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ private static final int CASE_SENSITIVITY_OFFSET = 0;
+ private static final int CASE_SENSITIVITY_MASK = 0b1;
+
+ private enum CaseSensitivity {
+ UNSPECIFIED, LCASE
+ }
+
+ private static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseSensitivity.UNSPECIFIED).collationId;
+ private static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseSensitivity.LCASE).collationId;
+ protected static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseSensitivity.UNSPECIFIED).buildCollation();
+ protected static Collation UTF8_BINARY_LCASE_COLLATION =
+ new CollationSpecUTF8Binary(CaseSensitivity.LCASE).buildCollation();
+
+ private final int collationId;
+
+ private CollationSpecUTF8Binary(CaseSensitivity caseSensitivity) {
+ this.collationId =
+ SpecifierUtils.setSpecValue(0, CASE_SENSITIVITY_OFFSET, caseSensitivity);
+ }
+
+ private static int collationNameToId(String originalName, String collationName)
+ throws SparkException {
+ if (UTF8_BINARY_COLLATION.collationName.equals(collationName)) {
+ return UTF8_BINARY_COLLATION_ID;
+ } else if (UTF8_BINARY_LCASE_COLLATION.collationName.equals(collationName)) {
+ return UTF8_BINARY_LCASE_COLLATION_ID;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+
+ private static CollationSpecUTF8Binary fromCollationId(int collationId) {
+ int caseConversionOrdinal = SpecifierUtils.getSpecValue(collationId,
+ CASE_SENSITIVITY_OFFSET, CASE_SENSITIVITY_MASK);
+ assert (SpecifierUtils.removeSpec(collationId,
+ CASE_SENSITIVITY_OFFSET, CASE_SENSITIVITY_MASK) == 0);
+ return new CollationSpecUTF8Binary(CaseSensitivity.values()[caseConversionOrdinal]);
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return new Collation("UTF8_BINARY", null, UTF8String::binaryCompare, "1.0",
+ s -> (long) s.hashCode(), true, true, false);
+ } else {
+ return new Collation("UTF8_BINARY_LCASE", null, UTF8String::compareLowerCase, "1.0",
+ s -> (long) s.toLowerCase().hashCode(), false, false, true);
+ }
+ }
+ }
+
+ private static class CollationSpecICU extends CollationSpec {
+
+ private enum CaseSensitivity {
+ CS, CI
+ }
+
+ private enum AccentSensitivity {
+ AS, AI
+ }
+
+ private static final int CASE_SENSITIVITY_OFFSET = 17;
+ private static final int CASE_SENSITIVITY_MASK = 0b1;
+ private static final int ACCENT_SENSITIVITY_OFFSET = 16;
+ private static final int ACCENT_SENSITIVITY_MASK = 0b1;
+
+ // Array of locale names, each locale id corresponds to the index in this array
+ private static final String[] ICULocaleNames;
+
+ // Mapping of locale names to corresponding `ULocale` instance
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+
+ // Used to parse user input collation names which are converted to uppercase
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+
+ // Reverse mapping of `ICULocaleNames`
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+
+ private static final String ICU_COLLATOR_VERSION = "153.120.0.0";
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
+ for (ULocale locale : locales) {
+ if (locale.getVariant().isEmpty()) {
+ String language = locale.getLanguage();
+ assert (!language.isEmpty());
+ StringBuilder builder = new StringBuilder(language);
+ String script = locale.getScript();
+ if (!script.isEmpty()) {
+ builder.append('_');
+ builder.append(script);
+ }
+ String country = locale.getISO3Country();
+ if (!country.isEmpty()) {
+ builder.append('_');
+ builder.append(country);
+ }
+ String localeName = builder.toString();
+ // locale names are unique
+ assert (!ICULocaleMap.containsKey(localeName));
+ ICULocaleMap.put(localeName, locale);
+ }
+ }
+ for (String localeName : ICULocaleMap.keySet()) {
+ String localeUppercase = localeName.toUpperCase();
+ // locale names are unique case-insensitively
+ assert (!ICULocaleMapUppercase.containsKey(localeUppercase));
+ ICULocaleMapUppercase.put(localeUppercase, localeName);
+ }
+ ICULocaleNames = ICULocaleMap.keySet().toArray(new String[0]);
+ Arrays.sort(ICULocaleNames);
+ // maximum number of locale ids as defined by binary layout
+ assert (ICULocaleNames.length <= (1 << 12));
+ for (int i = 0; i < ICULocaleNames.length; ++i) {
+ ICULocaleToId.put(ICULocaleNames[i], i);
+ }
+ }
+
+ private static final int UNICODE_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CS, AccentSensitivity.AS).collationId;
+ private static final int UNICODE_CI_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CI, AccentSensitivity.AS).collationId;
+
+ private final CaseSensitivity caseSensitivity;
+ private final AccentSensitivity accentSensitivity;
+ private final String locale;
+ private final int collationId;
+
+ private CollationSpecICU(String locale, CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ int collationId = ICULocaleToId.get(locale);
+ collationId = SpecifierUtils.setSpecValue(collationId, IMPLEMENTATION_PROVIDER_OFFSET,
+ ImplementationProvider.ICU);
+ collationId = SpecifierUtils.setSpecValue(collationId, CASE_SENSITIVITY_OFFSET,
+ caseSensitivity);
+ collationId = SpecifierUtils.setSpecValue(collationId, ACCENT_SENSITIVITY_OFFSET,
+ accentSensitivity);
+ this.collationId = collationId;
+ }
+
+ private static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ // search for the longest locale match because specifiers are designed to be different from
+ // script tag and country code, meaning the only valid locale name match can be
+ // the longest one
+ int lastPos = -1;
+ for (int i = 1; i <= collationName.length(); i++) {
+ String localeName = collationName.substring(0, i);
+ if (ICULocaleMapUppercase.containsKey(localeName)) {
+ lastPos = i;
+ }
+ }
+ if (lastPos == -1) {
+ throw collationInvalidNameException(originalName);
+ } else {
+ String locale = collationName.substring(0, lastPos);
+ int collationId = ICULocaleToId.get(ICULocaleMapUppercase.get(locale));
+
+ // try all combinations of AS/AI and CS/CI
+ CaseSensitivity caseSensitivity;
+ AccentSensitivity accentSensitivity;
+ if (collationName.equals(locale) ||
+ collationName.equals(locale + "_AS") ||
Review Comment:
Is `collationName` expected to be uppercase?
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -118,76 +119,433 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * Collation id is defined as 32-bit integer.
+ * We specify binary layouts for different classes of collations.
+ * Classes of collations are differentiated by most significant 3 bits (bit 31, 30 and 29),
+ * bit 31 being most significant and bit 0 being least significant.
+ * ---
+ * INDETERMINATE collation id binary layout:
+ * bit 31-0: 1
+ * INDETERMINATE collation id is equal to -1
+ * ---
+ * user-defined collation id binary layout:
+ * bit 31: 0
+ * bit 30: 1
+ * bit 29-0: undefined, reserved for future use
+ * ---
+ * UTF8_BINARY collation id binary layout:
+ * bit 31-22: zeroes
+ * bit 21-18: zeroes, reserved for space trimming
+ * bit 17-16: zeroes, reserved for version
+ * bit 15-3: zeroes
+ * bit 2: 0, reserved for accent sensitivity
+ * bit 1: 0, reserved for uppercase and case-insensitive
+ * bit 0: 0 = case-sensitive, 1 = lowercase
+ * ---
+ * ICU collation id binary layout:
+ * bit 31-30: zeroes
+ * bit 29: 1
+ * bit 28-24: zeroes
+ * bit 23-22: zeroes, reserved for version
+ * bit 21-18: zeroes, reserved for space trimming
+ * bit 17: 0 = case-sensitive, 1 = case-insensitive
+ * bit 16: 0 = accent-sensitive, 1 = accent-insensitive
+ * bit 15-14: zeroes, reserved for punctuation sensitivity
+ * bit 13-12: zeroes, reserved for first letter preference
+ * bit 11-0: locale id as specified in `ICULocaleToId` mapping
+ * ---
+ * Some illustrative examples of collation name to id mapping:
+ * - UTF8_BINARY -> 0
+ * - UTF8_BINARY_LCASE -> 1
+ * - UNICODE -> 0x20000000
+ * - UNICODE_AI -> 0x20010000
+ * - UNICODE_CI -> 0x20020000
+ * - UNICODE_CI_AI -> 0x20030000
+ * - af -> 0x20000001
+ * - af_CI_AI -> 0x20030001
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+
+ private enum DefinitionOrigin {
+ PREDEFINED, USER_DEFINED
+ }
+
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU
+ }
+
+ private static final int DEFINITION_ORIGIN_OFFSET = 30;
+ private static final int DEFINITION_ORIGIN_MASK = 0b1;
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b1;
+
+ private static final int INDETERMINATE_COLLATION_ID = -1;
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ private static ImplementationProvider getImplementationProvider(int collationId) {
+ return ImplementationProvider.values()[SpecifierUtils.getSpecValue(collationId,
+ IMPLEMENTATION_PROVIDER_OFFSET, IMPLEMENTATION_PROVIDER_MASK)];
+ }
+
+ private static DefinitionOrigin getDefinitionOrigin(int collationId) {
+ return DefinitionOrigin.values()[SpecifierUtils.getSpecValue(collationId,
+ DEFINITION_ORIGIN_OFFSET, DEFINITION_ORIGIN_MASK)];
+ }
+
+ private static Collation fetchCollation(int collationId) {
+ assert (collationId >= 0 && getDefinitionOrigin(collationId)
+ == DefinitionOrigin.PREDEFINED);
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ ImplementationProvider implementationProvider = getImplementationProvider(collationId);
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ private static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ private static final int CASE_SENSITIVITY_OFFSET = 0;
+ private static final int CASE_SENSITIVITY_MASK = 0b1;
+
+ private enum CaseSensitivity {
+ UNSPECIFIED, LCASE
+ }
+
+ private static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseSensitivity.UNSPECIFIED).collationId;
+ private static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseSensitivity.LCASE).collationId;
+ protected static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseSensitivity.UNSPECIFIED).buildCollation();
+ protected static Collation UTF8_BINARY_LCASE_COLLATION =
+ new CollationSpecUTF8Binary(CaseSensitivity.LCASE).buildCollation();
+
+ private final int collationId;
+
+ private CollationSpecUTF8Binary(CaseSensitivity caseSensitivity) {
+ this.collationId =
+ SpecifierUtils.setSpecValue(0, CASE_SENSITIVITY_OFFSET, caseSensitivity);
+ }
+
+ private static int collationNameToId(String originalName, String collationName)
+ throws SparkException {
+ if (UTF8_BINARY_COLLATION.collationName.equals(collationName)) {
+ return UTF8_BINARY_COLLATION_ID;
+ } else if (UTF8_BINARY_LCASE_COLLATION.collationName.equals(collationName)) {
+ return UTF8_BINARY_LCASE_COLLATION_ID;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+
+ private static CollationSpecUTF8Binary fromCollationId(int collationId) {
+ int caseConversionOrdinal = SpecifierUtils.getSpecValue(collationId,
+ CASE_SENSITIVITY_OFFSET, CASE_SENSITIVITY_MASK);
+ assert (SpecifierUtils.removeSpec(collationId,
+ CASE_SENSITIVITY_OFFSET, CASE_SENSITIVITY_MASK) == 0);
+ return new CollationSpecUTF8Binary(CaseSensitivity.values()[caseConversionOrdinal]);
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return new Collation("UTF8_BINARY", null, UTF8String::binaryCompare, "1.0",
+ s -> (long) s.hashCode(), true, true, false);
+ } else {
+ return new Collation("UTF8_BINARY_LCASE", null, UTF8String::compareLowerCase, "1.0",
+ s -> (long) s.toLowerCase().hashCode(), false, false, true);
+ }
+ }
+ }
+
+ private static class CollationSpecICU extends CollationSpec {
+
+ private enum CaseSensitivity {
+ CS, CI
+ }
+
+ private enum AccentSensitivity {
+ AS, AI
+ }
+
+ private static final int CASE_SENSITIVITY_OFFSET = 17;
+ private static final int CASE_SENSITIVITY_MASK = 0b1;
+ private static final int ACCENT_SENSITIVITY_OFFSET = 16;
+ private static final int ACCENT_SENSITIVITY_MASK = 0b1;
+
+ // Array of locale names, each locale id corresponds to the index in this array
+ private static final String[] ICULocaleNames;
+
+ // Mapping of locale names to corresponding `ULocale` instance
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+
+ // Used to parse user input collation names which are converted to uppercase
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+
+ // Reverse mapping of `ICULocaleNames`
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+
+ private static final String ICU_COLLATOR_VERSION = "153.120.0.0";
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
+ for (ULocale locale : locales) {
+ if (locale.getVariant().isEmpty()) {
+ String language = locale.getLanguage();
+ assert (!language.isEmpty());
+ StringBuilder builder = new StringBuilder(language);
+ String script = locale.getScript();
+ if (!script.isEmpty()) {
+ builder.append('_');
+ builder.append(script);
+ }
+ String country = locale.getISO3Country();
+ if (!country.isEmpty()) {
+ builder.append('_');
+ builder.append(country);
+ }
+ String localeName = builder.toString();
+ // locale names are unique
+ assert (!ICULocaleMap.containsKey(localeName));
+ ICULocaleMap.put(localeName, locale);
+ }
+ }
+ for (String localeName : ICULocaleMap.keySet()) {
+ String localeUppercase = localeName.toUpperCase();
+ // locale names are unique case-insensitively
+ assert (!ICULocaleMapUppercase.containsKey(localeUppercase));
+ ICULocaleMapUppercase.put(localeUppercase, localeName);
+ }
+ ICULocaleNames = ICULocaleMap.keySet().toArray(new String[0]);
+ Arrays.sort(ICULocaleNames);
+ // maximum number of locale ids as defined by binary layout
+ assert (ICULocaleNames.length <= (1 << 12));
+ for (int i = 0; i < ICULocaleNames.length; ++i) {
+ ICULocaleToId.put(ICULocaleNames[i], i);
+ }
+ }
+
+ private static final int UNICODE_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CS, AccentSensitivity.AS).collationId;
+ private static final int UNICODE_CI_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CI, AccentSensitivity.AS).collationId;
+
+ private final CaseSensitivity caseSensitivity;
+ private final AccentSensitivity accentSensitivity;
+ private final String locale;
+ private final int collationId;
+
+ private CollationSpecICU(String locale, CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ int collationId = ICULocaleToId.get(locale);
+ collationId = SpecifierUtils.setSpecValue(collationId, IMPLEMENTATION_PROVIDER_OFFSET,
+ ImplementationProvider.ICU);
+ collationId = SpecifierUtils.setSpecValue(collationId, CASE_SENSITIVITY_OFFSET,
+ caseSensitivity);
+ collationId = SpecifierUtils.setSpecValue(collationId, ACCENT_SENSITIVITY_OFFSET,
+ accentSensitivity);
+ this.collationId = collationId;
+ }
+
+ private static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ // search for the longest locale match because specifiers are designed to be different from
+ // script tag and country code, meaning the only valid locale name match can be
+ // the longest one
Review Comment:
```suggestion
// Search for the longest locale match because specifiers are designed to be different from
// script tag and country code, meaning the only valid locale name match can be
// the longest one.
```
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -118,76 +119,433 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * Collation id is defined as 32-bit integer.
+ * We specify binary layouts for different classes of collations.
+ * Classes of collations are differentiated by most significant 3 bits (bit 31, 30 and 29),
+ * bit 31 being most significant and bit 0 being least significant.
+ * ---
+ * INDETERMINATE collation id binary layout:
+ * bit 31-0: 1
+ * INDETERMINATE collation id is equal to -1
+ * ---
+ * user-defined collation id binary layout:
+ * bit 31: 0
+ * bit 30: 1
+ * bit 29-0: undefined, reserved for future use
+ * ---
+ * UTF8_BINARY collation id binary layout:
+ * bit 31-22: zeroes
+ * bit 21-18: zeroes, reserved for space trimming
+ * bit 17-16: zeroes, reserved for version
+ * bit 15-3: zeroes
+ * bit 2: 0, reserved for accent sensitivity
+ * bit 1: 0, reserved for uppercase and case-insensitive
+ * bit 0: 0 = case-sensitive, 1 = lowercase
+ * ---
+ * ICU collation id binary layout:
+ * bit 31-30: zeroes
+ * bit 29: 1
+ * bit 28-24: zeroes
+ * bit 23-22: zeroes, reserved for version
Review Comment:
I think it would be helpful (as part of a follow up PR) to provide a bit-centric view (right now we have a collation centric view.
Bit centric view:
```
31: 1 for INDETERMINATE (requires all other bits to be 1 as well), 0 for all other collations.
30: 0 for predefined, 1 for user-defined
29-24: Reserved
23-22: Reserved for version (here we have a discrepancy between UTF8_BINARY and ICU family).
21-18: Reserved for space trimming
17-0: Depend on collation family.
```
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -118,76 +119,433 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * Collation id is defined as 32-bit integer.
+ * We specify binary layouts for different classes of collations.
+ * Classes of collations are differentiated by most significant 3 bits (bit 31, 30 and 29),
+ * bit 31 being most significant and bit 0 being least significant.
+ * ---
+ * INDETERMINATE collation id binary layout:
+ * bit 31-0: 1
+ * INDETERMINATE collation id is equal to -1
+ * ---
+ * user-defined collation id binary layout:
+ * bit 31: 0
+ * bit 30: 1
+ * bit 29-0: undefined, reserved for future use
+ * ---
+ * UTF8_BINARY collation id binary layout:
+ * bit 31-22: zeroes
+ * bit 21-18: zeroes, reserved for space trimming
+ * bit 17-16: zeroes, reserved for version
+ * bit 15-3: zeroes
+ * bit 2: 0, reserved for accent sensitivity
+ * bit 1: 0, reserved for uppercase and case-insensitive
+ * bit 0: 0 = case-sensitive, 1 = lowercase
+ * ---
+ * ICU collation id binary layout:
+ * bit 31-30: zeroes
+ * bit 29: 1
+ * bit 28-24: zeroes
+ * bit 23-22: zeroes, reserved for version
+ * bit 21-18: zeroes, reserved for space trimming
+ * bit 17: 0 = case-sensitive, 1 = case-insensitive
+ * bit 16: 0 = accent-sensitive, 1 = accent-insensitive
+ * bit 15-14: zeroes, reserved for punctuation sensitivity
+ * bit 13-12: zeroes, reserved for first letter preference
+ * bit 11-0: locale id as specified in `ICULocaleToId` mapping
+ * ---
+ * Some illustrative examples of collation name to id mapping:
+ * - UTF8_BINARY -> 0
+ * - UTF8_BINARY_LCASE -> 1
+ * - UNICODE -> 0x20000000
+ * - UNICODE_AI -> 0x20010000
+ * - UNICODE_CI -> 0x20020000
+ * - UNICODE_CI_AI -> 0x20030000
+ * - af -> 0x20000001
+ * - af_CI_AI -> 0x20030001
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+
+ private enum DefinitionOrigin {
+ PREDEFINED, USER_DEFINED
+ }
+
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU
+ }
+
+ private static final int DEFINITION_ORIGIN_OFFSET = 30;
+ private static final int DEFINITION_ORIGIN_MASK = 0b1;
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b1;
+
+ private static final int INDETERMINATE_COLLATION_ID = -1;
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ private static ImplementationProvider getImplementationProvider(int collationId) {
+ return ImplementationProvider.values()[SpecifierUtils.getSpecValue(collationId,
+ IMPLEMENTATION_PROVIDER_OFFSET, IMPLEMENTATION_PROVIDER_MASK)];
+ }
+
+ private static DefinitionOrigin getDefinitionOrigin(int collationId) {
+ return DefinitionOrigin.values()[SpecifierUtils.getSpecValue(collationId,
+ DEFINITION_ORIGIN_OFFSET, DEFINITION_ORIGIN_MASK)];
+ }
+
+ private static Collation fetchCollation(int collationId) {
+ assert (collationId >= 0 && getDefinitionOrigin(collationId)
+ == DefinitionOrigin.PREDEFINED);
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ ImplementationProvider implementationProvider = getImplementationProvider(collationId);
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ private static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ private static final int CASE_SENSITIVITY_OFFSET = 0;
+ private static final int CASE_SENSITIVITY_MASK = 0b1;
+
+ private enum CaseSensitivity {
+ UNSPECIFIED, LCASE
+ }
+
+ private static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseSensitivity.UNSPECIFIED).collationId;
+ private static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseSensitivity.LCASE).collationId;
+ protected static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseSensitivity.UNSPECIFIED).buildCollation();
+ protected static Collation UTF8_BINARY_LCASE_COLLATION =
+ new CollationSpecUTF8Binary(CaseSensitivity.LCASE).buildCollation();
+
+ private final int collationId;
+
+ private CollationSpecUTF8Binary(CaseSensitivity caseSensitivity) {
+ this.collationId =
+ SpecifierUtils.setSpecValue(0, CASE_SENSITIVITY_OFFSET, caseSensitivity);
+ }
+
+ private static int collationNameToId(String originalName, String collationName)
+ throws SparkException {
+ if (UTF8_BINARY_COLLATION.collationName.equals(collationName)) {
+ return UTF8_BINARY_COLLATION_ID;
+ } else if (UTF8_BINARY_LCASE_COLLATION.collationName.equals(collationName)) {
+ return UTF8_BINARY_LCASE_COLLATION_ID;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+
+ private static CollationSpecUTF8Binary fromCollationId(int collationId) {
+ int caseConversionOrdinal = SpecifierUtils.getSpecValue(collationId,
+ CASE_SENSITIVITY_OFFSET, CASE_SENSITIVITY_MASK);
+ assert (SpecifierUtils.removeSpec(collationId,
+ CASE_SENSITIVITY_OFFSET, CASE_SENSITIVITY_MASK) == 0);
+ return new CollationSpecUTF8Binary(CaseSensitivity.values()[caseConversionOrdinal]);
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return new Collation("UTF8_BINARY", null, UTF8String::binaryCompare, "1.0",
+ s -> (long) s.hashCode(), true, true, false);
+ } else {
+ return new Collation("UTF8_BINARY_LCASE", null, UTF8String::compareLowerCase, "1.0",
+ s -> (long) s.toLowerCase().hashCode(), false, false, true);
+ }
+ }
+ }
+
+ private static class CollationSpecICU extends CollationSpec {
+
+ private enum CaseSensitivity {
+ CS, CI
+ }
+
+ private enum AccentSensitivity {
+ AS, AI
+ }
+
+ private static final int CASE_SENSITIVITY_OFFSET = 17;
+ private static final int CASE_SENSITIVITY_MASK = 0b1;
+ private static final int ACCENT_SENSITIVITY_OFFSET = 16;
+ private static final int ACCENT_SENSITIVITY_MASK = 0b1;
+
+ // Array of locale names, each locale id corresponds to the index in this array
+ private static final String[] ICULocaleNames;
+
+ // Mapping of locale names to corresponding `ULocale` instance
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+
+ // Used to parse user input collation names which are converted to uppercase
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+
+ // Reverse mapping of `ICULocaleNames`
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+
+ private static final String ICU_COLLATOR_VERSION = "153.120.0.0";
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
+ for (ULocale locale : locales) {
+ if (locale.getVariant().isEmpty()) {
+ String language = locale.getLanguage();
+ assert (!language.isEmpty());
+ StringBuilder builder = new StringBuilder(language);
+ String script = locale.getScript();
+ if (!script.isEmpty()) {
+ builder.append('_');
+ builder.append(script);
+ }
+ String country = locale.getISO3Country();
+ if (!country.isEmpty()) {
+ builder.append('_');
+ builder.append(country);
+ }
+ String localeName = builder.toString();
+ // locale names are unique
+ assert (!ICULocaleMap.containsKey(localeName));
+ ICULocaleMap.put(localeName, locale);
+ }
+ }
+ for (String localeName : ICULocaleMap.keySet()) {
+ String localeUppercase = localeName.toUpperCase();
+ // locale names are unique case-insensitively
+ assert (!ICULocaleMapUppercase.containsKey(localeUppercase));
+ ICULocaleMapUppercase.put(localeUppercase, localeName);
+ }
+ ICULocaleNames = ICULocaleMap.keySet().toArray(new String[0]);
+ Arrays.sort(ICULocaleNames);
+ // maximum number of locale ids as defined by binary layout
+ assert (ICULocaleNames.length <= (1 << 12));
+ for (int i = 0; i < ICULocaleNames.length; ++i) {
+ ICULocaleToId.put(ICULocaleNames[i], i);
+ }
+ }
+
+ private static final int UNICODE_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CS, AccentSensitivity.AS).collationId;
+ private static final int UNICODE_CI_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CI, AccentSensitivity.AS).collationId;
+
+ private final CaseSensitivity caseSensitivity;
+ private final AccentSensitivity accentSensitivity;
+ private final String locale;
+ private final int collationId;
+
+ private CollationSpecICU(String locale, CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ int collationId = ICULocaleToId.get(locale);
+ collationId = SpecifierUtils.setSpecValue(collationId, IMPLEMENTATION_PROVIDER_OFFSET,
+ ImplementationProvider.ICU);
+ collationId = SpecifierUtils.setSpecValue(collationId, CASE_SENSITIVITY_OFFSET,
+ caseSensitivity);
+ collationId = SpecifierUtils.setSpecValue(collationId, ACCENT_SENSITIVITY_OFFSET,
+ accentSensitivity);
+ this.collationId = collationId;
+ }
+
+ private static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ // search for the longest locale match because specifiers are designed to be different from
+ // script tag and country code, meaning the only valid locale name match can be
+ // the longest one
+ int lastPos = -1;
+ for (int i = 1; i <= collationName.length(); i++) {
+ String localeName = collationName.substring(0, i);
+ if (ICULocaleMapUppercase.containsKey(localeName)) {
+ lastPos = i;
+ }
+ }
+ if (lastPos == -1) {
+ throw collationInvalidNameException(originalName);
+ } else {
+ String locale = collationName.substring(0, lastPos);
+ int collationId = ICULocaleToId.get(ICULocaleMapUppercase.get(locale));
+
+ // try all combinations of AS/AI and CS/CI
Review Comment:
Please use full/regular sentences for comments. I do not plan to comment on this in the rest of the PR.
##########
sql/core/src/test/scala/org/apache/spark/sql/ICUCollationsMap.scala:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.util.{fileToString, stringToFile, CollationFactory}
+
+// scalastyle:off line.size.limit
+/**
+ * Guard against breaking changes in ICU locale names and codes supported by Collator class and provider by CollationFactory.
+ * Map is in form of rows of pairs (locale name, locale id); locale name consists of three parts:
+ * - 2-letter lowercase language code
+ * - 4-letter script code (optional)
+ * - 3-letter uppercase country code
+ *
+ * To re-generate collations map golden file, run:
+ * {{{
+ * SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly org.apache.spark.sql.ICUCollationsMap"
+ * }}}
+ */
+// scalastyle:on line.size.limit
+class ICUCollationsMap extends SparkFunSuite {
Review Comment:
Just curious: since this is a test, shouldn't the name be like `ICUCollationsMapSuite`?
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +118,490 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30: 0 = utf8-binary, 1 = ICU
+ * bit 29: 0 = case-sensitive, 1 = case-insensitive
+ * bit 28: 0 = accent-sensitive, 1 = accent-insensitive
+ * bit 27-26: 00 = unspecified, 01 = punctuation-sensitive, 10 = punctuation-insensitive
+ * bit 25-24: 00 = unspecified, 01 = first-lower, 10 = first-upper
+ * bit 23-22: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 21-20: 00 = unspecified, 01 = trim-left, 10 = trim-right, 11 = trim-both
+ * bit 19-18: zeroes, reserved for version
+ * bit 17-16: zeroes
+ * bit 15-0: locale id for ICU collations / zeroes for utf8-binary
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
- }
- }
+ private static class CollationSpec {
+ private enum ImplementationProvider {
+ UTF8_BINARY, ICU
+ }
+
+ private enum CaseSensitivity {
+ CS, CI
+ }
+
+ private enum AccentSensitivity {
+ AS, AI
+ }
+
+ private enum PunctuationSensitivity {
+ UNSPECIFIED, PS, PI
+ }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private enum FirstLetterPreference {
+ UNSPECIFIED, FU, FL
+ }
+
+ private enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ private enum SpaceTrimming {
+ UNSPECIFIED, LTRIM, RTRIM, TRIM
+ }
+
+ private static final int implementationProviderOffset = 30;
+ private static final int implementationProviderLen = 1;
+ private static final int caseSensitivityOffset = 29;
+ private static final int caseSensitivityLen = 1;
+ private static final int accentSensitivityOffset = 28;
+ private static final int accentSensitivityLen = 1;
+ private static final int punctuationSensitivityOffset = 26;
+ private static final int punctuationSensitivityLen = 2;
+ private static final int firstLetterPreferenceOffset = 24;
+ private static final int firstLetterPreferenceLen = 2;
+ private static final int caseConversionOffset = 22;
+ private static final int caseConversionLen = 2;
+ private static final int spaceTrimmingOffset = 20;
+ private static final int spaceTrimmingLen = 2;
+ private static final int localeOffset = 0;
+ private static final int localeLen = 16;
+
+ private static final String[] ICULocaleNames;
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
Review Comment:
I like the approach taken here. Thank you!
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
Review Comment:
Which gets us to an interesting problem: we use `toUpperCase` which depends on the JVM version, so technically it is not stable across time. Is it fair to say that for predefined collations everything is ASCII?
For user-defined collations we may want to think if we want to be as generic as what we are now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "mkaravel (via GitHub)" <gi...@apache.org>.
mkaravel commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1594837464
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
Review Comment:
What does this mean? Do we anticipate up to 4 versions max?
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
Review Comment:
What does `to-lower` and `to-upper` refer to? I know we have something in the PR description, but I would argue that it is even more important to have it here.
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
Review Comment:
This is a bit confusing to me. Maybe write it in a way that is completely unambiguous?
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
Review Comment:
```suggestion
* Collation id (32-bit integer) layout:
```
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
Review Comment:
As part of the documentation, I would add a few examples, notably what is the encoding for:
* UTF8_BINARY
* UTF8_BINARY_LCASE and/or UTF8_BINARY_UCASE
* UNICODE
* UNICODE_CI
* Some other more involved collation.
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
Review Comment:
Do we allow any kind of characters for the collation name?
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected static int parseSpecifiers(
+ String originalName, String collationName, int splitStart) throws SparkException {
+ int specifiers = 0;
+ String[] parts = collationName.substring(splitStart).split("_");
+ for (String part : parts) {
+ if (!part.isEmpty()) {
+ if (part.equals("UNSPECIFIED") || part.equals("INDETERMINATE")) {
+ throw collationInvalidNameException(originalName);
+ } else if (Arrays.stream(CaseSensitivity.values()).anyMatch(
+ (s) -> s.toString().equals(part))) {
+ specifiers |=
+ CaseSensitivity.valueOf(part).ordinal() << CASE_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(AccentSensitivity.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ AccentSensitivity.valueOf(part).ordinal() << ACCENT_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(CaseConversion.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ CaseConversion.valueOf(part).ordinal() << CASE_CONVERSION_OFFSET;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+ }
+ return specifiers;
+ }
+
+ protected abstract int getCollationId();
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ public static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.LCASE).getCollationId();
+ public static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).buildCollation();
+
+ private CollationSpecUTF8Binary(CaseConversion caseConversion) {
+ super(null, CaseSensitivity.CS, AccentSensitivity.AS, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int collationId = 0;
+ int specifiers = CollationSpec.parseSpecifiers(
+ originalName, collationName, "UTF8_BINARY".length());
+ if ((specifiers & ~(CASE_CONVERSION_MASK << CASE_CONVERSION_OFFSET)) != 0) {
+ throw collationInvalidNameException(originalName);
+ }
+ collationId |= specifiers;
+ return collationId;
+ }
+
+ @Override
+ protected int getCollationId() {
+ int collationId = 0;
+ collationId |= caseConversion.ordinal() << CASE_CONVERSION_OFFSET;
+ return collationId;
+ }
+
+ public static CollationSpecUTF8Binary fromCollationId(int collationId)
+ throws SparkException {
+ int originalCollationId = collationId;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
+ if (collationId != 0 || caseConversionOrdinal >= CaseConversion.values().length) {
Review Comment:
```suggestion
if (collationId != UTF8_BINARY_COLLATION_ID || caseConversionOrdinal >= CaseConversion.values().length) {
```
##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/common/DataTypeProtoConverter.scala:
##########
@@ -80,7 +81,7 @@ object DataTypeProtoConverter {
}
private def toCatalystStringType(t: proto.DataType.String): StringType =
- StringType(t.getCollationId)
+ StringType(if (t.getCollation.nonEmpty) t.getCollation else "UTF8_BINARY")
Review Comment:
Why would the collation ever be empty?
##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -30,31 +33,107 @@ import org.apache.spark.unsafe.types.UTF8String.{fromString => toUTF8}
class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ignore funsuite
test("collationId stability") {
- val utf8Binary = fetchCollation(0)
+ assert(UTF8_BINARY_COLLATION_ID == 0)
+ assert(INDETERMINATE_COLLATION_ID == 1 << 30)
Review Comment:
```suggestion
assert(INDETERMINATE_COLLATION_ID == (1 << 30))
```
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected static int parseSpecifiers(
+ String originalName, String collationName, int splitStart) throws SparkException {
+ int specifiers = 0;
+ String[] parts = collationName.substring(splitStart).split("_");
+ for (String part : parts) {
+ if (!part.isEmpty()) {
+ if (part.equals("UNSPECIFIED") || part.equals("INDETERMINATE")) {
+ throw collationInvalidNameException(originalName);
+ } else if (Arrays.stream(CaseSensitivity.values()).anyMatch(
+ (s) -> s.toString().equals(part))) {
+ specifiers |=
+ CaseSensitivity.valueOf(part).ordinal() << CASE_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(AccentSensitivity.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ AccentSensitivity.valueOf(part).ordinal() << ACCENT_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(CaseConversion.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ CaseConversion.valueOf(part).ordinal() << CASE_CONVERSION_OFFSET;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+ }
+ return specifiers;
+ }
+
+ protected abstract int getCollationId();
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ public static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.LCASE).getCollationId();
+ public static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).buildCollation();
+
+ private CollationSpecUTF8Binary(CaseConversion caseConversion) {
+ super(null, CaseSensitivity.CS, AccentSensitivity.AS, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int collationId = 0;
+ int specifiers = CollationSpec.parseSpecifiers(
+ originalName, collationName, "UTF8_BINARY".length());
+ if ((specifiers & ~(CASE_CONVERSION_MASK << CASE_CONVERSION_OFFSET)) != 0) {
+ throw collationInvalidNameException(originalName);
+ }
+ collationId |= specifiers;
+ return collationId;
+ }
+
+ @Override
+ protected int getCollationId() {
+ int collationId = 0;
+ collationId |= caseConversion.ordinal() << CASE_CONVERSION_OFFSET;
+ return collationId;
+ }
+
+ public static CollationSpecUTF8Binary fromCollationId(int collationId)
+ throws SparkException {
+ int originalCollationId = collationId;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
+ if (collationId != 0 || caseConversionOrdinal >= CaseConversion.values().length) {
+ throw SparkException.internalError("Invalid UTF8_BINARY collation id " +
+ originalCollationId);
+ } else {
+ CaseConversion caseConversion = CaseConversion.values()[caseConversionOrdinal];
+ return new CollationSpecUTF8Binary(caseConversion);
+ }
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ Comparator<UTF8String> comparator;
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ comparator = UTF8String::binaryCompare;
+ } else if (collationId == UTF8_BINARY_LCASE_COLLATION_ID) {
+ comparator = UTF8String::compareLowerCase;
+ } else {
+ comparator = (s1, s2) -> {
+ UTF8String convertedS1 = caseConversion(s1);
+ UTF8String convertedS2 = caseConversion(s2);
+ return convertedS1.binaryCompare(convertedS2);
+ };
+ }
+ return new Collation(
+ collationName(),
+ null,
+ comparator,
+ "1.0",
+ s -> (long) caseConversion(s).hashCode(),
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_LCASE_COLLATION_ID
+ );
+ }
+
+ private UTF8String caseConversion(UTF8String s) {
+ if (caseConversion == CaseConversion.LCASE) {
+ return s.toLowerCase();
+ } else if (caseConversion == CaseConversion.UCASE) {
+ return s.toUpperCase();
+ } else {
+ return s;
+ }
+ }
+
+ private String collationName() {
+ StringBuilder builder = new StringBuilder();
+ builder.append("UTF8_BINARY");
+ if (caseConversion != CaseConversion.UNSPECIFIED) {
+ builder.append('_');
+ builder.append(caseConversion.toString());
+ }
+ return builder.toString();
+ }
+ }
+
+ private static class CollationSpecICU extends CollationSpec {
+
+ private static final String[] ICULocaleNames;
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+ private static final String ICUCollatorVersion = "153.120.0.0";
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
+ for (ULocale locale : locales) {
+ if (locale.getVariant().isEmpty()) {
+ String language = locale.getLanguage();
+ assert (!language.isEmpty());
+ StringBuilder builder = new StringBuilder(language);
+ String script = locale.getScript();
+ if (!script.isEmpty()) {
+ builder.append('_');
+ builder.append(script);
+ }
+ String country = locale.getISO3Country();
+ if (!country.isEmpty()) {
+ builder.append('_');
+ builder.append(country);
+ }
+ String localeName = builder.toString();
+ assert (!ICULocaleMap.containsKey(localeName));
+ ICULocaleMap.put(localeName, locale);
+ }
+ }
+ for (String localeName : ICULocaleMap.keySet()) {
+ String localeUppercase = localeName.toUpperCase();
+ assert (!ICULocaleMapUppercase.containsKey(localeUppercase));
+ ICULocaleMapUppercase.put(localeUppercase, localeName);
+ }
+ ICULocaleNames = ICULocaleMap.keySet().toArray(new String[0]);
+ Arrays.sort(ICULocaleNames);
+ assert (ICULocaleNames.length <= (1 << 16));
+ for (int i = 0; i < ICULocaleNames.length; i++) {
+ ICULocaleToId.put(ICULocaleNames[i], i);
+ }
+ }
+
+ public static final int UNICODE_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CS, AccentSensitivity.AS,
+ CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UNICODE_CI_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CI, AccentSensitivity.AS,
+ CaseConversion.UNSPECIFIED).getCollationId();
+
+ private CollationSpecICU(String locale, CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity, CaseConversion caseConversion) {
+ super(locale, caseSensitivity, accentSensitivity, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int lastPos = -1;
+ for (int i = 1; i <= collationName.length(); i++) {
+ String localeName = collationName.substring(0, i);
+ if (ICULocaleMapUppercase.containsKey(localeName)) {
+ lastPos = i;
+ }
+ }
+ if (lastPos == -1) {
+ throw collationInvalidNameException(originalName);
+ } else {
+ int collationId = 0;
+ collationId |= ImplementationProvider.ICU.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+ collationId |= CollationSpec.parseSpecifiers(originalName, collationName, lastPos);
+ String normalizedLocaleName = ICULocaleMapUppercase.get(
+ collationName.substring(0, lastPos));
+ collationId |= ICULocaleToId.get(normalizedLocaleName) << LOCALE_OFFSET;
+ return collationId;
+ }
+ }
+
+ public static CollationSpecICU fromCollationId(int collationId) throws SparkException {
+ int originalCollationId = collationId;
+ int caseSensitivityOrdinal =
+ (collationId >> CASE_SENSITIVITY_OFFSET) & CASE_SENSITIVITY_MASK;
+ int accentSensitivityOrdinal =
+ (collationId >> ACCENT_SENSITIVITY_OFFSET) & ACCENT_SENSITIVITY_MASK;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ int localeOrdinal = (collationId >> LOCALE_OFFSET) & LOCALE_MASK;
+ collationId ^= ImplementationProvider.ICU.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+ collationId ^= caseSensitivityOrdinal << CASE_SENSITIVITY_OFFSET;
+ collationId ^= accentSensitivityOrdinal << ACCENT_SENSITIVITY_OFFSET;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
+ collationId ^= localeOrdinal << LOCALE_OFFSET;
+ if (collationId != 0 || caseConversionOrdinal >= CaseConversion.values().length ||
Review Comment:
```suggestion
if (collationId != UTF8_BINARY_COLLATION_ID || caseConversionOrdinal >= CaseConversion.values().length ||
```
##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -152,4 +231,168 @@ class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ig
}
})
}
+
+ test("test collation caching") {
+ Seq(
+ "UTF8_BINARY",
+ "UTF8_BINARY_LCASE",
+ "UTF8_BINARY_UCASE",
+ "UNICODE",
+ "UNICODE_LCASE",
+ "UNICODE_UCASE",
+ "UNICODE_CI",
+ "UNICODE_AI_CI",
+ "UNICODE_AI_CI_LCASE",
+ "UNICODE_AI_CI_UCASE"
+ ).foreach(collationId => {
+ val col1 = fetchCollation(collationId)
+ val col2 = fetchCollation(collationId)
+ assert(col1 eq col2) // reference equality
+ })
+ }
+
+ test("collations with ICU non-root localization") {
+ Seq(
+ // language only
+ "en",
+ "en_CS",
+ "en_CI",
+ "en_AS",
+ "en_AI",
+ "en_LCASE",
+ "en_UCASE",
+ // language + 3-letter country code
+ "en_USA",
+ "en_USA_CS",
+ "en_USA_CI",
+ "en_USA_AS",
+ "en_USA_AI",
+ "en_USA_LCASE",
+ "en_USA_UCASE",
+ // language + script code
+ "sr_Cyrl",
+ "sr_Cyrl_CS",
+ "sr_Cyrl_CI",
+ "sr_Cyrl_AS",
+ "sr_Cyrl_AI",
+ "sr_Cyrl_LCASE",
+ "sr_Cyrl_UCASE",
+ // language + script code + 3-letter country code
+ "sr_Cyrl_SRB",
+ "sr_Cyrl_SRB_CS",
+ "sr_Cyrl_SRB_CI",
+ "sr_Cyrl_SRB_AS",
+ "sr_Cyrl_SRB_AI",
+ "sr_Cyrl_SRB_LCASE",
+ "sr_Cyrl_SRB_UCASE"
+ ).foreach(collationICU => {
+ val col = fetchCollation(collationICU)
+ assert(col.collator.getLocale(ULocale.VALID_LOCALE) != ULocale.ROOT)
+ })
+ }
+
+ test("invalid names of collations with ICU non-root localization") {
+ Seq(
+ "en_US", // must use 3-letter country code
+ "enn",
+ "en_AAA",
+ "en_Something",
+ "en_Something_USA",
+ "en_CI_UNSPECIFIED",
+ "en_USA_UNSPECIFIED",
+ "en_USA_UNSPECIFIED_CI",
+ "en_INDETERMINATE",
+ "en_USA_INDETERMINATE",
+ "en_Latn_USA", // use en_USA instead
+ "en_Cyrl_USA",
+ "en_USA_AAA",
+ "sr_Cyrl_SRB_AAA"
+ ).foreach(collationName => {
+ val error = intercept[SparkException] {
+ fetchCollation(collationName)
+ }
+
+ assert(error.getErrorClass === "COLLATION_INVALID_NAME")
+ assert(error.getMessageParameters.asScala === Map("collationName" -> collationName))
+ })
+ }
+
+ test("collations name normalization for ICU non-root localization") {
+ Seq(
+ ("en_USA", "en_USA"),
+ ("en_CS", "en"),
+ ("en_AS", "en"),
+ ("en_CS_AS", "en"),
+ ("en_AS_CS", "en"),
+ ("en_CI", "en_CI"),
+ ("en_AI", "en_AI"),
+ ("en_AI_CI", "en_CI_AI"),
+ ("en_USA_AI_CI", "en_USA_CI_AI"),
+ ("en_USA_LCASE_AI_CI", "en_USA_CI_AI_LCASE"),
+ ("en_USA_LCASE_CI_AI", "en_USA_CI_AI_LCASE"),
+ ("en_USA_AI_LCASE_CI", "en_USA_CI_AI_LCASE"),
+ ("en_USA_CI_LCASE_AI", "en_USA_CI_AI_LCASE"),
+ // randomized case
+ ("EN_USA", "en_USA"),
+ ("eN_usA_ci_uCASe_aI", "en_USA_CI_AI_UCASE"),
+ ("SR_CYRL", "sr_Cyrl"),
+ ("sr_cyrl_srb", "sr_Cyrl_SRB"),
+ ("sR_cYRl_sRb", "sr_Cyrl_SRB")
+ ).foreach {
+ case (name, normalized) =>
+ val col = fetchCollation(name)
+ assert(col.collationName == normalized)
+ }
+ }
+
+ test("invalid collationId") {
+ val badCollationIds = Seq(
+ -1, // user-defined collation range
+ 1 << 31, // user-defined collation range
+ 1 << 12, // utf8-binary mandatory zero bit 12 breach
+ 1 << 13, // utf8-binary mandatory zero bit 13 breach
+ 1 << 14, // utf8-binary mandatory zero bit 14 breach
+ 1 << 15, // utf8-binary mandatory zero bit 15 breach
+ 1 << 16, // utf8-binary mandatory zero bit 16 breach
+ 1 << 17, // utf8-binary mandatory zero bit 17 breach
+ 1 << 18, // utf8-binary mandatory zero bit 18 breach
+ 1 << 19, // utf8-binary mandatory zero bit 19 breach
+ 1 << 20, // utf8-binary mandatory zero bit 20 breach
+ 1 << 23, // utf8-binary mandatory zero bit 23 breach
+ 1 << 24, // utf8-binary mandatory zero bit 24 breach
+ 1 << 25, // utf8-binary mandatory zero bit 25 breach
+ 1 << 26, // utf8-binary mandatory zero bit 26 breach
+ (1 << 29) | (1 << 12), // ICU mandatory zero bit 12 breach
+ (1 << 29) | (1 << 13), // ICU mandatory zero bit 13 breach
+ (1 << 29) | (1 << 14), // ICU mandatory zero bit 14 breach
+ (1 << 29) | (1 << 15), // ICU mandatory zero bit 15 breach
+ (1 << 29) | (1 << 16), // ICU mandatory zero bit 16 breach
+ (1 << 29) | (1 << 17), // ICU mandatory zero bit 17 breach
+ (1 << 29) | (1 << 18), // ICU mandatory zero bit 18 breach
+ (1 << 29) | (1 << 19), // ICU mandatory zero bit 19 breach
+ (1 << 29) | (1 << 20), // ICU mandatory zero bit 20 breach
+ (1 << 29) | (1 << 23), // ICU mandatory zero bit 23 breach
+ (1 << 29) | (1 << 24), // ICU mandatory zero bit 24 breach
+ (1 << 29) | (1 << 25), // ICU mandatory zero bit 25 breach
+ (1 << 29) | (1 << 26), // ICU mandatory zero bit 26 breach
+ 123, // utf8-binary with non-zero locale id
+ (1 << 29) | (1 << 12), // ICU with invalid locale id
+ (1 << 29) | 0xFFFF, // ICU with invalid locale id
+ (1 << 23) | (1 << 22), // utf8-binary with invalid case conversion
+ (1 << 29) | (1 << 23) | (1 << 22), // ICU with invalid case conversion
+ 1 << 27, // utf8-binary accent-insensitive
+ 1 << 28, // utf8-binary case-insensitive
+ INDETERMINATE_COLLATION_ID, // indeterminate collation
+ 1 << 30, // indeterminate collation
+ (1 << 30) | (1 << 29) // invalid implementation provider
+ )
+ badCollationIds.foreach(collationId => {
+ val e = intercept[SparkException](fetchCollationUnsafe(collationId))
+ // user cannot specify collation ids directly so this is an internal error
+ assert(e.getErrorClass === "INTERNAL_ERROR")
+ })
+ badCollationIds.foreach(collationId => {
+ assert(fetchCollation(collationId).collationName == "UTF8_BINARY")
+ })
Review Comment:
Again, I don't understand why we map invalid collation IDs to UTF8_BINARY. Wouldn't it be cleaner to throw a user-facing error?
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +118,490 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30: 0 = utf8-binary, 1 = ICU
+ * bit 29: 0 = case-sensitive, 1 = case-insensitive
+ * bit 28: 0 = accent-sensitive, 1 = accent-insensitive
+ * bit 27-26: 00 = unspecified, 01 = punctuation-sensitive, 10 = punctuation-insensitive
+ * bit 25-24: 00 = unspecified, 01 = first-lower, 10 = first-upper
+ * bit 23-22: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 21-20: 00 = unspecified, 01 = trim-left, 10 = trim-right, 11 = trim-both
+ * bit 19-18: zeroes, reserved for version
+ * bit 17-16: zeroes
+ * bit 15-0: locale id for ICU collations / zeroes for utf8-binary
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
- }
- }
+ private static class CollationSpec {
+ private enum ImplementationProvider {
+ UTF8_BINARY, ICU
+ }
+
+ private enum CaseSensitivity {
+ CS, CI
+ }
+
+ private enum AccentSensitivity {
+ AS, AI
+ }
+
+ private enum PunctuationSensitivity {
+ UNSPECIFIED, PS, PI
+ }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private enum FirstLetterPreference {
+ UNSPECIFIED, FU, FL
+ }
+
+ private enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ private enum SpaceTrimming {
+ UNSPECIFIED, LTRIM, RTRIM, TRIM
+ }
+
+ private static final int implementationProviderOffset = 30;
+ private static final int implementationProviderLen = 1;
+ private static final int caseSensitivityOffset = 29;
+ private static final int caseSensitivityLen = 1;
+ private static final int accentSensitivityOffset = 28;
+ private static final int accentSensitivityLen = 1;
+ private static final int punctuationSensitivityOffset = 26;
+ private static final int punctuationSensitivityLen = 2;
+ private static final int firstLetterPreferenceOffset = 24;
+ private static final int firstLetterPreferenceLen = 2;
+ private static final int caseConversionOffset = 22;
+ private static final int caseConversionLen = 2;
+ private static final int spaceTrimmingOffset = 20;
+ private static final int spaceTrimmingLen = 2;
+ private static final int localeOffset = 0;
+ private static final int localeLen = 16;
+
+ private static final String[] ICULocaleNames;
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
Review Comment:
Along the same lines: how stable is the list of available locales? Do we expect them to change?
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected static int parseSpecifiers(
+ String originalName, String collationName, int splitStart) throws SparkException {
+ int specifiers = 0;
+ String[] parts = collationName.substring(splitStart).split("_");
+ for (String part : parts) {
+ if (!part.isEmpty()) {
+ if (part.equals("UNSPECIFIED") || part.equals("INDETERMINATE")) {
+ throw collationInvalidNameException(originalName);
+ } else if (Arrays.stream(CaseSensitivity.values()).anyMatch(
+ (s) -> s.toString().equals(part))) {
+ specifiers |=
+ CaseSensitivity.valueOf(part).ordinal() << CASE_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(AccentSensitivity.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ AccentSensitivity.valueOf(part).ordinal() << ACCENT_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(CaseConversion.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ CaseConversion.valueOf(part).ordinal() << CASE_CONVERSION_OFFSET;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+ }
+ return specifiers;
+ }
+
+ protected abstract int getCollationId();
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ public static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.LCASE).getCollationId();
+ public static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).buildCollation();
+
+ private CollationSpecUTF8Binary(CaseConversion caseConversion) {
+ super(null, CaseSensitivity.CS, AccentSensitivity.AS, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int collationId = 0;
+ int specifiers = CollationSpec.parseSpecifiers(
+ originalName, collationName, "UTF8_BINARY".length());
+ if ((specifiers & ~(CASE_CONVERSION_MASK << CASE_CONVERSION_OFFSET)) != 0) {
+ throw collationInvalidNameException(originalName);
+ }
+ collationId |= specifiers;
+ return collationId;
+ }
+
+ @Override
+ protected int getCollationId() {
+ int collationId = 0;
+ collationId |= caseConversion.ordinal() << CASE_CONVERSION_OFFSET;
+ return collationId;
+ }
+
+ public static CollationSpecUTF8Binary fromCollationId(int collationId)
+ throws SparkException {
+ int originalCollationId = collationId;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
+ if (collationId != 0 || caseConversionOrdinal >= CaseConversion.values().length) {
+ throw SparkException.internalError("Invalid UTF8_BINARY collation id " +
+ originalCollationId);
+ } else {
+ CaseConversion caseConversion = CaseConversion.values()[caseConversionOrdinal];
+ return new CollationSpecUTF8Binary(caseConversion);
+ }
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ Comparator<UTF8String> comparator;
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ comparator = UTF8String::binaryCompare;
+ } else if (collationId == UTF8_BINARY_LCASE_COLLATION_ID) {
+ comparator = UTF8String::compareLowerCase;
+ } else {
+ comparator = (s1, s2) -> {
+ UTF8String convertedS1 = caseConversion(s1);
+ UTF8String convertedS2 = caseConversion(s2);
+ return convertedS1.binaryCompare(convertedS2);
+ };
+ }
+ return new Collation(
+ collationName(),
+ null,
+ comparator,
+ "1.0",
+ s -> (long) caseConversion(s).hashCode(),
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_LCASE_COLLATION_ID
+ );
+ }
+
+ private UTF8String caseConversion(UTF8String s) {
+ if (caseConversion == CaseConversion.LCASE) {
+ return s.toLowerCase();
+ } else if (caseConversion == CaseConversion.UCASE) {
+ return s.toUpperCase();
+ } else {
+ return s;
+ }
+ }
+
+ private String collationName() {
+ StringBuilder builder = new StringBuilder();
+ builder.append("UTF8_BINARY");
+ if (caseConversion != CaseConversion.UNSPECIFIED) {
+ builder.append('_');
+ builder.append(caseConversion.toString());
+ }
+ return builder.toString();
+ }
+ }
+
+ private static class CollationSpecICU extends CollationSpec {
+
+ private static final String[] ICULocaleNames;
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+ private static final String ICUCollatorVersion = "153.120.0.0";
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
+ for (ULocale locale : locales) {
+ if (locale.getVariant().isEmpty()) {
+ String language = locale.getLanguage();
+ assert (!language.isEmpty());
+ StringBuilder builder = new StringBuilder(language);
+ String script = locale.getScript();
+ if (!script.isEmpty()) {
+ builder.append('_');
+ builder.append(script);
+ }
+ String country = locale.getISO3Country();
+ if (!country.isEmpty()) {
+ builder.append('_');
+ builder.append(country);
+ }
+ String localeName = builder.toString();
+ assert (!ICULocaleMap.containsKey(localeName));
+ ICULocaleMap.put(localeName, locale);
+ }
+ }
+ for (String localeName : ICULocaleMap.keySet()) {
+ String localeUppercase = localeName.toUpperCase();
+ assert (!ICULocaleMapUppercase.containsKey(localeUppercase));
+ ICULocaleMapUppercase.put(localeUppercase, localeName);
+ }
+ ICULocaleNames = ICULocaleMap.keySet().toArray(new String[0]);
+ Arrays.sort(ICULocaleNames);
+ assert (ICULocaleNames.length <= (1 << 16));
+ for (int i = 0; i < ICULocaleNames.length; i++) {
+ ICULocaleToId.put(ICULocaleNames[i], i);
+ }
Review Comment:
Another question: Previously we used 0 and 1 for UTF8_BINARY and UTF8_BINARY_LCASE. Do these remain the same?
Also would it be possible to make sure that all collations (ICU and the custom ones) have unique IDs? One way to achieve this is to number the ICU collations starting from a fixed offset, say 100.
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -245,29 +599,26 @@ public static StringSearch getStringSearch(
* Returns the collation id for the given collation name.
*/
public static int collationNameToId(String collationName) throws SparkException {
- String normalizedName = collationName.toUpperCase();
- if (collationNameToIdMap.containsKey(normalizedName)) {
- return collationNameToIdMap.get(normalizedName);
- } else {
- Collation suggestion = Collections.min(List.of(collationTable), Comparator.comparingInt(
- c -> UTF8String.fromString(c.collationName).levenshteinDistance(
- UTF8String.fromString(normalizedName))));
-
- Map<String, String> params = new HashMap<>();
- params.put("collationName", collationName);
- params.put("proposal", suggestion.collationName);
-
- throw new SparkException(
- "COLLATION_INVALID_NAME", SparkException.constructMessageParams(params), null);
- }
+ return Collation.CollationSpec.collationNameToId(collationName);
+ }
+
+ public static Collation fetchCollationUnsafe(int collationId) throws SparkException {
+ return Collation.CollationSpec.fetchCollation(collationId);
}
public static Collation fetchCollation(int collationId) {
- return collationTable[collationId];
+ try {
+ return fetchCollationUnsafe(collationId);
+ } catch (SparkException e) {
+ return Collation.CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ }
Review Comment:
That seems a bit strange. Why would we default yo UTF8_BINATY if the look up fails as opposed to throwing an error?
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
Review Comment:
I would make this a method.
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected static int parseSpecifiers(
+ String originalName, String collationName, int splitStart) throws SparkException {
+ int specifiers = 0;
+ String[] parts = collationName.substring(splitStart).split("_");
+ for (String part : parts) {
+ if (!part.isEmpty()) {
+ if (part.equals("UNSPECIFIED") || part.equals("INDETERMINATE")) {
+ throw collationInvalidNameException(originalName);
+ } else if (Arrays.stream(CaseSensitivity.values()).anyMatch(
+ (s) -> s.toString().equals(part))) {
+ specifiers |=
+ CaseSensitivity.valueOf(part).ordinal() << CASE_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(AccentSensitivity.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ AccentSensitivity.valueOf(part).ordinal() << ACCENT_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(CaseConversion.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ CaseConversion.valueOf(part).ordinal() << CASE_CONVERSION_OFFSET;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+ }
+ return specifiers;
+ }
+
+ protected abstract int getCollationId();
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ public static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.LCASE).getCollationId();
+ public static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).buildCollation();
+
+ private CollationSpecUTF8Binary(CaseConversion caseConversion) {
+ super(null, CaseSensitivity.CS, AccentSensitivity.AS, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int collationId = 0;
+ int specifiers = CollationSpec.parseSpecifiers(
+ originalName, collationName, "UTF8_BINARY".length());
+ if ((specifiers & ~(CASE_CONVERSION_MASK << CASE_CONVERSION_OFFSET)) != 0) {
+ throw collationInvalidNameException(originalName);
+ }
+ collationId |= specifiers;
+ return collationId;
+ }
+
+ @Override
+ protected int getCollationId() {
+ int collationId = 0;
+ collationId |= caseConversion.ordinal() << CASE_CONVERSION_OFFSET;
+ return collationId;
+ }
+
+ public static CollationSpecUTF8Binary fromCollationId(int collationId)
+ throws SparkException {
+ int originalCollationId = collationId;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
+ if (collationId != 0 || caseConversionOrdinal >= CaseConversion.values().length) {
+ throw SparkException.internalError("Invalid UTF8_BINARY collation id " +
+ originalCollationId);
+ } else {
+ CaseConversion caseConversion = CaseConversion.values()[caseConversionOrdinal];
+ return new CollationSpecUTF8Binary(caseConversion);
+ }
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ Comparator<UTF8String> comparator;
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ comparator = UTF8String::binaryCompare;
+ } else if (collationId == UTF8_BINARY_LCASE_COLLATION_ID) {
+ comparator = UTF8String::compareLowerCase;
+ } else {
+ comparator = (s1, s2) -> {
+ UTF8String convertedS1 = caseConversion(s1);
+ UTF8String convertedS2 = caseConversion(s2);
+ return convertedS1.binaryCompare(convertedS2);
+ };
+ }
+ return new Collation(
+ collationName(),
+ null,
+ comparator,
+ "1.0",
+ s -> (long) caseConversion(s).hashCode(),
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_LCASE_COLLATION_ID
+ );
+ }
+
+ private UTF8String caseConversion(UTF8String s) {
+ if (caseConversion == CaseConversion.LCASE) {
+ return s.toLowerCase();
+ } else if (caseConversion == CaseConversion.UCASE) {
+ return s.toUpperCase();
+ } else {
+ return s;
+ }
+ }
+
+ private String collationName() {
+ StringBuilder builder = new StringBuilder();
+ builder.append("UTF8_BINARY");
+ if (caseConversion != CaseConversion.UNSPECIFIED) {
+ builder.append('_');
+ builder.append(caseConversion.toString());
+ }
+ return builder.toString();
+ }
+ }
+
+ private static class CollationSpecICU extends CollationSpec {
+
+ private static final String[] ICULocaleNames;
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+ private static final String ICUCollatorVersion = "153.120.0.0";
Review Comment:
Could you please add comments for each one of these members? (what they are for for example)
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected static int parseSpecifiers(
+ String originalName, String collationName, int splitStart) throws SparkException {
+ int specifiers = 0;
+ String[] parts = collationName.substring(splitStart).split("_");
+ for (String part : parts) {
+ if (!part.isEmpty()) {
+ if (part.equals("UNSPECIFIED") || part.equals("INDETERMINATE")) {
+ throw collationInvalidNameException(originalName);
+ } else if (Arrays.stream(CaseSensitivity.values()).anyMatch(
+ (s) -> s.toString().equals(part))) {
+ specifiers |=
+ CaseSensitivity.valueOf(part).ordinal() << CASE_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(AccentSensitivity.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ AccentSensitivity.valueOf(part).ordinal() << ACCENT_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(CaseConversion.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ CaseConversion.valueOf(part).ordinal() << CASE_CONVERSION_OFFSET;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+ }
+ return specifiers;
+ }
+
+ protected abstract int getCollationId();
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ public static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.LCASE).getCollationId();
+ public static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).buildCollation();
+
+ private CollationSpecUTF8Binary(CaseConversion caseConversion) {
+ super(null, CaseSensitivity.CS, AccentSensitivity.AS, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int collationId = 0;
+ int specifiers = CollationSpec.parseSpecifiers(
+ originalName, collationName, "UTF8_BINARY".length());
+ if ((specifiers & ~(CASE_CONVERSION_MASK << CASE_CONVERSION_OFFSET)) != 0) {
+ throw collationInvalidNameException(originalName);
+ }
+ collationId |= specifiers;
+ return collationId;
+ }
+
+ @Override
+ protected int getCollationId() {
+ int collationId = 0;
+ collationId |= caseConversion.ordinal() << CASE_CONVERSION_OFFSET;
+ return collationId;
+ }
+
+ public static CollationSpecUTF8Binary fromCollationId(int collationId)
+ throws SparkException {
+ int originalCollationId = collationId;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
+ if (collationId != 0 || caseConversionOrdinal >= CaseConversion.values().length) {
+ throw SparkException.internalError("Invalid UTF8_BINARY collation id " +
+ originalCollationId);
+ } else {
+ CaseConversion caseConversion = CaseConversion.values()[caseConversionOrdinal];
+ return new CollationSpecUTF8Binary(caseConversion);
+ }
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ Comparator<UTF8String> comparator;
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ comparator = UTF8String::binaryCompare;
+ } else if (collationId == UTF8_BINARY_LCASE_COLLATION_ID) {
+ comparator = UTF8String::compareLowerCase;
+ } else {
+ comparator = (s1, s2) -> {
+ UTF8String convertedS1 = caseConversion(s1);
+ UTF8String convertedS2 = caseConversion(s2);
+ return convertedS1.binaryCompare(convertedS2);
+ };
+ }
+ return new Collation(
+ collationName(),
+ null,
+ comparator,
+ "1.0",
+ s -> (long) caseConversion(s).hashCode(),
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_LCASE_COLLATION_ID
+ );
+ }
+
+ private UTF8String caseConversion(UTF8String s) {
+ if (caseConversion == CaseConversion.LCASE) {
+ return s.toLowerCase();
+ } else if (caseConversion == CaseConversion.UCASE) {
+ return s.toUpperCase();
+ } else {
+ return s;
+ }
+ }
+
+ private String collationName() {
+ StringBuilder builder = new StringBuilder();
+ builder.append("UTF8_BINARY");
+ if (caseConversion != CaseConversion.UNSPECIFIED) {
+ builder.append('_');
+ builder.append(caseConversion.toString());
+ }
+ return builder.toString();
+ }
+ }
+
+ private static class CollationSpecICU extends CollationSpec {
+
+ private static final String[] ICULocaleNames;
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+ private static final String ICUCollatorVersion = "153.120.0.0";
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
+ for (ULocale locale : locales) {
+ if (locale.getVariant().isEmpty()) {
+ String language = locale.getLanguage();
+ assert (!language.isEmpty());
+ StringBuilder builder = new StringBuilder(language);
+ String script = locale.getScript();
+ if (!script.isEmpty()) {
+ builder.append('_');
+ builder.append(script);
+ }
+ String country = locale.getISO3Country();
+ if (!country.isEmpty()) {
+ builder.append('_');
+ builder.append(country);
+ }
+ String localeName = builder.toString();
+ assert (!ICULocaleMap.containsKey(localeName));
+ ICULocaleMap.put(localeName, locale);
+ }
+ }
+ for (String localeName : ICULocaleMap.keySet()) {
+ String localeUppercase = localeName.toUpperCase();
+ assert (!ICULocaleMapUppercase.containsKey(localeUppercase));
+ ICULocaleMapUppercase.put(localeUppercase, localeName);
+ }
+ ICULocaleNames = ICULocaleMap.keySet().toArray(new String[0]);
+ Arrays.sort(ICULocaleNames);
+ assert (ICULocaleNames.length <= (1 << 16));
+ for (int i = 0; i < ICULocaleNames.length; i++) {
+ ICULocaleToId.put(ICULocaleNames[i], i);
+ }
+ }
+
+ public static final int UNICODE_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CS, AccentSensitivity.AS,
+ CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UNICODE_CI_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CI, AccentSensitivity.AS,
+ CaseConversion.UNSPECIFIED).getCollationId();
+
+ private CollationSpecICU(String locale, CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity, CaseConversion caseConversion) {
+ super(locale, caseSensitivity, accentSensitivity, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int lastPos = -1;
+ for (int i = 1; i <= collationName.length(); i++) {
+ String localeName = collationName.substring(0, i);
+ if (ICULocaleMapUppercase.containsKey(localeName)) {
+ lastPos = i;
+ }
Review Comment:
It is obvious that there no two locale names such that one is not a prefix of the other?
##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -30,31 +33,107 @@ import org.apache.spark.unsafe.types.UTF8String.{fromString => toUTF8}
class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ignore funsuite
test("collationId stability") {
- val utf8Binary = fetchCollation(0)
+ assert(UTF8_BINARY_COLLATION_ID == 0)
+ assert(INDETERMINATE_COLLATION_ID == 1 << 30)
+
+ val utf8Binary = fetchCollation(UTF8_BINARY_COLLATION_ID)
assert(utf8Binary.collationName == "UTF8_BINARY")
assert(utf8Binary.supportsBinaryEquality)
- val utf8BinaryLcase = fetchCollation(1)
+ val utf8BinaryLcase = fetchCollation(UTF8_BINARY_LCASE_COLLATION_ID)
assert(utf8BinaryLcase.collationName == "UTF8_BINARY_LCASE")
assert(!utf8BinaryLcase.supportsBinaryEquality)
- val unicode = fetchCollation(2)
+ val unicode = fetchCollation(UNICODE_COLLATION_ID)
assert(unicode.collationName == "UNICODE")
- assert(unicode.supportsBinaryEquality);
+ assert(unicode.supportsBinaryEquality)
- val unicodeCi = fetchCollation(3)
+ val unicodeCi = fetchCollation(UNICODE_CI_COLLATION_ID)
assert(unicodeCi.collationName == "UNICODE_CI")
assert(!unicodeCi.supportsBinaryEquality)
}
- test("fetch invalid collation name") {
- val error = intercept[SparkException] {
- fetchCollation("UTF8_BS")
+ test("UTF8_BINARY and ICU root locale collation names") {
+ // collation name already normalized
+ Seq(
+ "UTF8_BINARY",
+ "UTF8_BINARY_LCASE",
+ "UTF8_BINARY_UCASE",
+ "UNICODE",
+ "UNICODE_CI",
+ "UNICODE_AI",
+ "UNICODE_CI_AI",
+ "UNICODE_LCASE",
+ "UNICODE_UCASE"
+ ).foreach(collationName => {
+ val col = fetchCollation(collationName)
+ assert(col.collationName == collationName)
+ })
+ // collation name normalization
+ Seq(
+ // UTF8_BINARY
+ ("UTF8_BINARY_CS", "UTF8_BINARY"),
+ ("UTF8_BINARY_AS", "UTF8_BINARY"),
+ ("UTF8_BINARY_CS_AS", "UTF8_BINARY"),
+ ("UTF8_BINARY_AS_CS", "UTF8_BINARY"),
Review Comment:
Following the comment by @bart-samwel : let's try to limit having different names map to the same collation.
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected static int parseSpecifiers(
+ String originalName, String collationName, int splitStart) throws SparkException {
+ int specifiers = 0;
+ String[] parts = collationName.substring(splitStart).split("_");
+ for (String part : parts) {
+ if (!part.isEmpty()) {
+ if (part.equals("UNSPECIFIED") || part.equals("INDETERMINATE")) {
+ throw collationInvalidNameException(originalName);
+ } else if (Arrays.stream(CaseSensitivity.values()).anyMatch(
+ (s) -> s.toString().equals(part))) {
+ specifiers |=
+ CaseSensitivity.valueOf(part).ordinal() << CASE_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(AccentSensitivity.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ AccentSensitivity.valueOf(part).ordinal() << ACCENT_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(CaseConversion.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ CaseConversion.valueOf(part).ordinal() << CASE_CONVERSION_OFFSET;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+ }
+ return specifiers;
+ }
+
+ protected abstract int getCollationId();
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ public static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.LCASE).getCollationId();
+ public static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).buildCollation();
+
+ private CollationSpecUTF8Binary(CaseConversion caseConversion) {
+ super(null, CaseSensitivity.CS, AccentSensitivity.AS, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int collationId = 0;
+ int specifiers = CollationSpec.parseSpecifiers(
+ originalName, collationName, "UTF8_BINARY".length());
+ if ((specifiers & ~(CASE_CONVERSION_MASK << CASE_CONVERSION_OFFSET)) != 0) {
+ throw collationInvalidNameException(originalName);
+ }
+ collationId |= specifiers;
+ return collationId;
+ }
+
+ @Override
+ protected int getCollationId() {
+ int collationId = 0;
+ collationId |= caseConversion.ordinal() << CASE_CONVERSION_OFFSET;
+ return collationId;
+ }
+
+ public static CollationSpecUTF8Binary fromCollationId(int collationId)
+ throws SparkException {
+ int originalCollationId = collationId;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
Review Comment:
The more I see these bit operations all over the place the more I think we need to encapsulate the collation ID representation inside a class and put all these bit operations in there in a single place.
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected static int parseSpecifiers(
+ String originalName, String collationName, int splitStart) throws SparkException {
+ int specifiers = 0;
+ String[] parts = collationName.substring(splitStart).split("_");
+ for (String part : parts) {
+ if (!part.isEmpty()) {
+ if (part.equals("UNSPECIFIED") || part.equals("INDETERMINATE")) {
+ throw collationInvalidNameException(originalName);
+ } else if (Arrays.stream(CaseSensitivity.values()).anyMatch(
+ (s) -> s.toString().equals(part))) {
+ specifiers |=
+ CaseSensitivity.valueOf(part).ordinal() << CASE_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(AccentSensitivity.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ AccentSensitivity.valueOf(part).ordinal() << ACCENT_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(CaseConversion.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ CaseConversion.valueOf(part).ordinal() << CASE_CONVERSION_OFFSET;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+ }
+ return specifiers;
+ }
+
+ protected abstract int getCollationId();
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ public static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.LCASE).getCollationId();
+ public static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).buildCollation();
+
+ private CollationSpecUTF8Binary(CaseConversion caseConversion) {
+ super(null, CaseSensitivity.CS, AccentSensitivity.AS, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int collationId = 0;
+ int specifiers = CollationSpec.parseSpecifiers(
+ originalName, collationName, "UTF8_BINARY".length());
+ if ((specifiers & ~(CASE_CONVERSION_MASK << CASE_CONVERSION_OFFSET)) != 0) {
+ throw collationInvalidNameException(originalName);
+ }
+ collationId |= specifiers;
+ return collationId;
+ }
+
+ @Override
+ protected int getCollationId() {
+ int collationId = 0;
+ collationId |= caseConversion.ordinal() << CASE_CONVERSION_OFFSET;
+ return collationId;
+ }
+
+ public static CollationSpecUTF8Binary fromCollationId(int collationId)
+ throws SparkException {
+ int originalCollationId = collationId;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
+ if (collationId != 0 || caseConversionOrdinal >= CaseConversion.values().length) {
+ throw SparkException.internalError("Invalid UTF8_BINARY collation id " +
+ originalCollationId);
+ } else {
+ CaseConversion caseConversion = CaseConversion.values()[caseConversionOrdinal];
+ return new CollationSpecUTF8Binary(caseConversion);
+ }
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ Comparator<UTF8String> comparator;
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ comparator = UTF8String::binaryCompare;
+ } else if (collationId == UTF8_BINARY_LCASE_COLLATION_ID) {
+ comparator = UTF8String::compareLowerCase;
+ } else {
+ comparator = (s1, s2) -> {
+ UTF8String convertedS1 = caseConversion(s1);
+ UTF8String convertedS2 = caseConversion(s2);
+ return convertedS1.binaryCompare(convertedS2);
+ };
+ }
+ return new Collation(
+ collationName(),
+ null,
+ comparator,
+ "1.0",
+ s -> (long) caseConversion(s).hashCode(),
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_LCASE_COLLATION_ID
+ );
+ }
+
+ private UTF8String caseConversion(UTF8String s) {
+ if (caseConversion == CaseConversion.LCASE) {
+ return s.toLowerCase();
+ } else if (caseConversion == CaseConversion.UCASE) {
+ return s.toUpperCase();
+ } else {
+ return s;
+ }
+ }
+
+ private String collationName() {
+ StringBuilder builder = new StringBuilder();
+ builder.append("UTF8_BINARY");
+ if (caseConversion != CaseConversion.UNSPECIFIED) {
+ builder.append('_');
+ builder.append(caseConversion.toString());
+ }
+ return builder.toString();
+ }
+ }
+
+ private static class CollationSpecICU extends CollationSpec {
+
+ private static final String[] ICULocaleNames;
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+ private static final String ICUCollatorVersion = "153.120.0.0";
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
+ for (ULocale locale : locales) {
+ if (locale.getVariant().isEmpty()) {
+ String language = locale.getLanguage();
+ assert (!language.isEmpty());
+ StringBuilder builder = new StringBuilder(language);
+ String script = locale.getScript();
+ if (!script.isEmpty()) {
+ builder.append('_');
+ builder.append(script);
+ }
+ String country = locale.getISO3Country();
+ if (!country.isEmpty()) {
+ builder.append('_');
+ builder.append(country);
+ }
+ String localeName = builder.toString();
+ assert (!ICULocaleMap.containsKey(localeName));
+ ICULocaleMap.put(localeName, locale);
+ }
+ }
+ for (String localeName : ICULocaleMap.keySet()) {
+ String localeUppercase = localeName.toUpperCase();
+ assert (!ICULocaleMapUppercase.containsKey(localeUppercase));
+ ICULocaleMapUppercase.put(localeUppercase, localeName);
+ }
+ ICULocaleNames = ICULocaleMap.keySet().toArray(new String[0]);
+ Arrays.sort(ICULocaleNames);
+ assert (ICULocaleNames.length <= (1 << 16));
+ for (int i = 0; i < ICULocaleNames.length; i++) {
+ ICULocaleToId.put(ICULocaleNames[i], i);
+ }
+ }
+
+ public static final int UNICODE_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CS, AccentSensitivity.AS,
+ CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UNICODE_CI_COLLATION_ID =
+ new CollationSpecICU("UNICODE", CaseSensitivity.CI, AccentSensitivity.AS,
+ CaseConversion.UNSPECIFIED).getCollationId();
+
+ private CollationSpecICU(String locale, CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity, CaseConversion caseConversion) {
+ super(locale, caseSensitivity, accentSensitivity, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int lastPos = -1;
+ for (int i = 1; i <= collationName.length(); i++) {
Review Comment:
```suggestion
for (int i = 1; i <= collationName.length(); ++i) {
```
##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -30,31 +33,107 @@ import org.apache.spark.unsafe.types.UTF8String.{fromString => toUTF8}
class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ignore funsuite
test("collationId stability") {
- val utf8Binary = fetchCollation(0)
+ assert(UTF8_BINARY_COLLATION_ID == 0)
+ assert(INDETERMINATE_COLLATION_ID == 1 << 30)
+
+ val utf8Binary = fetchCollation(UTF8_BINARY_COLLATION_ID)
assert(utf8Binary.collationName == "UTF8_BINARY")
assert(utf8Binary.supportsBinaryEquality)
- val utf8BinaryLcase = fetchCollation(1)
+ val utf8BinaryLcase = fetchCollation(UTF8_BINARY_LCASE_COLLATION_ID)
assert(utf8BinaryLcase.collationName == "UTF8_BINARY_LCASE")
assert(!utf8BinaryLcase.supportsBinaryEquality)
- val unicode = fetchCollation(2)
+ val unicode = fetchCollation(UNICODE_COLLATION_ID)
assert(unicode.collationName == "UNICODE")
- assert(unicode.supportsBinaryEquality);
+ assert(unicode.supportsBinaryEquality)
- val unicodeCi = fetchCollation(3)
+ val unicodeCi = fetchCollation(UNICODE_CI_COLLATION_ID)
Review Comment:
These IDs should be stable as well, right? If yes, we should add asserts for them as well.
##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -30,31 +33,107 @@ import org.apache.spark.unsafe.types.UTF8String.{fromString => toUTF8}
class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ignore funsuite
test("collationId stability") {
- val utf8Binary = fetchCollation(0)
+ assert(UTF8_BINARY_COLLATION_ID == 0)
+ assert(INDETERMINATE_COLLATION_ID == 1 << 30)
+
+ val utf8Binary = fetchCollation(UTF8_BINARY_COLLATION_ID)
assert(utf8Binary.collationName == "UTF8_BINARY")
assert(utf8Binary.supportsBinaryEquality)
- val utf8BinaryLcase = fetchCollation(1)
+ val utf8BinaryLcase = fetchCollation(UTF8_BINARY_LCASE_COLLATION_ID)
assert(utf8BinaryLcase.collationName == "UTF8_BINARY_LCASE")
assert(!utf8BinaryLcase.supportsBinaryEquality)
- val unicode = fetchCollation(2)
+ val unicode = fetchCollation(UNICODE_COLLATION_ID)
assert(unicode.collationName == "UNICODE")
- assert(unicode.supportsBinaryEquality);
+ assert(unicode.supportsBinaryEquality)
- val unicodeCi = fetchCollation(3)
+ val unicodeCi = fetchCollation(UNICODE_CI_COLLATION_ID)
assert(unicodeCi.collationName == "UNICODE_CI")
assert(!unicodeCi.supportsBinaryEquality)
}
- test("fetch invalid collation name") {
- val error = intercept[SparkException] {
- fetchCollation("UTF8_BS")
+ test("UTF8_BINARY and ICU root locale collation names") {
+ // collation name already normalized
+ Seq(
+ "UTF8_BINARY",
+ "UTF8_BINARY_LCASE",
+ "UTF8_BINARY_UCASE",
+ "UNICODE",
+ "UNICODE_CI",
+ "UNICODE_AI",
+ "UNICODE_CI_AI",
+ "UNICODE_LCASE",
+ "UNICODE_UCASE"
+ ).foreach(collationName => {
+ val col = fetchCollation(collationName)
+ assert(col.collationName == collationName)
+ })
+ // collation name normalization
+ Seq(
+ // UTF8_BINARY
+ ("UTF8_BINARY_CS", "UTF8_BINARY"),
+ ("UTF8_BINARY_AS", "UTF8_BINARY"),
+ ("UTF8_BINARY_CS_AS", "UTF8_BINARY"),
+ ("UTF8_BINARY_AS_CS", "UTF8_BINARY"),
+ ("UTF8_BINARY_LCASE_CS", "UTF8_BINARY_LCASE"),
+ ("UTF8_BINARY_CS_LCASE", "UTF8_BINARY_LCASE"),
+ ("UTF8_BINARY_LCASE_AS", "UTF8_BINARY_LCASE"),
+ ("UTF8_BINARY_AS_LCASE", "UTF8_BINARY_LCASE"),
+ ("UTF8_BINARY_CS_LCASE_AS", "UTF8_BINARY_LCASE"),
+ ("UTF8_BINARY_AS_CS_LCASE", "UTF8_BINARY_LCASE"),
+ // ICU root locale
+ ("UNICODE_CS", "UNICODE"),
+ ("UNICODE_CS_AS", "UNICODE"),
+ ("UNICODE_CI_AS", "UNICODE_CI"),
+ ("UNICODE_AI_CS", "UNICODE_AI"),
+ ("UNICODE_AI_CI", "UNICODE_CI_AI"),
+ ("UNICODE_LCASE_CS", "UNICODE_LCASE"),
+ ("UNICODE_LCASE_AS", "UNICODE_LCASE"),
+ ("UNICODE_LCASE_CI", "UNICODE_CI_LCASE"),
+ ("UNICODE_UCASE_CI", "UNICODE_CI_UCASE"),
+ ("UNICODE_AI_UCASE_CI", "UNICODE_CI_AI_UCASE"),
Review Comment:
Are these real collations?
##########
connector/connect/common/src/main/protobuf/spark/connect/types.proto:
##########
@@ -101,7 +101,7 @@ message DataType {
message String {
uint32 type_variation_reference = 1;
- uint32 collation_id = 2;
+ string collation = 2;
Review Comment:
Why are we making this change?
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected static int parseSpecifiers(
+ String originalName, String collationName, int splitStart) throws SparkException {
+ int specifiers = 0;
+ String[] parts = collationName.substring(splitStart).split("_");
+ for (String part : parts) {
+ if (!part.isEmpty()) {
+ if (part.equals("UNSPECIFIED") || part.equals("INDETERMINATE")) {
+ throw collationInvalidNameException(originalName);
+ } else if (Arrays.stream(CaseSensitivity.values()).anyMatch(
+ (s) -> s.toString().equals(part))) {
+ specifiers |=
+ CaseSensitivity.valueOf(part).ordinal() << CASE_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(AccentSensitivity.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ AccentSensitivity.valueOf(part).ordinal() << ACCENT_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(CaseConversion.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ CaseConversion.valueOf(part).ordinal() << CASE_CONVERSION_OFFSET;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+ }
+ return specifiers;
+ }
+
+ protected abstract int getCollationId();
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ public static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.LCASE).getCollationId();
+ public static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).buildCollation();
+
+ private CollationSpecUTF8Binary(CaseConversion caseConversion) {
+ super(null, CaseSensitivity.CS, AccentSensitivity.AS, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int collationId = 0;
+ int specifiers = CollationSpec.parseSpecifiers(
+ originalName, collationName, "UTF8_BINARY".length());
+ if ((specifiers & ~(CASE_CONVERSION_MASK << CASE_CONVERSION_OFFSET)) != 0) {
+ throw collationInvalidNameException(originalName);
+ }
+ collationId |= specifiers;
+ return collationId;
+ }
+
+ @Override
+ protected int getCollationId() {
+ int collationId = 0;
+ collationId |= caseConversion.ordinal() << CASE_CONVERSION_OFFSET;
+ return collationId;
+ }
+
+ public static CollationSpecUTF8Binary fromCollationId(int collationId)
+ throws SparkException {
+ int originalCollationId = collationId;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
+ if (collationId != 0 || caseConversionOrdinal >= CaseConversion.values().length) {
+ throw SparkException.internalError("Invalid UTF8_BINARY collation id " +
+ originalCollationId);
+ } else {
+ CaseConversion caseConversion = CaseConversion.values()[caseConversionOrdinal];
+ return new CollationSpecUTF8Binary(caseConversion);
+ }
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ Comparator<UTF8String> comparator;
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ comparator = UTF8String::binaryCompare;
+ } else if (collationId == UTF8_BINARY_LCASE_COLLATION_ID) {
+ comparator = UTF8String::compareLowerCase;
+ } else {
+ comparator = (s1, s2) -> {
+ UTF8String convertedS1 = caseConversion(s1);
+ UTF8String convertedS2 = caseConversion(s2);
+ return convertedS1.binaryCompare(convertedS2);
+ };
+ }
+ return new Collation(
+ collationName(),
+ null,
+ comparator,
+ "1.0",
+ s -> (long) caseConversion(s).hashCode(),
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_LCASE_COLLATION_ID
+ );
+ }
+
+ private UTF8String caseConversion(UTF8String s) {
+ if (caseConversion == CaseConversion.LCASE) {
+ return s.toLowerCase();
+ } else if (caseConversion == CaseConversion.UCASE) {
+ return s.toUpperCase();
+ } else {
+ return s;
+ }
+ }
+
+ private String collationName() {
+ StringBuilder builder = new StringBuilder();
+ builder.append("UTF8_BINARY");
+ if (caseConversion != CaseConversion.UNSPECIFIED) {
+ builder.append('_');
+ builder.append(caseConversion.toString());
+ }
+ return builder.toString();
+ }
+ }
+
+ private static class CollationSpecICU extends CollationSpec {
+
+ private static final String[] ICULocaleNames;
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+ private static final String ICUCollatorVersion = "153.120.0.0";
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
+ for (ULocale locale : locales) {
+ if (locale.getVariant().isEmpty()) {
+ String language = locale.getLanguage();
+ assert (!language.isEmpty());
+ StringBuilder builder = new StringBuilder(language);
+ String script = locale.getScript();
+ if (!script.isEmpty()) {
+ builder.append('_');
+ builder.append(script);
+ }
+ String country = locale.getISO3Country();
+ if (!country.isEmpty()) {
+ builder.append('_');
+ builder.append(country);
+ }
+ String localeName = builder.toString();
+ assert (!ICULocaleMap.containsKey(localeName));
+ ICULocaleMap.put(localeName, locale);
+ }
+ }
+ for (String localeName : ICULocaleMap.keySet()) {
+ String localeUppercase = localeName.toUpperCase();
+ assert (!ICULocaleMapUppercase.containsKey(localeUppercase));
+ ICULocaleMapUppercase.put(localeUppercase, localeName);
+ }
Review Comment:
I am missing the logic behind having uppercase version of the locale name. Could we add a comment for that? (probably the best place is where we define the `ICULocaleMapUppercase` map.
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected static int parseSpecifiers(
+ String originalName, String collationName, int splitStart) throws SparkException {
+ int specifiers = 0;
+ String[] parts = collationName.substring(splitStart).split("_");
+ for (String part : parts) {
+ if (!part.isEmpty()) {
+ if (part.equals("UNSPECIFIED") || part.equals("INDETERMINATE")) {
+ throw collationInvalidNameException(originalName);
+ } else if (Arrays.stream(CaseSensitivity.values()).anyMatch(
+ (s) -> s.toString().equals(part))) {
+ specifiers |=
+ CaseSensitivity.valueOf(part).ordinal() << CASE_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(AccentSensitivity.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ AccentSensitivity.valueOf(part).ordinal() << ACCENT_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(CaseConversion.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ CaseConversion.valueOf(part).ordinal() << CASE_CONVERSION_OFFSET;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+ }
+ return specifiers;
+ }
+
+ protected abstract int getCollationId();
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ public static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.LCASE).getCollationId();
+ public static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).buildCollation();
+
+ private CollationSpecUTF8Binary(CaseConversion caseConversion) {
+ super(null, CaseSensitivity.CS, AccentSensitivity.AS, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int collationId = 0;
+ int specifiers = CollationSpec.parseSpecifiers(
+ originalName, collationName, "UTF8_BINARY".length());
+ if ((specifiers & ~(CASE_CONVERSION_MASK << CASE_CONVERSION_OFFSET)) != 0) {
+ throw collationInvalidNameException(originalName);
+ }
+ collationId |= specifiers;
+ return collationId;
+ }
+
+ @Override
+ protected int getCollationId() {
+ int collationId = 0;
+ collationId |= caseConversion.ordinal() << CASE_CONVERSION_OFFSET;
+ return collationId;
+ }
+
+ public static CollationSpecUTF8Binary fromCollationId(int collationId)
+ throws SparkException {
+ int originalCollationId = collationId;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
+ if (collationId != 0 || caseConversionOrdinal >= CaseConversion.values().length) {
+ throw SparkException.internalError("Invalid UTF8_BINARY collation id " +
+ originalCollationId);
+ } else {
+ CaseConversion caseConversion = CaseConversion.values()[caseConversionOrdinal];
+ return new CollationSpecUTF8Binary(caseConversion);
+ }
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ Comparator<UTF8String> comparator;
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ comparator = UTF8String::binaryCompare;
+ } else if (collationId == UTF8_BINARY_LCASE_COLLATION_ID) {
+ comparator = UTF8String::compareLowerCase;
+ } else {
+ comparator = (s1, s2) -> {
+ UTF8String convertedS1 = caseConversion(s1);
+ UTF8String convertedS2 = caseConversion(s2);
+ return convertedS1.binaryCompare(convertedS2);
+ };
+ }
+ return new Collation(
+ collationName(),
+ null,
+ comparator,
+ "1.0",
+ s -> (long) caseConversion(s).hashCode(),
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_LCASE_COLLATION_ID
+ );
+ }
+
+ private UTF8String caseConversion(UTF8String s) {
+ if (caseConversion == CaseConversion.LCASE) {
+ return s.toLowerCase();
+ } else if (caseConversion == CaseConversion.UCASE) {
+ return s.toUpperCase();
+ } else {
+ return s;
+ }
+ }
+
+ private String collationName() {
+ StringBuilder builder = new StringBuilder();
+ builder.append("UTF8_BINARY");
+ if (caseConversion != CaseConversion.UNSPECIFIED) {
+ builder.append('_');
+ builder.append(caseConversion.toString());
+ }
+ return builder.toString();
+ }
+ }
+
+ private static class CollationSpecICU extends CollationSpec {
+
+ private static final String[] ICULocaleNames;
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+ private static final String ICUCollatorVersion = "153.120.0.0";
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
+ for (ULocale locale : locales) {
+ if (locale.getVariant().isEmpty()) {
+ String language = locale.getLanguage();
+ assert (!language.isEmpty());
+ StringBuilder builder = new StringBuilder(language);
+ String script = locale.getScript();
+ if (!script.isEmpty()) {
+ builder.append('_');
+ builder.append(script);
+ }
+ String country = locale.getISO3Country();
+ if (!country.isEmpty()) {
+ builder.append('_');
+ builder.append(country);
+ }
+ String localeName = builder.toString();
+ assert (!ICULocaleMap.containsKey(localeName));
+ ICULocaleMap.put(localeName, locale);
+ }
+ }
+ for (String localeName : ICULocaleMap.keySet()) {
+ String localeUppercase = localeName.toUpperCase();
+ assert (!ICULocaleMapUppercase.containsKey(localeUppercase));
+ ICULocaleMapUppercase.put(localeUppercase, localeName);
+ }
+ ICULocaleNames = ICULocaleMap.keySet().toArray(new String[0]);
+ Arrays.sort(ICULocaleNames);
+ assert (ICULocaleNames.length <= (1 << 16));
+ for (int i = 0; i < ICULocaleNames.length; i++) {
+ ICULocaleToId.put(ICULocaleNames[i], i);
+ }
Review Comment:
I would add a comment here that the locale names are unique, and we have enforced that above with the `assert` when creating the `ICULocaleMap`.
Also do we care if the order is always the same? I believe not, but just checking.
##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -30,31 +33,107 @@ import org.apache.spark.unsafe.types.UTF8String.{fromString => toUTF8}
class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ignore funsuite
test("collationId stability") {
- val utf8Binary = fetchCollation(0)
+ assert(UTF8_BINARY_COLLATION_ID == 0)
Review Comment:
Could we please have a similar assertion for `UTF8_BINARY_LCASE_COLLATION_ID`?
I think this ID should be stable as well.
##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -152,4 +219,147 @@ class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ig
}
})
}
+
+ test("test collation caching") {
+ Seq(
+ "UTF8_BINARY",
+ "UTF8_BINARY_LCASE",
+ "UTF8_BINARY_UCASE",
+ "UNICODE",
+ "UNICODE_LCASE",
+ "UNICODE_UCASE",
+ "UNICODE_CI",
+ "UNICODE_AI_CI",
+ "UNICODE_AI_CI_LCASE",
+ "UNICODE_AI_CI_UCASE"
+ ).foreach(collationId => {
+ val col1 = fetchCollation(collationId)
+ val col2 = fetchCollation(collationId)
+ assert(col1 eq col2) // reference equality
+ })
+ }
+
+ test("collations with ICU non-root localization") {
+ Seq(
+ // language only
+ "en",
+ "en_CS",
+ "en_CI",
+ "en_AS",
+ "en_AI",
+ "en_LCASE",
+ "en_UCASE",
+ // language + 3-letter country code
+ "en_USA",
+ "en_USA_CS",
+ "en_USA_CI",
+ "en_USA_AS",
+ "en_USA_AI",
+ "en_USA_LCASE",
+ "en_USA_UCASE",
+ // language + script code
+ "sr_Cyrl",
+ "sr_Cyrl_CS",
+ "sr_Cyrl_CI",
+ "sr_Cyrl_AS",
+ "sr_Cyrl_AI",
+ "sr_Cyrl_LCASE",
+ "sr_Cyrl_UCASE",
+ // language + script code + 3-letter country code
+ "sr_Cyrl_SRB",
+ "sr_Cyrl_SRB_CS",
+ "sr_Cyrl_SRB_CI",
+ "sr_Cyrl_SRB_AS",
+ "sr_Cyrl_SRB_AI",
+ "sr_Cyrl_SRB_LCASE",
+ "sr_Cyrl_SRB_UCASE"
+ ).foreach(collationICU => {
+ val col = fetchCollation(collationICU)
+ assert(col.collator.getLocale(ULocale.VALID_LOCALE) != ULocale.ROOT)
+ })
+ }
+
+ test("invalid names of collations with ICU non-root localization") {
+ Seq(
+ "en_US", // must use 3-letter country code
+ "enn",
+ "en_AAA",
+ "en_Something",
+ "en_Something_USA",
+ "en_Latn_USA", // use en_USA instead
+ "en_Cyrl_USA",
+ "en_USA_AAA",
+ "sr_Cyrl_SRB_AAA"
+ ).foreach(collationName => {
+ val error = intercept[SparkException] {
+ fetchCollation(collationName)
+ }
+
+ assert(error.getErrorClass === "COLLATION_INVALID_NAME")
+ assert(error.getMessageParameters.asScala === Map("collationName" -> collationName))
+ })
+ }
+
+ test("collations name normalization for ICU non-root localization") {
+ Seq(
+ ("en_USA", "en_USA"),
+ ("en_CS", "en"),
+ ("en_AS", "en"),
+ ("en_CS_AS", "en"),
+ ("en_AI_CI", "en_CI_AI"),
+ ("en_USA_AI_CI", "en_USA_CI_AI"),
+ // randomized case
+ ("EN_USA", "en_USA"),
+ ("eN_usA_ci_uCASe_aI", "en_USA_CI_AI_UCASE"),
+ ("SR_CYRL", "sr_Cyrl"),
+ ("sr_cyrl_srb", "sr_Cyrl_SRB"),
+ ("sR_cYRl_sRb", "sr_Cyrl_SRB")
+ ).foreach {
+ case (name, normalized) =>
+ val col = fetchCollation(name)
+ assert(col.collationName == normalized)
+ }
+ }
+
+ test("invalid collationId") {
+ val badCollationIds = Seq(
+ -1, // user-defined collation range
Review Comment:
Why don't we make this a named constant that is int and be done with it?
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
Review Comment:
Please specify that bit 0 is the least significant bit in this numbering.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -772,12 +772,17 @@ object SQLConf {
" produced by a builtin function such as to_char or CAST")
.version("4.0.0")
.stringConf
- .checkValue(CollationFactory.isValidCollation,
+ .checkValue(
+ collationName => {
+ try {
+ CollationFactory.fetchCollation(collationName)
+ true
+ } catch {
+ case e: SparkException if e.getErrorClass == "COLLATION_INVALID_NAME" => false
+ }
+ },
Review Comment:
Why are we making this change as well? Why not keep the `isValidCollation` predicate?
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
Review Comment:
I think a safer approach would be to have a predefined BIG upper bound (probably through a SQL config) and throw an error if we go beyond this upper bound, or just instantiate the collation on the fly in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1593130617
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,438 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
Review Comment:
`ConcurrentHashMap` is used to enforce thread safety of static `fetchCollation` function. Added shortcut for `UTF8_BINARY` collation to skip using the map cache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "dbatomic (via GitHub)" <gi...@apache.org>.
dbatomic commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1592436706
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,438 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid implementation provider");
Review Comment:
Can you just add in error message that this has something to do with collations?
e.g. "Invalid collation implementation provider" so it's easier to root cause if we ever hit this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "mkaravel (via GitHub)" <gi...@apache.org>.
mkaravel commented on PR #46180:
URL: https://github.com/apache/spark/pull/46180#issuecomment-2101713318
> > User can use collation specifiers in any order except of locale which is mandatory and must go first. There is a one-to-one mapping between collation ids and collation names defined in CollationFactory.
>
> Does that mean that there are multiple collation names that have identical meaning but different names (with the specifiers in different order)? That seems confusing and error prone. Are we at least normalizing them internally somehow?
+1 on @bart-samwel 's concern. Is there a good reason to allow the order to be relaxed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "mkaravel (via GitHub)" <gi...@apache.org>.
mkaravel commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1605444475
##########
connector/connect/common/src/main/protobuf/spark/connect/types.proto:
##########
@@ -101,7 +101,7 @@ message DataType {
message String {
uint32 type_variation_reference = 1;
- uint32 collation_id = 2;
+ string collation = 2;
Review Comment:
It took me a bit, but I get it now. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606746841
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,438 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ public static int collationNameToId(String originalCollationName) throws SparkException {
+ String collationName = originalCollationName.toUpperCase();
+ try {
+ if (collationName.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName);
+ }
+ } catch (SparkException e) {
+ throw new SparkException(
+ "COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", originalCollationName)),
+ e);
+ }
+ }
+
+ protected static int parseSpecifiers(String specString) throws SparkException {
+ int specifiers = 0;
+ String[] parts = specString.split("_");
+ for (String part : parts) {
+ if (!part.isEmpty()) {
+ if (part.equals("UNSPECIFIED") || part.equals("INDETERMINATE")) {
+ throw new SparkException(part + " collation specifier reserved for internal use");
Review Comment:
Added tests to cover such problematic edge cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606729578
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected static int parseSpecifiers(
+ String originalName, String collationName, int splitStart) throws SparkException {
+ int specifiers = 0;
+ String[] parts = collationName.substring(splitStart).split("_");
+ for (String part : parts) {
+ if (!part.isEmpty()) {
+ if (part.equals("UNSPECIFIED") || part.equals("INDETERMINATE")) {
+ throw collationInvalidNameException(originalName);
+ } else if (Arrays.stream(CaseSensitivity.values()).anyMatch(
+ (s) -> s.toString().equals(part))) {
+ specifiers |=
+ CaseSensitivity.valueOf(part).ordinal() << CASE_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(AccentSensitivity.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ AccentSensitivity.valueOf(part).ordinal() << ACCENT_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(CaseConversion.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ CaseConversion.valueOf(part).ordinal() << CASE_CONVERSION_OFFSET;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+ }
+ return specifiers;
+ }
+
+ protected abstract int getCollationId();
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ public static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.LCASE).getCollationId();
+ public static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).buildCollation();
+
+ private CollationSpecUTF8Binary(CaseConversion caseConversion) {
+ super(null, CaseSensitivity.CS, AccentSensitivity.AS, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int collationId = 0;
+ int specifiers = CollationSpec.parseSpecifiers(
+ originalName, collationName, "UTF8_BINARY".length());
+ if ((specifiers & ~(CASE_CONVERSION_MASK << CASE_CONVERSION_OFFSET)) != 0) {
+ throw collationInvalidNameException(originalName);
+ }
+ collationId |= specifiers;
+ return collationId;
+ }
+
+ @Override
+ protected int getCollationId() {
+ int collationId = 0;
+ collationId |= caseConversion.ordinal() << CASE_CONVERSION_OFFSET;
+ return collationId;
+ }
+
+ public static CollationSpecUTF8Binary fromCollationId(int collationId)
+ throws SparkException {
+ int originalCollationId = collationId;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
+ if (collationId != 0 || caseConversionOrdinal >= CaseConversion.values().length) {
+ throw SparkException.internalError("Invalid UTF8_BINARY collation id " +
+ originalCollationId);
+ } else {
+ CaseConversion caseConversion = CaseConversion.values()[caseConversionOrdinal];
+ return new CollationSpecUTF8Binary(caseConversion);
+ }
+ }
+
+ @Override
+ protected Collation buildCollation() {
+ Comparator<UTF8String> comparator;
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ comparator = UTF8String::binaryCompare;
+ } else if (collationId == UTF8_BINARY_LCASE_COLLATION_ID) {
+ comparator = UTF8String::compareLowerCase;
+ } else {
+ comparator = (s1, s2) -> {
+ UTF8String convertedS1 = caseConversion(s1);
+ UTF8String convertedS2 = caseConversion(s2);
+ return convertedS1.binaryCompare(convertedS2);
+ };
+ }
+ return new Collation(
+ collationName(),
+ null,
+ comparator,
+ "1.0",
+ s -> (long) caseConversion(s).hashCode(),
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_COLLATION_ID,
+ collationId == UTF8_BINARY_LCASE_COLLATION_ID
+ );
+ }
+
+ private UTF8String caseConversion(UTF8String s) {
+ if (caseConversion == CaseConversion.LCASE) {
+ return s.toLowerCase();
+ } else if (caseConversion == CaseConversion.UCASE) {
+ return s.toUpperCase();
+ } else {
+ return s;
+ }
+ }
+
+ private String collationName() {
+ StringBuilder builder = new StringBuilder();
+ builder.append("UTF8_BINARY");
+ if (caseConversion != CaseConversion.UNSPECIFIED) {
+ builder.append('_');
+ builder.append(caseConversion.toString());
+ }
+ return builder.toString();
+ }
+ }
+
+ private static class CollationSpecICU extends CollationSpec {
+
+ private static final String[] ICULocaleNames;
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+ private static final String ICUCollatorVersion = "153.120.0.0";
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
+ for (ULocale locale : locales) {
+ if (locale.getVariant().isEmpty()) {
+ String language = locale.getLanguage();
+ assert (!language.isEmpty());
+ StringBuilder builder = new StringBuilder(language);
+ String script = locale.getScript();
+ if (!script.isEmpty()) {
+ builder.append('_');
+ builder.append(script);
+ }
+ String country = locale.getISO3Country();
+ if (!country.isEmpty()) {
+ builder.append('_');
+ builder.append(country);
+ }
+ String localeName = builder.toString();
+ assert (!ICULocaleMap.containsKey(localeName));
+ ICULocaleMap.put(localeName, locale);
+ }
+ }
+ for (String localeName : ICULocaleMap.keySet()) {
+ String localeUppercase = localeName.toUpperCase();
+ assert (!ICULocaleMapUppercase.containsKey(localeUppercase));
+ ICULocaleMapUppercase.put(localeUppercase, localeName);
+ }
+ ICULocaleNames = ICULocaleMap.keySet().toArray(new String[0]);
+ Arrays.sort(ICULocaleNames);
+ assert (ICULocaleNames.length <= (1 << 16));
+ for (int i = 0; i < ICULocaleNames.length; i++) {
+ ICULocaleToId.put(ICULocaleNames[i], i);
+ }
Review Comment:
Previous comment is outdated.
Resolving this conversation, clarified locale ids with comments around static members used in this static constructor and documented locale name - id mapping with golden file as well. New goal of `UTF8_BINARY` having collation id 0 and `UTF8_BINARY_LCASE` having collation id 1 is met with new changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606740054
##########
common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala:
##########
@@ -30,31 +33,107 @@ import org.apache.spark.unsafe.types.UTF8String.{fromString => toUTF8}
class CollationFactorySuite extends AnyFunSuite with Matchers { // scalastyle:ignore funsuite
test("collationId stability") {
- val utf8Binary = fetchCollation(0)
+ assert(UTF8_BINARY_COLLATION_ID == 0)
+ assert(INDETERMINATE_COLLATION_ID == 1 << 30)
+
+ val utf8Binary = fetchCollation(UTF8_BINARY_COLLATION_ID)
assert(utf8Binary.collationName == "UTF8_BINARY")
assert(utf8Binary.supportsBinaryEquality)
- val utf8BinaryLcase = fetchCollation(1)
+ val utf8BinaryLcase = fetchCollation(UTF8_BINARY_LCASE_COLLATION_ID)
assert(utf8BinaryLcase.collationName == "UTF8_BINARY_LCASE")
assert(!utf8BinaryLcase.supportsBinaryEquality)
- val unicode = fetchCollation(2)
+ val unicode = fetchCollation(UNICODE_COLLATION_ID)
assert(unicode.collationName == "UNICODE")
- assert(unicode.supportsBinaryEquality);
+ assert(unicode.supportsBinaryEquality)
- val unicodeCi = fetchCollation(3)
+ val unicodeCi = fetchCollation(UNICODE_CI_COLLATION_ID)
Review Comment:
Added assertions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606959835
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -118,76 +119,433 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * Collation id is defined as 32-bit integer.
+ * We specify binary layouts for different classes of collations.
+ * Classes of collations are differentiated by most significant 3 bits (bit 31, 30 and 29),
+ * bit 31 being most significant and bit 0 being least significant.
+ * ---
+ * INDETERMINATE collation id binary layout:
+ * bit 31-0: 1
+ * INDETERMINATE collation id is equal to -1
+ * ---
+ * user-defined collation id binary layout:
+ * bit 31: 0
+ * bit 30: 1
+ * bit 29-0: undefined, reserved for future use
+ * ---
+ * UTF8_BINARY collation id binary layout:
+ * bit 31-22: zeroes
+ * bit 21-18: zeroes, reserved for space trimming
+ * bit 17-16: zeroes, reserved for version
+ * bit 15-3: zeroes
+ * bit 2: 0, reserved for accent sensitivity
+ * bit 1: 0, reserved for uppercase and case-insensitive
+ * bit 0: 0 = case-sensitive, 1 = lowercase
+ * ---
+ * ICU collation id binary layout:
+ * bit 31-30: zeroes
+ * bit 29: 1
+ * bit 28-24: zeroes
+ * bit 23-22: zeroes, reserved for version
+ * bit 21-18: zeroes, reserved for space trimming
+ * bit 17: 0 = case-sensitive, 1 = case-insensitive
+ * bit 16: 0 = accent-sensitive, 1 = accent-insensitive
+ * bit 15-14: zeroes, reserved for punctuation sensitivity
+ * bit 13-12: zeroes, reserved for first letter preference
+ * bit 11-0: locale id as specified in `ICULocaleToId` mapping
+ * ---
+ * Some illustrative examples of collation name to id mapping:
+ * - UTF8_BINARY -> 0
+ * - UTF8_BINARY_LCASE -> 1
+ * - UNICODE -> 0x20000000
+ * - UNICODE_AI -> 0x20010000
+ * - UNICODE_CI -> 0x20020000
+ * - UNICODE_CI_AI -> 0x20030000
+ * - af -> 0x20000001
+ * - af_CI_AI -> 0x20030001
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+
+ private enum DefinitionOrigin {
Review Comment:
Added multiple clarifications across `CollationFactory`, resolving this. If we need more clarification, let's discuss the individual problematic points in the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606708799
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,438 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
Review Comment:
Outdated. We don't throw internal errors anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606722772
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 20-19: zeroes, reserved for space trimming
+ * bit 18-17: zeroes, reserved for version
+ * bit 16-12: zeroes
+ * bit 11-0: zeroes for utf8-binary / locale id for ICU
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
+ private abstract static class CollationSpec {
+ protected enum ImplementationProvider {
+ UTF8_BINARY, ICU, INDETERMINATE
+ }
+
+ protected enum CaseSensitivity {
+ CS, CI
+ }
+
+ protected enum AccentSensitivity {
+ AS, AI
+ }
+
+ protected enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ protected static final int IMPLEMENTATION_PROVIDER_OFFSET = 29;
+ protected static final int IMPLEMENTATION_PROVIDER_MASK = 0b11;
+ protected static final int CASE_SENSITIVITY_OFFSET = 28;
+ protected static final int CASE_SENSITIVITY_MASK = 0b1;
+ protected static final int ACCENT_SENSITIVITY_OFFSET = 27;
+ protected static final int ACCENT_SENSITIVITY_MASK = 0b1;
+ protected static final int CASE_CONVERSION_OFFSET = 21;
+ protected static final int CASE_CONVERSION_MASK = 0b11;
+ protected static final int LOCALE_OFFSET = 0;
+ protected static final int LOCALE_MASK = 0x0FFF;
+
+ protected static final int INDETERMINATE_COLLATION_ID =
+ ImplementationProvider.INDETERMINATE.ordinal() << IMPLEMENTATION_PROVIDER_OFFSET;
+
+ protected final CaseSensitivity caseSensitivity;
+ protected final AccentSensitivity accentSensitivity;
+ protected final CaseConversion caseConversion;
+ protected final String locale;
+ protected final int collationId;
+
+ protected CollationSpec(
+ String locale,
+ CaseSensitivity caseSensitivity,
+ AccentSensitivity accentSensitivity,
+ CaseConversion caseConversion) {
+ this.locale = locale;
+ this.caseSensitivity = caseSensitivity;
+ this.accentSensitivity = accentSensitivity;
+ this.caseConversion = caseConversion;
+ this.collationId = getCollationId();
+ }
+
+ private static final Map<Integer, Collation> collationMap = new ConcurrentHashMap<>();
+
+ public static Collation fetchCollation(int collationId) throws SparkException {
+ if (collationId == UTF8_BINARY_COLLATION_ID) {
+ return CollationSpecUTF8Binary.UTF8_BINARY_COLLATION;
+ } else if (collationMap.containsKey(collationId)) {
+ return collationMap.get(collationId);
+ } else {
+ CollationSpec spec;
+ int implementationProviderOrdinal =
+ (collationId >> IMPLEMENTATION_PROVIDER_OFFSET) & IMPLEMENTATION_PROVIDER_MASK;
+ if (implementationProviderOrdinal >= ImplementationProvider.values().length) {
+ throw SparkException.internalError("Invalid collation implementation provider");
+ } else {
+ ImplementationProvider implementationProvider = ImplementationProvider.values()[
+ implementationProviderOrdinal];
+ if (implementationProvider == ImplementationProvider.UTF8_BINARY) {
+ spec = CollationSpecUTF8Binary.fromCollationId(collationId);
+ } else if (implementationProvider == ImplementationProvider.ICU) {
+ spec = CollationSpecICU.fromCollationId(collationId);
+ } else {
+ throw SparkException.internalError("Cannot instantiate indeterminate collation");
+ }
+ Collation collation = spec.buildCollation();
+ collationMap.put(collationId, collation);
+ return collation;
+ }
+ }
+ }
+
+ protected static SparkException collationInvalidNameException(String collationName) {
+ return new SparkException("COLLATION_INVALID_NAME",
+ SparkException.constructMessageParams(Map.of("collationName", collationName)), null);
+ }
+
+ public static int collationNameToId(String collationName) throws SparkException {
+ String collationNameUpper = collationName.toUpperCase();
+ if (collationNameUpper.startsWith("UTF8_BINARY")) {
+ return CollationSpecUTF8Binary.collationNameToId(collationName, collationNameUpper);
+ } else {
+ return CollationSpecICU.collationNameToId(collationName, collationNameUpper);
+ }
+ }
+
+ protected static int parseSpecifiers(
+ String originalName, String collationName, int splitStart) throws SparkException {
+ int specifiers = 0;
+ String[] parts = collationName.substring(splitStart).split("_");
+ for (String part : parts) {
+ if (!part.isEmpty()) {
+ if (part.equals("UNSPECIFIED") || part.equals("INDETERMINATE")) {
+ throw collationInvalidNameException(originalName);
+ } else if (Arrays.stream(CaseSensitivity.values()).anyMatch(
+ (s) -> s.toString().equals(part))) {
+ specifiers |=
+ CaseSensitivity.valueOf(part).ordinal() << CASE_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(AccentSensitivity.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ AccentSensitivity.valueOf(part).ordinal() << ACCENT_SENSITIVITY_OFFSET;
+ } else if (Arrays.stream(CaseConversion.values()).anyMatch(
+ (s) -> part.equals(s.toString()))) {
+ specifiers |=
+ CaseConversion.valueOf(part).ordinal() << CASE_CONVERSION_OFFSET;
+ } else {
+ throw collationInvalidNameException(originalName);
+ }
+ }
+ }
+ return specifiers;
+ }
+
+ protected abstract int getCollationId();
+ protected abstract Collation buildCollation();
}
- }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private static class CollationSpecUTF8Binary extends CollationSpec {
+
+ public static final int UTF8_BINARY_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).getCollationId();
+ public static final int UTF8_BINARY_LCASE_COLLATION_ID =
+ new CollationSpecUTF8Binary(CaseConversion.LCASE).getCollationId();
+ public static Collation UTF8_BINARY_COLLATION =
+ new CollationSpecUTF8Binary(CaseConversion.UNSPECIFIED).buildCollation();
+
+ private CollationSpecUTF8Binary(CaseConversion caseConversion) {
+ super(null, CaseSensitivity.CS, AccentSensitivity.AS, caseConversion);
+ }
+
+ public static int collationNameToId(
+ String originalName, String collationName) throws SparkException {
+ int collationId = 0;
+ int specifiers = CollationSpec.parseSpecifiers(
+ originalName, collationName, "UTF8_BINARY".length());
+ if ((specifiers & ~(CASE_CONVERSION_MASK << CASE_CONVERSION_OFFSET)) != 0) {
+ throw collationInvalidNameException(originalName);
+ }
+ collationId |= specifiers;
+ return collationId;
+ }
+
+ @Override
+ protected int getCollationId() {
+ int collationId = 0;
+ collationId |= caseConversion.ordinal() << CASE_CONVERSION_OFFSET;
+ return collationId;
+ }
+
+ public static CollationSpecUTF8Binary fromCollationId(int collationId)
+ throws SparkException {
+ int originalCollationId = collationId;
+ int caseConversionOrdinal =
+ (collationId >> CASE_CONVERSION_OFFSET) & CASE_CONVERSION_MASK;
+ collationId ^= caseConversionOrdinal << CASE_CONVERSION_OFFSET;
Review Comment:
Moved all bitwise operations to utility class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1606717497
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +119,445 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30-29: 00 = utf8-binary, 01 = ICU, 10 = indeterminate (without spec implementation)
+ * bit 28: 0 for utf8-binary / 0 = case-sensitive, 1 = case-insensitive for ICU
+ * bit 27: 0 for utf8-binary / 0 = accent-sensitive, 1 = accent-insensitive for ICU
+ * bit 26-25: zeroes, reserved for punctuation sensitivity
+ * bit 24-23: zeroes, reserved for first letter preference
+ * bit 22-21: 00 = unspecified, 01 = to-lower, 10 = to-upper
Review Comment:
Outdated, removed support for lower/upper conversions with sole exception of `UTF8_BINARY_LCASE`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1612087300
##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/common/DataTypeProtoConverter.scala:
##########
@@ -177,7 +178,11 @@ object DataTypeProtoConverter {
case s: StringType =>
proto.DataType
.newBuilder()
- .setString(proto.DataType.String.newBuilder().setCollationId(s.collationId).build())
+ .setString(
+ proto.DataType.String
+ .newBuilder()
+ .setCollation(CollationFactory.fetchCollation(s.collationId).collationName)
Review Comment:
or is it specific to the python client?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]
Posted by "nikolamand-db (via GitHub)" <gi...@apache.org>.
nikolamand-db commented on PR #46180:
URL: https://github.com/apache/spark/pull/46180#issuecomment-2075286793
Please review collation team @dbatomic @stefankandic @uros-db @mihailom-db @stevomitric.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46841] Add collation support for ICU locales and collation specifiers [spark]
Posted by "dbatomic (via GitHub)" <gi...@apache.org>.
dbatomic commented on code in PR #46180:
URL: https://github.com/apache/spark/pull/46180#discussion_r1576141539
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##########
@@ -117,76 +118,490 @@ public Collation(
}
/**
- * Constructor with comparators that are inherited from the given collator.
+ * collation id (32-bit integer) layout:
+ * bit 31: 0 = predefined collation, 1 = user-defined collation
+ * bit 30: 0 = utf8-binary, 1 = ICU
+ * bit 29: 0 = case-sensitive, 1 = case-insensitive
+ * bit 28: 0 = accent-sensitive, 1 = accent-insensitive
+ * bit 27-26: 00 = unspecified, 01 = punctuation-sensitive, 10 = punctuation-insensitive
+ * bit 25-24: 00 = unspecified, 01 = first-lower, 10 = first-upper
+ * bit 23-22: 00 = unspecified, 01 = to-lower, 10 = to-upper
+ * bit 21-20: 00 = unspecified, 01 = trim-left, 10 = trim-right, 11 = trim-both
+ * bit 19-18: zeroes, reserved for version
+ * bit 17-16: zeroes
+ * bit 15-0: locale id for ICU collations / zeroes for utf8-binary
*/
- public Collation(
- String collationName,
- Collator collator,
- String version,
- boolean supportsBinaryEquality,
- boolean supportsBinaryOrdering,
- boolean supportsLowercaseEquality) {
- this(
- collationName,
- collator,
- (s1, s2) -> collator.compare(s1.toString(), s2.toString()),
- version,
- s -> (long)collator.getCollationKey(s.toString()).hashCode(),
- supportsBinaryEquality,
- supportsBinaryOrdering,
- supportsLowercaseEquality);
- }
- }
+ private static class CollationSpec {
+ private enum ImplementationProvider {
+ UTF8_BINARY, ICU
+ }
+
+ private enum CaseSensitivity {
+ CS, CI
+ }
+
+ private enum AccentSensitivity {
+ AS, AI
+ }
+
+ private enum PunctuationSensitivity {
+ UNSPECIFIED, PS, PI
+ }
- private static final Collation[] collationTable = new Collation[4];
- private static final HashMap<String, Integer> collationNameToIdMap = new HashMap<>();
-
- public static final int UTF8_BINARY_COLLATION_ID = 0;
- public static final int UTF8_BINARY_LCASE_COLLATION_ID = 1;
-
- static {
- // Binary comparison. This is the default collation.
- // No custom comparators will be used for this collation.
- // Instead, we rely on byte for byte comparison.
- collationTable[0] = new Collation(
- "UTF8_BINARY",
- null,
- UTF8String::binaryCompare,
- "1.0",
- s -> (long)s.hashCode(),
- true,
- true,
- false);
-
- // Case-insensitive UTF8 binary collation.
- // TODO: Do in place comparisons instead of creating new strings.
- collationTable[1] = new Collation(
- "UTF8_BINARY_LCASE",
- null,
- UTF8String::compareLowerCase,
- "1.0",
- (s) -> (long)s.toLowerCase().hashCode(),
- false,
- false,
- true);
-
- // UNICODE case sensitive comparison (ROOT locale, in ICU).
- collationTable[2] = new Collation(
- "UNICODE", Collator.getInstance(ULocale.ROOT), "153.120.0.0", true, false, false);
- collationTable[2].collator.setStrength(Collator.TERTIARY);
- collationTable[2].collator.freeze();
-
- // UNICODE case-insensitive comparison (ROOT locale, in ICU + Secondary strength).
- collationTable[3] = new Collation(
- "UNICODE_CI", Collator.getInstance(ULocale.ROOT), "153.120.0.0", false, false, false);
- collationTable[3].collator.setStrength(Collator.SECONDARY);
- collationTable[3].collator.freeze();
-
- for (int i = 0; i < collationTable.length; i++) {
- collationNameToIdMap.put(collationTable[i].collationName, i);
+ private enum FirstLetterPreference {
+ UNSPECIFIED, FU, FL
+ }
+
+ private enum CaseConversion {
+ UNSPECIFIED, LCASE, UCASE
+ }
+
+ private enum SpaceTrimming {
+ UNSPECIFIED, LTRIM, RTRIM, TRIM
+ }
+
+ private static final int implementationProviderOffset = 30;
+ private static final int implementationProviderLen = 1;
+ private static final int caseSensitivityOffset = 29;
+ private static final int caseSensitivityLen = 1;
+ private static final int accentSensitivityOffset = 28;
+ private static final int accentSensitivityLen = 1;
+ private static final int punctuationSensitivityOffset = 26;
+ private static final int punctuationSensitivityLen = 2;
+ private static final int firstLetterPreferenceOffset = 24;
+ private static final int firstLetterPreferenceLen = 2;
+ private static final int caseConversionOffset = 22;
+ private static final int caseConversionLen = 2;
+ private static final int spaceTrimmingOffset = 20;
+ private static final int spaceTrimmingLen = 2;
+ private static final int localeOffset = 0;
+ private static final int localeLen = 16;
+
+ private static final String[] ICULocaleNames;
+ private static final Map<String, ULocale> ICULocaleMap = new HashMap<>();
+ private static final Map<String, String> ICULocaleMapUppercase = new HashMap<>();
+ private static final Map<String, Integer> ICULocaleToId = new HashMap<>();
+
+ static {
+ ICULocaleMap.put("UNICODE", ULocale.ROOT);
+ ULocale[] locales = Collator.getAvailableULocales();
Review Comment:
`ICULocaleToId` array would need to be statically/compile time filled, right? i.e. we should have a separate array with list of all locales and a test that verifies that we covered all the locales in ICU?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org