[jira] [Created] (CALCITE-5003) MergeUnion on types with different collators produces wrong result

["Ruben Q L (Jira)" <ji...@apache.org>]

Ruben Q L created CALCITE-5003:
----------------------------------

             Summary: MergeUnion on types with different collators produces wrong result
                 Key: CALCITE-5003
                 URL: https://issues.apache.org/jira/browse/CALCITE-5003
             Project: Calcite
          Issue Type: Bug
          Components: core
    Affects Versions: 1.27.0
            Reporter: Ruben Q L


MergeUnion on types with different collators produces wrong result.

Problem can be reproduced with the following test (in {{EnumerableStringComparisonTest}}):
{code}
  @Test void testMergeUnionOnStringDifferentCollation() {
    tester()
        .query("?")
        .withHook(Hook.PLANNER, (Consumer<RelOptPlanner>) planner ->
            planner.removeRule(EnumerableRules.ENUMERABLE_UNION_RULE))
        .withRel(b -> {
          final RelBuilder builder = b.transform(c -> c.withSimplifyValues(false));
          return builder
              .values(builder.getTypeFactory().builder()
                      .add("name",
                          builder.getTypeFactory().createSqlType(SqlTypeName.VARCHAR)).build(),
                  "facilities", "HR", "administration", "Marketing")
              .values(createRecordVarcharSpecialCollation(builder),
                  "Marketing", "administration", "presales", "HR")
              .union(false)
              .sort(0)
              .build();
        })
        .explainHookMatches("" // It is important that we have MergeUnion in the plan
            + "EnumerableMergeUnion(all=[false])\n"
            + "  EnumerableSort(sort0=[$0], dir0=[ASC])\n"
            + "    EnumerableValues(tuples=[[{ 'facilities' }, { 'HR' }, { 'administration' }, { 'Marketing' }]])\n"
            + "  EnumerableSort(sort0=[$0], dir0=[ASC])\n"
            + "    EnumerableValues(tuples=[[{ 'Marketing' }, { 'administration' }, { 'presales' }, { 'HR' }]])\n")
        .returnsOrdered("name=administration\n"
            + "name=facilities\n"
            + "name=HR\n"
            + "name=Marketing\n"
            + "name=presales");
  }
{code}

which fails with:
{noformat}
java.lang.AssertionError: 
Expected: "name=administration\nname=facilities\nname=HR\nname=Marketing\nname=presales"
     but: was "name=administration\nname=HR\nname=Marketing\nname=administration\nname=facilities\nname=Marketing\nname=presales"
{noformat}

The problem is that, in case of different collators, the pre-requisite of the the MergeUnion (inputs sorted) is not fulfilled, since inputs are technically sorted, but not using the same sorting collator, so they are not comparable by the MergeUnion algorithm.

A possible solution could be not applying EnumerableMergeUnionRule in this case.
A more clever solution could be achieved if the rule pushes a Sort + Cast + input (and not just Sort + input) in case the input's key type differs collation-wise with the union's result type.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)