You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "KevinyhZou (via GitHub)" <gi...@apache.org> on 2024/08/15 03:45:41 UTC

[PR] [GLUTEN-6856][CH]Support arrays overlap [incubator-gluten]

KevinyhZou opened a new pull request, #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857

   ## What changes were proposed in this pull request?
   
   (Please fill in changes proposed in this fix)
   
   (Fixes: \#6856)
   
   ## How was this patch tested?
   
   test by spark ut
   
   
   (If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "KevinyhZou (via GitHub)" <gi...@apache.org>.
KevinyhZou commented on code in PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#discussion_r1767810732


##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";

Review Comment:
   CH already has a similar function `hasAny`, but it different from `arraysOverlap`, has so many corner cases, it will be very complex to convert arraysOverlap, as below
   ```
   parse arrays_overlap(arr1, arr2) as
                   if (isNull(arr1) || isNull(arr2))
                       return NULL
                   else if (isEmpty(arr1) || isEmpty(arr2))
                       return false;
                   else if (arr1.hasAny(arr2))
                   {
                       if (!arr1.has(NULL) || !arr2.has(NULL))
                           return true;
                       else if (arr1.intersect(arr2) != NULL)
                           return true
                       else
                           return NULL;
                   }
                   else if (arr1.has(NULL) || arr2.has(NULL))
                       return NULL;
                   else if (arr1.has(NaN) && arr2.has(NaN))
                       return true;
                   else
                       return false;
   ```
   
   and it is not easy to slove the case when array parameters contain `NaN`. So I think it is better to write a self-defined function. @taiyang-li 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "KevinyhZou (via GitHub)" <gi...@apache.org>.
KevinyhZou commented on code in PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#discussion_r1722607385


##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";

Review Comment:
   ok



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays overlap [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#issuecomment-2290509848

   https://github.com/apache/incubator-gluten/issues/6856


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "taiyang-li (via GitHub)" <gi...@apache.org>.
taiyang-li commented on code in PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#discussion_r1767903545


##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";
+    static FunctionPtr create(ContextPtr) { return std::make_shared<SparkFunctionArraysOverlap>(); }
+    SparkFunctionArraysOverlap() = default;
+    ~SparkFunctionArraysOverlap() override = default;
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+    size_t getNumberOfArguments() const override { return 2; }
+    String getName() const override { return name; }
+    bool useDefaultImplementationForNulls() const override { return false; }
+    bool useDefaultImplementationForConstants() const override { return false; }

Review Comment:
   什么clickhouse会替换const null? 没太理解 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#issuecomment-2304174434

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "KevinyhZou (via GitHub)" <gi...@apache.org>.
KevinyhZou commented on PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#issuecomment-2304177321

   The performance of `arrays_overlap` in gluten
   ![image](https://github.com/user-attachments/assets/215b1ef4-1df8-4dbc-b2a8-c039bec88b7b)
   
   
   The performance of `arrays_overlap` in spark
   ![image](https://github.com/user-attachments/assets/8de409a7-be9b-4821-8c60-07efbfd51001)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "KevinyhZou (via GitHub)" <gi...@apache.org>.
KevinyhZou commented on code in PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#discussion_r1767939298


##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";
+    static FunctionPtr create(ContextPtr) { return std::make_shared<SparkFunctionArraysOverlap>(); }
+    SparkFunctionArraysOverlap() = default;
+    ~SparkFunctionArraysOverlap() override = default;
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+    size_t getNumberOfArguments() const override { return 2; }
+    String getName() const override { return name; }
+    bool useDefaultImplementationForNulls() const override { return false; }
+    bool useDefaultImplementationForConstants() const override { return false; }
+
+    DB::DataTypePtr getReturnTypeImpl(const ColumnsWithTypeAndName &) const override
+    {
+        auto data_type = std::make_shared<DataTypeUInt8>();
+        return makeNullable(data_type);
+    }
+
+     ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t input_rows_count) const override
+     {
+        if (arguments.size() != 2)
+            throw Exception(ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH, "Function {} must have 2 arguments", getName());
+        
+        auto res = ColumnUInt8::create(input_rows_count, 0);
+        auto null_map = ColumnUInt8::create(input_rows_count, 0);
+        PaddedPODArray<UInt8> & res_data = res->getData();
+        PaddedPODArray<UInt8> & null_map_data = null_map->getData();
+        if (input_rows_count == 0)
+            return ColumnNullable::create(std::move(res), std::move(null_map));
+        
+        const ColumnArray * array_col_1 = nullptr,  * array_col_2 = nullptr;
+        const ColumnConst * const_col_1 = checkAndGetColumn<ColumnConst>(arguments[0].column.get());
+        const ColumnConst * const_col_2 = checkAndGetColumn<ColumnConst>(arguments[1].column.get());
+        if ((const_col_1 && const_col_1->onlyNull()) || (const_col_2 && const_col_2->onlyNull()))
+        {
+            null_map_data[0] = 1;
+            return ColumnNullable::create(std::move(res), std::move(null_map));
+        }
+        if (const_col_1)
+            array_col_1 = checkAndGetColumn<ColumnArray>(const_col_1->getDataColumnPtr().get());
+        else
+        {
+            const auto * null_col_1 = checkAndGetColumn<ColumnNullable>(arguments[0].column.get());
+            array_col_1 = checkAndGetColumn<ColumnArray>(null_col_1->getNestedColumnPtr().get());
+        }
+        if (const_col_2)
+            array_col_2 = checkAndGetColumn<ColumnArray>(const_col_2->getDataColumnPtr().get());
+        else
+        {
+            const auto * null_col_2 = checkAndGetColumn<ColumnNullable>(arguments[1].column.get());
+            array_col_2 = checkAndGetColumn<ColumnArray>(null_col_2->getNestedColumnPtr().get());
+        }
+        if (!array_col_1 || !array_col_2)
+            throw Exception(ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT, "Function {} 1st/2nd argument must be array type", getName());
+
+        const ColumnArray::Offsets & array_offsets_1 = array_col_1->getOffsets();
+        const ColumnArray::Offsets & array_offsets_2 = array_col_2->getOffsets();
+
+        size_t current_offset_1 = 0, current_offset_2 = 0;
+        size_t array_pos_1 = 0, array_pos_2 = 0;
+        for (size_t i = 0; i < array_col_1->size(); ++i)
+        {
+            if (arguments[0].column->isNullAt(i) || arguments[1].column->isNullAt(i))
+            {
+                null_map_data[i] = 1;
+                continue;
+            }
+            size_t array_size_1 = array_offsets_1[i] - current_offset_1;
+            size_t array_size_2 = array_offsets_2[i] - current_offset_2;
+            auto executeCompare = [&](const IColumn & col1, const IColumn & col2, const ColumnUInt8 * null_map1, const ColumnUInt8 * null_map2) -> void
+            {
+                for (size_t j = 0; j < array_size_1 && !res_data[i]; ++j)

Review Comment:
   HashSetWithStackMemory 可以用来存放FixedLength的数据(INT, FLOAT),通过调用getDataAt(i)。但是如果用来存放array(string) 这种不是FixedLength的情况,就会导致抛异常, 另外还要考虑到array中的复杂嵌套,会导致代码逻辑十分复杂。不是一个通用的方法



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "KevinyhZou (via GitHub)" <gi...@apache.org>.
KevinyhZou commented on code in PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#discussion_r1768232736


##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";
+    static FunctionPtr create(ContextPtr) { return std::make_shared<SparkFunctionArraysOverlap>(); }
+    SparkFunctionArraysOverlap() = default;
+    ~SparkFunctionArraysOverlap() override = default;
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+    size_t getNumberOfArguments() const override { return 2; }
+    String getName() const override { return name; }
+    bool useDefaultImplementationForNulls() const override { return false; }
+    bool useDefaultImplementationForConstants() const override { return false; }
+
+    DB::DataTypePtr getReturnTypeImpl(const ColumnsWithTypeAndName &) const override
+    {
+        auto data_type = std::make_shared<DataTypeUInt8>();
+        return makeNullable(data_type);
+    }
+
+     ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t input_rows_count) const override
+     {
+        if (arguments.size() != 2)
+            throw Exception(ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH, "Function {} must have 2 arguments", getName());
+        
+        auto res = ColumnUInt8::create(input_rows_count, 0);
+        auto null_map = ColumnUInt8::create(input_rows_count, 0);
+        PaddedPODArray<UInt8> & res_data = res->getData();
+        PaddedPODArray<UInt8> & null_map_data = null_map->getData();
+        if (input_rows_count == 0)
+            return ColumnNullable::create(std::move(res), std::move(null_map));
+        
+        const ColumnArray * array_col_1 = nullptr,  * array_col_2 = nullptr;
+        const ColumnConst * const_col_1 = checkAndGetColumn<ColumnConst>(arguments[0].column.get());
+        const ColumnConst * const_col_2 = checkAndGetColumn<ColumnConst>(arguments[1].column.get());
+        if ((const_col_1 && const_col_1->onlyNull()) || (const_col_2 && const_col_2->onlyNull()))
+        {
+            null_map_data[0] = 1;
+            return ColumnNullable::create(std::move(res), std::move(null_map));
+        }
+        if (const_col_1)
+            array_col_1 = checkAndGetColumn<ColumnArray>(const_col_1->getDataColumnPtr().get());
+        else
+        {
+            const auto * null_col_1 = checkAndGetColumn<ColumnNullable>(arguments[0].column.get());
+            array_col_1 = checkAndGetColumn<ColumnArray>(null_col_1->getNestedColumnPtr().get());
+        }
+        if (const_col_2)
+            array_col_2 = checkAndGetColumn<ColumnArray>(const_col_2->getDataColumnPtr().get());
+        else
+        {
+            const auto * null_col_2 = checkAndGetColumn<ColumnNullable>(arguments[1].column.get());
+            array_col_2 = checkAndGetColumn<ColumnArray>(null_col_2->getNestedColumnPtr().get());
+        }
+        if (!array_col_1 || !array_col_2)
+            throw Exception(ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT, "Function {} 1st/2nd argument must be array type", getName());
+
+        const ColumnArray::Offsets & array_offsets_1 = array_col_1->getOffsets();
+        const ColumnArray::Offsets & array_offsets_2 = array_col_2->getOffsets();
+
+        size_t current_offset_1 = 0, current_offset_2 = 0;
+        size_t array_pos_1 = 0, array_pos_2 = 0;
+        for (size_t i = 0; i < array_col_1->size(); ++i)
+        {
+            if (arguments[0].column->isNullAt(i) || arguments[1].column->isNullAt(i))
+            {
+                null_map_data[i] = 1;
+                continue;
+            }
+            size_t array_size_1 = array_offsets_1[i] - current_offset_1;
+            size_t array_size_2 = array_offsets_2[i] - current_offset_2;
+            auto executeCompare = [&](const IColumn & col1, const IColumn & col2, const ColumnUInt8 * null_map1, const ColumnUInt8 * null_map2) -> void
+            {
+                for (size_t j = 0; j < array_size_1 && !res_data[i]; ++j)

Review Comment:
   使用std::unordered_set 也是一样,对于复杂类似来说,需要判断是否有column 嵌套,然后再递归比较。相比较于两次for循环,可以直接使用ch 内置的column.compare 函数,直接在ch内部实现了复杂类型column的比较。使用set,也只是空间换时间。



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "taiyang-li (via GitHub)" <gi...@apache.org>.
taiyang-li commented on code in PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#discussion_r1767903264


##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";
+    static FunctionPtr create(ContextPtr) { return std::make_shared<SparkFunctionArraysOverlap>(); }
+    SparkFunctionArraysOverlap() = default;
+    ~SparkFunctionArraysOverlap() override = default;
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+    size_t getNumberOfArguments() const override { return 2; }
+    String getName() const override { return name; }
+    bool useDefaultImplementationForNulls() const override { return false; }

Review Comment:
   上面写的不是很清楚,设置成true的话,ch会处理arr1 = null or arr2 = null的逻辑, 不影响其他case. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "KevinyhZou (via GitHub)" <gi...@apache.org>.
KevinyhZou commented on code in PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#discussion_r1767939298


##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";
+    static FunctionPtr create(ContextPtr) { return std::make_shared<SparkFunctionArraysOverlap>(); }
+    SparkFunctionArraysOverlap() = default;
+    ~SparkFunctionArraysOverlap() override = default;
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+    size_t getNumberOfArguments() const override { return 2; }
+    String getName() const override { return name; }
+    bool useDefaultImplementationForNulls() const override { return false; }
+    bool useDefaultImplementationForConstants() const override { return false; }
+
+    DB::DataTypePtr getReturnTypeImpl(const ColumnsWithTypeAndName &) const override
+    {
+        auto data_type = std::make_shared<DataTypeUInt8>();
+        return makeNullable(data_type);
+    }
+
+     ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t input_rows_count) const override
+     {
+        if (arguments.size() != 2)
+            throw Exception(ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH, "Function {} must have 2 arguments", getName());
+        
+        auto res = ColumnUInt8::create(input_rows_count, 0);
+        auto null_map = ColumnUInt8::create(input_rows_count, 0);
+        PaddedPODArray<UInt8> & res_data = res->getData();
+        PaddedPODArray<UInt8> & null_map_data = null_map->getData();
+        if (input_rows_count == 0)
+            return ColumnNullable::create(std::move(res), std::move(null_map));
+        
+        const ColumnArray * array_col_1 = nullptr,  * array_col_2 = nullptr;
+        const ColumnConst * const_col_1 = checkAndGetColumn<ColumnConst>(arguments[0].column.get());
+        const ColumnConst * const_col_2 = checkAndGetColumn<ColumnConst>(arguments[1].column.get());
+        if ((const_col_1 && const_col_1->onlyNull()) || (const_col_2 && const_col_2->onlyNull()))
+        {
+            null_map_data[0] = 1;
+            return ColumnNullable::create(std::move(res), std::move(null_map));
+        }
+        if (const_col_1)
+            array_col_1 = checkAndGetColumn<ColumnArray>(const_col_1->getDataColumnPtr().get());
+        else
+        {
+            const auto * null_col_1 = checkAndGetColumn<ColumnNullable>(arguments[0].column.get());
+            array_col_1 = checkAndGetColumn<ColumnArray>(null_col_1->getNestedColumnPtr().get());
+        }
+        if (const_col_2)
+            array_col_2 = checkAndGetColumn<ColumnArray>(const_col_2->getDataColumnPtr().get());
+        else
+        {
+            const auto * null_col_2 = checkAndGetColumn<ColumnNullable>(arguments[1].column.get());
+            array_col_2 = checkAndGetColumn<ColumnArray>(null_col_2->getNestedColumnPtr().get());
+        }
+        if (!array_col_1 || !array_col_2)
+            throw Exception(ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT, "Function {} 1st/2nd argument must be array type", getName());
+
+        const ColumnArray::Offsets & array_offsets_1 = array_col_1->getOffsets();
+        const ColumnArray::Offsets & array_offsets_2 = array_col_2->getOffsets();
+
+        size_t current_offset_1 = 0, current_offset_2 = 0;
+        size_t array_pos_1 = 0, array_pos_2 = 0;
+        for (size_t i = 0; i < array_col_1->size(); ++i)
+        {
+            if (arguments[0].column->isNullAt(i) || arguments[1].column->isNullAt(i))
+            {
+                null_map_data[i] = 1;
+                continue;
+            }
+            size_t array_size_1 = array_offsets_1[i] - current_offset_1;
+            size_t array_size_2 = array_offsets_2[i] - current_offset_2;
+            auto executeCompare = [&](const IColumn & col1, const IColumn & col2, const ColumnUInt8 * null_map1, const ColumnUInt8 * null_map2) -> void
+            {
+                for (size_t j = 0; j < array_size_1 && !res_data[i]; ++j)

Review Comment:
    使用HashSetWithStackMemory,考虑到多层array,struct,map的复杂嵌套,会导致代码逻辑十分复杂,并且无法通过getDataAt(i) 从array(string) 的column中获取数据。不是一个通用的方法



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "taiyang-li (via GitHub)" <gi...@apache.org>.
taiyang-li commented on code in PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#discussion_r1722606167


##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";

Review Comment:
   can we contribute to upstream clickhouse first ? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#issuecomment-2313953407

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays overlap [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#issuecomment-2290510087

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#issuecomment-2298102631

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "taiyang-li (via GitHub)" <gi...@apache.org>.
taiyang-li commented on code in PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#discussion_r1767903545


##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";
+    static FunctionPtr create(ContextPtr) { return std::make_shared<SparkFunctionArraysOverlap>(); }
+    SparkFunctionArraysOverlap() = default;
+    ~SparkFunctionArraysOverlap() override = default;
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+    size_t getNumberOfArguments() const override { return 2; }
+    String getName() const override { return name; }
+    bool useDefaultImplementationForNulls() const override { return false; }
+    bool useDefaultImplementationForConstants() const override { return false; }

Review Comment:
   为什么clickhouse会替换const null? 没太理解 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#issuecomment-2363002676

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#issuecomment-2363189538

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "KevinyhZou (via GitHub)" <gi...@apache.org>.
KevinyhZou commented on code in PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#discussion_r1767939298


##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";
+    static FunctionPtr create(ContextPtr) { return std::make_shared<SparkFunctionArraysOverlap>(); }
+    SparkFunctionArraysOverlap() = default;
+    ~SparkFunctionArraysOverlap() override = default;
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+    size_t getNumberOfArguments() const override { return 2; }
+    String getName() const override { return name; }
+    bool useDefaultImplementationForNulls() const override { return false; }
+    bool useDefaultImplementationForConstants() const override { return false; }
+
+    DB::DataTypePtr getReturnTypeImpl(const ColumnsWithTypeAndName &) const override
+    {
+        auto data_type = std::make_shared<DataTypeUInt8>();
+        return makeNullable(data_type);
+    }
+
+     ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t input_rows_count) const override
+     {
+        if (arguments.size() != 2)
+            throw Exception(ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH, "Function {} must have 2 arguments", getName());
+        
+        auto res = ColumnUInt8::create(input_rows_count, 0);
+        auto null_map = ColumnUInt8::create(input_rows_count, 0);
+        PaddedPODArray<UInt8> & res_data = res->getData();
+        PaddedPODArray<UInt8> & null_map_data = null_map->getData();
+        if (input_rows_count == 0)
+            return ColumnNullable::create(std::move(res), std::move(null_map));
+        
+        const ColumnArray * array_col_1 = nullptr,  * array_col_2 = nullptr;
+        const ColumnConst * const_col_1 = checkAndGetColumn<ColumnConst>(arguments[0].column.get());
+        const ColumnConst * const_col_2 = checkAndGetColumn<ColumnConst>(arguments[1].column.get());
+        if ((const_col_1 && const_col_1->onlyNull()) || (const_col_2 && const_col_2->onlyNull()))
+        {
+            null_map_data[0] = 1;
+            return ColumnNullable::create(std::move(res), std::move(null_map));
+        }
+        if (const_col_1)
+            array_col_1 = checkAndGetColumn<ColumnArray>(const_col_1->getDataColumnPtr().get());
+        else
+        {
+            const auto * null_col_1 = checkAndGetColumn<ColumnNullable>(arguments[0].column.get());
+            array_col_1 = checkAndGetColumn<ColumnArray>(null_col_1->getNestedColumnPtr().get());
+        }
+        if (const_col_2)
+            array_col_2 = checkAndGetColumn<ColumnArray>(const_col_2->getDataColumnPtr().get());
+        else
+        {
+            const auto * null_col_2 = checkAndGetColumn<ColumnNullable>(arguments[1].column.get());
+            array_col_2 = checkAndGetColumn<ColumnArray>(null_col_2->getNestedColumnPtr().get());
+        }
+        if (!array_col_1 || !array_col_2)
+            throw Exception(ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT, "Function {} 1st/2nd argument must be array type", getName());
+
+        const ColumnArray::Offsets & array_offsets_1 = array_col_1->getOffsets();
+        const ColumnArray::Offsets & array_offsets_2 = array_col_2->getOffsets();
+
+        size_t current_offset_1 = 0, current_offset_2 = 0;
+        size_t array_pos_1 = 0, array_pos_2 = 0;
+        for (size_t i = 0; i < array_col_1->size(); ++i)
+        {
+            if (arguments[0].column->isNullAt(i) || arguments[1].column->isNullAt(i))
+            {
+                null_map_data[i] = 1;
+                continue;
+            }
+            size_t array_size_1 = array_offsets_1[i] - current_offset_1;
+            size_t array_size_2 = array_offsets_2[i] - current_offset_2;
+            auto executeCompare = [&](const IColumn & col1, const IColumn & col2, const ColumnUInt8 * null_map1, const ColumnUInt8 * null_map2) -> void
+            {
+                for (size_t j = 0; j < array_size_1 && !res_data[i]; ++j)

Review Comment:
   HashSetWithStackMemory 用来存放FixedLength的数据(INT, FLOAT),通过调用getDataAt(i)。但是如果用来存放string 这种不是FixedLength的情况,就会导致抛异常。不是一个通用的方法



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "taiyang-li (via GitHub)" <gi...@apache.org>.
taiyang-li commented on code in PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#discussion_r1767816482


##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";

Review Comment:
   What about array_join ? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "KevinyhZou (via GitHub)" <gi...@apache.org>.
KevinyhZou commented on code in PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#discussion_r1767939298


##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";
+    static FunctionPtr create(ContextPtr) { return std::make_shared<SparkFunctionArraysOverlap>(); }
+    SparkFunctionArraysOverlap() = default;
+    ~SparkFunctionArraysOverlap() override = default;
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+    size_t getNumberOfArguments() const override { return 2; }
+    String getName() const override { return name; }
+    bool useDefaultImplementationForNulls() const override { return false; }
+    bool useDefaultImplementationForConstants() const override { return false; }
+
+    DB::DataTypePtr getReturnTypeImpl(const ColumnsWithTypeAndName &) const override
+    {
+        auto data_type = std::make_shared<DataTypeUInt8>();
+        return makeNullable(data_type);
+    }
+
+     ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t input_rows_count) const override
+     {
+        if (arguments.size() != 2)
+            throw Exception(ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH, "Function {} must have 2 arguments", getName());
+        
+        auto res = ColumnUInt8::create(input_rows_count, 0);
+        auto null_map = ColumnUInt8::create(input_rows_count, 0);
+        PaddedPODArray<UInt8> & res_data = res->getData();
+        PaddedPODArray<UInt8> & null_map_data = null_map->getData();
+        if (input_rows_count == 0)
+            return ColumnNullable::create(std::move(res), std::move(null_map));
+        
+        const ColumnArray * array_col_1 = nullptr,  * array_col_2 = nullptr;
+        const ColumnConst * const_col_1 = checkAndGetColumn<ColumnConst>(arguments[0].column.get());
+        const ColumnConst * const_col_2 = checkAndGetColumn<ColumnConst>(arguments[1].column.get());
+        if ((const_col_1 && const_col_1->onlyNull()) || (const_col_2 && const_col_2->onlyNull()))
+        {
+            null_map_data[0] = 1;
+            return ColumnNullable::create(std::move(res), std::move(null_map));
+        }
+        if (const_col_1)
+            array_col_1 = checkAndGetColumn<ColumnArray>(const_col_1->getDataColumnPtr().get());
+        else
+        {
+            const auto * null_col_1 = checkAndGetColumn<ColumnNullable>(arguments[0].column.get());
+            array_col_1 = checkAndGetColumn<ColumnArray>(null_col_1->getNestedColumnPtr().get());
+        }
+        if (const_col_2)
+            array_col_2 = checkAndGetColumn<ColumnArray>(const_col_2->getDataColumnPtr().get());
+        else
+        {
+            const auto * null_col_2 = checkAndGetColumn<ColumnNullable>(arguments[1].column.get());
+            array_col_2 = checkAndGetColumn<ColumnArray>(null_col_2->getNestedColumnPtr().get());
+        }
+        if (!array_col_1 || !array_col_2)
+            throw Exception(ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT, "Function {} 1st/2nd argument must be array type", getName());
+
+        const ColumnArray::Offsets & array_offsets_1 = array_col_1->getOffsets();
+        const ColumnArray::Offsets & array_offsets_2 = array_col_2->getOffsets();
+
+        size_t current_offset_1 = 0, current_offset_2 = 0;
+        size_t array_pos_1 = 0, array_pos_2 = 0;
+        for (size_t i = 0; i < array_col_1->size(); ++i)
+        {
+            if (arguments[0].column->isNullAt(i) || arguments[1].column->isNullAt(i))
+            {
+                null_map_data[i] = 1;
+                continue;
+            }
+            size_t array_size_1 = array_offsets_1[i] - current_offset_1;
+            size_t array_size_2 = array_offsets_2[i] - current_offset_2;
+            auto executeCompare = [&](const IColumn & col1, const IColumn & col2, const ColumnUInt8 * null_map1, const ColumnUInt8 * null_map2) -> void
+            {
+                for (size_t j = 0; j < array_size_1 && !res_data[i]; ++j)

Review Comment:
   HashSetWithStackMemory 可以用来存放FixedLength的数据(INT, FLOAT),通过调用getDataAt(i)。但是如果用来存放array(string) 这种不是FixedLength的情况,就会导致抛异常, 考虑到多层array,struct,map的复杂嵌套,会导致代码逻辑十分复杂。不是一个通用的方法



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#issuecomment-2359909965

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "KevinyhZou (via GitHub)" <gi...@apache.org>.
KevinyhZou commented on code in PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#discussion_r1767810732


##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";

Review Comment:
   CH has a similar function `hasAny`, but it different from `arraysOverlap`, has so many corner cases, it will be very complex to convert arraysOverlap, as below
   ```
   parse arrays_overlap(arr1, arr2) as
                   if (isNull(arr1) || isNull(arr2))
                       return NULL
                   else if (isEmpty(arr1) || isEmpty(arr2))
                       return false;
                   else if (arr1.hasAny(arr2))
                   {
                       if (!arr1.has(NULL) || !arr2.has(NULL))
                           return true;
                       else if (arr1.intersect(arr2) != NULL)
                           return true
                       else
                           return NULL;
                   }
                   else if (arr1.has(NULL) || arr2.has(NULL))
                       return NULL;
                   else if (arr1.has(NaN) && arr2.has(NaN))
                       return true;
                   else
                       return false;
   ```
   
   and it is not easy to slove the case when array parameters contain `NaN`. So I think it is better to write a self-defined function. @taiyang-li 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "taiyang-li (via GitHub)" <gi...@apache.org>.
taiyang-li commented on code in PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#discussion_r1722742349


##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";

Review Comment:
   we could also contribute array_join to upstream. CH has strict code review and tests and it could help improve our code quality. 



##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";

Review Comment:
   we could also contribute array_join to upstream. CH has strict code reviews and tests and it could help improve our code quality. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "taiyang-li (via GitHub)" <gi...@apache.org>.
taiyang-li merged PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "KevinyhZou (via GitHub)" <gi...@apache.org>.
KevinyhZou commented on code in PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#discussion_r1767939298


##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";
+    static FunctionPtr create(ContextPtr) { return std::make_shared<SparkFunctionArraysOverlap>(); }
+    SparkFunctionArraysOverlap() = default;
+    ~SparkFunctionArraysOverlap() override = default;
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+    size_t getNumberOfArguments() const override { return 2; }
+    String getName() const override { return name; }
+    bool useDefaultImplementationForNulls() const override { return false; }
+    bool useDefaultImplementationForConstants() const override { return false; }
+
+    DB::DataTypePtr getReturnTypeImpl(const ColumnsWithTypeAndName &) const override
+    {
+        auto data_type = std::make_shared<DataTypeUInt8>();
+        return makeNullable(data_type);
+    }
+
+     ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t input_rows_count) const override
+     {
+        if (arguments.size() != 2)
+            throw Exception(ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH, "Function {} must have 2 arguments", getName());
+        
+        auto res = ColumnUInt8::create(input_rows_count, 0);
+        auto null_map = ColumnUInt8::create(input_rows_count, 0);
+        PaddedPODArray<UInt8> & res_data = res->getData();
+        PaddedPODArray<UInt8> & null_map_data = null_map->getData();
+        if (input_rows_count == 0)
+            return ColumnNullable::create(std::move(res), std::move(null_map));
+        
+        const ColumnArray * array_col_1 = nullptr,  * array_col_2 = nullptr;
+        const ColumnConst * const_col_1 = checkAndGetColumn<ColumnConst>(arguments[0].column.get());
+        const ColumnConst * const_col_2 = checkAndGetColumn<ColumnConst>(arguments[1].column.get());
+        if ((const_col_1 && const_col_1->onlyNull()) || (const_col_2 && const_col_2->onlyNull()))
+        {
+            null_map_data[0] = 1;
+            return ColumnNullable::create(std::move(res), std::move(null_map));
+        }
+        if (const_col_1)
+            array_col_1 = checkAndGetColumn<ColumnArray>(const_col_1->getDataColumnPtr().get());
+        else
+        {
+            const auto * null_col_1 = checkAndGetColumn<ColumnNullable>(arguments[0].column.get());
+            array_col_1 = checkAndGetColumn<ColumnArray>(null_col_1->getNestedColumnPtr().get());
+        }
+        if (const_col_2)
+            array_col_2 = checkAndGetColumn<ColumnArray>(const_col_2->getDataColumnPtr().get());
+        else
+        {
+            const auto * null_col_2 = checkAndGetColumn<ColumnNullable>(arguments[1].column.get());
+            array_col_2 = checkAndGetColumn<ColumnArray>(null_col_2->getNestedColumnPtr().get());
+        }
+        if (!array_col_1 || !array_col_2)
+            throw Exception(ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT, "Function {} 1st/2nd argument must be array type", getName());
+
+        const ColumnArray::Offsets & array_offsets_1 = array_col_1->getOffsets();
+        const ColumnArray::Offsets & array_offsets_2 = array_col_2->getOffsets();
+
+        size_t current_offset_1 = 0, current_offset_2 = 0;
+        size_t array_pos_1 = 0, array_pos_2 = 0;
+        for (size_t i = 0; i < array_col_1->size(); ++i)
+        {
+            if (arguments[0].column->isNullAt(i) || arguments[1].column->isNullAt(i))
+            {
+                null_map_data[i] = 1;
+                continue;
+            }
+            size_t array_size_1 = array_offsets_1[i] - current_offset_1;
+            size_t array_size_2 = array_offsets_2[i] - current_offset_2;
+            auto executeCompare = [&](const IColumn & col1, const IColumn & col2, const ColumnUInt8 * null_map1, const ColumnUInt8 * null_map2) -> void
+            {
+                for (size_t j = 0; j < array_size_1 && !res_data[i]; ++j)

Review Comment:
   HashSetWithStackMemory 可以用来存放FixedLength的数据(INT, FLOAT),通过调用getDataAt(i)。但是如果用来存放array(string) 这种不是FixedLength的情况,就会导致抛异常, 另外还要考虑到多层array,struct,map的复杂嵌套,会导致代码逻辑十分复杂。不是一个通用的方法



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "taiyang-li (via GitHub)" <gi...@apache.org>.
taiyang-li commented on code in PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#discussion_r1768119524


##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";
+    static FunctionPtr create(ContextPtr) { return std::make_shared<SparkFunctionArraysOverlap>(); }
+    SparkFunctionArraysOverlap() = default;
+    ~SparkFunctionArraysOverlap() override = default;
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+    size_t getNumberOfArguments() const override { return 2; }
+    String getName() const override { return name; }
+    bool useDefaultImplementationForNulls() const override { return false; }
+    bool useDefaultImplementationForConstants() const override { return false; }
+
+    DB::DataTypePtr getReturnTypeImpl(const ColumnsWithTypeAndName &) const override
+    {
+        auto data_type = std::make_shared<DataTypeUInt8>();
+        return makeNullable(data_type);
+    }
+
+     ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t input_rows_count) const override
+     {
+        if (arguments.size() != 2)
+            throw Exception(ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH, "Function {} must have 2 arguments", getName());
+        
+        auto res = ColumnUInt8::create(input_rows_count, 0);
+        auto null_map = ColumnUInt8::create(input_rows_count, 0);
+        PaddedPODArray<UInt8> & res_data = res->getData();
+        PaddedPODArray<UInt8> & null_map_data = null_map->getData();
+        if (input_rows_count == 0)
+            return ColumnNullable::create(std::move(res), std::move(null_map));
+        
+        const ColumnArray * array_col_1 = nullptr,  * array_col_2 = nullptr;
+        const ColumnConst * const_col_1 = checkAndGetColumn<ColumnConst>(arguments[0].column.get());
+        const ColumnConst * const_col_2 = checkAndGetColumn<ColumnConst>(arguments[1].column.get());
+        if ((const_col_1 && const_col_1->onlyNull()) || (const_col_2 && const_col_2->onlyNull()))
+        {
+            null_map_data[0] = 1;
+            return ColumnNullable::create(std::move(res), std::move(null_map));
+        }
+        if (const_col_1)
+            array_col_1 = checkAndGetColumn<ColumnArray>(const_col_1->getDataColumnPtr().get());
+        else
+        {
+            const auto * null_col_1 = checkAndGetColumn<ColumnNullable>(arguments[0].column.get());
+            array_col_1 = checkAndGetColumn<ColumnArray>(null_col_1->getNestedColumnPtr().get());
+        }
+        if (const_col_2)
+            array_col_2 = checkAndGetColumn<ColumnArray>(const_col_2->getDataColumnPtr().get());
+        else
+        {
+            const auto * null_col_2 = checkAndGetColumn<ColumnNullable>(arguments[1].column.get());
+            array_col_2 = checkAndGetColumn<ColumnArray>(null_col_2->getNestedColumnPtr().get());
+        }
+        if (!array_col_1 || !array_col_2)
+            throw Exception(ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT, "Function {} 1st/2nd argument must be array type", getName());
+
+        const ColumnArray::Offsets & array_offsets_1 = array_col_1->getOffsets();
+        const ColumnArray::Offsets & array_offsets_2 = array_col_2->getOffsets();
+
+        size_t current_offset_1 = 0, current_offset_2 = 0;
+        size_t array_pos_1 = 0, array_pos_2 = 0;
+        for (size_t i = 0; i < array_col_1->size(); ++i)
+        {
+            if (arguments[0].column->isNullAt(i) || arguments[1].column->isNullAt(i))
+            {
+                null_map_data[i] = 1;
+                continue;
+            }
+            size_t array_size_1 = array_offsets_1[i] - current_offset_1;
+            size_t array_size_2 = array_offsets_2[i] - current_offset_2;
+            auto executeCompare = [&](const IColumn & col1, const IColumn & col2, const ColumnUInt8 * null_map1, const ColumnUInt8 * null_map2) -> void
+            {
+                for (size_t j = 0; j < array_size_1 && !res_data[i]; ++j)

Review Comment:
   只是举个例子,如果上面的set不行还可以用std::unordered_set 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "KevinyhZou (via GitHub)" <gi...@apache.org>.
KevinyhZou commented on code in PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#discussion_r1767939298


##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";
+    static FunctionPtr create(ContextPtr) { return std::make_shared<SparkFunctionArraysOverlap>(); }
+    SparkFunctionArraysOverlap() = default;
+    ~SparkFunctionArraysOverlap() override = default;
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+    size_t getNumberOfArguments() const override { return 2; }
+    String getName() const override { return name; }
+    bool useDefaultImplementationForNulls() const override { return false; }
+    bool useDefaultImplementationForConstants() const override { return false; }
+
+    DB::DataTypePtr getReturnTypeImpl(const ColumnsWithTypeAndName &) const override
+    {
+        auto data_type = std::make_shared<DataTypeUInt8>();
+        return makeNullable(data_type);
+    }
+
+     ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t input_rows_count) const override
+     {
+        if (arguments.size() != 2)
+            throw Exception(ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH, "Function {} must have 2 arguments", getName());
+        
+        auto res = ColumnUInt8::create(input_rows_count, 0);
+        auto null_map = ColumnUInt8::create(input_rows_count, 0);
+        PaddedPODArray<UInt8> & res_data = res->getData();
+        PaddedPODArray<UInt8> & null_map_data = null_map->getData();
+        if (input_rows_count == 0)
+            return ColumnNullable::create(std::move(res), std::move(null_map));
+        
+        const ColumnArray * array_col_1 = nullptr,  * array_col_2 = nullptr;
+        const ColumnConst * const_col_1 = checkAndGetColumn<ColumnConst>(arguments[0].column.get());
+        const ColumnConst * const_col_2 = checkAndGetColumn<ColumnConst>(arguments[1].column.get());
+        if ((const_col_1 && const_col_1->onlyNull()) || (const_col_2 && const_col_2->onlyNull()))
+        {
+            null_map_data[0] = 1;
+            return ColumnNullable::create(std::move(res), std::move(null_map));
+        }
+        if (const_col_1)
+            array_col_1 = checkAndGetColumn<ColumnArray>(const_col_1->getDataColumnPtr().get());
+        else
+        {
+            const auto * null_col_1 = checkAndGetColumn<ColumnNullable>(arguments[0].column.get());
+            array_col_1 = checkAndGetColumn<ColumnArray>(null_col_1->getNestedColumnPtr().get());
+        }
+        if (const_col_2)
+            array_col_2 = checkAndGetColumn<ColumnArray>(const_col_2->getDataColumnPtr().get());
+        else
+        {
+            const auto * null_col_2 = checkAndGetColumn<ColumnNullable>(arguments[1].column.get());
+            array_col_2 = checkAndGetColumn<ColumnArray>(null_col_2->getNestedColumnPtr().get());
+        }
+        if (!array_col_1 || !array_col_2)
+            throw Exception(ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT, "Function {} 1st/2nd argument must be array type", getName());
+
+        const ColumnArray::Offsets & array_offsets_1 = array_col_1->getOffsets();
+        const ColumnArray::Offsets & array_offsets_2 = array_col_2->getOffsets();
+
+        size_t current_offset_1 = 0, current_offset_2 = 0;
+        size_t array_pos_1 = 0, array_pos_2 = 0;
+        for (size_t i = 0; i < array_col_1->size(); ++i)
+        {
+            if (arguments[0].column->isNullAt(i) || arguments[1].column->isNullAt(i))
+            {
+                null_map_data[i] = 1;
+                continue;
+            }
+            size_t array_size_1 = array_offsets_1[i] - current_offset_1;
+            size_t array_size_2 = array_offsets_2[i] - current_offset_2;
+            auto executeCompare = [&](const IColumn & col1, const IColumn & col2, const ColumnUInt8 * null_map1, const ColumnUInt8 * null_map2) -> void
+            {
+                for (size_t j = 0; j < array_size_1 && !res_data[i]; ++j)

Review Comment:
    使用HashSetWithStackMemory,考虑到多层array,struct,map的复杂嵌套,会导致代码逻辑十分复杂,并且无法通过getDataAt(i) 从array<string> 的column中获取数据。不是一个通用的方法



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "KevinyhZou (via GitHub)" <gi...@apache.org>.
KevinyhZou commented on code in PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#discussion_r1768107134


##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";
+    static FunctionPtr create(ContextPtr) { return std::make_shared<SparkFunctionArraysOverlap>(); }
+    SparkFunctionArraysOverlap() = default;
+    ~SparkFunctionArraysOverlap() override = default;
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+    size_t getNumberOfArguments() const override { return 2; }
+    String getName() const override { return name; }
+    bool useDefaultImplementationForNulls() const override { return false; }
+    bool useDefaultImplementationForConstants() const override { return false; }
+
+    DB::DataTypePtr getReturnTypeImpl(const ColumnsWithTypeAndName &) const override
+    {
+        auto data_type = std::make_shared<DataTypeUInt8>();
+        return makeNullable(data_type);
+    }
+
+     ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t input_rows_count) const override
+     {
+        if (arguments.size() != 2)
+            throw Exception(ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH, "Function {} must have 2 arguments", getName());
+        
+        auto res = ColumnUInt8::create(input_rows_count, 0);
+        auto null_map = ColumnUInt8::create(input_rows_count, 0);
+        PaddedPODArray<UInt8> & res_data = res->getData();
+        PaddedPODArray<UInt8> & null_map_data = null_map->getData();
+        if (input_rows_count == 0)
+            return ColumnNullable::create(std::move(res), std::move(null_map));
+        
+        const ColumnArray * array_col_1 = nullptr,  * array_col_2 = nullptr;
+        const ColumnConst * const_col_1 = checkAndGetColumn<ColumnConst>(arguments[0].column.get());
+        const ColumnConst * const_col_2 = checkAndGetColumn<ColumnConst>(arguments[1].column.get());
+        if ((const_col_1 && const_col_1->onlyNull()) || (const_col_2 && const_col_2->onlyNull()))
+        {
+            null_map_data[0] = 1;
+            return ColumnNullable::create(std::move(res), std::move(null_map));
+        }
+        if (const_col_1)
+            array_col_1 = checkAndGetColumn<ColumnArray>(const_col_1->getDataColumnPtr().get());
+        else
+        {
+            const auto * null_col_1 = checkAndGetColumn<ColumnNullable>(arguments[0].column.get());
+            array_col_1 = checkAndGetColumn<ColumnArray>(null_col_1->getNestedColumnPtr().get());
+        }
+        if (const_col_2)
+            array_col_2 = checkAndGetColumn<ColumnArray>(const_col_2->getDataColumnPtr().get());
+        else
+        {
+            const auto * null_col_2 = checkAndGetColumn<ColumnNullable>(arguments[1].column.get());
+            array_col_2 = checkAndGetColumn<ColumnArray>(null_col_2->getNestedColumnPtr().get());
+        }
+        if (!array_col_1 || !array_col_2)
+            throw Exception(ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT, "Function {} 1st/2nd argument must be array type", getName());
+
+        const ColumnArray::Offsets & array_offsets_1 = array_col_1->getOffsets();
+        const ColumnArray::Offsets & array_offsets_2 = array_col_2->getOffsets();
+
+        size_t current_offset_1 = 0, current_offset_2 = 0;
+        size_t array_pos_1 = 0, array_pos_2 = 0;
+        for (size_t i = 0; i < array_col_1->size(); ++i)
+        {
+            if (arguments[0].column->isNullAt(i) || arguments[1].column->isNullAt(i))

Review Comment:
   此处是多余逻辑,已经删除



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#issuecomment-2292904056

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "KevinyhZou (via GitHub)" <gi...@apache.org>.
KevinyhZou commented on code in PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#discussion_r1767831363


##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";
+    static FunctionPtr create(ContextPtr) { return std::make_shared<SparkFunctionArraysOverlap>(); }
+    SparkFunctionArraysOverlap() = default;
+    ~SparkFunctionArraysOverlap() override = default;
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+    size_t getNumberOfArguments() const override { return 2; }
+    String getName() const override { return name; }
+    bool useDefaultImplementationForNulls() const override { return false; }
+    bool useDefaultImplementationForConstants() const override { return false; }

Review Comment:
   if set to true, the input maybe const null, and clickhouse will replace this value , which would lead to differrent result 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "KevinyhZou (via GitHub)" <gi...@apache.org>.
KevinyhZou commented on code in PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#discussion_r1768106812


##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";
+    static FunctionPtr create(ContextPtr) { return std::make_shared<SparkFunctionArraysOverlap>(); }
+    SparkFunctionArraysOverlap() = default;
+    ~SparkFunctionArraysOverlap() override = default;
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+    size_t getNumberOfArguments() const override { return 2; }
+    String getName() const override { return name; }
+    bool useDefaultImplementationForNulls() const override { return false; }

Review Comment:
   已经设置为true



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "KevinyhZou (via GitHub)" <gi...@apache.org>.
KevinyhZou commented on code in PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#discussion_r1768232736


##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";
+    static FunctionPtr create(ContextPtr) { return std::make_shared<SparkFunctionArraysOverlap>(); }
+    SparkFunctionArraysOverlap() = default;
+    ~SparkFunctionArraysOverlap() override = default;
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+    size_t getNumberOfArguments() const override { return 2; }
+    String getName() const override { return name; }
+    bool useDefaultImplementationForNulls() const override { return false; }
+    bool useDefaultImplementationForConstants() const override { return false; }
+
+    DB::DataTypePtr getReturnTypeImpl(const ColumnsWithTypeAndName &) const override
+    {
+        auto data_type = std::make_shared<DataTypeUInt8>();
+        return makeNullable(data_type);
+    }
+
+     ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t input_rows_count) const override
+     {
+        if (arguments.size() != 2)
+            throw Exception(ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH, "Function {} must have 2 arguments", getName());
+        
+        auto res = ColumnUInt8::create(input_rows_count, 0);
+        auto null_map = ColumnUInt8::create(input_rows_count, 0);
+        PaddedPODArray<UInt8> & res_data = res->getData();
+        PaddedPODArray<UInt8> & null_map_data = null_map->getData();
+        if (input_rows_count == 0)
+            return ColumnNullable::create(std::move(res), std::move(null_map));
+        
+        const ColumnArray * array_col_1 = nullptr,  * array_col_2 = nullptr;
+        const ColumnConst * const_col_1 = checkAndGetColumn<ColumnConst>(arguments[0].column.get());
+        const ColumnConst * const_col_2 = checkAndGetColumn<ColumnConst>(arguments[1].column.get());
+        if ((const_col_1 && const_col_1->onlyNull()) || (const_col_2 && const_col_2->onlyNull()))
+        {
+            null_map_data[0] = 1;
+            return ColumnNullable::create(std::move(res), std::move(null_map));
+        }
+        if (const_col_1)
+            array_col_1 = checkAndGetColumn<ColumnArray>(const_col_1->getDataColumnPtr().get());
+        else
+        {
+            const auto * null_col_1 = checkAndGetColumn<ColumnNullable>(arguments[0].column.get());
+            array_col_1 = checkAndGetColumn<ColumnArray>(null_col_1->getNestedColumnPtr().get());
+        }
+        if (const_col_2)
+            array_col_2 = checkAndGetColumn<ColumnArray>(const_col_2->getDataColumnPtr().get());
+        else
+        {
+            const auto * null_col_2 = checkAndGetColumn<ColumnNullable>(arguments[1].column.get());
+            array_col_2 = checkAndGetColumn<ColumnArray>(null_col_2->getNestedColumnPtr().get());
+        }
+        if (!array_col_1 || !array_col_2)
+            throw Exception(ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT, "Function {} 1st/2nd argument must be array type", getName());
+
+        const ColumnArray::Offsets & array_offsets_1 = array_col_1->getOffsets();
+        const ColumnArray::Offsets & array_offsets_2 = array_col_2->getOffsets();
+
+        size_t current_offset_1 = 0, current_offset_2 = 0;
+        size_t array_pos_1 = 0, array_pos_2 = 0;
+        for (size_t i = 0; i < array_col_1->size(); ++i)
+        {
+            if (arguments[0].column->isNullAt(i) || arguments[1].column->isNullAt(i))
+            {
+                null_map_data[i] = 1;
+                continue;
+            }
+            size_t array_size_1 = array_offsets_1[i] - current_offset_1;
+            size_t array_size_2 = array_offsets_2[i] - current_offset_2;
+            auto executeCompare = [&](const IColumn & col1, const IColumn & col2, const ColumnUInt8 * null_map1, const ColumnUInt8 * null_map2) -> void
+            {
+                for (size_t j = 0; j < array_size_1 && !res_data[i]; ++j)

Review Comment:
   使用std::unordered_set 也是一样,对于复杂类型来说,需要判断是否有column 嵌套,然后再递归比较。相比较于两次for循环,可以直接使用ch 内置的column.compare 函数,直接在ch内部实现了复杂类型column的比较。使用set,也只是空间换时间。



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#issuecomment-2363221877

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "KevinyhZou (via GitHub)" <gi...@apache.org>.
KevinyhZou commented on code in PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#discussion_r1767820290


##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";

Review Comment:
   ch has a same named function `array_join`, but its function different from this. and I'll make a contribute to CH.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#issuecomment-2359843293

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#issuecomment-2293320246

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "KevinyhZou (via GitHub)" <gi...@apache.org>.
KevinyhZou commented on code in PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#discussion_r1767939298


##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";
+    static FunctionPtr create(ContextPtr) { return std::make_shared<SparkFunctionArraysOverlap>(); }
+    SparkFunctionArraysOverlap() = default;
+    ~SparkFunctionArraysOverlap() override = default;
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+    size_t getNumberOfArguments() const override { return 2; }
+    String getName() const override { return name; }
+    bool useDefaultImplementationForNulls() const override { return false; }
+    bool useDefaultImplementationForConstants() const override { return false; }
+
+    DB::DataTypePtr getReturnTypeImpl(const ColumnsWithTypeAndName &) const override
+    {
+        auto data_type = std::make_shared<DataTypeUInt8>();
+        return makeNullable(data_type);
+    }
+
+     ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t input_rows_count) const override
+     {
+        if (arguments.size() != 2)
+            throw Exception(ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH, "Function {} must have 2 arguments", getName());
+        
+        auto res = ColumnUInt8::create(input_rows_count, 0);
+        auto null_map = ColumnUInt8::create(input_rows_count, 0);
+        PaddedPODArray<UInt8> & res_data = res->getData();
+        PaddedPODArray<UInt8> & null_map_data = null_map->getData();
+        if (input_rows_count == 0)
+            return ColumnNullable::create(std::move(res), std::move(null_map));
+        
+        const ColumnArray * array_col_1 = nullptr,  * array_col_2 = nullptr;
+        const ColumnConst * const_col_1 = checkAndGetColumn<ColumnConst>(arguments[0].column.get());
+        const ColumnConst * const_col_2 = checkAndGetColumn<ColumnConst>(arguments[1].column.get());
+        if ((const_col_1 && const_col_1->onlyNull()) || (const_col_2 && const_col_2->onlyNull()))
+        {
+            null_map_data[0] = 1;
+            return ColumnNullable::create(std::move(res), std::move(null_map));
+        }
+        if (const_col_1)
+            array_col_1 = checkAndGetColumn<ColumnArray>(const_col_1->getDataColumnPtr().get());
+        else
+        {
+            const auto * null_col_1 = checkAndGetColumn<ColumnNullable>(arguments[0].column.get());
+            array_col_1 = checkAndGetColumn<ColumnArray>(null_col_1->getNestedColumnPtr().get());
+        }
+        if (const_col_2)
+            array_col_2 = checkAndGetColumn<ColumnArray>(const_col_2->getDataColumnPtr().get());
+        else
+        {
+            const auto * null_col_2 = checkAndGetColumn<ColumnNullable>(arguments[1].column.get());
+            array_col_2 = checkAndGetColumn<ColumnArray>(null_col_2->getNestedColumnPtr().get());
+        }
+        if (!array_col_1 || !array_col_2)
+            throw Exception(ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT, "Function {} 1st/2nd argument must be array type", getName());
+
+        const ColumnArray::Offsets & array_offsets_1 = array_col_1->getOffsets();
+        const ColumnArray::Offsets & array_offsets_2 = array_col_2->getOffsets();
+
+        size_t current_offset_1 = 0, current_offset_2 = 0;
+        size_t array_pos_1 = 0, array_pos_2 = 0;
+        for (size_t i = 0; i < array_col_1->size(); ++i)
+        {
+            if (arguments[0].column->isNullAt(i) || arguments[1].column->isNullAt(i))
+            {
+                null_map_data[i] = 1;
+                continue;
+            }
+            size_t array_size_1 = array_offsets_1[i] - current_offset_1;
+            size_t array_size_2 = array_offsets_2[i] - current_offset_2;
+            auto executeCompare = [&](const IColumn & col1, const IColumn & col2, const ColumnUInt8 * null_map1, const ColumnUInt8 * null_map2) -> void
+            {
+                for (size_t j = 0; j < array_size_1 && !res_data[i]; ++j)

Review Comment:
   HashSetWithStackMemory 可以用来存放FixedLength的数据(INT, FLOAT),通过调用getDataAt(i)。但是如果用来存放string 这种不是FixedLength的情况,就会导致抛异常。不是一个通用的方法



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#issuecomment-2360582567

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "KevinyhZou (via GitHub)" <gi...@apache.org>.
KevinyhZou commented on code in PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#discussion_r1767832973


##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";
+    static FunctionPtr create(ContextPtr) { return std::make_shared<SparkFunctionArraysOverlap>(); }
+    SparkFunctionArraysOverlap() = default;
+    ~SparkFunctionArraysOverlap() override = default;
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+    size_t getNumberOfArguments() const override { return 2; }
+    String getName() const override { return name; }
+    bool useDefaultImplementationForNulls() const override { return false; }

Review Comment:
   如果设置为true,ch 自己处理null 相关的逻辑,会导致两边的结果不一样。arrays_overlap里面如果有一个array含有null,就需要返回null。而ch里面,会返回true,因为两边同时含有NULL。



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#issuecomment-2290515921

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#issuecomment-2360093505

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "KevinyhZou (via GitHub)" <gi...@apache.org>.
KevinyhZou commented on code in PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#discussion_r1768106297


##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";
+    static FunctionPtr create(ContextPtr) { return std::make_shared<SparkFunctionArraysOverlap>(); }
+    SparkFunctionArraysOverlap() = default;
+    ~SparkFunctionArraysOverlap() override = default;
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+    size_t getNumberOfArguments() const override { return 2; }
+    String getName() const override { return name; }
+    bool useDefaultImplementationForNulls() const override { return false; }
+    bool useDefaultImplementationForConstants() const override { return false; }

Review Comment:
   测试了下可以设置为true,代码已经修改。应该是之前的测试有问题



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#issuecomment-2362996095

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "taiyang-li (via GitHub)" <gi...@apache.org>.
taiyang-li commented on code in PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#discussion_r1767819022


##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";
+    static FunctionPtr create(ContextPtr) { return std::make_shared<SparkFunctionArraysOverlap>(); }
+    SparkFunctionArraysOverlap() = default;
+    ~SparkFunctionArraysOverlap() override = default;
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+    size_t getNumberOfArguments() const override { return 2; }
+    String getName() const override { return name; }
+    bool useDefaultImplementationForNulls() const override { return false; }
+    bool useDefaultImplementationForConstants() const override { return false; }

Review Comment:
   why is it false? if it is false, we need to handle the case that all input arguments are constant, which could be avoided by setting it to true. 



##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";
+    static FunctionPtr create(ContextPtr) { return std::make_shared<SparkFunctionArraysOverlap>(); }
+    SparkFunctionArraysOverlap() = default;
+    ~SparkFunctionArraysOverlap() override = default;
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+    size_t getNumberOfArguments() const override { return 2; }
+    String getName() const override { return name; }
+    bool useDefaultImplementationForNulls() const override { return false; }
+    bool useDefaultImplementationForConstants() const override { return false; }
+
+    DB::DataTypePtr getReturnTypeImpl(const ColumnsWithTypeAndName &) const override
+    {
+        auto data_type = std::make_shared<DataTypeUInt8>();
+        return makeNullable(data_type);
+    }
+
+     ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t input_rows_count) const override
+     {
+        if (arguments.size() != 2)
+            throw Exception(ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH, "Function {} must have 2 arguments", getName());
+        
+        auto res = ColumnUInt8::create(input_rows_count, 0);
+        auto null_map = ColumnUInt8::create(input_rows_count, 0);
+        PaddedPODArray<UInt8> & res_data = res->getData();
+        PaddedPODArray<UInt8> & null_map_data = null_map->getData();
+        if (input_rows_count == 0)
+            return ColumnNullable::create(std::move(res), std::move(null_map));
+        
+        const ColumnArray * array_col_1 = nullptr,  * array_col_2 = nullptr;
+        const ColumnConst * const_col_1 = checkAndGetColumn<ColumnConst>(arguments[0].column.get());
+        const ColumnConst * const_col_2 = checkAndGetColumn<ColumnConst>(arguments[1].column.get());
+        if ((const_col_1 && const_col_1->onlyNull()) || (const_col_2 && const_col_2->onlyNull()))
+        {
+            null_map_data[0] = 1;
+            return ColumnNullable::create(std::move(res), std::move(null_map));
+        }
+        if (const_col_1)
+            array_col_1 = checkAndGetColumn<ColumnArray>(const_col_1->getDataColumnPtr().get());
+        else
+        {
+            const auto * null_col_1 = checkAndGetColumn<ColumnNullable>(arguments[0].column.get());
+            array_col_1 = checkAndGetColumn<ColumnArray>(null_col_1->getNestedColumnPtr().get());
+        }
+        if (const_col_2)
+            array_col_2 = checkAndGetColumn<ColumnArray>(const_col_2->getDataColumnPtr().get());
+        else
+        {
+            const auto * null_col_2 = checkAndGetColumn<ColumnNullable>(arguments[1].column.get());
+            array_col_2 = checkAndGetColumn<ColumnArray>(null_col_2->getNestedColumnPtr().get());
+        }
+        if (!array_col_1 || !array_col_2)
+            throw Exception(ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT, "Function {} 1st/2nd argument must be array type", getName());
+
+        const ColumnArray::Offsets & array_offsets_1 = array_col_1->getOffsets();
+        const ColumnArray::Offsets & array_offsets_2 = array_col_2->getOffsets();
+
+        size_t current_offset_1 = 0, current_offset_2 = 0;
+        size_t array_pos_1 = 0, array_pos_2 = 0;
+        for (size_t i = 0; i < array_col_1->size(); ++i)
+        {
+            if (arguments[0].column->isNullAt(i) || arguments[1].column->isNullAt(i))
+            {
+                null_map_data[i] = 1;
+                continue;
+            }
+            size_t array_size_1 = array_offsets_1[i] - current_offset_1;
+            size_t array_size_2 = array_offsets_2[i] - current_offset_2;
+            auto executeCompare = [&](const IColumn & col1, const IColumn & col2, const ColumnUInt8 * null_map1, const ColumnUInt8 * null_map2) -> void
+            {
+                for (size_t j = 0; j < array_size_1 && !res_data[i]; ++j)

Review Comment:
   两个for循环性能会比较差,可以用这个数据结构:
   using Set = HashSetWithStackMemory<StringRef, StringRefHash, 4>;



##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";
+    static FunctionPtr create(ContextPtr) { return std::make_shared<SparkFunctionArraysOverlap>(); }
+    SparkFunctionArraysOverlap() = default;
+    ~SparkFunctionArraysOverlap() override = default;
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+    size_t getNumberOfArguments() const override { return 2; }
+    String getName() const override { return name; }
+    bool useDefaultImplementationForNulls() const override { return false; }
+    bool useDefaultImplementationForConstants() const override { return false; }
+
+    DB::DataTypePtr getReturnTypeImpl(const ColumnsWithTypeAndName &) const override
+    {
+        auto data_type = std::make_shared<DataTypeUInt8>();
+        return makeNullable(data_type);
+    }
+
+     ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t input_rows_count) const override
+     {
+        if (arguments.size() != 2)
+            throw Exception(ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH, "Function {} must have 2 arguments", getName());
+        
+        auto res = ColumnUInt8::create(input_rows_count, 0);
+        auto null_map = ColumnUInt8::create(input_rows_count, 0);
+        PaddedPODArray<UInt8> & res_data = res->getData();
+        PaddedPODArray<UInt8> & null_map_data = null_map->getData();
+        if (input_rows_count == 0)
+            return ColumnNullable::create(std::move(res), std::move(null_map));
+        
+        const ColumnArray * array_col_1 = nullptr,  * array_col_2 = nullptr;
+        const ColumnConst * const_col_1 = checkAndGetColumn<ColumnConst>(arguments[0].column.get());
+        const ColumnConst * const_col_2 = checkAndGetColumn<ColumnConst>(arguments[1].column.get());
+        if ((const_col_1 && const_col_1->onlyNull()) || (const_col_2 && const_col_2->onlyNull()))
+        {
+            null_map_data[0] = 1;
+            return ColumnNullable::create(std::move(res), std::move(null_map));
+        }
+        if (const_col_1)
+            array_col_1 = checkAndGetColumn<ColumnArray>(const_col_1->getDataColumnPtr().get());

Review Comment:
   const里包的有可能是nullable array或array, 两种情况都需要考虑



##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";
+    static FunctionPtr create(ContextPtr) { return std::make_shared<SparkFunctionArraysOverlap>(); }
+    SparkFunctionArraysOverlap() = default;
+    ~SparkFunctionArraysOverlap() override = default;
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+    size_t getNumberOfArguments() const override { return 2; }
+    String getName() const override { return name; }
+    bool useDefaultImplementationForNulls() const override { return false; }
+    bool useDefaultImplementationForConstants() const override { return false; }
+
+    DB::DataTypePtr getReturnTypeImpl(const ColumnsWithTypeAndName &) const override
+    {
+        auto data_type = std::make_shared<DataTypeUInt8>();
+        return makeNullable(data_type);
+    }
+
+     ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t input_rows_count) const override
+     {
+        if (arguments.size() != 2)
+            throw Exception(ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH, "Function {} must have 2 arguments", getName());
+        
+        auto res = ColumnUInt8::create(input_rows_count, 0);
+        auto null_map = ColumnUInt8::create(input_rows_count, 0);
+        PaddedPODArray<UInt8> & res_data = res->getData();
+        PaddedPODArray<UInt8> & null_map_data = null_map->getData();
+        if (input_rows_count == 0)
+            return ColumnNullable::create(std::move(res), std::move(null_map));
+        
+        const ColumnArray * array_col_1 = nullptr,  * array_col_2 = nullptr;
+        const ColumnConst * const_col_1 = checkAndGetColumn<ColumnConst>(arguments[0].column.get());
+        const ColumnConst * const_col_2 = checkAndGetColumn<ColumnConst>(arguments[1].column.get());
+        if ((const_col_1 && const_col_1->onlyNull()) || (const_col_2 && const_col_2->onlyNull()))
+        {
+            null_map_data[0] = 1;
+            return ColumnNullable::create(std::move(res), std::move(null_map));
+        }
+        if (const_col_1)
+            array_col_1 = checkAndGetColumn<ColumnArray>(const_col_1->getDataColumnPtr().get());
+        else
+        {
+            const auto * null_col_1 = checkAndGetColumn<ColumnNullable>(arguments[0].column.get());

Review Comment:
   arguments[0]也许是nullable array或array, 两种情况都需要考虑。arguments[1]也一样



##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";
+    static FunctionPtr create(ContextPtr) { return std::make_shared<SparkFunctionArraysOverlap>(); }
+    SparkFunctionArraysOverlap() = default;
+    ~SparkFunctionArraysOverlap() override = default;
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+    size_t getNumberOfArguments() const override { return 2; }
+    String getName() const override { return name; }
+    bool useDefaultImplementationForNulls() const override { return false; }

Review Comment:
   这里设置成false的话,就必须自己处理arr1[i] = null或arr2[i] = null的问题了,增加了很多不必要的麻烦



##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";
+    static FunctionPtr create(ContextPtr) { return std::make_shared<SparkFunctionArraysOverlap>(); }
+    SparkFunctionArraysOverlap() = default;
+    ~SparkFunctionArraysOverlap() override = default;
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+    size_t getNumberOfArguments() const override { return 2; }
+    String getName() const override { return name; }
+    bool useDefaultImplementationForNulls() const override { return false; }
+    bool useDefaultImplementationForConstants() const override { return false; }
+
+    DB::DataTypePtr getReturnTypeImpl(const ColumnsWithTypeAndName &) const override
+    {
+        auto data_type = std::make_shared<DataTypeUInt8>();
+        return makeNullable(data_type);
+    }
+
+     ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t input_rows_count) const override
+     {
+        if (arguments.size() != 2)
+            throw Exception(ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH, "Function {} must have 2 arguments", getName());
+        
+        auto res = ColumnUInt8::create(input_rows_count, 0);
+        auto null_map = ColumnUInt8::create(input_rows_count, 0);
+        PaddedPODArray<UInt8> & res_data = res->getData();
+        PaddedPODArray<UInt8> & null_map_data = null_map->getData();
+        if (input_rows_count == 0)
+            return ColumnNullable::create(std::move(res), std::move(null_map));
+        
+        const ColumnArray * array_col_1 = nullptr,  * array_col_2 = nullptr;
+        const ColumnConst * const_col_1 = checkAndGetColumn<ColumnConst>(arguments[0].column.get());
+        const ColumnConst * const_col_2 = checkAndGetColumn<ColumnConst>(arguments[1].column.get());
+        if ((const_col_1 && const_col_1->onlyNull()) || (const_col_2 && const_col_2->onlyNull()))
+        {
+            null_map_data[0] = 1;
+            return ColumnNullable::create(std::move(res), std::move(null_map));
+        }
+        if (const_col_1)
+            array_col_1 = checkAndGetColumn<ColumnArray>(const_col_1->getDataColumnPtr().get());
+        else
+        {
+            const auto * null_col_1 = checkAndGetColumn<ColumnNullable>(arguments[0].column.get());
+            array_col_1 = checkAndGetColumn<ColumnArray>(null_col_1->getNestedColumnPtr().get());
+        }
+        if (const_col_2)
+            array_col_2 = checkAndGetColumn<ColumnArray>(const_col_2->getDataColumnPtr().get());
+        else
+        {
+            const auto * null_col_2 = checkAndGetColumn<ColumnNullable>(arguments[1].column.get());
+            array_col_2 = checkAndGetColumn<ColumnArray>(null_col_2->getNestedColumnPtr().get());
+        }
+        if (!array_col_1 || !array_col_2)
+            throw Exception(ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT, "Function {} 1st/2nd argument must be array type", getName());
+
+        const ColumnArray::Offsets & array_offsets_1 = array_col_1->getOffsets();
+        const ColumnArray::Offsets & array_offsets_2 = array_col_2->getOffsets();
+
+        size_t current_offset_1 = 0, current_offset_2 = 0;
+        size_t array_pos_1 = 0, array_pos_2 = 0;
+        for (size_t i = 0; i < array_col_1->size(); ++i)
+        {
+            if (arguments[0].column->isNullAt(i) || arguments[1].column->isNullAt(i))

Review Comment:
   for循环中高频调用虚函数 isNullAt会影响性能。既然拿到了null_map_data这里为什么不用? 
   



##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";
+    static FunctionPtr create(ContextPtr) { return std::make_shared<SparkFunctionArraysOverlap>(); }
+    SparkFunctionArraysOverlap() = default;
+    ~SparkFunctionArraysOverlap() override = default;
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+    size_t getNumberOfArguments() const override { return 2; }
+    String getName() const override { return name; }
+    bool useDefaultImplementationForNulls() const override { return false; }
+    bool useDefaultImplementationForConstants() const override { return false; }
+
+    DB::DataTypePtr getReturnTypeImpl(const ColumnsWithTypeAndName &) const override
+    {
+        auto data_type = std::make_shared<DataTypeUInt8>();
+        return makeNullable(data_type);
+    }
+
+     ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t input_rows_count) const override

Review Comment:
   尽量不要把所有的逻辑放在一个大函数里。参考src/Functions/regexpExtract.cpp的实现,按照两个参数是否为constant分成若干种情况,每种情况对应一个函数。



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#issuecomment-2359840961

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-6856][CH]Support arrays_overlap and fix array_join diff [incubator-gluten]

Posted by "KevinyhZou (via GitHub)" <gi...@apache.org>.
KevinyhZou commented on code in PR #6857:
URL: https://github.com/apache/incubator-gluten/pull/6857#discussion_r1767810732


##########
cpp-ch/local-engine/Functions/SparkFunctionArraysOverlap.cpp:
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnNullable.h>
+#include <Functions/IFunction.h>
+#include <Functions/FunctionFactory.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+
+using namespace DB;
+
+namespace DB
+{
+namespace ErrorCodes
+{
+    extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+}
+
+namespace local_engine
+{
+class SparkFunctionArraysOverlap : public IFunction
+{
+public:
+    static constexpr auto name = "sparkArraysOverlap";

Review Comment:
   CH already has a similar function `hasAny`, but it different from `arraysOverlap`, has so many corner cases, it will be very complex to convert arraysOverlap to hasAny, as below
   ```
   parse arrays_overlap(arr1, arr2) as
                   if (isNull(arr1) || isNull(arr2))
                       return NULL
                   else if (isEmpty(arr1) || isEmpty(arr2))
                       return false;
                   else if (arr1.hasAny(arr2))
                   {
                       if (!arr1.has(NULL) || !arr2.has(NULL))
                           return true;
                       else if (arr1.intersect(arr2) != NULL)
                           return true
                       else
                           return NULL;
                   }
                   else if (arr1.has(NULL) || arr2.has(NULL))
                       return NULL;
                   else if (arr1.has(NaN) && arr2.has(NaN))
                       return true;
                   else
                       return false;
   ```
   
   and it is not easy to slove the case when array parameters contain `NaN`. So I think it is better to write a self-defined function. @taiyang-li 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org