You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "abulbasar (via GitHub)" <gi...@apache.org> on 2024/12/04 11:53:20 UTC
[I] Acero kernel function returning int64 or float64 not working [arrow]
abulbasar opened a new issue, #44935:
URL: https://github.com/apache/arrow/issues/44935
### Describe the bug, including details regarding any error messages, version, and platform.
I am trying to create a custom function that takes an uniary input float64 and returns a float64. But, the output is not accurate. Mostly it is all 0. Look into the sample code to reproduce the issue
Arrow version: 1800.0.0
Gcc: 13
```
// main.cpp
#include <iostream>
#include <arrow/compute/api.h>
#include <arrow/status.h>
#include <arrow/type.h>
#include <arrow/array.h>
#include <arrow/builder.h>
#include <arrow/compute/exec.h>
#include <arrow/compute/function.h>
using namespace arrow;
using namespace arrow::compute;
// Define the function that computes the square of input values.
Status SquareFunctionExec(KernelContext* ctx, const ExecSpan& batch, ExecResult* out) {
// Ensure the input is numeric.
if (!batch[0].is_array() || batch[0].type()->id() != Type::DOUBLE) {
return Status::Invalid("Expected double type for input");
}
const auto& exec_batch = batch.ToExecBatch();
const std::shared_ptr<arrow::DoubleArray>& arg0 = exec_batch.values[0].array_as<arrow::DoubleArray>();
const int64_t length = arg0->length();
arrow::DoubleBuilder builder;
ARROW_RETURN_NOT_OK(builder.Resize(length));
for (int64_t i = 0; i < length; ++i)
{
if (arg0->IsNull(i))
builder.UnsafeAppendNull();
else
{
const double d = arg0->Value(i);
ARROW_RETURN_NOT_OK(builder.Append(d * d));
}
}
ARROW_ASSIGN_OR_RAISE(const std::shared_ptr<arrow::Array> output_array, builder.Finish());
out->value = output_array->data();
std::cout << " Inside kernel operation (Expected)" << output_array->ToString() << std::endl;
return arrow::Status::OK();
}
Status RegisterSquareFunction(FunctionRegistry* registry) {
auto square_func = std::make_shared<ScalarFunction>("square", Arity::Unary(), arrow::compute::FunctionDoc::Empty());
const auto signature = arrow::compute::KernelSignature::Make(
{arrow::float64()}, arrow::float64());
arrow::compute::ScalarKernel kernel(signature, &SquareFunctionExec);
RETURN_NOT_OK(square_func->AddKernel(std::move(kernel)));
return registry->AddFunction(std::move(square_func));
}
void assertOK(const arrow::Status& status)
{
if (!status.ok())
throw std::runtime_error(status.message());
}
int main() {
// Initialize the compute function registry.
const auto registry = arrow::compute::GetFunctionRegistry();
// Register the custom square function.
auto status = RegisterSquareFunction(registry);
if (!status.ok()) {
std::cerr << "Error registering custom function: " << status.ToString() << std::endl;
return 1;
}
// Use the function (example).
ExecContext exec_ctx;
FunctionOptions* options = nullptr;
arrow::DoubleBuilder builder;
// Append scalar values
assertOK(builder.Append(1.1));
assertOK(builder.Append(2.2));
assertOK(builder.Append(3.3));
// Finalize the array (convert builder to Array)
std::shared_ptr<arrow::DoubleArray> input_array;
assertOK(builder.Finish(&input_array));
const auto call_result = CallFunction("square", {input_array}, options, &exec_ctx);
if (call_result.ok()) {
const auto& output = call_result.ValueOrDie();
std::cout << "Result (Unexpected): " << output.ToString() << std::endl;
} else {
std::cerr << "Function execution failed: " << call_result.status().ToString() << std::endl;
}
return 0;
}
```
Cmake
```
cmake_minimum_required(VERSION 3.28)
project(ArrowTest)
set(CMAKE_CXX_STANDARD 23)
set(CMAKE_CXX_FLAGS "-Wall")
set(CMAKE_CXX_STANDARD_REQUIRED True)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wreturn-type")
set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -static-libgcc -static-libstdc++")
find_package(Arrow REQUIRED)
find_package(Parquet REQUIRED)
find_package(ArrowDataset REQUIRED)
message("Arrow_FOUND: ${Arrow_FOUND}")
message("Arrow Version: ${ARROW_VERSION}")
message("Arrow SO Version: ${ARROW_FULL_SO_VERSION}")
set(ARROW_LIBRARY_TARGET arrow_shared)
set(PARQUET_LIBRARY_TARGET parquet_shared)
set(ARROW_DATASET_LIBRARY_TARGET arrow_dataset)
set(ARROW_ACERO arrow_acero)
file(MAKE_DIRECTORY "${ARROW_INCLUDE_DIR}")
#add_library(${ARROW_LIBRARY_TARGET} SHARED IMPORTED)
#add_library(${PARQUET_LIBRARY_TARGET} SHARED IMPORTED)
add_executable(ArrowTest main.cpp)
target_link_libraries(ArrowTest
PRIVATE
${ARROW_ACERO}
${ARROW_LIBRARY_TARGET}
${ARROW_DATASET_LIBRARY_TARGET}
${PARQUET_LIBRARY_TARGET}
)
```
ExpectedOutput: [
1.2100000000000002,
4.840000000000001,
10.889999999999999
]
Actual Output: Array([
2.548682823166e-311,
0,
0
])
### Component(s)
C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Acero kernel function returning int64 or float64 not working [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #44935:
URL: https://github.com/apache/arrow/issues/44935#issuecomment-2517552064
Have you tried stepping in `SquareFunctionExec` to see what happens?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Acero kernel function returning int64 or float64 not working [arrow]
Posted by "abulbasar (via GitHub)" <gi...@apache.org>.
abulbasar commented on issue #44935:
URL: https://github.com/apache/arrow/issues/44935#issuecomment-2530782157
Setting the memory allocator to `NO_PREALLOCATE` resolved the issue.
```
kernel.mem_allocation = arrow::compute::MemAllocation::NO_PREALLOCATE;
```
Thank you @pitrou for the hint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Acero kernel function returning int64 or float64 not working [arrow]
Posted by "abulbasar (via GitHub)" <gi...@apache.org>.
abulbasar closed issue #44935: [C++] Acero kernel function returning int64 or float64 not working
URL: https://github.com/apache/arrow/issues/44935
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Acero kernel function returning int64 or float64 not working [arrow]
Posted by "abulbasar (via GitHub)" <gi...@apache.org>.
abulbasar commented on issue #44935:
URL: https://github.com/apache/arrow/issues/44935#issuecomment-2530566038
> Inside kernel operation (Expected)
Yes, the function produces expected result. In the sample code you you can see i have printed the output from that function. Search for ... "Inside kernel operation (Expected)"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Acero kernel function returning int64 or float64 not working [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #44935:
URL: https://github.com/apache/arrow/issues/44935#issuecomment-2517566270
One important thing: the ScalarKernel defaults are as follows:
https://github.com/apache/arrow/blob/ded148c2ef5dd441a3b6ab9496d0ed0aeb940f71/cpp/src/arrow/compute/kernel.h#L566-L569
So `out->value` has already been allocated, its validity bitmap has been computed from the input args, and you should just write inside its value buffer for non-null inputs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org