You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Sunyue (via GitHub)" <gi...@apache.org> on 2025/07/22 02:05:59 UTC
[I] Follow up on issue https://github.com/apache/arrow-go/issues/404 [arrow-go]
Sunyue opened a new issue, #446:
URL: https://github.com/apache/arrow-go/issues/446
### Describe the bug, including details regarding any error messages, version, and platform.
I had a try with enhancement from https://github.com/apache/arrow-go/issues/404 using v18.4.0, when executing exprs.ExecuteScalarExpression it still gives me an unimplemented error. The error is raised from arrow/compute/exprs/exec.go literalToDatum(), maybe those new PrecisionTimeType or PrecisionTimestampType should also be supported in this method? Thanks.
### Component(s)
Other
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Follow up on issue https://github.com/apache/arrow-go/issues/404 [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #446:
URL: https://github.com/apache/arrow-go/issues/446#issuecomment-3109027381
https://pkg.go.dev/github.com/apache/arrow-go/v18/arrow/compute#FilterRecordBatch should work for Decimal256, if not let me know :smile:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Follow up on issue https://github.com/apache/arrow-go/issues/404 [arrow-go]
Posted by "Sunyue (via GitHub)" <gi...@apache.org>.
Sunyue commented on issue #446:
URL: https://github.com/apache/arrow-go/issues/446#issuecomment-3100450731
Please find my test case below:
```go
import (
"testing"
"github.com/apache/arrow-go/v18/arrow"
"github.com/apache/arrow-go/v18/arrow/array"
"github.com/apache/arrow-go/v18/arrow/compute"
"github.com/apache/arrow-go/v18/arrow/compute/exprs"
"github.com/apache/arrow-go/v18/arrow/memory"
"github.com/stretchr/testify/require"
"github.com/substrait-io/substrait-go/v4/expr"
"github.com/substrait-io/substrait-go/v4/types"
)
func Test_Types(t *testing.T) {
t.Parallel()
tt := []struct {
name string
schema func() *arrow.Schema
record func(rq *require.Assertions, schema *arrow.Schema) arrow.Record
val func(rq *require.Assertions) expr.Literal
}{
{
name: "expect arrow.TIME64 (ns) ok",
schema: func() *arrow.Schema {
field := arrow.Field{
Name: "col",
Type: &arrow.Time64Type{Unit: arrow.Nanosecond},
Nullable: true,
}
return arrow.NewSchema([]arrow.Field{field}, nil)
},
record: func(rq *require.Assertions, schema *arrow.Schema) arrow.Record {
b := array.NewTime64Builder(memory.DefaultAllocator, &arrow.Time64Type{Unit: arrow.Nanosecond})
defer b.Release()
t1, err := arrow.Time64FromString("10:00:00.000000", arrow.Nanosecond)
rq.NoError(err, "Failed to create Time64 value")
b.AppendValues([]arrow.Time64{t1}, []bool{true})
return array.NewRecord(schema, []arrow.Array{b.NewArray()}, 1)
},
val: func(rq *require.Assertions) expr.Literal {
v, err := arrow.Time64FromString("11:00:00.000000", arrow.Nanosecond)
rq.NoError(err, "Failed to create Time64 value")
pt := &types.PrecisionTime{
Precision: int32(arrow.Nanosecond) * 3,
Value: int64(v),
}
lit, err := expr.NewLiteral(pt, true)
rq.NoError(err, "Failed to create literal")
return lit
},
},
}
for _, tc := range tt {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
ctx := t.Context()
rq := require.New(t)
schema := tc.schema()
record := tc.record(rq, schema)
extSet := exprs.GetExtensionIDSet(ctx)
builder := exprs.NewExprBuilder(extSet)
err := builder.SetInputSchema(schema)
rq.NoError(err, "Failed to set input schema")
b, err := builder.CallScalar("less", nil,
builder.FieldRef("col"),
builder.Literal(tc.val(rq)),
)
rq.NoError(err, "Failed to call scalar")
e, err := b.BuildExpr()
rq.NoError(err, "Failed to build expression")
ctx = exprs.WithExtensionIDSet(ctx, extSet)
dr := compute.NewDatum(record)
defer dr.Release()
_, err = exprs.ExecuteScalarExpression(ctx, schema, e, dr)
rq.NoError(err, "Failed to execute scalar expression")
})
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Follow up on issue https://github.com/apache/arrow-go/issues/404 [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #446:
URL: https://github.com/apache/arrow-go/issues/446#issuecomment-3103657887
Ah, I see where I missed the `literalToDatum`. I'll fix that one :smile:
As for Decimal256, the issue there is that Substrait explicitly does not support Decimal256, the precision for substrait decimal maxes out at 38, i.e. Decimal128, which is why the conversion hard codes to decimal128.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Follow up on issue https://github.com/apache/arrow-go/issues/404 [arrow-go]
Posted by "Sunyue (via GitHub)" <gi...@apache.org>.
Sunyue commented on issue #446:
URL: https://github.com/apache/arrow-go/issues/446#issuecomment-3105480397
@zeroshade Do you have any recommended work around for filtering Decimal 256 in arrow records or tables?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Follow up on issue https://github.com/apache/arrow-go/issues/404 [arrow-go]
Posted by "Sunyue (via GitHub)" <gi...@apache.org>.
Sunyue commented on issue #446:
URL: https://github.com/apache/arrow-go/issues/446#issuecomment-3100451491
Another thing is filtering on arrow.DECIMAL128 now works, but still not the case for arrow.DECIMAL256. It looks like when converting to types.DecimalType, both DECIMAL128 and DECIMAL256 are possible, but there are several place when converting from types.DecimalType back to arrow types, it was hard coded to DECIMAL128. Any chance make it work for DECIMAL256?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Follow up on issue https://github.com/apache/arrow-go/issues/404 [arrow-go]
Posted by "Sunyue (via GitHub)" <gi...@apache.org>.
Sunyue commented on issue #446:
URL: https://github.com/apache/arrow-go/issues/446#issuecomment-3139291534
> Ah, I see where I missed the `literalToDatum`. I'll fix that one 😄
>
> As for Decimal256, the issue there is that Substrait explicitly does not support Decimal256, the precision for substrait decimal maxes out at 38, i.e. Decimal128, which is why the conversion hard codes to decimal128.
@zeroshade For Decimal256, not sure if it is a good idea, but until substrait supports Decimal256, can we use substrait's `types.UserDefinedType` for Decimal256? The idea is:
- In `ToSubstraitType` return `types.UserDefinedType` for Decimal256, maybe put precision and scale into the `TypeParameters`?
- In `literalToDatum` use `ext.DecodeTypeArrow(t.TypeReference)` to decode the user defined type, if it is decimal256, then convert it back using the information that we store in the user defined type.
The only thing is for different precision/scale, user need to call `ExtensionIDRegistry.RegisterType` for each different combination, so that we can `EncodeType` and `DecodeTypeArrow`.
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Follow up on issue https://github.com/apache/arrow-go/issues/404 [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade closed issue #446: Follow up on issue https://github.com/apache/arrow-go/issues/404
URL: https://github.com/apache/arrow-go/issues/446
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org