You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "zeroshade (via GitHub)" <gi...@apache.org> on 2023/08/31 21:23:36 UTC
[GitHub] [arrow] zeroshade opened a new pull request, #37503: GH-37419: [Go][Parquet] Decimal256 support for pqarrow
zeroshade opened a new pull request, #37503:
URL: https://github.com/apache/arrow/pull/37503
<!--
Thanks for opening a pull request!
If this is your first pull request you can find detailed information on how
to contribute here:
* [New Contributor's Guide](https://arrow.apache.org/docs/dev/developers/guide/step_by_step/pr_lifecycle.html#reviews-and-merge-of-the-pull-request)
* [Contributing Overview](https://arrow.apache.org/docs/dev/developers/overview.html)
If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
-->
### Rationale for this change
When support for decimal128 was added to the Go Parquet lib, decimal256 wasn't yet implemented in the library. This adds proper support to read/write decimal256 with Parquet based on origin schemas and precision.
### Are these changes tested?
yes
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] conbench-apache-arrow[bot] commented on pull request #37503: GH-37419: [Go][Parquet] Decimal256 support for pqarrow
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #37503:
URL: https://github.com/apache/arrow/pull/37503#issuecomment-1709258245
After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 697a2cbaf834616a676f11f1b89b9200344de998.
There were no benchmark performance regressions. 🎉
The [full Conbench report](https://github.com/apache/arrow/runs/16564015156) has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on a diff in pull request #37503: GH-37419: [Go][Parquet] Decimal256 support for pqarrow
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #37503:
URL: https://github.com/apache/arrow/pull/37503#discussion_r1315999493
##########
go/parquet/pqarrow/column_readers.go:
##########
@@ -833,12 +888,51 @@ func transferDecimalBytes(rdr file.BinaryRecordReader, dt arrow.DataType) (*arro
}
ret := array.NewData(dt, length, []*memory.Buffer{
- input.Data().Buffers()[0], memory.NewBufferBytes(data),
+ in.Data().Buffers()[0], memory.NewBufferBytes(data),
}, nil, nullCount, 0)
defer ret.Release()
return array.MakeFromData(ret), nil
}
+ convert256 := func(in varOrFixedBin) (arrow.Array, error) {
+ length := in.Len()
+ data := make([]byte, arrow.Decimal256Traits.BytesRequired(length))
+ out := arrow.Decimal256Traits.CastFromBytes(data)
+
+ nullCount := in.NullN()
+ var err error
+ for i := 0; i < length; i++ {
+ if nullCount > 0 && in.IsNull(i) {
+ continue
+ }
+
+ rec := in.Value(i)
+ if len(rec) <= 0 {
+ return nil, fmt.Errorf("invalud BYTEARRAY length for type: %s", dt)
Review Comment:
nice catch! thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade merged pull request #37503: GH-37419: [Go][Parquet] Decimal256 support for pqarrow
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade merged PR #37503:
URL: https://github.com/apache/arrow/pull/37503
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #37503: GH-37419: [Go][Parquet] Decimal256 support for pqarrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #37503:
URL: https://github.com/apache/arrow/pull/37503#discussion_r1313327143
##########
go/parquet/pqarrow/column_readers.go:
##########
@@ -833,12 +888,51 @@ func transferDecimalBytes(rdr file.BinaryRecordReader, dt arrow.DataType) (*arro
}
ret := array.NewData(dt, length, []*memory.Buffer{
- input.Data().Buffers()[0], memory.NewBufferBytes(data),
+ in.Data().Buffers()[0], memory.NewBufferBytes(data),
}, nil, nullCount, 0)
defer ret.Release()
return array.MakeFromData(ret), nil
}
+ convert256 := func(in varOrFixedBin) (arrow.Array, error) {
+ length := in.Len()
+ data := make([]byte, arrow.Decimal256Traits.BytesRequired(length))
+ out := arrow.Decimal256Traits.CastFromBytes(data)
+
+ nullCount := in.NullN()
+ var err error
+ for i := 0; i < length; i++ {
+ if nullCount > 0 && in.IsNull(i) {
+ continue
+ }
+
+ rec := in.Value(i)
+ if len(rec) <= 0 {
+ return nil, fmt.Errorf("invalud BYTEARRAY length for type: %s", dt)
Review Comment:
Typo: invalud
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org