You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "loicalleyne (via GitHub)" <gi...@apache.org> on 2023/07/20 22:45:49 UTC
[GitHub] [arrow] loicalleyne opened a new pull request, #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
loicalleyne opened a new pull request, #36796:
URL: https://github.com/apache/arrow/pull/36796
<!--
Thanks for opening a pull request!
If this is your first pull request you can find detailed information on how
to contribute here:
* [New Contributor's Guide](https://arrow.apache.org/docs/dev/developers/guide/step_by_step/pr_lifecycle.html#reviews-and-merge-of-the-pull-request)
* [Contributing Overview](https://arrow.apache.org/docs/dev/developers/overview.html)
If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
-->
### Rationale for this change
First step in implementing an Avro reader.
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
### What changes are included in this PR?
Added Avro schema to Arrow schema converter.
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
### Are these changes tested?
yes
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
### Are there any user-facing changes?
Two exported functions:
**ArrowSchemaFromAvro** returns a new Arrow schema from an Avro schema JSON.
If the top level is of record type, set includeTopLevel to either make its fields top level fields in the resulting schema or nested in a single field.
_ArrowSchemaFromAvro(avroSchema []byte, includeTopLevel bool) (*arrow.Schema, error)_
**AvroPrimitiveToArrowType** returns the Arrow DataType equivalent to a Avro primitive type.
NOTE: Arrow Binary type is used as a catchall to avoid potential data loss.
_AvroPrimitiveToArrowType(avroFieldType string) arrow.DataType_
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please uncomment the line below and explain which changes are breaking.
-->
<!-- **This PR includes breaking changes to public APIs.** -->
<!--
Please uncomment the line below (and provide explanation) if the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld). We use this to highlight fixes to issues that may affect users without their knowledge. For this reason, fixing bugs that cause errors don't count, since those are usually obvious.
-->
<!-- **This PR contains a "Critical Fix".** -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] loicalleyne commented on a diff in pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne commented on code in PR #36796:
URL: https://github.com/apache/arrow/pull/36796#discussion_r1293980910
##########
go/arrow/avro/reader.go:
##########
@@ -0,0 +1,490 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package avro
+
+import (
+ "errors"
+ "fmt"
+ "io"
+ "sync"
+ "sync/atomic"
+
+ "github.com/apache/arrow/go/v13/arrow"
+ "github.com/apache/arrow/go/v13/arrow/array"
+ "github.com/apache/arrow/go/v13/arrow/internal/debug"
+ "github.com/apache/arrow/go/v13/arrow/memory"
+ "github.com/linkedin/goavro/v2"
+)
+
+// Reader wraps goavro/OCFReader and creates array.Records from a schema.
+type OCFReader struct {
+ r *goavro.OCFReader
+ schema *arrow.Schema
+
+ refs int64
+ bld *array.RecordBuilder
+ cur arrow.Record
+ err error
+
+ chunk int
+ done bool
+ next func() bool
+
+ mem memory.Allocator
+
+ topLevel bool
+ once sync.Once
+
+ fieldConverter []func(val string)
+ columnFilter []string
+ columnTypes map[string]arrow.DataType
+
+ stringsCanBeNull bool
+ nulls []string
+}
+
+// NewReader returns a reader that reads from an Avro OCF file and creates
+// arrow.Records from the converted schema.
+func NewOCFReader(r io.Reader, opts ...Option) *OCFReader {
+ ocfr, err := goavro.NewOCFReader(r)
+ if err != nil {
+ panic(fmt.Errorf("%w: could not create avro ocfreader", arrow.ErrInvalid))
+ }
+
+ rr := &OCFReader{
+ r: ocfr,
+ //schema: schema,
+ refs: 1,
+ chunk: 1,
+ stringsCanBeNull: false,
Review Comment:
removed reader from this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #36796:
URL: https://github.com/apache/arrow/pull/36796#issuecomment-1708715974
@loicalleyne looks like you've got some build/test failures. Other than fixing those issues, everything else looks good to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] loicalleyne commented on a diff in pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne commented on code in PR #36796:
URL: https://github.com/apache/arrow/pull/36796#discussion_r1281162168
##########
go/arrow/avro/schema.go:
##########
@@ -0,0 +1,409 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package avro reads Avro OCF files and presents the extracted data as records
+package avro
+
+import (
+ "encoding/json"
+ "fmt"
+ "math"
+ "strconv"
+
+ "github.com/apache/arrow/go/v13/arrow"
+)
+
+type schemaNode struct {
+ name string
+ ofType interface{}
+ logicalType string
+ precision int32
+ scale int32
+ size int
+ symbols []string
+ fields []interface{}
+}
+
+// ArrowSchemaFromAvro returns a new Arrow schema from an Avro schema JSON.
+// If the top level is of record type, set includeTopLevel to either make
+// its fields top level fields in the resulting schema or nested in a single field.
+func ArrowSchemaFromAvro(avroSchema []byte) (*arrow.Schema, error) {
+ var m map[string]interface{}
+ var node schemaNode
+ json.Unmarshal(avroSchema, &m)
+ if m["type"].(string) == "record" {
+ if _, ok := m["fields"]; ok {
+ for _, field := range m["fields"].([]interface{}) {
+ node.fields = append(node.fields, field.(map[string]interface{}))
+ }
+ if len(node.fields) == 0 {
+ return nil, fmt.Errorf("invalid avro schema: no top level record fields found")
+ }
+ }
+ }
+ fields := iterateFields(node.fields)
+ return arrow.NewSchema(fields, nil), nil
+}
+
+func iterateFields(f []interface{}) []arrow.Field {
+ var s []arrow.Field
+ for _, field := range f {
+ var n schemaNode
+ n.name = field.(map[string]interface{})["name"].(string)
+ n.ofType = field.(map[string]interface{})["type"]
+ switch n.ofType.(type) {
+ // Getting field type from within field object
+ case string:
+ switch n.ofType.(string) {
+ case "enum":
+ for _, symbol := range field.(map[string]interface{})["symbols"].([]interface{}) {
+ n.symbols = append(n.symbols, symbol.(string))
+ }
+ default:
+ if lt, ok := field.(map[string]interface{})["logicalType"]; ok {
+ n.logicalType = lt.(string)
+ }
+ if lt, ok := field.(map[string]interface{})["size"]; ok {
+ n.size = int(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["precision"]; ok {
+ n.precision = int32(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["scale"]; ok {
+ n.scale = int32(lt.(float64))
+ }
+ }
+ // Field type is an object
+ case map[string]interface{}:
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["logicalType"]; ok {
+ n.logicalType = lt.(string)
+ }
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["size"]; ok {
+ n.size = int(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["precision"]; ok {
+ n.precision = int32(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["scale"]; ok {
+ n.scale = int32(lt.(float64))
+ }
+ case []interface{}:
+
+ default:
+ if lt, ok := field.(map[string]interface{})["logicalType"]; ok {
+ n.logicalType = lt.(string)
+ }
+ if lt, ok := field.(map[string]interface{})["size"]; ok {
+ n.size = int(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["precision"]; ok {
+ n.precision = int32(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["scale"]; ok {
+ n.scale = int32(lt.(float64))
+ }
+ }
+ // Field is of type "record"
+ if nf, f := field.(map[string]interface{})["fields"]; f {
+ switch nf.(type) {
+ // primitive & complex types
+ case map[string]interface{}:
+ for _, v := range nf.(map[string]interface{})["fields"].([]interface{}) {
+ n.fields = append(n.fields, v.(map[string]interface{}))
+ }
+ // type unions
+ default:
+ for _, v := range nf.([]interface{}) {
+ n.fields = append(n.fields, v.(map[string]interface{}))
+ }
+ }
+ }
+ s = append(s, traverseNodes(n))
+ }
+ return s
+}
+
+func traverseNodes(node schemaNode) arrow.Field {
+ switch node.ofType.(type) {
+ case string:
+ // Avro primitive type
+ if len(node.fields) == 0 {
+ switch node.ofType.(string) {
+ case "boolean", "int", "long", "float", "double", "bytes", "string":
+ if node.logicalType != "" {
+ return avroLogicalToArrowField(node)
+ }
+ // Avro primitive type
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(node.ofType.(string))}
+ case "fixed":
+ // Duration type is not supported in github.com/linkedin/goavro
+ // Implementing as Binary for now.
+ switch node.logicalType {
+ case "decimal":
+ return avroLogicalToArrowField(node)
+ case "duration":
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.Binary}
+ //return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.MonthDayNanoInterval}
+ }
+ return arrow.Field{Name: node.name, Type: &arrow.FixedSizeBinaryType{ByteWidth: node.size}}
+ case "enum":
+ symbols := make(map[string]string)
+ for index, symbol := range node.symbols {
+ k := strconv.FormatInt(int64(index), 10)
+ symbols[k] = symbol
+ }
+ var dt arrow.DictionaryType = arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Uint64, ValueType: arrow.BinaryTypes.String, Ordered: false}
+ sl := len(symbols)
+ switch {
+ case sl <= math.MaxUint8:
+ dt.IndexType = arrow.PrimitiveTypes.Uint8
+ case sl > math.MaxUint8 && sl <= math.MaxUint16:
+ dt.IndexType = arrow.PrimitiveTypes.Uint16
+ case sl > math.MaxUint16 && sl <= math.MaxUint32:
+ dt.IndexType = arrow.PrimitiveTypes.Uint32
+ }
+ return arrow.Field{Name: node.name, Type: &dt, Nullable: true, Metadata: arrow.MetadataFrom(symbols)}
+ default:
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(node.ofType.(string))}
+ }
+ } else {
+ // avro "record" type, node has "fields" array
+ if node.ofType.(string) == "record" {
+ var n schemaNode
+ n.name = node.name
+ n.ofType = node.ofType
+ if len(node.fields) > 0 {
+ n.fields = append(n.fields, node.fields...)
+ }
+ f := iterateFields(n.fields)
+ return arrow.Field{Name: node.name, Type: arrow.StructOf(f...)}
+ }
+ }
+ // Avro complex types
+ case map[string]interface{}:
+ //var n schemaNode
+ //n.name = node.name
+ //n.ofType = node.ofType.(map[string]interface{})["type"]
+ switch node.logicalType {
+ case "":
+ return avroComplexToArrowField(node)
+ default:
+ return avroLogicalToArrowField(node)
+ }
+
+ // Avro union types
+ case []interface{}:
+ var unionTypes []string
+ for _, ft := range node.ofType.([]interface{}) {
+ switch ft.(type) {
+ // primitive types
+ case string:
+ if ft != "null" {
+ unionTypes = append(unionTypes, ft.(string))
+ }
+ continue
+ // complex types
+ case map[string]interface{}:
+ var n schemaNode
+ n.name = node.name
+ n.ofType = ft.(map[string]interface{})["type"]
+ if _, f := ft.(map[string]interface{})["fields"]; f {
+ for _, field := range ft.(map[string]interface{})["fields"].([]interface{}) {
+ n.fields = append(n.fields, field.(map[string]interface{}))
+ }
+ }
+ f := iterateFields(n.fields)
+ return arrow.Field{Name: node.name, Type: arrow.StructOf(f...)}
+ }
+ }
+ // Supported Avro union type is null + one other type.
+ // TODO: Complex AVRO union to Arrow Dense || Sparse Union.
+ if len(unionTypes) == 1 {
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(unionTypes[0])}
+ } else {
+ // BYTE_ARRAY is the catchall if union type is anything beyond null + one other type.
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.Binary}
+ }
+ }
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.Binary}
+}
+
+func avroLogicalToArrowField(node schemaNode) arrow.Field {
+ // Avro logical types
+ switch node.logicalType {
+ // The decimal logical type represents an arbitrary-precision signed decimal number of the form unscaled × 10-scale.
+ // A decimal logical type annotates Avro bytes or fixed types. The byte array must contain the two’s-complement
+ // representation of the unscaled integer value in big-endian byte order. The scale is fixed, and is specified
+ // using an attribute.
+ //
+ // The following attributes are supported:
+ // scale, a JSON integer representing the scale (optional). If not specified the scale is 0.
+ // precision, a JSON integer representing the (maximum) precision of decimals stored in this type (required).
+ case "decimal":
+ if node.precision <= 38 {
+ return arrow.Field{Name: node.name, Type: &arrow.Decimal128Type{Precision: node.precision, Scale: node.scale}}
+ } else {
+ return arrow.Field{Name: node.name, Type: &arrow.Decimal256Type{Precision: node.precision, Scale: node.scale}}
+ }
+
+ // The uuid logical type represents a random generated universally unique identifier (UUID).
+ // A uuid logical type annotates an Avro string. The string has to conform with RFC-4122
+ case "uuid":
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.String}
+
+ // The date logical type represents a date within the calendar, with no reference to a particular
+ // time zone or time of day.
+ // A date logical type annotates an Avro int, where the int stores the number of days from the unix epoch,
+ // 1 January 1970 (ISO calendar).
+ case "date":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Date32}
+
+ // The time-millis logical type represents a time of day, with no reference to a particular calendar,
+ // time zone or date, with a precision of one millisecond.
+ // A time-millis logical type annotates an Avro int, where the int stores the number of milliseconds
+ // after midnight, 00:00:00.000.
+ case "time-millis":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Time32ms}
+
+ // The time-micros logical type represents a time of day, with no reference to a particular calendar,
+ // time zone or date, with a precision of one microsecond.
+ // A time-micros logical type annotates an Avro long, where the long stores the number of microseconds
+ // after midnight, 00:00:00.000000.
+ case "time-micros":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Time64us}
+
+ // The timestamp-millis logical type represents an instant on the global timeline, independent of a
+ // particular time zone or calendar, with a precision of one millisecond. Please note that time zone
+ // information gets lost in this process. Upon reading a value back, we can only reconstruct the instant,
+ // but not the original representation. In practice, such timestamps are typically displayed to users in
+ // their local time zones, therefore they may be displayed differently depending on the execution environment.
+ // A timestamp-millis logical type annotates an Avro long, where the long stores the number of milliseconds
+ // from the unix epoch, 1 January 1970 00:00:00.000 UTC.
+ case "timestamp-millis":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Timestamp_ms}
+ // The timestamp-micros logical type represents an instant on the global timeline, independent of a
+ // particular time zone or calendar, with a precision of one microsecond. Please note that time zone
+ // information gets lost in this process. Upon reading a value back, we can only reconstruct the instant,
+ // but not the original representation. In practice, such timestamps are typically displayed to users
+ // in their local time zones, therefore they may be displayed differently depending on the execution environment.
+ // A timestamp-micros logical type annotates an Avro long, where the long stores the number of microseconds
+ // from the unix epoch, 1 January 1970 00:00:00.000000 UTC.
+ case "timestamp-micros":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Timestamp_us}
+ // The local-timestamp-millis logical type represents a timestamp in a local timezone, regardless of
+ // what specific time zone is considered local, with a precision of one millisecond.
+ // A local-timestamp-millis logical type annotates an Avro long, where the long stores the number of
+ // milliseconds, from 1 January 1970 00:00:00.000.
+ case "local-timestamp-millis":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Timestamp_ms}
Review Comment:
perhaps NoTZ versions would be better - I was going on the assumption that the users of the data would have to cast the timezone to the data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] loicalleyne commented on a diff in pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne commented on code in PR #36796:
URL: https://github.com/apache/arrow/pull/36796#discussion_r1290639987
##########
go/arrow/avro/common.go:
##########
@@ -0,0 +1,162 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package avro
+
+import (
+ "errors"
+ "fmt"
+
+ "github.com/apache/arrow/go/v13/arrow"
+ "github.com/apache/arrow/go/v13/arrow/memory"
+)
+
+var (
+ ErrMismatchFields = errors.New("arrow/avro: number of records mismatch")
+)
+
+// Option configures an Avro reader/writer.
+type Option func(config)
+type config interface{}
Review Comment:
fixed. Also moved the config functions to reader.go and removed common.go, common.go was a holdover from CSV having options for both reader and writer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on a diff in pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #36796:
URL: https://github.com/apache/arrow/pull/36796#discussion_r1272514353
##########
go/arrow/avro/schema_test.go:
##########
@@ -0,0 +1,530 @@
+package avro
Review Comment:
same as above, you need to add the apache license to the top of the file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on a diff in pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #36796:
URL: https://github.com/apache/arrow/pull/36796#discussion_r1281166530
##########
go/arrow/avro/schema.go:
##########
@@ -0,0 +1,409 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package avro reads Avro OCF files and presents the extracted data as records
+package avro
+
+import (
+ "encoding/json"
+ "fmt"
+ "math"
+ "strconv"
+
+ "github.com/apache/arrow/go/v13/arrow"
+)
+
+type schemaNode struct {
+ name string
+ ofType interface{}
+ logicalType string
+ precision int32
+ scale int32
+ size int
+ symbols []string
+ fields []interface{}
+}
+
+// ArrowSchemaFromAvro returns a new Arrow schema from an Avro schema JSON.
+// If the top level is of record type, set includeTopLevel to either make
+// its fields top level fields in the resulting schema or nested in a single field.
+func ArrowSchemaFromAvro(avroSchema []byte) (*arrow.Schema, error) {
+ var m map[string]interface{}
+ var node schemaNode
+ json.Unmarshal(avroSchema, &m)
+ if m["type"].(string) == "record" {
+ if _, ok := m["fields"]; ok {
+ for _, field := range m["fields"].([]interface{}) {
+ node.fields = append(node.fields, field.(map[string]interface{}))
+ }
+ if len(node.fields) == 0 {
+ return nil, fmt.Errorf("invalid avro schema: no top level record fields found")
+ }
+ }
+ }
+ fields := iterateFields(node.fields)
+ return arrow.NewSchema(fields, nil), nil
+}
Review Comment:
Yea, that's fine. MIT license is compatible with the Apache 2 license and can be bundled together as components
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] loicalleyne commented on a diff in pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne commented on code in PR #36796:
URL: https://github.com/apache/arrow/pull/36796#discussion_r1281147904
##########
go/arrow/avro/schema.go:
##########
@@ -0,0 +1,409 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package avro reads Avro OCF files and presents the extracted data as records
+package avro
+
+import (
+ "encoding/json"
+ "fmt"
+ "math"
+ "strconv"
+
+ "github.com/apache/arrow/go/v13/arrow"
+)
+
+type schemaNode struct {
+ name string
+ ofType interface{}
+ logicalType string
+ precision int32
+ scale int32
+ size int
+ symbols []string
+ fields []interface{}
+}
+
+// ArrowSchemaFromAvro returns a new Arrow schema from an Avro schema JSON.
+// If the top level is of record type, set includeTopLevel to either make
+// its fields top level fields in the resulting schema or nested in a single field.
+func ArrowSchemaFromAvro(avroSchema []byte) (*arrow.Schema, error) {
+ var m map[string]interface{}
+ var node schemaNode
+ json.Unmarshal(avroSchema, &m)
+ if m["type"].(string) == "record" {
+ if _, ok := m["fields"]; ok {
+ for _, field := range m["fields"].([]interface{}) {
+ node.fields = append(node.fields, field.(map[string]interface{}))
+ }
+ if len(node.fields) == 0 {
+ return nil, fmt.Errorf("invalid avro schema: no top level record fields found")
+ }
+ }
+ }
+ fields := iterateFields(node.fields)
+ return arrow.NewSchema(fields, nil), nil
+}
Review Comment:
Hamba has a MIT licence, is that ok to use in an Apache 2 licenced project?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-36760: [Go] Adding avro ocf reader - schema converter [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #36796:
URL: https://github.com/apache/arrow/pull/36796#issuecomment-1751306482
@loicalleyne should this be dropped/closed in favor of https://github.com/apache/arrow/pull/37115 ? It seems like this hasn't been updated or rebased
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] loicalleyne commented on a diff in pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne commented on code in PR #36796:
URL: https://github.com/apache/arrow/pull/36796#discussion_r1293980728
##########
go/arrow/avro/reader.go:
##########
@@ -0,0 +1,490 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package avro
+
+import (
+ "errors"
+ "fmt"
+ "io"
+ "sync"
+ "sync/atomic"
+
+ "github.com/apache/arrow/go/v13/arrow"
+ "github.com/apache/arrow/go/v13/arrow/array"
+ "github.com/apache/arrow/go/v13/arrow/internal/debug"
+ "github.com/apache/arrow/go/v13/arrow/memory"
+ "github.com/linkedin/goavro/v2"
+)
+
+// Reader wraps goavro/OCFReader and creates array.Records from a schema.
+type OCFReader struct {
+ r *goavro.OCFReader
+ schema *arrow.Schema
+
+ refs int64
+ bld *array.RecordBuilder
+ cur arrow.Record
+ err error
+
+ chunk int
+ done bool
+ next func() bool
+
+ mem memory.Allocator
+
+ topLevel bool
+ once sync.Once
+
+ fieldConverter []func(val string)
+ columnFilter []string
+ columnTypes map[string]arrow.DataType
+
+ stringsCanBeNull bool
+ nulls []string
+}
+
+// NewReader returns a reader that reads from an Avro OCF file and creates
+// arrow.Records from the converted schema.
+func NewOCFReader(r io.Reader, opts ...Option) *OCFReader {
+ ocfr, err := goavro.NewOCFReader(r)
+ if err != nil {
+ panic(fmt.Errorf("%w: could not create avro ocfreader", arrow.ErrInvalid))
+ }
+
+ rr := &OCFReader{
+ r: ocfr,
+ //schema: schema,
+ refs: 1,
+ chunk: 1,
+ stringsCanBeNull: false,
+ }
+ //rr.r.ReuseRecord = true
Review Comment:
removed reader from this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] loicalleyne commented on a diff in pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne commented on code in PR #36796:
URL: https://github.com/apache/arrow/pull/36796#discussion_r1316474144
##########
go/arrow/avro/schema.go:
##########
@@ -0,0 +1,407 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package avro reads Avro OCF files and presents the extracted data as records
+
+package avro
+
+import (
+ "fmt"
+ "math"
+ "strconv"
+
+ "github.com/apache/arrow/go/v14/arrow"
+ "github.com/apache/arrow/go/v14/internal/types"
+ avro "github.com/hamba/avro/v2"
+)
+
+type schemaNode struct {
+ name string
+ parent *schemaNode
+ schema avro.Schema
+ union bool
+ children []*schemaNode
+ arrowField arrow.Field
+ path []string
+ index, depth int32
+}
+
+var schemaCache avro.SchemaCache
+
+func newSchemaNode() *schemaNode { return &schemaNode{name: "", index: -1} }
+
+func (node *schemaNode) newChild(n string, s avro.Schema) *schemaNode {
+ child := &schemaNode{
+ name: n,
+ parent: node,
+ schema: s,
+ index: int32(len(node.children)),
+ depth: node.depth + 1}
+ node.children = append(node.children, child)
+ return child
+}
+func (node *schemaNode) Children() []*schemaNode { return node.children }
+
+func (node *schemaNode) Name() string { return node.name }
+
+// NamePath returns a slice of keys making up the path to the field
+func (node *schemaNode) NamePath() []string {
+ if len(node.path) == 0 {
+ var path []string
+ cur := node
+ for i := node.depth - 1; i >= 0; i-- {
+ path = append([]string{cur.name}, path...)
+ cur = cur.parent
+ }
+ return path
+ }
+ return node.path
+}
+
+// ArrowSchemaFromAvro returns a new Arrow schema from an Avro schema
+func ArrowSchemaFromAvro(schema avro.Schema) (s *arrow.Schema, err error) {
+ defer func() {
+ if r := recover(); r != nil {
+ s = nil
+ err = fmt.Errorf("invalid avro schema: could not convert schema")
+ }
+ }()
+ n := newSchemaNode()
+ n.schema = schema
+ c := n.newChild(n.schema.(avro.NamedSchema).Name(), n.schema)
+ arrowSchemafromAvro(c)
+ var fields []arrow.Field
+ for _, g := range c.Children() {
+ fields = append(fields, g.arrowField)
+ }
+ s = arrow.NewSchema(fields, nil)
+ return s, nil
+}
+
+func arrowSchemafromAvro(n *schemaNode) {
+ if ns, ok := n.schema.(avro.NamedSchema); ok {
+ schemaCache.Add(ns.Name(), ns)
+ }
+ switch st := n.schema.Type(); st {
+ case "record":
+ iterateFields(n)
+ case "enum":
+ symbols := make(map[string]string)
+ for index, symbol := range n.schema.(avro.PropertySchema).(*avro.EnumSchema).Symbols() {
+ k := strconv.FormatInt(int64(index), 10)
+ symbols[k] = symbol
+ }
+ var dt arrow.DictionaryType = arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Uint64, ValueType: arrow.BinaryTypes.String, Ordered: false}
+ sl := len(symbols)
+ switch {
+ case sl <= math.MaxUint8:
+ dt.IndexType = arrow.PrimitiveTypes.Uint8
+ case sl > math.MaxUint8 && sl <= math.MaxUint16:
+ dt.IndexType = arrow.PrimitiveTypes.Uint16
+ case sl > math.MaxUint16 && sl <= math.MaxUint32:
+ dt.IndexType = arrow.PrimitiveTypes.Uint32
+ }
+ n.arrowField = arrow.Field{Name: n.Name(), Type: &dt, Nullable: true, Metadata: arrow.MetadataFrom(symbols)}
+ case "array":
+ // logical items type
+ c := n.newChild(n.name, n.schema.(*avro.ArraySchema).Items())
+ if isLogicalSchemaType(n.schema.(*avro.ArraySchema).Items()) {
+ avroLogicalToArrowField(c)
+ } else {
+ arrowSchemafromAvro(c)
+ }
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.ListOf(c.arrowField.Type)}
+ case "map":
+ c := n.newChild(n.name, n.schema.(*avro.MapSchema).Values())
+ arrowSchemafromAvro(c)
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.MapOf(arrow.BinaryTypes.String, c.arrowField.Type), Metadata: c.arrowField.Metadata, Nullable: true}
+ case "union":
+ if n.schema.(*avro.UnionSchema).Nullable() {
+ if len(n.schema.(*avro.UnionSchema).Types()) > 1 {
+ n.schema = n.schema.(*avro.UnionSchema).Types()[1]
+ n.union = true
+ arrowSchemafromAvro(n)
+ }
+ }
+ // Avro "fixed" field type = Arrow FixedSize Primitive BinaryType
+ case "fixed":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.Name(), Type: &arrow.FixedSizeBinaryType{ByteWidth: n.schema.(*avro.FixedSchema).Size()}, Nullable: true}
+ }
+ case "string":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "bytes":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "int":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "long":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "float":
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ case "double":
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ case "boolean":
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ case "<ref>":
+ n.schema = schemaCache.Get(n.schema.(*avro.RefSchema).Schema().Name())
+ arrowSchemafromAvro(n)
+ case "null":
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.BinaryTypes.Binary, Nullable: true}
+ }
+}
+
+// iterate record Fields()
+func iterateFields(n *schemaNode) {
+ for _, f := range n.schema.(avro.PropertySchema).(*avro.RecordSchema).Fields() {
+ switch ft := f.Type().(type) {
+ // Avro "array" field type
+ case *avro.ArraySchema:
+ schemaCache.Add(f.Name(), ft.Items())
+ // logical items type
+ c := n.newChild(f.Name(), ft.Items())
+ if isLogicalSchemaType(ft.Items()) {
+ avroLogicalToArrowField(c)
+ } else {
+ arrowSchemafromAvro(c)
+ }
+ c.arrowField = arrow.Field{Name: c.name, Type: arrow.ListOf(c.arrowField.Type), Metadata: c.arrowField.Metadata, Nullable: true}
+ // Avro "enum" field type = Arrow dictionary type
+ case *avro.EnumSchema:
+ schemaCache.Add(f.Name(), f.Type())
+ c := n.newChild(f.Name(), f.Type())
+ symbols := make(map[string]string)
+ for index, symbol := range ft.Symbols() {
+ k := strconv.FormatInt(int64(index), 10)
+ symbols[k] = symbol
+ }
+ var dt arrow.DictionaryType = arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Uint64, ValueType: arrow.BinaryTypes.String, Ordered: false}
+ sl := len(symbols)
+ switch {
+ case sl <= math.MaxUint8:
+ dt.IndexType = arrow.PrimitiveTypes.Uint8
+ case sl > math.MaxUint8 && sl <= math.MaxUint16:
+ dt.IndexType = arrow.PrimitiveTypes.Uint16
+ case sl > math.MaxUint16 && sl <= math.MaxUint32:
+ dt.IndexType = arrow.PrimitiveTypes.Uint32
+ }
+ c.arrowField = arrow.Field{Name: f.Name(), Type: &dt, Nullable: true, Metadata: arrow.MetadataFrom(symbols)}
+ // Avro "fixed" field type = Arrow FixedSize Primitive BinaryType
+ case *avro.FixedSchema:
+ schemaCache.Add(f.Name(), f.Type())
+ if isLogicalSchemaType(f.Type()) {
+ c := n.newChild(f.Name(), f.Type())
+ avroLogicalToArrowField(c)
+ } else {
+ c := n.newChild(f.Name(), f.Type())
+ arrowSchemafromAvro(c)
+ }
+ case *avro.RecordSchema:
+ schemaCache.Add(f.Name(), f.Type())
+ c := n.newChild(f.Name(), f.Type())
+ iterateFields(c)
+ // Avro "map" field type - KVP with value of one type - keys are strings
+ case *avro.MapSchema:
+ schemaCache.Add(f.Name(), ft.Values())
+ c := n.newChild(f.Name(), ft.Values())
+ arrowSchemafromAvro(c)
+ if ft.Values().Type() == "union" {
+ c.arrowField = arrow.Field{Name: f.Name(), Type: arrow.MapOf(arrow.BinaryTypes.String, arrow.BinaryTypes.String), Metadata: c.arrowField.Metadata, Nullable: true}
+ } else {
+ c.arrowField = arrow.Field{Name: f.Name(), Type: arrow.MapOf(arrow.BinaryTypes.String, c.arrowField.Type), Metadata: c.arrowField.Metadata, Nullable: true}
+ }
+ case *avro.UnionSchema:
+ if ft.Nullable() {
+ if len(ft.Types()) > 1 {
+ schemaCache.Add(f.Name(), ft.Types()[1])
+ c := n.newChild(f.Name(), ft.Types()[1])
+ c.union = true
+ arrowSchemafromAvro(c)
+ }
+ }
+ default:
+ schemaCache.Add(f.Name(), f.Type())
+ if isLogicalSchemaType(f.Type()) {
+ c := n.newChild(f.Name(), f.Type())
+ avroLogicalToArrowField(c)
+ } else {
+ c := n.newChild(f.Name(), f.Type())
+ arrowSchemafromAvro(c)
+ }
+
+ }
+ }
+ var fields []arrow.Field
+ for _, child := range n.Children() {
+ fields = append(fields, child.arrowField)
+ }
+
+ namedSchema, ok := isNamedSchema(n.schema)
+ if !ok || namedSchema == n.name+"_data" || !n.union {
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.StructOf(fields...), Nullable: true}
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.StructOf(fields...), Nullable: true, Metadata: arrow.MetadataFrom(map[string]string{"typeName": namedSchema})}
+ }
+
+}
+
+func isLogicalSchemaType(s avro.Schema) bool {
+ lts, ok := s.(avro.LogicalTypeSchema)
+ if !ok {
+ return false
+ }
+ if lt := lts.Logical(); lt != nil {
+ return true
+ }
+ return false
+}
+
+func isNamedSchema(s avro.Schema) (string, bool) {
+ if ns, ok := s.(avro.NamedSchema); ok {
+ return ns.FullName(), ok
+ }
+ return "", false
+}
+
+// Avro primitive type.
+//
+// NOTE: Arrow Binary type is used as a catchall to avoid potential data loss.
Review Comment:
default removed, no fallback needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] loicalleyne commented on a diff in pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne commented on code in PR #36796:
URL: https://github.com/apache/arrow/pull/36796#discussion_r1281131718
##########
go/arrow/avro/common.go:
##########
@@ -0,0 +1,162 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package avro
+
+import (
+ "errors"
+ "fmt"
+
+ "github.com/apache/arrow/go/v13/arrow"
+ "github.com/apache/arrow/go/v13/arrow/memory"
+)
+
+var (
+ ErrMismatchFields = errors.New("arrow/avro: number of records mismatch")
+)
+
+// Option configures an Avro reader/writer.
+type Option func(config)
+type config interface{}
+
+// WithTopLevel specifies whether to include an Avro schema's top level record
+// or only its fields.
+func WithTopLevel(b bool) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.topLevel = b
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithAllocator specifies the Arrow memory allocator used while building records.
+func WithAllocator(mem memory.Allocator) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.mem = mem
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithChunk specifies the chunk size used while reading Avro OCF files.
+//
+// If n is zero or 1, no chunking will take place and the reader will create
+// one record per row.
+// If n is greater than 1, chunks of n rows will be read.
+// If n is negative, the reader will load the whole Avro OCF file into memory and
+// create one big record with all the rows.
+func WithChunk(n int) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.chunk = n
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// DefaultNullValues is the set of values considered as NULL values by default
+// when Reader is configured to handle NULL values.
+var DefaultNullValues = []string{"", "NULL", "null"}
+
+// WithNullReader sets options for a CSV Reader pertaining to NULL value
+// handling. If stringsCanBeNull is true, then a string that matches one of the
+// nullValues set will be interpreted as NULL. Numeric columns will be checked
+// for nulls in all cases. If no nullValues arguments are passed in, the
+// defaults set in NewReader() will be kept.
+//
+// When no NULL values is given, the default set is taken from DefaultNullValues.
+func WithNullReader(stringsCanBeNull bool, nullValues ...string) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.stringsCanBeNull = stringsCanBeNull
+
+ if len(nullValues) == 0 {
+ nullValues = DefaultNullValues
+ }
+ cfg.nulls = make([]string, len(nullValues))
+ copy(cfg.nulls, nullValues)
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithColumnTypes allows specifying optional per-column types (disabling
+// type inference on those columns).
+//
+// Will panic if used in conjunction with an explicit schema.
+func WithColumnTypes(types map[string]arrow.DataType) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ if cfg.schema != nil {
+ panic(fmt.Errorf("%w: cannot use WithColumnTypes with explicit schema", arrow.ErrInvalid))
+ }
+ cfg.columnTypes = types
+ default:
+ panic(fmt.Errorf("%w: WithColumnTypes only allowed for csv reader", arrow.ErrInvalid))
+ }
+ }
+}
+
+// WithIncludeColumns indicates the names of the columns from the AVRO file
+// that should actually be read and converted (in the slice's order).
+// If set and non-empty, columns not in this slice will be ignored.
+//
+// Will panic if used in conjunction with an explicit schema.
+func WithIncludeColumns(cols []string) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ if cfg.schema != nil {
+ panic(fmt.Errorf("%w: cannot use WithIncludeColumns with explicit schema", arrow.ErrInvalid))
+ }
Review Comment:
removed in latest commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] loicalleyne commented on pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne commented on PR #36796:
URL: https://github.com/apache/arrow/pull/36796#issuecomment-1709422930
github.com/hamba/avro/v2 requires go 1.19 so the builds for go 1.17 and go 1.18 will always fail. How many go releases worth of backwards compatibility does arrow need to maintain?
Also, schema.Equal() has a bug which is shown schema_test.go, the string representation is identical but the schema.Equal() comparison fails for some reason.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] loicalleyne commented on a diff in pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne commented on code in PR #36796:
URL: https://github.com/apache/arrow/pull/36796#discussion_r1272844364
##########
go/arrow/avro/schema_test.go:
##########
@@ -0,0 +1,530 @@
+package avro
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on a diff in pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #36796:
URL: https://github.com/apache/arrow/pull/36796#discussion_r1281093354
##########
go/arrow/avro/common.go:
##########
@@ -0,0 +1,162 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package avro
+
+import (
+ "errors"
+ "fmt"
+
+ "github.com/apache/arrow/go/v13/arrow"
+ "github.com/apache/arrow/go/v13/arrow/memory"
+)
+
+var (
+ ErrMismatchFields = errors.New("arrow/avro: number of records mismatch")
+)
+
+// Option configures an Avro reader/writer.
+type Option func(config)
+type config interface{}
Review Comment:
either use a separate config struct or just use the `OCFReader` as the config type, don't bother with it being an interface and doing the type switch. I don't think there's any benefit to doing it this way.
##########
go/arrow/avro/common.go:
##########
@@ -0,0 +1,162 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package avro
+
+import (
+ "errors"
+ "fmt"
+
+ "github.com/apache/arrow/go/v13/arrow"
+ "github.com/apache/arrow/go/v13/arrow/memory"
+)
+
+var (
+ ErrMismatchFields = errors.New("arrow/avro: number of records mismatch")
+)
+
+// Option configures an Avro reader/writer.
+type Option func(config)
+type config interface{}
+
+// WithTopLevel specifies whether to include an Avro schema's top level record
+// or only its fields.
+func WithTopLevel(b bool) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.topLevel = b
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithAllocator specifies the Arrow memory allocator used while building records.
+func WithAllocator(mem memory.Allocator) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.mem = mem
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithChunk specifies the chunk size used while reading Avro OCF files.
+//
+// If n is zero or 1, no chunking will take place and the reader will create
+// one record per row.
+// If n is greater than 1, chunks of n rows will be read.
+// If n is negative, the reader will load the whole Avro OCF file into memory and
+// create one big record with all the rows.
+func WithChunk(n int) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.chunk = n
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// DefaultNullValues is the set of values considered as NULL values by default
+// when Reader is configured to handle NULL values.
+var DefaultNullValues = []string{"", "NULL", "null"}
+
+// WithNullReader sets options for a CSV Reader pertaining to NULL value
+// handling. If stringsCanBeNull is true, then a string that matches one of the
+// nullValues set will be interpreted as NULL. Numeric columns will be checked
+// for nulls in all cases. If no nullValues arguments are passed in, the
+// defaults set in NewReader() will be kept.
+//
+// When no NULL values is given, the default set is taken from DefaultNullValues.
+func WithNullReader(stringsCanBeNull bool, nullValues ...string) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.stringsCanBeNull = stringsCanBeNull
+
+ if len(nullValues) == 0 {
+ nullValues = DefaultNullValues
+ }
+ cfg.nulls = make([]string, len(nullValues))
+ copy(cfg.nulls, nullValues)
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
Review Comment:
see previous comment. doesn't avro have a concept of a null value already so we don't need to have the "null reader" stuff that you need to do for a cSV or other text based reader. (For example, note that there's nothing like this in the JSON reader because if it needs to be null, it should be a JSON null)
##########
go/arrow/avro/common.go:
##########
@@ -0,0 +1,162 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package avro
+
+import (
+ "errors"
+ "fmt"
+
+ "github.com/apache/arrow/go/v13/arrow"
+ "github.com/apache/arrow/go/v13/arrow/memory"
+)
+
+var (
+ ErrMismatchFields = errors.New("arrow/avro: number of records mismatch")
+)
+
+// Option configures an Avro reader/writer.
+type Option func(config)
+type config interface{}
+
+// WithTopLevel specifies whether to include an Avro schema's top level record
+// or only its fields.
+func WithTopLevel(b bool) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.topLevel = b
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithAllocator specifies the Arrow memory allocator used while building records.
+func WithAllocator(mem memory.Allocator) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.mem = mem
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithChunk specifies the chunk size used while reading Avro OCF files.
+//
+// If n is zero or 1, no chunking will take place and the reader will create
+// one record per row.
+// If n is greater than 1, chunks of n rows will be read.
+// If n is negative, the reader will load the whole Avro OCF file into memory and
+// create one big record with all the rows.
+func WithChunk(n int) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.chunk = n
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// DefaultNullValues is the set of values considered as NULL values by default
+// when Reader is configured to handle NULL values.
+var DefaultNullValues = []string{"", "NULL", "null"}
+
+// WithNullReader sets options for a CSV Reader pertaining to NULL value
+// handling. If stringsCanBeNull is true, then a string that matches one of the
+// nullValues set will be interpreted as NULL. Numeric columns will be checked
+// for nulls in all cases. If no nullValues arguments are passed in, the
+// defaults set in NewReader() will be kept.
+//
+// When no NULL values is given, the default set is taken from DefaultNullValues.
+func WithNullReader(stringsCanBeNull bool, nullValues ...string) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.stringsCanBeNull = stringsCanBeNull
+
+ if len(nullValues) == 0 {
+ nullValues = DefaultNullValues
+ }
+ cfg.nulls = make([]string, len(nullValues))
+ copy(cfg.nulls, nullValues)
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithColumnTypes allows specifying optional per-column types (disabling
+// type inference on those columns).
+//
+// Will panic if used in conjunction with an explicit schema.
+func WithColumnTypes(types map[string]arrow.DataType) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ if cfg.schema != nil {
+ panic(fmt.Errorf("%w: cannot use WithColumnTypes with explicit schema", arrow.ErrInvalid))
+ }
+ cfg.columnTypes = types
+ default:
+ panic(fmt.Errorf("%w: WithColumnTypes only allowed for csv reader", arrow.ErrInvalid))
+ }
+ }
+}
+
+// WithIncludeColumns indicates the names of the columns from the AVRO file
+// that should actually be read and converted (in the slice's order).
+// If set and non-empty, columns not in this slice will be ignored.
+//
+// Will panic if used in conjunction with an explicit schema.
+func WithIncludeColumns(cols []string) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ if cfg.schema != nil {
+ panic(fmt.Errorf("%w: cannot use WithIncludeColumns with explicit schema", arrow.ErrInvalid))
+ }
+ cfg.columnFilter = cols
+ default:
+ panic(fmt.Errorf("%w: WithIncludeColumns only allowed on csv Reader", arrow.ErrInvalid))
+ }
+ }
+}
+
+func validate(schema *arrow.Schema) {
+ for i, f := range schema.Fields() {
+ switch ft := f.Type.(type) {
+ case *arrow.BooleanType:
+ case *arrow.Int8Type, *arrow.Int16Type, *arrow.Int32Type, *arrow.Int64Type:
+ case *arrow.Uint8Type, *arrow.Uint16Type, *arrow.Uint32Type, *arrow.Uint64Type:
+ case *arrow.Float16Type, *arrow.Float32Type, *arrow.Float64Type:
+ case *arrow.StringType, *arrow.LargeStringType:
+ case *arrow.TimestampType:
+ case *arrow.Date32Type, *arrow.Date64Type:
+ case *arrow.Decimal128Type, *arrow.Decimal256Type:
+ case *arrow.ListType, *arrow.LargeListType, *arrow.FixedSizeListType:
+ case *arrow.BinaryType, *arrow.LargeBinaryType, *arrow.FixedSizeBinaryType:
+ case arrow.ExtensionType:
+ case *arrow.NullType:
+ default:
+ panic(fmt.Errorf("arrow/csv: field %d (%s) has invalid data type %T", i, f.Name, ft))
+ }
+ }
+}
Review Comment:
should we even allow specifying an arrow schema rather than performing the conversion?
##########
go/arrow/avro/schema.go:
##########
@@ -0,0 +1,409 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package avro reads Avro OCF files and presents the extracted data as records
+package avro
+
+import (
+ "encoding/json"
+ "fmt"
+ "math"
+ "strconv"
+
+ "github.com/apache/arrow/go/v13/arrow"
+)
+
+type schemaNode struct {
+ name string
+ ofType interface{}
+ logicalType string
+ precision int32
+ scale int32
+ size int
+ symbols []string
+ fields []interface{}
+}
+
+// ArrowSchemaFromAvro returns a new Arrow schema from an Avro schema JSON.
+// If the top level is of record type, set includeTopLevel to either make
+// its fields top level fields in the resulting schema or nested in a single field.
+func ArrowSchemaFromAvro(avroSchema []byte) (*arrow.Schema, error) {
+ var m map[string]interface{}
+ var node schemaNode
+ json.Unmarshal(avroSchema, &m)
+ if m["type"].(string) == "record" {
+ if _, ok := m["fields"]; ok {
+ for _, field := range m["fields"].([]interface{}) {
+ node.fields = append(node.fields, field.(map[string]interface{}))
+ }
+ if len(node.fields) == 0 {
+ return nil, fmt.Errorf("invalid avro schema: no top level record fields found")
+ }
+ }
+ }
+ fields := iterateFields(node.fields)
+ return arrow.NewSchema(fields, nil), nil
+}
Review Comment:
Avro has a well defined spec for its JSON schema spec, why not create a struct to represent it and utilize `json` struct tags + `UnmarshalJSON` / `MarshalJSON` functions to simplify the unmarshalling?
Even better, https://pkg.go.dev/github.com/hamba/avro/v2#RecordSchema has a Parse function that will return objects which represent the schema for you allowing you to likely greatly simplify the `iterateFields` code to instead just iterate the objects created from the parsing.
##########
go/arrow/avro/schema.go:
##########
@@ -0,0 +1,409 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package avro reads Avro OCF files and presents the extracted data as records
+package avro
+
+import (
+ "encoding/json"
+ "fmt"
+ "math"
+ "strconv"
+
+ "github.com/apache/arrow/go/v13/arrow"
+)
+
+type schemaNode struct {
+ name string
+ ofType interface{}
+ logicalType string
+ precision int32
+ scale int32
+ size int
+ symbols []string
+ fields []interface{}
+}
+
+// ArrowSchemaFromAvro returns a new Arrow schema from an Avro schema JSON.
+// If the top level is of record type, set includeTopLevel to either make
+// its fields top level fields in the resulting schema or nested in a single field.
+func ArrowSchemaFromAvro(avroSchema []byte) (*arrow.Schema, error) {
+ var m map[string]interface{}
+ var node schemaNode
+ json.Unmarshal(avroSchema, &m)
+ if m["type"].(string) == "record" {
+ if _, ok := m["fields"]; ok {
+ for _, field := range m["fields"].([]interface{}) {
+ node.fields = append(node.fields, field.(map[string]interface{}))
+ }
+ if len(node.fields) == 0 {
+ return nil, fmt.Errorf("invalid avro schema: no top level record fields found")
+ }
+ }
+ }
+ fields := iterateFields(node.fields)
+ return arrow.NewSchema(fields, nil), nil
+}
+
+func iterateFields(f []interface{}) []arrow.Field {
+ var s []arrow.Field
+ for _, field := range f {
+ var n schemaNode
+ n.name = field.(map[string]interface{})["name"].(string)
+ n.ofType = field.(map[string]interface{})["type"]
+ switch n.ofType.(type) {
+ // Getting field type from within field object
+ case string:
+ switch n.ofType.(string) {
+ case "enum":
+ for _, symbol := range field.(map[string]interface{})["symbols"].([]interface{}) {
+ n.symbols = append(n.symbols, symbol.(string))
+ }
+ default:
+ if lt, ok := field.(map[string]interface{})["logicalType"]; ok {
+ n.logicalType = lt.(string)
+ }
+ if lt, ok := field.(map[string]interface{})["size"]; ok {
+ n.size = int(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["precision"]; ok {
+ n.precision = int32(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["scale"]; ok {
+ n.scale = int32(lt.(float64))
+ }
+ }
+ // Field type is an object
+ case map[string]interface{}:
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["logicalType"]; ok {
+ n.logicalType = lt.(string)
+ }
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["size"]; ok {
+ n.size = int(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["precision"]; ok {
+ n.precision = int32(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["scale"]; ok {
+ n.scale = int32(lt.(float64))
+ }
+ case []interface{}:
+
+ default:
+ if lt, ok := field.(map[string]interface{})["logicalType"]; ok {
+ n.logicalType = lt.(string)
+ }
+ if lt, ok := field.(map[string]interface{})["size"]; ok {
+ n.size = int(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["precision"]; ok {
+ n.precision = int32(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["scale"]; ok {
+ n.scale = int32(lt.(float64))
+ }
+ }
+ // Field is of type "record"
+ if nf, f := field.(map[string]interface{})["fields"]; f {
+ switch nf.(type) {
+ // primitive & complex types
+ case map[string]interface{}:
+ for _, v := range nf.(map[string]interface{})["fields"].([]interface{}) {
+ n.fields = append(n.fields, v.(map[string]interface{}))
+ }
+ // type unions
+ default:
+ for _, v := range nf.([]interface{}) {
+ n.fields = append(n.fields, v.(map[string]interface{}))
+ }
+ }
+ }
+ s = append(s, traverseNodes(n))
+ }
+ return s
+}
+
+func traverseNodes(node schemaNode) arrow.Field {
+ switch node.ofType.(type) {
+ case string:
+ // Avro primitive type
+ if len(node.fields) == 0 {
+ switch node.ofType.(string) {
+ case "boolean", "int", "long", "float", "double", "bytes", "string":
+ if node.logicalType != "" {
+ return avroLogicalToArrowField(node)
+ }
+ // Avro primitive type
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(node.ofType.(string))}
+ case "fixed":
+ // Duration type is not supported in github.com/linkedin/goavro
+ // Implementing as Binary for now.
+ switch node.logicalType {
+ case "decimal":
+ return avroLogicalToArrowField(node)
+ case "duration":
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.Binary}
+ //return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.MonthDayNanoInterval}
+ }
+ return arrow.Field{Name: node.name, Type: &arrow.FixedSizeBinaryType{ByteWidth: node.size}}
+ case "enum":
+ symbols := make(map[string]string)
+ for index, symbol := range node.symbols {
+ k := strconv.FormatInt(int64(index), 10)
+ symbols[k] = symbol
+ }
+ var dt arrow.DictionaryType = arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Uint64, ValueType: arrow.BinaryTypes.String, Ordered: false}
+ sl := len(symbols)
+ switch {
+ case sl <= math.MaxUint8:
+ dt.IndexType = arrow.PrimitiveTypes.Uint8
+ case sl > math.MaxUint8 && sl <= math.MaxUint16:
+ dt.IndexType = arrow.PrimitiveTypes.Uint16
+ case sl > math.MaxUint16 && sl <= math.MaxUint32:
+ dt.IndexType = arrow.PrimitiveTypes.Uint32
+ }
+ return arrow.Field{Name: node.name, Type: &dt, Nullable: true, Metadata: arrow.MetadataFrom(symbols)}
+ default:
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(node.ofType.(string))}
+ }
+ } else {
+ // avro "record" type, node has "fields" array
+ if node.ofType.(string) == "record" {
+ var n schemaNode
+ n.name = node.name
+ n.ofType = node.ofType
+ if len(node.fields) > 0 {
+ n.fields = append(n.fields, node.fields...)
+ }
+ f := iterateFields(n.fields)
+ return arrow.Field{Name: node.name, Type: arrow.StructOf(f...)}
+ }
+ }
+ // Avro complex types
+ case map[string]interface{}:
+ //var n schemaNode
+ //n.name = node.name
+ //n.ofType = node.ofType.(map[string]interface{})["type"]
+ switch node.logicalType {
+ case "":
+ return avroComplexToArrowField(node)
+ default:
+ return avroLogicalToArrowField(node)
+ }
+
+ // Avro union types
+ case []interface{}:
+ var unionTypes []string
+ for _, ft := range node.ofType.([]interface{}) {
+ switch ft.(type) {
+ // primitive types
+ case string:
+ if ft != "null" {
+ unionTypes = append(unionTypes, ft.(string))
+ }
+ continue
+ // complex types
+ case map[string]interface{}:
+ var n schemaNode
+ n.name = node.name
+ n.ofType = ft.(map[string]interface{})["type"]
+ if _, f := ft.(map[string]interface{})["fields"]; f {
+ for _, field := range ft.(map[string]interface{})["fields"].([]interface{}) {
+ n.fields = append(n.fields, field.(map[string]interface{}))
+ }
+ }
+ f := iterateFields(n.fields)
+ return arrow.Field{Name: node.name, Type: arrow.StructOf(f...)}
+ }
+ }
+ // Supported Avro union type is null + one other type.
+ // TODO: Complex AVRO union to Arrow Dense || Sparse Union.
+ if len(unionTypes) == 1 {
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(unionTypes[0])}
+ } else {
+ // BYTE_ARRAY is the catchall if union type is anything beyond null + one other type.
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.Binary}
+ }
+ }
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.Binary}
+}
+
+func avroLogicalToArrowField(node schemaNode) arrow.Field {
+ // Avro logical types
+ switch node.logicalType {
+ // The decimal logical type represents an arbitrary-precision signed decimal number of the form unscaled × 10-scale.
+ // A decimal logical type annotates Avro bytes or fixed types. The byte array must contain the two’s-complement
+ // representation of the unscaled integer value in big-endian byte order. The scale is fixed, and is specified
+ // using an attribute.
+ //
+ // The following attributes are supported:
+ // scale, a JSON integer representing the scale (optional). If not specified the scale is 0.
+ // precision, a JSON integer representing the (maximum) precision of decimals stored in this type (required).
+ case "decimal":
+ if node.precision <= 38 {
+ return arrow.Field{Name: node.name, Type: &arrow.Decimal128Type{Precision: node.precision, Scale: node.scale}}
+ } else {
+ return arrow.Field{Name: node.name, Type: &arrow.Decimal256Type{Precision: node.precision, Scale: node.scale}}
+ }
+
+ // The uuid logical type represents a random generated universally unique identifier (UUID).
+ // A uuid logical type annotates an Avro string. The string has to conform with RFC-4122
+ case "uuid":
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.String}
+
+ // The date logical type represents a date within the calendar, with no reference to a particular
+ // time zone or time of day.
+ // A date logical type annotates an Avro int, where the int stores the number of days from the unix epoch,
+ // 1 January 1970 (ISO calendar).
+ case "date":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Date32}
+
+ // The time-millis logical type represents a time of day, with no reference to a particular calendar,
+ // time zone or date, with a precision of one millisecond.
+ // A time-millis logical type annotates an Avro int, where the int stores the number of milliseconds
+ // after midnight, 00:00:00.000.
+ case "time-millis":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Time32ms}
+
+ // The time-micros logical type represents a time of day, with no reference to a particular calendar,
+ // time zone or date, with a precision of one microsecond.
+ // A time-micros logical type annotates an Avro long, where the long stores the number of microseconds
+ // after midnight, 00:00:00.000000.
+ case "time-micros":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Time64us}
+
+ // The timestamp-millis logical type represents an instant on the global timeline, independent of a
+ // particular time zone or calendar, with a precision of one millisecond. Please note that time zone
+ // information gets lost in this process. Upon reading a value back, we can only reconstruct the instant,
+ // but not the original representation. In practice, such timestamps are typically displayed to users in
+ // their local time zones, therefore they may be displayed differently depending on the execution environment.
+ // A timestamp-millis logical type annotates an Avro long, where the long stores the number of milliseconds
+ // from the unix epoch, 1 January 1970 00:00:00.000 UTC.
+ case "timestamp-millis":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Timestamp_ms}
+ // The timestamp-micros logical type represents an instant on the global timeline, independent of a
+ // particular time zone or calendar, with a precision of one microsecond. Please note that time zone
+ // information gets lost in this process. Upon reading a value back, we can only reconstruct the instant,
+ // but not the original representation. In practice, such timestamps are typically displayed to users
+ // in their local time zones, therefore they may be displayed differently depending on the execution environment.
+ // A timestamp-micros logical type annotates an Avro long, where the long stores the number of microseconds
+ // from the unix epoch, 1 January 1970 00:00:00.000000 UTC.
+ case "timestamp-micros":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Timestamp_us}
+ // The local-timestamp-millis logical type represents a timestamp in a local timezone, regardless of
+ // what specific time zone is considered local, with a precision of one millisecond.
+ // A local-timestamp-millis logical type annotates an Avro long, where the long stores the number of
+ // milliseconds, from 1 January 1970 00:00:00.000.
+ case "local-timestamp-millis":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Timestamp_ms}
+ // The local-timestamp-micros logical type represents a timestamp in a local timezone, regardless of
+ // what specific time zone is considered local, with a precision of one microsecond.
+ // A local-timestamp-micros logical type annotates an Avro long, where the long stores the number of
+ // microseconds, from 1 January 1970 00:00:00.000000.
+ case "local-timestamp-micros":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Timestamp_us}
+ // Avro primitive type
+ default:
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(node.ofType.(string))}
+ }
+ return arrow.Field{}
Review Comment:
shouldn't this be an error?
##########
go/arrow/avro/schema.go:
##########
@@ -0,0 +1,409 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package avro reads Avro OCF files and presents the extracted data as records
+package avro
+
+import (
+ "encoding/json"
+ "fmt"
+ "math"
+ "strconv"
+
+ "github.com/apache/arrow/go/v13/arrow"
+)
+
+type schemaNode struct {
+ name string
+ ofType interface{}
+ logicalType string
+ precision int32
+ scale int32
+ size int
+ symbols []string
+ fields []interface{}
+}
+
+// ArrowSchemaFromAvro returns a new Arrow schema from an Avro schema JSON.
+// If the top level is of record type, set includeTopLevel to either make
+// its fields top level fields in the resulting schema or nested in a single field.
+func ArrowSchemaFromAvro(avroSchema []byte) (*arrow.Schema, error) {
+ var m map[string]interface{}
+ var node schemaNode
+ json.Unmarshal(avroSchema, &m)
+ if m["type"].(string) == "record" {
+ if _, ok := m["fields"]; ok {
+ for _, field := range m["fields"].([]interface{}) {
+ node.fields = append(node.fields, field.(map[string]interface{}))
+ }
+ if len(node.fields) == 0 {
+ return nil, fmt.Errorf("invalid avro schema: no top level record fields found")
+ }
+ }
+ }
+ fields := iterateFields(node.fields)
+ return arrow.NewSchema(fields, nil), nil
+}
+
+func iterateFields(f []interface{}) []arrow.Field {
+ var s []arrow.Field
+ for _, field := range f {
+ var n schemaNode
+ n.name = field.(map[string]interface{})["name"].(string)
+ n.ofType = field.(map[string]interface{})["type"]
+ switch n.ofType.(type) {
+ // Getting field type from within field object
+ case string:
+ switch n.ofType.(string) {
+ case "enum":
+ for _, symbol := range field.(map[string]interface{})["symbols"].([]interface{}) {
+ n.symbols = append(n.symbols, symbol.(string))
+ }
+ default:
+ if lt, ok := field.(map[string]interface{})["logicalType"]; ok {
+ n.logicalType = lt.(string)
+ }
+ if lt, ok := field.(map[string]interface{})["size"]; ok {
+ n.size = int(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["precision"]; ok {
+ n.precision = int32(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["scale"]; ok {
+ n.scale = int32(lt.(float64))
+ }
+ }
+ // Field type is an object
+ case map[string]interface{}:
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["logicalType"]; ok {
+ n.logicalType = lt.(string)
+ }
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["size"]; ok {
+ n.size = int(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["precision"]; ok {
+ n.precision = int32(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["scale"]; ok {
+ n.scale = int32(lt.(float64))
+ }
+ case []interface{}:
+
+ default:
+ if lt, ok := field.(map[string]interface{})["logicalType"]; ok {
+ n.logicalType = lt.(string)
+ }
+ if lt, ok := field.(map[string]interface{})["size"]; ok {
+ n.size = int(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["precision"]; ok {
+ n.precision = int32(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["scale"]; ok {
+ n.scale = int32(lt.(float64))
+ }
+ }
+ // Field is of type "record"
+ if nf, f := field.(map[string]interface{})["fields"]; f {
+ switch nf.(type) {
+ // primitive & complex types
+ case map[string]interface{}:
+ for _, v := range nf.(map[string]interface{})["fields"].([]interface{}) {
+ n.fields = append(n.fields, v.(map[string]interface{}))
+ }
+ // type unions
+ default:
+ for _, v := range nf.([]interface{}) {
+ n.fields = append(n.fields, v.(map[string]interface{}))
+ }
+ }
+ }
+ s = append(s, traverseNodes(n))
+ }
+ return s
+}
+
+func traverseNodes(node schemaNode) arrow.Field {
+ switch node.ofType.(type) {
+ case string:
+ // Avro primitive type
+ if len(node.fields) == 0 {
+ switch node.ofType.(string) {
+ case "boolean", "int", "long", "float", "double", "bytes", "string":
+ if node.logicalType != "" {
+ return avroLogicalToArrowField(node)
+ }
+ // Avro primitive type
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(node.ofType.(string))}
+ case "fixed":
+ // Duration type is not supported in github.com/linkedin/goavro
+ // Implementing as Binary for now.
+ switch node.logicalType {
+ case "decimal":
+ return avroLogicalToArrowField(node)
+ case "duration":
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.Binary}
+ //return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.MonthDayNanoInterval}
+ }
+ return arrow.Field{Name: node.name, Type: &arrow.FixedSizeBinaryType{ByteWidth: node.size}}
+ case "enum":
+ symbols := make(map[string]string)
+ for index, symbol := range node.symbols {
+ k := strconv.FormatInt(int64(index), 10)
+ symbols[k] = symbol
+ }
+ var dt arrow.DictionaryType = arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Uint64, ValueType: arrow.BinaryTypes.String, Ordered: false}
+ sl := len(symbols)
+ switch {
+ case sl <= math.MaxUint8:
+ dt.IndexType = arrow.PrimitiveTypes.Uint8
+ case sl > math.MaxUint8 && sl <= math.MaxUint16:
+ dt.IndexType = arrow.PrimitiveTypes.Uint16
+ case sl > math.MaxUint16 && sl <= math.MaxUint32:
+ dt.IndexType = arrow.PrimitiveTypes.Uint32
+ }
+ return arrow.Field{Name: node.name, Type: &dt, Nullable: true, Metadata: arrow.MetadataFrom(symbols)}
+ default:
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(node.ofType.(string))}
+ }
+ } else {
+ // avro "record" type, node has "fields" array
+ if node.ofType.(string) == "record" {
+ var n schemaNode
+ n.name = node.name
+ n.ofType = node.ofType
+ if len(node.fields) > 0 {
+ n.fields = append(n.fields, node.fields...)
+ }
+ f := iterateFields(n.fields)
+ return arrow.Field{Name: node.name, Type: arrow.StructOf(f...)}
+ }
+ }
+ // Avro complex types
+ case map[string]interface{}:
+ //var n schemaNode
+ //n.name = node.name
+ //n.ofType = node.ofType.(map[string]interface{})["type"]
+ switch node.logicalType {
+ case "":
+ return avroComplexToArrowField(node)
+ default:
+ return avroLogicalToArrowField(node)
+ }
+
+ // Avro union types
+ case []interface{}:
+ var unionTypes []string
+ for _, ft := range node.ofType.([]interface{}) {
+ switch ft.(type) {
+ // primitive types
+ case string:
+ if ft != "null" {
+ unionTypes = append(unionTypes, ft.(string))
+ }
+ continue
+ // complex types
+ case map[string]interface{}:
+ var n schemaNode
+ n.name = node.name
+ n.ofType = ft.(map[string]interface{})["type"]
+ if _, f := ft.(map[string]interface{})["fields"]; f {
+ for _, field := range ft.(map[string]interface{})["fields"].([]interface{}) {
+ n.fields = append(n.fields, field.(map[string]interface{}))
+ }
+ }
+ f := iterateFields(n.fields)
+ return arrow.Field{Name: node.name, Type: arrow.StructOf(f...)}
+ }
+ }
+ // Supported Avro union type is null + one other type.
+ // TODO: Complex AVRO union to Arrow Dense || Sparse Union.
+ if len(unionTypes) == 1 {
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(unionTypes[0])}
+ } else {
+ // BYTE_ARRAY is the catchall if union type is anything beyond null + one other type.
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.Binary}
+ }
+ }
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.Binary}
+}
+
+func avroLogicalToArrowField(node schemaNode) arrow.Field {
+ // Avro logical types
+ switch node.logicalType {
+ // The decimal logical type represents an arbitrary-precision signed decimal number of the form unscaled × 10-scale.
+ // A decimal logical type annotates Avro bytes or fixed types. The byte array must contain the two’s-complement
+ // representation of the unscaled integer value in big-endian byte order. The scale is fixed, and is specified
+ // using an attribute.
+ //
+ // The following attributes are supported:
+ // scale, a JSON integer representing the scale (optional). If not specified the scale is 0.
+ // precision, a JSON integer representing the (maximum) precision of decimals stored in this type (required).
+ case "decimal":
+ if node.precision <= 38 {
+ return arrow.Field{Name: node.name, Type: &arrow.Decimal128Type{Precision: node.precision, Scale: node.scale}}
+ } else {
+ return arrow.Field{Name: node.name, Type: &arrow.Decimal256Type{Precision: node.precision, Scale: node.scale}}
+ }
+
+ // The uuid logical type represents a random generated universally unique identifier (UUID).
+ // A uuid logical type annotates an Avro string. The string has to conform with RFC-4122
+ case "uuid":
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.String}
+
+ // The date logical type represents a date within the calendar, with no reference to a particular
+ // time zone or time of day.
+ // A date logical type annotates an Avro int, where the int stores the number of days from the unix epoch,
+ // 1 January 1970 (ISO calendar).
+ case "date":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Date32}
+
+ // The time-millis logical type represents a time of day, with no reference to a particular calendar,
+ // time zone or date, with a precision of one millisecond.
+ // A time-millis logical type annotates an Avro int, where the int stores the number of milliseconds
+ // after midnight, 00:00:00.000.
+ case "time-millis":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Time32ms}
+
+ // The time-micros logical type represents a time of day, with no reference to a particular calendar,
+ // time zone or date, with a precision of one microsecond.
+ // A time-micros logical type annotates an Avro long, where the long stores the number of microseconds
+ // after midnight, 00:00:00.000000.
+ case "time-micros":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Time64us}
+
+ // The timestamp-millis logical type represents an instant on the global timeline, independent of a
+ // particular time zone or calendar, with a precision of one millisecond. Please note that time zone
+ // information gets lost in this process. Upon reading a value back, we can only reconstruct the instant,
+ // but not the original representation. In practice, such timestamps are typically displayed to users in
+ // their local time zones, therefore they may be displayed differently depending on the execution environment.
+ // A timestamp-millis logical type annotates an Avro long, where the long stores the number of milliseconds
+ // from the unix epoch, 1 January 1970 00:00:00.000 UTC.
+ case "timestamp-millis":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Timestamp_ms}
+ // The timestamp-micros logical type represents an instant on the global timeline, independent of a
+ // particular time zone or calendar, with a precision of one microsecond. Please note that time zone
+ // information gets lost in this process. Upon reading a value back, we can only reconstruct the instant,
+ // but not the original representation. In practice, such timestamps are typically displayed to users
+ // in their local time zones, therefore they may be displayed differently depending on the execution environment.
+ // A timestamp-micros logical type annotates an Avro long, where the long stores the number of microseconds
+ // from the unix epoch, 1 January 1970 00:00:00.000000 UTC.
+ case "timestamp-micros":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Timestamp_us}
+ // The local-timestamp-millis logical type represents a timestamp in a local timezone, regardless of
+ // what specific time zone is considered local, with a precision of one millisecond.
+ // A local-timestamp-millis logical type annotates an Avro long, where the long stores the number of
+ // milliseconds, from 1 January 1970 00:00:00.000.
+ case "local-timestamp-millis":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Timestamp_ms}
+ // The local-timestamp-micros logical type represents a timestamp in a local timezone, regardless of
+ // what specific time zone is considered local, with a precision of one microsecond.
+ // A local-timestamp-micros logical type annotates an Avro long, where the long stores the number of
+ // microseconds, from 1 January 1970 00:00:00.000000.
+ case "local-timestamp-micros":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Timestamp_us}
+ // Avro primitive type
+ default:
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(node.ofType.(string))}
+ }
+ return arrow.Field{}
+}
+
+func avroComplexToArrowField(node schemaNode) arrow.Field {
+ var n schemaNode
+ n.name = node.name
+ n.ofType = node.ofType.(map[string]interface{})["type"]
+ // Avro "array" field type
+ if i, ok := node.ofType.(map[string]interface{})["items"]; ok {
+ switch i.(string) {
+ case "int", "long", "float", "double", "bytes", "boolean", "string":
+ return arrow.Field{Name: node.name, Type: arrow.ListOf(AvroPrimitiveToArrowType(i.(string)))}
+ case "enum", "fixed", "map", "record", "array":
+ return arrow.Field{Name: node.name, Type: arrow.ListOf(avroComplexToArrowField(n).Type)}
+ case "decimal", "uuid", "date", "time-millis", "time-micros", "timestamp-millis", "timestamp-micros", "local-timestamp-millis", "local-timestamp-micros":
+ return arrow.Field{Name: node.name, Type: arrow.ListOf(avroLogicalToArrowField(n).Type)}
+ }
+ }
+ // Avro "enum" field type = Arrow dictionary type
+ if i, ok := node.ofType.(map[string]interface{})["symbols"]; ok {
+ symbols := make(map[string]string)
+ for index, symbol := range i.([]interface{}) {
+ k := strconv.FormatInt(int64(index), 10)
+ symbols[k] = symbol.(string)
+ }
+ var dt arrow.DictionaryType = arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Uint64, ValueType: arrow.BinaryTypes.String, Ordered: false}
+ sl := len(symbols)
+ switch {
+ case sl <= math.MaxUint8:
+ dt.IndexType = arrow.PrimitiveTypes.Uint8
+ case sl > math.MaxUint8 && sl <= math.MaxUint16:
+ dt.IndexType = arrow.PrimitiveTypes.Uint16
+ case sl > math.MaxUint16 && sl <= math.MaxUint32:
+ dt.IndexType = arrow.PrimitiveTypes.Uint32
+ }
+ return arrow.Field{Name: node.name, Type: &dt, Nullable: true, Metadata: arrow.MetadataFrom(symbols)}
+ }
+ // Avro "fixed" field type = Arrow FixedSize Primitive BinaryType
+ if i, ok := node.ofType.(map[string]interface{})["size"]; ok {
+ return arrow.Field{Name: node.name, Type: &arrow.FixedSizeBinaryType{ByteWidth: int(i.(float64))}}
+ }
+ // Avro "map" field type
+ if i, ok := node.ofType.(map[string]interface{})["values"]; ok {
+ return arrow.Field{Name: node.name, Type: arrow.MapOf(arrow.BinaryTypes.String, AvroPrimitiveToArrowType(i.(string)))}
+ }
+ // Avro "record" field type
+ if _, f := node.ofType.(map[string]interface{})["fields"]; f {
+ for _, field := range node.ofType.(map[string]interface{})["fields"].([]interface{}) {
+ n.fields = append(n.fields, field.(map[string]interface{}))
+ }
+ s := iterateFields(n.fields)
+ return arrow.Field{Name: n.name, Type: arrow.StructOf(s...)}
+ }
+ return arrow.Field{}
+}
+
+// AvroPrimitiveToArrowType returns the Arrow DataType equivalent to a
+// Avro primitive type.
+//
+// NOTE: Arrow Binary type is used as a catchall to avoid potential data loss.
+func AvroPrimitiveToArrowType(avroFieldType string) arrow.DataType {
+ switch avroFieldType {
+ // int: 32-bit signed integer
+ case "int":
+ return arrow.PrimitiveTypes.Int32
+ // long: 64-bit signed integer
+ case "long":
+ return arrow.PrimitiveTypes.Int64
+ // float: single precision (32-bit) IEEE 754 floating-point number
+ case "float":
+ return arrow.PrimitiveTypes.Float32
+ // double: double precision (64-bit) IEEE 754 floating-point number
+ case "double":
+ return arrow.PrimitiveTypes.Float64
+ // bytes: sequence of 8-bit unsigned bytes
+ case "bytes":
+ return arrow.BinaryTypes.Binary
+ // boolean: a binary value
+ case "boolean":
+ return arrow.FixedWidthTypes.Boolean
+ // string: unicode character sequence
+ case "string":
+ return arrow.BinaryTypes.String
+ // fallback to binary type for any unsupported type
+ default:
+ return arrow.BinaryTypes.Binary
Review Comment:
I'd prefer an error with `arrow.ErrNotImplemented` for any unsupported type
##########
go/go.mod:
##########
@@ -59,6 +59,7 @@ require (
github.com/golang/protobuf v1.5.2 // indirect
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 // indirect
github.com/kr/text v0.2.0 // indirect
+ github.com/linkedin/goavro/v2 v2.12.0 // indirect
Review Comment:
Might I suggest `github.com/hamba/avro/v2/` as it seems to be about an order of magnitude faster than goavro....?
##########
go/arrow/avro/common.go:
##########
@@ -0,0 +1,162 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package avro
+
+import (
+ "errors"
+ "fmt"
+
+ "github.com/apache/arrow/go/v13/arrow"
+ "github.com/apache/arrow/go/v13/arrow/memory"
+)
+
+var (
+ ErrMismatchFields = errors.New("arrow/avro: number of records mismatch")
+)
+
+// Option configures an Avro reader/writer.
+type Option func(config)
+type config interface{}
+
+// WithTopLevel specifies whether to include an Avro schema's top level record
+// or only its fields.
+func WithTopLevel(b bool) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.topLevel = b
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithAllocator specifies the Arrow memory allocator used while building records.
+func WithAllocator(mem memory.Allocator) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.mem = mem
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithChunk specifies the chunk size used while reading Avro OCF files.
+//
+// If n is zero or 1, no chunking will take place and the reader will create
+// one record per row.
+// If n is greater than 1, chunks of n rows will be read.
+// If n is negative, the reader will load the whole Avro OCF file into memory and
+// create one big record with all the rows.
+func WithChunk(n int) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.chunk = n
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// DefaultNullValues is the set of values considered as NULL values by default
+// when Reader is configured to handle NULL values.
+var DefaultNullValues = []string{"", "NULL", "null"}
+
+// WithNullReader sets options for a CSV Reader pertaining to NULL value
+// handling. If stringsCanBeNull is true, then a string that matches one of the
+// nullValues set will be interpreted as NULL. Numeric columns will be checked
+// for nulls in all cases. If no nullValues arguments are passed in, the
+// defaults set in NewReader() will be kept.
+//
+// When no NULL values is given, the default set is taken from DefaultNullValues.
+func WithNullReader(stringsCanBeNull bool, nullValues ...string) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.stringsCanBeNull = stringsCanBeNull
+
+ if len(nullValues) == 0 {
+ nullValues = DefaultNullValues
+ }
+ cfg.nulls = make([]string, len(nullValues))
+ copy(cfg.nulls, nullValues)
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithColumnTypes allows specifying optional per-column types (disabling
+// type inference on those columns).
+//
+// Will panic if used in conjunction with an explicit schema.
+func WithColumnTypes(types map[string]arrow.DataType) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ if cfg.schema != nil {
+ panic(fmt.Errorf("%w: cannot use WithColumnTypes with explicit schema", arrow.ErrInvalid))
+ }
+ cfg.columnTypes = types
+ default:
+ panic(fmt.Errorf("%w: WithColumnTypes only allowed for csv reader", arrow.ErrInvalid))
+ }
+ }
+}
+
+// WithIncludeColumns indicates the names of the columns from the AVRO file
+// that should actually be read and converted (in the slice's order).
+// If set and non-empty, columns not in this slice will be ignored.
+//
+// Will panic if used in conjunction with an explicit schema.
+func WithIncludeColumns(cols []string) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ if cfg.schema != nil {
+ panic(fmt.Errorf("%w: cannot use WithIncludeColumns with explicit schema", arrow.ErrInvalid))
+ }
+ cfg.columnFilter = cols
+ default:
+ panic(fmt.Errorf("%w: WithIncludeColumns only allowed on csv Reader", arrow.ErrInvalid))
+ }
+ }
+}
Review Comment:
I believe that Avro isn't order based, it's name and field based which means that you can probably get away with reading this into a `map[string]struct{}` to create a "set" of fields.
##########
go/arrow/avro/common.go:
##########
@@ -0,0 +1,162 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package avro
+
+import (
+ "errors"
+ "fmt"
+
+ "github.com/apache/arrow/go/v13/arrow"
+ "github.com/apache/arrow/go/v13/arrow/memory"
+)
+
+var (
+ ErrMismatchFields = errors.New("arrow/avro: number of records mismatch")
+)
+
+// Option configures an Avro reader/writer.
+type Option func(config)
+type config interface{}
+
+// WithTopLevel specifies whether to include an Avro schema's top level record
+// or only its fields.
+func WithTopLevel(b bool) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.topLevel = b
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithAllocator specifies the Arrow memory allocator used while building records.
+func WithAllocator(mem memory.Allocator) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.mem = mem
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithChunk specifies the chunk size used while reading Avro OCF files.
+//
+// If n is zero or 1, no chunking will take place and the reader will create
+// one record per row.
+// If n is greater than 1, chunks of n rows will be read.
+// If n is negative, the reader will load the whole Avro OCF file into memory and
+// create one big record with all the rows.
+func WithChunk(n int) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.chunk = n
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// DefaultNullValues is the set of values considered as NULL values by default
+// when Reader is configured to handle NULL values.
+var DefaultNullValues = []string{"", "NULL", "null"}
+
+// WithNullReader sets options for a CSV Reader pertaining to NULL value
+// handling. If stringsCanBeNull is true, then a string that matches one of the
+// nullValues set will be interpreted as NULL. Numeric columns will be checked
+// for nulls in all cases. If no nullValues arguments are passed in, the
+// defaults set in NewReader() will be kept.
+//
+// When no NULL values is given, the default set is taken from DefaultNullValues.
+func WithNullReader(stringsCanBeNull bool, nullValues ...string) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.stringsCanBeNull = stringsCanBeNull
+
+ if len(nullValues) == 0 {
+ nullValues = DefaultNullValues
+ }
+ cfg.nulls = make([]string, len(nullValues))
+ copy(cfg.nulls, nullValues)
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithColumnTypes allows specifying optional per-column types (disabling
+// type inference on those columns).
+//
+// Will panic if used in conjunction with an explicit schema.
+func WithColumnTypes(types map[string]arrow.DataType) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ if cfg.schema != nil {
+ panic(fmt.Errorf("%w: cannot use WithColumnTypes with explicit schema", arrow.ErrInvalid))
+ }
+ cfg.columnTypes = types
+ default:
+ panic(fmt.Errorf("%w: WithColumnTypes only allowed for csv reader", arrow.ErrInvalid))
+ }
+ }
+}
Review Comment:
We don't need type inference since Avro has a specific typed schema, so this is unnecessary entirely. We should just define what the mapping between Avro types and Arrow types are.
##########
go/arrow/avro/schema.go:
##########
@@ -0,0 +1,409 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package avro reads Avro OCF files and presents the extracted data as records
+package avro
+
+import (
+ "encoding/json"
+ "fmt"
+ "math"
+ "strconv"
+
+ "github.com/apache/arrow/go/v13/arrow"
+)
+
+type schemaNode struct {
+ name string
+ ofType interface{}
+ logicalType string
+ precision int32
+ scale int32
+ size int
+ symbols []string
+ fields []interface{}
+}
+
+// ArrowSchemaFromAvro returns a new Arrow schema from an Avro schema JSON.
+// If the top level is of record type, set includeTopLevel to either make
+// its fields top level fields in the resulting schema or nested in a single field.
+func ArrowSchemaFromAvro(avroSchema []byte) (*arrow.Schema, error) {
+ var m map[string]interface{}
+ var node schemaNode
+ json.Unmarshal(avroSchema, &m)
+ if m["type"].(string) == "record" {
+ if _, ok := m["fields"]; ok {
+ for _, field := range m["fields"].([]interface{}) {
+ node.fields = append(node.fields, field.(map[string]interface{}))
+ }
+ if len(node.fields) == 0 {
+ return nil, fmt.Errorf("invalid avro schema: no top level record fields found")
+ }
+ }
+ }
+ fields := iterateFields(node.fields)
+ return arrow.NewSchema(fields, nil), nil
+}
+
+func iterateFields(f []interface{}) []arrow.Field {
+ var s []arrow.Field
+ for _, field := range f {
+ var n schemaNode
+ n.name = field.(map[string]interface{})["name"].(string)
+ n.ofType = field.(map[string]interface{})["type"]
+ switch n.ofType.(type) {
+ // Getting field type from within field object
+ case string:
+ switch n.ofType.(string) {
+ case "enum":
+ for _, symbol := range field.(map[string]interface{})["symbols"].([]interface{}) {
+ n.symbols = append(n.symbols, symbol.(string))
+ }
+ default:
+ if lt, ok := field.(map[string]interface{})["logicalType"]; ok {
+ n.logicalType = lt.(string)
+ }
+ if lt, ok := field.(map[string]interface{})["size"]; ok {
+ n.size = int(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["precision"]; ok {
+ n.precision = int32(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["scale"]; ok {
+ n.scale = int32(lt.(float64))
+ }
+ }
+ // Field type is an object
+ case map[string]interface{}:
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["logicalType"]; ok {
+ n.logicalType = lt.(string)
+ }
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["size"]; ok {
+ n.size = int(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["precision"]; ok {
+ n.precision = int32(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["scale"]; ok {
+ n.scale = int32(lt.(float64))
+ }
+ case []interface{}:
+
+ default:
+ if lt, ok := field.(map[string]interface{})["logicalType"]; ok {
+ n.logicalType = lt.(string)
+ }
+ if lt, ok := field.(map[string]interface{})["size"]; ok {
+ n.size = int(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["precision"]; ok {
+ n.precision = int32(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["scale"]; ok {
+ n.scale = int32(lt.(float64))
+ }
+ }
+ // Field is of type "record"
+ if nf, f := field.(map[string]interface{})["fields"]; f {
+ switch nf.(type) {
+ // primitive & complex types
+ case map[string]interface{}:
+ for _, v := range nf.(map[string]interface{})["fields"].([]interface{}) {
+ n.fields = append(n.fields, v.(map[string]interface{}))
+ }
+ // type unions
+ default:
+ for _, v := range nf.([]interface{}) {
+ n.fields = append(n.fields, v.(map[string]interface{}))
+ }
+ }
+ }
+ s = append(s, traverseNodes(n))
+ }
+ return s
+}
+
+func traverseNodes(node schemaNode) arrow.Field {
+ switch node.ofType.(type) {
+ case string:
+ // Avro primitive type
+ if len(node.fields) == 0 {
+ switch node.ofType.(string) {
+ case "boolean", "int", "long", "float", "double", "bytes", "string":
+ if node.logicalType != "" {
+ return avroLogicalToArrowField(node)
+ }
+ // Avro primitive type
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(node.ofType.(string))}
+ case "fixed":
+ // Duration type is not supported in github.com/linkedin/goavro
+ // Implementing as Binary for now.
+ switch node.logicalType {
+ case "decimal":
+ return avroLogicalToArrowField(node)
+ case "duration":
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.Binary}
+ //return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.MonthDayNanoInterval}
+ }
+ return arrow.Field{Name: node.name, Type: &arrow.FixedSizeBinaryType{ByteWidth: node.size}}
+ case "enum":
+ symbols := make(map[string]string)
+ for index, symbol := range node.symbols {
+ k := strconv.FormatInt(int64(index), 10)
+ symbols[k] = symbol
+ }
+ var dt arrow.DictionaryType = arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Uint64, ValueType: arrow.BinaryTypes.String, Ordered: false}
+ sl := len(symbols)
+ switch {
+ case sl <= math.MaxUint8:
+ dt.IndexType = arrow.PrimitiveTypes.Uint8
+ case sl > math.MaxUint8 && sl <= math.MaxUint16:
+ dt.IndexType = arrow.PrimitiveTypes.Uint16
+ case sl > math.MaxUint16 && sl <= math.MaxUint32:
+ dt.IndexType = arrow.PrimitiveTypes.Uint32
+ }
+ return arrow.Field{Name: node.name, Type: &dt, Nullable: true, Metadata: arrow.MetadataFrom(symbols)}
+ default:
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(node.ofType.(string))}
+ }
+ } else {
+ // avro "record" type, node has "fields" array
+ if node.ofType.(string) == "record" {
+ var n schemaNode
+ n.name = node.name
+ n.ofType = node.ofType
+ if len(node.fields) > 0 {
+ n.fields = append(n.fields, node.fields...)
+ }
+ f := iterateFields(n.fields)
+ return arrow.Field{Name: node.name, Type: arrow.StructOf(f...)}
+ }
+ }
+ // Avro complex types
+ case map[string]interface{}:
+ //var n schemaNode
+ //n.name = node.name
+ //n.ofType = node.ofType.(map[string]interface{})["type"]
+ switch node.logicalType {
+ case "":
+ return avroComplexToArrowField(node)
+ default:
+ return avroLogicalToArrowField(node)
+ }
+
+ // Avro union types
+ case []interface{}:
+ var unionTypes []string
+ for _, ft := range node.ofType.([]interface{}) {
+ switch ft.(type) {
+ // primitive types
+ case string:
+ if ft != "null" {
+ unionTypes = append(unionTypes, ft.(string))
+ }
+ continue
+ // complex types
+ case map[string]interface{}:
+ var n schemaNode
+ n.name = node.name
+ n.ofType = ft.(map[string]interface{})["type"]
+ if _, f := ft.(map[string]interface{})["fields"]; f {
+ for _, field := range ft.(map[string]interface{})["fields"].([]interface{}) {
+ n.fields = append(n.fields, field.(map[string]interface{}))
+ }
+ }
+ f := iterateFields(n.fields)
+ return arrow.Field{Name: node.name, Type: arrow.StructOf(f...)}
+ }
+ }
+ // Supported Avro union type is null + one other type.
+ // TODO: Complex AVRO union to Arrow Dense || Sparse Union.
+ if len(unionTypes) == 1 {
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(unionTypes[0])}
+ } else {
+ // BYTE_ARRAY is the catchall if union type is anything beyond null + one other type.
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.Binary}
+ }
+ }
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.Binary}
+}
+
+func avroLogicalToArrowField(node schemaNode) arrow.Field {
+ // Avro logical types
+ switch node.logicalType {
+ // The decimal logical type represents an arbitrary-precision signed decimal number of the form unscaled × 10-scale.
+ // A decimal logical type annotates Avro bytes or fixed types. The byte array must contain the two’s-complement
+ // representation of the unscaled integer value in big-endian byte order. The scale is fixed, and is specified
+ // using an attribute.
+ //
+ // The following attributes are supported:
+ // scale, a JSON integer representing the scale (optional). If not specified the scale is 0.
+ // precision, a JSON integer representing the (maximum) precision of decimals stored in this type (required).
+ case "decimal":
+ if node.precision <= 38 {
+ return arrow.Field{Name: node.name, Type: &arrow.Decimal128Type{Precision: node.precision, Scale: node.scale}}
+ } else {
+ return arrow.Field{Name: node.name, Type: &arrow.Decimal256Type{Precision: node.precision, Scale: node.scale}}
+ }
Review Comment:
probably easier to simplify this function and switch to something like:
```go
var dt arrow.DataType
switch node.logicalType {
// ...
case "decimal":
id := arrow.DECIMAL128
if node.precision > 38 {
id = arrow.DECIMAL256
}
dt, _ = arrow.NewDecimalType(id, node.precision, node.scale)
case "uuid":
dt = types.NewUUIDType() // import "github.com/apache/arrow/v13/internal/types"
// etc.....
}
return arrow.Field{Name: node.name, Type: dt}
```
##########
go/arrow/avro/common.go:
##########
@@ -0,0 +1,162 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package avro
+
+import (
+ "errors"
+ "fmt"
+
+ "github.com/apache/arrow/go/v13/arrow"
+ "github.com/apache/arrow/go/v13/arrow/memory"
+)
+
+var (
+ ErrMismatchFields = errors.New("arrow/avro: number of records mismatch")
+)
+
+// Option configures an Avro reader/writer.
+type Option func(config)
+type config interface{}
+
+// WithTopLevel specifies whether to include an Avro schema's top level record
+// or only its fields.
+func WithTopLevel(b bool) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.topLevel = b
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithAllocator specifies the Arrow memory allocator used while building records.
+func WithAllocator(mem memory.Allocator) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.mem = mem
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithChunk specifies the chunk size used while reading Avro OCF files.
+//
+// If n is zero or 1, no chunking will take place and the reader will create
+// one record per row.
+// If n is greater than 1, chunks of n rows will be read.
+// If n is negative, the reader will load the whole Avro OCF file into memory and
+// create one big record with all the rows.
+func WithChunk(n int) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.chunk = n
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// DefaultNullValues is the set of values considered as NULL values by default
+// when Reader is configured to handle NULL values.
+var DefaultNullValues = []string{"", "NULL", "null"}
Review Comment:
doesn't Avro have a concept of null already so you don't need to interpret strings as null?
##########
go/arrow/avro/common.go:
##########
@@ -0,0 +1,162 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package avro
+
+import (
+ "errors"
+ "fmt"
+
+ "github.com/apache/arrow/go/v13/arrow"
+ "github.com/apache/arrow/go/v13/arrow/memory"
+)
+
+var (
+ ErrMismatchFields = errors.New("arrow/avro: number of records mismatch")
+)
+
+// Option configures an Avro reader/writer.
+type Option func(config)
+type config interface{}
+
+// WithTopLevel specifies whether to include an Avro schema's top level record
+// or only its fields.
+func WithTopLevel(b bool) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.topLevel = b
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithAllocator specifies the Arrow memory allocator used while building records.
+func WithAllocator(mem memory.Allocator) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.mem = mem
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithChunk specifies the chunk size used while reading Avro OCF files.
+//
+// If n is zero or 1, no chunking will take place and the reader will create
+// one record per row.
+// If n is greater than 1, chunks of n rows will be read.
+// If n is negative, the reader will load the whole Avro OCF file into memory and
+// create one big record with all the rows.
+func WithChunk(n int) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.chunk = n
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// DefaultNullValues is the set of values considered as NULL values by default
+// when Reader is configured to handle NULL values.
+var DefaultNullValues = []string{"", "NULL", "null"}
+
+// WithNullReader sets options for a CSV Reader pertaining to NULL value
+// handling. If stringsCanBeNull is true, then a string that matches one of the
+// nullValues set will be interpreted as NULL. Numeric columns will be checked
+// for nulls in all cases. If no nullValues arguments are passed in, the
+// defaults set in NewReader() will be kept.
+//
+// When no NULL values is given, the default set is taken from DefaultNullValues.
+func WithNullReader(stringsCanBeNull bool, nullValues ...string) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.stringsCanBeNull = stringsCanBeNull
+
+ if len(nullValues) == 0 {
+ nullValues = DefaultNullValues
+ }
+ cfg.nulls = make([]string, len(nullValues))
+ copy(cfg.nulls, nullValues)
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithColumnTypes allows specifying optional per-column types (disabling
+// type inference on those columns).
+//
+// Will panic if used in conjunction with an explicit schema.
+func WithColumnTypes(types map[string]arrow.DataType) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ if cfg.schema != nil {
+ panic(fmt.Errorf("%w: cannot use WithColumnTypes with explicit schema", arrow.ErrInvalid))
+ }
+ cfg.columnTypes = types
+ default:
+ panic(fmt.Errorf("%w: WithColumnTypes only allowed for csv reader", arrow.ErrInvalid))
+ }
+ }
+}
+
+// WithIncludeColumns indicates the names of the columns from the AVRO file
+// that should actually be read and converted (in the slice's order).
+// If set and non-empty, columns not in this slice will be ignored.
+//
+// Will panic if used in conjunction with an explicit schema.
+func WithIncludeColumns(cols []string) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ if cfg.schema != nil {
+ panic(fmt.Errorf("%w: cannot use WithIncludeColumns with explicit schema", arrow.ErrInvalid))
+ }
Review Comment:
why do we need the explicit schema to do this? we have the avro schema so we can throw an error if the field is not found etc.
##########
go/arrow/avro/schema.go:
##########
@@ -0,0 +1,409 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package avro reads Avro OCF files and presents the extracted data as records
+package avro
+
+import (
+ "encoding/json"
+ "fmt"
+ "math"
+ "strconv"
+
+ "github.com/apache/arrow/go/v13/arrow"
+)
+
+type schemaNode struct {
+ name string
+ ofType interface{}
+ logicalType string
+ precision int32
+ scale int32
+ size int
+ symbols []string
+ fields []interface{}
+}
+
+// ArrowSchemaFromAvro returns a new Arrow schema from an Avro schema JSON.
+// If the top level is of record type, set includeTopLevel to either make
+// its fields top level fields in the resulting schema or nested in a single field.
+func ArrowSchemaFromAvro(avroSchema []byte) (*arrow.Schema, error) {
+ var m map[string]interface{}
+ var node schemaNode
+ json.Unmarshal(avroSchema, &m)
+ if m["type"].(string) == "record" {
+ if _, ok := m["fields"]; ok {
+ for _, field := range m["fields"].([]interface{}) {
+ node.fields = append(node.fields, field.(map[string]interface{}))
+ }
+ if len(node.fields) == 0 {
+ return nil, fmt.Errorf("invalid avro schema: no top level record fields found")
+ }
+ }
+ }
+ fields := iterateFields(node.fields)
+ return arrow.NewSchema(fields, nil), nil
+}
+
+func iterateFields(f []interface{}) []arrow.Field {
+ var s []arrow.Field
+ for _, field := range f {
+ var n schemaNode
+ n.name = field.(map[string]interface{})["name"].(string)
+ n.ofType = field.(map[string]interface{})["type"]
+ switch n.ofType.(type) {
+ // Getting field type from within field object
+ case string:
+ switch n.ofType.(string) {
+ case "enum":
+ for _, symbol := range field.(map[string]interface{})["symbols"].([]interface{}) {
+ n.symbols = append(n.symbols, symbol.(string))
+ }
+ default:
+ if lt, ok := field.(map[string]interface{})["logicalType"]; ok {
+ n.logicalType = lt.(string)
+ }
+ if lt, ok := field.(map[string]interface{})["size"]; ok {
+ n.size = int(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["precision"]; ok {
+ n.precision = int32(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["scale"]; ok {
+ n.scale = int32(lt.(float64))
+ }
+ }
+ // Field type is an object
+ case map[string]interface{}:
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["logicalType"]; ok {
+ n.logicalType = lt.(string)
+ }
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["size"]; ok {
+ n.size = int(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["precision"]; ok {
+ n.precision = int32(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["scale"]; ok {
+ n.scale = int32(lt.(float64))
+ }
+ case []interface{}:
+
+ default:
+ if lt, ok := field.(map[string]interface{})["logicalType"]; ok {
+ n.logicalType = lt.(string)
+ }
+ if lt, ok := field.(map[string]interface{})["size"]; ok {
+ n.size = int(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["precision"]; ok {
+ n.precision = int32(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["scale"]; ok {
+ n.scale = int32(lt.(float64))
+ }
+ }
+ // Field is of type "record"
+ if nf, f := field.(map[string]interface{})["fields"]; f {
+ switch nf.(type) {
+ // primitive & complex types
+ case map[string]interface{}:
+ for _, v := range nf.(map[string]interface{})["fields"].([]interface{}) {
+ n.fields = append(n.fields, v.(map[string]interface{}))
+ }
+ // type unions
+ default:
+ for _, v := range nf.([]interface{}) {
+ n.fields = append(n.fields, v.(map[string]interface{}))
+ }
+ }
+ }
+ s = append(s, traverseNodes(n))
+ }
+ return s
+}
+
+func traverseNodes(node schemaNode) arrow.Field {
+ switch node.ofType.(type) {
+ case string:
+ // Avro primitive type
+ if len(node.fields) == 0 {
+ switch node.ofType.(string) {
+ case "boolean", "int", "long", "float", "double", "bytes", "string":
+ if node.logicalType != "" {
+ return avroLogicalToArrowField(node)
+ }
+ // Avro primitive type
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(node.ofType.(string))}
+ case "fixed":
+ // Duration type is not supported in github.com/linkedin/goavro
+ // Implementing as Binary for now.
+ switch node.logicalType {
+ case "decimal":
+ return avroLogicalToArrowField(node)
+ case "duration":
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.Binary}
+ //return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.MonthDayNanoInterval}
+ }
+ return arrow.Field{Name: node.name, Type: &arrow.FixedSizeBinaryType{ByteWidth: node.size}}
+ case "enum":
+ symbols := make(map[string]string)
+ for index, symbol := range node.symbols {
+ k := strconv.FormatInt(int64(index), 10)
+ symbols[k] = symbol
+ }
+ var dt arrow.DictionaryType = arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Uint64, ValueType: arrow.BinaryTypes.String, Ordered: false}
+ sl := len(symbols)
+ switch {
+ case sl <= math.MaxUint8:
+ dt.IndexType = arrow.PrimitiveTypes.Uint8
+ case sl > math.MaxUint8 && sl <= math.MaxUint16:
+ dt.IndexType = arrow.PrimitiveTypes.Uint16
+ case sl > math.MaxUint16 && sl <= math.MaxUint32:
+ dt.IndexType = arrow.PrimitiveTypes.Uint32
+ }
+ return arrow.Field{Name: node.name, Type: &dt, Nullable: true, Metadata: arrow.MetadataFrom(symbols)}
+ default:
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(node.ofType.(string))}
+ }
+ } else {
+ // avro "record" type, node has "fields" array
+ if node.ofType.(string) == "record" {
+ var n schemaNode
+ n.name = node.name
+ n.ofType = node.ofType
+ if len(node.fields) > 0 {
+ n.fields = append(n.fields, node.fields...)
+ }
+ f := iterateFields(n.fields)
+ return arrow.Field{Name: node.name, Type: arrow.StructOf(f...)}
+ }
+ }
+ // Avro complex types
+ case map[string]interface{}:
+ //var n schemaNode
+ //n.name = node.name
+ //n.ofType = node.ofType.(map[string]interface{})["type"]
+ switch node.logicalType {
+ case "":
+ return avroComplexToArrowField(node)
+ default:
+ return avroLogicalToArrowField(node)
+ }
+
+ // Avro union types
+ case []interface{}:
+ var unionTypes []string
+ for _, ft := range node.ofType.([]interface{}) {
+ switch ft.(type) {
+ // primitive types
+ case string:
+ if ft != "null" {
+ unionTypes = append(unionTypes, ft.(string))
+ }
+ continue
+ // complex types
+ case map[string]interface{}:
+ var n schemaNode
+ n.name = node.name
+ n.ofType = ft.(map[string]interface{})["type"]
+ if _, f := ft.(map[string]interface{})["fields"]; f {
+ for _, field := range ft.(map[string]interface{})["fields"].([]interface{}) {
+ n.fields = append(n.fields, field.(map[string]interface{}))
+ }
+ }
+ f := iterateFields(n.fields)
+ return arrow.Field{Name: node.name, Type: arrow.StructOf(f...)}
+ }
+ }
+ // Supported Avro union type is null + one other type.
+ // TODO: Complex AVRO union to Arrow Dense || Sparse Union.
+ if len(unionTypes) == 1 {
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(unionTypes[0])}
+ } else {
+ // BYTE_ARRAY is the catchall if union type is anything beyond null + one other type.
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.Binary}
+ }
+ }
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.Binary}
+}
+
+func avroLogicalToArrowField(node schemaNode) arrow.Field {
+ // Avro logical types
+ switch node.logicalType {
+ // The decimal logical type represents an arbitrary-precision signed decimal number of the form unscaled × 10-scale.
+ // A decimal logical type annotates Avro bytes or fixed types. The byte array must contain the two’s-complement
+ // representation of the unscaled integer value in big-endian byte order. The scale is fixed, and is specified
+ // using an attribute.
+ //
+ // The following attributes are supported:
+ // scale, a JSON integer representing the scale (optional). If not specified the scale is 0.
+ // precision, a JSON integer representing the (maximum) precision of decimals stored in this type (required).
+ case "decimal":
+ if node.precision <= 38 {
+ return arrow.Field{Name: node.name, Type: &arrow.Decimal128Type{Precision: node.precision, Scale: node.scale}}
+ } else {
+ return arrow.Field{Name: node.name, Type: &arrow.Decimal256Type{Precision: node.precision, Scale: node.scale}}
+ }
+
+ // The uuid logical type represents a random generated universally unique identifier (UUID).
+ // A uuid logical type annotates an Avro string. The string has to conform with RFC-4122
+ case "uuid":
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.String}
+
+ // The date logical type represents a date within the calendar, with no reference to a particular
+ // time zone or time of day.
+ // A date logical type annotates an Avro int, where the int stores the number of days from the unix epoch,
+ // 1 January 1970 (ISO calendar).
+ case "date":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Date32}
+
+ // The time-millis logical type represents a time of day, with no reference to a particular calendar,
+ // time zone or date, with a precision of one millisecond.
+ // A time-millis logical type annotates an Avro int, where the int stores the number of milliseconds
+ // after midnight, 00:00:00.000.
+ case "time-millis":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Time32ms}
+
+ // The time-micros logical type represents a time of day, with no reference to a particular calendar,
+ // time zone or date, with a precision of one microsecond.
+ // A time-micros logical type annotates an Avro long, where the long stores the number of microseconds
+ // after midnight, 00:00:00.000000.
+ case "time-micros":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Time64us}
+
+ // The timestamp-millis logical type represents an instant on the global timeline, independent of a
+ // particular time zone or calendar, with a precision of one millisecond. Please note that time zone
+ // information gets lost in this process. Upon reading a value back, we can only reconstruct the instant,
+ // but not the original representation. In practice, such timestamps are typically displayed to users in
+ // their local time zones, therefore they may be displayed differently depending on the execution environment.
+ // A timestamp-millis logical type annotates an Avro long, where the long stores the number of milliseconds
+ // from the unix epoch, 1 January 1970 00:00:00.000 UTC.
+ case "timestamp-millis":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Timestamp_ms}
+ // The timestamp-micros logical type represents an instant on the global timeline, independent of a
+ // particular time zone or calendar, with a precision of one microsecond. Please note that time zone
+ // information gets lost in this process. Upon reading a value back, we can only reconstruct the instant,
+ // but not the original representation. In practice, such timestamps are typically displayed to users
+ // in their local time zones, therefore they may be displayed differently depending on the execution environment.
+ // A timestamp-micros logical type annotates an Avro long, where the long stores the number of microseconds
+ // from the unix epoch, 1 January 1970 00:00:00.000000 UTC.
+ case "timestamp-micros":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Timestamp_us}
+ // The local-timestamp-millis logical type represents a timestamp in a local timezone, regardless of
+ // what specific time zone is considered local, with a precision of one millisecond.
+ // A local-timestamp-millis logical type annotates an Avro long, where the long stores the number of
+ // milliseconds, from 1 January 1970 00:00:00.000.
+ case "local-timestamp-millis":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Timestamp_ms}
Review Comment:
the default `arrow.FixedWidthTypes.Timestamp_ms` sets the timezone to "UTC". You should manually create `&arrow.TimestampType{Unit: arrow.Millisecond}` for the local timestamp (or maybe we should create `NoTZ` versions of these vars that you can use for this....?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] loicalleyne commented on a diff in pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne commented on code in PR #36796:
URL: https://github.com/apache/arrow/pull/36796#discussion_r1281175339
##########
go/arrow/avro/schema.go:
##########
@@ -0,0 +1,409 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package avro reads Avro OCF files and presents the extracted data as records
+package avro
+
+import (
+ "encoding/json"
+ "fmt"
+ "math"
+ "strconv"
+
+ "github.com/apache/arrow/go/v13/arrow"
+)
+
+type schemaNode struct {
+ name string
+ ofType interface{}
+ logicalType string
+ precision int32
+ scale int32
+ size int
+ symbols []string
+ fields []interface{}
+}
+
+// ArrowSchemaFromAvro returns a new Arrow schema from an Avro schema JSON.
+// If the top level is of record type, set includeTopLevel to either make
+// its fields top level fields in the resulting schema or nested in a single field.
+func ArrowSchemaFromAvro(avroSchema []byte) (*arrow.Schema, error) {
+ var m map[string]interface{}
+ var node schemaNode
+ json.Unmarshal(avroSchema, &m)
+ if m["type"].(string) == "record" {
+ if _, ok := m["fields"]; ok {
+ for _, field := range m["fields"].([]interface{}) {
+ node.fields = append(node.fields, field.(map[string]interface{}))
+ }
+ if len(node.fields) == 0 {
+ return nil, fmt.Errorf("invalid avro schema: no top level record fields found")
+ }
+ }
+ }
+ fields := iterateFields(node.fields)
+ return arrow.NewSchema(fields, nil), nil
+}
+
+func iterateFields(f []interface{}) []arrow.Field {
+ var s []arrow.Field
+ for _, field := range f {
+ var n schemaNode
+ n.name = field.(map[string]interface{})["name"].(string)
+ n.ofType = field.(map[string]interface{})["type"]
+ switch n.ofType.(type) {
+ // Getting field type from within field object
+ case string:
+ switch n.ofType.(string) {
+ case "enum":
+ for _, symbol := range field.(map[string]interface{})["symbols"].([]interface{}) {
+ n.symbols = append(n.symbols, symbol.(string))
+ }
+ default:
+ if lt, ok := field.(map[string]interface{})["logicalType"]; ok {
+ n.logicalType = lt.(string)
+ }
+ if lt, ok := field.(map[string]interface{})["size"]; ok {
+ n.size = int(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["precision"]; ok {
+ n.precision = int32(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["scale"]; ok {
+ n.scale = int32(lt.(float64))
+ }
+ }
+ // Field type is an object
+ case map[string]interface{}:
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["logicalType"]; ok {
+ n.logicalType = lt.(string)
+ }
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["size"]; ok {
+ n.size = int(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["precision"]; ok {
+ n.precision = int32(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["scale"]; ok {
+ n.scale = int32(lt.(float64))
+ }
+ case []interface{}:
+
+ default:
+ if lt, ok := field.(map[string]interface{})["logicalType"]; ok {
+ n.logicalType = lt.(string)
+ }
+ if lt, ok := field.(map[string]interface{})["size"]; ok {
+ n.size = int(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["precision"]; ok {
+ n.precision = int32(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["scale"]; ok {
+ n.scale = int32(lt.(float64))
+ }
+ }
+ // Field is of type "record"
+ if nf, f := field.(map[string]interface{})["fields"]; f {
+ switch nf.(type) {
+ // primitive & complex types
+ case map[string]interface{}:
+ for _, v := range nf.(map[string]interface{})["fields"].([]interface{}) {
+ n.fields = append(n.fields, v.(map[string]interface{}))
+ }
+ // type unions
+ default:
+ for _, v := range nf.([]interface{}) {
+ n.fields = append(n.fields, v.(map[string]interface{}))
+ }
+ }
+ }
+ s = append(s, traverseNodes(n))
+ }
+ return s
+}
+
+func traverseNodes(node schemaNode) arrow.Field {
+ switch node.ofType.(type) {
+ case string:
+ // Avro primitive type
+ if len(node.fields) == 0 {
+ switch node.ofType.(string) {
+ case "boolean", "int", "long", "float", "double", "bytes", "string":
+ if node.logicalType != "" {
+ return avroLogicalToArrowField(node)
+ }
+ // Avro primitive type
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(node.ofType.(string))}
+ case "fixed":
+ // Duration type is not supported in github.com/linkedin/goavro
+ // Implementing as Binary for now.
+ switch node.logicalType {
+ case "decimal":
+ return avroLogicalToArrowField(node)
+ case "duration":
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.Binary}
+ //return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.MonthDayNanoInterval}
+ }
+ return arrow.Field{Name: node.name, Type: &arrow.FixedSizeBinaryType{ByteWidth: node.size}}
+ case "enum":
+ symbols := make(map[string]string)
+ for index, symbol := range node.symbols {
+ k := strconv.FormatInt(int64(index), 10)
+ symbols[k] = symbol
+ }
+ var dt arrow.DictionaryType = arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Uint64, ValueType: arrow.BinaryTypes.String, Ordered: false}
+ sl := len(symbols)
+ switch {
+ case sl <= math.MaxUint8:
+ dt.IndexType = arrow.PrimitiveTypes.Uint8
+ case sl > math.MaxUint8 && sl <= math.MaxUint16:
+ dt.IndexType = arrow.PrimitiveTypes.Uint16
+ case sl > math.MaxUint16 && sl <= math.MaxUint32:
+ dt.IndexType = arrow.PrimitiveTypes.Uint32
+ }
+ return arrow.Field{Name: node.name, Type: &dt, Nullable: true, Metadata: arrow.MetadataFrom(symbols)}
+ default:
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(node.ofType.(string))}
+ }
+ } else {
+ // avro "record" type, node has "fields" array
+ if node.ofType.(string) == "record" {
+ var n schemaNode
+ n.name = node.name
+ n.ofType = node.ofType
+ if len(node.fields) > 0 {
+ n.fields = append(n.fields, node.fields...)
+ }
+ f := iterateFields(n.fields)
+ return arrow.Field{Name: node.name, Type: arrow.StructOf(f...)}
+ }
+ }
+ // Avro complex types
+ case map[string]interface{}:
+ //var n schemaNode
+ //n.name = node.name
+ //n.ofType = node.ofType.(map[string]interface{})["type"]
+ switch node.logicalType {
+ case "":
+ return avroComplexToArrowField(node)
+ default:
+ return avroLogicalToArrowField(node)
+ }
+
+ // Avro union types
+ case []interface{}:
+ var unionTypes []string
+ for _, ft := range node.ofType.([]interface{}) {
+ switch ft.(type) {
+ // primitive types
+ case string:
+ if ft != "null" {
+ unionTypes = append(unionTypes, ft.(string))
+ }
+ continue
+ // complex types
+ case map[string]interface{}:
+ var n schemaNode
+ n.name = node.name
+ n.ofType = ft.(map[string]interface{})["type"]
+ if _, f := ft.(map[string]interface{})["fields"]; f {
+ for _, field := range ft.(map[string]interface{})["fields"].([]interface{}) {
+ n.fields = append(n.fields, field.(map[string]interface{}))
+ }
+ }
+ f := iterateFields(n.fields)
+ return arrow.Field{Name: node.name, Type: arrow.StructOf(f...)}
+ }
+ }
+ // Supported Avro union type is null + one other type.
+ // TODO: Complex AVRO union to Arrow Dense || Sparse Union.
+ if len(unionTypes) == 1 {
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(unionTypes[0])}
+ } else {
+ // BYTE_ARRAY is the catchall if union type is anything beyond null + one other type.
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.Binary}
+ }
+ }
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.Binary}
+}
+
+func avroLogicalToArrowField(node schemaNode) arrow.Field {
+ // Avro logical types
+ switch node.logicalType {
+ // The decimal logical type represents an arbitrary-precision signed decimal number of the form unscaled × 10-scale.
+ // A decimal logical type annotates Avro bytes or fixed types. The byte array must contain the two’s-complement
+ // representation of the unscaled integer value in big-endian byte order. The scale is fixed, and is specified
+ // using an attribute.
+ //
+ // The following attributes are supported:
+ // scale, a JSON integer representing the scale (optional). If not specified the scale is 0.
+ // precision, a JSON integer representing the (maximum) precision of decimals stored in this type (required).
+ case "decimal":
+ if node.precision <= 38 {
+ return arrow.Field{Name: node.name, Type: &arrow.Decimal128Type{Precision: node.precision, Scale: node.scale}}
+ } else {
+ return arrow.Field{Name: node.name, Type: &arrow.Decimal256Type{Precision: node.precision, Scale: node.scale}}
+ }
+
+ // The uuid logical type represents a random generated universally unique identifier (UUID).
+ // A uuid logical type annotates an Avro string. The string has to conform with RFC-4122
+ case "uuid":
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.String}
+
+ // The date logical type represents a date within the calendar, with no reference to a particular
+ // time zone or time of day.
+ // A date logical type annotates an Avro int, where the int stores the number of days from the unix epoch,
+ // 1 January 1970 (ISO calendar).
+ case "date":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Date32}
+
+ // The time-millis logical type represents a time of day, with no reference to a particular calendar,
+ // time zone or date, with a precision of one millisecond.
+ // A time-millis logical type annotates an Avro int, where the int stores the number of milliseconds
+ // after midnight, 00:00:00.000.
+ case "time-millis":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Time32ms}
+
+ // The time-micros logical type represents a time of day, with no reference to a particular calendar,
+ // time zone or date, with a precision of one microsecond.
+ // A time-micros logical type annotates an Avro long, where the long stores the number of microseconds
+ // after midnight, 00:00:00.000000.
+ case "time-micros":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Time64us}
+
+ // The timestamp-millis logical type represents an instant on the global timeline, independent of a
+ // particular time zone or calendar, with a precision of one millisecond. Please note that time zone
+ // information gets lost in this process. Upon reading a value back, we can only reconstruct the instant,
+ // but not the original representation. In practice, such timestamps are typically displayed to users in
+ // their local time zones, therefore they may be displayed differently depending on the execution environment.
+ // A timestamp-millis logical type annotates an Avro long, where the long stores the number of milliseconds
+ // from the unix epoch, 1 January 1970 00:00:00.000 UTC.
+ case "timestamp-millis":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Timestamp_ms}
+ // The timestamp-micros logical type represents an instant on the global timeline, independent of a
+ // particular time zone or calendar, with a precision of one microsecond. Please note that time zone
+ // information gets lost in this process. Upon reading a value back, we can only reconstruct the instant,
+ // but not the original representation. In practice, such timestamps are typically displayed to users
+ // in their local time zones, therefore they may be displayed differently depending on the execution environment.
+ // A timestamp-micros logical type annotates an Avro long, where the long stores the number of microseconds
+ // from the unix epoch, 1 January 1970 00:00:00.000000 UTC.
+ case "timestamp-micros":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Timestamp_us}
+ // The local-timestamp-millis logical type represents a timestamp in a local timezone, regardless of
+ // what specific time zone is considered local, with a precision of one millisecond.
+ // A local-timestamp-millis logical type annotates an Avro long, where the long stores the number of
+ // milliseconds, from 1 January 1970 00:00:00.000.
+ case "local-timestamp-millis":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Timestamp_ms}
+ // The local-timestamp-micros logical type represents a timestamp in a local timezone, regardless of
+ // what specific time zone is considered local, with a precision of one microsecond.
+ // A local-timestamp-micros logical type annotates an Avro long, where the long stores the number of
+ // microseconds, from 1 January 1970 00:00:00.000000.
+ case "local-timestamp-micros":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Timestamp_us}
+ // Avro primitive type
+ default:
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(node.ofType.(string))}
+ }
+ return arrow.Field{}
+}
+
+func avroComplexToArrowField(node schemaNode) arrow.Field {
+ var n schemaNode
+ n.name = node.name
+ n.ofType = node.ofType.(map[string]interface{})["type"]
+ // Avro "array" field type
+ if i, ok := node.ofType.(map[string]interface{})["items"]; ok {
+ switch i.(string) {
+ case "int", "long", "float", "double", "bytes", "boolean", "string":
+ return arrow.Field{Name: node.name, Type: arrow.ListOf(AvroPrimitiveToArrowType(i.(string)))}
+ case "enum", "fixed", "map", "record", "array":
+ return arrow.Field{Name: node.name, Type: arrow.ListOf(avroComplexToArrowField(n).Type)}
+ case "decimal", "uuid", "date", "time-millis", "time-micros", "timestamp-millis", "timestamp-micros", "local-timestamp-millis", "local-timestamp-micros":
+ return arrow.Field{Name: node.name, Type: arrow.ListOf(avroLogicalToArrowField(n).Type)}
+ }
+ }
+ // Avro "enum" field type = Arrow dictionary type
+ if i, ok := node.ofType.(map[string]interface{})["symbols"]; ok {
+ symbols := make(map[string]string)
+ for index, symbol := range i.([]interface{}) {
+ k := strconv.FormatInt(int64(index), 10)
+ symbols[k] = symbol.(string)
+ }
+ var dt arrow.DictionaryType = arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Uint64, ValueType: arrow.BinaryTypes.String, Ordered: false}
+ sl := len(symbols)
+ switch {
+ case sl <= math.MaxUint8:
+ dt.IndexType = arrow.PrimitiveTypes.Uint8
+ case sl > math.MaxUint8 && sl <= math.MaxUint16:
+ dt.IndexType = arrow.PrimitiveTypes.Uint16
+ case sl > math.MaxUint16 && sl <= math.MaxUint32:
+ dt.IndexType = arrow.PrimitiveTypes.Uint32
+ }
+ return arrow.Field{Name: node.name, Type: &dt, Nullable: true, Metadata: arrow.MetadataFrom(symbols)}
+ }
+ // Avro "fixed" field type = Arrow FixedSize Primitive BinaryType
+ if i, ok := node.ofType.(map[string]interface{})["size"]; ok {
+ return arrow.Field{Name: node.name, Type: &arrow.FixedSizeBinaryType{ByteWidth: int(i.(float64))}}
+ }
+ // Avro "map" field type
+ if i, ok := node.ofType.(map[string]interface{})["values"]; ok {
+ return arrow.Field{Name: node.name, Type: arrow.MapOf(arrow.BinaryTypes.String, AvroPrimitiveToArrowType(i.(string)))}
+ }
+ // Avro "record" field type
+ if _, f := node.ofType.(map[string]interface{})["fields"]; f {
+ for _, field := range node.ofType.(map[string]interface{})["fields"].([]interface{}) {
+ n.fields = append(n.fields, field.(map[string]interface{}))
+ }
+ s := iterateFields(n.fields)
+ return arrow.Field{Name: n.name, Type: arrow.StructOf(s...)}
+ }
+ return arrow.Field{}
+}
+
+// AvroPrimitiveToArrowType returns the Arrow DataType equivalent to a
+// Avro primitive type.
+//
+// NOTE: Arrow Binary type is used as a catchall to avoid potential data loss.
+func AvroPrimitiveToArrowType(avroFieldType string) arrow.DataType {
+ switch avroFieldType {
+ // int: 32-bit signed integer
+ case "int":
+ return arrow.PrimitiveTypes.Int32
+ // long: 64-bit signed integer
+ case "long":
+ return arrow.PrimitiveTypes.Int64
+ // float: single precision (32-bit) IEEE 754 floating-point number
+ case "float":
+ return arrow.PrimitiveTypes.Float32
+ // double: double precision (64-bit) IEEE 754 floating-point number
+ case "double":
+ return arrow.PrimitiveTypes.Float64
+ // bytes: sequence of 8-bit unsigned bytes
+ case "bytes":
+ return arrow.BinaryTypes.Binary
+ // boolean: a binary value
+ case "boolean":
+ return arrow.FixedWidthTypes.Boolean
+ // string: unicode character sequence
+ case "string":
+ return arrow.BinaryTypes.String
+ // fallback to binary type for any unsupported type
+ default:
+ return arrow.BinaryTypes.Binary
Review Comment:
The only unimplemented type is Duration logical type which is unimplemented in linkedin/goavro but is implemented in hamba/avro. If we change to hamba for the reader then all types would be covered and we can just get rid of the fallback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on a diff in pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #36796:
URL: https://github.com/apache/arrow/pull/36796#discussion_r1316384892
##########
go/arrow/avro/schema.go:
##########
@@ -0,0 +1,407 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package avro reads Avro OCF files and presents the extracted data as records
+
+package avro
+
+import (
+ "fmt"
+ "math"
+ "strconv"
+
+ "github.com/apache/arrow/go/v14/arrow"
+ "github.com/apache/arrow/go/v14/internal/types"
+ avro "github.com/hamba/avro/v2"
+)
+
+type schemaNode struct {
+ name string
+ parent *schemaNode
+ schema avro.Schema
+ union bool
+ children []*schemaNode
+ arrowField arrow.Field
+ path []string
+ index, depth int32
+}
+
+var schemaCache avro.SchemaCache
+
+func newSchemaNode() *schemaNode { return &schemaNode{name: "", index: -1} }
+
+func (node *schemaNode) newChild(n string, s avro.Schema) *schemaNode {
+ child := &schemaNode{
+ name: n,
+ parent: node,
+ schema: s,
+ index: int32(len(node.children)),
+ depth: node.depth + 1}
+ node.children = append(node.children, child)
+ return child
+}
+func (node *schemaNode) Children() []*schemaNode { return node.children }
+
+func (node *schemaNode) Name() string { return node.name }
+
+// NamePath returns a slice of keys making up the path to the field
+func (node *schemaNode) NamePath() []string {
+ if len(node.path) == 0 {
+ var path []string
+ cur := node
+ for i := node.depth - 1; i >= 0; i-- {
+ path = append([]string{cur.name}, path...)
+ cur = cur.parent
+ }
+ return path
+ }
+ return node.path
+}
+
+// ArrowSchemaFromAvro returns a new Arrow schema from an Avro schema
+func ArrowSchemaFromAvro(schema avro.Schema) (s *arrow.Schema, err error) {
+ defer func() {
+ if r := recover(); r != nil {
+ s = nil
+ err = fmt.Errorf("invalid avro schema: could not convert schema")
+ }
+ }()
+ n := newSchemaNode()
+ n.schema = schema
+ c := n.newChild(n.schema.(avro.NamedSchema).Name(), n.schema)
+ arrowSchemafromAvro(c)
+ var fields []arrow.Field
+ for _, g := range c.Children() {
+ fields = append(fields, g.arrowField)
+ }
+ s = arrow.NewSchema(fields, nil)
+ return s, nil
+}
+
+func arrowSchemafromAvro(n *schemaNode) {
+ if ns, ok := n.schema.(avro.NamedSchema); ok {
+ schemaCache.Add(ns.Name(), ns)
+ }
+ switch st := n.schema.Type(); st {
+ case "record":
+ iterateFields(n)
+ case "enum":
+ symbols := make(map[string]string)
+ for index, symbol := range n.schema.(avro.PropertySchema).(*avro.EnumSchema).Symbols() {
+ k := strconv.FormatInt(int64(index), 10)
+ symbols[k] = symbol
+ }
+ var dt arrow.DictionaryType = arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Uint64, ValueType: arrow.BinaryTypes.String, Ordered: false}
+ sl := len(symbols)
+ switch {
+ case sl <= math.MaxUint8:
+ dt.IndexType = arrow.PrimitiveTypes.Uint8
+ case sl > math.MaxUint8 && sl <= math.MaxUint16:
+ dt.IndexType = arrow.PrimitiveTypes.Uint16
+ case sl > math.MaxUint16 && sl <= math.MaxUint32:
+ dt.IndexType = arrow.PrimitiveTypes.Uint32
+ }
+ n.arrowField = arrow.Field{Name: n.Name(), Type: &dt, Nullable: true, Metadata: arrow.MetadataFrom(symbols)}
+ case "array":
+ // logical items type
+ c := n.newChild(n.name, n.schema.(*avro.ArraySchema).Items())
+ if isLogicalSchemaType(n.schema.(*avro.ArraySchema).Items()) {
+ avroLogicalToArrowField(c)
+ } else {
+ arrowSchemafromAvro(c)
+ }
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.ListOf(c.arrowField.Type)}
+ case "map":
+ c := n.newChild(n.name, n.schema.(*avro.MapSchema).Values())
+ arrowSchemafromAvro(c)
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.MapOf(arrow.BinaryTypes.String, c.arrowField.Type), Metadata: c.arrowField.Metadata, Nullable: true}
+ case "union":
+ if n.schema.(*avro.UnionSchema).Nullable() {
+ if len(n.schema.(*avro.UnionSchema).Types()) > 1 {
+ n.schema = n.schema.(*avro.UnionSchema).Types()[1]
+ n.union = true
+ arrowSchemafromAvro(n)
+ }
+ }
+ // Avro "fixed" field type = Arrow FixedSize Primitive BinaryType
+ case "fixed":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.Name(), Type: &arrow.FixedSizeBinaryType{ByteWidth: n.schema.(*avro.FixedSchema).Size()}, Nullable: true}
+ }
+ case "string":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "bytes":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "int":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "long":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "float":
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ case "double":
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ case "boolean":
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ case "<ref>":
+ n.schema = schemaCache.Get(n.schema.(*avro.RefSchema).Schema().Name())
+ arrowSchemafromAvro(n)
+ case "null":
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.BinaryTypes.Binary, Nullable: true}
+ }
+}
+
+// iterate record Fields()
+func iterateFields(n *schemaNode) {
+ for _, f := range n.schema.(avro.PropertySchema).(*avro.RecordSchema).Fields() {
+ switch ft := f.Type().(type) {
+ // Avro "array" field type
+ case *avro.ArraySchema:
+ schemaCache.Add(f.Name(), ft.Items())
+ // logical items type
+ c := n.newChild(f.Name(), ft.Items())
+ if isLogicalSchemaType(ft.Items()) {
+ avroLogicalToArrowField(c)
+ } else {
+ arrowSchemafromAvro(c)
+ }
+ c.arrowField = arrow.Field{Name: c.name, Type: arrow.ListOf(c.arrowField.Type), Metadata: c.arrowField.Metadata, Nullable: true}
+ // Avro "enum" field type = Arrow dictionary type
+ case *avro.EnumSchema:
+ schemaCache.Add(f.Name(), f.Type())
+ c := n.newChild(f.Name(), f.Type())
+ symbols := make(map[string]string)
+ for index, symbol := range ft.Symbols() {
+ k := strconv.FormatInt(int64(index), 10)
+ symbols[k] = symbol
+ }
+ var dt arrow.DictionaryType = arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Uint64, ValueType: arrow.BinaryTypes.String, Ordered: false}
+ sl := len(symbols)
+ switch {
+ case sl <= math.MaxUint8:
+ dt.IndexType = arrow.PrimitiveTypes.Uint8
+ case sl > math.MaxUint8 && sl <= math.MaxUint16:
+ dt.IndexType = arrow.PrimitiveTypes.Uint16
+ case sl > math.MaxUint16 && sl <= math.MaxUint32:
+ dt.IndexType = arrow.PrimitiveTypes.Uint32
+ }
+ c.arrowField = arrow.Field{Name: f.Name(), Type: &dt, Nullable: true, Metadata: arrow.MetadataFrom(symbols)}
+ // Avro "fixed" field type = Arrow FixedSize Primitive BinaryType
+ case *avro.FixedSchema:
+ schemaCache.Add(f.Name(), f.Type())
+ if isLogicalSchemaType(f.Type()) {
+ c := n.newChild(f.Name(), f.Type())
+ avroLogicalToArrowField(c)
+ } else {
+ c := n.newChild(f.Name(), f.Type())
+ arrowSchemafromAvro(c)
+ }
+ case *avro.RecordSchema:
+ schemaCache.Add(f.Name(), f.Type())
+ c := n.newChild(f.Name(), f.Type())
+ iterateFields(c)
+ // Avro "map" field type - KVP with value of one type - keys are strings
+ case *avro.MapSchema:
+ schemaCache.Add(f.Name(), ft.Values())
+ c := n.newChild(f.Name(), ft.Values())
+ arrowSchemafromAvro(c)
+ if ft.Values().Type() == "union" {
+ c.arrowField = arrow.Field{Name: f.Name(), Type: arrow.MapOf(arrow.BinaryTypes.String, arrow.BinaryTypes.String), Metadata: c.arrowField.Metadata, Nullable: true}
+ } else {
+ c.arrowField = arrow.Field{Name: f.Name(), Type: arrow.MapOf(arrow.BinaryTypes.String, c.arrowField.Type), Metadata: c.arrowField.Metadata, Nullable: true}
+ }
+ case *avro.UnionSchema:
+ if ft.Nullable() {
+ if len(ft.Types()) > 1 {
+ schemaCache.Add(f.Name(), ft.Types()[1])
+ c := n.newChild(f.Name(), ft.Types()[1])
+ c.union = true
+ arrowSchemafromAvro(c)
+ }
+ }
+ default:
+ schemaCache.Add(f.Name(), f.Type())
Review Comment:
could there end up being multiple *different* schemas with the same name? Would that be a problem for this cache?
##########
go/arrow/avro/schema.go:
##########
@@ -0,0 +1,407 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package avro reads Avro OCF files and presents the extracted data as records
+
+package avro
+
+import (
+ "fmt"
+ "math"
+ "strconv"
+
+ "github.com/apache/arrow/go/v14/arrow"
+ "github.com/apache/arrow/go/v14/internal/types"
+ avro "github.com/hamba/avro/v2"
+)
+
+type schemaNode struct {
+ name string
+ parent *schemaNode
+ schema avro.Schema
+ union bool
+ children []*schemaNode
+ arrowField arrow.Field
+ path []string
+ index, depth int32
+}
+
+var schemaCache avro.SchemaCache
+
+func newSchemaNode() *schemaNode { return &schemaNode{name: "", index: -1} }
+
+func (node *schemaNode) newChild(n string, s avro.Schema) *schemaNode {
+ child := &schemaNode{
+ name: n,
+ parent: node,
+ schema: s,
+ index: int32(len(node.children)),
+ depth: node.depth + 1}
+ node.children = append(node.children, child)
+ return child
+}
+func (node *schemaNode) Children() []*schemaNode { return node.children }
+
+func (node *schemaNode) Name() string { return node.name }
+
+// NamePath returns a slice of keys making up the path to the field
+func (node *schemaNode) NamePath() []string {
+ if len(node.path) == 0 {
+ var path []string
+ cur := node
+ for i := node.depth - 1; i >= 0; i-- {
+ path = append([]string{cur.name}, path...)
+ cur = cur.parent
+ }
+ return path
+ }
+ return node.path
+}
+
+// ArrowSchemaFromAvro returns a new Arrow schema from an Avro schema
+func ArrowSchemaFromAvro(schema avro.Schema) (s *arrow.Schema, err error) {
+ defer func() {
+ if r := recover(); r != nil {
+ s = nil
+ err = fmt.Errorf("invalid avro schema: could not convert schema")
+ }
+ }()
+ n := newSchemaNode()
+ n.schema = schema
+ c := n.newChild(n.schema.(avro.NamedSchema).Name(), n.schema)
+ arrowSchemafromAvro(c)
+ var fields []arrow.Field
+ for _, g := range c.Children() {
+ fields = append(fields, g.arrowField)
+ }
+ s = arrow.NewSchema(fields, nil)
+ return s, nil
+}
+
+func arrowSchemafromAvro(n *schemaNode) {
+ if ns, ok := n.schema.(avro.NamedSchema); ok {
+ schemaCache.Add(ns.Name(), ns)
+ }
+ switch st := n.schema.Type(); st {
+ case "record":
+ iterateFields(n)
+ case "enum":
+ symbols := make(map[string]string)
+ for index, symbol := range n.schema.(avro.PropertySchema).(*avro.EnumSchema).Symbols() {
+ k := strconv.FormatInt(int64(index), 10)
+ symbols[k] = symbol
+ }
+ var dt arrow.DictionaryType = arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Uint64, ValueType: arrow.BinaryTypes.String, Ordered: false}
+ sl := len(symbols)
+ switch {
+ case sl <= math.MaxUint8:
+ dt.IndexType = arrow.PrimitiveTypes.Uint8
+ case sl > math.MaxUint8 && sl <= math.MaxUint16:
+ dt.IndexType = arrow.PrimitiveTypes.Uint16
+ case sl > math.MaxUint16 && sl <= math.MaxUint32:
+ dt.IndexType = arrow.PrimitiveTypes.Uint32
+ }
+ n.arrowField = arrow.Field{Name: n.Name(), Type: &dt, Nullable: true, Metadata: arrow.MetadataFrom(symbols)}
+ case "array":
+ // logical items type
+ c := n.newChild(n.name, n.schema.(*avro.ArraySchema).Items())
+ if isLogicalSchemaType(n.schema.(*avro.ArraySchema).Items()) {
+ avroLogicalToArrowField(c)
+ } else {
+ arrowSchemafromAvro(c)
+ }
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.ListOf(c.arrowField.Type)}
+ case "map":
+ c := n.newChild(n.name, n.schema.(*avro.MapSchema).Values())
+ arrowSchemafromAvro(c)
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.MapOf(arrow.BinaryTypes.String, c.arrowField.Type), Metadata: c.arrowField.Metadata, Nullable: true}
+ case "union":
+ if n.schema.(*avro.UnionSchema).Nullable() {
+ if len(n.schema.(*avro.UnionSchema).Types()) > 1 {
+ n.schema = n.schema.(*avro.UnionSchema).Types()[1]
+ n.union = true
+ arrowSchemafromAvro(n)
+ }
+ }
+ // Avro "fixed" field type = Arrow FixedSize Primitive BinaryType
+ case "fixed":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.Name(), Type: &arrow.FixedSizeBinaryType{ByteWidth: n.schema.(*avro.FixedSchema).Size()}, Nullable: true}
+ }
+ case "string":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "bytes":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "int":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "long":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "float":
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ case "double":
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ case "boolean":
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ case "<ref>":
+ n.schema = schemaCache.Get(n.schema.(*avro.RefSchema).Schema().Name())
+ arrowSchemafromAvro(n)
+ case "null":
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.BinaryTypes.Binary, Nullable: true}
Review Comment:
should this be type `arrow.Null` ?
##########
go/arrow/avro/schema.go:
##########
@@ -0,0 +1,407 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package avro reads Avro OCF files and presents the extracted data as records
+
+package avro
+
+import (
+ "fmt"
+ "math"
+ "strconv"
+
+ "github.com/apache/arrow/go/v14/arrow"
+ "github.com/apache/arrow/go/v14/internal/types"
+ avro "github.com/hamba/avro/v2"
+)
+
+type schemaNode struct {
+ name string
+ parent *schemaNode
+ schema avro.Schema
+ union bool
+ children []*schemaNode
+ arrowField arrow.Field
+ path []string
+ index, depth int32
+}
+
+var schemaCache avro.SchemaCache
Review Comment:
this should probably not be global to the package, otherwise multiple calls to read schemas might get cached schemas that aren't correct....?
##########
go/arrow/avro/schema.go:
##########
@@ -0,0 +1,407 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package avro reads Avro OCF files and presents the extracted data as records
+
+package avro
+
+import (
+ "fmt"
+ "math"
+ "strconv"
+
+ "github.com/apache/arrow/go/v14/arrow"
+ "github.com/apache/arrow/go/v14/internal/types"
+ avro "github.com/hamba/avro/v2"
+)
+
+type schemaNode struct {
+ name string
+ parent *schemaNode
+ schema avro.Schema
+ union bool
+ children []*schemaNode
+ arrowField arrow.Field
+ path []string
+ index, depth int32
+}
+
+var schemaCache avro.SchemaCache
+
+func newSchemaNode() *schemaNode { return &schemaNode{name: "", index: -1} }
+
+func (node *schemaNode) newChild(n string, s avro.Schema) *schemaNode {
+ child := &schemaNode{
+ name: n,
+ parent: node,
+ schema: s,
+ index: int32(len(node.children)),
+ depth: node.depth + 1}
+ node.children = append(node.children, child)
+ return child
+}
+func (node *schemaNode) Children() []*schemaNode { return node.children }
+
+func (node *schemaNode) Name() string { return node.name }
+
+// NamePath returns a slice of keys making up the path to the field
+func (node *schemaNode) NamePath() []string {
+ if len(node.path) == 0 {
+ var path []string
+ cur := node
+ for i := node.depth - 1; i >= 0; i-- {
+ path = append([]string{cur.name}, path...)
+ cur = cur.parent
+ }
+ return path
+ }
+ return node.path
+}
+
+// ArrowSchemaFromAvro returns a new Arrow schema from an Avro schema
+func ArrowSchemaFromAvro(schema avro.Schema) (s *arrow.Schema, err error) {
+ defer func() {
+ if r := recover(); r != nil {
+ s = nil
+ err = fmt.Errorf("invalid avro schema: could not convert schema")
+ }
+ }()
+ n := newSchemaNode()
+ n.schema = schema
+ c := n.newChild(n.schema.(avro.NamedSchema).Name(), n.schema)
+ arrowSchemafromAvro(c)
+ var fields []arrow.Field
+ for _, g := range c.Children() {
+ fields = append(fields, g.arrowField)
+ }
+ s = arrow.NewSchema(fields, nil)
+ return s, nil
+}
+
+func arrowSchemafromAvro(n *schemaNode) {
+ if ns, ok := n.schema.(avro.NamedSchema); ok {
+ schemaCache.Add(ns.Name(), ns)
+ }
+ switch st := n.schema.Type(); st {
+ case "record":
+ iterateFields(n)
+ case "enum":
+ symbols := make(map[string]string)
+ for index, symbol := range n.schema.(avro.PropertySchema).(*avro.EnumSchema).Symbols() {
+ k := strconv.FormatInt(int64(index), 10)
+ symbols[k] = symbol
+ }
+ var dt arrow.DictionaryType = arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Uint64, ValueType: arrow.BinaryTypes.String, Ordered: false}
+ sl := len(symbols)
+ switch {
+ case sl <= math.MaxUint8:
+ dt.IndexType = arrow.PrimitiveTypes.Uint8
+ case sl > math.MaxUint8 && sl <= math.MaxUint16:
+ dt.IndexType = arrow.PrimitiveTypes.Uint16
+ case sl > math.MaxUint16 && sl <= math.MaxUint32:
+ dt.IndexType = arrow.PrimitiveTypes.Uint32
+ }
+ n.arrowField = arrow.Field{Name: n.Name(), Type: &dt, Nullable: true, Metadata: arrow.MetadataFrom(symbols)}
+ case "array":
+ // logical items type
+ c := n.newChild(n.name, n.schema.(*avro.ArraySchema).Items())
+ if isLogicalSchemaType(n.schema.(*avro.ArraySchema).Items()) {
+ avroLogicalToArrowField(c)
+ } else {
+ arrowSchemafromAvro(c)
+ }
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.ListOf(c.arrowField.Type)}
+ case "map":
+ c := n.newChild(n.name, n.schema.(*avro.MapSchema).Values())
+ arrowSchemafromAvro(c)
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.MapOf(arrow.BinaryTypes.String, c.arrowField.Type), Metadata: c.arrowField.Metadata, Nullable: true}
+ case "union":
+ if n.schema.(*avro.UnionSchema).Nullable() {
+ if len(n.schema.(*avro.UnionSchema).Types()) > 1 {
+ n.schema = n.schema.(*avro.UnionSchema).Types()[1]
+ n.union = true
+ arrowSchemafromAvro(n)
+ }
+ }
+ // Avro "fixed" field type = Arrow FixedSize Primitive BinaryType
+ case "fixed":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.Name(), Type: &arrow.FixedSizeBinaryType{ByteWidth: n.schema.(*avro.FixedSchema).Size()}, Nullable: true}
+ }
+ case "string":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "bytes":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "int":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "long":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "float":
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ case "double":
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ case "boolean":
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ case "<ref>":
+ n.schema = schemaCache.Get(n.schema.(*avro.RefSchema).Schema().Name())
+ arrowSchemafromAvro(n)
+ case "null":
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.BinaryTypes.Binary, Nullable: true}
+ }
+}
+
+// iterate record Fields()
+func iterateFields(n *schemaNode) {
+ for _, f := range n.schema.(avro.PropertySchema).(*avro.RecordSchema).Fields() {
+ switch ft := f.Type().(type) {
+ // Avro "array" field type
+ case *avro.ArraySchema:
+ schemaCache.Add(f.Name(), ft.Items())
+ // logical items type
+ c := n.newChild(f.Name(), ft.Items())
+ if isLogicalSchemaType(ft.Items()) {
+ avroLogicalToArrowField(c)
+ } else {
+ arrowSchemafromAvro(c)
+ }
+ c.arrowField = arrow.Field{Name: c.name, Type: arrow.ListOf(c.arrowField.Type), Metadata: c.arrowField.Metadata, Nullable: true}
+ // Avro "enum" field type = Arrow dictionary type
+ case *avro.EnumSchema:
+ schemaCache.Add(f.Name(), f.Type())
+ c := n.newChild(f.Name(), f.Type())
+ symbols := make(map[string]string)
+ for index, symbol := range ft.Symbols() {
+ k := strconv.FormatInt(int64(index), 10)
+ symbols[k] = symbol
+ }
+ var dt arrow.DictionaryType = arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Uint64, ValueType: arrow.BinaryTypes.String, Ordered: false}
+ sl := len(symbols)
+ switch {
+ case sl <= math.MaxUint8:
+ dt.IndexType = arrow.PrimitiveTypes.Uint8
+ case sl > math.MaxUint8 && sl <= math.MaxUint16:
+ dt.IndexType = arrow.PrimitiveTypes.Uint16
+ case sl > math.MaxUint16 && sl <= math.MaxUint32:
+ dt.IndexType = arrow.PrimitiveTypes.Uint32
+ }
+ c.arrowField = arrow.Field{Name: f.Name(), Type: &dt, Nullable: true, Metadata: arrow.MetadataFrom(symbols)}
+ // Avro "fixed" field type = Arrow FixedSize Primitive BinaryType
+ case *avro.FixedSchema:
+ schemaCache.Add(f.Name(), f.Type())
+ if isLogicalSchemaType(f.Type()) {
+ c := n.newChild(f.Name(), f.Type())
+ avroLogicalToArrowField(c)
+ } else {
+ c := n.newChild(f.Name(), f.Type())
+ arrowSchemafromAvro(c)
+ }
+ case *avro.RecordSchema:
+ schemaCache.Add(f.Name(), f.Type())
+ c := n.newChild(f.Name(), f.Type())
+ iterateFields(c)
+ // Avro "map" field type - KVP with value of one type - keys are strings
+ case *avro.MapSchema:
+ schemaCache.Add(f.Name(), ft.Values())
+ c := n.newChild(f.Name(), ft.Values())
+ arrowSchemafromAvro(c)
+ if ft.Values().Type() == "union" {
+ c.arrowField = arrow.Field{Name: f.Name(), Type: arrow.MapOf(arrow.BinaryTypes.String, arrow.BinaryTypes.String), Metadata: c.arrowField.Metadata, Nullable: true}
+ } else {
+ c.arrowField = arrow.Field{Name: f.Name(), Type: arrow.MapOf(arrow.BinaryTypes.String, c.arrowField.Type), Metadata: c.arrowField.Metadata, Nullable: true}
+ }
+ case *avro.UnionSchema:
+ if ft.Nullable() {
+ if len(ft.Types()) > 1 {
+ schemaCache.Add(f.Name(), ft.Types()[1])
+ c := n.newChild(f.Name(), ft.Types()[1])
+ c.union = true
+ arrowSchemafromAvro(c)
+ }
+ }
+ default:
+ schemaCache.Add(f.Name(), f.Type())
+ if isLogicalSchemaType(f.Type()) {
+ c := n.newChild(f.Name(), f.Type())
+ avroLogicalToArrowField(c)
+ } else {
+ c := n.newChild(f.Name(), f.Type())
+ arrowSchemafromAvro(c)
+ }
+
+ }
+ }
+ var fields []arrow.Field
+ for _, child := range n.Children() {
+ fields = append(fields, child.arrowField)
+ }
+
+ namedSchema, ok := isNamedSchema(n.schema)
+ if !ok || namedSchema == n.name+"_data" || !n.union {
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.StructOf(fields...), Nullable: true}
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.StructOf(fields...), Nullable: true, Metadata: arrow.MetadataFrom(map[string]string{"typeName": namedSchema})}
+ }
+
+}
+
+func isLogicalSchemaType(s avro.Schema) bool {
+ lts, ok := s.(avro.LogicalTypeSchema)
+ if !ok {
+ return false
+ }
+ if lt := lts.Logical(); lt != nil {
+ return true
+ }
+ return false
+}
+
+func isNamedSchema(s avro.Schema) (string, bool) {
+ if ns, ok := s.(avro.NamedSchema); ok {
+ return ns.FullName(), ok
+ }
+ return "", false
+}
+
+// Avro primitive type.
+//
+// NOTE: Arrow Binary type is used as a catchall to avoid potential data loss.
Review Comment:
instead of returning a "NotImplemented" error?
##########
go/arrow/avro/schema.go:
##########
@@ -0,0 +1,407 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package avro reads Avro OCF files and presents the extracted data as records
+
+package avro
+
+import (
+ "fmt"
+ "math"
+ "strconv"
+
+ "github.com/apache/arrow/go/v14/arrow"
+ "github.com/apache/arrow/go/v14/internal/types"
+ avro "github.com/hamba/avro/v2"
+)
+
+type schemaNode struct {
+ name string
+ parent *schemaNode
+ schema avro.Schema
+ union bool
+ children []*schemaNode
+ arrowField arrow.Field
+ path []string
+ index, depth int32
+}
+
+var schemaCache avro.SchemaCache
+
+func newSchemaNode() *schemaNode { return &schemaNode{name: "", index: -1} }
+
+func (node *schemaNode) newChild(n string, s avro.Schema) *schemaNode {
+ child := &schemaNode{
+ name: n,
+ parent: node,
+ schema: s,
+ index: int32(len(node.children)),
+ depth: node.depth + 1}
+ node.children = append(node.children, child)
+ return child
+}
+func (node *schemaNode) Children() []*schemaNode { return node.children }
+
+func (node *schemaNode) Name() string { return node.name }
+
+// NamePath returns a slice of keys making up the path to the field
+func (node *schemaNode) NamePath() []string {
+ if len(node.path) == 0 {
+ var path []string
+ cur := node
+ for i := node.depth - 1; i >= 0; i-- {
+ path = append([]string{cur.name}, path...)
+ cur = cur.parent
+ }
+ return path
+ }
+ return node.path
+}
+
+// ArrowSchemaFromAvro returns a new Arrow schema from an Avro schema
+func ArrowSchemaFromAvro(schema avro.Schema) (s *arrow.Schema, err error) {
+ defer func() {
+ if r := recover(); r != nil {
+ s = nil
+ err = fmt.Errorf("invalid avro schema: could not convert schema")
+ }
+ }()
+ n := newSchemaNode()
+ n.schema = schema
+ c := n.newChild(n.schema.(avro.NamedSchema).Name(), n.schema)
+ arrowSchemafromAvro(c)
+ var fields []arrow.Field
+ for _, g := range c.Children() {
+ fields = append(fields, g.arrowField)
+ }
+ s = arrow.NewSchema(fields, nil)
+ return s, nil
+}
+
+func arrowSchemafromAvro(n *schemaNode) {
+ if ns, ok := n.schema.(avro.NamedSchema); ok {
+ schemaCache.Add(ns.Name(), ns)
+ }
+ switch st := n.schema.Type(); st {
+ case "record":
+ iterateFields(n)
+ case "enum":
+ symbols := make(map[string]string)
+ for index, symbol := range n.schema.(avro.PropertySchema).(*avro.EnumSchema).Symbols() {
+ k := strconv.FormatInt(int64(index), 10)
+ symbols[k] = symbol
+ }
+ var dt arrow.DictionaryType = arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Uint64, ValueType: arrow.BinaryTypes.String, Ordered: false}
+ sl := len(symbols)
+ switch {
+ case sl <= math.MaxUint8:
+ dt.IndexType = arrow.PrimitiveTypes.Uint8
+ case sl > math.MaxUint8 && sl <= math.MaxUint16:
+ dt.IndexType = arrow.PrimitiveTypes.Uint16
+ case sl > math.MaxUint16 && sl <= math.MaxUint32:
+ dt.IndexType = arrow.PrimitiveTypes.Uint32
+ }
+ n.arrowField = arrow.Field{Name: n.Name(), Type: &dt, Nullable: true, Metadata: arrow.MetadataFrom(symbols)}
+ case "array":
+ // logical items type
+ c := n.newChild(n.name, n.schema.(*avro.ArraySchema).Items())
+ if isLogicalSchemaType(n.schema.(*avro.ArraySchema).Items()) {
+ avroLogicalToArrowField(c)
+ } else {
+ arrowSchemafromAvro(c)
+ }
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.ListOf(c.arrowField.Type)}
+ case "map":
+ c := n.newChild(n.name, n.schema.(*avro.MapSchema).Values())
+ arrowSchemafromAvro(c)
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.MapOf(arrow.BinaryTypes.String, c.arrowField.Type), Metadata: c.arrowField.Metadata, Nullable: true}
+ case "union":
+ if n.schema.(*avro.UnionSchema).Nullable() {
+ if len(n.schema.(*avro.UnionSchema).Types()) > 1 {
+ n.schema = n.schema.(*avro.UnionSchema).Types()[1]
+ n.union = true
+ arrowSchemafromAvro(n)
+ }
+ }
+ // Avro "fixed" field type = Arrow FixedSize Primitive BinaryType
+ case "fixed":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.Name(), Type: &arrow.FixedSizeBinaryType{ByteWidth: n.schema.(*avro.FixedSchema).Size()}, Nullable: true}
+ }
+ case "string":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "bytes":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "int":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "long":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "float":
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ case "double":
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ case "boolean":
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ case "<ref>":
+ n.schema = schemaCache.Get(n.schema.(*avro.RefSchema).Schema().Name())
+ arrowSchemafromAvro(n)
+ case "null":
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.BinaryTypes.Binary, Nullable: true}
+ }
+}
+
+// iterate record Fields()
+func iterateFields(n *schemaNode) {
+ for _, f := range n.schema.(avro.PropertySchema).(*avro.RecordSchema).Fields() {
+ switch ft := f.Type().(type) {
+ // Avro "array" field type
+ case *avro.ArraySchema:
+ schemaCache.Add(f.Name(), ft.Items())
+ // logical items type
+ c := n.newChild(f.Name(), ft.Items())
+ if isLogicalSchemaType(ft.Items()) {
+ avroLogicalToArrowField(c)
+ } else {
+ arrowSchemafromAvro(c)
+ }
+ c.arrowField = arrow.Field{Name: c.name, Type: arrow.ListOf(c.arrowField.Type), Metadata: c.arrowField.Metadata, Nullable: true}
+ // Avro "enum" field type = Arrow dictionary type
+ case *avro.EnumSchema:
+ schemaCache.Add(f.Name(), f.Type())
+ c := n.newChild(f.Name(), f.Type())
+ symbols := make(map[string]string)
+ for index, symbol := range ft.Symbols() {
+ k := strconv.FormatInt(int64(index), 10)
+ symbols[k] = symbol
+ }
+ var dt arrow.DictionaryType = arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Uint64, ValueType: arrow.BinaryTypes.String, Ordered: false}
+ sl := len(symbols)
+ switch {
+ case sl <= math.MaxUint8:
+ dt.IndexType = arrow.PrimitiveTypes.Uint8
+ case sl > math.MaxUint8 && sl <= math.MaxUint16:
+ dt.IndexType = arrow.PrimitiveTypes.Uint16
+ case sl > math.MaxUint16 && sl <= math.MaxUint32:
+ dt.IndexType = arrow.PrimitiveTypes.Uint32
+ }
+ c.arrowField = arrow.Field{Name: f.Name(), Type: &dt, Nullable: true, Metadata: arrow.MetadataFrom(symbols)}
+ // Avro "fixed" field type = Arrow FixedSize Primitive BinaryType
+ case *avro.FixedSchema:
+ schemaCache.Add(f.Name(), f.Type())
+ if isLogicalSchemaType(f.Type()) {
+ c := n.newChild(f.Name(), f.Type())
+ avroLogicalToArrowField(c)
+ } else {
+ c := n.newChild(f.Name(), f.Type())
+ arrowSchemafromAvro(c)
+ }
+ case *avro.RecordSchema:
+ schemaCache.Add(f.Name(), f.Type())
+ c := n.newChild(f.Name(), f.Type())
+ iterateFields(c)
+ // Avro "map" field type - KVP with value of one type - keys are strings
+ case *avro.MapSchema:
+ schemaCache.Add(f.Name(), ft.Values())
+ c := n.newChild(f.Name(), ft.Values())
+ arrowSchemafromAvro(c)
+ if ft.Values().Type() == "union" {
+ c.arrowField = arrow.Field{Name: f.Name(), Type: arrow.MapOf(arrow.BinaryTypes.String, arrow.BinaryTypes.String), Metadata: c.arrowField.Metadata, Nullable: true}
+ } else {
+ c.arrowField = arrow.Field{Name: f.Name(), Type: arrow.MapOf(arrow.BinaryTypes.String, c.arrowField.Type), Metadata: c.arrowField.Metadata, Nullable: true}
+ }
+ case *avro.UnionSchema:
+ if ft.Nullable() {
+ if len(ft.Types()) > 1 {
+ schemaCache.Add(f.Name(), ft.Types()[1])
+ c := n.newChild(f.Name(), ft.Types()[1])
+ c.union = true
+ arrowSchemafromAvro(c)
+ }
+ }
+ default:
+ schemaCache.Add(f.Name(), f.Type())
+ if isLogicalSchemaType(f.Type()) {
+ c := n.newChild(f.Name(), f.Type())
+ avroLogicalToArrowField(c)
+ } else {
+ c := n.newChild(f.Name(), f.Type())
+ arrowSchemafromAvro(c)
+ }
+
+ }
+ }
+ var fields []arrow.Field
+ for _, child := range n.Children() {
+ fields = append(fields, child.arrowField)
+ }
+
+ namedSchema, ok := isNamedSchema(n.schema)
+ if !ok || namedSchema == n.name+"_data" || !n.union {
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.StructOf(fields...), Nullable: true}
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.StructOf(fields...), Nullable: true, Metadata: arrow.MetadataFrom(map[string]string{"typeName": namedSchema})}
+ }
+
+}
+
+func isLogicalSchemaType(s avro.Schema) bool {
+ lts, ok := s.(avro.LogicalTypeSchema)
+ if !ok {
+ return false
+ }
+ if lt := lts.Logical(); lt != nil {
+ return true
+ }
+ return false
+}
+
+func isNamedSchema(s avro.Schema) (string, bool) {
+ if ns, ok := s.(avro.NamedSchema); ok {
+ return ns.FullName(), ok
+ }
+ return "", false
+}
+
+// Avro primitive type.
+//
+// NOTE: Arrow Binary type is used as a catchall to avoid potential data loss.
+func avroPrimitiveToArrowType(avroFieldType string) arrow.DataType {
+ switch avroFieldType {
+ // int: 32-bit signed integer
+ case "int":
+ return arrow.PrimitiveTypes.Int32
+ // long: 64-bit signed integer
+ case "long":
+ return arrow.PrimitiveTypes.Int64
+ // float: single precision (32-bit) IEEE 754 floating-point number
+ case "float":
+ return arrow.PrimitiveTypes.Float32
+ // double: double precision (64-bit) IEEE 754 floating-point number
+ case "double":
+ return arrow.PrimitiveTypes.Float64
+ // bytes: sequence of 8-bit unsigned bytes
+ case "bytes":
+ return arrow.BinaryTypes.Binary
+ // boolean: a binary value
+ case "boolean":
+ return arrow.FixedWidthTypes.Boolean
+ // string: unicode character sequence
+ case "string":
+ return arrow.BinaryTypes.String
+ // fallback to binary type for any unsupported type
+ // (shouldn't happen as all types specified in avro 1.11.1 are supported)
+ default:
+ return arrow.BinaryTypes.Binary
+ }
+}
+
+func avroLogicalToArrowField(n *schemaNode) arrow.Field {
+ var dt arrow.DataType
+ // Avro logical types
+ switch lt := n.schema.(avro.LogicalTypeSchema).Logical(); lt.Type() {
+ // The decimal logical type represents an arbitrary-precision signed decimal number of the form unscaled × 10-scale.
+ // A decimal logical type annotates Avro bytes or fixed types. The byte array must contain the two’s-complement
+ // representation of the unscaled integer value in big-endian byte order. The scale is fixed, and is specified
+ // using an attribute.
+ //
+ // The following attributes are supported:
+ // scale, a JSON integer representing the scale (optional). If not specified the scale is 0.
+ // precision, a JSON integer representing the (maximum) precision of decimals stored in this type (required).
+ case "decimal":
+ id := arrow.DECIMAL128
+ if lt.(*avro.DecimalLogicalSchema).Precision() > 38 {
Review Comment:
use the constant, `decimal128.MaxPrecision`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] loicalleyne commented on a diff in pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne commented on code in PR #36796:
URL: https://github.com/apache/arrow/pull/36796#discussion_r1316464248
##########
go/arrow/avro/schema.go:
##########
@@ -0,0 +1,407 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package avro reads Avro OCF files and presents the extracted data as records
+
+package avro
+
+import (
+ "fmt"
+ "math"
+ "strconv"
+
+ "github.com/apache/arrow/go/v14/arrow"
+ "github.com/apache/arrow/go/v14/internal/types"
+ avro "github.com/hamba/avro/v2"
+)
+
+type schemaNode struct {
+ name string
+ parent *schemaNode
+ schema avro.Schema
+ union bool
+ children []*schemaNode
+ arrowField arrow.Field
+ path []string
+ index, depth int32
+}
+
+var schemaCache avro.SchemaCache
Review Comment:
ok, I think the only way to not have a global variable would be to pass a pointer to the schema cache to the schemaNode object and all its children; will do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #36796:
URL: https://github.com/apache/arrow/pull/36796#issuecomment-1710789009
@loicalleyne Hmmm, i didn't realize that. I think i'm fine with us bumping the minimum to go1.19 and dropping the go1.17/go1.18 builds. I'll put together a PR for that. Let's see if we can dig in and figure out the bug in the `schema.Equals` method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] loicalleyne commented on a diff in pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne commented on code in PR #36796:
URL: https://github.com/apache/arrow/pull/36796#discussion_r1281181640
##########
go/go.mod:
##########
@@ -59,6 +59,7 @@ require (
github.com/golang/protobuf v1.5.2 // indirect
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 // indirect
github.com/kr/text v0.2.0 // indirect
+ github.com/linkedin/goavro/v2 v2.12.0 // indirect
Review Comment:
had a question about licence compatibility but a Google search answered it, will look to moving avro reader to Hamba
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] loicalleyne commented on a diff in pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne commented on code in PR #36796:
URL: https://github.com/apache/arrow/pull/36796#discussion_r1290603822
##########
go/arrow/avro/schema.go:
##########
@@ -0,0 +1,409 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package avro reads Avro OCF files and presents the extracted data as records
+package avro
+
+import (
+ "encoding/json"
+ "fmt"
+ "math"
+ "strconv"
+
+ "github.com/apache/arrow/go/v13/arrow"
+)
+
+type schemaNode struct {
+ name string
+ ofType interface{}
+ logicalType string
+ precision int32
+ scale int32
+ size int
+ symbols []string
+ fields []interface{}
+}
+
+// ArrowSchemaFromAvro returns a new Arrow schema from an Avro schema JSON.
+// If the top level is of record type, set includeTopLevel to either make
+// its fields top level fields in the resulting schema or nested in a single field.
+func ArrowSchemaFromAvro(avroSchema []byte) (*arrow.Schema, error) {
+ var m map[string]interface{}
+ var node schemaNode
+ json.Unmarshal(avroSchema, &m)
+ if m["type"].(string) == "record" {
+ if _, ok := m["fields"]; ok {
+ for _, field := range m["fields"].([]interface{}) {
+ node.fields = append(node.fields, field.(map[string]interface{}))
+ }
+ if len(node.fields) == 0 {
+ return nil, fmt.Errorf("invalid avro schema: no top level record fields found")
+ }
+ }
+ }
+ fields := iterateFields(node.fields)
+ return arrow.NewSchema(fields, nil), nil
+}
+
+func iterateFields(f []interface{}) []arrow.Field {
+ var s []arrow.Field
+ for _, field := range f {
+ var n schemaNode
+ n.name = field.(map[string]interface{})["name"].(string)
+ n.ofType = field.(map[string]interface{})["type"]
+ switch n.ofType.(type) {
+ // Getting field type from within field object
+ case string:
+ switch n.ofType.(string) {
+ case "enum":
+ for _, symbol := range field.(map[string]interface{})["symbols"].([]interface{}) {
+ n.symbols = append(n.symbols, symbol.(string))
+ }
+ default:
+ if lt, ok := field.(map[string]interface{})["logicalType"]; ok {
+ n.logicalType = lt.(string)
+ }
+ if lt, ok := field.(map[string]interface{})["size"]; ok {
+ n.size = int(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["precision"]; ok {
+ n.precision = int32(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["scale"]; ok {
+ n.scale = int32(lt.(float64))
+ }
+ }
+ // Field type is an object
+ case map[string]interface{}:
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["logicalType"]; ok {
+ n.logicalType = lt.(string)
+ }
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["size"]; ok {
+ n.size = int(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["precision"]; ok {
+ n.precision = int32(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["scale"]; ok {
+ n.scale = int32(lt.(float64))
+ }
+ case []interface{}:
+
+ default:
+ if lt, ok := field.(map[string]interface{})["logicalType"]; ok {
+ n.logicalType = lt.(string)
+ }
+ if lt, ok := field.(map[string]interface{})["size"]; ok {
+ n.size = int(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["precision"]; ok {
+ n.precision = int32(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["scale"]; ok {
+ n.scale = int32(lt.(float64))
+ }
+ }
+ // Field is of type "record"
+ if nf, f := field.(map[string]interface{})["fields"]; f {
+ switch nf.(type) {
+ // primitive & complex types
+ case map[string]interface{}:
+ for _, v := range nf.(map[string]interface{})["fields"].([]interface{}) {
+ n.fields = append(n.fields, v.(map[string]interface{}))
+ }
+ // type unions
+ default:
+ for _, v := range nf.([]interface{}) {
+ n.fields = append(n.fields, v.(map[string]interface{}))
+ }
+ }
+ }
+ s = append(s, traverseNodes(n))
+ }
+ return s
+}
+
+func traverseNodes(node schemaNode) arrow.Field {
+ switch node.ofType.(type) {
+ case string:
+ // Avro primitive type
+ if len(node.fields) == 0 {
+ switch node.ofType.(string) {
+ case "boolean", "int", "long", "float", "double", "bytes", "string":
+ if node.logicalType != "" {
+ return avroLogicalToArrowField(node)
+ }
+ // Avro primitive type
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(node.ofType.(string))}
+ case "fixed":
+ // Duration type is not supported in github.com/linkedin/goavro
+ // Implementing as Binary for now.
+ switch node.logicalType {
+ case "decimal":
+ return avroLogicalToArrowField(node)
+ case "duration":
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.Binary}
+ //return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.MonthDayNanoInterval}
+ }
+ return arrow.Field{Name: node.name, Type: &arrow.FixedSizeBinaryType{ByteWidth: node.size}}
+ case "enum":
+ symbols := make(map[string]string)
+ for index, symbol := range node.symbols {
+ k := strconv.FormatInt(int64(index), 10)
+ symbols[k] = symbol
+ }
+ var dt arrow.DictionaryType = arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Uint64, ValueType: arrow.BinaryTypes.String, Ordered: false}
+ sl := len(symbols)
+ switch {
+ case sl <= math.MaxUint8:
+ dt.IndexType = arrow.PrimitiveTypes.Uint8
+ case sl > math.MaxUint8 && sl <= math.MaxUint16:
+ dt.IndexType = arrow.PrimitiveTypes.Uint16
+ case sl > math.MaxUint16 && sl <= math.MaxUint32:
+ dt.IndexType = arrow.PrimitiveTypes.Uint32
+ }
+ return arrow.Field{Name: node.name, Type: &dt, Nullable: true, Metadata: arrow.MetadataFrom(symbols)}
+ default:
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(node.ofType.(string))}
+ }
+ } else {
+ // avro "record" type, node has "fields" array
+ if node.ofType.(string) == "record" {
+ var n schemaNode
+ n.name = node.name
+ n.ofType = node.ofType
+ if len(node.fields) > 0 {
+ n.fields = append(n.fields, node.fields...)
+ }
+ f := iterateFields(n.fields)
+ return arrow.Field{Name: node.name, Type: arrow.StructOf(f...)}
+ }
+ }
+ // Avro complex types
+ case map[string]interface{}:
+ //var n schemaNode
+ //n.name = node.name
+ //n.ofType = node.ofType.(map[string]interface{})["type"]
+ switch node.logicalType {
+ case "":
+ return avroComplexToArrowField(node)
+ default:
+ return avroLogicalToArrowField(node)
+ }
+
+ // Avro union types
+ case []interface{}:
+ var unionTypes []string
+ for _, ft := range node.ofType.([]interface{}) {
+ switch ft.(type) {
+ // primitive types
+ case string:
+ if ft != "null" {
+ unionTypes = append(unionTypes, ft.(string))
+ }
+ continue
+ // complex types
+ case map[string]interface{}:
+ var n schemaNode
+ n.name = node.name
+ n.ofType = ft.(map[string]interface{})["type"]
+ if _, f := ft.(map[string]interface{})["fields"]; f {
+ for _, field := range ft.(map[string]interface{})["fields"].([]interface{}) {
+ n.fields = append(n.fields, field.(map[string]interface{}))
+ }
+ }
+ f := iterateFields(n.fields)
+ return arrow.Field{Name: node.name, Type: arrow.StructOf(f...)}
+ }
+ }
+ // Supported Avro union type is null + one other type.
+ // TODO: Complex AVRO union to Arrow Dense || Sparse Union.
+ if len(unionTypes) == 1 {
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(unionTypes[0])}
+ } else {
+ // BYTE_ARRAY is the catchall if union type is anything beyond null + one other type.
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.Binary}
+ }
+ }
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.Binary}
+}
+
+func avroLogicalToArrowField(node schemaNode) arrow.Field {
+ // Avro logical types
+ switch node.logicalType {
+ // The decimal logical type represents an arbitrary-precision signed decimal number of the form unscaled × 10-scale.
+ // A decimal logical type annotates Avro bytes or fixed types. The byte array must contain the two’s-complement
+ // representation of the unscaled integer value in big-endian byte order. The scale is fixed, and is specified
+ // using an attribute.
+ //
+ // The following attributes are supported:
+ // scale, a JSON integer representing the scale (optional). If not specified the scale is 0.
+ // precision, a JSON integer representing the (maximum) precision of decimals stored in this type (required).
+ case "decimal":
+ if node.precision <= 38 {
+ return arrow.Field{Name: node.name, Type: &arrow.Decimal128Type{Precision: node.precision, Scale: node.scale}}
+ } else {
+ return arrow.Field{Name: node.name, Type: &arrow.Decimal256Type{Precision: node.precision, Scale: node.scale}}
+ }
+
+ // The uuid logical type represents a random generated universally unique identifier (UUID).
+ // A uuid logical type annotates an Avro string. The string has to conform with RFC-4122
+ case "uuid":
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.String}
+
+ // The date logical type represents a date within the calendar, with no reference to a particular
+ // time zone or time of day.
+ // A date logical type annotates an Avro int, where the int stores the number of days from the unix epoch,
+ // 1 January 1970 (ISO calendar).
+ case "date":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Date32}
+
+ // The time-millis logical type represents a time of day, with no reference to a particular calendar,
+ // time zone or date, with a precision of one millisecond.
+ // A time-millis logical type annotates an Avro int, where the int stores the number of milliseconds
+ // after midnight, 00:00:00.000.
+ case "time-millis":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Time32ms}
+
+ // The time-micros logical type represents a time of day, with no reference to a particular calendar,
+ // time zone or date, with a precision of one microsecond.
+ // A time-micros logical type annotates an Avro long, where the long stores the number of microseconds
+ // after midnight, 00:00:00.000000.
+ case "time-micros":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Time64us}
+
+ // The timestamp-millis logical type represents an instant on the global timeline, independent of a
+ // particular time zone or calendar, with a precision of one millisecond. Please note that time zone
+ // information gets lost in this process. Upon reading a value back, we can only reconstruct the instant,
+ // but not the original representation. In practice, such timestamps are typically displayed to users in
+ // their local time zones, therefore they may be displayed differently depending on the execution environment.
+ // A timestamp-millis logical type annotates an Avro long, where the long stores the number of milliseconds
+ // from the unix epoch, 1 January 1970 00:00:00.000 UTC.
+ case "timestamp-millis":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Timestamp_ms}
+ // The timestamp-micros logical type represents an instant on the global timeline, independent of a
+ // particular time zone or calendar, with a precision of one microsecond. Please note that time zone
+ // information gets lost in this process. Upon reading a value back, we can only reconstruct the instant,
+ // but not the original representation. In practice, such timestamps are typically displayed to users
+ // in their local time zones, therefore they may be displayed differently depending on the execution environment.
+ // A timestamp-micros logical type annotates an Avro long, where the long stores the number of microseconds
+ // from the unix epoch, 1 January 1970 00:00:00.000000 UTC.
+ case "timestamp-micros":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Timestamp_us}
+ // The local-timestamp-millis logical type represents a timestamp in a local timezone, regardless of
+ // what specific time zone is considered local, with a precision of one millisecond.
+ // A local-timestamp-millis logical type annotates an Avro long, where the long stores the number of
+ // milliseconds, from 1 January 1970 00:00:00.000.
+ case "local-timestamp-millis":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Timestamp_ms}
+ // The local-timestamp-micros logical type represents a timestamp in a local timezone, regardless of
+ // what specific time zone is considered local, with a precision of one microsecond.
+ // A local-timestamp-micros logical type annotates an Avro long, where the long stores the number of
+ // microseconds, from 1 January 1970 00:00:00.000000.
+ case "local-timestamp-micros":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Timestamp_us}
+ // Avro primitive type
+ default:
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(node.ofType.(string))}
+ }
+ return arrow.Field{}
+}
+
+func avroComplexToArrowField(node schemaNode) arrow.Field {
+ var n schemaNode
+ n.name = node.name
+ n.ofType = node.ofType.(map[string]interface{})["type"]
+ // Avro "array" field type
+ if i, ok := node.ofType.(map[string]interface{})["items"]; ok {
+ switch i.(string) {
+ case "int", "long", "float", "double", "bytes", "boolean", "string":
+ return arrow.Field{Name: node.name, Type: arrow.ListOf(AvroPrimitiveToArrowType(i.(string)))}
+ case "enum", "fixed", "map", "record", "array":
+ return arrow.Field{Name: node.name, Type: arrow.ListOf(avroComplexToArrowField(n).Type)}
+ case "decimal", "uuid", "date", "time-millis", "time-micros", "timestamp-millis", "timestamp-micros", "local-timestamp-millis", "local-timestamp-micros":
+ return arrow.Field{Name: node.name, Type: arrow.ListOf(avroLogicalToArrowField(n).Type)}
+ }
+ }
+ // Avro "enum" field type = Arrow dictionary type
+ if i, ok := node.ofType.(map[string]interface{})["symbols"]; ok {
+ symbols := make(map[string]string)
+ for index, symbol := range i.([]interface{}) {
+ k := strconv.FormatInt(int64(index), 10)
+ symbols[k] = symbol.(string)
+ }
+ var dt arrow.DictionaryType = arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Uint64, ValueType: arrow.BinaryTypes.String, Ordered: false}
+ sl := len(symbols)
+ switch {
+ case sl <= math.MaxUint8:
+ dt.IndexType = arrow.PrimitiveTypes.Uint8
+ case sl > math.MaxUint8 && sl <= math.MaxUint16:
+ dt.IndexType = arrow.PrimitiveTypes.Uint16
+ case sl > math.MaxUint16 && sl <= math.MaxUint32:
+ dt.IndexType = arrow.PrimitiveTypes.Uint32
+ }
+ return arrow.Field{Name: node.name, Type: &dt, Nullable: true, Metadata: arrow.MetadataFrom(symbols)}
+ }
+ // Avro "fixed" field type = Arrow FixedSize Primitive BinaryType
+ if i, ok := node.ofType.(map[string]interface{})["size"]; ok {
+ return arrow.Field{Name: node.name, Type: &arrow.FixedSizeBinaryType{ByteWidth: int(i.(float64))}}
+ }
+ // Avro "map" field type
+ if i, ok := node.ofType.(map[string]interface{})["values"]; ok {
+ return arrow.Field{Name: node.name, Type: arrow.MapOf(arrow.BinaryTypes.String, AvroPrimitiveToArrowType(i.(string)))}
+ }
+ // Avro "record" field type
+ if _, f := node.ofType.(map[string]interface{})["fields"]; f {
+ for _, field := range node.ofType.(map[string]interface{})["fields"].([]interface{}) {
+ n.fields = append(n.fields, field.(map[string]interface{}))
+ }
+ s := iterateFields(n.fields)
+ return arrow.Field{Name: n.name, Type: arrow.StructOf(s...)}
+ }
+ return arrow.Field{}
+}
+
+// AvroPrimitiveToArrowType returns the Arrow DataType equivalent to a
+// Avro primitive type.
+//
+// NOTE: Arrow Binary type is used as a catchall to avoid potential data loss.
+func AvroPrimitiveToArrowType(avroFieldType string) arrow.DataType {
+ switch avroFieldType {
+ // int: 32-bit signed integer
+ case "int":
+ return arrow.PrimitiveTypes.Int32
+ // long: 64-bit signed integer
+ case "long":
+ return arrow.PrimitiveTypes.Int64
+ // float: single precision (32-bit) IEEE 754 floating-point number
+ case "float":
+ return arrow.PrimitiveTypes.Float32
+ // double: double precision (64-bit) IEEE 754 floating-point number
+ case "double":
+ return arrow.PrimitiveTypes.Float64
+ // bytes: sequence of 8-bit unsigned bytes
+ case "bytes":
+ return arrow.BinaryTypes.Binary
+ // boolean: a binary value
+ case "boolean":
+ return arrow.FixedWidthTypes.Boolean
+ // string: unicode character sequence
+ case "string":
+ return arrow.BinaryTypes.String
+ // fallback to binary type for any unsupported type
+ default:
+ return arrow.BinaryTypes.Binary
Review Comment:
hamba/avro supports duration logical type, all avro primitive and logical types are now implemented.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] loicalleyne commented on a diff in pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne commented on code in PR #36796:
URL: https://github.com/apache/arrow/pull/36796#discussion_r1290607426
##########
go/arrow/avro/schema.go:
##########
@@ -0,0 +1,409 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package avro reads Avro OCF files and presents the extracted data as records
+package avro
+
+import (
+ "encoding/json"
+ "fmt"
+ "math"
+ "strconv"
+
+ "github.com/apache/arrow/go/v13/arrow"
+)
+
+type schemaNode struct {
+ name string
+ ofType interface{}
+ logicalType string
+ precision int32
+ scale int32
+ size int
+ symbols []string
+ fields []interface{}
+}
+
+// ArrowSchemaFromAvro returns a new Arrow schema from an Avro schema JSON.
+// If the top level is of record type, set includeTopLevel to either make
+// its fields top level fields in the resulting schema or nested in a single field.
+func ArrowSchemaFromAvro(avroSchema []byte) (*arrow.Schema, error) {
+ var m map[string]interface{}
+ var node schemaNode
+ json.Unmarshal(avroSchema, &m)
+ if m["type"].(string) == "record" {
+ if _, ok := m["fields"]; ok {
+ for _, field := range m["fields"].([]interface{}) {
+ node.fields = append(node.fields, field.(map[string]interface{}))
+ }
+ if len(node.fields) == 0 {
+ return nil, fmt.Errorf("invalid avro schema: no top level record fields found")
+ }
+ }
+ }
+ fields := iterateFields(node.fields)
+ return arrow.NewSchema(fields, nil), nil
+}
+
+func iterateFields(f []interface{}) []arrow.Field {
+ var s []arrow.Field
+ for _, field := range f {
+ var n schemaNode
+ n.name = field.(map[string]interface{})["name"].(string)
+ n.ofType = field.(map[string]interface{})["type"]
+ switch n.ofType.(type) {
+ // Getting field type from within field object
+ case string:
+ switch n.ofType.(string) {
+ case "enum":
+ for _, symbol := range field.(map[string]interface{})["symbols"].([]interface{}) {
+ n.symbols = append(n.symbols, symbol.(string))
+ }
+ default:
+ if lt, ok := field.(map[string]interface{})["logicalType"]; ok {
+ n.logicalType = lt.(string)
+ }
+ if lt, ok := field.(map[string]interface{})["size"]; ok {
+ n.size = int(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["precision"]; ok {
+ n.precision = int32(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["scale"]; ok {
+ n.scale = int32(lt.(float64))
+ }
+ }
+ // Field type is an object
+ case map[string]interface{}:
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["logicalType"]; ok {
+ n.logicalType = lt.(string)
+ }
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["size"]; ok {
+ n.size = int(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["precision"]; ok {
+ n.precision = int32(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["scale"]; ok {
+ n.scale = int32(lt.(float64))
+ }
+ case []interface{}:
+
+ default:
+ if lt, ok := field.(map[string]interface{})["logicalType"]; ok {
+ n.logicalType = lt.(string)
+ }
+ if lt, ok := field.(map[string]interface{})["size"]; ok {
+ n.size = int(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["precision"]; ok {
+ n.precision = int32(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["scale"]; ok {
+ n.scale = int32(lt.(float64))
+ }
+ }
+ // Field is of type "record"
+ if nf, f := field.(map[string]interface{})["fields"]; f {
+ switch nf.(type) {
+ // primitive & complex types
+ case map[string]interface{}:
+ for _, v := range nf.(map[string]interface{})["fields"].([]interface{}) {
+ n.fields = append(n.fields, v.(map[string]interface{}))
+ }
+ // type unions
+ default:
+ for _, v := range nf.([]interface{}) {
+ n.fields = append(n.fields, v.(map[string]interface{}))
+ }
+ }
+ }
+ s = append(s, traverseNodes(n))
+ }
+ return s
+}
+
+func traverseNodes(node schemaNode) arrow.Field {
+ switch node.ofType.(type) {
+ case string:
+ // Avro primitive type
+ if len(node.fields) == 0 {
+ switch node.ofType.(string) {
+ case "boolean", "int", "long", "float", "double", "bytes", "string":
+ if node.logicalType != "" {
+ return avroLogicalToArrowField(node)
+ }
+ // Avro primitive type
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(node.ofType.(string))}
+ case "fixed":
+ // Duration type is not supported in github.com/linkedin/goavro
+ // Implementing as Binary for now.
+ switch node.logicalType {
+ case "decimal":
+ return avroLogicalToArrowField(node)
+ case "duration":
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.Binary}
+ //return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.MonthDayNanoInterval}
+ }
+ return arrow.Field{Name: node.name, Type: &arrow.FixedSizeBinaryType{ByteWidth: node.size}}
+ case "enum":
+ symbols := make(map[string]string)
+ for index, symbol := range node.symbols {
+ k := strconv.FormatInt(int64(index), 10)
+ symbols[k] = symbol
+ }
+ var dt arrow.DictionaryType = arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Uint64, ValueType: arrow.BinaryTypes.String, Ordered: false}
+ sl := len(symbols)
+ switch {
+ case sl <= math.MaxUint8:
+ dt.IndexType = arrow.PrimitiveTypes.Uint8
+ case sl > math.MaxUint8 && sl <= math.MaxUint16:
+ dt.IndexType = arrow.PrimitiveTypes.Uint16
+ case sl > math.MaxUint16 && sl <= math.MaxUint32:
+ dt.IndexType = arrow.PrimitiveTypes.Uint32
+ }
+ return arrow.Field{Name: node.name, Type: &dt, Nullable: true, Metadata: arrow.MetadataFrom(symbols)}
+ default:
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(node.ofType.(string))}
+ }
+ } else {
+ // avro "record" type, node has "fields" array
+ if node.ofType.(string) == "record" {
+ var n schemaNode
+ n.name = node.name
+ n.ofType = node.ofType
+ if len(node.fields) > 0 {
+ n.fields = append(n.fields, node.fields...)
+ }
+ f := iterateFields(n.fields)
+ return arrow.Field{Name: node.name, Type: arrow.StructOf(f...)}
+ }
+ }
+ // Avro complex types
+ case map[string]interface{}:
+ //var n schemaNode
+ //n.name = node.name
+ //n.ofType = node.ofType.(map[string]interface{})["type"]
+ switch node.logicalType {
+ case "":
+ return avroComplexToArrowField(node)
+ default:
+ return avroLogicalToArrowField(node)
+ }
+
+ // Avro union types
+ case []interface{}:
+ var unionTypes []string
+ for _, ft := range node.ofType.([]interface{}) {
+ switch ft.(type) {
+ // primitive types
+ case string:
+ if ft != "null" {
+ unionTypes = append(unionTypes, ft.(string))
+ }
+ continue
+ // complex types
+ case map[string]interface{}:
+ var n schemaNode
+ n.name = node.name
+ n.ofType = ft.(map[string]interface{})["type"]
+ if _, f := ft.(map[string]interface{})["fields"]; f {
+ for _, field := range ft.(map[string]interface{})["fields"].([]interface{}) {
+ n.fields = append(n.fields, field.(map[string]interface{}))
+ }
+ }
+ f := iterateFields(n.fields)
+ return arrow.Field{Name: node.name, Type: arrow.StructOf(f...)}
+ }
+ }
+ // Supported Avro union type is null + one other type.
+ // TODO: Complex AVRO union to Arrow Dense || Sparse Union.
+ if len(unionTypes) == 1 {
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(unionTypes[0])}
+ } else {
+ // BYTE_ARRAY is the catchall if union type is anything beyond null + one other type.
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.Binary}
+ }
+ }
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.Binary}
+}
+
+func avroLogicalToArrowField(node schemaNode) arrow.Field {
+ // Avro logical types
+ switch node.logicalType {
+ // The decimal logical type represents an arbitrary-precision signed decimal number of the form unscaled × 10-scale.
+ // A decimal logical type annotates Avro bytes or fixed types. The byte array must contain the two’s-complement
+ // representation of the unscaled integer value in big-endian byte order. The scale is fixed, and is specified
+ // using an attribute.
+ //
+ // The following attributes are supported:
+ // scale, a JSON integer representing the scale (optional). If not specified the scale is 0.
+ // precision, a JSON integer representing the (maximum) precision of decimals stored in this type (required).
+ case "decimal":
+ if node.precision <= 38 {
+ return arrow.Field{Name: node.name, Type: &arrow.Decimal128Type{Precision: node.precision, Scale: node.scale}}
+ } else {
+ return arrow.Field{Name: node.name, Type: &arrow.Decimal256Type{Precision: node.precision, Scale: node.scale}}
+ }
+
+ // The uuid logical type represents a random generated universally unique identifier (UUID).
+ // A uuid logical type annotates an Avro string. The string has to conform with RFC-4122
+ case "uuid":
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.String}
+
+ // The date logical type represents a date within the calendar, with no reference to a particular
+ // time zone or time of day.
+ // A date logical type annotates an Avro int, where the int stores the number of days from the unix epoch,
+ // 1 January 1970 (ISO calendar).
+ case "date":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Date32}
+
+ // The time-millis logical type represents a time of day, with no reference to a particular calendar,
+ // time zone or date, with a precision of one millisecond.
+ // A time-millis logical type annotates an Avro int, where the int stores the number of milliseconds
+ // after midnight, 00:00:00.000.
+ case "time-millis":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Time32ms}
+
+ // The time-micros logical type represents a time of day, with no reference to a particular calendar,
+ // time zone or date, with a precision of one microsecond.
+ // A time-micros logical type annotates an Avro long, where the long stores the number of microseconds
+ // after midnight, 00:00:00.000000.
+ case "time-micros":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Time64us}
+
+ // The timestamp-millis logical type represents an instant on the global timeline, independent of a
+ // particular time zone or calendar, with a precision of one millisecond. Please note that time zone
+ // information gets lost in this process. Upon reading a value back, we can only reconstruct the instant,
+ // but not the original representation. In practice, such timestamps are typically displayed to users in
+ // their local time zones, therefore they may be displayed differently depending on the execution environment.
+ // A timestamp-millis logical type annotates an Avro long, where the long stores the number of milliseconds
+ // from the unix epoch, 1 January 1970 00:00:00.000 UTC.
+ case "timestamp-millis":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Timestamp_ms}
+ // The timestamp-micros logical type represents an instant on the global timeline, independent of a
+ // particular time zone or calendar, with a precision of one microsecond. Please note that time zone
+ // information gets lost in this process. Upon reading a value back, we can only reconstruct the instant,
+ // but not the original representation. In practice, such timestamps are typically displayed to users
+ // in their local time zones, therefore they may be displayed differently depending on the execution environment.
+ // A timestamp-micros logical type annotates an Avro long, where the long stores the number of microseconds
+ // from the unix epoch, 1 January 1970 00:00:00.000000 UTC.
+ case "timestamp-micros":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Timestamp_us}
+ // The local-timestamp-millis logical type represents a timestamp in a local timezone, regardless of
+ // what specific time zone is considered local, with a precision of one millisecond.
+ // A local-timestamp-millis logical type annotates an Avro long, where the long stores the number of
+ // milliseconds, from 1 January 1970 00:00:00.000.
+ case "local-timestamp-millis":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Timestamp_ms}
Review Comment:
implemented as `&arrow.TimestampType{Unit: arrow.Millisecond}`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] loicalleyne commented on a diff in pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne commented on code in PR #36796:
URL: https://github.com/apache/arrow/pull/36796#discussion_r1316460761
##########
go/arrow/avro/schema.go:
##########
@@ -0,0 +1,407 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package avro reads Avro OCF files and presents the extracted data as records
+
+package avro
+
+import (
+ "fmt"
+ "math"
+ "strconv"
+
+ "github.com/apache/arrow/go/v14/arrow"
+ "github.com/apache/arrow/go/v14/internal/types"
+ avro "github.com/hamba/avro/v2"
+)
+
+type schemaNode struct {
+ name string
+ parent *schemaNode
+ schema avro.Schema
+ union bool
+ children []*schemaNode
+ arrowField arrow.Field
+ path []string
+ index, depth int32
+}
+
+var schemaCache avro.SchemaCache
+
+func newSchemaNode() *schemaNode { return &schemaNode{name: "", index: -1} }
+
+func (node *schemaNode) newChild(n string, s avro.Schema) *schemaNode {
+ child := &schemaNode{
+ name: n,
+ parent: node,
+ schema: s,
+ index: int32(len(node.children)),
+ depth: node.depth + 1}
+ node.children = append(node.children, child)
+ return child
+}
+func (node *schemaNode) Children() []*schemaNode { return node.children }
+
+func (node *schemaNode) Name() string { return node.name }
+
+// NamePath returns a slice of keys making up the path to the field
+func (node *schemaNode) NamePath() []string {
+ if len(node.path) == 0 {
+ var path []string
+ cur := node
+ for i := node.depth - 1; i >= 0; i-- {
+ path = append([]string{cur.name}, path...)
+ cur = cur.parent
+ }
+ return path
+ }
+ return node.path
+}
+
+// ArrowSchemaFromAvro returns a new Arrow schema from an Avro schema
+func ArrowSchemaFromAvro(schema avro.Schema) (s *arrow.Schema, err error) {
+ defer func() {
+ if r := recover(); r != nil {
+ s = nil
+ err = fmt.Errorf("invalid avro schema: could not convert schema")
+ }
+ }()
+ n := newSchemaNode()
+ n.schema = schema
+ c := n.newChild(n.schema.(avro.NamedSchema).Name(), n.schema)
+ arrowSchemafromAvro(c)
+ var fields []arrow.Field
+ for _, g := range c.Children() {
+ fields = append(fields, g.arrowField)
+ }
+ s = arrow.NewSchema(fields, nil)
+ return s, nil
+}
+
+func arrowSchemafromAvro(n *schemaNode) {
+ if ns, ok := n.schema.(avro.NamedSchema); ok {
+ schemaCache.Add(ns.Name(), ns)
+ }
+ switch st := n.schema.Type(); st {
+ case "record":
+ iterateFields(n)
+ case "enum":
+ symbols := make(map[string]string)
+ for index, symbol := range n.schema.(avro.PropertySchema).(*avro.EnumSchema).Symbols() {
+ k := strconv.FormatInt(int64(index), 10)
+ symbols[k] = symbol
+ }
+ var dt arrow.DictionaryType = arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Uint64, ValueType: arrow.BinaryTypes.String, Ordered: false}
+ sl := len(symbols)
+ switch {
+ case sl <= math.MaxUint8:
+ dt.IndexType = arrow.PrimitiveTypes.Uint8
+ case sl > math.MaxUint8 && sl <= math.MaxUint16:
+ dt.IndexType = arrow.PrimitiveTypes.Uint16
+ case sl > math.MaxUint16 && sl <= math.MaxUint32:
+ dt.IndexType = arrow.PrimitiveTypes.Uint32
+ }
+ n.arrowField = arrow.Field{Name: n.Name(), Type: &dt, Nullable: true, Metadata: arrow.MetadataFrom(symbols)}
+ case "array":
+ // logical items type
+ c := n.newChild(n.name, n.schema.(*avro.ArraySchema).Items())
+ if isLogicalSchemaType(n.schema.(*avro.ArraySchema).Items()) {
+ avroLogicalToArrowField(c)
+ } else {
+ arrowSchemafromAvro(c)
+ }
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.ListOf(c.arrowField.Type)}
+ case "map":
+ c := n.newChild(n.name, n.schema.(*avro.MapSchema).Values())
+ arrowSchemafromAvro(c)
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.MapOf(arrow.BinaryTypes.String, c.arrowField.Type), Metadata: c.arrowField.Metadata, Nullable: true}
+ case "union":
+ if n.schema.(*avro.UnionSchema).Nullable() {
+ if len(n.schema.(*avro.UnionSchema).Types()) > 1 {
+ n.schema = n.schema.(*avro.UnionSchema).Types()[1]
+ n.union = true
+ arrowSchemafromAvro(n)
+ }
+ }
+ // Avro "fixed" field type = Arrow FixedSize Primitive BinaryType
+ case "fixed":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.Name(), Type: &arrow.FixedSizeBinaryType{ByteWidth: n.schema.(*avro.FixedSchema).Size()}, Nullable: true}
+ }
+ case "string":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "bytes":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "int":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "long":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "float":
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ case "double":
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ case "boolean":
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ case "<ref>":
+ n.schema = schemaCache.Get(n.schema.(*avro.RefSchema).Schema().Name())
+ arrowSchemafromAvro(n)
+ case "null":
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.BinaryTypes.Binary, Nullable: true}
Review Comment:
yes probably, changing it to arrow.Null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-36760: [Go] Adding avro ocf reader - schema converter [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #36796:
URL: https://github.com/apache/arrow/pull/36796#issuecomment-1751362577
let's close this one and just use #37115 to track the whole thing. Please rebase that one and have a look at my comments there and close this one. It's likely easier to just keep it all in there than go back and forth between these and have you having to maintain both.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] loicalleyne commented on a diff in pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne commented on code in PR #36796:
URL: https://github.com/apache/arrow/pull/36796#discussion_r1281129698
##########
go/arrow/avro/common.go:
##########
@@ -0,0 +1,162 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package avro
+
+import (
+ "errors"
+ "fmt"
+
+ "github.com/apache/arrow/go/v13/arrow"
+ "github.com/apache/arrow/go/v13/arrow/memory"
+)
+
+var (
+ ErrMismatchFields = errors.New("arrow/avro: number of records mismatch")
+)
+
+// Option configures an Avro reader/writer.
+type Option func(config)
+type config interface{}
+
+// WithTopLevel specifies whether to include an Avro schema's top level record
+// or only its fields.
+func WithTopLevel(b bool) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.topLevel = b
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithAllocator specifies the Arrow memory allocator used while building records.
+func WithAllocator(mem memory.Allocator) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.mem = mem
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithChunk specifies the chunk size used while reading Avro OCF files.
+//
+// If n is zero or 1, no chunking will take place and the reader will create
+// one record per row.
+// If n is greater than 1, chunks of n rows will be read.
+// If n is negative, the reader will load the whole Avro OCF file into memory and
+// create one big record with all the rows.
+func WithChunk(n int) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.chunk = n
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// DefaultNullValues is the set of values considered as NULL values by default
+// when Reader is configured to handle NULL values.
+var DefaultNullValues = []string{"", "NULL", "null"}
+
+// WithNullReader sets options for a CSV Reader pertaining to NULL value
+// handling. If stringsCanBeNull is true, then a string that matches one of the
+// nullValues set will be interpreted as NULL. Numeric columns will be checked
+// for nulls in all cases. If no nullValues arguments are passed in, the
+// defaults set in NewReader() will be kept.
+//
+// When no NULL values is given, the default set is taken from DefaultNullValues.
+func WithNullReader(stringsCanBeNull bool, nullValues ...string) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.stringsCanBeNull = stringsCanBeNull
+
+ if len(nullValues) == 0 {
+ nullValues = DefaultNullValues
+ }
+ cfg.nulls = make([]string, len(nullValues))
+ copy(cfg.nulls, nullValues)
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithColumnTypes allows specifying optional per-column types (disabling
+// type inference on those columns).
+//
+// Will panic if used in conjunction with an explicit schema.
+func WithColumnTypes(types map[string]arrow.DataType) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ if cfg.schema != nil {
+ panic(fmt.Errorf("%w: cannot use WithColumnTypes with explicit schema", arrow.ErrInvalid))
+ }
+ cfg.columnTypes = types
+ default:
+ panic(fmt.Errorf("%w: WithColumnTypes only allowed for csv reader", arrow.ErrInvalid))
+ }
+ }
+}
Review Comment:
removed in the latest commit
##########
go/arrow/avro/common.go:
##########
@@ -0,0 +1,162 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package avro
+
+import (
+ "errors"
+ "fmt"
+
+ "github.com/apache/arrow/go/v13/arrow"
+ "github.com/apache/arrow/go/v13/arrow/memory"
+)
+
+var (
+ ErrMismatchFields = errors.New("arrow/avro: number of records mismatch")
+)
+
+// Option configures an Avro reader/writer.
+type Option func(config)
+type config interface{}
+
+// WithTopLevel specifies whether to include an Avro schema's top level record
+// or only its fields.
+func WithTopLevel(b bool) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.topLevel = b
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithAllocator specifies the Arrow memory allocator used while building records.
+func WithAllocator(mem memory.Allocator) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.mem = mem
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithChunk specifies the chunk size used while reading Avro OCF files.
+//
+// If n is zero or 1, no chunking will take place and the reader will create
+// one record per row.
+// If n is greater than 1, chunks of n rows will be read.
+// If n is negative, the reader will load the whole Avro OCF file into memory and
+// create one big record with all the rows.
+func WithChunk(n int) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.chunk = n
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// DefaultNullValues is the set of values considered as NULL values by default
+// when Reader is configured to handle NULL values.
+var DefaultNullValues = []string{"", "NULL", "null"}
+
+// WithNullReader sets options for a CSV Reader pertaining to NULL value
+// handling. If stringsCanBeNull is true, then a string that matches one of the
+// nullValues set will be interpreted as NULL. Numeric columns will be checked
+// for nulls in all cases. If no nullValues arguments are passed in, the
+// defaults set in NewReader() will be kept.
+//
+// When no NULL values is given, the default set is taken from DefaultNullValues.
+func WithNullReader(stringsCanBeNull bool, nullValues ...string) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.stringsCanBeNull = stringsCanBeNull
+
+ if len(nullValues) == 0 {
+ nullValues = DefaultNullValues
+ }
+ cfg.nulls = make([]string, len(nullValues))
+ copy(cfg.nulls, nullValues)
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithColumnTypes allows specifying optional per-column types (disabling
+// type inference on those columns).
+//
+// Will panic if used in conjunction with an explicit schema.
+func WithColumnTypes(types map[string]arrow.DataType) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ if cfg.schema != nil {
+ panic(fmt.Errorf("%w: cannot use WithColumnTypes with explicit schema", arrow.ErrInvalid))
+ }
+ cfg.columnTypes = types
+ default:
+ panic(fmt.Errorf("%w: WithColumnTypes only allowed for csv reader", arrow.ErrInvalid))
+ }
+ }
+}
+
+// WithIncludeColumns indicates the names of the columns from the AVRO file
+// that should actually be read and converted (in the slice's order).
+// If set and non-empty, columns not in this slice will be ignored.
+//
+// Will panic if used in conjunction with an explicit schema.
+func WithIncludeColumns(cols []string) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ if cfg.schema != nil {
+ panic(fmt.Errorf("%w: cannot use WithIncludeColumns with explicit schema", arrow.ErrInvalid))
+ }
+ cfg.columnFilter = cols
+ default:
+ panic(fmt.Errorf("%w: WithIncludeColumns only allowed on csv Reader", arrow.ErrInvalid))
+ }
+ }
+}
Review Comment:
removed in the latest commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] loicalleyne commented on a diff in pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne commented on code in PR #36796:
URL: https://github.com/apache/arrow/pull/36796#discussion_r1281158088
##########
go/arrow/avro/schema.go:
##########
@@ -0,0 +1,409 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package avro reads Avro OCF files and presents the extracted data as records
+package avro
+
+import (
+ "encoding/json"
+ "fmt"
+ "math"
+ "strconv"
+
+ "github.com/apache/arrow/go/v13/arrow"
+)
+
+type schemaNode struct {
+ name string
+ ofType interface{}
+ logicalType string
+ precision int32
+ scale int32
+ size int
+ symbols []string
+ fields []interface{}
+}
+
+// ArrowSchemaFromAvro returns a new Arrow schema from an Avro schema JSON.
+// If the top level is of record type, set includeTopLevel to either make
+// its fields top level fields in the resulting schema or nested in a single field.
+func ArrowSchemaFromAvro(avroSchema []byte) (*arrow.Schema, error) {
+ var m map[string]interface{}
+ var node schemaNode
+ json.Unmarshal(avroSchema, &m)
+ if m["type"].(string) == "record" {
+ if _, ok := m["fields"]; ok {
+ for _, field := range m["fields"].([]interface{}) {
+ node.fields = append(node.fields, field.(map[string]interface{}))
+ }
+ if len(node.fields) == 0 {
+ return nil, fmt.Errorf("invalid avro schema: no top level record fields found")
+ }
+ }
+ }
+ fields := iterateFields(node.fields)
+ return arrow.NewSchema(fields, nil), nil
+}
+
+func iterateFields(f []interface{}) []arrow.Field {
+ var s []arrow.Field
+ for _, field := range f {
+ var n schemaNode
+ n.name = field.(map[string]interface{})["name"].(string)
+ n.ofType = field.(map[string]interface{})["type"]
+ switch n.ofType.(type) {
+ // Getting field type from within field object
+ case string:
+ switch n.ofType.(string) {
+ case "enum":
+ for _, symbol := range field.(map[string]interface{})["symbols"].([]interface{}) {
+ n.symbols = append(n.symbols, symbol.(string))
+ }
+ default:
+ if lt, ok := field.(map[string]interface{})["logicalType"]; ok {
+ n.logicalType = lt.(string)
+ }
+ if lt, ok := field.(map[string]interface{})["size"]; ok {
+ n.size = int(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["precision"]; ok {
+ n.precision = int32(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["scale"]; ok {
+ n.scale = int32(lt.(float64))
+ }
+ }
+ // Field type is an object
+ case map[string]interface{}:
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["logicalType"]; ok {
+ n.logicalType = lt.(string)
+ }
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["size"]; ok {
+ n.size = int(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["precision"]; ok {
+ n.precision = int32(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["scale"]; ok {
+ n.scale = int32(lt.(float64))
+ }
+ case []interface{}:
+
+ default:
+ if lt, ok := field.(map[string]interface{})["logicalType"]; ok {
+ n.logicalType = lt.(string)
+ }
+ if lt, ok := field.(map[string]interface{})["size"]; ok {
+ n.size = int(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["precision"]; ok {
+ n.precision = int32(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["scale"]; ok {
+ n.scale = int32(lt.(float64))
+ }
+ }
+ // Field is of type "record"
+ if nf, f := field.(map[string]interface{})["fields"]; f {
+ switch nf.(type) {
+ // primitive & complex types
+ case map[string]interface{}:
+ for _, v := range nf.(map[string]interface{})["fields"].([]interface{}) {
+ n.fields = append(n.fields, v.(map[string]interface{}))
+ }
+ // type unions
+ default:
+ for _, v := range nf.([]interface{}) {
+ n.fields = append(n.fields, v.(map[string]interface{}))
+ }
+ }
+ }
+ s = append(s, traverseNodes(n))
+ }
+ return s
+}
+
+func traverseNodes(node schemaNode) arrow.Field {
+ switch node.ofType.(type) {
+ case string:
+ // Avro primitive type
+ if len(node.fields) == 0 {
+ switch node.ofType.(string) {
+ case "boolean", "int", "long", "float", "double", "bytes", "string":
+ if node.logicalType != "" {
+ return avroLogicalToArrowField(node)
+ }
+ // Avro primitive type
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(node.ofType.(string))}
+ case "fixed":
+ // Duration type is not supported in github.com/linkedin/goavro
+ // Implementing as Binary for now.
+ switch node.logicalType {
+ case "decimal":
+ return avroLogicalToArrowField(node)
+ case "duration":
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.Binary}
+ //return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.MonthDayNanoInterval}
+ }
+ return arrow.Field{Name: node.name, Type: &arrow.FixedSizeBinaryType{ByteWidth: node.size}}
+ case "enum":
+ symbols := make(map[string]string)
+ for index, symbol := range node.symbols {
+ k := strconv.FormatInt(int64(index), 10)
+ symbols[k] = symbol
+ }
+ var dt arrow.DictionaryType = arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Uint64, ValueType: arrow.BinaryTypes.String, Ordered: false}
+ sl := len(symbols)
+ switch {
+ case sl <= math.MaxUint8:
+ dt.IndexType = arrow.PrimitiveTypes.Uint8
+ case sl > math.MaxUint8 && sl <= math.MaxUint16:
+ dt.IndexType = arrow.PrimitiveTypes.Uint16
+ case sl > math.MaxUint16 && sl <= math.MaxUint32:
+ dt.IndexType = arrow.PrimitiveTypes.Uint32
+ }
+ return arrow.Field{Name: node.name, Type: &dt, Nullable: true, Metadata: arrow.MetadataFrom(symbols)}
+ default:
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(node.ofType.(string))}
+ }
+ } else {
+ // avro "record" type, node has "fields" array
+ if node.ofType.(string) == "record" {
+ var n schemaNode
+ n.name = node.name
+ n.ofType = node.ofType
+ if len(node.fields) > 0 {
+ n.fields = append(n.fields, node.fields...)
+ }
+ f := iterateFields(n.fields)
+ return arrow.Field{Name: node.name, Type: arrow.StructOf(f...)}
+ }
+ }
+ // Avro complex types
+ case map[string]interface{}:
+ //var n schemaNode
+ //n.name = node.name
+ //n.ofType = node.ofType.(map[string]interface{})["type"]
+ switch node.logicalType {
+ case "":
+ return avroComplexToArrowField(node)
+ default:
+ return avroLogicalToArrowField(node)
+ }
+
+ // Avro union types
+ case []interface{}:
+ var unionTypes []string
+ for _, ft := range node.ofType.([]interface{}) {
+ switch ft.(type) {
+ // primitive types
+ case string:
+ if ft != "null" {
+ unionTypes = append(unionTypes, ft.(string))
+ }
+ continue
+ // complex types
+ case map[string]interface{}:
+ var n schemaNode
+ n.name = node.name
+ n.ofType = ft.(map[string]interface{})["type"]
+ if _, f := ft.(map[string]interface{})["fields"]; f {
+ for _, field := range ft.(map[string]interface{})["fields"].([]interface{}) {
+ n.fields = append(n.fields, field.(map[string]interface{}))
+ }
+ }
+ f := iterateFields(n.fields)
+ return arrow.Field{Name: node.name, Type: arrow.StructOf(f...)}
+ }
+ }
+ // Supported Avro union type is null + one other type.
+ // TODO: Complex AVRO union to Arrow Dense || Sparse Union.
+ if len(unionTypes) == 1 {
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(unionTypes[0])}
+ } else {
+ // BYTE_ARRAY is the catchall if union type is anything beyond null + one other type.
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.Binary}
+ }
+ }
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.Binary}
+}
+
+func avroLogicalToArrowField(node schemaNode) arrow.Field {
+ // Avro logical types
+ switch node.logicalType {
+ // The decimal logical type represents an arbitrary-precision signed decimal number of the form unscaled × 10-scale.
+ // A decimal logical type annotates Avro bytes or fixed types. The byte array must contain the two’s-complement
+ // representation of the unscaled integer value in big-endian byte order. The scale is fixed, and is specified
+ // using an attribute.
+ //
+ // The following attributes are supported:
+ // scale, a JSON integer representing the scale (optional). If not specified the scale is 0.
+ // precision, a JSON integer representing the (maximum) precision of decimals stored in this type (required).
+ case "decimal":
+ if node.precision <= 38 {
+ return arrow.Field{Name: node.name, Type: &arrow.Decimal128Type{Precision: node.precision, Scale: node.scale}}
+ } else {
+ return arrow.Field{Name: node.name, Type: &arrow.Decimal256Type{Precision: node.precision, Scale: node.scale}}
+ }
Review Comment:
making the change - will be in next commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] loicalleyne commented on a diff in pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne commented on code in PR #36796:
URL: https://github.com/apache/arrow/pull/36796#discussion_r1281129390
##########
go/arrow/avro/common.go:
##########
@@ -0,0 +1,162 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package avro
+
+import (
+ "errors"
+ "fmt"
+
+ "github.com/apache/arrow/go/v13/arrow"
+ "github.com/apache/arrow/go/v13/arrow/memory"
+)
+
+var (
+ ErrMismatchFields = errors.New("arrow/avro: number of records mismatch")
+)
+
+// Option configures an Avro reader/writer.
+type Option func(config)
+type config interface{}
+
+// WithTopLevel specifies whether to include an Avro schema's top level record
+// or only its fields.
+func WithTopLevel(b bool) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.topLevel = b
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithAllocator specifies the Arrow memory allocator used while building records.
+func WithAllocator(mem memory.Allocator) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.mem = mem
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithChunk specifies the chunk size used while reading Avro OCF files.
+//
+// If n is zero or 1, no chunking will take place and the reader will create
+// one record per row.
+// If n is greater than 1, chunks of n rows will be read.
+// If n is negative, the reader will load the whole Avro OCF file into memory and
+// create one big record with all the rows.
+func WithChunk(n int) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.chunk = n
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// DefaultNullValues is the set of values considered as NULL values by default
+// when Reader is configured to handle NULL values.
+var DefaultNullValues = []string{"", "NULL", "null"}
Review Comment:
this is removed in the latest commit
##########
go/arrow/avro/common.go:
##########
@@ -0,0 +1,162 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package avro
+
+import (
+ "errors"
+ "fmt"
+
+ "github.com/apache/arrow/go/v13/arrow"
+ "github.com/apache/arrow/go/v13/arrow/memory"
+)
+
+var (
+ ErrMismatchFields = errors.New("arrow/avro: number of records mismatch")
+)
+
+// Option configures an Avro reader/writer.
+type Option func(config)
+type config interface{}
+
+// WithTopLevel specifies whether to include an Avro schema's top level record
+// or only its fields.
+func WithTopLevel(b bool) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.topLevel = b
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithAllocator specifies the Arrow memory allocator used while building records.
+func WithAllocator(mem memory.Allocator) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.mem = mem
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithChunk specifies the chunk size used while reading Avro OCF files.
+//
+// If n is zero or 1, no chunking will take place and the reader will create
+// one record per row.
+// If n is greater than 1, chunks of n rows will be read.
+// If n is negative, the reader will load the whole Avro OCF file into memory and
+// create one big record with all the rows.
+func WithChunk(n int) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.chunk = n
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// DefaultNullValues is the set of values considered as NULL values by default
+// when Reader is configured to handle NULL values.
+var DefaultNullValues = []string{"", "NULL", "null"}
+
+// WithNullReader sets options for a CSV Reader pertaining to NULL value
+// handling. If stringsCanBeNull is true, then a string that matches one of the
+// nullValues set will be interpreted as NULL. Numeric columns will be checked
+// for nulls in all cases. If no nullValues arguments are passed in, the
+// defaults set in NewReader() will be kept.
+//
+// When no NULL values is given, the default set is taken from DefaultNullValues.
+func WithNullReader(stringsCanBeNull bool, nullValues ...string) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.stringsCanBeNull = stringsCanBeNull
+
+ if len(nullValues) == 0 {
+ nullValues = DefaultNullValues
+ }
+ cfg.nulls = make([]string, len(nullValues))
+ copy(cfg.nulls, nullValues)
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
Review Comment:
removed in the latest commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on a diff in pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #36796:
URL: https://github.com/apache/arrow/pull/36796#discussion_r1293969142
##########
go/arrow/avro/reader.go:
##########
@@ -0,0 +1,490 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package avro
+
+import (
+ "errors"
+ "fmt"
+ "io"
+ "sync"
+ "sync/atomic"
+
+ "github.com/apache/arrow/go/v13/arrow"
+ "github.com/apache/arrow/go/v13/arrow/array"
+ "github.com/apache/arrow/go/v13/arrow/internal/debug"
+ "github.com/apache/arrow/go/v13/arrow/memory"
+ "github.com/linkedin/goavro/v2"
+)
+
+// Reader wraps goavro/OCFReader and creates array.Records from a schema.
+type OCFReader struct {
+ r *goavro.OCFReader
+ schema *arrow.Schema
+
+ refs int64
+ bld *array.RecordBuilder
+ cur arrow.Record
+ err error
+
+ chunk int
+ done bool
+ next func() bool
+
+ mem memory.Allocator
+
+ topLevel bool
+ once sync.Once
+
+ fieldConverter []func(val string)
+ columnFilter []string
+ columnTypes map[string]arrow.DataType
+
+ stringsCanBeNull bool
+ nulls []string
+}
+
+// NewReader returns a reader that reads from an Avro OCF file and creates
+// arrow.Records from the converted schema.
+func NewOCFReader(r io.Reader, opts ...Option) *OCFReader {
+ ocfr, err := goavro.NewOCFReader(r)
+ if err != nil {
+ panic(fmt.Errorf("%w: could not create avro ocfreader", arrow.ErrInvalid))
+ }
+
+ rr := &OCFReader{
+ r: ocfr,
+ //schema: schema,
Review Comment:
extraneous comment?
##########
go/arrow/avro/reader.go:
##########
@@ -0,0 +1,490 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package avro
+
+import (
+ "errors"
+ "fmt"
+ "io"
+ "sync"
+ "sync/atomic"
+
+ "github.com/apache/arrow/go/v13/arrow"
+ "github.com/apache/arrow/go/v13/arrow/array"
+ "github.com/apache/arrow/go/v13/arrow/internal/debug"
+ "github.com/apache/arrow/go/v13/arrow/memory"
+ "github.com/linkedin/goavro/v2"
+)
+
+// Reader wraps goavro/OCFReader and creates array.Records from a schema.
+type OCFReader struct {
+ r *goavro.OCFReader
+ schema *arrow.Schema
+
+ refs int64
+ bld *array.RecordBuilder
+ cur arrow.Record
+ err error
+
+ chunk int
+ done bool
+ next func() bool
+
+ mem memory.Allocator
+
+ topLevel bool
+ once sync.Once
+
+ fieldConverter []func(val string)
+ columnFilter []string
+ columnTypes map[string]arrow.DataType
+
+ stringsCanBeNull bool
+ nulls []string
+}
+
+// NewReader returns a reader that reads from an Avro OCF file and creates
+// arrow.Records from the converted schema.
+func NewOCFReader(r io.Reader, opts ...Option) *OCFReader {
+ ocfr, err := goavro.NewOCFReader(r)
+ if err != nil {
+ panic(fmt.Errorf("%w: could not create avro ocfreader", arrow.ErrInvalid))
+ }
+
+ rr := &OCFReader{
+ r: ocfr,
+ //schema: schema,
+ refs: 1,
+ chunk: 1,
+ stringsCanBeNull: false,
+ }
+ //rr.r.ReuseRecord = true
Review Comment:
same
##########
go/arrow/avro/reader.go:
##########
@@ -0,0 +1,490 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package avro
+
+import (
+ "errors"
+ "fmt"
+ "io"
+ "sync"
+ "sync/atomic"
+
+ "github.com/apache/arrow/go/v13/arrow"
+ "github.com/apache/arrow/go/v13/arrow/array"
+ "github.com/apache/arrow/go/v13/arrow/internal/debug"
+ "github.com/apache/arrow/go/v13/arrow/memory"
+ "github.com/linkedin/goavro/v2"
+)
+
+// Reader wraps goavro/OCFReader and creates array.Records from a schema.
+type OCFReader struct {
+ r *goavro.OCFReader
+ schema *arrow.Schema
+
+ refs int64
+ bld *array.RecordBuilder
+ cur arrow.Record
+ err error
+
+ chunk int
+ done bool
+ next func() bool
+
+ mem memory.Allocator
+
+ topLevel bool
+ once sync.Once
+
+ fieldConverter []func(val string)
+ columnFilter []string
+ columnTypes map[string]arrow.DataType
+
+ stringsCanBeNull bool
+ nulls []string
+}
+
+// NewReader returns a reader that reads from an Avro OCF file and creates
+// arrow.Records from the converted schema.
+func NewOCFReader(r io.Reader, opts ...Option) *OCFReader {
+ ocfr, err := goavro.NewOCFReader(r)
+ if err != nil {
+ panic(fmt.Errorf("%w: could not create avro ocfreader", arrow.ErrInvalid))
+ }
+
+ rr := &OCFReader{
+ r: ocfr,
+ //schema: schema,
+ refs: 1,
+ chunk: 1,
+ stringsCanBeNull: false,
Review Comment:
unneeded since Avro can represent null values
##########
go/arrow/avro/schema.go:
##########
@@ -0,0 +1,512 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package avro reads Avro OCF files and presents the extracted data as records
+
+package avro
+
+import (
+ "encoding/json"
+ "fmt"
+ "math"
+ "strconv"
+
+ "github.com/apache/arrow/go/v13/arrow"
+ "github.com/apache/arrow/go/v13/internal/types"
+)
+
+type schemaNode struct {
+ name string
+ ofType interface{}
+ logicalType string
+ precision int32
+ scale int32
+ size int
+ symbols []string
+ fields []interface{}
+}
+
+// ArrowSchemaFromAvro returns a new Arrow schema from an Avro schema JSON.
+func ArrowSchemaFromAvro(avroSchema []byte) (*arrow.Schema, error) {
+ var m map[string]interface{}
+ var node schemaNode
+ json.Unmarshal(avroSchema, &m)
+ if name, ok := m["name"]; ok {
+ node.name = name.(string)
+ } else {
+ node.name = "NameStub"
+ }
Review Comment:
were we gonna try using the hambra library for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] loicalleyne commented on a diff in pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne commented on code in PR #36796:
URL: https://github.com/apache/arrow/pull/36796#discussion_r1293987728
##########
go/arrow/avro/schema.go:
##########
@@ -0,0 +1,512 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package avro reads Avro OCF files and presents the extracted data as records
+
+package avro
+
+import (
+ "encoding/json"
+ "fmt"
+ "math"
+ "strconv"
+
+ "github.com/apache/arrow/go/v13/arrow"
+ "github.com/apache/arrow/go/v13/internal/types"
+)
+
+type schemaNode struct {
+ name string
+ ofType interface{}
+ logicalType string
+ precision int32
+ scale int32
+ size int
+ symbols []string
+ fields []interface{}
+}
+
+// ArrowSchemaFromAvro returns a new Arrow schema from an Avro schema JSON.
+func ArrowSchemaFromAvro(avroSchema []byte) (*arrow.Schema, error) {
+ var m map[string]interface{}
+ var node schemaNode
+ json.Unmarshal(avroSchema, &m)
+ if name, ok := m["name"]; ok {
+ node.name = name.(string)
+ } else {
+ node.name = "NameStub"
+ }
Review Comment:
The input schema is a json in canonical format which is identical whether it's coming from hamba/avro or linkedin/goavro.
With the changes suggested previously the current code produces a valid arrow schema and only needs to be run once per avro OCF so I focused my efforts on the reader which I did move over to use /hamba/avro.
Moving the schema converter to hamba might be better for a future version if we want to add extended support for Avro unions beyond the currently supported "null + 1 on other type".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-36760: [Go] Adding avro ocf reader - schema converter [arrow]
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne commented on PR #36796:
URL: https://github.com/apache/arrow/pull/36796#issuecomment-1751331867
@zeroshade it's up to you, I submitted the schema converter and the reader in separate PRs to keep the scope small.
To the best of my recollection #37115 contains the exactly the same code. Let me know if you want me to rebase one or both of them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] loicalleyne commented on a diff in pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne commented on code in PR #36796:
URL: https://github.com/apache/arrow/pull/36796#discussion_r1316472927
##########
go/arrow/avro/schema.go:
##########
@@ -0,0 +1,407 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package avro reads Avro OCF files and presents the extracted data as records
+
+package avro
+
+import (
+ "fmt"
+ "math"
+ "strconv"
+
+ "github.com/apache/arrow/go/v14/arrow"
+ "github.com/apache/arrow/go/v14/internal/types"
+ avro "github.com/hamba/avro/v2"
+)
+
+type schemaNode struct {
+ name string
+ parent *schemaNode
+ schema avro.Schema
+ union bool
+ children []*schemaNode
+ arrowField arrow.Field
+ path []string
+ index, depth int32
+}
+
+var schemaCache avro.SchemaCache
+
+func newSchemaNode() *schemaNode { return &schemaNode{name: "", index: -1} }
+
+func (node *schemaNode) newChild(n string, s avro.Schema) *schemaNode {
+ child := &schemaNode{
+ name: n,
+ parent: node,
+ schema: s,
+ index: int32(len(node.children)),
+ depth: node.depth + 1}
+ node.children = append(node.children, child)
+ return child
+}
+func (node *schemaNode) Children() []*schemaNode { return node.children }
+
+func (node *schemaNode) Name() string { return node.name }
+
+// NamePath returns a slice of keys making up the path to the field
+func (node *schemaNode) NamePath() []string {
+ if len(node.path) == 0 {
+ var path []string
+ cur := node
+ for i := node.depth - 1; i >= 0; i-- {
+ path = append([]string{cur.name}, path...)
+ cur = cur.parent
+ }
+ return path
+ }
+ return node.path
+}
+
+// ArrowSchemaFromAvro returns a new Arrow schema from an Avro schema
+func ArrowSchemaFromAvro(schema avro.Schema) (s *arrow.Schema, err error) {
+ defer func() {
+ if r := recover(); r != nil {
+ s = nil
+ err = fmt.Errorf("invalid avro schema: could not convert schema")
+ }
+ }()
+ n := newSchemaNode()
+ n.schema = schema
+ c := n.newChild(n.schema.(avro.NamedSchema).Name(), n.schema)
+ arrowSchemafromAvro(c)
+ var fields []arrow.Field
+ for _, g := range c.Children() {
+ fields = append(fields, g.arrowField)
+ }
+ s = arrow.NewSchema(fields, nil)
+ return s, nil
+}
+
+func arrowSchemafromAvro(n *schemaNode) {
+ if ns, ok := n.schema.(avro.NamedSchema); ok {
+ schemaCache.Add(ns.Name(), ns)
+ }
+ switch st := n.schema.Type(); st {
+ case "record":
+ iterateFields(n)
+ case "enum":
+ symbols := make(map[string]string)
+ for index, symbol := range n.schema.(avro.PropertySchema).(*avro.EnumSchema).Symbols() {
+ k := strconv.FormatInt(int64(index), 10)
+ symbols[k] = symbol
+ }
+ var dt arrow.DictionaryType = arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Uint64, ValueType: arrow.BinaryTypes.String, Ordered: false}
+ sl := len(symbols)
+ switch {
+ case sl <= math.MaxUint8:
+ dt.IndexType = arrow.PrimitiveTypes.Uint8
+ case sl > math.MaxUint8 && sl <= math.MaxUint16:
+ dt.IndexType = arrow.PrimitiveTypes.Uint16
+ case sl > math.MaxUint16 && sl <= math.MaxUint32:
+ dt.IndexType = arrow.PrimitiveTypes.Uint32
+ }
+ n.arrowField = arrow.Field{Name: n.Name(), Type: &dt, Nullable: true, Metadata: arrow.MetadataFrom(symbols)}
+ case "array":
+ // logical items type
+ c := n.newChild(n.name, n.schema.(*avro.ArraySchema).Items())
+ if isLogicalSchemaType(n.schema.(*avro.ArraySchema).Items()) {
+ avroLogicalToArrowField(c)
+ } else {
+ arrowSchemafromAvro(c)
+ }
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.ListOf(c.arrowField.Type)}
+ case "map":
+ c := n.newChild(n.name, n.schema.(*avro.MapSchema).Values())
+ arrowSchemafromAvro(c)
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.MapOf(arrow.BinaryTypes.String, c.arrowField.Type), Metadata: c.arrowField.Metadata, Nullable: true}
+ case "union":
+ if n.schema.(*avro.UnionSchema).Nullable() {
+ if len(n.schema.(*avro.UnionSchema).Types()) > 1 {
+ n.schema = n.schema.(*avro.UnionSchema).Types()[1]
+ n.union = true
+ arrowSchemafromAvro(n)
+ }
+ }
+ // Avro "fixed" field type = Arrow FixedSize Primitive BinaryType
+ case "fixed":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.Name(), Type: &arrow.FixedSizeBinaryType{ByteWidth: n.schema.(*avro.FixedSchema).Size()}, Nullable: true}
+ }
+ case "string":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "bytes":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "int":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "long":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "float":
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ case "double":
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ case "boolean":
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ case "<ref>":
+ n.schema = schemaCache.Get(n.schema.(*avro.RefSchema).Schema().Name())
+ arrowSchemafromAvro(n)
+ case "null":
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.BinaryTypes.Binary, Nullable: true}
+ }
+}
+
+// iterate record Fields()
+func iterateFields(n *schemaNode) {
+ for _, f := range n.schema.(avro.PropertySchema).(*avro.RecordSchema).Fields() {
+ switch ft := f.Type().(type) {
+ // Avro "array" field type
+ case *avro.ArraySchema:
+ schemaCache.Add(f.Name(), ft.Items())
+ // logical items type
+ c := n.newChild(f.Name(), ft.Items())
+ if isLogicalSchemaType(ft.Items()) {
+ avroLogicalToArrowField(c)
+ } else {
+ arrowSchemafromAvro(c)
+ }
+ c.arrowField = arrow.Field{Name: c.name, Type: arrow.ListOf(c.arrowField.Type), Metadata: c.arrowField.Metadata, Nullable: true}
+ // Avro "enum" field type = Arrow dictionary type
+ case *avro.EnumSchema:
+ schemaCache.Add(f.Name(), f.Type())
+ c := n.newChild(f.Name(), f.Type())
+ symbols := make(map[string]string)
+ for index, symbol := range ft.Symbols() {
+ k := strconv.FormatInt(int64(index), 10)
+ symbols[k] = symbol
+ }
+ var dt arrow.DictionaryType = arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Uint64, ValueType: arrow.BinaryTypes.String, Ordered: false}
+ sl := len(symbols)
+ switch {
+ case sl <= math.MaxUint8:
+ dt.IndexType = arrow.PrimitiveTypes.Uint8
+ case sl > math.MaxUint8 && sl <= math.MaxUint16:
+ dt.IndexType = arrow.PrimitiveTypes.Uint16
+ case sl > math.MaxUint16 && sl <= math.MaxUint32:
+ dt.IndexType = arrow.PrimitiveTypes.Uint32
+ }
+ c.arrowField = arrow.Field{Name: f.Name(), Type: &dt, Nullable: true, Metadata: arrow.MetadataFrom(symbols)}
+ // Avro "fixed" field type = Arrow FixedSize Primitive BinaryType
+ case *avro.FixedSchema:
+ schemaCache.Add(f.Name(), f.Type())
+ if isLogicalSchemaType(f.Type()) {
+ c := n.newChild(f.Name(), f.Type())
+ avroLogicalToArrowField(c)
+ } else {
+ c := n.newChild(f.Name(), f.Type())
+ arrowSchemafromAvro(c)
+ }
+ case *avro.RecordSchema:
+ schemaCache.Add(f.Name(), f.Type())
+ c := n.newChild(f.Name(), f.Type())
+ iterateFields(c)
+ // Avro "map" field type - KVP with value of one type - keys are strings
+ case *avro.MapSchema:
+ schemaCache.Add(f.Name(), ft.Values())
+ c := n.newChild(f.Name(), ft.Values())
+ arrowSchemafromAvro(c)
+ if ft.Values().Type() == "union" {
+ c.arrowField = arrow.Field{Name: f.Name(), Type: arrow.MapOf(arrow.BinaryTypes.String, arrow.BinaryTypes.String), Metadata: c.arrowField.Metadata, Nullable: true}
+ } else {
+ c.arrowField = arrow.Field{Name: f.Name(), Type: arrow.MapOf(arrow.BinaryTypes.String, c.arrowField.Type), Metadata: c.arrowField.Metadata, Nullable: true}
+ }
+ case *avro.UnionSchema:
+ if ft.Nullable() {
+ if len(ft.Types()) > 1 {
+ schemaCache.Add(f.Name(), ft.Types()[1])
+ c := n.newChild(f.Name(), ft.Types()[1])
+ c.union = true
+ arrowSchemafromAvro(c)
+ }
+ }
+ default:
+ schemaCache.Add(f.Name(), f.Type())
+ if isLogicalSchemaType(f.Type()) {
+ c := n.newChild(f.Name(), f.Type())
+ avroLogicalToArrowField(c)
+ } else {
+ c := n.newChild(f.Name(), f.Type())
+ arrowSchemafromAvro(c)
+ }
+
+ }
+ }
+ var fields []arrow.Field
+ for _, child := range n.Children() {
+ fields = append(fields, child.arrowField)
+ }
+
+ namedSchema, ok := isNamedSchema(n.schema)
+ if !ok || namedSchema == n.name+"_data" || !n.union {
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.StructOf(fields...), Nullable: true}
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.StructOf(fields...), Nullable: true, Metadata: arrow.MetadataFrom(map[string]string{"typeName": namedSchema})}
+ }
+
+}
+
+func isLogicalSchemaType(s avro.Schema) bool {
+ lts, ok := s.(avro.LogicalTypeSchema)
+ if !ok {
+ return false
+ }
+ if lt := lts.Logical(); lt != nil {
+ return true
+ }
+ return false
+}
+
+func isNamedSchema(s avro.Schema) (string, bool) {
+ if ns, ok := s.(avro.NamedSchema); ok {
+ return ns.FullName(), ok
+ }
+ return "", false
+}
+
+// Avro primitive type.
+//
+// NOTE: Arrow Binary type is used as a catchall to avoid potential data loss.
+func avroPrimitiveToArrowType(avroFieldType string) arrow.DataType {
+ switch avroFieldType {
+ // int: 32-bit signed integer
+ case "int":
+ return arrow.PrimitiveTypes.Int32
+ // long: 64-bit signed integer
+ case "long":
+ return arrow.PrimitiveTypes.Int64
+ // float: single precision (32-bit) IEEE 754 floating-point number
+ case "float":
+ return arrow.PrimitiveTypes.Float32
+ // double: double precision (64-bit) IEEE 754 floating-point number
+ case "double":
+ return arrow.PrimitiveTypes.Float64
+ // bytes: sequence of 8-bit unsigned bytes
+ case "bytes":
+ return arrow.BinaryTypes.Binary
+ // boolean: a binary value
+ case "boolean":
+ return arrow.FixedWidthTypes.Boolean
+ // string: unicode character sequence
+ case "string":
+ return arrow.BinaryTypes.String
+ // fallback to binary type for any unsupported type
+ // (shouldn't happen as all types specified in avro 1.11.1 are supported)
+ default:
+ return arrow.BinaryTypes.Binary
+ }
+}
+
+func avroLogicalToArrowField(n *schemaNode) arrow.Field {
+ var dt arrow.DataType
+ // Avro logical types
+ switch lt := n.schema.(avro.LogicalTypeSchema).Logical(); lt.Type() {
+ // The decimal logical type represents an arbitrary-precision signed decimal number of the form unscaled × 10-scale.
+ // A decimal logical type annotates Avro bytes or fixed types. The byte array must contain the two’s-complement
+ // representation of the unscaled integer value in big-endian byte order. The scale is fixed, and is specified
+ // using an attribute.
+ //
+ // The following attributes are supported:
+ // scale, a JSON integer representing the scale (optional). If not specified the scale is 0.
+ // precision, a JSON integer representing the (maximum) precision of decimals stored in this type (required).
+ case "decimal":
+ id := arrow.DECIMAL128
+ if lt.(*avro.DecimalLogicalSchema).Precision() > 38 {
Review Comment:
changed to use constant in comparison
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] loicalleyne commented on a diff in pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne commented on code in PR #36796:
URL: https://github.com/apache/arrow/pull/36796#discussion_r1316465887
##########
go/arrow/avro/schema.go:
##########
@@ -0,0 +1,407 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package avro reads Avro OCF files and presents the extracted data as records
+
+package avro
+
+import (
+ "fmt"
+ "math"
+ "strconv"
+
+ "github.com/apache/arrow/go/v14/arrow"
+ "github.com/apache/arrow/go/v14/internal/types"
+ avro "github.com/hamba/avro/v2"
+)
+
+type schemaNode struct {
+ name string
+ parent *schemaNode
+ schema avro.Schema
+ union bool
+ children []*schemaNode
+ arrowField arrow.Field
+ path []string
+ index, depth int32
+}
+
+var schemaCache avro.SchemaCache
+
+func newSchemaNode() *schemaNode { return &schemaNode{name: "", index: -1} }
+
+func (node *schemaNode) newChild(n string, s avro.Schema) *schemaNode {
+ child := &schemaNode{
+ name: n,
+ parent: node,
+ schema: s,
+ index: int32(len(node.children)),
+ depth: node.depth + 1}
+ node.children = append(node.children, child)
+ return child
+}
+func (node *schemaNode) Children() []*schemaNode { return node.children }
+
+func (node *schemaNode) Name() string { return node.name }
+
+// NamePath returns a slice of keys making up the path to the field
+func (node *schemaNode) NamePath() []string {
+ if len(node.path) == 0 {
+ var path []string
+ cur := node
+ for i := node.depth - 1; i >= 0; i-- {
+ path = append([]string{cur.name}, path...)
+ cur = cur.parent
+ }
+ return path
+ }
+ return node.path
+}
+
+// ArrowSchemaFromAvro returns a new Arrow schema from an Avro schema
+func ArrowSchemaFromAvro(schema avro.Schema) (s *arrow.Schema, err error) {
+ defer func() {
+ if r := recover(); r != nil {
+ s = nil
+ err = fmt.Errorf("invalid avro schema: could not convert schema")
+ }
+ }()
+ n := newSchemaNode()
+ n.schema = schema
+ c := n.newChild(n.schema.(avro.NamedSchema).Name(), n.schema)
+ arrowSchemafromAvro(c)
+ var fields []arrow.Field
+ for _, g := range c.Children() {
+ fields = append(fields, g.arrowField)
+ }
+ s = arrow.NewSchema(fields, nil)
+ return s, nil
+}
+
+func arrowSchemafromAvro(n *schemaNode) {
+ if ns, ok := n.schema.(avro.NamedSchema); ok {
+ schemaCache.Add(ns.Name(), ns)
+ }
+ switch st := n.schema.Type(); st {
+ case "record":
+ iterateFields(n)
+ case "enum":
+ symbols := make(map[string]string)
+ for index, symbol := range n.schema.(avro.PropertySchema).(*avro.EnumSchema).Symbols() {
+ k := strconv.FormatInt(int64(index), 10)
+ symbols[k] = symbol
+ }
+ var dt arrow.DictionaryType = arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Uint64, ValueType: arrow.BinaryTypes.String, Ordered: false}
+ sl := len(symbols)
+ switch {
+ case sl <= math.MaxUint8:
+ dt.IndexType = arrow.PrimitiveTypes.Uint8
+ case sl > math.MaxUint8 && sl <= math.MaxUint16:
+ dt.IndexType = arrow.PrimitiveTypes.Uint16
+ case sl > math.MaxUint16 && sl <= math.MaxUint32:
+ dt.IndexType = arrow.PrimitiveTypes.Uint32
+ }
+ n.arrowField = arrow.Field{Name: n.Name(), Type: &dt, Nullable: true, Metadata: arrow.MetadataFrom(symbols)}
+ case "array":
+ // logical items type
+ c := n.newChild(n.name, n.schema.(*avro.ArraySchema).Items())
+ if isLogicalSchemaType(n.schema.(*avro.ArraySchema).Items()) {
+ avroLogicalToArrowField(c)
+ } else {
+ arrowSchemafromAvro(c)
+ }
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.ListOf(c.arrowField.Type)}
+ case "map":
+ c := n.newChild(n.name, n.schema.(*avro.MapSchema).Values())
+ arrowSchemafromAvro(c)
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.MapOf(arrow.BinaryTypes.String, c.arrowField.Type), Metadata: c.arrowField.Metadata, Nullable: true}
+ case "union":
+ if n.schema.(*avro.UnionSchema).Nullable() {
+ if len(n.schema.(*avro.UnionSchema).Types()) > 1 {
+ n.schema = n.schema.(*avro.UnionSchema).Types()[1]
+ n.union = true
+ arrowSchemafromAvro(n)
+ }
+ }
+ // Avro "fixed" field type = Arrow FixedSize Primitive BinaryType
+ case "fixed":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.Name(), Type: &arrow.FixedSizeBinaryType{ByteWidth: n.schema.(*avro.FixedSchema).Size()}, Nullable: true}
+ }
+ case "string":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "bytes":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "int":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "long":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "float":
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ case "double":
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ case "boolean":
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ case "<ref>":
+ n.schema = schemaCache.Get(n.schema.(*avro.RefSchema).Schema().Name())
+ arrowSchemafromAvro(n)
+ case "null":
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.BinaryTypes.Binary, Nullable: true}
+ }
+}
+
+// iterate record Fields()
+func iterateFields(n *schemaNode) {
+ for _, f := range n.schema.(avro.PropertySchema).(*avro.RecordSchema).Fields() {
+ switch ft := f.Type().(type) {
+ // Avro "array" field type
+ case *avro.ArraySchema:
+ schemaCache.Add(f.Name(), ft.Items())
+ // logical items type
+ c := n.newChild(f.Name(), ft.Items())
+ if isLogicalSchemaType(ft.Items()) {
+ avroLogicalToArrowField(c)
+ } else {
+ arrowSchemafromAvro(c)
+ }
+ c.arrowField = arrow.Field{Name: c.name, Type: arrow.ListOf(c.arrowField.Type), Metadata: c.arrowField.Metadata, Nullable: true}
+ // Avro "enum" field type = Arrow dictionary type
+ case *avro.EnumSchema:
+ schemaCache.Add(f.Name(), f.Type())
+ c := n.newChild(f.Name(), f.Type())
+ symbols := make(map[string]string)
+ for index, symbol := range ft.Symbols() {
+ k := strconv.FormatInt(int64(index), 10)
+ symbols[k] = symbol
+ }
+ var dt arrow.DictionaryType = arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Uint64, ValueType: arrow.BinaryTypes.String, Ordered: false}
+ sl := len(symbols)
+ switch {
+ case sl <= math.MaxUint8:
+ dt.IndexType = arrow.PrimitiveTypes.Uint8
+ case sl > math.MaxUint8 && sl <= math.MaxUint16:
+ dt.IndexType = arrow.PrimitiveTypes.Uint16
+ case sl > math.MaxUint16 && sl <= math.MaxUint32:
+ dt.IndexType = arrow.PrimitiveTypes.Uint32
+ }
+ c.arrowField = arrow.Field{Name: f.Name(), Type: &dt, Nullable: true, Metadata: arrow.MetadataFrom(symbols)}
+ // Avro "fixed" field type = Arrow FixedSize Primitive BinaryType
+ case *avro.FixedSchema:
+ schemaCache.Add(f.Name(), f.Type())
+ if isLogicalSchemaType(f.Type()) {
+ c := n.newChild(f.Name(), f.Type())
+ avroLogicalToArrowField(c)
+ } else {
+ c := n.newChild(f.Name(), f.Type())
+ arrowSchemafromAvro(c)
+ }
+ case *avro.RecordSchema:
+ schemaCache.Add(f.Name(), f.Type())
+ c := n.newChild(f.Name(), f.Type())
+ iterateFields(c)
+ // Avro "map" field type - KVP with value of one type - keys are strings
+ case *avro.MapSchema:
+ schemaCache.Add(f.Name(), ft.Values())
+ c := n.newChild(f.Name(), ft.Values())
+ arrowSchemafromAvro(c)
+ if ft.Values().Type() == "union" {
+ c.arrowField = arrow.Field{Name: f.Name(), Type: arrow.MapOf(arrow.BinaryTypes.String, arrow.BinaryTypes.String), Metadata: c.arrowField.Metadata, Nullable: true}
+ } else {
+ c.arrowField = arrow.Field{Name: f.Name(), Type: arrow.MapOf(arrow.BinaryTypes.String, c.arrowField.Type), Metadata: c.arrowField.Metadata, Nullable: true}
+ }
+ case *avro.UnionSchema:
+ if ft.Nullable() {
+ if len(ft.Types()) > 1 {
+ schemaCache.Add(f.Name(), ft.Types()[1])
+ c := n.newChild(f.Name(), ft.Types()[1])
+ c.union = true
+ arrowSchemafromAvro(c)
+ }
+ }
+ default:
+ schemaCache.Add(f.Name(), f.Type())
+ if isLogicalSchemaType(f.Type()) {
+ c := n.newChild(f.Name(), f.Type())
+ avroLogicalToArrowField(c)
+ } else {
+ c := n.newChild(f.Name(), f.Type())
+ arrowSchemafromAvro(c)
+ }
+
+ }
+ }
+ var fields []arrow.Field
+ for _, child := range n.Children() {
+ fields = append(fields, child.arrowField)
+ }
+
+ namedSchema, ok := isNamedSchema(n.schema)
+ if !ok || namedSchema == n.name+"_data" || !n.union {
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.StructOf(fields...), Nullable: true}
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.StructOf(fields...), Nullable: true, Metadata: arrow.MetadataFrom(map[string]string{"typeName": namedSchema})}
+ }
+
+}
+
+func isLogicalSchemaType(s avro.Schema) bool {
+ lts, ok := s.(avro.LogicalTypeSchema)
+ if !ok {
+ return false
+ }
+ if lt := lts.Logical(); lt != nil {
+ return true
+ }
+ return false
+}
+
+func isNamedSchema(s avro.Schema) (string, bool) {
+ if ns, ok := s.(avro.NamedSchema); ok {
+ return ns.FullName(), ok
+ }
+ return "", false
+}
+
+// Avro primitive type.
+//
+// NOTE: Arrow Binary type is used as a catchall to avoid potential data loss.
Review Comment:
there aren't any unimplemented avro Primitive types, I could just remove the default: completely, what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-36760: [Go] Adding avro ocf reader - schema converter [arrow]
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne closed pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
URL: https://github.com/apache/arrow/pull/36796
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on a diff in pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #36796:
URL: https://github.com/apache/arrow/pull/36796#discussion_r1272513931
##########
go/arrow/avro/schema.go:
##########
@@ -0,0 +1,386 @@
+// Package avro reads Avro OCF files and presents the extracted data as records
Review Comment:
You need to add the Apache license to the top of the file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36796:
URL: https://github.com/apache/arrow/pull/36796#issuecomment-1644730597
:warning: GitHub issue #36760 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] loicalleyne commented on a diff in pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne commented on code in PR #36796:
URL: https://github.com/apache/arrow/pull/36796#discussion_r1281165680
##########
go/arrow/avro/schema.go:
##########
@@ -0,0 +1,409 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package avro reads Avro OCF files and presents the extracted data as records
+package avro
+
+import (
+ "encoding/json"
+ "fmt"
+ "math"
+ "strconv"
+
+ "github.com/apache/arrow/go/v13/arrow"
+)
+
+type schemaNode struct {
+ name string
+ ofType interface{}
+ logicalType string
+ precision int32
+ scale int32
+ size int
+ symbols []string
+ fields []interface{}
+}
+
+// ArrowSchemaFromAvro returns a new Arrow schema from an Avro schema JSON.
+// If the top level is of record type, set includeTopLevel to either make
+// its fields top level fields in the resulting schema or nested in a single field.
+func ArrowSchemaFromAvro(avroSchema []byte) (*arrow.Schema, error) {
+ var m map[string]interface{}
+ var node schemaNode
+ json.Unmarshal(avroSchema, &m)
+ if m["type"].(string) == "record" {
+ if _, ok := m["fields"]; ok {
+ for _, field := range m["fields"].([]interface{}) {
+ node.fields = append(node.fields, field.(map[string]interface{}))
+ }
+ if len(node.fields) == 0 {
+ return nil, fmt.Errorf("invalid avro schema: no top level record fields found")
+ }
+ }
+ }
+ fields := iterateFields(node.fields)
+ return arrow.NewSchema(fields, nil), nil
+}
+
+func iterateFields(f []interface{}) []arrow.Field {
+ var s []arrow.Field
+ for _, field := range f {
+ var n schemaNode
+ n.name = field.(map[string]interface{})["name"].(string)
+ n.ofType = field.(map[string]interface{})["type"]
+ switch n.ofType.(type) {
+ // Getting field type from within field object
+ case string:
+ switch n.ofType.(string) {
+ case "enum":
+ for _, symbol := range field.(map[string]interface{})["symbols"].([]interface{}) {
+ n.symbols = append(n.symbols, symbol.(string))
+ }
+ default:
+ if lt, ok := field.(map[string]interface{})["logicalType"]; ok {
+ n.logicalType = lt.(string)
+ }
+ if lt, ok := field.(map[string]interface{})["size"]; ok {
+ n.size = int(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["precision"]; ok {
+ n.precision = int32(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["scale"]; ok {
+ n.scale = int32(lt.(float64))
+ }
+ }
+ // Field type is an object
+ case map[string]interface{}:
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["logicalType"]; ok {
+ n.logicalType = lt.(string)
+ }
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["size"]; ok {
+ n.size = int(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["precision"]; ok {
+ n.precision = int32(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["type"].(map[string]interface{})["scale"]; ok {
+ n.scale = int32(lt.(float64))
+ }
+ case []interface{}:
+
+ default:
+ if lt, ok := field.(map[string]interface{})["logicalType"]; ok {
+ n.logicalType = lt.(string)
+ }
+ if lt, ok := field.(map[string]interface{})["size"]; ok {
+ n.size = int(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["precision"]; ok {
+ n.precision = int32(lt.(float64))
+ }
+ if lt, ok := field.(map[string]interface{})["scale"]; ok {
+ n.scale = int32(lt.(float64))
+ }
+ }
+ // Field is of type "record"
+ if nf, f := field.(map[string]interface{})["fields"]; f {
+ switch nf.(type) {
+ // primitive & complex types
+ case map[string]interface{}:
+ for _, v := range nf.(map[string]interface{})["fields"].([]interface{}) {
+ n.fields = append(n.fields, v.(map[string]interface{}))
+ }
+ // type unions
+ default:
+ for _, v := range nf.([]interface{}) {
+ n.fields = append(n.fields, v.(map[string]interface{}))
+ }
+ }
+ }
+ s = append(s, traverseNodes(n))
+ }
+ return s
+}
+
+func traverseNodes(node schemaNode) arrow.Field {
+ switch node.ofType.(type) {
+ case string:
+ // Avro primitive type
+ if len(node.fields) == 0 {
+ switch node.ofType.(string) {
+ case "boolean", "int", "long", "float", "double", "bytes", "string":
+ if node.logicalType != "" {
+ return avroLogicalToArrowField(node)
+ }
+ // Avro primitive type
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(node.ofType.(string))}
+ case "fixed":
+ // Duration type is not supported in github.com/linkedin/goavro
+ // Implementing as Binary for now.
+ switch node.logicalType {
+ case "decimal":
+ return avroLogicalToArrowField(node)
+ case "duration":
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.Binary}
+ //return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.MonthDayNanoInterval}
+ }
+ return arrow.Field{Name: node.name, Type: &arrow.FixedSizeBinaryType{ByteWidth: node.size}}
+ case "enum":
+ symbols := make(map[string]string)
+ for index, symbol := range node.symbols {
+ k := strconv.FormatInt(int64(index), 10)
+ symbols[k] = symbol
+ }
+ var dt arrow.DictionaryType = arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Uint64, ValueType: arrow.BinaryTypes.String, Ordered: false}
+ sl := len(symbols)
+ switch {
+ case sl <= math.MaxUint8:
+ dt.IndexType = arrow.PrimitiveTypes.Uint8
+ case sl > math.MaxUint8 && sl <= math.MaxUint16:
+ dt.IndexType = arrow.PrimitiveTypes.Uint16
+ case sl > math.MaxUint16 && sl <= math.MaxUint32:
+ dt.IndexType = arrow.PrimitiveTypes.Uint32
+ }
+ return arrow.Field{Name: node.name, Type: &dt, Nullable: true, Metadata: arrow.MetadataFrom(symbols)}
+ default:
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(node.ofType.(string))}
+ }
+ } else {
+ // avro "record" type, node has "fields" array
+ if node.ofType.(string) == "record" {
+ var n schemaNode
+ n.name = node.name
+ n.ofType = node.ofType
+ if len(node.fields) > 0 {
+ n.fields = append(n.fields, node.fields...)
+ }
+ f := iterateFields(n.fields)
+ return arrow.Field{Name: node.name, Type: arrow.StructOf(f...)}
+ }
+ }
+ // Avro complex types
+ case map[string]interface{}:
+ //var n schemaNode
+ //n.name = node.name
+ //n.ofType = node.ofType.(map[string]interface{})["type"]
+ switch node.logicalType {
+ case "":
+ return avroComplexToArrowField(node)
+ default:
+ return avroLogicalToArrowField(node)
+ }
+
+ // Avro union types
+ case []interface{}:
+ var unionTypes []string
+ for _, ft := range node.ofType.([]interface{}) {
+ switch ft.(type) {
+ // primitive types
+ case string:
+ if ft != "null" {
+ unionTypes = append(unionTypes, ft.(string))
+ }
+ continue
+ // complex types
+ case map[string]interface{}:
+ var n schemaNode
+ n.name = node.name
+ n.ofType = ft.(map[string]interface{})["type"]
+ if _, f := ft.(map[string]interface{})["fields"]; f {
+ for _, field := range ft.(map[string]interface{})["fields"].([]interface{}) {
+ n.fields = append(n.fields, field.(map[string]interface{}))
+ }
+ }
+ f := iterateFields(n.fields)
+ return arrow.Field{Name: node.name, Type: arrow.StructOf(f...)}
+ }
+ }
+ // Supported Avro union type is null + one other type.
+ // TODO: Complex AVRO union to Arrow Dense || Sparse Union.
+ if len(unionTypes) == 1 {
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(unionTypes[0])}
+ } else {
+ // BYTE_ARRAY is the catchall if union type is anything beyond null + one other type.
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.Binary}
+ }
+ }
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.Binary}
+}
+
+func avroLogicalToArrowField(node schemaNode) arrow.Field {
+ // Avro logical types
+ switch node.logicalType {
+ // The decimal logical type represents an arbitrary-precision signed decimal number of the form unscaled × 10-scale.
+ // A decimal logical type annotates Avro bytes or fixed types. The byte array must contain the two’s-complement
+ // representation of the unscaled integer value in big-endian byte order. The scale is fixed, and is specified
+ // using an attribute.
+ //
+ // The following attributes are supported:
+ // scale, a JSON integer representing the scale (optional). If not specified the scale is 0.
+ // precision, a JSON integer representing the (maximum) precision of decimals stored in this type (required).
+ case "decimal":
+ if node.precision <= 38 {
+ return arrow.Field{Name: node.name, Type: &arrow.Decimal128Type{Precision: node.precision, Scale: node.scale}}
+ } else {
+ return arrow.Field{Name: node.name, Type: &arrow.Decimal256Type{Precision: node.precision, Scale: node.scale}}
+ }
+
+ // The uuid logical type represents a random generated universally unique identifier (UUID).
+ // A uuid logical type annotates an Avro string. The string has to conform with RFC-4122
+ case "uuid":
+ return arrow.Field{Name: node.name, Type: arrow.BinaryTypes.String}
+
+ // The date logical type represents a date within the calendar, with no reference to a particular
+ // time zone or time of day.
+ // A date logical type annotates an Avro int, where the int stores the number of days from the unix epoch,
+ // 1 January 1970 (ISO calendar).
+ case "date":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Date32}
+
+ // The time-millis logical type represents a time of day, with no reference to a particular calendar,
+ // time zone or date, with a precision of one millisecond.
+ // A time-millis logical type annotates an Avro int, where the int stores the number of milliseconds
+ // after midnight, 00:00:00.000.
+ case "time-millis":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Time32ms}
+
+ // The time-micros logical type represents a time of day, with no reference to a particular calendar,
+ // time zone or date, with a precision of one microsecond.
+ // A time-micros logical type annotates an Avro long, where the long stores the number of microseconds
+ // after midnight, 00:00:00.000000.
+ case "time-micros":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Time64us}
+
+ // The timestamp-millis logical type represents an instant on the global timeline, independent of a
+ // particular time zone or calendar, with a precision of one millisecond. Please note that time zone
+ // information gets lost in this process. Upon reading a value back, we can only reconstruct the instant,
+ // but not the original representation. In practice, such timestamps are typically displayed to users in
+ // their local time zones, therefore they may be displayed differently depending on the execution environment.
+ // A timestamp-millis logical type annotates an Avro long, where the long stores the number of milliseconds
+ // from the unix epoch, 1 January 1970 00:00:00.000 UTC.
+ case "timestamp-millis":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Timestamp_ms}
+ // The timestamp-micros logical type represents an instant on the global timeline, independent of a
+ // particular time zone or calendar, with a precision of one microsecond. Please note that time zone
+ // information gets lost in this process. Upon reading a value back, we can only reconstruct the instant,
+ // but not the original representation. In practice, such timestamps are typically displayed to users
+ // in their local time zones, therefore they may be displayed differently depending on the execution environment.
+ // A timestamp-micros logical type annotates an Avro long, where the long stores the number of microseconds
+ // from the unix epoch, 1 January 1970 00:00:00.000000 UTC.
+ case "timestamp-micros":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Timestamp_us}
+ // The local-timestamp-millis logical type represents a timestamp in a local timezone, regardless of
+ // what specific time zone is considered local, with a precision of one millisecond.
+ // A local-timestamp-millis logical type annotates an Avro long, where the long stores the number of
+ // milliseconds, from 1 January 1970 00:00:00.000.
+ case "local-timestamp-millis":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Timestamp_ms}
+ // The local-timestamp-micros logical type represents a timestamp in a local timezone, regardless of
+ // what specific time zone is considered local, with a precision of one microsecond.
+ // A local-timestamp-micros logical type annotates an Avro long, where the long stores the number of
+ // microseconds, from 1 January 1970 00:00:00.000000.
+ case "local-timestamp-micros":
+ return arrow.Field{Name: node.name, Type: arrow.FixedWidthTypes.Timestamp_us}
+ // Avro primitive type
+ default:
+ return arrow.Field{Name: node.name, Type: AvroPrimitiveToArrowType(node.ofType.(string))}
+ }
+ return arrow.Field{}
Review Comment:
the premise is that the schema would be taken from a OCF that was written with a valid schema and that there should always be a valid primitive or complex type passed into the function; I put the empty field because the compiler complained about a missing return.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] loicalleyne commented on a diff in pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne commented on code in PR #36796:
URL: https://github.com/apache/arrow/pull/36796#discussion_r1281132461
##########
go/arrow/avro/common.go:
##########
@@ -0,0 +1,162 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package avro
+
+import (
+ "errors"
+ "fmt"
+
+ "github.com/apache/arrow/go/v13/arrow"
+ "github.com/apache/arrow/go/v13/arrow/memory"
+)
+
+var (
+ ErrMismatchFields = errors.New("arrow/avro: number of records mismatch")
+)
+
+// Option configures an Avro reader/writer.
+type Option func(config)
+type config interface{}
+
+// WithTopLevel specifies whether to include an Avro schema's top level record
+// or only its fields.
+func WithTopLevel(b bool) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.topLevel = b
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithAllocator specifies the Arrow memory allocator used while building records.
+func WithAllocator(mem memory.Allocator) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.mem = mem
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithChunk specifies the chunk size used while reading Avro OCF files.
+//
+// If n is zero or 1, no chunking will take place and the reader will create
+// one record per row.
+// If n is greater than 1, chunks of n rows will be read.
+// If n is negative, the reader will load the whole Avro OCF file into memory and
+// create one big record with all the rows.
+func WithChunk(n int) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.chunk = n
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// DefaultNullValues is the set of values considered as NULL values by default
+// when Reader is configured to handle NULL values.
+var DefaultNullValues = []string{"", "NULL", "null"}
+
+// WithNullReader sets options for a CSV Reader pertaining to NULL value
+// handling. If stringsCanBeNull is true, then a string that matches one of the
+// nullValues set will be interpreted as NULL. Numeric columns will be checked
+// for nulls in all cases. If no nullValues arguments are passed in, the
+// defaults set in NewReader() will be kept.
+//
+// When no NULL values is given, the default set is taken from DefaultNullValues.
+func WithNullReader(stringsCanBeNull bool, nullValues ...string) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ cfg.stringsCanBeNull = stringsCanBeNull
+
+ if len(nullValues) == 0 {
+ nullValues = DefaultNullValues
+ }
+ cfg.nulls = make([]string, len(nullValues))
+ copy(cfg.nulls, nullValues)
+ default:
+ panic(fmt.Errorf("arrow/avro: unknown config type %T", cfg))
+ }
+ }
+}
+
+// WithColumnTypes allows specifying optional per-column types (disabling
+// type inference on those columns).
+//
+// Will panic if used in conjunction with an explicit schema.
+func WithColumnTypes(types map[string]arrow.DataType) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ if cfg.schema != nil {
+ panic(fmt.Errorf("%w: cannot use WithColumnTypes with explicit schema", arrow.ErrInvalid))
+ }
+ cfg.columnTypes = types
+ default:
+ panic(fmt.Errorf("%w: WithColumnTypes only allowed for csv reader", arrow.ErrInvalid))
+ }
+ }
+}
+
+// WithIncludeColumns indicates the names of the columns from the AVRO file
+// that should actually be read and converted (in the slice's order).
+// If set and non-empty, columns not in this slice will be ignored.
+//
+// Will panic if used in conjunction with an explicit schema.
+func WithIncludeColumns(cols []string) Option {
+ return func(cfg config) {
+ switch cfg := cfg.(type) {
+ case *OCFReader:
+ if cfg.schema != nil {
+ panic(fmt.Errorf("%w: cannot use WithIncludeColumns with explicit schema", arrow.ErrInvalid))
+ }
+ cfg.columnFilter = cols
+ default:
+ panic(fmt.Errorf("%w: WithIncludeColumns only allowed on csv Reader", arrow.ErrInvalid))
+ }
+ }
+}
+
+func validate(schema *arrow.Schema) {
+ for i, f := range schema.Fields() {
+ switch ft := f.Type.(type) {
+ case *arrow.BooleanType:
+ case *arrow.Int8Type, *arrow.Int16Type, *arrow.Int32Type, *arrow.Int64Type:
+ case *arrow.Uint8Type, *arrow.Uint16Type, *arrow.Uint32Type, *arrow.Uint64Type:
+ case *arrow.Float16Type, *arrow.Float32Type, *arrow.Float64Type:
+ case *arrow.StringType, *arrow.LargeStringType:
+ case *arrow.TimestampType:
+ case *arrow.Date32Type, *arrow.Date64Type:
+ case *arrow.Decimal128Type, *arrow.Decimal256Type:
+ case *arrow.ListType, *arrow.LargeListType, *arrow.FixedSizeListType:
+ case *arrow.BinaryType, *arrow.LargeBinaryType, *arrow.FixedSizeBinaryType:
+ case arrow.ExtensionType:
+ case *arrow.NullType:
+ default:
+ panic(fmt.Errorf("arrow/csv: field %d (%s) has invalid data type %T", i, f.Name, ft))
+ }
+ }
+}
Review Comment:
removed in latest commit, left over from CSV implementation that was used as a skeleton for this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] loicalleyne commented on a diff in pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne commented on code in PR #36796:
URL: https://github.com/apache/arrow/pull/36796#discussion_r1293980483
##########
go/arrow/avro/reader.go:
##########
@@ -0,0 +1,490 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package avro
+
+import (
+ "errors"
+ "fmt"
+ "io"
+ "sync"
+ "sync/atomic"
+
+ "github.com/apache/arrow/go/v13/arrow"
+ "github.com/apache/arrow/go/v13/arrow/array"
+ "github.com/apache/arrow/go/v13/arrow/internal/debug"
+ "github.com/apache/arrow/go/v13/arrow/memory"
+ "github.com/linkedin/goavro/v2"
+)
+
+// Reader wraps goavro/OCFReader and creates array.Records from a schema.
+type OCFReader struct {
+ r *goavro.OCFReader
+ schema *arrow.Schema
+
+ refs int64
+ bld *array.RecordBuilder
+ cur arrow.Record
+ err error
+
+ chunk int
+ done bool
+ next func() bool
+
+ mem memory.Allocator
+
+ topLevel bool
+ once sync.Once
+
+ fieldConverter []func(val string)
+ columnFilter []string
+ columnTypes map[string]arrow.DataType
+
+ stringsCanBeNull bool
+ nulls []string
+}
+
+// NewReader returns a reader that reads from an Avro OCF file and creates
+// arrow.Records from the converted schema.
+func NewOCFReader(r io.Reader, opts ...Option) *OCFReader {
+ ocfr, err := goavro.NewOCFReader(r)
+ if err != nil {
+ panic(fmt.Errorf("%w: could not create avro ocfreader", arrow.ErrInvalid))
+ }
+
+ rr := &OCFReader{
+ r: ocfr,
+ //schema: schema,
Review Comment:
my mistake, the reader wasn't supposed to be part of this PR at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] loicalleyne commented on a diff in pull request #36796: GH-36760: [Go] Adding avro ocf reader - schema converter
Posted by "loicalleyne (via GitHub)" <gi...@apache.org>.
loicalleyne commented on code in PR #36796:
URL: https://github.com/apache/arrow/pull/36796#discussion_r1316460234
##########
go/arrow/avro/schema.go:
##########
@@ -0,0 +1,407 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package avro reads Avro OCF files and presents the extracted data as records
+
+package avro
+
+import (
+ "fmt"
+ "math"
+ "strconv"
+
+ "github.com/apache/arrow/go/v14/arrow"
+ "github.com/apache/arrow/go/v14/internal/types"
+ avro "github.com/hamba/avro/v2"
+)
+
+type schemaNode struct {
+ name string
+ parent *schemaNode
+ schema avro.Schema
+ union bool
+ children []*schemaNode
+ arrowField arrow.Field
+ path []string
+ index, depth int32
+}
+
+var schemaCache avro.SchemaCache
+
+func newSchemaNode() *schemaNode { return &schemaNode{name: "", index: -1} }
+
+func (node *schemaNode) newChild(n string, s avro.Schema) *schemaNode {
+ child := &schemaNode{
+ name: n,
+ parent: node,
+ schema: s,
+ index: int32(len(node.children)),
+ depth: node.depth + 1}
+ node.children = append(node.children, child)
+ return child
+}
+func (node *schemaNode) Children() []*schemaNode { return node.children }
+
+func (node *schemaNode) Name() string { return node.name }
+
+// NamePath returns a slice of keys making up the path to the field
+func (node *schemaNode) NamePath() []string {
+ if len(node.path) == 0 {
+ var path []string
+ cur := node
+ for i := node.depth - 1; i >= 0; i-- {
+ path = append([]string{cur.name}, path...)
+ cur = cur.parent
+ }
+ return path
+ }
+ return node.path
+}
+
+// ArrowSchemaFromAvro returns a new Arrow schema from an Avro schema
+func ArrowSchemaFromAvro(schema avro.Schema) (s *arrow.Schema, err error) {
+ defer func() {
+ if r := recover(); r != nil {
+ s = nil
+ err = fmt.Errorf("invalid avro schema: could not convert schema")
+ }
+ }()
+ n := newSchemaNode()
+ n.schema = schema
+ c := n.newChild(n.schema.(avro.NamedSchema).Name(), n.schema)
+ arrowSchemafromAvro(c)
+ var fields []arrow.Field
+ for _, g := range c.Children() {
+ fields = append(fields, g.arrowField)
+ }
+ s = arrow.NewSchema(fields, nil)
+ return s, nil
+}
+
+func arrowSchemafromAvro(n *schemaNode) {
+ if ns, ok := n.schema.(avro.NamedSchema); ok {
+ schemaCache.Add(ns.Name(), ns)
+ }
+ switch st := n.schema.Type(); st {
+ case "record":
+ iterateFields(n)
+ case "enum":
+ symbols := make(map[string]string)
+ for index, symbol := range n.schema.(avro.PropertySchema).(*avro.EnumSchema).Symbols() {
+ k := strconv.FormatInt(int64(index), 10)
+ symbols[k] = symbol
+ }
+ var dt arrow.DictionaryType = arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Uint64, ValueType: arrow.BinaryTypes.String, Ordered: false}
+ sl := len(symbols)
+ switch {
+ case sl <= math.MaxUint8:
+ dt.IndexType = arrow.PrimitiveTypes.Uint8
+ case sl > math.MaxUint8 && sl <= math.MaxUint16:
+ dt.IndexType = arrow.PrimitiveTypes.Uint16
+ case sl > math.MaxUint16 && sl <= math.MaxUint32:
+ dt.IndexType = arrow.PrimitiveTypes.Uint32
+ }
+ n.arrowField = arrow.Field{Name: n.Name(), Type: &dt, Nullable: true, Metadata: arrow.MetadataFrom(symbols)}
+ case "array":
+ // logical items type
+ c := n.newChild(n.name, n.schema.(*avro.ArraySchema).Items())
+ if isLogicalSchemaType(n.schema.(*avro.ArraySchema).Items()) {
+ avroLogicalToArrowField(c)
+ } else {
+ arrowSchemafromAvro(c)
+ }
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.ListOf(c.arrowField.Type)}
+ case "map":
+ c := n.newChild(n.name, n.schema.(*avro.MapSchema).Values())
+ arrowSchemafromAvro(c)
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.MapOf(arrow.BinaryTypes.String, c.arrowField.Type), Metadata: c.arrowField.Metadata, Nullable: true}
+ case "union":
+ if n.schema.(*avro.UnionSchema).Nullable() {
+ if len(n.schema.(*avro.UnionSchema).Types()) > 1 {
+ n.schema = n.schema.(*avro.UnionSchema).Types()[1]
+ n.union = true
+ arrowSchemafromAvro(n)
+ }
+ }
+ // Avro "fixed" field type = Arrow FixedSize Primitive BinaryType
+ case "fixed":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.Name(), Type: &arrow.FixedSizeBinaryType{ByteWidth: n.schema.(*avro.FixedSchema).Size()}, Nullable: true}
+ }
+ case "string":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "bytes":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "int":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "long":
+ if isLogicalSchemaType(n.schema) {
+ avroLogicalToArrowField(n)
+ } else {
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ }
+ case "float":
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ case "double":
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ case "boolean":
+ n.arrowField = arrow.Field{Name: n.name, Type: avroPrimitiveToArrowType(string(st)), Nullable: true}
+ case "<ref>":
+ n.schema = schemaCache.Get(n.schema.(*avro.RefSchema).Schema().Name())
+ arrowSchemafromAvro(n)
+ case "null":
+ n.arrowField = arrow.Field{Name: n.name, Type: arrow.BinaryTypes.Binary, Nullable: true}
+ }
+}
+
+// iterate record Fields()
+func iterateFields(n *schemaNode) {
+ for _, f := range n.schema.(avro.PropertySchema).(*avro.RecordSchema).Fields() {
+ switch ft := f.Type().(type) {
+ // Avro "array" field type
+ case *avro.ArraySchema:
+ schemaCache.Add(f.Name(), ft.Items())
+ // logical items type
+ c := n.newChild(f.Name(), ft.Items())
+ if isLogicalSchemaType(ft.Items()) {
+ avroLogicalToArrowField(c)
+ } else {
+ arrowSchemafromAvro(c)
+ }
+ c.arrowField = arrow.Field{Name: c.name, Type: arrow.ListOf(c.arrowField.Type), Metadata: c.arrowField.Metadata, Nullable: true}
+ // Avro "enum" field type = Arrow dictionary type
+ case *avro.EnumSchema:
+ schemaCache.Add(f.Name(), f.Type())
+ c := n.newChild(f.Name(), f.Type())
+ symbols := make(map[string]string)
+ for index, symbol := range ft.Symbols() {
+ k := strconv.FormatInt(int64(index), 10)
+ symbols[k] = symbol
+ }
+ var dt arrow.DictionaryType = arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Uint64, ValueType: arrow.BinaryTypes.String, Ordered: false}
+ sl := len(symbols)
+ switch {
+ case sl <= math.MaxUint8:
+ dt.IndexType = arrow.PrimitiveTypes.Uint8
+ case sl > math.MaxUint8 && sl <= math.MaxUint16:
+ dt.IndexType = arrow.PrimitiveTypes.Uint16
+ case sl > math.MaxUint16 && sl <= math.MaxUint32:
+ dt.IndexType = arrow.PrimitiveTypes.Uint32
+ }
+ c.arrowField = arrow.Field{Name: f.Name(), Type: &dt, Nullable: true, Metadata: arrow.MetadataFrom(symbols)}
+ // Avro "fixed" field type = Arrow FixedSize Primitive BinaryType
+ case *avro.FixedSchema:
+ schemaCache.Add(f.Name(), f.Type())
+ if isLogicalSchemaType(f.Type()) {
+ c := n.newChild(f.Name(), f.Type())
+ avroLogicalToArrowField(c)
+ } else {
+ c := n.newChild(f.Name(), f.Type())
+ arrowSchemafromAvro(c)
+ }
+ case *avro.RecordSchema:
+ schemaCache.Add(f.Name(), f.Type())
+ c := n.newChild(f.Name(), f.Type())
+ iterateFields(c)
+ // Avro "map" field type - KVP with value of one type - keys are strings
+ case *avro.MapSchema:
+ schemaCache.Add(f.Name(), ft.Values())
+ c := n.newChild(f.Name(), ft.Values())
+ arrowSchemafromAvro(c)
+ if ft.Values().Type() == "union" {
+ c.arrowField = arrow.Field{Name: f.Name(), Type: arrow.MapOf(arrow.BinaryTypes.String, arrow.BinaryTypes.String), Metadata: c.arrowField.Metadata, Nullable: true}
+ } else {
+ c.arrowField = arrow.Field{Name: f.Name(), Type: arrow.MapOf(arrow.BinaryTypes.String, c.arrowField.Type), Metadata: c.arrowField.Metadata, Nullable: true}
+ }
+ case *avro.UnionSchema:
+ if ft.Nullable() {
+ if len(ft.Types()) > 1 {
+ schemaCache.Add(f.Name(), ft.Types()[1])
+ c := n.newChild(f.Name(), ft.Types()[1])
+ c.union = true
+ arrowSchemafromAvro(c)
+ }
+ }
+ default:
+ schemaCache.Add(f.Name(), f.Type())
Review Comment:
As far as I can determine from the spec and samples I've seen, in Avro a referenced field has to be defined first in the depth-first, left-to-right traversal of the JSON parse tree so if the name is reused it would get overwritten in the schema cache before being referenced again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org