You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "aslatter (via GitHub)" <gi...@apache.org> on 2025/08/24 19:40:18 UTC
[I] "error encountered during schema visitor: arrow/array: index out of range" while reading S3 table metadata [iceberg-go]
aslatter opened a new issue, #547:
URL: https://github.com/apache/iceberg-go/issues/547
### Apache Iceberg version
None
### Please describe the bug 🐞
[schema.json](https://github.com/user-attachments/files/21958694/schema.json)
[parquet_data.zip](https://github.com/user-attachments/files/21958704/parquet_data.zip)
I don't understand the bug, so I'll describe what I was doing to run into it, and I'll also show my attempt at a minimal repro.
I had enabled "S3 Table Metadata", which stores meta-data about an S3 bucket in an iceberg table. whenever I tried to read from the tables, when performing a scan of the table using iceberg-go, I would get "error encountered during schema visitor: arrow/array: index out of range" errors.
These errors go away if I apply the following patch to `arrow-go`:
```patch
diff --git a/arrow/array/struct.go b/arrow/array/struct.go
index 6883712c..4f943ac6 100644
--- a/arrow/array/struct.go
+++ b/arrow/array/struct.go
@@ -192,7 +192,7 @@ func (a *Struct) setData(data *Data) {
a.fields = make([]arrow.Array, len(data.childData))
for i, child := range data.childData {
if data.offset != 0 || child.Len() != data.length {
- sub := NewSliceData(child, int64(data.offset), int64(data.offset+data.length))
+ sub := NewSliceData(child, int64(data.offset), int64(data.offset+child.Len()))
a.fields[i] = MakeFromData(sub)
sub.Release()
} else {
```
I'm just banging rocks together - I have no idea if this patch is a good idea or not.
In an attempt to produce a minimal repro, I grabbed the iceberg schema and raw parquet data from the table, and loaded them with the following go program:
<details>
<summary>Click to expand</summary>
```go
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"os"
"github.com/apache/iceberg-go"
"github.com/apache/iceberg-go/table"
"github.com/apache/arrow-go/v18/arrow/memory"
"github.com/apache/arrow-go/v18/parquet"
pfile "github.com/apache/arrow-go/v18/parquet/file"
"github.com/apache/arrow-go/v18/parquet/pqarrow"
)
func main() {
if err := mainErr(); err != nil {
fmt.Fprintln(os.Stdout, "error:", err)
os.Exit(1)
}
}
func mainErr() error {
var parquetfile string
var icebergSchemaFile string
flag.StringVar(&parquetfile, "parquet-file", "", "parquet file to load")
flag.StringVar(&icebergSchemaFile, "iceberg-schema-file", "", "iceberg schema to use to interpret parquet file")
flag.Parse()
var schema iceberg.Schema
schemaBytes, err := os.ReadFile(icebergSchemaFile)
if err != nil {
return fmt.Errorf("reading iceberg schema file: %s", err)
}
err = json.Unmarshal(schemaBytes, &schema)
if err != nil {
return fmt.Errorf("unmarhcalling iceberg schema: %s", err)
}
parquetFile, err := os.Open(parquetfile)
if err != nil {
return fmt.Errorf("opening parquet file: %s")
}
ctx := context.Background()
return dumpParquetWithSchema(ctx, parquetFile, &schema)
}
func dumpParquetWithSchema(ctx context.Context, file parquet.ReaderAtSeeker, schema *iceberg.Schema) error {
pr, err := pfile.NewParquetReader(file)
if err != nil {
return fmt.Errorf("creating parquet reader: %s", err)
}
defer pr.Close()
arrowReader, err := pqarrow.NewFileReader(pr, pqarrow.ArrowReadProperties{
BatchSize: 1024,
}, memory.DefaultAllocator)
if err != nil {
return fmt.Errorf("creating pqarrow file reader: %s", err)
}
ar, err := arrowReader.GetRecordReader(ctx, nil, nil)
if err != nil {
return fmt.Errorf("creating arrow reader: %s", err)
}
defer ar.Release()
arrowFileSchema := ar.Schema()
fileSchema, err := table.ArrowSchemaToIceberg(arrowFileSchema, false, nil)
if err != nil {
return fmt.Errorf("convert parquet file schema to iceberg schema: %s", err)
}
for ar.Next() {
rec := ar.Record()
newRec, err := table.ToRequestedSchema(ctx, schema, fileSchema, rec, false, false, false)
if err != nil {
return fmt.Errorf("table to requested schema: %s", err)
}
err = json.NewEncoder(os.Stdout).Encode(newRec)
if err != nil {
return fmt.Errorf("encoding arrow rec as json: %s", err)
}
rec.Release()
}
return nil
}
```
</details>
It fails with the same error as the table-scan described above, and the same patch to arrow-go makes the error go away.
However if I do a raw dump of the parquet file with arrow-go without trying to apply the iceberg-schema to it, I don't get any errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] "error encountered during schema visitor: arrow/array: index out of range" while reading S3 table metadata [iceberg-go]
Posted by "aslatter (via GitHub)" <gi...@apache.org>.
aslatter commented on issue #547:
URL: https://github.com/apache/iceberg-go/issues/547#issuecomment-3218349772
My original program for working with s3-metadata which ran into this is the following:
<details>
<summary>Click to expand</summary>
```go
package main
import (
"context"
"encoding/json"
"errors"
"flag"
"fmt"
"os"
"strings"
"github.com/apache/iceberg-go"
"github.com/apache/iceberg-go/catalog/rest"
"github.com/aws/aws-sdk-go-v2/aws/arn"
"github.com/aws/aws-sdk-go-v2/config"
)
func main() {
if err := mainErr(); err != nil {
fmt.Fprintln(os.Stdout, "error:", err)
os.Exit(1)
}
}
func mainErr() error {
var tableBucketARNString string
var tableString string
flag.StringVar(&tableBucketARNString, "table-bucket", "", "s3 table-bucket")
flag.StringVar(&tableString, "table", "", "table within s3 table-bucket, in the form namespace.table_name")
flag.Parse()
if tableBucketARNString == "" {
return errors.New("flag -table-bucket is required")
}
if tableString == "" {
return errors.New("flag -table is required")
}
bucketARN, err := arn.Parse(tableBucketARNString)
if err != nil {
return fmt.Errorf("parsing bucket arn: %s", err)
}
tableIdent := strings.Split(tableString, ".")
ctx := context.Background()
cfg, err := config.LoadDefaultConfig(ctx)
if err != nil {
return fmt.Errorf("loading AWS config: %s", err)
}
awsRegion := cfg.Region
if awsRegion == "" {
return errors.New("missing AWS region")
}
cat, err := rest.NewCatalog(ctx, "unknown", fmt.Sprintf("https://s3tables.%s.amazonaws.com/iceberg", awsRegion),
rest.WithAwsConfig(cfg),
rest.WithWarehouseLocation(tableBucketARNString),
rest.WithSigV4(),
rest.WithSigV4RegionSvc(awsRegion, bucketARN.Service),
)
if err != nil {
return fmt.Errorf("opening iceberg catalog: %s", err)
}
inventory, err := cat.LoadTable(ctx, tableIdent, iceberg.Properties{})
_, riter, err := inventory.Scan().ToArrowRecords(ctx)
if err != nil {
return fmt.Errorf("scanning inventory table: %s", err)
}
for rec, err := range riter {
if err != nil {
return fmt.Errorf("iterating over scanned records: %s", err)
}
err = json.NewEncoder(os.Stdout).Encode(rec)
if err != nil {
return fmt.Errorf("encoding json for record: %s", err)
}
}
return nil
}
```
</details>
Example:
```
export AWS_REGION=<aws region>
AWS_ACCOUNT_ID=12345678
S3_BUCKET_NAME="your-bucket-with-metadata-enabled"
go run ./ -table-bucket arn:aws:s3tables:$AWS_REGION:$AWS_ACCOUNT_ID:bucket/aws-s3 -table b_${S3_BUCKET_NAME}.inventory
```
(I had inventory enabled - if you only have "journal" enabled you'd use that instead of "inventory")
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] "error encountered during schema visitor: arrow/array: index out of range" while reading S3 table metadata [iceberg-go]
Posted by "aslatter (via GitHub)" <gi...@apache.org>.
aslatter commented on issue #547:
URL: https://github.com/apache/iceberg-go/issues/547#issuecomment-3688418127
(I did test with my original, more complex test-program that actually queries s3-tables, and that seems to be fixed)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] "error encountered during schema visitor: arrow/array: index out of range" while reading S3 table metadata [iceberg-go]
Posted by "aslatter (via GitHub)" <gi...@apache.org>.
aslatter commented on issue #547:
URL: https://github.com/apache/iceberg-go/issues/547#issuecomment-3688416265
It looks good to me. Do you know what the arrow-go fix was?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] "error encountered during schema visitor: arrow/array: index out of range" while reading S3 table metadata [iceberg-go]
Posted by "aslatter (via GitHub)" <gi...@apache.org>.
aslatter closed issue #547: "error encountered during schema visitor: arrow/array: index out of range" while reading S3 table metadata
URL: https://github.com/apache/iceberg-go/issues/547
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] "error encountered during schema visitor: arrow/array: index out of range" while reading S3 table metadata [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #547:
URL: https://github.com/apache/iceberg-go/issues/547#issuecomment-3687397105
@aslatter I believe this has been fixed in the latest version of arrow-go, can you try it again using main? (which is using arrow-go v18.5.0)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org