You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Sunyue (via GitHub)" <gi...@apache.org> on 2025/12/29 10:14:03 UTC

[I] [GO] parquet: dict spaced eof exception [arrow-go]

Sunyue opened a new issue, #619:
URL: https://github.com/apache/arrow-go/issues/619

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   Hi, I am reading parquet from S3 using the parquet reader, basically by providing a reader with `ReadAt(buf []byte, off int64)` implemented with `Range` header like `bytes=<off>-<off+len(buf)-1>. When I enable `BufferedStreamEnabled`, when reading certain columns parquet reader ends in error `parquet: dict spaced eof exception`. But when I disable `BufferedStreamEnabled` reading same parquet file works fine.
   I can't see where the error is, could you kindly help on this? Thanks!
   
   ### Component(s)
   
   Parquet


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [GO] parquet: dict spaced eof exception [arrow-go]

Posted by "Sunyue (via GitHub)" <gi...@apache.org>.
Sunyue commented on issue #619:
URL: https://github.com/apache/arrow-go/issues/619#issuecomment-3701277102

   Reading from an os.File can also end in same error. Please find program to reproduce this issue below:
   
   ```go
   package main
   
   import (
   	"context"
   	"log"
   	"os"
   
   	"github.com/apache/arrow-go/v18/arrow/memory"
   	"github.com/apache/arrow-go/v18/parquet"
   	"github.com/apache/arrow-go/v18/parquet/file"
   	"github.com/apache/arrow-go/v18/parquet/pqarrow"
   )
   
   func main() {
   	pFile, err := os.Open("<path_to_file>")
   	if err != nil {
   		log.Fatal(err)
   	}
   
   	ctx := context.Background()
   	opts := &parquet.ReaderProperties{
   		BufferSize:            1024,
   		FileDecryptProps:      nil,
   		BufferedStreamEnabled: true,
   	}
   
   	base, err := file.NewParquetReader(pFile, file.WithReadProps(opts))
   	if err != nil {
   		log.Fatal(err)
   	}
   
   	pReader, err := pqarrow.NewFileReader(base, pqarrow.ArrowReadProperties{
   		Parallel:  true,
   		BatchSize: 3000,
   	}, memory.DefaultAllocator)
   	if err != nil {
   		log.Fatal(err)
   	}
   
   	numOfRowGroups := pReader.ParquetReader().NumRowGroups()
   
   	for rgIdx := range numOfRowGroups {
   		rReader, err := pReader.GetRecordReader(ctx, nil, []int{rgIdx})
   		if err != nil {
   			log.Fatal(err)
   		}
   
   		for rReader.Next() {
   			batch := rReader.RecordBatch()
   			log.Printf("got batch: %v records", batch.NumRows())
   		}
   	}
   }
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [GO] parquet: dict spaced eof exception [arrow-go]

Posted by "Sunyue (via GitHub)" <gi...@apache.org>.
Sunyue commented on issue #619:
URL: https://github.com/apache/arrow-go/issues/619#issuecomment-3803240651

   @zeroshade could you help to have a look at this issue?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [GO] parquet: dict spaced eof exception [arrow-go]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade closed issue #619: [GO] parquet: dict spaced eof exception
URL: https://github.com/apache/arrow-go/issues/619


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [GO] parquet: dict spaced eof exception [arrow-go]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #619:
URL: https://github.com/apache/arrow-go/issues/619#issuecomment-3806156418

   I've put up a fix, please take a look


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [GO] parquet: dict spaced eof exception [arrow-go]

Posted by "Sunyue (via GitHub)" <gi...@apache.org>.
Sunyue commented on issue #619:
URL: https://github.com/apache/arrow-go/issues/619#issuecomment-3701357567

   [test2.snappy.parquet.zip](https://github.com/user-attachments/files/24389797/test2.snappy.parquet.zip) You can reproduce using this file with above test program


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org