You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "daniel-adam-tfs (via GitHub)" <gi...@apache.org> on 2025/08/18 08:49:05 UTC
[PR] Feature/improve rle [arrow-go]
daniel-adam-tfs opened a new pull request, #477:
URL: https://github.com/apache/arrow-go/pull/477
### Rationale for this change
Performance improvement of reading data from RLE-dictionary encoded parquet columns.
### What changes are included in this PR?
The DictionaryConverter interface was changed to DictionaryConverter[T] interface, so the methods could take []T as a parameter instead of interface{}. This required changed to several types implementing the interface.
The template code generation for RleDecoder was removed and the methods for the parquet types are now handled by the generic TypedRleDecoder type.
Additionally, some utils.Min/utils.Max calls were replace with calls to min/max intrinsics
### Are these changes tested?
Yes, tests verifying the RLE dictionary already existed and I've added BenchmarkReadInt32Column to measure the performance improvement.
### Are there any user-facing changes?
Some exported types functions were changed:
DictionaryConverter -> DictionaryConverter[T]
RleDecoder -> split into RleDecoder and TypeRleDecoder[T]
func (b *BitReader) GetAligned(nbytes int, v interface{}) bool -> split into getAlignedUint8, getAlignedUint16, getAlignedUint32, getAlignedUint64
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement RLE dictionary decoder using generics [arrow-go]
Posted by "daniel-adam-tfs (via GitHub)" <gi...@apache.org>.
daniel-adam-tfs commented on code in PR #477:
URL: https://github.com/apache/arrow-go/pull/477#discussion_r2297395458
##########
parquet/file/file_reader_test.go:
##########
@@ -927,3 +928,61 @@ func TestListColumns(t *testing.T) {
}
}
}
+
+func BenchmarkReadInt32Column(b *testing.B) {
+ b.Skip("rle-dict-int32-snappy.parquet not available")
+
+ dir := os.Getenv("PARQUET_TEST_DATA")
+ if dir == "" {
+ dir = "../../parquet-testing/data"
+ b.Log("PARQUET_TEST_DATA not set, using ../../parquet-testing/data")
+ }
+ require.DirExists(b, dir)
+
+ filePath := filepath.Join(dir, "rle-dict-int32-snappy.parquet")
+ reader, err := file.OpenParquetFile(filePath, false)
+ if err != nil {
+ b.Fatalf("Expected no error while opening parquet file %q, got %v", filePath, err)
+ }
+ defer reader.Close()
+
+ int32ColIdx := reader.MetaData().Schema.Root().FieldIndexByName("int32")
+ if int32ColIdx < 0 {
+ b.Fatalf("Expected to find int32 column in schema, got index %d", int32ColIdx)
+ }
+
+ numValues := reader.NumRows()
+ values := make([]int32, numValues)
+ b.StopTimer()
Review Comment:
`ResetTimer` doesn't restart the timer, so what's the reason for the swap? The only difference I can think of is that the clock counters will be updated with data from the interval. Ie.
The current way:
1) timer is stopped
2) timer counters are zero-ed
The swapped way:
1) timer counters are zero-ed (but the timer keeps running)
2) timer is stopped (and the counters are updated with the data from interval between 1) and 2) )
Am I missing something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement RLE dictionary decoder using generics [arrow-go]
Posted by "daniel-adam-tfs (via GitHub)" <gi...@apache.org>.
daniel-adam-tfs commented on code in PR #477:
URL: https://github.com/apache/arrow-go/pull/477#discussion_r2281737562
##########
parquet/file/file_reader_test.go:
##########
@@ -927,3 +928,61 @@ func TestListColumns(t *testing.T) {
}
}
}
+
+func BenchmarkReadInt32Column(b *testing.B) {
+ b.Skip("rle-dict-int32-snappy.parquet not available")
Review Comment:
@zeroshade I've generated this file with a single int32 column, 1 000 000 rows overall, 10 row groups and 1 00 unique values, v2 format and snappy compression. It's 113KB in size, if there's no other file you'd prefer to use for testing then we can add it to the data repo and remove the `b.Skip` call.
##########
parquet/file/file_reader_test.go:
##########
@@ -927,3 +928,61 @@ func TestListColumns(t *testing.T) {
}
}
}
+
+func BenchmarkReadInt32Column(b *testing.B) {
+ b.Skip("rle-dict-int32-snappy.parquet not available")
Review Comment:
@zeroshade I've generated this file with a single int32 column, 1 000 000 rows overall, 10 row groups and 100 unique values, v2 format and snappy compression. It's 113KB in size, if there's no other file you'd prefer to use for testing then we can add it to the data repo and remove the `b.Skip` call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement RLE dictionary decoder using generics [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #477:
URL: https://github.com/apache/arrow-go/pull/477#discussion_r2282799560
##########
parquet/internal/utils/bit_reader.go:
##########
@@ -152,24 +169,67 @@ func (b *BitReader) GetAligned(nbytes int, v interface{}) bool {
if n != nbytes {
return false
}
+
// zero pad the bytes
- memory.Set(b.raw[n:typBytes], 0)
-
- switch v := v.(type) {
Review Comment:
i think making it generic is the good call, using `unsafe.Sizeof` is the right way to do this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement RLE dictionary decoder using generics [arrow-go]
Posted by "daniel-adam-tfs (via GitHub)" <gi...@apache.org>.
daniel-adam-tfs commented on code in PR #477:
URL: https://github.com/apache/arrow-go/pull/477#discussion_r2285167839
##########
parquet/file/file_reader_test.go:
##########
@@ -927,3 +928,61 @@ func TestListColumns(t *testing.T) {
}
}
}
+
+func BenchmarkReadInt32Column(b *testing.B) {
+ b.Skip("rle-dict-int32-snappy.parquet not available")
Review Comment:
Hmm I've used a python to generate it, it's simple so it shouldn't be to hard to rewrite the code to Go and generate a file with the same parameters.
I haven't used the writers available in this lib, is there a test or something that writes a RLE-encoded column so I check how to do it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement RLE dictionary decoder using generics [arrow-go]
Posted by "daniel-adam-tfs (via GitHub)" <gi...@apache.org>.
daniel-adam-tfs commented on PR #477:
URL: https://github.com/apache/arrow-go/pull/477#issuecomment-3229059976
> LGTM are we all set here @daniel-adam-tfs ?
Gimme a minute, I was stuck in meetings all day today, but I can finish this now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement RLE dictionary decoder using generics [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #477:
URL: https://github.com/apache/arrow-go/pull/477#discussion_r2282768623
##########
parquet/file/file_reader_test.go:
##########
@@ -927,3 +928,61 @@ func TestListColumns(t *testing.T) {
}
}
}
+
+func BenchmarkReadInt32Column(b *testing.B) {
+ b.Skip("rle-dict-int32-snappy.parquet not available")
Review Comment:
Would it be difficult to generate the file in `t.TempDir()` during the test, do the benchmark, then clean it up? That would be my preference, otherwise we can add a `testdata` folder and add it there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement RLE dictionary decoder using generics [arrow-go]
Posted by "daniel-adam-tfs (via GitHub)" <gi...@apache.org>.
daniel-adam-tfs commented on code in PR #477:
URL: https://github.com/apache/arrow-go/pull/477#discussion_r2284898520
##########
parquet/file/file_reader_test.go:
##########
@@ -927,3 +928,61 @@ func TestListColumns(t *testing.T) {
}
}
}
+
+func BenchmarkReadInt32Column(b *testing.B) {
+ b.Skip("rle-dict-int32-snappy.parquet not available")
+
+ dir := os.Getenv("PARQUET_TEST_DATA")
+ if dir == "" {
+ dir = "../../parquet-testing/data"
+ b.Log("PARQUET_TEST_DATA not set, using ../../parquet-testing/data")
+ }
+ require.DirExists(b, dir)
+
+ filePath := filepath.Join(dir, "rle-dict-int32-snappy.parquet")
+ reader, err := file.OpenParquetFile(filePath, false)
+ if err != nil {
+ b.Fatalf("Expected no error while opening parquet file %q, got %v", filePath, err)
+ }
+ defer reader.Close()
+
+ int32ColIdx := reader.MetaData().Schema.Root().FieldIndexByName("int32")
+ if int32ColIdx < 0 {
+ b.Fatalf("Expected to find int32 column in schema, got index %d", int32ColIdx)
+ }
+
+ numValues := reader.NumRows()
+ values := make([]int32, numValues)
+ b.StopTimer()
Review Comment:
I definitely shouldn't have 3 calls to `StopTimer` in this benchmark, I think I forgot to remove the one on 961.
The timer is in `On` state as this benchmark runs and `ResetTimer` doesn't stop it. I want to measure just the `ReadBatch` call, so I stop it to avoid including execution of lines 958-971 into the benchmark timing. Or do you think we should include it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement RLE dictionary decoder using generics [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #477:
URL: https://github.com/apache/arrow-go/pull/477#discussion_r2298581925
##########
parquet/file/file_reader_test.go:
##########
@@ -927,3 +928,61 @@ func TestListColumns(t *testing.T) {
}
}
}
+
+func BenchmarkReadInt32Column(b *testing.B) {
+ b.Skip("rle-dict-int32-snappy.parquet not available")
+
+ dir := os.Getenv("PARQUET_TEST_DATA")
+ if dir == "" {
+ dir = "../../parquet-testing/data"
+ b.Log("PARQUET_TEST_DATA not set, using ../../parquet-testing/data")
+ }
+ require.DirExists(b, dir)
+
+ filePath := filepath.Join(dir, "rle-dict-int32-snappy.parquet")
+ reader, err := file.OpenParquetFile(filePath, false)
+ if err != nil {
+ b.Fatalf("Expected no error while opening parquet file %q, got %v", filePath, err)
+ }
+ defer reader.Close()
+
+ int32ColIdx := reader.MetaData().Schema.Root().FieldIndexByName("int32")
+ if int32ColIdx < 0 {
+ b.Fatalf("Expected to find int32 column in schema, got index %d", int32ColIdx)
+ }
+
+ numValues := reader.NumRows()
+ values := make([]int32, numValues)
+ b.StopTimer()
Review Comment:
Ah, that's my mistake. The current version looks good!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement RLE dictionary decoder using generics [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #477:
URL: https://github.com/apache/arrow-go/pull/477#discussion_r2282789332
##########
parquet/internal/encoding/typed_encoder.go:
##########
@@ -307,43 +307,36 @@ func (dc *dictConverter[T]) IsValid(idxes ...utils.IndexType) bool {
return min >= 0 && int(min) < len(dc.dict) && int(max) >= 0 && int(max) < len(dc.dict)
}
-func (dc *dictConverter[T]) Fill(out any, val utils.IndexType) error {
- o := out.([]T)
+func (dc *dictConverter[T]) IsValidSingle(idx utils.IndexType) bool {
+ dc.ensure(idx)
+ return int(idx) >= 0 && int(idx) < len(dc.dict)
+}
+
+func (dc *dictConverter[T]) Fill(o []T, val utils.IndexType) error {
if err := dc.ensure(val); err != nil {
return err
}
-
o[0] = dc.dict[val]
for i := 1; i < len(o); i *= 2 {
copy(o[i:], o[:i])
}
return nil
}
-func (dc *dictConverter[T]) FillZero(out any) {
- o := out.([]T)
+func (dc *dictConverter[T]) FillZero(o []T) {
o[0] = dc.zeroVal
for i := 1; i < len(o); i *= 2 {
copy(o[i:], o[:i])
}
}
-func (dc *dictConverter[T]) Copy(out any, vals []utils.IndexType) error {
- o := out.([]T)
+func (dc *dictConverter[T]) Copy(o []T, vals []utils.IndexType) error {
for idx, val := range vals {
o[idx] = dc.dict[val]
}
return nil
}
-type Int32DictConverter = dictConverter[int32]
-type Int64DictConverter = dictConverter[int64]
-type Float32DictConverter = dictConverter[float32]
-type Float64DictConverter = dictConverter[float64]
-type Int96DictConverter = dictConverter[parquet.Int96]
-type ByteArrayDictConverter = dictConverter[parquet.ByteArray]
-type FixedLenByteArrayDictConverter = dictConverter[parquet.FixedLenByteArray]
Review Comment:
These are publicly exported, why are we removing these aliases?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement RLE dictionary decoder using generics [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #477:
URL: https://github.com/apache/arrow-go/pull/477#issuecomment-3228848999
LGTM are we all set here @daniel-adam-tfs ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement RLE dictionary decoder using generics [arrow-go]
Posted by "daniel-adam-tfs (via GitHub)" <gi...@apache.org>.
daniel-adam-tfs commented on PR #477:
URL: https://github.com/apache/arrow-go/pull/477#issuecomment-3229182319
If the tests fit, you must acquit (or LGTM)!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement RLE dictionary decoder using generics [arrow-go]
Posted by "daniel-adam-tfs (via GitHub)" <gi...@apache.org>.
daniel-adam-tfs commented on code in PR #477:
URL: https://github.com/apache/arrow-go/pull/477#discussion_r2297894865
##########
parquet/internal/utils/bit_reader.go:
##########
@@ -152,24 +169,67 @@ func (b *BitReader) GetAligned(nbytes int, v interface{}) bool {
if n != nbytes {
return false
}
+
// zero pad the bytes
- memory.Set(b.raw[n:typBytes], 0)
-
- switch v := v.(type) {
Review Comment:
Annoying, I benchmarked it now and the non-generic version is about 2% faster according to the benchmark. In this case I'm alright with a little bit of code duplication for 2% better performance, but what's your call?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement RLE dictionary decoder using generics [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #477:
URL: https://github.com/apache/arrow-go/pull/477#discussion_r2282772306
##########
parquet/file/file_reader_test.go:
##########
@@ -927,3 +928,61 @@ func TestListColumns(t *testing.T) {
}
}
}
+
+func BenchmarkReadInt32Column(b *testing.B) {
+ b.Skip("rle-dict-int32-snappy.parquet not available")
+
+ dir := os.Getenv("PARQUET_TEST_DATA")
+ if dir == "" {
+ dir = "../../parquet-testing/data"
+ b.Log("PARQUET_TEST_DATA not set, using ../../parquet-testing/data")
+ }
+ require.DirExists(b, dir)
+
+ filePath := filepath.Join(dir, "rle-dict-int32-snappy.parquet")
+ reader, err := file.OpenParquetFile(filePath, false)
+ if err != nil {
+ b.Fatalf("Expected no error while opening parquet file %q, got %v", filePath, err)
+ }
+ defer reader.Close()
+
+ int32ColIdx := reader.MetaData().Schema.Root().FieldIndexByName("int32")
+ if int32ColIdx < 0 {
+ b.Fatalf("Expected to find int32 column in schema, got index %d", int32ColIdx)
+ }
+
+ numValues := reader.NumRows()
+ values := make([]int32, numValues)
+ b.StopTimer()
Review Comment:
you don't need `StopTimer` before you call `ResetTimer`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement RLE dictionary decoder using generics [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #477:
URL: https://github.com/apache/arrow-go/pull/477#discussion_r2282799560
##########
parquet/internal/utils/bit_reader.go:
##########
@@ -152,24 +169,67 @@ func (b *BitReader) GetAligned(nbytes int, v interface{}) bool {
if n != nbytes {
return false
}
+
// zero pad the bytes
- memory.Set(b.raw[n:typBytes], 0)
-
- switch v := v.(type) {
Review Comment:
i think making it generic is the good call
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement RLE dictionary decoder using generics [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #477:
URL: https://github.com/apache/arrow-go/pull/477#discussion_r2286311805
##########
parquet/file/file_reader_test.go:
##########
@@ -927,3 +928,61 @@ func TestListColumns(t *testing.T) {
}
}
}
+
+func BenchmarkReadInt32Column(b *testing.B) {
+ b.Skip("rle-dict-int32-snappy.parquet not available")
Review Comment:
The default logic currently will enable dictionary encoding, but if you want to be explicit you can use the writer property `parquet.WithDictionaryFor("path.to.column", true)` or `parquet.WithDictionaryDefault(true)` to explicitly tell the writer to use dictionary encoding when writing the column.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement RLE dictionary decoder using generics [arrow-go]
Posted by "daniel-adam-tfs (via GitHub)" <gi...@apache.org>.
daniel-adam-tfs commented on code in PR #477:
URL: https://github.com/apache/arrow-go/pull/477#discussion_r2281753735
##########
parquet/internal/utils/bit_reader.go:
##########
@@ -152,24 +169,67 @@ func (b *BitReader) GetAligned(nbytes int, v interface{}) bool {
if n != nbytes {
return false
}
+
// zero pad the bytes
- memory.Set(b.raw[n:typBytes], 0)
-
- switch v := v.(type) {
Review Comment:
We could make this generic:
func GetAligned[T ...](b *BitReader, v *T)
I didn't because I wanted to avoid casting to interface{} so I can do a type switch, but in my cpu profiles that switch was pretty fast. The slowest thing in the original code was the call to reflect package, but in the generic version unsafe.Sizeof(*v) can be called, which is much quicker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement RLE dictionary decoder using generics [arrow-go]
Posted by "daniel-adam-tfs (via GitHub)" <gi...@apache.org>.
daniel-adam-tfs commented on code in PR #477:
URL: https://github.com/apache/arrow-go/pull/477#discussion_r2281737562
##########
parquet/file/file_reader_test.go:
##########
@@ -927,3 +928,61 @@ func TestListColumns(t *testing.T) {
}
}
}
+
+func BenchmarkReadInt32Column(b *testing.B) {
+ b.Skip("rle-dict-int32-snappy.parquet not available")
Review Comment:
@zeroshade I've generated this file with a single int32 column, 1 000 000 rows overall, 10 row groups and 1 000 unique values, v2 format and snappy compression. It's 113KB in size, if there's no other file you'd prefer to use for testing then we can add it to the data repo and remove the `b.Skip` call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement RLE dictionary decoder using generics [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #477:
URL: https://github.com/apache/arrow-go/pull/477#discussion_r2282774597
##########
parquet/file/file_reader_test.go:
##########
@@ -927,3 +928,61 @@ func TestListColumns(t *testing.T) {
}
}
}
+
+func BenchmarkReadInt32Column(b *testing.B) {
+ b.Skip("rle-dict-int32-snappy.parquet not available")
+
+ dir := os.Getenv("PARQUET_TEST_DATA")
+ if dir == "" {
+ dir = "../../parquet-testing/data"
+ b.Log("PARQUET_TEST_DATA not set, using ../../parquet-testing/data")
+ }
+ require.DirExists(b, dir)
+
+ filePath := filepath.Join(dir, "rle-dict-int32-snappy.parquet")
+ reader, err := file.OpenParquetFile(filePath, false)
+ if err != nil {
+ b.Fatalf("Expected no error while opening parquet file %q, got %v", filePath, err)
+ }
+ defer reader.Close()
+
+ int32ColIdx := reader.MetaData().Schema.Root().FieldIndexByName("int32")
+ if int32ColIdx < 0 {
+ b.Fatalf("Expected to find int32 column in schema, got index %d", int32ColIdx)
+ }
+
+ numValues := reader.NumRows()
+ values := make([]int32, numValues)
+ b.StopTimer()
+ b.ResetTimer()
+ for i := 0; i < b.N; i++ {
+ startIndex := 0
+ for rg := 0; rg < reader.NumRowGroups(); rg++ {
+ b.StopTimer()
Review Comment:
Swap this with the line above it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement RLE dictionary decoder using generics [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade merged PR #477:
URL: https://github.com/apache/arrow-go/pull/477
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement RLE dictionary decoder using generics [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #477:
URL: https://github.com/apache/arrow-go/pull/477#discussion_r2286315828
##########
parquet/file/file_reader_test.go:
##########
@@ -927,3 +928,61 @@ func TestListColumns(t *testing.T) {
}
}
}
+
+func BenchmarkReadInt32Column(b *testing.B) {
+ b.Skip("rle-dict-int32-snappy.parquet not available")
+
+ dir := os.Getenv("PARQUET_TEST_DATA")
+ if dir == "" {
+ dir = "../../parquet-testing/data"
+ b.Log("PARQUET_TEST_DATA not set, using ../../parquet-testing/data")
+ }
+ require.DirExists(b, dir)
+
+ filePath := filepath.Join(dir, "rle-dict-int32-snappy.parquet")
+ reader, err := file.OpenParquetFile(filePath, false)
+ if err != nil {
+ b.Fatalf("Expected no error while opening parquet file %q, got %v", filePath, err)
+ }
+ defer reader.Close()
+
+ int32ColIdx := reader.MetaData().Schema.Root().FieldIndexByName("int32")
+ if int32ColIdx < 0 {
+ b.Fatalf("Expected to find int32 column in schema, got index %d", int32ColIdx)
+ }
+
+ numValues := reader.NumRows()
+ values := make([]int32, numValues)
+ b.StopTimer()
Review Comment:
you would want to swap 956 and 957 (call `StopTimer` after `ResetTimer`). Then you can remove the one at 961 so you just start, do readbatch, and then stop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement RLE dictionary decoder using generics [arrow-go]
Posted by "daniel-adam-tfs (via GitHub)" <gi...@apache.org>.
daniel-adam-tfs commented on PR #477:
URL: https://github.com/apache/arrow-go/pull/477#issuecomment-3195808226
I've attached benchmarks for the commits from my machine:
```
% benchstat BenchmarkInt32Column.18-4-base.txt BenchmarkInt32Column.18-4-commit7.txt
goos: darwin
goarch: amd64
pkg: github.com/apache/arrow-go/v18/parquet/file
cpu: Intel(R) Core(TM) i5-1038NG7 CPU @ 2.00GHz
│ BenchmarkInt32Column.18-4-base.txt │ BenchmarkInt32Column.18-4-commit7.txt │
│ sec/op │ sec/op vs base │
ReadInt32Column-8 6.024m ± 2% 3.329m ± 3% -44.74% (p=0.000 n=8)
│ BenchmarkInt32Column.18-4-base.txt │ BenchmarkInt32Column.18-4-commit7.txt │
│ B/op │ B/op vs base │
ReadInt32Column-8 1300.8Ki ± 0% 292.0Ki ± 0% -77.55% (p=0.000 n=8)
│ BenchmarkInt32Column.18-4-base.txt │ BenchmarkInt32Column.18-4-commit7.txt │
│ allocs/op │ allocs/op vs base │
ReadInt32Column-8 71046.0 ± 0% 562.0 ± 0% -99.21% (p=0.000 n=8)
```
[BenchmarkInt32Column.18-4-base.txt](https://github.com/user-attachments/files/21831684/BenchmarkInt32Column.18-4-base.txt)
[BenchmarkInt32Column.18-4-commit3.txt](https://github.com/user-attachments/files/21831685/BenchmarkInt32Column.18-4-commit3.txt)
[BenchmarkInt32Column.18-4-commit4.txt](https://github.com/user-attachments/files/21831686/BenchmarkInt32Column.18-4-commit4.txt)
[BenchmarkInt32Column.18-4-commit5.txt](https://github.com/user-attachments/files/21831687/BenchmarkInt32Column.18-4-commit5.txt)
[BenchmarkInt32Column.18-4-commit6.txt](https://github.com/user-attachments/files/21831688/BenchmarkInt32Column.18-4-commit6.txt)
[BenchmarkInt32Column.18-4-commit7.txt](https://github.com/user-attachments/files/21831689/BenchmarkInt32Column.18-4-commit7.txt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement RLE dictionary decoder using generics [arrow-go]
Posted by "daniel-adam-tfs (via GitHub)" <gi...@apache.org>.
daniel-adam-tfs commented on code in PR #477:
URL: https://github.com/apache/arrow-go/pull/477#discussion_r2304859407
##########
parquet/internal/utils/bit_reader.go:
##########
@@ -152,24 +169,67 @@ func (b *BitReader) GetAligned(nbytes int, v interface{}) bool {
if n != nbytes {
return false
}
+
// zero pad the bytes
- memory.Set(b.raw[n:typBytes], 0)
-
- switch v := v.(type) {
Review Comment:
Reverted back to the non-generic code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement RLE dictionary decoder using generics [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #477:
URL: https://github.com/apache/arrow-go/pull/477#discussion_r2282789332
##########
parquet/internal/encoding/typed_encoder.go:
##########
@@ -307,43 +307,36 @@ func (dc *dictConverter[T]) IsValid(idxes ...utils.IndexType) bool {
return min >= 0 && int(min) < len(dc.dict) && int(max) >= 0 && int(max) < len(dc.dict)
}
-func (dc *dictConverter[T]) Fill(out any, val utils.IndexType) error {
- o := out.([]T)
+func (dc *dictConverter[T]) IsValidSingle(idx utils.IndexType) bool {
+ dc.ensure(idx)
+ return int(idx) >= 0 && int(idx) < len(dc.dict)
+}
+
+func (dc *dictConverter[T]) Fill(o []T, val utils.IndexType) error {
if err := dc.ensure(val); err != nil {
return err
}
-
o[0] = dc.dict[val]
for i := 1; i < len(o); i *= 2 {
copy(o[i:], o[:i])
}
return nil
}
-func (dc *dictConverter[T]) FillZero(out any) {
- o := out.([]T)
+func (dc *dictConverter[T]) FillZero(o []T) {
o[0] = dc.zeroVal
for i := 1; i < len(o); i *= 2 {
copy(o[i:], o[:i])
}
}
-func (dc *dictConverter[T]) Copy(out any, vals []utils.IndexType) error {
- o := out.([]T)
+func (dc *dictConverter[T]) Copy(o []T, vals []utils.IndexType) error {
for idx, val := range vals {
o[idx] = dc.dict[val]
}
return nil
}
-type Int32DictConverter = dictConverter[int32]
-type Int64DictConverter = dictConverter[int64]
-type Float32DictConverter = dictConverter[float32]
-type Float64DictConverter = dictConverter[float64]
-type Int96DictConverter = dictConverter[parquet.Int96]
-type ByteArrayDictConverter = dictConverter[parquet.ByteArray]
-type FixedLenByteArrayDictConverter = dictConverter[parquet.FixedLenByteArray]
Review Comment:
These are publicly exported, why are we removing these aliases?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement RLE dictionary decoder using generics [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #477:
URL: https://github.com/apache/arrow-go/pull/477#discussion_r2298575858
##########
parquet/internal/utils/bit_reader.go:
##########
@@ -152,24 +169,67 @@ func (b *BitReader) GetAligned(nbytes int, v interface{}) bool {
if n != nbytes {
return false
}
+
// zero pad the bytes
- memory.Set(b.raw[n:typBytes], 0)
-
- switch v := v.(type) {
Review Comment:
Agreed, a little duplication for the better performance is the way to go here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org