You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "fallintoplace (via GitHub)" <gi...@apache.org> on 2026/10/21 21:28:22 UTC

[PR] perf(parquet): decode narrow packed levels directly [arrow-go]

fallintoplace opened a new pull request, #1404:
URL: https://github.com/apache/arrow-go/pull/1404

   ## What
   Decode 1-bit and 2-bit Parquet levels directly into `int16` output.
   
   ## Why
   The current path unpacks values into `uint32` scratch space before converting them to `int16`.
   
   ## Implementation
   - Expand narrow packed bytes directly into the output.
   - Keep the existing path for wider levels and trailing values.
   - Add parity coverage for slices, truncated data, and sequential reads.
   
   ## Benchmark
   Paired benchmark of the copied previous `GetBatchLevels` implementation and the actual updated `BitReader.GetBatchLevels` method. Go 1.26.3, Apple M1 Pro (`darwin/arm64`), `GOMAXPROCS=1`; 10 runs of 150 ms per case.
   
   | Metric | Before | After | Change |
   | --- | ---: | ---: | ---: |
   | 1-bit, mostly defined, 1,024 levels | 742.9 ns/op | 324.6 ns/op | -56.32% |
   | 1-bit, mostly defined, 65,536 levels | 43.84 µs/op | 17.50 µs/op | -60.09% |
   | 1-bit, alternating, 1,024 levels | 740.1 ns/op | 330.5 ns/op | -55.34% |
   | 1-bit, alternating, 65,536 levels | 44.45 µs/op | 17.43 µs/op | -60.80% |
   | 1-bit, varied, 1,024 levels | 744.3 ns/op | 325.4 ns/op | -56.28% |
   | 1-bit, varied, 65,536 levels | 43.98 µs/op | 17.75 µs/op | -59.64% |
   | 2-bit, mostly defined, 1,024 levels | 753.7 ns/op | 572.3 ns/op | -24.07% |
   | 2-bit, mostly defined, 65,536 levels | 45.10 µs/op | 30.10 µs/op | -33.25% |
   | 2-bit, alternating, 1,024 levels | 753.9 ns/op | 521.0 ns/op | -30.90% |
   | 2-bit, alternating, 65,536 levels | 45.32 µs/op | 29.97 µs/op | -33.85% |
   | 2-bit, varied, 1,024 levels | 756.7 ns/op | 525.6 ns/op | -30.54% |
   | 2-bit, varied, 65,536 levels | 44.88 µs/op | 31.11 µs/op | -30.69% |
   | Geometric mean, 12 cases | 5.777 µs/op | 3.114 µs/op | -46.10% |
   
   All cases improved. Both paths use 48 B/op and 1 alloc/op. Command: `GOMAXPROCS=1 go test ./parquet/internal/utils -run '^$' -bench '^BenchmarkPackedLevelDecode$' -benchmem -benchtime=150ms -count=10 -cpu=1`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(parquet): decode narrow packed levels directly [arrow-go]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade merged PR #1404:
URL: https://github.com/apache/arrow-go/pull/1404


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(parquet): decode narrow packed levels directly [arrow-go]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #1404:
URL: https://github.com/apache/arrow-go/pull/1404#discussion_r4362291200


##########
parquet/internal/utils/bit_reader.go:
##########
@@ -619,24 +620,35 @@ func (b *BitReader) GetBatchLevels(bits uint, out []int16, maxLevel int16) (int,
 	if _, err := b.reader.Seek(b.byteoffset, io.SeekStart); err != nil {
 		return i, maxCount, err
 	}
-	for i < length {
-		unpackSize := utils.Min(buflen, length-i)
-		numUnpacked, err := unpack32(b.reader, b.unpackBuf[:unpackSize], int(bits))
-
-		for k, val := range b.unpackBuf[:numUnpacked] {
-			level := int16(val)
-			out[i+k] = level
-			if level == maxLevel {
-				maxCount++
-			}
-		}
-		i += numUnpacked
-		b.byteoffset += int64(numUnpacked * int(bits) / 8)
+	if bits == 1 || bits == 2 {

Review Comment:
   On amd64 the 2-bit case comes out slower than the existing unpack path. Pinned, 5×300 ms runs of `BenchmarkPackedLevelDecode`: width=2 alternating n=65536 30.1–30.5 µs before vs 32.5–33.9 µs after; mostly-defined n=65536 30.9–31.3 vs 32.8–33.2; n=1024 cases 4–5% slower. 1-bit is 1.2–1.4× faster. Could this be `if bits == 1` unless the 2-bit loop can be made faster on amd64 too?



##########
parquet/internal/utils/bit_reader.go:
##########
@@ -658,6 +670,74 @@ func (b *BitReader) GetBatchLevels(bits uint, out []int16, maxLevel int16) (int,
 	return i, maxCount, nil
 }
 
+// getBatchLevelsNarrow consumes complete 32-value groups of packed 1-bit or
+// 2-bit levels. It must be called with a byte-aligned reader; the caller decodes
+// any trailing values with next. Only complete groups are written on short reads.
+func (b *BitReader) getBatchLevelsNarrow(width uint, out []int16, maxLevel int16) (int, int64, error) {
+	const valuesPerGroup = 32
+	groupBytes := 4 * int(width)
+	toDecode := len(out) / valuesPerGroup * valuesPerGroup
+	packed := arrow.Uint32Traits.CastToBytes(b.unpackBuf[:])
+	decoded := 0
+	var maxCount int64
+
+	for decoded < toDecode {
+		batch := min(toDecode-decoded, buflen)
+		batchBytes := batch * int(width) / 8
+		nread, err := io.ReadFull(b.reader, packed[:batchBytes])
+		complete := nread / groupBytes * valuesPerGroup
+		completeBytes := complete * int(width) / 8
+		dst := out[decoded : decoded+complete]
+
+		if width == 1 {
+			for _, value := range packed[:completeBytes] {
+				dst[0] = int16(value & 1)
+				dst[1] = int16((value >> 1) & 1)
+				dst[2] = int16((value >> 2) & 1)
+				dst[3] = int16((value >> 3) & 1)
+				dst[4] = int16((value >> 4) & 1)
+				dst[5] = int16((value >> 5) & 1)
+				dst[6] = int16((value >> 6) & 1)
+				dst[7] = int16(value >> 7)
+				switch maxLevel {
+				case 0:
+					maxCount += int64(8 - mathbits.OnesCount8(value))
+				case 1:
+					maxCount += int64(mathbits.OnesCount8(value))
+				}
+				dst = dst[8:]
+			}
+		} else {
+			// Each 2-bit value occupies one of four pairs in the byte.

Review Comment:
   This is the loop that regresses on amd64 (see the comment above). If you keep it, please include amd64 numbers alongside the M1 ones.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org