You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by Zehua Zou <ze...@gmail.com> on 2026/02/27 03:47:23 UTC
RE: Re: Flaky test in github PR test
Hi Aaron and Gang Wu,
I have opened a PR to fix this issue by lazily initializing data and reusing temporary arrays. Please take a look.
Issue link: https://github.com/apache/parquet-java/issues/3404
PR link: https://github.com/apache/parquet-java/pull/3405
On 2026/01/20 08:37:20 Gang Wu wrote:
> Hi Aaron,
>
> Sorry for the late reply!
>
> Below is the table to roughly estimate memory consumption with different
> bit widths:
>
> | bitWidth | arrays | per-array length | per-array bytes | total bytes |
> total (binary)|
> |---------:|-------:|-----------------:|----------------:|---------------:|--------------:|
> | 32 | 16 | 268,435,520 | 1,073,742,080 | 17,179,873,280 |
> 16.00 GiB |
> | 31 | 8 | 268,435,520 | 1,073,742,080 | 8,589,936,640 |
> 8.00 GiB |
> | 30 | 4 | 268,435,520 | 1,073,742,080 | 4,294,968,320 |
> 4.00 GiB |
> | 29 | 2 | 268,435,520 | 1,073,742,080 | 2,147,484,160 |
> 2.00 GiB |
> | 28 | 1 | 268,435,520 | 1,073,742,080 | 1,073,742,080 |
> 1.00 GiB |
> | 27 | 1 | 134,217,792 | 536,871,168 | 536,871,168 |
> 512.00 MiB |
> | 26 | 1 | 67,108,928 | 268,435,712 | 268,435,712 |
> 256.00 MiB |
> | 25 | 1 | 33,554,496 | 134,217,984 | 134,217,984 |
> 128.00 MiB |
> | 24 | 1 | 16,777,280 | 67,108,480 | 67,108,480 |
> 64.00 MiB |
> | 20 | 1 | 1,048,640 | 4,194,560 | 4,194,560 |
> 4.00 MiB |
> | 10 | 1 | 1,088 | 4,352 | 4,352 |
> 4.25 KiB |
> | 1 | 1 | 64 | 256 | 256 |
> 0.25 KiB |
>
> I think your proposed change has a slightly different pattern of
> generated numbers.
> If we want to keep the original pattern, a quick fix might be lazily
> generating arrays
> to reduce the peak memory size. WDYT?
>
> Best,
> Gang
>
>
> On Wed, Jan 7, 2026 at 6:23 AM Aaron Niskode-Dossett via dev <
> dev@parquet.apache.org> wrote:
>
> > The
> > test
> > TestByteBitPacking512VectorLE.unpackValuesUsingVectorBitWidth(TestByteBitPacking512VectorLE
> > is flaky in the Parquet github PR testing environment [1].
> >
> > I gave the error to Codex (the OpenAI coding agent) and asked it to fix the
> > test. However, since I don't have enough confidence in my own
> > understanding of the problem or the fix, I have not opened a PR. The fix
> > can be found on my fork here
> > <
> > https://github.com/dossett/parquet-java/commit/7635c8599524aadee1164fc2168801c51390b118
> > >
> > .
> >
> > The codex summary of the problem and the fix is this:
> >
> > We addressed CI OOMs in TestByteBitPacking512VectorLE
> > (parquet-encoding-vector) by bounding the test input size while keeping the
> > same correctness coverage. The original getRangeData could allocate arrays
> > on the order of hundreds of millions of ints per bit width, which can
> > consume tens of GB of heap and fail in constrained CI environments.
> >
> > The updated test generates a single bounded dataset (min 64, max 2^20
> > values) and spans the full legal value range for each bit width (including
> > the full signed int range for 32‑bit). The vector and scalar pack/unpack
> > paths are still compared for equality across bit widths, but without the
> > unbounded memory stress that was causing flakiness.
> >
> > I would appreciate any feedback on that or alternatively other ways to
> > address the flaky test, I found it very frustrating recently when I was
> > opening several PRs.
> >
> > Cheers, Aaron
> >
> > [1] Example failure:
> >
> > https://github.com/apache/parquet-java/actions/runs/20671204311/job/59352228516?pr=3385
> >
> > --
> > Aaron Niskode-Dossett, Data Engineering -- Etsy
> >
>
Re: Re: Flaky test in github PR test
Posted by Aaron Niskode-Dossett via dev <de...@parquet.apache.org>.
Oh great, thank you for picking this up!
On Thu, Feb 26, 2026 at 9:47 PM Zehua Zou <ze...@gmail.com> wrote:
> Hi Aaron and Gang Wu,
>
> I have opened a PR to fix this issue by lazily initializing data and
> reusing temporary arrays. Please take a look.
>
> Issue link: https://github.com/apache/parquet-java/issues/3404
>
> PR link: https://github.com/apache/parquet-java/pull/3405
>
>
> On 2026/01/20 08:37:20 Gang Wu wrote:
> > Hi Aaron,
> >
> > Sorry for the late reply!
> >
> > Below is the table to roughly estimate memory consumption with different
> > bit widths:
> >
> > | bitWidth | arrays | per-array length | per-array bytes | total bytes
> |
> > total (binary)|
> >
> |---------:|-------:|-----------------:|----------------:|---------------:|--------------:|
> > | 32 | 16 | 268,435,520 | 1,073,742,080 |
> 17,179,873,280 |
> > 16.00 GiB |
> > | 31 | 8 | 268,435,520 | 1,073,742,080 |
> 8,589,936,640 |
> > 8.00 GiB |
> > | 30 | 4 | 268,435,520 | 1,073,742,080 |
> 4,294,968,320 |
> > 4.00 GiB |
> > | 29 | 2 | 268,435,520 | 1,073,742,080 |
> 2,147,484,160 |
> > 2.00 GiB |
> > | 28 | 1 | 268,435,520 | 1,073,742,080 |
> 1,073,742,080 |
> > 1.00 GiB |
> > | 27 | 1 | 134,217,792 | 536,871,168 |
> 536,871,168 |
> > 512.00 MiB |
> > | 26 | 1 | 67,108,928 | 268,435,712 |
> 268,435,712 |
> > 256.00 MiB |
> > | 25 | 1 | 33,554,496 | 134,217,984 |
> 134,217,984 |
> > 128.00 MiB |
> > | 24 | 1 | 16,777,280 | 67,108,480 |
> 67,108,480 |
> > 64.00 MiB |
> > | 20 | 1 | 1,048,640 | 4,194,560 |
> 4,194,560 |
> > 4.00 MiB |
> > | 10 | 1 | 1,088 | 4,352 |
> 4,352 |
> > 4.25 KiB |
> > | 1 | 1 | 64 | 256 |
> 256 |
> > 0.25 KiB |
> >
> > I think your proposed change has a slightly different pattern of
> > generated numbers.
> > If we want to keep the original pattern, a quick fix might be lazily
> > generating arrays
> > to reduce the peak memory size. WDYT?
> >
> > Best,
> > Gang
> >
> >
> > On Wed, Jan 7, 2026 at 6:23 AM Aaron Niskode-Dossett via dev <
> > dev@parquet.apache.org> wrote:
> >
> > > The
> > > test
> > >
> TestByteBitPacking512VectorLE.unpackValuesUsingVectorBitWidth(TestByteBitPacking512VectorLE
> > > is flaky in the Parquet github PR testing environment [1].
> > >
> > > I gave the error to Codex (the OpenAI coding agent) and asked it to
> fix the
> > > test. However, since I don't have enough confidence in my own
> > > understanding of the problem or the fix, I have not opened a PR. The
> fix
> > > can be found on my fork here
> > > <
> > >
> https://github.com/dossett/parquet-java/commit/7635c8599524aadee1164fc2168801c51390b118
> > > >
> > > .
> > >
> > > The codex summary of the problem and the fix is this:
> > >
> > > We addressed CI OOMs in TestByteBitPacking512VectorLE
> > > (parquet-encoding-vector) by bounding the test input size while
> keeping the
> > > same correctness coverage. The original getRangeData could allocate
> arrays
> > > on the order of hundreds of millions of ints per bit width, which can
> > > consume tens of GB of heap and fail in constrained CI environments.
> > >
> > > The updated test generates a single bounded dataset (min 64, max 2^20
> > > values) and spans the full legal value range for each bit width
> (including
> > > the full signed int range for 32‑bit). The vector and scalar
> pack/unpack
> > > paths are still compared for equality across bit widths, but without
> the
> > > unbounded memory stress that was causing flakiness.
> > >
> > > I would appreciate any feedback on that or alternatively other ways to
> > > address the flaky test, I found it very frustrating recently when I was
> > > opening several PRs.
> > >
> > > Cheers, Aaron
> > >
> > > [1] Example failure:
> > >
> > >
> https://github.com/apache/parquet-java/actions/runs/20671204311/job/59352228516?pr=3385
> > >
> > > --
> > > Aaron Niskode-Dossett, Data Engineering -- Etsy
> > >
> >
--
Aaron Niskode-Dossett, Data Engineering -- Etsy