beam.io.ReadFromParquetBatched min_bundle_size

[Cindy McMullen <cm...@twitter.com>]

Hi -

It appears as though the 'min_bundle_size' parameter is ignored.  Can
someone verify?  I didn't see an issue in JIRA for this.

I'm following the docs here
<https://beam.apache.org/releases/pydoc/2.30.0/apache_beam.io.parquetio.html>
.

Example code:

mbs = 10000000 * 10 #  10 MBytes
( p
             | 'ReadParquetBatched' >>
beam.io.ReadFromParquetBatched(pq_file, min_bundle_size=mbs)
            | beam.Map(lambda table: print(table.shape))
)

I've experimented w/ various 'min_bundle_size' values, and they result in
the same value:  100 rows, 4 cols

Re: beam.io.ReadFromParquetBatched min_bundle_size

[Brian Hulette <bh...@google.com>]

Hi Cindy,
min_bundle_size is a parameter propagated from FileBasedSource [1] that
controls how Beam performs initial splitting of files. Splitting is likely
constrained in the Parquet case, since the format's columnar nature
restricts how effective splitting can be. I haven't looked at the code but
I imagine we only allow splitting between row groups.

+Heejong Lee <he...@google.com> wrote the original Parquet connector and
may be able to comment in more detail.

[1]
https://beam.apache.org/releases/pydoc/2.30.0/apache_beam.io.filebasedsource.html

On Fri, Dec 24, 2021 at 11:36 AM Cindy McMullen <cm...@twitter.com>
wrote:

> Hi -
>
> It appears as though the 'min_bundle_size' parameter is ignored.  Can
> someone verify?  I didn't see an issue in JIRA for this.
>
> I'm following the docs here
> <https://beam.apache.org/releases/pydoc/2.30.0/apache_beam.io.parquetio.html>
> .
>
> Example code:
>
> mbs = 10000000 * 10 #  10 MBytes
> ( p
>              | 'ReadParquetBatched' >>
> beam.io.ReadFromParquetBatched(pq_file, min_bundle_size=mbs)
>             | beam.Map(lambda table: print(table.shape))
> )
>
> I've experimented w/ various 'min_bundle_size' values, and they result in
> the same value:  100 rows, 4 cols
>