Next Parquet Sync

[Julien Le Dem <ju...@apache.org>]

The next Parquet Sync is happening now at 9:30am PT - 12:30pm ET - 6:30pm
CET
(sorry I didn't send a reminder earlier)
To join the invite:
https://calendar.app.google/KM28Ci71B2DMoHo4A
Everybody is welcome, bring your topic or just listen in.
Best
Julien

Parquet Sync Notes September 11th 2024

[Julien Le Dem <ju...@apache.org>]

Attendees:

   -

   Micah
   -

   Gene Pang: Variant type plan review
   -

   Rok: need help on encryption. _metadata problem
   -

   Ashish
   -

   Julien

Notes:

   -

   Variant:
   -

      Commitment to get Variant in Parquet
      -

      Gene is working on the plan for moving the implementation into
      parquet. He shared on the mailing list for feedback.
      -

      Next step: We need to finalize the plan. plan doc
      <https://docs.google.com/document/d/1guEzBQjzOEEZvvibeZjNraKmZHWtxQR95O_DvtZU0xw/edit?usp=sharing>
      -

      Goals:
      -

         Do we need to release Variant independently?
         -

            Spec released with parquet-format
            -

            Java implementation can be released with parquet-java as long
            as we can do releases easily in the near future.
            -

            Discussion on the need to have a separate repo
            -

               We’d like to avoid parquet repos proliferation.
               -

         Separate jar with its own dependencies
         -

            To facilitate reuse by other projects (ex: Spark, …), there
            should be a separate artifact with minimal dependencies.
That can be done
            as a separate maven module in parquet-java.



   -

   Parquet cpp implementation is in Arrow
   -

   Parquet Rust implementation:
   -

      2 implementations
      -

         Arrow
         -

         Polars
         -

   Interval type in Parquet is a mess and needs improvement [Micah]
   -

   We’ll need another vote on the plan when finalized.


   -

   Encryption:
   -

      Use case: Need to use  _metadata file but it is not encrypted.
      -

      Work in progress to add this but currently the encryption doesn’t
      work properly. AD strings unique per file, reader does not use them
      correctly.
      -

      Need to solve it in a way that we can merge it and add encryption
      support to _metadata.
      -

      Giddon is helpin on the ML.
      -

      PR: https://github.com/apache/arrow/pull/41821
      -

   Action Items:
   -

      Gene to update doc with release requirements etc
      -

      Micah, Julien, …: Add feedback in the doc
      -

      Julien: enable people to add the recurring google calendar invite to
      their calendar.


On Wed, Sep 11, 2024 at 9:34 AM Julien Le Dem <ju...@apache.org> wrote:

> The next Parquet Sync is happening now at 9:30am PT - 12:30pm ET - 6:30pm
> CET
> (sorry I didn't send a reminder earlier)
> To join the invite:
> https://calendar.app.google/KM28Ci71B2DMoHo4A
> Everybody is welcome, bring your topic or just listen in.
> Best
> Julien
>

Re: Next Parquet Sync

[Julien Le Dem <ju...@apache.org>]

It came to my attention in the sync today that the link only lets you join
the next meeting and doesn't add you to the recurring invite.
I will look into a solution for people to be able to add themselves to the
recurring meeting. (suggestions welcome)
In the meantime feel free to respond privately if you want me to add your
email manually to the invite.
Next meeting: https://calendar.app.google/kZaYv5kHJR715sP46

On Wed, Sep 11, 2024 at 9:34 AM Julien Le Dem <ju...@apache.org> wrote:

> The next Parquet Sync is happening now at 9:30am PT - 12:30pm ET - 6:30pm
> CET
> (sorry I didn't send a reminder earlier)
> To join the invite:
> https://calendar.app.google/KM28Ci71B2DMoHo4A
> Everybody is welcome, bring your topic or just listen in.
> Best
> Julien
>