Interest in adding the float16 logical type to the Parquet spec

[Anja <an...@gmail.com>]

Hello!

Is there interest in having the float16 logical type standardised in the
Parquet spec? I am proposing a PR for Arrow that will write float16 to
Parquet as FixedSizeBinary:https://issues.apache.org/jira/browse/ARROW-17464
but for the sake of portability between data analysis tools, it would of
course be a lot better to have this type standardised in the format itself.

Previous requests for this have been here:
https://issues.apache.org/jira/browse/PARQUET-1647 and here:
https://issues.apache.org/jira/browse/PARQUET-758 .

With the development of neural networks, half-precision floating points are
becoming more popular:
https://en.wikipedia.org/wiki/Half-precision_floating-point_format ; I do
think that a demand exists for its support. I am new to the project, but am
happy to contribute development time if there is support for this feature,
and guidance.

Warm regards,

Anja

Re: Interest in adding the float16 logical type to the Parquet spec

[Micah Kornfield <em...@gmail.com>]

Just as a follow-up on the proposal PR [1].  A blocker came up based on the
fact that we have never fully addressed how statistics for floating point
values  (PARQUET-1222 <https://issues.apache.org/jira/browse/PARQUET-1222>).

[1] https://github.com/apache/parquet-format/pull/184

On Wed, Aug 24, 2022 at 7:22 AM Xinli shang <sh...@uber.com.invalid> wrote:

> Hi Anja,
>
> Thanks for your interest! We encourage people's new proposals. Go head to
> make a proposal and the community can review it.
>
> Xinli
>
> On Tue, Aug 23, 2022 at 4:53 PM Anja <an...@gmail.com> wrote:
>
> > Hello!
> >
> > Is there interest in having the float16 logical type standardised in the
> > Parquet spec? I am proposing a PR for Arrow that will write float16 to
> > Parquet as FixedSizeBinary:
> > https://issues.apache.org/jira/browse/ARROW-17464
> > but for the sake of portability between data analysis tools, it would of
> > course be a lot better to have this type standardised in the format
> itself.
> >
> > Previous requests for this have been here:
> > https://issues.apache.org/jira/browse/PARQUET-1647 and here:
> > https://issues.apache.org/jira/browse/PARQUET-758 .
> >
> > With the development of neural networks, half-precision floating points
> are
> > becoming more popular:
> > https://en.wikipedia.org/wiki/Half-precision_floating-point_format ; I
> do
> > think that a demand exists for its support. I am new to the project, but
> am
> > happy to contribute development time if there is support for this
> feature,
> > and guidance.
> >
> > Warm regards,
> >
> > Anja
> >
>
>
> --
> Xinli Shang
>

Re: Interest in adding the float16 logical type to the Parquet spec

[Xinli shang <sh...@uber.com.INVALID>]

Hi Anja,

Thanks for your interest! We encourage people's new proposals. Go head to
make a proposal and the community can review it.

Xinli

On Tue, Aug 23, 2022 at 4:53 PM Anja <an...@gmail.com> wrote:

> Hello!
>
> Is there interest in having the float16 logical type standardised in the
> Parquet spec? I am proposing a PR for Arrow that will write float16 to
> Parquet as FixedSizeBinary:
> https://issues.apache.org/jira/browse/ARROW-17464
> but for the sake of portability between data analysis tools, it would of
> course be a lot better to have this type standardised in the format itself.
>
> Previous requests for this have been here:
> https://issues.apache.org/jira/browse/PARQUET-1647 and here:
> https://issues.apache.org/jira/browse/PARQUET-758 .
>
> With the development of neural networks, half-precision floating points are
> becoming more popular:
> https://en.wikipedia.org/wiki/Half-precision_floating-point_format ; I do
> think that a demand exists for its support. I am new to the project, but am
> happy to contribute development time if there is support for this feature,
> and guidance.
>
> Warm regards,
>
> Anja
>


-- 
Xinli Shang