[DISCUSS] complete FILE proposal

[Alkis Evlogimenos via dev <de...@parquet.apache.org>]

Hello,

The FILE PR [1] is in good shape — path/size/offset/etag are in, and
there's an active thread on adding content_type. Following last week's sync
and the discussing in the comments of the PR, we took a step back and
looked at the difference in complexity between having two separate types:
one narrowly focused on referencing a file and later introducing one that
can accommodate inline/in-file/packed representations and it all came down
to the addition of a single field (along with how fields are interpreted).
Rather than have multiple types that significantly overlap in capability
and representation, we think it make sense to provide a single type that
can address all necessary features. The changes in short:

   - make path optional — an absent path means a self-reference to the
   current file (already raised in review; makes self-referencing files
   trivially moveable)
   - add inline for storing bytes directly in the value
   - generalize etag into a self-describing checksum (it carries an etag as
   one case)

FILE then spans inline/in-file/external as one type. Defining the full set
now avoids a second type later that every reader and writer would have to
handle.

Details in the "FILE — complete type" tab of the design doc [2]. Feedback
welcome.

Regards,
Alkis

[1] https://github.com/apache/parquet-format/pull/585
[2]
https://docs.google.com/document/d/1AiwrstqkwkBoOZqgOkm9JGwSMcNeHyLR7EEj1CVqpZQ/edit

Re: [DISCUSS] complete FILE proposal

[Daniel Weeks <dw...@apache.org>]

I wanted to follow up based on the discussion in the parquet sync and
highlight a few things:

(Notes summary here
<https://docs.google.com/document/d/11auIypU3ZfUvuusyWm1SLTFodnUsdf3b-jpG_R4MmKc/edit?tab=t.xf2812q29vzd#bookmark=id.xgdbdak3iq3p>,
transcript here
<https://docs.google.com/document/d/11auIypU3ZfUvuusyWm1SLTFodnUsdf3b-jpG_R4MmKc/edit?tab=t.rzuip2y0am75#heading=h.22t1qst27vv3>
)

One of the nice features of the additional inline and self-referencing path
is that the read path is trivial regardless of where the data is stored.
This leaves the complexity up to the writer implementations to decide which
approach they want to use, but consumers aren't burdened by those choices.

Other than that, the proposal is still very much inline with the
principals we discussed before about keeping the type minimal.

I think the next steps from here are to update the PR to reflect these
changes and start making some progress on reference implementations.

I'd love to hear if there is disagreement with this as a path forward.

Thanks,
Dan


On Tue, Jun 30, 2026 at 8:37 AM Alkis Evlogimenos via dev <
dev@parquet.apache.org> wrote:

> Hello,
>
> The FILE PR [1] is in good shape — path/size/offset/etag are in, and
> there's an active thread on adding content_type. Following last week's sync
> and the discussing in the comments of the PR, we took a step back and
> looked at the difference in complexity between having two separate types:
> one narrowly focused on referencing a file and later introducing one that
> can accommodate inline/in-file/packed representations and it all came down
> to the addition of a single field (along with how fields are interpreted).
> Rather than have multiple types that significantly overlap in capability
> and representation, we think it make sense to provide a single type that
> can address all necessary features. The changes in short:
>
>    - make path optional — an absent path means a self-reference to the
>    current file (already raised in review; makes self-referencing files
>    trivially moveable)
>    - add inline for storing bytes directly in the value
>    - generalize etag into a self-describing checksum (it carries an etag as
>    one case)
>
> FILE then spans inline/in-file/external as one type. Defining the full set
> now avoids a second type later that every reader and writer would have to
> handle.
>
> Details in the "FILE — complete type" tab of the design doc [2]. Feedback
> welcome.
>
> Regards,
> Alkis
>
> [1] https://github.com/apache/parquet-format/pull/585
> [2]
>
> https://docs.google.com/document/d/1AiwrstqkwkBoOZqgOkm9JGwSMcNeHyLR7EEj1CVqpZQ/edit
>