You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "emkornfield (via GitHub)" <gi...@apache.org> on 2025/12/11 07:47:40 UTC
[PR] GH-541: Document status of file_path [parquet-format]
emkornfield opened a new pull request, #542:
URL: https://github.com/apache/parquet-format/pull/542
### Rationale for this change
`file_path` does not have any proper implementations and we should clarify its current usage, and that future
usage must go through a formal feature addition process.
### What changes are included in this PR?
Clarification on the `file_path` field status.
### Do these changes have PoC implementations?
No.
Closes #541
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-541: Document status of file_path [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #542:
URL: https://github.com/apache/parquet-format/pull/542#discussion_r2637249548
##########
src/main/thrift/parquet.thrift:
##########
@@ -958,6 +958,22 @@ union ColumnCryptoMetaData {
struct ColumnChunk {
/** File where column data is stored. If not set, assumed to be same file as
* metadata. This path is relative to the current file.
+ *
+ * As of December 2025, the only known use-case for this field is writing summary
+ * parquet files (i.e. "_metadata" files). These files consolidate footers from
+ * multiple parquet files to allow for efficient reading of footers to avoid file
+ * listing costs and prune out files that do not need to be read based on statistics.
+ * This is legacy feature as modern table formats (e.g. Iceberg, Hudi and Delta Lake)
+ * are more scalable and serve effectively the same purpose.
Review Comment:
This is at attempt to summarize this thread: https://lists.apache.org/thread/ootf2kmyg3p01b1bvplpvp4ftd1bt72d
It seems like there are potential correctness issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-541: Document status of file_path [parquet-format]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #542:
URL: https://github.com/apache/parquet-format/pull/542#discussion_r2636111324
##########
src/main/thrift/parquet.thrift:
##########
@@ -958,6 +958,24 @@ union ColumnCryptoMetaData {
struct ColumnChunk {
/** File where column data is stored. If not set, assumed to be same file as
* metadata. This path is relative to the current file.
+ *
+ * As of December 2025, the only known use-case for this field is writing summary
+ * parquet files (i.e. "_metadata" files). These files consolidate footers from
+ * multiple parquet files to allow for efficient reading of footers to avoid file
+ * listing costs and prune out files that do not need to be read based on statistics.
+ * This is legacy feature as modern table formats (e.g. Iceberg, Hudi and Delta Lake)
+ * are more scalable and serve effectively the same purpose.
+ *
+ * There is no other known use-case for this field. Specifically, there are no known
+ * readers that will read externally stored column data if this field is populated
+ * within a standard parquet file.
+ *
+ * Writers should not populate this field except for in parquet summary files. Readers
+ * should ensure this field is empty.
Review Comment:
These statements are effectively removing it from the spec, which I feel is too strong of a position for this clarification. I think it's fair to say that "readers should validate this field is empty if they do not support reading external column data", but prohibiting it is not a clarification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-541: Document status of file_path [parquet-format]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #542:
URL: https://github.com/apache/parquet-format/pull/542#discussion_r2673774670
##########
src/main/thrift/parquet.thrift:
##########
@@ -958,6 +958,21 @@ union ColumnCryptoMetaData {
struct ColumnChunk {
/** File where column data is stored. If not set, assumed to be same file as
* metadata. This path is relative to the current file.
+ *
+ * As of December 2025, the only known use-case for this field is writing summary
+ * parquet files (i.e. "_metadata" files). These files consolidate footers from
+ * multiple parquet files to allow for efficient reading of footers to avoid file
+ * listing costs and prune out files that do not need to be read based on statistics.
+ *
+ * These files do not appear to have ever been formally specified in the specification.
+ * and are potentially problematic from a correctness perspective [1].
+ *
+ * [1] https://lists.apache.org/thread/ootf2kmyg3p01b1bvplpvp4ftd1bt72d
+ *
+ * There is no other known usage of this field. Specifically, there are no known
+ * reference implementations that will read externally stored column data if this field is populated
+ * within a standard parquet file. Making use of the field for this purpose is
+ * not considered part of the Parquet specification.
Review Comment:
```suggestion
* within a standard parquet file. Making use of the field for this purpose is
* discouraged and may be removed in a later revision of the specification.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-541: Document status of file_path [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #542:
URL: https://github.com/apache/parquet-format/pull/542#discussion_r2660269228
##########
src/main/thrift/parquet.thrift:
##########
@@ -958,6 +958,21 @@ union ColumnCryptoMetaData {
struct ColumnChunk {
/** File where column data is stored. If not set, assumed to be same file as
* metadata. This path is relative to the current file.
+ *
+ * As of December 2025, the only known use-case for this field is writing summary
+ * parquet files (i.e. "_metadata" files). These files consolidate footers from
+ * multiple parquet files to allow for efficient reading of footers to avoid file
+ * listing costs and prune out files that do not need to be read based on statistics.
+ *
+ * These files do not appear to have ever been formally specified in the specification.
+ * and are potentially problematic from a correctness perspective [1].
+ *
+ * [1] https://lists.apache.org/thread/ootf2kmyg3p01b1bvplpvp4ftd1bt72d
+ *
+ * There is no other known usage of this field. Specifically, there are no known
+ * readers that will read externally stored column data if this field is populated
Review Comment:
```suggestion
* reference implementations that will read externally stored column data if this field is populated
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-541: Document status of file_path [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #542:
URL: https://github.com/apache/parquet-format/pull/542#discussion_r2635889996
##########
src/main/thrift/parquet.thrift:
##########
@@ -958,6 +958,22 @@ union ColumnCryptoMetaData {
struct ColumnChunk {
/** File where column data is stored. If not set, assumed to be same file as
* metadata. This path is relative to the current file.
+ *
+ * As of December 2025, there are no known released Parquet implementations
+ * that make use of this field when reading columns. Readers should check
+ * that the field is empty before retrieving a column from within the file,
+ * and error if it isn't.
+ *
+ * Writers should not populate this field.
+ *
+ * Any new use of this field must go through the normal Parquet feature
+ * addition process. CONTRIBUTING.md in the parquet-format repository
+ * provides details on the process.
+ *
+ * One known use-case for this field is to batch parquet footers together
Review Comment:
I rephrased above, starting with the known use-case.
##########
src/main/thrift/parquet.thrift:
##########
@@ -958,6 +958,22 @@ union ColumnCryptoMetaData {
struct ColumnChunk {
/** File where column data is stored. If not set, assumed to be same file as
* metadata. This path is relative to the current file.
+ *
+ * As of December 2025, there are no known released Parquet implementations
+ * that make use of this field when reading columns. Readers should check
+ * that the field is empty before retrieving a column from within the file,
+ * and error if it isn't.
+ *
+ * Writers should not populate this field.
+ *
+ * Any new use of this field must go through the normal Parquet feature
+ * addition process. CONTRIBUTING.md in the parquet-format repository
+ * provides details on the process.
Review Comment:
removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-541: Document status of file_path [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #542:
URL: https://github.com/apache/parquet-format/pull/542#discussion_r2636134644
##########
src/main/thrift/parquet.thrift:
##########
@@ -958,6 +958,24 @@ union ColumnCryptoMetaData {
struct ColumnChunk {
/** File where column data is stored. If not set, assumed to be same file as
* metadata. This path is relative to the current file.
+ *
+ * As of December 2025, the only known use-case for this field is writing summary
+ * parquet files (i.e. "_metadata" files). These files consolidate footers from
+ * multiple parquet files to allow for efficient reading of footers to avoid file
+ * listing costs and prune out files that do not need to be read based on statistics.
+ * This is legacy feature as modern table formats (e.g. Iceberg, Hudi and Delta Lake)
+ * are more scalable and serve effectively the same purpose.
+ *
+ * There is no other known use-case for this field. Specifically, there are no known
+ * readers that will read externally stored column data if this field is populated
+ * within a standard parquet file.
+ *
+ * Writers should not populate this field except for in parquet summary files. Readers
+ * should ensure this field is empty.
Review Comment:
I just removed this text entirely. I think this is mostly covered by the having new use go through a formal proposal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-541: Document status of file_path [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #542:
URL: https://github.com/apache/parquet-format/pull/542#discussion_r2636139058
##########
src/main/thrift/parquet.thrift:
##########
@@ -958,6 +958,22 @@ union ColumnCryptoMetaData {
struct ColumnChunk {
/** File where column data is stored. If not set, assumed to be same file as
* metadata. This path is relative to the current file.
+ *
+ * As of December 2025, the only known use-case for this field is writing summary
+ * parquet files (i.e. "_metadata" files). These files consolidate footers from
+ * multiple parquet files to allow for efficient reading of footers to avoid file
+ * listing costs and prune out files that do not need to be read based on statistics.
+ * This is legacy feature as modern table formats (e.g. Iceberg, Hudi and Delta Lake)
+ * are more scalable and serve effectively the same purpose.
+ *
+ * There is no other known usage of this field. Specifically, there are no known
+ * readers that will read externally stored column data if this field is populated
+ * within a standard parquet file.
Review Comment:
```suggestion
* within a standard parquet file. Making use of the field for this purpose is currently
* not considered part of the parquet specification.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-541: Document status of file_path [parquet-format]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #542:
URL: https://github.com/apache/parquet-format/pull/542#discussion_r2615876120
##########
src/main/thrift/parquet.thrift:
##########
@@ -958,6 +958,22 @@ union ColumnCryptoMetaData {
struct ColumnChunk {
/** File where column data is stored. If not set, assumed to be same file as
* metadata. This path is relative to the current file.
+ *
+ * As of December 2025, there are no known released Parquet implementations
+ * that make use of this field when reading columns. Readers should check
+ * that the field is empty before retrieving a column from within the file,
+ * and error if it isn't.
+ *
+ * Writers should not populate this field.
+ *
+ * Any new use of this field must go through the normal Parquet feature
+ * addition process. CONTRIBUTING.md in the parquet-format repository
+ * provides details on the process.
+ *
+ * One known use-case for this field is to batch parquet footers together
Review Comment:
I found this somewhat confusing as above it says there are no implementations that make use of this implementation but then this paragraph explains a usecase.
Maybe we could say something like
> As of December 2025, there are no known open source Parquet implementations that support
> reading external data via this field. Some query engines (which ones??) make use of this field
> to batch can batch parquet footers together into a single file that serve as an index, but this is
> not common.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-541: Document status of file_path [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #542:
URL: https://github.com/apache/parquet-format/pull/542#discussion_r2636111539
##########
src/main/thrift/parquet.thrift:
##########
@@ -958,6 +958,24 @@ union ColumnCryptoMetaData {
struct ColumnChunk {
/** File where column data is stored. If not set, assumed to be same file as
* metadata. This path is relative to the current file.
+ *
+ * As of December 2025, the only known use-case for this field is writing summary
+ * parquet files (i.e. "_metadata" files). These files consolidate footers from
+ * multiple parquet files to allow for efficient reading of footers to avoid file
+ * listing costs and prune out files that do not need to be read based on statistics.
+ * This is legacy feature as modern table formats (e.g. Iceberg, Hudi and Delta Lake)
+ * are more scalable and serve effectively the same purpose.
+ *
+ * There is no other known use-case for this field. Specifically, there are no known
Review Comment:
Yep, usage makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-541: Document status of file_path [parquet-format]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #542:
URL: https://github.com/apache/parquet-format/pull/542#discussion_r2637151736
##########
src/main/thrift/parquet.thrift:
##########
@@ -958,6 +958,22 @@ union ColumnCryptoMetaData {
struct ColumnChunk {
/** File where column data is stored. If not set, assumed to be same file as
* metadata. This path is relative to the current file.
+ *
+ * As of December 2025, the only known use-case for this field is writing summary
+ * parquet files (i.e. "_metadata" files). These files consolidate footers from
+ * multiple parquet files to allow for efficient reading of footers to avoid file
+ * listing costs and prune out files that do not need to be read based on statistics.
+ * This is legacy feature as modern table formats (e.g. Iceberg, Hudi and Delta Lake)
+ * are more scalable and serve effectively the same purpose.
Review Comment:
Seem to me that calling this "legacy" may be too opinionated -- maybe we could tone down the language with something like
> Note that table formats (e.g. Iceberg, Hudi and Delta Lake) offer a
> superset of this functionality.
##########
src/main/thrift/parquet.thrift:
##########
@@ -958,6 +958,22 @@ union ColumnCryptoMetaData {
struct ColumnChunk {
/** File where column data is stored. If not set, assumed to be same file as
* metadata. This path is relative to the current file.
+ *
+ * As of December 2025, the only known use-case for this field is writing summary
+ * parquet files (i.e. "_metadata" files). These files consolidate footers from
Review Comment:
do we have a link that describes what a summary file is and what implementations support it?
This is what came back from a quick google search: https://stackoverflow.com/questions/53150801/what-is-the-parquet-summary-file
But I didn't see any mention of this in the format repository: https://github.com/search?q=repo%3Aapache%2Fparquet-format%20summary&type=code
##########
src/main/thrift/parquet.thrift:
##########
@@ -958,6 +958,22 @@ union ColumnCryptoMetaData {
struct ColumnChunk {
/** File where column data is stored. If not set, assumed to be same file as
* metadata. This path is relative to the current file.
+ *
+ * As of December 2025, the only known use-case for this field is writing summary
+ * parquet files (i.e. "_metadata" files). These files consolidate footers from
+ * multiple parquet files to allow for efficient reading of footers to avoid file
+ * listing costs and prune out files that do not need to be read based on statistics.
+ * This is legacy feature as modern table formats (e.g. Iceberg, Hudi and Delta Lake)
+ * are more scalable and serve effectively the same purpose.
+ *
+ * There is no other known usage of this field. Specifically, there are no known
+ * readers that will read externally stored column data if this field is populated
+ * within a standard parquet file. Making use of the field for this purpose is currently
+ * not considered part of the Parquet specification.
+ *
+ * Any new use of this field must go through the normal Parquet feature
+ * addition process.
Review Comment:
this might also sound a bit too prescriptive -- maybe we can just leave it informative (remove the last two sentences)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-541: Document status of file_path [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield merged PR #542:
URL: https://github.com/apache/parquet-format/pull/542
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-541: Document status of file_path [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #542:
URL: https://github.com/apache/parquet-format/pull/542#discussion_r2637253206
##########
src/main/thrift/parquet.thrift:
##########
@@ -958,6 +958,22 @@ union ColumnCryptoMetaData {
struct ColumnChunk {
/** File where column data is stored. If not set, assumed to be same file as
* metadata. This path is relative to the current file.
+ *
+ * As of December 2025, the only known use-case for this field is writing summary
+ * parquet files (i.e. "_metadata" files). These files consolidate footers from
+ * multiple parquet files to allow for efficient reading of footers to avoid file
+ * listing costs and prune out files that do not need to be read based on statistics.
+ *
+ * These files do not appear to have ever been formally specified in the specification.
+ * and are potentially problematic from a correctness perspective [1].
+ *
+ * [1] https://lists.apache.org/thread/ootf2kmyg3p01b1bvplpvp4ftd1bt72d
+ *
+ * There is no other known usage of this field. Specifically, there are no known
+ * readers that will read externally stored column data if this field is populated
+ * within a standard parquet file. Making use of the field for this purpose is
+ * not considered part of the Parquet specification.
+ *
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-541: Document status of file_path [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on PR #542:
URL: https://github.com/apache/parquet-format/pull/542#issuecomment-3676042853
> Should we add a row to https://parquet.apache.org/docs/file-format/implementationstatus/ for this?
I think we need to update implementation status phrasing to indicate the "_metadata" use case is what is meant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-541: Document status of file_path [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #542:
URL: https://github.com/apache/parquet-format/pull/542#discussion_r2636129974
##########
src/main/thrift/parquet.thrift:
##########
@@ -958,6 +958,24 @@ union ColumnCryptoMetaData {
struct ColumnChunk {
/** File where column data is stored. If not set, assumed to be same file as
* metadata. This path is relative to the current file.
+ *
+ * As of December 2025, the only known use-case for this field is writing summary
+ * parquet files (i.e. "_metadata" files). These files consolidate footers from
+ * multiple parquet files to allow for efficient reading of footers to avoid file
+ * listing costs and prune out files that do not need to be read based on statistics.
+ * This is legacy feature as modern table formats (e.g. Iceberg, Hudi and Delta Lake)
+ * are more scalable and serve effectively the same purpose.
+ *
+ * There is no other known usage of this field. Specifically, there are no known
+ * readers that will read externally stored column data if this field is populated
+ * within a standard parquet file.
+ *
+ * Writers should not populate this field except for in parquet summary files. Readers
+ * should ensure this field is empty.
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-541: Document status of file_path [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #542:
URL: https://github.com/apache/parquet-format/pull/542#discussion_r2636141636
##########
src/main/thrift/parquet.thrift:
##########
@@ -958,6 +958,23 @@ union ColumnCryptoMetaData {
struct ColumnChunk {
/** File where column data is stored. If not set, assumed to be same file as
* metadata. This path is relative to the current file.
+ *
+ * As of December 2025, the only known use-case for this field is writing summary
+ * parquet files (i.e. "_metadata" files). These files consolidate footers from
+ * multiple parquet files to allow for efficient reading of footers to avoid file
+ * listing costs and prune out files that do not need to be read based on statistics.
+ * This is legacy feature as modern table formats (e.g. Iceberg, Hudi and Delta Lake)
+ * are more scalable and serve effectively the same purpose.
+ *
+ * There is no other known usage of this field. Specifically, there are no known
+ * readers that will read externally stored column data if this field is populated
+ * within a standard parquet file. Making use of the field for this purpose is currently
+ * not considered part of the Parquet specification.
+ *
+ *
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-541: Document status of file_path [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #542:
URL: https://github.com/apache/parquet-format/pull/542#discussion_r2637249271
##########
src/main/thrift/parquet.thrift:
##########
@@ -958,6 +958,22 @@ union ColumnCryptoMetaData {
struct ColumnChunk {
/** File where column data is stored. If not set, assumed to be same file as
* metadata. This path is relative to the current file.
+ *
+ * As of December 2025, the only known use-case for this field is writing summary
+ * parquet files (i.e. "_metadata" files). These files consolidate footers from
Review Comment:
I don't think this was ever officially part of the parquet specification as far as I can tell.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-541: Document status of file_path [parquet-format]
Posted by "alkis (via GitHub)" <gi...@apache.org>.
alkis commented on code in PR #542:
URL: https://github.com/apache/parquet-format/pull/542#discussion_r2610047452
##########
src/main/thrift/parquet.thrift:
##########
@@ -958,6 +958,22 @@ union ColumnCryptoMetaData {
struct ColumnChunk {
/** File where column data is stored. If not set, assumed to be same file as
* metadata. This path is relative to the current file.
+ *
+ * As of December 2025, there are no known released Parquet implementations
+ * that make use of this field when reading columns. Readers should check
+ * that the field is empty before retrieving a column from within the file,
+ * and error if it isn't.
+ *
+ * Writers should not populate this field.
+ *
+ * Any new use of this field must go through the normal Parquet feature
+ * addition process. CONTRIBUTING.md in the parquet-format repository
+ * provides details on the process.
Review Comment:
nit: I wouldn't mention CONTRIBUTING.md here - seems something that can be found easily without explicit mention.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-541: Document status of file_path [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #542:
URL: https://github.com/apache/parquet-format/pull/542#discussion_r2637249685
##########
src/main/thrift/parquet.thrift:
##########
@@ -958,6 +958,22 @@ union ColumnCryptoMetaData {
struct ColumnChunk {
/** File where column data is stored. If not set, assumed to be same file as
* metadata. This path is relative to the current file.
+ *
+ * As of December 2025, the only known use-case for this field is writing summary
+ * parquet files (i.e. "_metadata" files). These files consolidate footers from
+ * multiple parquet files to allow for efficient reading of footers to avoid file
+ * listing costs and prune out files that do not need to be read based on statistics.
+ * This is legacy feature as modern table formats (e.g. Iceberg, Hudi and Delta Lake)
+ * are more scalable and serve effectively the same purpose.
+ *
+ * There is no other known usage of this field. Specifically, there are no known
+ * readers that will read externally stored column data if this field is populated
+ * within a standard parquet file. Making use of the field for this purpose is currently
+ * not considered part of the Parquet specification.
+ *
+ * Any new use of this field must go through the normal Parquet feature
+ * addition process.
Review Comment:
Yes, I think this is now implied by the first paragraph, I'll remove.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-541: Document status of file_path [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #542:
URL: https://github.com/apache/parquet-format/pull/542#discussion_r2637252592
##########
src/main/thrift/parquet.thrift:
##########
@@ -958,6 +958,22 @@ union ColumnCryptoMetaData {
struct ColumnChunk {
/** File where column data is stored. If not set, assumed to be same file as
* metadata. This path is relative to the current file.
+ *
+ * As of December 2025, the only known use-case for this field is writing summary
+ * parquet files (i.e. "_metadata" files). These files consolidate footers from
+ * multiple parquet files to allow for efficient reading of footers to avoid file
+ * listing costs and prune out files that do not need to be read based on statistics.
+ * This is legacy feature as modern table formats (e.g. Iceberg, Hudi and Delta Lake)
+ * are more scalable and serve effectively the same purpose.
+ *
+ * There is no other known usage of this field. Specifically, there are no known
+ * readers that will read externally stored column data if this field is populated
+ * within a standard parquet file. Making use of the field for this purpose is currently
+ * not considered part of the Parquet specification.
+ *
+ * Any new use of this field must go through the normal Parquet feature
+ * addition process.
Review Comment:
removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-541: Document status of file_path [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #542:
URL: https://github.com/apache/parquet-format/pull/542#discussion_r2636140814
##########
src/main/thrift/parquet.thrift:
##########
@@ -958,6 +958,23 @@ union ColumnCryptoMetaData {
struct ColumnChunk {
/** File where column data is stored. If not set, assumed to be same file as
* metadata. This path is relative to the current file.
+ *
+ * As of December 2025, the only known use-case for this field is writing summary
+ * parquet files (i.e. "_metadata" files). These files consolidate footers from
+ * multiple parquet files to allow for efficient reading of footers to avoid file
+ * listing costs and prune out files that do not need to be read based on statistics.
+ * This is legacy feature as modern table formats (e.g. Iceberg, Hudi and Delta Lake)
+ * are more scalable and serve effectively the same purpose.
+ *
+ * There is no other known usage of this field. Specifically, there are no known
+ * readers that will read externally stored column data if this field is populated
+ * within a standard parquet file. Making use of the field for this purpose is currently
+ * not considered part of the parquet specification.
Review Comment:
```suggestion
* not considered part of the Parquet specification.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-541: Document status of file_path [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on PR #542:
URL: https://github.com/apache/parquet-format/pull/542#issuecomment-3837963706
Thanks everyone for the reviews. Merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-541: Document status of file_path [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #542:
URL: https://github.com/apache/parquet-format/pull/542#discussion_r2637252510
##########
src/main/thrift/parquet.thrift:
##########
@@ -958,6 +958,22 @@ union ColumnCryptoMetaData {
struct ColumnChunk {
/** File where column data is stored. If not set, assumed to be same file as
* metadata. This path is relative to the current file.
+ *
+ * As of December 2025, the only known use-case for this field is writing summary
+ * parquet files (i.e. "_metadata" files). These files consolidate footers from
Review Comment:
I reworded this section.
##########
src/main/thrift/parquet.thrift:
##########
@@ -958,6 +958,22 @@ union ColumnCryptoMetaData {
struct ColumnChunk {
/** File where column data is stored. If not set, assumed to be same file as
* metadata. This path is relative to the current file.
+ *
+ * As of December 2025, the only known use-case for this field is writing summary
+ * parquet files (i.e. "_metadata" files). These files consolidate footers from
+ * multiple parquet files to allow for efficient reading of footers to avoid file
+ * listing costs and prune out files that do not need to be read based on statistics.
+ * This is legacy feature as modern table formats (e.g. Iceberg, Hudi and Delta Lake)
+ * are more scalable and serve effectively the same purpose.
Review Comment:
added this to the text.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-541: Document status of file_path [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #542:
URL: https://github.com/apache/parquet-format/pull/542#discussion_r2637252904
##########
src/main/thrift/parquet.thrift:
##########
@@ -958,6 +958,22 @@ union ColumnCryptoMetaData {
struct ColumnChunk {
/** File where column data is stored. If not set, assumed to be same file as
* metadata. This path is relative to the current file.
+ *
+ * As of December 2025, the only known use-case for this field is writing summary
+ * parquet files (i.e. "_metadata" files). These files consolidate footers from
+ * multiple parquet files to allow for efficient reading of footers to avoid file
+ * listing costs and prune out files that do not need to be read based on statistics.
+ *
+ * These files do not appear to have ever been formally specified in the specification.
+ * and are potentially problematic from correctness perspective [1].
Review Comment:
```suggestion
* and are potentially problematic from a correctness perspective [1].
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-541: Document status of file_path [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #542:
URL: https://github.com/apache/parquet-format/pull/542#discussion_r2636129559
##########
src/main/thrift/parquet.thrift:
##########
@@ -958,6 +958,24 @@ union ColumnCryptoMetaData {
struct ColumnChunk {
/** File where column data is stored. If not set, assumed to be same file as
* metadata. This path is relative to the current file.
+ *
+ * As of December 2025, the only known use-case for this field is writing summary
+ * parquet files (i.e. "_metadata" files). These files consolidate footers from
+ * multiple parquet files to allow for efficient reading of footers to avoid file
+ * listing costs and prune out files that do not need to be read based on statistics.
+ * This is legacy feature as modern table formats (e.g. Iceberg, Hudi and Delta Lake)
+ * are more scalable and serve effectively the same purpose.
+ *
+ * There is no other known usage of this field. Specifically, there are no known
+ * readers that will read externally stored column data if this field is populated
+ * within a standard parquet file.
+ *
+ * Writers should not populate this field except for in parquet summary files. Readers
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-541: Document status of file_path [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #542:
URL: https://github.com/apache/parquet-format/pull/542#discussion_r2636149126
##########
src/main/thrift/parquet.thrift:
##########
@@ -958,6 +958,24 @@ union ColumnCryptoMetaData {
struct ColumnChunk {
/** File where column data is stored. If not set, assumed to be same file as
* metadata. This path is relative to the current file.
+ *
+ * As of December 2025, the only known use-case for this field is writing summary
+ * parquet files (i.e. "_metadata" files). These files consolidate footers from
+ * multiple parquet files to allow for efficient reading of footers to avoid file
+ * listing costs and prune out files that do not need to be read based on statistics.
+ * This is legacy feature as modern table formats (e.g. Iceberg, Hudi and Delta Lake)
+ * are more scalable and serve effectively the same purpose.
+ *
+ * There is no other known use-case for this field. Specifically, there are no known
+ * readers that will read externally stored column data if this field is populated
+ * within a standard parquet file.
+ *
+ * Writers should not populate this field except for in parquet summary files. Readers
+ * should ensure this field is empty.
Review Comment:
Not sure if this was also referring to below as well. But I updated some language above this specific sentence in the paragraph above to also clarify that using this for reading external columns is not considered part of the specification (which is reflected by current implementation status).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-541: Document status of file_path [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on PR #542:
URL: https://github.com/apache/parquet-format/pull/542#issuecomment-3676092937
https://github.com/apache/parquet-site/pull/145 to update implementation status
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-541: Document status of file_path [parquet-format]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #542:
URL: https://github.com/apache/parquet-format/pull/542#discussion_r2636096307
##########
src/main/thrift/parquet.thrift:
##########
@@ -958,6 +958,24 @@ union ColumnCryptoMetaData {
struct ColumnChunk {
/** File where column data is stored. If not set, assumed to be same file as
* metadata. This path is relative to the current file.
+ *
+ * As of December 2025, the only known use-case for this field is writing summary
+ * parquet files (i.e. "_metadata" files). These files consolidate footers from
+ * multiple parquet files to allow for efficient reading of footers to avoid file
+ * listing costs and prune out files that do not need to be read based on statistics.
+ * This is legacy feature as modern table formats (e.g. Iceberg, Hudi and Delta Lake)
+ * are more scalable and serve effectively the same purpose.
+ *
+ * There is no other known use-case for this field. Specifically, there are no known
Review Comment:
```suggestion
* There is no other known usage of this field. Specifically, there are no known
```
I think we should clarify since I believe there are use cases, but there's no implementation. I don't think it's correct to assert there is no use case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-541: Document status of file_path [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #542:
URL: https://github.com/apache/parquet-format/pull/542#discussion_r2749024372
##########
src/main/thrift/parquet.thrift:
##########
@@ -958,6 +958,21 @@ union ColumnCryptoMetaData {
struct ColumnChunk {
/** File where column data is stored. If not set, assumed to be same file as
* metadata. This path is relative to the current file.
+ *
+ * As of December 2025, the only known use-case for this field is writing summary
+ * parquet files (i.e. "_metadata" files). These files consolidate footers from
+ * multiple parquet files to allow for efficient reading of footers to avoid file
+ * listing costs and prune out files that do not need to be read based on statistics.
+ *
+ * These files do not appear to have ever been formally specified in the specification.
+ * and are potentially problematic from a correctness perspective [1].
+ *
+ * [1] https://lists.apache.org/thread/ootf2kmyg3p01b1bvplpvp4ftd1bt72d
+ *
+ * There is no other known usage of this field. Specifically, there are no known
+ * reference implementations that will read externally stored column data if this field is populated
+ * within a standard parquet file. Making use of the field for this purpose is
+ * not considered part of the Parquet specification.
Review Comment:
I think the consensus is that making use of the field needs to be revisited with a strong use-case, I'd prefer to leave the wording as is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org