You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "brkyvz (via GitHub)" <gi...@apache.org> on 2026/06/09 22:04:48 UTC
[PR] Introduces a new LogicalType: FILE [parquet-format]
brkyvz opened a new pull request, #585:
URL: https://github.com/apache/parquet-format/pull/585
<!--
Thanks for opening a pull request!
If you're new to Parquet-Format, information on how to contribute can be found here: https://parquet.apache.org/docs/contribution-guidelines/contributing
Please open a GitHub issue for this pull request: https://github.com/apache/parquet-format/issues/new/choose
and format pull request title as below:
GH-${GITHUB_ISSUE_ID}: ${SUMMARY}
or simply use the title below if it is a minor issue:
MINOR: ${SUMMARY}
-->
### Rationale for this change
Introduces a new type called File as a typed FileReference. The design document is [here](https://docs.google.com/document/d/1AiwrstqkwkBoOZqgOkm9JGwSMcNeHyLR7EEj1CVqpZQ/edit?tab=t.0#heading=h.k8qyue4jj4rn).
The motivation is as follows:
```
Unstructured data ingestion is getting extremely popular with the advances in Generative AI. Today, our only means of dealing with unstructured data is to store it as a binary blob inside Parquet, or point to files that exist in some object store with a string. These solutions fail to address these use cases, because of scalability, usability, and governance issues.
We would like to introduce a new logical type annotation in Parquet called “File” for storing a struct that contains a path reference to a file with additional metadata. This reference may be to a file that exists (or expected to exist) in storage at a given path. We’d like to define the minimum required list of fields that would allow a client to correctly read the referenced data. Any additional metadata can be optionally stored by engines and table formats as necessary adjacent to this type.
```
### What changes are included in this PR?
Introduces the specification for FileType.
### Do these changes have PoC implementations?
Yes: https://github.com/apache/parquet-java/pull/3608/changes
<!-- Please uncomment the line below and replace ${GITHUB_ISSUE_ID} with the actual Github issue id. -->
<!-- Closes #${GITHUB_ISSUE_ID} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3546295539
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
Added self-referencing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3632340820
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | invalid |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | - | set | - | invalid |
+| – | - | - | set | invalid |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+`size` must be set whenever `offset` is set, so any offset-based read always carries an
+explicit `size`. A self-reference (`path` not set) must set `offset`, and therefore also
+`size`. `size` may be omitted only for a whole-file external reference, where the range
+runs to the end of the referenced file.
+
+A self-reference points within the same Parquet file using `offset` and `size` (both
+required); the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is when `path` is not set, never an absolute path back to
+the current file, so a file containing self-references is renamed or relocated as a
+single unit.
+
+The bytes referenced by a self-reference are compressed with the same `CompressionCodec`
+as the one specified for the `inline` column.
+
+#### Validation
+
+* A value must resolve to some referenced data. It resolves only if `inline`, `path`, or
+ `offset` is set; if none of them are set, the value does not resolve and is invalid, even
+ if `size` is set. Use column nullability to represent a null value.
+* A self-reference (`path` not set) must set `offset`. A value with neither `path` nor
+ `offset` set (and not `inline`) does not resolve and is invalid.
+* `size` must be set whenever `offset` is set. A value that sets `offset` without `size`
+ is invalid. Because a self-reference must set `offset`, it must also set `size`.
+* If `inline` is set, it supplies the bytes; producers may instead treat `inline` and the
+ locator fields as mutually exclusive.
+* Field names within a `FILE`-annotated group must not be renamed.
+* Additional metadata about the file (e.g., modification timestamp) must
+ be stored adjacent to this group by engines or table formats, not inside it.
+
+Statistics may be collected for the individual fields of a `FILE`-annotated group
+according to the sort order of each field's logical type.
+
+This is an example of a `FILE`-annotated group that defines all fields:
+
+```
+optional group my_file (FILE) {
+ optional binary path (STRING);
+ optional int64 offset;
+ optional int64 size;
+ optional binary content_type (STRING);
+ optional binary checksum (STRING);
+ optional binary inline;
+}
+```
+
+Because every field is optional, a group need only define the fields it uses. A group
+whose values are always stored inline may define just `inline`:
Review Comment:
I added ```and optionally `content_type` as additional metadata:```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3632343404
##########
LogicalTypes.md:
##########
@@ -635,7 +635,183 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty[1]). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+[1] Implementations are not expected to treat empty strings as null
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type), as defined by RFC 2046, of the resolved bytes (e.g., `image/png`).
+When not set, the type can be assumed as `application/octet-stream`.
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:<digest>`, where `<digest>` is encoded according to the `Encoding`
+column below. Readers should ignore unknown algorithms. The recognized algorithms
+are:
+
+| Algorithm | Encoding | Notes |
+|-----------|---------------|----------------------------------------------------------|
+| `ETAG` | opaque | the object-store eTag, not recomputable |
+| `MD5` | lowercase hex | as defined in RFC 6151 represented as 32 hex characters |
Review Comment:
done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3560128768
##########
LogicalTypes.md:
##########
@@ -635,7 +635,133 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within this file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name. Field IDs
+may also be used for projection. All fields are optional:
+
+| Field | Type | Required |
+|----------------|------------|----------|
+| `path` | STRING | No |
+| `offset` | INT64 | No |
+| `size` | INT64 | No |
+| `content_type` | STRING | No |
+| `checksum` | STRING | No |
+| `inline` | BYTE_ARRAY | No |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+##### path
+
+An opaque path string to an external file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied on top of the user provided data. If `path` is absent,
+the value refers to this file (a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not provided, readers must treat the value as 0.
+If provided and non-zero, readers must seek to this offset to retrieve the referenced data.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if provided.
+A value of 0 indicates empty referenced data. If not provided, the range runs to the end
+of the referenced data.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. It generalizes the storage-system eTag. The
+recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are present are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|----------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | – | external `path`, `[offset, EOF)` |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | – | set | – | this file, `[offset, EOF)` (self-reference) |
+| – | – | – | set | this file, `[0, size)` (self-reference) |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+A self-reference typically points within the same Parquet file using `offset` and
+`size`; the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is the absence of `path`, never an absolute path back to
+the current file, so a file containing self-references is renamed or relocated as a
+single unit.
+
+The referenced bytes are compressed with the same `CompressionCodec` as the one
+specified for the `inline` column.
Review Comment:
@alkis thoughts? Is there a reason that the data needs to be compressed additionally? Most formats may be optimal anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "alkis (via GitHub)" <gi...@apache.org>.
alkis commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3430299756
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
Can we make this optional and say if it is NULL or empty the path is the parquet file itself? This makes a parquet file self contained and moveable trivially (renaming the file itself does not require a re-encoding pass).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3501044097
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
Yes, it would make adding the file type a forward-incompatible change, this might be a reason to exclude it from the spec for the time being. CC @alkis
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3580748953
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | invalid |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | - | set | - | invalid |
+| – | - | - | set | invalid |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+`size` must be set whenever `offset` is set, so any offset-based read always carries an
+explicit `size`. A self-reference (`path` not set) must set `offset`, and therefore also
+`size`. `size` may be omitted only for a whole-file external reference, where the range
+runs to the end of the referenced file.
+
+A self-reference points within the same Parquet file using `offset` and `size` (both
+required); the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is when `path` is not set, never an absolute path back to
+the current file, so a file containing self-references is renamed or relocated as a
+single unit.
+
+The bytes referenced by a self-reference are compressed with the same `CompressionCodec`
+as the one specified for the `inline` column.
+
+#### Validation
+
+* A value must resolve to some referenced data. It resolves only if `inline`, `path`, or
+ `offset` is set; if none of them are set, the value does not resolve and is invalid, even
+ if `size` is set. Use column nullability to represent a null value.
+* A self-reference (`path` not set) must set `offset`. A value with neither `path` nor
+ `offset` set (and not `inline`) does not resolve and is invalid.
+* `size` must be set whenever `offset` is set. A value that sets `offset` without `size`
+ is invalid. Because a self-reference must set `offset`, it must also set `size`.
+* If `inline` is set, it supplies the bytes; producers may instead treat `inline` and the
+ locator fields as mutually exclusive.
+* Field names within a `FILE`-annotated group must not be renamed.
+* Additional metadata about the file (e.g., modification timestamp) must
+ be stored adjacent to this group by engines or table formats, not inside it.
+
+Statistics may be collected for the individual fields of a `FILE`-annotated group
Review Comment:
Which kind of type family does `FILE` belong to: primitive, complex, or something else?
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | invalid |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | - | set | - | invalid |
+| – | - | - | set | invalid |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+`size` must be set whenever `offset` is set, so any offset-based read always carries an
+explicit `size`. A self-reference (`path` not set) must set `offset`, and therefore also
+`size`. `size` may be omitted only for a whole-file external reference, where the range
+runs to the end of the referenced file.
+
+A self-reference points within the same Parquet file using `offset` and `size` (both
+required); the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is when `path` is not set, never an absolute path back to
+the current file, so a file containing self-references is renamed or relocated as a
+single unit.
+
+The bytes referenced by a self-reference are compressed with the same `CompressionCodec`
+as the one specified for the `inline` column.
+
+#### Validation
+
+* A value must resolve to some referenced data. It resolves only if `inline`, `path`, or
+ `offset` is set; if none of them are set, the value does not resolve and is invalid, even
+ if `size` is set. Use column nullability to represent a null value.
+* A self-reference (`path` not set) must set `offset`. A value with neither `path` nor
+ `offset` set (and not `inline`) does not resolve and is invalid.
+* `size` must be set whenever `offset` is set. A value that sets `offset` without `size`
+ is invalid. Because a self-reference must set `offset`, it must also set `size`.
+* If `inline` is set, it supplies the bytes; producers may instead treat `inline` and the
+ locator fields as mutually exclusive.
+* Field names within a `FILE`-annotated group must not be renamed.
+* Additional metadata about the file (e.g., modification timestamp) must
+ be stored adjacent to this group by engines or table formats, not inside it.
+
+Statistics may be collected for the individual fields of a `FILE`-annotated group
+according to the sort order of each field's logical type.
+
+This is an example of a `FILE`-annotated group that defines all fields:
+
+```
+optional group my_file (FILE) {
+ optional binary path (STRING);
+ optional int64 offset;
+ optional int64 size;
+ optional binary content_type (STRING);
+ optional binary checksum (STRING);
+ optional binary inline;
+}
+```
+
+Because every field is optional, a group need only define the fields it uses. A group
+whose values are always stored inline may define just `inline`:
Review Comment:
Why defined `content_type` below but here says `just inline`? Seems inconsistent.
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | invalid |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | - | set | - | invalid |
+| – | - | - | set | invalid |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
Review Comment:
Why this is invalid? Shouldn't it be unset, namely no file is associated with this row?
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | invalid |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | - | set | - | invalid |
+| – | - | - | set | invalid |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+`size` must be set whenever `offset` is set, so any offset-based read always carries an
+explicit `size`. A self-reference (`path` not set) must set `offset`, and therefore also
+`size`. `size` may be omitted only for a whole-file external reference, where the range
+runs to the end of the referenced file.
+
+A self-reference points within the same Parquet file using `offset` and `size` (both
+required); the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is when `path` is not set, never an absolute path back to
+the current file, so a file containing self-references is renamed or relocated as a
+single unit.
+
+The bytes referenced by a self-reference are compressed with the same `CompressionCodec`
Review Comment:
The does not look right. Same column can have different codecs across different row groups. V2 pages can even decide whether to compress individually. How can we tell whether a self-referenced data is compressed or not?
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | invalid |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | - | set | - | invalid |
+| – | - | - | set | invalid |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+`size` must be set whenever `offset` is set, so any offset-based read always carries an
+explicit `size`. A self-reference (`path` not set) must set `offset`, and therefore also
+`size`. `size` may be omitted only for a whole-file external reference, where the range
+runs to the end of the referenced file.
+
+A self-reference points within the same Parquet file using `offset` and `size` (both
+required); the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is when `path` is not set, never an absolute path back to
+the current file, so a file containing self-references is renamed or relocated as a
+single unit.
+
+The bytes referenced by a self-reference are compressed with the same `CompressionCodec`
+as the one specified for the `inline` column.
+
+#### Validation
+
+* A value must resolve to some referenced data. It resolves only if `inline`, `path`, or
+ `offset` is set; if none of them are set, the value does not resolve and is invalid, even
+ if `size` is set. Use column nullability to represent a null value.
+* A self-reference (`path` not set) must set `offset`. A value with neither `path` nor
+ `offset` set (and not `inline`) does not resolve and is invalid.
+* `size` must be set whenever `offset` is set. A value that sets `offset` without `size`
+ is invalid. Because a self-reference must set `offset`, it must also set `size`.
+* If `inline` is set, it supplies the bytes; producers may instead treat `inline` and the
+ locator fields as mutually exclusive.
+* Field names within a `FILE`-annotated group must not be renamed.
+* Additional metadata about the file (e.g., modification timestamp) must
+ be stored adjacent to this group by engines or table formats, not inside it.
+
+Statistics may be collected for the individual fields of a `FILE`-annotated group
+according to the sort order of each field's logical type.
Review Comment:
```suggestion
according to the sort order of each field.
```
Or physical and logical type?
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
Review Comment:
Could you elaborate when it could be `relative` and any restriction on using a relative path?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3582891953
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | invalid |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | - | set | - | invalid |
+| – | - | - | set | invalid |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
Review Comment:
Yes, I think the intent is that if you make the struct non-null, then at least one of the values that tracks content must be used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3606837664
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
If I understand correctly `path` was not renamed to `field` yet?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3483657310
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
Review Comment:
I think there's a strong case to include content_type here, but with well defined limits so it doesn't turn into a general optional-metadata bucket.
Concretely:
- content_type is a media type per [RFC 6838](https://www.rfc-editor.org/rfc/rfc6838) and should be an IANA-registered type (https://www.iana.org/assignments/media-types/media-types.xhtml). I'd lean toward constraining it further, but I don't think we can do that cleanly, and I'm not sure forbidding things like multipart actually buys anyone anything.
- It is matched case-insensitively, readers should compare on the type/subtype.
- If the field is missing, or a reader sees a value it can't parse or doesn't recognize, it is treated as application/octet-stream. That keeps readers from having to fail on a bad value, and it means "missing" and "octet-stream" get the same handling. Whatever an engine does for octet-stream is what it should do when the field is absent.
The point of the constraint is that every engine, reader, or utility handed a FILE gets a known, bounded set of behaviors it can expect from the string, with a defined default. This is basically what the cloud storage providers do today, and what HTTP does with a missing Content-Type, so we're matching the most common existing behavior rather than inventing new semantics.
To be clear on scope, this is not an argument for arbitrary optional metadata.
This solves the self-describing type in the offset/size case Stefan raised, where path alone can't tell you the modality of a packed slice.
cc @rdblue (who was chatting with me about this earlier today)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3399076238
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
Review Comment:
I think it would be better to use `repetition` instead of `required`, and either `required` or `optional` in the table.
I would also state that the struct must contain exactly these fields, rather than saying they are all required since "required" has a special meaning here (the Parquet type's repetition).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "alkis (via GitHub)" <gi...@apache.org>.
alkis commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3545284569
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,35 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to a file, or to a range of
+ * bytes that may be stored inline, elsewhere in this file, or in an external
+ * file. All fields are optional and are identified by name:
+ * - path (STRING): an opaque location string (e.g. s3://bucket/file.jpg);
+ * if absent, the value refers to this file (self-reference)
+ * - offset (INT64): start of the byte range; if absent, treated as 0
+ * - size (INT64): byte length of the referenced data; required when `offset` is set
+ * or `path` is absent. May be omitted only for a whole-file external reference
Review Comment:
We should make this required when either `offset` or `path` is set.
For self-reference (`path` not set) `offset` is set.
For external reference: if full `size` gives us the size of the file (avoiding a HEAD request if object store), else `offset` is set and it is required.
Alternatively we can say `required` unless `inline` is set?
##########
LogicalTypes.md:
##########
@@ -635,7 +635,164 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name. Field IDs
+may also be used for projection. Every field is optional both in the schema and in the
+data: a writer may omit any field from the group definition, and any field that is
+present has a field repetition type of `OPTIONAL`. A group need only define the fields
+it uses (for example, an inline-only group may define just `inline`, and an external
+reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+An opaque path string to an external file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied on top of the user-provided data. If `path` is
+not set, the value refers to the current file (a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set
+or `path` is not set. It may be omitted only for a whole-file external reference (`path`
+set, `offset` not set), in which case the range runs to the end of the referenced file.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. It generalizes the storage-system eTag. The
+recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
Review Comment:
nit: align the rightmost `|`
##########
LogicalTypes.md:
##########
@@ -635,7 +635,164 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name. Field IDs
+may also be used for projection. Every field is optional both in the schema and in the
+data: a writer may omit any field from the group definition, and any field that is
+present has a field repetition type of `OPTIONAL`. A group need only define the fields
+it uses (for example, an inline-only group may define just `inline`, and an external
+reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+An opaque path string to an external file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied on top of the user-provided data. If `path` is
+not set, the value refers to the current file (a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set
+or `path` is not set. It may be omitted only for a whole-file external reference (`path`
+set, `offset` not set), in which case the range runs to the end of the referenced file.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. It generalizes the storage-system eTag. The
+recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | – | – | set | current file, `[0, size)` (self-reference) |
+| – | – | set | set | current file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+`size` must be set whenever `offset` is set or `path` is not set, so a self-reference
+and any offset-based read always carry an explicit `size`. It may be omitted only for a
+whole-file external reference, where the range runs to the end of the referenced file.
+
+A self-reference typically points within the same Parquet file using `offset` and
+`size`; the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is when `path` is not set, never an absolute path back to
+the current file, so a file containing self-references is renamed or relocated as a
+single unit.
+
+The bytes referenced by a self-reference are compressed with the same `CompressionCodec`
+as the one specified for the `inline` column.
+
+#### Validation
+
+* A value must resolve to some referenced data. If none of `inline`, `path`, `offset`, or
+ `size` is set, the value does not resolve and is invalid; use column nullability to
+ represent a null value.
+* `size` must be set whenever `offset` is set or `path` is not set. A value that sets
+ `offset` without `size`, or is a self-reference (no `path`) without `size`, is invalid.
Review Comment:
Also needs update.
##########
LogicalTypes.md:
##########
@@ -635,7 +635,164 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name. Field IDs
+may also be used for projection. Every field is optional both in the schema and in the
+data: a writer may omit any field from the group definition, and any field that is
+present has a field repetition type of `OPTIONAL`. A group need only define the fields
+it uses (for example, an inline-only group may define just `inline`, and an external
+reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+An opaque path string to an external file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied on top of the user-provided data. If `path` is
+not set, the value refers to the current file (a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set
+or `path` is not set. It may be omitted only for a whole-file external reference (`path`
+set, `offset` not set), in which case the range runs to the end of the referenced file.
Review Comment:
This need the equivalent update in the comment in the thrift udl.
##########
LogicalTypes.md:
##########
@@ -635,7 +635,164 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name. Field IDs
+may also be used for projection. Every field is optional both in the schema and in the
+data: a writer may omit any field from the group definition, and any field that is
+present has a field repetition type of `OPTIONAL`. A group need only define the fields
+it uses (for example, an inline-only group may define just `inline`, and an external
+reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+An opaque path string to an external file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied on top of the user-provided data. If `path` is
+not set, the value refers to the current file (a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set
+or `path` is not set. It may be omitted only for a whole-file external reference (`path`
+set, `offset` not set), in which case the range runs to the end of the referenced file.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. It generalizes the storage-system eTag. The
+recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
Review Comment:
Let's update this table with the updated table to the design doc:
<img width="646" height="437" alt="Image" src="https://github.com/user-attachments/assets/778fdbd2-3d75-4918-b754-69b401a5f1fd" />
##########
LogicalTypes.md:
##########
@@ -635,7 +635,164 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name. Field IDs
+may also be used for projection. Every field is optional both in the schema and in the
+data: a writer may omit any field from the group definition, and any field that is
+present has a field repetition type of `OPTIONAL`. A group need only define the fields
+it uses (for example, an inline-only group may define just `inline`, and an external
+reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+An opaque path string to an external file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied on top of the user-provided data. If `path` is
+not set, the value refers to the current file (a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set
+or `path` is not set. It may be omitted only for a whole-file external reference (`path`
+set, `offset` not set), in which case the range runs to the end of the referenced file.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. It generalizes the storage-system eTag. The
+recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | – | – | set | current file, `[0, size)` (self-reference) |
+| – | – | set | set | current file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+`size` must be set whenever `offset` is set or `path` is not set, so a self-reference
+and any offset-based read always carry an explicit `size`. It may be omitted only for a
+whole-file external reference, where the range runs to the end of the referenced file.
Review Comment:
Update this to match.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3574036778
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | invalid |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | - | set | - | invalid |
+| – | - | - | set | invalid |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+`size` must be set whenever `offset` is set, so any offset-based read always carries an
+explicit `size`. A self-reference (`path` not set) must set `offset`, and therefore also
+`size`. `size` may be omitted only for a whole-file external reference, where the range
+runs to the end of the referenced file.
+
+A self-reference points within the same Parquet file using `offset` and `size` (both
+required); the bytes are written between column chunks and are not otherwise referenced by
Review Comment:
I don't think that we should specify where bytes must be written. Other restrictions apply, but it would be reasonable to place these just before the indexes and footer as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3546989116
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
Please read again:
> Every type defines precisely how their values are encoded and how they should be interpreted. I see no reason why FILE would be an exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3606837664
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
If I understand correctly `path` was not renamed to `uri` yet?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3605759083
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
Review Comment:
base64 seems to be shorter in terms of length, but hex does seem to be more standard for these. The hex length for sha-256 is 64 characters whereas base64 is 44 characters. Do we care about space usage here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3606005520
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | invalid |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | - | set | - | invalid |
+| – | - | - | set | invalid |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+`size` must be set whenever `offset` is set, so any offset-based read always carries an
+explicit `size`. A self-reference (`path` not set) must set `offset`, and therefore also
+`size`. `size` may be omitted only for a whole-file external reference, where the range
+runs to the end of the referenced file.
+
+A self-reference points within the same Parquet file using `offset` and `size` (both
+required); the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is when `path` is not set, never an absolute path back to
+the current file, so a file containing self-references is renamed or relocated as a
+single unit.
+
+The bytes referenced by a self-reference are compressed with the same `CompressionCodec`
+as the one specified for the `inline` column.
+
+#### Validation
+
+* A value must resolve to some referenced data. It resolves only if `inline`, `path`, or
+ `offset` is set; if none of them are set, the value does not resolve and is invalid, even
+ if `size` is set. Use column nullability to represent a null value.
+* A self-reference (`path` not set) must set `offset`. A value with neither `path` nor
+ `offset` set (and not `inline`) does not resolve and is invalid.
+* `size` must be set whenever `offset` is set. A value that sets `offset` without `size`
+ is invalid. Because a self-reference must set `offset`, it must also set `size`.
+* If `inline` is set, it supplies the bytes; producers may instead treat `inline` and the
+ locator fields as mutually exclusive.
+* Field names within a `FILE`-annotated group must not be renamed.
+* Additional metadata about the file (e.g., modification timestamp) must
+ be stored adjacent to this group by engines or table formats, not inside it.
+
+Statistics may be collected for the individual fields of a `FILE`-annotated group
Review Comment:
It should be treated like a struct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3605989612
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
Review Comment:
I don't think we should define how relative should be treated here. A table format may decide that it is relative to the parquet file itself or relative to the base path of a table. We're just saying that both forms (absolute and relative) are okay
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3605983983
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
Review Comment:
yep!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3390327658
##########
LogicalTypes.md:
##########
@@ -635,7 +635,72 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
+#### path
+
+An opaque path string to the referenced file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied. This is the only required field.
+
+#### size
+
+The length of the content in bytes. Must be zero or a positive integer if provided.
+A value of 0 indicates an empty file.
+
+#### offset
+
+A byte offset for range reads. If not provided, readers must treat the value as 0.
+If provided and non-zero, readers must seek to this offset and read `size` bytes.
+If `offset` is provided, `size` must also be provided.
+
+#### etag
+
+An eTag value provided by the storage system (e.g., from S3 or Azure Blob Storage).
+Can be used to detect whether the referenced file has been updated. If the reference
+points to a byte range within a file, the eTag applies to the entire file.
+
+Validation rules for readers and writers:
Review Comment:
Do we need a separate heading here? Seems like this should all be under `####Validation` not `etag`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3636193721
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
Given that not everybody will be reading the spec in detail, I think naming the field `uri` has a stronger chance of producing valid FILE columns :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3632321023
##########
LogicalTypes.md:
##########
@@ -635,7 +635,133 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within this file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name. Field IDs
+may also be used for projection. All fields are optional:
+
+| Field | Type | Required |
+|----------------|------------|----------|
+| `path` | STRING | No |
+| `offset` | INT64 | No |
+| `size` | INT64 | No |
+| `content_type` | STRING | No |
+| `checksum` | STRING | No |
+| `inline` | BYTE_ARRAY | No |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+##### path
+
+An opaque path string to an external file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied on top of the user provided data. If `path` is absent,
+the value refers to this file (a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not provided, readers must treat the value as 0.
+If provided and non-zero, readers must seek to this offset to retrieve the referenced data.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if provided.
+A value of 0 indicates empty referenced data. If not provided, the range runs to the end
+of the referenced data.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. It generalizes the storage-system eTag. The
+recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are present are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|----------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | – | external `path`, `[offset, EOF)` |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | – | set | – | this file, `[offset, EOF)` (self-reference) |
+| – | – | – | set | this file, `[0, size)` (self-reference) |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+A self-reference typically points within the same Parquet file using `offset` and
+`size`; the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is the absence of `path`, never an absolute path back to
+the current file, so a file containing self-references is renamed or relocated as a
+single unit.
+
+The referenced bytes are compressed with the same `CompressionCodec` as the one
+specified for the `inline` column.
Review Comment:
I removed this after more discussion. Most multimodal data is already compressed, e.g. jpeg, png, video formats, ... Inline data will already be compressed, the self-referenced data doesn't need to be compressed as well. Other formats don't do any additional compression
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3637539927
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
@brkyvz what is the argument against naming it `uri`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "sfc-gh-sgrafberger (via GitHub)" <gi...@apache.org>.
sfc-gh-sgrafberger commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3462769363
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
Review Comment:
A bit of background information for people not very familiar with unstructured data and semantic query processing yet:
When thinking about real-world use of unstructured data, there are many domains where mixing different modalities within one table column is quite natural.
E.g., once you look at the law domain, you might have court cases with different kinds of evidence: videos, images, pdf scans of police reports, etc. Other columns in the table might then be columns like "case_id", "case_name", "judge_id", "file_source", ... Other domain examples would be patient records in healthcare that mix medical imaging, pdf scans of written medical reports, prescriptions, etc. Or think about the kind of unstructured data a reporter might be working with.
Customers couldn't really do large-scale automated analysis of unstructured data before the rise of LLMs, and are now discovering step-by-step what is possible. Being able to efficiently answer basic questions like what kind of unstructured data you are dealing with sounds like a natural requirement to me to support currently emerging workloads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "divjotarora (via GitHub)" <gi...@apache.org>.
divjotarora commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3393836020
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
+#### Fields
+
+##### path
+
+An opaque path string to the referenced file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied. This is the only required field.
+
+##### size
+
+The length of the content in bytes. Must be zero or a positive integer if provided.
+A value of 0 indicates an empty file. If not provided, the length of the referenced
+content is unknown and the entirety of the content can be read.
+
+##### offset
+
+A byte offset indicating the start of a content slice within the referenced file.
+If not provided, readers must treat the value as 0.
+If provided and non-zero, readers must seek to this offset and read `size` bytes to retrieve the referenced data.
+If `offset` is provided, `size` must also be provided.
+
+##### etag
+
+An eTag value provided by the storage system (e.g., from S3 or Azure Blob Storage).
+Can be used to detect whether the referenced file has been updated. If the reference
+points to a byte range within a file, the eTag applies to the entire file.
+
+#### Validation
+
+* The `path` field is required and must be present. Readers must reject a `FILE`-annotated
+ group that does not contain `path`.
+* If `offset` is present and non-zero, `size` must also be provided.
+* Additional metadata about the file (e.g., content type, modification timestamp) should
Review Comment:
I think we should just specify that extra fields must not be added to this struct. @wgtmac wdyt of just saying:
> Additional metadata about the file (e.g. content type, modification timestamp) must not be stored inside this struct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3561877552
##########
LogicalTypes.md:
##########
@@ -635,7 +635,133 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within this file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name. Field IDs
+may also be used for projection. All fields are optional:
+
+| Field | Type | Required |
+|----------------|------------|----------|
+| `path` | STRING | No |
+| `offset` | INT64 | No |
+| `size` | INT64 | No |
+| `content_type` | STRING | No |
+| `checksum` | STRING | No |
+| `inline` | BYTE_ARRAY | No |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+##### path
+
+An opaque path string to an external file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied on top of the user provided data. If `path` is absent,
+the value refers to this file (a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not provided, readers must treat the value as 0.
+If provided and non-zero, readers must seek to this offset to retrieve the referenced data.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if provided.
+A value of 0 indicates empty referenced data. If not provided, the range runs to the end
+of the referenced data.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. It generalizes the storage-system eTag. The
+recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are present are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|----------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | – | external `path`, `[offset, EOF)` |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | – | set | – | this file, `[offset, EOF)` (self-reference) |
+| – | – | – | set | this file, `[0, size)` (self-reference) |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+A self-reference typically points within the same Parquet file using `offset` and
+`size`; the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is the absence of `path`, never an absolute path back to
+the current file, so a file containing self-references is renamed or relocated as a
+single unit.
+
+The referenced bytes are compressed with the same `CompressionCodec` as the one
+specified for the `inline` column.
Review Comment:
> Should we point out CompressionCodec can differ per page?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3610128858
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | invalid |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | - | set | - | invalid |
+| – | - | - | set | invalid |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+`size` must be set whenever `offset` is set, so any offset-based read always carries an
+explicit `size`. A self-reference (`path` not set) must set `offset`, and therefore also
+`size`. `size` may be omitted only for a whole-file external reference, where the range
+runs to the end of the referenced file.
+
+A self-reference points within the same Parquet file using `offset` and `size` (both
+required); the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is when `path` is not set, never an absolute path back to
+the current file, so a file containing self-references is renamed or relocated as a
+single unit.
+
+The bytes referenced by a self-reference are compressed with the same `CompressionCodec`
Review Comment:
I may have missed the relevant discussion on this so please correct me if I was wrong.
`CompressionCodec` defined by a `ColumnMetadata` is to compress all data pages of that column chunk in that specific row group. The description here means that we enforce all inline blocks referenced by this FILE column to always use the same `CompressionCodec` in the same row group? Is this too restrictive? Why can't we choose them to be uncompressed (like bloom filter) or compressed by a different scheme?
For other non-self-referenced cases, we can deduce compression codec based on `content_type` field, right? Shouldn't we do the same thing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3549459771
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
URIs can be relative.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3546348597
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
I also don't understand why you're pushing back against this. Mandating a precise definition won't hurt anyone, while leaving the definition ambiguous will definitely hurt downstream users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3540587787
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
Review Comment:
Added content_type and checksum
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3540732884
##########
LogicalTypes.md:
##########
@@ -635,7 +635,164 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name. Field IDs
+may also be used for projection. Every field is optional both in the schema and in the
+data: a writer may omit any field from the group definition, and any field that is
+present has a field repetition type of `OPTIONAL`. A group need only define the fields
+it uses (for example, an inline-only group may define just `inline`, and an external
+reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+An opaque path string to an external file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied on top of the user-provided data. If `path` is
+not set, the value refers to the current file (a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set
+or `path` is not set. It may be omitted only for a whole-file external reference (`path`
+set, `offset` not set), in which case the range runs to the end of the referenced file.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. It generalizes the storage-system eTag. The
+recognized algorithms are:
Review Comment:
```suggestion
recognized algorithms are (readers should ignore unknown algorithms):
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3634138549
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
@rok we weren't planning on renaming the field. We just added that the data inside should be a URI. That's why I left the name as `path`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3632350959
##########
LogicalTypes.md:
##########
@@ -635,7 +635,183 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty[1]). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+[1] Implementations are not expected to treat empty strings as null
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type), as defined by RFC 2046, of the resolved bytes (e.g., `image/png`).
+When not set, the type can be assumed as `application/octet-stream`.
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:<digest>`, where `<digest>` is encoded according to the `Encoding`
+column below. Readers should ignore unknown algorithms. The recognized algorithms
+are:
+
+| Algorithm | Encoding | Notes |
+|-----------|---------------|----------------------------------------------------------|
+| `ETAG` | opaque | the object-store eTag, not recomputable |
+| `MD5` | lowercase hex | as defined in RFC 6151 represented as 32 hex characters |
+| `CRC32` | lowercase hex | as defined in RFC 3385, represented as 8 hex characters |
+| `CRC32C` | lowercase hex | as defined in RFC 9260, represented as 8 hex characters |
+| `SHA-256` | lowercase hex | as defined in RFC 6234, represented as 64 hex characters |
+
+`<digest>` encodings are:
+
+* `lowercase hex`: the digest bytes rendered as lowercase hexadecimal, two
+ characters per byte and no separators (e.g. `MD5:d41d8cd98f00b204e9800998ecf8427e`).
+* `opaque`: the token supplied verbatim by the object store, used only for
+ equality comparison and not otherwise interpreted.
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | invalid |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | - | set | - | invalid |
+| – | - | - | set | invalid |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3632364223
##########
LogicalTypes.md:
##########
@@ -635,7 +635,183 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty[1]). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+[1] Implementations are not expected to treat empty strings as null
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type), as defined by RFC 2046, of the resolved bytes (e.g., `image/png`).
+When not set, the type can be assumed as `application/octet-stream`.
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:<digest>`, where `<digest>` is encoded according to the `Encoding`
+column below. Readers should ignore unknown algorithms. The recognized algorithms
+are:
+
+| Algorithm | Encoding | Notes |
+|-----------|---------------|----------------------------------------------------------|
+| `ETAG` | opaque | the object-store eTag, not recomputable |
+| `MD5` | lowercase hex | as defined in RFC 6151 represented as 32 hex characters |
+| `CRC32` | lowercase hex | as defined in RFC 3385, represented as 8 hex characters |
+| `CRC32C` | lowercase hex | as defined in RFC 9260, represented as 8 hex characters |
Review Comment:
done, thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3500070984
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
If it's meant to be a URI, can we mandate it and call the field "uri" instead?
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
+ * - size (INT64, optional): the length of the content in bytes; must be zero or positive
+ * - offset (INT64, optional): byte offset for range reads; if provided, size must also be provided
+ * - etag (STRING, optional): eTag from the storage system for staleness detection
Review Comment:
Isn't this a bit specific? Do some storage systems support something other than a ETag? Sooner or later, people will want to put other kinds of metadata, such as Last-Modified, Content-Type...
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
Why would a Parquet file reference itself recursively?
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
+#### Fields
+
+##### path
+
+An opaque path string to the referenced file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied. This is the only required field.
Review Comment:
The example shows a URI but this mandates no URI encoding??
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
+ * - size (INT64, optional): the length of the content in bytes; must be zero or positive
+ * - offset (INT64, optional): byte offset for range reads; if provided, size must also be provided
Review Comment:
Why would there be an offset here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3578561843
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
Review Comment:
A stupid issue: not set means it's NULL, and when not null, generally it should be non-empty RFC 3986 value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "sfc-gh-sgrafberger (via GitHub)" <gi...@apache.org>.
sfc-gh-sgrafberger commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3426902642
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
+#### Fields
+
+##### path
+
+An opaque path string to the referenced file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied. This is the only required field.
+
+##### size
+
+The length of the content in bytes. Must be zero or a positive integer if provided.
+A value of 0 indicates an empty file. If not provided, the length of the referenced
+content is unknown and the entirety of the content can be read.
+
+##### offset
+
+A byte offset indicating the start of a content slice within the referenced file.
+If not provided, readers must treat the value as 0.
+If provided and non-zero, readers must seek to this offset and read `size` bytes to retrieve the referenced data.
+If `offset` is provided, `size` must also be provided.
+
+##### etag
+
+An eTag value provided by the storage system (e.g., from S3 or Azure Blob Storage).
+Can be used to detect whether the referenced file has been updated. If the reference
+points to a byte range within a file, the eTag applies to the entire file.
+
+#### Validation
+
+* The `path` field is required and must be present. Readers must reject a `FILE`-annotated
+ group that does not contain `path`.
+* If `offset` is present and non-zero, `size` must also be provided.
+* Additional metadata about the file (e.g., content type, modification timestamp) should
Review Comment:
I also think that without optional metadata we risk non-interoperable implementations of those fields.
These additional fields are directly linked to the files and even specific versions of a file (see the `etag` field). If additional fields like `content_type` and `last_modified` are stored separately, they can get out of sync. If a file gets updated but related fields like `last_modified` are not updated in sync, it will lead to wrong results to any reader that relies on `last_modified` being in sync with the file version. If we had these additional fields as part of the file datatype itself, it would solve such problems by design: every engine can rely on such fields being up-to-date if they are present, and leave out fields it doesn't understand when writing file columns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3500816189
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
Embedding blobs in between column groups. This allows one to avoid blobs throwing off row-group sizing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3501496463
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
I think a self reference like this feels very awkward. If data is supposed to be inlined, I don't think it needs to live inside the File type in Parquet. An engine can decide to store data as an inline record or an external file reference. I think we should keep this format to external file references.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3580712295
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | invalid |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | - | set | - | invalid |
+| – | - | - | set | invalid |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+`size` must be set whenever `offset` is set, so any offset-based read always carries an
+explicit `size`. A self-reference (`path` not set) must set `offset`, and therefore also
+`size`. `size` may be omitted only for a whole-file external reference, where the range
+runs to the end of the referenced file.
+
+A self-reference points within the same Parquet file using `offset` and `size` (both
+required); the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is when `path` is not set, never an absolute path back to
+the current file, so a file containing self-references is renamed or relocated as a
+single unit.
+
+The bytes referenced by a self-reference are compressed with the same `CompressionCodec`
Review Comment:
The does not look right. Same column can have different codecs across different row groups. V2 pages can even decide whether to compress individually. Even if all row groups of the same column use the same codec, how can we tell whether self-referenced data is compressed or not?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "alkis (via GitHub)" <gi...@apache.org>.
alkis commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3578116292
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | invalid |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | - | set | - | invalid |
+| – | - | - | set | invalid |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+`size` must be set whenever `offset` is set, so any offset-based read always carries an
+explicit `size`. A self-reference (`path` not set) must set `offset`, and therefore also
+`size`. `size` may be omitted only for a whole-file external reference, where the range
+runs to the end of the referenced file.
+
+A self-reference points within the same Parquet file using `offset` and `size` (both
+required); the bytes are written between column chunks and are not otherwise referenced by
Review Comment:
Yes bytes can be placed anywhere. For all we care they could be in hidden thrift extensions in page headers, reader should be able to fetch them with offset+size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3574045612
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | invalid |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | - | set | - | invalid |
+| – | - | - | set | invalid |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+`size` must be set whenever `offset` is set, so any offset-based read always carries an
+explicit `size`. A self-reference (`path` not set) must set `offset`, and therefore also
+`size`. `size` may be omitted only for a whole-file external reference, where the range
+runs to the end of the referenced file.
+
+A self-reference points within the same Parquet file using `offset` and `size` (both
+required); the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is when `path` is not set, never an absolute path back to
Review Comment:
The phrase "never an absolute path back to the current file" sounds like you're saying "path must not be an absolute reference to the current file and must instead be unset". If that's true, I'd use a more clear phrasing to instruct people to replace self-references with `null`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "alkis (via GitHub)" <gi...@apache.org>.
alkis commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3578120501
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
Review Comment:
Good point @rdblue. Perhaps we can switch to hex if it is more familiar?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3550186899
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
I don't understand the push against uri. A path is a subset of URI, ARN is a URI, database path is an URI. URI comes with the contract that it is a a string identifying a resource. If the idea is to add a free form metadata field then path is not the correct name either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3550186899
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
I don't understand the push against uri. A path is a subset of URI, ARN is a URI, database path is an URI. It comes with a contract it's a reference to a resource. If the idea is to add a free form metadata field then path is not the correct name either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3550186899
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
I don't understand the push against uri. A path is a subset of URI, ARN is a URI, database path is an URI. URI comes with the contract that it identifies a resource. If the idea is to add a free form metadata field then path is not the correct name either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3546340925
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
> It's not parquet's job to enforce how users refer to their objects imho
Why not? Every type defines precisely how their values are encoded and how they should be interpreted. I see no reason why FILE would be an exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3546372471
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
FTR, I'll be -1 on any iteration of the spec that leaves the interpretation of the path ambiguous.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3560126291
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
Alright, will update this as URI. Thanks for pressing on this @pitrou
I wasn't aware ARNs were actually valid URIs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield merged PR #585:
URL: https://github.com/apache/parquet-format/pull/585
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3391709795
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
Review Comment:
After looking at the reference implementation I find this wording a bit confusing. The group "must" contain these fields, except it really only "must" contain `path`, and "may" contain the other three. I thought the `Required` column in the following table only referred to whether the field repetition was 'required' or 'optional'.
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
+#### Fields
+
+##### path
+
+An opaque path string to the referenced file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied. This is the only required field.
+
+##### size
+
+The length of the content in bytes. Must be zero or a positive integer if provided.
+A value of 0 indicates an empty file. If not provided, the length of the referenced
+content is unknown and the entirety of the content can be read.
+
+##### offset
+
+A byte offset indicating the start of a content slice within the referenced file.
+If not provided, readers must treat the value as 0.
+If provided and non-zero, readers must seek to this offset and read `size` bytes to retrieve the referenced data.
+If `offset` is provided, `size` must also be provided.
Review Comment:
Again, looking at the implementation in Rust, I don't see this enforced. What if the group definition is
```
optional group my_file (FILE) {
required binary path (STRING);
optional int64 offset;
}
```
I think we need to specify this at both the field and data levels, i.e. if the group contains an 'offset' field, it must also contain a 'size' field; if the optional 'offset' field is populated, it must be a non-negative integer, and a value for 'size' must also present.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3393072883
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
+#### Fields
+
+##### path
+
+An opaque path string to the referenced file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied. This is the only required field.
+
+##### size
+
+The length of the content in bytes. Must be zero or a positive integer if provided.
+A value of 0 indicates an empty file. If not provided, the length of the referenced
Review Comment:
Do we want to advise the reader to ignore negative value here?
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
+#### Fields
+
+##### path
+
+An opaque path string to the referenced file (e.g., `s3://bucket/file.jpg`). No special
Review Comment:
If this is opaque, should we remove `No special encoding (e.g., URI encoding) is applied`? It seems valid if users want to apply URI encoding.
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
Review Comment:
> identified by name
Do we need to mention its case sensitivity?
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
Review Comment:
```suggestion
The annotated group must contain the following fields, identified by name (not by order).
Field IDs (if exist) may also be used for projection:
```
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
+#### Fields
+
+##### path
+
+An opaque path string to the referenced file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied. This is the only required field.
+
+##### size
+
+The length of the content in bytes. Must be zero or a positive integer if provided.
+A value of 0 indicates an empty file. If not provided, the length of the referenced
+content is unknown and the entirety of the content can be read.
+
+##### offset
+
+A byte offset indicating the start of a content slice within the referenced file.
+If not provided, readers must treat the value as 0.
+If provided and non-zero, readers must seek to this offset and read `size` bytes to retrieve the referenced data.
+If `offset` is provided, `size` must also be provided.
+
+##### etag
+
+An eTag value provided by the storage system (e.g., from S3 or Azure Blob Storage).
+Can be used to detect whether the referenced file has been updated. If the reference
+points to a byte range within a file, the eTag applies to the entire file.
+
+#### Validation
+
+* The `path` field is required and must be present. Readers must reject a `FILE`-annotated
+ group that does not contain `path`.
+* If `offset` is present and non-zero, `size` must also be provided.
+* Additional metadata about the file (e.g., content type, modification timestamp) should
Review Comment:
Why adding this restriction? Do we want to enforce Parquet reader implementations to validate against this rule? I can foresee that users may use another group type (a.k.a struct) to wrap a `FILE` type and a `VARIANT` type for extra metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "sfc-gh-sgrafberger (via GitHub)" <gi...@apache.org>.
sfc-gh-sgrafberger commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3460832583
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
Review Comment:
One argument from the Parquet sync last week against this concern was that most engines anyway just infer content_type based on the file extension. However, this spec PR includes an `offset` field and says a reader should "seek to this offset and read size bytes to retrieve the referenced data" within the referenced file. But this means that a single physical container file can pack byte ranges for multiple logical files, and path then points to the container, not the logical entry.
In that case the extension of path tells you nothing about the modality of a given slice. The container can hold images, audio, and documents at different offsets. Extension-based inference only works when one path = one homogeneous file, which offset/size explicitly break. That leaves either byte-sniffing each slice at query time, or carrying the type in side metadata.
The second option seems to be something we want to avoid: a reader can read the FILE struct but still can't tell how to interpret the bytes at offset/size without additional knowledge, which breaks the "if you can read the type, you can understand everything in the type" goal.
So it seems to me that offset/size make content_type necessary, at least whenever offset is non-zero, for the type to stay self-describing. Is there something I'm missing that keeps the bytes interpretable without it? Or is the officially recommended path to always do byte sniffing at query runtime?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3605807621
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | invalid |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | - | set | - | invalid |
+| – | - | - | set | invalid |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+`size` must be set whenever `offset` is set, so any offset-based read always carries an
+explicit `size`. A self-reference (`path` not set) must set `offset`, and therefore also
+`size`. `size` may be omitted only for a whole-file external reference, where the range
+runs to the end of the referenced file.
+
+A self-reference points within the same Parquet file using `offset` and `size` (both
+required); the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is when `path` is not set, never an absolute path back to
+the current file, so a file containing self-references is renamed or relocated as a
+single unit.
+
+The bytes referenced by a self-reference are compressed with the same `CompressionCodec`
+as the one specified for the `inline` column.
+
+#### Validation
+
+* A value must resolve to some referenced data. It resolves only if `inline`, `path`, or
+ `offset` is set; if none of them are set, the value does not resolve and is invalid, even
+ if `size` is set. Use column nullability to represent a null value.
+* A self-reference (`path` not set) must set `offset`. A value with neither `path` nor
+ `offset` set (and not `inline`) does not resolve and is invalid.
+* `size` must be set whenever `offset` is set. A value that sets `offset` without `size`
+ is invalid. Because a self-reference must set `offset`, it must also set `size`.
Review Comment:
Ideally fail at write time, but is there a precedent for enforcing that? From your past comments it seems like we don't necessarily enforce contents of columns. Invalid files can be treated as `null` files in my opinion if we can't enforce it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3606355373
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | invalid |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | - | set | - | invalid |
+| – | - | - | set | invalid |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+`size` must be set whenever `offset` is set, so any offset-based read always carries an
+explicit `size`. A self-reference (`path` not set) must set `offset`, and therefore also
+`size`. `size` may be omitted only for a whole-file external reference, where the range
+runs to the end of the referenced file.
+
+A self-reference points within the same Parquet file using `offset` and `size` (both
+required); the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is when `path` is not set, never an absolute path back to
+the current file, so a file containing self-references is renamed or relocated as a
+single unit.
+
+The bytes referenced by a self-reference are compressed with the same `CompressionCodec`
Review Comment:
reworded to
```
The bytes referenced by a self-reference use the `CompressionCodec`
defined by the `inline` column chunk's `ColumnMetadata`.
```
is that better @wgtmac ?
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
Review Comment:
Added RFCs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #585:
URL: https://github.com/apache/parquet-format/pull/585#issuecomment-5051740470
@brkyvz can you reply on the [path thread](https://github.com/apache/parquet-format/pull/585#discussion_r3606837664)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on PR #585:
URL: https://github.com/apache/parquet-format/pull/585#issuecomment-5060378553
@rok @pitrou Renamed `path` to `uri`!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3501503397
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
+ * - size (INT64, optional): the length of the content in bytes; must be zero or positive
+ * - offset (INT64, optional): byte offset for range reads; if provided, size must also be provided
Review Comment:
if you're pointing to a slice of a file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3550802077
##########
LogicalTypes.md:
##########
@@ -635,7 +635,133 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within this file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name. Field IDs
+may also be used for projection. All fields are optional:
+
+| Field | Type | Required |
+|----------------|------------|----------|
+| `path` | STRING | No |
+| `offset` | INT64 | No |
+| `size` | INT64 | No |
+| `content_type` | STRING | No |
+| `checksum` | STRING | No |
+| `inline` | BYTE_ARRAY | No |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+##### path
+
+An opaque path string to an external file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied on top of the user provided data. If `path` is absent,
+the value refers to this file (a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not provided, readers must treat the value as 0.
+If provided and non-zero, readers must seek to this offset to retrieve the referenced data.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if provided.
+A value of 0 indicates empty referenced data. If not provided, the range runs to the end
+of the referenced data.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. It generalizes the storage-system eTag. The
+recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are present are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|----------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | – | external `path`, `[offset, EOF)` |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | – | set | – | this file, `[offset, EOF)` (self-reference) |
+| – | – | – | set | this file, `[0, size)` (self-reference) |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+A self-reference typically points within the same Parquet file using `offset` and
+`size`; the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is the absence of `path`, never an absolute path back to
+the current file, so a file containing self-references is renamed or relocated as a
+single unit.
+
+The referenced bytes are compressed with the same `CompressionCodec` as the one
+specified for the `inline` column.
Review Comment:
Should we point out `CompressionCodec` can differ per page?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3546622258
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
@pitrou Engines can build support for handling URIs out of the box, but I'm curious why you're so opposed to users being able to do their own things with UDFs in engines. You're asking to block out potential use cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3540721969
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,35 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to a file, or to a range of
+ * bytes that may be stored inline, elsewhere in this file, or in an external
+ * file. All fields are optional and are identified by name:
Review Comment:
Could we clarify terminology of:
- set - I assume this means non-null value?
- absent - Does this mean not set or the column is missing
- omitted - Does this mean not set or the column is missing
- optional - (Does this mean they have repetition level optional?
I think these might be more clearly defined in logical types. Maybe just point readers to logical_types.md for a the full definition to avoid too much duplication in case things change in one place or the other?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3540134684
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,33 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to a file, or to a range of
+ * bytes that may be stored inline, elsewhere in this file, or in an external
+ * file. All fields are optional and are identified by name:
+ * - path (STRING): an opaque location string (e.g. s3://bucket/file.jpg);
+ * if absent, the value refers to this file (self-reference)
+ * - offset (INT64): start of the byte range; if absent, treated as 0
+ * - size (INT64): byte length of the referenced data; if absent, the range runs to
+ * the end of the referenced data
Review Comment:
Wouldn't this be when `path` is not set (i.e. absent)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3632352610
##########
LogicalTypes.md:
##########
@@ -635,7 +635,183 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty[1]). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+[1] Implementations are not expected to treat empty strings as null
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type), as defined by RFC 2046, of the resolved bytes (e.g., `image/png`).
+When not set, the type can be assumed as `application/octet-stream`.
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:<digest>`, where `<digest>` is encoded according to the `Encoding`
+column below. Readers should ignore unknown algorithms. The recognized algorithms
+are:
+
+| Algorithm | Encoding | Notes |
+|-----------|---------------|----------------------------------------------------------|
+| `ETAG` | opaque | the object-store eTag, not recomputable |
+| `MD5` | lowercase hex | as defined in RFC 6151 represented as 32 hex characters |
+| `CRC32` | lowercase hex | as defined in RFC 3385, represented as 8 hex characters |
+| `CRC32C` | lowercase hex | as defined in RFC 9260, represented as 8 hex characters |
+| `SHA-256` | lowercase hex | as defined in RFC 6234, represented as 64 hex characters |
+
+`<digest>` encodings are:
+
+* `lowercase hex`: the digest bytes rendered as lowercase hexadecimal, two
+ characters per byte and no separators (e.g. `MD5:d41d8cd98f00b204e9800998ecf8427e`).
+* `opaque`: the token supplied verbatim by the object store, used only for
+ equality comparison and not otherwise interpreted.
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | invalid |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | - | set | - | invalid |
+| – | - | - | set | invalid |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+`size` must be set whenever `offset` is set, so any offset-based read always carries an
+explicit `size`. A self-reference (`path` not set) must set `offset`, and therefore also
+`size`. `size` may be omitted only for a whole-file external reference, where the range
+runs to the end of the referenced file.
+
+A self-reference points within the same Parquet file using `offset` and `size` (both
+required). A self-reference is when `path` is not set. A file containing self-references
+can be renamed or relocated as a single unit.
+
+The bytes referenced by a self-reference use the `CompressionCodec`
+defined by the `inline` column chunk's `ColumnMetadata`.
+
+#### Validation
+
+* A value must resolve to some referenced data. It resolves only if `inline`, `path`, or
+ `offset` is set; if none of them are set, the value does not resolve and is invalid, even
+ if `size` is set.
+* A self-reference (`path` not set) must set `offset`. A value with neither `path` nor
+ `offset` set (and not `inline`) does not resolve and is invalid.
+* `size` must be set whenever `offset` is set. A value that sets `offset` without `size`
+ is invalid. Because a self-reference must set `offset`, it must also set `size`.
+* If `inline` is set, it supplies the bytes readers; producers may treat `inline` and the
Review Comment:
added `for readers`. There was a comment before around reader confusion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3606116415
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
Review Comment:
If we cared about length, why would we use binary?
I think if we're using a string representation, then we should go with the most human-friendly form, which is the hex string most people are used to. That is also good if you don't only have machine validation of the checksum or etag.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3431157040
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
Review Comment:
I actually brought this up today in the Parquet sync, it's the odd one out of this pack of properties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "alkis (via GitHub)" <gi...@apache.org>.
alkis commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3435643120
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
Review Comment:
I concur. The type should be minimal. Also we should make all fields optional and define the semantics of them missing:
- `path`: the file path if present, otherwise this is the path of the current file
- `offset`: the start of the binary data, otherwise 0
- `size`: the size of the binary data, otherwise the length of the file. `size`, given or derived, is capped such that `offset + size <= file_size`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3545989695
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,35 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to a file, or to a range of
+ * bytes that may be stored inline, elsewhere in this file, or in an external
+ * file. All fields are optional and are identified by name:
+ * - path (STRING): an opaque location string (e.g. s3://bucket/file.jpg);
+ * if absent, the value refers to this file (self-reference)
+ * - offset (INT64): start of the byte range; if absent, treated as 0
+ * - size (INT64): byte length of the referenced data; required when `offset` is set
+ * or `path` is absent. May be omitted only for a whole-file external reference
Review Comment:
what if you have a mutable external file? Knowing that you want to avoid putting in the size, so that you always fetch the latest size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3546362450
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
I've heard of use cases where people reference objects using AWS ARNs which isn't a URI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "alkis (via GitHub)" <gi...@apache.org>.
alkis commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3561789828
##########
LogicalTypes.md:
##########
@@ -635,7 +635,133 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within this file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name. Field IDs
+may also be used for projection. All fields are optional:
+
+| Field | Type | Required |
+|----------------|------------|----------|
+| `path` | STRING | No |
+| `offset` | INT64 | No |
+| `size` | INT64 | No |
+| `content_type` | STRING | No |
+| `checksum` | STRING | No |
+| `inline` | BYTE_ARRAY | No |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+##### path
+
+An opaque path string to an external file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied on top of the user provided data. If `path` is absent,
+the value refers to this file (a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not provided, readers must treat the value as 0.
+If provided and non-zero, readers must seek to this offset to retrieve the referenced data.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if provided.
+A value of 0 indicates empty referenced data. If not provided, the range runs to the end
+of the referenced data.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. It generalizes the storage-system eTag. The
+recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are present are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|----------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | – | external `path`, `[offset, EOF)` |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | – | set | – | this file, `[offset, EOF)` (self-reference) |
+| – | – | – | set | this file, `[0, size)` (self-reference) |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+A self-reference typically points within the same Parquet file using `offset` and
+`size`; the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is the absence of `path`, never an absolute path back to
+the current file, so a file containing self-references is renamed or relocated as a
+single unit.
+
+The referenced bytes are compressed with the same `CompressionCodec` as the one
+specified for the `inline` column.
Review Comment:
The reason is because we are making the assumption that the type of data stored in the file column is homogeneous. I argue that's a good assumption. If this column contains text, there is little reason to assume it will be a string when small and an image when large. Ergo if the writer chose to compress the column it should compress the self-referenced packed blobs too.
I wanted to make all packed references compressed as well but there was some pushback for that with the argument that external may be referenced by different parquet files and it may choose its own compression scheme (or none at all). I can see arguments both ways - it is a tradeoff.
Making all packed references inherit the CompressionCodec makes the spec more consistent.
Making only self-references inherit the CompressionCodec is friendlier to shared external references.
We pick one of the two, it is a tradeoff.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3582883813
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
Review Comment:
I think I would just add this to the table and use whatever is normal for the use case. I think most of these use hex. What was the rationale for base64?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "sfc-gh-sgrafberger (via GitHub)" <gi...@apache.org>.
sfc-gh-sgrafberger commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3462769363
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
Review Comment:
A bit of background information for people not very familiar with unstructured data and semantic query processing yet:
When thinking about real-world use of unstructured data, there are many domains where mixing different modalities within one table column is quite natural.
E.g., once you look at the law domain, you might have court cases with different kinds of evidence: videos, images, pdf scans of police reports, etc. Other columns in the table might then be columns like "case_id", "case_name", "judge_id", "file_source", ... Other domain examples would be patient records in healthcare that mix medical imaging, pdf scans of written medical reports, prescriptions, etc. Or think about the kind of unstructured data a reporter might be working with.
Users couldn't really do large-scale automated analysis of unstructured data before the rise of LLMs, and are now discovering step-by-step what is possible. Being able to efficiently answer basic questions like what kind of unstructured data you are dealing with sounds like a natural requirement to me to support currently emerging workloads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "sfc-gh-sgrafberger (via GitHub)" <gi...@apache.org>.
sfc-gh-sgrafberger commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3462769363
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
Review Comment:
A bit of background information for people not very familiar with semantic query processing and unstructured data yet:
When thinking about real-world use of unstructured data, there are many domains where mixing different modalities within one table column is quite natural.
E.g., once you look at the law domain, you might have court cases with different kinds of evidence: videos, images, pdf scans of police reports, etc. Other columns in the table might then be columns like "case_id", "case_name", "judge_id", "file_source", ... Other domain examples would be patient records in healthcare that mix medical imaging, pdf scans of written medical reports, prescriptions, etc. Or think about the kind of unstructured data a reporter might be working with.
DB users couldn't really do large-scale automated analysis of unstructured data before the rise of LLMs, and are now discovering step-by-step what is possible. Being able to efficiently answer basic questions like what kind of unstructured data you are dealing with sounds like a natural requirement to me to support currently emerging workloads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3500889126
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
Doesn't this require a new version of Parquet? I'm thinking specifically of how you copy a parquet file by reading and writing it if there are new blob references in between column groups you don't understand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3399093831
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
Review Comment:
I think that using "identified by name" conflicts with the next sentence. I would remove both. You don't need to state how these are identified.
If you want to be more clear, then you could add requirements like not renaming the fields. You could also say that the fields can be reordered, which would also remove the ability to project by order (which would be weird anyway).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "sfc-gh-nsharma (via GitHub)" <gi...@apache.org>.
sfc-gh-nsharma commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3404593056
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
+#### Fields
+
+##### path
+
+An opaque path string to the referenced file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied. This is the only required field.
+
+##### size
+
+The length of the content in bytes. Must be zero or a positive integer if provided.
+A value of 0 indicates an empty file. If not provided, the length of the referenced
+content is unknown and the entirety of the content can be read.
+
+##### offset
+
+A byte offset indicating the start of a content slice within the referenced file.
+If not provided, readers must treat the value as 0.
+If provided and non-zero, readers must seek to this offset and read `size` bytes to retrieve the referenced data.
+If `offset` is provided, `size` must also be provided.
+
+##### etag
+
+An eTag value provided by the storage system (e.g., from S3 or Azure Blob Storage).
+Can be used to detect whether the referenced file has been updated. If the reference
+points to a byte range within a file, the eTag applies to the entire file.
+
+#### Validation
+
+* The `path` field is required and must be present. Readers must reject a `FILE`-annotated
+ group that does not contain `path`.
+* If `offset` is present and non-zero, `size` must also be provided.
+* Additional metadata about the file (e.g., content type, modification timestamp) should
Review Comment:
How are we thinking about various engines adding very common fields like content type, modification timestamp? If they are not part of the parquet file format, I suspect there will be lot of non-interoperable implementations with those fields. Do we expect this file type to evolve to support those in the future?
e.g. these two definitions won't be interoperable because they wrap the FILE struct differently. It would be easier/less error-prone if data type allowed extra fields or a variant to capture additional attributes so that engines don't need a wrapper on top of the file object.
```
FileEngine1 (outer struct) {
file_ref FILE
content_type STRING
last_modified TIMESTAMP
}
FileEngine2 (outer struct) {
fileRef FILE
mimeType STRING
lastModified TIMESTAMP
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "sfc-gh-sgrafberger (via GitHub)" <gi...@apache.org>.
sfc-gh-sgrafberger commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3402334755
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
Review Comment:
I really think we should include `content_type` as field here. I fully understand that we don't want to have all kinds of optional metadata like `last_modified` as part of the spec, as this optional metadata is only needed for a smaller subset of queries and differs between different engines. However, the `content_type` is required information for a large fraction of common AI/ML workloads, and there is a good reason why existing FILE datatypes in engines like BigQuery or Snowflake have it as part of the datatype.
Many existing AI and ML-related functions on FILEs only work for certain modalities. E.g., an `AI_TRANSCRIBE` function only makes sense for audio and video data. Many common ML preprocessing techniques/functions only work for certain modalities, e.g., downscaling the resolution of images, turning image colours into grey scale, cropping and rotating pictures, downsampling the fps of a video etc. Some AI models only support text data, while others also support modalities like images. Routing AI function calls to compatible models critically depends on modalities, conceptually. If this isn't part of the datatype, a large fraction of FILE usecases would have to repeatedly read the first n bytes of a file to try to infer the modality, resulting in a big, unnecessary performance overhead that could be easily avoided.
The main reason to add a FILE datatype to the spec is to better support new Gen AI workloads with unstructured data. I think not having a `content_type` field would critically impact the user experience for the intended main usecase of FILE.
Can we please reconsider adding `content_type`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3501508360
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
+ * - size (INT64, optional): the length of the content in bytes; must be zero or positive
+ * - offset (INT64, optional): byte offset for range reads; if provided, size must also be provided
+ * - etag (STRING, optional): eTag from the storage system for staleness detection
Review Comment:
We chose ETag since it's a standard HTTP convention. I do think content-type makes sense as well, but we had removed it and maybe are considering adding it back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3501066243
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
I don't think this would be a forward-incompatible change. The FILE type carries no guarantees about the lifecycle of the content it tracks. The file this content lives in may be deleted out from under the reference.
Allowing a self-reference like this doesn't carry a requirement to preserve blocks within a Parquet file and rewrite FILE references when carrying them forward. That would conflict with the "no guarantee" point above and I don't think anyone has (or would) suggest adding this as a requirement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3556813316
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
> I think it's reasonable to clarify that we expect the location to be a URI, but are you saying that implementations must parse and validate the contents of the column?
We are talking about the spec, not what implementations do. The STRING type is specified as UTF8. Putting non-UTF8 data in a STRING column would be invalid Parquet. Readers will probably not UTF8-validate the contents, they assume the writer is well-behaved.
This is the same here: we can specify this to be a URI and let readers decide if they want to be strict about it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "sfc-gh-nsharma (via GitHub)" <gi...@apache.org>.
sfc-gh-nsharma commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3556378958
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
I still don't see why parquet path should specify how relative path should be resolved. I understand the argument behind it being a URI but not how path resolution should work (relative to table base or file itself)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3606005520
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | invalid |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | - | set | - | invalid |
+| – | - | - | set | invalid |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+`size` must be set whenever `offset` is set, so any offset-based read always carries an
+explicit `size`. A self-reference (`path` not set) must set `offset`, and therefore also
+`size`. `size` may be omitted only for a whole-file external reference, where the range
+runs to the end of the referenced file.
+
+A self-reference points within the same Parquet file using `offset` and `size` (both
+required); the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is when `path` is not set, never an absolute path back to
+the current file, so a file containing self-references is renamed or relocated as a
+single unit.
+
+The bytes referenced by a self-reference are compressed with the same `CompressionCodec`
+as the one specified for the `inline` column.
+
+#### Validation
+
+* A value must resolve to some referenced data. It resolves only if `inline`, `path`, or
+ `offset` is set; if none of them are set, the value does not resolve and is invalid, even
+ if `size` is set. Use column nullability to represent a null value.
+* A self-reference (`path` not set) must set `offset`. A value with neither `path` nor
+ `offset` set (and not `inline`) does not resolve and is invalid.
+* `size` must be set whenever `offset` is set. A value that sets `offset` without `size`
+ is invalid. Because a self-reference must set `offset`, it must also set `size`.
+* If `inline` is set, it supplies the bytes; producers may instead treat `inline` and the
+ locator fields as mutually exclusive.
+* Field names within a `FILE`-annotated group must not be renamed.
+* Additional metadata about the file (e.g., modification timestamp) must
+ be stored adjacent to this group by engines or table formats, not inside it.
+
+Statistics may be collected for the individual fields of a `FILE`-annotated group
Review Comment:
It should be treated like a struct. So the statistics can be tracked for the fields inside it as leaf columns
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #585:
URL: https://github.com/apache/parquet-format/pull/585#issuecomment-5059255400
> > @brkyvz can you reply on the [path thread](https://github.com/apache/parquet-format/pull/585#discussion_r3606837664)?
>
> @rok sorry that was hidden in the UI. Responded! We weren't planning on renaming the field. Just explaining that it is a URI
See my reply in the thread above. I don't think "we weren't planning" sufficiently answers this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3545980655
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,35 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to a file, or to a range of
+ * bytes that may be stored inline, elsewhere in this file, or in an external
+ * file. All fields are optional and are identified by name:
Review Comment:
Yeah, let me remove most of the definition here and just point to `logical_types.md` to avoid duplication and potential divergence
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3540741001
##########
LogicalTypes.md:
##########
@@ -635,7 +635,164 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name. Field IDs
+may also be used for projection. Every field is optional both in the schema and in the
+data: a writer may omit any field from the group definition, and any field that is
+present has a field repetition type of `OPTIONAL`. A group need only define the fields
+it uses (for example, an inline-only group may define just `inline`, and an external
+reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+An opaque path string to an external file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied on top of the user-provided data. If `path` is
+not set, the value refers to the current file (a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set
+or `path` is not set. It may be omitted only for a whole-file external reference (`path`
+set, `offset` not set), in which case the range runs to the end of the referenced file.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. It generalizes the storage-system eTag. The
+recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | – | – | set | current file, `[0, size)` (self-reference) |
Review Comment:
Bytes 0-3 of an inline file should always be `PAR1` or the relevant magic number of the parquet file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3549523308
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
To answer the "it can be left to engines" argument: this is fine for closed environments where the Parquet file never leaves some company cloud of sorts (and even there, several Parquet implementations may be involved).
But Parquet files are a widespread way of publishing and sharing columnar data. If FILE columns cannot be interpreted unambiguously, then ingesting a Parquet file needs additional hands-holding from the user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3550186899
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
I don't understand the push against uri. A path is a subset of URI, ARN is a URI, database path is an URI comes with the contract that it identifies a resource. If the idea is to add a free form metadata field then path is not the correct name either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3550791817
##########
LogicalTypes.md:
##########
@@ -635,7 +635,172 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+An opaque path string to an external file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied on top of the user-provided data. If `path` is
+not set, the value refers to the current file (a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set).
Review Comment:
Perhaps worth specifying?
```suggestion
external reference (`path` set). `offset must not be < 0.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "divjotarora (via GitHub)" <gi...@apache.org>.
divjotarora commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3393882322
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
+#### Fields
+
+##### path
+
+An opaque path string to the referenced file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied. This is the only required field.
+
+##### size
+
+The length of the content in bytes. Must be zero or a positive integer if provided.
+A value of 0 indicates an empty file. If not provided, the length of the referenced
+content is unknown and the entirety of the content can be read.
+
+##### offset
Review Comment:
Should we specify that it must be non-negative?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "divjotarora (via GitHub)" <gi...@apache.org>.
divjotarora commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3393840657
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
+#### Fields
+
+##### path
+
+An opaque path string to the referenced file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied. This is the only required field.
+
+##### size
+
+The length of the content in bytes. Must be zero or a positive integer if provided.
+A value of 0 indicates an empty file. If not provided, the length of the referenced
Review Comment:
IMO we should specify that readers error on invalid values, not silently ignore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3399122146
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
+#### Fields
+
+##### path
+
+An opaque path string to the referenced file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied. This is the only required field.
+
+##### size
+
+The length of the content in bytes. Must be zero or a positive integer if provided.
+A value of 0 indicates an empty file. If not provided, the length of the referenced
+content is unknown and the entirety of the content can be read.
+
+##### offset
+
+A byte offset indicating the start of a content slice within the referenced file.
+If not provided, readers must treat the value as 0.
+If provided and non-zero, readers must seek to this offset and read `size` bytes to retrieve the referenced data.
+If `offset` is provided, `size` must also be provided.
+
+##### etag
+
+An eTag value provided by the storage system (e.g., from S3 or Azure Blob Storage).
+Can be used to detect whether the referenced file has been updated. If the reference
+points to a byte range within a file, the eTag applies to the entire file.
Review Comment:
What should writers do when there is no etag provided by the system, like in a local FS? What about systems that have checksums but don't call them etags?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "sfc-gh-sgrafberger (via GitHub)" <gi...@apache.org>.
sfc-gh-sgrafberger commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3402334755
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
Review Comment:
I really think we should include `content_type` as field here. I fully understand that we don't want to have all kinds of optional metadata like `last_modified` as part of the spec, as this optional metadata is only needed for a smaller subset of queries and differs between different engines. However, the `content_type` is required information for a large fraction of common AI/ML workloads, and there is a good reason why existing FILE datatypes in engines like BigQuery or Snowflake have it as part of the datatype.
Many existing AI and ML-related functions on FILEs only work for certain modalities. E.g., an `AI_TRANSCRIBE` function only makes sense for audio and video data. Many common ML preprocessing techniques/functions only work for certain modalities, e.g., downscaling the resolution of images, turning image colours into grey scale, cropping and rotating pictures, downsampling the fps of a video etc. Some AI models only support text data, while others also support modalities like images. Routing AI function calls to compatible models critically depends on modalities, conceptually.
The main reason to add a FILE datatype to the spec is to better support new Gen AI workloads with unstructured data. I think not having a `content_type` field would critically impact the user experience for the intended main usecase of FILE.
Can we please reconsider adding `content_type`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3483819042
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
Review Comment:
+1 for Russell's argument here. I think it is valuable to be able to filter on type and if we are compacting multiple blobs into a container file we wouldn't be able to rely on extension alone. As long as we have guidelines for behavior so that this is useful across engines, I'm good including this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3430869570
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
Review Comment:
I'm neutral on whether the Parquet spec should stay minimal or include convenient metadata (content type, metadata map etc.) for common use cases
But if we are leaning toward keeping the spec simple, I don't think `etag` should be part of the spec either. `etag` feels more like storage specific metadata than a portable field with consistent semantics across storage systems
Would be happy to hear more thoughts on this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3540138852
##########
LogicalTypes.md:
##########
@@ -635,7 +635,133 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within this file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name. Field IDs
+may also be used for projection. All fields are optional:
+
+| Field | Type | Required |
+|----------------|------------|----------|
+| `path` | STRING | No |
+| `offset` | INT64 | No |
+| `size` | INT64 | No |
+| `content_type` | STRING | No |
+| `checksum` | STRING | No |
+| `inline` | BYTE_ARRAY | No |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+##### path
+
+An opaque path string to an external file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied on top of the user provided data. If `path` is absent,
+the value refers to this file (a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not provided, readers must treat the value as 0.
+If provided and non-zero, readers must seek to this offset to retrieve the referenced data.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if provided.
+A value of 0 indicates empty referenced data. If not provided, the range runs to the end
Review Comment:
Here too, I think "if `path` is not set or `offset` is set"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3574012296
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
Review Comment:
Minor: would be nice to have a reference like the URI reference. I thought that we also wanted to have an assumed MIME type, `application/octet-stream`? I'm fine if it isn't there, but I remember it coming up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3550186899
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
I don't understand the push against uri. A path is a subset of URI, ARN is a URI, database path is an URI. It comes with a contract that it identifies reference to a resource. If the idea is to add a free form metadata field then path is not the correct name either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3546303957
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
I don't think a URI should be mandated. It's not parquet's job to enforce how users refer to their objects imho
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3546362450
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
I've heard of use cases where people reference objects using AWS ARNs which isn't a URI. I don't want to mandate them to change their company policy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "divjotarora (via GitHub)" <gi...@apache.org>.
divjotarora commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3393844105
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
+#### Fields
+
+##### path
+
+An opaque path string to the referenced file (e.g., `s3://bucket/file.jpg`). No special
Review Comment:
I think this is saying that implementations must not apply encoding on top of what the user provides. We should clarify the wording
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3399125350
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
+#### Fields
+
+##### path
+
+An opaque path string to the referenced file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied. This is the only required field.
+
+##### size
+
+The length of the content in bytes. Must be zero or a positive integer if provided.
+A value of 0 indicates an empty file. If not provided, the length of the referenced
+content is unknown and the entirety of the content can be read.
+
+##### offset
+
+A byte offset indicating the start of a content slice within the referenced file.
+If not provided, readers must treat the value as 0.
+If provided and non-zero, readers must seek to this offset and read `size` bytes to retrieve the referenced data.
+If `offset` is provided, `size` must also be provided.
+
+##### etag
+
+An eTag value provided by the storage system (e.g., from S3 or Azure Blob Storage).
+Can be used to detect whether the referenced file has been updated. If the reference
+points to a byte range within a file, the eTag applies to the entire file.
+
+#### Validation
+
+* The `path` field is required and must be present. Readers must reject a `FILE`-annotated
+ group that does not contain `path`.
Review Comment:
So this makes a file completely unreadable? 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3399111940
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
+#### Fields
+
+##### path
+
+An opaque path string to the referenced file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied. This is the only required field.
+
+##### size
+
+The length of the content in bytes. Must be zero or a positive integer if provided.
+A value of 0 indicates an empty file. If not provided, the length of the referenced
Review Comment:
We can't require readers to fail, we can only say what it means when the length is -1. Failure is a responsibility of whatever uses this data, and it's even hard to require that component to fail because users don't want their query to fail for one bad row. They expect reasonable fallback behavior.
Also, does 0 indicate an empty file or does it indicate 0-length content? With `offset`, I think this indicates that the content length is 0, not that the underlying file is empty.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3390152490
##########
LogicalTypes.md:
##########
@@ -635,7 +635,72 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
+#### path
+
+An opaque path string to the referenced file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied. This is the only required field.
+
+#### size
+
+The length of the content in bytes. Must be zero or a positive integer if provided.
+A value of 0 indicates an empty file.
+
+#### offset
+
+A byte offset for range reads. If not provided, readers must treat the value as 0.
+If provided and non-zero, readers must seek to this offset and read `size` bytes.
Review Comment:
```suggestion
If provided and non-zero, readers must seek to this offset and read `size` bytes to retrieve the referenced data.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3390154859
##########
LogicalTypes.md:
##########
@@ -635,7 +635,72 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
+#### path
+
+An opaque path string to the referenced file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied. This is the only required field.
+
+#### size
+
+The length of the content in bytes. Must be zero or a positive integer if provided.
+A value of 0 indicates an empty file.
Review Comment:
maybe be explicit about semantics if not provided.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3617556394
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | invalid |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | - | set | - | invalid |
+| – | - | - | set | invalid |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+`size` must be set whenever `offset` is set, so any offset-based read always carries an
+explicit `size`. A self-reference (`path` not set) must set `offset`, and therefore also
+`size`. `size` may be omitted only for a whole-file external reference, where the range
+runs to the end of the referenced file.
+
+A self-reference points within the same Parquet file using `offset` and `size` (both
+required); the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is when `path` is not set, never an absolute path back to
+the current file, so a file containing self-references is renamed or relocated as a
+single unit.
+
+The bytes referenced by a self-reference are compressed with the same `CompressionCodec`
Review Comment:
> For other non-self-referenced cases, we can deduce compression codec based on content_type field, right? Shouldn't we do the same thing?
The intent of the field is to capture the content type of the data for filters like "find all PNG images". I don't expect this to describe compression because that wouldn't be very useful (all `application/zip`, for example).
I think the `content_type` field would behave like the [HTTP `Content-Type` header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Content-Type) that is the original media type before compression. HTTP uses a separate [`Content-Encoding` header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Content-Encoding) for compression.
This leaves the question of how to determine the equivalent of `Content-Encoding` for external references. We should probably spend some time considering that more.
> The description here means that we enforce all inline blocks referenced by this FILE column to always use the same CompressionCodec in the same row group? Is this too restrictive? Why can't we choose them to be uncompressed (like bloom filter) or compressed by a different scheme?
Yes, I think that reading is accurate. Looks like this is a way to determine compression, but only for blobs that are located within the Parquet file itself as a self-reference.
I don't think that this is too restrictive since generic compression is typically uniform for files. I don't expect people to customize the compression used for these. We want compression to be possible (mandating uncompressed doesn't seem like a good choice), but we don't need to track it per value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3632335122
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | invalid |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | - | set | - | invalid |
+| – | - | - | set | invalid |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+`size` must be set whenever `offset` is set, so any offset-based read always carries an
+explicit `size`. A self-reference (`path` not set) must set `offset`, and therefore also
+`size`. `size` may be omitted only for a whole-file external reference, where the range
+runs to the end of the referenced file.
+
+A self-reference points within the same Parquet file using `offset` and `size` (both
+required); the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is when `path` is not set, never an absolute path back to
+the current file, so a file containing self-references is renamed or relocated as a
+single unit.
+
+The bytes referenced by a self-reference are compressed with the same `CompressionCodec`
Review Comment:
After more discussion, I removed this requirement. Most multi-modal formats are already compressed and don't need further compression. If you want text to be compressed, you can use `inline`, self-referenced data can be just read without additional compression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3632390771
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | invalid |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | - | set | - | invalid |
+| – | - | - | set | invalid |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+`size` must be set whenever `offset` is set, so any offset-based read always carries an
+explicit `size`. A self-reference (`path` not set) must set `offset`, and therefore also
+`size`. `size` may be omitted only for a whole-file external reference, where the range
+runs to the end of the referenced file.
+
+A self-reference points within the same Parquet file using `offset` and `size` (both
+required); the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is when `path` is not set, never an absolute path back to
+the current file, so a file containing self-references is renamed or relocated as a
+single unit.
+
+The bytes referenced by a self-reference are compressed with the same `CompressionCodec`
+as the one specified for the `inline` column.
+
+#### Validation
+
+* A value must resolve to some referenced data. It resolves only if `inline`, `path`, or
+ `offset` is set; if none of them are set, the value does not resolve and is invalid, even
+ if `size` is set. Use column nullability to represent a null value.
+* A self-reference (`path` not set) must set `offset`. A value with neither `path` nor
+ `offset` set (and not `inline`) does not resolve and is invalid.
+* `size` must be set whenever `offset` is set. A value that sets `offset` without `size`
+ is invalid. Because a self-reference must set `offset`, it must also set `size`.
Review Comment:
Writers should fail, my opinion is that readers should return `null` as the file reference other than outright failing, otherwise it makes it impossible to read the rest of the data that may be valid in the file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3574055613
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | invalid |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | - | set | - | invalid |
+| – | - | - | set | invalid |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+`size` must be set whenever `offset` is set, so any offset-based read always carries an
+explicit `size`. A self-reference (`path` not set) must set `offset`, and therefore also
+`size`. `size` may be omitted only for a whole-file external reference, where the range
+runs to the end of the referenced file.
+
+A self-reference points within the same Parquet file using `offset` and `size` (both
+required); the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is when `path` is not set, never an absolute path back to
+the current file, so a file containing self-references is renamed or relocated as a
+single unit.
+
+The bytes referenced by a self-reference are compressed with the same `CompressionCodec`
+as the one specified for the `inline` column.
+
+#### Validation
+
+* A value must resolve to some referenced data. It resolves only if `inline`, `path`, or
+ `offset` is set; if none of them are set, the value does not resolve and is invalid, even
+ if `size` is set. Use column nullability to represent a null value.
+* A self-reference (`path` not set) must set `offset`. A value with neither `path` nor
+ `offset` set (and not `inline`) does not resolve and is invalid.
+* `size` must be set whenever `offset` is set. A value that sets `offset` without `size`
+ is invalid. Because a self-reference must set `offset`, it must also set `size`.
Review Comment:
How should implementations behave if `offset` is set but `size` is not? Are they required to fail?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3574023416
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
Review Comment:
An example or a few examples would help here because of the use of base64 in the first paragraph.
Are we really going to use base64 for formats that are normally hexadecimal like etags?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "alkis (via GitHub)" <gi...@apache.org>.
alkis commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3435605201
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
+#### Fields
+
+##### path
+
+An opaque path string to the referenced file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied. This is the only required field.
+
+##### size
+
+The length of the content in bytes. Must be zero or a positive integer if provided.
+A value of 0 indicates an empty file. If not provided, the length of the referenced
+content is unknown and the entirety of the content can be read.
+
+##### offset
+
+A byte offset indicating the start of a content slice within the referenced file.
+If not provided, readers must treat the value as 0.
+If provided and non-zero, readers must seek to this offset and read `size` bytes to retrieve the referenced data.
+If `offset` is provided, `size` must also be provided.
+
+##### etag
+
+An eTag value provided by the storage system (e.g., from S3 or Azure Blob Storage).
+Can be used to detect whether the referenced file has been updated. If the reference
+points to a byte range within a file, the eTag applies to the entire file.
+
+#### Validation
+
+* The `path` field is required and must be present. Readers must reject a `FILE`-annotated
+ group that does not contain `path`.
Review Comment:
As I posted in another comment we should make `path` optional and if it is missing it means the reference is to the same file. This has two advantages:
1. make already supported self-reference (path == "current-file") moveable. On a file rename the self-reference is still valid. Ergo a file with only self-references is trivially moveable.
2. it removes a non-representable state from the type system: non-existent `path` is not an error anymore, no need to check it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3606313683
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | invalid |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | - | set | - | invalid |
+| – | - | - | set | invalid |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+`size` must be set whenever `offset` is set, so any offset-based read always carries an
+explicit `size`. A self-reference (`path` not set) must set `offset`, and therefore also
+`size`. `size` may be omitted only for a whole-file external reference, where the range
+runs to the end of the referenced file.
+
+A self-reference points within the same Parquet file using `offset` and `size` (both
+required); the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is when `path` is not set, never an absolute path back to
+the current file, so a file containing self-references is renamed or relocated as a
+single unit.
+
+The bytes referenced by a self-reference are compressed with the same `CompressionCodec`
+as the one specified for the `inline` column.
+
+#### Validation
+
+* A value must resolve to some referenced data. It resolves only if `inline`, `path`, or
+ `offset` is set; if none of them are set, the value does not resolve and is invalid, even
+ if `size` is set. Use column nullability to represent a null value.
+* A self-reference (`path` not set) must set `offset`. A value with neither `path` nor
+ `offset` set (and not `inline`) does not resolve and is invalid.
+* `size` must be set whenever `offset` is set. A value that sets `offset` without `size`
+ is invalid. Because a self-reference must set `offset`, it must also set `size`.
+* If `inline` is set, it supplies the bytes; producers may instead treat `inline` and the
+ locator fields as mutually exclusive.
+* Field names within a `FILE`-annotated group must not be renamed.
+* Additional metadata about the file (e.g., modification timestamp) must
+ be stored adjacent to this group by engines or table formats, not inside it.
+
+Statistics may be collected for the individual fields of a `FILE`-annotated group
+according to the sort order of each field's logical type.
Review Comment:
it's just referring to the sort order defined in the logical type definition for each field. Tried to make this clearer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3608070444
##########
LogicalTypes.md:
##########
@@ -635,7 +635,183 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty[1]). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+[1] Implementations are not expected to treat empty strings as null
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type), as defined by RFC 2046, of the resolved bytes (e.g., `image/png`).
+When not set, the type can be assumed as `application/octet-stream`.
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:<digest>`, where `<digest>` is encoded according to the `Encoding`
+column below. Readers should ignore unknown algorithms. The recognized algorithms
+are:
+
+| Algorithm | Encoding | Notes |
+|-----------|---------------|----------------------------------------------------------|
+| `ETAG` | opaque | the object-store eTag, not recomputable |
+| `MD5` | lowercase hex | as defined in RFC 6151 represented as 32 hex characters |
+| `CRC32` | lowercase hex | as defined in RFC 3385, represented as 8 hex characters |
+| `CRC32C` | lowercase hex | as defined in RFC 9260, represented as 8 hex characters |
+| `SHA-256` | lowercase hex | as defined in RFC 6234, represented as 64 hex characters |
+
+`<digest>` encodings are:
+
+* `lowercase hex`: the digest bytes rendered as lowercase hexadecimal, two
+ characters per byte and no separators (e.g. `MD5:d41d8cd98f00b204e9800998ecf8427e`).
+* `opaque`: the token supplied verbatim by the object store, used only for
+ equality comparison and not otherwise interpreted.
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | invalid |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | - | set | - | invalid |
+| – | - | - | set | invalid |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
Review Comment:
nit: this table uses both dash (-) and em dash (–) to indicate null input. Let's just use dash?.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3621080717
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | invalid |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | - | set | - | invalid |
+| – | - | - | set | invalid |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+`size` must be set whenever `offset` is set, so any offset-based read always carries an
+explicit `size`. A self-reference (`path` not set) must set `offset`, and therefore also
+`size`. `size` may be omitted only for a whole-file external reference, where the range
+runs to the end of the referenced file.
+
+A self-reference points within the same Parquet file using `offset` and `size` (both
+required); the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is when `path` is not set, never an absolute path back to
+the current file, so a file containing self-references is renamed or relocated as a
+single unit.
+
+The bytes referenced by a self-reference are compressed with the same `CompressionCodec`
Review Comment:
Thanks @rdblue for the clarification! Now I'm clear with the self-reference case. I agree that we need to think more about the equivalent of `Content-Encoding` for external references.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on PR #585:
URL: https://github.com/apache/parquet-format/pull/585#issuecomment-5052106247
> @brkyvz can you reply on the [path thread](https://github.com/apache/parquet-format/pull/585#discussion_r3606837664)?
@rok sorry that was hidden in the UI. Responded! We weren't planning on renaming the field. Just explaining that it is a URI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on PR #585:
URL: https://github.com/apache/parquet-format/pull/585#issuecomment-5082722479
Vote passed: https://lists.apache.org/thread/lbtdvq63382fgl2049zb6vn396h9lmfz and all comments addressed. going to merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3501501339
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
it's not meant to be a URI. it can be anything, but everyone asked for an example, so I provided a URI example. It's up to the clients on what they store and how they read/consume it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "sfc-gh-nsharma (via GitHub)" <gi...@apache.org>.
sfc-gh-nsharma commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3501756573
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
If self-reference doesn't carry a requirement to preserve blocks, is there a huge benefit of choosing it over external references? With inline blobs, lifecycle of the the blob is tied with the file object. It seems like self-reference is giving the same lifecycle guarantee as the external reference. I can see that it makes the auth story simpler compared to external files but not if that is the main motivation for introducing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3504188572
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
+ * - size (INT64, optional): the length of the content in bytes; must be zero or positive
+ * - offset (INT64, optional): byte offset for range reads; if provided, size must also be provided
Review Comment:
Ok, so this should be stated more clearly instead of talking about "ranged reads" (an implementation detail).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3504195529
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
+ * - size (INT64, optional): the length of the content in bytes; must be zero or positive
+ * - offset (INT64, optional): byte offset for range reads; if provided, size must also be provided
+ * - etag (STRING, optional): eTag from the storage system for staleness detection
Review Comment:
Ok, but the problem is there are multiple pieces of data that can be of interest depending on the usage and the backend storage. So we might need something more sophisticated than a single STRING column for ETag (a MAP perhaps?).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3560194248
##########
LogicalTypes.md:
##########
@@ -635,7 +635,172 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+An opaque path string to an external file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied on top of the user-provided data. If `path` is
+not set, the value refers to the current file (a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set).
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | external `path`, `[offset, EOF)` |
Review Comment:
good catch, thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3573978844
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
Review Comment:
Nit: we should be clear that implementations are not expected to treat empty strings as null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3574059404
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | invalid |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | - | set | - | invalid |
+| – | - | - | set | invalid |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+`size` must be set whenever `offset` is set, so any offset-based read always carries an
+explicit `size`. A self-reference (`path` not set) must set `offset`, and therefore also
+`size`. `size` may be omitted only for a whole-file external reference, where the range
+runs to the end of the referenced file.
+
+A self-reference points within the same Parquet file using `offset` and `size` (both
+required); the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is when `path` is not set, never an absolute path back to
+the current file, so a file containing self-references is renamed or relocated as a
+single unit.
+
+The bytes referenced by a self-reference are compressed with the same `CompressionCodec`
+as the one specified for the `inline` column.
+
+#### Validation
+
+* A value must resolve to some referenced data. It resolves only if `inline`, `path`, or
+ `offset` is set; if none of them are set, the value does not resolve and is invalid, even
+ if `size` is set. Use column nullability to represent a null value.
+* A self-reference (`path` not set) must set `offset`. A value with neither `path` nor
+ `offset` set (and not `inline`) does not resolve and is invalid.
+* `size` must be set whenever `offset` is set. A value that sets `offset` without `size`
+ is invalid. Because a self-reference must set `offset`, it must also set `size`.
+* If `inline` is set, it supplies the bytes; producers may instead treat `inline` and the
+ locator fields as mutually exclusive.
Review Comment:
Readers may not because locator fields may be persisted to record provenance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3574050633
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | invalid |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | - | set | - | invalid |
+| – | - | - | set | invalid |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+`size` must be set whenever `offset` is set, so any offset-based read always carries an
+explicit `size`. A self-reference (`path` not set) must set `offset`, and therefore also
+`size`. `size` may be omitted only for a whole-file external reference, where the range
+runs to the end of the referenced file.
+
+A self-reference points within the same Parquet file using `offset` and `size` (both
+required); the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is when `path` is not set, never an absolute path back to
+the current file, so a file containing self-references is renamed or relocated as a
+single unit.
+
+The bytes referenced by a self-reference are compressed with the same `CompressionCodec`
+as the one specified for the `inline` column.
+
+#### Validation
+
+* A value must resolve to some referenced data. It resolves only if `inline`, `path`, or
+ `offset` is set; if none of them are set, the value does not resolve and is invalid, even
+ if `size` is set. Use column nullability to represent a null value.
Review Comment:
> Use column nullability to represent a null value
But empty string is considered unset?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3391753789
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
+#### Fields
+
+##### path
+
+An opaque path string to the referenced file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied. This is the only required field.
+
+##### size
+
+The length of the content in bytes. Must be zero or a positive integer if provided.
+A value of 0 indicates an empty file. If not provided, the length of the referenced
+content is unknown and the entirety of the content can be read.
+
+##### offset
+
+A byte offset indicating the start of a content slice within the referenced file.
+If not provided, readers must treat the value as 0.
+If provided and non-zero, readers must seek to this offset and read `size` bytes to retrieve the referenced data.
+If `offset` is provided, `size` must also be provided.
Review Comment:
Again, looking at the implementation in Rust, I don't see this enforced. What if the group definition is
```
optional group my_file (FILE) {
required binary path (STRING);
optional int64 offset;
}
```
I think we need to specify this at both the field and data levels, i.e. if the group contains an 'offset' field, it must also contain a 'size' field; if the optional 'offset' field is populated, it must be a non-negative integer, and a value for 'size' must also present.
Edit: I think the `MAP` description in this file is a good example of what I'm wanting here:
> - The key field encodes the map's key type. This field must have repetition required and must always be present. It must always be the first field of the repeated key_value group.
> - The value field encodes the map's value type and repetition. This field can be required, optional, or omitted. It must always be the second field of the repeated key_value group if present. If not present, it can be represented as a map with all null values or as a set of keys.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3546255649
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
Review Comment:
Made it case sensitive, based on the precedent with VARIANT, where the keys are case sensitive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3550729730
##########
LogicalTypes.md:
##########
@@ -635,7 +635,172 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+An opaque path string to an external file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied on top of the user-provided data. If `path` is
+not set, the value refers to the current file (a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set).
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | external `path`, `[offset, EOF)` |
Review Comment:
As per text below ("`size` must be set whenever `offset` is set, so any offset-based read always carries an explicit `size`") this should be invalid?
```suggestion
| – | set | set | - | external `path`, invalid |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3501510957
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
+#### Fields
+
+##### path
+
+An opaque path string to the referenced file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied. This is the only required field.
Review Comment:
no additional URI encoding on top of it. Essentially, what the client provides, it gets stored - there's no additional manipulation of the input
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "sfc-gh-sgrafberger (via GitHub)" <gi...@apache.org>.
sfc-gh-sgrafberger commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3462769363
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
Review Comment:
A bit of background information for people not very familiar with unstructured data and semantic query processing yet:
When thinking about real-world use of unstructured data, there are many domains where mixing different modalities within one table column is quite natural.
E.g., once you look at the law domain, you might have court cases with different kinds of evidence: videos, images, pdf scans of police reports, etc. Other columns in the table might then be columns like "case_id", "case_name", "judge_id", "file_source", ... Other domain examples would be patient records in healthcare that mix medical imaging, pdf scans of written medical reports, prescriptions, etc. Or think about the kind of unstructured data a reporter might be working with.
DB users couldn't really do large-scale automated analysis of unstructured data before the rise of LLMs, and are now discovering step-by-step what is possible. Being able to efficiently answer basic questions like what kind of unstructured data you are dealing with sounds like a natural requirement to me to support currently emerging workloads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "sfc-gh-nsharma (via GitHub)" <gi...@apache.org>.
sfc-gh-nsharma commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3404593056
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
+#### Fields
+
+##### path
+
+An opaque path string to the referenced file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied. This is the only required field.
+
+##### size
+
+The length of the content in bytes. Must be zero or a positive integer if provided.
+A value of 0 indicates an empty file. If not provided, the length of the referenced
+content is unknown and the entirety of the content can be read.
+
+##### offset
+
+A byte offset indicating the start of a content slice within the referenced file.
+If not provided, readers must treat the value as 0.
+If provided and non-zero, readers must seek to this offset and read `size` bytes to retrieve the referenced data.
+If `offset` is provided, `size` must also be provided.
+
+##### etag
+
+An eTag value provided by the storage system (e.g., from S3 or Azure Blob Storage).
+Can be used to detect whether the referenced file has been updated. If the reference
+points to a byte range within a file, the eTag applies to the entire file.
+
+#### Validation
+
+* The `path` field is required and must be present. Readers must reject a `FILE`-annotated
+ group that does not contain `path`.
+* If `offset` is present and non-zero, `size` must also be provided.
+* Additional metadata about the file (e.g., content type, modification timestamp) should
Review Comment:
How are we thinking about various engines adding very common fields like content type, modification timestamp? If they are not part of the parquet file format, I suspect there will be lot of non-interoperable implementations with those fields. Do we expect this file type to evolve to support those in the future?
e.g. these two definitions won't be interoperable because they wrap the FILE struct differently
```
FileEngine1 (outer struct) {
file_ref FILE
content_type STRING
last_modified TIMESTAMP
}
FileEngine2 (outer struct) {
fileRef FILE
mimeType STRING
lastModified TIMESTAMP
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on PR #585:
URL: https://github.com/apache/parquet-format/pull/585#issuecomment-4937103509
Addressed your comments, @rok @pitrou please take a look. There's only one unaddressed comment on the CompressionCodec, which I will address after talking to @alkis
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3547224471
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
I think [Rfc for URI](https://datatracker.ietf.org/doc/html/rfc3986) covers all cases mentioned.
We could specify something like
> A URI-reference as defined by RFC 3986, encoded as a Parquet STRING. If absolute, it identifies an external resource. If relative, readers must resolve it against the URI of the Parquet file containing this value using RFC 3986 section 5. If `uri` is not set, the value refers to the current Parquet file.
>
> Readers may support an implementation-defined set of URI schemes. If the scheme is unsupported or the base URI is unavailable for a relative reference, the value is not resolvable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3390313092
##########
LogicalTypes.md:
##########
@@ -635,7 +635,72 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
+#### path
+
+An opaque path string to the referenced file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied. This is the only required field.
+
+#### size
+
+The length of the content in bytes. Must be zero or a positive integer if provided.
+A value of 0 indicates an empty file.
+
+#### offset
+
+A byte offset for range reads. If not provided, readers must treat the value as 0.
Review Comment:
This isn't actually "for range reads". It's to indicate that the content referenced content is a slice of the referenced file. The term "range read" implies that it's still the full file but the act of reading it just uses a particular range. I find the terminology a little strange.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3399131082
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field | Type | Required |
+|----------|--------|----------|
+| `path` | STRING | Yes |
+| `size` | INT64 | No |
+| `offset` | INT64 | No |
+| `etag` | STRING | No |
+
+#### Fields
+
+##### path
+
+An opaque path string to the referenced file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied. This is the only required field.
+
+##### size
+
+The length of the content in bytes. Must be zero or a positive integer if provided.
+A value of 0 indicates an empty file. If not provided, the length of the referenced
+content is unknown and the entirety of the content can be read.
+
+##### offset
+
+A byte offset indicating the start of a content slice within the referenced file.
+If not provided, readers must treat the value as 0.
+If provided and non-zero, readers must seek to this offset and read `size` bytes to retrieve the referenced data.
+If `offset` is provided, `size` must also be provided.
+
+##### etag
+
+An eTag value provided by the storage system (e.g., from S3 or Azure Blob Storage).
+Can be used to detect whether the referenced file has been updated. If the reference
+points to a byte range within a file, the eTag applies to the entire file.
+
+#### Validation
+
+* The `path` field is required and must be present. Readers must reject a `FILE`-annotated
+ group that does not contain `path`.
+* If `offset` is present and non-zero, `size` must also be provided.
+* Additional metadata about the file (e.g., content type, modification timestamp) should
+ be stored adjacent to this struct by engines or table formats, not inside it.
+
+Statistics may be collected for the individual fields of a `FILE`-annotated group
+according to the sort order of each field's logical type.
+
+This is an example `FILE`-annotated group in Parquet:
+
+```
+optional group my_file (FILE) {
+ required binary path (STRING);
+ optional int64 size;
+ optional int64 offset;
+ optional binary etag (STRING);
+}
+```
+
+*Compatibility*
Review Comment:
I wouldn't include this. Seems unnecessary to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3399080409
##########
LogicalTypes.md:
##########
@@ -635,7 +635,76 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to an external file, along with
+the minimum metadata required to read it. It is intended for use cases such as
+storing file inventories, manifests, and unstructured data references (e.g., images
+or audio files stored in object storage).
+
+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
Review Comment:
I think it should be that the struct "must contain the following fields". That avoids the confusing meaning of "required".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "alkis (via GitHub)" <gi...@apache.org>.
alkis commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3527075524
##########
LogicalTypes.md:
##########
@@ -635,7 +635,133 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within this file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name. Field IDs
+may also be used for projection. All fields are optional:
+
+| Field | Type | Required |
+|----------------|------------|----------|
+| `path` | STRING | No |
+| `offset` | INT64 | No |
+| `size` | INT64 | No |
+| `content_type` | STRING | No |
+| `checksum` | STRING | No |
+| `inline` | BYTE_ARRAY | No |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+##### path
+
+An opaque path string to an external file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied on top of the user provided data. If `path` is absent,
+the value refers to this file (a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not provided, readers must treat the value as 0.
+If provided and non-zero, readers must seek to this offset to retrieve the referenced data.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if provided.
+A value of 0 indicates empty referenced data. If not provided, the range runs to the end
+of the referenced data.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. It generalizes the storage-system eTag. The
+recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are present are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|----------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | – | external `path`, `[offset, EOF)` |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | – | set | – | this file, `[offset, EOF)` (self-reference) |
Review Comment:
This would be invalid with the new requirements for `size`.
##########
LogicalTypes.md:
##########
@@ -635,7 +635,133 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within this file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name. Field IDs
+may also be used for projection. All fields are optional:
+
+| Field | Type | Required |
+|----------------|------------|----------|
+| `path` | STRING | No |
+| `offset` | INT64 | No |
+| `size` | INT64 | No |
+| `content_type` | STRING | No |
+| `checksum` | STRING | No |
+| `inline` | BYTE_ARRAY | No |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+##### path
+
+An opaque path string to an external file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied on top of the user provided data. If `path` is absent,
+the value refers to this file (a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not provided, readers must treat the value as 0.
+If provided and non-zero, readers must seek to this offset to retrieve the referenced data.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if provided.
+A value of 0 indicates empty referenced data. If not provided, the range runs to the end
+of the referenced data.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. It generalizes the storage-system eTag. The
+recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are present are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|----------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | – | external `path`, `[offset, EOF)` |
Review Comment:
This would be invalid with the new requirements for `size`.
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,33 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to a file, or to a range of
+ * bytes that may be stored inline, elsewhere in this file, or in an external
+ * file. All fields are optional and are identified by name:
+ * - path (STRING): an opaque location string (e.g. s3://bucket/file.jpg);
+ * if absent, the value refers to this file (self-reference)
+ * - offset (INT64): start of the byte range; if absent, treated as 0
+ * - size (INT64): byte length of the referenced data; if absent, the range runs to
+ * the end of the referenced data
Review Comment:
This was brought up at the design doc. Self-references with omitted size are virtually always wrong, so the only gain we have from this rule is for the last chunk of packed references (external with offset). I suggest we harden this slightly to:
```suggestion
* - size (INT64): byte length of the referenced data; required when `path` or `offset` are defined
```
##########
LogicalTypes.md:
##########
@@ -635,7 +635,133 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within this file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name. Field IDs
+may also be used for projection. All fields are optional:
+
+| Field | Type | Required |
+|----------------|------------|----------|
+| `path` | STRING | No |
+| `offset` | INT64 | No |
+| `size` | INT64 | No |
+| `content_type` | STRING | No |
+| `checksum` | STRING | No |
+| `inline` | BYTE_ARRAY | No |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+##### path
+
+An opaque path string to an external file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied on top of the user provided data. If `path` is absent,
+the value refers to this file (a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not provided, readers must treat the value as 0.
+If provided and non-zero, readers must seek to this offset to retrieve the referenced data.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if provided.
+A value of 0 indicates empty referenced data. If not provided, the range runs to the end
Review Comment:
```suggestion
The byte length of the referenced data. Must be zero or a positive integer if provided. It must be set if `path` or `offset` are set.
```
##########
LogicalTypes.md:
##########
@@ -635,7 +635,133 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within this file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name. Field IDs
+may also be used for projection. All fields are optional:
+
+| Field | Type | Required |
+|----------------|------------|----------|
+| `path` | STRING | No |
+| `offset` | INT64 | No |
+| `size` | INT64 | No |
+| `content_type` | STRING | No |
+| `checksum` | STRING | No |
+| `inline` | BYTE_ARRAY | No |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+##### path
+
+An opaque path string to an external file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied on top of the user provided data. If `path` is absent,
+the value refers to this file (a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not provided, readers must treat the value as 0.
+If provided and non-zero, readers must seek to this offset to retrieve the referenced data.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if provided.
+A value of 0 indicates empty referenced data. If not provided, the range runs to the end
+of the referenced data.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. It generalizes the storage-system eTag. The
+recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are present are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|----------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | – | external `path`, `[offset, EOF)` |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | – | set | – | this file, `[offset, EOF)` (self-reference) |
+| – | – | – | set | this file, `[0, size)` (self-reference) |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+A self-reference typically points within the same Parquet file using `offset` and
+`size`; the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is the absence of `path`, never an absolute path back to
+the current file, so a file containing self-references is renamed or relocated as a
+single unit.
+
+The referenced bytes are compressed with the same `CompressionCodec` as the one
+specified for the `inline` column.
Review Comment:
Let's make this clearer.
```suggestion
The bytes referenced by a self-reference are compressed with the same `CompressionCodec` as the one specified for the `inline` column.
```
##########
LogicalTypes.md:
##########
@@ -635,7 +635,133 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within this file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name. Field IDs
+may also be used for projection. All fields are optional:
+
+| Field | Type | Required |
+|----------------|------------|----------|
+| `path` | STRING | No |
+| `offset` | INT64 | No |
+| `size` | INT64 | No |
+| `content_type` | STRING | No |
+| `checksum` | STRING | No |
+| `inline` | BYTE_ARRAY | No |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+##### path
+
+An opaque path string to an external file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied on top of the user provided data. If `path` is absent,
+the value refers to this file (a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not provided, readers must treat the value as 0.
+If provided and non-zero, readers must seek to this offset to retrieve the referenced data.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if provided.
+A value of 0 indicates empty referenced data. If not provided, the range runs to the end
+of the referenced data.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. It generalizes the storage-system eTag. The
+recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are present are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|----------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
Review Comment:
This would be invalid with the new requirements for `size`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "sfc-gh-nsharma (via GitHub)" <gi...@apache.org>.
sfc-gh-nsharma commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3547286134
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
I think it should be left to engines. e.g iceberg could choose to resolve relative paths against table's base location. Other engines might do it differently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3540731616
##########
LogicalTypes.md:
##########
@@ -635,7 +635,164 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name. Field IDs
+may also be used for projection. Every field is optional both in the schema and in the
+data: a writer may omit any field from the group definition, and any field that is
+present has a field repetition type of `OPTIONAL`. A group need only define the fields
+it uses (for example, an inline-only group may define just `inline`, and an external
+reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+An opaque path string to an external file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied on top of the user-provided data. If `path` is
+not set, the value refers to the current file (a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set
+or `path` is not set. It may be omitted only for a whole-file external reference (`path`
+set, `offset` not set), in which case the range runs to the end of the referenced file.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. It generalizes the storage-system eTag. The
Review Comment:
```suggestion
`<algorithm>:base64(<digest bytes>)`. The
```
nit: not sure if this is relevant given the table below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3547061935
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
but we don't force a special format on what users store in their string columns.
I'm sorry if I'm misunderstanding what you're trying to suggest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3582740017
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
Review Comment:
nit: What does the notes mean here? Shouldn't we just link to algorithm definitions like RFC 6234 or whatnot?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3580712295
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | invalid |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | - | set | - | invalid |
+| – | - | - | set | invalid |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+`size` must be set whenever `offset` is set, so any offset-based read always carries an
+explicit `size`. A self-reference (`path` not set) must set `offset`, and therefore also
+`size`. `size` may be omitted only for a whole-file external reference, where the range
+runs to the end of the referenced file.
+
+A self-reference points within the same Parquet file using `offset` and `size` (both
+required); the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is when `path` is not set, never an absolute path back to
+the current file, so a file containing self-references is renamed or relocated as a
+single unit.
+
+The bytes referenced by a self-reference are compressed with the same `CompressionCodec`
Review Comment:
The does not look right. Same column can have different codecs across different row groups. V2 pages can even decide whether to compress individually. Even if all row groups on the same column use the same codec, how can we tell whether self-referenced data is compressed or not?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3504185481
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
Disagreed. The standard should say how the field is to be interpreted. If there is no standard interpretation then this is not a proper spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3640932555
##########
src/main/thrift/parquet.thrift:
##########
Review Comment:
```suggestion
* GEOGRAPHY - undefined
* FILE - undefined
```
##########
LogicalTypes.md:
##########
@@ -635,7 +635,182 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `uri`).
+
+| Field | Type |
+|----------------|------------|
+| `uri` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `uri` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty[1]). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+[1] Implementations are not expected to treat empty strings as null
+
+##### uri
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `uri` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`uri` not set); it is optional for an
+external reference (`uri` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`uri` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type), as defined by RFC 2046, of the resolved bytes (e.g., `image/png`).
+When not set, the type can be assumed as `application/octet-stream`.
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:<digest>`, where `<digest>` is encoded according to the `Encoding`
+column below. Readers should ignore unknown algorithms. The recognized algorithms
+are:
+
+| Algorithm | Encoding | Notes |
+|-----------|---------------|----------------------------------------------------------|
+| `ETAG` | opaque | the object-store eTag, not recomputable |
+| `MD5` | lowercase hex | as defined in RFC 1321 represented as 32 hex characters |
+| `CRC32` | lowercase hex | as defined in RFC 2083, represented as 8 hex characters |
+| `CRC32C` | lowercase hex | as defined in RFC 3385, represented as 8 hex characters |
+| `SHA-256` | lowercase hex | as defined in RFC 6234, represented as 64 hex characters |
+
+`<digest>` encodings are:
+
+* `lowercase hex`: the digest bytes rendered as lowercase hexadecimal, two
+ characters per byte and no separators (e.g. `MD5:d41d8cd98f00b204e9800998ecf8427e`).
+* `opaque`: the token supplied verbatim by the object store, used only for
+ equality comparison and not otherwise interpreted.
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `uri`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`uri`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `uri`, `offset`, and `size` are
+set:
+
+| `inline` | `uri` | `offset` | `size` | Resolves to |
+|----------|-------|----------|--------|-------------------------------------------------------|
+| set | - | - | - | the inline bytes |
+| - | set | - | - | whole external file at `uri` |
+| - | set | set | - | invalid |
+| - | set | - | set | external `uri`, `[0, size)` |
+| - | set | set | set | external `uri`, `[offset, offset + size)` |
+| - | - | set | - | invalid |
+| - | - | - | set | invalid |
+| - | - | set | set | this file, `[offset, offset + size)` (self-reference) |
+| - | - | - | - | nothing - invalid |
+
+`size` must be set whenever `offset` is set, so any offset-based read always carries an
+explicit `size`. A self-reference (`uri` not set) must set `offset`, and therefore also
+`size`. `size` may be omitted only for a whole-file external reference, where the range
+runs to the end of the referenced file.
+
+A self-reference points within the same Parquet file using `offset` and `size` (both
+required). A self-reference is when `uri` is not set. A file containing self-references
+can be renamed or relocated as a single unit.
+
Review Comment:
```suggestion
can be renamed or relocated as a single unit.
Parquet files containing self-references must not use Parquet modular encryption.
Self-referenced byte ranges are not Parquet encryption modules and therefore cannot
be encrypted or authenticated independently. Encryption of external files referenced
by `uri` is outside the scope of the Parquet format.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on PR #585:
URL: https://github.com/apache/parquet-format/pull/585#issuecomment-4910477457
@etseidl @sfc-gh-sgrafberger @alkis @rdblue addressed your comments, can you please take another look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3555463492
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Review Comment:
@pitrou I think it's reasonable to clarify that we expect the location to be a URI, but are you saying that implementations must parse and validate the contents of the column? I don't think that we enforce things like that anywhere else. But if I understand correctly, you're just saying that the field definition should state that the value in the field is a URI and would not require expensive validation. Is that right?
I don't want to set a precedent that logical types can require per-value validation. That would mean a lot of work in the write path for little gain. Validating a URI would require parsing it because the parts have different sets of valid characters and escape requirements (`_` is not allowed in a scheme, nor are escaped characters).
In the more general case of requiring per-value validation, we could introduce negatives for forward compatibility. Avro lets you define enums that are translated into ordinals when stored. This ended up being surprisingly restrictive because adding a symbol breaks forward compatibility. In most cases, readers using old schemas will automatically project and ignore new columns. Using a string works without breaking the reader, but adding an enum symbol causes a failure to resolve the read schema. Something similar could easily happen if we require per-value validation: even widening that validation is a forward-incompatible change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "brkyvz (via GitHub)" <gi...@apache.org>.
brkyvz commented on PR #585:
URL: https://github.com/apache/parquet-format/pull/585#issuecomment-5007767098
@rdblue @wgtmac @mapleFU @RussellSpitzer Thank you for the feedback! Addressed your comments. Hopefully things are clearer now! Let me know if they aren't
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3606132563
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | invalid |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | - | set | - | invalid |
+| – | - | - | set | invalid |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+`size` must be set whenever `offset` is set, so any offset-based read always carries an
+explicit `size`. A self-reference (`path` not set) must set `offset`, and therefore also
+`size`. `size` may be omitted only for a whole-file external reference, where the range
+runs to the end of the referenced file.
+
+A self-reference points within the same Parquet file using `offset` and `size` (both
+required); the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is when `path` is not set, never an absolute path back to
+the current file, so a file containing self-references is renamed or relocated as a
+single unit.
+
+The bytes referenced by a self-reference are compressed with the same `CompressionCodec`
+as the one specified for the `inline` column.
+
+#### Validation
+
+* A value must resolve to some referenced data. It resolves only if `inline`, `path`, or
+ `offset` is set; if none of them are set, the value does not resolve and is invalid, even
+ if `size` is set. Use column nullability to represent a null value.
+* A self-reference (`path` not set) must set `offset`. A value with neither `path` nor
+ `offset` set (and not `inline`) does not resolve and is invalid.
+* `size` must be set whenever `offset` is set. A value that sets `offset` without `size`
+ is invalid. Because a self-reference must set `offset`, it must also set `size`.
Review Comment:
A "must" requirement is sufficient for writers. I think that requirement implies a failure at write time because otherwise the requirements are violated.
I'm more concerned about read time, which is why I'm asking. If this is invalid, what is the expected reader behavior? In the `offset=null` case, `size` is understood to be the length of the file. That doesn't seem right, but my guess is that readers will produce bytes up to the end of the file unless you specifically state that this should fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3606900148
##########
LogicalTypes.md:
##########
@@ -635,7 +635,183 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty[1]). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+[1] Implementations are not expected to treat empty strings as null
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type), as defined by RFC 2046, of the resolved bytes (e.g., `image/png`).
+When not set, the type can be assumed as `application/octet-stream`.
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:<digest>`, where `<digest>` is encoded according to the `Encoding`
+column below. Readers should ignore unknown algorithms. The recognized algorithms
+are:
+
+| Algorithm | Encoding | Notes |
+|-----------|---------------|----------------------------------------------------------|
+| `ETAG` | opaque | the object-store eTag, not recomputable |
+| `MD5` | lowercase hex | as defined in RFC 6151 represented as 32 hex characters |
Review Comment:
I would propose we reference [RFC 1321](https://datatracker.ietf.org/doc/html/rfc1321) (The MD5 Message-Digest Algorithm) over [RFC 6151](https://datatracker.ietf.org/doc/html/rfc6151) (Updated Security Considerations for the MD5 Message-Digest and the HMAC-MD5 Algorithms).
```suggestion
| `MD5` | lowercase hex | as defined in RFC 1321 represented as 32 hex characters |
```
##########
LogicalTypes.md:
##########
@@ -635,7 +635,183 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty[1]). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+[1] Implementations are not expected to treat empty strings as null
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type), as defined by RFC 2046, of the resolved bytes (e.g., `image/png`).
+When not set, the type can be assumed as `application/octet-stream`.
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:<digest>`, where `<digest>` is encoded according to the `Encoding`
+column below. Readers should ignore unknown algorithms. The recognized algorithms
+are:
+
+| Algorithm | Encoding | Notes |
+|-----------|---------------|----------------------------------------------------------|
+| `ETAG` | opaque | the object-store eTag, not recomputable |
+| `MD5` | lowercase hex | as defined in RFC 6151 represented as 32 hex characters |
+| `CRC32` | lowercase hex | as defined in RFC 3385, represented as 8 hex characters |
+| `CRC32C` | lowercase hex | as defined in RFC 9260, represented as 8 hex characters |
Review Comment:
These might fit better - I am in no way an expert on CRC32(c) [RFC 2083](https://datatracker.ietf.org/doc/html/rfc2083#page-15), [RFC 3385](https://datatracker.ietf.org/doc/html/rfc3385#section-5.1), but we would want to specify exact alghoritms used.
```suggestion
| `CRC32` | lowercase hex | as defined in RFC 2083, represented as 8 hex characters |
| `CRC32C` | lowercase hex | as defined in RFC 3385, represented as 8 hex characters |
```
##########
LogicalTypes.md:
##########
@@ -635,7 +635,183 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty[1]). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+[1] Implementations are not expected to treat empty strings as null
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type), as defined by RFC 2046, of the resolved bytes (e.g., `image/png`).
+When not set, the type can be assumed as `application/octet-stream`.
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:<digest>`, where `<digest>` is encoded according to the `Encoding`
+column below. Readers should ignore unknown algorithms. The recognized algorithms
+are:
+
+| Algorithm | Encoding | Notes |
+|-----------|---------------|----------------------------------------------------------|
+| `ETAG` | opaque | the object-store eTag, not recomputable |
+| `MD5` | lowercase hex | as defined in RFC 6151 represented as 32 hex characters |
+| `CRC32` | lowercase hex | as defined in RFC 3385, represented as 8 hex characters |
+| `CRC32C` | lowercase hex | as defined in RFC 9260, represented as 8 hex characters |
+| `SHA-256` | lowercase hex | as defined in RFC 6234, represented as 64 hex characters |
+
+`<digest>` encodings are:
+
+* `lowercase hex`: the digest bytes rendered as lowercase hexadecimal, two
+ characters per byte and no separators (e.g. `MD5:d41d8cd98f00b204e9800998ecf8427e`).
+* `opaque`: the token supplied verbatim by the object store, used only for
+ equality comparison and not otherwise interpreted.
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | invalid |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | - | set | - | invalid |
+| – | - | - | set | invalid |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+`size` must be set whenever `offset` is set, so any offset-based read always carries an
+explicit `size`. A self-reference (`path` not set) must set `offset`, and therefore also
+`size`. `size` may be omitted only for a whole-file external reference, where the range
+runs to the end of the referenced file.
+
+A self-reference points within the same Parquet file using `offset` and `size` (both
+required). A self-reference is when `path` is not set. A file containing self-references
+can be renamed or relocated as a single unit.
+
+The bytes referenced by a self-reference use the `CompressionCodec`
+defined by the `inline` column chunk's `ColumnMetadata`.
+
+#### Validation
+
+* A value must resolve to some referenced data. It resolves only if `inline`, `path`, or
+ `offset` is set; if none of them are set, the value does not resolve and is invalid, even
+ if `size` is set.
+* A self-reference (`path` not set) must set `offset`. A value with neither `path` nor
+ `offset` set (and not `inline`) does not resolve and is invalid.
+* `size` must be set whenever `offset` is set. A value that sets `offset` without `size`
+ is invalid. Because a self-reference must set `offset`, it must also set `size`.
+* If `inline` is set, it supplies the bytes readers; producers may treat `inline` and the
Review Comment:
```suggestion
* If `inline` is set, it supplies the bytes; producers may treat `inline` and the
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Introduces a new LogicalType: FILE [parquet-format]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3610124773
##########
LogicalTypes.md:
##########
@@ -635,7 +635,173 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection. Every field
+is optional both in the schema and in the data: a writer may omit any field from the group
+definition, and any field that is present has a field repetition type of `OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group may define
+just `inline`, and an external reference may define just `path`).
+
+| Field | Type |
+|----------------|------------|
+| `path` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top
+of the user-provided data. If `path` is not set, the value refers to the current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set.
+It may be omitted only for a whole-file external reference (`path` set, `offset` not set),
+in which case the range runs to the end of the referenced file. Because a self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. Readers should ignore unknown
+algorithms. The recognized algorithms are:
+
+| Algorithm | Notes |
+|-----------|-----------------------------------------------------------------|
+| `ETAG` | the object-store eTag — equality-only, not recomputable |
+| `MD5` | the usual S3/HTTP eTag and Content-MD5 |
+| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) |
+| `CRC32C` | common in object stores, hardware-accelerated |
+| `SHA-256` | e.g. S3 additional checksums |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set | – | – | – | the inline bytes |
+| – | set | – | – | whole external file at `path` |
+| – | set | set | - | invalid |
+| – | set | – | set | external `path`, `[0, size)` |
+| – | set | set | set | external `path`, `[offset, offset + size)` |
+| – | - | set | - | invalid |
+| – | - | - | set | invalid |
+| – | – | set | set | this file, `[offset, offset + size)` (self-reference) |
+| – | – | – | – | nothing — invalid |
+
+`size` must be set whenever `offset` is set, so any offset-based read always carries an
+explicit `size`. A self-reference (`path` not set) must set `offset`, and therefore also
+`size`. `size` may be omitted only for a whole-file external reference, where the range
+runs to the end of the referenced file.
+
+A self-reference points within the same Parquet file using `offset` and `size` (both
+required); the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is when `path` is not set, never an absolute path back to
+the current file, so a file containing self-references is renamed or relocated as a
+single unit.
+
+The bytes referenced by a self-reference are compressed with the same `CompressionCodec`
+as the one specified for the `inline` column.
+
+#### Validation
+
+* A value must resolve to some referenced data. It resolves only if `inline`, `path`, or
+ `offset` is set; if none of them are set, the value does not resolve and is invalid, even
+ if `size` is set. Use column nullability to represent a null value.
+* A self-reference (`path` not set) must set `offset`. A value with neither `path` nor
+ `offset` set (and not `inline`) does not resolve and is invalid.
+* `size` must be set whenever `offset` is set. A value that sets `offset` without `size`
+ is invalid. Because a self-reference must set `offset`, it must also set `size`.
+* If `inline` is set, it supplies the bytes; producers may instead treat `inline` and the
+ locator fields as mutually exclusive.
+* Field names within a `FILE`-annotated group must not be renamed.
+* Additional metadata about the file (e.g., modification timestamp) must
+ be stored adjacent to this group by engines or table formats, not inside it.
+
+Statistics may be collected for the individual fields of a `FILE`-annotated group
+according to the sort order of each field's logical type.
+
+This is an example of a `FILE`-annotated group that defines all fields:
+
+```
+optional group my_file (FILE) {
+ optional binary path (STRING);
+ optional int64 offset;
+ optional int64 size;
+ optional binary content_type (STRING);
+ optional binary checksum (STRING);
+ optional binary inline;
+}
+```
+
+Because every field is optional, a group need only define the fields it uses. A group
+whose values are always stored inline may define just `inline`:
Review Comment:
I don't see this is addressed. At least `inline_file` struct below still defines `content_type`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org