Re: [PR] PARQUET-2492: Add extension points for all thrift messages [parquet-format]

["alkis (via GitHub)" <gi...@apache.org>]

alkis commented on code in PR #254:
URL: https://github.com/apache/parquet-format/pull/254#discussion_r1629831010


##########
README.md:
##########
@@ -285,6 +285,61 @@ There are many places in the format for compatible extensions:
 - Encodings: Encodings are specified by enum and more can be added in the future.
 - Page types: Additional page types can be added and safely skipped.
 
+### Thrift extensions
+Thrift is used for metadata. The Thrift spec mandates that unknown fields are
+skipped. To facilitate extensions Parquet reserves field-id `32767` of *every*
+struct as an ignorable extension point. More specifically Parquet guarantees
+that field-id `32767` will *never* be seen in the official Thrift IDL. The type
+of this field is always `binary`. 
+
+Such extensions can easily be appended to an existing Thrift serialized message
+without any special APIs. Sample `C++` implementation is provided:
+
+```c++
+std::string AppendExtension(std::string thrift, std::string ext) {
+  auto append_uleb = [](uint32_t x, std::string* out) {
+    while (true) {
+      int c = x & 0x7F;
+      if ((x >>= 7) == 0) {
+        out->push_back(c);
+        return;
+      } else {
+        out->push_back(c | 0x80);
+      }
+    }
+  };
+  thrift.pop_back();  // remove the trailing 0
+  thrift += "\x08\xFF\xFF\x01";
+  append_uleb(ext.size(), &thrift);
+  thrift += ext;
+  thrift += "\x00";  // add the trailing 0 back
+  return thrift;
+}
+```
+
+Additionally the binary extension MUST have a specific form in order to be
+unambiguously identifiable by parsers that know of it and, as a corollary,
+impossible to be accidentally generated by user data.
+
+```
+N bytes: the extension data

Review Comment:
   After thinking some more, I simplified the specification further.
   
   I removed all the fields from the encoding itself except the magic 3 byte disambiguator.
   
   The rest are moved into an example section that describes how one might design a `FileMetaData` replacement.
   
   The extension mechanism is very flexible, one might extend `ColumnMetaData` with some extra information and in that design the extension may be very different than the `FileMetaData` replacement.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org